Поэтому моя роль как data engineer, на основе информации выше, разбираться с этим, чтобы на выходе я мог написать что-то вроде (взял у Facebook data engineer и немного изменил):
- Managed a 10 PB+ data platform
- Consolidated and conformed company-wide growth metrics (across Amazon Events and marketing efforts) into a single, company-wide view.
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x.
- Reduced core notification data set latencies from 36 hours to < 8 hours.
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.
+ надо обязательно упомянуть Privacy (GDPR, и все другие вещи, про удаление клиентских данных и compliance)
- Managed a 10 PB+ data platform
- Consolidated and conformed company-wide growth metrics (across Amazon Events and marketing efforts) into a single, company-wide view.
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x.
- Reduced core notification data set latencies from 36 hours to < 8 hours.
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.
+ надо обязательно упомянуть Privacy (GDPR, и все другие вещи, про удаление клиентских данных и compliance)
1 comment
Log in to comment
А можете привести примеры (или может даже подробно описать принципы) того, как там ставят задачи для DE ?