На этой неделе мы будем выкладывать видео с Moscow Python Conf Online, которая прошла в марте этого года.
Первое видео: Mastering a data pipeline with Python / Robson Luis Monteiro Junior (Microsoft)
Building data pipelines are a consolidated task, there are a vast number of tools that automate and help developers to create data pipelines with few clicks on the cloud. It might solve non-complex or well-defined standard problems. This presentation is a demystification of years of experience and painful mistakes using Python as a core to create reliable data pipelines and manage insanely amount of valuable data. Let's cover how each piece fits into this puzzle: data acquisition, ingestion, transformation, storage, workflow management and serving. Also, we'll walk through best practices and possible issues. We'll cover PySpark vs Dask and Pandas, Airflow, and Apache Arrow as a new approach.
https://www.youtube.com/watch?v=25fUlUsmg38
Первое видео: Mastering a data pipeline with Python / Robson Luis Monteiro Junior (Microsoft)
Building data pipelines are a consolidated task, there are a vast number of tools that automate and help developers to create data pipelines with few clicks on the cloud. It might solve non-complex or well-defined standard problems. This presentation is a demystification of years of experience and painful mistakes using Python as a core to create reliable data pipelines and manage insanely amount of valuable data. Let's cover how each piece fits into this puzzle: data acquisition, ingestion, transformation, storage, workflow management and serving. Also, we'll walk through best practices and possible issues. We'll cover PySpark vs Dask and Pandas, Airflow, and Apache Arrow as a new approach.
https://www.youtube.com/watch?v=25fUlUsmg38
Comments
Be the first to add a comment
Log in to comment