This intensive three-day workshop is designed to equip professionals with the skills to construct and refine high-performance data-processing workloads leveraging PySpark, Pandas, and Polars within Kubernetes-based ecosystems.
Attendees will gain a functional understanding of how Spark applications operate on Kubernetes, focusing on how configuration choices at the application level directly impact performance, scalability, resource utilisation, and operational costs. The curriculum addresses critical optimisation domains, such as executor sizing, memory allocation, dynamic allocation, partitioning strategies, shuffle mechanics, the impact of small-file issues, and efficient Parquet handling.
Additionally, the course tackles frequent challenges associated with Pandas, such as memory constraints and out-of-memory errors, while introducing Polars as a high-performance alternative for specific data-processing tasks. Through practical exercises, participants will learn to diagnose performance and memory bottlenecks, evaluate various configuration approaches, and implement optimisation strategies for realistic ETL and machine learning use cases.
The core focus of the course remains on practical decision-making: mastering the ability to pinpoint bottlenecks, select the most appropriate tooling, configure Spark effectively, and achieve a balance between performance and infrastructure resource consumption and cost.
Read more...
