Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Foundations of Big Data and Spark

• Overview of the Big Data ecosystem and Spark's role in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining when to apply each approach
• Creating and configuring SparkSession, along with a grasp of application configuration fundamentals

Module 2: PySpark DataFrames

• Reading from and writing to enterprise data sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations like window functions, timestamp handling, and processing nested data
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes and Docker Basics for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods regarding Spark driver and executor processes—a conceptual view for cluster users rather than administrators
• What happens when a Spark job runs on Kubernetes: pod scheduling, resource requests and limits, and mapping session settings to these resources
• Distinguishing between what you control via Spark session parameters and what the platform team manages through cluster provisioning—and why this boundary is critical for tuning
• Practical exercise: Examine a running Spark-on-Kubernetes job in the sandbox cluster and identify the driver and executor pods

Module 4: In-Depth Spark Session Configuration on Kubernetes

• Executor count versus size: analysing memory-versus-cores trade-offs and reasoning through them
• Memory allocation for drivers and executors, memory overhead, and how these translate into pod resources
• Dynamic allocation: behaviour on Kubernetes, scenarios where it conserves resources, and where it may not
• Practical exercise: Execute the same job under varying executor and core configurations and compare runtime and resource usage

Module 5: Scaling Behaviour and Cost Optimisation

• How adding or removing nodes affects job behaviour, completion time, and resource consumption
• Comparing performance and cost trade-offs between many small executors and fewer large ones
• Shuffle behaviour and tuning shuffle partitions, including estimating the cost impact of different configuration choices
• Practical exercise: Scale the sandbox cluster from five to ten nodes and monitor the impact on job completion time and resource consumption

Module 6: Efficient Data Ingestion and Partitioning

• The small-files problem: why sources split into numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing
• Controlling partition sizes during both reading and writing
• Writing Parquet efficiently to prevent reintroducing the small-files problem downstream
• Practical exercise: Load a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and compare performance before and after optimisation

Module 7: Pandas Memory Management and Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognising their symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory blow-ups when writing large datasets to CSV
• Utilising chunked processing and dtype optimisation for resource-constrained environments
• Practical exercise: Replicate a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation, and memory behaviour
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Practical exercise: Rewrite a Pandas-heavy transformation in Polars and compare memory usage and processing speed

Module 9: Applying Optimisation to ETL and ML Workloads

• Applying configuration, partitioning, and memory-management principles across a realistic ETL pipeline
• Understanding optimisation considerations specific to machine-learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Practical exercise: Complete an end-to-end mini-project combining data loading, transformation, and a simple model-training step, while participants tune the Spark configuration themselves