Kafka for Administrators Training Course

Fundamentals of Kafka Administration

• The role of Kafka in modern data platforms and typical production responsibilities.
• Key concepts for operators: brokers, topics, partitions, offsets, and consumer groups.
• Replication basics: leaders and followers, in-sync replicas, and availability trade-offs.
• Operational highlights and standard terminology found in runbooks.

KRaft Mode and Cluster Architecture

• KRaft fundamentals: controllers, metadata quorum, elections, and their operational significance.
• Deployment planning: sizing for throughput, partitions, retention, and future growth.
• Node roles and layouts: combined vs. dedicated controllers, and fault domain considerations.
• Lab: Examine KRaft metadata, validate quorum health, and interpret controller logs.

Installation, Configuration, and Daily Operations

• Installation methods (packages, tarballs, containers) and standardization best practices in enterprise environments.
• Core broker settings affecting reliability: listeners, replication, log directories, and retention.
• Safe service management: startup sequencing, graceful shutdown, and validation checks.
• Lab: Deploy a multi-node cluster, verify broker registration, and confirm baseline produce and consume operations.

Managing Topics, Partitions, and Data Distribution

• Topic lifecycle management via Kafka CLI: creating, describing, updating configurations, and deleting.
• Selecting appropriate partitions and replication factors for real-world workloads, including common anti-patterns.
• Reassignment and balancing: determining when to move partitions and verifying progress safely.
• Lab: Create topics, initiate partition reassignment, simulate a broker failure, and confirm recovery.

Production Security for Kafka

• TLS implementation for client and inter-broker traffic: certificates, trust chains, and validation procedures.
• SASL authentication: selecting common mechanisms and preventing misconfiguration.
• ACL-based authorization: implementing least-privilege patterns for admins, producers, and consumers.
• Lab: Enable TLS and SASL, verify client connectivity, and apply ACLs for specific application roles.

Observability, Reliability, and Troubleshooting

• Essential monitoring: controller health, under-replicated partitions, request latency, and disk/network saturation.
• Logs and metrics: analyzing broker logs and exporting metrics via JMX to standard observability stacks.
• Operational playbooks: executing rolling restarts, making safe configuration changes, and managing disk-full or ISR issues.
• Lab: Establish a minimal alert set, diagnose a degraded cluster, and restore healthy replication.

Upgrades and Disaster Recovery Preparedness

• Kafka upgrade planning: compatibility checks, staging, and rollback strategies.
• Backup and recovery expectations: identifying recoverable assets and restoring configurations.
• Overview of cross-cluster replication and the use of MirrorMaker 2 for DR and migrations.
• Conclusion: Operational checklists, handover documentation, and next steps for production rollout.