Kafka Fundamentals for Beginners
BeginnerLearn the core concepts of Apache Kafka including topics, producers, consumers, and brokers. This course covers distributed systems, Kafka architecture, installation, and hands-on practice.
1.1 Overview of Distributed Systems
🎥Understanding distributed systems vs monolithic architecture and communication patterns.
1.2 Why Kafka?
🎥Learn what makes Kafka ideal for real-time data processing. Explore use cases including real-time analytics, event sourcing, and log aggregation, with examples from Uber, LinkedIn, and Netflix.
1.3 Brief History of Kafka
🎥Discover Kafka's journey from LinkedIn in 2010 to becoming an Apache top-level project. Learn about key milestones including replication (2014), Kafka Streams (2016), and KSQL (2017).
2.1.0 Installing Kafka With Zookeeper
🎥Traditional Kafka installation with Zookeeper for Windows, Mac, Linux, and Docker. Covers environment setup, Zookeeper configuration, and Kafka server startup.
2.1.1 Installing Kafka Without Zookeeper (KRaft Mode)
🎥Complete guide to installing Kafka 3.8.0 in KRaft mode without Zookeeper. Covers installation steps for Linux, Mac, Windows, and Docker with detailed configuration examples.
2.2 Kafka Configuration Files
🎥Overview of Kafka configuration files and key settings.
2.3 Setting Up Zookeeper
🎥Understanding Zookeeper's role and how to configure it for Kafka.
2.4 Understanding KRaft Mode
🎥Learn Kafka's KRaft consensus protocol that eliminates Zookeeper dependency. Covers installation on Linux, Mac, Windows, and Docker with complete configuration examples.
3.1 Kafka Architecture
🎥Deep dive into Kafka's distributed system architecture.
3.2 Key Components
Understanding brokers, producers, consumers, and topics in Kafka.
3.3 Topics and Partitions
How Kafka distributes and stores data using topics and partitions.
3.4 Real-Time vs Batch Processing
Understanding the differences between real-time and batch processing in Kafka.
3.5 Cluster Scaling
Learn how to scale Kafka clusters for increased data volumes and traffic.
4.1 Creating and Managing Topics
Commands and configuration for managing Kafka topics.
4.2 Producing Messages
Using the command line to produce messages and best practices.
4.3 Understanding Kafka Schema
Introduction to schema management in Kafka.
4.4 Schema Evolution
Backward compatibility and schema evolution examples.
4.5 Consuming Messages
Consumer groups, offsets, and message consumption patterns.
4.6 Performance Testing
NFT and performance testing strategies for Kafka.
4.7 In-Sync Replicas and Acknowledgements
Ensuring data reliability with ISR and acknowledgement modes.
5.1 Kafka API Introduction
Overview of Kafka's Producer and Consumer APIs.
5.2 Producer API
Sending data to Kafka using the Producer API.
5.3 Consumer API
Learn to read data from Kafka using Consumer API. Covers consumer groups, polling, offsets, and graceful shutdown with complete Java implementation examples.
5.4 Consumer Offset Management
Master offset management for reliable Kafka consumers. Covers auto vs manual commits, consumer groups, partition assignment, and offset reset strategies with code examples.
5.5 Message Guarantees
At-least-once, at-most-once, and exactly-once delivery semantics.
6.1 Kafka Connect
Integrating external systems with Kafka Connect.
6.2 Running Kafka Connect Locally
Set up Kafka Connect using Docker to stream data from MySQL to file. Learn JDBC Source and File Sink connectors for real-time data integration.
6.3 Kafka Streams
Real-time stream processing with Kafka Streams library. Covers stateless vs stateful operations, exactly-once processing, and event-time windowing.
6.4 Running Kafka Streams Locally
Build and run Kafka Streams application locally using Java and Docker. Filter and transform transaction data in real-time with hands-on examples.
6.5 Kafka Streams Deep Dive
Master KStream, KTable, and GlobalKTable with advanced joins, aggregations, and state management. Includes concurrency and scaling strategies.