Producing Messages

Overview

This lesson covers producing messages to Kafka using command-line tools, including best practices for testing and debugging.

Why Use Command Line?

Command-line tools are ideal for:

• Quick testing and debugging
• Simulating data flows for NFT testing
• Verifying broker connections
• Validating ACL configurations
• Prototyping before application development

Environment Setup

Use Docker Compose with three services:

• Zookeeper: Coordinates the cluster
• Kafka: Broker handling message production/consumption
• Schema Registry: Stores and serves Avro schemas for validation

Producing Messages Without Keys

Round-Robin Distribution

When no key is specified, Kafka distributes messages across partitions using round-robin or its internal algorithm.

Example: Simple Messages

1docker exec -it kafka kafka-console-producer.sh \\
2  --topic payment \\
3  --bootstrap-server localhost:9092
4> {"user_id": 1, "amount": 100.0, "currency": "USD"}
5> {"user_id": 2, "amount": 250.0, "currency": "AUD"}

Messages are distributed evenly across partitions without control over which partition receives which message.

Producing Messages With Keys

Key-Based Partitioning

Keys determine which partition receives the message, ensuring all messages with the same key go to the same partition.

Enable Key Parsing

1docker exec -it kafka kafka-console-producer.sh \\
2  --topic payment \\
3  --bootstrap-server localhost:9092 \\
4  --property "parse.key=true" \\
5  --property "key.separator=:"
6> 1:{"user_id": 1, "amount": 100.0, "currency": "USD"}
7> 2:{"user_id": 2, "amount": 250.0, "currency": "AUD"}

Benefits of Keys

• Messages with same key always go to same partition
• Order preserved for related messages
• Better control over data distribution
• Enables stateful processing

Working with Schema Registry

Why Use Schemas?

Schemas provide:

• Data structure definition
• Contract between producers and consumers
• Prevention of mismatched data
• Validation at write time

Avro Schema Example

1{
2  "type": "record",
3  "name": "Payment",
4  "fields": [
5    {"name": "user_id", "type": "int"},
6    {"name": "amount", "type": "float"},
7    {"name": "currency", "type": "string"}
8  ]
9}

Register Schema

1docker exec -i schema-registry curl -X POST \\
2  -H "Content-Type: application/vnd.schemaregistry.v1+json" \\
3  --data '{"schema": "..."}' \\
4  http://localhost:8081/subjects/payment-value/versions

Produce with Schema

Once registered, producers can send messages that conform to the schema, with the Schema Registry automatically validating the data structure.

Topic Creation

Before producing messages, create the topic:

1docker exec -it kafka kafka-topics.sh \\
2  --create --topic payment \\
3  --bootstrap-server localhost:9092 \\
4  --partitions 2 \\
5  --replication-factor 1

Best Practices

Message Keys

• Use keys for related data that needs ordering
• Choose keys that distribute load evenly
• Avoid keys with high cardinality imbalance

Schema Management

• Always use schemas in production
• Version schemas properly
• Test schema changes before deployment
• Monitor Schema Registry health

Performance

• Batch messages when possible
• Use compression for large messages
• Monitor producer metrics
• Tune buffer sizes appropriately

Summary

Producing messages effectively requires:

• Understanding key-based vs keyless distribution
• Using schemas for data consistency
• Proper topic configuration
• Following best practices for performance and reliability

The command-line tools provide a foundation for understanding Kafka producers before implementing application-level producers.