Kafka with KRaft (Kafka Raft)

                                                            image credit Kafka

1. What is Kafka?

Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, real-time data streaming. It is used for publish-subscribe messaging, event sourcing, log processing, and real-time analytics.

Key Features:

• Scalability: Distributes data across multiple brokers.
• Durability: Stores data persistently.
• High Throughput: Handles millions of messages per second.
• Fault Tolerance: Replicates data across nodes.

Core Components:

• Producers: Send messages (events) to Kafka.
• Topics: Logical channels where messages are stored.
• Partitions: Sub-divisions of topics for parallel processing.
• Consumers: Read messages from topics.
• Brokers: Kafka servers that store and manage data.
• Zookeeper (or KRaft in newer versions): Manages metadata and leader election.

---

2. How Kafka Works (Step-by-Step)

Step 1: Producers Publish Messages

• Producers send messages to a topic.
• Kafka distributes messages among partitions (using round-robin or key-based distribution).

Step 2: Brokers Store Messages

• Brokers receive data and store it in partition logs.
• Messages persist based on retention policies.

Step 3: Consumers Subscribe to Topics

• Consumers read messages sequentially from partitions.
• Kafka maintains an offset (position of last-read message).
• Consumers can commit offsets for fault tolerance.

Step 4: Message Processing

• Consumers process messages and may send them to other Kafka topics, databases, or services.

Example: Kafka in Action

Imagine an e-commerce platform:

• Order Service (Producer) sends order events to orders-topic.
• Inventory Service (Consumer) updates stock.
• Shipping Service (Consumer) prepares delivery.

---

3. What is Zookeeper in Kafka?

Apache Zookeeper is a distributed coordination service used by Kafka for:

1. Broker Metadata Management: Keeps track of brokers in the cluster.
2. Leader Election: Ensures high availability by selecting leader brokers.
3. Topic and Partition Management: Stores information about topics and their partitions.

Limitations of Zookeeper:

• Single Point of Failure: Zookeeper failure affects Kafka.
• Operational Overhead: Requires separate setup and maintenance.
• Scalability Issues: Slower performance in large clusters.

---

                                    image credit confluent

4. KRaft (Kafka Raft) – Replacing Zookeeper

Kafka Raft (KRaft) is a Zookeeper-free mode introduced in Kafka 2.8+ and production-ready in Kafka 3.3+.

Why KRaft?

• Removes dependency on Zookeeper.
• Improves scalability: No separate cluster needed.
• Simplifies deployment: Kafka manages its metadata internally.

How KRaft Works:

• Uses the Raft consensus algorithm for leader election and replication.
• One broker acts as the Controller to manage metadata.
• Other brokers synchronize metadata through replicated logs.

---

5. How to Set Up Kafka with KRaft (Without Zookeeper)

Step 1: Install Kafka

Download Kafka:

wget https://downloads.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz
tar -xzf kafka_2.13-3.5.0.tgz
cd kafka_2.13-3.5.0

Step 2: Generate KRaft Metadata

Run:

bin/kafka-storage.sh format -t $(uuidgen) -c config/kraft/server.properties

This initializes metadata storage.

Step 3: Configure Kafka for KRaft

Edit config/kraft/server.properties:

process.roles=controller,broker
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
log.dirs=/tmp/kafka-logs

Step 4: Start Kafka (Without Zookeeper)

bin/kafka-server-start.sh config/kraft/server.properties

Step 5: Create a Topic

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Step 6: Start a Producer

bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

Enter messages.

Step 7: Start a Consumer

bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

---

Conclusion

• Kafka is a high-throughput event streaming system.
• Zookeeper was used for metadata management but had scalability issues.
• KRaft (Kafka Raft) replaces Zookeeper with an internal Raft-based system.
• Setting up Kafka with KRaft is easier and more scalable.

 You can find more article and tutorial about Kafka here.

Python Kafka

 

Developing Microservices with Python, REST API, Nginx, and Kafka (End-to-End)

Here's a step-by-step guide to developing microservices with the mentioned technologies:

1. Define Your Microservices:

• Break down Functionality: Identify distinct functionalities within your application that can be independent services. These services should have well-defined APIs for communication.
• Example: If you're building an e-commerce application, separate services could manage user accounts, products, orders, and payments.

2. Develop Python Microservices with RESTful APIs:

• Choose a Python framework: Popular options include Flask, FastAPI, and Django REST Framework.
• Develop each microservice as a separate Python application with clearly defined endpoints for API calls (GET, POST, PUT, DELETE).
• Use libraries like requests for making API calls between services if needed.
• Implement data persistence for each service using databases (e.g., PostgreSQL, MongoDB) or other storage solutions.

3. Setup Nginx as a Reverse Proxy:

• Nginx acts as a single entry point for external traffic directed to your application.
• Configure Nginx to route incoming requests to the appropriate microservice based on the URL path.
• You can use tools like uvicorn (with ASGI frameworks) or gunicorn (with WSGI frameworks) to serve your Python applications behind Nginx.

4. Implement Communication with Kafka:

• Producers: Use Kafka producer libraries for Python (e.g., confluent-kafka-python) to send messages (events) to specific Kafka topics relevant to your application's needs.
• Consumers: Each microservice can subscribe to relevant Kafka topics to receive events published by other services. Implement consumer logic to react to these events and update its data or perform actions accordingly.
• Kafka acts as a decoupling mechanism, allowing services to communicate asynchronously and avoid tight coupling.

5. Build and Deploy:

• Containerization: Consider containerizing your Python applications using Docker for easier deployment and management.
• Orchestration: Use container orchestration tools like Docker Swarm or Kubernetes to manage scaling and deployment across multiple servers (if needed).

Example Workflow:

1. User sends a request to Nginx.
2. Nginx routes the request based on the URL path to the appropriate microservice.
3. Microservice processes the request, interacts with its database/storage, and generates a response.
4. If necessary, the microservice publishes an event to a Kafka topic.
5. Other microservices subscribed to that topic receive the event and react accordingly, updating data or performing actions.
6. The response from the original microservice is sent back through Nginx to the user.

Additional Considerations:

• Configuration Management: Tools like Consul or Etcd can be used to manage configuration settings for microservices and Kafka.
• Logging and Monitoring: Implement logging and monitoring solutions (e.g., Prometheus, Grafana) to track performance and troubleshoot issues.
• Security: Secure your API endpoints and consider authentication and authorization mechanisms. Explore libraries like python-jose for JWT (JSON Web Token) based authentication.

Resources:

• Flask Tutorial: https://palletsprojects.com/p/flask/
• FastAPI Tutorial: https://github.com/tiangolo/full-stack-fastapi-template
• Django REST Framework Tutorial: https://www.django-rest-framework.org/tutorial/quickstart/
• Nginx Configuration Guide: https://docs.nginx.com/nginx/admin-guide/web-server/web-server/
• Confluent Kafka Python Client: https://docs.confluent.io/platform/current/clients/api-docs/confluent-kafka-python.html

Remember: This is a high-level overview. Each step involves further research and configuration based on your specific requirements.

While you can't directly implement a full-fledged Kafka-like system in pure Python due to its distributed nature and complex features, you can create a multithreaded event bus using libraries or build a basic version yourself. Here are two approaches:

1. Using a Third-Party Library:

• Libraries: Consider libraries like kombu (built on top of RabbitMQ) or geventhub (https://docs.readthedocs.io/) that provide multithreaded message queues with features like publishers, subscribers, and concurrency handling. These libraries handle the low-level details, allowing you to focus on the event bus logic.

2. Building a Basic Event Bus:

Here's a basic implementation to illustrate the core concepts:

Python

from queue import Queue
from threading import Thread

class EventBus:
  def __init__(self):
    self.subscribers = {}  # Dictionary to store subscribers for each topic
    self.event_queue = Queue()  # Queue to hold events

  def subscribe(self, topic, callback):
    """
    Subscribes a callback function to a specific topic.

    Args:
        topic: The topic to subscribe to.
        callback: The function to be called when an event is published to the topic.
    """
    if topic not in self.subscribers:
      self.subscribers[topic] = []
    self.subscribers[topic].append(callback)

  def publish(self, topic, event):
    """
    Publishes an event to a specific topic.

    Args:
        topic: The topic to publish the event to.
        event: The event data to be sent to subscribers.
    """
    self.event_queue.put((topic, event))

  def run(self):
    """
    Starts a thread to handle event processing from the queue.
    """
    def process_events():
      while True:
        topic, event = self.event_queue.get()
        for callback in self.subscribers.get(topic, []):
          callback(event)  # Call each subscriber's callback with the event

    event_thread = Thread(target=process_events)
    event_thread.start()

# Example usage
def callback1(event):
  print("Callback 1 received event:", event)

def callback2(event):
  print("Callback 2 received event:", event)

event_bus = EventBus()
event_bus.subscribe("my_topic", callback1)
event_bus.subscribe("my_topic", callback2)
event_bus.publish("my_topic", {"data": "This is an event!"})
event_bus.run()  # Start the event processing thread

Explanation:

1. EventBus Class:

   • subscribers: Dictionary to store lists of callback functions for each topic.
   • event_queue: Queue to hold events published to different topics.
   • subscribe: Registers a callback function for a specific topic.
   • publish: Adds an event to the queue with the corresponding topic.
   • run: Creates a separate thread to process events from the queue. The thread loops, retrieves events from the queue, and calls the registered callback functions for the matching topic with the event data.

2. Example Usage:

   • Defines two callback functions (callback1 and callback2) to be called when an event is published.
   • Creates an EventBus instance.
   • Subscribes both callbacks to the topic "my_topic".
   • Publishes an event to "my_topic" with some data.
   • Starts the event processing thread using run().

This is a basic multithreaded event bus. For a fully-fledged system, you'd need to consider additional features:

• Thread Safety: Implement synchronization mechanisms like locks to ensure safe access to shared resources (e.g., the queue) from multiple threads.
• Error Handling: Handle potential errors like queue full exceptions or exceptions raised by subscriber callbacks.
• Serialization/Deserialization: If events contain complex data structures, consider using libraries like pickle or json to serialize them before sending and deserialize them on the receiving end.

Remember, this is a simplified example. Consider exploring the libraries mentioned earlier for more robust event bus implementations in Python.

You can search for more articles and tutorials here on this blog.