Apache kafka Setup in Google cloud


This blog post guides you through setting up a basic Kafka environment on Google Cloud Platform for learning purposes. Kafka is a powerful distributed event streaming platform used for real-time data processing. We'll walk through launching a Kafka cluster, creating a topic, and sending and consuming messages.

Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.

Prerequisites:

  • A Google Cloud Platform account

Steps:

  1. Deploying Kafka:
  • Head over to the Google Cloud Marketplace: https://console.cloud.google.com/marketplace/product/google/kafka
  • Click on "LAUNCH" and proceed with the deployment configuration.
  • Important:
    • For service account, you can choose an existing one or create a new one with appropriate permissions.
    • Select a deployment region closest to you for optimal performance.
    • Keep the disk space settings at default for this learning exercise.
  • Once done, click on "Create" to deploy the Kafka cluster.
  1. Connecting via SSH:
  • Once deployed, access the Kafka cluster instance through SSH using the provided credentials


  1. Starting Kafka Services:

  • Navigate to the Kafka installation directory:

  • Bash

    cd /opt/kafka/
  • Start Zookeeper, which manages the cluster coordination:
  • Bash
    sudo bin/zookeeper-server-start.sh config/zookeeper.properties

  • Open a new SSH terminal and start the Kafka broker service:
  • Bash
    cd /opt/kafka/
sudo bin/kafka-server-start.sh config/server.properties
       
  • These commands will launch the essential Kafka services for your learning environment
  1. Creating a Topic:

  • Open another SSH terminal and navigate to the Kafka directory:

  • Bash

    cd /opt/kafka/
  • Create a topic named "quickstart-events" to hold your messages:
    

    Bash
    sudo bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
  • This command should output "Created topic quickstart-events." verifying its creation.
        

  1. Verifying Topic Details:

  • Use below command to see details about the created topic:

  • Bash

    sudo bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
  • You should see information about the topic, including the number of partitions and replicas.
      

  1. Producing Messages:

  • Kafka uses a producer-consumer model for data flow. The producer writes messages to a topic.

  • Use the producer console:

  • Bash

    sudo bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
  • Start typing your messages and press Enter to send them to the "quickstart-events" topic.
  • Press Ctrl+C to stop sending messages.
  1. Consuming Messages:

  • Open another SSH terminal and use the consumer console:

  • Bash

    sudo bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
  • The consumer will start displaying any messages sent to the "quickstart-events" topic, demonstrating successful message flow.




Comments

Post a Comment

Popular posts from this blog

SOLID Principle (Quick Read)

Image
 The famous design principle comprises 5 design strategies that a developer should follow for a successful application development Single Responsibility Principle Open/ Close Principle Liskov Substitution Principle Interface Segregation Principle Dependency Inversion Single Responsibility Principle(SRP) No code unit(function/class/package) should have more than one responsibility.   Bad Ex:   Class Bird  {     String result;;    public fly(Bird bird)    {     if(bird==piegeon)       result="flys 20 meter hight";    else if(bird==hen)       result="flys 5 meter";    else        result="not measured yet";    } } This class should design such a way that it should work well for all the types of birds current design is hectic to maintain for the following reasons  difficult to test. difficult in parallel programming  understanding the code...
Read more

Design Patterns

 GOF (Gang Of Four) came up with the concepts of design patterns. We can classify the deign patterns in to thress. Creational Design Pattern Structural Design pattern Behavioural Design pattern
Read more

Building a Smart Holiday Booking System with Agent-to-Agent Communication

Image
 Building a Multi-Agent Holiday Booking System with the A2A Protocol (An MVP Approach) The world of AI is rapidly moving towards "agentic systems" — autonomous AI agents that can perform complex, multi-step tasks by collaborating with each other. The challenge, however, has always been standardization: how do you get agents built on different frameworks, by different teams, to communicate effectively? This is the problem the  Agent2Agent (A2A) protocol , an open standard, aims to solve. It provides a common language for agents to discover, communicate, and collaborate securely. In this blog post, we'll walk through a Minimum Viable Product (MVP) approach to a real-world scenario: building a holiday booking system using the A2A protocol in python. Design Architecture (MVP) The Problem: A Siloed Booking Experience Imagine a traditional holiday booking website. It might have separate sections for flights, hotels, and cabs. Each of these services is handled by a different int...
Read more