Skip to main content

What is advertised.listeners in Kafka?

This is the title of the webpage!
Hi guys, 
Today we gonna talk about Kafka Broker Properties.
More Specifically, advertised.listeners property. If you have seen the server.properties file in Kafka there are two properties with listener settings.
  1. #listeners=PLAINTEXT://:9092
  2. #advertised.listeners=PLAINTEXT://your.host.name:9092
why the hell we need two listeners for our broker?
usually, Kafka brokers talk to each other and register themselves in zookeeper using listeners property. So for all internal cluster communication happens over what you set in listeners property.
But if you have a complex network, for example, consider if your cluster is on the cloud which has an internal network and also external IP on which rest of the work can connect to your cluster, in that case, you have to set advertised.listeners property with {EXTERNAL_IP}://{EXTERNAL_PORT}.
For Example,
If Internal IP is 10.168.4.9 and port is 9092 and External IP is 35.196.212.10 and port is 3101 then your property will look like,
listeners=PLAINTEXT://10.168.4.9:9092 &
advertised.listeners = PLAINTEXT://35.196.212.10:3101
So broker will register this value in zookeeper and when the external world wants to connect to your Kafka Cluster they can connect over the network which you provide in “advertised. listeners” property.
Bonus Tip: 
This is also true in case of Kafka running inside the Kubernetes Cluster. In that case, you might want to open NodePort on your worker node and provide node_ip and port as “advertised.listeners” to allow the outside world to communicate to Kafka cluster.

Comments

  1. Hello Suraj, my kafka broker is running inside a kubernetes cluseter. I added the advertised.listeners with a public IP and exposed it outside. but i'm not able to connect from outside. any idea?

    ReplyDelete
    Replies
    1. Did you expose it as nodePort ?

      Delete
    2. same issue with me as well but i haven't setup up a cluster ,its in a single VM.
      when i try adding my public ip and produce messages from my local machine its not working

      Delete

Post a Comment

Popular posts from this blog

Cloud Computing Weekly Digest #1

This is the title of the webpage! Cloud Computing Weekly Newsletter is a weekly digest of all the major development and updates that happened in major cloud providers like GCP, AWS, Azure. Photo by  Alex Machado  on  Unsplash

Serverless API Data Ingestion in Google BigQuery: Part 1 (Introduction)

  Ingesting API Data in Google BigQuery the Serverless way! API To Google BigQuery In the era of cloud computing, Serverless has become a buzzword that we keep hearing about, And eventually, we get convinced that serverless is the way to go for companies of all sizes because of various advantages. The basic advantages of the Serverless approach are : No Server Management Scalability Pay as you go In this article, we will also explore how we can use the Serverless approach to build our data Ingestion pipeline in Google Cloud. Serverless Offerings In GCP GCP offers plenty of Serverless Services in various areas such as mentioned below. Computing : Cloud Run, Cloud Function, App Engine Data warehouse : Google BigQuery Object Storage: Google Cloud Storage Workflow management : Cloud Workflow Scheduler: Cloud Scheduler Technically, the combination of the above tools is enough to build API data ingestion in GCP. We can build Two patterns to ingest API data in Google BigQu...