Skip to main content

What is advertised.listeners in Kafka?

This is the title of the webpage!
Hi guys, 
Today we gonna talk about Kafka Broker Properties.
More Specifically, advertised.listeners property. If you have seen the server.properties file in Kafka there are two properties with listener settings.
  1. #listeners=PLAINTEXT://:9092
  2. #advertised.listeners=PLAINTEXT://your.host.name:9092
why the hell we need two listeners for our broker?
usually, Kafka brokers talk to each other and register themselves in zookeeper using listeners property. So for all internal cluster communication happens over what you set in listeners property.
But if you have a complex network, for example, consider if your cluster is on the cloud which has an internal network and also external IP on which rest of the work can connect to your cluster, in that case, you have to set advertised.listeners property with {EXTERNAL_IP}://{EXTERNAL_PORT}.
For Example,
If Internal IP is 10.168.4.9 and port is 9092 and External IP is 35.196.212.10 and port is 3101 then your property will look like,
listeners=PLAINTEXT://10.168.4.9:9092 &
advertised.listeners = PLAINTEXT://35.196.212.10:3101
So broker will register this value in zookeeper and when the external world wants to connect to your Kafka Cluster they can connect over the network which you provide in “advertised. listeners” property.
Bonus Tip: 
This is also true in case of Kafka running inside the Kubernetes Cluster. In that case, you might want to open NodePort on your worker node and provide node_ip and port as “advertised.listeners” to allow the outside world to communicate to Kafka cluster.

Comments

  1. Sundar29 September 2019 at 15:16

    Hello Suraj, my kafka broker is running inside a kubernetes cluseter. I added the advertised.listeners with a public IP and exposed it outside. but i'm not able to connect from outside. any idea?

    ReplyDelete
    Replies
    1. Suraj9 October 2019 at 10:31

      Did you expose it as nodePort ?

      Delete
    2. Anonymous31 December 2019 at 01:22

      same issue with me as well but i haven't setup up a cluster ,its in a single VM.
      when i try adding my public ip and produce messages from my local machine its not working

      Delete

Post a Comment

Popular posts from this blog

Beginners Guide to Machine Learning on GCP

This is the title of the webpage! This blog covers basic knowledge needed to get started ML journey on GCP. It provides foundational knowledge which will help readers to gain some level of confidence understanding ML ecosystem on GCP from where they can master each component. Introduction to Machine Learning Machine Learning is a way to use some set of algorithms to derive predictive analytics from data . It is different than Business Intelligence and Data Analytics in a sense that In BI and Data analytics Businesses make decision based on historical data, but In case of Machine Learning , Businesses  predict  the future based on the historical data. Example, It’s a difference between  what happened to the business  vs  what will happen to the business .Its like making BI much smarter and scalable so that it can predict future rather than just showing the state of the business. ML is based on Standard algorithms which are used to create use case specific model
Post a Comment
Read more

Using NPM Library in Google BigQuery UDF

  Javascript UDF’s are cool and using with NPM library is a whole new world to explore! Background One of the main reason to build ETL pipeline was to do data transformation on data before loading into the data warehouse. The only reason we were doing that because data warehouses were not capable to handle these data transformations due to several reasons such as performance and flexibility. In the era of modern data warehouses like Google BigQuery or SnowFlake , things have changed. These data warehouses can process terabyte and petabyte data within seconds and minutes. Considering this much improvement, now performing data transformation within a data warehouse make more sense. Hence to create common transformation logic via UDF (user-defined functions). In this blog, we will see how can we utilize the power of javascript UDF and NPM library to generate data in BigQuery. What is UDF? From Google Cloud Documentations: A user-defined function (UDF) lets you create a function by using
Post a Comment
Read more