apache-kafka - 卡夫卡监听器不工作！它被隔离在 Intranet 中

Question

我的 Kafka 节点托管在 Google Cloud Dataproc 中。但是，我们意识到通过默认初始化脚本安装的 Kafka 设置为只允许内网访问。它与外界完全隔绝。谷歌云网络外的生产者无法将消息发布到 Kafka，并且 Kafka 消息无法链接到其外网订阅者。

评论

我已将生产者 IP 列入白名单

通读其他 StackOverflow、博客文章和文档后。我认为这可能是由于in的advertised.listeners部分原因。Socket Server Settings/usr/lib/kafka/server.properties

第一个解决方案

我添加了advertised.listeners=PLAINTEXT://[External_IP]:19092

然后sudo /etc/init.d/kafka-server restart

结果

但是，当我尝试使用 Kafkacat 或 telnet 时，它总是失败。我还用各种端口测试了 Advertisementd.listeners

来自https://rmoff.net/2018/08/02/kafka-listeners-explained/的第二个解决方案

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().


->>>>>>> I added below listener config according to https://rmoff.net/2018/08/02/kafka-listeners-explained/

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=EXTERNAL://[External_IP]:19092,INTERNAL://[Internal_IP]:9092
inter.broker.listener.name=INTERNAL

结果

结果和上面一样，不工作。

防火墙规则[更新]

这是我当前的防火墙规则配置。我做错了吗？

谁能帮我解决这个问题？

Answer 1

这是适用于我的集群的方法：

我从第二个解决方案中设置了以下属性：

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=EXTERNAL://[External_IP]:19092,INTERNAL://[Internal_IP]:9092
inter.broker.listener.name=INTERNAL

我创建了一个防火墙规则，将端口 19092 开放给我的个人开发机器 IP，并将其应用于网络。从我的机器上，我尝试远程登录 kafka 服务器，我得到了：

$ telnet [EXTERNAL-IP] 19092
Trying [EXTERNAL-IP]...
Connected to [EXTERNAL-IP].
Escape character is '^]'.

然后我尝试使用 kafkacat，但出现错误。在调试中运行，我看到错误是因为我没有设置任何主题：

%7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: ===== Received metadata: application requested =====
%7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: ClusterId: jYxfi6zzR0euAovYyKCFZg, ControllerId: -1
%7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: 0 brokers, 0 topics
%7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: No brokers or topics in metadata: should retry
%7|1578351264.551|REQERR|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: MetadataRequest failed: Local: Partial response: explicit actions Retry
%7|1578351264.551|RETRY|rdkafka#producer-1| [thrd:[EXTERNAL-IP]:19092/bootstrap]: [EXTERNAL-IP]:19092/bootstrap: Retrying MetadataRequest (v2, 25 bytes, retry 1/2, prev CorrId 3) in 100ms

请注意，我尝试从集群外部连接到 kafka 服务器。在问题中，telnet 和 kafkacat 与 kafka 服务器（kafka-tng-w-0）在同一台机器上运行。

Answer 2

这是一个示例 docker-compose.yaml 文件。

version: '2'

services:

  zookeeper:
    image: strimzi/kafka:0.20.0-kafka-2.6.0
    command: [
      "sh", "-c",
      "bin/zookeeper-server-start.sh config/zookeeper.properties"
    ]
    ports:
      - "2181:2181"
    environment:
      LOG_DIR: /tmp/logs

  kafka:
    image: strimzi/kafka:0.20.0-kafka-2.6.0 
    command: [
      "sh", "-c",
      "bin/kafka-server-start.sh config/server.properties --override 
listeners=$${KAFKA_LISTENERS} --override 
advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override 
zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
    ]
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      LOG_DIR: "/tmp/logs"
      # Dev GQ - Laptop
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.23.240.1:9092
      # AWS Pre-Prod  
      #KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://11.122.200.229:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

这是一个示例 Quarkus application.properties 文件，其中 kafka 引导服务器配置为 docker-compose.yaml 中的广告侦听器。

# Configure the SmallRye Kafka connector
# Dev GQ - Laptop
mp.messaging.connector.smallrye-kafka.bootstrap.servers=172.23.240.1:9092
# AWS Pre-Prod
#mp.messaging.connector.smallrye-kafka.bootstrap.servers=11.122.200.229:9092
quarkus.kafka.health.enabled=true

# Configure the Kafka sink (we write to it)
mp.messaging.outgoing.generated-price.connector=smallrye-kafka
mp.messaging.outgoing.generated-price.topic=prices
mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer

# Configure the Kafka source (we read from it)
mp.messaging.incoming.prices.connector=smallrye-kafka
mp.messaging.incoming.prices.topic=prices
# ..... more codes