Evaluate use of credentials in Kafka (auth + performance)

[GregorioBlazquez]

As part of the Kafka-based architecture replacing Cygnus (Kafka + Kafka Streams with Faust + Kafka Connect), we should explore enabling credentials in Kafka and assess its performance impact.

• Identify suitable auth mechanisms.
  

• Test integration with Kafka Connect and clients.
  

• Benchmark latency and throughput with/without auth.
  

This would help us prepare for secure production environments.

[GregorioBlazquez]

Kafka authentication: first step with SASL/PLAIN

We could start by testing SASL/PLAIN, as it's simple and supported by:

• Kafka (cp-kafka)
• Kafka Connect
• Faust (aiokafka)

It’s easy to enable in Docker Compose without major changes.

Minimal broker config example:

  kafka:
    image: confluentinc/cp-kafka:7.9.0
    environment:
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_LISTENERS=SASL_PLAINTEXT://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
      - KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://kafka:9093,PLAINTEXT://kafka:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=SASL_PLAINTEXT
      - KAFKA_SASL_ENABLED_MECHANISMS=PLAIN
      - KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
      - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/secrets/kafka_server_jaas.conf

JAAS config example (kafka_server_jaas.conf):

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="admin-secret"
  user_admin="admin-secret"
  user_faust="faust-pass"
  user_connect="connect-pass";
};

We could keep port 9092 open with no auth during testing, and gradually switch components to authenticate.