S
AlgoSimplified

Security, Authentication, and Authorization

Master Kafka security: SASL mechanisms, SSL/TLS encryption, ACL configuration, quotas, and implementing production-grade security for Kafka clusters.

Course Navigation

← Back to Kafka Course
Previous Lesson
Schema Registry and Data Governance
Current Lesson
9 of 10
Next Lesson
Production Operations and Advanced Patterns
Progress 90%

Why Kafka Security Matters

Kafka without security is like leaving your house unlocked. Here's what can go wrong:

🚨 Security Nightmares

  • Data theft - Anyone can read your messages
  • Data tampering - Malicious producers inject bad data
  • Service disruption - Attackers flood your cluster
  • Compliance violations - GDPR, HIPAA, SOX requirements
  • Reputation damage - Security breaches make headlines

In production, security isn't optional - it's mandatory.

Kafka Security Layers

Kafka security has multiple layers, each protecting different aspects:

1. Network Security (SSL/TLS)

  • Encrypts data in transit - Prevents eavesdropping
  • Server authentication - Clients verify broker identity
  • Client authentication - Brokers verify client identity

2. Authentication (SASL)

  • Who you are - Identity verification
  • Multiple mechanisms - PLAIN, SCRAM, GSSAPI, OAUTHBEARER
  • Integration ready - Works with LDAP, Kerberos, OAuth

3. Authorization (ACL)

  • What you can do - Permission management
  • Fine-grained control - Per-topic, per-operation
  • Role-based access - Groups and permissions

4. Encryption at Rest

  • Disk encryption - LUKS, BitLocker
  • Application-level encryption - Encrypt message payloads
  • Key management - Secure key storage and rotation

SASL Authentication Mechanisms

Choose the right authentication method for your environment:

PLAIN (Simple but Insecure)

⚠️ Use Only for Testing

  • Username/password in plain text
  • No encryption - credentials visible in logs
  • No password hashing
  • Good for: Development, testing, internal networks
# server.properties
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
listeners=SASL_PLAINTEXT://localhost:9092
security.inter.broker.protocol=SASL_PLAINTEXT

# JAAS config
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="admin-secret"
  user_admin="admin-secret"
  user_alice="alice-secret";
};

SCRAM (Recommended for Production)

✅ Production Ready

  • Password hashing - SHA-256 or SHA-512
  • Salt-based - Prevents rainbow table attacks
  • Challenge-response - No password transmission
  • Good for: Production, cloud deployments
# Create SCRAM users
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'SCRAM-SHA-256=[password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' \
  --entity-type users --entity-name alice

# Server configuration
sasl.enabled.mechanisms=SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
listeners=SASL_SSL://localhost:9092
security.inter.broker.protocol=SASL_SSL

GSSAPI (Kerberos Integration)

🏢 Enterprise Integration

  • Kerberos integration - Enterprise authentication
  • Single sign-on - No separate passwords
  • Ticket-based - Time-limited access
  • Good for: Enterprise environments with Kerberos

OAUTHBEARER (OAuth 2.0)

🌐 Modern Authentication

  • OAuth 2.0 integration - Modern auth standard
  • JWT tokens - Stateless authentication
  • Third-party integration - Google, Microsoft, etc.
  • Good for: Cloud-native, microservices

SSL/TLS Encryption Setup

Encrypt data in transit with SSL/TLS:

1. Generate Certificates

# Create CA certificate
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

# Create keystore for broker
keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

# Create truststore for clients
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert

# Sign broker certificate
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

2. Configure Broker

# server.properties
listeners=SSL://localhost:9092
security.inter.broker.protocol=SSL
ssl.keystore.location=/path/to/kafka.server.keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password
ssl.truststore.location=/path/to/kafka.server.truststore.jks
ssl.truststore.password=truststore-password
ssl.client.auth=required

3. Configure Client

# Client configuration
bootstrap.servers=localhost:9092
security.protocol=SSL
ssl.truststore.location=/path/to/kafka.client.truststore.jks
ssl.truststore.password=truststore-password
ssl.keystore.location=/path/to/kafka.client.keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password

ACL (Access Control Lists)

Control who can do what with fine-grained permissions:

ACL Operations

Topic Operations

  • Read - Consume from topic
  • Write - Produce to topic
  • Create - Create new topics
  • Delete - Delete topics
  • Alter - Modify topic configuration
  • Describe - View topic metadata

Cluster Operations

  • ClusterAction - Administrative operations
  • Create - Create resources
  • Alter - Modify configurations
  • Describe - View cluster state

ACL Examples

# Allow user 'alice' to read from 'user-events' topic
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:alice \
  --operation Read --topic user-events

# Allow user 'bob' to write to 'orders' topic
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:bob \
  --operation Write --topic orders

# Allow group 'analytics' to read from any topic
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:analytics \
  --operation Read --topic '*' --group analytics

# Allow user 'admin' to do anything
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:admin \
  --operation All --topic '*' --cluster

Quotas and Rate Limiting

Prevent resource abuse with quotas:

Producer Quotas

# Limit producer to 10MB/s
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'producer_byte_rate=10485760' \
  --entity-type users --entity-name alice

# Limit producer to 1000 requests/s
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'producer_byte_rate=10485760,request_percentage=1000' \
  --entity-type users --entity-name alice

Consumer Quotas

# Limit consumer to 5MB/s
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'consumer_byte_rate=5242880' \
  --entity-type users --entity-name alice

Encryption at Rest

Protect data when it's stored on disk:

1. Disk Encryption

# LUKS encryption (Linux)
cryptsetup luksFormat /dev/sdb
cryptsetup luksOpen /dev/sdb kafka-data
mkfs.ext4 /dev/mapper/kafka-data
mount /dev/mapper/kafka-data /var/kafka-logs

2. Application-Level Encryption

from cryptography.fernet import Fernet
import json

# Generate encryption key
key = Fernet.generate_key()
cipher = Fernet(key)

# Encrypt message before sending
def encrypt_message(message):
    message_json = json.dumps(message)
    encrypted_data = cipher.encrypt(message_json.encode())
    return encrypted_data

# Decrypt message after receiving
def decrypt_message(encrypted_data):
    decrypted_data = cipher.decrypt(encrypted_data)
    message = json.loads(decrypted_data.decode())
    return message

Security Monitoring

Monitor security events and anomalies:

Key Metrics to Monitor

  • Authentication failures - Brute force attempts
  • Authorization denials - Unauthorized access attempts
  • SSL handshake failures - Certificate issues
  • Quota violations - Resource abuse
  • Unusual access patterns - Anomaly detection

Security Alerts

  • Multiple auth failures - Potential attack
  • New user access - Unauthorized account
  • Quota exceeded - Resource abuse
  • SSL errors - Certificate problems

Production Security Checklist

✅ Must Have

  • SSL/TLS encryption - All network traffic
  • SASL authentication - SCRAM or better
  • ACL authorization - Fine-grained permissions
  • Quota limits - Prevent resource abuse
  • Security monitoring - Alert on anomalies

✅ Nice to Have

  • Encryption at rest - Disk encryption
  • Key rotation - Regular certificate updates
  • Audit logging - Track all access
  • Network segmentation - Isolate Kafka cluster
  • Backup encryption - Encrypt backups

Key Takeaways

  1. Security is not optional - Implement from day one
  2. Use SCRAM for authentication - Avoid PLAIN in production
  3. Enable SSL/TLS everywhere - Encrypt all network traffic
  4. Implement fine-grained ACLs - Principle of least privilege
  5. Monitor security events - Detect attacks early

Next Steps

Ready for production operations? Check out our final lesson on Production Operations and Advanced Patterns where we'll learn how to run Kafka in production with Kubernetes, monitoring, and advanced architectural patterns.