Kafka security-Encryption and Authentication using SSL

1. Overview
   For Kafka release 0.9.X, the following security measures are currently supported:

   • Authentication of connections to brokers from clients (producers and consumers), other brokers and tools, using either SSL or SASL (Kerberos)

   • Authentication of connections from brokers to ZooKeeper

   • Encryption of data transferred between brokers and clients, between brokers, or between brokers and tools using SSL (Note that there is a performance degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)
   • Authorization of read / write operations by clients
   • Authorization is pluggable and integration with external authorization services is supported
     as the picture display:

2. Encryption and Authentication using SSL
   I. architecture
   Provided we have 3 servers. Server 1 generate CA, server 2 and Server 3 use the CA which is generated on server 1.
   The architecture display as below:
   

Note: This document just focus on how to configure SSL omiting the Zookeeper and Kafka deployment.

II. configuration of server 1 :
firstly create one directory for SSL:

 cd $KAFKA_HOME   #your own Kafka home  
 mkdir SSL  
 cd SSL  

then, run the below command one by one:

keytool -keystore server.keystore.jks -alias $HOSTNAME -validity 365 -genkey  
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365  

keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert  
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert  
keytool -keystore server.keystore.jks -alias $HOSTNAME -certreq -file cert-file  

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:test123  

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert  
keytool -keystore server.keystore.jks -alias $HOSTNAME -import -file cert-signed  

Kafka broker configuration:

cd $KAFKA_HOME/config  
vi server.properties  

listeners=PLAINTEXT://:9092,SSL://:9093 #add ',SSL://:9093'  
advertised.listeners=SSL://10.103.219.67:9093 # advertised.listeners  

ssl.keystore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/server.keystore.jks  
ssl.keystore.password=test123  
ssl.key.password=test123  
ssl.truststore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/server.truststore.jks  
ssl.truststore.password=test123  
ssl.client.auth=requested  
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1  
ssl.keystore.type=JKS  
ssl.truststore.type=JKS  
security.inter.broker.protocol=SSL #enable SSL for inter-broker communication  

Kafka client configuration:

cd $KAFKA_HOME/config  
vi client-ssl.properties  

#add the below configuration:  
security.protocol=SSL  
ssl.truststore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/client.truststore.jks  
ssl.truststore.password=test123  
ssl.keystore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/server.keystore.jks  
ssl.keystore.password=test123  
ssl.key.password=test123  
#ssl.enable.protocols=TLSv1.2,TLSv1.1,TLSv1  
ssl.truststore.type=JKS  
ssl.keystore.type=JKS  

III. configuration of server 2 (server 3 is the same with server 2) :
firstly create one directory for SSL:

cd $KAFKA_HOME  #your own Kafka home  
mkdir SSL  
cd SSL  

then, run the below command one by one:

keytool -keystore client.keystore.jks -alias $HOSTNAME -validity 365 -genkey  
keytool -keystore client.keystore.jks -alias $HOSTNAME -certreq -file cert-file-$HOSTNAME  

mkdir CA   

then copy the generated file at the server 1 to the CA directory, in my case, /usr/local/kafka_2.11-0.9.0.0/SSL/CA:

scp -r /usr/local/kafka_2.11-0.9.0.0/SSL/* casb-68:/usr/local/kafka_2.11-0.9.0.0/SSL/CA  

then, continue to run the below command one by one:

openssl x509 -req -CA CA/ca-cert -CAkey CA/ca-key -in cert-file-$HOSTNAME -out cert-signed-$HOSTNAME -days 365 -CAcreateserial -passin pass:test123  
keytool -keystore client.keystore.jks -alias CARoot -import -file CA/ca-cert  
keytool -keystore client.keystore.jks -alias $HOSTNAME -import -file cert-signed-$HOSTNAME  

Kafka broker configuration:

cd $KAFKA_HOME/config  
vi server.properties  
#add the below configuration:  
listeners=PLAINTEXT://:9092,SSL://:9093 #add ',SSL://:9093'  
advertised.listeners=SSL://10.103.219.68:9093 # advertised.listeners  
ssl.keystore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/CA/server.keystore.jks  
ssl.keystore.password=test123  
ssl.key.password=test123  
ssl.truststore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/CA/server.truststore.jks  
ssl.truststore.password=test123  
ssl.client.auth=requested  
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1  
ssl.keystore.type=JKS  
ssl.truststore.type=JKS  
security.inter.broker.protocol=SSL #enable SSL for inter-broker communication  

Kafka client configuration:

cd $KAFKA_HOME/config  
vi client-ssl.properties  

#add the below configuration:  
security.protocol=SSL  
ssl.truststore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/client.keystore.jks  
ssl.truststore.password=test123  
ssl.keystore.location=/usr/local/kafka_2.11-0.9.0.0/SSL/client.keystore.jks  
ssl.keystore.password=test123  
ssl.key.password=test123  
#ssl.enable.protocols=TLSv1.2,TLSv1.1,TLSv1  
ssl.truststore.type=JKS  
ssl.keystore.type=JKS  

IV. example :
1). check zookeeper status is running
2). start all brokers
3). create topic
then you can
run at command line:

kafka-console-producer.sh --broker-list casb-68:9093 --topic test-ssl --producer.config client-ssl.properties  
kafka-console-consumer.sh --bootstrap-server casb-67:9093 --topic test-ssl --new-consumer --consumer.config client-ssl.properties  

run at java IDE:

Properties props = new Properties();   
//other the ordinary properties, add the below properties  
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, " ");  
props.put(ProducerConfig.CLIENT_ID_CONFIG, " ");  //same with the client-ssl.properties  
props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, " ");  //same with the client-ssl.properties  
props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, " ");  //same with the client-ssl.properties  
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, " ");  //same with the client-ssl.properties  
props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, " ");  //same with the client-ssl.properties  
props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "JKS");   
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");   

KafkaProducer producer = new KafkaProducer(props);   
KafkaConsumer consumer = new KafkaConsumer(props);