Running a minimal Kafka instance on K8S

A couple of days ago I ran into an issue with our hosted Kafka environment, and we had to give a demo. So I was looking for a quick way to run a simple ephemeral Kafka instance on our kubernetes cluster. I looked around a bit, and couldn’t find one working out of the box, so I adapted one I found and which almost worked (https://dzone.com/articles/ultimate-guide-to-installing-kafka-docker-on-kuber) which used the standard docker images from here

Setup

The complete setup will consist out of:

• A single deployment for zookeeper. This will create a single zookeeper pod, with the name zoo1
• A single deployment for kafka. This will create a single kafka pod, with the name kafka-broker0
• two services; one that exposes zookeeper, and one that exposes the kafka-broker
• a final deployment that also adds kafkacat so we can quickly test whether our broker is working and accessible

---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
    - name: client
      port: 2181
      protocol: TCP
    - name: follower
      port: 2888
      protocol: TCP
    - name: leader
      port: 3888
      protocol: TCP
  selector:
    app: zookeeper-1
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
    - port: 9092
      name: kafka-port
      protocol: TCP
  selector:
    app: kafka
    id: "0"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
        - name: zoo1
          image: digitalwonderland/zookeeper
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_ID
              value: "1"
            - name: ZOOKEEPER_SERVER_1
              value: zoo1
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker0
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
        - name: kafka
          image: wurstmeister/kafka
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_ADVERTISED_PORT
              value: "9092"
            - name: KAFKA_ADVERTISED_HOST_NAME
              value: kafka-service
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: zoo1:2181
            - name: KAFKA_BROKER_ID
              value: "0"
            - name: KAFKA_CREATE_TOPICS
              value: sample.topic:1:1
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-cat
spec:
  template:
    metadata:
      labels:
        app: kafka-cat
    spec:
      containers:
        - name: kafka-cat
          image: confluentinc/cp-kafkacat
          command: ["/bin/sh"]
          args: ["-c", "trap : TERM INT; sleep infinity & wait"]

Starting Kafka

To deploy it simple save this configuration as kubernetes-kafka-ephemeral.yml:

$ kubectl apply -f ./kubernetes-kafka-ephemeral.yml
service/zoo1 created
service/kafka-service created
deployment.extensions/zookeeper-deployment-1 created
deployment.extensions/kafka-broker0 created
deployment.extensions/kafka-cat created

And looking at the resources you’ll see that the correct items have been created (the kafka broker might restart if zookeeper isn’t up yet, but that’ll resolve itself):

 
$ kubectl get services
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka-service   ClusterIP   10.105.141.140   <none>        9092/TCP                     72s
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP                      49m
zoo1            ClusterIP   10.108.158.62    <none>        2181/TCP,2888/TCP,3888/TCP   72s
 
$ kubectl get pods 
NAME                                      READY   STATUS    RESTARTS   AGE
kafka-broker0-677f5b7ccf-gsstk            1/1     Running   0          76s
kafka-cat-84b9cfdf45-x9wk7                1/1     Running   0          76s
zookeeper-deployment-1-684478678c-d8m5m   1/1     Running   0          76s
 
$ kubectl get deployments
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
kafka-broker0            1/1     1            1           80s
kafka-cat                1/1     1            1           80s
zookeeper-deployment-1   1/1     1            1           80s

Using Kafka

kafkacat runs in a pod which won’t exit, so you can just exec into the pod, and see if everything works. Easiest way to test is to just open two shells and run the kafkacat commands (kafkacat -P -b kafka-service:9092 -t test and kafkacat -b kafka-service:9092 -t test):