Getting Started with Apache Kafka

Am doing some toe-dipping into Apache Kafka.

Linux:

Windows:

Commands from the Apache quick start documentation:

  • This gave me a good overview of what the system is doing.
> bin/zookeeper-server-start.sh config/zookeeper.properties
> bin/kafka-server-start.sh config/server.properties
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
> bin/kafka-topics.sh --list --zookeeper localhost:2181
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties
[Now edit these new files and set the following properties:
config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1
config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2]
> bin/kafka-server-start.sh config/server-1.properties &
> bin/kafka-server-start.sh config/server-2.properties &
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
> ps | grep server-1.properties
> kill -9 7564
[Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:]
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

Netstat, Telnet

In my past life I was so used to using the ‘netstat’ command. So I found it funny when I got to re-use it again after ~5 years.

Netstat:

  • netstat -nltp
  • netstat -rn
netstat [address_family_options] [–tcp|-t] [–udp|-u] [–raw|-w] [–listening|-l] [–all|-a] [--numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–symbolic|-N] [–extend|-e[–extend|-e]] [–timers|-o] [–program|-p] [–verbose|-v] [–continuous|-c]
netstat {–route|-r} [address_family_options] [–extend|-e[–extend|-e]] [–verbose|-v] [–numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–continuous|-c]

 

There are so many more flags (see reference)

Telnet:

  • used telnet to verify connectivity with zookeeper
  • at the telnet prompt, type in ruok and press ENTER. if everything’s fine, ZooKeeper will say imok and end the Telnet session

References: