Zoe - The Kafka CLI for humans
Zoe is a command line tool to interact with kafka in an easy and intuitive way. Wanna see this in action ? check out this demo...
Zoe really shines when it comes to interacting with cloud hosted kafka clusters (kubernetes, AWS, etc.) due to its ability to offload consumption and execution to kubernetes pods or lambda functions (more runners will be supported in the future).
Try out zoe right from your browser!
You can use our Katacoda course to try out zoe and learn about the basics without even installing it.
Key features
Here are some of the most interesting features of zoe :
- Supports offloading consumption of data to lambda functions, kubernetes pods, etc. for parallelism (ex. adding
-r kuberneteswould offload all the requests to multiple pods in a configured kubernetes cluster). - Consume kafka topics from a specific point in time (ex. using
--from 'PT5hfrom the last 5 hours). - Filter data based on content (ex. using
--filter "id == '12345'"filters records with the selected id). - Monitor consumer groups' offsets.
- Upload avro schemas from a
.avscor.avdlfile using different naming strategies. - ... and more.
Go to the install section for instructions on how to install the zoe CLI.
Quickstart
# Initialize zoe configuration
zoe config init
# The generated config points to a local kafka cluster (localhost:29092). You can edit it using the following command
zoe config edit
# You can inspect the list of clusters you have in your config
zoe -o table config clusters list
# You can now use zoe to interact with the clusters
zoe -o table topics list
Sample commands
Read 10 records from the input topic from the local kafka cluster (aliases for topics and clusters are set in the configuration) :
zoe --cluster local topics consume -n 10
{"_id":"5b199196ce456e001424256a","text":"Cats can distinguish different flavors in water.","type":"cat","user":{"_id":"5a9ac18c7478810ea6c06381","name":{"first":"Alex","last":"Wohlbruck"}},"upvotes":6,"userUpvoted":null}
{"_id":"5b1b411d841d9700146158d9","text":"The Egyptian Mau’s name is derived from the Middle...","type":"cat","user":{"_id":"5a9ac18c7478810ea6c06381","name":{"first":"Alex","last":"Wohlbruck"}},"upvotes":5,"userUpvoted":null}
{"_id":"591d9b2f227c1a0020d26823","text":"Every year, nearly four million cats are eaten in ...","type":"cat","user":{"_id":"5a9ac18c7478810ea6c06381","name":{"first":"Alex","last":"Wohlbruck"}},"upvotes":4,"userUpvoted":null}
{"_id":"59951d5ef2db18002031693c","text":"America’s cats, including housecats that adventure...","type":"cat","user":{"_id":"5a9ac18c7478810ea6c06381","name":{"first":"Alex","last":"Wohlbruck"}},"upvotes":4,"userUpvoted":null}
{"_id":"5a4d76916ef087002174c28b","text":"A cat’s nose pad is ridged with a unique pattern, ...","type":"cat","user":{"_id":"5a9ac18c7478810ea6c06381","name":{"first":"Alex","last":"Wohlbruck"}},"upvotes":4,"userUpvoted":null}
Read the 10 records starting from the last 6 hours and format the output data as a table :
zoe --cluster local \
--output table
topics consume \
-n 10 \
--from 'PT6h'
┌──────────────────────────┬───────────────────────────────────────────────────────┬──────┬───────────────────────────────────────────┬─────────┬─────────────┐
│ _id │ text │ type │ user │ upvotes │ userUpvoted │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 599f89639a11040c4a163440 │ Here is a video of some cats in zero gravity. yout... │ cat │ _id: "5a9ac18c7478810ea6c06381" │ 5 │ null │
│ │ │ │ name: {"first":"Alex","last":"Wohlbruck"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 591d9b2f227c1a0020d26823 │ Every year, nearly four million cats are eaten in ... │ cat │ _id: "5a9ac18c7478810ea6c06381" │ 4 │ null │
│ │ │ │ name: {"first":"Alex","last":"Wohlbruck"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 5b3bd7d24cf4e10014bfd199 │ The myth that a cat has nine lives comes from thei... │ cat │ _id: "5a9ac18c7478810ea6c06381" │ 3 │ null │
│ │ │ │ name: {"first":"Alex","last":"Wohlbruck"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 5b453e380fd3a600147f32f3 │ Exposure to UV light with hairless or partially-ha... │ cat │ _id: "5a9ac18c7478810ea6c06381" │ 3 │ null │
│ │ │ │ name: {"first":"Alex","last":"Wohlbruck"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 590b9d90229d260020af0b06 │ Evidence suggests domesticated cats have been arou... │ cat │ _id: "5a9ac18c7478810ea6c06381" │ 2 │ null │
│ │ │ │ name: {"first":"Alex","last":"Wohlbruck"} │ │ │
└──────────────────────────┴───────────────────────────────────────────────────────┴──────┴───────────────────────────────────────────┴─────────┴─────────────┘
Filter only records that belong to Kasimir :
zoe --cluster local \
--output table \
topics consume \
-n 10 \
--from 'PT6h' \
--filter "user.name.first == 'Kasimir'"
┌──────────────────────────┬───────────────────────────────────────────────────────┬──────┬───────────────────────────────────────────┬─────────┬─────────────┐
│ _id │ text │ type │ user │ upvotes │ userUpvoted │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 58e009390aac31001185ed10 │ Most cats are lactose intolerant, and milk can cau... │ cat │ _id: "58e007480aac31001185ecef" │ 2 │ null │
│ │ │ │ name: {"first":"Kasimir","last":"Schulz"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 58e007cc0aac31001185ecf5 │ Cats are the most popular pet in the United States... │ cat │ _id: "58e007480aac31001185ecef" │ 2 │ null │
│ │ │ │ name: {"first":"Kasimir","last":"Schulz"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 58e008c50aac31001185ed0e │ The world's richest cat is worth $13 million after... │ cat │ _id: "58e007480aac31001185ecef" │ 2 │ null │
│ │ │ │ name: {"first":"Kasimir","last":"Schulz"} │ │ │
├──────────────────────────┼───────────────────────────────────────────────────────┼──────┼───────────────────────────────────────────┼─────────┼─────────────┤
│ 58e009790aac31001185ed14 │ The technical term for "hairball" is "bezoar." │ cat │ _id: "58e007480aac31001185ecef" │ 4 │ null │
│ │ │ │ name: {"first":"Kasimir","last":"Schulz"} │ │ │
└──────────────────────────┴───────────────────────────────────────────────────────┴──────┴───────────────────────────────────────────┴─────────┴─────────────┘
Do the same search as before but against my production kafka cluster and by offloading the consumption to kubernetes and spinning up 10 pods in parallel :
zoe --cluster my-production-kafka \
--output table \
--runner kubernetes \
topics consume \
-n 10 \
--from 'PT6h' \
--filter "user.name.first == 'Kasimir'" \
--jobs 10
Display the offsets of my consumer group my-group :
zoe -o table -v -c my-cluster groups offsets my-consumer-group-name
┌──────────┬───────────┬───────────────┬───────────┬─────┬──────────┐
│ topic │ partition │ currentOffset │ endOffset │ lag │ metadata │
├──────────┼───────────┼───────────────┼───────────┼─────┼──────────┤
│ my-topic │ 3 │ 49993793 │ 49993793 │ 0 │ │
├──────────┼───────────┼───────────────┼───────────┼─────┼──────────┤
│ my-topic │ 4 │ 50275990 │ 50275991 │ 1 │ │
├──────────┼───────────┼───────────────┼───────────┼─────┼──────────┤
│ my-topic │ 1 │ 50155761 │ 50155762 │ 1 │ │
├──────────┼───────────┼───────────────┼───────────┼─────┼──────────┤
│ my-topic │ 2 │ 50212574 │ 50212574 │ 0 │ │
├──────────┼───────────┼───────────────┼───────────┼─────┼──────────┤
│ my-topic │ 0 │ 50027346 │ 50027347 │ 1 │ │
└──────────┴───────────┴───────────────┴───────────┴─────┴──────────┘
Maintainers
Contributors
Contributor's guide coming soon.