It’s Cuter Than Kafka: MQTT
I’ve been prototyping distributed data sharing and needed a pub/sub tool that would be easy to work with, understandable, and reliable and came across MQTT. With the huge message payload size (256MB) you can transfer, for example, dbt models and json payloads without needing to use any other protocol. This isn’t to say that not all roads lead to Kafka, but it is nice to have a tool that is quick to deploy, easy to understand, has some inherent scalability, and because of it’s barebones nature allows you to really grasp what problems a more heavyweight tool like Kafka is solving. Once I started working on a service for storing messages into SQLite, thinking about ways to store a global log, and who’s the authoritative source it’s become more apparent Kafka will eventually become a part of this project. But for now I’ll enjoy my cute prototype.