23. Kafka as Messaging Middleware¶
23.1. Context and Problem Statement¶
To compose service with the mico application we need a messaging middleware that is highly scalable and supports the Enterprise Integration Patterns.
23.2. Decision Drivers¶
- Should be a well known and proven solution
- Support a pipes and filters style architecture
23.3. Considered Options¶
23.3.1. Apache Kafka¶
Pro: Proven system with high scalability. Well suited for pipes and filters architectures.
Contra: Topics have different semantics than traditional message queues. Some patterns don’t work well/at all with the topic semantic.
23.3.2. Some other messaging middleware¶
This option was not explored further.
23.4. Decision Outcome¶
We want to use Kafka as our messaging middleware.
23.4.1. Important things to know about Kafka¶
Kafka topics have a different semantic than traditional message queues. Kafka also does NOT log wether a message was consumed. It only stores the last comitted offset for a consumer. Because of that and the different topic semantic some patterns need to be adapted to work with Kafka.
For more detailed information about the inner workings of Kafka consult the following links: