Kafka or RabbitMQ?
When scaling the application horizontally or vertically is not enough to handle the tremendous traffic that will come to your application, the choice to use a message broker/queue comes to mind.
Of all the available options, the choice always stops at these two due to the mature support of Kafka and RabbitMQ.
Although many blogs will praise Kafka and sometimes RabbitMQ, choosing the right option solely depends on the use case or architecture of the system you are dealing with. To understand them better, let’s try to understand the characteristics of both.
RabbitMQ
It is a message broker which uses queues to store messages created by “Producers”. The messages placed in the queue can be consumed by "Consumers" applications. The queue is non-persistent, and messages are removed with the acknowledgement. Another characteristic of the queue is that it is single-threaded and can only scale up to an extent.
Let's try to understand this by example.
Suppose we are making an API which triggers a process to store some data on the blockchain. Ignoring the state of the transaction(success/failure/pending) and with the scalability of the API, let’s assume that it can process and submit a transaction in 20 seconds. The data being stored is a picture coming from another service at the rate of 10 per minute.
So, the service can handle three requests per minute (60/20). But we have to handle the load of 10 requests per minute.
That’s a good use-case where we can use RabbitMQ:
RabbitMQ also provides Smart Routing that enables to define of prefixes and routing of the requests to multiple queues, which can be pulled/pushed to different consumers.
RabbitMQ queues are single-threaded. So, the only scope of multithreading is possible at the consumer level, but you can use pre-fetch to restrict load. It simply means how many messages the queue can throw to consumers without getting acknowledged. Now, it's on the consumer to handle this prefetch load correctly.
Kafka
Per the official website, Kafka is said to be an event streaming platform distributed in nature. So why are we comparing it with RabbitMQ, a message broker? The answer is that it is an enhanced version of a message queue/broker capable of more throughput, which can even manage the high data load in the form of consistent streams.
In Kafka terminologies, it contains topics, a log of events, that a consumer can subscribe to listen to. As it is distributed in nature, the different topics can be replicated inside a Kafka cluster for horizontal scaling.
It works on ‘Pub-Sub’ Model, contrary to ‘Push-Pull’ Model of RabbitMQ. Another advantage of Kafka is that it stores messages for a more extended period of configurable time, compared to RabbiqMQ queues, in which messages are removed once played. Because of the storage of logs, Kafka can replay the messages.
So if your system is simple enough and has few direct consumers that don’t need to handle high throughput of data, RabbitMQ is a good choice, but if your system needs to be a step ahead and work with high throughput streams of data and have multiple consumers wanting to subscribe to different topics, using Kafka is a good option.
If you want to know more about these two, Vanlightly’s blog is the best resource, so I don’t need to make this article lengthy and compare.
https://jack-vanlightly.com/blog/2017/12/3/rabbitmq-vs-kafka-series-introduction#:~:text=The%20hype%20has%20been%20centered,scale%20where%20RabbitMQ%20has%20problems.
