<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Understanding Systems]]></title><description><![CDATA[Exploring software architecture, distributed systems, and the engineering decisions behind modern software.]]></description><link>https://understandingsystems.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6a64c35c24e2815e3b168429/74f07ac3-a408-4022-be73-e93e4acc4aa9.png</url><title>Understanding Systems</title><link>https://understandingsystems.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 20:03:38 GMT</lastBuildDate><atom:link href="https://understandingsystems.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Why RabbitMQ Instead of Kafka? Choosing the Right Tool for the Problem]]></title><description><![CDATA[Learn why RabbitMQ was the right fit for my distributed scientific data pipeline and how to choose between RabbitMQ and Kafka.]]></description><link>https://understandingsystems.hashnode.dev/why-rabbitmq-instead-of-kafka-choosing-the-right-tool-for-the-problem</link><guid isPermaLink="true">https://understandingsystems.hashnode.dev/why-rabbitmq-instead-of-kafka-choosing-the-right-tool-for-the-problem</guid><category><![CDATA[distributed systems]]></category><category><![CDATA[rabbitmq]]></category><category><![CDATA[Apache Kafka]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[System Design]]></category><dc:creator><![CDATA[Jay Solanki]]></dc:creator><pubDate>Sat, 25 Jul 2026 18:06:30 GMT</pubDate><content:encoded><![CDATA[<blockquote>
<p><strong>Series:</strong> Building a Distributed Scientific Data Pipeline (Part 2)</p>
</blockquote>
<p>In the previous article, I explored why my Distributed Scientific Data Pipeline (DSDP) needed a message broker in the first place.</p>
<p>That answered one architectural question.</p>
<p>The next one was much harder.</p>
<blockquote>
<p><strong>If we need a message broker, why RabbitMQ instead of Kafka?</strong></p>
</blockquote>
<p>At first, I assumed RabbitMQ and Kafka were simply two different tools that solved the same problem.</p>
<p>The more I learned, the more I realized that assumption was wrong.</p>
<p>When I first started learning distributed systems, I assumed RabbitMQ and Kafka were simply two different tools that solved the same problem.</p>
<p>The more I learned, the more I realized that assumption was wrong.</p>
<p>Both enable communication between independent services.</p>
<p>But they were designed to solve fundamentally different architectural problems.</p>
<p>The real question isn't:</p>
<blockquote>
<p><strong>Which technology is better?</strong></p>
</blockquote>
<p>The better question is:</p>
<blockquote>
<p><strong>Which problem am I trying to solve?</strong></p>
</blockquote>
<p>That shift in thinking completely changed how I evaluate technologies.</p>
<hr />
<h2>RabbitMQ Thinks in Tasks</h2>
<p>Imagine you're running a bakery.</p>
<p>Customers place orders.</p>
<p>The baker prepares them.</p>
<p>Once an order is completed, everyone moves on.</p>
<pre><code class="language-text">                           Customer

                              ↓

                         Order Queue

                              ↓

                            Baker

                              ↓

                        Order Completed
</code></pre>
<p>The purpose of the queue is simple.</p>
<p>Deliver work reliably.</p>
<p>Once the work has been completed, the queue doesn't need to remember it forever.</p>
<p>RabbitMQ follows the same philosophy.</p>
<p>A producer publishes work.</p>
<p>RabbitMQ stores that work until a consumer is ready.</p>
<p>Once the message has been processed successfully, RabbitMQ can safely remove it.</p>
<p>Its job is reliable delivery.</p>
<hr />
<h2>Kafka Thinks in Events</h2>
<p>Kafka approaches the problem very differently.</p>
<p>Instead of asking:</p>
<blockquote>
<p>"Who should process this task?"</p>
</blockquote>
<p>Kafka asks:</p>
<blockquote>
<p>"How can we preserve the history of everything that happened?"</p>
</blockquote>
<p>Imagine a notebook.</p>
<p>Every new event gets written onto the next page.</p>
<pre><code class="language-text">Event 1

Event 2

Event 3

Event 4

Event 5

...
</code></pre>
<p>Nothing gets erased.</p>
<p>Different applications can read from different places.</p>
<p>One application might be reading today's events.</p>
<p>Another might replay yesterday's.</p>
<p>Another could start from the very beginning.</p>
<p>Kafka isn't primarily designed to deliver tasks.</p>
<p>It's designed to preserve an ordered stream of events.</p>
<p>That's why it's often described as an <strong>event streaming platform</strong> rather than simply a message broker.</p>
<hr />
<h2>Imagine Four Different Teams</h2>
<p>Suppose you're building YouTube.</p>
<p>Every second, millions of events are generated.</p>
<pre><code class="language-text">User watched a video

User liked a video

User searched "Docker"

User subscribed

User commented
</code></pre>
<p>Now imagine four different teams inside the company.</p>
<h3>Team 1</h3>
<p>Recommendation Engine</p>
<p>Needs every viewing event.</p>
<hr />
<h3>Team 2</h3>
<p>Analytics</p>
<p>Needs every user interaction.</p>
<hr />
<h3>Team 3</h3>
<p>Fraud Detection</p>
<p>Needs every login event.</p>
<hr />
<h3>Team 4</h3>
<p>Marketing</p>
<p>Needs search history and user behavior.</p>
<p>Now ask yourself.</p>
<p>Should Team 1 process an event and make it disappear before everyone else can see it?</p>
<p>Of course not.</p>
<p>Every team needs access to exactly the same history.</p>
<p>This is where Kafka shines.</p>
<p>Multiple independent systems can consume the same stream of events without interfering with one another.</p>
<hr />
<h2>Why Not Just Store Everything in a Database?</h2>
<p>This question confused me for quite a while.</p>
<p>If every event eventually ends up inside a database anyway...</p>
<p>Why keep another copy inside Kafka?</p>
<p>The answer lies in understanding the difference between <strong>state</strong> and <strong>history</strong>.</p>
<p>Imagine a user's email address.</p>
<p>Today the database stores:</p>
<pre><code class="language-text">jay@company.com
</code></pre>
<p>But what was the previous email?</p>
<p>Unless you've built an auditing system, that information is usually gone.</p>
<p>Kafka records something completely different.</p>
<pre><code class="language-text">                         User Created

                               ↓

                         Email Changed

                               ↓

                        Password Changed
   
                               ↓

                      Subscription Upgraded

                               ↓

                     Subscription Cancelled
</code></pre>
<p>Instead of storing only the latest state, Kafka preserves every event that produced that state.</p>
<p>That history becomes incredibly valuable for analytics, auditing, event sourcing, rebuilding systems, and machine learning pipelines.</p>
<hr />
<h2>Why RabbitMQ Was the Better Choice for This Project</h2>
<p>Once I understood what Kafka was actually designed to do, the decision became much simpler.</p>
<p>My Distributed Scientific Data Pipeline doesn't require multiple independent consumer groups.</p>
<p>It doesn't require replaying events months later.</p>
<p>It doesn't require maintaining a permanent event log.</p>
<p>Its architecture is much simpler.</p>
<pre><code class="language-text">                     Synthetic Detector
                              ↓
        
                           RabbitMQ
                              ↓

                      Processing Worker
                              ↓

                         TimescaleDB
</code></pre>
<p>Each component has a single responsibility:</p>
<ul>
<li><p>The detector generates scientific events.</p>
</li>
<li><p>RabbitMQ reliably delivers those events.</p>
</li>
<li><p>The processing worker consumes and processes them.</p>
</li>
<li><p>TimescaleDB stores them permanently.</p>
</li>
</ul>
<p>Once an event reaches the database, it already has a permanent home.</p>
<p>RabbitMQ doesn't need to store it forever.</p>
<p>Its responsibility ends once reliable delivery has been achieved.</p>
<p>Introducing Kafka would have added another system responsible for storing the same events.</p>
<p>That duplication makes perfect sense when you need event replay, multiple independent consumers, or a long-lived event log.</p>
<p>My architecture required none of those capabilities.</p>
<p>Adding Kafka would have increased operational complexity without solving an actual problem.</p>
<p>That duplication makes perfect sense for many architectures.</p>
<p>It simply wasn't solving a problem my system actually had.</p>
<hr />
<h2>The Bigger Lesson</h2>
<p>Before this project, I thought choosing technologies meant comparing features.</p>
<p>RabbitMQ vs Kafka.</p>
<p>PostgreSQL vs MongoDB.</p>
<p>Docker vs Podman.</p>
<p>Now I approach those questions differently.</p>
<p>I first ask:</p>
<blockquote>
<p><strong>What problem is my architecture trying to solve?</strong></p>
</blockquote>
<p>Only after I understand the problem do I begin comparing technologies.</p>
<p>That simple shift has changed the way I learn software engineering.</p>
<p>Instead of collecting tools, I'm learning the reasoning behind them.</p>
<hr />
<h2>Final Thoughts</h2>
<p>One of the biggest lessons I learned while building this project is that software architecture isn't about choosing the most powerful technology.</p>
<p>It's about choosing the technology that best fits the problem you're solving.</p>
<p>RabbitMQ wasn't the "best" messaging technology.</p>
<p>It was simply the best fit for this architecture.</p>
<p>RabbitMQ wasn't the "best" messaging technology.</p>
<p>It was simply the best fit for this architecture.</p>
<p>That distinction changed the way I think about software engineering.</p>
<p>Today, when I evaluate a new technology, I don't begin by asking:</p>
<blockquote>
<p><strong>"What can this technology do?"</strong></p>
</blockquote>
<p>Instead, I ask:</p>
<blockquote>
<p><strong>"What problem was this technology designed to solve?"</strong></p>
</blockquote>
<p>Once that answer is clear, choosing between different technologies becomes much less about trends and much more about engineering.</p>
<hr />
<h2>Key Takeaway</h2>
<blockquote>
<p><strong>Good architecture doesn't begin with technologies.</strong></p>
<p><strong>It begins with understanding the problem.</strong></p>
</blockquote>
<hr />
<blockquote>
<p><strong>This article is part of my <em>Building a Distributed Scientific Data Pipeline</em> series, where I document the architectural decisions, engineering trade-offs, and distributed systems concepts I encounter while building a real-world software project from first principles.</strong></p>
</blockquote>
<p><strong>Next article:</strong> <em>What Actually Happens When a Producer Publishes a Message to RabbitMQ?</em></p>
]]></content:encoded></item><item><title><![CDATA[Why We Chose RabbitMQ Instead of Kafka for Our Distributed Scientific Data Pipeline ]]></title><description><![CDATA[Understanding the problem before choosing the technology.]]></description><link>https://understandingsystems.hashnode.dev/why-we-chose-rabbitmq-instead-of-kafka-for-our-distributed-scientific-data-pipeline</link><guid isPermaLink="true">https://understandingsystems.hashnode.dev/why-we-chose-rabbitmq-instead-of-kafka-for-our-distributed-scientific-data-pipeline</guid><category><![CDATA[distributed systems]]></category><category><![CDATA[rabbitmq]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[System Design]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[Jay Solanki]]></dc:creator><pubDate>Sat, 25 Jul 2026 17:09:14 GMT</pubDate><content:encoded><![CDATA[<p><strong>Series:</strong> Building a Distributed Scientific Data Pipeline (Part 1)</p>
<blockquote>
<p>Every distributed system begins with a series of engineering decisions. This series documents the architectural reasoning behind the Distributed Scientific Data Pipeline, focusing not just on how technologies work, but why they were chosen.</p>
</blockquote>
<blockquote>
<p><strong>While</strong> building my <strong>Distributed Scientific Data Pipeline (DSDP)</strong>, I reached a point where multiple independent services needed to communicate with one another.</p>
</blockquote>
<p><strong>At first, the obvious solution seemed to be connecting them directly. But as the system grew, a simple question emerged:</strong></p>
<blockquote>
<p><strong>How should these services communicate without becoming tightly coupled?</strong></p>
</blockquote>
<p>As I explored different approaches, RabbitMQ appeared in almost every discussion about asynchronous communication. Most resources focused on how to use RabbitMQ—how to install it, publish messages, and consume them.</p>
<p>But I found myself asking a different question.</p>
<blockquote>
<p><strong>Why RabbitMQ?</strong></p>
</blockquote>
<p>Why not Kafka?</p>
<p>Why not Redis Streams?</p>
<p>Why not let one service call another directly?</p>
<p>I realized that if I couldn't answer those questions, I wasn't really learning RabbitMQ—I was only learning its API.</p>
<p>This article isn't a RabbitMQ tutorial.</p>
<p>Instead, it's the reasoning process that led me to choose a message broker in the first place.</p>
<hr />
<h2>Thinking From First Principles</h2>
<p>Before introducing any technology, let's remove it completely.</p>
<p>Imagine RabbitMQ doesn't exist.</p>
<p>Imagine Kafka doesn't exist.</p>
<p>Our system consists of two components:</p>
<ul>
<li><p>A detector that generates scientific events.</p>
</li>
<li><p>A worker that processes those events.</p>
</li>
</ul>
<pre><code class="language-text">Detector
    │
    V
Worker
</code></pre>
<p>The implementation might look something like this:</p>
<pre><code class="language-python">event = detector.generate_event()
worker.process(event)
</code></pre>
<p>At first glance, this seems perfectly reasonable.</p>
<p>The detector generates an event.</p>
<p>The worker processes it.</p>
<p>Simple.</p>
<p>So why introduce another component at all?</p>
<hr />
<h2>The Problem</h2>
<p>Let's imagine the detector generates <strong>100 events every second</strong>.</p>
<p>Our worker can also process <strong>100 events every second</strong>.</p>
<p>Everything works smoothly.</p>
<p>Now imagine the detector becomes much faster.</p>
<p>Instead of generating 100 events every second, it now produces <strong>10,000 events every second</strong>.</p>
<p>Unfortunately, our worker hasn't changed.</p>
<p>It can still process only 100 events per second.</p>
<p>Immediately, we have a problem.</p>
<pre><code class="language-text">Detector
████████████████████

Worker
██
</code></pre>
<p>The detector is producing events much faster than the worker can consume them.</p>
<p>If the detector directly calls:</p>
<pre><code class="language-python">worker.process(event)
</code></pre>
<p>the worker quickly becomes overwhelmed.</p>
<p>The detector must wait.</p>
<p>And that's where the design begins to break down.</p>
<hr />
<h2>Why Waiting Is a Problem</h2>
<p>Imagine the detector is connected to a real scientific experiment.</p>
<p>It continuously generates observations from the physical world.</p>
<p>Those observations don't stop simply because our software is busy.</p>
<p>You can't tell an experiment:</p>
<blockquote>
<p>"Please stop collecting data for a few minutes while the processing service catches up."</p>
</blockquote>
<p>Reality doesn't wait for software.</p>
<p>The producer of information should never have to slow down because another component is temporarily overloaded.</p>
<p>This means the detector and the worker shouldn't depend on each other's processing speed.</p>
<p>They need to become independent.</p>
<hr />
<h2>What Does the System Actually Need?</h2>
<p>At this point, I stopped thinking about RabbitMQ and started thinking about the problem.</p>
<p>The system needed a way to:</p>
<ul>
<li><p>Allow producers and consumers to work independently.</p>
</li>
<li><p>Prevent data loss when processing falls behind.</p>
</li>
<li><p>Handle temporary spikes in workload.</p>
</li>
<li><p>Scale by adding more workers when necessary.</p>
</li>
<li><p>Keep services loosely coupled.</p>
</li>
</ul>
<p>Only after identifying these requirements did it make sense to evaluate different technologies.</p>
<hr />
<h2>We Need Something in the Middle</h2>
<p>Instead of sending events directly to the worker, imagine placing another component between them.</p>
<pre><code class="language-plaintext"> Detector
    │
    V

    ?

    │
    V
 Worker
</code></pre>
<p>What should this mysterious component do?</p>
<p>Ideally, it should tell the detector:</p>
<blockquote>
<p>"I'll accept your event immediately."</p>
</blockquote>
<p>Then, independently, it should tell the worker:</p>
<blockquote>
<p>"Process this whenever you're ready."</p>
</blockquote>
<p>Now the detector doesn't care whether the worker is busy.</p>
<p>It simply hands over the event and continues generating the next one.</p>
<p>The worker processes events at its own pace.</p>
<p>This simple idea is the foundation of asynchronous communication.</p>
<hr />
<h2>A Bakery Analogy</h2>
<p>Imagine a bakery with only one baker.</p>
<p>Without an order queue, every customer has to wait until the baker finishes the current order.</p>
<pre><code class="language-text">Customer
   │
   V
 Baker
</code></pre>
<p>Now imagine introducing an order queue.</p>
<pre><code class="language-text">Customers
    │
    V
Order Queue
    │
    V
 Baker
</code></pre>
<p>Customers place their orders and leave.</p>
<p>The baker completes one order at a time.</p>
<p>Nobody blocks anyone else.</p>
<p>The queue absorbs the difference between how quickly orders arrive and how quickly they can be completed.</p>
<p>Software systems face the same challenge.</p>
<hr />
<h2>Enter RabbitMQ</h2>
<p>This is precisely the problem RabbitMQ is designed to solve.</p>
<p>Instead of forcing the detector and the worker to communicate directly, RabbitMQ sits between them.</p>
<pre><code class="language-text">Detector
    │
    V
RabbitMQ
    │
    V
Worker
</code></pre>
<p>When the detector generates an event, RabbitMQ immediately accepts it.</p>
<p>The detector is free to continue generating more events.</p>
<p>Meanwhile, RabbitMQ safely stores those events until a worker is ready to process them.</p>
<p>The worker processes messages at its own pace without slowing down the detector.</p>
<p>Instead of tightly coupling two services together, RabbitMQ acts as a reliable intermediary.</p>
<p>Each component has a single responsibility:</p>
<ul>
<li><p>The detector produces events.</p>
</li>
<li><p>RabbitMQ delivers them reliably.</p>
</li>
<li><p>The worker processes those events and stores the results.</p>
</li>
</ul>
<p>Each service can now evolve, scale, and fail independently.</p>
<hr />
<h2>Lessons Learned</h2>
<p>The interesting realization wasn't that RabbitMQ was the answer.</p>
<p>The real realization was that I had been asking the wrong question.</p>
<p>I wasn't trying to choose between RabbitMQ and Kafka.</p>
<p>I first needed to understand whether the system required a message broker at all.</p>
<p>Once that became clear, evaluating different technologies became much simpler.</p>
<hr />
<h2>What's Next</h2>
<p>We've answered one important architectural question:</p>
<blockquote>
<p><strong>Why introduce a message broker in the first place?</strong></p>
</blockquote>
<p>The next question is equally important:</p>
<blockquote>
<p><strong>If RabbitMQ solves this problem, why not Kafka?</strong></p>
</blockquote>
<hr />
<blockquote>
<p><strong>This article is part of my <em>Building a Distributed Scientific Data Pipeline</em> series, where I document the architectural decisions, engineering trade-offs, and distributed systems concepts I encounter while building a real-world software project from first principles.</strong></p>
</blockquote>
]]></content:encoded></item></channel></rss>