Let's talk a bit about event-driven microservices. Event streaming is a really powerful substrate for microservices, so it sees a lot of use, but it comes with caveats that are worth exploring before you fully commit to the event streaming model.
An event-driven microservice subscribes to messages several topics in a distributed log (probably Kafka) and publishes to several other topics. It processes data asynchronously and batches its I/O. A Kafka broker gives you service discovery, load balancing, horizontal scaling, fault tolerance, guaranteed delivery, and best-in-class throughput; it's a pretty good deal.
All these benefits come bundled together, however, which makes event streaming contagious, in the way that async functions are contagious. In order for an event-driven service to processes messages at speed, it can't perform a separate set of network round-trips for each one. It shouldn't be calling APIs or querying a database; rather, it should be ingesting streamed versions of those things. Any synchronous operations will block the pipeline and throw the service's performance off a cliff. The service's output also has to be asynchronous, which propagates all these constraints to downstream services, too.
So asynchronous streams can easily spread all through your system, and for every problem stream processing solves, it creates a new one. Synchronous APIs will respond to clients with errors if they get too busy, but an asynchronous pipeline will gladly fill its disks with an enormous backlog of work. You have to either set up backpressure or be very careful about how quickly you add to your queue. Also, Kafka forces you to handle queued messages in order, so things like exponential backoff become difficult to implement in practice. In order to reprocess a message at a later time, you have to forward that message to a "retry" topic which circles back to an earlier stage in your pipeline. Now you have copies of the same message hanging around in both the normal topic and the retry topic.
This is my biggest problem with event streams, actually: You often end up with more than one canonical representations of the same domain entity. With a conventional database-backed service, you can check the current state of an entity by querying the relevant database table. Meanwhile, in order to completely understand the status of an entity in a streaming system, you'll have to trawl through a laundry list of topics, each representing a different view of the entity, or a different stage of processing for the entity, or a join between that entity and something else.
To be clear, I like Kafka. It's a really powerful tool. My point is that legibility is as much of a consideration when designing a system as performance is. Asynchronous systems perform better, but synchronous systems are often easier to reason about, because they can maintain a single source of truth for each entity in their domain. You can balance these properties and get the benefits of both paradigms, but that requires deliberate boundaries between the synchronous and asynchronous parts of your system.
Patterns for fitting synchronous and asynchronous systems together:
-
Workflow engine + worker nodes. A scheduler with a database can serve as a synchronous source of truth while dispatching, monitoring, and retrying asynchronous tasks across a cluster of workers. This centralizes application state while remaining highly scalable.
Also, a dynamic scheduler will easily be able to describe workflow topologies and error-handling flows that would be too complex to represent with static Kafka topics.
-
If you do want a conventional streaming pipeline, a synchronous control plane can be a great enhancement. The control plane can handle observability and configuration concerns, trigger replays, and even answer debugging queries about the internal states of stream operators.