STREAMING ARCHITECTURE 101_
I spent three weeks building a Kinesis pipeline before I understood what streaming actually was. I had read the docs. I had watched the re:Invent talks. I still got it wrong.
Here is what I wish someone had told me first.
> THE REQUEST/RESPONSE TRAP
Most engineers learn to build systems around one contract: you ask, something answers. HTTP. RPC. SQL. The caller waits. The response comes back. Done.
This model is intuitive. It is also a ceiling.
When your order service calls your inventory service synchronously, you have created a dependency. If inventory is slow, orders are slow. If inventory is down, orders are down. You have coupled two systems at the worst possible layer — runtime.
Streaming breaks that coupling.
> THE MENTAL MODEL SHIFT
Stop thinking about calling services. Start thinking about emitting facts.
An event is an immutable record of something that happened. order.placed. payment.processed. inventory.reserved. These are not requests — they are statements. The producer does not care who is listening. The consumer does not care who produced the event.
The log becomes the source of truth. Not the database. The database is a projection — a materialized view of event history. This one takes time to internalize.
Time becomes a first-class citizen. Events have timestamps. You can replay them. You can reprocess history with new logic. You cannot do that with a REST call that happened six months ago.
> THE PATTERNS
EVENT SOURCING — State is derived from the event log, not stored directly. Your user's current address is the result of every address.updated event applied in sequence. Powerful for audit trails and temporal queries. Expensive to implement correctly.
CQRS — Command Query Responsibility Segregation. Separate your write model from your read model. Commands mutate state and emit events. Queries read from optimized projections. Not always necessary, but it pairs naturally with event sourcing.
PUB/SUB — Producers publish to a topic. Consumers subscribe. Neither knows about the other. SNS, EventBridge, Kafka. The simplest form of streaming decoupling. Start here.
> WHEN NOT TO USE STREAMING
This is the part nobody writes about.
Do not stream a simple CRUD app. If your users are creating records and reading them back, a database and a REST API are the right tools. Adding Kinesis to a to-do list app is not architecture — it is resume-driven development.
Do not stream when you need strong consistency. Eventual consistency is a feature of streaming systems. If your business logic requires that a read immediately reflects a write, streaming will fight you.
Do not stream when your team cannot operate it. Kinesis shards, consumer lag, dead-letter queues, poison pill messages — these are real operational concerns. If nobody on your team has dealt with them before, you will learn on production traffic. That is a bad classroom.
> PRACTICAL FIRST STEPS
Start with SQS before you touch Kinesis. SQS is simpler, cheaper, and handles most use cases. Kinesis is for ordered, high-throughput streams where consumer position matters. Most applications do not need that.
Get your event schema right early. Changing an event schema after consumers are in production is painful. Use a schema registry. Version your events. Treat them like a public API — because they are.
Pick one bounded context and stream it. Not the whole system. One domain, one event type, one consumer. Prove the pattern works in your environment before you commit to it everywhere.
The over-engineered Kinesis pipeline I built in week one? I replaced it with SQS in week four. The business logic did not change. The operational complexity dropped by half.