System Architecture

Use When Recommended

• Your team is fewer than ~20 engineers
• You're building a new product / MVP
• Domain boundaries are unclear
• You want simple local development and debugging
• You need fast iteration speed
• Deployment simplicity matters (one artifact, one pipeline)

Don't Use When Caution

• Multiple teams need independent deployment cadences
• Parts of the system need radically different scaling profiles (e.g., CPU-heavy image processing vs I/O-heavy API)
• Regulatory/compliance requires strict service isolation
• You genuinely have 100+ engineers on one codebase and build times are measured in hours

Network Latency

In-process function calls take nanoseconds. Network calls take milliseconds. A request that touches 5 services adds 5x network round trips. Multiply by retries and you have a latency problem.

Distributed Transactions

No more simple database transactions. You need sagas, compensating transactions, eventual consistency, and idempotency everywhere. This is genuinely hard to get right.

Operational Complexity

You need: service discovery, distributed tracing, centralized logging, health checks, circuit breakers, container orchestration, CI/CD per service, contract testing, API versioning.

Debugging

A bug that spans 3 services requires correlating logs across systems, understanding async message flows, and reasoning about eventual consistency states. Stack traces don't cross network boundaries.

Event Sourcing

Store every state change as an immutable event. Current state = replay all events. Gives you a complete audit log and the ability to rebuild state at any point in time.

Hard Increases storage, requires snapshots for performance, eventual consistency.

CQRS

Command Query Responsibility Segregation — separate the write model (commands) from the read model (queries). Write to a normalized store, project into denormalized read views.

When Read/write patterns differ drastically. Don't use for simple CRUD.

Event Bus

A central channel (Kafka, RabbitMQ, EventBridge) where producers emit events and consumers subscribe. Decouples producers from consumers completely.

Enables Loose coupling, independent scaling, async processing.

FaaS Functions as a Service

AWS Lambda, Google Cloud Functions, Azure Functions. Deploy individual functions triggered by events (HTTP, queue, schedule). Auto-scales to zero.

BaaS Backend as a Service

Firebase, Supabase, AWS Amplify. Managed services for auth, database, storage, push notifications. Write frontend code, let the platform handle the backend.

REST Wins When

• Building public/external APIs
• HTTP caching is important
• Team is familiar with REST
• Simple CRUD operations
• Browser clients are primary

gRPC Wins When

• Service-to-service communication
• Low latency required
• Bidirectional streaming needed
• Strict typing via Protobuf
• Polyglot services (code gen)

GraphQL Wins When

• Multiple client types (mobile, web)
• Clients need flexible queries
• Avoiding over-fetching matters
• Aggregating multiple backends
• Rapid frontend iteration

Fire-and-Forget

Producer sends and moves on. No response expected. Best for notifications, logging, analytics events.

Simple No confirmation

Request-Reply (Async)

Producer sends a message with a correlation ID and reply-to queue. Consumer processes and replies. Good for distributed RPC over messaging.

Decoupled Complex

Backpressure

When consumers can't keep up, the system must handle it: drop messages, buffer, slow down producers, or scale consumers. Kafka handles this via consumer lag; RabbitMQ via queue limits.

Critical at scale

Pick RabbitMQ when

• You need complex routing (topic, header, fanout exchanges)
• Task distribution with acknowledgments
• Request-reply pattern over messaging
• Message priority support needed
• Moderate throughput is sufficient

Pick Kafka when

• You need event replay / reprocessing
• High throughput event streaming (100K+ events/s)
• Change data capture (CDC) pipelines
• Multiple consumers reading same events independently
• Event sourcing / audit trail

Pick SQS when

• You're on AWS and want zero ops overhead
• Simple decoupling between services
• Dead letter queue for failed messages
• You don't need replay or consumer groups

Pick NATS when

• Ultra-low latency required
• Edge / IoT deployments
• Simple pub/sub without persistence (core NATS)
• Lightweight footprint (single binary, no JVM)

mTLS (Mutual TLS)

Automatic encryption and identity verification between all services. Zero-trust networking without changing application code.

Observability

Automatic distributed tracing, metrics (latency, error rates, throughput) for every service-to-service call. No instrumentation code needed.

Traffic Management

Canary deployments, A/B testing, traffic shifting, header-based routing, fault injection for chaos testing.

Resilience

Retries, timeouts, circuit breakers, rate limiting — all configured via policies, not code.

Good Fit

• Multiple teams own different parts of the graph
• Frontend needs to query across domains in one request
• You already have GraphQL and want to split a monolith graph
• Schema governance is important

Avoid When

• You have fewer than 3 services (use a single GraphQL server)
• Team doesn't have GraphQL expertise
• REST aggregation with a BFF would be simpler
• Performance is critical (federation adds latency for cross-subgraph queries)

Cache-Aside (Lazy Loading) Most Common

App checks cache first. On miss, reads from DB and writes to cache. App manages both reads and writes. Simple, good for read-heavy workloads.

Risk: Cache and DB can drift. Use TTL to bound staleness.

Write-Through

Every write goes to cache AND database synchronously. Cache is always up-to-date. Higher write latency (two writes per operation).

Best for: Data that's read immediately after writing.

Write-Behind (Write-Back)

Write to cache immediately, asynchronously flush to database. Very fast writes. Risk of data loss if cache node fails before flush.

Best for: Write-heavy workloads where some data loss is acceptable.

Read-Through

Cache itself is responsible for loading from DB on miss (cache acts as the data source). Simplifies app code but requires cache library support.

Best for: When you want the cache layer to handle all data loading logic.

Idempotency Critical

If a request might be retried, the operation MUST be idempotent (same result if called multiple times). Use idempotency keys: client sends a unique key, server deduplicates. Essential for payment operations.

Timeout Cascades

Service A calls B calls C. If C is slow, B waits, A waits. Set timeouts at each level, with outer timeouts greater than inner ones. Example: A timeout = 5s, B timeout to C = 3s. This prevents thread pool exhaustion.

Bulkhead Pattern

Isolate different call types into separate thread pools or connection pools. A slow dependency in pool A doesn't exhaust resources for pool B. Like ship bulkheads — one leak doesn't sink the ship.

Retry Budget

Limit total retries across the system. "Retry at most 10% of original requests." Prevents retry storms where retries generate more load than original traffic, worsening an outage.

Structured Logging

Log as JSON, not free text. Include: timestamp, level, service, correlation_id, user_id, message. Makes logs searchable and parseable.

Correlation IDs

Generate a unique ID at the entry point (API Gateway). Pass it through every service call (header: X-Correlation-ID). Include it in every log line. Lets you trace a request across all services.

What to Monitor

The RED method: Rate (requests/sec), Errors (error rate), Duration (latency percentiles p50, p95, p99). Plus: saturation (CPU, memory, disk, connections).

SLI Indicator

A measurable metric. Example: "99.2% of requests completed in under 300ms over the last 28 days." The actual measurement.

SLO Objective

Internal target. "We aim for 99.9% availability." Triggers alerts and engineering priorities when at risk. More stringent than the SLA.

SLA Agreement

External contract with customers. "We guarantee 99.5% uptime or credits are issued." Has financial/legal consequences.

K8s Makes Sense When

• 10+ microservices that need orchestration
• Need auto-scaling, self-healing, rolling updates
• Multi-team, multi-service deployments
• You have (or can hire) Kubernetes expertise
• Running on multiple clouds or on-premises

K8s Is Overkill When

• 1-3 services (use Docker Compose or ECS)
• Small team without K8s experience
• Serverless/PaaS would be simpler (Heroku, Fly.io, Railway)
• You're managing more YAML than application code

Blue-Green

Run two identical environments. Deploy to the idle one (green), test it, then switch traffic. Instant rollback by switching back.

Fast rollback 2x resources

Canary

Route a small percentage (1-5%) of traffic to the new version. Monitor error rates and latency. Gradually increase if healthy. Roll back if issues detected.

Low risk Needs good observability

Rolling

Replace instances one at a time. Default in Kubernetes. No extra resources needed. Slower rollback (must roll forward or replace all).

Resource efficient Mixed versions during deploy