Performance

1 Why Performance Matters

Performance is not a feature — it is the product. Every millisecond of delay compounds into lost revenue, frustrated users, and wasted infrastructure spend. But knowing when to optimize is just as important as knowing how.

The Business Case for Speed

"Premature optimization is the root of all evil." — Donald Knuth (often misquoted out of context)

When NOT to Optimize

Knuth's full quote continues: "Yet we should not pass up our opportunities in that critical 3%." The real message: don't optimize without data. Here's when to hold off:

2 Latency vs Throughput

Latency is how long one operation takes. Throughput is how many operations complete per unit of time. They are related but not inverses — optimizing one can hurt the other.

The Bathtub Analogy

Percentiles: Why Averages Lie

Average latency hides your worst users. If 99 requests take 10ms and 1 takes 5 seconds, the average is ~60ms. Looks fine. But that 1 request represents a real user having a terrible experience — and it's probably your most important user (heavy users make more requests and are more likely to hit tail latency).

Coordinated Omission

A subtle but devastating measurement bug. Most load testing tools measure latency from when the request was sent, not from when it should have been sent. If the server stalls for 5 seconds and the client is waiting, the stall only shows up in one request's latency — but it delayed all the requests that were supposed to fire during that 5 seconds. Gil Tene's wrk2 and HdrHistogram are designed to combat this.

3 Latency Numbers Every Programmer Should Know

Jeff Dean's famous list, updated for modern hardware (~2024-2025 era). Memorize the orders of magnitude — the exact numbers change yearly, but the ratios stay remarkably stable.

Visual Scale (logarithmic, 1 ns = 1 second analogy)

Architectural Implications

4 Amdahl's Law & Scalability

Amdahl's Law defines the theoretical maximum speedup from parallelizing a computation. It is the single most important formula in performance engineering.

Universal Scalability Law (USL)

Neil Gunther's USL extends Amdahl's Law by adding a coherence penalty (the cost of keeping shared state consistent across processors). While Amdahl's Law says throughput plateaus, USL predicts that throughput can actually decrease past a certain point — a phenomenon called retrograde scalability.

Practical Implications

• Find the serial bottleneck first. If 20% of your pipeline is serial, max speedup is 5x regardless of hardware.
• Measure before adding cores. Going from 4 to 8 cores gives at most 2x. Going from 64 to 128 gives almost nothing.
• Reduce coordination. Shared locks, distributed consensus, and cache coherence are the enemies of scalability.
• Partition, don't share. Independent shards scale linearly. Shared-everything architectures hit USL's coherence wall.

5 Little's Law

The most useful formula in capacity planning, and one of the few results in queueing theory that requires almost no assumptions.

Practical Examples

The Queueing Trap

When utilization approaches 100%, queueing delay explodes. For an M/M/1 queue, the average wait time is:

W_queue = W_service / (1 - utilization)

At 50% utilization, wait = 2x service time. At 90%, wait = 10x. At 99%, wait = 100x. This is why you should never run a system above ~70-80% utilization — the latency penalty is not linear.

6 Profiling

Profiling is the act of measuring where your program spends its time and memory. Without profiling, optimization is guesswork. With profiling, you find the 5% of code that accounts for 95% of the cost.

CPU Profiling Methods

Wall-Clock vs CPU Time

Profiling Tools

Memory Profiling

• Heap dumps — snapshot of every object on the heap. Find what's using memory. Tools: jmap (Java), pprof (Go), memray (Python).
• Allocation profiling — tracks where objects are allocated. More useful than heap dumps for GC pressure. async-profiler's alloc mode is excellent.
• Leak detection — compare heap dumps over time. Growing object counts = leak. In Go, pprof's inuse_objects vs alloc_objects tells the story.

7 Flame Graphs

Invented by Brendan Gregg, flame graphs are the single most effective visualization for understanding CPU profiles. Once you learn to read them, you'll never go back to flat profiles.

How to Read a Flame Graph

Flame vs Icicle

Differential Flame Graphs

Compare two profiles (before/after optimization, or two versions). Functions that got slower appear in red; functions that got faster appear in blue. Generate with Brendan Gregg's difffolded.pl or use SpeedScope's compare mode.

Generating Flame Graphs

1. Java: async-profiler -d 30 -f profile.html <pid> — produces an interactive HTML flame graph directly.
2. Go: go tool pprof -http=:8080 cpu.prof — built-in flame graph view in the web UI.
3. Linux (any language): perf record -g -F 99 -p <pid> -- sleep 30, then perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg
4. Node.js: node --prof app.js, then process with 0x or clinic flame.

8 Benchmarking

Benchmarking is deceptively hard. The difference between a meaningful benchmark and a misleading one is enormous. Most benchmarks in blog posts are wrong.

Micro vs Macro Benchmarks

Common Pitfalls

HTTP Benchmarking Tools

How to Report Results

A good benchmark report includes: P50, P95, P99 latencies, throughput (req/s), error rate, resource utilization (CPU, memory), test duration, hardware specs, software versions, and reproduction instructions. Without these, the numbers are useless.

9 Load Testing

Load testing validates that your system meets its performance requirements under realistic conditions. Different types of load tests answer different questions.

Types of Load Tests

Load Testing Tools

Designing Realistic Load Profiles

• Analyze real traffic — use access logs to understand request distribution, think time, session patterns
• Use varied data — hitting the same URL with the same parameters tests the cache, not your system
• Model think time — real users pause between clicks; remove think time and you're testing something unrealistic
• Include all request types — a mix of reads, writes, uploads, searches at realistic ratios
• Ramp up gradually — don't jump to full load; ramp over 2-5 minutes to let connection pools warm up

10 Monitoring & Alerting

Monitoring is how you know your system is healthy in production. Good monitoring prevents outages. Great monitoring tells you why the outage happened in under 60 seconds.

The RED Method (for Services)

For every service, monitor these three things. Rate tells you if traffic changed. Errors tell you if something broke. Duration tells you if something slowed down. Together they cover most failure modes.

The USE Method (for Resources)

For every resource (CPU, disk, network, memory), check: How busy is it? Is work queuing? Are there errors? Brendan Gregg designed this to systematically find bottlenecks.

SLOs, SLIs, and Error Budgets

Prometheus + Grafana Stack

This is the de facto open-source monitoring stack. Prometheus pulls metrics via HTTP (pull model), stores them as time series, and supports powerful queries. Grafana visualizes them. Alertmanager handles routing and deduplication of alerts.

Golden Signals (Google SRE)

Google's four golden signals are similar to RED but add Saturation: how "full" a service is. A system at 95% memory is one bad deploy from OOM-killing. Monitor: Latency, Traffic, Errors, Saturation.

11 CPU Performance

Modern CPUs are marvels of prediction and speculation. Understanding how they work (and fail) is essential for writing truly fast code.

The Instruction Pipeline

Branch Prediction

CPUs predict which branch an if/else will take and start executing speculatively. Modern predictors are >95% accurate on typical code. But when they're wrong, the penalty is 10-20 cycles (the pipeline must be flushed and restarted).

Cache Lines

CPUs don't read individual bytes from memory — they read 64-byte cache lines. When you access one byte, the adjacent 63 bytes come along for free. This has profound implications:

• Spatial locality wins — accessing sequential memory is fast because it's already in cache
• Struct padding matters — a struct that spans two cache lines requires two memory accesses
• Array of structs vs struct of arrays — if you only access one field, SoA keeps that field's data contiguous in cache

False Sharing

NUMA (Non-Uniform Memory Access)

Multi-socket servers have memory attached to each socket. Accessing local memory takes ~100ns. Accessing remote memory (another socket) takes ~150-300ns. For performance-critical applications on multi-socket machines, pin threads to cores and allocate memory on the local NUMA node.

Cache Hierarchy

12 Algorithmic Complexity

Big O notation is the first tool for performance analysis. But real-world performance is messier than theory suggests — constant factors, cache effects, and input distributions all matter.

Big O Refresher (Practical Focus)

When O(n²) is Actually Fine

Real-World Examples Where Algorithm Choice Made 100x Difference

Amortized Complexity

Some operations are expensive occasionally but cheap on average. ArrayList.add() is O(1) amortized — most adds are instant, but every ~Nth add triggers an O(n) resize. The total cost over N operations is still O(n), so each one is O(1) amortized. This is fine for throughput but can cause latency spikes — which matters in real-time systems.

13 Concurrency Performance

Concurrency is where hardware parallelism meets software coordination. Getting it right yields linear speedups. Getting it wrong yields bugs and makes things slower.

Lock Contention

A lock serializes access to shared state. Under contention (many threads competing for the same lock), the lock becomes a bottleneck — effectively making your parallel program serial. Strategies to reduce contention:

• Reduce critical section size — hold the lock for the minimum time necessary
• Use lock striping — ConcurrentHashMap uses 16 segments instead of one lock
• Use read-write locks — allow concurrent readers, only serialize writers
• Eliminate shared state — thread-local storage, sharding, immutable data

Lock-Free Data Structures

Thread Pool Sizing

Context Switching Cost

Every thread context switch costs 1-10 microseconds of direct overhead, plus indirect costs from cache pollution (L1/L2 caches go cold). With 10,000 threads, the scheduler alone can consume 100% of a core. This is why event-loop architectures (Node.js, Netty, Nginx) outperform thread-per-request models at high concurrency.

14 Memory Performance

On modern hardware, memory access patterns often matter more than algorithmic complexity. A cache-friendly O(n log n) sort beats a cache-hostile O(n) scan. Understanding the memory hierarchy is the key to unlocking hardware performance.

Cache Hierarchy Latency

Spatial & Temporal Locality

Data-Oriented Design

Traditional OOP organizes data around objects. Data-oriented design organizes data around access patterns — keeping the data you process together actually together in memory.

Struct of Arrays vs Array of Structs

struct Particle {
  x, y, z: f32,
  vx, vy, vz: f32,
  mass: f32,
  color: u32,
}
particles: [Particle; N]

struct Particles {
  x: [f32; N],
  y: [f32; N],
  z: [f32; N],
  vx: [f32; N],
  ...
}

15 Memory Management

Allocation and deallocation are among the most frequent operations in any program. How they're managed — manually, by GC, or via pools — can make or break performance.

Garbage Collection Tuning

Object Pooling

For frequently allocated/deallocated objects (buffers, connections, message objects), pooling eliminates GC pressure. Java: use ThreadLocal pools or libraries like Netty's pooled byte buffers. Go: sync.Pool. Key rule: only pool objects that are expensive to create or created at extremely high rates.

Memory Leaks: Common Patterns

Detection Tools

Java: Eclipse MAT for heap dump analysis, jmap -histo for quick overview, JFR for allocation tracking. Go: pprof heap profiles. Python: tracemalloc, objgraph. General: compare heap snapshots over time — anything growing linearly is a leak candidate.

16 Data Structures & Performance

Choosing the right data structure is the highest-leverage optimization decision. The best algorithm with the wrong data structure is like a Formula 1 engine in a shopping cart.

ArrayList vs LinkedList: The Cache Locality Victory

HashMap Internals

• Load factor — Java's default is 0.75 (resize when 75% full). Lower = faster lookups, more memory. Higher = slower lookups, less memory.
• Hash function quality — poor hash functions create clusters, degrading O(1) to O(n). Java 8+ uses a tree structure for long chains.
• Initial capacity — if you know you'll have 1000 entries, create with capacity 1334 (1000/0.75) to avoid resizing.

Choosing the Right Structure

Bloom Filters

17 Disk I/O

Despite SSDs making random I/O much faster than spinning disks, the gap between memory and disk remains enormous. Understanding disk I/O patterns is critical for any data-intensive application.

Sequential vs Random I/O

Key I/O Concepts

io_uring (Linux 5.1+)

A modern async I/O interface for Linux that eliminates syscall overhead. Uses shared ring buffers between user and kernel space. Supports batching multiple I/O operations into a single submission. Benchmarks show 2-3x throughput improvement over epoll for I/O-heavy workloads. Libraries: liburing (C), io-uring crate (Rust), tokio-uring.

Write-Ahead Log (WAL) Pattern

Sequential writes are fast. Random writes are slow. WAL converts random writes into sequential appends, giving both durability and speed. Used by: PostgreSQL, SQLite, LevelDB, Kafka, etcd, every serious database.

18 Network Performance

Network is usually the slowest link in a distributed system. A datacenter round-trip is 500 us — 5,000x a memory access. Every network call you can eliminate is a massive win.

Bandwidth vs Latency

TCP Tuning

HTTP/2 Multiplexing

Compression Trade-offs

19 Database Performance

The database is the most common bottleneck in web applications. A single missing index can turn a 2ms query into a 20-second table scan. Database optimization has the highest ROI of any performance work.

Query Optimization

Indexing Strategy

The N+1 Problem

Connection Pooling

Each database connection costs ~5-10 MB of memory on the server. Creating a connection takes 5-50ms (TCP + auth). Connection pools (HikariCP for Java, PgBouncer for PostgreSQL, pgx pool for Go) reuse connections. HikariCP's default pool size of 10 handles more load than most people expect — see the HikariCP wiki's "About Pool Sizing" page, which argues that fewer connections with less contention outperform many connections competing for CPU/disk.

Other Database Optimizations

• Read replicas — route read queries to replicas to reduce primary load. Watch for replication lag on writes-then-reads.
• Query caching — cache frequent query results in Redis/Memcached. Invalidate on writes. Massive win for read-heavy workloads.
• Denormalization — store pre-computed data to avoid expensive JOINs. Trade write complexity for read speed. Common in analytics and reporting.
• Batch operations — INSERT 1000 rows in one statement instead of 1000 individual INSERTs. 10-100x faster.

20 Caching Strategies

"There are only two hard things in computer science: cache invalidation and naming things." — Phil Karlton. Caching is the single most impactful performance technique, and the single hardest to get right.

Where to Cache

Each layer reduces load on the layer behind it. Browser caching eliminates network requests entirely. CDN caching serves content from edge nodes close to users. App-level caching (Redis/Memcached) avoids expensive computations and database queries.

Cache Patterns

Cache Invalidation Strategies

• TTL (Time To Live) — simplest. Data expires after N seconds. Good enough for most cases. Tune TTL based on data freshness requirements.
• Event-based invalidation — on data change, explicitly delete or update cache entries. More complex but ensures consistency.
• Version keys — include a version number in the cache key. Increment on change. Old entries expire naturally via TTL.

Cache Stampede (Thundering Herd)

Hot Key Problem

A single cache key receiving millions of requests per second can overwhelm a single Redis shard. Solutions: replicate the key across multiple shards with randomized suffixes (key:1, key:2, ..., key:N), use local in-process caching for the hottest keys, or use a dedicated "hot key" cache tier.

21 Async & Non-Blocking

Async programming lets a single thread handle thousands of concurrent operations by not blocking on I/O. It's the foundation of every high-performance server.

Event Loops

Virtual Threads (Java 21+)

When Async Does NOT Help

Backpressure

When a producer generates data faster than a consumer can process it, backpressure is the mechanism to slow the producer down. Without it, queues grow unbounded until the system OOMs.

Options: block the producer (bounded queue), drop messages (load shedding), sample (rate limiting), or signal the producer to slow down (reactive streams, TCP window).

Queue-Based Decoupling

Place a message queue (Kafka, RabbitMQ, SQS) between components to absorb traffic spikes. The queue acts as a shock absorber — producers can burst at 10x the normal rate as long as the average rate doesn't exceed consumer capacity. Also enables independent scaling and fault isolation.

22 Serialization Performance

Serialization is converting in-memory objects to bytes for storage or transmission. It happens on every API call, every database query, every cache operation. Even small improvements compound massively.

Format Comparison

Protobuf is typically 2-10x faster than JSON for serialization/deserialization and produces 2-3x smaller payloads. For internal service-to-service communication, there's rarely a reason to use JSON.

Compression: Speed vs Ratio

Rule of thumb: Use Zstd unless you have a specific reason not to. It offers the best speed-to-ratio trade-off, with a tunable compression level from "faster than LZ4" to "smaller than gzip."

23 JVM Performance

The JVM is one of the most sophisticated runtime environments ever built. Understanding its internals unlocks performance that rivals C/C++ for many workloads — but misconfiguration can make it surprisingly slow.

GC Tuning: G1 vs ZGC

Heap Sizing

JIT Warmup

The JVM starts by interpreting bytecode. After ~10,000 invocations, the JIT compiler (C2) compiles hot methods to optimized native code. This warmup period can last 30-60 seconds in large applications. Strategies:

• Accept warmup and design SLAs around it (route traffic gradually after deploy)
• Use CDS (Class Data Sharing) and AppCDS to speed up startup
• GraalVM native-image for ahead-of-time compilation (eliminates warmup but loses peak JIT performance)
• JVM warmup scripts that exercise hot paths before accepting traffic

Escape Analysis

The JIT compiler can prove that an object never "escapes" the method that created it. Such objects can be allocated on the stack (free) instead of the heap (GC-managed). This is why small, short-lived objects in hot loops are often "free" in Java — the JIT eliminates the allocation entirely.

JFR (Java Flight Recorder)

Built into the JVM since Java 11 (free in OpenJDK). Low overhead (~1-2%) continuous profiling: CPU, memory allocation, GC events, thread states, I/O, lock contention. Enable with -XX:StartFlightRecording=duration=60s,filename=recording.jfr. Analyze with JDK Mission Control (JMC). The best "always on" profiling tool for production JVMs.

Other JVM Tricks

• String deduplication (-XX:+UseStringDeduplication with G1) — if your heap is 30%+ String objects (common in web apps), this can save significant memory
• Compressed oops — enabled by default for heaps <32GB. Use object references as 32-bit offsets instead of 64-bit pointers. Stay under 32GB heap if possible.
• Class loading overhead — thousands of classes = slow startup. Minimize dependencies. Use jlink to create minimal runtime images.

24 Scaling Patterns

Scaling is the art of handling more load without degrading performance. The approach depends on what's bottlenecked — CPU, memory, I/O, or network.

Vertical vs Horizontal Scaling

Stateless Services

The prerequisite for horizontal scaling. If a service stores no local state, any instance can handle any request. Session state moves to Redis or a cookie. File uploads go to S3. All shared state lives in a database. Now you can auto-scale from 2 to 200 instances transparently.

Sharding

Scaling Patterns Summary

25 Performance Anti-Patterns

These are the most common performance mistakes. Each one is easy to commit and can cause 10-1000x slowdowns. If your system is slow, start by checking for these.

The Hall of Shame

26 Performance Culture

Performance is not a one-time project — it's a culture. Without ongoing investment, systems get slower with every release. Building performance into your engineering culture is the only way to stay fast at scale.

Performance Budgets

Regression Detection in CI

1. Establish a baseline — run benchmarks on a dedicated (not shared) machine. Record P50, P95, P99 for key operations.
2. Run on every PR — compare the PR's benchmarks against the baseline. Flag regressions >5% as warnings, >15% as failures.
3. Track trends — store benchmark results over time. A 1% regression per week = 67% slower in a year. Trend detection catches slow degradation that per-PR checks miss.
4. Automate alerts — post benchmark results as PR comments. Make performance visible to every engineer, every day.

Continuous Profiling

Capacity Planning

Use current metrics + growth projections to anticipate when you'll need more capacity. Little's Law is your friend: if traffic grows 10% monthly and you're at 60% utilization, you have ~5 months before you hit 100%. Plan capacity 3-6 months ahead. Factor in seasonal peaks (Black Friday, back-to-school).

Performance Review Checklist

"Performance is a feature. Treat it like one — with requirements, testing, monitoring, and dedicated ownership." — Every engineering team that has been paged at 3am for a latency spike