Part of my notebook. Terse by design.
The latency ladder
Orders of magnitude, from registers to cross-planet. Knowing which rung an operation sits on is most of capacity planning.
| Range | What lives here |
|---|---|
| 1 ns | CPU register access, clock cycle |
| 1-10 ns | L1/L2 cache |
| 10-100 ns | L3 cache; main memory on modern chips (M1 at the low end) |
| 100 ns-1 µs | Linux system call; MD5 of a 64-bit number (~200 ns) |
| 1-10 µs | Thread context switch; copy 64 KB in memory |
| 10-100 µs | NGINX processing an HTTP request (~50 µs); read 1 MB sequentially from RAM; SSD 8K page read (~100 µs) |
| 100 µs-1 ms | SSD write (~10x read); intra-zone cloud round trip; Redis GET incl. network (~1 ms) |
| 1-10 ms | Inter-zone round trip; HDD seek (~5 ms) |
| 10-100 ms | US coast-to-coast or US-Europe round trip; read 1 GB from RAM |
| 100 ms-1 s | bcrypt (~300 ms, deliberately); cross-region TLS handshake; US-Singapore round trip; read 1 GB from SSD |
| 1-10 s | Transfer 1 GB over the network within a region |
Quick disk/network variants: 1 MB sequential from SSD ~1 ms, from HDD ~30 ms, over 1 Gbps ~10 ms. Packet CA → Netherlands → CA: ~150 ms.
The whole cache hierarchy
People say “cache” and mean the twelfth layer of twelve:
- CPU: L1 (16-128 KB, per core) → L2 (128 KB-1 MB) → L3 (2-64 MB, shared) → sometimes L4; plus the TLB for virtual→physical address translation.
- OS: page cache (recently used disk blocks in RAM), inode cache.
- Browser: HTTP cache per response headers.
- CDN: static content at the edge.
- Load balancer / reverse proxy caches.
- Application/distributed: Redis, Memcached.
- Database: buffer pools, materialized views precomputing query results.
Cache write patterns
- Cache-aside: on miss, read DB, fill cache, return. The default.
- Write-through: write cache + DB together; confirm after both. Simple consistency, write latency.
- Write-around: write DB only; cache fills on read. Protects cache from write-heavy churn.
- Write-behind: write cache, confirm, flush to DB asynchronously (often via queue). Fast and risky.
- Refresh-ahead: refresh hot entries before expiry.
Redis notes
Why it’s fast: everything in memory; single-threaded core with I/O multiplexing (no lock contention, no context switches); efficient data structures (skip lists, hash tables, sets). Multiple instances per machine if you need more cores.
Use cases that keep recurring: object cache, session store, distributed locks (see the locking note before trusting them), rate limiting, leaderboards via sorted sets, lightweight queues.
Precautions from production: set TTLs deliberately, and plan for the thundering herd on cold restart (below).
Why Kafka is fast
Two mechanical facts: sequential I/O (append-only log; writing blocks one after another is far faster than random writes, even to disk) and zero-copy reads (DMA moves bytes from page cache to socket without a round trip through userspace).
Rate limiting
Stripe’s framing stuck with me: not “1000 requests per second” but “20 requests in flight at once,” enforced with a token bucket in Redis (take a token per request, drip tokens back at a fixed rate, empty bucket → reject).
The four strategies:
- Token bucket: accumulating budget of tokens; allows bursts.
- Leaky bucket: fixed drain rate, overflow discarded; smooth constant flow, no bursts.
- Fixed window: “3000/hour” but all 3000 can arrive in the first minute.
- Sliding window: fixed-window benefits with the burst smoothed out; Redis expiring keys make this easy.
Where to implement: reverse proxy (NGINX), Redis, API gateway, or app code. Practices that matter: fail open if the limiter itself errors, return a clear 429, keep a kill switch, and dark-launch limiters first to observe what they would block before they block it.
Thundering herd
An unexpected traffic influx (cache flush, mass reconnect, cron alignment) overwhelms a service and the failure cascades to dependents. The toolkit:
- Exponential backoff with jitter: randomized retry timing spreads the load and gives the resource room to recover.
- Request queueing for cache-miss storms so the origin sees one fill, not thousands.
- Load balancing so the herd at least distributes evenly.
- Rate limiting against API abuse, batch jobs, and DDoS.
- Circuit breakers: the electrical MCB analogy; stop calling a failing dependency.
- Load shedding: rolling blackouts as a last resort; some service beats no service.
A great postmortem in this genre: Slack’s incident of 2-22-22, where mass client reconnects met an under-provisioned dependency.
C10K → C10M
Around 1999-2000 Dan Kegel named the C10K problem: web servers choked on ~10,000 concurrent connections because of thread-per-connection models and kernel I/O limits. The fix was event-driven, non-blocking I/O: NGINX (2004) handled 100k+ connections on commodity hardware, and kernel improvements (epoll, zero-copy, SO_REUSEPORT, smarter NICs) kept raising the ceiling. Today NGINX/Envoy/HAProxy/Go servers hold millions of keep-alive connections, and the frontier conversation is C10M.