At Near I had what was officially one job title, lead platform engineer, and what were actually two jobs. Job one: keep a real-time bidding engine answering a quarter of a million auctions per second without embarrassing us. Job two: make sure the rest of engineering didn’t have to rediscover how to build a service every time someone had a product idea.

I’ve written before about the latency side of that first job. This post is about the shape of the whole thing, because the shape is what I’d get wrong if I did it again without notes.

The bidder: 50 machines, 4 data centers, no heroics

The fleet was simple to describe:

us-east ─┐
         ├─ 35 machines        ~5K QPS each
us-west ─┘
FRT      ── 10 machines        ──────────────
SG       ──  5 machines        ≈ 250K QPS total

Each box did about 5K QPS. Exchanges (InMobi, Rubicon, AdMixer, and more as we grew) sent us OpenRTB bid requests, we had ~100ms end to end to answer, and geography ate most of that budget before our code ran a single line. That’s why the topology looks the way it does: you can’t serve a Singapore auction from Virginia and stay inside the latency budget.

Onboarding a new exchange sounds like a protocol exercise (OpenRTB is a spec, how hard can it be?), but every integration taught us that the spec is where agreement ends. One exchange’s “optional” field is another’s hard requirement, everyone extends the ext object differently, and traffic shape varies wildly: one partner sends you a firehose of low-value requests, another sends fewer requests where losing any one of them actually costs money. We ended up treating each exchange as a capacity-planning event, not a parsing task. Add the exchange, watch the per-machine QPS, and re-derive the fleet math before the traffic forces you to.

Redis, tuned like you actually pay the bill

The bidder’s hot path leaned on Redis hard, and Redis is single-threaded, which means a 32GB 8-core machine running one Redis instance is a machine you’re paying for and mostly not using. So: eight instances per box, fronted by Twemproxy, one instance roughly per core. Redis got dedicated machines, because sharing a box with a JVM is a good way to end up debugging mystery latency spikes.

The less obvious tuning was operational, and each item came from a bad day:

  • TTLs were a cost dial, not a correctness dial. Every cached object’s TTL was a trade between memory cost and hit rate. We tuned them per key family instead of picking one number and moving on.
  • RDB over AOF, because restarts have an SLA too. Replaying an append-only log on a fat instance took long enough that a “quick bounce” wasn’t. Snapshot-based recovery got instances back in the pool fast. Persistence writes were disabled on masters entirely; the slave did the disk work while the master did the serving.
  • maxmemory at 80%, not 100%. The gap is headroom for fork-time copy-on-write during snapshots and for the OS to breathe. Set it to the full machine and your first snapshot under load will hit the OOM killer.
  • Hit ratio as a first-class metric. Redis’s stats give you hits and misses; we kept a moving average in Grafana with an alert on it. A sagging hit ratio is the earliest signal that a TTL is wrong or a new traffic pattern arrived. It fires days before latency does.

Elasticsearch and the art of not searching

Campaign targeting data lived in Elasticsearch, replicated master-to-slave across the DCs so every region queried locally. The tuning that mattered most wasn’t a knob, it was index design: we cut indexes along the dimensions that filtered hardest, things like country_mediatype_*. A query that lands on a pre-narrowed index does a fraction of the work of one that filters the same predicates inside a giant shared index. Boring, but it did more than any setting we ever changed.

Same philosophy, different system: fresh bid requests used to make a synchronous HTTP call to an ID-generation service, on the hot path, for a value the bid response didn’t need. We cut the call, answered the auction, and let a Kafka consumer do the ID work in the background. The rule we took from it: the hot path should only compute what the response requires, and everything else can move to a queue.

Job two: NEEV, or paving the road everyone was already driving on

What surprised me about running a high-QPS system with a small team is that the bidder isn’t where the engineering time goes. It goes into the fifteen ordinary services around it, each solving caching, queues, and auth from scratch, each slightly differently, each with its own bugs.

So we built NEEV (Near Excellency and Engineering Vault), an internal platform layer:

  • Service templates. A new DropWizard service came out of a generator with logging, metrics, health checks, and deploy config already correct. Day one was writing business logic, not plumbing.
  • service-core. A family of libraries (near-cache, near-queue, near-kafka, near-mongo, near-mailer, near-commons) wrapping infrastructure with our conventions baked in: connection handling, serialization, retries, metrics. Naming things near-* was half branding, half a grep-able audit trail of who used what.
  • near-a3, authentication and authorization built with Spring Security and JWT, done once at the platform level instead of nine slightly different ways across services.
  • Stripe integration as a platform capability, so a product team touching payments inherited a hardened path instead of a fresh chance to mishandle card flows.

Was some of this NIH? Partly. But the payoff was real: the time for a new engineer to ship a production service went from weeks to days, and code review got faster too, because every service was recognizably the same species.

The proof it worked was Engage, our self-serve platform where clients managed their own ad spend and targeting. DropWizard behind, AngularJS in front, assembled largely from NEEV parts. It shipped fast because the boring layers already existed, and it removed a whole category of ops tickets by letting customers do things themselves.

What I’d tell past me

The bidder was the resume line. 250K QPS across four DCs sounds impressive at dinner. But the leverage was the platform work. Tuning Redis made one system faster. NEEV made every future system cheaper. If you lead platform engineering and your calendar is 100% hot path, you’re only doing the visible half of the job.

And write down the fleet math. Fifty machines, 5K each, four regions. The day someone asks “can we take this new exchange’s traffic?”, the answer should come from arithmetic, not guesswork.