Building Low-Latency ML Inference at Ad-serving Scale

How we kept p95 latency under 20 ms while moving from in-memory models to remote neural inference