Topic
16 articles on Performance, written from production systems rather than documentation.
Asking a model to reason step by step improves some tasks and slows every one of them. The trade is latency and tokens, paid on every single call.
Teams pick one model and route everything through it. Most systems have several jobs with different requirements, and one answer fits none of them well.
Long sessions fill the window and something has to go. What you drop, and what you replace it with, decides whether the system degrades or forgets.
Approximate search is approximate on purpose. Knowing which results you are giving up explains retrieval failures that look like model problems.
Vector search handles paraphrase well and exact terms badly. Keyword search is the reverse, and users send both kinds of query in the same session.
Caching the stable prefix of a prompt cuts cost and time to first token substantially. It only works if the stable part is genuinely first.
Retrieval optimises for recall and the model needs precision. A reranker sits between them and fixes the mismatch without touching the index.
Approximate indexes want to be in memory, and memory is the bill. Sizing that before you commit is the difference between a plan and a surprise.
Transcription accuracy on clean audio is a solved problem. Production audio is noisy, overlapping and full of the terms your domain depends on.
Once you have sent a 200 and started writing tokens, there is no status code left for the failure that happens halfway through the answer.
Three settings control the same thing from different angles. Turning all of them is how teams get output that is either rigid or quietly unhinged.
Cost, latency and the point at which text gets cut off are all counted in tokens. Reasoning about the system in words will mislead you in all three.
Whether it needs designing from scratch or rescuing from its own success, tell me what you’re building and I’ll tell you how I’d architect it.
Open to remote and hybrid work worldwide