Topic, page 2
Page 2 of articles on Performance.
A model that describes an image accurately is not yet a system. Frame rate, cost per frame and what happens on a wrong answer decide the architecture.
A million-token window suggests you can stop being selective about context. Attention is uneven across a long window, and cost scales with what you send.
Running a model locally is free in the sense that a server is free. The bill is idle capacity, and the arithmetic decides this more than principle does.
Batching helps throughput and hurts latency. Caching helps latency and can hurt consistency. Most performance debates are about which number matters more.
Whether it needs designing from scratch or rescuing from its own success, tell me what you’re building and I’ll tell you how I’d architect it.
Open to remote and hybrid work worldwide