TL;DR
A report from The Pragmatic Engineer describes inference engineering as a growing set of practices for software teams adopting open-weight AI models. It explains how teams can start with a provider’s shared API and consider dedicated GPUs as usage, latency or rate-limit needs change.
The Pragmatic Engineer has published a report on inference engineering, the practical work of deploying and operating AI models, as software companies consider alternatives to APIs from frontier AI labs. Drawing on interviews with Modal, Fireworks AI and Baseten, the report describes a path from shared, multi-tenant model APIs to dedicated GPU deployments, and the operational trade-offs teams encounter along the way.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The report frames the move to open-weight models as one response to AI infrastructure costs. It cites earlier reported savings of 50% at Uber, 90% at Pinterest and 56% at AT&T, but does not provide the calculation periods, baselines or underlying methodology for those figures in the supplied material. The figures should be read as examples reported by the publication, not as a forecast for other companies.
Teams can either operate their own GPU infrastructure or use an inference provider. The report cites Revolut’s 56 NVIDIA B200 units as an example of a company running hardware itself, a route that requires substantial setup and operational work. Providers instead operate GPU fleets and offer access to models through APIs, which can make initial adoption faster.
The report says the first step for many teams is to replace a frontier lab’s API with a provider’s shared API. At higher usage, teams may encounter rate limits, inconsistent performance, tail latency or cache issues. Dedicated GPUs can offer more control but require engineers to make decisions about model size, throughput, interactive response times, KV-cache capacity and failover. The report also discusses speculative decoding as a possible way to use otherwise idle compute during token generation.
Engineering Choices Behind Model Serving
The topic matters because model selection is only part of the cost and performance equation. Once a software product depends on an AI model, the way it is served can affect response times, reliability and operating costs. Engineers may need to compare those factors with the effort of managing infrastructure or changing providers.
The report’s staged approach gives teams a practical decision framework rather than treating dedicated GPUs as an automatic upgrade. A shared API may suit early or variable workloads; rate limits, unstable latency or high usage costs can prompt a closer look at dedicated capacity. The right choice depends on a company’s workload and service requirements, details that cannot be inferred from token prices alone.
That makes inference engineering relevant to software engineers beyond AI research teams. It involves familiar production concerns—capacity planning, caching, reliability and vendor evaluation—applied to systems that generate model outputs. Provider descriptions and reported savings are not independent comparisons, so teams still need to test performance and cost for their own use cases.
Products Worth Considering

Samsung Galaxy Tab A11+ Kids Edition, 128GB, Royal Blue

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

Apple Watch Series 12 GPS 42mm Smartwatch
As an affiliate, we earn on qualifying purchases.
From Frontier APIs to Open Models
The report distinguishes inference providers from frontier AI labs. Both can provide an API for sending prompts and receiving streamed tokens, but the providers discussed primarily host open-weight models; frontier labs train their own foundation models. The report says training frontier-scale models can cost millions of dollars, while post-training open models is less expensive.
It describes different provider approaches. Modal emphasizes programmable cloud infrastructure and open-source tooling. Fireworks focuses on specialized workloads, dedicated deployments and tuning. Baseten presents an inference platform with additional tools for agent development. These are company descriptions reported by the publication, not independent evaluations of the vendors.
The report says its authors had no commercial affiliation with the named providers and were not paid to mention them. It credits interviews with Richard Gong of Modal, Fireworks co-founder and CTO Dmytro Dzhulgakov, and Philip Kiely of Baseten. The supplied source text ends during an example about a Fireworks announcement, so that example cannot be described in further detail here.
“When you run code on Modal, you should be able to see all the code that’s running.”
— Richard Gong, inference lead at Modal, as quoted by The Pragmatic Engineer
Costs and Performance Need Testing
The supplied source does not state the report’s publication date or give enough detail to verify the cited company savings, including their baselines, time periods and accounting methods. It also provides no comparative benchmark showing how Modal, Fireworks and Baseten perform on the same workload. Vendor selection and savings cannot be determined from the reported figures alone.
It is also unclear how much dedicated infrastructure a particular team would need, or whether moving away from a shared API would improve its latency or total costs. Those outcomes depend on model choice, request patterns, service-level needs and provider terms. The source’s truncated final example leaves the details of the Fireworks product announcement unspecified.
Teams Can Compare Workloads
The report does not announce a formal next milestone or give a deployment timeline. Its practical next step for engineering teams is to evaluate open models and providers against their own requirements, then monitor performance after any API change. Relevant measures include latency, cache behavior, rate limits, reliability and total cost.
Teams that outgrow a shared setup can assess whether dedicated GPUs meet their needs, while accounting for capacity, failover and operating work. The source does not identify which provider or deployment model is best for a given company; that remains a workload-specific decision.
Key Questions
What does inference engineering mean?
It refers to the engineering work involved in serving AI models, including latency, caching, capacity, reliability and cost.
How are inference providers different from frontier AI labs?
The report says inference providers mainly host open-weight models and sell access to them, while frontier labs train their own foundation models. Both may offer APIs to customers.
When might a team move from a shared API to dedicated GPUs?
The report identifies rate limits, inconsistent performance and high usage costs as possible signs to examine dedicated deployments. They are prompts for evaluation, not proof that dedicated GPUs will be cheaper or faster.
Are the reported cost reductions guaranteed for other companies?
No. The publication cites reductions at Uber, Pinterest and AT&T, but the supplied material does not give the comparison baselines or methods. Other teams would need to test costs using their own workloads.
Source: newsletter.pragmaticengineer.com
