A monthly estimate built from “requests times token price” is appealingly clean. It often misses the things that make the feature useful: document ingestion, search, tool calls, retries, moderation where appropriate, monitoring and people reviewing the difficult cases. You need a budget for the system, not only the model.
Count the whole workflow
Write out one completed user journey, from opening the screen to receiving a result that can be used. For a document assistant, that might be identity check, retrieval, generation, citation display and a correction when a source is stale. Measure both typical and difficult requests. A request that fails and needs a human handoff still consumes resources.
- Model input and output per stage.
- Search, storage and data refresh.
- Hosting, logging and third-party services.
- Review, support and evaluation time.
Build a range, not a single promise
Estimate low, typical and high usage with explicit assumptions: active users, tasks per user, average input size, average output size and retry rate. Multiply by current provider rates for the exact model and tier. Add a separate line for storage and search, another for engineering operations and another for human review. Keep assumptions next to the numbers so an estimate can be revised when real traces arrive.
Be especially careful with context growth. A long conversation or large retrieved document can make later requests much larger than the first. Caching can help some repeated input, but it depends on provider rules and the structure of your calls. Do not assume every input token receives the cached rate.
The expensive failure is not always a bill
A wrong answer that requires a person to investigate, a delayed support interaction or a disclosure caused by weak permissions can cost more than the API. Avoid hiding those risks behind a small infrastructure line item. Set quality and safety thresholds and test the workflow on a representative sample before projecting savings.
A human-in-the-loop step may look like extra cost, but it can make an early deployment viable while evidence accumulates. As the system improves, measure which cases can safely move to lighter review. A trustworthy budget includes the work of learning.
- Track time saved per completed task, not only requests served.
- Log the kinds of correction people make.
- Revisit model choice and retrieval size with real usage data.
A worksheet someone can copy
Make columns for task type, daily volume, input tokens, output tokens, retries, retrieval calls, average review minutes and completion rate. Multiply a row by the published provider price for the chosen mode, then add infrastructure and review time. Divide the total by tasks completed correctly, not by requests attempted. Keep a separate note for unusual high-volume days.
The worksheet will be imperfect on day one; that is expected. Its value is in making assumptions visible. After launch, replace estimates with measured distributions, including the longest requests and the ones requiring a human. If the cost rises, you can see whether more people arrived, context got longer, retries increased or the product actually helped more people finish.
- Track low, typical and high usage.
- Use current provider rates, recorded with a date.
- Update assumptions with actual request traces.
Use the estimate to shape the product
If the cost per task is too high, the first move may be a better UX: shorter context, clearer questions, fewer unnecessary calls or a conventional rule for a routine step. A cheaper model is one option, but not the only one. Sometimes an answer should be retrieved and displayed directly without generation.
Update the worksheet each month with real traces and provider prices. The best budget is not a fixed forecast. It is a decision tool that tells you which part of the product to improve next.