LLM costs rarely explode because a team wrote one bad prompt. They explode because every prompt drags too much messy, duplicated, irrelevant, or poorly structured data into the context window.
That is why one of the most important AI engineering lessons right now sounds like classic data engineering: model the data before you send it to the model.
The new cost center hiding in your data layer
For years, engineering teams optimized databases around query performance, storage cost, reporting accuracy, and operational reliability. Now there is another constraint: tokens.
Every transcript, support ticket, product event, and CRM note that gets passed into an LLM has a cost. It also affects latency, retrieval quality, and answer reliability. The larger and noisier the context, the more expensive and less predictable the AI workflow becomes.
This is the core argument in dbt’s article, Model for the token, not the table. The post describes a team that was burning through Gong’s API to feed AI workflows. Instead of repeatedly calling Gong and pushing raw or semi-structured transcript data into AI prompts, they modeled the transcripts in the warehouse with dbt. The result: token costs dropped by 20x.
That is not just a prompt optimization story. It is a modernization story.
The team did not solve the problem by swapping models, adding a clever cache, or asking developers to write shorter prompts. They solved it by improving the shape, governance, and accessibility of the underlying data. In other words, the AI system became cheaper because the data system became better maintained.
Why token cost is really an architecture problem
A token budget is a design constraint. If your AI workflow summarizes sales calls, extracts customer objections, generates account intelligence, or powers an internal agent, it needs context. But context is only useful when it is relevant, compact, and trustworthy.
Raw operational data is rarely any of those things.
Call transcripts may include greetings, filler words, repeated names, timestamp artifacts, bot messages, and irrelevant chatter. CRM notes may be inconsistent. Support histories may span years and include outdated product behavior. Product analytics may use event names that changed three times across different versions of an application.
When AI features consume this data directly, three bad things happen:
- Costs rise because prompts include unnecessary text.
- Quality drops because useful signals are buried in noise.
- Reliability suffers because the system cannot distinguish current, governed facts from stale or duplicate information.
Traditional data modeling addresses exactly these problems. Transformations remove noise. Business logic standardizes meaning. Lineage explains where data came from. Tests catch unexpected changes. Documentation helps teams understand what each field represents.
The difference is that the downstream consumer is no longer only a dashboard or analyst. It is also an LLM, an agent, a retrieval pipeline, or an AI feature embedded in a product.
What the Gong transcript example teaches
The Gong example from dbt is useful because it is concrete. Many teams are building AI workflows around sales calls, customer conversations, and account data. The obvious first implementation is to call the source API, fetch transcripts, and hand them to the LLM.
That works for a demo. It often fails at scale.
Repeatedly pulling source data from an API creates operational fragility. Sending full transcripts into prompts burns tokens. Asking each application team to clean and summarize the same data creates duplicated logic. And if the source schema changes, multiple AI workflows may break in different ways.
By moving transcript modeling into the warehouse with dbt, the team created a reusable layer between the source system and the AI workflow. Instead of feeding the LLM raw transcripts, they could feed it curated, structured, and purpose-built representations of the conversation.
For example, a modeled transcript layer might include:
- Speaker-separated conversation turns
- Cleaned text with filler and artifacts removed
- Standardized timestamps and call metadata
- Extracted topics, objections, competitors, and next steps
- Account, opportunity, and contact joins
- Summaries at different levels of detail
- Flags for sensitive or low-confidence content
This lets an AI workflow retrieve the right slice of context instead of the entire conversation. It also lets teams reuse the same modeled data across sales coaching, customer intelligence, churn analysis, onboarding, and executive reporting.
The 20x token reduction matters because it shows that AI cost control is not only about choosing a cheaper model. It is about reducing waste before inference happens.
Agent reliability starts before the agent
In a related dbt article, Why agentic projects fail and how to fix them, dbt argues that many agentic AI projects fail because of issues in the underlying data. That point should resonate with anyone who has maintained production software systems.
Agents are only as reliable as the tools and data they use. If an agent queries inconsistent metrics, outdated customer records, undocumented tables, or conflicting definitions of revenue, it will produce confident but unreliable output. The failure may look like an AI problem, but the root cause is often a data maintenance problem.
This is especially important for CTOs and modernization leaders. Many organizations are layering AI features on top of systems that were not designed for AI consumption. Legacy databases, SaaS exports, event streams, and manual spreadsheets may all feed the same retrieval pipeline. Without a governed modeling layer, the agent has to reason over chaos.
That is not a fair job for the model.
A modern AI stack needs a semantic and operational contract between enterprise data and AI applications. dbt-style modeling helps create that contract by defining transformations, tests, documentation, ownership, and lineage in code. Recent dbt Core releases, including the v1.12 GA announcement, continue to reinforce the importance of maintainable transformation workflows for analytics and AI-adjacent use cases.
Model for consumption, not just storage
The phrase “model for the token, not the table” is powerful because it challenges a long-standing habit. Database tables are often designed around source systems, normalization, reporting needs, or ingestion convenience. AI systems need something different: context objects.
A context object is the information package an AI workflow actually needs to complete a task. It may combine multiple tables, include derived fields, remove irrelevant details, and compress long histories into structured summaries.
For a customer-success agent, the right context object might include current plan, renewal date, open support issues, recent usage trends, last executive interaction, and known risks. It probably does not need every raw clickstream event or every ticket comment.
For a code modernization assistant, the right context might include service ownership, dependency maps, runtime versions, recent incidents, deprecated libraries, and architectural decision records. It does not need every line of every log file.
For a sales call summarizer, the right context might include the cleaned transcript, identified speakers, opportunity stage, account segment, and prior call summary. It does not need the entire CRM export.
This shift requires engineering teams to ask different modeling questions:
- What decision or action will the AI system support?
- What is the minimum context needed to support that action?
- Which fields are authoritative?
- Which data should be excluded for privacy, cost, or quality reasons?
- How should context be versioned and tested?
- How will we know if the context has drifted?
Those are data engineering questions, but they are now also AI product questions.
Practical implications for modernization teams
For teams modernizing applications and adding AI capabilities, the lesson is straightforward: do not bolt AI directly onto brittle data flows.
Instead, treat AI adoption as an opportunity to improve the data architecture around your systems. At Vibgrate, we see this pattern across modernization efforts: teams that understand their dependencies, data contracts, ownership boundaries, and upgrade paths are better positioned to add reliable automation. AI does not remove the need for maintenance discipline. It raises the value of it.
Here are practical steps engineering leaders can take.
1. Audit AI workflows for token waste
Look at your highest-volume AI use cases. Identify what data is being passed into prompts or retrieval pipelines. Then ask how much of that context is actually used.
Common waste patterns include full transcripts when summaries would work, entire documents when specific sections are needed, duplicated records across joins, and raw event streams when aggregates would be enough.
2. Create modeled AI-ready datasets
Do not make every AI feature team clean the same data in application code. Build reusable modeled datasets in the warehouse. Use transformation tools such as dbt to create tested, documented layers for common AI contexts: customers, accounts, conversations, incidents, services, dependencies, and product usage.
3. Add lineage and ownership
When an AI output is wrong, teams need to debug the data path. Where did the context come from? Which transformation changed it? Who owns the definition? Lineage turns AI troubleshooting from guesswork into engineering work.
4. Design for retrieval quality
Retrieval is not only a vector database problem. Chunking, metadata, filtering, freshness, and semantic consistency all depend on upstream modeling. Better-modeled data leads to better retrieval, which leads to better answers with fewer tokens.
5. Govern sensitive and stale data
AI workflows can accidentally expose information that should not be in a prompt. Modeling layers give teams a place to redact, filter, classify, and enforce policies before data reaches the model.
6. Treat token budgets like performance budgets
Engineering teams already manage latency, memory, cloud spend, and database query cost. Token usage should be monitored the same way. Define budgets per workflow, track regressions, and make context size visible in code reviews and release checks.
The modernization takeaway
The teams that succeed with AI will not be the ones that send the most data to the largest model. They will be the ones that send the right data, in the right shape, at the right time.
The Gong transcript example shows that disciplined data modeling can produce dramatic AI cost reductions. The broader dbt argument about agentic failures reinforces the same point: AI reliability depends on the quality of the systems beneath it.
For developers, this means data transformations, tests, and documentation are now part of AI engineering. For CTOs, it means modernization roadmaps should connect application upgrades, data governance, and AI adoption instead of treating them as separate initiatives.
Modeling for the token is not a temporary optimization. It is a durable architecture principle for the next generation of software systems.