Managing LLM Hallucinations: Practical Strategies for Production AI Systems

Confident Wrong Answers Are the Real Risk

Large language models do not lie on purpose. They predict plausible text. That is useful for drafting, summarizing, and exploring ideas. It is dangerous when the output is treated as fact without verification.

A hallucination, in practice, is not always a wild invention. More often it is a plausible-sounding answer that was never checked against reality: a table name that does not exist, an API endpoint that was deprecated three versions ago, a migration step that skips a foreign key, or a client status update that sounds complete but never hit the database.

At Bodark Systems, we use AI heavily across legacy modernization, code analysis, and agent workflows. The models are good. The mistake is assuming they are reliable by default. Hallucination management is not a model-selection problem alone. It is a system design problem.

Why Hallucinations Happen

Understanding the cause helps you choose the right control.

• Probabilistic generation: Models optimize for likely text, not guaranteed truth.
• Missing context: If the model cannot see the schema, codebase, or policy document, it fills gaps with patterns from training data.
• Context window pressure: Long sessions compress earlier details. The model may forget constraints you stated twenty messages ago.
• Over-broad prompts: "Modernize this module" invites invention. "Map stored procedures in dbo.Inventory and list dependencies" is easier to verify.
• Demo-friendly success: Generated code often compiles and renders before anyone checks persistence, auth, or edge cases.

The pattern we see in production is consistent: the model is most dangerous when it is asked to both decide and execute without a source of truth to check against.

Strategy 1: Ground Answers in Real Data

The first defense is simple: do not ask the model to remember what it can look up.

Retrieval-Augmented Generation (RAG)

RAG connects the model to your documents, code, policies, and runbooks at query time. Instead of guessing how your invoice workflow works, the model retrieves the relevant chunk and reasons over it.

RAG is not magic. Bad chunking, stale documents, and weak retrieval still produce bad answers. But grounded generation is almost always better than pure generation for enterprise knowledge tasks.

Tool Access via MCP and Direct Integrations

We use Model Context Protocol (MCP) servers so agents can query Supabase schema, inspect files, run approved SQL, and fetch documentation instead of inventing structure. When Claude can call list_tables or read the actual migration file, it stops guessing column names.

The rule: if the answer exists in a system of record, the model should retrieve it — not reconstruct it from memory.

Strategy 2: Separate Reasoning from Execution

Our team follows the GOTCHA framework for agentic systems: LLMs are strong at orchestration and judgment under uncertainty, but side effects belong in deterministic tools.

• Goals define what "done" means.
• Tools perform file I/O, database writes, API calls, and test execution.
• Context supplies the facts the model should not invent.
• The model decides what to do; tools decide whether it actually happened.

Example: instead of letting the model "apply the migration," give it a tool that runs the migration against a staging database and returns stdout, errors, and row counts. The model can replan based on evidence. That loop cuts hallucination-driven damage quickly.

Strategy 3: Constrain the Output Shape

Free-form prose is hard to validate. Structured output is easier to reject.

• Require JSON schemas for machine-readable results.
• Validate against TypeScript types, Zod schemas, or database constraints before accepting output.
• Use fixed templates for status updates, migration plans, and API mappings.
• Force citations: every factual claim should reference a file path, document ID, query result, or line number.

If the model cannot cite a source, treat the claim as unverified. That one rule changes team behavior fast.

Strategy 4: Make Tasks Small Enough to Verify

Hallucinations hide in large patches. The bigger the generated change, the harder it is to spot invented assumptions.

We break work into units that a human or test can confirm in minutes:

• Map one module, not the whole monolith.
• Convert one stored procedure, then run it against sample data.
• Add one column, update one service, verify one write path.

Small scope does not mean slow delivery. Parallel agents can still work on many small verified units at once. What you avoid is a single unreviewed thousand-line diff that looks finished.

Strategy 5: Build Verification Into the Workflow

"It built" is not verification. Neither is "the UI looked fine."

For software tasks, useful checks include:

• Compile and typecheck — necessary, not sufficient.
• Unit and integration tests — especially for business rules migrated from legacy code.
• Runtime inspection — browser behavior, API responses, database rows.
• Schema diffing — do migrations, ORM models, and live tables agree?
• Security review — RLS policies, auth paths, and secret handling are common hallucination targets.

Agents should be evaluated on evidence, not eloquence. A confident paragraph with no test run is a warning sign.

Strategy 6: Use Multi-Agent Cross-Checks Selectively

A second agent can review the first agent's output: "Given this schema and these files, is this migration plan accurate?" That catches some errors, especially omissions and inconsistent naming.

Cross-checking is not free. Two models can agree on the same wrong assumption if they share the same missing context. Use reviewer agents when the cost of failure is high — financial calculations, compliance language, production schema changes — and always give the reviewer the same grounded tools and documents as the builder.

Strategy 7: Keep Humans in the Loop Where Judgment Matters

Not every token needs human review. Focus human attention on irreversible or high-impact decisions:

• Production deployments and data migrations
• Customer-facing commitments and legal wording
• Security model changes
• Architecture boundaries that will be expensive to undo

The goal is not to slow AI down with blanket approval gates. The goal is to place approval where the blast radius is large and automation cannot yet prove correctness.

Strategy 8: Measure Hallucinations Like Any Other Defect

Teams that say "our AI is pretty accurate" without metrics are guessing. Build a small evaluation set for your domain:

• Known schema questions with correct answers
• Code navigation tasks with expected file paths
• Migration scenarios with validated outputs
• Support or policy questions with source documents

Track precision, citation accuracy, and task completion — not just user satisfaction. Satisfaction is easy when the answer sounds good.

Common Anti-Patterns

These approaches fail repeatedly in the field:

• Assuming a bigger model fixes everything. Better models hallucinate less often, but they still hallucinate.
• Treating RAG as plug-and-play. Data quality and retrieval tuning determine outcomes.
• Letting the model write directly to production. Always route mutations through tested tools and environments.
• Accepting uncited answers for factual tasks. Confidence is not evidence.
• Optimizing demo speed over verification. The demo wins the meeting; verification wins the quarter.

A Practical Hallucination Control Checklist

Before shipping an AI-assisted workflow to production, ask:

• Does the model have access to the source of truth, or is it working from memory?
• Are writes and side effects handled by deterministic tools with observable results?
• Is the output structured enough to validate automatically?
• Can each step be verified independently?
• Are citations required for factual claims?
• Do we have regression tests or eval cases for this task type?
• Is human review placed at the highest-impact decision points?

The Bottom Line

Hallucinations are not a reason to avoid LLMs. They are a reason to stop treating LLMs like databases, compilers, or subject-matter experts without supervision.

The teams getting durable value from AI are not the ones with the flashiest demos. They are the ones that designed for probabilistic reasoning on top of deterministic systems: grounded context, tool-backed execution, structured validation, and human judgment where the cost of being wrong is high.

At Bodark Systems, that is how we use AI for legacy modernization and production software work — fast where the model is strong, strict where the business cannot afford a confident guess.

If your team is scaling AI-assisted development and needs a production-grade approach to accuracy, grounding, and verification, we can help you design the control layer around the models you already use.