The company you’re working for is about to spend millions on AI, or probably they already have. So often, the models are cutting-edge, the vendors promise magic, and yet—the AI is still wrong. Frequently, embarrassingly wrong. I’ve seen companies burn up millions of dollars because of a badly formed AI data architecture. Contracts split mid-sentence, financial tables disconnected from their headers, and customer conversations fragmented into nonsense. This isn’t an intelligence problem. It’s a data problem.
Welcome to Data Hell.
I’ve watched this scenario unfold repeatedly across enterprises diving head-first into AI: executives, dazzled by sophisticated models and shiny agentic search tools, overlook the mundane but crucial task of data preparation. The result? AI hallucinations, inaccurate retrievals, and costly errors. It’s the equivalent of handing Shakespeare to a reader—but only after tearing the pages at random intervals. No model, no matter how advanced, can reliably reconstruct meaning from that chaos.
The good news is there’s a clear, systematic way out. It’s not another expensive model, vendor switch, or bigger budget. It’s the diligent, meticulous work of proper data chunking and preparation—work most companies ignore because it’s not glamorous. After working with companies and watching some very public mistakes, I’ve distilled this into five critical principles that will transform data hell into something better: an organized data structure that AI can actually access.
FAQ: What’s chunking anyway?
Chunking is the essential process of breaking down large documents into smaller, meaningful segments—"chunks"—that an AI model can process effectively.
Done right, each chunk is a standalone piece of coherent information, making retrieval accurate and preventing costly misunderstandings.
Done wrong, chunks split sentences, separate context, and confuse meaning, causing AI to "hallucinate" or generate false responses.
Effective chunking respects natural document boundaries (like sentences, paragraphs, or sections), optimizes size for relevance and cost, and strategically overlaps content to preserve meaning. In short, chunking determines whether your AI reliably understands your data or confidently gets things wrong.
Good chunking depends on respecting semantic meaning. I lay out five principles for that below, but let me start with one example here—Context Coherence. Context Coherence is the principle that reminds you to figure out strategies that align to your data structures with where you chunk in order to preserve that meaning as you build your data system.
But that’s not all. In addition to laying out five principles for proper chunking, I cover lots of examples from various industries, including an extensive discussion of agentic search and Excel, and that’s not all! I’m including one more thing: a 61 page guide I put together on the principles of chunking.
Think of it as a companion piece to my guide to RAG a couple of weeks ago. If RAG answers: how should I structure an AI data retrieval system, chunking answers how should I prepare my data for RAG?
Inside the guide you’ll find detailed configurations for 10 crucial data types, code examples you can immediately implement, benchmark results to keep an eye on as you build, and troubleshooting tools to guide you through every obstacle.
Escaping data hell isn’t about chasing magical solutions; it’s about mastering the foundational work your competitors won’t bother with. Do it right, and you’ll have AI that doesn’t just demo impressively—it actually delivers.
Listen to this episode with a 7-day free trial
Subscribe to Nate’s Substack to listen to this post and get 7 days of free access to the full post archives.
