A friend and coworker, Martin Emde, dropped an offhand observation that's been rattling around my head ever since: when Claude Code fetches an llms.txt file, the structure comes back squashed. Indentation gone, the careful nesting collapsed into mush. He figured a well-built HTML page might actually make better WebFetch input than the plain-text file that was designed for exactly this.
Which is a weird thing to be true. llms.txt is a plain text file. There's nothing to parse, nothing to render, no markup to strip. It's the format whose entire pitch is "here's content already laid out for a language model." So what's eating it?
I went looking, because "huh, that's weird" is how most of my evenings start. The answer turned out to be more interesting than the bug, and it changes how much you can trust WebFetch any time the exact words matter.
The model never touches the page
Here's what actually happens when you WebFetch a URL.
WebFetch grabs the page. Converts the HTML to markdown. Then it hands that markdown to a separate, smaller, faster model with a prompt like "answer this question about the content," and the only thing that comes back to your main agent is that little model's answer.
Read that again. Your capable, expensive, top-of-the-line model never sees the page. A smaller one reads it, writes up what it found, and passes you the notes. What lands in your context is a summary of a conversion of the page. It's a game of telephone with exactly one middleman, and the middleman is a language model doing its best.
So nothing was wrong with Martin's llms.txt. The summarizer reworded it on the way through.
The docs are upfront about this now
When I first dug into this, the docs were quiet about the whole arrangement. They've since caught up, and credit where it's due, the Claude Code tools reference is pretty candid about it now:
WebFetch takes a URL and a prompt describing what to extract. It fetches the page, converts the response to Markdown when the server returns HTML, and runs the prompt against the content using a small, fast model. For most fetches, Claude receives that model's answer, not the raw page. The conversion step is not configurable.
This makes WebFetch lossy by design.
"Lossy by design." That's an honest line, and I'm glad it's there. The docs even tell you the escape hatch: "use curl via Bash for the unprocessed page."
The trouble is nobody reads tool docs that way. They're reference material. You skim them once when you're wiring something up, and then they just sit there. Nobody re-reads the WebFetch behavior section while an agent is firing off six fetches in the middle of a task. So the behavior ends up documented and invisible at the same time: right there on the page, and nowhere near your attention when a fetch scrolls past, returns something plausible, and you move on. The one time the paraphrase actually bites you, "well, it's in the docs" is cold comfort. That gap is most of why this post exists.
So this isn't secret behavior, and I'm not here to act like I cracked a vault. The top-level shape is documented. What the docs don't do is quantify the loss, and the loss has some sharp edges worth knowing about before you trust a fetch with something that has to be exactly right.
The part that bites
The small model runs under a copyright guard. In practice that means verbatim quotes are capped at 125 characters, and everything past that gets reworded. Not "maybe reworded." Reworded by default, because that's the safe behavior the wrapper prompt asks for.
That's fine when you wanted the gist. It is not fine when you wanted the bytes:
- Exact API signatures and method names come back paraphrased, and sometimes "cleaned up" into something that doesn't exist.
- Config snippets, version pins, and error strings (the stuff you need to match character for character) are not safe to trust out of a fetch.
- An
llms.txtor a README you fetched "to get the content" comes back as a recap of the content.
If you've ever had an agent confidently fetch the docs and then hand you a method signature that's almost right, this is probably why. The little model wasn't lying. It was paraphrasing, and a paraphrase of connect(host, opts = {}) can turn into connect(host, options) and nobody flags it.
And then the conversion eats the rest
The 125-character cap is the summarizer. There's a second category of loss that happens before the summarizer, in the HTML-to-markdown step. WebFetch uses stock Turndown with no plugins and no options, and that has consequences:
- No main-content extraction. There's no Readability pass, so the whole document converts: nav bars, footers, cookie banners, the lot. All of it becomes noise the little model has to wade through before it answers your question.
- Tables flatten. No GFM plugin means HTML tables don't become markdown tables. They become run-together cell text. A nice comparison grid turns into word salad.
- Most attributes get dropped. Turndown only keeps what its built-in rules use:
hrefon links,src/alt/titleon images. Everything else is gone. Image alt text survives, which is nice, but a button whose only label isaria-label="Close"is simply invisible to the model. So is anything leaning onrole,data-*,class, orid.
There's also a hard size cap: the converted markdown gets truncated at 100,000 characters before the little model ever sees it. Big page, late content, you'll never know it got cut.
Which brings us back to llms.txt
The irony I can't get over: Anthropic publishes an llms.txt. They're a listed early adopter of the format. And WebFetch, Anthropic's own tool, is one of the things that flattens it.
llms.txt actually dodges most of the damage above. It's text/plain, so it skips the HTML mangling entirely. No Turndown, no dropped aria, no flattened tables, because there's no HTML to convert. It passes through nearly verbatim into the block the little model reads (still under that 100k cap).
But "nearly verbatim into the summarizer" is not the same as "into your context." The summarize step still happens. The 125-character quote cap still applies. So the structure that makes llms.txt useful, the deliberate nesting and ordering someone hand-tuned for a model, gets reworded right back out. The format built to feed a language model cleanly gets handed to a language model that paraphrases it. That's the whole catch Martin spotted, mechanism and all.
So what do you actually do
When you need the real text, curl it:
curl -s https://example.com/llms.txt
That lands the full thing in context with no summarizer in the middle (assuming the host is on your session's network allowlist). It's the move for llms.txt, for exact docs, for API references, for anything where being off by a word is being wrong. The docs recommend the same thing, so this isn't me fighting the tool, it's using the right one.
And to be clear, WebFetch is still great. "What does this page say," "is there a section about rate limits," "roughly how does this API work," that's exactly what it's for, and the summarize step earns its keep there. It's just not the tool for "quote me the exact thing." Match the tool to whether you need the meaning or the bytes.
Two small tells worth pocketing while we're here:
- A WebFetch result that's exactly 215 characters is an HTTP 404. That's the canned "use an authenticated tool instead" message, and once you've seen it you'll recognize it on sight.
- You'll see this tool called both
Fetch(in the UI) andWebFetch(in transcripts, permission rules,allowedTools, hooks). Same tool.Fetchis just the display label;WebFetchis the real name everywhere that matters programmatically.
One more thing, and yes this is a little smug: that "lossy by design" quote up above? I curl'd the docs page to get it. I didn't WebFetch it. Because WebFetch would have laundered the very page that documents WebFetch, handed me a paraphrase of the sentence I wanted to quote exactly, and I'd have been right back where this post started. Eat your own dog food and all that.
How I actually know any of this
Everything above is checkable, and I'd rather show the method than ask you to take my word for it. This is the part I wish more "here's how the tool works internally" posts included, so here's how you can go poke at it yourself.
Two tools, because neither one sees the whole picture alone.
Spelunk the binary. Claude Code ships as a single Bun-compiled binary, and a compiled binary still has all its string literals sitting right there in the file. So strings over the binary plus grep gets you a shocking amount: the prompt templates, the size caps, the element-removal lists, the tool wiring. That's how I know the conversion is stock Turndown with style/script/noscript/iframe removed, where the 100k cap lives, and what the little model's prompt actually says.
The one trick: don't anchor your greps on the minified function names. Those get reshuffled every release, so a name that works today is garbage next week. Anchor on the stable strings instead, the human-readable prompt text and UI labels that have to stay constant because they're user-facing or load-bearing. Find those, then walk outward to the code around them. I keep the whole routine as a harness-binary-spelunking skill so I'm not re-deriving it from scratch each time, but the core of it is genuinely just strings, grep, and stubbornness.
Cross-check with real sessions. The binary tells you what the machinery is. It doesn't prove the machinery actually ran on a given fetch. For that I use cq, which lets me run SQL over my own Claude Code session transcripts. Join the tool calls to their results, pull up an old WebFetch, and you can see the input (URL plus prompt) and the output (the little model's answer) side by side. I had an old fetch of a project's docs where I'd asked it to "quote the description" and got back a tidy summary with only sub-50-character snippets. That's the 125-char cap showing up in the wild, not in a string table.
The catch that makes you need both: the intermediate prompt, the one handed to the little summarizer model, never gets written to the transcript. That sub-call happens inside the tool and vanishes. So the binary shows you the prompt that the transcript can't, and the transcript proves the behavior that the binary can only imply. Neither half is enough on its own.
That's the actual answer to "how do you know." It's strings, some SQL, and a coworker with a good eye for when something smells off.
So: thanks Martin, for noticing the squashed llms.txt and not just shrugging at it. And the next time your agent fetches a page and hands you something that's almost right, you know which model to blame, and you know to reach for curl.