The web library, read in order — from how an agent reads a page (pixels vs DOM/markdown) through the extraction and crawler tools (Firecrawl, Crawl4AI, Jina, Tabstack), web search and deep research (Tavily, Exa, Linkup, GPT Researcher, open deep-research agents), acting on pages (Browser Use, Stagehand, Playwright MCP, Skyvern, Browserbase/Steel/Browserless, and computer use), and the access and safety layer (web bot auth, llms.txt vs robots.txt, browser prompt injection).
The fight in browser automation isn't whether an agent can click. It's whether it reads the page's accessibility tree or its pixels — and which failure you'd rather debug at 3 a.m.
All three turn a webpage into clean markdown an LLM can read. They are not competing on that — they sit on three different rungs, and picking by star count gets the rung wrong.
Mozilla shipped a one-call API that turns any URL into structured JSON, cited research, or a finished browser task. The pitch isn't the features — it's that it obeys robots.txt on purpose.
They all give an agent the web, but they hand it back at different stages of doneness — raw links, cleaned pages, semantic matches, or a finished sourced answer. The price tracks exactly how much reading they did for you.
Three open-source answers to Deep Research, and they disagree on one thing — how the research loop is controlled. One project's benchmark proves that choice is the whole game.
OpenAI and Google ship deep-research as a closed feature. These seven open repositories let you run the same plan-search-read-synthesize loop on your own models, your own sources, and — if you want — entirely on your own machine.
A deep research agent hands you a long, confident, well-structured report. Grading it means measuring two different things at once — how good it reads, and whether a single sentence is actually supported.
Three projects give an agent a browser, but they disagree on what a page even is — pixels, DOM, or accessibility tree — and that one choice sets your token bill.
Both drive a real browser from natural language. But one reads the DOM and one looks at pixels — and that single perception choice decides your cost per step, your reliability on ugly sites, and whether you can even ship it in a closed product.
Your agent's automation framework drives the browser. This layer decides where that browser actually runs — and whether the sites it visits let it in.
A browser agent running through Playwright MCP spends roughly four times the tokens of the same task run through the CLI. The gap is real — but the cheap path isn't free. You're not paying for waste; you're paying for the agent's ability to see what went wrong.
Two ways to build an agent that drives software: send it screenshots and let it move the cursor, or hand it the page's structure and let it act on elements. The split isn't old vs new — it's general vs reliable.
For 25 years the web tried to detect bots by behavior and kept losing. Web Bot Auth gives up on detection and asks the bot to sign its name instead — and the big agent makers have already started doing it.
A year on, the data is in — almost nobody reads your llms.txt. The files that move the needle are the one that blocks crawlers and the content that earns a citation.
Nearly a year after the first Comet and Atlas exploits, the browsers' own makers say prompt injection may never be fully solved. The reason is structural, not a bug waiting for a patch.