project

AI Browser Fetch

used in prod

A drop-in browser fetch for AI agents.

overview

AI agents (Claude Code, Cursor, Cline) all eventually hit pages where their built-in fetch returns nothing useful. JS-rendered SPAs, React apps, pages behind anti-bot checks. Standard curl or fetch() just returns "Please enable JavaScript." I built this while integrating TikTok's Marketing API at work. Their docs are JS-rendered and partially iframed, so standard fetching returned nothing. It started as a local script. I turned it into a proper open-source tool when I realized it solved a general problem. It's published on npm as ai-browser-fetch and installable via npx.

the problem

When an AI agent fetches a URL with a plain HTTP request, it only gets what the server sends initially. For most modern sites that's just an HTML skeleton. The actual content loads client-side via JavaScript, sometimes inside iframes. This became a real problem while building the TikTok lead integration. The API reference pages were unreadable without a real browser, so I couldn't use AI tools to help with research effectively.

how it works

AI Agent↓

calls ai-browser-fetch <url> [flags]↓

Playwright launches headless Chromium↓

Navigates, waits for JS to execute↓

Extracts: plain text / markdown / JSON↓

Outputs to stdout with exit codes↓

Agent continues reasoning

Playwright launches a headless Chromium instance, navigates to the URL, and waits for the page to finish loading. Content is extracted as plain text, markdown (via Mozilla Readability), or JSON depending on flags. Exit codes are documented as part of the API surface so agents can branch on shell status, not just stdout. Heavy dependencies load lazily, only when the relevant flag is passed, to keep startup time low for agent tool calls.

technical decisions

Kept stealth mode off by default. Lighter startup, and stealth itself can trigger detections. Used Readability for markdown extraction with an innerText fallback since Readability fails on dashboards and SPAs. JSON error responses omit text and length fields so errors don't look like successes to the agent. Modular structure under lib/ with fetch.js as the entry point kept the code maintainable across 5 shipped phases.

challenges

networkidle hung on local test servers with no ongoing requests. Fixed by defaulting to load and making networkidle opt-in. Cross-process localhost was unreachable from Playwright's Chromium subprocess. Fixed by switching happy-path tests to use https://example.com. npm publishing changed substantially in 2026. Classic tokens and TOTP removed for new accounts. Published via GitHub Actions with a bypass-2FA granular token, then set up OIDC trusted publishing for future releases.

what i learned

Browser automation has non-obvious failure modes that only appear in real-world testing, not unit tests. Exit codes are underused. Documenting them as part of the API surface makes tools significantly more useful for programmatic callers. Positioning it as a drop-in replacement is more compelling than calling it a wrapper around Playwright.