Netscraper

Watch it work

JSON

XState + A2UI. That’s it.

The whole actor is one JSON file: an XState machine, plus A2UI views and JSONata routes. No code ships with it.

Target

TypeScript · WASM. Runs anywhere.

One runtime for Node and for a browser tab, where it runs inside QuickJS compiled to WebAssembly.

BYOD · bring your own driver

Any driver, same records.

• Playwright
• Puppeteer
• raw Chrome DevTools Protocol
• attach to a running browser
• CloakBrowser as the browser

In the replay, an AI agent (Claude, working only through Netscraper’s own tools) is asked for TikTok’s explore feed, logged out. It reads how open-source TikTok scrapers work, then browses like a visitor. When a login prompt covers the page it closes it, checks that it has gone, and carries on scrolling until it can show that paging really fired. Then it writes the manifest, tests it on 20 items, matches 5/5 target records field for field, and runs it at scale: 100 of 100 videos in 64 seconds, in 17 tool calls.

…but why?

A conventional scraper is a program. The site’s structure is buried in its code, the browser library is wired through it, and the program ships however you ship programs. When the site changes, which it will, you are back in the build-and-deploy loop to change one selector.

Netscraper splits the scraper in two. The runtime is generic and never changes per site. Everything site-specific lives in the manifest, and the manifest is data. The properties below follow from that split.

Portable

The same file runs on Playwright, Puppeteer and a raw Chrome DevTools Protocol socket, with byte-identical output in the project’s end-to-end test. It also runs in an ordinary browser tab as WebAssembly. Nothing is installed per site.

Cheap to change

A fix is a new manifest. An agent in the Studio re-reads the page, patches the file and proves it against what it saw. The recorded session took 17 tool calls from a blank page to a production run.

Continuously testable

Parity compares a run’s output with records captured from the live page, field by field. Each view carries a structural fingerprint, so a redesign is detected on the next run rather than discovered weeks later in bad data.

Failures are declared

Login prompts, cookie banners, captchas, missing accounts and empty pages are named states in the file, not exceptions in code. Work is split into idempotent units, so a crashed run restarts without duplicating records.

Small

The recorded manifest is about 24 KB as formatted JSON, 4 KB gzipped. Gzipped, the WebAssembly runtime and engine add about 505 KB, so the whole stack fits on a 1.44 MB floppy with room to spare.

Anatomy of a manifest

The design rests on a standard in the middle. A2UI is Google’s open protocol for agents to describe user interfaces as data: components such as text, lists and buttons, bound to a data model, sent as a stream of small JSON messages. Netscraper uses it the other way round. It describes an existing web page as if it were an A2UI interface. The rest of the system never sees the DOM, only a typed surface it can read and press buttons on.

Views

Each kind of page is a view: a match rule, A2UI components and a data model. The model is filled from the DOM, or better, from the page’s own network responses: a net pointer names a response by URL and selects from it with JSONata, a small query language for JSON. Capture is passive. Netscraper reads what the page already received and never replays or forges a request.

Machine

Navigation is an XState v5 state machine written as JSON. Its states navigate, wait for a view, or press an A2UI button; its guards read the data model. A selector never appears in it. Interrupt views, such as a login prompt, sit over the page as their own surface: the machine dismisses them and a deep history state resumes exactly where it was.

Routes

A route turns a surface into records: a key for de-duplication, a JSONata expression, a JSON Schema every record is checked against. Field names follow the most-used commercial actors, so the output drops in where theirs went.

Those three lines under machine.on and resume are the whole interrupt story. Whatever the machine is doing when a login prompt appears, it jumps to interrupted, presses the prompt’s own close button through an A2UI action, and returns to the exact nested state it left. The interrupted step runs again, which is safe because every unit of work is idempotent.

Bring your own driver

Everything the runtime does to a browser goes through one small interface, and it evaluates expressions as strings, never closures. That is what lets one engine drive a browser from another process, another machine or a WebAssembly sandbox.

In a browser tab, the runtime is bundled into QuickJS, a small JavaScript engine compiled to WebAssembly, running in a Web Worker. It drives a sandboxed browser on the server over a page-scoped DevTools socket. In the recorded session it started in 202 ms and collected 20 of 20 explore records in 50.1 seconds.

The run

On the live site, logged out, from Australia: 100 of 100 videos in 64 seconds, about 1.6 a second, after matching 5/5 target records field for field on a 20-item test.

An earlier run the same day was four times faster per record (158 ms) and stopped at 66 of 100. Its browsing had reported success on a page that never actually scrolled, so the manifest was designed against a feed that never loaded its second screen. Scrolling is now verified step by step, and the agent may not write a manifest until it can show the page moved, nothing covers it and paging fired. The run is slower as a result, but it completes.

What it costs

These figures are not like for like. A commercial actor’s price buys residential proxies, someone to fix it when the site changes, and support. What they show is that running the engine is cheap, and that fixing it, the part that used to need an engineer and a release, is now a supervised agent session measured in minutes.