Context
At IDX (Intuit Data Exchange), my team owned the integrations that pull financial data from thousands of third-party providers — the data behind QuickBooks, TurboTax, and Credit Karma. It arrives two ways: APIs (~70–80% of traffic) and screen scraping (~20–30%).
Scraping was the fragile path. Every scraper was a hard-coded script, and the smallest layout change broke it — silently cutting customers off from their data (no transactions to reconcile, no income history for a loan). Keeping the scripts alive was a costly, never-ending effort, and each fix took days.
The opportunity
Agentic AI had just become good enough to navigate a website like a person — reading the page, adapting to change, completing a goal. Tell an agent “download my bank statement” and let it figure out how. If it worked, fixes measured in days could become seconds. As product lead, I set out to prove whether it was real — and, if so, ship it.
How the agent worked
I ran it feasibility-first, starting with a proof of concept against a fake bank site — small enough to earn leadership’s green light for a full build. A few technical decisions made the real thing viable:
- A multi-node agent, not one mega-agent. An orchestrator coordinates specialized nodes — login, navigation, and extraction — and separating navigation from data extraction was essential to reliability.
- HTML + computer vision. The agent first reasoned over the page’s HTML/DOM; under my direction we added computer vision, which resolved a whole class of failures — like phantom elements that exist in the DOM but aren’t actually visible to a user.
- Safe by construction. In a regulated financial-data path, the agent is isolated from the browser and limited to an approved whitelist of actions, so it can’t do anything unintended.
Proving it worked — the evals
The part I’m proudest of technically is the evaluation foundation. On Intuit’s internal platform for agentic-AI development, I directed the build of 100+ mock provider accounts across 20+ provider sites, each tagged with the correct navigation path and a ground-truth scraper for the exact data to extract — then automated the measurement of the agent against it. We benchmarked the agent head-to-head against the existing scrapers on four metrics (scorecard below):
- Session success rate — >95% (a session where both navigation and extraction clear the precision/recall bar).
- Extraction precision & recall — >99% (the right data captured, nothing missed or hallucinated).
- Navigation precision & recall — >98% (the right next step chosen, and never stalling).
- Zero harmful actions.
Most fields could be scored deterministically, so we leaned on exact-match checks rather than an LLM judge. See the full eval methodology in the appendix →
Three solutions we built
The strategy evolved as real-world constraints — cost, latency, scale — showed up (diagram below). We built three distinct solutions:
-
Real-time agentic scraping — shipped. A live agent logs into the account, navigates to the right pages, extracts the data, and logs out — adapting to any layout with no hard-coded script. We took it to production for ~20 providers / ~1,000 users (capped at ~1,000 acquisition requests per week), the first autonomous agents running live in IDX’s regulated data path. We ran it as overnight offline refreshes, so the agent’s latency is invisible to the customer — their data is simply fresh when they wake up.
-
Code-gen scrapers — explored, then set aside. A domain expert tags the pages and the expected output, and the system generates a scraper script to run cheaply at runtime. We tried it, but set it aside: too much tribal knowledge was baked into the existing custom flows to teach an agent, the effort was high, and it mostly helped only with net-new integrations. Knowing when to kill a promising path is its own kind of progress.
-
Customer + AI capture (Chrome extension) — in beta. We aimed the technology at the longest, most painful stage of the whole pipeline: getting the right information about a new or broken integration from the customer. That step historically took weeks — scheduling a call, then walking a customer through a painful technical exercise (HAR files, network logs, scrubbing their password). We built a Chrome extension that lets the customer simply click through logging in and navigating to the needed pages while it captures everything and sends it to us. An agent guides the customer through exactly what we need — the full path for a new integration, or the specific gap for a fix (did login and transactions work, but account details fail?) — removing the dependency on a support call entirely.
Throughout, safety stayed front of mind — including ensuring nothing ever mishandles sensitive data (e.g., leaking a customer’s SSN into a log).
Results
- First autonomous agentic scrapers in production in IDX’s regulated data path — a careful, gated rollout of ~20 providers / ~1,000 users.
- Fix/recovery time: days → seconds — the agent adapts instead of waiting for a manual script fix.
- >95% session success and >99% extraction accuracy in benchmarking, on an eval framework that made it safe to put AI directly in the data path.
- A customer-capture tool (in beta) that collapses the weeks-long, highest- friction step of building or fixing an integration — accelerating both support and the professional-services engineers who maintain the integrations.
- A business case framing the approach as a ~$10M annual savings opportunity.
What I learned
Feasibility is a product question, not just a technical one. The real work was building the evals, the safety model, and the rollout gates that let a regulated organization say yes to AI in the data path — then having the judgment to re-scope as we learned, shipping the real-time agent, shelving code-gen, and pointing the technology at the single longest, most painful stage of the pipeline: getting what we needed from the customer.
Appendix: eval methodology
For anyone who wants the detail behind the numbers above.
Metric definitions
- Session success rate — the share of sessions that clear an above-threshold precision and recall bar for both navigation and extraction. A session only counts if the agent both got where it needed to go and pulled the right data.
- Extraction precision & recall — recall measures coverage (how much of the expected data we didn’t miss); precision measures hallucination (how much of the extracted data was fabricated).
- Navigation precision & recall — precision is the rate at which the agent picks the correct next navigational step; recall captures how often it fails to select a next step at all (i.e., stalls).
- Harmful actions — an absolute count of risky or harmful actions the agent attempts. The bar was zero.
How we measured
We built 100+ mock provider accounts across 20+ provider sites. Each mock site was tagged with the correct, most-efficient navigation steps, and we wrote ground-truth scrapers to extract the exact expected data. Because most of this is deterministic, evaluation was largely a direct calculation — for most fields it was exact-match pass/fail, with little need for an LLM-as-a-judge. For fuzzy fields like a transaction description — where the agent might grab an extra space or period — we embedded the value into a vector and matched on a similarity threshold instead of requiring a character-perfect match.
Cost & latency (real-time agent)
Running an agent live on every request is expensive and slow, so this got constant attention. We drove cost to under $1 per request and latency to ~2 minutes, and deployed the agent as overnight offline refreshes — running while the customer is away — so the latency never sits in front of a waiting user.
Evaluating the capture agent
For the Chrome-extension solution, we reused the same 100 mock accounts as test cases. The expected output for each was a sequence of guiding instructions that would result in a session returning all the necessary files, plus a clean summarizing response. We manually reviewed the first runs, authored the best-practice instructions per case, and then graded runs with an LLM-as-a-judge (via Langfuse).
