Security Research / Agentic AI Systems — Primary Finding
Browser automation tools that drive AI agents send page DOM, accessibility trees, and screenshots to cloud LLM APIs — and without deliberate mitigations, sensitive form fields including passwords, session tokens, SSNs, and credit card numbers travel with them.
Every action step in a browser agent involves at least one — and often all three — of the following representations of the current page state being serialized into the LLM prompt:
| Data stream | Contents | Sensitive data risk | Default behavior |
|---|---|---|---|
| Accessibility tree snapshot | Role, name, state, and description of every DOM element: textboxes, labels, buttons, current values | Critical — password field labels and values appear verbatim | Sent on every step unless filtered |
| Screenshot (vision mode) | Full-page image, processed by vision-capable model via OCR-equivalent interpretation | Critical — anything visible on screen, including autofilled credentials | Enabled by default when use_vision=True |
| Raw DOM / page text | Semantic text extracted from page, potentially including hidden elements | High — hidden form field values, data attributes, inline JS variables | Used as fallback or enrichment |
A single Amazon product page generates 100,000–150,000 tokens of raw HTML; accessibility-tree pruning reduces this to a workable size while still preserving the semantic structure that includes all form field states. The arXiv paper Building Browser Agents: Architecture, Security, and Practical Solutions (2511.19477) confirms that "form field contents — potentially including passwords — are transmitted within accessibility tree snapshots sent to the LLM" and treats snapshot filtering as a mitigation, not a default.
# What the LLM actually sees in an unmitigated browser-use step:
ref=12 textbox 'Email' value='[email protected]' focusable
ref=13 textbox 'Password' value='hunter2!' type=password focusable
ref=14 checkbox 'Remember me' checked
ref=15 button 'Sign in'
# This snapshot is serialized into the API request body
# and transmitted to Anthropic, OpenAI, or whichever provider handles the step.
The Brave Security Blog (Unseeable Prompt Injections in Screenshots, Oct 2025) documents the screenshot vector in detail: both Perplexity Comet and the Fellou browser were found to transmit extracted screenshot text "to the LLM without distinguishing it from the user's query" — meaning visible page content overrides the trust boundary entirely.
dumps() and dumpd() serialization functions fail to escape user-controlled dictionaries with 'lc' keys, allowing attackers to instantiate arbitrary objects within trusted namespaces. A crafted LLM response can trigger exfiltration of environment variables — including API keys and credentials stored in .env files. Fixed in versions 1.2.5 and 0.3.81.CRITICAL
.env transmitted in request headers. Affects all v1.3 deployments. No patch merged as of disclosure.HIGH
sensitive_data parameter with GPT-4 and GPT-4 Mini, the agent inserts the literal placeholder keys (x_password) rather than the actual values into form fields. This means the substitution mechanism silently fails and — depending on how the agent recovers — may pass actual credential values into the LLM context on retry. Bug label applied; unresolved as of report date.MEDIUM
"Prompt injection, much like scams and social engineering on the web, is unlikely to ever be fully 'solved.'"
— OpenAI, responding to disclosure of AI browser vulnerabilities, TechCrunch Dec 2025The research community has converged on browser-based AI agents as a distinct and serious threat class. Key academic work:
font-size: 0px), off-screen positioning, and XML/SVG CDATA encapsulation — all designed to be invisible to humans but readable by agents scraping DOM content.OWASP lists LLM02:2025 Sensitive Information Disclosure as a top-ten risk, specifically calling out agents that include unredacted PII, credentials, and financial data in LLM context. The OWASP AI Agent Security Cheat Sheet recommends automatic detection and redaction of SSN, credit card, and credential patterns before context is assembled — but notes most production deployments do not implement this.
browser-use is the dominant open-source framework for Playwright-based AI agents. It ships a sensitive_data parameter specifically designed to prevent credentials from reaching the LLM. The design is sound in principle:
# Intended flow with sensitive_data:
agent = Agent(
task="Log in to GitHub",
sensitive_data={
"x_user": "myusername",
"x_pass": "mypassword"
},
use_vision=False # required — vision leaks screenshot
)
# LLM sees only: x_user and x_pass (placeholders)
# Real values are injected post-LLM via Chrome DevTools Protocol
The security model is: LLM plans and generates the action sequence using placeholder tokens, then a local layer resolves placeholders to real values and performs DOM injection — so real credentials never enter any API request. The SECURITY.md additionally recommends allowed_domains to prevent the agent from navigating to attacker-controlled pages and resolving credentials there.
GitHub Issue #1062 (open, unresolved): With OpenAI models, sensitive_data fails silently — the agent inserts literal placeholder keys into form fields rather than resolved values. The recovery path may pass actual credentials into the LLM on retry.
Vision mode enabled by default: Screenshots sent to vision-capable models include anything visible — autofilled usernames, error messages with account numbers, session data displayed on-screen. Must be explicitly disabled.
No allowed_domains = no guard: SECURITY.md warns that sensitive_data without allowed_domains "does not stop navigation by itself" — an agent can be injected into navigating to a phishing page and resolving credentials there.
Telemetry: browser-use collects tool usage, session duration, and model information by default. URL content and page content are stated as not collected, but this is a policy claim rather than an architectural one.
Ranked by effectiveness, based on current research consensus:
1 — Never pass credentials through the LLM context at all. Use a local credential injection layer (1Password's Secure Agentic Autofill, Vaultwarden + Cerberus KeyRouter, or equivalent) that intercepts placeholder tokens and resolves them via CDP without the LLM ever seeing real values. This is an architectural mitigation — the only one that eliminates the root cause.
2 — Set use_vision=False whenever the task involves any authenticated session or page that displays account-level data. Screenshot-based leakage is invisible to the user and hard to detect after the fact.
3 — Always configure Browser(allowed_domains=[...]) when using sensitive_data. Without it, the credential scope is unbounded and prompt injection can redirect credential delivery.
4 — Prefer storage_state='./auth.json' (cookie-based auth) over password-based login flows. This eliminates the credential-in-context problem entirely for sites that support it.
5 — Implement pre-LLM context sanitization: scan the accessibility tree snapshot for patterns matching [type=password], SSN regex (\d{3}-\d{2}-\d{4}), and card numbers before assembling the LLM payload. OWASP recommends full redaction to [REDACTED] for restricted-tier data.
6 — Run agents against a dedicated browser profile with no stored passwords, autofill data, or authenticated sessions for services outside the task scope.
7 — Treat LLM API call logs as a sensitive data store. If the API call logs are retained by your provider or your own infrastructure, any unredacted sensitive data that reached the LLM is now in those logs.
8 — For high-sensitivity workflows, run local models (Ollama, llama.cpp) rather than cloud API providers. This keeps sensitive data off third-party infrastructure entirely, though it trades capability.
Sources