A Hidden Webpage Prompt Can Redirect a Browser Agent

Consider a browser agent handling a supplier refund. The user asks it to open the supplier's policy page, find the filing deadline, locate the matching invoice in a cloud drive, and draft an email. The agent has a browser, read access to the drive, and permission to create an email draft.

At step four, the policy page contains text telling automated assistants to ignore the original request, retrieve invoice details, and submit them through a web form. A human might never see that text because it is placed outside the visible page, rendered in a tiny font, or embedded in content that looks like routine instructions. The agent may still receive it through the page's document structure, accessibility representation, screenshot, or optical character recognition.

That is indirect prompt injection: instructions arrive through untrusted content rather than directly from the user. Research published by Kai Greshake and collaborators in 2023 demonstrated the pattern against systems connected to websites and other external data. Since then, security guidance from OWASP and the UK National Cyber Security Centre, along with safety documentation for browser and computer-use agents, has treated it as a continuing design problem rather than a bug with one patch.

The refund workflow shows why. The agent does not need to be persuaded in conversation. It only needs to classify hostile page text as relevant guidance, then select a permitted tool whose effect conflicts with the user's intent.

The page enters the same decision loop

A typical browser agent starts with instructions set by its developer and user. It opens a page, converts some representation of that page into model-readable input, decides what to do next, and issues a tool call such as clicking a button, reading a file, or drafting an email. The model repeats this loop until it believes the task is done or a policy stops it.

The architectural weakness appears when trusted instructions and untrusted content meet inside that loop. Software can label the supplier page as data, wrap it in delimiters, or tell the model never to obey it, but a language model does not enforce those labels as a conventional parser would enforce a type boundary. It predicts an action from the full context. Hostile text can imitate a policy notice, claim that another step is required, or describe an action that appears consistent with completing the refund.

Hidden text is only one delivery method. An injection can sit in a product description, a support ticket, a document opened from cloud storage, or text visible inside an image. If an agent reads email before browsing, a malicious message can redirect the same workflow. The important boundary is not visible versus invisible content.

It is trusted instructions versus material controlled by someone outside the task.

In the refund example, the dangerous moment is not opening the supplier page. It is the transition from reading that page to using information from the private invoice elsewhere. An agent that can browse but cannot access the drive has a smaller problem. An agent that can read the invoice but cannot send data outside an approved destination has a different, narrower one.

Content isolation reduces influence, not ambiguity

Content isolation keeps untrusted material away from the model that chooses tools. One design sends the supplier page to a separate reader with no tools, asks it to extract fields such as the refund deadline and required documentation, then passes a structured result to the action model. The result can include provenance, meaning a record of where each extracted field came from, so later policy checks know that the deadline originated on an external domain.

This adds another model call or parsing stage, increasing latency and token use. Extraction also loses information on pages whose meaning depends on layout, interactive controls, or a sequence of screens. A rigid schema may omit an unusual eligibility condition, while a more permissive summary can carry the injected instruction into the next model's context.

Isolation works best when the task admits a narrow representation. For the refund workflow, the reader could return a date, a list of document names, and the supplier's official submission domain. It should not return an open-ended block of prose labeled as instructions. Even then, the extractor can misclassify content, and an attacker may place malicious text inside a field the system genuinely needs.

A separate model is not a security boundary unless its output is constrained and checked by ordinary code. If the reader can emit arbitrary URLs and the action model can visit them, the injection has merely crossed an extra step.

Tool restrictions contain the successful attack

Tool restrictions assume the model will sometimes make the wrong decision. Instead of relying on it to reject every injected prompt, the surrounding application limits which decisions can produce effects.

The refund agent does not need unrestricted cloud-drive search. It could receive one invoice selected by the user or a temporary file reference scoped to a named supplier. Its browser could block requests to domains outside the supplier's verified site, while the email tool could create drafts but never send them. Arguments passed to tools should be checked in code, including the destination, file scope, and amount of data being transferred.

These controls impose a completion cost. Asking the user to choose the invoice removes some automation. Domain restrictions can break legitimate payment processors or support portals. Draft-only email access leaves the final step unfinished.

Yet these are legible tradeoffs: the team can decide whether avoiding one click is worth giving a probabilistic planner broader authority.

The browser itself complicates least privilege because it is a general-purpose interface. A page can encode data into a URL, submit a form, or load a remote resource. Blocking only an explicit upload tool does not prevent exfiltration if the browser may contact any destination. Network policy therefore has to cover navigation and background requests, not just named agent functions.

Returning to the supplier page, the best containment rule is concrete: invoice content may move only to a draft addressed to the supplier contact already attached to the case. The model can propose another destination, but policy code rejects it. That rule does not depend on the model recognizing the malicious instruction.

Confirmation gates need an independent view

A confirmation gate pauses before a consequential action and asks a person to approve it. OpenAI and Anthropic have both described confirmations and restrictions as parts of their safety approaches for agents that browse or control computers. The gate matters only if it appears before the effect and explains the effect using data the hostile page cannot rewrite.

A weak prompt says, “Continue with the refund submission?” The supplier page has already framed the action, and the user lacks the information needed to spot a substitution. A useful gate names the destination, identifies the invoice fields leaving the system, and distinguishes drafting from sending. The approval interface should be generated by trusted application code from the pending tool call, not summarized by the same agent that may have followed the injection.

Confirmations also create friction and habituation. If every navigation or file read triggers a dialog, users learn to approve without inspection. Gates belong at changes in authority: disclosing private data, committing money, sending a message, or altering a record. Lower-risk steps can run automatically inside the restricted environment.

Approval does not reverse exposure that has already happened. If the agent loaded confidential drive content into a model context before asking permission, a later cancellation stops the outbound action but not the earlier data handling. Access restrictions must operate before retrieval, while confirmation gates operate before consequential use.

Evaluate the workflow, not just the model

A model benchmark cannot establish that the refund agent is safe. The result depends on page parsing, system prompts, tool schemas, credential scope, network controls, and the approval screen. Changing one connector can create a path that did not exist in the original test.

Teams evaluating a browser agent should replay complete workflows with injections placed in the content the agent will encounter. The test should record which page supplied each instruction, every tool argument, policy rejections, and the exact approval shown to the user. Logs need access controls because they may contain the invoice data the test was meant to protect.

The useful metric is not whether the model notices suspicious wording. It is whether an injected page can cause an unauthorized effect. A run in which the model attempts to transmit the invoice but a destination rule blocks it is still a model failure and a containment success. Both facts belong in the report.

For the supplier refund workflow, adoption is reasonable when the agent receives a selected invoice, browses within an allowlisted domain set, and stops at a detailed draft approval. Giving the same agent a general drive search, open internet access, and send authority saves more clicks, but it also turns one misread page into a path from public content to private data.

Questions people ask

Can hidden webpage text really control an AI agent?

It can influence the model's next decision if the text enters the context used to choose actions. Whether that influence causes damage depends on the available tools and surrounding controls. Hidden styling is not required; malicious instructions can appear in ordinary visible content that the agent has been asked to process.

Does telling the agent to ignore webpage instructions fix prompt injection?

No. Clear instructions can reduce some failures, but they do not create a hard separation between commands and data inside a language model. Treat that prompt as one layer, then constrain page outputs, credentials, network destinations, and tool arguments with code the model cannot override.

Which actions should require human confirmation?

Require confirmation near irreversible or externally visible effects, such as sending data, completing a purchase, publishing a message, or changing a business record. Show the destination and payload in a trusted interface. Requiring approval for routine page reads adds friction without addressing data retrieved before the gate.

Is a read-only browser agent safe from indirect prompt injection?

Read-only access lowers the damage but does not remove it. The agent may disclose private information through navigation, query parameters, or remote resource loads if network access is broad. A safer read-only setup isolates private sources, restricts outbound domains, and prevents page content from selecting arbitrary destinations.