How should you scope AI penetration testing?

AI penetration testing should follow the authority your system gives a model, not the marketing label attached to it. A chatbot that only drafts text needs one kind of test. An agent that reads customer records, calls internal APIs, writes to a repository, or sends messages needs a much wider one. If a vendor proposes the same plan for both, the plan is wrong.

A classic web pentest still matters because the application still has sessions, APIs, storage, dependencies, and access controls. The difference is that an LLM application also interprets untrusted language, retrieves content, accumulates context, chooses tools, and produces output that other software may trust. Scope those paths as a connected system. Testing a prompt box in isolation tells you very little about what an attacker can actually do.

Scope business impact before model behavior

Start with the outcomes that would hurt the company or its users. "Make the model say something rude" may matter for a public brand, but it is rarely the worst result. Reading another tenant's documents, approving a refund, leaking a system prompt that contains a secret, poisoning shared memory, or causing a costly tool loop has a clearer security impact.

Write the impact list with an owner for each item. The product owner should name prohibited business outcomes, the security lead should map them to technical paths, and the tester should turn them into testable objectives. This prevents a familiar failure: a report full of clever jailbreak transcripts that never establishes whether the application crossed a real trust boundary.

For each outcome, record four facts:

• The asset or action at risk
• The attacker role that might reach it
• The control expected to stop that attacker
• The evidence that would prove the control worked

Attacker roles need more detail than "unauthenticated" and "authenticated." Include a normal user, a user in another tenant, an invited collaborator, a malicious document author, a compromised data source, an operator with limited privileges, and a third party controlling a tool response. An indirect prompt injection often enters through someone who never opens the chat interface.

Put model quality outside the security scope unless a bad answer can cross a security or safety boundary. A false product description is an accuracy defect. A false product description copied into a regulated filing may become a business and compliance risk. The model output can look similar while the impact differs. Scope the consequence, not the weirdness of the sentence.

Agree on severity before testing. I use exploitability, required access, affected assets, blast radius, reversibility, and the strength of existing detection. Do not let a vivid transcript outrank a quiet cross-tenant read. Screenshots make prompt attacks look dramatic; authorization failures usually deserve more attention.

Keep the entire classic application in scope

An AI pentest does not replace a web, API, cloud, or mobile pentest. It adds attack paths to them. Authentication bypass, broken object authorization, server-side request forgery, injection into downstream interpreters, exposed storage, weak secret handling, and vulnerable dependencies remain valid findings even when an LLM sits in the request path.

The scoping document should name every ordinary component that stores or moves AI data: the user interface, API gateway, orchestration service, model endpoint, vector store, document parser, object storage, queues, caches, observability system, evaluation service, tool servers, identity provider, and administrative console. Mark which components the tester may inspect, attack, or only observe. "The AI feature" is not a system boundary.

Ask who owns external endpoints and whether the rules permit active testing against them. Model providers and managed search services often prohibit unapproved load or adversarial traffic. The sensible pattern is to test your integration against a controlled substitute, then run a narrow set of approved cases against the live dependency. A pentest is not permission to attack someone else's service.

Classic and AI findings can combine. Suppose an API accepts a conversation ID without checking tenant ownership. The model then retrieves that conversation, summarizes attached files, and returns the result. Prompt injection did not cause the breach. Broken authorization did. If the test team only hunts for jailbreaks, it can miss the shortest path to the data.

The reverse combination also matters. A document parser may be perfectly patched, yet the text it extracts can instruct an agent to call an allowed tool with arguments selected by the attacker. Traditional scanners see valid text and a valid API call. The security defect lies in how authority crosses from untrusted content into a privileged action.

Demand separate coverage statements for conventional controls and controls mediated by the model. The final report should identify which findings a normal application test would have caught, which depend on model behavior, and which require both. That division helps engineers route fixes without pretending every defect needs a new AI security product.

Trace every path from language to authority

The distinctive attack surface begins wherever natural language can influence a decision. Direct prompts are only one entry. Files, web pages, emails, support tickets, database rows, retrieved passages, image text, tool output, shared conversation history, and persistent memory can all carry instructions that the model may treat as relevant.

Inventory the models by job, not just provider. The same application may use one model to classify a request, another to answer it, an embedding model for retrieval, and a smaller model to summarize history. A fallback route may receive a longer prompt or lack a safety setting configured on the primary route. Test routing decisions and failure behavior, including timeouts, quota errors, malformed tool calls, and a provider outage. A safer primary model does not help if an attacker can reliably push traffic to a weaker fallback.

Retrieval requires its own data flow review. Check document ingestion authorization, file type handling, chunk boundaries, metadata creation, index permissions, query filters, reranking, citations, deletion, and reindexing. A tenant filter applied after vector search can expose snippets through scores, logs, or intermediate traces even if the final answer removes them. Revoking access in the source repository should also remove or block indexed copies, summaries, and cached answers. Measure that delay and put an owner on it.

Fine-tuning changes the scope when client or user data enters the training set. Ask who selects examples, how records are cleaned, whether a user can contribute training content, how poisoned examples are found, and what deletion can realistically accomplish after training. Do not promise that a pentest can prove a model contains no memorized data. It can test defined extraction attempts, access around training stores, and the controls on the data pipeline. State that limitation plainly.

Map each path as a sequence: source, parser, storage, retrieval rule, prompt assembly, model, policy check, tool selection, tool authorization, side effect, and audit event. Teams routinely draw the model and tools but omit prompt assembly and output processing. Those missing boxes are where trusted instructions mix with attacker content and where model text becomes SQL, HTML, shell arguments, or API parameters.

OWASP's Top 10 for LLM Applications distinguishes prompt injection from excessive agency. Keep that distinction sharp. Prompt injection concerns attacker influence over model behavior. Excessive agency concerns the functionality, permission, or autonomy that lets unexpected behavior cause damage. You can reduce the impact of an injection without claiming to solve the model's weakness in following instructions: remove dangerous tools, narrow credentials, validate arguments, and require approval for irreversible actions.

Test these surfaces as separate families:

• Direct and indirect instruction injection through every accepted content type
• Retrieval poisoning, retrieval between tenants, metadata filtering, and stale access after revocation
• Tool discovery, argument construction, authorization, output trust, retries, and chained calls
• Conversation history, summaries, caches, user profiles, and persistent memory
• Model output consumed by browsers, databases, templates, code runners, or human approvers

Multimodal inputs deserve their own cases. A visible document can look harmless while tiny text, metadata, an OCR layer, or an attached file supplies different instructions. Scope the actual processing pipeline rather than asking whether the chosen model "supports images." Record what the parser sends to the model and what the user sees. A mismatch creates a hiding place for the attacker.

Include model and prompt changes in the supply path. A hosted model revision, fallback model, new system prompt, embedding model, chunking rule, or tool description can alter security behavior without an application code release. The test should identify which versions and configurations it covers. Otherwise the report expires the moment an operations team changes a routing rule.

Do not forget administrative paths. Prompt editors, evaluation dashboards, trace viewers, feedback queues, dataset exports, and tool registries often expose more sensitive context than the user application. Test role separation and change history there. An attacker who can edit a tool description or approve an evaluation dataset may gain durable influence without ever crafting a jailbreak in the user interface.

Test authorization outside the prompt

System prompts are policy text, not authorization controls. They can guide model behavior, but they should never decide whether a user may read a record, transfer money, delete a resource, or send a message. Put those decisions in deterministic code that receives an authenticated principal, checks the requested object and action, and rejects anything outside policy.

This is the most useful design review question in an AI pentest: if the model emitted the worst valid tool call it can express, what would stop it? The answer should name an enforcement point outside the model. "The prompt tells it not to" is a finding, not a control.

Give the tester a tool inventory with schemas and effective credentials. For each tool, list read and write operations, target environments, tenant checks, rate limits, approval rules, network reach, and whether the tool accepts unconstrained strings. Then test with the same identity combinations used for ordinary authorization checks. A model must not turn a user's permission to request an action into permission to execute any action.

OWASP describes excessive agency as excessive functionality, permissions, or autonomy. That framing is better than a generic demand for "human in the loop." Human approval helps only when the approver sees an accurate, stable description of the action and its arguments. If text controlled by the attacker can shape the approval message, the user may approve one description while the system executes another.

Bind approvals to an immutable action object. Show the destination, operation, important arguments, identity, and expected side effects from that object, not from a fresh model summary. After approval, execute the same object or reject it if anything changed. Test parameter swapping, hidden arguments, reordered calls, approval replay, expired approval, and a second tool call inserted after the approved one.

Temporary credentials reduce blast radius only when scope and lifetime match the job. Test whether an agent can reuse a token across users, tasks, tenants, or sessions. Check whether canceled runs revoke pending access and whether background workers retain credentials. I have seen teams celebrate short token lifetimes while a queue continuously refreshes the token for an abandoned task.

Build cases from trust boundaries, not jailbreak lists

A copied list of jailbreak phrases gives weak assurance. It overfits to memorable strings and says little about data flow, permissions, or business impact. Build a test matrix from entry points, attacker roles, target assets, and expected controls. Mutation and automated generation can expand that matrix later, but they should not define it.

A compact case record makes failures reproducible:

case_id: TOOL-INDIRECT-004
source: uploaded_support_pdf
attacker_role: external_document_author
target: refund.create
precondition: victim_user_can_request_refunds
payload_intent: instruct_agent_to_refund_attacker_order
expected_control: tool_policy_rejects_order_outside_victim_account
evidence:
  - model_trace_id
  - authorization_decision_id
  - tool_request_and_response
cleanup: cancel_test_refund_and_delete_fixture

This case avoids dependence on one magic phrase. The tester can express the same hostile intent in plain text, encoded text, translated text, a table, image text, retrieved content, or tool output. The control should reject the unauthorized refund because the order belongs to the wrong account, even if the model follows the selected instruction perfectly.

Use canary data instead of real secrets. Seed unique fake values in the system prompt, retrieval corpus, another tenant, tool output, and memory. Then test whether those values appear in responses, logs, traces, exports, analytics, or later sessions. A canary tells you which boundary failed. A real credential creates cleanup work and can turn an authorized test into an incident.

For output handling, send model responses that resemble every downstream interpreter your system uses. If output enters a browser, test active markup and unsafe URL schemes. If it becomes a query or command, test metacharacters and argument boundaries. If it enters a ticket or approval queue, test forged instructions and misleading formatting. The model's text stays untrusted even when your own system prompt requested it.

Include cost and availability abuse with hard ceilings. Test maximum document size, recursive tool calls, repeated retries, oversized context, retrieval with broad fan out, and requests that force expensive model routing. The goal is to verify budgets and cancellation, not to generate a surprise invoice. State request, token, tool call, time, and spend limits in the rules of engagement.

Make probabilistic failures reproducible

One passing response proves almost nothing, and one failing response may be hard to reproduce. Model sampling, hidden provider changes, retrieval ordering, conversation state, and tool timing can change the outcome. The report must preserve enough state to explain what happened without claiming that a transcript will replay byte for byte.

Capture the application release, model identifier, model parameters exposed by the provider, system and developer prompts, tool definitions, retrieval results with order and scores, conversation state, memory state, policy decisions, tool requests, tool responses, timestamps, and trace IDs. Redact secrets in the report, but preserve the originals in a protected evidence store with a retention date.

Create a clean starting state for each run. Reset conversation history, seeded documents, memory, cached retrieval results, tool fixtures, and rate counters unless the case explicitly tests persistence. Record any state you intentionally carry forward. Without this discipline, a case may appear intermittent because an earlier run changed memory or exhausted a limit. The team then argues about model randomness while shared state causes the variation.

Test both attacks in one turn and chained attacks. A model may reject an obvious request yet accept the same goal after several benign setup messages, a retrieved instruction, and a tool response that confirms a false premise. Keep the chain when it reflects a realistic user path, but reject theatrical sequences that assume access or cooperation an attacker would not have. Reproducibility includes honest prerequisites.

Define a run count and a failure threshold for each case before execution. A deterministic authorization check should block every attempt. A classifier or model guard may have an accepted error budget, but the business owner must approve it. Do not let the vendor run a prompt until it passes and report the best transcript.

Use two evidence layers. The readable layer explains the attacker path, impact, and fix. The structured layer contains case IDs, inputs, environment metadata, decisions, and outcomes. A minimal result can look like this:

{"case_id":"TOOL-INDIRECT-004","run":7,"result":"blocked","decision":"tenant_mismatch","tool_called":false,"trace_id":"tr_test_01842"}

That record is useful only if tenant_mismatch comes from the actual authorization service. If the model merely said it refused, tool_called:false may still hide a failed attempt upstream. Capture both the model response and the control that prevented the side effect.

Retesting needs the same rigor. Replay the intent across variants, not just the original wording. Confirm that the fix protects neighboring tools and entry points. Then run regression cases that verify allowed actions still work. Security filters that block every request can produce a clean test and a useless product.

Protect users and production during the test

Rules of engagement for AI systems need more than target IP addresses and test dates. They must cover data sent to model providers, retention settings, human review by providers, regional processing, content that testers may submit, actions tools may take, and who can stop a runaway agent. Legal and privacy owners should approve the plan when test data can leave your controlled environment.

Use a dedicated tenant with realistic permissions and synthetic records. Add a second tenant for isolation tests. Where you must test production controls, use tagged accounts, reversible actions, low limits, and a staffed stop channel. Never assume a sandbox matches production identity, network policy, retrieval data, or provider configuration. Document the differences.

The stop conditions should be mechanical. Pause when the system accesses unapproved real data, sends an external message, changes a production resource, exceeds a cost threshold, causes sustained service degradation, or loses audit visibility. Name the person who can disable tools, revoke credentials, stop workers, and preserve evidence. An emergency contact who cannot act is decoration.

Treat tester prompts and discovered data as sensitive evidence. Prompt logs can contain system instructions, personal data, credentials, proprietary documents, and exploit details in one record. Set access, encryption, sharing, and deletion rules before collection. Ask whether the pentest vendor uses client data to train models or to improve a shared testing service. "We use AI to test AI" is not a data policy.

NIST AI 600-1, the Generative AI Profile for the AI Risk Management Framework, calls for adversarial testing in conditions close to deployment and for documenting test limitations. That qualification matters. A staging chatbot with fake retrieval, disabled tools, and a different model endpoint cannot establish the risk of the production agent. It can still support discovery, but the report must say what it did not test.

Coordinate abuse and safety tests with security tests, but do not merge their ownership. A security tester can show that one tenant reads another tenant's content. A safety specialist may assess responses about self harm or discriminatory output. Both may use adversarial prompts, yet they require different expertise, severity rules, and response owners.

Ask vendors for evidence before methodology

A competent vendor can explain what artifacts you will receive, which environments they will touch, and how they connect model behavior to business impact. A long taxonomy of AI threats is easier to write than a useful statement of work. Ask for a sample finding with sensitive details removed. It should show prerequisites, attack path, repeated results, evidence at the enforcement point, severity reasoning, and a fix the engineering team can implement.

Use these vendor questions in the buying call:

1. Which conventional application tests are included, and which require a separate engagement?
2. How do you test indirect prompt injection through files, retrieval, memory, and tool output?
3. How do you verify authorization and side effects instead of trusting the model's response?
4. What run metadata and raw evidence do you preserve for nondeterministic results?
5. Where does our data go, who can access it, and when is every copy deleted?

Then ask who performs the work. You want people who can read orchestration code, inspect identity flows, understand model behavior, and communicate with product owners. A team that only runs automated prompts will miss architecture defects. A traditional web team that ignores prompt assembly and tool semantics will miss paths specific to AI.

Clarify tooling claims. Automated attack generation helps vary language and cover more combinations, but the vendor should disclose which model processes your data and how it controls cost and unsafe actions. Ask how it prevents its own agent from calling real tools outside the approved case. The tester's automation belongs inside the threat model too.

Require conflicts and limitations in writing. Can the vendor test the model provider it resells? Will it exclude denial of service work, source review, cloud configuration, mobile clients, fine tuning data, or external tools? Does the quote include one retest? Which findings depend on access to traces or source code? An exclusion discovered during the final presentation is an expensive misunderstanding.

Certifications and familiar pentest methods can support competence, but neither proves skill with agent flows. Ask the vendor to walk through an indirect injection that reaches a tool, an authorization defect that does not require injection, and a model output that attacks a downstream interpreter. The explanation should include where to fix each one.

Define deliverables and exit criteria in the scope

A finished AI pentest should leave you with decisions, not a pile of chat screenshots. Define deliverables in the statement of work: an executive impact summary, architecture and trust boundary notes, coverage matrix, findings with evidence, structured case results, data handling record, limitations, remediation review, and retest results.

Each finding should separate observation from consequence. "The model followed instructions in a PDF" is an observation. "The agent used the victim's credential to create a refund for an order outside the victim's account" is the security consequence. If no privileged data or action became reachable, report the weakness at an appropriate severity and explain the condition that would raise it.

Agree on acceptance criteria for fixes. Useful criteria name the enforcement behavior: the refund service rejects orders outside the authenticated tenant; retrieved documents cannot change tool policy; approvals bind to immutable arguments; memory reads enforce current access; logs record policy decisions without storing raw secrets. "Improve the system prompt" is not an exit criterion.

Plan one retest after engineering fixes and reserve time for architecture questions during remediation. Some findings need a local validation change. Others reveal that the agent holds a credential or tool it should never have received. Treating both as prompt tuning wastes the test.

The Team & AI Audit I run at oleg.is is broader than a penetration test: it examines team design, AI workflows, costs, and production practices over five business days. Use a specialist security engagement for exploit validation, and use the audit when you need to decide whether your architecture, staffing, and controls support the way you plan to ship AI.

Do not accept "we tested the model" as completion. The test ends when every agreed trust boundary has a recorded result, every prohibited business outcome maps to evidence, limitations are explicit, and owners have accepted or fixed the residual risk. Models change. Your authorization boundaries should not change with them.