How to write an AI vendor assessment questionnaire

An AI vendor assessment questionnaire should expose how a supplier's service behaves with your data, not test whether its sales team knows security vocabulary. Twenty answerable questions can do that if each one names the boundary, asks for evidence, and leaves room for an honest exception.

Most questionnaires fail because they mix four different things: the vendor's company controls, the AI application, the underlying model provider, and the customer's own configuration. A supplier answers "yes" because one layer has a control, procurement reads that answer as applying to every layer, and both sides discover the gap after a sensitive workflow is live. I have reviewed enough vendor packs to distrust a clean page of yeses. A qualified answer with an owner and a date tells me more.

The questionnaire below is an initial instrument for a startup or small business buying an AI service. It covers data, models, security incidents, and exit in exactly 20 questions. It does not replace a privacy review, contract review, penetration test, or threat model for the intended use. It tells you where those deeper reviews need to spend time.

Define the service before asking about controls

The supplier can answer consistently only when you name the exact product, deployment, and intended use. "Assess Vendor X" is too broad. A company may sell a public chatbot, an enterprise workspace, an API, and a private deployment with different retention, subprocessors, model choices, and administrative controls.

Put a short scope block above the questions. Identify the service and plan, expected data classes, users, integrations, regions, and whether the system will only draft content or also take actions. Ask the vendor to state which legal entity contracts with you and which team owns the answers. Date the response and give it an expiry, usually tied to contract renewal or a material service change rather than an arbitrary annual ritual.

This scope also stops a common dodge: answering for the supplier's internal corporate environment when the question concerns the service you will buy. SOC 2 coverage for office systems does not prove that prompts are excluded from model training. An ISO certificate does not tell you whether an agent can call your payment API. Certifications can support an answer, but they cannot substitute for one.

Describe the planned workflow in operational terms. State who submits data, whether other people's data appears, what the output influences, which systems connect, and what a mistaken or hostile output could change. "Customer support assistant" is still too vague: one version drafts replies for review, while another reads private tickets and issues credits. Those deployments need different evidence and different approval gates even when the vendor and product are identical.

Keep corporate assurance and service behavior in separate columns. Corporate assurance covers matters such as hiring, access reviews, vulnerability management, and business continuity. Service behavior covers prompt retention, model routing, retrieval boundaries, tool permissions, and output handling. The first can support the second, but never infer a product control just because the company has a mature security program.

Use a response format that allows "yes," "no," "partial," "not applicable," and "unknown." Require a short explanation for every choice, including yes. For each control, ask for the owner, evidence reference, and any dependency on customer configuration. A supplier that cannot disclose sensitive evidence can name the document, issuer, scope, and date under confidentiality rather than attaching it to email.

NIST AI RMF groups work under Govern, Map, Measure, and Manage. That separation is useful here because it stops policy from masquerading as measurement or response. The 20 questions deliberately ask what the system does and how the supplier verifies it, then ask who acts when it fails.

Questions 1-5 establish the data boundary

Data questions must trace content through collection, processing, storage, training, logs, support, and deletion. "We do not sell your data" answers almost none of that. Use these five questions verbatim, then replace bracketed examples in your scope note with your actual data classes.

1. What customer data does the service receive or create, including prompts, files, outputs, embeddings, metadata, feedback, logs, and support content, and where does each category flow?
2. For each data category, what is the purpose, storage location or region, default retention period, and shortest configurable retention period?
3. Does any customer data influence model training, fine-tuning, evaluation, human review, or product improvement, and what exact setting or contract term prevents each use?
4. Which subprocessors and model providers can receive customer data, what do they receive, and how will you notify us before that list or purpose changes?
5. How can we retrieve, correct, restrict, and delete customer data, including backups, derived data, embeddings, logs, and content held by subprocessors?

A complete answer to question 1 should look like a small data inventory, not a paragraph about encryption. Ask the supplier to distinguish content from operational metadata. Metadata can reveal project names, user identities, timestamps, file names, IP addresses, token volume, and integration targets even when the supplier never stores prompt text.

Question 3 needs separate answers for training, evaluation, human review, and general improvement. Vendors often collapse them into "training" and then reserve broad evaluation rights elsewhere. Ask whether the exclusion is the default for your plan, an administrator setting, a flag for the account, or a negotiated term. Record who can change it and whether the change applies prospectively or to data already collected.

Deletion deserves a lifecycle answer. Immediate removal from the active database can coexist with months in backups, indefinitely retained abuse logs, or embeddings that remain searchable. You do not need magical deletion from immutable backups. You do need a defined expiry, access limits during that period, and deletion propagated to downstream processors.

Questions 6-10 make model claims testable

Model questions should identify what can change, how the supplier evaluates changes, and which decisions still belong to people. A model card can help, but your risk comes from the assembled application: model, retrieval, prompts, tools, filters, permissions, and user interface.

6. Which model families, versions, hosting arrangements, and fallback models can process our data, and can we pin or approve material model changes?
7. What evaluations do you run before release for our use case, including accuracy, harmful output, data leakage, prompt injection, and tool misuse, and what release thresholds apply?
8. How do you monitor model and application behavior after release, investigate regressions, and roll back a model, prompt, retrieval source, or tool configuration?
9. Where does the service make or recommend consequential decisions, what information explains an output, and which decisions require human approval?
10. What technical limits prevent the model or agent from accessing data and taking actions beyond the current user's permissions and intended task?

Do not accept "we use the latest model" as an answer to question 6. Automatic upgrades trade operational convenience for unreviewed behavior change. If pinning is impossible, ask for notice, a change log, regression results relevant to your workflow, and a rollback path. A supplier can provide useful control without disclosing model weights or proprietary prompts.

Question 7 separates an evaluation name from an operating threshold. "We test for hallucinations" is a claim. A useful response defines the test set owner, sampling method, metric, acceptance threshold, failure handling, and last run. For a drafting assistant, an error may be annoying. For an agent that issues refunds, the same error reaches a system of record. The evaluation must match the action.

OWASP's Top 10 for LLM Applications treats prompt injection, sensitive information disclosure, supply chain weakness, improper output handling, and excessive agency as distinct failure modes. That is the right distinction. A system prompt that tells the model to ignore malicious instructions does not enforce authorization. Question 10 should produce evidence about scoped credentials, permission checks on the server, tool allowlists, argument validation, transaction limits, approval gates, and isolation between tenants.

Keep retrieval and training separate as well. Retrieval places selected material into the application's context when a request runs; training or fine-tuning changes model behavior through a different process. A vendor can truthfully exclude your data from training while still storing it in a retrieval index, sending retrieved passages to a model provider, and retaining those passages in traces. Ask about each path rather than treating "not trained on your data" as a complete privacy answer.

The same precision applies to human approval. A button labeled approve provides little protection if the reviewer cannot see the proposed tool, target, arguments, source material, and expected effect. Ask what context the interface presents, whether the request can change after approval, and whether the log binds the approved request to the action that ran.

Questions 11-15 reveal security and incident readiness

Security questions should follow an incident from prevention through detection, containment, notice, and learning. A generic security policy proves that someone wrote a policy. It does not show whether the AI service emits the events needed to investigate a leaked prompt or an unauthorized tool call.

11. Which controls protect customer data in transit, at rest, and between tenants, and how do you test tenant isolation and privileged access?
12. What service events do you log for authentication, administration, data access, model requests, retrieval, tool calls, policy blocks, and exports, and which logs can customers obtain?
13. How do you find and handle abuse specific to AI, including prompt injection, sensitive output, poisoned retrieval content, model extraction, and unauthorized agent actions?
14. What is your incident process, who is on call, what evidence can you preserve, and within what contractual period will you notify us of an incident affecting our data or service?
15. During the last relevant incident or exercise, what failed, how long did detection and containment take, and which corrective actions remain open?

For question 11, ask which party manages keys and whether backups share the same protection, but do not let an encryption discussion consume the review. Most practical failures happen while authorized systems process plaintext: a support engineer opens the wrong tenant, a retrieval query crosses an index boundary, or an agent uses an overbroad credential. Access design and isolation tests matter as much as algorithms.

Question 12 should yield an event catalogue with fields and retention, not a promise that "comprehensive logging" exists. You need enough context to connect a user, tenant, model, policy version, retrieval sources, tool request, approval, and result. You probably do not want raw sensitive prompts duplicated into a logging system. Ask how the supplier minimizes or redacts content while keeping investigation identifiers.

Question 15 makes many vendors uncomfortable, so permit an anonymized incident or tabletop exercise. "We have had no incidents" is not the end of the discussion. Ask for the last exercise involving model misuse or access between tenants and the tracker for corrective actions. Mature teams can discuss what their controls missed without exposing another customer.

Define the notification trigger before arguing over hours. Suppliers may start the clock when they confirm harm, while buyers expect notice when the supplier reasonably suspects unauthorized access. Those can be days apart. State the event that triggers notice, the channel and contacts, the minimum initial facts, update frequency, and how the supplier handles an investigation that later proves harmless.

Also separate an AI quality failure from a security incident without creating a blind spot between them. One wrong summary belongs in normal quality handling. Repeated citations between users, a bypassed approval, or model output that exposes secrets may require security response even if nobody has confirmed exploitation. The supplier should explain who can escalate signals from model monitoring into the incident process.

Questions 16-20 decide whether you can leave

Exit questions belong in the initial assessment because negotiating them after termination gives the supplier all the bargaining power. Portability is more than exporting chat history. You may depend on prompts, agent definitions, retrieval indexes, evaluation sets, audit records, user mappings, and integration configuration.

16. What customer content, configuration, prompts, agent definitions, logs, evaluation records, and metadata can we export, in which documented formats, and through what interface?
17. If we terminate, what is the timeline for access cutoff, export, deletion from active systems, backup expiry, subprocessor deletion, and written confirmation?
18. Which service components, model features, or proprietary formats would we need to replace, and what migration assistance is available at what contracted cost?
19. What happens to service access and our data if you discontinue a feature, replace a model, suffer a prolonged outage, change control, or cease operations?
20. Which obligations survive termination, including confidentiality, security, incident notice, legal holds, audit evidence, and restrictions on data use?

Test question 16 before signing if the workflow matters. Request a sample export or run one with synthetic data. Check whether the archive contains stable identifiers and documented relationships, and whether another system can parse it without reconstructing state from screenshots. A CSV of conversation titles is not portability if the business process depends on tool definitions and source citations.

Question 18 should expose economic dependency as well as technical dependency. A standard file format can still be difficult to use when the supplier withholds schema documentation, batches exports manually, or charges an unbounded fee for professional services. Record included assistance, rates, lead time, volume limits, and the person authorized to request an export.

Do not demand that every vendor guarantee continued operation after insolvency. Ask for a credible failure mode: scheduled exports, short retention of a recoverable snapshot, source escrow where justified, or a manual process your team has rehearsed. The right control depends on whether the service drafts marketing copy or operates production infrastructure.

Evidence turns prose into an assessable answer

Every material answer needs evidence proportional to the risk, but "attach all policies" creates work without clarity. Ask for a narrow artifact and verify that its scope matches the product and plan. Useful evidence includes a diagram of data flows, retention table, subprocessor register, model change record, evaluation summary, access review, isolation test, event catalogue, incident exercise report, sample export, or deletion certificate template.

Give the supplier a compact response object so answers arrive in a comparable shape. A spreadsheet works, but structured text handles long qualifications and version control better. This example is deliberately simple:

{
  "question_id": 10,
  "response": "partial",
  "explanation": "Tool access inherits workspace roles; payment actions require a separate customer approval rule.",
  "customer_action": "Enable the approval rule before connecting the payment system.",
  "evidence": ["Agent authorization design v3", "Tool call isolation test 2026-04"],
  "owner": "Head of Product Security",
  "valid_until": "2027-04-30"
}

The value "partial" is often more useful than yes or no. It names a boundary the contract, deployment plan, or risk acceptance can address. Do not punish honest qualifications by automatically lowering the vendor's score. If your process rewards unconditional yes answers, suppliers will learn to remove detail.

Check evidence at three levels. First, confirm identity: title, owner, version, and date. Second, confirm scope: legal entity, service, plan, region, and system components. Third, confirm substance: does the artifact demonstrate the claim, or merely repeat it? A policy that says access reviews happen quarterly is weaker than a dated review record with findings and closure status.

Sensitive artifacts need controlled review. Let the supplier share a redacted extract in a data room, present it during a call, or provide an independent assessor's scoped conclusion. "Confidential" cannot excuse every missing answer, but procurement should not create a new security problem by emailing penetration test details to a large distribution list.

Score exposure and exceptions, not vendor polish

A useful scoring model separates the inherent impact of the intended use from the supplier's control strength. Otherwise a polished writing tool with limited risk can appear equivalent to an agent with access to production and finance systems. Score the planned deployment first, then judge whether each answer reduces that exposure.

Use four impact bands: low for public or synthetic data with no actions, moderate for internal data or reversible workflow effects, high for confidential or personal data and important recommendations, and severe for privileged access or actions that can materially affect people, money, or production. The names matter less than the decision rules. Write examples from your business beside each band.

For each question, record one disposition: accepted, accepted with customer configuration, remediation required before launch, temporary exception, or disqualifying gap. Add an owner and due date to anything except accepted. Averages hide fatal gaps, so define gates. For example, a severe usage scenario may require tenant isolation evidence, contractual incident notice, no training on customer content, scoped agent credentials, and a tested export before approval regardless of total score.

Do not add points for a familiar logo, a long policy library, or confident meeting performance. Small vendors sometimes have thin documentation and strong architecture. Large vendors sometimes send an assurance pack that never answers how your selected feature works. Score the scoped service, then record company viability and your contractual position as separate commercial risks.

Cloud Security Alliance's AI Consensus Assessment Initiative Questionnaire maps a much larger question set to its AI Controls Matrix. Use that kind of control library when the first pass exposes high risk or when a regulated customer needs detailed mapping. Sending hundreds of questions to every supplier at intake produces copied policy language and weeks of delay. Twenty targeted questions produce a map for deeper diligence.

A clean yes can conceal a broken deployment

Consider a support team buying an AI agent that reads tickets, searches internal articles, and may issue account credits. The vendor says customer data is not used for training, supports access based on roles, encrypts data, logs activity, and has an incident plan. A conventional questionnaire records five yeses and procurement approves the tool.

The deployment still fails. Ticket attachments enter the prompt and remain in abuse monitoring logs longer than the buyer expected. Retrieval uses one shared index with application filters, but the isolation test covers only the primary database. The agent inherits a service credential that can issue credits for any account. Logs record the final answer but omit retrieved document identifiers, tool arguments, and the approving user. Nobody can reconstruct why a credit happened.

The 20 questions expose each gap without assuming malice. Question 1 inventories attachments and logs. Question 2 asks retention by category. Question 10 asks how permissions constrain actions. Questions 11 and 12 demand isolation evidence and an event catalogue. Question 14 fixes notification timing, while question 16 tests whether the buyer can recover the configuration and audit trail.

The correct decision might still be yes, with a narrower launch. Disable credits, restrict retrieval to a synthetic knowledge base, set log retention contractually, and run the isolation test before adding confidential tickets. That is a controlled deployment, not a security team's vague promise to "monitor closely."

This failure also shows why a vendor review cannot carry the whole decision. The supplier controls its service. You control data classification, identity configuration, connected systems, approval thresholds, and user behavior. Put each customer action beside the vendor answer and block production access until its owner closes it.

Put the answers into the contract and operating plan

Questionnaire answers decay unless the contract and service owner keep them alive. Move material commitments into the data processing terms, security schedule, order form, or service levels. A sales response that conflicts with standard terms will be hard to enforce when an incident occurs.

Contract the items that would change your purchase decision: permitted data uses, training exclusion, retention, subprocessor notice, incident notification period, audit cooperation, notice of material changes to models, export rights, deletion timing, and exit assistance. Keep implementation details in a referenced security document when they need to evolve. Require notice or equivalent protection if the supplier changes a control you relied on.

Turn customer dependencies into launch tasks. If the answer says an administrator must disable history, configure regional storage, restrict connectors, or require approval for tool calls, name the administrator and retain proof of the setting. Recheck those controls after plan changes and major releases. An assessment approved for a drafting pilot does not authorize an autonomous production agent six months later.

Assign a service owner who receives subprocessor notices, notices about model changes, incident communications, and service retirement announcements. Procurement mailboxes often collect notices without anyone judging the impact. The owner should trigger reassessment when data classes, integrations, agency, models, regions, or contract terms change.

For founders without a dedicated security or procurement function, the Team & AI Audit at oleg.is can examine the intended use, vendor answers, and internal deployment choices as part of a broader review lasting five business days. The useful outcome is a decision record with conditions, owners, and costs, not another binder of policies.

Send twenty questions and insist on precise nouns

The best vendor assessment starts with a narrow scope, twenty questions, and an answer format that welcomes qualifications. Give the supplier enough context to answer for the service you will actually deploy. Set a reasonable deadline, offer one clarification call, and ask responders to mark trade secrets for controlled review rather than omitting them.

Reject pronouns that blur responsibility. When an answer says "we do not retain it," ask which legal entity, which system, which data category, and which processor "we" and "it" mean. When it says "encryption standard for the industry," ask for the boundary and who owns the encryption keys. When it says "human oversight," ask which person approves which action, in which interface, with which evidence.

Do not use the questionnaire to force every supplier into the same architecture. A no answer can be acceptable for an intended use with limited impact, and a yes can be inadequate for a severe one. Record the exposure, evidence, exception, customer dependency, and decision owner. That record lets you explain the purchase after the salesperson, reviewer, and model version have all changed.

Send the questionnaire before the final commercial negotiation, then test the two claims that matter most to your intended use. For an agent, test authorization and rollback. For a knowledge assistant, test tenant isolation and deletion. For an API that handles high volume, test behavior after model changes and export. The document opens the review; the evidence decides it.