Questions to ask an AI vendor before you sign.

The security questionnaire comes back complete: encryption, access control, backups, audit reports, all in order. None of it tells you which model the product runs on, who can change that model, whether your data trains anything, or how good the product is on your cases rather than in the demo.

AI vendor due diligence needs those questions too. Add them to your existing questionnaire as a separate section, route the data-protection answers to your DPO and the contract points to counsel, and insist on a trial on your own examples. Here are about twenty, grouped, with what a good answer sounds like.

What do standard vendor questionnaires miss?

They were written for software that behaves the same way until the vendor ships a release. An AI product can change its behavior without one, when the model underneath is updated. Your data may flow on to a model provider as a sub-processor. Quality is not a feature you can tick; it depends on your cases, your languages and your documents. Shared indexes and caches can leak between customers in ways a classic application can’t. And leaving takes more than a data export: prompts, configurations and evaluation results are part of what you would need to rebuild elsewhere.

If a vendor can’t show quality on your cases, you haven’t seen quality yet.

Which questions cover data handling?

1. Which data goes to which model provider, and where is it processed and stored? A good answer is a diagram per feature, not “we use a major provider”.
2. Who are your sub-processors, and how are we told about changes? Look for a published list and notice before changes.
3. Is any of our data used to train or improve a model, yours or your provider’s? A good answer is yes or no, per model, with the clause that says so.
4. How long are prompts, outputs, files and logs kept, and can we shorten it? Ask per feature; retention often differs between them.
5. How is our data deleted, including from indexes, caches, backups and tuned models, and how is deletion confirmed? A good answer includes written confirmation and a timeline you can test.

KVKK and GDPR questions, such as transfers abroad, legal basis and the data processing terms, go to your DPO. Our note on where your data goes lists the hops worth asking about.

Here is an illustrative exchange on question 3. Vendor A answers: “We take privacy very seriously and never misuse customer data.” That is not an answer. Follow up: yes or no, for each model; the clause; what “improving the service” covers; and whether vendor staff read customer conversations. Vendor B answers: “No. Customer content is not used to train our models or our provider’s; see section 7 of our data processing terms. Prompts are kept for thirty days for abuse monitoring, in the EU, then deleted.” That is specific, so verify it: read the clause, check the provider’s terms for the plan the vendor uses, and confirm the retention setting in the admin console.

Which questions cover models and change?

6. Which models does the product use, for which features, and can we restrict them, for example to a region or a self-hosted option?
7. Who can change the model or the prompts, and how much notice do we get? Look for notice before changes, release notes and a way to test first.
8. How do you check that a model change didn’t reduce quality? A good answer names a fixed evaluation set and shares the results.
9. What happens if your model provider has an outage or withdraws a model? Ask for the fallback and the switching plan.

Which questions cover quality on your own cases?

10. Can we run a trial on our own examples, against our own criteria? The right answer is yes: a few hundred of your real cases, scoring rules agreed in advance, and results case by case rather than one average.
11. How do you measure quality in production, and what will we see? Sampled reviews, override rates and trends, not only uptime.
12. How does the product show its sources and its uncertainty to users? Look for citations to the exact passage and a visible “not sure” state.
13. How was it tested in our languages and on our terminology? For Turkish or Arabic, ask for tests with native reviewers on text like yours.

Which questions cover security and incidents?

14. How do you defend against prompt injection and leakage between users or customers? Look for permissions enforced outside the model, separate indexes and caches per customer, and adversarial tests you can see; the same principles sit behind our guardrails.
15. Which actions can the product take in our systems, with which permissions, and which need a person’s approval? Approval steps should be enforced by the product, not left to a prompt.
16. What do you log, who can read it, and can we get the records for our cases? You will need them for disputes and audits.
17. How and when will you tell us about incidents, including AI-specific ones such as harmful output or data shown to the wrong customer?

Which questions cover exit?

18. What can we export, and in which formats: data, configurations, prompts, evaluation results, logs?
19. What happens to our data when the contract ends, and how is deletion confirmed?
20. If a model is tuned for us, who owns it, and can we take it with us? Settle this before anything is tuned.
21. How long is the notice period, and what help do we get during a transition? Look for a transition period with support and exports included.

Some points belong in the contract, not the questionnaire. Take these to counsel: notice of model changes and the right to test first, data deletion and its confirmation, training use, sub-processor changes, incident notification, and liability, meaning who bears the cost when a harmful output or a data incident starts on the vendor’s side.

Make the trial a condition

Add these questions as an AI section to the questionnaire you already use, and make the trial on your own examples a condition before any contract. If you are running a tender, our AI RFP template shows how to get answers you can compare. If you are still deciding whether to buy at all, our note on build or buy covers that decision, and testing both options on the same evaluation set is part of our data and AI foundations work.

This note is general information, not legal advice.

Sources

1. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 National Institute of Standards and Technology (NIST) nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
2. ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system International Organization for Standardization (ISO) www.iso.org/standard/42001
3. OWASP Top 10 for Large Language Model Applications OWASP Gen AI Security Project genai.owasp.org/llm-top-10
4. Regulation (EU) 2016/679 (General Data Protection Regulation), official text EUR-Lex eur-lex.europa.eu/eli/reg/2016/679/oj
5. Personal Data Protection Law (Law No. 6698), English text Personal Data Protection Authority (KVKK) www.kvkk.gov.tr/Icerik/6649/Personal-Data-Protection-Law