Why a question list helps
Product demonstrations are designed to impress, so they show the best case. Your job in a vendor conversation is to learn about the ordinary case and the bad case: what happens to your information, what support looks like on a bad day, and how easy it is to leave. A prepared list keeps you from being swept along and lets you compare vendors on equal terms.
Send the questions in writing where possible, and keep the answers. Written replies are easier to compare and to refer back to. For anything involving sensitive data or legal commitments, have the contract and the answers reviewed by someone qualified, such as a lawyer or your security lead.
Data handling
This is usually the most important area.
- What data does the tool collect, and where is it stored?
- Is our input used to train or improve the vendor’s models? Can we opt out, and is the opt-out in the contract?
- How long is data retained, and can we delete it on request? The reasoning behind this is explained in what happens to the data you enter.
- Who at the vendor, or at its subcontractors, can see our content?
- What security practices and independent assessments can you describe or document?
- How do you handle a data breach, and how quickly would we be told?
- Does the tool support the access controls we need, such as separate accounts and permission levels?
If your use involves personal information, health details or other regulated data, the questions get stricter. Basic principles of data protection and protecting customer data apply, and the advice of a privacy professional is worth having.
Accuracy, limits and honesty
Reasonable vendors will talk about limitations. Be wary of any that do not.
- What tasks is the tool not designed for?
- How often does it produce wrong or invented output, and how do you suggest we check for it? Tools can generate fabricated answers, so ask what safeguards exist.
- What do you know about bias in the system, and how is it tested? See the background on algorithmic bias.
- Can you explain, in plain language, how the tool reaches its outputs, and what training data it relies on?
- Does it show sources or confidence levels, or does it just present an answer?
- Does it support human review steps?
A vendor who promises perfect accuracy is overpromising. Treat guarantees of savings or flawless results with caution.
Support and reliability
- What support is included, through which channels, and during what hours?
- What response times do you commit to, and are they in the contract?
- What is the uptime record, and what happens if the service goes down?
- How are changes and updates announced? Will features we rely on change without notice?
- Is training or onboarding provided, and is it free?
- Is there documentation that ordinary users can follow?
Pricing and contract terms
- What exactly is included in the price, and what costs extra?
- How is pricing structured: per user, by usage, or by tier? What triggers a higher charge?
- How long is the term, and how do renewals and price changes work?
- Can we start with a trial or a small number of users?
- What setup or integration fees apply?
Use the answers to build your estimate of total cost, which will usually be higher than the subscription alone.
Integration and ownership
- Which of our existing systems does it connect to, and how?
- Who owns the content we produce with the tool? There are open questions around content ownership, so ask for the vendor’s position in writing.
- Can we export our data, templates and settings in standard formats?
Exit terms
Ask early, while you still have leverage.
- How do we cancel, and what notice is needed?
- What happens to our data on cancellation, and how soon is it deleted?
- Are there early-termination penalties?
- What happens if you are acquired or shut down the product?
Clear answers here make a later decision about tool retirement much easier.
Accountability and responsibility
- If the tool causes harm or a mistake, what does your contract say about responsibility? Questions of accountability are better discussed before a problem than after.
- Do you disclose when AI is involved in your own service, and what do you require of us? See guidance on AI disclosure.
How to read the answers
Look for specific, documented, consistent answers. Warning signs include vague phrases like “industry-standard security” with nothing behind them, reluctance to put statements in writing, pressure to sign quickly, and reluctance to discuss limits. Note any question the vendor could not answer and decide whether it matters for your use. Missing answers are information too.
Finally, check the answers against reality in a small pilot project. A vendor’s claims and your actual results do not always match, and a limited trial is the fair way to find out.