AI governance buyer's checklist: 12 questions to ask every vendor

9 min read · Updated May 16, 2026

Every AI governance vendor sounds the same on the homepage: “Redact sensitive data before it reaches ChatGPT.” The differences only surface during a real evaluation - and by then you've usually already burned three weeks on a proof of concept that won't answer the questions that actually matter. This is the checklist we wish every buyer had open in front of them on the first vendor call.

We've grouped the twelve questions into five categories: coverage, detection, operations, compliance, and commercial. Skip any that aren't relevant to your shop. The point is to make every vendor answer the same twelve questions so you can compare apples-to-apples.

Coverage: what surfaces does it actually cover?

1. Which AI tools are covered today - by SKU, not by roadmap?

ChatGPT, Claude, Gemini, and Copilot are table stakes. Ask about the tools your team actually uses: Cursor, Perplexity, Grok, the Anthropic API, internal RAG apps, and the long tail of unmanaged AI accounts. “On the roadmap” is not coverage. Get a written list of supported tools and the specific surfaces - browser, desktop app, mobile, API.

2. Does it cover unauthenticated and temporary chats?

ChatGPT supports temporary chats and a signed-out flow that doesn't require sign-in. Tools that only inspect network traffic often see only signed-in traffic and miss these entirely. Same problem with personal Google accounts using Gemini. Ask specifically: “If a person opens ChatGPT in a non-corporate browser profile and pastes a customer email, is that covered?”

3. Browser extension, device agent, or network proxy?

Each has tradeoffs. A browser extension is the lightest-touch deploy but misses native desktop apps and IDE assistants. A network proxy catches traffic on the corporate network but can't see a personal hotspot or a certificate-pinned client. A device-level agent catches far more of it, at the cost of a slightly heavier deploy. Pick deliberately; don't let a vendor pick for you with a hand-wave.

Detection: what does it actually catch?

4. What detection layers does it run?

Three are common: regex / deterministic patterns (good for credit cards, API keys, IFSC codes - cheap, fast, high-precision), named-entity recognition (good for names, addresses, organisations - medium cost), and contextual model-based detection (good for “is this paragraph an internal memo” - higher cost, higher recall). Ask which stages run, in what order, and what happens when one stage flags but the next doesn't.

5. Where does redaction physically happen?

The agent should intercept on the device; where it redacts is a separate question. Ask: “Does the original, un-redacted prompt ever leave the person's device unredacted, and where does it get redacted?” A defensible answer names the exact service or deployment - the vendor's own service, or the customer's own deployment - and states plainly that the original is discarded, not retained.

6. Does it block, redact, or only alert?

Hard blocking trains people to route around the tool (personal phone, personal account, screenshot). Alert-only is invisible to the person and lets the sensitive data through anyway. Redaction - replacing “Ravi Mehta” with [PERSON_1] - lets the work continue without exposing the value. Most teams want redaction as the default with alerts layered on top for admins, not a global block switch.

Operations: can your team actually run it?

7. What does the audit log look like during a review?

Ask for a real screenshot or export. The minimum useful set: timestamp, user identity, tool, surface (browser / desktop / IDE / CLI), categories triggered, and a redacted preview of the content. If the log only shows counts, you can't investigate.

8. What's the added latency on a typical prompt?

Anything over ~250 ms is noticeable; over ~500 ms and people will route around it. Ask for a real number, not a range. If the answer is “depends on the model tier,” ask for the p50 and p95 on the cheapest tier you'd realistically use.

9. How does it deploy and update at scale?

MDM channels (Intune, JAMF, Kandji, Group Policy) for the initial install. Auto-update with a rollback path for new versions. Confirm the agent self-repairs a broken interception state without an IT ticket - this is the single biggest source of “the AI tool stopped working” complaints.

Compliance: does it pass the audit?

10. Which regulations does it map to, and how?

The honest list depends on jurisdiction: DPDP Act (India), GDPR (EU), HIPAA (US healthcare), PCI DSS (cardholder data), ISO 27001 (general). A good vendor can show you which categories in their detection map to which clauses or controls - not just claim “compliant” on its own, which is meaningless without the specific controls in scope and the organisational measures that sit alongside them.

11. Can data stay in-region or on-prem if you're regulated?

For Indian DPDP, healthcare, or financial-services buyers this is usually non-negotiable. Ask about: data residency for the vendor's cloud control plane, whether redaction itself ever sends content outside the customer's own deployment (Q5 again), and whether a self-hosted or in-VPC deployment is supported.

Commercial: can you actually buy it?

12. Published per-seat pricing - or "contact sales"?

Both models exist. Published per-seat means you can budget without three vendor calls; “contact sales” usually means custom enterprise contracts and a longer procurement cycle. Neither is wrong, but match it to your buying cycle. If you're a 50-person team and the vendor only sells to 500+ enterprises, you'll get deprioritised in support.

Bonus: things that aren't on this list (on purpose)

Detection accuracy percentages. Every vendor claims 95%+. Without a shared benchmark dataset, the number is meaningless. Run a proof of concept on your own data instead.
Logo walls. Big customers prove enterprise-readiness, not that the product fits you. Ask for a reference customer of similar size and stack.
Analyst placement. Useful signal, not a buying criterion. It lags the market considerably.

Sources & further reading

OWASP - Top 10 for LLM Applications
India Ministry of Electronics & IT - Digital Personal Data Protection Act, 2023