Evaluation checklist
Questions to ask before you buy
32 questions for any agentic analytics product or platform, grouped into six areas. Each comes with what a good answer includes, so the people scoring vendor responses agree on what they are looking for. It names no vendors and favours none.
Use it as RFP questions or as the agenda for a proof of concept. Score each answer 0 to 3 against the good answer, before the demos rather than after.
Semantic layer
Whether the product reads your definitions or asks you to write a second set just for it.
Where do metric and dimension definitions live, and in what format? Can we export them?
A good answer: Definitions stored as code or in a documented format that other tools can read, exportable without a services engagement.
Answer: ____________________________ Score (0 to 3): ___
Can dashboards, notebooks, and the agent all read the same definitions?
A good answer: One definition served to every consumer. No separate agent-only copy to keep in sync.
Answer: ____________________________ Score (0 to 3): ___
How does the agent choose between two definitions with similar names?
A good answer: It uses the names and descriptions in the semantic layer, asks a clarifying question when a request is ambiguous, and shows which definition it used.
Answer: ____________________________ Score (0 to 3): ___
How are changes to definitions versioned, reviewed, and rolled back?
A good answer: Version history, a review step, and a way to see which version of a definition produced a given answer.
Answer: ____________________________ Score (0 to 3): ___
What happens when a question needs data the semantic layer does not cover?
A good answer: Behaviour you configure: refuse, answer with a clear "ungoverned" label, or route to an analyst.
Answer: ____________________________ Score (0 to 3): ___
Governance and security
Whether the agent can reach anything its user cannot, and whether you can prove it afterwards.
Does the agent act as the requesting user or as a service account?
A good answer: The user's identity is passed through, and permissions are checked on every request.
Answer: ____________________________ Score (0 to 3): ___
Where are row and column rules enforced, and does the agent path skip any of them?
A good answer: Enforced in the data platform or catalog, the same way for the agent, BI tools, and SQL clients.
Answer: ____________________________ Score (0 to 3): ___
What does the audit log record for each agent answer?
A good answer: The person, the agent, the question, the queries it generated, the datasets touched, and the time. Exportable to your security tooling.
Answer: ____________________________ Score (0 to 3): ___
What can the agent do besides read, and how is each action approved?
A good answer: Read-only by default. Writes or outside actions need explicit configuration and a person to approve them.
Answer: ____________________________ Score (0 to 3): ___
Where are questions, answers, and logs stored, for how long, and are they used to train models?
A good answer: A documented location and retention period you control, and no training on your data unless you agree to it in writing.
Answer: ____________________________ Score (0 to 3): ___
How are credentials to data sources handled?
A good answer: Short-lived, narrowly scoped credentials. No long-lived keys stored in agent configuration.
Answer: ____________________________ Score (0 to 3): ___
Openness and lock-in
What it would cost to replace this product, or any one part of it, later.
Which table formats and catalogs can it read in place, without copying data?
A good answer: Open table formats such as Apache Iceberg read where they sit, through standard catalog protocols such as the Iceberg REST catalog.
Answer: ____________________________ Score (0 to 3): ___
Which models can we use? Can we bring our own or run one ourselves?
A good answer: Several model providers, including self-hosted models, switchable by configuration.
Answer: ____________________________ Score (0 to 3): ___
Does it support open agent interfaces such as MCP, in either direction?
A good answer: Documented support, with the same permissions applied as every other path.
Answer: ____________________________ Score (0 to 3): ___
If we leave, what do we take with us?
A good answer: Definitions, evaluation question sets, logs, and configuration in documented formats, with a stated export process.
Answer: ____________________________ Score (0 to 3): ___
Which components are proprietary, and what would replacing each one involve?
A good answer: The vendor names them plainly and describes the migration path for each.
Answer: ____________________________ Score (0 to 3): ___
Accuracy and evaluation
Whether you can measure correctness on your own questions, before and after you buy.
Can we run our own set of questions with known answers, on our data, before we buy?
A good answer: Yes, in a trial or proof of concept, with every result open to inspection.
Answer: ____________________________ Score (0 to 3): ___
How does the agent show its work?
A good answer: The generated SQL or query plan, the definitions used, the data as-of time, and row counts, visible to the person asking.
Answer: ____________________________ Score (0 to 3): ___
What does it do when it is unsure or the data is missing?
A good answer: It says so, asks a clarifying question, or declines. The threshold is configurable.
Answer: ____________________________ Score (0 to 3): ___
How are regressions caught when the model, prompts, or definitions change?
A good answer: Evaluation runs on every change, built in or connected to your own pipeline, with a history of results.
Answer: ____________________________ Score (0 to 3): ___
Were any published accuracy figures measured on data like ours?
A good answer: The benchmark and method are named, and the vendor will measure on your data instead.
Answer: ____________________________ Score (0 to 3): ___
Cost and latency
What drives the bill, and how long a real multi-step question takes.
What is the pricing unit (seats, questions, tokens, compute), and what makes it go up?
A good answer: A clear unit and a worked estimate for your expected usage, including model and query compute.
Answer: ____________________________ Score (0 to 3): ___
Where does query compute run, and who pays for it?
A good answer: Stated plainly. It can use engines you already run, and it does not require a second copy of the data.
Answer: ____________________________ Score (0 to 3): ___
How long does a typical multi-step question take, and how long does a slow one take?
A good answer: Median and slow-case times measured on your data during the trial.
Answer: ____________________________ Score (0 to 3): ___
What caching or acceleration applies, and how does it interact with permissions and freshness?
A good answer: Caches respect permissions, and answers show how fresh the underlying data is.
Answer: ____________________________ Score (0 to 3): ___
What limits exist on runaway usage?
A good answer: Budgets per user or team, query limits, and alerts before costs pass a threshold.
Answer: ____________________________ Score (0 to 3): ___
Operations
Who runs it, how you watch it, and what adding the next domain takes.
Who operates it: you, us, or both? What are the service levels?
A good answer: A written split of responsibilities and a service level agreement.
Answer: ____________________________ Score (0 to 3): ___
How do we monitor usage, failures, refusals, and corrections?
A good answer: Logs and reports for each, exportable to your own monitoring.
Answer: ____________________________ Score (0 to 3): ___
Where can it be deployed, and where does data travel?
A good answer: Deployment options by cloud and region, and a diagram of every data flow, including model calls.
Answer: ____________________________ Score (0 to 3): ___
How are model and product updates rolled out? Can we pin a version?
A good answer: Release notes, a staging option, and the ability to delay an update until your evaluation passes.
Answer: ____________________________ Score (0 to 3): ___
What does adding a new business domain involve, and who does the work?
A good answer: Concrete steps and effort estimates that your own team can follow without mandatory services.
Answer: ____________________________ Score (0 to 3): ___
What security attestations do you hold, and how are incidents reported to customers?
A good answer: Current independent reports available on request, and a defined notification process and timeline.
Answer: ____________________________ Score (0 to 3): ___
Before you send it
- Collect your own question set with known answers first, and insist on running it on your data during the trial. The rollout playbook covers how to build one in the first two weeks.
- Ask every vendor the same questions in the same order, and agree the scoring before anyone sees a demo.
- Weight the areas to your situation. The lock-in exercise shows why governance and openness tend to decide more of the long-term cost than the model choice.
- If the readiness assessment puts you at Exploring, most of these answers will matter less than the definitions and access work on your own side.