# Agentic analytics evaluation checklist

Vendor-neutral questions for choosing an agentic analytics product or platform, grouped by area, with what a good answer includes. Use it as RFP questions or as the agenda for a proof of concept. Score each answer 0 (no answer or a bad one) to 3 (meets the good answer in full).

Source: https://agenticanalyticsnow.com/evaluation-checklist/ (Alex Merced, Agentic Analytics Now). Run the readiness assessment first: https://agenticanalyticsnow.com/readiness/

## Semantic layer

Whether the product reads your definitions or asks you to write a second set just for it.

### 1. Where do metric and dimension definitions live, and in what format? Can we export them?

- [ ] Asked
- Vendor answer:
- A good answer includes: Definitions stored as code or in a documented format that other tools can read, exportable without a services engagement.
- Score (0 to 3):

### 2. Can dashboards, notebooks, and the agent all read the same definitions?

- [ ] Asked
- Vendor answer:
- A good answer includes: One definition served to every consumer. No separate agent-only copy to keep in sync.
- Score (0 to 3):

### 3. How does the agent choose between two definitions with similar names?

- [ ] Asked
- Vendor answer:
- A good answer includes: It uses the names and descriptions in the semantic layer, asks a clarifying question when a request is ambiguous, and shows which definition it used.
- Score (0 to 3):

### 4. How are changes to definitions versioned, reviewed, and rolled back?

- [ ] Asked
- Vendor answer:
- A good answer includes: Version history, a review step, and a way to see which version of a definition produced a given answer.
- Score (0 to 3):

### 5. What happens when a question needs data the semantic layer does not cover?

- [ ] Asked
- Vendor answer:
- A good answer includes: Behaviour you configure: refuse, answer with a clear "ungoverned" label, or route to an analyst.
- Score (0 to 3):

## Governance and security

Whether the agent can reach anything its user cannot, and whether you can prove it afterwards.

### 6. Does the agent act as the requesting user or as a service account?

- [ ] Asked
- Vendor answer:
- A good answer includes: The user's identity is passed through, and permissions are checked on every request.
- Score (0 to 3):

### 7. Where are row and column rules enforced, and does the agent path skip any of them?

- [ ] Asked
- Vendor answer:
- A good answer includes: Enforced in the data platform or catalog, the same way for the agent, BI tools, and SQL clients.
- Score (0 to 3):

### 8. What does the audit log record for each agent answer?

- [ ] Asked
- Vendor answer:
- A good answer includes: The person, the agent, the question, the queries it generated, the datasets touched, and the time. Exportable to your security tooling.
- Score (0 to 3):

### 9. What can the agent do besides read, and how is each action approved?

- [ ] Asked
- Vendor answer:
- A good answer includes: Read-only by default. Writes or outside actions need explicit configuration and a person to approve them.
- Score (0 to 3):

### 10. Where are questions, answers, and logs stored, for how long, and are they used to train models?

- [ ] Asked
- Vendor answer:
- A good answer includes: A documented location and retention period you control, and no training on your data unless you agree to it in writing.
- Score (0 to 3):

### 11. How are credentials to data sources handled?

- [ ] Asked
- Vendor answer:
- A good answer includes: Short-lived, narrowly scoped credentials. No long-lived keys stored in agent configuration.
- Score (0 to 3):

## Openness and lock-in

What it would cost to replace this product, or any one part of it, later.

### 12. Which table formats and catalogs can it read in place, without copying data?

- [ ] Asked
- Vendor answer:
- A good answer includes: Open table formats such as Apache Iceberg read where they sit, through standard catalog protocols such as the Iceberg REST catalog.
- Score (0 to 3):

### 13. Which models can we use? Can we bring our own or run one ourselves?

- [ ] Asked
- Vendor answer:
- A good answer includes: Several model providers, including self-hosted models, switchable by configuration.
- Score (0 to 3):

### 14. Does it support open agent interfaces such as MCP, in either direction?

- [ ] Asked
- Vendor answer:
- A good answer includes: Documented support, with the same permissions applied as every other path.
- Score (0 to 3):

### 15. If we leave, what do we take with us?

- [ ] Asked
- Vendor answer:
- A good answer includes: Definitions, evaluation question sets, logs, and configuration in documented formats, with a stated export process.
- Score (0 to 3):

### 16. Which components are proprietary, and what would replacing each one involve?

- [ ] Asked
- Vendor answer:
- A good answer includes: The vendor names them plainly and describes the migration path for each.
- Score (0 to 3):

## Accuracy and evaluation

Whether you can measure correctness on your own questions, before and after you buy.

### 17. Can we run our own set of questions with known answers, on our data, before we buy?

- [ ] Asked
- Vendor answer:
- A good answer includes: Yes, in a trial or proof of concept, with every result open to inspection.
- Score (0 to 3):

### 18. How does the agent show its work?

- [ ] Asked
- Vendor answer:
- A good answer includes: The generated SQL or query plan, the definitions used, the data as-of time, and row counts, visible to the person asking.
- Score (0 to 3):

### 19. What does it do when it is unsure or the data is missing?

- [ ] Asked
- Vendor answer:
- A good answer includes: It says so, asks a clarifying question, or declines. The threshold is configurable.
- Score (0 to 3):

### 20. How are regressions caught when the model, prompts, or definitions change?

- [ ] Asked
- Vendor answer:
- A good answer includes: Evaluation runs on every change, built in or connected to your own pipeline, with a history of results.
- Score (0 to 3):

### 21. Were any published accuracy figures measured on data like ours?

- [ ] Asked
- Vendor answer:
- A good answer includes: The benchmark and method are named, and the vendor will measure on your data instead.
- Score (0 to 3):

## Cost and latency

What drives the bill, and how long a real multi-step question takes.

### 22. What is the pricing unit (seats, questions, tokens, compute), and what makes it go up?

- [ ] Asked
- Vendor answer:
- A good answer includes: A clear unit and a worked estimate for your expected usage, including model and query compute.
- Score (0 to 3):

### 23. Where does query compute run, and who pays for it?

- [ ] Asked
- Vendor answer:
- A good answer includes: Stated plainly. It can use engines you already run, and it does not require a second copy of the data.
- Score (0 to 3):

### 24. How long does a typical multi-step question take, and how long does a slow one take?

- [ ] Asked
- Vendor answer:
- A good answer includes: Median and slow-case times measured on your data during the trial.
- Score (0 to 3):

### 25. What caching or acceleration applies, and how does it interact with permissions and freshness?

- [ ] Asked
- Vendor answer:
- A good answer includes: Caches respect permissions, and answers show how fresh the underlying data is.
- Score (0 to 3):

### 26. What limits exist on runaway usage?

- [ ] Asked
- Vendor answer:
- A good answer includes: Budgets per user or team, query limits, and alerts before costs pass a threshold.
- Score (0 to 3):

## Operations

Who runs it, how you watch it, and what adding the next domain takes.

### 27. Who operates it: you, us, or both? What are the service levels?

- [ ] Asked
- Vendor answer:
- A good answer includes: A written split of responsibilities and a service level agreement.
- Score (0 to 3):

### 28. How do we monitor usage, failures, refusals, and corrections?

- [ ] Asked
- Vendor answer:
- A good answer includes: Logs and reports for each, exportable to your own monitoring.
- Score (0 to 3):

### 29. Where can it be deployed, and where does data travel?

- [ ] Asked
- Vendor answer:
- A good answer includes: Deployment options by cloud and region, and a diagram of every data flow, including model calls.
- Score (0 to 3):

### 30. How are model and product updates rolled out? Can we pin a version?

- [ ] Asked
- Vendor answer:
- A good answer includes: Release notes, a staging option, and the ability to delay an update until your evaluation passes.
- Score (0 to 3):

### 31. What does adding a new business domain involve, and who does the work?

- [ ] Asked
- Vendor answer:
- A good answer includes: Concrete steps and effort estimates that your own team can follow without mandatory services.
- Score (0 to 3):

### 32. What security attestations do you hold, and how are incidents reported to customers?

- [ ] Asked
- Vendor answer:
- A good answer includes: Current independent reports available on request, and a defined notification process and timeline.
- Score (0 to 3):

## Before you send it

- Run your own question set with known answers on your own data during the trial. Published accuracy figures on other data tell you little.
- Ask every vendor the same questions in the same order, and score the answers before the demos, not after.
- Weight the areas to your situation. For most teams, governance and openness decide more of the long-term cost than the model does.