Ninety days, one domain, one decision

The work

• Take the readiness assessment with the platform and analytics engineering leads, and keep the result link.
• Choose one domain using the readiness result and the use-case library. Prefer the domain with the best existing definitions over the one with the most enthusiasm.
• Collect 30 to 50 real questions from that domain, with the answers analysts actually gave.
• Record how long those questions take to answer today.
• Write the success measures: share of the question set answered correctly, time to answer, and which failures are unacceptable.

Exit criteria

• A named accountable owner and a standing group with the roles above.
• A question set with known answers, stored where the team can version it.
• Written success measures and a one-page scope that names what is out of scope.

Risks

• Scope grows to several domains before one works.
• Questions are written by the project team instead of collected from users, so the set is easier than real use.

The work

• Write the metrics and dimensions the question set needs as code in the semantic layer, each with an owner and a description.
• Settle the term collisions the questions expose, and give each meaning its own name.
• Confirm the authoritative copy of each dataset involved, and record it in the catalog.
• Put row and column rules in the catalog or policy layer, and design how the user’s identity reaches it through the agent.
• Add freshness and basic quality checks to the datasets in scope.

Exit criteria

• An analyst can answer every question in the set from the definitions alone.
• Access rules tested on the agent path with at least two users who have different permissions.
• Freshness checks running, with an owner for each dataset.

Risks

• Definition work takes longer than planned. It is the most common schedule slip, and it is the work that decides accuracy.
• Security review starts late and blocks the build phase.

The work

• If you are selecting a product, run the evaluation checklist and a trial on your own question set.
• Connect the agent through open interfaces (SQL, APIs, or an MCP server) to the semantic layer, running as the user.
• Run the full question set, log every generated query, and review each wrong answer.
• Fix failures in definitions and descriptions first. Treat prompt patches as a last resort and write each one down.
• Decide and test what the agent does when it is unsure or the data is not covered.

Exit criteria

• The question set passes the threshold agreed in week 2.
• Every answer shows the definitions used and the data as-of time.
• The audit trail names the person behind every query.
• A written list of known failure types, shared with pilot users.

Risks

• Accuracy is improved by prompt patches that do not generalise, instead of by better definitions.
• The question set is quietly trimmed to the questions that pass.

The work

• Give access to one team in the chosen domain, with a named channel for reporting wrong answers.
• Review corrections and refusals every week. Each correction becomes an item on the definition backlog.
• Track cost per question and time to answer alongside accuracy.
• Watch for workarounds: new extracts, copied tables, or questions routed around the agent.

Exit criteria

• Pilot users rely on answers for real decisions in the domain.
• Corrections per week are falling, and each one has been resolved or logged.
• Cost per question is known and fits the success measures.
• No access incidents, confirmed from the audit trail.

Risks

• Users accept plausible wrong numbers because the answers read well.
• Usage costs arrive higher than the trial suggested.
• Teams outside the pilot build their own copies to get access sooner.

The work

• Compare results with the success measures written in week 2.
• Decide to expand, extend the pilot, or stop, and write down why.
• If expanding, choose the second domain and start its definitions work. Adding users to a domain that already works is a permissions change; adding a domain is a definitions project.
• Move the question set into the change process so it runs whenever the model, prompts, or definitions change.
• Retake the readiness assessment and compare it with the week 1 result.

Exit criteria

• A written decision, shared with the standing group and sponsor.
• If expanding: the second domain, its owner, and its question set are named.
• An operating runbook: who maintains definitions, who reviews corrections, who owns cost.

Risks

• Expansion by user count instead of by domain, which spreads the pilot’s definition gaps across more people.
• Success declared on enthusiasm rather than the written measures.