AI governance · Free implementation template
AI Vendor Evaluation Scorecard
Compare AI platforms using workflow fit, integration depth, governance, observability, implementation effort, and measurable business value.
Introduction
What the AI Vendor Evaluation Scorecard is for.
Compare AI platforms using workflow fit, integration depth, governance, observability, implementation effort, and measurable business value.
A completed governance template is the operating contract for an autonomous system: what it may do, on whose authority, where it must stop, and who is accountable when it is wrong. This one covers fit, governance, operations, economics, and it is written for executives, it leaders, procurement. Each section is a decision that has to exist before the workflow can be built or automated — the prompts are there to force the decision, not to be filled in for their own sake.
A completed template is also a structured brief. ARIA can read it as the objective, the systems involved, and the constraints that apply, which is what turns the specification into something that runs rather than something that gets filed.
- Category
- AI governance
- Sections
- 4
- Format
- Markdown, no signup
- Best for
- Executives, IT leaders, Procurement
The problem
Specifications fail on the decisions nobody wrote down.
The difference between governed execution and an open-ended agent is invisible in a demo. Both produce a plausible result. The difference only appears when the system encounters a case nobody specified, and by then the question is whether it stopped or improvised.
Scope written as a prohibition list is the reliable failure. The action nobody thought to prohibit is the one that causes the incident, because an implicit permission is still a permission.
The third failure is accountability. A capability that no named person owns is a capability nobody fixes, and it degrades quietly until something forces attention.
You're likely here because
- Vendor comparison is happening on feature lists
- Demos look identical and the differences are unclear
- Data handling questions surface late in procurement
How it works
From a blank AI Vendor Evaluation Scorecard to a running workflow.
Each stage is a decision that has to be made once. Making them in this order is what keeps the later ones answerable — you cannot set an automation boundary before you know who owns the record.
Step 01
Enumerate permitted actions
Scope is an allow-list, not a deny-list. Anything not named is not permitted, which is what makes the boundary hold against cases nobody anticipated.
Step 02
Bound the data access
Name the systems and record types the capability may read, and the identity it acts under. Access follows the workflow, not the convenience of the implementer.
Step 03
Write evaluable stop conditions
Each stop condition must be expressible as a check the runtime can perform. "Stop if something looks wrong" is not a control.
Step 04
Place the approval gates
Legal, clinical, financial, employment, and safety decisions require explicit human authority regardless of how reliable the automated path becomes.
Step 05
Name the accountable owner
One person accountable for the capability’s outcomes, with a review cadence. This is the field that keeps the rest of the document current.
The template
Every section, with the prompts that force the decision.
Section 01
Fit
Section 02
Governance
Section 03
Operations
Section 04
Economics
After it is filled in
What the completed specification is actually consumed by.
Filling in the AI Vendor Evaluation Scorecard produces an artifact, and an artifact on its own changes nothing. What matters is what consumes it afterwards.
Handed to the runtime, a completed governance specification becomes the boundary the execution layer enforces: what may run unattended, what requires approval and from whom, where the system must stop, and what is written to the audit trail regardless. The reason it is worth writing down is that these limits are invisible in a demo and decisive in production.
The test of a good one is what happens on the case nobody specified. A system with a real boundary stops and escalates with its reasoning; a system without one improvises and produces a plausible result that nobody can trace afterwards. If the specification does not make that distinction concrete, it has recorded an intention rather than a control.
Implementation path
How to actually fill it in.
- 01
Start from the outcome the capability is responsible for, then work backwards to the minimum permissions that outcome requires.
- 02
Write the stop conditions as checks against observable state, and test each one deliberately before go-live.
- 03
Run the capability in recommend-only mode until its recommendations are boringly correct.
- 04
Log every action with its trigger, the context used, and the approval that permitted it.
- 05
Review the register on a fixed cadence — scope drifts as the capability proves useful, and that drift is where the risk accumulates.
Controls this specification sets
Controls that matter.
Control 01
Permitted actions are an explicit allow-list; anything unnamed is refused.
Control 02
Consequential decisions stay under human authority regardless of measured reliability.
Control 03
Every action is attributable to the trigger, the context, and the identity that permitted it.
Control 04
Capability scope changes require the same review as the original approval.
Worked examples
When teams reach for this one.
Comparing shortlisted vendors
Scoring against your own criteria rather than each vendor’s strengths is what makes the comparison meaningful.
Surfacing data-handling terms early
Data residency, retention, and training-use questions are cheap before contracting and expensive after.
Justifying the decision afterwards
A completed scorecard is the record of why, which matters when the decision is revisited.
Limitations
What this template does not do.
- Scores reflect your weightings; they are not an objective ranking.
- Vendor self-reported capabilities need verification against a trial with your own data.
- The AI vendor landscape changes fast enough that evaluations date within months.
- It does not assess vendor viability or roadmap risk.
FAQ
Questions about the AI Vendor Evaluation Scorecard.
Who should fill in the AI Vendor Evaluation Scorecard?
The people who own the outcome, together. This template is written for executives, it leaders, procurement, and the value comes from surfacing disagreement between them rather than from one person completing it alone.
How long does it take?
A focused session, usually one to two hours for a first pass. Sections that take much longer are normally pointing at a real disagreement, which is the template doing its job rather than failing.
Is it free, and is there a signup?
Free, with no gate. The Markdown download link at the top of this page returns the full template immediately.
What happens after it is filled in?
The completed template becomes the specification the build works from. You can hand it to ARIA to resolve into an objective and an execution path, hand it to an internal team, or use it to brief a vendor — the artifact is the same in each case.
Does this replace the tooling?
No. It specifies what the tooling has to do. AI governance tools differ in how much of this they can enforce, which is itself useful information when you are comparing them.
How often should it be revisited?
When the process it describes changes, and on a scheduled review otherwise. A specification that no longer matches what runs is worse than none, because people still trust it.
More ai governance templates
Other ai governance specifications.
AI Agent Design Canvas
Define an agent by outcome, tools, context, memory, permissions, stop conditions, and human escalation before implementation.
Open the template →
Human-in-the-Loop Decision Matrix
Choose where AI may act, where it should recommend, and where humans must retain final authority.
Open the template →
30-Day AI Pilot Plan
Run a bounded AI pilot with a measurable baseline, explicit scope, control points, evaluation criteria, and go/no-go decision.
Open the template →
Keep exploring
Related reading and next steps.
Start with ARIA
Hand the brief to ARIA.
A completed template is a structured brief. Tell ARIA the objective and it resolves the systems involved and the execution path — so the specification becomes a running workflow, not another document.
- ARIA acts only through the systems and permissions you connect.
- Connections use scoped credentials you can change or revoke.
- Actions are recorded, and consequential ones can require approval.
Start here
Fill it in, then hand it to ARIA.
A completed template is a structured brief. ARIA can resolve it into the objective, the systems involved, and the execution path — so the specification becomes a running workflow rather than another document.