TL;DR
Evaluate AI vendors with a pilot on your data, clear success metrics, written answers on training/retention, pricing modeled at 10x usage, and an exit plan. Buy commodity AI; build where your workflow is the product. Magical demos are not diligence.
Introduction
AI sales motions optimize for wow. Your job is boredom: fit, data, cost at success, and divorce. This checklist works when the "procurement team" is one founder with a credit card.
Three Purchase Types
- Internal productivity — ChatGPT, Cursor. Lower integration risk; still set data controls.
- Embedded product AI — model APIs inside your app. Care about latency, unit economics, evals.
- Workflow SaaS with AI — support/CRM/marketing tools. Lock-in lives in your operational data.
Diligence depth should track customer-data exposure and switching cost.
12-Point Scorecard (1–5 Each)
- Task fit on your samples
- Time-to-value for a tiny team
- Training-policy clarity
- Retention and deletion
- Access controls
- Export completeness
- Pricing clarity at 1x and 10x
- Failure-mode honesty
- Visible evals/analytics
- Support during pilot
- Roadmap credibility
- Replaceability
Under 36/60: walk. 36–48: careful pilot. 48+: negotiate.
Pilot Design
Define owner, 20–50 real examples, baseline metric, target, max spend, end date. Run blind comparisons when you can. Log failure types: wrong, incomplete, unsafe, slow. Decide adopt / extend / kill — no endless "another month."
Data Questions in Writing
Is content used for training? Opt-out default? Region of storage? Retention of prompts/files/embeddings? Staff access? Subprocessors? Breach process? Deletion confirmation?
Foggy answers = high risk.
Pricing Reality
Model seats, usage, overage, "production" security add-ons, human review time, and engineering integration. Ask for 10x volume pricing. Cheap pilots with expensive production tiers are a classic trap.
Build vs Buy
Buy commodity (transcription, generic summarization) when speed beats differentiation.
Build when the AI workflow is the product or proprietary data creates a loop you must own.
Wrap models but own prompts, evals, UX, and data plumbing — or quality decays silently.
Exit Plan Before Marriage
What must export? How long to reimplement? Will customers notice a provider swap? Prefer open-ish formats and API export. Fear proprietary graphs you cannot extract.
Security Lite
Least-privilege keys, staging vs prod separation, no production dumps in shared demos, rotate on contractor offboarding. Read SOC2 claims for scope, not logos.
Negotiation Without Much Leverage
Shorter term, renewal notice, price lock, training opt-out on the order form, pilot credits, written success criteria.
Red Flags
ROI guarantees, sample-only quality, deckware docs, call-time pressure, logo-slide references, data policy bait-and-switch after upload.
Composite Example
Two support AI tools, 3-week pilot, 80 tickets. Flashy tool invents refunds. Quieter tool cites KB and escalates. Team picks quiet tool, keeps human send on billing, revisits autonomy later. Demo beauty ≠ production fit.
Decision Record Template
Problem, scorecard, pilot deltas, data answers, 10x cost, exit plan, decision, review date — one Notion page per vendor.
Operating Cadence After Purchase
30 days in: re-check metrics vs pilot. 90 days: renegotiate or rip out. Annually: re-read data policy diffs. Assign an owner; orphaned AI tools accumulate risk.
Field Notes From Operators
The difference between AI tourism and AI leverage is usually process. Operators who win write the decision criteria down, keep prompts in a shared library, and schedule a weekly review of what the tools got wrong. They treat model output like an intern draft: useful, sometimes impressive, never automatically trusted with customers, cash, or compliance.
When something works, they turn it into a checklist. When something fails, they add a ban rule. Over a quarter, that creates a private playbook no generic Twitter thread can replace. Your advantage is not access to the same models everyone else has — it is the taste and constraints you encode around them.
Use the rest of this guide as a living document. Revisit after each quarter's tooling changes. Delete advice that no longer matches how you work. Add examples from your own tickets, deploys, and sales calls so the next hire inherits judgment, not just subscriptions.
If you only remember one practice, make it this: every AI-assisted external action needs a named human owner. Ownership is what converts speed into responsibility. Without it, you get probabilistic chaos with a friendly chatbot UI.
Practical Exercises
Exercise A — Constraint writing. Take a vague prompt you used last week. Rewrite it with stakes, options, criteria, and a required disconfirming paragraph. Compare outputs. Keep the better template in your library.
Exercise B — Data minimization. Redo a support or research prompt with redaction. Note whether quality dropped. If not, make redaction the default.
Exercise C — Cost model. Pick one AI SaaS tool. Estimate monthly cost at 10x usage. Decide in writing whether that curve is acceptable.
Exercise D — Stop list. Add three company-specific "do not use AI for X" rules based on your product risk. Share them with anyone who has a seat.
Exercise E — Postmortem. Find one bad AI output that reached a customer or commit. Document cause, blast radius, and the process change that prevents repeats.
These exercises take a few hours total and prevent months of quiet drift.
Worked Walkthrough
Imagine a solo founder considering an AI support add-on at $400/month. They run a 21-day pilot on 60 tickets, measure edit distance and CSAT, discover the bot mishandles plan-change requests, and keep draft-only mode while rewriting three KB articles. They revisit autonomy next quarter. That sequence — hypothesis, timebox, metric, decision — is the entire craft. Apply it to vendors, content systems, coding agents, and prompting for strategy. The tools change; the discipline does not.
Another walkthrough: a founder wants "thought leadership" via AI. They brief one distinctive thesis from a real customer failure, draft with AI, rewrite the opening by hand, add a checklist artifact, publish, and pitch a single niche newsletter. One earned mention follows. That beats twenty interchangeable posts with zero distribution. Systems create that behavior on purpose.
Extended Playbook
Operators who get durable value from AI treat it like infrastructure with runbooks. They keep a shared prompt library, a stop list, and a weekly 30-minute review of failures. They measure outcomes (tickets resolved well, features shipped safely, pages that earn citations) instead of chat volume. They assign a human owner to every customer-facing automation. They budget for business-tier privacy settings the moment real customer data appears. They refuse multi-year vendor contracts before a pilot proves fit on their own messy inputs.
They also write. Short specs before coding agents. Decision memos before strategy chats. Briefs before content drafts. The writing is the control surface; the model is the accelerator. Without writing, you get confident entropy.
Another durable habit: version your rules. When a bad output ships, add a ban. When a good pattern repeats, add a template. After a quarter you have a private operating system that competitors cannot copy by buying the same subscriptions. That is the real stack advantage — taste encoded as process.
Finally, schedule deletion. Old chats, unused seats, stale connectors, and zombie pilots accumulate risk and cost. A monthly cleanup is as important as a new tool trial. Speed without subtraction is how startups drown in dashboards.
Scenarios and Responses
Scenario: investor asks for AI strategy. Answer with jobs-to-be-done, current tools, data boundaries, and what you will not automate. Do not perform science fiction.
Scenario: customer asks if a human will help. Promise a human path and keep it. Autonomy theater destroys NPS.
Scenario: contractor pastes a client list into a consumer model. Revoke access, delete where possible, review legal duties, update acceptable use training immediately.
Scenario: coding agent "fixes" auth broadly. Revert, re-spec, implement in a thin slice, add a regression checklist.
Scenario: content calendar demands five AI posts/week. Cut to one deep piece with distribution. Volume without voice trains readers to ignore you.
Scenario: vendor demo looks perfect. Demand a pilot on your samples with kill criteria and 10x pricing before you celebrate.
Checklists You Can Run Tomorrow
- Approved AI tools list exists and is shared
- Training opt-out enabled on business accounts
- One-page acceptable use for data
- Prompt library with at least five templates
- Stop list with at least five bans
- Pilot template for any new AI vendor
- Content brief template with voice kit link
- Support escalation matrix if AI drafts exist
- Weekly failure review on the calendar
- Monthly seat and connector cleanup
If fewer than six boxes are checked, fix basics before buying another tool.
Extended Playbook
Operators who get durable value from AI treat it like infrastructure with runbooks. They keep a shared prompt library, a stop list, and a weekly 30-minute review of failures. They measure outcomes (tickets resolved well, features shipped safely, pages that earn citations) instead of chat volume. They assign a human owner to every customer-facing automation. They budget for business-tier privacy settings the moment real customer data appears. They refuse multi-year vendor contracts before a pilot proves fit on their own messy inputs.
They also write. Short specs before coding agents. Decision memos before strategy chats. Briefs before content drafts. The writing is the control surface; the model is the accelerator. Without writing, you get confident entropy.
Another durable habit: version your rules. When a bad output ships, add a ban. When a good pattern repeats, add a template. After a quarter you have a private operating system that competitors cannot copy by buying the same subscriptions. That is the real stack advantage — taste encoded as process.
Finally, schedule deletion. Old chats, unused seats, stale connectors, and zombie pilots accumulate risk and cost. A monthly cleanup is as important as a new tool trial. Speed without subtraction is how startups drown in dashboards.
Scenarios and Responses
Scenario: investor asks for AI strategy. Answer with jobs-to-be-done, current tools, data boundaries, and what you will not automate. Do not perform science fiction.
Scenario: customer asks if a human will help. Promise a human path and keep it. Autonomy theater destroys NPS.
Scenario: contractor pastes a client list into a consumer model. Revoke access, delete where possible, review legal duties, update acceptable use training immediately.
Scenario: coding agent "fixes" auth broadly. Revert, re-spec, implement in a thin slice, add a regression checklist.
Scenario: content calendar demands five AI posts/week. Cut to one deep piece with distribution. Volume without voice trains readers to ignore you.
Scenario: vendor demo looks perfect. Demand a pilot on your samples with kill criteria and 10x pricing before you celebrate.
Key Takeaways
- Pilot on your data with kill criteria.
- Get training/retention in writing; model 10x cost.
- Buy commodity; build moats; own evals in-product.
- Plan export/exit first.
- Reject hype, fog, and forced long contracts.
FAQ
How do I evaluate an AI vendor?
Scorecard + real-sample pilot + data/pricing/exit review.
Build or buy?
Buy commodity; build differentiators; wrap models but own quality loops.
What contract terms matter?
Data use, retention/deletion, export, pricing uplift, subprocessors.
How long should a pilot run?
2–6 weeks with metrics and an end date.
Common red flags?
Vague training answers, demo-only quality, no export, multi-year pressure.
Security questionnaire?
Yes when customer data is in play.