Key takeaways
The short version
- Study repeated decisions and information bottlenecks before brainstorming model features.
- Prioritize with value, data, feasibility, adoption, risk, and operating cost—not novelty alone.
- Fund an evaluation path and workflow change, not just a prototype that produces plausible output.
01
Start with work, not a catalogue of AI features
Observe a real process from trigger to outcome. Look for people searching across documents, extracting information, classifying cases, drafting repetitive material, comparing options, or coordinating handoffs. Record frequency, time, delays, rework, variation, and the cost of mistakes. Ask which step constrains the result rather than which step sounds easiest to demonstrate.
Separate the user, the person accountable for the decision, and the buyer. A tool that saves one role a few minutes may create review work for another. Include exception handling and the current baseline. Existing automation, clearer policy, better search, or a form redesign may solve the constraint more reliably than a model.
02
Score the whole opportunity, including adoption and risk
Assess expected value, frequency, data readiness, technical feasibility, integration effort, evaluation clarity, user adoption, error impact, privacy, security, and ongoing cost. Use broad evidence ranges rather than invented precision. A high-volume task with modest per-case improvement may be more valuable than a rare executive workflow with a dramatic demo.
Apply hard gates before a weighted score. If the required data cannot be used lawfully, success cannot be checked, or a wrong output can cause unacceptable harm without review, the opportunity needs redesign. Ranking creates a portfolio conversation; it does not remove the need for judgment or accountable ownership.
- Value: what outcome changes, for whom, and how often?
- Viability: can the data, workflow, integrations, and operating model support it?
- Risk: what happens when output is wrong, missing, delayed, or exposed?
03
Inspect whether the data represents the job
Data existence is not data readiness. Sample the documents, records, labels, permissions, freshness, language, and edge cases needed for the task. Identify sources of truth and whether historical outcomes are trustworthy. Retrieval over inconsistent policy documents will reproduce inconsistency faster; a classifier trained on past decisions may preserve undesirable patterns.
Define access and retention before moving content into a model workflow. Keep tenant and role permissions intact, minimize sensitive fields, and establish provenance. Budget for content cleanup, integration, labeling, and evaluation set creation. These are often the core product work, not preparation surrounding the “real” AI feature.
04
Choose the right level of assistance
Match authority to consequence and evidence. Search, summarization, drafting, recommendation, and action each require different controls. An assistive interface may deliver value quickly while letting a domain expert handle ambiguity. Deterministic code should enforce rules, calculations, permissions, and transactions that do not benefit from probabilistic behavior.
Design the handoff between model and person. Show sources or relevant evidence, communicate uncertainty without fake precision, and make correction easy. Avoid forcing users to inspect long generated text when structured choices would be clearer. The product should reduce cognitive work, not move it into a new verification queue.
05
Prototype to answer a decision
Build the smallest end-to-end slice that can test quality on representative cases and fit into the workflow. A prompt playground can explore behavior, but it does not validate permissions, retrieval, latency, integration, review time, or recovery. Include the surrounding application early enough to measure the real user task.
Create an evaluation set before tuning toward memorable examples. Compare with the current baseline and a simpler alternative. Track task success, unsupported claims, critical errors, review effort, latency, and cost. Run blinded domain review when subjective quality matters. Document where the prototype fails and whether those failures can be controlled.
06
Make a product investment decision, not a demo decision
Estimate the full operating model: model and infrastructure cost, integration maintenance, evaluation, monitoring, support, policy updates, human review, and vendor change. Identify an owner for quality after launch. Consider reversibility and provider portability in proportion to business importance rather than promising an abstract form of independence.
Proceed when evidence supports useful quality, manageable risk, user adoption, and credible economics. Redesign when the valuable part needs narrower scope or lower authority. Stop when the baseline is already better or the organization cannot operate the controls. A disciplined “not now” preserves attention for opportunities where AI can produce durable value.
Worth building means worth operating
A valuable AI initiative connects a bounded workflow, representative data, measurable quality, responsible authority, and an adopted operating model. That standard filters out novelty without dismissing real opportunity—and gives promising ideas a much stronger route to production.