How to Validate an AI Product Idea Before You Build It

Most companies now feel pressure to add AI to their products. The hard part is knowing which ideas deserve the investment. If you want to validate an AI product idea, you need to test more than demand. You also need to test whether people will trust the output and whether the math works at scale.

This guide walks through the process we use. It works for a new AI product or an AI feature inside an existing one. Most of the steps happen before anyone trains or connects a model.

Why AI product ideas fail

Adoption is wide, but results are thin. In McKinsey's 2026 State of AI survey, nearly nine in ten respondents said their organizations use AI in at least one business function. Only 37 percent attributed any EBIT impact to AI, and about 6 percent qualified as high performers (McKinsey, 2026).

Other research points to the same gap. RAND researchers interviewed 65 data scientists and engineers and reported that, by some estimates, more than 80 percent of AI projects fail. That is about twice the rate of IT projects without AI. The most common root cause they found was leaders misunderstanding or miscommunicating the problem the AI should solve (RAND, 2024). In 2024, Gartner predicted that 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025. It named poor data quality, weak risk controls, rising costs, and unclear business value as the causes (Gartner, 2024).

In our work, these failures fall into three groups:

• No real need. The feature solves a problem people don't have, or solves it in a way they don't want.

• No trust. The output is wrong often enough, or unclear enough, that people stop using it.

• No margin. Each use costs more in compute, review, or support than the customer will pay.

Each of these can be tested with customers before a full build.

Step 1: Name the job and the bet

Start by writing down two things.

The job. What is the customer trying to get done, in their words? "Summarize claims notes" is a feature. "Decide which claims need a second look before end of day" is a job. Jobs keep the team focused on the outcome.

The bet. What has to be true for this idea to work? Write each assumption as a statement you can test. For example:

• Claims reviewers spend more than an hour a day sorting notes.

• They would act on an AI flag if they could see why it was raised.

• A wrong flag costs a few minutes. A missed flag costs much more.

• The team would pay for this as part of their current plan.

Rank the assumptions by risk. Test the riskiest one first. This mirrors the approach in our guide to validating a product idea, with extra attention to trust and cost.

Step 2: Test desirability with interviews

Talk to 5 to 8 people per customer group. Ask about their current work, their workarounds, and the last time the problem came up. Avoid pitching the AI idea early. People are polite about concepts and honest about their past behavior.

Listen for:

• How often the problem happens and what it costs them.

• What they use today, including spreadsheets, scripts, or a coworker.

• Whether they have tried AI tools for this, and why they kept or dropped them.

• Who would need to approve a new tool.

If only 1 or 2 of 8 people describe the problem as painful, the idea may be weaker than it looks. If 6 of 8 describe the same workaround, you likely have a real job to serve. For help with sample sizes, see how many user interviews you need.

Step 3: Test trust with a Wizard of Oz or concierge prototype

This step is where AI ideas differ most from other products. You can learn how people respond to AI output before you build the model.

Wizard of Oz prototype. The participant uses what looks like a working AI feature. Behind the scenes, a person writes or selects the response. This lets you test the interface, the tone, and the format of answers with real users.

Concierge prototype. You deliver the service by hand, openly. A team member does the work the AI would do, and the customer receives the result. This tests whether the outcome is valuable at all.

In both cases, watch what people do with the output:

• Do they read it, skim it, or ignore it?

• Do they check it against another source?

• Do they act on it, edit it, or redo the work themselves?

• What would make them trust it more: sources, confidence labels, or an easy way to correct it?

You can also plant a few wrong answers on purpose. This shows how people catch errors and how much one bad answer damages their trust. Our guide to testing AI assistants with real customers covers what to look for in these sessions.

Step 4: Test accuracy needs and error tolerance

Every AI feature makes mistakes. The question is how many mistakes users can live with, and which kinds.

Ask participants to walk through the cost of an error in their own work. A wrong product suggestion in a shopping app is a small annoyance. A wrong dosage note in a care plan is serious. The acceptable error rate depends on that cost and on how easily the user can spot and fix the mistake.

Useful questions:

• If this were wrong once in 10 uses, would you still use it? Once in 50?

• What happens if you miss an error?

• Which is worse here: a false alarm or a missed case?

Turn the answers into a target. For example, a hypothetical team might set the bar at "no more than 1 wrong flag in 20, and zero missed high-value claims in testing." That target becomes a requirement for the engineering team. It also tells you early if the bar is out of reach with current models and data.

Step 5: Test willingness to pay and cost per use

An AI feature can be loved and still lose money. Test both sides of the math.

Willingness to pay. Use a fake door test or a priced landing page to see how many people click to buy or join a waitlist. For business buyers, ask who holds the budget and what they pay today for the same job. Surveys of 30 to 100 or more people can show price ranges once interviews tell you what to ask.

Cost per use. Estimate what each use costs to run. Include model calls, data storage, human review, and support. Then compare it to what each use is worth to the customer.

Here is a simple hypothetical calculation. Say a feature costs $0.04 per request in model calls and $0.06 in review time. That is $0.10 per use. If an average user makes 300 requests a month, the cost is $30 per user per month (300 × $0.10). If customers will pay $25 a month, the feature loses $5 per user before any other cost. You can then decide whether to raise the price, limit usage, or change the design.

How to validate an AI product idea: risks and tests

Use this table to match each AI-specific risk with a test.

The last row matters. RAND and Gartner both list data quality among the main causes of failure. Bring engineering in early to check whether the data exists. Our guide to desirability, viability, and feasibility explains how to balance all three.

Step 6: Decide to build, change, or stop

Pull the evidence into a short readout. For each assumption, mark it as supported, mixed, or not supported. Then make a call.

• Build when need, trust, accuracy, and margin all hold. Write the requirements, including the accuracy target and cost limit.

• Change when one part fails but the job is real. You might narrow the use case, add a human review step, or change the price.

• Stop when the need is weak or the economics can't work. A stop decision made after four weeks of research costs far less than one made after six months of building.

Write the decision down with the evidence behind it. A clear research readout helps leaders agree, and a solid requirements document helps the build team start fast.

Frequently asked questions

How long does it take to validate an AI product idea?

Most teams can run the first round in 3 to 6 weeks. That covers interviews, a Wizard of Oz or concierge prototype, and a first pass at pricing and cost per use.

Do I need a working model to test an AI feature?

No. A Wizard of Oz prototype uses a person behind the scenes to produce the output. This lets you test trust and usefulness before you invest in a model.

What is an acceptable error rate for an AI feature?

It depends on what an error costs the user and how easily they can catch it. Ask users directly, test with planted errors, and set a target before you build.

How do I estimate the cost per use of an AI feature?

Add up model calls, data storage, human review, and support for one use. Multiply by expected monthly usage per customer and compare the total to what that customer will pay.

What if the idea fails validation?

Look at which test failed. If the job is real but trust or cost failed, change the design. If the need itself is weak, stop and move the budget to a stronger idea.

Get help validating your AI product idea

Our Innovation Retainer runs this process on a quarterly cycle. We generate and test ideas with your customers, run Wizard of Oz and concierge prototypes, test pricing and cost, and deliver build, change, or stop readouts with requirements ready for your team or a vendor. See how it works or start a conversation.

Sources

• Dan Tinkoff, Lieven Van der Veken, Michael Chui, and Tara Balakrishnan, "The state of AI in 2026: On the road to ROI," McKinsey & Company, 2026

• James Ryseff, Brandon F. De Bruhl, and Sydne J. Newberry, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI," RAND Corporation, 2024

• Gartner, "Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025," Gartner press release, 2024

• Cost per use example is hypothetical: 300 requests × $0.10 per request = $30 per user per month