Figuring out what to actually build

For a few years the interesting question was capability. Could a machine draft this, read that, answer the other? That question is largely settled. Settling it removed the constraint that used to do the choosing for you.

Now almost anything is technically possible. So the shortlist is no longer set by what the technology can reach. It gets set by whoever was in the room, what a vendor demonstrated last month, and which executive saw something impressive at a conference.

So the failure mode changed shape. Organisations are not failing because they picked something too hard. They are failing because they picked something nobody needed. They also measured it in a way that could never show anything. They only found out both facts after spending a year.

One finding in the MIT NANDA research deserves more attention than the headline it came wrapped in. Only 40% of the companies surveyed had bought an official large language model subscription. Yet workers at over 90% of those same companies reported regularly using personal AI tools for work. The report puts it plainly: almost every single person used one in some form for their job.

Sit with what that means. Inside most organisations there is a live, unmanaged, self-funded experiment running right now. Hundreds of people have worked out for themselves which parts of their job AI genuinely helps with. They chose the tasks themselves. They kept using it where it worked and quietly stopped where it did not. Nobody had to run a workshop.

That is the most honest signal about what to build that any organisation has. Almost nobody reads it. Meanwhile the formal process assembles a shortlist from a vendor slide. The two lists rarely resemble each other.

The first thing I want to know is what people are already doing without permission. I do not want to shut it down. I want to read it.

Every idea that reaches a shortlist sounds reasonable in the room. That is what makes the room a bad instrument. These four questions reliably separate candidates worth money from candidates worth a conversation. An idea that fails any single one should be killed rather than parked.

• Evidence. Is somebody already doing this work by hand, at volume, and can you name them? An idea with a named person doing it badly today is worth ten that begin "imagine if we could".
• Edge. Does doing this well depend on something only you have, like your data, your process or your customer relationships? If it does not, you are building what you will be able to buy as a feature within a year. Buying it will be cheaper.
• Measurement. Is there a number that already exists? Does somebody already look at it? Would it move if this worked? Inventing a new metric alongside the project is how a project becomes unfalsifiable.
• Ownership. Who runs this in six months, when it breaks and the enthusiasm has moved on? A candidate with no named owner is just an intention.

The same research asked executives to allocate a hypothetical hundred pounds across functions. Sales and marketing took around seventy per cent of it. The report is direct about why that is a problem. Back-office automation often yields better returns. The concentration reflects easier attribution, not actual value.

This is worth understanding as a mechanism rather than a mistake. It repeats everywhere and nobody involved is being stupid. A marketing use case produces a number that moves within a quarter. It also gives somebody a story to tell upward. A back-office use case produces time returned to people whose time was never costed. It sits in a process nobody outside the team can see.

One of the executives quoted in the research says it almost exactly. How do you sell an idea to a chief executive when it will not directly move revenue or reduce a measurable cost? That is no failure of nerve. It is what happens when the appraisal system only recognises one kind of evidence.

This is why the measurement question sits inside the filters rather than after them. A genuinely valuable candidate may have no number attached. Then the work is to find the number before the funding conversation. It is not to abandon the candidate.

The same study traced what happens to enterprise-grade AI tools once organisations start looking. Sixty per cent evaluated something. Twenty per cent got as far as a pilot. Five per cent reached production.

Read that funnel carefully, because it says something counterintuitive. The idea that gets rejected is not the expensive failure. The expensive one gets piloted, absorbs a year of attention from your best people, and then quietly goes nowhere. Pilots are cheap to start and enormously expensive to finish. Almost nothing in a normal organisation is designed to stop one.

So the deliverable that matters from this stage is not a list of possibilities. Anyone can produce those. Increasingly anyone can produce forty of them in an afternoon. The deliverable is a shortlist with the kills attached. The reasoning is written down where people can argue with it.

Writing the kills down matters more than it sounds. An idea rejected without a recorded reason comes back in nine months with a new sponsor. The organisation pays to have the same argument twice.

You will be told that 95% of AI pilots fail. It usually arrives without a source, just before a recommendation to buy something. It comes from the MIT NANDA report. It is worth knowing what it actually says, because the version in circulation is not quite it.

The finding is that 95% of organisations were getting zero return, defined as no measurable profit and loss impact. 5% were extracting millions. That is a claim about measured financial return. It is a different and narrower claim than "failed". The report also attributes the divide to approach rather than to model quality or regulation.

The caveats are in the document and rarely survive the retelling. It describes itself as preliminary findings. It is not peer reviewed. It rests on 52 organisations interviewed and 153 senior leaders surveyed across four industry conferences. That is a self-selected sample. Its own research note says the figures are directionally accurate based on individual interviews, not official company reporting. It also says sample sizes vary by category. The definition of success differs between organisations.

There is also a plain internal inconsistency. The summary line for the investment section says half of budgets go to sales and marketing. The body of the same section says approximately seventy per cent. Both numbers are in the report, a few lines apart. I use the seventy because it is attached to the described method. I mention the discrepancy because anyone quoting either figure at you has probably not opened the document.

None of this makes the research worthless. The shadow usage finding and the pilot funnel are the most useful things published on this subject in a while. They are evidence to reason with, rather than a number to wave.

I start with the people doing the work, not the leadership team that commissioned the exercise. The people standing in the bottlenecks know them with great precision. That conversation costs nothing beyond the willingness to hear that a favoured idea solves nothing.

I ask what people already use unofficially. I ask it so nobody gets in trouble. The answer is worth more than punishing the breach.

Then each candidate goes through the four filters with the reasoning visible. You can disagree with a judgement rather than with a verdict. You end up able to run this yourself next time. That is the only version that survives my leaving.

And I will tell you when the answer is to buy something ordinary, or to fix a process, or to do nothing this year. That happens more often than the market admits. Saying it is most of what you are actually paying for.