Ask an AI to review your architecture proposal. It will find things to praise. Ask if your approach is sound. It will tell you "of course it is". Ask what could go wrong. It will list some things, helpfully, then conclude that your approach is solid overall.

This is not a criticism of the technology. The potential is high and the possibilities exciting, but there are consequences given how it was built.

AI systems are trained on human feedback: humans rate responses, and the model learns to produce responses that get high ratings. Raters tend to prefer confident, helpful, positive answers. So the model learns to give confident, helpful, positive answers. The result is overconfidence, erroneous minimisation of uncertainty, and the phenomenon known as "hallucinations". It is the direct and predictable result of optimising for human approval.

Understanding this changes how you use it.

The failure mode is specific: AI is excellent at many things, and particularly unreliable at one of them; telling you that your idea is wrong.

What AI Is Good At

The list is substantial:

  • Generating options quickly when you are stuck.
  • Explaining unfamiliar code.
  • Writing code for tests you have already designed / written.
  • Writing tests for logic you have already designed.
  • Spotting inconsistencies in a design you have described.
  • Rubber-ducking a problem at 11pm when nobody else is available.
  • Drafting prose.
  • Scanning code for edge cases you missed.
  • Translating intent into boilerplate.

These are real, substantial gains. A senior engineer who uses AI well is more productive than one who does not. The underlying capability is not overrated, even if the hype around specific applications often is.

Where It Falls Short

Knowing your context. Understanding organisational constraints you have not mentioned. Making judgment calls that require weighing incommensurable priorities.

And: telling you that the direction you have committed to is wrong.

This last one matters because of how most engineers use AI. You have a plan. You ask AI to help you execute it. AI helps you execute it. The plan goes forward. At no point did AI meaningfully interrogate whether the plan was right in any significant way.

The problem is not that AI cannot see the flaw. Often it can; the training process has simply made it reluctant to lead with that observation. Ask "how do I implement X?" and you get an implementation of X. Ask "is X a good approach?" and you will probably be told it is reasonable, with some caveats, followed by a confident explanation of how to proceed.

Asking Different Questions

The practical response is to ask different questions before you start executing.

"What are the weakest points in this design?" gets better answers than "does this design look right?" "What assumptions am I making that could be wrong?" gets better answers than "have I missed anything?" "Argue against this approach" gets better answers than "what do you think?"

Framing questions around interrogation rather than confirmation exploits what AI is actually good at: pattern matching against a vast library of how systems have gone wrong. It has encountered more failed architectures than any individual engineer. Ask it to look for failure modes, not to validate your solution.

The other shift is sequencing. Use AI to challenge the plan before you are committed to it, not to help you execute a plan you have already decided on. The cost of changing direction in a design conversation is zero. The cost of changing direction after three weeks of implementation is not.

In Practice

In an architecture session: describe the problem and the proposed approach, then ask AI to steelman the alternatives and identify the assumptions that would have to hold for your approach to be correct. You will often find one assumption you had not named.

In code review: ask AI to find the edge cases you are not handling, not to tell you whether the code is good. The former produces actionable output; the latter produces praise with caveats.

In design work: draft the design, then ask AI to take the opposing position. Not "what could be improved" but "make the case against this". The objections that come back are usually real ones.

A Working Convention

Over time I have settled on a working convention for AI collaboration that formalises this cognitive stance. The core is a separation between a planning mode, in which the AI is explicitly directed to challenge assumptions and surface hidden context before any execution happens, and a tactical mode, in which the scope is agreed and execution is the focus.

The convention is open source and available at resonance-protocol.org. It is not required reading for any of the above. The underlying principle is simple enough: tell AI what cognitive stance you want it to take, and it will largely take it. The training process creates defaults. You can, to a great degree, override them.