How accounting professionals can stay confident, accountable and compliant in an AI world

We caught up with AAT president Lucy Cohen, who has researched the impact of AI in accountancy and finance roles extensively, to see what skills are needed in a rapidly changing industry and the risks that come with it.

Are current accounting professionals too trusting of automation?

Trust is the wrong axis. The problem is that two completely different categories of technology are being treated as one thing.

A deterministic, rule-based system produces outputs that are logical, traceable and reproducible. A probabilistic generative system produces outputs sampled from a distribution, where the same input may not give the same answer twice and the working cannot be recovered afterwards. Most practitioners are, if anything, under-trusting of the first and over-trusting of the second. Frequently both are inside the same product, and nothing in the interface tells you which one just posted your client’s transaction.

The behavioural evidence on this is not encouraging. Research on clinical decision support found clinicians switched from correct to incorrect answers in around five percent of cases after being shown an erroneous system recommendation. Automation bias gets worse, not better, as output quality improves, because plausible, well-formatted, internally consistent output gives the reviewer nothing to catch on. A system that produced obviously silly answers would be safer, because it would create friction.

What risks do you think accountants underestimate when adopting AI-powered bookkeeping tools?

The most underestimated risk is that firms evaluate these tools on accuracy when the real exposure is verifiability.

A financial record has to do three things. The accounting equation has to hold across the whole system at once. The correctness of the record has to be demonstrable from source evidence if it is challenged. And there has to be an unbroken trail from the figure back to the primary document. Those three properties together are what make something a financial record rather than a number in a database.

A probabilistic system can produce the right answer and still fail the second and third of those on every single output, because it samples from a distribution rather than deriving from a rule. There is no derivation chain to inspect. So the record may be numerically correct and still legally undefendable, and you will not be able to tell which is which by looking at it.

The second thing firms underestimate is that machine errors behave differently from human ones. A bookkeeper having a bad afternoon makes scattered, random mistakes. A model that misclassifies a supplier misclassifies that supplier consistently, across every client, in a way that looks entirely plausible and entirely consistent with the population. Random error shows up as an anomaly. Correlated error shows up as a pattern, which is to say it does not show up at all.

The third is liability. When the figure is wrong, HMRC does not pursue the software vendor. It pursues the taxpayer and, professionally, their agent.

What warning signs should firms look for when evaluating AI-led accounting software?

Ask the vendor these questions and pay close attention to which ones produce a straight answer.

Show me the derivation. For any given posting, can the system tell me which rule fired and which piece of source evidence it was applied to? If the answer is a confidence percentage, you have been given a probability, not a derivation.

Run it twice. Same input, same output, every time? If not, it is sampling.

Is classification memory pooled across clients? If the system learns from all customers, another business’s coding decisions are shaping your client’s records, and you have no visibility of whose.

Does it change its own behaviour without anyone approving the change? Silent self-updating means the logic that produced last year’s accounts no longer exists to be re-examined.

Can AI-derived text post on its own? Receipt reads, OCR descriptions and generated narrative should be advisory input to a matching process. They should never post unattended.

Is the audit trail a record of derivation or a record of approval? A log saying a human clicked yes is a governance record. It is not an audit trail.

One more. Treat “HMRC-recognised” with care. Recognition means the product connects correctly to HMRC’s API in the required format. It is a compatibility test, not an accuracy assessment, and it does not examine the processing architecture at all.

What new skills will accountants need as AI becomes more embedded in finance functions?

Not prompt engineering. Four things:

1. Rule specification: The ability to express an accounting treatment as explicit, testable logic that someone else can implement and audit. This is a much deeper skill than it sounds, because it demands that you know why a treatment is correct rather than simply that it is.

2. System-level assurance: Reviewing populations rather than transactions. Sampling design, control totals, exception profiling, movement analysis. Knowing what a healthy output distribution looks like and noticing when it stops looking like that.

3. Exception judgement: Once routine processing is genuinely handled, everything reaching a human is by definition novel, ambiguous or material. The remaining work is harder than the work that went away.

4. Architectural interrogation of suppliers: Someone in every firm needs to be able to hold a technical conversation with a vendor and know when they are being sold a confidence score dressed up as an audit trail.

Where do you think the line should be drawn between automation and professional oversight?

The line should be drawn by architecture, not by transaction value or volume. Which is an unpopular answer, so let me give you the evidence.

We ran a deterministic accounting system across a population of UK micro-businesses. In automated operation with escalation only for exceptions, it processed at twelve minutes per client per month against a legacy manual baseline of seventy-one minutes, with a zero defect rate against materiality. Then we put qualified accountants into the processing loop. Identical rules, equivalent data, the only change being human operators actively driving each stage rather than receiving escalations.

Processing time went to forty-nine minutes. Error rates returned to documented human norms of one to three percent, having been between thirty-two and ninety-six times lower under full automation. The accountants created a new category that was slower than automation and carried the error profile of manual processing.

So for a deterministic system, oversight belongs at the level of system state rather than individual output. Humans should own the rules, the change control on those rules, the aggregate monitoring, the exceptions and the accountability. Inserting them into routine processing adds cost and subtracts accuracy.

For a probabilistic system the line sits somewhere else entirely, and much further back. It should not be permitted to write to the ledger at all. It can suggest. Something deterministic, or somebody qualified, decides.

Are we moving towards a future where accountants manage AI systems rather than accounting records?

We are already there in parts of the profession, but I would resist the framing slightly. Managing the system is managing the records. It happens at a higher level of abstraction, and the consequences of getting it wrong are larger rather than smaller, because a rule error affects an entire population rather than one entry.

The real risk in that transition is treating it as an IT responsibility. The person who specifies how a transaction should be treated has to be the person who understands why that treatment is correct under the relevant standard and the relevant tax law. Once that specification is handed to someone without the accounting knowledge, you have automated something nobody in the building can defend.

Accountability does not transfer to the vendor when the processing does. That is worth saying out loud more often than the profession currently says it.

What does a “safe” use of AI in accounting look like in your view?

Safe use has hard boundaries rather than good intentions. Mine are these:

• Anything AI-derived is advisory input to a decision, never the decision. A generated description or an extracted figure is a suggestion offered to a matching process. It never posts on its own.
• Classification runs deterministically against a versioned ruleset that a qualified human approved, under change control, with a history you can reconstruct.
• Classification memory stays scoped to the individual client. Nothing is pooled.
• Posting is forward-only. Corrections happen as new entries. Filed periods do not reopen and history does not get rewritten.
• Escalation is a designed feature with defined triggers, not the thing that happens when the system runs out of ideas.
• Assurance is built into the processing rather than inspected afterwards, because after the fact review of plausible output catches very little.

None of this rules out generative AI in a practice. It is genuinely excellent at drafting client communications, summarising legislation, interrogating a set of accounts you already trust, and producing first-draft narrative. All of that is safe precisely because none of it enters the ledger.

If you could give one piece of advice to AAT members about AI adoption, what would it be?

Ask your software provider one question, about one real posting: show me the rule that produced this and the evidence it was derived from. If they cannot answer it, you are putting your name to a figure you have no way of defending. And when it is challenged, it will be your name on it. Not theirs.

Further reading

Ask an accountant: How you’re testing out AI in practice

Artificial Intelligence 101

From Accounting to advisory: why the role of the accountant is changing

Harry Rogers is AAT Comment’s news writer.