AI Governance in Care Settings: What CQC Is Already Looking For

A care provider was rated Inadequate by the Care Quality Commission last August. One of the contributing factors was ungoverned AI use — confidential care records processed through AI tools with no risk assessment, no governance framework, and no evidence that anyone had paused to consider whether the use was appropriate. This is not a future risk. It is a documented outcome that has already occurred.

The case, shared publicly by Kevin Humphreys, CEO of Oakland Care Group, at the South East Social Care Alliance Annual Conference this week, represents a significant moment in how the care sector should understand AI and CQC compliance. The regulator is not waiting for formal AI-specific guidance to be published before it acts. It is already assessing whether AI use in care settings is governed or ungoverned, and it is already incorporating that assessment into ratings decisions.

The Distinction That Matters

The correct inference from this case is not that AI use is dangerous. It is that ungoverned AI use is dangerous. Those are not the same thing, and treating them as equivalent would lead providers to avoid AI tools entirely — an overcorrection that leaves real efficiency and quality benefits on the table — rather than implementing the governance that makes AI use defensible.

A care provider who uses AI tools with a documented governance framework — knowing what tools are in use across the service, what data those tools process, who is responsible for oversight, how AI outputs are reviewed before entering care records, and how that governance could be demonstrated to an inspector — is in a fundamentally different position from the provider in Kevin's case. The difference is not the technology. It is the governance.

The CQC's assessment framework asks inspectors to evaluate whether a service is well-led. Well-led services have systems for identifying and managing risk. The use of AI tools in care documentation, without any assessment of the risks that use introduces or any framework for managing those risks, is a governance failure in exactly the terms the Well-led key question is designed to identify. It is not a technology question. It is a leadership and governance question — which is precisely why it can contribute to an Inadequate rating.

What Ungoverned AI Use Actually Looks Like

Understanding why the case described by Kevin Humphreys produced a CQC finding requires understanding what ungoverned AI use looks like in practice in a care setting.

A care worker or manager who uses a general-purpose AI tool — a chatbot, a writing assistant, a summarisation tool — to help draft care records, progress notes, or care plans, without any organisational assessment of whether that tool is appropriate, is using AI without governance. The care records that result may read correctly. They may use appropriate language and structure. But they have been produced through a process the organisation has not evaluated, using a tool whose data handling the organisation has not assessed, in a way that leaves no record of AI involvement for anyone reviewing the documentation later.

When a CQC inspector asks to see the care records and governance documentation for a service that has been doing this, what they find is documentation that may not reflect what actually happened — because AI tools can plausibly describe care that was not provided, in language that sounds appropriate, without any of the evidential grounding that genuine care records require. They also find no evidence that the organisation understood or managed the risk. Both findings are serious.

The confidentiality dimension in Kevin's case adds a further layer. Processing confidential care records through an AI tool — entering patient names, medical histories, care needs, or other personal information into a general-purpose AI system — is a data protection concern as well as a governance one. The ICO's Better Records Together campaign, which moved to enforcement this week with reprimands, enforcement notices, and financial penalties already issued for care records failures, sits alongside the CQC finding rather than separately from it. A provider who processes confidential care records through an ungoverned AI tool is potentially failing two regulators simultaneously.

What a Governed Alternative Looks Like

Kevin Humphreys described his own demonstration at the conference as "two glass boxes" — AI tools that are auditable, transparent in their reasoning, and structured so that humans remain in every decision. That framing is useful because it describes the principle rather than the specific technology: the test of governed AI use is not which tool you are using but whether you can see into what it is doing, who is responsible for reviewing its outputs, and whether there is a record of that review.

Translated into practical governance requirements for a CQC-regulated care service, this means five things.

1. AI Systems Register

A record of what AI tools are in use across the service, what they are used for, what data they process, and who authorised their use. This is the AI Systems Register — the baseline document that allows an organisation to demonstrate it knows what AI is being used within its operation.

2. Risk Assessment

A risk assessment for each AI tool in use, covering data protection risks, accuracy risks, the potential for AI outputs to be used in care records without sufficient human review, and the mitigations in place for each identified risk. This is the document that transforms AI use from something that happened informally into something the organisation has evaluated and managed.

3. Acceptable Use Standard

An acceptable use standard that sets out what AI can and cannot be used for within the service — specifically distinguishing between uses where AI assistance is appropriate and reviewed, and uses where AI output should not enter care records without significant human verification. This standard gives staff clarity and gives inspectors evidence that the organisation has thought carefully about appropriate boundaries.

4. Output Review Process

A process for reviewing AI outputs before they are used in care records, with a record of that review. The review record is the document that closes the loop — it shows that a human read, assessed, and took responsibility for the content before it became part of a formal care record. Without this record, the fact of human review cannot be demonstrated retrospectively.

5. Supplier Due Diligence

A supplier due diligence process for any AI tools provided by third parties — assessing how the supplier handles data, what their terms of service say about how inputs are used, whether the tool has been assessed for use in health and care settings, and what the organisation's rights are if things go wrong.

These five requirements are not onerous for a well-run service. They are the minimum governance infrastructure that distinguishes an organisation that has thought about AI from one that has not.

The Timing of This Conversation

The ICO's Better Records Together campaign moved to active enforcement this week. The NMC is consulting on changes to nursing education standards. The NHS has rolled out AI tools to 500,000 staff. The Medical Protection Society has published a report on the AI liability gap for clinicians. The Government is urging the NHS to take more risks with AI adoption while professional bodies warn that the workforce plan relies too heavily on unproven AI assumptions.

All of these developments share a common thread: the pace of AI adoption in health and care is outrunning the governance frameworks designed to make that adoption safe. Kevin Humphreys put it directly at the SESCA conference: the technology is ready. The question is whether governance is keeping pace.

For care providers making decisions now about which AI tools to use and how to use them, the answer is clear. The governance framework needs to exist before the AI tool is used, not assembled in response to a CQC finding. The risk assessment needs to be completed before confidential care records are processed through an AI system, not after a data protection concern has been raised. The record of human review needs to exist before an inspector asks for it, not reconstructed from memory after the fact.

The case that Kevin shared is not an isolated failure of one provider in unusual circumstances. It is an early indicator of the enforcement direction the regulator is already moving in. Providers who treat it as a warning and act now are in a fundamentally different position from those who treat it as someone else's problem.