I keep seeing the same pattern in mid-market organizations across various industries.

Leadership makes a smart call to move quickly with AI. The pressure is real. Boards are asking questions, and peers are rolling out pilots that now feel like table stakes. So the company hires an AI guru, a talented practitioner tasked with building AI use cases fast and proving value in short order.

At first, everything goes to plan. Use cases get built. Productivity goes up. Demos impress the executive team.

Then reality steps in.

It usually shows up in one of two ways:

• A CxO surfaces a regulatory reference like the 2026 FINRA Annual Regulatory Oversight Report, an EU AI Act briefing, or a board-level risk update.
• Or a client, partner, or investor sends a security or due diligence questionnaire asking a deceptively simple (but often challenging) question: “Please describe your AI Governance program.”

That is when we get the call.

“We need AI governance, yesterday!”

Sometimes the organization has no formal AI governance program at all. In other cases, it has the start of one: an AI Acceptable Use Policy, a risk register template, and an AI Risk Management Committee. But the program is still incomplete and not yet operationalized.

In almost every case, leadership turns to the same person:

“You need to build our AI governance program.”

And by you, they mean the AI guru.

On paper, that seems logical. This person knows the models, data, tools, and roadmap. They know how the company uses AI. Who better to document it and help govern it?

In practice, this is where organizations fall into a trap.

The friction nobody wants to acknowledge

The AI guru was hired, and in many cases rewarded, to move fast. Governance, risk management, and documentation are not why they come to work each morning. Most see governance as a necessary speed bump at best and, at worst, as something that slows innovation without a clear payoff.

So what happens?

They do the minimum needed to get past the immediate hurdle:

• An initial AI inventory.
• A pilot use case in the risk register.
• A project to update the third-party risk management (TPRM) program. 
• A diagram or two that looks good in a slide deck.

This is not malicious. It is rational behavior tied to their main job: building and deploying AI.

The bigger problem comes next.

When governance becomes self-governance

Once the documentation is in place, leadership may assume the job is done. Then the same AI guru, or the same AI team, is expected to:

• Operate the AI systems.
• Expand AI use cases and extend existing systems.
• Assess and manage AI risks.
• Validate that governance controls are working.
• Answer regulator or client questions about compliance.

At that point, the organization has moved from incomplete governance to something more troubling: a segregation of duties problem.

The people designing and deploying AI systems are now also the ones judging whether those systems work as they should, producing the records needed to show compliance, and deciding whether governance is in place.

That is not governance. It is self-attestation.

This approach fails every meaningful framework test

What stands out is that the major frameworks all address this issue, even if they use different terms.

NIST AI Risk Management Framework (AI RMF)

The AI RMF’s Govern function focuses on organizational oversight, accountability, and clear lines of responsibility that stay separate from system development and operations. Governance sets risk appetite, watches outcomes, and escalates issues. It is not there to defend design choices.

When the same team does both jobs, independence disappears.

EU AI Act

The EU AI Act places heavy emphasis on risk management, human oversight, and accountability throughout the AI lifecycle. That oversight has to be real, not symbolic. In high-risk cases, the Act assumes that someone other than the implementer is checking whether controls are sufficient and whether risks are being managed effectively.

Self-governance breaks that assumption.

ISO/IEC 42001 (AI Management Systems)

ISO 42001 does not spell out “segregation of duties,” but it does not need to. Like other ISO management system standards, it calls for:

• Clearly assigned roles to keep the AI management system in line with requirements.
• Independent reporting on system performance to top management.
• Monitoring, review, and ongoing improvement that are not compromised by conflicts of interest.

The point is clear: the person building the system should not be the only one deciding whether governance works.

ISO/IEC 27001

ISO 27001 is more direct. Its Annex A calls for separating conflicting duties to reduce the risk of errors, abuse, and control failures. When a governance function audits or validates its own work, that is a known anti-pattern. Many firms are now learning that lesson the hard way with their AI governance.

The mid-market reality: “We don’t have infinite staff.”

To be fair, mid-market organizations rarely have the luxury of fully separate teams for everything. Expecting a financial services style three-lines-of-defense model to materialize overnight is not realistic.

But the frameworks do not ask for perfection. They ask for intent and integrity.

What they do not accept is a claim that governance exists when:

• It is owned by the same role that gains from fewer constraints.
• It lacks the authority to slow, question, or stop an AI initiative.
• It reports sideways instead of up.

Auditors, regulators, and sophisticated clients are getting better at spotting this.

What good looks like (and what we see working)

The organizations that handle this well do not sideline their AI experts. They reposition them.

Common patterns we see working:

• AI practitioners keep building and operating systems.
• A separate governance function, such as risk, compliance, security, privacy, or an AI steering committee, owns oversight, validation, and reporting.
• When full separation is not possible, documented compensating controls are put in place: independent reviews by an external party, formal challenge processes, management sign-off, and clear escalation paths.

The difference matters: the AI guru informs governance but does not own it.

The uncomfortable truth

AI governance is not failing in the mid-market because organizations do not care about risk. It is failing because they are asking the wrong people to own it.

Speed and governance both matter, but they pull in opposite directions. Asking one role to optimize for both almost always means governance loses.

Over the next few years, the organizations most able to scale AI will not be the ones moving fastest. They will be the ones who see independent governance not as bureaucracy, but as what lets them move fast with minimal risk.