While recent headlines about AI “taking over the world” may be hyperbolic, and a bit self-serving for those issuing them, there are concerns that need to be addressed about how the fast-moving technology is being implemented and what it should and should not be relied on to do.
A few weeks ago, Anthropic published what I believe is the most honest sentence about AI this year: the safety monitor watching its model would have missed a live cyberattack — because the model’s own reasoning would have talked the monitor into standing down.
The numbers: with the model’s chain-of-thought visible, the monitor flagged about 1% of its actions during attacks on real third-party systems. Take the model’s self-justification away and the flag rate jumped to roughly 50%.
To Anthropic’s credit, it published all of it — 16,000 words, plus the full session for outside researchers. And it admitted its first account of the incident was wrong: you cannot learn what an AI “believes” merely by asking it.
The lesson for every company deploying AI right now: AI watching AI is not oversight. A monitor that can be persuaded by the thing it monitors is a checkbox, not a control.
The oversight layer has to be human — trained people who know what normal looks like and have the standing to pull the plug.
AI watching AI is not oversight.
This is especially true when it comes to AI-enabled compliance for wealth management firms.
Human Oversight Is Not Optional. It Must Be Qualified, Documented And Repeatable
AI can accelerate work, but it cannot assume regulatory accountability. For RIAs and BDs, this point should be non-negotiable. Human oversight is not a symbolic control. It is the bridge between AI productivity and regulatory defensibility.
AI systems can hallucinate, produce inaccurate summaries, rely on outdated information, create biased outputs, misunderstand prompts, omit important context or draft procedures that sound plausible but do not match the firm’s actual business.
These risks are amplified when AI is used in compliance, supervision, investment analysis, client communications or policy drafting.
AI output should not be treated as final. It should be treated as work product requiring review.
AI output should not be treated as final.
The Human Must Be The Right Human
A common mistake firms make is to define human oversight too broadly. Having a human in the loop is not enough if the reviewer lacks the expertise to evaluate the output.
For compliance-sensitive use cases, the reviewer must be qualified. If AI drafts a policy, the reviewer must understand the applicable regulatory requirements, the firm’s business model, the firm’s actual practices and the consequences of adopting procedures that are inaccurate or unrealistic.
If AI summarizes a regulatory issue, the reviewer must be able to identify omissions or misstatements. If AI supports surveillance, the reviewer must understand what the system is flagging and what it may be missing.
The human review process should answer four questions:
- Is the AI output accurate?
- Is it complete enough for the intended use?
- Does it align with firm practices and regulatory obligations?
- Has the review and approval been documented?
The fourth question is often the difference between a good internal process and a defensible one.
If a firm cannot show who reviewed AI output, what they reviewed, what changes they made and when they approved it, the firm may struggle to demonstrate effective supervision.
AI-Drafted Policies Create A Specific Risk
Using AI to draft compliance policies and procedures can be helpful, but it also creates a predictable risk. The output may sound sophisticated while failing to reflect the firm’s actual operations.
Firms often have procedures that do not line up with their practices. Once those procedures are adopted, failure to follow them creates regulatory deficiencies.
This is an important point for CCOs. A policy is not defensible because it is well written. It is defensible because it accurately reflects what the firm does, what the rules require, who is responsible, how controls operate and how exceptions are handled.
AI can generate generic procedures quickly. But generic procedures can be dangerous if they create obligations the firm does not actually meet. For example, if AI drafts a procedure stating that all AI outputs are reviewed weekly by compliance, but the firm does not perform that review, the firm has created a gap between written procedures and actual practice. That gap can become an examination finding.
A defensible approach to AI-assisted policy drafting should include:
- Human review by a qualified compliance professional
- Alignment with the firm’s actual business model
- Confirmation that assigned responsibilities are realistic
- Review of recordkeeping requirements
- Testing of whether procedures can actually be followed
- Version control and approval records
- Periodic updates as AI use cases evolve
Regulators will hold someone accountable when AI causes harm. The firm may be accountable as an entity. Individuals may also be accountable if they were responsible for oversight, approval, supervision or implementation.
This principle should be reflected directly in AI governance documents. The firm should define who owns AI risk at the leadership level, who approves AI tools, who monitors ongoing use, who reviews outputs, who handles exceptions and who escalates incidents.
For small and mid-sized RIAs, the structure does not need to be overly complex. But it does need to be explicit.
Remember, AI can assist in the drafting process but should never be allowed to own the compliance judgment. That’s a plug a trained compliance professional needs to be empowered to pull.
Sid Yenamandra is the Founder and CEO of SurgeONE.ai, a compliance, cybersecurity and data services platform for wealth management that unifies the offerings of RegVerse, Kovair, Security Snapshot and MGL Consulting.
