Don't Let Your AI Agents Run Wild: A Policy Field Report

How do you deploy agentic AI without becoming the next compliance horror story?

That's the question every automation lead is asking. The technology is moving fast, and the rules are still settling. But there's a clear path forward if you're willing to think like an engineer, not a lawyer.

The Scenario That Should Scare You

Imagine you're the head of automation at a mid-sized insurer. Your CEO just read a pitch about "agentic automation" and wants a pilot. You're told to build a system that handles claims triage. It should read incoming claims, classify them, and route them to the right adjuster or automated process. Sounds simple. But here's the catch: your company is subject to the EU AI Act because you operate in Europe. And if you're not in Europe, you're probably still subject to it if you serve EU customers.

The EU AI Act is the first comprehensive legal framework for AI worldwide (European Commission). It classifies AI systems by risk. Your claims triage bot might not be "high-risk" under the Act, but if it's making decisions that affect people's access to insurance, you need to be careful. The Act's prohibited practices include emotion recognition in the workplace and social scoring (European Commission). You're not doing that. But you're still building an autonomous system that could make mistakes.

The Tech Stack: RPA as the Safety Net

You've heard the buzz about AI agents replacing RPA. Don't believe it. The dominant architecture is hybrid: AI agents do the thinking, RPA does the doing (Wikipedia). RPA is deterministic and brittle, while agentic AI handles variability but introduces uncertainty (Wikipedia). That's actually a good thing. You want your agent to decide which claims need human review, but you want a robot to execute the routine steps reliably.

For your pilot, you'll use a large language model as the reasoning core. It will use tool calling to access your claims database and send emails (IBM). You'll also need a way for the agent to call external tools. The Model Context Protocol (MCP) is an open standard that does exactly that. It's like a USB-C port for AI applications (Model Context Protocol). You can use it to connect your agent to your claims system and other services. Microsoft's Foundry Agent Service supports remote MCP servers, so you can plug in an Azure DevOps server if needed (Microsoft Foundry).

But here's the thing: the agent is not a black box. You need to design it with human oversight. IBM calls this human-in-the-loop (HITL), and it's a key part of iterative refinement (IBM). You should set up checkpoints where a human approves any high-stakes action, like sending a denial letter.

The Ethical Checklist: NIST and the EU AI Act

Now, let's talk policy. You don't have to wait for the EU AI Act's high-risk obligations to kick in. Start with NIST's AI Risk Management Framework. It's voluntary, but it's a solid blueprint. Released in January 2023, it focuses on trustworthiness: valid, reliable, safe, secure, resilient, accountable, and transparent (NIST).

Your checklist should include:

• Define the human oversight points. You must have a human review any decision that materially affects a customer.
• Ensure data quality. The AI Act will require high-quality datasets for high-risk systems (European Commission). Start now.
• Log everything. The AI Act requires activity logging for high-risk systems (European Commission). You'll need an audit trail.
• Test for bias. Your agent might inadvertently discriminate. Use the NIST framework to evaluate.

The EU AI Act's rules on general-purpose AI models became effective in August 2025 (European Commission). That's your LLM. You need to document how you're using it.

The Financial Reality: Don't Overbuild

You don't need a massive system. Start small. The RPA market is growing, but so is the cost of doing nothing. The global RPA market was estimated at $4.68 billion in 2025 (Grand View Research). That's a lot of companies investing in automation. But you can start with a single process.

Here's a concrete example: your claims triage agent. IBM cites a case where a multi-agent legal research assistant cut contract review time from 90 minutes to 45 minutes by routing simple queries through a low-cost classifier first (IBM). You can do the same. Use your LLM to classify claims as low, medium, or high complexity. For low-complexity claims, have an RPA bot process them automatically. For high-complexity ones, route to a human adjuster.

But watch out: even state-of-the-art models fail on real-world tasks. When SWE-bench was introduced, the best model solved only 1.96% of 2,294 real-world coding issues (SWE-bench paper). That's a reminder that your agent will make mistakes. You need fallbacks.

Your Move: Start Small, Govern Hard

Here's my recommendation: don't try to boil the ocean. Pick one process, like claims triage, and build a hybrid system with clear human oversight. Use MCP to connect your agent to your tools. Document everything under NIST's framework. And if you're in the EU, map your system to the AI Act's risk categories now, even if you're not required to until December 2027 for high-risk systems (European Commission).

The biggest barrier to adoption isn't technology; it's organizational change and human oversight (Wikipedia). That's good news. You can start today.

Bottom line

Your single best move is to design your agentic AI with a human-in-the-loop checkpoint at every decision that matters, and to document that design against NIST's RMF. That way, when the EU AI Act comes knocking, you're already compliant.

Sources

• Robotic process automation (Wikipedia) - https://en.wikipedia.org/wiki/Robotic_process_automation
• IBM (AI agents) - https://www.ibm.com/think/topics/ai-agents
• European Commission (EU AI Act) - https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
• NIST AI Risk Management Framework - https://www.nist.gov/itl/ai-risk-management-framework
• Model Context Protocol (official docs) - https://modelcontextprotocol.io/introduction
• Grand View Research (RPA market) - https://www.grandviewresearch.com/industry-analysis/robotic-process-automation-rpa-market