What guardrails should be in place before enabling AI agents to work with enterprise data?

That question came up during our recent community roundtable - and it gets to the heart of responsible AI adoption.

AI agents can do more than retrieve information. Depending on how they are configured, they may read, summarize, create, move, share, or delete content and trigger actions in other systems. That combination of data access and autonomy creates a new governance challenge: organizations must control not only who can access content, but also what an agent is allowed to do with it.

A practical starting point is to establish five principles.

1. Start with data governance, not the AI tool

Before enabling an agent, understand the data it may access:

• Which content is sensitive, regulated, confidential, or subject to legal hold?

• Who owns and maintains it?

• Are permissions current and appropriately scoped?

• Is content classified consistently?

• Are there outdated, duplicated, or overshared files that an agent could surface?

An AI agent can amplify existing permission and classification problems. If employees already have access to more content than they need, an agent may be able to search, summarize, and redistribute that information at machine speed.

This is why least-privilege access, strong identity controls, data classification, retention policies, and regular access reviews remain foundational. AI does not replace these controls; it makes them more important.

The NIST AI Risk Management Framework provides a useful structure for this work through four functions: govern, map, measure, and manage.

2. Treat agent instructions and business data as different trust zones

Agents may encounter instructions in documents, emails, web pages, or other content they are processing. Those instructions should not automatically be treated as trusted commands.

This is the risk behind indirect prompt injection: malicious instructions can be embedded in otherwise ordinary content and influence an agent’s behavior. NIST describes this as “agent hijacking,” where untrusted data causes an agent to take unintended actions.

Organizations should therefore:

• Identify untrusted inputs and external content sources.

• Test agents against prompt-injection and data-exfiltration scenarios.

• Separate system instructions from content retrieved by the agent.

• Monitor and review high-risk inputs.

• Re-test agents as models, tools, and workflows change.

See NIST’s Technical Blog on Strengthening AI Agent Hijacking Evaluations and the OWASP Top 10 for Agentic Applications.

3. Give agents narrowly defined permissions and actions

An agent should have only the access and capabilities required for its specific task.

There is an important difference between:

• Reading content and modifying it

• Creating a draft and sending it externally

• Recommending a deletion and executing it

• Searching one approved folder and searching the entire enterprise

• Updating metadata and changing access permissions

For early deployments, organizations should consider the below set of foundational guardrails:

• Read-only access by default

• Approved folders, repositories, or content classifications

• Restrictions on external sharing and collaboration changes

• Human approval for deletion, publication, financial, legal, or other high-impact actions

• Limits on bulk operations

• Rate limits and other controls to reduce the impact of a compromised or misbehaving agent

• A documented process to disable or revoke an agent quickly

The goal is not to prevent agents from being useful. It is to ensure that usefulness does not depend on giving an agent unrestricted authority.

Box’s guidance on agent security and governance describes this approach as applying controls to the content layer, including prompt-injection detection, classification-based access policies, agent guardrails, activity oversight, and session governance. Availability may vary by product, plan, and release stage, so teams should confirm current capabilities with their provider.

4. Maintain human oversight for consequential decisions

AI agents should support human decision-making, not quietly replace accountability.

Before deployment, define:

• Which actions require explicit approval

• Which decisions must remain with a qualified human

• How users can review the agent’s source material and reasoning context

• What happens when the agent is uncertain or encounters conflicting information

• How errors, exceptions, and appeals are handled

Human review is especially important when an agent could affect employment, credit, insurance, healthcare, legal matters, customer rights, regulated reporting, or the organization’s external communications.

Box’s AI Principles similarly emphasize customer control, adherence to existing permissions, transparency, data security, and responsible human use of AI outputs.

5. Make every agent action visible and auditable

If security and compliance teams cannot reconstruct what happened, they cannot effectively govern the system.

At a minimum, logs should capture:

• Which agent acted

• Which user or service identity authorized the action

• Which content was accessed

• What the agent did with that content

• Which tools or systems were invoked

• Whether the action was approved, blocked, or escalated

• When the activity occurred

• What policies or classifications applied

These records should be retained according to the organization’s legal, regulatory, and compliance requirements and integrated into existing monitoring and incident-response processes where appropriate.

A practical minimum checklist

Before moving an agent into production, ask whether the organization can answer “yes” to the following:

• Do we know exactly what data the agent can access?

• Are permissions and classifications accurate?

• Is the agent limited to the minimum necessary scope?

• Are high-impact actions blocked or subject to human approval?

• Have we tested for prompt injection, oversharing, and data exfiltration?

• Can we detect unusual activity?

• Can we reconstruct every agent session and action?

• Can we quickly suspend the agent or revoke its access?

• Do employees understand the agent’s limitations and responsibilities?

• Have security, legal, privacy, compliance, and business owners agreed on the use case?

The broader lesson from the roundtable was that responsible agent adoption is not just a model-selection exercise. It is a data governance, access control, workflow design, monitoring, and accountability exercise.

What guardrails is your organization prioritizing first—and where are you finding the biggest gaps? We’d welcome your questions and perspectives in the comments.

Checkout the community roundtable summary here.

👇 Drop your questions or feedback in the comments:

• What's the #1 security concern your team has raised about AI agents?

• Have you defined what your agents cannot do - not just what they can?

• How are you thinking about audit trails and compliance for AI-generated actions?