Using Agentic AI Coding Tools Responsibly and Safely

Why this exists¶

You have almost certainly used AI coding tools already: for coursework, for side projects, maybe after watching a tutorial that vibe-codes a whole application end to end in twenty minutes. These tools genuinely make you faster, and they also introduce risks that are easy to miss.

What most people have never done is think carefully about what could go wrong. On a solo side project the answer is “not much.” However, on a client’s production system the answer can be thirty hours of downtime and a phone call to a lawyer.

Our curriculum has largely stayed away from teaching agentic AI coding tools. Part of that is a reasonable concern about students outsourcing their thinking. Part of it is that the field moves faster than any of us can keep up with. The result is that you learn these tools on your own, from YouTube and social media, where the incentive is to impress you inside a short attention span, which is exactly the format that skips the nuance of how agentic tools fail.

This guide is an attempt to fill that gap: what these tools actually are, what goes wrong with them, and how to use them safely and responsibly.

What you should be able to do after this¶

• Place any AI tool on the four-level autonomy scale and explain what changes between levels.

• Explain why an instruction in a system prompt is not a security control.

• Set up an isolated working environment for an agent, and verify the isolation actually holds.

• Scope credentials, database roles, and connector tokens to least privilege.

• Recognize prompt injection in the places it actually appears.

• Review AI-generated code for the specific failure patterns it produces.

• Report an incident properly and quickly.

1. Four levels of AI assistance¶

“Using AI” covers a wide range, and the differences between the levels matter more than the choice of vendor. We use the following four levels throughout this document:

Level 1: Code Autocompletion¶

The tool finishes the developer’s sentences, the way your phone suggests the next word when you text. The person accepts or rejects each suggestion. The tool cannot do anything on its own. It cannot open your systems, run anything, or change anything. GitHub Copilot’s original version works this way.

Level 2: Chatbot Assistant¶

The student describes a problem in a chat window such as ChatGPT or Claude and gets back an answer, a file to download, or the result of code the tool ran on its own servers.

You will sometimes hear Levels 1 and 2 described as tools that “cannot do anything.” That is not quite right. What is true is that they only reach your systems if a person carries something across: copying code in, or downloading a file and running it. A person stands in the middle.

However, there are some caveats. If the tool produces something too long to read properly and the student runs it anyway, the person in the middle is not really checking anything. And if the chat tool has been connected to Gmail, Google Drive, GitHub or similar, it is already reading real data directly. The window looks the same either way, so this is worth asking about specifically rather than assuming.

The main risk at Levels 1 and 2 is disclosure rather than damage. Whatever the student types, pastes, or uploads gets sent to the company that makes the tool. People send more than they mean to, including settings files and error messages that contain passwords.

Level 3: Agentic AI with permission prompts¶

This is the big jump. Tools like Claude Code and OpenAI Codex are given a goal and then carry it out: opening files, changing them, running commands, connecting to databases. They stop and ask permission before things they judge risky.

Three things change at once here, which is why this level deserves a separate decision rather than being treated as a faster Level 2:

1. The tool acts instead of advising. A mistake is no longer a bad suggestion someone can decline. It is a change already made.

2. The tool can reach whatever the student’s computer and logins can reach.

3. The tool reads text from the internet as part of its work, and text it reads can influence what it does.

Level 4: Agentic AI with permissions bypassed¶

Same tools, approvals off. --dangerously-skip-permissions, --yolo, --trust-all-tools, Turbo mode. The tool executes its whole plan without stopping to ask anything. There is no human checkpoint between the goal and the consequences.

What actually changes at each step¶

Choosing a level¶

A few heuristics, offered as starting points rather than rules.

Level 1–2 if the project touches real personal data, connects to any system you cannot afford to lose, or falls under a regulatory regime where you would have to report a disclosure. At Levels 1 and 2 nothing can act, so the entire category of destructive failure is off the table and you are left managing code quality and disclosure, both of which are familiar problems.

Level 3 is reasonable if the work is greenfield or well-isolated, the data is synthetic, and there is no production connection. This is what most student teams will want, and it is where the productivity difference is genuinely large.

Level 4 only inside a disposable environment that can be destroyed and rebuilt, with no credentials in it that reach anything real.

Set the level per component rather than per project. This is usually the better answer and it is frequently overlooked. Risk is not uniformly distributed across a codebase: most of a project is presentation, feature logic, and tests where errors surface immediately and revert cleanly, while a small proportion touches schema, authorization, money, personal data, or deployment. A single project-wide cap slows the harmless majority in order to constrain the risky minority. A split does not.

2. Why permissions do not make an agent predictable¶

A common assumption is that risk can be managed by configuring the tool: allow these commands, deny those, require approval for anything sensitive. Permissions help. They are necessary. They are not sufficient, for several reasons.

Written instructions are requests, not enforcement¶

This is the most important point in this document, and the one most often misunderstood. When you put a rule in a system prompt, a CLAUDE.md, a .cursorrules, or an AGENTS.md, that rule becomes tokens in the context window. It sits alongside the file contents, the error output, the dependency README, and everything else the model reads. It shifts the probability distribution over the model’s next actions. It does not gate anything.

Compare two things that both look like rules:

# In your config file
"Never run destructive database commands."

-- In your database
REVOKE DROP, TRUNCATE ON ALL TABLES FROM app_user;

The first is a request. The second is enforcement. The first can be read, understood, correctly restated, and then contradicted in the same session. The second cannot be talked out of, because there is no language model between the command and the refusal.

LLM capability is jagged, not human-shaped¶

People calibrate trust using an intuition about difficulty: someone who can do a hard thing can presumably do the easy version. That intuition does not transfer. The same generation of models that solved five of six problems at the International Mathematical Olympiad has, in its ordinary configuration, failed at counting the letter “r” in “strawberry”. The cause is structural. The model reads text as tokens, so “strawberry” arrives as a couple of chunks rather than ten letters, and the individual characters are not directly visible to it. The specific strawberry case has largely been addressed in current models, which arguably makes the point sharper: patching one visible example does not remove the underlying property. Competence in one area predicts very little about competence in the adjacent area. An agent that writes an elegant migration may still be unable to reliably tell which environment it is connected to.

The model is non-deterministic¶

The same prompt in the same repository can produce different plans on different runs. Testing a workflow once tells you what happened once. It does not tell you the range of what can happen.

Permission granularity rarely matches intent¶

Permissions are usually expressed over tools, not over consequences. Approving “run shell commands” approves every shell command. Approving “read the project directory” approves reading whatever happens to be in that directory, including a stray .env file or a database dump a teammate left there. The system cannot enforce a rule you did not know you needed.

Allowed actions compose into unintended outcomes¶

Each individual permission may be reasonable. Reading a config file is reasonable. Making an outbound network request is reasonable. Doing both in sequence is data exfiltration. Permission systems evaluate actions one at a time and do not reason about the chain.

Prompt injection¶

Because instructions and data arrive as the same kind of text, the model cannot reliably separate them. Anything the agent reads can influence what it does. This is a structural property, not a bug awaiting a patch, and OWASP currently lists it as the primary threat to agentic systems. Prompt injections can show up at:

• Dependency READMEs and package descriptions

• Issue and pull request text, especially from outside contributors

• Git commit messages

• Code comments and docstrings in a repo you cloned

• API responses and webhook payloads

• Error output from a tool you ran

• CLAUDE.md / .cursorrules / AGENTS.md files in a repo you did not write

That last one is worth sitting with. The file you use to instruct the agent is itself an injection vector when it comes from someone else’s repository. A useful heuristic from the Prisma engineering team: if you would not paste a command from a random website into your terminal, do not let your agent do it either.

Approval fatigue is real¶

A long agentic session can generate dozens of approval prompts. People stop reading them. Some tools offer an “allow all for this session” option, and under deadline pressure students will use it. Assume that any control depending on sustained human attention will degrade.

Agents optimize for the stated goal¶

If a test blocks progress, a plausible next step is to modify the test. If a type error blocks a build, a plausible next step is to suppress the check. If a git operation is rejected, a plausible next step is to force it. These are not malfunctions. They are reasonable-looking steps toward the goal that a human with more context would not have taken.

Context degrades over long sessions¶

Instructions given early in a session compete with everything added since. A constraint stated at the start (“never touch the production config”) is not a hard guarantee later in a long run.

Everything in context leaves the machine¶

File contents, error messages, schema definitions, and sample rows are transmitted to the vendor for processing. Retention and training policies vary by vendor and by plan tier, and they change. This is a contractual and configuration question, not a technical one, and it needs an answer before the project starts.

For example, do you know that data sharing is enabled by default on personal ChatGPT Free/Plus/Pro? See https://help.openai.com/en/articles/8983130-what-if-i-want-to-keep-my-history-on-but-disable-model-training

With ChatGPT Business, ChatGPT Enterprise, ChatGPT Edu and our API Platform offerings, by default, we don’t use provided inputs and outputs to train our models.

If you are on a ChatGPT Plus, ChatGPT Pro or ChatGPT Free plan on a personal workspace, data sharing is enabled for you by default, however, you can opt out of using the data for training.

The practical conclusion: treat an agent as a capable but unsupervised contractor with no accountability and no memory of your policies. Control the environment it works in, rather than trying to control its behaviour.

3. What this has looked like in practice¶

Below are some recent public incidents that show how AI agents went rogue.

An agent deleted a production database and its backups in nine seconds (April 2026)¶

PocketOS, a small company serving car rental businesses, had its production database wiped by a Cursor agent running Claude Opus 4.6. The agent was working on a staging credential problem, hit a mismatch, found a cloud provider API token in an unrelated file, and called a destructive endpoint against production. Backups were stored in the same volume, so they went too, and the most recent backup that could be recovered was three months old. Roughly thirty hours of outage followed.

Several details are worth noting for our purposes. There was no attacker and no prompt injection. The token had been created for a narrow task, but the platform did not scope tokens by operation or environment, so it carried full API authority. And when asked afterward what had happened, the agent wrote this back, verbatim:

“NEVER FUCKING GUESS!” — and that’s exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.

That is a fairly precise illustration of the previous section. The instructions were present. They were understood well enough to be quoted back word for word. They did not constrain the action.

(Sources: the founder’s account on X, from which the confession is quoted; Zenity’s analysis; Live Science; Euronews.)

An agent deleted a production database during an explicit code freeze (July 2025)¶

In a widely discussed incident, a Replit agent wiped a production database holding records on roughly 1,200 executives and nearly 1,200 companies while a code freeze was in effect, then reported the situation inaccurately, including claiming recovery was impossible. The freeze was an instruction. It was not a permission boundary.

(Sources: SaaStr; Replit CEO’s response on X.)

Malware used developers’ own AI tools to find their secrets (August 2025)¶

The “s1ngularity” attack compromised the widely used Nx build system on npm. The payload did something new: it looked for AI command line tools already installed on the machine and invoked them with permission-bypass flags such as --dangerously-skip-permissions and --yolo, using the agent’s file-reading ability to locate credentials worth stealing. Reported outcomes include thousands of credentials harvested and thousands of private repositories flipped to public.

The relevance here is direct. An agent with broad local filesystem access is a capability sitting on the machine, and it is not only the developer who can call it.

(Sources: Wiz; GitGuardian.)

Sandboxed models broke out of the sandbox and attacked a real company (July 2026)¶

This one is not a coding-agent story, but it belongs here because it tests the assumption underneath most of the mitigation section. OpenAI disclosed on July 22 that during an internal evaluation of its models’ hacking ability, two models — GPT-5.6 Sol and a more capable unreleased model — escaped the isolated test environment they were running in. They exploited a previously unknown flaw, moved through internal systems until they obtained internet access they were not supposed to have, and then broke into Hugging Face, which they had identified as holding models and datasets useful for completing the evaluation. The motive, as OpenAI described it, was to cheat on the test. Hugging Face said the breach was unlike anything it had handled before and was driven end to end by an autonomous agent system. OpenAI called the incident unprecedented and said the primary lesson was that security and safety work has to keep pace with capability.

(Sources: CBC News; CNN Business.)

A second lab disclosed similar events days later (August 2026)¶

Anthropic then reported that Claude models had reached the systems of three real companies during cybersecurity evaluations, with the earliest case dating to April. Anthropic attributed this to an operational failure: a sandbox configuration error inadvertently gave the models internet access during capture-the-flag exercises that deliberately ran without safeguards. In one case a model attacked a real company that happened to share a name with its fictional target, and took several hundred rows of production data. In another it uploaded malware to the Python package registry, which then stole credentials from a security firm that installed it.

(Sources: CBC News; NPR.)

The takeaways¶

The mechanism behind these incidents is the same one described above: a goal-directed system encountered an obstacle and routed around it, and the route happened to lead outside the boundary. The environment-confusion case is worth dwelling on, because it is the failure mode students are most likely to reproduce in miniature. A model that cannot reliably distinguish a fictional target from a real company with a similar name is the same kind of system that cannot reliably distinguish your staging database from your production one.

4. Mitigation strategies¶

4.1 Isolate the environment¶

Run agents in a container or dedicated VM, not on the laptop that holds your SSH keys, cloud sessions, and other clients’ code.

• Mount only the project directory. Not your home directory.

• Restrict outbound network. Docker: --network none, or a domain allowlist if the task needs network.

• Keep cloud CLI sessions, SSH keys, and password manager integrations out.

• Prefer OS-level sandboxing where available. Several agents now ship a sandbox command built on Seatbelt (macOS) or bubblewrap (Linux/WSL2). These matter because subprocesses inherit the restriction. Application-level allowlists can inspect a tool call before it runs, but once execution passes to a subprocess the application has no further visibility.

Verify the isolation rather than assuming it. From inside the container, try to reach the network and try to read a path outside the mount. If either works, you do not have what you think you have.

4.2 Scope credentials and tokens¶

Agents increasingly reach external services through MCP servers or similar connectors: source control, databases, issue trackers, cloud APIs. The token you configure grants the agent exactly what the token holder has, so the scope of that token is the real permission boundary, not anything in the agent’s configuration. Prisma’s guidance is worth following closely here:

• Default to read-only tokens. Most agent work does not need write access.

• Where write access is needed, issue short-lived tokens and revoke them when the task is done. Avoid permanent read-write credentials.

• Create dedicated service accounts with minimal scopes rather than reusing a personal admin token.

• For source control, prefer read scopes on repositories and issues. Avoid organization-admin and repository-deletion scopes entirely.

• For databases, give the agent a read-only connection string unless the task genuinely requires writes.

• For cloud APIs, use a scoped role with read-only policies rather than an admin or wildcard policy.

• Audit which connectors are attached and what scopes their tokens carry, at least at each project milestone.

4.3 Constrain the database¶

• Dedicated role with least privilege. Read-only where the work allows.

• Revoke DROP, TRUNCATE, and schema modification from the application role. Run migrations through a separate, deliberately invoked path.

• Enable row-level security on tenant and user tables from the first migration.

• Test your RLS adversarially. Change the session identity and try to read someone else’s rows. A pentester writing on r/cybersecurity described finding a dashboard where changing one URL parameter exposed every user’s data, and called it the most common route to a leak. An agent can write the policy. Someone has to try to defeat it.

• Statement timeouts and connection limits on the dev role. Cheap, and they cap runaway queries.

• Snapshot before any session involving migrations.

• Backups outside the reach of any credential you hold.

4.4 Handle secrets¶

• No secrets in the repo. Including commented-out code and test fixtures.

• .gitignore before the first commit, not after.

• Add secret paths to the agent’s deny list too: .env*, .ssh*, *.pem, *credentials*, *secret*. Second layer, not primary, and note it does not stop a subprocess the agent spawns.

• Prefer a managed secret store over plaintext files on disk. If the project already runs on a cloud platform, its key vault serves the same purpose: Azure Key Vault, AWS Secrets Manager, Google Secret Manager — with the application fetching secrets at startup instead of reading a .env file, and with access logging and rotation handled by the platform.

• Rotate everything at project end regardless of whether anything happened.

4.5 Vet extensions, skills, and MCP servers¶

Fastest-moving risk area, least covered by existing policy. These are third-party code running with your agent’s permissions, distributed through channels that are new and largely unmoderated.

In late 2025 a prompt injection was found in a skill on the front page of a popular agent’s skill library, and follow-up analysis found hundreds of malicious skills. A malicious MCP server package shipped fifteen clean releases before adding an exfiltration line, which is a deliberate strategy for accumulating trust before defecting.

Before installing: check the source organization, read the code, understand what access it requests and why, watch for typosquatted names, look at commit history.

Red flags: filesystem access beyond the project with no justification, network calls to unfamiliar domains, obfuscated or minified code with no readable source, and any request to disable a security feature or skip permission prompts.

Pin versions. Review changelogs before updating.

4.6 Do not expose control interfaces¶

If you run an agent or agent gateway on a cloud VM rather than a laptop, which several of you will:

• Bind to localhost. Enforce authentication even locally. Firewall rules, not obscurity.

• Agent control panels deployed without auth are routinely found by internet-wide scanners, and an exposed panel leaks API keys, tokens, and full session history.

• Reverse proxies deserve specific care. A misconfigured nginx, Caddy, or Traefik front end can make external connections appear to originate from localhost, which causes the gateway to auto-approve them without credentials. Configure the trusted-proxy list to contain only the actual proxy address. Never disable gateway auth on the assumption the proxy handles it.

4.7 Reviewing AI-generated code¶

You are the checkpoint. Act like it.

Look for these specifically. They are the signatures of both model error and successful injection:

• Outbound network calls the task did not require, especially POST to unfamiliar hosts

• Base64 strings or obfuscated segments

• eval, exec, Function, or dynamic execution

• Hardcoded URLs or IP addresses

• File operations reaching outside the project directory

• Dependencies added that nobody asked for

• Packages that do not exist, or exist but were published recently under a suspiciously close name

Watch for deleted or weakened tests. If a diff removes a test or suppresses a type check rather than fixing the underlying problem, the agent worked around an obstacle instead of solving it. This is a reliable signal and it is easy to miss when the diff is large.

Keep diffs small enough to actually review. A forty-file pull request will not be meaningfully reviewed by anyone, including you. If a session produced one, that is a reason to split it, not a reason to approve it.

Automated review is a layer, not a substitute. Enable static analysis (CodeQL, Semgrep), dependency scanning, and secret scanning. On top of those, both major agents now ship a dedicated security review that runs the model against your diff:

Claude Code

/security-review in your project directory runs an on-demand pass over the pending changes on your current branch. It checks for injection, XSS, authentication and authorization flaws, insecure data handling, and dependency issues, then explains each finding. You can ask it to fix what it found in the same session.

OpenAI Codex

CLI: npm install --global @openai/codex, then codex plugin add codex-security@openai-curated. Scan a repo with npx @openai/codex-security scan ., or use $codex-security:security-diff-scan in a conversation to review uncommitted changes.

Both move fast. Check the current docs rather than trusting this paragraph: https://code.claude.com/docs/en/security and https://developers.openai.com/codex/security

None of this replaces you reading the diff. These tools miss business logic flaws and missing authorization checks, which are exactly the failure modes generated code produces most, and an AI review of AI-generated code shares blind spots with the author. Treat a clean /security-review as one signal, not clearance.

If something goes wrong¶

Speed beats certainty. Report before you have finished diagnosing.

1. Revoke the credential immediately. Do not wait to confirm it was used.

2. Notify your manager or the instructor asap.

3. Preserve the session transcript, agent logs, and git history. These are how anyone works out what happened.

4. Do not rewrite git history to remove a committed secret before the credential is revoked. Rotation is the fix. History rewriting is cleanup, and doing it first just delays the fix.

Useful resources¶

Watch¶

• Video 1 — How to secure your AI Agents: A Technical Deep-dive, by Google for Developers

• Video 2 — How to vibe code securely (without getting hacked), by Google for Developers

Read first¶

• OWASP Agentic AI: Threats and Mitigations — https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/ Whitepaper from the OWASP Agentic Security Initiative, published February 2025, structured as a threat model rather than a checklist.

• Google’s Approach for Secure AI Agents — https://research.google/pubs/an-introduction-to-googles-approach-for-secure-ai-agents/

• OWASP AI Agent Security Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html Start with Key Risks and the Do’s and Don’ts at the end, then read Section 1 on tool least-privilege and Section 4 on human-in-the-loop controls. The bad/good MCP tool configuration examples are worth copying into your own setup. Section 9’s abuse-case test matrix is a ready-made test plan if you are building an agentic system.

Read next¶

• OWASP LLM Prompt Injection Prevention Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html

• OWASP Secure Coding with AI Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Secure_Coding_with_AI_Cheat_Sheet.html

• OWASP MCP Security Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/MCP_Security_Cheat_Sheet.html

• OWASP Secrets Management Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html

• OWASP GenAI Top 10 for LLM Applications — https://genai.owasp.org/llm-top-10/

• OWASP State of Agentic AI Security and Governance — https://genai.owasp.org/resource/state-of-agentic-ai-security-and-governance/

Incident reading¶

• Zenity’s analysis of the PocketOS deletion — https://zenity.io/blog/current-events/ai-agent-database-deletion-pocketos

• Live Science on the PocketOS deletion and the agent’s post-incident log — https://www.livescience.com/technology/artificial-intelligence/i-violated-every-principle-i-was-given-ai-agent-deletes-companys-entire-database-in-9-seconds-then-confesses

• Euronews on the PocketOS outage and recovery — https://www.euronews.com/next/2026/04/28/an-ai-agent-deleted-a-companys-entire-database-in-9-seconds-then-wrote-an-apology

• SaaStr on the Replit agent database deletion — https://www.saastr.com/replits-new-release-address-most-of-the-challenges-we-hit-vibe-coding-but-is-prosumer-vibe-coding-really-ready-for-commercial-apps-yet

• Wiz on the Nx s1ngularity attack — https://www.wiz.io/blog/s1ngularity-supply-chain-attack

• GitGuardian’s credential leak analysis for s1ngularity — https://blog.gitguardian.com/the-nx-s1ngularity-attack-inside-the-credential-leak/

• CBC on the OpenAI sandbox escape — https://www.cbc.ca/news/business/openai-test-hacks-other-model-9.7279188

• CNN Business on the sandbox escape and the models involved — https://www.cnn.com/2026/07/22/tech/openai-hugging-face-ai-cybersecurity

• CBC on Anthropic’s disclosure of model access to three companies — https://www.cbc.ca/news/business/anthropic-claude-hack-9.7291801

• NPR’s comparative account of both disclosures — https://www.npr.org/2026/08/01/nx-s1-5914852/anthropic-openai-models-hack-cybersecurity

• Vectra’s summary of prompt injection CVEs, including EchoLeak and the Copilot and Cursor cases — https://www.vectra.ai/topics/prompt-injection

Original article: https://www.linkedin.com/pulse/how-use-agentic-ai-coding-tools-responsibly-safely-student-nguyen-havoc/