What is your coding agent loading?
Five questions to ask any agent security tool — including ours
The BAS Guard maintainers
Disclosure: We build one of the tools discussed below. We’ve tried to describe the alternatives as we’d want ours described, and we list our own gaps at the end.
In 2026, agent skills became a supply chain. Then they were breached.
- Over 1,100 malicious skills on ClawHub harvested crypto keys from the machines that installed them.
- The Mini Shai-Hulud worm wrote hooks into projects’ .claude/settings.json. Nothing was installed. A file already on disk gained a few new lines, and from then on the agent ran attacker code on every matching tool call.
- An MCP server sent WALLET_PRIVATE_KEY to its own host. It was doing exactly what its configuration said it would do. Nobody had read the configuration.
Three attacks. Three entirely different places: a marketplace skill, a project hook, a configured server.
That last point is worth sitting with. No single checkpoint would have caught all three. A scanner that inspects marketplace listings never sees a hook added to a local file. A tool that audits your installed environment once a week never sees a server that was configured yesterday.
So before you pick a tool, ask it five questions.
The five questions
1. Where and when does it look?
There are at least five distinct positions, and most tools occupy one or two:
| Position | Catches | Misses |
|---|---|---|
| Registry-side (marketplace scans its own listings) | Malicious published skills | Anything you install from outside, anything added locally |
| Before install (you point it at a package) | What you remember to check | Everything you forget |
| The installed environment (scan what's on disk) | Current state | Changes between scans |
| At load time (a gate fires when the agent loads something) | Drift, late edits | Behaviour after loading |
| At runtime (intercept what the agent actually does) | Live misbehaviour | Nothing — but it's the hardest to build |
The Mini Shai-Hulud case is instructive: a registry-side scanner and a before-install checker would both report clean, forever, because nothing was ever installed.
Ask: Which of these five does it cover, and what is it structurally blind to?
2. Does it execute what it scans?
Some MCP scanners start the server to read its tool descriptions. That is the most reliable way to learn what a server advertises. It is also the act of running untrusted code on your machine in order to find out whether it is safe to run on your machine.
For most of the day this is fine. For a corpus of known-malicious samples it is not fine at all.
Ask: Does it execute, and if so, in what isolation?
3. Does content leave your machine?
Cloud verdicts are the norm. So is upload-on-unknown: scan locally, and when the tool sees something it doesn’t recognise, send it up for analysis.
This is often the right trade-off — it’s how a central service learns. But it should be a decision you made, not a default you discovered. The files involved here are not ordinary source: they contain credential paths, internal hostnames, MCP endpoints, and the instruction files that describe how your team works.
Ask: What is sent, when, and can you see it before it goes?
4. Is the verdict reproducible?
Two failure modes, opposite in character:
- LLM verdicts vary between runs. The same file can come back clean on Monday and flagged on Tuesday. Useful as a second opinion; unusable as a gate, and unusable as evidence.
- Hash lookups break on one newline. Precise, and trivially evaded by anyone who knows you’re using one.
If a score is going to appear in a report, in a CI gate, or in an argument with a vendor, someone will eventually need to reproduce it.
Ask: Can I re-run this and get the same number? Can someone else?
5. What happens to content it never analysed?
This is the question almost nobody asks, and it is the one that produces the worst outcomes.
A tool that can’t parse a file has three options: report it as a gap, report nothing, or report it as clean. The third is the dangerous one, and it is common — not usually by design, but because a clean report and a silent skip look identical in a UI.
Cisco’s skill scanner, for instance, refuses to scan a directory tree with no SKILL.md. That’s a defensible engineering choice. But it means a quarter of a typical corpus is never examined, and if the output doesn’t say so, “no findings” and “no look” become the same sentence.
Ask: Show me the list of things it didn’t check, and why.
What’s available now
The free tier is crowded, which is good news.
| Tool | What it does well |
|---|---|
| ClawHub and skills.sh | Verdicts directly on listings. Zero effort — but only for marketplace browsing. |
| Cisco DefenseClaw and Snyk Agent Scan | Cover 10–14 agents between them. Best starting point if you run more than one coding agent. |
| Gen's Sage | Checks skills at session start — closer to load time than install time. |
Run more than one of these. They occupy different positions on the five-question grid, and the overlap is smaller than the marketing suggests.
The uncomfortable part: accuracy
Everyone’s numbers are worse than they look.
Independent testing bypassed most scanners within an hour. In one published evaluation, a flag-everything baseline beat eight open-source scanners on F1 — that is, a tool that simply marks every input as malicious scored better than eight real products.
Cisco deserves credit for publishing held-out recall:
- 7.7% with rules alone
- 66.7% with an LLM judge
Those are honest numbers from a serious team, and they are the best public figures in the category.
Read them carefully. 7.7% is the static-rules result — the deterministic, reproducible, offline part. The jump to 66.7% comes from a language model, which brings back question 4.
There is no tool in this category whose accuracy you should take on faith, including ours.
Where we stand
BAS Guard makes a specific set of trade-offs. They are not the only reasonable ones.
| Question | Our answer |
|---|---|
| Where / when does it look? | The installed environment, and at load time via a hook gate. Not registry-side, not runtime. |
| Does it execute? | Never. It reads; it does not run. |
| Does content leave? | No, by default — and a test in CI enforces zero network calls. The optional LLM second opinion is the one exception, and it's opt-in with a preview command. |
| Reproducible? | The 0–100 score is deterministic and can be recomputed offline. The optional LLM can only raise risk, never lower it, and never moves the deterministic score. |
| Unread content? | Reported as unread. Never assumed clean. |
Our limits, plainly:
- It covers Claude Code only. If you run Cursor, Codex, or anything else alongside it, we don’t see those. Start with a multi-agent scanner.
- We have no published accuracy numbers yet. We’ve made the score reproducible, which makes it checkable — but checkable is not the same as published, and until we publish, you should weigh our claims accordingly.
What to do regardless of what you install
Review .claude/settings.json like code. It runs.
Hooks in that file execute on matching tool calls, silently, with your shell and your credentials. Auto-approve rules remove the confirmation step that most people assume is still there. Neither of these is a vulnerability — they’re features, working as designed. They just happen to be the most consequential lines in your repository that nobody reads in review.
Put the file in your review checklist. Diff it when it changes. That costs nothing and catches the case that defeated every registry-side scanner in 2026.
We’d rather you pick the right tool than ours. If you find a gap in the five questions above — a position on the timeline we’ve missed, or a failure mode we haven’t named — tell us and we’ll update this post.