Why Do Unindexed Agent Skills Cause Token Context Overhead?
Unindexed agent skills cause token context overhead because most coding assistants default to eager preamble injection whenever instructions lack a lightweight discovery index. Every rule inside an AGENTS.md file, a GEMINI.md file, or an alwaysApply: true Cursor rule consumes input tokens on every single turn of a session, even when the current task never touches that workflow.
When developers start using AI coding agents, putting three formatting bullets into a root file feels free. Over six months, however, a team accumulates deployment runbooks, database migration checklists, API error conventions, PR review rubrics, and framework workarounds. According to the official AGENTS.md specification, the open format now guides coding agents in "over 60k open-source projects"—yet plain markdown files carry no metadata schema for conditional loading.
Because multi-turn coding sessions re-send the system preamble with every tool call and follow-up message, eager instructions compound quadratically across long debugging loops. A static 9,000-token ruleset in Cursor or OpenAI Codex CLI does not cost 9,000 tokens once; across a 12-turn refactoring run, it burns 108,000 input tokens before the model reads a single line of repository source code. Understanding this distinction between eager rule injection and progressive disclosure is why separating baseline invariants from modular workflows is critical in any skills vs system prompts architecture.
The Context Drift Moment: When 40 Rules Quietly Break Reasoning
A staff engineer maintains 42 custom workflows split across .cursor/rules/*.mdc, a monorepo AGENTS.md, and a personal ~/.claude/skills folder across two laptops. Half the Cursor rules carry alwaysApply: true or catch-all **/* globs left over from earlier sprints, totaling roughly 11,400 tokens of persistent startup text, while the nested microservice AGENTS.md silently crosses 34 KiB after a teammate appends GraphQL error guidelines. During a high-stakes authentication refactor, Codex drops the nested file altogether past its byte limit, and Cursor hallucinates an older REST pattern because the relevant rule sits buried in the middle of an overloaded preamble—leaving the engineer wondering why an agent that worked flawlessly with five rules suddenly ignores explicit repository constraints at 40.
How Much Context Do Cursor, Codex, Antigravity, Gemini CLI, and Claude Code Spend at Startup?
Cursor, OpenAI Codex CLI, Google Antigravity, Gemini CLI, and Claude Code use fundamentally different loading pipelines for rules and skills at session startup. Tools that concatenate raw markdown files front-load full instruction payloads immediately, whereas progressive-disclosure runtimes index short headers first and defer full bodies until triggered.
In official Cursor Rules documentation, the loading behavior is explicit: "When applied, rule contents are included at the start of the model context." If a rule in .cursor/rules/*.mdc sets alwaysApply: true, Cursor ignores both its globs and description fields and injects the entire file into every chat and Composer turn. By contrast, OpenAI Codex AGENTS.md documentation notes that "Codex skips empty files and stops adding files once the combined size reaches the limit defined by project_doc_max_bytes (32 KiB by default)."
| Agent Platform | Startup Loading Behavior | Documented Budget / Cap | Primary Overhead Failure Mode |
|---|---|---|---|
| Cursor IDE (.cursor/rules/*.mdc) | Injects every rule with alwaysApply: true plus matching globs at the start of model context. | No hard byte cap; attention & prompt window degrade past ~8k–15k tokens. | Overly broad globs ("**/*") trigger irrelevant rules and push code context out. |
| OpenAI Codex CLI (AGENTS.md) | Concatenates global ~/.codex/AGENTS.md and nested repo AGENTS.md files once per session. | 32 KiB (32,768 bytes) combined limit via project_doc_max_bytes default. | Stops appending nested AGENTS.md files once 32 KiB is reached. |
| Google Antigravity & Gemini CLI (GEMINI.md) | Loads workspace and global instruction files plus active skill metadata into session context. | Bounded by active workspace skill indexing and model context policy. | Large persistent GEMINI.md blocks dilute attention on multi-file edits. |
| Claude Code (SKILL.md + CLAUDE.md) | Loads CLAUDE.md eagerly; indexes SKILL.md names & YAML descriptions lazily. | 1% context window listing budget; compaction keeps 5k tokens/skill (25k total). | Drops descriptions for least-used skills if the 1% listing budget overflows. |
Notice the architectural asymmetry: a 3,104-byte SKILL.md file (roughly 776 tokens total) costs only ~25–50 tokens at startup in Claude Code because only its YAML name and description enter the index. That same 3,104-byte checklist stored as an always-applied Cursor rule or pasted into AGENTS.md or GEMINI.md burns all ~776 tokens on every single turn. To standardize which directives belong in persistent root files versus scoped rules, follow a three-tier multi-agent rules architecture.
Auditing Startup Token Overhead: A Multi-Agent POSIX Measurement Script
You can measure your exact startup token tax across Cursor, OpenAI Codex CLI, Google Antigravity, Gemini CLI, and Claude Code in under two seconds using a POSIX shell audit that separates eager preamble bytes from deferred SKILL.md bodies.
In benchmark tests executed against a representative multi-agent fixture, an eager 3,656-byte alwaysApply: true Cursor rule (ts-strict.mdc) consumed ~914 tokens per turn and a 2,823-byte AGENTS.md consumed ~706 tokens per turn, whereas a 3,104-byte SKILL.md (~776 total tokens) loaded only its frontmatter header at startup. The POSIX script below inspects your live repository and home directories, separates YAML frontmatter from skill bodies, estimates token counts at ~4 bytes per token, and alerts you whenever combined AGENTS.md payloads approach OpenAI Codex's 32,768-byte ceiling:
#!/bin/sh
# audit-skill-tokens.sh — Measure eager preamble vs deferred skill bytes across agents
set -eu
CODEX_MAX_BYTES=32768
eager_bytes=0
agents_bytes=0
skill_index_bytes=0
skill_deferred_bytes=0
echo "=== 1. Eager Context Files (Cursor alwaysApply, AGENTS.md, GEMINI.md, CLAUDE.md) ==="
for f in AGENTS.md ~/.codex/AGENTS.md GEMINI.md ~/.gemini/GEMINI.md CLAUDE.md ~/.claude/CLAUDE.md; do
[ -f "$f" ] || continue
b=$(wc -c < "$f" | tr -d ' ')
t=$(( (b + 3) / 4 ))
eager_bytes=$((eager_bytes + b))
case "$f" in *AGENTS.md) agents_bytes=$((agents_bytes + b)) ;; esac
printf " [EAGER] %-40s %6d bytes (~%5d tokens)
" "$f" "$b" "$t"
done
for f in .cursor/rules/*.mdc; do
[ -f "$f" ] || continue
b=$(wc -c < "$f" | tr -d ' ')
t=$(( (b + 3) / 4 ))
if grep -qE '^alwaysApply:[[:space:]]*true' "$f"; then
eager_bytes=$((eager_bytes + b))
printf " [ALWAYS] %-40s %6d bytes (~%5d tokens)
" "$f" "$b" "$t"
else
printf " [SCOPED] %-40s %6d bytes (~%5d tokens when matched)
" "$f" "$b" "$t"
fi
done
echo ""
echo "=== 2. Progressive Skills (SKILL.md frontmatter vs deferred body) ==="
for f in .claude/skills/*/SKILL.md ~/.claude/skills/*/SKILL.md ~/.codex/skills/*/SKILL.md; do
[ -f "$f" ] || continue
total=$(wc -c < "$f" | tr -d ' ')
front=$(awk 'BEGIN{c=0} /^---$/{c++; print; next} c==1{print} c>=2{exit}' "$f" | wc -c | tr -d ' ')
body=$((total - front))
skill_index_bytes=$((skill_index_bytes + front))
skill_deferred_bytes=$((skill_deferred_bytes + body))
printf " [SKILL] %-40s index: %4d B (~%3d tok) | body: %5d B (~%4d tok)
" "$f" "$front" "$(( (front + 3) / 4 ))" "$body" "$(( (body + 3) / 4 ))"
done
echo ""
echo "=== Summary ==="
printf " Persistent startup overhead: %d bytes (~%d tokens/turn)
" "$((eager_bytes + skill_index_bytes))" "$(( (eager_bytes + skill_index_bytes + 3) / 4 ))"
printf " Deferred skill payload saved: %d bytes (~%d tokens/turn)
" "$skill_deferred_bytes" "$(( (skill_deferred_bytes + 3) / 4 ))"
if [ "$agents_bytes" -gt "$CODEX_MAX_BYTES" ]; then
printf " WARNING: Combined AGENTS.md (%d bytes) exceeds Codex 32 KiB (%d B) limit!
" "$agents_bytes" "$CODEX_MAX_BYTES"
fiOnce you identify oversized rules or duplicate workflows across subdirectories, pair this measurement with our guide on auditing and pruning obsolete AI agent skills to retire dead files safely before converting active rules.
What Happens When Agent Skill Listings Exceed Runtime Context Budgets?
When agent skill listings or static preambles exceed runtime context budgets, coding assistants do not raise obvious compilation errors—instead, they silently truncate trailing instructions, strip semantic trigger descriptions, or evict invoked skills during auto-compaction.
In official Claude Code Skills documentation, Anthropic explicitly documents this silent degradation: "When the listing overflows, Claude Code drops descriptions starting with the skills you invoke least, so the skills you use most keep their full text." That skill-listing budget scales at 1% of the model's context window. Even more critically, during long sessions when auto-compaction summarizes history, Claude Code re-attaches only the first 5,000 tokens of each invoked skill within a combined 25,000-token compaction cap—meaning oversized skills lose their trailing output contracts mid-session.
Across non-Claude tools, overflow failures are equally stealthy:
- OpenAI Codex CLI silent cutoff: Once concatenated global and nested
AGENTS.mdfiles crossproject_doc_max_bytes(32 KiB by default), deeper repository rules are skipped entirely without halting execution. - Claude Code description stripping: Running
/doctoror/skill-doctorinside Claude Code exposes when an unpruned library overflows the 1% listing budget and strips keywords needed for automatic skill matching. - Cursor attention dilution: Packing 10,000+ tokens into
alwaysApply: truerules inside.cursor/rules/*.mdcforces critical constraints into the middle of the prompt window, increasing hallucinated APIs and ignored formatting checks.
Before and After: Converting Eager Monolithic Rules Into Lazy-Loaded Skills
Converting an eager, always-injected rule file into a scoped Cursor rule or a progressive SKILL.md package eliminates 90%+ of its per-turn startup footprint while preserving full behavioral fidelity when the task actually needs it.
Look at the difference between a bloated alwaysApply: true rule (loaded on every turn, including simple CSS tweaks) and a split progressive structure:
# BEFORE: .cursor/rules/backend-standards.mdc (Eager — ~2,400 tokens burned EVERY turn)
---
description: Backend standards, Prisma migrations, JWT auth, and Stripe webhook verification
alwaysApply: true
---
[...180 lines of database migration commands, SQL index checklists, and Stripe webhook HMAC verification...]
# -----------------------------------------------------------------------
# AFTER (Option A — Cursor Scoped): .cursor/rules/prisma-migrations.mdc (0 tokens until .sql/.prisma touched)
---
description: Enforce safe PostgreSQL migrations and index checks on schema edits
globs: ["prisma/**/*.prisma", "migrations/**/*.sql"]
alwaysApply: false
---
[...Focused 35-line migration checklist...]
# AFTER (Option B — Cross-Agent Progressive Skill): ~/.claude/skills/stripe-webhook-audit/SKILL.md (~38 startup tokens)
---
name: stripe-webhook-audit
description: Verify Stripe webhook HMAC signature validation, idempotency keys, and replay protection. Use when editing payment webhook handlers.
---
## Verification Steps
1. Confirm raw request body buffer is passed to constructEvent().
2. Reject requests missing valid Stripe-Signature headers.
3. Verify idempotency key persistence before mutating subscription state.In the "Before" state, a 20-turn session across unrelated frontend components wastes ~48,000 input tokens repeating backend webhook and SQL migration instructions. In the "After" state, Cursor incurs 0 tokens until a .prisma or .sql file enters context, and Claude Code or Codex spends ~38 tokens indexing the skill's YAML frontmatter until a payment handler is actually audited. If you maintain multiple coding assistants simultaneously, see how to structure one reusable cross-agent skill library across Cursor, Codex, Google Antigravity, and Claude Code.
How Do You Eliminate Token Overhead Across Multiple Machines and Agents?
Eliminating token overhead permanently across multiple laptops and AI tools requires separating universal repository invariants from on-demand skills and situational prompts, rather than copying giant markdown preambles into every workspace. Follow this 5-step optimization protocol:
- Measure startup preamble bytes across every agent file: Run
audit-skill-tokens.shacrossAGENTS.md,GEMINI.md,CLAUDE.md,.cursor/rules/*.mdc, andSKILL.mdfolders to establish your baseline per-turn tax. - Strip
alwaysApply: truefrom task-specific Cursor rules: Restrict always-applied Cursor rules to under 1,000 tokens of universal project truths and attach everything else to explicitglobsor manual@rulementions. - Split monolithic
AGENTS.mdfiles below Codex 32 KiB caps: Keep rootAGENTS.mdfiles under 1,500 tokens of build, test, and security invariants so Codex CLI never truncates nested overrides past 32,768 bytes. - Move procedural workflows into lazy-loaded
SKILL.mdpackages: Use concise, keyword-dense YAML descriptions under 250–400 characters and verify listing health with/doctorand/skill-doctorso your skill index stays inside the 1% context window budget. Once your catalog crosses 500+ skills, replace startup listings entirely with hybrid sqlite-vec and BM25 semantic skill search exposed on demand. - Retrieve ad-hoc prompts via Mac hotkey palette instead of context preambles: Keep situational prompts (incident postmortems, SQL explainers, regex builders) completely out of agent startup files and insert them only in the specific turn you need them—or review when to leave deterministic templates in editor snippets vs agent rules.
Prompttly is a skill manager for AI agents — one library for your skills and prompts that syncs into Claude Code, Codex, ChatGPT, and Claude and is one hotkey away on your Mac, so your setup follows you across every machine, repo, and tool.
Prompttly attacks token context overhead from two angles at once. First, its two-way Mac sync writes clean, properly scoped SKILL.md directories directly into ~/.claude/skills and ~/.codex/skills so coding agents use progressive disclosure instead of bloated monolithic files. Second, Prompttly's macOS global hotkey palette opens your entire library in under 200ms and inserts any prompt into Cursor, Antigravity, Ghostty, ChatGPT, or Claude on demand—meaning situational prompts consume zero startup context tokens until the exact moment you trigger them. You can generate lean, frontmatter-validated skills right now with the free Claude Skill Creator, or browse our full archive on the Prompttly Resources hub and learn how to insert prompts into any Mac app without copy-paste.
Frequently Asked Questions on Agent Skill Token Overhead
How many tokens do unindexed agent skills waste per session?
A typical developer setup with 12 to 20 eager-loaded Cursor rules, a sprawling AGENTS.md file, and unpruned root instructions injects 8,000 to 15,000 tokens before the first user message. Across a 10-turn debugging session, that persistent preamble repeats on every round trip and burns 80,000 to 150,000 input tokens.
What happens when AGENTS.md exceeds OpenAI Codex CLI limits?
According to official OpenAI Codex documentation, Codex reads AGENTS.md files sequentially and stops adding files once the combined size reaches project_doc_max_bytes, which defaults to 32 KiB (32,768 bytes, or roughly 8,192 tokens). Any nested directory instructions beyond that cap are silently dropped.
How does Claude Code limit skill context overhead at startup?
Claude Code uses progressive disclosure: at startup it loads only each skill name and YAML description within a character budget equal to 1% of the model context window. If descriptions overflow that 1% cap, Claude Code drops descriptions from the least-invoked skills and flags contributors in /doctor and /skill-doctor.
How can I keep a 100-prompt library fast without bloating agent context?
Keep startup files (AGENTS.md, GEMINI.md, and always-applied .cursor/rules) under 1,500 tokens for repo invariants, store procedural workflows in lazy-loaded SKILL.md modules, and use a skill manager like Prompttly with a global macOS hotkey palette to insert situational prompts on demand in under 200ms.
Related Prompt Resources
Keep Your Full Skill Library Ready Without Bloating Model Context
Stop cramming 40 workflows into AGENTS.md or always-on Cursor rules. Sync lean, progressive skills across Claude Code and Codex and pull any prompt into any app in under 200ms with Prompttly's Mac hotkey palette.