# Copilot Cost Audit

Audit any repo for GitHub Copilot cost optimizations ahead of the June 1 usage-based billing switch.

## Output format

Returns a **markdown checklist** with a risk score at the top — paste directly into a GitHub issue or PR:

```md
## Copilot Cost Audit — ⚠️ MODERATE

> 1 high-priority · 2 medium-priority · 6 already good

- [ ] Trim `copilot-instructions.md` — currently 8,200 chars (~2,050 tokens), injected on every request
- [ ] Add build/test commands to `copilot-instructions.md`
- [x] Output controls present
- [ ] Replace hardcoded premium models: `src/api.js:14: model: "gpt-4"`
- [x] No MCP servers configured — no token overhead
```

**Risk score:** `🔴 HIGH` / `⚠️ MODERATE` / `🟡 LOW-MODERATE` / `🟢 LOW`

## What It Checks

1. **copilot-instructions.md presence + output controls** — Highest-ROI single fix. Flags missing file or missing output controls.
2. **Instructions file size** — Flags if over ~6,000 chars (~1,500 tokens). Injected on every request.
3. **Build/test commands documented** — Saves the agent from wasting turns discovering them.
4. **Duplicate instruction files** — Checks `AGENTS.md`, `CLAUDE.md`, `.cursorrules`. Overlapping files stack tokens.
5. **copilot-setup-steps.yml** — Flags if missing. Without it the coding agent reinstalls deps from scratch every session.
6. **VSCode hidden instructions** — Scans `.vscode/settings.json` for `codeGeneration.instructions` which inject hidden context on every chat turn.
7. **MCP servers** — Counts configured servers across `.mcp.json`, `.claude/mcp.json`, `.vscode/mcp.json`. Each costs ~100–500 tokens/step.
8. **Agent code + prompt caching** — Scans for `anthropic`/`openai` imports, flags missing `cache_control`.
9. **Hardcoded premium models** — Detects model assignments across JS, TS, Python, YAML, JSON, and `.env` files using assignment-aware patterns (not just bare string grep).

## Usage

This is a native **Copilot CLI extension**. Drop the file below into `.github/extensions/copilot-cost-audit/extension.mjs` in your repo, then ask Copilot CLI:

> "run the copilot cost audit"

## Installation

Create `.github/extensions/copilot-cost-audit/extension.mjs` with the code below. No extra dependencies — uses `@github/copilot-sdk` which is already available in Copilot CLI.

```js
import { joinSession } from "@github/copilot-sdk/extension";
import { readFileSync, existsSync } from "fs";
import { join } from "path";

const session = await joinSession({
  tools: [
    {
      name: "copilot_cost_audit",
      description:
        "Audit this repo for GitHub Copilot cost optimizations. Returns a markdown checklist with a risk score.",
      parameters: { type: "object", properties: {} },
      handler: async () => {
        const cwd = process.cwd();
        const { execSync } = await import("child_process");

        // Each item: { label, checked, severity: 'high'|'medium'|'ok' }
        const items = [];

        // --- 1. copilot-instructions.md presence + output controls ---
        const instrPath = join(cwd, ".github/copilot-instructions.md");
        if (!existsSync(instrPath)) {
          items.push({
            checked: false,
            severity: "high",
            label:
              "Create `.github/copilot-instructions.md` with output controls — highest-ROI single fix\n" +
              "  ```md\n" +
              "  ## Response Style\n" +
              "  - Code only, no explanation unless asked.\n" +
              "  - Bullets over paragraphs. No preamble.\n" +
              "  ```",
          });
        } else {
          const instr = readFileSync(instrPath, "utf-8");
          const hasOutputControls =
            /code only|bullets over|no preamble|no explanation/i.test(instr);

          items.push({
            checked: hasOutputControls,
            severity: hasOutputControls ? "ok" : "high",
            label: hasOutputControls
              ? "`copilot-instructions.md` has output controls"
              : "Add output controls to `.github/copilot-instructions.md`\n" +
                "  ```md\n" +
                "  ## Response Style\n" +
                "  - Code only, no explanation unless asked.\n" +
                "  - Bullets over paragraphs. No preamble.\n" +
                "  ```",
          });

          // --- 1b. File size ---
          const chars = instr.length;
          const tokens = Math.round(chars / 4);
          const tooBig = chars > 6000;
          items.push({
            checked: !tooBig,
            severity: tooBig ? "high" : "ok",
            label: tooBig
              ? `Trim \`copilot-instructions.md\` — currently ${chars} chars (~${tokens} tokens), injected on every request. Target: under 6,000 chars`
              : `\`copilot-instructions.md\` size OK — ${chars} chars (~${tokens} tokens)`,
          });

          // --- 1c. Build/test commands ---
          const hasBuildRef =
            /npm run|yarn|pnpm|pytest|go test|cargo test|mvn|gradle|make/i.test(instr);
          items.push({
            checked: hasBuildRef,
            severity: hasBuildRef ? "ok" : "medium",
            label: hasBuildRef
              ? "`copilot-instructions.md` documents build/test commands"
              : "Add build/test commands to `copilot-instructions.md` so the agent doesn't waste turns discovering them\n" +
                "  ```md\n" +
                "  ## Commands\n" +
                "  - Build: `npm run build`\n" +
                "  - Test: `npm test`\n" +
                "  ```",
          });
        }

        // --- 2. Duplicate instruction files ---
        const instrFiles = [
          { path: join(cwd, "AGENTS.md"), label: "AGENTS.md" },
          { path: join(cwd, "CLAUDE.md"), label: "CLAUDE.md" },
          { path: join(cwd, ".cursorrules"), label: ".cursorrules" },
          { path: join(cwd, ".github/copilot-instructions.md"), label: ".github/copilot-instructions.md" },
        ].filter((f) => existsSync(f.path));

        const tooManyInstr = instrFiles.length > 2;
        items.push({
          checked: !tooManyInstr,
          severity: tooManyInstr ? "medium" : "ok",
          label: tooManyInstr
            ? `Consolidate instruction files — ${instrFiles.length} found (${instrFiles.map((f) => f.label).join(", ")}). Overlapping files stack tokens across agent sessions`
            : `Instruction file count OK — ${instrFiles.map((f) => f.label).join(", ")}`,
        });

        // --- 3. copilot-setup-steps.yml ---
        const setupPath = join(cwd, ".github/workflows/copilot-setup-steps.yml");
        const hasSetup = existsSync(setupPath);
        items.push({
          checked: hasSetup,
          severity: hasSetup ? "ok" : "medium",
          label: hasSetup
            ? "`copilot-setup-steps.yml` present — agent sessions have cached dependencies"
            : "Add `.github/workflows/copilot-setup-steps.yml` — without it the coding agent reinstalls deps from scratch every session\n" +
              "  See: https://docs.github.com/en/copilot/customizing-copilot/customizing-the-development-environment-for-copilot-coding-agent",
        });

        // --- 4. VSCode hidden instructions ---
        const vscodePath = join(cwd, ".vscode/settings.json");
        if (existsSync(vscodePath)) {
          try {
            const vs = readFileSync(vscodePath, "utf-8");
            const hasHidden =
              /github\.copilot\.chat\.codeGeneration\.instructions/i.test(vs);
            items.push({
              checked: !hasHidden,
              severity: hasHidden ? "medium" : "ok",
              label: hasHidden
                ? "Review `github.copilot.chat.codeGeneration.instructions` in `.vscode/settings.json` — injects hidden context on every chat turn"
                : "`.vscode/settings.json` has no hidden per-file Copilot instructions",
            });
          } catch {
            items.push({ checked: false, severity: "medium", label: "`.vscode/settings.json` found but couldn't parse it" });
          }
        }

        // --- 5. MCP servers ---
        const mcpPaths = [
          join(cwd, ".mcp.json"),
          join(cwd, ".claude/mcp.json"),
          join(cwd, ".vscode/mcp.json"),
        ];
        let mcpCount = 0;
        for (const p of mcpPaths) {
          if (existsSync(p)) {
            try {
              const obj = JSON.parse(readFileSync(p, "utf-8"));
              const servers = obj.mcpServers ?? obj.servers ?? {};
              const count = Object.keys(servers).length;
              mcpCount += count;
              if (count > 0) {
                items.push({
                  checked: false,
                  severity: "medium",
                  label: `Review MCP servers in \`${p.replace(cwd + "/", "")}\` — ${count} configured (${Object.keys(servers).join(", ")}), each costs ~100–500 tokens/step`,
                });
              }
            } catch {
              items.push({ checked: false, severity: "medium", label: `\`${p.replace(cwd + "/", "")}\` found but couldn't parse it` });
            }
          }
        }
        if (mcpCount === 0) {
          items.push({ checked: true, severity: "ok", label: "No MCP servers configured — no token overhead" });
        }

        // --- 6. Agent code + prompt caching ---
        let agentFiles = [];
        try {
          const out = execSync(
            `grep -rl "anthropic\\|openai\\|tool_use\\|cache_control" --include="*.ts" --include="*.js" --include="*.mjs" ${cwd} 2>/dev/null || true`,
            { encoding: "utf-8" }
          ).trim();
          agentFiles = out ? out.split("\n").filter(Boolean) : [];
        } catch {}

        if (agentFiles.length === 0) {
          items.push({ checked: true, severity: "ok", label: "No agent/LLM API code found — no prompt caching overhead" });
        } else {
          for (const f of agentFiles) {
            const src = readFileSync(f, "utf-8");
            const hasCaching = /cache_control/.test(src);
            const label = f.replace(cwd + "/", "");
            items.push({
              checked: hasCaching,
              severity: hasCaching ? "ok" : "medium",
              label: hasCaching
                ? `\`${label}\`: prompt caching wired up`
                : `\`${label}\`: LLM calls found but no \`cache_control\` — add prompt caching to save on repeated context\n  See: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching`,
            });
          }
        }

        // --- 7. Hardcoded premium models (improved patterns for #9) ---
        // Matches JS/TS/Python/YAML model assignments, not just bare strings
        const modelRegex =
          /(?:model\s*[:=]\s*["'`]|"model"\s*:\s*"|model_name\s*=\s*["'])(claude-opus|claude-3-5|claude-3-opus|gpt-4[o\-\d]*|gpt-5[^\s"']*)/gi;

        let modelHits = [];
        try {
          // Broader file scan: also catch .yaml, .yml, .py, .json
          const out = execSync(
            `grep -rn "claude-opus\\|claude-3-5\\|claude-3-opus\\|gpt-4\\|gpt-5" ` +
              `--include="*.ts" --include="*.js" --include="*.mjs" ` +
              `--include="*.json" --include="*.yaml" --include="*.yml" --include="*.py" ` +
              `--include="*.env*" ` +
              `${cwd} 2>/dev/null || true`,
            { encoding: "utf-8" }
          ).trim();
          modelHits = out ? out.split("\n").filter(Boolean) : [];
        } catch {}

        if (modelHits.length === 0) {
          items.push({ checked: true, severity: "ok", label: "No hardcoded premium model names found" });
        } else {
          items.push({
            checked: false,
            severity: "high",
            label:
              "Replace hardcoded premium models with a cheaper model for non-critical tasks:\n" +
              modelHits.map((l) => "  - `" + l.replace(cwd + "/", "") + "`").join("\n"),
          });
        }

        // --- Scoring ---
        const highCount = items.filter((i) => i.severity === "high" && !i.checked).length;
        const medCount = items.filter((i) => i.severity === "medium" && !i.checked).length;
        const risk =
          highCount >= 2 ? "🔴 HIGH"
          : highCount === 1 || medCount >= 3 ? "⚠️ MODERATE"
          : medCount >= 1 ? "🟡 LOW-MODERATE"
          : "🟢 LOW";

        // --- Render markdown checklist ---
        const checklist = items
          .map((i) => `- [${i.checked ? "x" : " "}] ${i.label}`)
          .join("\n");

        return (
          `## Copilot Cost Audit — ${risk}\n\n` +
          `> ${highCount} high-priority · ${medCount} medium-priority · ${items.filter((i) => i.checked).length} already good\n\n` +
          checklist
        );
      },
    },
  ],
});
```