How to build an AI test planning agent in VS Code that reads your requirement and your codebase together — and generates test plans, test cases, and risk analysis from both.

On a Thursday afternoon, a team shipped a password reset feature. It passed code review twice, passed 47 automated tests, and passed a two-hour manual QA session by an engineer with eight years of experience.

Eleven days later, a security researcher emailed them. Using nothing but a password reset link, he had verified an email address he did not own.

What Is an AI Test Planning Agent?

An AI test planning agent is an AI assistant, configured through a Markdown instruction file inside your repository, that analyzes a requirement document against your live codebase and automatically generates a testing strategy — impact analysis, test plan, test cases, regression scope, and risk register. Unlike a generic AI test case generator, it cites real file paths from your project.

That last part is the entire difference. A chatbot given a user story writes test cases anyone could write. An agent with repository access writes test cases about your system.

Why Traditional Test Planning Misses Bugs

Every testing effort begins with four words: “What should we test?”

Most QA engineers are guessing. Educated guessing, informed by scar tissue and pattern recognition — but guessing. Here’s why.

A tester receives this:

PR #482 — Add password reset via email verification link.

And is expected to produce a test plan for a system with 340 source files, 12 database tables, 6 API middlewares, and three years of decisions they were not present for.

The tester doesn’t lack skill. They lack context. The requirement lives in Jira. The truth lives in the codebase. The connection between them lives in someone’s head — usually someone who left the company.

An agent that reads both will find things a human reading them separately cannot. Not because it’s smarter, but because it never gets tired of tracing a function call across nine files.

Related: How to write a regression test strategy · Risk-based testing explained

How to Set Up an AI Test Planning Agent in VS Code

Project structure

password-service/
│
├── src/
│   ├── components/
│   │   └── ResetPasswordForm.tsx
│   ├── controllers/
│   │   ├── authController.js
│   │   └── userController.js
│   ├── services/
│   │   ├── tokenService.js        ← remember this one
│   │   └── mailService.js
│   └── database/
│       └── migrations/
│
├── tests/
│   ├── e2e/
│   └── api/
│
└── test-agent.md                  ← the entire tooling

No platform, no license, no vendor. A Markdown file and a coding agent that can read your repository.

The test-agent.md instruction file

Most people write a three-line prompt and wonder why the output is generic. The quality of your agent is the quality of this file:

# AI Test Planning Agent

## Role
You are a Senior QA Engineer with a security-testing background.
You have full read access to this repository.

## Non-Negotiable Rules
1. Every claim about the system must cite a real file path and line range.
   Format: src/services/tokenService.js:41-58
2. If you cannot find something in the code, write
   "NOT FOUND IN CODEBASE" — never assume, never invent.
3. Trace data end to end. For every new input, follow it:
   UI → API → validation → service → database → response.
4. Before writing test cases, list what the change might BREAK.
   Regression thinking comes before new-feature thinking.
5. Flag any place where the requirement and the code disagree.

## Process
Step 1 — Parse the requirement into functional and non-functional needs.
Step 2 — Map the requirement onto real files. Produce an impact surface.
Step 3 — Identify shared resources: shared tables, shared utilities,
         shared middleware, shared token logic. These are where bugs hide.
Step 4 — Produce a risk register, ranked by severity × likelihood.
Step 5 — Only then generate test artifacts.

## Deliverables
1. Impact Analysis (with file citations)
2. Risk Register
3. Test Plan (objective, scope, exclusions, entry/exit criteria)
4. Test Scenarios and Detailed Test Cases
5. Regression Checklist mapped to existing test files
6. Automation Recommendations with runnable code
7. Open Questions for the developer

Two rules do most of the work. Rule 2 is your hallucination brake — without it, the agent invents plausible middleware that doesn’t exist. Rule 3, Step 3 is the one that found the bug.

How the AI Agent Analyzes Your Codebase

Step 1: Requirement parsing

Feature: Password Reset via Email

- User submits email address on /forgot-password
- System sends a reset link valid for 30 minutes
- Link opens a form to set a new password
- Password must be 8+ characters
- User is logged in after successful reset

Five bullet points. A human tester reads this and starts thinking about weak passwords, expired links, and invalid emails.

Step 2: Impact mapping

The agent doesn’t start with test cases. It follows the data.

Frontend
  src/components/ResetPasswordForm.tsx        NEW
  src/components/LoginForm.tsx:88             MODIFIED — "Forgot password?" link

Backend
  src/controllers/authController.js:120-190   NEW — requestReset, submitReset
  src/services/tokenService.js:41-58          REUSED  ⚠
  src/services/mailService.js:12              REUSED
  src/middleware/rateLimiter.js               NOT APPLIED TO NEW ROUTES  ⚠

Database
  tokens table        REUSED — no schema change  ⚠
  users.password_hash UPDATED
  sessions table      NOT TOUCHED  ⚠

Four warning flags. Three are on things the pull request didn’t change.

That is the whole trick. The diff shows you what moved. The bugs are in what didn’t.

What the AI Agent Found That Four Reviewers Missed

// src/services/tokenService.js:41
async function validateToken(rawToken) {
  const record = await db.tokens.findOne({ token: hash(rawToken) });
  if (!record) return null;
  if (record.expires_at < Date.now()) return null;
  return record;                      // ← purpose is never checked
}

Four engineers reviewed this pull request. None opened tokenService.js, because the diff didn’t change it.

Three More Risks the AI Test Agent Flagged

🟠 HIGH — Sessions survive the reset. The sessions table is never touched during reset (authController.js:184). If an attacker holds an active session, the victim resetting their password does not evict them. See OWASP’s session management guidance.

🟠 HIGH — User enumeration. /forgot-password returns 404 "No account found" for unknown emails and 200 "Check your inbox" for known ones (authController.js:131). Anyone can now confirm whether an email is registered. Fix: identical response and timing, always.

🟡 MEDIUM — No rate limit on the reset endpoint. rateLimiter.js is applied to /login (routes/auth.js:14) but not to the two new routes. Unlimited reset requests means unlimited outbound email — a spam vector aimed at your users, paid for by your SMTP bill, ending with your domain on a blocklist.

The AI-Generated Test Plan

Objective: Verify password reset end to end, with emphasis on token integrity and session invalidation.

In scope: Reset request, token lifecycle, password update, session handling, email delivery, rate limiting.

Out of scope: SMTP provider reliability, email client rendering.

Exit criteria: All 🔴 and 🟠 findings resolved and covered by automated tests; regression suite green.

AI-generated test cases

The first five you’d have written yourself. The last four are the reason this is worth doing.

Regression checklist mapped to real test files

tests/e2e/login.spec.ts          — token path changed, re-run
tests/e2e/signup.spec.ts         — shares validateToken(), re-run
tests/api/verify-email.spec.ts   — MUST re-run, directly affected  ⚠
tests/e2e/profile.spec.ts        — unaffected, skip

Not “run the regression suite.” Which files, and why.

How to Automate These Tests with Playwright

test('reset token cannot be redeemed as email verification', async ({ request }) => {
  const { token } = await requestPasswordReset('victim@example.com');

  const res = await request.post('/api/verify-email', {
    data: { token }
  });

  expect(res.status()).toBe(400);
  expect(await res.text()).toContain('Invalid token');
});

test('active sessions are terminated after password reset', async ({ browser }) => {
  const attacker = await loginAs('victim@example.com', 'oldPassword123');

  await completePasswordReset('victim@example.com', 'newPassword456');

  const res = await attacker.request.get('/api/me');
  expect(res.status()).toBe(401);
});

Full API testing syntax is in the Playwright documentation.

Limitations of AI Test Case Generation

If I only told you the wins, you’d try this, hit a wall, and quit. So:

1. It hallucinates without the citation rule. Drop Rule 2 and it will confidently describe middleware you never wrote. Treat uncited claims as fiction.
2. It struggles past a certain repo size. Large monorepos exceed what the agent can hold at once. Point it at a subtree, not the root.
3. It cannot know your business. It won’t know that enterprise customers use SSO and must never receive a reset email. Put domain knowledge in the agent file.
4. It over-generates. Left alone it hands you 80 test cases where 25 matter. Tell it to rank by risk and cap the output.
5. It is a first draft, not a verdict. Every finding above still needed a human to open the file and confirm. The agent’s value isn’t being right — it’s being fast at pointing.

How to Build Your Own AI Test Planning Agent in 5 Minutes

1. Create test-agent.md in your repository root and paste the instruction file above.
2. Open any coding agent with read access to your repository.
3. Paste your most recent pull request description underneath.
4. Run it.
5. Read only the section titled shared resources.

That last step is the whole game. Shared tables, shared utilities, shared middleware, shared token logic. Every finding in this post came from something two features had in common and neither one owned.

Will AI Replace QA Engineers?

No — but the job is changing, and pretending otherwise helps nobody.

The scarce skill used to be reading a large unfamiliar system quickly. That just got cheap. What remains is the work that was always the real work:

• Knowing which risks matter to this business
• Deciding what “good enough to ship” means
• Designing the questions worth asking
• Judging whether the agent is right

The tester who fought the codebase for two days to build a mental map was doing valuable work. The tester who gets that map in ninety seconds and spends those two days on risks nobody thought to name is doing something better.

Frequently Asked Questions

What is an AI test planning agent?

An AI test planning agent is an AI assistant configured by a Markdown instruction file in your repository. It analyzes a requirement document against your live source code and generates an impact analysis, test plan, test cases, regression scope, and risk register, citing real file paths from your project.

Can AI generate test cases from a requirement document?

Yes. Any capable model can produce test cases from a user story or SRS. The difference with a codebase-aware agent is that it also identifies which existing modules the change affects, which is where the highest-value test cases come from.

How accurate are AI-generated test cases?

Accurate enough to be a strong first draft, not accurate enough to ship unreviewed. Require file-path citations for every claim so you can verify findings in seconds. Expect roughly a third of the output to be redundant.

Which AI model works best for test planning?

Any model with repository read access and a large context window works. The instruction file matters more than the model choice — a well-specified agent on a mid-tier model outperforms a vague prompt on a frontier one.

Do I need a paid tool to build this?

No. The setup is one Markdown file plus a coding agent you likely already have. There is no separate platform, license, or vendor involved.

Does this replace manual QA?

No. It replaces the slowest part of manual QA — building a mental model of an unfamiliar system. Exploratory testing, business judgment, and final sign-off remain human work.

Have you tried this on your own codebase? I’d like to hear what it found — especially the things it got wrong. Leave a comment below.