Run the Test Suite in the Sandbox

Static analysis is helpful, but if tests are red and we ignore them, that is code review cosplay.

The whole reason Sandboxes exist is so we can actually execute the code, not just read it. The model is allowed to be wrong, and the tests are the ground truth for at least one definition of "works." So let's run them.

Outcome

Extend the lifecycle to run pnpm test inside the Sandbox after the file collection step, and return the test command's exit code, stdout, and stderr.

Fast Track

After file collection, run pnpm install with cwd: 'repo' so dependencies are present.
Run pnpm test with the same working directory and capture the result.
Return testResult: { exitCode, stdout, stderr } from the lifecycle.

Hands-on exercise

Open src/sandbox-lifecycle.ts and add the test step:

import { Sandbox } from '@vercel/sandbox';
 
const INTERESTING_PATHS = [
  'repo/package.json',
  'repo/src/index.ts',
  'repo/src/app.ts',
  'repo/lib/auth.ts'
];
 
export type TestResult = {
  exitCode: number;
  stdout: string;
  stderr: string;
};
 
export type LifecycleResult = {
  sandboxName: string;
  cloneExitCode: number;
  files: Array<{ path: string; content: string }>;
  testResult: TestResult;
};
 
export async function runSandboxLifecycle(repoUrl: string): Promise<LifecycleResult> {
  const sandbox = await Sandbox.create({ persistent: false, timeout: 10 * 60 * 1000 });
 
  try {
    const clone = await sandbox.runCommand('git', ['clone', '--depth', '1', repoUrl, 'repo']);
    if (clone.exitCode !== 0) {
      throw new Error(`Clone failed: ${await clone.stderr()}`);
    }
 
    const files: Array<{ path: string; content: string }> = [];
    for (const fullPath of INTERESTING_PATHS) {
      const content = await sandbox.readFileToBuffer({ path: fullPath });
      if (content) {
        files.push({
          path: fullPath.replace(/^repo\//, ''),
          content: content.toString('utf8')
        });
      }
    }
 
    const install = await sandbox.runCommand({ cmd: 'pnpm', args: ['install'], cwd: 'repo' });
    if (install.exitCode !== 0) {
      throw new Error(`Install failed: ${await install.stderr()}`);
    }
    const test = await sandbox.runCommand({ cmd: 'pnpm', args: ['test'], cwd: 'repo' });
 
    return {
      sandboxName: sandbox.name,
      cloneExitCode: clone.exitCode,
      files,
      testResult: {
        exitCode: test.exitCode,
        stdout: await test.stdout(),
        stderr: await test.stderr()
      }
    };
  } finally {
    await sandbox.stop();
  }
}

A couple of intentional choices.

We're not throwing when pnpm test exits non-zero. Failing tests are a finding, not a crash. The whole point of running them is that the failures become part of the review.

We're also not parsing anything yet. Parsing is the next lesson. Right now we just need to prove the tests ran and we captured the output.

Troubleshooting: pnpm is not available

Some Sandbox base images don't have pnpm. If pnpm install returns "command not found", we'll handle that in 4.3 when we detect the lockfile and pick the matching tool. For now, pick a repo that uses pnpm.

Troubleshooting: install takes forever

pnpm install on a real repo can take 30-90 seconds. That's normal for a cold Sandbox. We'll speed it up with snapshots in Chapter 5.

Untrusted scripts can access the network by default

Package installation and tests execute repository-controlled scripts. Sandbox isolates them from your machine, but its default network policy allows internet access. A production reviewer should allow only the registries needed for installation, then deny or tightly restrict egress before running untrusted tests.

Try It

Pick a repo that uses pnpm and has a test script (a small one — install time matters):

pnpm review https://github.com/<a-pnpm-repo-with-tests>

Expected output (the trailing test summary varies):

Reviewing https://github.com/<...>...
Sandbox: repo-review-...
Collected 2 file(s) for analysis.
Overall risk: low
Findings: 3
  [medium] ...
  ...

You won't see the test output in the terminal yet because the CLI isn't logging it. To verify it's there, temporarily add console.log(lifecycle.testResult) in the CLI right after the lifecycle call. You should see something like:

{
  exitCode: 0,
  stdout: '> repo@1.0.0 test\n> vitest run\n\n  ✓ src/sum.test.ts (3)\n...',
  stderr: ''
}

Or if tests failed:

{
  exitCode: 1,
  stdout: '... FAIL  src/sum.test.ts > adds correctly ...',
  stderr: ''
}

Either way, the suite ran and we captured the result. Remove the debug log before committing.

Commit

git add src/sandbox-lifecycle.ts
git commit -m "feat(sandbox): run repo test suite and capture the result"

Done-When

pnpm install runs in the Sandbox before tests
pnpm test runs after install
testResult: { exitCode, stdout, stderr } is returned from the lifecycle
Non-zero test exit codes don't crash the lifecycle

Solution

src/sandbox-lifecycle.ts

import { Sandbox } from '@vercel/sandbox';
 
const INTERESTING_PATHS = [
  'repo/package.json',
  'repo/src/index.ts',
  'repo/src/app.ts',
  'repo/lib/auth.ts'
];
 
export type TestResult = {
  exitCode: number;
  stdout: string;
  stderr: string;
};
 
export type LifecycleResult = {
  sandboxName: string;
  cloneExitCode: number;
  files: Array<{ path: string; content: string }>;
  testResult: TestResult;
};
 
export async function runSandboxLifecycle(repoUrl: string): Promise<LifecycleResult> {
  const sandbox = await Sandbox.create({ persistent: false, timeout: 10 * 60 * 1000 });
 
  try {
    const clone = await sandbox.runCommand('git', ['clone', '--depth', '1', repoUrl, 'repo']);
    if (clone.exitCode !== 0) {
      throw new Error(`Clone failed: ${await clone.stderr()}`);
    }
 
    const files: Array<{ path: string; content: string }> = [];
    for (const fullPath of INTERESTING_PATHS) {
      const content = await sandbox.readFileToBuffer({ path: fullPath });
      if (content) {
        files.push({
          path: fullPath.replace(/^repo\//, ''),
          content: content.toString('utf8')
        });
      }
    }
 
    const install = await sandbox.runCommand({ cmd: 'pnpm', args: ['install'], cwd: 'repo' });
    if (install.exitCode !== 0) {
      throw new Error(`Install failed: ${await install.stderr()}`);
    }
    const test = await sandbox.runCommand({ cmd: 'pnpm', args: ['test'], cwd: 'repo' });
 
    return {
      sandboxName: sandbox.name,
      cloneExitCode: clone.exitCode,
      files,
      testResult: {
        exitCode: test.exitCode,
        stdout: await test.stdout(),
        stderr: await test.stderr()
      }
    };
  } finally {
    await sandbox.stop();
  }
}