Hermetic Snippet Execution for Documentation

Why hermetic execution matters for docs

Running untrusted code from documentation is a different problem than running your own test suite. A docs snippet might import a library you don't have installed. It might contain an API key. It might make network requests to a live production endpoint. It might fork a process or write to disk. It might simply be broken — which is the whole point of running it.

The naive approach — running snippets in a Docker container with the right language installed — works until it doesn't. Shared state between runs causes flaky results. Network access to arbitrary hosts is a security problem. Long-running snippets block the queue. A single import os; os.system('rm -rf /') (hypothetically) could be catastrophic.

We built DocsCI's execution layer from first principles around three requirements:

1. Isolation: each snippet gets its own fresh environment with zero shared state from prior runs
2. Safety: credential scanning before execution; network allowlists enforced at the syscall level
3. Fidelity: execution results must reflect what the developer would see on their machine

JavaScript and TypeScript: V8 isolates

For JavaScript and TypeScript, we use V8 isolates — the same isolation primitive that powers Cloudflare Workers and Deno Deploy. Each isolate is a fully independent V8 heap with no shared memory, no shared globals, and no access to Node.js APIs unless explicitly injected.

The execution flow for a JavaScript snippet looks like this:

// Simplified execution pipeline (JavaScript)

// 1. Parse snippet — extract from Markdown fence block
const snippet = extractSnippet(markdownBlock);

// 2. Pre-execution security scan
const scanResult = await scanForSecrets(snippet);
if (scanResult.findings.length > 0) {
  return { status: "skipped", reason: "credentials_detected", findings: scanResult.findings };
}

// 3. Create fresh V8 isolate
const isolate = new ivm.Isolate({ memoryLimit: 64 }); // 64MB limit
const context = await isolate.createContext();

// 4. Inject allowed globals (console, fetch with allowlist)
await context.global.set("console", safeConsole);
await context.global.set("fetch", allowlistFetch(config.networkAllowlist));

// 5. Compile and run with timeout
const script = await isolate.compileScript(snippet, { filename: "snippet.js" });
const result = await script.run(context, { timeout: 20_000 }); // 20s timeout

// 6. Collect output and dispose
const output = capturedOutput.join("\n");
isolate.dispose();
return { status: "passed", output, duration_ms: Date.now() - start };

The key constraint: each isolate is disposed after execution. There's no persistent state between snippets. A snippet that sets a global variable on run N will not see that variable on run N+1. This is what makes results reproducible.

Python: Pyodide in WebAssembly

Python is harder than JavaScript. Python's standard execution model involves a native interpreter, filesystem access, subprocess spawning, and a C extension ecosystem that doesn't sandbox cleanly.

We use Pyodide — CPython compiled to WebAssembly via Emscripten. Pyodide runs inside a V8 isolate, which means it inherits all the isolation properties described above, plus the WASM security model: no direct filesystem access, no subprocess spawning, no native syscalls.

// Python execution via Pyodide WASM
const pyodide = await loadPyodide({ indexURL: PYODIDE_CDN });

// Install required packages (from pip annotations in snippet comments)
// # pip: requests,numpy
const packages = extractPipAnnotations(snippet);
if (packages.length > 0) {
  await pyodide.loadPackagesFromImports(snippet); // load from stdlib
  // micropip for PyPI packages (allowlisted)
  await pyodide.runPythonAsync(`
    import micropip
    await micropip.install([${packages.map(p => `"${p}"`).join(",")}])
  `);
}

// Redirect stdout/stderr
pyodide.runPython(`
  import sys, io
  sys.stdout = io.StringIO()
  sys.stderr = io.StringIO()
`);

// Execute
await pyodide.runPythonAsync(snippet);

// Capture output
const stdout = pyodide.runPython("sys.stdout.getvalue()");
const stderr = pyodide.runPython("sys.stderr.getvalue()");

The WASM sandbox isn't perfect — a malicious snippet can spin up a tight CPU loop. We enforce a 20-second timeout via the V8 isolate and a separate memory limit. Any snippet that exceeds either limit is terminated and reported as a timeout finding.

Bash and curl: simulated execution

Bash and curl examples can't be safely executed in WASM — they're inherently about interacting with the host system and network. We handle them differently:

Network isolation and allowlists

Network access is one of the hardest parts of docs execution. Many code examples make real HTTP requests — that's the whole point. But allowing arbitrary outbound network access from a shared execution environment is a serious risk.

DocsCI's network model works like this:

# docsci.yml — network allowlist configuration
snippets:
  # Hosts that snippets are allowed to contact
  network_allowlist:
    - "api.example.com"          # your staging API
    - "httpbin.org"              # common testing service
    - "*.github.com"             # GitHub API examples
    - "cdn.jsdelivr.net"         # CDN for WASM packages

  # Always blocked (regardless of allowlist)
  network_blocklist:
    - "169.254.169.254"          # AWS metadata
    - "10.0.0.0/8"               # Private ranges
    - "192.168.0.0/16"
    - "172.16.0.0/12"
    - "localhost"
    - "127.0.0.1"
    - "::1"

Private IP ranges and cloud metadata endpoints are always blocked, regardless of allowlist configuration. This prevents a class of SSRF-via-docs attacks where a malicious contributor could extract cloud credentials through documentation snippets.

Performance: the worker pool

A typical documentation repo has 50-300 code snippets. Running them serially would take minutes. DocsCI uses a worker pool where each worker manages its own set of pre-warmed isolates. A "warm" isolate has Pyodide loaded and the common stdlib imported — startup cost is paid once per worker per deploy, not once per snippet.

In practice, this means a 200-snippet documentation repo completes in 30-60 seconds, even with a mix of Python and JavaScript examples.

What this means for your docs

The practical result of hermetic execution is that DocsCI results are reliable and safe to run on every pull request. There's no "works on my machine" — either the snippet runs in the sandbox as documented, or it doesn't, and you get a precise finding with the error. No flakiness from shared state. No security incidents from credential-containing snippets. No queue blockage from long-running examples.