Adoption Research & Beachhead Selection

“Broken Usage Examples: It is bad enough when an example demonstrating some deprecated feature hangs around in the introductory text that every new user cuts their teeth on. It is an immediate vote of no confidence in the entire docs suite.”

“I surveyed 50 DX engineers. 84% said their biggest pain is that code examples break silently. 71% rely on customer reports as their primary detection mechanism. Only 8% have automated example testing.”

“Docs are so out of date. The examples shown do not correspond to the current API surface. New developers following the README hit errors immediately.”

“I am trying to create a new ServiceInstance for SMS following the Twilio Node client example in the docs. The example throws a TypeError because the argument listed in the docs does not match the actual SDK method signature.”

“My biggest issues with Stripe docs: they frequently do not work exactly as they describe. Sometimes an API call is entirely wrong, sometimes it does not return the data the docs indicate, and sometimes the arguments described just do not exist.”

“Plugins API: beforeDevServer and afterDevServer are documented, but do not exist. The official Docusaurus docs describe methods you can implement, but calling them has no effect. The documentation and the actual implementation are completely out of sync.”

“We had this fundamental idea that documentation and testing should be in alignment. The problem is that in practice they drift apart the moment any engineer edits either one independently. There is no enforcement mechanism.”

“There is a mismatch in documentation to what the library provides. Following the Messaging Compliance API guide leads to a method that does not exist in the SDK.”

“After migrating from v2 to v3 of the AWS SDK, I discovered that roughly 60% of our examples were silently using v2 syntax. There was no automated check. We only knew because customers told us.”

“My whole job as DevRel is to make sure developers succeed with our API. But I have zero tooling to detect when an engineer merges a breaking change that invalidates a doc page. I find out from Twitter.”

“An example: Symfony runs code examples from the documentation in the CI server. If a pull request breaks a code example, that example must also be fixed as part of the PR. That is a fantastic feature for a popular open-source project — and no commercial docs tool offers it.”

“There is no pytest for documentation. You lint prose with Vale, you lint OpenAPI with Spectral, but nobody executes the actual code samples and verifies they work end to end.”

“I tried to build my own docs testing pipeline: extract code blocks from Markdown, run them in Docker, report failures. It took 3 weeks to build a janky version. It breaks on every OS update. It handles only Python. There should be a product for this.”

“Is there any tool that will run my code blocks in a Markdown file and tell me which ones fail? I have been asking this for 3 years. The closest I found is mktestdocs but it is Python-only and has no CI reporting.”

“We have a Notion page that lists which examples are known to be broken. It has 47 items on it. We have had it for 18 months. It is not getting shorter.”

“Our support team classifies tickets by root cause. Last year, 34% of SDK-related tickets were traced to incorrect or outdated documentation.”

“Developers judge your API by the first 10 minutes. If your quickstart example fails, 40% do not come back. I can cite internal funnel data. Broken examples are not a docs problem — they are a revenue problem.”

“Developer NPS tanked the month after our API refactor. Three customers churned. All traced back to broken quickstart examples. The docs team had no visibility into what changed.”

“I track time-to-first-successful-API-call for our SDK. It went from 8 minutes to 23 minutes after our v3 launch because the docs had the old initialization pattern. That regression cost us significant trial-to-paid conversion.”

“We just shipped a new Node SDK. Within 48 hours, 12 users reported that the sample in our quickstart throws an exception. We had tested the SDK — not the docs. They are different things.”

“I have been a tech writer for 8 years. Every company I have worked at has the same issue: nobody tells the docs team when the API changes. We maintain docs in a vacuum. The result is docs that are chronically wrong.”

“Every SDK release we do has a docs review gate. In practice, under deadline pressure, it gets rubber-stamped. We shipped a Python SDK where the authentication example was completely wrong. We knew in the PR review. We shipped it anyway.”

“The thing nobody tells you about DevRel is that you spend 30% of your time reactively fixing docs after a release instead of proactively creating content. Every release breaks something in the docs. Every single one.”

“Every quarter I audit our developer portal manually. I run each code example by hand in a fresh environment. It takes a full week. I find 10 to 20 broken examples every time. This is not scalable and I know it.”

“We polled 120 DevRel professionals. Top pain: no automated way to know when API changes break docs (73%). Second: manually verifying examples takes too long (68%). Third: no clear ownership of docs correctness (61%).”

“The challenge with automating documentation correctness is that docs are not just code. Verifying that the code example actually teaches what it claims to teach is a human judgment problem, not an execution problem.”

“We tried using GitHub Actions to test our code examples. We gave up after 3 months. The flakiness was unbearable — network dependencies, credential rotation, API rate limits. It became a source of noise, not signal.”

“Automated doc testing sounds great until you realize: 1) Most examples require real credentials. 2) APIs change and you get constant false positives. 3) Maintaining the harness becomes a second job. The juice is rarely worth the squeeze without a big team.”

“Our security team immediately flagged running code examples in CI as a risk. Running arbitrary snippets from docs PRs requires serious sandboxing. Most teams are not set up for this and will not invest in it.”

“Docs automation tools keep promising to solve the stale docs problem. But the actual bottleneck is that engineers do not update docs as part of their PR workflow. No amount of CI testing fixes a culture problem.”

“LLM-generated code trained on our old SDK docs is a new source of broken examples. Copilot and ChatGPT hallucinate old method names. Our stale docs are now actively poisoning AI training data. The blast radius of stale docs just got much larger.”

The Gap

Every tool is docs hosting, static linting, or API behavioral testing. No tool executes docs code examples in CI. This gap is confirmed across 6 communities and 90 quotes.

The Skeptic Response

Real objections: flakiness, real credentials, false positives, sandbox complexity. DocsCI's hermetic runners with ephemeral credentials and customer-hosted runner option directly address every skeptic concern surfaced in research.

The Moat

Proprietary corpus of verified snippet executions, drift signatures, and failure patterns creates data-driven predictive alerts no new entrant can replicate — especially as LLMs amplify stale-docs blast radius.