Webshooter

Problem

Writing user documentation or training material for a web app normally means manually clicking through every page and interaction, taking screenshots, and assembling them into a document by hand. That process goes stale the moment the UI changes. There was no tool that could walk a whole site the way a real user would — including clicking and hovering elements, not just loading pages — and turn that into a ready-to-read documentation gallery without a person capturing every image manually.

The problem compounds across a portfolio of sites and internal apps instead of existing once. Every one of them eventually needs onboarding material, and re-doing this manual capture process per site, per redesign, is the kind of repeated cost that never shows up as a single line item but adds up across a whole stack of properties.

What was built

Webshooter crawls a target site page by page, and on each page interacts with every meaningful clickable and hoverable element. It captures a before screenshot (with the target element outlined in red) and an after screenshot for each interaction. It compiles the results into an HTML gallery with a filterable sidebar table of contents, full-page and viewport captures per page, before/after pairs grouped by interaction type, and a lightbox for full-size viewing, plus a print-optimized PDF version of the same report.

It ships two ways to run it: a one-dependency CLI (crawler.py + reporter.py, Playwright only) for local use, and a FastAPI/HTMX web app that wraps the same engine so a crawl can be started from a browser, watched live, and its output browsed or downloaded. It's deployed as a standing service on the owner's infrastructure behind its own subdomain and access control.

Technical approach

The crawler and reporter are deliberately kept as separable stages. crawler.py writes a manifest.json index (page metadata, interaction records, bounding boxes) alongside PNGs, and reporter.py reads that manifest independently to emit the HTML/PDF report. That split means a report can be rebuilt or redesigned from a past crawl without re-crawling the live site. PDF generation reuses Playwright itself instead of a separate PDF library — it points a headless browser at the already-generated HTML report with print-background enabled.

Several safety rails are hard constraints in the crawl loop. Traversal stays same-domain-only. A regex-based deny list skips interactive elements whose labels match destructive actions (logout, delete, remove, unsubscribe). Form fields get filled with obviously fake sample data that is never submitted. And a fresh page reload runs before every single interaction, so state never drifts or compounds across captures.

Authenticated crawling is opt-in per run. A session is captured once locally via a setup command that opens a real browser for a manual login, and the resulting session file is uploaded through the web app's Sessions page instead of credentials ever being typed into the app itself. The web app layer adds a SQLite metadata store for run/session history and a Keychain-backed session store, with an in-memory backend for tests, so captured auth sessions never touch disk unencrypted. It also adds an async job runner with progress reporting and timeout handling wired into the app's lifecycle.

It runs as a native launchd service on the host machine instead of in the containerized agent platform, specifically so Chromium never has to enter that platform's Docker image. It's exposed through its own dedicated tunnel isolated from the tunnel carrying other production services, with access gated by identity-based access control plus an app-level shared secret as fallback.

Creative approach

Craft

The report format treats documentation as a reading experience, not a raw file dump. A filterable table-of-contents sidebar groups interactions by type — clicks, hovers, forms — instead of dumping them chronologically, and a full-size lightbox sits on every image. CSS specifically tuned for print (sidebar hidden, deliberate page breaks between sections, backgrounds preserved) means the same artifact works as both a browsable web gallery and a printable manual, without a second export pass.

Every page capture ships two versions instead of one — a full-page scrolled shot and a viewport-only shot. A training reader needs both "here's the whole page" context and "here's what a real user actually sees on load" without scrolling; picking only one loses one of those two questions. The PDF path was deliberately built on top of the same HTML report instead of a separate PDF library. Playwright points its own headless browser at the already-rendered report with print backgrounds enabled, so the print output is guaranteed to match what a reader sees on screen instead of drifting into a second, separately maintained rendering path.

Reframe

The key idea is capturing "what happens when you click this" as a first-class artifact, not just "what does this page look like." Most site-screenshotting tools stop at full-page captures. This one drives real interactions — hover, click — and pairs a before shot with a red-outlined target element against an after shot. That is what actually makes the output usable as step-by-step training material, instead of a static image dump.

That same instinct — capture the artifact, not a description of it — extends to how the crawler and reporter are split. The crawler's only output contract is a manifest plus PNGs, and the reporter treats that manifest as its complete input. Redesigning the report format is therefore a reporter-only change, one that can be tested and iterated against a real past crawl without touching the site being documented again.

Process and what failed

The build moved from a CLI prototype into a properly designed web app with its own spec and TDD plan instead of growing organically. It picked up real infrastructure hardening along the way. A Keychain-backed session store got a swappable in-memory test backend. An isolated deploy tunnel was kept separate from the shared production tunnel carrying other live services. The service runs natively on the host, a deliberate decision made to avoid shipping a full Chromium install inside another platform's container image.

The safety rails in the crawl loop — same-domain-only traversal, a destructive-action deny list, fake but never-submitted form data, a fresh page reload before every single interaction — exist for a specific reason. The earlier CLI-only version had no such guardrails against an authenticated crawl accidentally triggering a real logout or delete action mid-run. Folding the crawler into the web app was the point at which those constraints got written down and enforced instead of relied on as an operator habit.

Deployment access itself went through two layers instead of one after the initial build. An identity-gated access policy sits in front as the primary gate, with an app-level shared secret kept as defense in depth that can be disabled once the identity layer is confirmed working.

Outcome

Webshooter is deployed as a standing internal tool with its own subdomain, launchd service, and dedicated tunnel, callable either as a CLI or through its web app. It functions as one of the owner's internal utility tools, not a customer-facing product, used to generate documentation and training material for other Evolve-family sites and apps.

The deploy runbook that stood it up is itself a six-step, order-dependent sequence: environment setup, secret provisioning into the Keychain, launchd service load with an explicit health-check curl, dedicated tunnel creation and DNS routing, an identity-based access policy, and finally an end-to-end verification pass. That last step runs one small public crawl and one authenticated crawl before the tool is considered live. The sequence is evidence the "internal tool" framing didn't mean skipping the operational rigor a customer-facing service would get.