StoryFlow

Problem

Turning raw property walkthrough video into finished, branded marketing edits normally means either paying an editor per listing or burning hours in an NLE for every clip. Neither scales across a high listing volume. Neither produces the three edit variants — different pacing and look — a listing needs, without redoing the work three times. The tool needed to take a folder of unedited footage and hand back finished, deliverable edits without a human touching a timeline. The build also had a hard resource constraint baked in from the start: everything runs on a single 16GB Mac Mini, with no cloud GPU and no local generative video or image models permitted. The whole pipeline had to be analysis and classification, not generation, and had to fit a memory budget the machine also uses for other services.

What was built

StoryFlow watches a folder for new walkthrough footage and runs it through a six-stage pipeline: ingest and scene detection, semantic room classification, quality scoring per clip, slot assignment against a story template, automated finishing (color grade, stabilization, retiming, titles, audio), and final render. Each stage is a hard gate. The next stage cannot start until the previous one is built, tested, and confirmed against real footage. The finishing layer alone was built out as a standalone toolkit: color correction, three preset looks (Clean/Dynamic/Cinematic), stabilization, vertical/horizontal reframing, lower-third titles, and loudness-normalized audio, composable per clip and per output variant. A gate-review aid, storyflow contact-sheet, writes a labeled grid — one representative frame per detected clip, captioned with timecodes. Confirming Stage 1's scene cuts becomes a glance instead of a manual scrub through raw footage. The target story template, home-tour-45.json, encodes a 45-second home tour as a JSON graph of slots (exterior, entryway, living room, and more). Each slot carries a required/optional flag, a duration range, semantic targets the slot is looking for, preferred camera motion, and things to avoid (blur, duplicate angles) — the artifact Stage 4's assignment scoring is graded against.

Technical approach

The pipeline splits across Docker and host process on purpose. Postgres (pgvector-enabled from day one) and Redis run in Docker, but the arq analysis worker runs as a launchd host service, so the memory-heavy embedding and ffmpeg render steps don't compete with a hard-capped Docker VM memory ceiling. Ingest uses ffprobe for metadata, PySceneDetect's content detector for scene boundaries, and writes one proxy JPEG per second per clip for review. Idempotency is enforced with a unique checksum index and ON CONFLICT DO NOTHING at the database level, after a code-review pass caught a check-then-insert race under concurrent ingest. Quality scoring computes blur (Laplacian variance), stability (Farneback optical flow), exposure (histogram clipping), and framing (Hough horizon tilt) as pure functions normalized 0-1 against synthetic-footage reference constants. The slot-assignment scoring formula is treated as a frozen contract once merged: 0.35M + 0.20Q + 0.15F + 0.10B + 0.10D + 0.10U - P. Two terms, B and the motion half of F, are held as neutral 0.5 placeholders until music-beat and motion signals exist, explicitly not renormalized around their absence. The target machine's ffmpeg build lacks vid.stab, zscale, and drawtext. The finishing layer substitutes deshake for stabilization, stays SDR-first instead of HDR tonemap, and renders titles as OpenCV PNG overlays composited via a two-input ffmpeg filter honoring the delivery codec.

Creative approach

Craft

The finishing layer's three variant looks (Clean, Dynamic, Cinematic) are built as composable primitives, not baked presets, so a .cube LUT can drop in per variant to override the grade without touching the pipeline. Vertical (1080x1920) and horizontal reframing are both first-class outputs from the same assembled edit. One walkthrough needs to serve a vertical social cut and a horizontal listing cut without a second manual pass. The finishing toolkit is broken into single-purpose primitives — capabilities (runtime ffmpeg filter/encoder probing and fail-loud selection), color, stabilize, retime, titles, audio, reframe, deliver, and recipe (the three named looks composed from the rest). Each is independently testable, before the Stage 5 assembler composes them per clip and per variant. That assembler was verified end to end on a real composed case: stabilize, then the cinematic look (grade plus vignette), then a vertical 1080x1920 reframe, encoded with hardware HEVC and a lower-third overlay, producing a correct file. A separate three-segment case cross-dissolved the segments with computed xfade offsets and landed on the exact duration the offset formula predicted. Titles render as OpenCV-drawn PNG overlays composited through a two-input ffmpeg filter, not through ffmpeg's own text-drawing filter, because that filter isn't present in the target machine's build. The visual result — a lower-third with the intended type — is identical to what drawtext would have produced, just reached through a different renderer.

Reframe

The project is explicitly scoped as an automated pipeline, not an editing tool: 'NOT an interactive NLE, node compositing, real-time grading, multicam, or a DAW.' That constraint is what makes stage gating possible. Each stage's output is a deterministic, testable artifact — a scene list, a quality score, an assignment, a render — not a human-in-the-loop decision. A director-review UI (Stage 6) can sit on top of the pipeline as an override layer, instead of being the pipeline itself. The same discipline shows up in the slot-assignment scoring formula. Two of its seven terms, beat compatibility and the motion half of framing fit, are held as neutral 0.5 placeholders instead of being dropped or estimated, because the signals that would compute them (music-beat analysis, motion classification) don't exist yet. The formula is frozen with honest gaps in it, instead of being quietly renormalized to hide what isn't built. Storyflow also refuses to guess: an unclassifiable clip in Stage 2 stays unclassified, instead of being forced into the nearest room-type bucket. That matches the same never-impute-missing-data rule the pipeline applies everywhere else.

Process and what failed

Real footage deliberately never enters the repo, and only the owner can place it on the machine. That means the Stage 1 gate (confirming scene cuts land where a human would cut) and the Stage 2 confusion-matrix validation are both structurally blocked pending that footage. The pipeline's correctness on synthetic ffmpeg-generated fixtures is proven, but calibration against real walkthroughs is not. Code review surfaced a real concurrency bug — a duplicate ingest race on checksum — that was fixed with a unique index, not application-level locking. The finishing layer's stabilization approach had to be reworked around a missing ffmpeg filter (vid.stab), discovered only by probing the actual build on the target machine.

Outcome

Stages 1, 3, and the finishing/assembly toolkit are built, tested (420 passing tests, ruff clean), and code-reviewed. Stage 1's real-footage gate and Stage 2's semantic classification remain blocked on the owner supplying and labeling walkthrough footage. That's a deliberate constraint — real client footage never enters the repo — not a stalled build. Stage 4's slot-assignment scoring is specified and frozen but not yet coded. The finishing/assembler layer still needs its music-bed and beat-align add-ons and a wire-up to real Stage 4 assignments before it can run unattended end to end. It is an internal Mac Mini tool with no external users, no auth beyond the local network, and no marketing surface by design. The infrastructure it depends on (Postgres, Redis) is deliberately not even started as a resident service until real footage is ready to process.