LafitteWare LLC / Detailed showcase
ClipFactory
An AI-assisted media product that turns long recordings into ranked clip candidates and edited exports. I built the selection logic, editorial feedback loop, and processing pipeline—from explainable scores to face-aware framing and branded captions.
Sole builder · Product architecture, AI selection & media engineering
Review before rendering
The problem
Clip selection is an editorial judgment, not just a timestamp search. A useful system needs to rank moments, explain its choices, learn the editor’s preferences, keep timing consistent, and turn approved decisions into reliable media outputs.
What I built
Built a FastAPI application and a separate worker process coordinated through a durable PostgreSQL job queue. Ingestion, word-timed transcription, candidate discovery, creative briefs, human review, and rendering form distinct stages. An explainable eight-signal scoring system runs alongside optional LLM discovery. Stored decisions feed a bounded learning loop that proposes versioned selection profiles. OpenCV face tracking and FFmpeg handle framing, captions, audio, and export verification.
The result
Implemented an end-to-end media product with inspectable candidate scores, editorial feedback, versioned selection and brand profiles, account isolation, subscription integration, and recoverable background processing. The system connects editorial decisions to rendered outputs while preserving the evidence behind its recommendations.
Architecture / Simplified overview
How the pieces connect
- 01Ingest + word timing
- 02Discover + score
- 03Review + learn
- 04Frame + render
- 05Verify + export
Inside ClipFactory
Intelligence with an editorial feedback loop.
One timeline, separate processing stages.
- 01
Ingest
Probe the media and prepare word-timed captions or speech recognition.
- 02
Discover
Segment the transcript, rank moments, and merge optional LLM candidates.
- 03
Review
Inspect the score and brief; approve, reject, or adjust the boundaries.
- 04
Produce
Reframe, caption, normalize audio, render, and verify the export.
FastAPI handles review and API requests. PostgreSQL persists jobs and application state; a separate worker performs media processing.
Explainable selection / Default profile
What goes into
the Clip Score?
Each signal scores 0–10. The weighted sum is scaled to 0–100, with the components stored alongside the candidate.
These are implementation weights, not measured audience outcomes. Editorial feedback can propose a different profile.
- Hook
- 22%
- Novelty
- 18%
- Standalone clarity
- 15%
- Tension
- 10%
- Emotion
- 10%
- Information density
- 10%
- Boundary quality
- 7.5%
- Duration fit
- 7.5%
Architecture that survives interrupted work
The web process owns accounts, review screens, and API requests; a separate worker runs media jobs. PostgreSQL stores records and queue state. Atomic job claims, database-clock leases, heartbeats, retry limits, and idempotent fingerprints coordinate processing and recovery. Versioned database migrations run before startup. Media is shared between the web and worker processes, while job and entitlement records preserve processing and billing state.
AI clip selection and the engagement score
Word-level transcripts provide a shared timeline for segmentation, boundary snapping, captions, and cuts. A deterministic discovery pass ranks candidates using hook strength, novelty, tension, emotion, information density, standalone clarity, boundary quality, and duration fit. Optional LLM discovery adds candidates and editorial briefs; overlap checks merge the results. The 0–100 Clip Score exposes the contributing signals so an editor can understand the ranking. It estimates editorial usefulness from engagement-related cues; audience engagement or virality is not a measured prediction.
Training through editorial feedback
The learning process uses the account’s own approvals, rejections, and boundary edits. It keeps the latest verdict for each candidate and attributes feedback to the selection-profile version that produced it. After at least 40 decisions, including 12 approvals and 12 rejections, the evaluator compares signal values across the two groups and proposes bounded weight adjustments. It re-scores stored candidates and requires a Brier-score improvement of at least 0.005 before saving a proposed version. This is retrospective calibration on the same feedback sample; it does not establish performance on unseen clips.
Learning language and editorial preferences
The evaluator can add up to three literal phrases per proposal when they recur in approved text and are rare in rejected text. Phrases are escaped before compilation, keeping transcript content from becoming executable matching patterns. Repeated boundary edits can shift the preferred clip duration by a bounded amount. Proposals retain sample counts, signal differences, and before/after evaluation evidence. Profiles are immutable versions and require review by default; automatic activation requires explicit operator and profile opt-in. This trains editorial selection preferences without fine-tuning a foundation model.
Speaker awareness and stable reframing
The implemented framing layer samples video at two frames per second and uses OpenCV’s frontal-face detector to follow the largest detected face. A deadzone ignores small movements; a hold interval requires a new position to persist before a smooth pan. The resulting keyframes become a bounded FFmpeg crop expression, with a centered fallback if detection fails. This provides visual speaker framing for suitable footage. Audio diarization and identifying the active speaker in a multi-person scene are separate capabilities that this implementation does not establish.
Editing, branding, and verified exports
Editors approve or reject candidates and adjust clip boundaries before rendering. The renderer supports 9:16, 4:5, 1:1, and 16:9 canvases; word-highlighted ASS captions; brand profiles; silence-cut edit lists; loudness normalization; and watermark handling. Existing word-timed source captions can bypass speech recognition when they pass timing checks; otherwise faster-whisper supplies the transcript. Exports include H.264 video, subtitle and metadata sidecars, and thumbnails. Output probing and frame-hash checks help detect invalid or frozen renders.