Demo Pipeline

Demo Pipeline processes CS2 demos and turns them into useful data, clips, and analysis outputs.

It owns the real end-to-end contract for demo downloads, parsing/analysis, generated database rows, queue transitions, storage behavior, retries, and failure modes.

System map

Main components

Showrunner

Showrunner is the operator control surface for demo processing.

It can:

• create FACEIT download jobs
• create analysis jobs from existing storage files
• trigger worker polling
• inspect queues, pipeline rows, logs, and Cloud Run instance counts
• retry/reset/delete jobs and artifacts

Showrunner is intentionally environment-isolated by deployment boundary, not by a UI environment switch.

• Production: showrunner.flashback.gg from master
• Staging/dev: showrunner-dev.flashback.gg from dev

See Showrunner for runtime guards, Vercel env targeting, and operating notes.

Faceit Downloader

demo-downloader reads demo_download queue messages, downloads FACEIT demos, uploads the raw .dem.zst file to Supabase Storage, and advances the pipeline to analysis.

Staging service:

• demo-downloader-dev

Production service:

• demo-downloader

See Faceit Downloader.

Analyst

analyst reads demo_analyze queue messages, parses/analyzes demos, stores analysis JSON in R2, updates demo metadata, and advances the pipeline.

Staging service:

• analyst-dev

Production service:

• analyst

See Analyst.

Demtovi

Demtovi is part of the downstream processing/toolchain for demo-derived outputs.

See Demtovi.

Data and queue flow

For FACEIT jobs, the normal pipeline is:

1. Showrunner creates a demos row and demo_pipeline row through create_pipeline_job.
2. The job must include a resolved FACEIT demo resource URL.
   • Showrunner API accepts demo_resource_url.
   • If omitted, Showrunner falls back to the FACEIT Data API.
   • If no demo URL is available, job creation fails instead of enqueueing broken work.
3. Database RPC enqueues a demo_download message.
4. Downloader polls demo_download, claims the pipeline row, downloads the demo, and uploads raw storage.
5. Downloader calls advance_demo_pipeline_stage, which enqueues demo_analyze.
6. Analyst polls demo_analyze, parses the demo, writes analysis JSON, and updates demos.analysis_result_path / demos.analyzed_at.
7. Later stages can generate additional outputs such as video artifacts.

The key ownership rule: queue transitions and persistent state changes happen through database rows/RPCs, not in client state.

Environment isolation

Staging and production must never share mutable runtime resources.

Concern	Staging / dev	Production
Git branch	dev	master
Showrunner domain	showrunner-dev.flashback.gg	showrunner.flashback.gg
Vercel target	Preview branch dev	Production
App marker	NEXT_PUBLIC_APP_ENV=staging	NEXT_PUBLIC_APP_ENV=production
Downloader	demo-downloader-dev	demo-downloader
Analyst	analyst-dev	analyst
Supabase	dev/staging project	production project
R2 analysis bucket	flashback-demo-json-files-dev	flashback-demo-json-files
Cron	disabled by default	enabled

Showrunner has fail-closed runtime guards for dangerous misconfiguration:

• staging refuses production worker URLs
• staging refuses production R2 bucket
• production refuses *-dev worker URLs
• production refuses staging R2 bucket
• staging cron is disabled unless explicitly enabled
• generic/non-dev previews do not have Supabase/R2/worker mutation secrets

Deployment map

Staging processing flow

Deployment model

The demo-pipeline repository default branch is dev.

Showrunner / Vercel

Vercel is connected to GitHub with:

• project: showrunner
• root directory: showrunner
• production branch: master
• branch domain for dev: showrunner-dev.flashback.gg

Vercel uses an ignored-build command so Showrunner only deploys when files under its Vercel root changed.

Cloud Run workers / GitHub Actions

GitHub Actions deploy Cloud Run workers by changed directory:

• changes under faceit-downloader/** deploy only downloader
• changes under analyst/** deploy only analyst
• workflow dispatch can deploy both, demo-downloader, or analyst

Branch mapping:

• push to dev deploys staging services
• push to master deploys production services

Staging validation status

A staging E2E smoke test was completed with a recent P3chstein FACEIT match:

• downloader-dev successfully downloaded and uploaded a demo
• analyst-dev successfully processed a demo
• analysis JSON was written to flashback-demo-json-files-dev
• the same analysis key was verified absent from production R2
• staging test database rows and storage artifacts were cleaned up afterwards

The remaining FACEIT UI/API creation test should be rerun after FACEIT API limits refresh, using showrunner-dev.flashback.gg rather than direct RPC calls.

E2E boundary rule

The demo pipeline owns real end-to-end correctness for downloading, analyzing/parsing, generated database rows, queue transitions, storage behavior, retries, and failure modes.

Other systems, including Flashback challenge tests, should be able to mock the pipeline with deterministic outputs once this contract is covered by dedicated pipeline tests.

Implications:

• Flashback challenge E2E tests should not need to run the real parser/analyzer to prove challenge product behavior.
• Mocked outputs must match the real contract: tables, fields, statuses, queue messages, and error states produced by the pipeline.
• If the pipeline contract changes, update pipeline E2E tests and any reusable mocks/fixtures consumed by Flashback tests.
• Real demo-pipeline E2Es should run nightly and be manually triggerable.