Render Engine

The render engine lives in render/ and turns project.json [final] into either a final MP4 (video projects) or N still PNG slides (carousel projects). It reads the project, renders the visual content via React + Puppeteer, and either composites with source footage via ffmpeg (video) or screenshots each slide directly (carousel).

How Rendering Works

Invocation

montaj render <project>
# or directly:
node render/render.js          <project.json> [--out <path>] [--workers <n>] [--clean]
node render/render-carousel.js --project-json <project.json> [--out <path>] [--clean]

Project status must be "final" before rendering. The render is non-destructive, source files are never modified.

Render dispatch by project type

montaj render and the HTTP endpoint POST /api/projects/{id}/render both read projectType from project.json and dispatch:

The video output path depends on the caller. The CLI (montaj render / direct node render.js) writes <project>/render/<name>.mp4, where <name> is the project's name field. The HTTP endpoint (POST /api/projects/{id}/render) writes <project>/output/<id>.mp4 so the file appears in the /outputs listing and Hub's delivery pipeline picks it up. Pass --out <path> on the CLI to override either path.

The video pipeline (Pipeline section below) covers the MP4 path. Carousel rendering is much simpler:

1. Read project.slides[].
2. For each slide: bundle its image + overlay elements as JSX via esbuild, load in Puppeteer, screenshot the slide (at 2× device pixels by default. See High-DPI output), write slide_NN.png (zero-padded, 1-based).
3. Write manifest.json listing each slide's filename + dimensions.

No segments, no ffmpeg, no audio. The carousel render modal in the UI streams the renderer's log lines and, on completion, opens a full-screen overlay with every slide as a clickable thumbnail and a Download all (.zip) button (GET /api/projects/{id}/render-zip). The zip excludes manifest.json. That's a renderer-side artifact for tooling, not user-facing.

High-DPI output (--scale)

Carousel renders default to --scale 2: each slide rasterizes at 2× the project's settings.resolution (e.g. portrait 1080 × 1350 → 2160 × 2700) so slides stay crisp on desktop and Retina displays without any flag. Pass --scale 1 for design-resolution (1×) output, or --scale 3 for 3× (print-quality PDFs, large-display use).

node render/render-carousel.js --project-json project.json            # 2× by default
node render/render-carousel.js --project-json project.json --scale 1  # design resolution

Also available on:

• montaj render (carousel projects render at 2×; pass --scale 1 for 1×; ignored for video).
• POST /api/projects/{id}/render (2× by default; ?scale=1 for 1×; carousel projects only).

The default applies end-to-end, including the auto-render triggered when a project reaches final. The design canvas stays at the base resolution, overlay coordinates and font sizes are unchanged. Only the output PNG resolution scales: the default 2× render is ~4× the pixel area (and file size) of 1×, and --scale 3 is ~9×.

Carousel manifest fields

manifest.json is written alongside the PNGs after every carousel render. Fields:

At scale=1, designWidth/designHeight and width/height are identical; at the default scale=2 the width/height pixel dims are double the design dims.

Pipeline

project.json
    │
    ├─ 1. Validate + resolve paths
    ├─ 2. Collect segment specs + video/image items
    ├─ 3. Normalize pre-pass (project working color space)
    ├─ 4. Process video items (remove_bg if flagged)
    ├─ 5. Bundle JSX → HTML  (one per overlay/caption)
    ├─ 6. Render HTML → FFV1/MKV  (Puppeteer pool)
    ├─ 7. Segment-based compose → concat
    ├─ 8. Mix audio tracks → <name>.mp4 (CLI) / output/<id>.mp4 (HTTP)
    └─ 9. derive-sdr  (--export sdr|both on an HDR project only) → <name>-sdr.mp4

Project Color Space

Each project has an explicit working color space stored at settings.colorSpace in project.json. This setting drives the codec, pixel format, and color metadata the entire render pipeline emits, from the normalize pre-pass through the segment encoder to the final concat.

Three color spaces are supported:

Smart-detect at init. When clips are added to a project (montaj run or montaj init), each clip's color transfer is probed and the project color space is the modal (most common) value across all clips. Outliers are converted on the fly, HDR sources in an SDR project are tonemapped per-segment, SDR sources in an HDR project are stretched into the HDR container. This matches the FCP/Resolve pattern: one SDR clip dropped into an iPhone-HDR project is treated as SDR-graded content shown on an HDR canvas, not a reason to flip the whole project down to SDR.

• All clips HLG → hdr_hlg.
• All clips PQ → hdr_pq.
• 27 HLG + 1 SDR → hdr_hlg (modal wins; the 1 SDR clip is stretched into HLG on the fly).
• 27 SDR + 1 HLG → sdr_bt709 (modal wins; the 1 HLG clip is tonemapped).
• Tied modes (no clear majority). Tiebreaks: HLG+PQ tied → hdr_pq (larger gamut, clean HLG→PQ conversion); SDR tied with HDR → sdr_bt709 (conservative, tonemap-down is well-defined, inverse-stretch is creative when there's no signal of intent).
• No clips probed → sdr_bt709 default.

Override. Pass --color-space {sdr_bt709|hdr_hlg|hdr_pq|auto} to montaj init (default auto runs the smart-detect rules above), or include "colorSpace" in the HTTP intake JSON, to force a specific working space regardless of source detection.

Legacy projects. Projects created before settings.colorSpace existed (or any project where the field was deleted) trigger the same smart-detect rules at render time. The detected value is written back to project.json so subsequent renders skip the detection step. This means an older project will render correctly on first run with no manual migration: the engine fills in what's missing.

Per-color-space behavior in the segment encoder. SDR projects emit libx264 yuv420p with bt709 color metadata; HDR projects emit libx265 yuv420p10le with bt2020nc colorimetry plus the appropriate transfer (arib-std-b67 for HLG, smpte2084 for PQ with static HDR10 mastering metadata). Sources whose color space conflicts with the project are converted at the per-item filter chain in the segment encoder: the Montaj Vivid LUT for HDR→SDR (see Montaj Vivid below), a stretch into the HDR container for SDR→HDR, and a zscale transfer-curve conversion for HLG↔PQ.

Montaj Vivid

Every HDR→SDR conversion in Montaj, editing proxies, renders, exports, embedded thumbnails, single-frame previews, goes through one signature look, Montaj Vivid, applied everywhere by construction rather than chosen per feature.

The look ships as an open 3D LUT in the repo, montaj_assets/luts/montaj-vivid-v1.cube, registered as the default curve (vivid1) in the manifest montaj_assets/luts/looks.json. A second curve is available for renders that want flatter highlights: montaj-vivid-v1-neutral.cube (id vivid1-neutral, labeled "Neutral brights"). The neutral curve is never used automatically. It's only ever selected explicitly, via --sdr-curve on a derived SDR export (see SDR Export Choice below) or the matching sdr-curve param on sample_frame.

The same look is baked into editing proxies as well as final exports, so what you see while editing an HDR project is the same grade the SDR export ships, preview and export agree. That parity holds only for the default curve: proxies and the live preview always use vivid1, so an export made with vivid1-neutral will look slightly different from what you saw while editing.

A build of ffmpeg missing the zscale/lut3d filters falls back to a generic tonemap instead of failing the render, with a loud warning printed to the render log. montaj doctor checks for lut3d support.

Step 3: Normalize Pre-Pass

The normalize pre-pass is color-space-aware. A source is conformant when its color transfer and bit depth match the project's working color space, and its keyframe interval is ≤ 2.0s (required for the segment encoder's input-level fast seek). When all three hold, the source passes through with no transcode, iPhone HDR HLG clips in an hdr_hlg project are essentially a no-op at intake.

When a source conflicts, normalize emits the project's working format using the encoder, pixel format, and color args from the color-space spec:

• sdr_bt709 project: libx264 -pix_fmt yuv420p with bt709 stream metadata. HDR sources are tone-mapped through the Montaj Vivid LUT chain (see Montaj Vivid above), preceded by a light denoise pass, because the vivid curve brightens midtones in a way that would otherwise amplify phone-camera shadow grain. A build missing zscale/lut3d falls back to a generic tonemap, with a loud warning.
• hdr_hlg project: libx265 -pix_fmt yuv420p10le with bt2020nc / arib-std-b67 stream metadata.
• hdr_pq project: libx265 -pix_fmt yuv420p10le with bt2020nc / smpte2084 stream metadata + static HDR10 mastering metadata.

All paths emit AAC 48 kHz audio and force IDR keyframes every ~1s.

Resolution is preserved. Source clips remain at their native resolution through the entire pipeline; the segment encoder scales per-item at compose time via the scale= filter in encode-segment.js. This avoids the permanent quality loss of intake-time downscaling and preserves headroom for crops, zooms, and re-frames.

Parallel execution: Both init-time and render-time pre-pass normalize loops run with a concurrency cap of 2. Memory-heavy 4K HDR encodes are the worst case; 2 workers stays within bounds on systems with ≥8GB free RAM. The cap applies to both libx264 (SDR projects) and libx265 (HDR projects): both are preset-bound CPU encodes.

Normalization creates _normalized_<colorSpace>.mp4 files alongside the originals (e.g. clip_normalized_sdr_bt709.mp4 or clip_normalized_hdr_hlg.mp4), originals are never modified. Namespacing by color space lets a project flip between SDR and HDR without colliding with cached normalize output. The lib/normalize.py module backs this and is also used at ingest time (project/init.py) and for AI-generated clips.

Cached output is reused across renders. When the deterministic _normalized_<colorSpace>.mp4 output already exists and its mtime is at least as fresh as the source, the render-time pre-pass skips the re-encode entirely. Replacing or re-recording a source file (which advances its mtime) correctly invalidates the cache. Net result: legacy projects pay the normalize cost once, not on every render.

After normalization, every source entering the compose pipeline conforms to the project's working color space. The segment encoder still handles per-item scaling at compose time, and applies in-line color conversion for any source that arrives in a different color space than the project. Resolution is intentionally NOT unified at intake.

Step 5: JSX Bundling

Each overlay/caption JSX component is compiled into a self-contained HTML page using esbuild. The page exposes window.__setFrame(n) so Puppeteer can drive it frame-by-frame.

Step 6: Puppeteer Rendering

A pool of N Chromium browsers (default: os.cpus().length) renders each segment in parallel.

Per-job flow:

1. Open a new page, set viewport to design resolution (1080 x 1920)
2. Navigate to the bundled HTML file
3. For each frame: call window.__setFrame(f), wait for paint confirmation, screenshot to PNG
4. Encode PNG sequence to FFV1 in MKV container
5. If segment exceeds chunk size, split into chunks and concatenate after encoding

Step 7: Segment-Based Compositing

Compositing uses a segment-based pipeline with three stages:

normalized video items + Puppeteer segments
    │
    ├─ 1. segment-plan.js   → plan segments at clip/overlay boundaries
    ├─ 2. encode-segment.js → encode each segment independently
    └─ 3. ffmpeg concat      → join segments via concat demuxer

Segment planning: the timeline is divided into segments at every clip and overlay boundary. Each segment is a contiguous time range where the set of active layers does not change.

Segment encoding: each segment is encoded independently with its own ffmpeg call. Items at non-project resolution are scaled by the per-item scale= filter. This is what enables source-resolution preservation at intake. Segments encode in parallel through a worker pool (default 2 concurrent encodes); tune it with the MONTAJ_SEGMENT_WORKERS environment variable.

Concat: all segments are joined via the ffmpeg concat demuxer with -c:v copy (no re-encode). This is near-instant.

Step 8: Audio Mixing

Independent audio tracks (music, voiceover, sound effects) are mixed in a final pass via mix-audio.js. Handles volume, ducking (sidechaincompress), delay offsets, and in/out points.

Four Render Stages (Summary)

montaj render runs up to four stages:

1. Normalize + base video: normalize all sources, then trim and prepare source clips. Canvas projects (no video track) generate a synthetic black base from overlay durations.
2. Overlay segments: each JSX overlay is bundled with esbuild, rendered frame-by-frame in headless Chromium, and encoded to lossless FFV1/MKV. Segments are rendered at design resolution (1080 x 1920) regardless of output resolution.
3. Compose + mix: segment-based pipeline encodes each timeline segment independently, concats them, then mixes audio tracks.
4. Derive SDR (conditional): only for an HDR project rendered with --export sdr or --export both. One additional ffmpeg pass converts the finished HDR master into an SDR sibling through the Montaj Vivid LUT chain. Skipped for --export auto (the default) and for SDR projects. See SDR Export Choice below.

Output Encoding

Per-segment encoding follows the project's color space (see Project Color Space above). Within a single render every segment shares one codec and pixel format, so the concat demuxer can stream-copy video without re-encoding:

• SDR projects (sdr_bt709): libx264 -preset fast -crf 18 -pix_fmt yuv420p with bt709 stream-level color metadata.
• HDR projects (hdr_hlg, hdr_pq): libx265 -preset fast -crf 22 -pix_fmt yuv420p10le with bt2020nc colorimetry plus the project's transfer curve (arib-std-b67 for HLG, smpte2084 + static HDR10 mastering metadata for PQ).

Per-frame setparams and per-stream color args come from the color-space spec, ensuring downstream players read the same colorimetry the encoder produced.

SDR Export Choice

By default an HDR project renders its HDR master untouched. montaj render (and the HTTP render route, via an optional JSON body) also accepts a render-time choice about which file(s) come out the other end:

montaj render --export auto                               # HDR master only (default), unchanged
montaj render --export sdr                                # one SDR file, no HDR master kept
montaj render --export both                               # HDR master + a derived SDR sibling
montaj render --export both --sdr-curve vivid1-neutral    # pick the "Neutral brights" curve

"Primary output path" is the same naming convention as render dispatch by project type above, <project>/render/<name>.mp4 for the CLI, <project>/output/<id>.mp4 for the HTTP endpoint. --export has no effect on SDR projects: there's no HDR master to derive from, so a sdr or both request logs a one-line notice and renders exactly like auto.

How the SDR file is made. The SDR rendition is never a second render of the project. It's derived from the already-finished HDR master in one additional ffmpeg pass. The master's pixels go through the same Montaj Vivid LUT chain used everywhere else in the product, then encode to the sdr_bt709 spec (libx264, bt709). Audio is stream-copied bit-identically rather than re-encoded, and the file is finalized with +faststart for progressive playback. One full render either way: never a second compose pass.

--sdr-curve <id> picks which look curve the derived SDR file uses: vivid1 (default, "Montaj Vivid") or vivid1-neutral ("Neutral brights", flatter highlights). It affects the export only. The live preview and editing proxies always use the default curve, so an export made with vivid1-neutral looks slightly different from what you saw while editing.

In the editor. An HDR project's render dialog opens on an export choice (Match footage (HDR), SDR, or Both) instead of starting the render immediately. An Advanced section offers the same curve picker, with a thumbnail of the project rendered through each curve, and a note that flips to a heads-up when the selected curve won't match the preview. SDR projects skip this choice entirely and start rendering as soon as the dialog opens, same as before.

Intermediate Files

<project>/
├── render/
│   ├── segments/           Puppeteer FFV1/MKV files + composed segment files
│   │   ├── <id>-chunk-0.mkv    (Puppeteer renders)
│   │   ├── seg-000.mp4         (composed segments)
│   │   └── ...
│   └── <name>.mp4          Final output — CLI / direct node invocation
└── output/
    └── <id>.mp4            Final output — HTTP endpoint (appears in /outputs listing)

Intermediate files are kept by default and reused on re-runs. Use --clean to delete them after compositing. Set MONTAJ_KEEP_SEGMENTS=1 to preserve composed segment files for debugging (they are cleaned up by default after concat).

Clip Seeking

Each video clip is fed as:

-ss <inPoint> -t <duration> -i <src>

Use -t duration (not -to outPoint). After the normalize pre-pass, every clip has frequent keyframes (every ~1s), so fast seek (-ss before -i) lands accurately. -t stops once it has read duration seconds of content, or at EOF, whichever comes first. This matters for clips where the source runs shorter than its timeline slot (e.g. a 24fps clip normalized to 30fps can lose frames at the tail): with -to, ffmpeg would hold the last frame past EOF instead of stopping; -t simply stops.

Render over HTTP

The serve layer exposes the render pipeline over HTTP. (Paths shown with the /api prefix used elsewhere in these docs.)

Synchronous (SSE): the default

POST /api/projects/{id}/render streams event: log / event: done / event: error Server-Sent Events until the render finishes. The render runs detached from the connection. A dropped client (e.g. a Cloudflare tunnel's ~100 s wall) only stops the SSE viewer: the render keeps running and the output still lands in <project>/output/<id>.mp4. (This changed in v3.2.3; previously a dropped connection killed the render.)

Asynchronous: poll for status

POST /api/projects/{id}/render?async=1 returns 202 immediately with {"projectId": "<id>", "status": "running"} and kicks the render in the background. Poll:

GET /api/projects/{id}/render/status → response shapes:

{ "status": "running", "phase": "rendering" }
{ "status": "done",    "phase": "done", "outputPath": "/abs/path/to/output/<id>.mp4" }
{ "status": "error",   "phase": "encoding", "error": "<message>" }
{ "status": "idle" }

phase is a coarse, user-facing progress signal that advances preparing → rendering → captions → encoding → done. The terminal state persists after completion, so a late poll still returns done + outputPath. Use async mode when a render may outlive the proxy/tunnel request wall.

Cancel

POST /api/projects/{id}/render/cancel aborts the in-flight render (kills the whole process group, so child ffmpeg/headless-browser processes die too). This is the only way to abort a render: a dropped SSE connection no longer does. Re-issuing POST /render supersedes a previous tracked render rather than returning 409.

Scale (carousel only)

Append ?scale=1|2|3 to POST /render. Default is 2 (2× device pixels, crisp on Retina). Ignored for video projects.

Custom Overlays

Overlays are custom JSX components written by the agent. There are no built-in overlay templates: every overlay is a React component the agent writes, styled to the editing prompt and brand context.

Overlay Item in project.json

{
  "id": "ov-hook",
  "type": "overlay",
  "src": "/abs/path/to/project/overlays/hook.jsx",
  "props": { "text": "She built an AI employee" },
  "start": 0.0,
  "end": 3.0
}

offsetX, offsetY, and scale are applied by the render engine as a CSS transform on the component container. The JSX component itself is unaware of them.

Component Globals

Overlay components have access to these globals:

Recharts Globals

Overlay JSX can render SVG charts via Recharts. The following components are available as globals, no import needed:

BarChart, Bar, LineChart, Line, PieChart, Pie, Cell, XAxis, YAxis, CartesianGrid, Tooltip, Legend, ResponsiveContainer

Carousel-only scope. Charts work in carousel slides today. Video overlays don't have a chart story yet.

Three system overlay templates (bar-chart, line-chart, pie-chart) ship out of the box for carousel projects. See the Image Carousel docs for the data shape per template and property panel details.

When writing a custom chart overlay, always set isAnimationActive={false} on every chart primitive and size the chart with explicit width/height props rather than <ResponsiveContainer>: the screenshot fires synchronously and won't wait for async resize observers. See skills/write-overlay/SKILL.md (Charts / Recharts section) for the full authoring rules.

How Overlays Are Rendered

1. The JSX file is bundled into a self-contained HTML page by esbuild
2. Puppeteer opens the page in headless Chrome at 1080 x 1920 (transparent background)
3. For each frame: window.__setFrame(n) increments, screenshot to PNG with alpha
4. ffmpeg encodes the PNG sequence into a transparent FFV1/MKV video segment
5. The segment is composited onto the source footage in the segment-based compose pipeline

HDR projects: still images referenced from overlay JSX (e.g. <img src="..."> or CSS background-image) are automatically converted to HDR-encoded PNGs before Puppeteer screenshots them, so they composite at the correct brightness and colour against HDR footage. No configuration needed, conversion happens on first render and the result is cached on disk. Text, shapes, and other non-image content in the same overlay are unaffected.

Parallelism

Overlay rendering is CPU-bound. Two levels of parallelism:

• Segment-level: all overlay segments are independent and rendered simultaneously by a worker pool
• Frame chunking: segments above 1,000 frames (~33s at 30fps) are split into chunks, each rendered by a separate worker

caption track (18,000 frames) → 18 chunks × 1,000 frames → 18 workers
lower-third (135 frames)      → 1 chunk → 1 worker
flash (9 frames)              → 1 chunk → 1 worker

Configurable via ~/.montaj/config.json:

{ "render": { "workers": 8, "chunkSize": 1000 } }

Browser Recycling

Each Puppeteer worker restarts its browser every 5 jobs. After many segments, browser processes accumulate memory and can start timing out. Recycling flushes that state.

Design Resolution

Overlays are always rendered at 1080 x 1920 regardless of output resolution. The pipeline upscales at compose time for higher resolutions (e.g., 2x for 4K).

Caption Templates

Caption templates are pre-built React components referenced by style name. Unlike overlays (which are always custom JSX), captions use built-in templates selected by the agent or user.

Available Styles

How Captions Work

1. transcribe step generates word-level timestamps via whisper.cpp
2. caption step converts the transcript into a caption track with a chosen style
3. Caption data is stored inline in project.json (not as a file pointer)
4. At render time, the template component receives the caption segments and renders frame-by-frame via Puppeteer

Caption Data Format

{
  "id": "captions",
  "type": "caption",
  "style": "word-by-word",
  "segments": [
    {
      "text": "Hello world",
      "start": 0.0,
      "end": 1.2,
      "words": [
        { "word": "Hello", "start": 0.0, "end": 0.5 },
        { "word": "world", "start": 0.5, "end": 1.2 }
      ]
    }
  ]
}

Each segment contains:

• text: the full segment text
• start / end: segment time window (seconds)
• words: individual word timestamps for styles that animate per-word

Rendering

Caption templates produce the same output as custom overlays: rendered frame-by-frame by Puppeteer, composited into the video by ffmpeg. The template component uses interpolate and spring utilities for animation.

Choosing a Style

Via CLI:

montaj step caption --input transcript.json --style word-by-word
montaj step caption --input transcript.json --style karaoke
montaj step caption --input transcript.json --style pop
montaj step caption --input transcript.json --style subtitle
montaj step caption --input transcript.json --style highlight-box
montaj step caption --input transcript.json --style outline
montaj step caption --input transcript.json --style clean

Via the editing prompt:

"add word-by-word captions"
"karaoke-style captions, bold text"

Editing Captions

In the UI review phase, you can:

• Click a caption segment to edit its text inline
• Drag segments to adjust their timing
• Change the caption style

Changes update the segments array in the caption track of project.json.

GPU Acceleration

The Montaj render pipeline is mostly CPU-bound. GPU acceleration applies at one specific step.

Where GPU Applies

ffmpeg detects and uses available hardware encoders automatically, providing a 5-10x speedup on the final encode.

CPU-Bound Stages

Puppeteer Frame Rendering

The main bottleneck. Each overlay/caption segment is rendered frame-by-frame in headless Chrome. Two parallelism strategies:

1. Segment-level: all segments rendered simultaneously via a worker pool (default: CPU core count)
2. Frame chunking: segments above 1,000 frames are split into chunks for parallel rendering

ffmpeg Compositing

Each segment is composited independently: the per-segment filter graph is simple since all sources are pre-normalized. GPU filters exist but are limited.

Configuration

Control worker count and chunk size via ~/.montaj/config.json:

{
  "render": {
    "workers": 8,
    "chunkSize": 1000
  }
}

Or via CLI flags:

montaj render --workers 4

Color Space Handling

Color space is a per-project setting (settings.colorSpace), not a fixed pipeline assumption. iPhone HDR clips (HEVC, BT.2020/HLG 10-bit) flow through an HDR project with no transformation: the working format already matches their native color space. In an SDR project they're tone-mapped through the Montaj Vivid LUT chain (see Montaj Vivid above), like the rest of ffmpeg's per-segment filtering, this is a CPU-bound pass with no GPU acceleration path, so the "Final encode" GPU row above doesn't apply to it. A build missing zscale/lut3d falls back to a generic tonemap. Sources whose color space conflicts with the project are converted at intake or, for late-arriving items, in the segment encoder.

Run montaj doctor to verify your ffmpeg has zscale and lut3d support. zscale is required for any cross-color-space conversion (HDR↔SDR, HLG↔PQ), and lut3d for the Montaj Vivid LUT itself; on systems missing either, only direct passes (matching project color space) and a degraded HDR→SDR fallback work. montaj install ffmpeg rebuilds ffmpeg with libzimg (adds zscale) on macOS.

Troubleshooting