Core Concepts

Architecture

Montaj is a video editing tool harness that mounts on top of your existing agent framework. It is not an agent. It is the toolkit the agent uses. You bring Claude, Cursor, or any agent; Montaj gives it the tools to edit video.

System Overview

┌──────────────────────────────────────────────────────────────┐
│                       LOCAL UI  (ui/)                         │
│                    browser → montaj serve                     │
│                                                              │
│  ┌──────────────┐                    ┌──────────────────┐    │
│  │  1. UPLOAD   │                    │   3. REVIEW      │    │
│  │  drop clips  │                    │  timeline        │    │
│  │  write prompt│                    │  preview player  │    │
│  │  POST /run   │                    │  caption editor  │    │
│  └──────┬───────┘                    └────────┬─────────┘    │
│         │          ┌──────────────┐           │              │
│         │          │ 2. LIVE VIEW │           │              │
│         │          │ SSE stream   │───────────┘              │
│         │          └──────┬───────┘                          │
└─────────┼─────────────────┼──────────────────────────────────┘
          │                 │
          ▼                 │
┌───────────────────────────┴──────────────────────────────────┐
│                      montaj serve                             │
│                  local HTTP + SSE server                       │
│                                                              │
│  POST /api/run    → creates project.json [pending]           │
│  GET  /api/projects → list projects; ?status=pending         │
│  file watcher     → detects project.json writes → SSE        │
└─────────────────────────┬────────────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────────────┐
│                     AGENT (external)                          │
│                  Claude, Cursor, etc.                         │
│                                                              │
│  reads project.json [pending]                                │
│  reads workflows/<name>.json                                 │
│  calls steps as tools at its own discretion                  │
│  writes project.json as work progresses → file watcher → SSE│
│  marks [draft] when done                                     │
└──────────────────────────────────────────────────────────────┘
                          │
                          ▼
              ┌────────────────────────┐
              │   human review (UI)    │
              │   optional tweaks      │
              └────────────┬───────────┘
                           │
                           ▼
              ┌────────────────────────┐
              │     RENDER PASS        │
              │  React + Puppeteer     │
              │  + ffmpeg              │
              └────────────┬───────────┘
                           │
                           ▼
                      final MP4

One package answers "what is on screen at time T" for the whole pipeline: montaj_assets/timeline-core/. The editor's preview player, the render pass, and the sample_frame diagnostic tool all resolve the timeline through it instead of each keeping their own copy of that logic. Its main entry point is resolveAt(project, t, { variant }), which returns everything active at instant t, used directly by the editor preview and by sample_frame; the render pass builds its ffmpeg segment plan from the same package's lower-level primitives (sourceWindow, boundariesFrom, activeIn) rather than re-deriving that logic.

Proxies and Playback

Every clip gets a full-source editing proxy at import: all-intra 720p H.264 + Opus, named <clip>_proxy_<look>.mp4 (<look> is a look-version tag; it changes whenever the active color look does, which naturally invalidates old proxies), so the editor can scrub and seek instantly instead of decoding the original master. The editor's preview plays the proxy (proxySrc) when one exists; the render pipeline ignores it entirely, so every export always encodes from the original-quality source, never the proxy. A clips project's children that share one lazy source converge on a single proxy rather than each encoding its own. Proxy generation can be skipped per run (--no-proxy) or per workflow ("proxy": false); without a proxy, or before one is ready, the editor just falls back to playing the master directly.

Agent Interfaces

Montaj exposes three interfaces for agents to call steps. All are optional. All wrap the same CLI executables.

CLI

The agent runs montaj commands directly via shell access:

montaj trim clip.mp4 --start 2.5 --end 8.3
montaj transcribe clip.mp4 --model base.en
montaj resize clip.mp4 --ratio 9:16

Works with any agent that has shell access, Claude Code, Cursor, or any framework that can execute shell commands.

MCP

Montaj runs as a local MCP server (montaj mcp), started automatically by the MCP client. The agent calls steps as native tools: no shell access required.

{
  "mcpServers": {
    "montaj": { "command": "montaj", "args": ["mcp"] }
  }
}

New steps are picked up automatically, adding steps/my-step.py + .json makes it available as an MCP tool with no extra configuration.

HTTP API

montaj serve exposes a step execution API alongside the browser UI:

POST /api/steps/trim        body: { input: "clip.mp4", start: 2.5, end: 8.3 }
POST /api/steps/transcribe  body: { input: "clip.mp4", model: "medium.en" }
GET  /api/steps             returns: list of available steps with schemas

All API routes are namespaced under /api/ so they never collide with React Router paths.

Summary

Directory Structure

Three scopes. Same format at every level:

~/Montaj/                       # workspace — all projects live here
  2024-11-01-my-ad/             # one directory per project
    project.json
    clip1_trimmed.mp4
    ...

~/.montaj/                      # user-global config + extensions
  steps/                        # user custom steps
  workflows/                    # user custom workflows
  config.json                   # global defaults (workspaceDir, model, etc.)
  credentials.json              # API credentials (0600 permissions)

montaj/                         # built-in (ships with montaj)
  steps/                        # native steps
  connectors/                   # external API wrappers
  workflows/                    # bundled workflows

The workspace location defaults to ~/Montaj. Override via the MONTAJ_WORKSPACE_DIR env var or ~/.montaj/config.json:

export MONTAJ_WORKSPACE_DIR=/Volumes/FastSSD/Montaj

{ "workspaceDir": "/Volumes/FastSSD/Montaj" }

Precedence: MONTAJ_WORKSPACE_DIR > ~/.montaj/config.json > default. The env var is useful for containerized deployments where dropping a config file is awkward.

Step Resolution Order

When the agent or CLI calls a step, Montaj resolves it in this order:

1. Project-local: ./steps/<name>
2. User-global: ~/.montaj/steps/<name>
3. Built-in: montaj/steps/<name>

Prefix in workflow files makes scope explicit:

Output Convention

All steps follow a strict contract:

• stdout: the result, either a file path or JSON. Nothing else.
• stderr: errors only: {"error":"code","message":"detail"}
• exit 0 on success, exit 1 on failure

This makes steps composable at the shell level:

FILE=$(montaj step rm_fillers --input clip.mp4 --model base.en)
FILE=$(montaj step trim --input "$FILE" --start 5 --end 90)
FILE=$(montaj step resize --input "$FILE" --ratio 9:16)

Versioning

Two layers:

• Git (milestone): montaj run initializes the workspace as a git repo. Commits are created automatically at state transitions (pending, draft, human save). montaj checkpoint "<name>" creates a named commit before risky operations.
• In-memory undo stack (fine-grained): the UI maintains an undo stack for the current review session. Every caption, overlay, or trim edit is undoable without touching disk.

Project Format

project.json is the single format that flows through the entire pipeline. One file, three states.

Project Types

The schema below describes video projects (editing, music_video, ai_video, broll). Carousel projects share the same header (id, status, projectType, name, workflow, editingPrompt, settings.resolution, assets) but replace tracks/audio/fps with a flat slides[] array. See the carousel page for details.

clips is a workflow (not a projectType). It fans out N child projects each with projectType: "editing". See Clips.

Lifecycle

Schema

{
  "version": "0.2",
  "id": "<uuid>",
  "status": "pending",
  "workflow": "overlays",
  "editingPrompt": "tight cuts, remove filler, 9:16",
  "settings": {
    "resolution": [1080, 1920],
    "fps": 30
  },
  "tracks": [
    {
      "id": "trk-0",
      "items": [
        {
          "id": "clip-0",
          "type": "video",
          "src": "/abs/path/clip.mp4",
          "start": 0.0,
          "end": 0.0
        }
      ]
    }
  ],
  "assets": [],
  "audio": {}
}

Settings

Identity

Each project gets a UUID (id) at init time. This is the stable identifier. The workspace directory name (~/Montaj/<date>-<name>/) is human-readable but not the identity. The optional name field is a label; it does not need to be unique.

The optional derivedFrom field is present on clip projects created by the clips workflow; it holds the source project's UUID as a provenance tag (the source project may no longer exist after find_clips completes).

Tracks

tracks is a top-level array of track objects ({ id, items, volume?, muted?, enabled? }) not a bare array of arrays. tracks[0] is the primary video track; its clips live in tracks[0].items. Overlay tracks start at index 1, each with its own items array. volume/muted/enabled are optional per-track settings that fold into every item on the track, in both preview and render: volume multiplies against each item's own volume (a clip already balanced against its neighbors keeps that balance when the track is turned down), muted silences every item on the track, and enabled: false skips the track entirely, hidden from preview and left out of the rendered export.

A project still on disk in the older bare-array-of-arrays shape ("tracks": [[item, item]]) is read correctly everywhere: the CLI, the render pipeline, and the editor all understand both shapes, and is converted to the object shape automatically the first time it's opened in the editor. When authoring or editing project.json directly (as an agent does), always write the object shape shown above.

Video Items

{
  "id": "clip-0",
  "type": "video",
  "src": "/abs/path/to/original.MOV",
  "inPoint": 2.1,
  "outPoint": 8.4,
  "start": 0.0,
  "end": 6.3
}

Overlay Items

{
  "id": "ov-hook",
  "type": "overlay",
  "src": "/abs/path/to/project/overlays/hook.jsx",
  "props": { "text": "She built an AI employee" },
  "start": 0.0,
  "end": 3.0
}

Captions

captions is a top-level object (not a track). It always renders above all tracks, topmost in the compositing stack.

{
  "captions": {
    "style": "word-by-word",
    "segments": [
      {
        "text": "Hello world",
        "start": 0.0,
        "end": 1.2,
        "words": [
          { "word": "Hello", "start": 0.0, "end": 0.5 },
          { "word": "world", "start": 0.5, "end": 1.2 }
        ]
      }
    ]
  }
}

Styles: word-by-word, pop, karaoke, subtitle, highlight-box, outline, clean. Caption data is always inlined, never a file pointer. start/end are output-timeline timestamps (after trim and concat).

Per-segment positioning. Each entry in segments[] may also carry its own offsetX, offsetY, and scale to move or resize that one segment independently of the rest, e.g. to clear a beat where something else (a whiteboard card, a lower-third, a screenshot) sits at the bottom of frame.

These fields are honored only by the JSX/Puppeteer render path. The older ffmpeg drawtext render branch can't position segments individually, so projects rendering through it see no change from these fields. Projects that never set them render exactly as before.

Assets

Image files (logos, watermarks). Each has id, absolute src, type: "image", and optional name.

# CLI
montaj run ./clips --prompt "..." --assets logo.png

# HTTP
POST /api/run  body: { "assets": ["/path/logo.png"] }

Audio

The audio field stores the project's independent audio tracks under audio.tracks[], music beds, voiceover spines, sound effects. Each track carries src, volume, start/end, inPoint/outPoint, fadeIn/fadeOut, and optional ducking. One track is one contiguous slice of one file; there's no multi-segment cut list for audio, which is why a cut-down voiceover is materialized to a single file first.

Voiceover

broll projects carry a top-level voiceover block: src is the file you supplied at intake (audio, or a video whose picture is discarded), and cleanedSrc is written by the agent after the voiceover has been cleaned. Only src is required. See B-Roll.

Live Updates

The agent writes project.json as it works: every write is picked up by the file watcher and pushed to the browser via SSE. The timeline builds live as the agent makes decisions.

State Transitions

montaj run       → project.json [pending]
agent completes  → project.json [draft]
human reviews    → project.json [final]
montaj render    → reads [final] → final.mp4

Trim Specs

The trim spec architecture is the core innovation in Montaj's editing pipeline. Editing steps output trim specs, not video files. A trim spec describes which ranges of the original source file to keep:

{
  "input": "/abs/path/original.MOV",
  "keeps": [[0.0, 5.3], [6.1, 12.4]]
}

Why This Matters

Before this architecture, every editing step re-encoded the full video. A five-clip workflow running silence removal + filler removal produced fifteen video encodes before the final concat. For 4K HEVC footage this caused multi-minute timeouts per step.

With trim specs, no video is decoded or encoded until materialize_cut. Editing steps work on audio only (for analysis) and pass timestamps forward. The entire set of cuts (silence boundaries, filler removals, take selections) is accumulated as trim spec refinements and applied in a single ffmpeg filter_complex pass at encode time.

Data Flow

waveform_trim(clip.MOV)
  → {input: "clip.MOV", keeps: [[2.1, 8.4], [9.0, 15.2]]}

transcribe({input: "clip.MOV", keeps: [...]})
  → extracts audio only at keep ranges
  → runs whisper on the joined audio
  → remaps word timestamps back to original timeline

rm_fillers({input: "clip.MOV", keeps: [...]})
  → extracts audio at keeps, detects fillers
  → subtracts filler timestamps from keeps
  → {input: "clip.MOV", keeps: [[2.1, 7.8], [9.2, 15.2]]}  ← refined

materialize_cut({inputs: [spec1.json, spec2.json, ...]})
  → ONE filter_complex per clip, applying all accumulated cuts
  → ONE encode pass total
  → final.mp4

Trim Spec Rules

1. Editing steps always receive the original source file path, never a re-encoded intermediate
2. Trim specs chain: each step refines the keeps list, preserving the original input path throughout
3. materialize_cut is the only step that encodes video: it handles both the normal pipeline encode and cases where a subsequent step (e.g. remove_bg) requires a physical video file
4. Uses input-level seeking (-ss/-t placed before -i), ffmpeg seeks at the container level so only the requested segment is decoded
5. HEVC source files are handled automatically: no pre-conversion needed

Trim Spec Steps

These steps produce or refine trim specs:

Trim Spec Consumers

These steps consume trim specs by encoding video:

Important: The src Field

Any video clip item in project.json (in any track) must have src pointing to a real video file (.MOV, .mp4, etc.): never a spec JSON file. For clips derived from trim specs, read spec["input"] for src, and spec["keeps"] to derive inPoint/outPoint.

The same rule protects proxySrc: it is written by Montaj and is preview-only. Never author it, never copy it into src, and never use it as a render input. A proxy path in src means the export comes from a 720p proxy instead of the original.

Multi-keep specs expand into multiple clip items, each with their own inPoint/outPoint.

Workflows

Workflows are suggested editing plans, which steps to use and their default params. The agent reads the plan, reads the prompt, and decides the actual execution.

A workflow is not a deterministic execution pipeline. The agent may reorder steps, adjust params, skip steps that don't apply, or add steps not in the list, whatever the prompt and content call for.

Workflow File Format

{
  "name": "overlays",
  "description": "Multi-clip edit — silence trim, transcribe, select best takes, remove fillers, concat, caption, overlays, resize to 9:16.",
  "steps": [
    { "id": "probe",             "uses": "montaj/probe" },
    { "id": "snapshot",          "uses": "montaj/snapshot" },
    { "id": "silence",           "uses": "montaj/waveform_trim",  "foreach": "clips", "params": { "threshold": "-30", "min-silence": 0.3 } },
    { "id": "transcribe",        "uses": "montaj/transcribe",     "foreach": "clips", "needs": ["silence"],           "params": { "model": "base.en" } },
    { "id": "select_takes",      "uses": "montaj/select_takes",                       "needs": ["transcribe"] },
    { "id": "fillers",           "uses": "montaj/rm_fillers",     "foreach": "clips", "needs": ["select_takes"],      "params": { "model": "base.en" } },
    { "id": "transcribe_final",  "uses": "montaj/transcribe",                         "needs": ["fillers"],           "params": { "model": "base.en" } },
    { "id": "caption",           "uses": "montaj/caption",                            "needs": ["transcribe_final"],  "params": { "style": "word-by-word" } },
    { "id": "overlays",          "uses": "montaj/overlay",                            "needs": ["caption"],           "params": { "style": "auto" } },
    { "id": "resize",            "uses": "montaj/resize",                             "needs": ["overlays"],          "params": { "ratio": "9:16" } }
  ]
}

Step Fields

Parallel Execution

needs is the dependency graph. The agent fires all steps with no unmet needs simultaneously, then re-evaluates after each completes. Steps in the same "wave" run in parallel.

Example execution waves for the overlays workflow:

Wave 1 (parallel): probe, snapshot, silence x N (foreach clips)
Wave 2 (parallel): transcribe x N (foreach clips — needs silence)
Wave 3:            select_takes (needs transcribe)
Wave 4 (parallel): fillers x N (foreach clips — needs select_takes)
Wave 5:            transcribe_final (needs fillers)
Wave 6:            caption (needs transcribe_final)
Wave 7:            overlays (needs caption)
Wave 8:            resize (needs overlays)

foreach: <path> fans out a step across all entries in a dotted-path collection on the project. Common values: "clips" (all project clips), "storyboard.scenes", "storyboard.imageRefs", "storyboard.styleRefs" (used by the ai_video workflow). The agent runs them as parallel tool calls and collects the outputs before proceeding to steps that need them.

Bundled Workflows

Agent-Authored Steps

Two step names in workflows are not CLI executables. They are tasks the agent performs itself:

montaj/select_takes

The agent reads transcripts from all clips, groups segments by content similarity (repeated takes), selects the best take of each section, and trims each clip accordingly using montaj/trim.

montaj/overlay

The agent writes custom JSX overlay files and adds them to tracks in project.json. There are no built-in overlay templates: every overlay is a custom React component the agent authors.

Deviation Rules

The agent should follow the assigned workflow and deviate only when the prompt explicitly requires it:

• "no captions" → skip caption
• "keep it raw" → skip rm_fillers, waveform_trim
• "YouTube format" → resize 16:9

Managing Workflows

montaj workflow list                    # list all available workflows
montaj workflow new <name>              # scaffold a new workflow file
montaj workflow edit <name>             # open in the node graph UI
montaj workflow run <name> ./clips --prompt "..."  # run a specific workflow

All workflow files are equal, fork any of them, save under a new name, and it becomes available immediately.