Plugin Development Guide

How to build a Solar Sailer plugin: a new Process-menu tool with its own settings dialog, progress reporting, and timeline edits — one Python file plus a manifest, no repo checkout, no TypeScript, no build step. This is "Tier 1" of the plugin system. For driving the editor from an external script instead (any language, over HTTP), see Scripting Solar Sailer — that's Tier 0, and it's often the fastest way to prototype an idea before turning it into a plugin.

Team plugins are trusted code: your server.py runs inside the editor's own Python server with full app privileges, exactly like a built-in module. There is no sandbox. Don't ship a plugin you wouldn't run as a program.

Quick start

1. Copy the plugin template folder (ask a teammate for a copy, or fetch it over HTTP — see below; it's self-contained) into your plugins directory. Find the directory in Edit → Preferences… → Plugins, which shows the path with a Copy button. The folder's name doesn't matter; the plugin's identity is the id inside plugin.json.
2. Click Reload plugins in that same Preferences section. Your plugin appears in the Process menu; load errors appear on the plugin's row in Preferences instead.
3. Edit server.py, click Reload plugins, run again. That's the whole iteration loop — no app restart, no build.

A plugin folder contains exactly two required files (plus anything else you like — a README, test data):

my-plugin/
  plugin.json    ← manifest: identity, menu entry, settings schema, completion mode
  server.py      ← one Python file: registers a ModuleDefinition, does the work

If you develop in a working folder elsewhere, launch the app with PROKO_PLUGINS_DEV_DIR=<your folder> — it is scanned before the installed plugins directory, and a dev copy of a plugin id wins over an installed copy.

No repo checkout? Everything above is also served over HTTP (bearer-authed, same token as the rest of the API — this is how the embedded AI agent and external scripts build plugins on an installed app): GET /plugins/guide returns this guide, GET /plugins/template returns the template files as {"files": {name: content}}, GET /plugins/list returns the plugins directory path (pluginsDir) and per-plugin load status, and POST /plugins/reload is the Reload button. Runs start with POST /modules/{id}/run {"media_ids": [...], "settings": {...}} and are watched via GET /modules/tasks/status.

The hard rules

These are the same contracts the built-in modules and the embedded AI agent live under. They are what makes a plugin's edits safe: one undo step, link groups respected, validation applied, nothing half-applied.

• A plugin's ONLY paths to changing the timeline are the four completion modes (ranges, review, report, transforms — see below). All of them end in the editor's command layer or the review tab. There is no API for a plugin to touch Redux state, dispatch actions, or mutate timeline data directly — do not look for one, and do not write timeline-shaped files for the app to pick up. If your tool needs an edit the completion modes can't express, that's a feature request for the plugin system (Tier 2 / new commands), not something to hack around.
• Never edit .sailer project files on disk. The running editor's state is the source of truth; the file is an internal serialization.
• Transcript writes go through TranscriptDoc.replace_span (server/services/transcript_doc.py) — never hand-roll a word_segments rewrite. It atomically rewrites the span, rejoins sentence text, re-runs word-timing refinement, and re-anchors review candidates. Hand-rolled rewrites corrupt sentence pills and silently drop review rows. Reads and writes of transcript files use transcript_lock + write_transcript_atomic (server/services/transcript_io.py); slow work (an LLM call) must follow the 3-phase pattern: read under lock → release → slow work → re-lock → re-read + merge → atomic write.
• AI calls go through the app's wrappers — from server.services.llm_router import call_llm (chat/vision, pass a tier) and from server.services.embeddings_router import embed_batch. Never import provider SDKs (openai, anthropic, …) or skell_e_router directly. The wrappers carry retry/backoff and cost-ledger attribution, and they're the seam tests mock. Provider keys come from the system environment; your plugin never reads or writes key files.
• Dependencies: Python stdlib plus the packages the app already bundles — the server ships with numpy, scipy, librosa, soxr, pyloudnorm, psutil, opencv-python (cv2), av, matplotlib, requests, mediapipe, fastapi. ffmpeg is on PATH. Nothing else: no pip on user machines, no vendored lib/ folders in v1, and compiled dependencies are out regardless. If your idea needs more, raise it — don't bundle it.
• Respect cancel_event and report progress (details below). A plugin that ignores cancellation makes the Cancel button in the status bar a lie.
• The reload contract is one file. "Reload plugins" re-executes server.py fresh. Helper modules you import stay cached (sys.modules) and background threads you start are not stopped — so keep everything in server.py, don't start threads that outlive your run() (a run-scoped worker or watcher thread is fine; the template uses one for cancellation), and don't do side-effectful work at import time beyond register(...). Anything beyond that shape needs a full app restart.

plugin.json — manifest reference

The manifest is validated on load (server-side and again in the renderer); every violation is listed on the plugin's row in Preferences → Plugins. Reference example: the template's plugin.json.

Field	Required	Meaning
id	yes	Registry id. Lowercase snake_case. Must be unique (a collision with a built-in module or another plugin is a load error), must equal the name= in server.py's register(...), and a few ids are reserved for the app's own background tasks (batch, conform, waveform, poster, sprite, proxy, capability, audio_quality) — the load error names the offender.
name	yes	Menu / dialog label, imperative ("Delete Long Silences").
version	yes	Informational in v1; shown in Preferences.
description	no	Shown to the agent and in tooling.
menuOrder	no	Process-menu position. Built-ins occupy 10–100; plugins default to 200 (after them).
activeLabel	no	Present-progressive status-bar label while running ("Detecting silences"). Without it the imperative name is shown.
requires	no	Capabilities every selected media item must satisfy: audio, video, transcript, sync_groups. Drives menu greying and the run dialog's Run gate.
batchable	no	Default true — the plugin appears in Batch Process. Set false to opt out.
settings	no	Ordered field list → the auto-generated settings dialog (next section).
completion	yes	{ "mode": "ranges" | "review" | "report" | "transforms", "rangeAction"?: "delete" | "disable", "ripple"?: boolean } — how a finished run becomes results (below). rangeAction/ripple are valid in ranges mode only.

Settings schema — the generated dialog

Each entry in settings becomes a control, rendered with the editor's own design system. The dialog's values arrive in your run() as the settings dict, keyed by key. All fields need key, type, label, default; all support optional help (muted line under the control), devOnly (only visible in Dev Mode), visibleWhen / enabledWhen ({ "key": "<other field>", "equals": <value> } — hide or grey this field until another field matches).

Type	Control	Extra fields
boolean	checkbox	—
number	number input	min, max, step (fractional values need a fractional step)
slider	slider + exact-value input	min, max required; step
select	dropdown	options: [{ "value", "label" }, …]; default must be one of the values
text	text input	placeholder

The key dev_mode is reserved — the app injects it into every run's settings (true while Dev Mode is on), so your plugin can gate diagnostics on settings.get("dev_mode", False).

server.py — the module contract

server.py runs once at load. Its only import-time job is registering a ModuleDefinition:

from server.modules.registry import register
from server.modules.types import ModuleDefinition, ModuleResult

def run(*, media_ids, file_paths_by_id, project_path, settings,
        cancel_event=None, progress_callback=None) -> ModuleResult:
    ...

register(ModuleDefinition(
    name="my_plugin_id",          # == manifest "id", enforced at load
    display_name="My Plugin",
    description="One line.",
    func=run,
))

The run signature is the app's runner contract — keyword-only, exactly these names. What you get:

• media_ids — the files the user checked in the run dialog (media ids, stable per project).
• file_paths_by_id — media id → absolute file path on disk. You have direct read access (trusted code).
• project_path — the project's data-sidecar root (<project>.sailer.data). It is already the .data root — don't append .data. Put any files your plugin persists under a subfolder named after your plugin id.
• settings — the dict from the generated dialog (plus dev_mode).
• cancel_event — a threading.Event; check cancel_event.is_set() at loop/phase boundaries and return ModuleResult(status="cancelled") promptly when set. Kill subprocesses you started.
• progress_callback — call with a float 0..1; the status bar shows it. Cap your own reporting at 0.95 and let completion signal the rest. If you fan out to threads, aggregate monotonically.

Return a ModuleResult: status is one of completed | needs_review | failed | cancelled; errors is a list of human-readable strings — always say which media and why. errors only reaches the user when status is failed. On a partially-successful completed run (3 files, 1 unreadable), put the skip lines into summary.report too — that's the visibility channel the completion toast shows; errors alone would hide the problem behind a green toast. summary is the completion-mode payload (next section). Runs execute in a worker pool off the server's event loop, and one run receives the whole batch of checked files — the app does not fan out per media, so parallelism across media happens inside your run(), by you (bounded ThreadPoolExecutor; shut it down with cancel_futures=True on cancel).

Everything else built-in modules can use, you can use — same process, same services. The patterns the built-in modules follow for caching, the transcript lock dance, progress aggregation, and testing seams apply to plugins verbatim; what does NOT apply to you are the core-file wiring steps a built-in needs (import lists, descriptors, parity fixture) — the manifest replaces all of them.

Completion modes — how results land

Declared in the manifest's completion.mode. This is the result contract: the app owns turning your summary into timeline edits through its command layer.

ranges — your run returns time ranges per media; the app deletes or disables them on the timeline: one undo step, linked audio/video kept in sync, trims and multi-clip layouts handled. Return:

ModuleResult(status="completed", summary={
    "range_edits": {
        "ranges": {"<media_id>": [{"start_sec": 12.4, "end_sec": 15.9}, ...]},
        "action": "delete",   # optional per-run override of manifest rangeAction
        "ripple": True,       # optional per-run override: close gaps (delete only)
    },
    # Optional; newline-separated. Shown as the completion toast's detail
    # lines — put counts AND any per-media skip notes here (see errors note above).
    "report": "Found 12 silences.\nSkipped intro.mp4: no audio stream",
})

Times are media time — seconds from the start of the source file, NOT timeline positions. The app maps them through every clip that uses that media, trims included. Missing range_edits is surfaced as an error; zero ranges is a friendly "nothing found" toast (which still shows your report lines).

review — for transcript-anchored suggestions a human should approve. Your run writes module-tagged candidates into the transcript sidecar server-side — {"word_id_start": ..., "word_id_end": ..., "text": ..., "module": "<your plugin id>"} appended to review_candidates — using TranscriptDoc (never a hand-rolled rewrite; see hard rules). The app refreshes transcripts on completion and seeds your rows into the Review tab, where the user applies or dismisses each one. Transcript plugins only in v1 (rows anchor to word spans). One shared-resolution caveat: resolving a row marks the underlying word reviewed for ALL modules, so two modules anchored on the same word share resolution state in v1.

report — analysis only, no edits. summary.report (string; first line = toast message, further lines = details) becomes a completion toast.

transforms — your run returns time ranges per media, each carrying a visual transform patch; the app splits the timeline clips at the range boundaries (linked audio/video kept in sync) and patches the carved-out video segments in place — one undo step, nothing moves, no cuts. Return:

ModuleResult(status="completed", summary={
    "transform_edits": {
        "transforms": {
            "<media_id>": [
                {"start_sec": 12.4, "end_sec": 15.9,
                 "transform": {"scale": 1.5, "positionX": -0.25, "positionY": -0.25}},
            ],
        },
    },
    # Optional, same toast-details channel as ranges mode.
    "report": "Zoomed 3 close-ups.\nSkipped intro.mp4: no face found",
})

transform takes any of positionX/positionY/scale/scaleX/scaleY/rotation/opacity/cropLeft/cropTop/cropRight/cropBottom — same meaning, bounds, and semantics as the editor's setClipTransform command: omitted properties keep the segment's current value, explicit null resets one to its default. The parser is strict and loud: an unknown key, an empty patch, or a non-number value rejects the whole result with an error toast — nothing half-applies, and a typo never reads as "nothing found". Each range's start_sec must be >= 0 and strictly less than its end_sec — a reversed or zero-length range rejects the whole result as malformed. Times are media time, mapped through every VIDEO clip that uses the media (trims and split clips included; audio-only media contribute nothing — there is nothing to see). Ranges carrying the identical patch may touch or overlap (they merge); ranges with different patches must not overlap each other. Read "Transform semantics" below before emitting scale/position — a bare scale does not zoom around the center.

Worked example — a centered zoom by factor s on the point (u, v) (fractions 0–1 of the frame, e.g. a detected face center), full-frame video, using the formula from the next section:

s = 1.5
u, v = 0.62, 0.35            # face-box center from your detector
zoom = {"scale": s, "positionX": 0.5 - s * u, "positionY": 0.5 - s * v}
ranges = [{"start_sec": 12.4, "end_sec": 15.9, "transform": zoom}]

Transform semantics — for tools that scale or reposition video

The transforms completion mode above is the native way for a plugin to emit clip transforms; Tier 0 scripts and the embedded agent's command API apply setClipTransform/transformRanges directly (see Scripting Solar Sailer). Whichever route you use, one non-obvious rule prevents mis-framed results:

Scale and position anchor at the TOP-LEFT of the fitted frame, not the center — only rotation pivots around the center. scale grows the picture right and down from its fixed top-left corner; positionX/positionY then offset that corner as fractions of the project width/height. A bare scale: s with no position drifts the picture right/down by (s−1)/2 of the frame, so every zoom needs a matching position:

• Plain centered zoom (full-frame video, no crop): positionX = positionY = −(s−1)/2.
• Zoom keeping source point (u, v) (fractions 0–1 of the frame, e.g. a face-box center) in the middle of the preview: positionX = 0.5 − s·u, positionY = 0.5 − s·v.
• Letterboxed/pillarboxed media (aspect mismatch, or cropped): replace the frame with the aspect-fit rect of the (cropped) source into the project resolution W×H — positionX = (W/2 − fitX − u·fitW·s)/W, positionY = (H/2 − fitY − v·fitH·s)/H.

That's exactly what the renderer does: scale multiplies the fit rect from its fixed top-left, and position is added to the fit origin.

Registration side effects you get for free

Registering means your plugin is a module: a generated Process-menu entry (name + menuOrder, greyed via requires), the generated settings dialog, status-bar progress with cancel, a Batch Process row (unless batchable: false), review seeding (review mode), and agent visibility — the embedded chat agent can run your plugin like any built-in module, with zero extra work. The agent can also write plugins: point it at this guide and the template.

Iterating — the reload loop

• Reload plugins (Preferences → Plugins) re-scans the plugin directories, re-executes every plugin's server.py fresh, and updates the Process menu live. Errors land on your plugin's row; a broken reload never takes down built-in modules, and a plugin that fails to load simply drops out of the menu until you fix and reload again.
• Reload is refused while any plugin has a run in flight (finish or cancel it first) — the app won't yank a registration out from under a running task.
• The narrow contract again: single server.py, no import-time side effects beyond register(), helper imports stay cached, background threads aren't stopped. Restart the app when you're outside that shape.
• Removing a plugin = delete its folder + Reload (or restart).

What a plugin cannot do (v1)

Each limit is deliberate, and each has an unlock path we're tracking: no importing new files into the media bin (planned with drag-and-drop import), no custom panels or per-plugin React UI (Tier 2, deferred), no rendering effects/transitions (the editor's effects system doesn't exist yet — when it lands, plugins inherit it as commands), no driving the chat agent, no extra Python packages.

Testing your plugin

• Fastest loop: run server.py's run() directly from a scratch script or pytest — it's a plain function; pass media_ids/file_paths_by_id pointing at a small test file and assert on the returned ModuleResult. The pattern our own template tests use: synthesize a small WAV, load the plugin through the real plugin loader, then assert on the result.
• Mock AI calls at the wrapper seam (monkeypatch.setattr("<your module>.call_llm", fake)).
• In-app: use a throwaway project, and remember every applied edit is one undo step — Ctrl+Z is your reset button.

Sharing with the team

Team plugins are distributed as folders — a git repo or shared drive the team copies from into their plugins directory (Preferences shows the path; then Reload). There is no marketplace or auto-update in v1; version bumps are new folder copies.