Observability

Every run is recorded in enough detail to rebuild what happened after the fact. Recording never changes a run. If an event cannot be written, one warning goes to the error stream and the run goes on.

Events

factory2/lib/Telemetry.psm1 writes JSON Lines, always at debug level, using OpenTelemetry’s field names and W3C style trace and span ids but not its protocol.

One file per process. Each process writes <run folder>/events/<component>.<pid>.jsonl. No two processes share a file, so an elevated process and an ordinary one never disagree about sharing and a partial line from one never spoils another. A reader merges the files by time. Every line is flushed as it is written, so a killed process loses nothing.

One line per event. Each line carries these fields.

Field	Meaning
ts	UTC time in milliseconds.
run_id, trace_id	The run, shared by every process in it.
span_id, parent_span_id	The process’s own id, minted at its first event, and the id of the process that started it.
seq, pid, comp	Order within the process, the process id and the component (factory, accept, vm-session, vm-e2e).
level, sev_num	debug, info, warn or error and the matching OpenTelemetry severity number.
event	The event name.
attrs	The event’s own data, as JSON.

Children find the run through the environment. FACTORY_RUN_DIR, FACTORY_RUN_ID, FACTORY_TRACE_ID and FACTORY_PARENT_SPAN are inherited. Processes started with -Verb RunAs may not inherit them, so the scripts that start one also pass the run on the command line (Get-TelemetryArguments) and the child calls Restore-Telemetry.

Big output goes to a file. Anything large goes to a numbered file under artifacts/ and the event names its path. The test runner writes the whole output of every native process there, and heartbeat events every 30 seconds show whether a quiet process is hung.

The events worth knowing are run.start, stage, agent, gate.plan, gate.scope, gate.step, gate.slowest_tests, process.start, process.heartbeat, process.end, process.stopped_early, vm.queue, vm.request, claim.wait, main.moved, rebase.conflict, diagnosis, stall, merge.refused, run.done, run.stop and run.error. Errors that are deliberately ignored are still written, as ignored.error, so none is silent.

Reading runs with runs.py

scripts/runs.py loads every events/*.jsonl under .work into a DuckDB view named ev. It needs pip install duckdb.

Command	Shows
summary	One line per run, with how it ended, seconds, cycles, Claude cost, diagnoses, stalls and warnings.
slow	The slowest steps across runs, from processes, gate steps and agents.
tests	The slowest tests and the tests that fail most, from the TRX files.
agents	Each agent step with turns, tokens, cost, permission denials and outcome.
failures	Every failed layer with its diagnosis, stalls, stops and errors.
timeline [RUN]	One run’s events in order, newest run by default.
trace [RUN] [OUT.json]	One run as a Chrome trace, to open at ui.perfetto.dev.
sql "select ..."	Any query over the ev view.

Attributes stay as JSON, so asking for a key an event does not have gives a null and never an error.

Test timings

Each dotnet test run adds the TRX logger. After it, the gate reads the file and writes a gate.slowest_tests event with the ten slowest tests and the names of failed ones, so slow and flaky tests can be followed across runs.

Agent stream capture

scripts/run-agent/Invoke-Agent.ps1 runs each agent with --output-format stream-json --verbose and writes what it sees as it arrives.

File	Holds
<name>.stream.jsonl	Every line of the stream, written and flushed as it arrives.
<name>.debug.log	The CLI’s debug log.
<name>.out.json	The final result message of the stream.
<name>.result.json	The wrapper’s own summary, with outcome, exit code, seconds, models and turns.
<name>.err.txt	Standard error.

Notable messages become events, so you can watch an agent work. They are agent.init, agent.tool_use, agent.tool_result, agent.permission_denied, agent.api_retry, agent.hook_problem and agent.result. The factory adds an agent event per step with model, outcome, seconds, turns, cost, tokens and permission denials.

Other records

• factory.log in the run folder has one line per stage with the seconds since the last stage, and a clock that shows hours.
• The catch log records which layer caught what. See Failure handling.
• The VM session keeps its own session.log and per-request vm.request events.