Testing design

The factory tests at three levels and no others. Unit tests and integration tests are the plugin’s dotnet test projects. E2E tests are driven in real Flare. The design rule is that a run tests only what its change can reach, and the full suites wait for the release gate.

Open the diagram full size

Level	What it is	When it runs
Unit tests and integration tests	The plugin’s test projects, run with dotnet test and a filter.	Every cycle of every run, limited to the test classes the change can reach.
E2E tests, as acceptance	An agent uses the change in real Flare inside a VM, as a writer would.	Once per run, only when a writer could notice the change or the harness changed.
Required checks	The plugin’s own CI on the pull request, build and feedback-worker.	Every pull request.
Full suites	Every test project and the whole E2E suite.	Only in the nightly release gate.

Acceptance replaces running E2E test scripts for each change. An E2E test script is a fixed list of steps. The acceptance agent reads the issue’s “done when” and checks what the issue asked for, plus whether a writer can use the result without trouble.

How a change picks its tests

Open the diagram full size

Invoke-Gate in factory2/lib/Gate.psm1 takes the patch and does the following.

1. Read the changed paths

Get-PatchPath lists every path the patch names, old and new, so a rename or delete counts.

2. Make a test plan

Get-TestPlan reads every csproj under src, tests and tools. It records each project’s references and whether it is a test project, meaning it uses the test SDK and does not set IsTestProject to false. For each changed path it decides what to run.

• Documentation under docs, corpus, changes and site, and Markdown at the repository root, runs nothing. A Markdown file inside a project does run, because a test can read it.
• A change to a project runs every test project that references it, directly or through other projects.
• Build configuration (csproj, sln, props, targets, lock files, nuget.config, global.json, anything under ref/ or lib/) and any path in no known project run the whole unit suite. The plan records why.
• A change to the style guide or the baselines starts from the Core projects and also runs the bench gate, which is scripts/check-baseline.ps1 -SkipPolicy in the plugin.
• A change under tests/e2e does not run dotnet test. It runs the E2E test script linter on the changed scripts, or on all of them when a helper changed.

3. Narrow it with the code graph

Get-AffectedTest narrows each planned project to the test classes the change can reach.

1. For a changed test file, the classes in that file that hold a [Fact] or [Theory] are added.
2. For any other C# file, every class, struct, enum, interface and record it declares becomes a symbol of the form file::Type.
3. For each symbol the gate runs python -m graphify affected <symbol> --depth 3 inside the run’s private copy of the graph. Every tests/<project>/...cs line in the answer adds that file’s test classes.
4. The result is a dotnet test --filter of FullyQualifiedName~.ClassName joined with |, one filter per project.

Two rules keep this honest.

• A type the change adds is not in the graph yet, because the graph is built from main. The query reports no unique match and the gate treats that as having nothing to find, not as an error.
• A project with no reachable test runs nothing. Nothing broader is added. The change’s own tests are always in the result, and a change that nothing tests runs nothing here.

A graph that is missing is an infrastructure stop, because the gate cannot know what to run.

4. Build and run

The gate builds FlareAI.sln in Release against the ref/ stubs, the same way the plugin’s CI does on a runner without Flare. When the style guide or fixture templates changed, it runs the fixture generator and keeps the regenerated rule fixtures as part of the change. Then it runs each project with TreatNoTestsAsError=true passed after --, so a filter that matches nothing fails instead of passing.

Test output is read as it arrives. If three tests fail over 40 seconds or more, the run is mostly waiting on timeouts, so the gate stops it there and says so. A short failure summary (project, failing test names, the first 40 lines of the error) goes to the next implement pass and to the diagnosis agent.

The code graph

scripts/graph.ps1 keeps a Graphify graph of the plugin repository.

Command	What it does
graph.ps1	Brings the shared graph up to date with main using code analysis only, and only if main has moved since the last build.
graph.ps1 -Rebuild	The full pass, which adds a semantic pass over the docs through the gateway. A scheduled task runs it every night.
graph.ps1 -Snapshot <dir>	Also copies the graph to <dir>, so a run reads a copy no other run can change.
graph.ps1 -Affected Symbol	Prints what a symbol can affect, tests included.

The shared graph lives under .work/graph/plugin/graphify-out and is not committed. A named mutex lets one process change or copy it at a time. A note in the folder records which commit of main it was built from, so an update with nothing to do is skipped. Graphify’s own cache means only files that changed since the last pass cost anything.

Each cycle of each run calls graph.ps1 -Snapshot into its own work folder and reads that copy, so another run updating the shared graph never changes what this run reads.

The release gate

scripts/release-gate.ps1 runs from a scheduled task at 01:15 each night, after the graph rebuild at 23:45. It

1. clones main of the plugin with its tags and finds the last release tag
2. exits quietly if nothing under src/ or styleguide/ changed since that tag, because the nightly release would not release either
3. runs vm-e2e.ps1 -Full against main, which is every E2E test script except the AI ones unless the gateway key is set up
4. on a pass, switches the plugin’s nightly workflow back on, but only if this script was the one that switched it off
5. on a failure, or when the VM cannot be used, disables the nightly workflow and comments on the tracker issue with the tail of the log

-DryRun runs the suite and changes nothing on GitHub.