EvidenceRig

EvidenceRig turns declarative experiment specifications into immutable execution plans and reproducible evidence.

EvidenceRig is a declarative experimentation and evidence orchestration platform for reproducible computational research. You describe an experiment as data. The platform compiles it into a canonical, immutable plan, binds a human approval to that plan's hash, executes it under a recorded runtime, and keeps every result, artifact, decision and state transition as content-addressed evidence.

The distinguishing capability is not batch execution. It is being able to answer, long after the fact and without trusting anybody's memory: what was planned, what actually ran, under which exact runtime, what evidence was produced, which data was visible when, and why a gate promoted or stopped the work.

CLI: rig. Status: pre-release. See current status.

What it does

Capability	What it means in practice
Deterministic planning	The same specification compiles to the same plan bytes on any machine. No timestamps, host paths, locale or map ordering leak into a plan.
Immutable plans	An ExperimentPlan is canonical JSON with a SHA-256 identity. Every default is materialized, so nothing is hidden behind a framework.
Approvals bound to identity	Approval binds the plan hash, not editable source bytes. Change one parameter and the approval no longer applies.
Content-addressed artifacts	An artifact is named by the SHA-256 of its exact bytes. A path never silently refers to different data.
Provenance	A typed graph links spec, plan, approval, runtime, component digests, attempts, artifacts, analyses, decisions and reports.
Evidence lifecycle	Append-only events, an authoritative state projection, checksummed checkpoints, and immutable attempt records.
Checkpoints and recovery	Restart reconciles durable evidence before doing new work. Success is never inferred from a process disappearing; contradictions fail closed.
Controlled held-out data	Evidence cohorts carry roles and release policies, so confirmation and final-validation data stay sealed until the declared conditions are met.
Validation and gates	Typed validators check results; gates are deterministic predicates over declared metrics, with recorded reasons.
Human-comprehensible status	Commands explain state, progress, resource use, runtime identity and the next gate in plain terms.

What it is not: a workflow programming language, a cluster scheduler, a CI runner, a notebook platform, or a way to make a result look more reproducible than it is.

Where it came from

EvidenceRig grew out of a real system. FlyAdmin is a read-only Linux and Kubernetes diagnostic research project that needed staged scientific experiment matrices with exact approval, immutable container provenance, typed attempts, resource limits, deterministic promotion and a separate adjudication phase. The runtime it produced works, and it is still the authoritative executor for its own campaigns.

That relationship is now one-directional:

                EvidenceRig  (generic platform)
                     ^
                     |
   FlyAdmin / FlyCoinRH / other experiment systems  (consumers, case studies)

FlyAdmin and FlyCoinRH are historical origin, compatibility targets, real-world case studies, and future backend/plugin consumers. They do not define the generic core. No FlyAdmin or neuroscience vocabulary belongs in EvidenceRig generic contracts, schemas or packages.

Repository layout

Path	What it is
docs/rig/	EvidenceRig product documentation: architecture, contracts, CLI, CI/CD, migration plan, ADRs, methodologies, templates, examples
schemas/rig/v1/	EvidenceRig v1 contract schemas (JSON Schema 2020-12)
testdata/rig/v1/	EvidenceRig v1 conformance data: canonical-JSON vectors, node-identity vectors, a hand-authored Plan fixture
testdata/rig/legacy-m1b/, testdata/rig/canonical/	The R0 conformance corpus: a frozen characterization of the legacy runtime, never regenerated casually
tools/rig/	Small Python reference implementations used by tests and CI: canonicalization, schema loading, node-identity computation, deterministic-ordering and Plan semantic-invariant checks. Not the product runtime.
src/flyadmin/	The legacy FlyAdmin runtime, including the M1B campaign executor being migrated from. Still authoritative for its own campaigns.
scripts/, configs/, systemd/, schemas/*.json	Legacy preparation, configuration, installed-path and schema assets
roadmap.md	Both tracks: the EvidenceRig product roadmap and the FlyAdmin scientific roadmap

Current status

EvidenceRig is contract-first and pre-implementation. Milestones are defined in docs/rig/migration-plan.md.

Milestone	State
R0 — current-state semantic freeze and conformance corpus	complete
R0.5 — legacy runtime hardening	complete
R1 — generic EvidenceRig contracts and schemas	complete; approved for implementation after human architecture review
R2 — Go CLI and deterministic compiler	complete; rig validate, rig plan, planner only
R3 — legacy adapter and read-only observer UX	complete; rig status, rig inspect, rig verify (legacy M1B only, read-only)
R4 — managed compatibility launch adapter	R4.2 complete; rig run requests execution via the existing systemd unit (legacy M1B only). R4.1 (plan registry) and R4.3 (observer collision fix) deferred
R5 — generic component and backend protocol	R5.1 complete; internal/component describes, validates, invokes, and verifies one bounded component invocation. R5.2 (worker-pool backend) and R5.3 (FlyCoinRH adapter) deferred
R6–R9	planned

rig compiles an ExperimentSpec into an ExperimentPlan; it does not execute one. Its legacy status/inspect/verify commands only ever read an existing campaign directory, rig run only ever requests execution through the existing systemd-owned flyadmin-m1b@<id>.service unit, and internal/component can invoke one isolated component process and verify its result -- none of this schedules work, owns lifecycle state, or invokes Python/Docker directly. Until a milestone explicitly transfers ownership after installed-path acceptance, the legacy systemd -> controller -> Docker -> CampaignRunner chain remains the authoritative executor for existing campaigns.

Start here

• docs/rig/README.md — documentation map and reading order
• docs/rig/product-identity.md — name, CLI, schema namespace, module and image conventions, versioning
• docs/rig/architecture.md — architecture and subsystem disposition
• docs/rig/contracts.md — the v1 contract family, what each object is and who owns it
• docs/rig/cli.md — the rig command surface
• docs/rig/examples/ — a FlyCoinRH case study and a non-domain worker-pool benchmark, described by the same schemas
• AGENTS.md — rules for humans and agents modifying this repository

Running the tests

The legacy runtime requires POSIX (fcntl leases). Contract and schema tests additionally need jsonschema and PyYAML; they skip cleanly without them.

PYTHONPATH=src:tests:tools python3 -m unittest discover -s tests -t tests

pip install -e '.[dev]'

Some legacy tests depend on host-specific installed paths and on generated artifact directories that are not checked in; those failures are environmental and are listed in docs/rig/current-state-inventory.md.