Reporting and FileSink

Reporting is a boundary, not a side effect hidden inside eval.Run. The core package exposes one small seam:

type Sink interface {
	WriteReport(context.Context, eval.Report) error
}

reportjson.FileSink implements that seam for a local directory. It encodes a validated report as canonical report/v1 JSON and writes the final file as <report-id>.json.

Atomic file writes

FileSink.WriteReport follows this sequence inside the caller-provided directory:

1. Check the context and validate the report ID as one safe path component.
2. Encode the report with the redacted report/v1 codec.
3. Open a randomly named hidden temporary file with exclusive creation.
4. Write all bytes and call fsync on that temporary file.
5. Close it, then rename it over the final .json name.
6. Remove the temporary file when any write, close, or rename step fails.

The temporary file is in the same directory as the final file, so the rename is atomic on the target filesystem. A crash can leave at most an obvious hidden temporary artifact, not a partially visible final report. os.Root scopes the open and rename to the sink directory. Empty IDs, separators, . and .., and path escapes are rejected before a write can reach another location.

func persist(ctx context.Context, report eval.Report, dir string) error {
	sink := reportjson.NewFileSink(dir)
	if err := sink.WriteReport(ctx, report); err != nil {
		var pathErr *reportjson.InvalidReportIDError
		if errors.As(err, &pathErr) {
			return fmt.Errorf("report id cannot be a file name: %w", err)
		}
		return err
	}
	return nil
}

The sink is safe for concurrent distinct reports because each write opens its own root and chooses its own random temporary name. A report with an invalid identity or failed validation writes no file.

report/v1 is redacted by default

The wire form is a versioned envelope. The payload shape is:

Encode and Decode are canonical and strict. They reject unknown versions, unknown fields, trailing JSON values, invalid UTF-8, non-finite measurements, and reports that violate whole-report invariants. Collections are sorted by stable identity for byte-stable output; summary status counts use a fixed status order.

The codec is a projection, not an observation round trip:

This lets a report travel through storage or a CI artifact without accidentally turning raw conversation, tool arguments, judge reasons, or provider errors into a log field. Use a separately controlled trace store when raw interaction data is required for debugging.

Read the redacted report

func readReport(ctx context.Context, dir, id string) (eval.Report, error) {
	raw, err := os.ReadFile(filepath.Join(dir, id+".json"))
	if err != nil {
		return eval.Report{}, err
	}
	report, err := reportjson.Decode(raw)
	if err != nil {
		return eval.Report{}, err
	}
	if err := report.Validate(); err != nil {
		return eval.Report{}, err
	}
	return report, nil
}

After decoding, inspect safe summary and assessment fields. Do not expect the conversation to be present, and do not treat an empty Finding.Message as a lost verdict explanation. The finding code and evidence references identify which safe fact the evaluator used.

Sink composition

The runner does not know whether a sink writes to disk, a database, or an observability system. A caller can fan a completed report to multiple sinks:

type multiSink []eval.Sink

func (s multiSink) WriteReport(ctx context.Context, r eval.Report) error {
	for _, sink := range s {
		if err := sink.WriteReport(ctx, r); err != nil {
			return err
		}
	}
	return nil
}

Run-level failures and sink-level failures remain different. A cancelled eval.Run returns a partial report with a context error; a sink may still write that partial report if the caller chooses. A sink error does not rewrite any assessment status.

The run contract explains how Summary and Provenance are produced. The model judge guide calls out why raw judge reasons are not safe wire fields.

Source

The sink contract is implemented by reportjson/sink.go, and the redacted codec is in reportjson/codec.go.

Proof

The sink sequence and path boundary are implemented in reportjson/sink.go. The envelope, redaction projection, canonical ordering, strict decoder, and target-error classes are implemented in reportjson/codec.go. Atomicity, redaction, canonical round trips, path rejection, and no-write-on-invalid tests are in reportjson/codec_test.go. The complete runnable example is examples/report/example_test.go.