Table of Contents

Class ScenarioReport

Namespace
Valheim.Testing.Game
Assembly
Valheim.Testing.Game.dll

The result of one run: its steps, each in a StepPhase, the four states derived from them, provenance, log scans and linked evidence. Write(string) writes result.json (schema Schema) and junit.xml (one test suite per phase).

public sealed class ScenarioReport
Inheritance
object
ScenarioReport

Examples

A runner records each phase, scans the logs after its owned processes stop, and writes both result formats even when an earlier step failed. The complete owned-server lifecycle is in examples/FullLifecycle/ExampleMod.SystemTests.

static void Run(string outputDirectory, Action validatePlan, Action assertTerrain, Action restore)
{
    var report = new ScenarioReport("terrain");
    try
    {
        report.Step(StepPhase.Preflight, "validate plan", validatePlan);
        report.Step("check terrain", assertTerrain);
    }
    finally
    {
        try { report.Step(StepPhase.Cleanup, "restore", restore); }
        finally { report.Write(outputDirectory); }
    }
}

Constructors

ScenarioReport(string)

Creates an empty report; it cannot pass until at least one step runs and all recorded steps pass.

public ScenarioReport(string name)

Parameters

name string

Properties

CleanupVerified

At least one Cleanup step, and every one passed.

public bool CleanupVerified { get; }

Property Value

bool

Evidence

Every piece of evidence attached with Attach(IEvidence) or Attach(EvidenceReference), in attachment order, filled in by Write(string): snapshots, review stills and clips, each with its SHA-256.

public List<EvidenceReference> Evidence { get; }

Property Value

List<EvidenceReference>

Logs

The teardown log scans (ScanLogs(IEnumerable<RunLog>, IReadOnlyDictionary<string, LogClassification>?)), one per log.

public List<LogFileScan> Logs { get; }

Property Value

List<LogFileScan>

Name

The run name written into both result files and each JUnit phase suite.

public string Name { get; }

Property Value

string

Passed

Every step passed, in every phase. A run whose cleanup failed has not passed.

public bool Passed { get; }

Property Value

bool

PinOrigins

Each standalone-run pin group (runtimeManifest, runtimePins, worldManifest, worldUid and plugins.<guid>) and each prepared owned client's prefixed installPins and plugin pins, to plan when supplied and checked, or observed when bound from what the run staged.

public IReadOnlyDictionary<string, string> PinOrigins { get; }

Property Value

IReadOnlyDictionary<string, string>

Plugins

Per actor (server, a client's name, the host), the plugins its game confirmed with strict pins when it started: plugin GUID to the MD5 ValheimCLI compares, or absent.

public IReadOnlyDictionary<string, IReadOnlyDictionary<string, string>> Plugins { get; }

Property Value

IReadOnlyDictionary<string, IReadOnlyDictionary<string, string>>

PreflightPassed

Every Preflight step passed (true when there were none).

public bool PreflightPassed { get; }

Property Value

bool

Provenance

Caller-supplied source, fixture and environment facts written into result.json.

public Dictionary<string, string> Provenance { get; }

Property Value

Dictionary<string, string>

RuntimeReady

PreflightPassed and every Setup step passed: the runtime was ready.

public bool RuntimeReady { get; }

Property Value

bool

ScenarioPassed

RuntimeReady, at least one Scenario step, and every one passed.

public bool ScenarioPassed { get; }

Property Value

bool

Schema

The result.json schema: 2 since steps carry a phase and the four states are reported; 3 since every result names the toolkit that ran (Toolkit) and each actor's in-game plugin hashes (Plugins); 4 since loaded assembly identities are separate from package coordinates; 5 since standalone runs record whether each pin group came from the plan or was observed from what the run staged.

public int Schema { get; }

Property Value

int

Steps

Recorded outcomes in completion order. Concurrent actor starts may complete in either order.

public List<StepResult> Steps { get; }

Property Value

List<StepResult>

Toolkit

The toolkit packages, ValheimCLI transport and runner that ran, each released, candidate or unreleased (read when written).

public ToolkitProvenance Toolkit { get; }

Property Value

ToolkitProvenance

Methods

Attach(EvidenceReference)

Link evidence already written elsewhere, such as ReviewCaptureReceipt.Evidence or ReviewClipReceipt.Evidence. Write(string) stores its path relative to the report directory when it lies inside it.

public void Attach(EvidenceReference published)

Parameters

published EvidenceReference

Attach(IEvidence)

Attach evidence for the next Write(string) to serialize to evidence/<kind>-NNN.json and link from result.json, such as a TerrainSiteSnapshot or AreaObjectSnapshot with its exact replies.

public void Attach(IEvidence evidence)

Parameters

evidence IEvidence

RecordFailure(string, Exception)

Records a failure outside any step (for example in the runner around the scenario), so the report cannot pass. It goes in the phase of the last recorded step, where the run had got to (Preflight before any).

public void RecordFailure(string name, Exception error)

Parameters

name string
error Exception

RecordFailure(StepPhase, string, Exception)

Records a failure in phase outside any step, so the report cannot pass.

public void RecordFailure(StepPhase phase, string name, Exception error)

Parameters

phase StepPhase
name string
error Exception

RecordPlugins(string, IReadOnlyDictionary<string, string>)

Records the plugin pins of pins (the world expectations, such as its UID, are left out) as actor's. Call it once the actor's game confirmed them.

public void RecordPlugins(string actor, IReadOnlyDictionary<string, string> pins)

Parameters

actor string
pins IReadOnlyDictionary<string, string>

ScanLogs(IEnumerable<RunLog>, IReadOnlyDictionary<string, LogClassification>?)

Scans a run's logs after its processes stopped (LogScanner) and records a scan run logs step: it fails when a failure pattern matched or a required log is missing, and names each with its first occurrence. Warnings are counted in Logs only. classifications changes a pattern's severity for this run, with a written reason. It never throws, because it runs at teardown; returns whether the step passed.

public bool ScanLogs(IEnumerable<RunLog> logs, IReadOnlyDictionary<string, LogClassification>? classifications = null)

Parameters

logs IEnumerable<RunLog>
classifications IReadOnlyDictionary<string, LogClassification>

Returns

bool

Step(string, Action)

Records a Scenario step: runs action, records its outcome and time, rethrows its failure.

public void Step(string name, Action action)

Parameters

name string
action Action

Step(StepPhase, string, Action)

Records a step in phase: runs action, records its outcome and time, rethrows its failure.

public void Step(StepPhase phase, string name, Action action)

Parameters

phase StepPhase
name string
action Action

StepAsync(string, Func<Task>, string?)

The async twin of Step(string, Action): records the awaited action's outcome and time and rethrows its failure. With failure, the failure is rethrown as an System.InvalidOperationException with that message and the original as its inner exception.

public Task StepAsync(string name, Func<Task> action, string? failure = null)

Parameters

name string
action Func<Task>
failure string

Returns

Task

StepAsync(StepPhase, string, Func<Task>, string?)

The async twin of Step(StepPhase, string, Action).

public Task StepAsync(StepPhase phase, string name, Func<Task> action, string? failure = null)

Parameters

phase StepPhase
name string
action Func<Task>
failure string

Returns

Task

StepWithEvidenceOnFailure(string, Action, Func<IEvidence>)

Run an assertion, capturing evidence only if it fails. A failed capture is recorded as its own failed step; the original assertion failure remains the thrown exception.

public void StepWithEvidenceOnFailure(string name, Action assertion, Func<IEvidence> capture)

Parameters

name string
assertion Action
capture Func<IEvidence>

Write(string)

Writes result.json, one JUnit suite per StepPhase in junit.xml, and attached evidence beneath evidence/ in directory. It creates the directory if needed and overwrites those result files if they already exist there; callers should provide a new run directory. A linked EvidenceReference outside that directory retains its absolute path, so attach only evidence safe to expose with the result. This method can throw on a file-system failure; run it from cleanup even after a failed step.

public void Write(string directory)

Parameters

directory string
Edit this page