Deterministic test data

Determinism is the property that makes a Tales suite replayable. Same binary version + same seed + same .tales files = identical generated values, byte for byte.

This guide explains how Tales achieves that, what the contract is, and where it deliberately doesn’t apply.

The contract

When you run:

tales test ./e2e/pass --seed 1234

You get the following guarantees:

• Every generate(...) call returns the same value on every run.
• The order in which scenarios produce their generated values does not depend on --parallel.
• Retries inside a step (retry { ... }) re-use the same generated values, so polling does not unexpectedly mutate inputs.
• Multipart upload boundaries (which are random) are deterministic when seeded.

You don’t get a guarantee about:

• Server timestamps, IDs assigned by the server, or any other server-side non-determinism.
• The current time (now_unix, now_rfc3339), these are intentionally live.
• The order log lines appear on stderr when --parallel > 1.

How the seed mixer works

Each call to generate(name) mixes the following inputs:

1. The global --seed value.
2. The scenario name.
3. The keyword call path (the chain of calling step names) when the call happens inside a keyword, empty otherwise.
4. The step name.
5. The generator name.
6. The expression path, where in the step the call appears.

The hash of those inputs seeds a deterministic generator just for that call. The implementation lives in internal/runtime/seed.go.

Why expression path matters

Two generate("user_email") calls in the same step appear at different expression paths, so they produce different values:

step "http" "create_two_users" {
  request {
    body {
      json = {
        primary   = { email = generate("user_email") }   // value A
        secondary = { email = generate("user_email") }   // value B (different)
      }
    }
  }
}

If you want the same value used twice, capture once:

step "http" "create_two_users" {
  vars {
    shared_email = generate("user_email")
  }

  request {
    body {
      json = {
        primary   = { email = vars.shared_email }
        secondary = { email = vars.shared_email }
      }
    }
  }
}

Why the same keyword called twice gets distinct values

When a generator runs inside a keyword, the call path (the calling step name, plus the outer call sites for nested keywords) is part of the seed. So calling the same keyword from two different steps produces different values, not duplicates:

keyword "register_user" {
  step "http" "register" {
    request {
      body { json = { email = generate("user_email") } }
    }
    expect { status = 201 }
  }
}

scenario "two distinct signups" {
  step "keyword" "signup_first"  { name = "register_user" }  // email A
  step "keyword" "signup_second" { name = "register_user" }  // email B (different)
}

This is what lets a register_user keyword be reused across a suite without the second registration colliding with the first (for example, a backend rejecting a duplicate email). A single keyword invocation still produces the same value on every run for the same seed.

Why parallelism doesn’t break determinism

Each scenario’s generated values depend only on the scenario name + the seed (plus step / generator / path). They do not depend on the order scenarios start, finish, or interleave. So --parallel 1 and --parallel 16 produce the same generated values for the same --seed.

Why retries don’t break determinism

The seed mixer does not include an “attempt number”. A retry replays the same step with the same vars, same generate(...) outputs, same JSON body. Only the wall clock (and the server’s response) can differ between attempts.

Where wall-clock helpers fit

now_unix() and now_rfc3339() read the current time. They are non-deterministic by design, there is no useful seedable “now”.

The right pattern is to compute the timestamp once per step in a vars block and reference it elsewhere:

step "http" "signed_call" {
  vars {
    ts = now_unix()       // captured once, stable for this step
  }

  request {
    headers = { X-Timestamp = "${vars.ts}" }
    body {
      json = {
        client_time = vars.ts   // same value
      }
    }
  }
}

See Signing webhooks for the canonical recipe.

Reproducing a CI failure locally

A red CI build with --seed 1234 reproduces exactly with:

tales test ./e2e/pass --seed 1234

If the failure was timing-sensitive (a 500 from a slow database, for example), the local run might not reproduce it, that’s an actual flake, not a seed issue. Use --report-jsonl to see the wall-clock durations from the failing run and compare.

Cross-version determinism

The seed contract is stable across patch versions (v0.1.0 → v0.1.1). Major or minor version bumps may change the generator implementations, when they do, the changelog calls it out so you know to regenerate your golden fixtures.

Use deterministic seeds in CI, exploratory seeds locally

A common pattern: pin the CI seed (--seed 1234) for reproducibility, but use --seed $(date +%s) locally when you want to randomise inputs. When the random seed surfaces a bug, save the seed in the bug report and add it as a regression fixture.

Tracking determinism over time

If you want to guarantee a scenario’s generated data is byte-stable across releases, snapshot the JSONL output of a known-seed run and diff future runs against it. Any change to the generator algorithm shows up immediately.

# Capture a baseline
tales test ./e2e/pass --seed 1234 --report-jsonl ./baseline.jsonl

# After upgrading Tales
tales test ./e2e/pass --seed 1234 --report-jsonl ./current.jsonl
diff <(jq -S . baseline.jsonl) <(jq -S . current.jsonl)

A clean diff means full determinism preserved. A diff means a generator or assertion changed, read the release notes.