Skip to content

Re-read the QEMU integration test work #31

Open

Description

@chrysh

Headed for the OCP demo on light review. Listed worst-first by how much
it would cost to have got wrong. All of it is additive and test-only
except OpenPRoT#541, which changes
services/orchestrator/driver and is merged to ocp-global-demo-wip as
bd062c9.

Merged to ocp-global-demo-wip on 2026-10-08, in this order: the harness
seed as d5fcaf5, OpenPRoT#533 as 8ba0756,
OpenPRoT#536 as 6f11f2f, OpenPRoT#537 as ca7e35f and
OpenPRoT#538 as 81da269. The rest still sit on
cq-demo-integration.

Read first, these are not test directories:

Read after, test-only:

  • tools: Add the flash image builder OpenPRoT/openprot#533 tools/flash_image, merged as 8ba0756. Host
    build tool. The layout goes through the real
    Region/Slot/ImageLayout constructors on purpose, so a test image
    cannot describe a layout the orchestrator would refuse.

  • services/test: Add the mock BMC boot model OpenPRoT/openprot#536 services/test/mock-bmc, merged as 6f11f2f.
    The boot model. Two behaviours only: a late boot is Boots with a
    long delay, so lateness is a property of the deadline rather than
    of the device.

  • target/ast10x0: Add the mock BMC integration test OpenPRoT/openprot#538 target/ast10x0/tests/integration/mock_bmc,
    merged as 81da269. The first scenario. The negative case is
    automated now, as mock_bmc/device_hangs.

  • target/ast10x0/tests/integration/pldm_update. A full DSP0267
    update between an agent and a firmware device, three apps in one
    image. No PR yet; it is commit 331321c on cq-demo-integration.
    The firmware device stages into RAM and the agent's bytes are
    checked against a known pattern, so the check to make is whether
    anything here could pass without the whole sequence running.

  • The boot scenario's RoT app now calls bring_up, so the state
    machine, the platform driver, the device table and CheckpointWalk
    are the shipped code rather than a hand-rolled sequence. No PR
    yet; it is commit ef645ea on cq-demo-integration. Read the board
    stubs: the verifier authenticates everything, and that is a
    decision about the demo, not an oversight.

  • The PLDM scenario's firmware device now reports to a real
    orchestrator in a fourth app and waits for its answer. No PR yet;
    it is commit 9bb3b7e on cq-demo-integration. Two gaps are in the
    commit message on purpose: the driver's Updatable stages nothing,
    and a refusal is recorded rather than enforced. Check that the
    three-message protocol cannot let a step run before its answer.

  • Wiring the two test boards to the verifier seam
    driver: Wire IncrementalVerifier into the update pump OpenPRoT/openprot#541 added is 980d4de. It leaves two verdicts on
    the same candidate, the device's and the RoT's, and the update
    needs both.

  • The PLDM update test failed about six runs in ten before dc7cc19.
    The agent could reach the bus before the device claimed its
    endpoint id, so discovery is retried now. Twenty consecutive suite
    runs pass. Worth a look at whether the retry hides anything else.

  • target/ast10x0/tests/integration/README.md says how to run each
    scenario and how to make it fail. Read it first: it is the
    shortest path into the rest of this list.

  • The three negative checks are test targets now, not hand edits. No
    PR yet; it is commit bf43011 on cq-demo-integration, and
    scenario.bzl plus SCENARIO_CFGS is the part worth reading. Check
    the claim each negative makes: it has to name the failure it
    arranged, not just observe that the run did not finish.

  • The firmware device stages into SPI NOR on CS1 now, and the runner
    reads the backing file back after QEMU exits. No PR yet; commits
    28c2063 and ea245b6 on cq-demo-integration. ea245b6 touches
    qemu_runner.py and defs.bzl, so it is not test-only: every QEMU
    test in the tree runs through that file. The claim to check is
    that a guest printing PASS still fails the test when the flash
    does not match, which was verified by flipping a byte.

  • full_update, the whole demo script in one image: five apps, an
    update over PLDM followed by the device being reset into it and
    reporting ready a second time. Commit 632122b on
    cq-demo-integration, which is pushed to the 9elements fork. Two
    things to check. The RoT owns the sentinel here rather than the
    firmware device, through the rot_owns_verdict flag, which should
    appear in exactly one BUILD file and leave the other scenarios'
    verdicts where they were. And the claim is the ordering, so look
    at whether a stale ready report could stand in for the second
    boot: the reset path clears the latch, and the run requires the
    reset count to go up across the activation.

  • full_update's own negatives, commit 74220c6 on
    cq-demo-integration, pushed to the fork. device_hangs keeps the
    device from coming up at all; device_stays_down lets the update
    succeed and then keeps the device from returning. The claim to
    check is that each passes on its own outcome and nothing else: a
    run that commits has to fail both, and device_stays_down's pass
    has to show the update accepted and activating in the log before
    the checkpoint failure, or it is passing on a broken update rather
    than on a device that never came back.

  • Discovery, transfer error, cancel and offer-before-supervising scenarios OpenPRoT/openprot#557, merged as f604054.
    Type 0 terminus discovery before offering an update,
    transfer_error (agent errors on one data request),
    cancel_mid_transfer (agent withdraws two chunks in), and
    offer_before_supervising (offer arrives before supervision,
    refused as Busy). The discovery moves the start-order retry to
    GetPLDMTypes. The scenario RoT's accept_update returns
    Option<RejectReason> instead of bool, so the reason is decided
    where the cause is known. Eleven scenarios, twenty-two targets,
    all green.

  • Rename mock BMC to mock downstream device OpenPRoT/openprot#559, merged as 0e2ff6c.
    Renames mock-bmc to mock-downstream-device: crate, type and
    scenario. The mock models the device on the other end of the reset
    and ready lines, not the BMC. pldm/firmware_update keeps the old
    name, where card B really is the update agent.

  • ast10x0: Ask for one chunk twice in the update scenario OpenPRoT/openprot#560, merged as 4519639.
    A twelfth scenario, pldm_update/retry_chunk. The agent answers one
    RequestFirmwareData with RetryRequestFwData, the device waits out
    FD_T2, asks for the same window again and the update completes.
    Nothing else makes the device request a window twice, so the
    re-delivery branch in its download callback had no caller. The test
    device reads the kernel clock now; with FdOps::now returning a
    constant its FD_T1 and FD_T2 windows never elapsed and no retry
    could fire. Checked against the failure it exists for: with 08763db
    reverted the run ends at "FD: the retry was asked for and the update
    did not complete". The second commit drops the commit timeout from
    the gap list, since Arm the commit watchdog from the machine's window OpenPRoT/openprot#554 was closed and nothing arms a deadline on
    BootConfirmed.

  • orchestrator: Confirm the eRoT's own update trial at the next boot OpenPRoT/openprot#565, merged as bffec2c.
    mock_downstream_device/self_trial. The board gets a session and the
    eRoT's own floor, the session starts at TrialPending with the trial
    image running, and the run passes when the chain coming up has moved
    it to Committed. What Ready proves is only that the chain under the
    eRoT came up, which is the bar the 2026-10-12 decision settled on.
    Checked against the failure it exists for: with the ConfirmSelfTrial
    emit removed from Ready entry the run ends at "the chain came up and
    the self trial was not confirmed". Only this scenario settles;
    pldm_update and full_update still wire self_image: None.

  • orchestrator: Move the SVN floor advance to the update agent's ask OpenPRoT/openprot#568, merged as 33b0e3e.
    The SVN floor advance moves off the confirmed boot onto
    Event::SvnCommitRequest, which the event loop injects when the update
    agent sends UpdateSecurityRevision. No firmware device parks at
    SvnCommitPending yet, so full_update and pldm_update dispatch that
    event themselves, the way they already dispatch BootConfirmed.
    GAPS.md says what a green run therefore does not say: the 0x22
    callback is not in the pinned pldm-lib, and the event-loop glue that
    reads the status and answers is unwritten.

What to look for rather than reading line by line: each test's green has
to mean something. For every scenario, ask what would still be true if
the test were lying, and whether something rules that out.

Merge commits go here as they land, since rebase merging rewrites the
SHAs and the PR numbers alone get harder to follow later.

Not a re-read, a change to make:

  • services/orchestrator/driver, the update pump's verifier state.
    update_verifier is an Option only so the session can be moved
    out to poll it and moved back. The hole that leaves is real: a
    bail-out between the two has to hand the verifier back by hand
    (drop_session), and a path that forgets would leave the driver
    unable to verify anything ever again. Putting the session in the
    phase (UpdatePhase::Authenticating(V::Session)) removes the
    Option and the last defensive arm with it. The cost is that
    UpdateJob becomes generic over the board, which threads through
    pending_update and every method that touches it.
    driver: Wire IncrementalVerifier into the update pump OpenPRoT/openprot#541 went as far as collapsing Authenticating
    into Staged, so the phase and the verifier can no longer
    disagree; nothing is wrong today.

Activity

  1. chrysh commented on Oct 8, 2026

    @chrysh
    CollaboratorAuthor

    OpenPRoT#546 merged to ocp-global-demo-wip as 9982f92 (measurement downstream wiring). OpenPRoT#547 landed via the same rebase-merge.

  2. chrysh commented on Oct 8, 2026

    @chrysh
    CollaboratorAuthor

    Merge SHAs for the integration-test stack, all landed 2026-10-08 on
    ocp-global-demo-wip. The stack was rebase-merged, so these are the SHAs on the
    branch, not the ones the PRs carried.

    Foundations:

    OpenPRoT#538 81da269e target/ast10x0: Add the mock BMC integration test

    OpenPRoT#550 target/ast10x0: Add the PLDM update integration test

    • ca789635 Add the PLDM update integration test
    • 01d4aa64 Run the real orchestrator in the boot scenario
    • d4904212 Put the orchestrator behind the PLDM update
    • b1dda205 Give the test boards an update verifier
    • 508aa747 Stop the PLDM update test failing at discovery

    OpenPRoT#551 target/ast10x0: Stage the update into flash on CS1

    • a26114f9 Describe how to run the integration tests
    • a717d689 Make the negative scenarios test targets
    • 9deba1b2 Stage the update into flash on CS1
    • fcec9bf3 Check what the guest left in CS1
    • 382457ce Say what the flash check proves in the README
    • 7687d2bb Command the firmware device over the shipped IPC seam
    • f0290ecf Fix and explain the memory map comments
    • 35cf1bb3 Fix review findings in the PLDM update tests

    OpenPRoT#552 target/ast10x0: Add the full update scenario

    • ea24f013 Write down what the integration tests do not cover
    • 5fb39f31 Add the full update scenario
    • 5eb7a513 Give the full update scenario its negatives
    • 763dce68 Document untested parts of the DSP0267 Type 5 flow
    • 6aa5bfd7 Reword test comments to stand on their own

    OpenPRoT#556 bb1d6752 Fix the two builds the merges broke. Not part of the stack; it
    cleans up after it.

    State at bb1d6752: bazel test --config=virt_ast10x0 //target/ast10x0/tests/integration/... is 16 of 16 green. Eight scenarios, each
    with a no-panics twin. target/ast10x0/tests/integration/GAPS.md landed with
    OpenPRoT#552 and is the list of what a green run still does not say.

    One thing for the re-read list. The eight *_no_panics_test targets all take
    1.7 seconds, which looks like a fixed cost rather than real work, and it is:
    rust_binary_no_panics_test runs pigweed's panic_detector --image over the
    built ELF on the host. It sets PW_RUNNER_PASSTHROUGH=1 so the QEMU wrapper
    does not start QEMU at all. These are static scans for reachable panic sites,
    not runs, so half of that 16 is not exercising any scenario.

  3. chrysh commented on Oct 13, 2026

    @chrysh
    CollaboratorAuthor

    Merge SHAs for everything that landed on ocp-global-demo-wip after the
    2026-10-08 stack, through e86ec87f. Same caveat as before: rebase-merged, so
    these are the branch SHAs, not the ones the PRs carried.

    Integration-test work, the part this issue is about:

    Supporting, outside the tests but reached by them:

    Crypto client, unrelated to this issue but on the same branch:

    Housekeeping: OpenPRoT#562 e86ec87f third_party: Bump ruzstd to 0.9.1

    The suite is twelve scenarios and twenty-four targets now, up from eight and
    sixteen. Four of the five new scenarios close gaps GAPS.md had listed as
    uncovered: Type 0 discovery, a transfer error, a cancel mid-transfer, and an
    offer arriving before the platform is supervised. The fifth, the retried chunk,
    exposed OpenPRoT#555: a device told to come back for a window ended its session instead
    of waiting out FD_T2, because the retry timer and a state mismatch both reported
    FdInitiatorModeError.

    What is left on the re-read list from the earlier comment is unchanged. The
    no-panics observation still stands: of the twenty-four targets, twelve are QEMU
    runs and twelve are host-side panic scans.

  4. chrysh commented on Oct 13, 2026

    @chrysh
    CollaboratorAuthor

    PR OpenPRoT#561 merged 2026-10-13, rebased as seven commits on ocp-global-demo-wip:

    • 8d114688 capabilities: Implement SelfUpdate for ()
    • 810e7dee driver: Give the board the eRoT's own update session
    • a91fd3c6 orchestrator: Add the eRoT itself as a component kind
    • 2dd30149 sm: Pass over the eRoT in the chain walk
    • 3577376d target/ast10x0: Put the eRoT at chain position 0
    • 4428f35d config: Require the eRoT at chain position 0
    • 1327d39c target/ast10x0: Count the device table instead of restating it

    This one touches the integration tests rather than adding scenarios: every
    device table gains the eRoT at position 0, so the managed device is now
    ComponentId(1) in all three images, and index 0 of each per-component array
    holds a value nothing reads. The suite stayed at 24 of 24 through the change.

    Two things to look at on a re-read. DeviceConfig::reset_signal now returns
    Option<&R>, which every board meets as a compile error. And chain_of gained
    two const asserts requiring the eRoT at position 0 and nowhere else; the eRoT
    rule lives only there, so a bare chain built through Chain::try_from, which
    is what the SM unit tests and the QEMU runtime test do, may still start with
    any kind. That split is on purpose but it is the kind of thing worth checking
    has not drifted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions