methodical-ssz

This tool generates code, for marshaling, unmarshaling and merkleization (with bindings for go-hashtree) of go types. The original generated code patterns and library methods were based on code generated by fastssz. Many thanks to Ferran and the other contributors!

Generate ssz methodsets

The source types are parsed using go/types, which locates the source code for a given package path through go's local package discovery utilities, so a current limitation is that you need to fetch the package to your go package tree before using the tool. This can be done with go get:

go get github.com/prysmaticlabs/prysm/v7@latest

Once this is done, code generation can be run against go types in the desired package:

go run ./cmd/ssz gen --output=beacon-state.ssz.go --type-names BeaconState,BeaconStateAltair,BeaconStateBellatrix github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconState
Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconStateAltair
Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconStateBellatrix

The generated file can then be placed in a local directory for the source package so that the generated methods become part of the methodset of the source type.

Generator config yaml

The gen subcommand accepts a --config flag pointing at a yaml file, as an alternative to the --package/--type-names flags. The config names the source package and the types to generate, and is also where per-type codegen customization lives. (Note this is a different file from the spectest mapping yaml described below — projects typically keep several generator configs, for example one per fork.)

The minimal config is equivalent to the cli flags:

package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
types:
  - name: BeaconState
  - name: BeaconStateAltair

go run ./cmd/ssz gen --config=beacon-state.yaml --output=beacon-state.ssz.go

ProgressiveContainer

A type entry can be marked as an SSZ ProgressiveContainer (see the progressive merkleization spec) by adding a progressive key:

package: github.com/example/mytypes
types:
  - name: IndexedAttestation
    progressive: {}

A marked type's merkleization methods are generated as ProgressiveHashTreeRoot() and ProgressiveHashTreeRootWith(hh *ssz.Hasher), with the standard HashTreeRoot/HashTreeRootWith generated as thin wrappers that call the progressive forms. Callers — including the generated methods of enclosing container types — always use the standard names, and marked types redirect internally, so nesting progressive and regular containers requires no further configuration. Serialization (SizeSSZ/MarshalSSZ/UnmarshalSSZ) is unaffected: the spec serializes progressive types identically to their non-progressive counterparts.

An empty progressive: {} object derives the container's active_fields bitvector as all-ones, with one position per Go struct field. When fields have been removed from (or reserved in) the container's schema across forks, the gaps are expressed with inactive_indices — a list of 0-indexed positions in the bitvector forced to 0. The final bitvector length is the Go field count plus the number of inactive indices:

types:
  - name: ExecutionPayloadHeader
    progressive:
      # the type declares 3 go fields; positions 1 and 3 are reserved,
      # so active_fields = [1, 0, 1, 0, 1]
      inactive_indices: [1, 3]

The spec's validity rules are enforced at generation time: a progressive container must have at least one field, active_fields is limited to 256 positions, must not end in an inactive position, and inactive indices must be unique and in range.

ProgressiveList, ProgressiveBitlist, ProgressiveByteList

Progressive collections have no spec limit, so they are also declared in the config rather than struct tags: a type entry's fields key maps Go struct field names to an object with a type key.

types:
  - name: BeaconState
    progressive: {}
    fields:
      Balances:          {type: "ProgressiveList"}     # go type: []uint64
      ParticipationBits: {type: "ProgressiveBitlist"}  # go type: go-bitfield Bitlist
      ExtraData:         {type: "ProgressiveByteList"} # go type: []byte

ProgressiveList is the progressive merkleization form of the ssz List type (ProgressiveByteList is the byte-slice special case, equivalent to the spec's alias for ProgressiveList[byte]). ProgressiveBitlist applies to a named bitlist type backed by a byte slice, like github.com/OffchainLabs/go-bitfield's Bitlist. Hashing uses the spec's merkleize_progressive with a length mixin, with no limit. ProgressiveList family fields still use the ssz-size/ssz-max struct tags, so element dimensions can be resolved, and the field's ssz-max (present on ProgressiveBitlist fields optionally) is enforced at unmarshal time as a defensive check: generated UnmarshalSSZ rejects inputs exceeding it, exactly like the non-progressive form, so malicious inputs cannot trigger unbounded allocation. Progressive collection fields may be used in any generated type; the containing type does not need to be marked progressive.

Generate spectests for a package

This tool has a subcommand to generate go tests from the ssz_static tests in the ethereum/consensus-spec-tests repository. The generated tests can be run as normal go tests, which is helpful in identifying specific failure cases and walking through generated code with a debugger.

spectest tarball

To get the spec test fixtures for codegen, go to the consensus-spec-test repository releases page and download the most recent pre-release or latest spec tests release.

curl -L https://github.com/ethereum/consensus-spec-tests/releases/download/v1.3.0-rc.2-hotfix/mainnet.tar.gz > mainnet-v1.3.0-rc.2.tar.gz

yaml config

The spectest generation tool needs a mapping between go types and consensus spec container types. This mapping is described as a yaml config file. The mappings for prysm types are committed to the repo in a go testdata fixture directory, in specs/testdata/prysm.yaml. The following snippet illustrates the format:

package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
preset: mainnet
defs:
  - fork: phase0
    types:
      - name: AggregateAndProof
        type_name: AggregateAttestationAndProof
      - name: Attestation
      # ...
      - name: BeaconBlock
      # ...
  - fork: altair
    types:
      - name: BeaconBlock
        type_name: BeaconBlockAltair
      - name: BeaconBlockBody
        type_name: BeaconBlockBodyAltair

package is the go package containing the types for code generation. Each yaml holds the description for one go package. preset is one of mainnet or minimal. At this time only mainnet is supported as preset customization is specific to an implementation's build tooling. defs contains a list of yaml objects, with the fork key matching the fork directory in the spectest layout, and the types key containing a list of objects connecting the name of the type from consensus specs / spectest directory names, to the type_name found in the package. If type_name is not present, name is the default, so for go types with the same spelling and capitalization as the consensus type, the name field alone is all that's needed to specify the type.

The config reader processes type mappings in canonical fork order, so if a type's schema has not changed since the previous fork, it does not need to be redeclared. For instance the mapping for Attestation is only described once in the phase0 mapping; the same go type will be used to execute the spectest for the Attestation value for all subsequent forks. BeaconBlock, on the other hand, has been redefined at every fork, so tests will use BeaconBlockAltair for spectests in the altair tree and so on. Any types observed in the tarball that don't have an entry defined in the config yaml will be skipped with a warning.

spectest subcommand

The generated directory is a good place to stick generated tests as it is already in the .gitignore for the project. Assuming the tarball in the example above has been downloaded to the repo directory, running the following command there will generate spectests for all prysm types described in the test fixture yaml config:

go run ./cmd/ssz spectest --release-uri=file://$PWD/mainnet-v1.3.0-rc.2.tar.gz --config=$PWD/specs/testdata/prysm.yaml --output=$PWD/generated

Run the spectest like normal go tests:

go test ./generated
ok  	github.com/OffchainLabs/methodical-ssz/generated	1.003s

gen-spectest: generate spectests in your own module

The spectest subcommand above runs from inside this repository and builds a self-contained package (copying the type definitions and regenerating their methodsets), which requires methodical-ssz to depend on the package being tested. When your project already consumes methodical-ssz, that creates an awkward circularity — and the tests exercise a copy rather than the methodsets you ship.

The gen-spectest subcommand is the alternate mode for that workflow: run it from the directory of the module that owns the types. It performs no package loading and no code generation — the yaml config (same format as above) supplies the type names and the types package import path, and the tarball supplies the fixtures — so the tool needs no access to your types at all. The generated tests import your package and exercise the ssz methodsets already committed in your tree (produced by gen).

go run github.com/OffchainLabs/methodical-ssz/cmd/ssz gen-spectest \
  --release-uri=file://$PWD/mainnet-v1.3.0-rc.2.tar.gz \
  --config=./prysm.yaml \
  --output=./proto/prysm/v1alpha1/spectest

The output directory receives a test-only package (package spectest by default, override with --package-name) containing methodical_test.go and the fixture files under testdata/. Run it like any go test:

go test ./proto/prysm/v1alpha1/spectest

Since the fixtures are large and reproducible from the release tarball, the output directory is a good candidate for your .gitignore.

When the types span multiple Go packages (for example most consensus types in proto/prysm/v1alpha1 but the execution-layer containers in proto/engine/v1), a type entry can override the top-level package: with its own package key, and a single generated test package will import each used package under a deterministic alias:

package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1   # default
preset: mainnet
defs:
  - fork: bellatrix
    types:
      - name: BeaconBlock
        type_name: BeaconBlockBellatrix          # default package
      - name: ExecutionPayload
        package: github.com/prysmaticlabs/prysm/v7/proto/engine/v1 # override

A blank package means the top-level default; fork inheritance carries the package along with the type name, so a later fork can move a type to a different package. Only packages whose types matched a fixture are imported.