feat: add bloatnet marker support

[LouisTsai-Csie]

🗒️ Description

Background

Bloatnet benchmark tests were introduced in PR #2040 and #2090. Because of the nature of bloatnet, using execute together with a stub address is a more ideal solution compared to the fill and consume method.

With the fill command, the pre-deployment state of contracts is encoded in the genesis file. However, this becomes problematic when the state configuration in bloatnet is too large. By leveraging execute, we can directly send transactions to the network to interact with contracts.

Instead of creating a new folder for bloatnet, the test cases are placed under the existing benchmark/ folder, with a specific marker to distinguish them.

Following the discussion in issue #1926, the test_bloatnet.py file is placed under benchmark/state/bloatnet/. All tests that require the combination of execute and stub addresses are grouped under the stateful/ folder. PR #2101 would also benefit from this structure.

Based on this design, I did not add a separate bloatnet marker. Instead, I extended the marker to stateful marker, which is shared across all stateful tests.

Verification

• Flag: -m stateful runs only the bloatnet-related tests (excluding benchmark tests).
• Flag: -m benchmark runs only the benchmark tests (excluding bloatnet tests).

To verify:

1. Add a test in the test_bloatnet.py file, then run:

• uv run fill -v tests/benchmark/stateful/bloatnet/test_bloatnet.py -m benchmark → should not run
• uv run fill -v tests/benchmark/stateful/bloatnet/test_bloatnet.py -m stateful → should run

2. Add the state marker to another file under benchmark/ (e.g. test_block_full_data), then run:

• uv run fill -v tests/benchmark/test_worst_blocks.py::test_block_full_data -m benchmark → should not run
• uv run fill -v tests/benchmark/test_worst_blocks.py::test_block_full_data -m stateful → should run

Documentation:

1. For test cases under state/ folder, the command should include the -m stateful flag
2. For test cases not under state/ folder but label with stateful, the command include the -m stateful flag
3. For the remaining cases, it should include -m benchmark

Notes

I am open to discussion, please let me know if you want to rename the marker, or

🔗 Related Issues or PRs

PR #2040 , #2090 and #2101

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
• Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
• Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

[CPerezz]

Make sure to mark any existing benchmarks in /bloatnet once #2186 is merged.

[spencer-tb]

Can we change from -m state to -m stateful - the same for the folder name: state/ to stateful? Since we have -m state_test already!

The logic all makes sense to me. Having 2 conftest's was the better option after reviewing this ;)

Will re-review once 2186 is merged so we can check the logic.

[LouisTsai-Csie]

We just need to move the file under benchmark/stateful/bloatnet and the marker would be applied!

[spencer-tb]

LGTM!

[danceratopz]

Thanks a lot for tackling this @LouisTsai-Csie! The PR description is really helpful and documents the expectation very well 🙏

The benchmark paths aren't correct in the following, but it should explain my suggestion.

With the additional requirement to "filter" stateful tests, I'm wondering if we're adding too much complexity to our pytest setup via markers in order to run the correct subset of tests. The logic alone for tests/benchmark/conftest.py is quite complex and then we overlay the logic from tests/benchmark/stateful/conftest.py on top of this.

I won't block this, but I think the requirement is actually quite simple and perhaps we can quickly consider an alternative approach before merging. My proposal would be to remove all of this conftest logic and use an approach that modifies pytest's testpaths at the CLI logic stage.

I.e., adding a mode flag, None by default (=just fill regular tests), but that can be set to "benchmark" or "stateful".

uv run fill --mode=benchmark

Then, we add (simpler) logic in src/cli/pytest_commands to modify fill and execute's ini config from (and others...):

• https://github.com/LouisTsai-Csie/execution-spec-tests/blob/7e4d3e214252fa878a7ac5c381fcab0d48d1c40a/src/cli/pytest_commands/pytest_ini_files/pytest-fill.ini#L5
  https://github.com/LouisTsai-Csie/execution-spec-tests/blob/7e4d3e214252fa878a7ac5c381fcab0d48d1c40a/src/cli/pytest_commands/pytest_ini_files/pytest-execute.ini#L5

to overwite testpaths with the relevant path (needs a well-organized structure underneath ./tests/benchmarks/). For this, we don't modify or add new ini files, but rather modify it using this functionality:

-o testpaths=tests/benchmark/stateful

From pytest --help:

  -o OVERRIDE_INI, --override-ini=OVERRIDE_INI
                        Override ini option with "option=value" style, e.g. `-o
                        xfail_strict=True -o cache_dir=cache`.

For normal filling mode (no benchmarking), we can append:

uv run pytest --help | grep ignore
  --ignore=path         Ignore path during collection (multi-allowed)

to ignore ./tests/benchmarks in regular filling.

[danceratopz]

Thanks a lot for tackling this @LouisTsai-Csie! The PR description is really helpful and documents the expectation very well 🙏

The benchmark paths aren't correct in the following, but it should explain my suggestion.

With the additional requirement to "filter" stateful tests, I'm wondering if we're adding too much complexity to our pytest setup via markers in order to run the correct subset of tests. The logic alone for tests/benchmark/conftest.py is quite complex and then we overlay the logic from tests/benchmark/stateful/conftest.py on top of this.

I won't block this, but I think the requirement is actually quite simple and perhaps we can quickly consider an alternative approach before merging. My proposal would be to remove all of this conftest logic and use an approach that modifies pytest's testpaths at the CLI logic stage.

I.e., adding a mode flag, None by default (=just fill regular tests), but that can be set to "benchmark" or "stateful".

uv run fill --mode=benchmark

Then, we add (simpler) logic in src/cli/pytest_commands to modify fill and execute's ini config from (and others...):

* https://github.com/LouisTsai-Csie/execution-spec-tests/blob/7e4d3e214252fa878a7ac5c381fcab0d48d1c40a/src/cli/pytest_commands/pytest_ini_files/pytest-fill.ini#L5
  https://github.com/LouisTsai-Csie/execution-spec-tests/blob/7e4d3e214252fa878a7ac5c381fcab0d48d1c40a/src/cli/pytest_commands/pytest_ini_files/pytest-execute.ini#L5

to overwite testpaths with the relevant path (needs a well-organized structure underneath ./tests/benchmarks/). For this, we don't modify or add new ini files, but rather modify it using this functionality:

-o testpaths=tests/benchmark/stateful

From pytest --help:

  -o OVERRIDE_INI, --override-ini=OVERRIDE_INI
                        Override ini option with "option=value" style, e.g. `-o
                        xfail_strict=True -o cache_dir=cache`.

For normal filling mode (no benchmarking), we can append:

uv run pytest --help | grep ignore
  --ignore=path         Ignore path during collection (multi-allowed)

to ignore ./tests/benchmarks in regular filling.

Hey @LouisTsai-Csie, tried to add a clearer formulation here:

• feat(fill,execute,benchmark): test selection via pytest's testpaths instead of markers via --mode #2208

As we discussed, happy to merge this as-is if there are other priorities and tackle the simpler solution later.

[marioevz]

I think this is great, I agree there's a better approach but I would like for this to get merged so it's not lost in the Weld transition.

[marioevz]

We just need to move the file under benchmark/stateful/bloatnet and the marker would be applied!

I'm rebasing and moving the files to the new folder before merging 👍

[marioevz]

to overwite testpaths with the relevant path (needs a well-organized structure underneath ./tests/benchmarks/). For this, we don't modify or add new ini files, but rather modify it using this functionality:

I'm not super sure if this will seamlessly work when we transition to the EELS repo, I think it would work but I would defer implementing this approach until we move the code.

[danceratopz]

Thanks @LouisTsai-Csie, one of us can update the implementation to the approach in #2208 after the Weld!

to overwite testpaths with the relevant path (needs a well-organized structure underneath ./tests/benchmarks/). For this, we don't modify or add new ini files, but rather modify it using this functionality:

I'm not super sure if this will seamlessly work when we transition to the EELS repo, I think it would work but I would defer implementing this approach until we move the code.

It should be as simple as updating the specified testpaths, but yes, it's one more thing that will break / to remember. Let's get this merged!