Last month the Production Engineering team embarked on a new project called Thunderdome to create a way to perform side by side experiments on IPFS Gateway implementations using real traffic.

This week we're shipping the first stage: an end to end experimental test bed with tooling to support it.

To bring this together we created:

• A cli and frontend called dealgood that replays traffic to each of the backend gateways in an experiment. It has a neat TUI that can be used to test interactively, but most of the time it's intended to be used in headless mode.
• A new environment for running experiments. We're currently using ECS but have some ideas to enable bring your own infra.
• Super simple experiment definitions: a dockerfile and some configuration overrides.
• Deployment and teardown of experiments via Terraform
• Some example experiments: peered vs non peered and comparison of tracing sampling levels

All this is open source, of course, and available at https://github.com/ipfs-shipyard/thunderdome

Here's what the dealgood TUI looks like:

And here's an Asciinema recording of a run of an experiment of a peered gateway vs a non-peered one. Asciinema renders the graph lines a little weirdly though!

https://asciinema.org/a/i23zeJu6MN7tU0mcDm623y3CP

Next up in our plan is grafana integration aiming to get metrics from every experiment immediately available in grafana as well as tracing support.

Follow our progress on the Prod Eng board: https://github.com/orgs/protocol/projects/17/views/2

More about Thunderdome including planned phases and timeline: https://www.notion.so/pl-strflt/Project-2-Thunderdome-9329254f11a441d586de9146f51aeedc