Common Motivation. Active development of Filecoin protocol and its implementation led to significant amount of various software components creation. Currently there are two major technological stacks / programming languages used: Rust and Go. Rust codebase contains 40 GitHub repositories and Go codebase contains 75 GitHub repositories within Filecoin-project organisation. Software components are developed and maintained by a number of distributed teams asynchronously. In such circumstances we need specific system that could guarantee some form of quality:

• Ideally any arbitrary change to any software component in the stack should not break build of high-level software components - Filecoin nodes;
• Ideally Filecoin build breaking that has been caused due to a change in some of external dependencies (of any component in the stack) should be alarmed promptly to correspondent team of developers;
• Correctness of proofs tests from https://github.com/filecoin-project/rust-fil-proofs;
• Ideally any arbitrary change to any software component should not reduce performance, unless it is considered by business goal;
• Ideally any arbitrary change to any software component should not decrease test coverage of this component;
• In case of performance issue, Buildbot should be able to make hints on where to start looking for a problem - either in proofs stack or in high-level caller code;
• TODO adding more requirements;

Prototyping. As soon as it is difficult to suggest optimal solution for such reporting system, considering huge amount of existing software (written on 2 different programming languages) it is worth to start with some prototype that will cover some limited amount of existing software. For this purpose a limited number of Rust components are selected: https://github.com/filecoin-project/merkletree and all Rust high-level crates that utilise it. Their location is https://github.com/filecoin-project/rust-fil-proofs repository. Schematically this can be visualised as following:

https://miro.com/app/board/uXjVO6IqRQo=/

For simplicity, initial implementation of the prototype will cover following requirements: 1) email-alarming in case of failed build and executing tests of https://github.com/filecoin-project/merkletree repository; 2) If https://github.com/filecoin-project/merkletree is “correct” (it can be built and tests are successful), try to build https://github.com/filecoin-project/rust-fil-proofs and execute its tests, including big tests that cover 32Gib / 64Gib sectors used in Filecoin’s production. So system should support some form of subscribing - and once mentioned incompatible is detected, system should send reports to all subscribed emails. Additionally, system should support targeting specific branch. Obviously, by default it will target master, but this may be useful at least for verifying that functionality works as expected.

The common methodology for correctness evaluation is scheduled integration testing of https://github.com/filecoin-project/rust-fil-proofs components performed each 4 hours for fast tests and continuously - for slow tests. Cron daemon can be used for scheduling integration tests execution. Ansible can be used for automation of buildbot deployment.

Implementation details. All infrastructure code can be found in https://github.com/filecoin-project/proofs-buildbot. Initially build bot can be implemented in some ad-hoc manner. E.g. it can be set of (pre-deployed to our worker-gpu-7) bash scripts with scheduled execution that abstractly implements scheduling integration tests execution via three following workflows:

Workflow 1:

• Clone ‣ repository, try building it. If it fails, notify subscribed emails about failed build (this can prevent any sort of flaky codebase at some degree) and finish workflow with error (send report);
• Try executing https://github.com/filecoin-project/merkletree tests. If it fails, notify subscribed emails about failed tests and finish workflow with error (send report);