This doc was originally posted on HackMD at this url.

Enrico, 2022-10-10

Kudos to everyone who worked on this epic before me - I took the liberty of copy/pasting some of your thoughts here.

Context

Bacalhau (currently v0.3.x) can take workloads composed by a single job but to execute a multi-step workload they have to manually submit and track each job, this (1) poses a significant burden on the user, (2) makes reproducing a pipeline difficult, (3) data origin can only be determined by manually logging each step and backtracking.

In the effort to offering broader support towards modern multi-step workloads, this document aims at designing a compelling pipelining feature that is user-friendly and allows for complex workloads.

In this context, a Pipeline is completely user-defined meaning they write a pipeline spec detailing what/how each step (i.e. a Bacalhau job) is related to one another.

This document is based on prior work:

• Job Pipelines as DAGs #190
• Support in the design of Bacalhau’s connection to external orchestration systems (e.g. Airflow) by polyphene.io
• Bacalhau Master Plan Roadmap - Sept. 2022
• https://docs.google.com/document/d/1Whyj-e17YW6VGILw2VB4ErwvjvtqDa2H2BSRcmo_mMo/edit#
• https://filecoinproject.slack.com/files/TEHTVS1L6/F0420QETMU3
• let me know if I missed anything that should be on this list!

Personas

• User: a computational scientist, interested in consuming jobs/pipelines. Can be a data engineer as well. In short, they're the bacalhau docker run cli users.
• Compute Provider (CP): a third-party willing to run a Bacalhau node on their infrastructure. In short, they're the bacalhau serve users. We consider this persona too because any new piece of infrastructure must be easily deployable.

User stories