Author: @Enrico Rotundo
Status: Discussion
Last Updated Date: 2022-12-31
We keep the provisioning of external services (e.g. Airflow) out of the scope of Bacalhau, instead, we provide instructions and a sample instance to play with.
On one hand, we’d like to provide a smooth UX where users do not need to deploy an Airflow instance themselves, and provide it directly via Bacalhau (e.g. by self-hosting Airflow in a Bacalhau job?).
On the other hand, Bacalhau is a computing platform and should refrain from entering the business of managing/provisioning third-party services. Furthermore, as described in Job Scope, Bacalhau should keep responsibilities separated and be concerned only with compute-related affairs.
The provisioning of an Airflow instance is not going to be performed by Bacalhau itself. We choose a bring-your-own-Airflow (BYOA 😁) approach and assume the user will either have a running Airflow instance or grab one of the many deployment options (SaaS, Helm charts, etc.) described in https://airflow.apache.org/ecosystem/.
We will provide the following: