KALE (Kubeflow Automated pipeLines Engine) is a project that aims at simplifyingthe Data Science experience of deploying Kubeflow Pipelines workflows.
Kubeflow is a great platform for orchestrating complex workflows on topKubernetes and Kubeflow Pipeline provides the mean to create reusable componentsthat can be executed as part of workflows. The self-service nature of Kubeflowmake it extremely appealing for Data Science use, at it provides an easy accessto advanced distributed jobs orchestration, re-usability of components, JupyterNotebooks, rich UIs and more. Still, developing and maintaining Kubeflowworkflows can be hard for data scientists, who may not be experts in workingorchestration platforms and related SDKs. Additionally, data science ofteninvolve processes of data exploration, iterative modelling and interactiveenvironments (mostly Jupyter notebook).
Kale bridges this gap by providing a simple UI to define Kubeflow Pipelinesworkflows directly from you JupyterLab interface, without the need to change asingle line of code.
Read more about Kale and how it works in this Medium post:Automating Jupyter Notebook Deployments to Kubeflow Pipelines with Kale
Install the Kale backend from PyPI and the JupyterLab extension. You can find aset of curated Notebooks in theexamples repository
# install kale
pip install kubeflow-kale
# install jupyter lab
pip install "jupyterlab>=2.0.0,<3.0.0"
# install the extension
jupyter labextension install kubeflow-kale-labextension
# verify extension status
jupyter labextension list
# run
jupyter lab
To build images to be used as a NotebookServer in Kubeflow, refer to theDockerfile in the docker
folder.
Head over to FAQ to read about some known issues and some of thelimitations imposed by the Kale data marshalling model.
Create a new Python virtual environment with Python >= 3.6
. Then:
cd backend/
pip install -e .[dev]
# run tests
pytest -x -vv
The JupyterLab Python package comes with its own yarn wrapper, called jlpm
.While using the previously installed venv, install JupyterLab by running:
pip install "jupyterlab>=2.0.0,<3.0.0"
You can then run the following to install the Kale extension:
cd labextension/
# install dependencies from package.lock
jlpm install
# build extension
jlpm run build
# list installed jp extensions
jlpm labextension list
# install Kale extension
jlpm labextension install .
# for development:
# build and watch
jlpm run watch
# in another shell, run JupyterLab in watch mode
jupyter lab --no-browser --watch
This repository useshuskyto set up git hooks.
For husky
to function properly, you need to have yarn
installed and in yourPATH
. The reason that is required is that husky
is installed viajlpm install
and jlpm
is a yarn
wrapper. (Similarly, if it was installedusing the npm
package manager, then npm
would have to be in PATH
.)
Currently installed git hooks:
pre-commit
: Run a prettier check on staged files, usingpretty-quick