Skip to main content

Kubeflow Cluster Setup

Current Configuration​

We have a fork of the Kubeflow Cluster distribution, based on V1.7.1 release. The kubeflow cluster setup is very sensitive to changes in GCP APIs. The Almanak fork provides a concise representation of what was deployed, for ease of maintenance of the current kubeflow cluster.

https://github.com/almanak-co/almanak-kubeflow-distribution

The fork can also be used to deploy new kubeflow clusters using the existing management cluster in prod kf-mgmt-clstr-2

Existing management cluster settings: https://github.com/almanak-co/almanak-kubeflow-distribution/blob/master/management/env.sh

Existing kubeflow cluster settings: https://github.com/almanak-co/almanak-kubeflow-distribution/blob/master/kubeflow/env.sh

Making changes to existing deployments / new deployments​

Changes in existing settings should be made to the env.sh of the respective cluster (See above section)

If a new kubeflow cluster is required, make a copy of the the kubeflow package in the git repo and change your working directory to point to it. Manually step through the shell commands provided in the deploy kubeflow script below to deploy.

To apply changes to existing kubeflow clusters or to deploy a new cluster, use the following script: https://github.com/almanak-co/almanak-kubeflow-distribution/blob/master/almanak_scripts/USE_ME_deploy_kubeflow_cluster.sh

To update configuration of the management cluster, use the following script: https://github.com/almanak-co/almanak-kubeflow-distribution/blob/master/almanak_scripts/USE_ME_gcp_deploy_management_cluster.sh

Providing a Service Account access to an existing Kubeflow namespace​

To create a role binding for the Service Account to be used by the Almanak KFP SDK to use an existing user namespace, use the following command:

kubectl create rolebinding <role-binding-name> \ --clusterrole=kubeflow-admin \ --user=<service account email address> \
 --namespace=<existing user namespace>