How to build a mlflow server in cloud

We can work together in data with MLFlow

In the last post (medium) about mlflow we see just a little description about mlops and the module of mlflow tracking. In this post i’m gonna write about how you can work in a centralized way in your organization.

In the work environment you always need to share your work with your coworkers, if you need some help, share some knowledge or even a presentation for the c-suite. As you read in the last post, mlflow brings help with the ML lifecycle in your organization, and gives us a possibility to manage the chaos when we work with ML or AI.

So, if you are introducing yourself in the mlops world, this post is probably gonna be very useful. In this video we can get the idea of how to configure the cloud server (vm) to work with this, and I’m gonna put some instructions to configure the remote server.

What i’m talking about

As the mlflow website describes, MLflow runs can be recorded to local files, to a SQLAlchemy compatible database, or remotely to a tracking server. MLflow artifacts can be persisted to local files and a variety of remote file storage solutions. For storing runs and artifacts, MLflow uses two components for storage: backend store and artifact store. While the backend store persists MLflow entities (runs, parameters, metrics, tags, notes, metadata, etc), the artifact store persists artifacts (files, models, images, in-memory objects, or model summary, etc).

So, in the video, present how we can build a vm and work together in different environments and get lots of metrics so we can compare and improve our models in a centralized way. This differs a little from github, because in the data world, not only you work with code: you have a lot of metadata, hyperparameters and more

I work a little in the security layer, just enough to have a user and password and some kind of login. I don’t recommended to put this in production or even dev, but only at your own risk, but if you do this, at least configure your ip to only access to the vm.

You can use your shell o the google cloud shell to run this commands

gcloud compute instances create mlflow-log-server \
--machine-type n2-standard-2 \
--zone us-central1-a \
--tags mlflow-log-server \
--metadata startup-script='#! /bin/bash
sudo apt update
sudo apt-get -y install tmux
echo Installing python3-pip
sudo apt install -y python3-pip
export PATH="$HOME/.local/bin:$PATH"
pip3 install mlflow google-cloud-storage'

if you check, the metadata arg help us to install all the components necessary for mlflow server runs

gcloud compute firewall-rules create mlflow-log-server \
--direction=INGRESS --priority=999 --network=default \
--action=ALLOW --rules=tcp:5000 --source-ranges=0.0.0.0/0 \
--target-tags=mlflow-server

With this you can create a firewall rule.

This happens in the vm

mlflow --version
[/python]
[python]
mlflow server \
--backend-store-uri sqlite:///mlflow.db \
--default-artifact-root gs://<bucket> \
--host <ip-internal>

# install nginx in your vm

sudo apt-get install nginx apache2-utils

# create users

sudo htpasswd -c /etc/nginx/.htpasswd root # you can change
sudo htpasswd /etc/nginx/.htpasswd user # you can change

# open a nano editor and modify the nginx configuration

sudo nano /etc/nginx/sites-enabled/default
#just add the lines in the location section
server {
location / {

proxy_pass http://localhost:5000;
auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
#control x to exit
#Restart the nginx 
sudo service nginx restart
# Run in the vm
mlflow server \
--backend-store-uri sqlite:///mlflow.db \
--default-artifact-root gs://<bucket> \
--host localhost

With this you can connect with the <remote-ip> of your vm and check the experimentations

All the work you can do is the same as in the last post, the modification of the script is in the github link.

Thanks for reading, this post is for a series of posts about mlflow.