Skip to content

Getting started with Hopsworks.ai (Azure)#

Hopsworks.ai is our managed platform for running Hopsworks and the Feature Store in the cloud. It integrates seamlessly with third party platforms such as Databricks, SageMaker and KubeFlow. This guide shows how to set up Hopsworks.ai with your organization's Azure account.

Step 1: Connecting your Azure account#

Hopsworks.ai deploys Hopsworks clusters to your Azure account. To enable this, you have to create a service principal and a custom role for Hopsworks.ai granting access to either a subscription or resource group.

Step 1.1: Creating a service principal for Hopsworks.ai#

On Hopsworks.ai, go to Settings/Cloud Accounts and choose to Configure Azure:

Cloud account settings
Cloud account settings

Select Add subscription key:

Add subscription keys
Add subscription keys

The Azure account configuration will show you the required steps and permissions. Ensure that you have the Azure CLI installed Install the Azure CLI and are logged in Sign in with Azure CLI.

Copy the Azure CLI command from the first step and open a terminal:

Connect your Azure Account
Connect your Azure Account

Paste the command into the terminal and execute it:

Add service principal
Add service principal

At this point, you might get the following error message. This means that your Azure user does not have sufficient permissions to add the service principal. In this case, please ask your Azure administrator to add it for you or give you the required permissions.

Error

az ad sp create --id d4abcc44-2c40-40bd-9bba-986df591c28f

When using this permission, the backing application of the service principal being created must in the local tenant.

Step 1.2: Creating a custom role for Hopsworks.ai#

Proceed to the Azure Portal and open either a Subscription or Resource Group that you want to use for Hopsworks.ai. Select Add and choose Add custom role. Granting access to a Subscription will grant access to all Resource Groups in that Subscription. If you are uncertain if that is what you want, then start with a Resource Group.

Add custom role
Add custom role

Name the role and proceed to Assignable scopes:

Name custom role
Name custom role

Ensure the scope is set to the Subscription or Resource Group you want to use. You can change it here if required. Proceed to the JSON tab:

Review assignable scope
Review assignable scope

Select Edit and replace the actions part of the JSON with the one from Hopsworks.ai Azure account configuration workflow.

Note

If the access rights provided by Hopsworks.ai Azure account configuration workflow are too permissive, you can go to Limiting Azure permissions for more details on how to reduce the permissions.

Hopsworks.ai permission list
Hopsworks.ai permission list

Press Save, proceed to Review + create and create the role:

Update permission JSON
Update permission JSON

Step 1.3: Assigning the custom role to Hopsworks.ai#

Back in the Subscription or Resource Group overview, select Add and choose Add role assignment:

Add role assignment
Add role assignment

Choose the custom role you just created, select User, group, or service principal to Assign access to and select the hopsworks.ai service principal. Press Save:

Configure Hopsworks.ai as role assignment
Configure Hopsworks.ai as role assignment

Go back to the Hopsworks.ai Azure account configuration workflow and proceed to the next step. Copy the CLI command shown:

Configure subscription and tenant id
Configure subscription and tenant id

Paste the CLI command into your terminal and execute it. Note that you might have multiple entries listed here. If so, ensure that you pick the subscription that you want to use.

Show subscription and tenant id
Show subscription and tenant id

Copy the value of id and paste it into the Subscription id field on Hopsworks.ai. Go back to the terminal and copy the value of tenantId. Ensure to NOT use the tenantId under managedByTenants. Paste the value into the Tenant ID field on Hopsworks.ai and press Finish.

Congratulations, you have successfully connected you Azure account to Hopsworks.ai.

Store subscription and tenant id
Store subscription and tenant id

Step 2: Creating and configuring a storage#

The Hopsworks clusters deployed by hopsworks.ai store their data in a container in your Azure account. To enable this you need to create a storage account and a User Assigned Managed Identity to give the Hopsworks cluster access to the storage.

Step 2.1: Creating a User Assigned Managed Identity#

Proceed to the Azure Portal and open the Resource Group that you want to use for Hopsworks.ai. Click on Add.

Add to resource group
Add to resource group

Search for User Assigned Managed Identity and click on it.

Search User Assigned Managed Identity
Search User Assigned Managed Identity

Click on Create. Then, select the Location you want to use and name the identity. Click on Review + create. Finally click on Create.

Create a User Assigned Managed Identity
Create a User Assigned Managed Identity

Step 2.2: Creating a Storage account#

Proceed to the Azure Portal and open the Resource Group that you want to use for Hopsworks.ai. Click on Add.

Add to resource group
Add to resource group

Search for Storage account and click on it.

Search Storage Account Identity
Search Storage Account Identity

Click on Create, name your storage account, select the Location you want to use and click on Review + create. Finally click on Create.

Create a Storage Account
Create a Storage Account

Step 2.3: Give the Managed Identity access to the storage#

Proceed to the Storage Account you just created and click on Access Control (IAM) (1). Click on Add (2), then click on Add role assignment (3). In Role select Storage Blob Data Contributor (4). In Assign access to select User assigned managed identity (5). Select the identity you created in step 2.1 (6). Click on Save (7).

Add role assignment to storage
Add role assignment to storage

Step 3: Adding a ssh key to your resource group#

When deploying clusters, Hopsworks.ai installs a ssh key on the cluster's instances so that you can access them if necessary. For this purpose you need to add a ssh key to your resource group.

Proceed to the Azure Portal and open the Resource Group that you want to use for Hopsworks.ai. Click on Add.

Add to resource group
Add to resource group

Search for SSH Key and click on it. Click on Create. Then, name your key pair and choose between Generate a new key pair and Upload existing public key. Click on Review + create. Finally click on Create.

Create a SSH key
Add to resource group

Step 4: Deploying a Hopsworks cluster#

In Hopsworks.ai, select Create cluster:

Create a Hopsworks cluster
Create a Hopsworks cluster

Select the Location in which you want your cluster to run (1), name your cluster (2) and select the Resource Group (3) in which you created your storage account and user assigned managed identity (see above).

Select the Instance type (4) and Local storage (5) size for the cluster Head node.

Select the number of workers you want to start the cluster with (6). Select the Instance type (7) and Local storage size (8) for the worker nodes.

Note

It is possible to add or remove workers once the cluster is running.

Enter the name of the storage account (9) you created above in Azure Storage account name and name the container in which the data wil be stored in Azure Container name (10).

Note

You can choose to use a container already existing in your storage account by using the name of this container, but you need to first make sure that this container is empty.

Press Next (11):

General configuration
General configuration

Select the SSH key that you want to use to access cluster instances:

Choose SSH key
Choose SSH key

Select the User assigned managed identity that you created above:

Choose the User assigned managed identity
Choose the User assigned managed identity

Select the Virtual Network or choose to automatically create a new one:

Choose virtual network
Choose virtual network

Select the Subnet or choose to automatically create a new one:

Choose subnet
Choose subnet

Select the Security group or choose to automatically create a new one:

Choose security group
Choose security group

Choose the user management you want. Select Managed to manage users via Hopsworks.ai, LDAP to integrate with your organization's LDAP/ActiveDirectory server or Disabled to manage users manually from within Hopsworks:

Choose user management type
Choose user management type

Review all information and select Create:

Review cluster information
Review cluster information

The cluster will start. This will take a few minutes:

Booting Hopsworks cluster
Booting Hopsworks cluster

As soon as the cluster has started, you will be able to log in to your new Hopsworks cluster with the username and password provided. You will also be able to stop, restart or terminate the cluster.

Running Hopsworks cluster
Running Hopsworks cluster

Step 5: Outside Access to the Feature Store#

By default, only the Hopsworks REST API (and UI) is accessible by clients on external networks, like the Internet. To integrate with external platforms and access APIs for services such as the Feature Store, you have to open the service's ports.

Open ports by going to Services tab, selecting a service and pressing Update. This will update the Security Group attached to the Hopsworks cluster to allow incoming traffic on the relevant ports.

Outside Access to the Feature Store
Outside Access to the Feature Store

Step 4: Next steps#

Check out our other guides for how to get started with Hopsworks and the Feature Store: