Skip to content

Hopsworks.ai Terraform Provider#

Managed.hopsworks.ai allows users to create and manage their clusters using the Hopsworks.ai terraform provider. In this guide, we first provide brief description on how to get started on AWS, AZURE, and GCP, then we show how to import an existing cluster to be managed by terraform.

Getting Started with AWS#

Complete the following steps to start using Hopsworks.ai Terraform Provider on AWS.

  1. Create a managed.hopsworks.ai API KEY as described in details here, and export the API KEY as follows 
    export HOPSWORKSAI_API_KEY=<YOUR_API_KEY>
  2. Download the proper Terraform CLI for your os from here.
  3. Install the AWS CLI and run aws configurre to configure your AWS credentials.

Example#

In this section, we provide a simple example to create a Hopsworks cluster on AWS along with all its required resources (ssh key, S3 bucket, and instance profile with the required permissions).

  1. In your terminal, run the following to create a demo directory and cd to it 
    mkdir demo
cd demo
  2. In this empty directory, create an empty file main.tf. Open the file and paste the following configurations to it then save it. 
    terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.16.0"
    }
    hopsworksai = {
      source = "logicalclocks/hopsworksai"
    }
  }
}

variable "region" {
  type    = string
  default = "us-east-2"
}

provider "aws" {
  region = var.region
}

provider "hopsworksai" {
}

# Create the required aws resources, an ssh key, an s3 bucket, and an instance profile with the required Hopsworks permissions
module "aws" {
  source  = "logicalclocks/helpers/hopsworksai//modules/aws"
  region  = var.region
  version = "2.3.0"
}

# Create a cluster with no workers
resource "hopsworksai_cluster" "cluster" {
  name    = "tf-hopsworks-cluster"
  ssh_key = module.aws.ssh_key_pair_name

  head {
    instance_type = "m5.2xlarge"
  }

  aws_attributes {
    region               = var.region
    instance_profile_arn = module.aws.instance_profile_arn
    bucket {
      name = module.aws.bucket_name
    }
  }

  rondb {
    single_node {
      instance_type = "t3a.xlarge"
    }
  }

  autoscale {
    non_gpu_workers {
      instance_type       = "m5.2xlarge"
      disk_size           = 256
      min_workers         = 1
      max_workers         = 5
      standby_workers     = 0.5
      downscale_wait_time = 300
    }
  }

  open_ports {
    ssh = true
  }
}

output "hopsworks_cluster_url" {
  value = hopsworksai_cluster.cluster.url
}
  3. Initialize the terraform directory by running the following command 
    terraform init
  4. Now you can apply the changes to create all required resources 
    terraform apply

  5. Once terraform finishes creating the resources, it will output the url to the newly created cluster. Notice that for now, you have to navigate to your managed.hopsworks.ai dashboard to get your login credentials.

  6. After you finish working with the cluster, you can terminate it along with the other AWS resources using the following command 

    terraform destroy

Getting Started with AZURE#

Complete the following steps to start using Hopsworks.ai Terraform Provider on AZURE.

  1. Create a managed.hopsworks.ai API KEY as described in details here, and export the API KEY as follows 
    export HOPSWORKSAI_API_KEY=<YOUR_API_KEY>
  2. Download the proper Terraform CLI for your os from here.
  3. Install the AZURE CLI and run az login to configure your AZURE credentials.

Example#

In this section, we provide a simple example to create a Hopsworks cluster on AZURE along with all its required resources (ssh key, storage account, acr registry, and user assigned managed identity with the required permissions).

  1. In your terminal, run the following to create a demo directory and cd to it 
    mkdir demo
cd demo
  2. In this empty directory, create an empty file main.tf. Open the file and paste the following configurations to it then save it. 
    terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.8.0"
    }
    hopsworksai = {
      source = "logicalclocks/hopsworksai"
    }
  }
}

variable "resource_group" {
  type    = string
}

provider "azurerm" {
  features {}
  skip_provider_registration = true
}

provider "hopsworksai" {
}

data "azurerm_resource_group" "rg" {
  name = var.resource_group
}

# Create the required azure resources, an ssh key, a storage account, and an user assigned managed identity with the required Hopsworks permissions
module "azure" {
  source         = "logicalclocks/helpers/hopsworksai//modules/azure"
  resource_group = var.resource_group
  version        = "2.3.0"
}

# Create an ACR registry 
resource "azurerm_container_registry" "acr" {
  name                = replace(module.azure.storage_account_name, "storageaccount", "acr")
  resource_group_name = module.azure.resource_group
  location            = module.azure.location
  sku                 = "Premium"
  admin_enabled       = false
  retention_policy {
    enabled = true
    days    = 7
  }
}

# Create a cluster with no workers
resource "hopsworksai_cluster" "cluster" {
  name    = "tf-hopsworks-cluster"
  ssh_key = module.azure.ssh_key_pair_name

  head {
    instance_type = "Standard_D8_v3"
  }

 azure_attributes {
    location                       = module.azure.location
    resource_group                 = module.azure.resource_group
    user_assigned_managed_identity = module.azure.user_assigned_identity_name
    container {
      storage_account = module.azure.storage_account_name
    }
    acr_registry_name = azurerm_container_registry.acr.name
  }

  rondb {
    single_node {
      instance_type = "Standard_D4s_v4"
    }
  }

  autoscale {
    non_gpu_workers {
      instance_type       = "Standard_D8_v3"
      disk_size           = 256
      min_workers         = 1
      max_workers         = 5
      standby_workers     = 0.5
      downscale_wait_time = 300
    }
  }

  open_ports {
    ssh = true
  }
}

output "hopsworks_cluster_url" {
  value = hopsworksai_cluster.cluster.url
}
  3. Initialize the terraform directory by running the following command 
    terraform init
  4. Now you can apply the changes to create all required resources. Replace the placeholders with your Azure resource group 
    terraform apply -var="resource_group=<YOUR_RESOURCE_GROUP>"

  5. Once terraform finishes creating the resources, it will output the url to the newly created cluster. Notice that for now, you have to navigate to your managed.hopsworks.ai dashboard to get your login credentials.

  6. After you finish working with the cluster, you can terminate it along with the other AZURE resources using the following command 

    terraform destroy -var="resource_group=<YOUR_RESOURCE_GROUP>"

Getting Started with GCP#

Complete the following steps to start using Hopsworks.ai Terraform Provider on GCP.

  1. Create a managed.hopsworks.ai API KEY as described in details here, and export the API KEY as follows 
    export HOPSWORKSAI_API_KEY=<YOUR_API_KEY>
  2. Download the proper Terraform CLI for your os from here.
  3. Install the Google Cloud CLI and run gcloud init to configure your GCP credentials.

Example#

In this section, we provide a simple example to create a Hopsworks cluster on GCP along with all its required resources (Google storage bucket and service account with the required permissions).

  1. In your terminal, run the following to create a demo directory and cd to it 
    mkdir demo
cd demo
  2. In this empty directory, create an empty file main.tf. Open the file and paste the following configurations to it then save it. 
    terraform {
  required_version = ">= 0.14.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "5.13.0"
    }
    hopsworksai = {
      source = "logicalclocks/hopsworksai"
    }
    time = {
      source  = "hashicorp/time"
      version = "0.10.0"
    }
  }
}

variable "region" {
  type    = string
  default = "europe-north1"
}

variable "project" {
  type = string
}

provider "google" {
  region  = var.region
  project = var.project
}

provider "hopsworksai" {
}

provider "time" {

}

# Create required google resources, a storage bucket and an service account with the required hopsworks permissions
data "hopsworksai_gcp_service_account_custom_role_permissions" "service_account" {

}

resource "google_project_iam_custom_role" "service_account_role" {
  role_id     = "tf.HopsworksAIInstances"
  title       = "Hopsworks AI Instances"
  description = "Role that allows Hopsworks AI Instances to access resources"
  permissions = data.hopsworksai_gcp_service_account_custom_role_permissions.service_account.permissions
}

resource "google_service_account" "service_account" {
  account_id   = "tf-hopsworks-ai-instances"
  display_name = "Hopsworks AI instances"
  description  = "Service account for Hopsworks AI instances"
}

resource "google_project_iam_binding" "service_account_role_binding" {
  project = var.project
  role    = google_project_iam_custom_role.service_account_role.id

  members = [
    google_service_account.service_account.member
  ]
}

resource "google_storage_bucket" "bucket" {
  name          = "tf-hopsworks-bucket"
  location      = var.region
  force_destroy = true
}

resource "time_sleep" "wait_60_seconds" {
  depends_on      = [google_project_iam_binding.service_account_role_binding]
  create_duration = "60s"
}

# Create a simple cluster with two workers with two different configuration
data "google_compute_zones" "available" {
  region = var.region
}

locals {
  zone = data.google_compute_zones.available.names.0
}

resource "hopsworksai_cluster" "cluster" {
  name = "tf-cluster"

  head {
    instance_type = "e2-standard-8"
  }

  gcp_attributes {
    project_id            = var.project
    region                = var.region
    zone                  = local.zone
    service_account_email = google_service_account.service_account.email
    bucket {
      name = google_storage_bucket.bucket.name
    }
  }

  rondb {
    single_node {
      instance_type = "e2-highmem-4"
    }
  }

  autoscale {
    non_gpu_workers {
      instance_type       = "e2-standard-8"
      disk_size           = 256
      min_workers         = 1
      max_workers         = 5
      standby_workers     = 0.5
      downscale_wait_time = 300
    }
  }

  open_ports {
    ssh = true
  }

  # waiting for 60 seconds after service account permissions has been granted 
  # to avoid permissions validation failure on hopsworks when creating the cluster 
  depends_on = [time_sleep.wait_60_seconds]
}

output "hopsworks_cluster_url" {
  value = hopsworksai_cluster.cluster.url
}
  3. Initialize the terraform directory by running the following command 
    terraform init
  4. Now you can apply the changes to create all required resources. Replace the placeholders with your GCP project id 
    terraform apply -var="project=<YOUR_PROJECT_ID>"

  5. Once terraform finishes creating the resources, it will output the url to the newly created cluster. Notice that for now, you have to navigate to your managed.hopsworks.ai dashboard to get your login credentials.

  6. After you finish working with the cluster, you can terminate it along with the other GCP resources using the following command 

    terraform destroy -var="project=<YOUR_PROJECT_ID>"

Importing an existing cluster to terraform#

In this section, we show how to use terraform import to manage your existing Hopsworks cluster.

  • Step 1: In your managed.hopsworks.ai dashboard, choose the cluster you want to import to terraform, then go to the Details tab and copy the Id as shown in the figure below

Details tab
Click on the Details tab and copy the Id

  • Step 2: In your terminal, create an empty directory and cd to it. 

    mkdir import-demo
cd import-demo

  • Step 3: In this empty directory, create an empty file versions.tf. Open the file and paste the following configurations.

Note

Notice that you need to change these configurations depending on your cluster, in this example, the Hopsworks cluster reside in region us-east-2 on AWS.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.16.0"
    }
    hopsworksai = {
      source = "logicalclocks/hopsworksai"
    }
  }
}

provider "aws" {
  region = us-east-2
}

provider "hopsworksai" {
}

  • Step 4: Initialize the terraform directory by running the following command 

    terraform init

  • Step 5: Create another file main.tf. Open the file and paste the following configuration. 

    resource "hopsworksai_cluster" "cluster" {
}

  • Step 6: Import the cluster state using terraform import, in this step you need the cluster id from Step 1 (33ae7ae0-d03c-11eb-84e2-af555fb63565). 

    terraform import hopsworksai_cluster.cluster 33ae7ae0-d03c-11eb-84e2-af555fb63565

The output should be similar to the following snippet 

hopsworksai_cluster.cluster: Importing from ID "33ae7ae0-d03c-11eb-84e2-af555fb63565"...
hopsworksai_cluster.cluster: Import prepared!
  Prepared hopsworksai_cluster for import
hopsworksai_cluster.cluster: Refreshing state... [id=33ae7ae0-d03c-11eb-84e2-af555fb63565]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

  • Step 7: At that moment the local terraform state is updated, however, if we try to run terraform plan or terraform apply it will complain about missing configurations. The reason is that our local resource configuration in main.tf is empty, we should populate it using the terraform state commands as shown below: 

    terraform show -no-color > main.tf

  • Step 8: If you try to run terraform plan again, the command will complain that the read-only attributes are set (Computed attributes) as shown below. The solution is to remove these attributes from the main.tf and retry again until you have no errors. 

    Error: Computed attributes cannot be set

  on main.tf line 3, in resource "hopsworksai_cluster" "cluster":
   3:     activation_state               = "stoppable"

Computed attributes cannot be set, but a value was set for "activation_state".


Error: Computed attributes cannot be set

  on main.tf line 6, in resource "hopsworksai_cluster" "cluster":
   6:     cluster_id                     = "33ae7ae0-d03c-11eb-84e2-af555fb63565"

Computed attributes cannot be set, but a value was set for "cluster_id".


Error: Computed attributes cannot be set

  on main.tf line 7, in resource "hopsworksai_cluster" "cluster":
   7:     creation_date                  = "2021-06-18T15:51:07+02:00"

Computed attributes cannot be set, but a value was set for "creation_date".


Error: Invalid or unknown key

  on main.tf line 8, in resource "hopsworksai_cluster" "cluster":
   8:     id                             = "33ae7ae0-d03c-11eb-84e2-af555fb63565"



Error: Computed attributes cannot be set

  on main.tf line 13, in resource "hopsworksai_cluster" "cluster":
  13:     start_date                     = "2021-06-18T15:51:07+02:00"

Computed attributes cannot be set, but a value was set for "start_date".


Error: Computed attributes cannot be set

  on main.tf line 14, in resource "hopsworksai_cluster" "cluster":
  14:     state                          = "running"

Computed attributes cannot be set, but a value was set for "state".


Error: Computed attributes cannot be set

  on main.tf line 17, in resource "hopsworksai_cluster" "cluster":
  17:     url                            = "https://33ae7ae0-d03c-11eb-84e2-af555fb63565.dev-cloud.hopsworks.ai/hopsworks/#!/"

Computed attributes cannot be set, but a value was set for "url".

  • Step 9: Once you have fixed all the errors, you should get the following output when running terraform plan. With that, you can proceed as normal to manage this cluster locally using terraform. 

    hopsworksai_cluster.cluster: Refreshing state... [id=33ae7ae0-d03c-11eb-84e2-af555fb63565]

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Next Steps#