Scaling up#
If you run into limitations due to the instance types you chose during a cluster creation it is possible to scale up the instances to overcome these limitations.
Scaling up the workers#
If spark jobs are not starting in your cluster it may come from the fact that you don't have worker resources to run them. As workers are stateless the best way to solve this problem is to add new workers with enough resources to handle your job. Or to configure autoscalling to automatically add the workers when needed.
Scaling up the head node#
You may run into the need to scale up the head node for different reasons. For example:
- You are running a cluster without dedicated RonDB nodes and have a workload with a high demand on the online feature store.
- You are running a cluster without managed containers and want to run an important number of jupyter notebooks simultaneously.
While we are working on implementing a solution to add these features to an existing cluster you can use the following approach to run your head node on an instance with more vcores and memory to handle more load.
To scale up the head node you first have to stop your cluster.
Once the cluster is stopped you can go to the Details tab and click on the head node instance type.
This will open a new window. Select the type of instance you want to change to and click on Review and submit
Verify your choice and click on Modify
Note
If you set up your account with AWS in a period predating the introduction of this feature you may need to add the indicated permission to your hopsworks.ai role.
You can now start your cluster. The head node will be started on an instance type of the new type you chose.
Scaling up the RonDB nodes#
If you are running a cluster with dedicated RonDB nodes and have a workload with a high demand on the online feature store you may need to scale up the RonDB Datanodes and MySQLd nodes. For this stop the cluster.
Once the cluster is stopped you can go to the RonDB tab. To scale MySQLd or API nodes, click on the instance type for the node you want to scale up. To scale all datanodes, click on the Change button over their instance types. Datanodes cannot be scaled individually.
This will open a new window. Select the type of instance you want to change to and click on Review and submit
Verify your choice and click on Modify
Note
If you set up your account with AWS in a period predating the introduction of this feature you may need to add the indicated permission to your hopsworks.ai role.
You can now start your cluster. The nodes will be started on an instance type of the new type you chose.