Limiting AWS permissions#
Hopsworks.ai requires a set of permissions to be able to manage resources in the user’s AWS account. By default, these permissions are set to easily allow a wide range of different configurations and allow us to automate as many steps as possible. While we ensure to never access resources we shouldn’t, we do understand that this might not be enough for your organization or security policy. This guide explains how to lock down AWS permissions following the IT security policy principle of least privilege allowing Hopsworks.ai to only access resources in a specific VPC.
Limiting the cross-account role permissions#
Step 1: Create a VPC#
To restrict Hopsworks.ai from accessing resources outside of a specific VPC, you need to create a new VPC
connected to an Internet Gateway. This can be achieved in the AWS Management Console following this guide:
Create the VPC.
The option VPC with a Single Public Subnet from the Launch VPC Wizard should work out of the box.
Alternatively, an existing VPC such as the default VPC can be used and Hopsworks.ai will be restricted to this VPC.
Note the VPC ID of the VPC you want to use for the following steps.
Note
Make sure you enable DNS hostnames for your VPC
After you have created the VPC either Create a Security Group or use VPC's default. Make sure that the VPC allow the following traffic.
Inbound traffic#
The Security Group and/or Network ACLs
need to be configured so that at least port 80 is reachable from the internet otherwise you will have to use self signed certificate in your Hopsworks cluster.
It is imperative the Security Group allows Inbound traffic from any Instance within the same Security Group in any (TCP) port. All VMs of the Cluster should be able to communicate with each other.
Outbound traffic#
Clusters created on Hopsworks.ai need to be able to send http requests to api.hopsworks.ai. The api.hopsworks.ai domain use a content delivery network for better performance. This result in the impossibility to predict which IP the request will be sent to. If you require a list of static IPs to allow outbound traffic from your security group, use the static IPs option during cluster creation.
Similar to Inbound traffic, the Security Group in place must allow Outbound traffic in any (TCP) port towards any VM withing the same Security Group.
Note
If you intend to use the managed users option on your Hopsworks cluster you should also allow outbound traffic to cognito-idp.us-east-2.amazonaws.com and managedhopsworks-prod.auth.us-east-2.amazoncognito.com.
Step 2: Create an instance profile#
You need to create an instance profile that will identify all instances started by Hopsworks.ai. Follow this guide to create a role to be used by EC2 with no permissions attached: Creating a Role for an AWS Service (Console). Take note of the ARN of the role you just created.
You will need to add permissions to the instance profile to give access to the S3 bucket where Hopsworks will store its data. For more details about these permissions check our guide here. Check bellow for more information on restricting the permissions given the instance profile.
Step 3: Set permissions of the cross-account role#
During the account setup for Hopsworks.ai, you were asked to create and provide a cross-account role. If you don’t remember which role you used then you can find it in Settings/Account Settings in Hopsworks.ai. Edit this role in the AWS Management Console and overwrite the existing inline policy with the following policy.
Note that you have to replace [INSTANCE_PROFILE_NAME] and [VPC_ID] for multiple occurrences in the given policy.
If you want to learn more about how this policy works check out: How to Help Lock Down a User’s Amazon EC2 Capabilities to a Single VPC.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "NonResourceBasedPermissions",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeVpcs",
"ec2:DescribeVolumes",
"ec2:DescribeSubnets",
"ec2:DescribeKeyPairs",
"ec2:DescribeInstanceStatus",
"iam:ListInstanceProfiles",
"ec2:DescribeSecurityGroups",
"ec2:DescribeVpcAttribute",
"ec2:DescribeRouteTables"
],
"Resource": "*"
},
{
"Sid": "IAMPassRoleToInstance",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/[INSTANCE_PROFILE_NAME]"
},
{
"Sid": "EC2RunInstancesOnlyWithGivenRole",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"ArnLike": {
"ec2:InstanceProfile": "arn:aws:iam::*:instance-profile/[INSTANCE_PROFILE_NAME]"
}
}
},
{
"Sid": "EC2RunInstancesOnlyInGivenVpc",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:subnet/*",
"Condition": {
"ArnLike": {
"ec2:vpc": "arn:aws:ec2:*:*:vpc/[VPC_ID]"
}
}
},
{
"Sid": "AllowInstanceActions",
"Effect": "Allow",
"Action": [
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:StartInstances",
"ec2:CreateTags",
"ec2:AssociateIamInstanceProfile"
],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"ArnLike": {
"ec2:InstanceProfile": "arn:aws:iam::*:instance-profile/[INSTANCE_PROFILE_NAME]"
}
}
},
{
"Sid": "RemainingRunInstancePermissions",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:*:*:volume/*",
"arn:aws:ec2:*::image/*",
"arn:aws:ec2:*::snapshot/*",
"arn:aws:ec2:*:*:network-interface/*",
"arn:aws:ec2:*:*:key-pair/*",
"arn:aws:ec2:*:*:security-group/*"
]
},
{
"Sid": "EC2VpcNonResourceSpecificActions",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup"
],
"Resource": "*",
"Condition": {
"ArnLike": {
"ec2:vpc": "arn:aws:ec2:*:*:vpc/[VPC_ID]"
}
}
},
{
"Sid": "EC2BackupCreation",
"Effect": "Allow",
"Action": [
"ec2:RegisterImage",
"ec2:DeregisterImage",
"ec2:DescribeImages",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot",
"ec2:DescribeSnapshots"
],
"Resource": "*"
}
]
}
Step 4: Create your Hopsworks instance#
You can now create a new Hopsworks instance in Hopsworks.ai by selecting the configured instance profile, VPC and security group during instance configuration. Selecting any other VPCs or instance profiles will result in permissions errors.
Step 5: Supporting multiple VPCs#
The policy can be extended to give Hopsworks.ai access to multiple VPCs. See: Creating a Condition with Multiple Keys or Values.
Backup permissions#
The following permissions are only needed for the backup feature. You can remove them if you are not going to create backups or if you do not have access to this Enterprise feature.
{
"Sid": "EC2BackupCreation",
"Effect": "Allow",
"Action": [
"ec2:RegisterImage",
"ec2:DeregisterImage",
"ec2:DescribeImages",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot",
"ec2:DescribeSnapshots"
],
"Resource": "*",
}
Other removable permissions#
The following permissions are needed to give an early warning if your VPC and security groups are badly configured. You can remove them if you don't need it.
"ec2:DescribeVpcAttribute",
"ec2:DescribeRouteTables"
{
"Sid": "EC2VpcNonResourceSpecificActions",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup"
],
"Resource": "*",
"Condition": {
"ArnLike": {
"ec2:vpc": "arn:aws:ec2:*:*:vpc/[VPC_ID]"
}
}
}
Limiting the instance profile permissions#
Backups#
If you do not intend to take backups or if you do not have access to this Enterprise feature you can remove the permissions that are only used by the backup feature when configuring instance profile permissions . For this remove the following permissions from the instance profile:
"S3:PutLifecycleConfiguration",
"S3:GetLifecycleConfiguration",
"S3:PutBucketVersioning",
"S3:ListBucketVersions",
"S3:DeleteObjectVersion",
CloudWatch Logs#
Hopsworks put its logs in Amazon CloudWatch so that you can access them without having to ssh into the machine. If you are not interested in this feature you can remove the following from your instance profile policy:
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ec2:DescribeVolumes",
"ec2:DescribeTags",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
}
Upgrade permissions#
Removing upgrade permissions#
If you do not intend to upgrade your cluster to newer versions of Hopsworks, then you can remove the upgrade permissions statement from the instance profile that you have created here. For this remove the following statement from your instance profile
{
"Sid": "UpgradePermissions",
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes",
"ec2:DetachVolume",
"ec2:AttachVolume",
"ec2:ModifyInstanceAttribute"
],
"Resource": "*"
}
Limiting upgrade permissions#
You can use tags to restrict the upgrade permissions to only the resources created for your cluster. For this attach a tag to your cluster during the cluster creation. Then, replace the upgrade permissions with the following policy instead. First you need to replace REGION and ACCOUNT with your region and account where you run your cluster, then replace HEAD_NODE_INSTANCE_ID with your aws instance id of the head node and HEAD_NODE_VOLUME_ID with the volume id attached to the head node, and finally replace the TAG_KEY and TAG_VALUE with your the tag name and value that is used with your cluster.
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"AllowAttachVolumeForUpgrade",
"Effect":"Allow",
"Action":"ec2:AttachVolume",
"Resource":"arn:aws:ec2:REGION:ACCOUNT:volume/HEAD_NODE_VOLUME_ID"
},
{
"Sid":"AllowAttachVolumeForUpgradeOnlyTaggedInstance",
"Effect":"Allow",
"Action":"ec2:AttachVolume",
"Resource":"arn:aws:ec2:REGION:ACCOUNT:instance/*",
"Condition":{
"StringEquals":{
"ec2:ResourceTag/TAG_KEY":"TAG_VALUE"
}
}
},
{
"Sid":"AllowDetachVolumeForUpgrade",
"Effect":"Allow",
"Action":"ec2:DetachVolume",
"Resource":"arn:aws:ec2:REGION:ACCOUNT:volume/HEAD_NODE_VOLUME_ID"
},
{
"Sid":"AllowDetachVolumeForUpgradeOnlyTaggedInstance",
"Effect":"Allow",
"Action":"ec2:DetachVolume",
"Resource":"arn:aws:ec2:REGION:ACCOUNT:instance/*",
"Condition":{
"StringEquals":{
"ec2:ResourceTag/TAG_KEY":"TAG_VALUE"
}
}
},
{
"Sid":"AllowModifyInstanceAttributeForUpgrade",
"Effect":"Allow",
"Action":"ec2:ModifyInstanceAttribute",
"Resource":[
"arn:aws:ec2:REGION:ACCOUNT:instance/HEAD_NODE_INSTANCE_ID",
"arn:aws:ec2:REGION:ACCOUNT:volume/HEAD_NODE_VOLUME_ID"
]
},
{
"Sid":"AllowDescribeVolumesForUpgrade",
"Effect":"Allow",
"Action":"ec2:DescribeVolumes",
"Resource":"*"
}
]
}