Set up Self-hosted Environments on AWS
This feature is currently in private beta. If you wish to participate, you must first request access. A member of our team will follow up to discuss the steps to get the feature enabled.
Meroxa does the heavy-lifting required to deploy and run a Self-hosted Environment in your Amazon Web Services (AWS) cloud account.
This guide will take you through steps to get started with Self-hosted Environments on AWS. Including a list of supported AWS regions, how to create a new IAM Policy, and how to create a new IAM User.
The following steps assume you meet all the requirements to get started with access and privileges to create IAM users with a set policy. If you do not, you will need to ask your AWS administrator to perform these steps.
Supported AWS Regions
Self-hosted Environments may only be provisioned in the following Amazon Web Services (AWS) regions:
us-east-1(N. Virginia)us-east-2(Ohio)us-west-2(Oregon)ap-northeast-1(Tokyo)eu-central-1(Frankfurt)
Create a new IAM Policy
- AWS Console
- AWS CLI
- Log in to your AWS Console.
- Go to your IAM dashboard.
- In the Policies dashboard, create a new policy using the permissions defined in Policy Configuration below.
- Configure the AWS CLI to log in to your AWS account.
- Save to a
.jsonfile the Policy Configuration defined below. - Set the following environment variables:
export MEROXA_AWS_POLICY=<policy_name> export MEROXA_AWS_POLICY_PATH=<path_to_file>- Replace
<policy_name>with the desired name for the policy. - Replace
<path_to_file>with the path to the file created in the previous step.
- Replace
- Run
aws iam create-policy --policy-name $MEROXA_AWS_POLICY --policy-document file://$MEROXA_AWS_POLICY_PATH.
Policy Configuration
This policy enables the IAM User to take the actions required to successfully provision an environment in your Amazon Web Services (AWS) cloud account. Remember to give this policy an identifiable name as it will be to the IAM user you create. If you have any questions around what is required, please don't hesitate to contact us.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "cloudformation",
"Effect": "Allow",
"Action": [
"cloudformation:*"
],
"Resource": [
"arn:aws:cloudformation:*:*:stack/meroxa-vpc-*/*",
"arn:aws:cloudformation:*:*:stack/meroxa-eks-*/*",
"arn:aws:cloudformation:*:*:stack/meroxa-deploy-custom-eks-resource/*",
"arn:aws:cloudformation:*:*:stack/meroxa-msk-*/*"
]
},
{
"Sid": "cloudwatch",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DeleteLogGroup",
"logs:DeleteLogStream",
"logs:DeleteRetentionPolicy",
"logs:DescribeLogGroups",
"logs:FilterLogEvents",
"logs:GetLogEvents",
"logs:GetLogGroupFields",
"logs:PutLogEvents",
"logs:PutRetentionPolicy",
"logs:TagLogGroup",
"logs:UntagLogGroup"
],
"Resource": [
"arn:aws:logs:*:*:log-group:*:log-stream:*",
"arn:aws:logs:*:*:log-group:/meroxa/msk/meroxa-msk-*"
]
},
{
"Sid": "ecr",
"Effect": "Allow",
"Action": [
"ecr:*"
],
"Resource": [
"arn:aws:ecr:*:*:repository/meroxa-eks-ecr-*"
]
},
{
"Sid": "eks0",
"Effect": "Allow",
"Action": [
"eks:*"
],
"Resource": [
"arn:aws:eks:*:*:nodegroup/*/meroxa-default-nodegroup/*"
]
},
{
"Sid": "eks1",
"Effect": "Allow",
"Action": [
"eks:CreateNodegroup",
"eks:DeleteCluster",
"eks:DescribeCluster",
"eks:ListNodegroups",
"eks:TagResource",
"eks:UntagResource",
"eks:UpdateClusterConfig",
"eks:UpdateClusterVersion",
"eks:*Addon"
],
"Resource": [
"arn:aws:eks:*:*:cluster/*",
"arn:aws:eks:*:*:addon/*"
]
},
{
"Sid": "iam0",
"Effect": "Allow",
"Action": [
"iam:SimulatePrincipalPolicy",
"iam:GetUserPolicy",
"iam:ListAttachedUserPolicies"
],
"Resource": [
"arn:aws:iam::*:user/*"
]
},
{
"Sid": "iam1",
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:CreateRole",
"iam:CreateServiceLinkedRole",
"iam:DeleteRole",
"iam:DeleteRolePermissionsBoundary",
"iam:DeleteRolePolicy",
"iam:DeleteServiceLinkedRole",
"iam:DetachRolePolicy",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:GetServiceLinkedRoleDeletionStatus",
"iam:ListAttachedRolePolicies",
"iam:ListRolePolicies",
"iam:PassRole",
"iam:PutRolePermissionsBoundary",
"iam:PutRolePolicy",
"iam:TagRole",
"iam:UntagRole",
"iam:UpdateAssumeRolePolicy",
"iam:UpdateRole",
"iam:CreateRole",
"iam:CreateOpenIDConnectProvider",
"iam:DeleteOpenIDConnectProvider",
"iam:GetOpenIDConnectProvider",
"iam:TagOpenIDConnectProvider"
],
"Resource": [
"arn:aws:iam::*:policy/*",
"arn:aws:iam::*:role/meroxa-eks-msk-*",
"arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup",
"arn:aws:iam::*:role/meroxa-eks-add-on-*",
"arn:aws:iam::*:oidc-provider/*"
]
},
{
"Sid": "s3",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::meroxa-config-*",
"arn:aws:s3:::meroxa-source-*"
]
},
{
"Sid": "all",
"Effect": "Allow",
"Action": [
"ec2:AllocateAddress",
"ec2:CreateDhcpOptions",
"ec2:AssociateAddress",
"ec2:AssociateDhcpOptions",
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateEgressOnlyInternetGateway",
"ec2:CreateInternetGateway",
"ec2:CreateNatGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:CreateVpcEndpoint",
"ec2:DeleteDhcpOptions",
"ec2:DeleteEgressOnlyInternetGateway",
"ec2:DeleteInternetGateway",
"ec2:DeleteNatGateway",
"ec2:DeleteNetworkInterface",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteTags",
"ec2:DeleteVpc",
"ec2:DeleteVpcEndpoints",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcs",
"ec2:DetachInternetGateway",
"ec2:DetachNetworkInterface",
"ec2:DisassociateAddress",
"ec2:DisassociateRouteTable",
"ec2:ModifySecurityGroupRules",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:ReleaseAddress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress",
"ec2:UpdateSecurityGroupRuleDescriptionsEgress",
"ec2:UpdateSecurityGroupRuleDescriptionsIngress",
"ecr:GetAuthorizationToken",
"eks:CreateCluster",
"kms:CancelKeyDeletion",
"kms:CreateGrant",
"kms:CreateKey",
"kms:Decrypt",
"kms:DescribeKey",
"kms:DisableKey",
"kms:EnableKey",
"kms:Encrypt",
"kms:GenerateDataKey*",
"kms:GetKeyPolicy",
"kms:GetKeyRotationStatus",
"kms:GetPublicKey",
"kms:ListKeys",
"kms:ListResourceTags",
"kms:PutKeyPolicy",
"kms:ReEncrypt*",
"kms:RetireGrant",
"kms:RevokeGrant",
"kms:ScheduleKeyDeletion",
"kms:TagResource",
"kms:UntagResource",
"kms:UpdateKeyDescription",
"kafka:CreateCluster",
"kafka:CreateConfiguration",
"kafka:DeleteCluster",
"kafka:DeleteConfiguration",
"kafka:DescribeCluster",
"kafka:DescribeConfiguration",
"kafka:DescribeConfigurationRevision",
"kafka:GetBootstrapBrokers",
"kafka:ListClusters",
"kafka:ListConfigurationRevisions",
"kafka:ListConfigurations",
"kafka:ListTagsForResource",
"kafka:RebootBroker",
"kafka:TagResource",
"kafka:UntagResource",
"kafka:UpdateBrokerCount",
"kafka:UpdateBrokerStorage",
"kafka:UpdateBrokerType",
"kafka:UpdateClusterConfiguration",
"kafka:UpdateClusterKafkaVersion",
"kafka:UpdateConfiguration",
"logs:CancelExportTask",
"logs:CreateLogDelivery",
"logs:DeleteDestination",
"logs:DeleteLogDelivery",
"logs:DeleteQueryDefinition",
"logs:DeleteResourcePolicy",
"logs:DescribeDestinations",
"logs:DescribeExportTasks",
"logs:DescribeQueries",
"logs:DescribeQueryDefinitions",
"logs:DescribeResourcePolicies",
"logs:GetLogDelivery",
"logs:GetLogRecord",
"logs:GetQueryResults",
"logs:ListLogDeliveries",
"logs:PutDestination",
"logs:PutDestinationPolicy",
"logs:PutQueryDefinition",
"logs:PutResourcePolicy",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:UpdateLogDelivery",
"servicequotas:GetServiceQuota"
],
"Resource": "*"
}
]
}
Create a new IAM User
- AWS Console
- AWS CLI
- Go to your IAM dashboard.
- In the Users dashboard, create a new user.
- Assign the IAM policy you created from the previous step to the newly created user.
- Generate access keys for the newly created user and store your Access key ID and Secret access key somewhere safe and secure. You will need them when creating your Environment.
- Configure the AWS CLI to log in to your AWS account.
- Set the following environment variables
export MEROXA_AWS_POLICY=<policy_name> export MEROXA_AWS_USER=<user_name>- Replace
<policy_name>with the name for the IAM policy you created in the previous step. - Replace
<user_name>with the desired name for user.
- Replace
- Run the following commands:
aws iam create-user --user-name $MEROXA_AWS_USER export POLICYARN=$(aws iam list-policies --query "Policies[?PolicyName==\`${MEROXA_AWS_POLICY}\`].{ARN:Arn}" --output text) aws iam attach-user-policy --user-name $MEROXA_AWS_USER --policy-arn $POLICYARN aws iam create-access-key --user-name $MEROXA_AWS_USER- Store your AccessKeyId and SecretAccessKey generated by the last command somewhere safe and secure. You will need them when creating your Environment.
🎉 You now have an IAM User to create your Self-hosted Environment!
You can only use the IAM User used to create the VPC. You may still rotate your Secret Access Key and update your environment configuration. However, any newly generated IAM Users will not work with your Self-hosted Environment.