Skip to main content

Set up Self-hosted Environments on AWS

BETA FEATURE

This feature is currently in beta. If you wish to participate, you must first request access. A member of our team will follow up to discuss the steps to get the feature enabled.

Meroxa does the heavy-lifting required to deploy and run a Self-hosted Environment in your Amazon Web Services (AWS) cloud account.

This guide will take you through steps to get started with Self-hosted Environments on AWS. Including a list of supported AWS regions, how to create a new IAM Policy, and how to create a new IAM User.

The following steps assume you meet all the requirements to get started with access and privileges to create IAM users with a set policy. If you do not, you will need to ask your AWS administrator to perform these steps.

Supported AWS Regions

Self-hosted Environments may only be provisioned in the following Amazon Web Services (AWS) regions:

  • us-east-1 (N. Virginia)
  • us-east-2 (Ohio)
  • us-west-2 (Oregon)
  • ap-northeast-1 (Tokyo)
  • eu-central-1 (Frankfurt)

Create a new IAM Policy

  1. Log in to your AWS Console.
  2. Go to your IAM dashboard.
  3. Under Access management, click Policies.
  4. In your Policies dashboard, click the button Create Policy.
  5. Next to the Visual editor tab, select the tab JSON.
  6. Copy and paste the Policy Configuration below.
  7. Click the button Next: Tags.
  8. You may skip the Add tags (optional) step and click the button Next: Review.
  9. Under Review policy provide an identifiable Name. You will add this policy to your IAM User later.
  10. Click the button Create policy.

You should now see your new policy in your Policies dashboard list with the type Customer managed.

Policy Configuration

This policy enables the IAM User to take the actions required to successfully provision an environment in your Amazon Web Services (AWS) cloud account. Remember to give this policy an identifiable name as it will be to the IAM user you create. If you have any questions around what is required, please don't hesitate to contact us.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:DeleteObject",
"s3:GetBucketPolicy",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::meroxa-config-*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"ec2:AllocateAddress",
"ec2:AssociateAddress",
"ec2:AssociateDhcpOptions",
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateDhcpOptions",
"ec2:CreateEgressOnlyInternetGateway",
"ec2:CreateInternetGateway",
"ec2:CreateNatGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:CreateVpcEndpoint",
"ec2:DeleteDhcpOptions",
"ec2:DeleteEgressOnlyInternetGateway",
"ec2:DeleteInternetGateway",
"ec2:DeleteNatGateway",
"ec2:DeleteNetworkInterface",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteTags",
"ec2:DeleteVpc",
"ec2:DeleteVpcEndpoints",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcs",
"ec2:DetachInternetGateway",
"ec2:DetachNetworkInterface",
"ec2:DisassociateAddress",
"ec2:DisassociateRouteTable",
"ec2:ModifySecurityGroupRules",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:ReleaseAddress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress",
"ec2:UpdateSecurityGroupRuleDescriptionsEgress",
"ec2:UpdateSecurityGroupRuleDescriptionsIngress"
],
"Resource": "*"
},{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStacks",
"cloudformation:ListStackResources",
"cloudformation:UpdateStack"
],
"Resource": [
"arn:aws:cloudformation:*:*:stack/meroxa-vpc-*/*",
"arn:aws:cloudformation:*:*:stack/meroxa-eks-*/*",
"arn:aws:cloudformation:*:*:stack/meroxa-deploy-custom-eks-resource/*",
"arn:aws:cloudformation:*:*:stack/meroxa-msk-*/*"
]
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"kafka:CreateCluster",
"kafka:CreateConfiguration",
"kafka:DeleteCluster",
"kafka:DeleteConfiguration",
"kafka:DescribeCluster",
"kafka:DescribeConfiguration",
"kafka:DescribeConfigurationRevision",
"kafka:GetBootstrapBrokers",
"kafka:ListClusters",
"kafka:ListConfigurationRevisions",
"kafka:ListConfigurations",
"kafka:ListTagsForResource",
"kafka:RebootBroker",
"kafka:TagResource",
"kafka:UntagResource",
"kafka:UpdateBrokerCount",
"kafka:UpdateBrokerStorage",
"kafka:UpdateBrokerType",
"kafka:UpdateClusterConfiguration",
"kafka:UpdateClusterKafkaVersion",
"kafka:UpdateConfiguration"
],
"Resource": "*"
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": [
"eks:CreateNodegroup",
"eks:DeleteCluster",
"eks:DescribeCluster",
"eks:ListNodegroups",
"eks:TagResource",
"eks:UntagResource",
"eks:UpdateClusterConfig",
"eks:UpdateClusterVersion"
],
"Resource": [
"arn:aws:eks:*:*:cluster/*"
]
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": [
"eks:DeleteNodegroup",
"eks:DescribeNodegroup",
"eks:ListNodegroups",
"eks:TagResource",
"eks:UntagResource",
"eks:UpdateNodegroupConfig",
"eks:UpdateNodegroupVersion"
],
"Resource": [
"arn:aws:eks:*:*:nodegroup/*/meroxa-default-nodegroup/*"
]
},
{
"Sid": "VisualEditor6",
"Effect": "Allow",
"Action": [
"eks:CreateCluster"
],
"Resource": "*"
},
{
"Sid": "VisualEditor7",
"Effect": "Allow",
"Action": [
"kms:CancelKeyDeletion",
"kms:CreateGrant",
"kms:CreateKey",
"kms:Decrypt",
"kms:DescribeKey",
"kms:DisableKey",
"kms:EnableKey",
"kms:Encrypt",
"kms:GenerateDataKey*",
"kms:GetKeyPolicy",
"kms:GetKeyRotationStatus",
"kms:GetPublicKey",
"kms:ListKeys",
"kms:PutKeyPolicy",
"kms:ReEncrypt*",
"kms:ReEncryptTo",
"kms:RetireGrant",
"kms:RevokeGrant",
"kms:ScheduleKeyDeletion",
"kms:TagResource",
"kms:UntagResource",
"kms:UpdateKeyDescription"
],
"Resource": "*"
},
{
"Sid": "VisualEditor8",
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:CreateRole",
"iam:CreateServiceLinkedRole",
"iam:DeleteRole",
"iam:DeleteRolePermissionsBoundary",
"iam:DeleteRolePolicy",
"iam:DeleteServiceLinkedRole",
"iam:DetachRolePolicy",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:GetUserPolicy",
"iam:GetServiceLinkedRoleDeletionStatus",
"iam:ListAttachedRolePolicies",
"iam:ListAttachedUserPolicies",
"iam:ListRolePolicies",
"iam:PassRole",
"iam:PutRolePermissionsBoundary",
"iam:PutRolePolicy",
"iam:TagRole",
"iam:UntagRole",
"iam:UpdateAssumeRolePolicy",
"iam:UpdateRole"
],
"Resource": [
"arn:aws:iam::*:policy/*",
"arn:aws:iam::*:role/meroxa-eks-msk-*",
"arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup"
]
},
{
"Sid": "VisualEditor9",
"Effect": "Allow",
"Action": ["iam:SimulatePrincipalPolicy"
],
"Resource": [
"arn:aws:iam::*:user/*"
]
},
{
"Sid": "VisualEditor10",
"Effect": "Allow",
"Action": [
"logs:CancelExportTask",
"logs:CreateLogDelivery",
"logs:DeleteDestination",
"logs:DeleteLogDelivery",
"logs:DeleteQueryDefinition",
"logs:DeleteResourcePolicy",
"logs:DescribeDestinations",
"logs:DescribeExportTasks",
"logs:DescribeQueries",
"logs:DescribeQueryDefinitions",
"logs:DescribeResourcePolicies",
"logs:GetLogDelivery",
"logs:GetLogRecord",
"logs:GetQueryResults",
"logs:ListLogDeliveries",
"logs:PutDestination",
"logs:PutDestinationPolicy",
"logs:PutQueryDefinition",
"logs:PutResourcePolicy",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:UpdateLogDelivery"
],
"Resource": "*"
},
{
"Sid": "VisualEditor11",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DeleteLogGroup",
"logs:DeleteLogStream",
"logs:DeleteRetentionPolicy",
"logs:DescribeLogGroups",
"logs:FilterLogEvents",
"logs:GetLogEvents",
"logs:GetLogGroupFields",
"logs:PutLogEvents",
"logs:PutRetentionPolicy",
"logs:TagLogGroup",
"logs:UntagLogGroup"
],
"Resource": [
"arn:aws:logs:*:*:log-group:*:log-stream:*",
"arn:aws:logs:*:*:log-group:/meroxa/msk/meroxa-msk-*"
]
},
{
"Sid": "VisualEditor12",
"Effect": "Allow",
"Action": [
"servicequotas:GetServiceQuota"
],
"Resource": [
"*"
]
}
]
}

Create a new IAM User

  1. Go to your IAM dashboard.
  2. Under Access management, click Users.
  3. In your Users dashboard, click the button Add users.
  4. Under Set user details provide an identifiable User name.
  5. Under Select AWS access type, check the box next to Access key - Programmatic access.
  6. Click the botton Next: Permissions.
  7. Under Set permissions, click the option Attach existing policies directly.
  8. Next to Filter policies, use the search to find your recently created IAM policy.
  9. In the resulting policy list, check the box next to your IAM policy.
  10. Click the button Next: Tags.
  11. You may skip the Add tags (optional) step and click the button Next: Review.
  12. Check to make sure everything looks correct, then click the button Create user.
  13. You will see confirmation that your IAM user has been created.
  14. Copy and paste your Access key ID and Secret access key somewhere safe and secure.
  15. You're done, go ahead and click the button Close.

🎉 You now have an IAM User to create your Self-hosted Environment!

Important

You can only use the IAM User used to create the VPC. You may still rotate your Secret Access Key and update your environment configuration. However, any newly generated IAM Users will not work with your Self-hosted Environment.

What's Next?

Create a Self-hosted Environment