Skip to main content

Terraform EC2 Auto-Discovery Configuration

Report an Issue

This guide shows you how to use Terraform to configure Teleport and AWS to automatically enroll EC2 instances in your cluster.

How it works​

The teleport-discovery-aws Terraform module creates resources in your Teleport cluster and AWS account to enable Teleport EC2 auto-discovery.

The Teleport Discovery Service queries the AWS API to list EC2 instances in your account. For each EC2 instance that it discovers, the Discovery Service uses AWS Systems Manager (SSM) to install Teleport on the instance and join it to the cluster as a Teleport-protected server.

Prerequisites​

  • A running Teleport cluster. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl and tsh clients.

    Installing tctl and tsh clients

    1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

      TELEPORT_DOMAIN=teleport.example.com:443
      TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"

    2. Follow the instructions for your platform to install tctl and tsh clients:

      Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

      curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

      In Finder double-click the pkg file to begin installation.

      danger

      Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

  • EC2 instances running Ubuntu/Debian/RHEL/Amazon Linux 2/Amazon Linux 2023 and SSM Agent version 3.1 or greater
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email@example.com to your Teleport username: 
    tsh login --proxy=teleport.example.com --user=email@example.com
    tctl status
    Cluster  teleport.example.com
    Version  19.0.0-dev
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
note

All EC2 instances added to the Teleport cluster by the Discovery Service must include the AmazonSSMManagedInstanceCore IAM policy in order to receive commands from the Discovery Service. For a list of permissions included in the policy, see the AWS documentation.

Install Teleport to run the Discovery Service​

tip

If you already have a running Discovery Service instance, then assign its discovery_group to discovery-group-name and proceed to use the teleport-discovery-aws Terraform module.

All Teleport Cloud clusters run the Discovery Service for you, so Teleport Cloud subscribers can also skip all of these installation steps.

Step 1/3. Install Teleport​

If you plan on running the Discovery Service on the same host already running another Teleport service (Auth or Proxy, for example), then you can skip this step.

Install Teleport on a host instance that will run the Discovery Service:

To install a Teleport Agent on your Linux server:

The recommended installation method is the cluster install script. It will select the correct version, edition, and installation mode for your cluster.

  1. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https://).

  2. Run your cluster's install script:

    curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash

Step 2/3. Configure the Discovery Service​

If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service:

tctl tokens add --type=discovery

Save the generated token in /tmp/token on the Node (EC2 instance) that will run the Discovery Service.

warning

Discovery Service exposes a configuration parameter - discovery_service.discovery_group - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services.

When running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud resources or a different value if they are watching different cloud resources.

It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.

  • 2 Discovery Services configured with discovery_group: "prod" polling data from Production account.
  • 2 Discovery Services configured with discovery_group: "staging" polling data from Staging account.

Assign teleport.example.com:443 to the host and port of the Teleport Proxy Service in your cluster, and discovery-group-name to a name that identifies a group of resources that you will enroll:

# teleport.yaml
version: v3
teleport:
  join_params:
    # token_name can be a literal token string or a path to a file.
    # File path is preferable to avoid including a secret in the config file.
    token_name: "/tmp/token"
    method: token
  proxy_server: "teleport.example.com:443"
auth_service:
  enabled: false
proxy_service:
  enabled: false
ssh_service:
  enabled: false
discovery_service:
  enabled: true
  discovery_group: "discovery-group-name"

Step 3/3. Start Teleport​

Grant the Discovery Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Discovery Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Discovery Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Have multiple sources of AWS credentials?

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Discovery Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service.

On the host where you will run the Discovery Service, enable and start Teleport:

sudo systemctl enable teleport
sudo systemctl start teleport

You can check the status of the Discovery Service with systemctl status teleport and view its logs with journalctl -fu teleport.

Use the teleport-discovery-aws Terraform module​

Step 1/5. Configure AWS Terraform provider​

Configure the AWS Terraform provider and AWS IAM permissions for Terraform to manage AWS resources.

AWS IAM permissions required for AWS Terraform provider

The AWS Terraform provider will need the following AWS IAM permissions to manage AWS resources created by the teleport-discovery-aws module:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TerraformIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "ManageIamRole",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:GetRole",
        "iam:ListInstanceProfilesForRole",
        "iam:ListRolePolicies",
        "iam:ListRoles",
        "iam:ListRoleTags",
        "iam:TagRole",
        "iam:UntagRole",
        "iam:UpdateAssumeRolePolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ManageIamPolicy",
      "Effect": "Allow",
      "Action": [
        "iam:CreatePolicy",
        "iam:CreatePolicyVersion",
        "iam:DeletePolicy",
        "iam:DeletePolicyVersion",
        "iam:GetPolicy",
        "iam:GetPolicyVersion",
        "iam:ListPolicies",
        "iam:ListPolicyTags",
        "iam:ListPolicyVersions",
        "iam:TagPolicy",
        "iam:UntagPolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ManageRolePolicyAttachments",
      "Effect": "Allow",
      "Action": [
        "iam:AttachRolePolicy",
        "iam:DetachRolePolicy",
        "iam:ListAttachedRolePolicies"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ManageOidcProvider",
      "Effect": "Allow",
      "Action": [
        "iam:CreateOpenIDConnectProvider",
        "iam:DeleteOpenIDConnectProvider",
        "iam:GetOpenIDConnectProvider",
        "iam:ListOpenIDConnectProviders",
        "iam:TagOpenIDConnectProvider",
        "iam:UntagOpenIDConnectProvider",
        "iam:UpdateOpenIDConnectProviderThumbprint"
      ],
      "Resource": "*"
    }
  ]
}

Step 2/5. Configure Teleport Terraform provider​

There are several ways to configure the Teleport Terraform provider depending on how you intend to run Terraform, for example in CI, Spacelift, or some other remote environment, but for a quick start, see the Local Demo guide.

The local demo guide walks through configuring the Teleport Terraform provider with local Teleport credentials, which is typically as simple as logging in with tsh and running a tctl command from the same shell that you use to run terraform commands:

tsh login
eval "$(tctl terraform env)"

If you are running Terraform in a remote environment, such as a cloud VM, an on-prem server, or CI/CD pipelines, refer to Using the Teleport Terraform Provider to find the appropriate guide for your use case.

Step 3/5. Configure the Terraform module inputs​

Add the teleport-discovery-aws module to your Terraform configuration.

module "aws_discovery" {
  source = "terraform.releases.teleport.dev/teleport/discovery/aws"
  version = "~> 18.0"

  # Required inputs:
  # Assign example.teleport.sh:443 to your Teleport cluster's proxy public address in host:port form.
  teleport_proxy_public_addr = "https://example.teleport.sh:443"
  # teleport_discovery_group_name must match the discovery group name in your Discovery Service config file.
  # Teleport Cloud clusters run the Discovery Service in the group name "cloud-discovery-group".
  # Do not modify this input unless you intend to run your own Discovery Service.
  teleport_discovery_group_name = "cloud-discovery-group"
  # Discover EC2 instances
  match_aws_resource_types = ["ec2"]

  # Optional inputs:
  # Apply the additional AWS tag "origin=example" to all AWS resources created by this module
  apply_aws_tags = { origin = "example" }
  # Apply the additional Teleport label "origin=example" to all Teleport resources created by this module
  apply_teleport_resource_labels = { origin = "example" }
  # match_aws_regions is a list of regions in which the Discovery Service will match and discover EC2 instances.
  # The wildcard "*" is the default and matches all regions.
  match_aws_regions = ["*"]
  # match_aws_tags are AWS tags to match when discovering EC2 instances.
  # Only EC2 instances with matching tags will be discovered.
  # The wildcard {"*" : ["*"]} is the default and matches all tags.
  match_aws_tags = { "*" : ["*"] }
}

Add a Terraform output for the module so that Terraform will display its outputs:

output "aws_discovery" {
  value = module.aws_discovery
}

See the teleport-discovery-aws reference for a complete description of the module inputs and outputs.

Step 4/5. Apply the Terraform module​

terraform init
terraform apply

Terraform should plan to create the following resources:

  • AWS IAM role for Teleport Discovery Service to assume
  • AWS IAM policy that grants the AWS permissions necessary for Teleport to discover resources in AWS
  • AWS IAM policy attachment to attach the IAM policy to the Discovery Service IAM role
  • AWS OIDC Provider for Teleport Discovery Service to assume an IAM role using OIDC.
  • Teleport discovery_config cluster resource that configures Teleport for AWS resource discovery.
  • Teleport integration cluster resource for AWS OIDC
  • Teleport token cluster resource that allows Teleport nodes to use AWS IAM credentials to join the cluster

Review the Terraform plan and confirm the plan actions.

After Terraform finishes applying the plan, it should display the module outputs:

aws_discovery = {
  "aws_oidc_provider_arn" = "arn:aws:iam::123456789012:oidc-provider/example.teleport.sh"
  "teleport_discovery_config_name" = "discovery-aws-account-123456789012"
  "teleport_discovery_service_iam_policy_arn" = "arn:aws:iam::123456789012:policy/teleport-discovery-<timestamp>"
  "teleport_discovery_service_iam_role_arn" = "arn:aws:iam::123456789012:role/teleport-discovery-<timestamp>"
  "teleport_integration_name" = "discovery-aws-account-123456789012"
  "teleport_provision_token_name" = "discovery-aws-account-123456789012"
}

The AWS resources should have the following tags:

  • origin=example
  • teleport.dev/cluster=<cluster-name>
  • teleport.dev/integration=discovery-aws-account-<account-id>
  • teleport.dev/iac-tool=terraform

The Teleport resources should have the following labels:

  • origin=example
  • teleport.dev/iac-tool=terraform

Step 5/5. Check discovery status​

After applying the Terraform module, the Teleport Discovery Service should start to discover EC2 instances in your AWS account and enroll them in your cluster as protected resources.

note

It may take a few minutes for EC2 instances to be discovered and enrolled.

Navigate to the Teleport Web UI and select Zero Trust Access > Integrations. By default, the integration created by the teleport-discovery-aws Terraform module is named discovery-aws-account-<aws-account-id>.

Click on the integration for your AWS account to review the discovery status.

The integration page provides an overview of how many EC2 instances have been discovered and any issues encountered during the discovery process.

In the example above we can see there was an issue with the AWS SSM agent on the instance.

If we navigate to the "issues" tab and click on "Details" we can see more information about the issue:

In this case the issue is that the EC2 instance does not have an IAM role with the AmazonSSMManagedInstanceCore IAM policy attached to it, so the instance cannot receive SSM commands.

We can fix this by attaching the AmazonSSMManagedInstanceCore IAM policy to the EC2 instance's role and waiting for the Discovery Service to reattempt the installation with SSM. It may take several minutes for the next discovery scan to run and install Teleport on the instance.

When all of the discovered EC2 instances have successfully joined the Teleport cluster, the overview page will display a healthy status:

Updating module configuration​

The module inputs can be changed and re-applied to adjust the AWS discovery integration.

For example, if you had previously used specific AWS regions (rather than the wildcard default for all regions), then you can adjust match_aws_regions to include additional regions for EC2 discovery and re-apply the module to start enrolling instances in those regions as well.

-match_aws_regions = ["us-west-1"]
+match_aws_regions = ["us-west-1", "us-east-1"]

Apply Terraform again:

terraform apply

Review the Terraform plan before confirming the changes.

After Terraform finishes applying its plan, the Discovery Service will pick up the change to the dynamic discovery_config and begin to enroll EC2 instances in us-east-1 as well.

Troubleshooting​

If Installs are showing failed or instances are failing to appear check the Command history in AWS System Manager -> Node Management -> Run Command. Select the instance-id of the Target to review Errors.

cannot unmarshal object into Go struct field​

If you encounter an error similar to the following:

invalid format in plugin properties map[destinationPath:/tmp/installTeleport.sh sourceInfo:map[url:[https://example.teleport.sh:443/webapi/scripts/installer/preprod-installer](https://example.teleport.sh/webapi/scripts/installer/preprod-installer)] sourceType:HTTP];
error json: cannot unmarshal object into Go struct field DownloadContentPlugin.sourceInfo of type string

It is likely that you're running an older SSM agent version. Upgrade to SSM agent version 3.1 or greater to resolve.

InvalidInstanceId: Instances [[i-123]] not in a valid state for account 456​

The following problems can cause this error:

  • The Discovery Service doesn't have permission to access the managed node.
  • AWS Systems Manager Agent (SSM Agent) isn't running. Verify that SSM Agent is running.
  • SSM Agent isn't registered with the SSM endpoint. Try reinstalling SSM Agent.
  • The discovered instance does not have permission to receive SSM commands, verify the instance includes the AmazonSSMManagedInstanceCore IAM policy.

See SSM RunCommand error codes and troubleshooting information in AWS documentation for more details:

Next steps​