Using AWS Secrets Manager in the Elastic CI Stack for AWS

The Elastic CI Stack for AWS supports reading a Buildkite Agent token from the AWS Systems Manager Parameter Store. The token can be stored in a plaintext parameter, or encrypted with a KMS Key for access control purposes. You can also store your Buildkite Agent token using AWS Secrets Manager if you need the advanced functionality it offers over the Parameter Store.

For example, AWS Secrets Manager can automatically rotate and revoke secrets using Lambda functions, and replicate secrets across multiple regions in your account.

Storing agent tokens

To store your Buildkite Agent token as an AWS Secrets Manager secret, configure the Elastic CI Stack for AWS's BuildkiteAgentTokenParameterStorePath parameter to reference your secret with the special parameter path /aws/reference/secretsmanager/your_Secrets_Manager_secret_ID. Parameter Store will transparently fetch the token from AWS Secrets Manager when this parameter is read.

See the AWS documentation on Referencing AWS Secrets Manager secrets from Parameter Store parameters for more details.

To ensure your Elastic CI Stack for AWS has access to the secret:

  • Provide the Key ID (not the alias) used to encrypt the Secrets Manager secret to the BuildkiteAgentTokenParameterStoreKMSKey parameter. An IAM policy with kms:Decrypt permission for this key is included in the CloudFormation template.
  • Use the CloudFormation stacks' Resources tab to find the AutoscalingLambdaExecutionRole and IAMRole roles, use their Amazon Resource Name (ARN) in the policy below.
  • Secret Manager will capture a role's Unique ID when saving the resource policy; if you re-create the IAM role you must save the resource policy again to grant access.
  • Use the Secret Manager secret's resource policy to grant secretsmanager:GetSecretValue permission to both the instance IAM role and the scaling Lambda IAM Role.
{
  "Version" : "2012-10-17",
  "Statement" : [ {
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : [
        "arn:aws:iam::[redacted]:role/buildkite-stack-AutoscalingLambdaExecutionRole",
        "arn:aws:iam::[redacted]:role/buildkite-stack-Role"
      ]
    },
    "Action" : "secretsmanager:GetSecretValue",
    "Resource" : "*"
  } ]
}

Multi-region replication

It is also possible to replicate your Buildkite Agent token to multiple regions using AWS Secret Manager's multi-region replication. You can then deploy an Elastic CI Stack for AWS to each region and use the Parameter Store reference path to read the secret from the regionally replicated secret.

Some additional points to keep in mind when using multi-region replication:

  • Ensure each region's IAM role has ssm:GetParameter permission for the region it will be retrieving the secret from.
    • By default, the template will grant permission to only the region it is deployed to, limiting the role's utility to the stack's region. This isn't a problem but a caveat to be aware of. Don't expect to use the same role in multiple regions.
  • Ensure each region's IAM role has kms:Decrypt permission for the key used to encrypt the secret in that region.
    • You can do this with the AWS Secrets Manager key in Secrets Manager, and looking up the underlying CMK ID of that key alias in each region the stack template is deployed to. Provide that value for the BuildkiteAgentTokenParameterStoreKMSKey parameter for the stack in that region.
  • Apply a resource policy to the primary Secrets Manager secret that grants secretsmanager:GetSecretValue for each region's IAM role and wait for that to be replicated.

Now, changes to the agent token secret (either made by hand or using Automatic Secret Rotation) will be replicated from the primary region to each replica region.

The Elastic CI Stack for AWS will only retrieve the Buildkite Agent token once when the instance boots. You should refresh your Auto Scaling Group instances after rotating and replicating the secret, and before revoking the old token.