Storing Lambda Secrets in AWS Systems Manager

At Enova, we explore and leverage Amazon Web Services for many of our systems. I recently had a need to quickly deploy a simple relay Lambda to establish secure communication with a legacy system. The solution also needed to be highly configurable for multiple environments. I found that the combination of Serverless Framework and AWS Systems Manager Parameter Store was both the quickest and most powerful way to accomplish the task.

Serverless Framework

For this project I used Serverless Framework to define my Lambda. It is essentially a wrapper around CloudFormation, allowing me to define the whole service configuration within a single YAML file and to deploy to each separate environment with a single command. A great introduction to using Serverless Framework to define a Go service can be found here.

Systems Manager

AWS Systems Manager (SSM) is designed specifically for management of resources. Available resources include parameters, accessible through the Parameter Store. The parameters are stored in a hierarchical structure, making them easy to retrieve. Having a single store for parameters is less error prone than other options. Parameters can also be encrypted using AWS Key Management Service (KMS). This makes it a great place for storing secrets.

Serverless Configuration

Your Serverless configuration file will need to include AWS Identity and Access Management (IAM) policies to allow access to SSM, and to any KMS keys you use for encrypted values. These configurations can be very specific, or very broad. You can allow read access access to all parameters by adding

iamManagedPolicies:
  - 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'

You can allow read access access to specific parameters by adding

iamRoleStatements:
  - Effect: 'Allow'
    Action: 'ssm:GetParameters'
    Resource:
      - 'arn:aws:ssm:aws-region:#{AWS::AccountId}:parameter/foo'
      - 'arn:aws:ssm:aws-region:#{AWS::AccountId}:parameter/bar'

You also need to add access to the key used to encrypt/decrypt any Secure String parameters:

iamRoleStatements:
  - Effect: Allow
    Action:
      - KMS:Decrypt
    Resource: "arn:aws:kms:aws-region:#{AWS::AccountId}:key/exampleUUID"

Retrieving the values

It’s very simple to retrieve the values in process using the AWS SDK.  This allows you to retrieve the most up to date value, and limit exposure of unencrypted values. A simple Go example is here.

The values can also be added as variables in the serverless.yml file by referencing them directly, which sets the values at deploy:

${ssm:/path/to/service/myParam}

To apply the values into specific environment variables that your application can retrieve, you need to create a custom section for each item.

custom:
  serviceHost:
    local: localhost
    dev: service.dev.example.com
    staging: ${ssm:/myapp/staging/service-host} #get from ssm
    prod: ${ssm:/myapp/prod/service-host} #get from ssm

You also need to then reference that value in your environment section.

environment:
  SERVICE_HOST: ${self:custom.serviceHost.${self:provider.stage}}

You will then find this value by getting “SERVICE_HOST” from the environment.

Os.Getenv("SERVICE_HOST")

If the parameter needs to be decrypted, you need to add ~true to the end of it, but be aware that this decrypts the value at deploy.

${ssm:/path/to/secureparam~true}

You could also get the encrypted value by omitting the ~true, and decrypt with KMS within your application.

Limitations of Standard Parameters

Standard parameters are available freely, and are sufficient for most applications. Advanced parameters have more functionality, but have a cost associated. Limitations for standard parameters include:

  • Max 10,000 parameters per account
  • Max length of 4k
  • Max 100 past values per parameter

Conclusion

The Serverless Framework simplifies the setup of cloud deployed services by allowing you to define your entire deployment in one file. Storing your parameters in SSM Parameter Store gives you a centralized, encrypted, secure storage facility that is less error prone and more secure than traditional methods. The values can be retrieved simply in any language through the AWS SDK or through Serverless configuration.