CloudFormation Custom Resources

Extending the reach of CloudFormation templates can feel like a tricky task, and sometimes the built-in resources just don’t cut it. You might be in a spot where you need a special action, like interacting with an API outside of AWS, or configuring a custom service that AWS doesn’t support out of the box. That’s where the magic of CloudFormation Custom Resources comes into play, and they can be a huge help to those who know how to use them right. You can unlock a whole new world of possibilities that can transform your infrastructure as code approach. This article will walk you through the world of CloudFormation Custom Resources, giving you the knowledge to harness their power and create solutions that go beyond what’s typical.

Table of Contents

Understanding CloudFormation Custom Resources

CloudFormation Custom Resources are, in essence, your way of teaching CloudFormation new tricks. They let you define resources that aren’t built into the service by default. You can think of them as building blocks that you can use to make your infrastructure as code more complete, and aligned to your exact needs.

Custom Resources are like a bridge between CloudFormation and your own logic, they allow you to add custom provisioning steps. They come in handy when you need to execute tasks that CloudFormation doesn’t support natively, from interacting with third-party APIs to carrying out complex setup actions that go beyond typical resource deployments. You can add custom logic to your templates and make your deployments far more flexible and versatile.

Here’s how they generally work: you set up a custom resource in your CloudFormation template. This resource refers to a Lambda function, or a CloudFormation macro that includes the actions you want to perform during the resource’s creation, update or deletion. When CloudFormation processes your template, it sends requests to the Lambda function (or your chosen provider) to complete the task.

Key Concepts of CloudFormation Custom Resources

To fully grasp how Custom Resources work, it is important to understand the key parts that make them up. These pieces fit together to ensure the custom actions you create are carried out correctly. These include resource providers, request types, and response formats.

Resource Providers

At the heart of every CloudFormation Custom Resource lies a resource provider. This can be a Lambda function you write, or an AWS CloudFormation macro, or a third-party service that you tell CloudFormation to call to carry out your needed actions. The provider is the engine that carries out the custom logic to manage your resources. The function receives the CloudFormation request and does the actions needed by it, it can be creating a resource or updating or deleting it.

Request Types

When CloudFormation needs to do a resource action with custom resources, it sends a request that explains what needs to be done. These requests come in three forms:

Create: This request means that CloudFormation needs to set up a new resource for you. It is sent when the custom resource is first seen in the CloudFormation template. This type of request needs to return details like resource IDs for future reference.
Update: When a resource has to be changed, or updated this request type is used by CloudFormation. It is sent when a custom resource’s properties change in the template. This request needs to manage changes to existing resources, ensuring that all updates are done properly.
Delete: This type of request is sent when a resource is no longer needed and it has to be removed. It is used when a custom resource is removed from the CloudFormation template. This request needs to remove the resource, and cleaning any data or settings properly.

Understanding these request types makes it easier for you to create providers that can manage the life cycle of your resources effectively.

Response Format

After a provider does the actions requested, it sends back a formatted response to CloudFormation. This response must be in a clear and specific format to confirm the actions taken. The most important parts of this response are:

Status: This says if the action was a success or failure, with messages that confirm the outcomes.
PhysicalResourceId: This is a unique identifier of the resource set up by the provider. This identifier is used by CloudFormation to track the resources.
Data: This provides any extra information for the resource. It can be data for use by other resources in your CloudFormation stack.

The responses are key to making sure CloudFormation can track and manage your custom resources properly.

Use Cases for CloudFormation Custom Resources

Custom Resources bring a ton of versatility to your infrastructure as code practice. Let’s explore some of the areas where they stand out the most.

Integrating with Third-Party APIs

One of the most useful use cases of Custom Resources is working with APIs outside of AWS. Think about setting up a database with a third-party provider, or checking a domain name with an external registrar. These actions are far out of the reach of CloudFormation default resources, but with Custom Resources, they become a natural extension of your infrastructure setup.

You can set up Custom Resources to call external APIs, sending requests with details pulled from your CloudFormation template. Once done, the API responses can set up new settings, create user accounts, or even configure domain names with a lot of ease, using the parameters you need, and the data you want.

Deploying Custom Configurations

CloudFormation does a great job with the core AWS services, but what happens when you need more detailed control over the environment that you manage? CloudFormation Custom Resources allow you to carry out very specific setups, such as installing and configuring apps on EC2 instances, setting up settings for Kubernetes clusters, or even building and deploying custom services within your environment.

With Custom Resources, you can design the logic for your configurations, making sure that every part of your infrastructure is set up exactly how you need it. You get a lot of control, from the apps that are installed to the services you start up, and how they are configured, all of this within your infrastructure as code workflows.

Managing Resources Not Supported by CloudFormation

Sometimes, you will need to manage things that CloudFormation does not support. These could be legacy systems, or a non-AWS service that is part of your ecosystem. In this case, custom resources act as an adapter.

By using custom resources, you can add those resources into your CloudFormation templates, making it easier to manage all of your resources from a single place. You can then treat those unsupported services like they are normal resources in your template, setting them up, updating them and deleting them all as part of your deployments. This makes your infrastructure simpler and more consistent.

Automating Complex Workflows

You might find that setting up your infrastructure needs a series of steps, and not just setting up individual resources. With custom resources, you can automate such complex workflows, like setting up settings across different systems, or carrying out tasks in a series.

With a custom resource, your steps are tied into your infrastructure as code, and this also makes it easy to make changes, rollbacks, and repeats, while making sure everything goes smoothly.

Creating CloudFormation Custom Resources

Creating a CloudFormation Custom Resource is a multi-step process that involves both defining the resource in your CloudFormation template and coding the provider logic. The following steps walk you through the process of setting up custom resources.

Step 1: Define the Custom Resource in Your Template

You must start by declaring a custom resource in your CloudFormation template. The template must describe the type of provider that you will use (Lambda, or Macro). Here is a basic example of how you can declare a custom resource using a Lambda function provider:

Resources:
  MyCustomResource:
    Type: 'AWS::CloudFormation::CustomResource'
    Properties:
      ServiceToken: !GetAtt MyLambdaFunction.Arn
      MyProperty: 'MyValue'

Type: This parameter specifies that it’s a custom resource using the AWS::CloudFormation::CustomResource type.
Properties: This part includes the custom configurations:
- ServiceToken: This needs the ARN of the Lambda function that will carry out the logic.
- MyProperty: This is your own specific configuration that you need to pass to your function. You can add any property that your logic needs.

This simple template gives a basic structure that you will need, with a ServiceToken, and your custom properties.

Step 2: Implement the Provider Logic

After you have defined your resource in your template, you have to code the logic for the provider that will manage the actions you need. In the case of a Lambda function, you will use Python, Node.js, or other languages you like, depending on your needs. Here’s a Python example:

import json
import boto3
import cfnresponse

def lambda_handler(event, context):
    print(f'Request received: {json.dumps(event)}')
    try:
        request_type = event['RequestType']
        properties = event['ResourceProperties']
        physical_resource_id = event.get('PhysicalResourceId')

        if request_type == 'Create':
            # Custom logic for resource creation
            physical_resource_id = 'my-resource-' + str(hash(json.dumps(properties)))
            response_data = {'Message': 'Resource created successfully'}
            cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data, physical_resource_id)

        elif request_type == 'Update':
            # Custom logic for resource update
            response_data = {'Message': 'Resource updated successfully'}
            cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data, physical_resource_id)

        elif request_type == 'Delete':
            # Custom logic for resource deletion
            response_data = {'Message': 'Resource deleted successfully'}
            cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data, physical_resource_id)

        else:
            response_data = {'Message': f'Unknown request type: {request_type}'}
            cfnresponse.send(event, context, cfnresponse.FAILED, response_data, physical_resource_id)

    except Exception as e:
        print(f'Error: {str(e)}')
        response_data = {'Error': str(e)}
        cfnresponse.send(event, context, cfnresponse.FAILED, response_data, physical_resource_id)

This example shows a generic structure for the provider, it handles the different types of requests, Create, Update, and Delete.

lambda_handler(event, context): this is the main function for a Lambda function, the entry point that does the actions based on events.
event: This parameter has the details about the CloudFormation request, such as the request type, and resource properties.
context: this parameter includes details about the Lambda function and its environment.
cfnresponse.send(event, context, status, response_data, physical_resource_id): this is a function that sends a response back to CloudFormation with the status of the request, any data, and the resource ID.

This Python logic should be set up in AWS Lambda, and the ARN of that function needs to be passed as a ServiceToken for the custom resource.

Step 3: Deploy and Test the Custom Resource

After you have defined your template and implemented the logic, it is time for deployment. Here’s how to go about it:

Package and Upload your Lambda function: You must package your Lambda code into a deployment package and then upload it to AWS Lambda.
Deploy your CloudFormation template: Use the AWS console, or the AWS CLI, to create or update your CloudFormation stack using the template with your custom resource.
Monitor the deployment: Keep an eye on the CloudFormation events to make sure there are no errors, and that the custom resource is doing what it is supposed to do.
Test the resource: after a successful deployment, you have to check that your custom resource is working fine. You can change or delete the stack to see that your logic is properly executed by your provider.

After you deploy and test the custom resource, it can be used again in the future to help you automate resource setups and other tasks.

Best Practices for CloudFormation Custom Resources

When using CloudFormation Custom Resources, some practices can make your setup process more efficient and also make it easier to maintain and scale. Here are some best practices:

Use a Versioning Strategy for Your Providers

As you develop the logic of your resource providers, it is best to use a versioning system that will let you track and manage your code changes over time. This is critical when you use Lambda functions as your provider. By having a versioning strategy, you can keep track of changes, rollbacks, and also ensure that each template version uses a compatible provider version. This avoids any errors that may happen due to non compatible providers, and template changes.

Implement Proper Error Handling

Error handling is a key part of a stable system. In the provider logic, you have to implement robust error handling, logging, and alerting mechanisms, so you can track problems. You must use try/catch blocks in your code to capture errors, logging them, and also send a failure message back to CloudFormation with the details, this will make it easier to debug your custom resources if errors happen.

Secure Your Providers

Security must be a priority when using Custom Resources, you must follow the security best practices when designing your provider logic, making sure that your Lambda functions have the least level of privileges required for their execution. Avoid putting sensitive data into your template directly, and use secrets management services such as AWS Secrets Manager to keep this data safe. This is key for protecting the data that your infrastructure manages.

Keep your Providers Lightweight

Provider logic must be lightweight and do the task it needs to do fast. In the case of Lambda functions, long execution times can add to the deployment time, and also to the costs. Optimize your logic, and use efficient algorithms to make sure that your providers are able to perform quickly. This will make your infrastructure management smoother and cost effective.

Document Your Custom Resources

Documenting your custom resources is needed for maintainability and to improve collaboration between team members. You need to create clear documentation for each custom resource, showing its purpose, parameters, the logic of the provider, and any dependencies that it may have. Having proper documentation will make it easier to understand the purpose of a given custom resource, making your setup process easier to maintain.

Advanced Techniques for CloudFormation Custom Resources

When you have a good grasp of the basics of custom resources, you can start to explore more complex techniques to improve their function and usage. Here are some advanced techniques you can use to create more advanced and powerful custom resources.

Using AWS SDK for Complex Interactions

When working with Custom Resources, you might have the need to interact with multiple AWS services. The AWS SDK, gives you a standardized set of functions to help you to interact with AWS services from within your providers. Instead of managing the logic to interact with each API, the SDK gives you a simple way to handle everything.

You can use the AWS SDK in your provider to call any API, create or delete resources, or read their properties, improving the versatility and the features that you can build using your custom resources. The SDK saves you the complexity of directly working with each API, making your logic easier to handle and maintain.

Implementing State Management

State management is a key part of custom resources that need to have more complex setups. When your resources need to be updated, your providers must know the past states of that resource. By tracking the state you can see any changes that happen during updates and also delete them properly.

State management can be achieved using different mechanisms, like storing a resource state in a database, or you can also store a state in the PhysicalResourceId, but there are trade offs, since there are limits to the length of the PhysicalResourceId. Using state management will improve the accuracy and efficiency of your resources.

Using CloudFormation Macros for Template Transformations

CloudFormation Macros can transform your templates, they do this before CloudFormation processes the template. This gives you ways to do actions that are not possible through custom resources. You can implement logic that does text replacement, or parameter validation. You can use a macro to make your templates more flexible. This lets you make your templates far easier to manage by using a CloudFormation macro.

Creating Nested Custom Resources

You can nest custom resources with other CloudFormation stacks, this is useful for those complex setups. Nested stacks make your setup process cleaner and also let you divide your templates into smaller parts that you can manage and reuse. This improves your template’s structure, making it more manageable, and it allows a better team work approach, where different people work on different parts of a complex infrastructure.

Troubleshooting Common Issues

Even when following best practices, you might find yourself facing issues when setting up custom resources. Here are some usual problems and solutions to help you solve them faster.

Lambda Function Timeouts

Lambda functions have time limits for the execution time, and if your logic goes beyond this time limit, it will be terminated by AWS, and this will lead to deployment failures. If this happens you can optimize the code, or you can also increase the timeout limit. But, you need to look at any bottlenecks that might be causing the issue.

Incorrect Response Format

CloudFormation needs the correct JSON format for the responses from your providers. If your responses do not match the required structure it can cause deployment failures and confusion. Always validate the responses, using the correct fields, and correct formatting.

Permissions Errors

If the Lambda function you use does not have the required permissions to access resources, you will see errors during resource deployments. Check your IAM role to make sure it has all the permissions required.

Logical Errors in the Provider Logic

Logical errors in your provider code can lead to unexpected behaviors and also to deployments that don’t go as planned. Use tests to find the mistakes before deployment, and use logging in your code to track what is going on during execution, this will make it easier to debug logical errors.

CloudFormation Resource Limit Issues

You might be hitting a CloudFormation resource limit, where a CloudFormation template has a limit on how many custom resources it can manage. Check the current limits in the AWS documentation, or redesign your architecture by using nested stacks to avoid the error.