Using CloudFormation to route the webhooks

What I find most interesting about AWS, is that there’s no distinction between “normal services” (e.g. DynamoDB) and “infrastructure services” (e.g. CloudFormation). In both cases there are APIs. Your code can use the DynamoDB API to save the data, as well as it can use the CloudFormation API to understand which componets your system consists of.

We’re building a system that tracks users on the website, tests their behavior against the predefined patterns, and gives the users recommendations based on the matching patterns. The tracking/matching functionality is a 3rd-party service (lets call it GTM, because it’s very similar to GTM) – this service calls our webhooks when something interesting happens.

Due to company’s security policy, we all are supposed to use a single AWS account. Thanks to CloudFormation, we can have as many environments as we need (we have 1 environment per every feature branch + developers can deploy their personal environments as they want).

Every environment is a set of CloudFormation stacks that have names prefixed with an environment identifier. E.g. if you create an environment myenv345, you’ll get CloudFormation stacks named myenv345Services and myenv345Databases. Because we all use the same AWS account, this account has a bunch of stacks named myenv345Services, feature1Services, demoServices, etc.

GTM, on the other hand, doesn’t have any notion of environments – there’s just one environment, one webhook URL it can talk to.

To work this around, we’ve made GTM look at the browser cookie named envId and make this cookie’s value a part of the webhook request. Developer comes to the website and runs a one-liner like this one:

1
document.cookie = "envId=myenv345";

This makes GTM include myenv345 to every HTTP request it sends to our AWS-hosted part of the system, as long as that particular developer is playing with the website.

On our end, we’ve built a service (an AWS API Gateway backed by a Lambda function) that receives all the webhook requests and decides who should be the actual receiver based on the envId attribute of the request.

Because all our environments are CloudFormation stacks, you can guess the name of the ‘Services’ stack like this: envId + "Services". So, if you have envId=myenv345, your ‘Services’ stack is myenv345Services.

Once you have the stack name, you can use the CloudFormation API to look up this stack’s details, namely outputs. In our case, one of the outputs is the environment-specific webhook URL. Here’s some pseudocode:

1
2
3
4
5
6
7
function handle(request) {
    const envId = request.body.envId;
    const stack = cloudFormation.getStack(envId + "Services");
    const targetUrl = stack.outputs["GTMEndpointUrl"];
    httpClient.post(targetUrl, request.body);
    return "200";
}

This component works perfectly. The only issue we had so far was timeouts on the GTM side. It takes time for a Lambda to start, so if you only get requests few times a day, every request will take a significant amount of time to process. We solved it by processing request asynchronously – API Gateway responds with 200 immediately once it has the request, and then Lambda has as much time as it needs to actually do the processing.