AWS Lambda here, there and everywhere
Everybody is talking about serverless, and with serverless comes serverless functions. Small pieces of code (in theory) that receive an event (input) and return a message (output). They do not need a server. Therefore they are serverless, and they scale on demand. On the AWS platform, a serverless function is called a Lambda. In theory, you write the code, and AWS takes care of the rest. There is a reason why I put “in theory” in some locations in the text. Even writing the hello world sample does need some authorization configuration, and when running complete Docker images as a lambda, you cannot talk about small pieces of code. Time to focus on typical usage patterns for lambdas.
Usage patterns for Lambdas
When thinking about Lambdas, think of small pieces of functionality. In your average MVC application, you can think of the request handlers, services, and repository methods. The big difference is, we do not create a class with multiple methods. Instead, we make separate Lamdas for each method. But what about reuse and libraries? That is where Layers come in, more on those things later on. Example usage of lambdas, you require an API to be used by a mobile app. You can specify an API file using OpenApi or create the AWS API Gateway service’s endpoints. Next, you write a lambda to handle the API call, get some data from DynamoDB and return the response. Another example, when deploying an AWS Elasticsearch cluster using CDK, you use a lambda to interact with Elasticsearch for configuration queries. You can read more about deploying elasticsearch in another blog post of mine, Deploying a secure AWS Elasticsearch cluster using CDK. A final example, when sending notifications on arrival of a new order in DynamoDB, create a lambda function triggered by the event that sends a message to AWS SNS.
Talking about it is easy, time to create a few lamdas. The next sections give code samples of lambdas and how to manage them using AWS CDK. You can find the code in our Github repository.
Running examples – Hello world.
All good? Time for some running examples. To make setting up the examples easier, we use AWS CDK, and what is easier for a Lambda then creating a HelloWorld lambda? The following code block shows how to create a lambda that receives an event with two parameters. The first parameter, ‘say,’ is the message you want to bring. The second parameter, ‘to’, is the person to say the message to. In the lambda, we log the famous and customizable hello world message to the console. The final line uses the context to find the log stream’s name to send the logs. The structure for lambda logs is first the log-group. This name is the name returned by deploying the stack using CDK. Within the log-group, there are multiple streams. We obtain the right stream using the context as mentioned before.
const helloLogsLambda = new lambda.Function(this, "HelloLogsLambda", {
runtime: lambda.Runtime.NODEJS_12_X,
handler: "index.handler",
code: lambda.Code.fromInline(`
exports.handler = async function (event, context) {
console.log(event.say + " " + event.to + "!");
return context.logStreamName;
};
`),
});
Use CDK to deploy the stack to AWS. Next, we want to execute the lambda, you can use the console, but I prefer to use the AWS command-line tool. We use two different modules, the lambda module for interacting with lambda functions and the logs module, to obtain the right log stream. Another thing to notice is the way to send a payload to a lambda through the command-line tool. We need to base64 encode the payload before sending it. With sed, we find the name for the log stream. The log-group is the same as the name of the lambda. The script is as follows.
#!/bin/bash
lambda_name="AwsLambdasStack-HelloLogsLambdaA3F63B77-130WFH89QU9AN"
payload=$(echo '{"say": "Hello", "to": "Lambda" }' | openssl base64)
aws lambda invoke --function-name $lambda_name out --payload "$payload"
sed -i'' -e 's/"//g' out
sleep 15
aws logs get-log-events --log-group-name /aws/lambda/$lambda_name --log-stream-name $(cat out) --limit 5
The next block shows the output from the script. The output is verbose. There is a start event, the console log event, the end event, and a report.
{
"events": [
{
"timestamp": 1616341610877,
"message": "START RequestId: 7d75dba5-0c54-45b1-8378-7c8cfce71873 Version: $LATEST\n",
"ingestionTime": 1616341619926
},
{
"timestamp": 1616341610893,
"message": "2021-03-21T15:46:50.878Z\t7d75dba5-0c54-45b1-8378-7c8cfce71873\tINFO\tHello Lambda!\n",
"ingestionTime": 1616341619926
},
{
"timestamp": 1616341610895,
"message": "END RequestId: 7d75dba5-0c54-45b1-8378-7c8cfce71873\n",
"ingestionTime": 1616341619926
},
{
"timestamp": 1616341610895,
"message": "REPORT RequestId: 7d75dba5-0c54-45b1-8378-7c8cfce71873\tDuration: 18.12 ms\tBilled Duration: 19 ms\tMemory Size: 128 MB\tMax Memory Used: 64 MB\tInit Duration: 148.96 ms\t\n",
"ingestionTime": 1616341619926
}
],
"nextForwardToken": "f/36045622418351923997532045574640095047475813476767367171",
"nextBackwardToken": "b/36045622417950510583958494358092452118568142969659719680"
}
Running examples – Sending a message.
We now know how to create a lambda and how to call it using the command line. Time to upper the game and introduce the Simple Notification Service to send an email with the content coming from a lambda. We again call the lambda from the command line and send the message to the configured email address. The first part of the stack is creating the SNS Topic. The code block below shows the creation of the Topic as well as adding an email type subscription.
const snsTopic = new sns.Topic(this, 'LambdaSnsTopic', {
displayName: 'Lambda SNS Topic to send email',
});
snsTopic.addSubscription(new subs.EmailSubscription("your_email@here.com"));
Next, we create a Lambda that sends a message to the Topic. This time we do it a little bit different. Instead of embedded JavaScript for the lambda implementation, we use a separate file. We place the file in the lambda folder. This location is important, as we need it for the CDK configuration. The lambda itself is easy, but now we have another way to store the lambda. The following code block shows the lambda.
const { SNSClient, PublishCommand } = require("@aws-sdk/client-sns");
exports.handler = async (event, context) => {
const sns = new SNSClient(process.env.REGION)
const publishCommand = new PublishCommand({
Message: event.message,
TopicArn: process.env.SNS_TOPIC_ARN,
});
await sns.send(publishCommand);
console.log("Send a message to the SNS Topic: " + process.env.SNS_TOPIC_ARN);
return context.logStreamName;
}
Next, we create the lambda in AWS using CDK, and we grant publisher rights to the lambda on the created Topic.
const sendToSNSLambda = new lambda.Function(this, 'SendToSNSLambda', {
runtime: lambda.Runtime.NODEJS_12_X,
code: lambda.Code.fromAsset('lambda'),
handler: 'send-message.handler',
environment: {
SNS_TOPIC_ARN: snsTopic.topicArn,
REGION: this.region,
}
});
snsTopic.grantPublish(sendToSNSLambda);
Of course, we need a new script to execute the lambda from the command line. I only show the two lines that changed in the script compared to the previous bash script.
lambda_name="AwsLambdasStack-SendToSNSLambdaE1D5DD6A-T6Q6W0KDYCTV"
payload=$(echo '{"message": "Hi, hope you like this message from AWS." }' | openssl base64)
Now when executing the lambda, I get the following email message:
Running examples – Provide a REST endpoint.
The AWS command-line tool is a good tool to execute a lambda. However, we prefer having a REST endpoint that a mobile app or a website can call—time to meet the AWS API Gateway. The API Gateway configures an endpoint that, when called, executes the configured lambda. This example contains the API Gateway and a lambda that again posts a message to SNS like in the previous example. The request it performs is a POST with a JSON document in the body. The JSON document contains a field message. The contents of this field are sent to SNS and, in the end, to an email. Beware, the usage of the API Gateway here is elementary. Explaining all the nuts and bolts of the gateway is not the focus of this blog as we focus on the lambdas.
The code for exposing the lambda is the same as for the previous example. The lambda itself is not. Notice that we are dealing with strings in the request and the response. The lambda has to parse into and from JSON itself. The response is also worthy of checking out. Without a correct structure, the lambda is doing fine, but the gateway throws an error. The following code block shows the lambda code.
exports.handler = async (event) => {
const sns = new SNSClient(process.env.REGION)
const jsonBody = JSON.parse(event.body);
const publishCommand = new PublishCommand({
Message: jsonBody.message,
TopicArn: process.env.SNS_TOPIC_ARN,
});
await sns.send(publishCommand);
return {
"isBase64Encoded": false,
"statusCode": 200,
"headers": {
"Access-Control-Allow-Origin": '*'
},
"body": JSON.stringify({
"message": "OK"
})
};
}
What about the gateway? As mentioned before, creating a simple gateway is also very simple in the CDK. Beware, we expose the root resource, which is not a good practice. Check the gateway documentation for more information.
new apigateway.LambdaRestApi(this, 'JettroLambdaApi', {
handler: gatewayLambda,
});
Running a CDK deploy exposes the URL of the gateway:
AwsLambdasStack.JettroLambdaApiEndpoint0A243061 = https://943pdmvxn7.execute-api.eu-west-1.amazonaws.com/prod/
Send a POST request using any tool you like. I prefer using the Intellij HTTP client for this.
### Send POST request with json body
POST https://943pdmvxn7.execute-api.eu-west-1.amazonaws.com/prod/
Content-Type: application/json
{
"message": "Hello from API Gateway using HTTP client."
}
Layers
When writing code in a lambda, you can still use libraries. You can add the library immediately to your lambda package. However, this means you have to redeploy it each time you deploy your lambda. Also, you cannot easily reuse the code among your lambdas. These challenges are why AWS provides layers. A layer contains libraries created by yourself but also offered by AWS or third parties. Layers are extracted in a language-specific path within your lambda. When creating your lambda using Nodejs, the path to the libraries in a layer is “/opt/nodejs”.
Of course, we want to use a layer to improve our sample application. Our Layer provides a function to create a response in the correct format for the API Gateway.
The code for the Layer is straightforward. The folder layout of the Layer is essential. If you make a mistake, you get errors telling you the Layer could not be found. You have to adhere to the following structure:
-nodejs
–response-layer
—responses.js
–package.json
The name in the package.json is “nodejs”, and the main is “index.js”. Below is the code for the layer in the file “responses.js”
exports.createResponse = (message) => {
return {
"isBase64Encoded": false,
"statusCode": 200,
"headers": {
"Access-Control-Allow-Origin": '*'
},
"body": JSON.stringify({
"message": message
})
};
}
Below is the definition of the Layer and the lambda in CDK. Notice the path to the Layer and adding the Layer to the layers property of the lambda.
const responseLayer = new lambda.LayerVersion(this, "ResponseLayer", {
code: lambda.Code.fromAsset("./layers/response-layer"),
compatibleRuntimes: [lambda.Runtime.NODEJS_12_X],
});
const gatewayLambda = new lambda.Function(this, 'GatewaySendToSNSLambda', {
runtime: lambda.Runtime.NODEJS_12_X,
code: lambda.Code.fromAsset('./lambdas/lambda-gateway'),
handler: 'index.handler',
environment: {
SNS_TOPIC_ARN: snsTopic.topicArn,
REGION: this.region,
},
layers:[responseLayer],
});
The final piece of the puzzle is importing the “createResponse” method from the module, do notice the path used in the require part.
const { createResponse } = require("/opt/nodejs/response-layer/responses");
Step functions
As I mentioned in the title, Lambdas are everywhere. I cannot give examples of all different scenario’s. But using Lambdas in step functions is an important one to understand. A single Lambda is ideal for doing small tasks, but you need orchestration capabilities when dealing with more advanced tasks. You want to create an error flow. You want to do different actions based on some state—time to meet AWS Step Functions.
With AWS Step Functions, you create a workflow or state diagram. Our flow consists of a few steps:
- Retrieve parameters from the request
- Check if the message contains the word “bad.”
- NO: Send a message to the Topic and send an email.
- YES: Don’t do anything
I am nog showing the CDK code for lamdas again. But for each step in the workflow that calls a lambda, we need to create a LambdaInvoke task. The following code block shows creating the task.
const parseRequest = new tasks.LambdaInvoke(this, "ParseRequest", {
lambdaFunction: parseRequestLambda,
outputPath: "$.Payload",
});
The next block shows how to use these tasks to create a state machine definition and the state machine itself.
const definition = parseRequest
.next(checkBadWords)
.next(
new sfn.Choice(this, 'No Bad Words')
.when(sfn.Condition.booleanEquals('$.valid', false), sendMessageFailed)
.when(sfn.Condition.booleanEquals('$.valid', true), sendMessage)
.otherwise(waitX)
);
const stateMachine = new sfn.StateMachine(this, "StateMachine", {
definition,
timeout: cdk.Duration.minutes(5),
});
In the console, you can find the workflow with all the steps. For each stage, the input and output are available. That makes it easier to debug. The sample code also shows how to call the step functions from an API Gateway.
Concluding
As you can see, Lambdas are the glue of the AWS Cloud. You find them everywhere. They are also very powerful to create a high available and scalable system. I have shown a number of typical usage scenarios. And working with Lambdas together with AWS CDK makes it easy to deploy them. If you want to have a better look at the sample code, check out the link to our Github repo containing a number of CDK experiments. This blog post uses the aws-lambdas and the aws-step-functions modules. Please contact me if you have questions or improvements.
https://github.com/luminis-ams/aws-cdk-examples
