Useful tools for local development with AWS services
Over the last 2.5 years, I’ve been working with AWS and a wide range of its services. During this time I noticed that for most projects it’s useful to be able to test your application against AWS services without having to deploy or move your code into the cloud. There are several free solutions available for you to use depending on the services required by your project. In this post, I’ll describe some of the tools that I use.
DynamoDB local
At one of my previous projects, we made extensive use of the combination of DynamoDB and Elasticsearch for storing and querying data. The fact that DynamoDB is a managed database service with immense scale and performance benefits, makes DynamoDB a great fit for high traffic applications.
As a user, it’s quite simple to use as it’s a key-value store. Most of the other AWS databases are managed instances of existing services, however, DynamoDB is an AWS specific service which you can’t really download and install locally. Luckily back in 2018 AWS introduced a simpler way to work with DynamoDB utilizing DynamoDB local, a dockerized version of DynamoDB which you can simply run as a docker container to develop and test against.
Running DynamoDB local is as simple as executing:
$ docker run -p 8000:8000 amazon/dynamodb-local
Or if it’s part of a bigger set of dependencies you could leverage docker-compose.
With that it’s a matter of running:
$ docker-compose up
And you should see something like:
With the AWS CLI you can easily query for available tables:
$ aws dynamodb list-tables --endpoint-url http://localhost:8000
Which should result in something like:
And of course, you can use the AWS SDK with your preferred language as well.
I hear you thinking: are there no limitations? Yes of course there are some limitations with using DynamoDB local compared to the managed service. For instance, parallel scans are not supported (they will happen sequentially). Most limitations are nicely outlined in the DynamoDB Developer guide.
Amazon also provides docker images for other services like AWS Step functions Local and OpenDistro for Elasticsearch. Be sure to check out the Amazon repo on DockerHub for other usefull images.
LocalStack
Now when you’re developing a simple service that only depends on DynamoDB, DynamoDB local is a good choice. However, once you start to leverage more and more services it might be worthwhile to look for other options as not all services are available as single docker images.
When you’re building services that are part of a microservices architecture, you’re probably using other AWS services like SNS, SQS, and perhaps S3. This is where a tool like LocalStack can add a lot of value. So what is LocalStack?
LocalStack is a project open-sourced by Atlassian that provides an easy way to develop AWS cloud applications directly from your localhost. It spins up a testing environment on your local machine that provides almost the same feature parity and APIs as the real AWS cloud environment, minus the scaling and robustness of course.
Localstack focuses primarily on providing a local AWS cloud environment that adheres to the AWS APIs and offers a free and pro version, which you can leverage depending on your requirements. In my experience, the free/community version offers a lot of value and supports a whole range of services.
You can install LocalStack via pip if you’re familiar with python and its package system or you can use it via docker(compose). On my Mac, I found that installing LocalStack as a python package was a bit of a hassle, so I always prefer to use it via docker-compose.
Using LocalStack with docker-compose is as simple as creating a docker-compose.yml file with the content:
If you’re running on a Mac be sure to prepend TMPDIR=/private$TMPDIR before running.
$ TMPDIR=/private$TMPDIR docker-compose up
Afterwards, you should see something similar to the following output.
As you can see, it starts a whole bunch of services out of the box. If you don’t use all those services you can also provide a list of services required when starting localstack by providing a SERVICES variable like:
$ TMPDIR=/private$TMPDIR SERVICES=s3,sqs docker-compose up
Now you should see in the startup output that it only started S3 and SQS.
Update: I just learned that homebrew also supports installing LocalStack. I’ve not used it, so can’t say if it’s any good, but it looks pretty simple 🙂
$ brew install localstack
If you don’t want to manually start LocalStack via docker-compose, but want to start it for instance during your build/test phase, you can also leverage testcontainers and just add a localstack rule to your Unit test:
AWSLocal
If you’re already using LocalStack, it’s worthwhile to also install awslocal. It’s a CLI that proxies the AWS CLI and adds the --endpoint-url http://localhost:4566/ after every command, so you don’t have to.
You can install it by running
$ pip install awscli-local
Now we can just run:
$ awslocal dynamodb list-tables
which just like with DynamoDB local would return:
Commandeer
Localstack used to come with a Web UI, which is now marked as deprecated. As an alternative, I would recommend using Commandeer. It’s a very useful tool and also supports working with LocalStack (next to a whole bunch of other services). It can give a nice overview of the services started with localstack, but also offers dashboards and UIs for example DynamoDB in which you can explore data, create tables, etc.
If you know of other useful tools that help you in your day to day work working with and developing on the AWS platform, feel free to leave a comment or tweet at @jreijn or @luminis_eu!