How to use AWS Lambda to build a tweetbot

Inspired by Vicki I decided to build a Tweetbot – the code is available here. You can follow the tweetbot online – the architecture that Vicki proposed is basically what I did, only I made a few changes in the code.

Time taken

It’s worth pointing out first that it took me approximately 32 hours of work to get this al to work. I think a big part of that is that I wasn’t familiar with AWS IAM. Like anything – I think the first time you build something it’s more tricky than normal.

Technical goals of this project:

The technical goals of this project was to get my head around how to use AWS Lambdas – because before starting I didn’t know what they were. I also wanted to understand how to set up a CI system from scratch. I used Travis CI because of it’s integration with github.  The other reason for this project is that I wanted to stretch my software engineering skills. All in all I consider it a success.

Learnings from AWS:

• Getting IAM to work, it’s quite tricky and I had to read lots of documentation and stackoverflow to get this to work.
• Total time spent about 32 hours, about half of that was spent with errors – of connections between lambdas and S3 and all.
• This is tricky and I still am not very good at all of this. The docs aren’t in my opinion super useful.
• Don’t mix up lambda execution roles and IAM roles.
• Make sure to configure your AWS CLI correctly – I got confused by different profiles.

Travis CI set up

• It wasn’t too difficult to get a Travis CI setup working.
• https://stackoverflow.com/questions/37267916/how-to-run-aws-configure-in-a-travis-deploy-script was a useful link for helping me understand how to configure credentials.

TODO:

• Still haven’t got SSM working – I’ve no idea about this.
• It needs more tests like tests before CI runs, and more tests locally.

AWS Lambdas

AWS Lambdas are super useful. Here is a short list of what they can do(much more in the docs):

• Run when a file is placed in an S3 bucket and move the file to another location
• Process logs with Kinesis streams and store the logs in DynamoDB
• Compile a static website based on a new code change and reload the site
• Backup files and put them into cold storage

When you think about it – many business processes, fit into this sort of workflow. For example you may have someone uploading a pdf of a document to a server, an AWS Lambda could resize that document and then put it into cold storage after a certain period of time.

A good thing to keep in mind is the limitations of lambdas:

• 512 MB disk space
• Execution timeout after 5 minutes of activity
• 128 to 1536 MB
• Have boto included as a runtime library

I had some trouble uploading the AWS Lambda manually – see Vickis blog post for details of how to solve that.

Conclusion

All in all this was a successful side project. I learned a ton about AWS Lambdas and I probably need to spend more time getting my head around AWS IAM. Also investing in bash scripts and tools to get the zipping of AWS Lambda done is worth it.

I also want to add in more images, so there can be more varied tweets.

Something I need to get my head around is how to do this with Zappa.