A Cloud Guru Challenge - Improve Application Performance using ElastiCache Redis
My name is Michael Connaker, and I am an experienced Site Reliability Engineer and Cloud Engineer.
With over a decade of experience in Information Technology, I have worked across a broad spectrum of disciplines, including networking, systems administration, desktop support, and cloud management. Currently I specialize as a Site Reliability Engineer and Cloud Engineer, where my expertise in Infrastructure as Code (IaC), GitOps, Configuration Managment and CI/CD tools is used to automate deployments for improved efficiency, scalability, and cost optimization. I have a strong ability to drive collaboration with clients and teams, ensuring seamless execution and delivery of results.
I have extensive leadership experience, serving as a Subject Matter Expert (SME) to clients and colleagues, as well as a Team Lead managing a team of 8 to 12 members. In this role, I provided guidance and direction, collaborated with management to develop and enhance onboarding procedures, SLAs, SLOs, technical documentation, and shaped the team’s vision and strategy. Additionally, as an Agile leader, I facilitated sprints, reported progress to senior leadership, and led Scrum of Scrums and retrospectives to drive continuous improvement.
I am educated with a Bachelors of Arts in General Studies and Associates of Applied Science in Computer Networking & Systems Administration. I have also achieved several industry certifications.
Outside the technical realm, I’m an avid reader with a passion for Sci-Fi and Supernatural genres, and a dedicated gamer, having explored countless virtual worlds across multiple platforms. Yet, my most fulfilling adventure is fatherhood to my two teenage daughters and a newborn son — a role that brings immense joy and fresh challenges every day.
Hey Guys!
Welcome to my next blog where we are covering another challenge presented by A Cloud Guru. For this challenge, we'll be improving application performance using ElastiCache Redis. The challenge can be found here.
For this challenge, A Cloud Guru has listed out steps that are required to complete the challenge.
They are:
- Deploy RDS PostgresSQL database instance
- Deploy ec2 instance to run the app. Include python3 and its modules psycopg2, flask, configparser, redis. Add the python application to the server from GitHub.
- Deploy ElastiCache Redis
- Convert application to use Redis
Github Application: https://github.com/ACloudGuru/elastic-cache-challenge
My GitHub Submission: https://github.com/mconnaker/acg-app-performance-challenge
The Goal: The application itself has been artificially slowed to represent an under-provisioned database server. The goal of this challenge is to add a cache in front of the database, not to optimize the stored procedure.
Alright, let's go ahead and break this down. First, we'll need to deploy the resources in AWS. There are two ways we can do this - manually through the AWS Management Console or through IaC. Next, we'll need to make a few manual changes on the ec2, which include creating a proxy on the server and a folder where the application will reside in. Lastly, after doing initial testing, we'll modify the application to use Redis.
Deployment using Infrastructure as Code
In a nutshell, IaC is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Using IaC allows your code to be the single source of truth, have version control, traceability/accountability and decreases mistakes made through manual processes. IaC also allows testing, monitoring and reviewing errors that happen when attempting to deploy the code.
For the Big 3 in Cloud Computing (AWS, GCP and Azure), each has its own version of IaC. AWS CloudFormation, Azure ARM Templates and GCP Cloud Deployment Manager. There are also open source deployment tools, such as Terraform, Ansible, Chef and Puppet.
For this challenge I will deploy my infrastructure using IaC utilizing Terraform. For version control, I will use Github. With Terraform, you can use modules or write it from scratch. Modules is a packaged Terraform configuration that makes it easy to deploy.
My personal experience is with Cloud Formation. For the challenge, I decided to write my Terraform code from scratch to better understand how the deployment process works.
The Terraform code I wrote deploys ec2, RDS, ElasticCache Redis, Keypair, VPC, subnet, nacl, route table, Security Groups and key pair. I also deployed .sh install configuration setup for the ec2, which can be done using terraform. The configuration file deploys Python3, the modules, postgresql, nginx, gcc, development tools and git. The file can also be used to clone the github file where the python application resides.
Post Deployment
After deploying the infrastructure using Terraform, a few changes on the ec2 must be done. These are more minor configurations, such as grabbing the repository for the application from GitHub, creating a folder in the home directory and configuring proxy in Nginx.
Pre-Caching
Once the app is running, the elapsed time is always above 5 seconds. This is again due to the fact that the application itself has been artificially slowed to represent an under-provisioned database server
Now that we've done the initial testing, we can add Redis cache to the application.
Caching with ElastiCache Redis
Now that we know the application is working and the slowness is showing, we'll implement ElastiCache Redis to boost the performance. We'll do a cache-aside strategy, where we'll query the cache first to see if the data is there.
In Cache-aside strategy, the application looks to the cache (Redis) first, if no data is found it will query the database and populate the cache with the result.
We'll take a step further and add in an invalidation period using Time to Live (TTL). Invalidation is used when the relevant records are modified in the database, allowing the cache to stay fresh.
To this end, we'll make 3 simple modifications to the application.
First, we'll add in import json, import redis and create a variable called rcache that will have the information to access the ElastiCache Redis.
Second, we'll modify our code existing code. Our first step is to modify a slice of the code. We'll rename def fetch(sql) to def dbfetch(sql).
Finally, we'll add in Redis. the SQL statement is used as a key in Redis, and the cache is examined to see if a value is present. If a value is not present, the SQL statement is used to query the database. The result of the database query is stored in Redis. Finally, we'll use TTL to set how long the key will be stored until it expires. Once it hits the TTL, Redis will evict the key to free the associated memory.
example:
In order to get the SQL statement, we'll need to call on the function, def dbfetch(sql). We'll also use to json serialize and deserialize the results.
Once the app is modified and ran again, we'll see that the time elapse drops from 5 seconds to milliseconds.
And thats it.
Final Thoughts
This challenge was very fun to work with. Coding in general is something I lack experience in, so I am glad I got to learn a few new things with python.
Having built the environment and reviewing it, I find that there are some improvements that could be had for this challenge / project. First is deploying into ECS and taking advantage of Docker containerization. With Docker, we could add Nginx and the application as two separate containers. Second would be using aws Code Pipeline, Deploy or Jenkins and pipeline for CI/CD into the ECS.
Many thanks to A Cloud Guru for presenting this challenge.
