We build, develop and support software for national and international organisations and HNW Individuals.
May 01
8:27 AM
AWS Cost Reduction
We have to continually keep an eye on AWS Costs and where possible we have to make changes in applications to find a more cost efficient method to getting the same output. Below is an example of reducing the size of an RDS Instance and implementing AWS Elasticache to take some of the database load. The end result $2,400 in savings a month.
One of our clients has a pretty significant YouTube Channel. We manage the full backend of this channel for them. Including the live streaming and the copyright management. Probably not the usual hardcore technical features that are expected of us but allows them to focus on creating their content which creates tens of thousands of AdSense revenue each month.
One of our clients, is a nationwide recruiter and we support them on all aspects of their own in house built ERP, which allows them to manage thousands of staff at any one time. We are rebuilding one aspect of this platform which allows for the management of staff to individual shifts. We are designing this using REACT JS, and AWS SQS. React ensure the state stays the same without having to do multiple refreshes with requests from the database/server. AWS SQS ensures we can manage all the POST/UPDATE requests that the application makes without any lack of performance.
Have recently been asked to help improve the search speed for a company that sells personalised number plates. Currently was using a MySQL RDS Instance, where each table had 53million + rows. However good the indexing there is only so much you can do with MySQL. This is where we fired up AWS DMS, which automatically streams any updates on the MySQL Instance and fires them into Opensearch (Scalable search, analytics engine for real-time data). The WebApp then queries the Opensearch domain. Gone from 10s + query to under 300ms, and we can now also reduce the instance size of RDS to save some additional £££.
One of those annoying jobs that has to be done once in a while, updating OS to the latest version. Most of our clients run off AWS Linux instances, with AL2 coming to end of life in June this year. So been ensuring any old instances are up to date, which is AWS Linux 2023. Feels like a spring clean but important to ensure everything is optimal
Perhaps one of the frequent requests we get is in regards to performance. Its very easy to solve performance issues by throwing lots of cash at hardware resources, who wants an ongoing increase in their monthly spend. Optimisation and in time we have to start splitting the load of web applications across multiple microservices. Caching, queuing and lambda are the main ones we use. However decent indexing of DB’s is a must. As well as a healthy use of CRON’s to build caches away from client requests. All in a days work
In order to be hands off with this, there are a number of scenarios we are having to cater for to ensure that if there is a problem with the multi step process then we can fix automatically without having to raise a software engineer to debug and fix. Essentially we are putting a virtual “lock” in place for every file as we go through the process. The “lock” is only released when we have received the streaming url back from AWS SNS letting us know the job is complete. Now we are going to leave this running for a good few days to see if we receive any hiccups if not then we can move to the next phase of making the content available for publishing to the customers Content Management Platform.*The attached is a processing log table we have used to record every single step of the process, helps us massively to debug
This work has now completed, actually its all rather impressive. In the end we decided to split the task into 4 distinct areas. DownloadExtract AudioTranscribeSynopsis Generation and EncodingThe transcription is running from an old mining rig so AWS not needed, we have baked this into the back up int he case of failure. However as the workflow is not time sensitive we dont have to worry about it too much. The real art is taking a complex process and make it simple. We are using www.clearcron.com to help with our automation and CRON management.
A client has requested that they want to auto publish their ‘tv shows’ onto their streaming platform. Wait for itMAM>Download file>ffmpeg extract audio>create transcription from audio>send audio to AI model to create synopsis>original file for transcoding>payload to API endpoint to create a new item in the CMS.Want to keep costs as low as possible, so got a few 3070RTX GPU’s which will smash the transcriptions using Whisper. This week have mostly been proving all the steps, which are working then to add them together next week.
As part of the awards season, yesterday evening we took a live video feed for the BAFTA’s and distributed to a Nigerian only audience. Plenty of tech was used, including FairPlay, WideVine and PlayReady DRM Profiles. Also we had to use a combination of methods to ensure the stream could only be accessed within the licensed region, including geo location and account authentication. As always plenty of AWS Services. The show went off without a hitch
We have to continually keep an eye on AWS Costs and where possible we have to make changes in applications to find a more cost efficient method to getting the same output. Below is an example of reducing the size of an RDS Instance and implementing AWS Elasticache to take some of the database load. The end result $2,400 in savings a month.
One of our clients has a pretty significant YouTube Channel. We manage the full backend of this channel for them. Including the live streaming and the copyright management. Probably not the usual hardcore technical features that are expected of us but allows them to focus on creating their content which creates tens of thousands of AdSense revenue each month.
One of our clients, is a nationwide recruiter and we support them on all aspects of their own in house built ERP, which allows them to manage thousands of staff at any one time. We are rebuilding one aspect of this platform which allows for the management of staff to individual shifts. We are designing this using REACT JS, and AWS SQS. React ensure the state stays the same without having to do multiple refreshes with requests from the database/server. AWS SQS ensures we can manage all the POST/UPDATE requests that the application makes without any lack of performance.
Have recently been asked to help improve the search speed for a company that sells personalised number plates. Currently was using a MySQL RDS Instance, where each table had 53million + rows. However good the indexing there is only so much you can do with MySQL. This is where we fired up AWS DMS, which automatically streams any updates on the MySQL Instance and fires them into Opensearch (Scalable search, analytics engine for real-time data). The WebApp then queries the Opensearch domain. Gone from 10s + query to under 300ms, and we can now also reduce the instance size of RDS to save some additional £££.
One of those annoying jobs that has to be done once in a while, updating OS to the latest version. Most of our clients run off AWS Linux instances, with AL2 coming to end of life in June this year. So been ensuring any old instances are up to date, which is AWS Linux 2023. Feels like a spring clean but important to ensure everything is optimal
Perhaps one of the frequent requests we get is in regards to performance. Its very easy to solve performance issues by throwing lots of cash at hardware resources, who wants an ongoing increase in their monthly spend. Optimisation and in time we have to start splitting the load of web applications across multiple microservices. Caching, queuing and lambda are the main ones we use. However decent indexing of DB’s is a must. As well as a healthy use of CRON’s to build caches away from client requests. All in a days work
In order to be hands off with this, there are a number of scenarios we are having to cater for to ensure that if there is a problem with the multi step process then we can fix automatically without having to raise a software engineer to debug and fix. Essentially we are putting a virtual “lock” in place for every file as we go through the process. The “lock” is only released when we have received the streaming url back from AWS SNS letting us know the job is complete. Now we are going to leave this running for a good few days to see if we receive any hiccups if not then we can move to the next phase of making the content available for publishing to the customers Content Management Platform.*The attached is a processing log table we have used to record every single step of the process, helps us massively to debug
This work has now completed, actually its all rather impressive. In the end we decided to split the task into 4 distinct areas. DownloadExtract AudioTranscribeSynopsis Generation and EncodingThe transcription is running from an old mining rig so AWS not needed, we have baked this into the back up int he case of failure. However as the workflow is not time sensitive we dont have to worry about it too much. The real art is taking a complex process and make it simple. We are using www.clearcron.com to help with our automation and CRON management.
A client has requested that they want to auto publish their ‘tv shows’ onto their streaming platform. Wait for itMAM>Download file>ffmpeg extract audio>create transcription from audio>send audio to AI model to create synopsis>original file for transcoding>payload to API endpoint to create a new item in the CMS.Want to keep costs as low as possible, so got a few 3070RTX GPU’s which will smash the transcriptions using Whisper. This week have mostly been proving all the steps, which are working then to add them together next week.
As part of the awards season, yesterday evening we took a live video feed for the BAFTA’s and distributed to a Nigerian only audience. Plenty of tech was used, including FairPlay, WideVine and PlayReady DRM Profiles. Also we had to use a combination of methods to ensure the stream could only be accessed within the licensed region, including geo location and account authentication. As always plenty of AWS Services. The show went off without a hitch
