Apr 10

3:19 PM

REACT UI for ERP

One of our clients, is a nationwide recruiter and we support them on all aspects of their own in house built ERP, which allows them to manage thousands of staff at any one time. We are rebuilding one aspect of this platform which allows for the management of staff to individual shifts. We are designing this using REACT JS, and AWS SQS. React ensure the state stays the same without having to do multiple refreshes with requests from the database/server. AWS SQS ensures we can manage all the POST/UPDATE requests that the application makes without any lack of performance.

Apr 07

10:26 AM

Search Improvements

Have recently been asked to help improve the search speed for a company that sells personalised number plates. Currently was using a MySQL RDS Instance, where each table had 53million + rows. However good the indexing there is only so much you can do with MySQL. This is where we fired up AWS DMS, which automatically streams any updates on the MySQL Instance and fires them into Opensearch (Scalable search, analytics engine for real-time data). The WebApp then queries the Opensearch domain. Gone from 10s + query to under 300ms, and we can now also reduce the instance size of RDS to save some additional £££.

Mar 28

12:21 PM

AWS Instance Update

One of those annoying jobs that has to be done once in a while, updating OS to the latest version. Most of our clients run off AWS Linux instances, with AL2 coming to end of life in June this year. So been ensuring any old instances are up to date, which is AWS Linux 2023. Feels like a spring clean but important to ensure everything is optimal

Mar 19

7:48 AM

Performance

Perhaps one of the frequent requests we get is in regards to performance. Its very easy to solve performance issues by throwing lots of cash at hardware resources, who wants an ongoing increase in their monthly spend. Optimisation and in time we have to start splitting the load of web applications across multiple microservices. Caching, queuing and lambda are the main ones we use. However decent indexing of DB’s is a must. As well as a healthy use of CRON’s to build caches away from client requests. All in a days work

Mar 06

1:41 PM

Content Auto Publishing (Testing & Monitoring)

In order to be hands off with this, there are a number of scenarios we are having to cater for to ensure that if there is a problem with the multi step process then we can fix automatically without having to raise a software engineer to debug and fix. Essentially we are putting a virtual “lock” in place for every file as we go through the process. The “lock” is only released when we have received the streaming url back from AWS SNS letting us know the job is complete. Now we are going to leave this running for a good few days to see if we receive any hiccups if not then we can move to the next phase of making the content available for publishing to the customers Content Management Platform.*The attached is a processing log table we have used to record every single step of the process, helps us massively to debug

Mar 03

10:26 AM

Content Auto Publishing

This work has now completed, actually its all rather impressive. In the end we decided to split the task into 4 distinct areas. DownloadExtract AudioTranscribeSynopsis Generation and EncodingThe transcription is running from an old mining rig so AWS not needed, we have baked this into the back up int he case of failure. However as the workflow is not time sensitive we dont have to worry about it too much. The real art is taking a complex process and make it simple. We are using www.clearcron.com to help with our automation and CRON management.

Feb 21

5:00 PM

Content Auto Publishing

A client has requested that they want to auto publish their ‘tv shows’ onto their streaming platform. Wait for itMAM>Download file>ffmpeg extract audio>create transcription from audio>send audio to AI model to create synopsis>original file for transcoding>payload to API endpoint to create a new item in the CMS.Want to keep costs as low as possible, so got a few 3070RTX GPU’s which will smash the transcriptions using Whisper. This week have mostly been proving all the steps, which are working then to add them together next week.

Feb 17

5:54 PM

BAFTA's Streaming

As part of the awards season, yesterday evening we took a live video feed for the BAFTA’s and distributed to a Nigerian only audience. Plenty of tech was used, including FairPlay, WideVine and PlayReady DRM Profiles. Also we had to use a combination of methods to ensure the stream could only be accessed within the licensed region, including geo location and account authentication. As always plenty of AWS Services. The show went off without a hitch

Feb 11

5:51 PM

Dashboard Performance (cont)

After last weeks AWS setup, we have data now coming out of this setup but in order to help improve performance we are also using Elasticache. We are using a CRON to schedule an update every 15 minutes, which gets the data from Athena and caches this for millisecond performance on write back into the app. Wish sometimes it was simpler for these things, but all we can try and do is to maintain the highest level of performance while ensuring the minimal cost for the client. Enough micros services for now, but overall very pleased with the result. 👍👍👍

Feb 09

8:14 AM

Dashboard Performance

One of our clients has a very real requirement for a real time dashboard, essentially giving their staff all the information they need to ensure that the margins they are working too are met and tracked week by week. This data is stored in an RDS Cluster, however to get the data we need requires scanning some 45 tables with some tables having 50million rows of data in each. Which while it works to query the database is a little slow, a different solution is required. We are using AWS Glue to get data from the cluster, which stores the data on S3 as parquert files, will then use Athena to get the data into the app via an API. A performance improvement 100 times quicker than querying the database alone. 🚀🚀