Today we have started to change the way multimedia content is served on MastodonApp.UK to ensure we can continue to scale the site while gaining the best value for money from where we store and serve our content to you.
As of around 5PM on the 1st May 2023 we made the most significant change to our media content management since November of 2022 when we moved media from being stored locally on our original server to AWS S3 for storage. We have now started the first in a multi-step plan to migrate our media away from AWS S3 in an effort to save a significant cost we currently face. If all has gone to plan, you shouldn't notice any changes in the website.
The initial change that was deployed as the introduction of what we refer to as a "Media Proxy Server", this server is the first place your web browser will now reach when you're attempting to load an image on our website. This proxy server is configured to first look in our new object storage location (At the time of writing, an empty bucket) before attempting to load from the soon to be legacy AWS S3 endpoint before rendering a static error page if it can't load the content at all.
The major change which is yet to come is the actual migration of data from AWS S3 to our new object storage provider. The reason for writing this blog post is to help explain what this is going to look like and to ensure transparency on our part as we move your data from one provider and geolocation to another.
As it currently stands we are heavy users of Cloudflare primarily as their CDN which has helped the site stay online through some of the most significant peaks in traffic we've ever seen. Cloudflare have an object storage offering (Cloudflare R2) which is considerably cheaper than AWS S3 while still ensuring the durability and resiliency of data.
The major limitation for us moving to Cloudflare compared to S3 is we have less control over the physical location that the media will be stored, with AWS we deployed any of our assets (Including the S3 object store) to the AWS London Region (Eu-West-2) which ensured, like our servers, all data is primarily residing in the United Kingdom and specifically in London. With Cloudflare they will dynamically move data across their data centres based on what their systems thinks will give our users the best performance, while we have requested it favourites Western Europe data centres, we have limited control beyond this to more accurately specify where the data is residing.
We will over the coming months be evaluating other storage providers in an effort to identify if there is a better provider that can offer us the value for money we need to be able to sustain the site and persist your photo's and videos for the longer term while also giving us control over your data. Our first and highest priority is being able to share how and where we are storing and processing your data and ensuring we are doing so in a responsible and substantial way which enables us to continue operating the site.
As always if you have any questions or concerns please do let me know, you can always reach me on Mastodon my tagging my handle: @[email protected]
