Media sharing is one of the hottest markets on internet. Customers have a staggering appetite for placing photos and video on social networking sites and sharing their photo/video in custom online photo/video albums.
The growing popularity of media sharing means scaling problem for site owners who face ever-increasing storage and bandwidth requirements and increased go-to-market pressure to deliver faster than the competition.
In this entry we will discuss an example of a highly available, durable and cost effective media sharing and processing platform using amazon web service.
Sharing content file involves uploading media files to online service. In this configuration, an elastic load balancer distributes incoming network traffic to upload servers, a dynamic fleet of Amazon Elastic Compute Cloud (Amazon EC2) instances. Amazon CloudWatch monitors these servers and Auto-Scaling manages them, automatically scaling EC2 capability up or down based on load. In the above diagram, a separate end-point to receive media upload was created in order to off-load this task from the website's servers.
Original uploaded files are stored in Amazon Simple Storage Service (Amazon S3) , a high available and durable storage service.
To submit a new file to be processed, upload web servers push a message into a Amazon Simple Queue Service (Amazon SQS) queue. This queue acts as a communication pipeline between file reception and file processing components.
The processing pipeline is a dedicated group of Amazon EC2 instances used to execute any kind of post processing task on the uploaded files ( eg. video transcoding, image resizing etc). To automatically adjust the needed capacity, Auto Scaling manages this group. We can use Spot instances to dynamically extend the capacity of the group and to significantly reduce the cost of file processing.
Once the processing is completed. Amazon S3 stores the output files. Original files can be stored with high durability.
Media related files can be put in a relational database like Amazon Relational Database Service (Amazon RDS) or in key-value store like Amazon SimpleDB.
A third fleet of EC2 instances is dedicated to host the website front- end of media sharing service.
Media files are distribute from Amazon S3 to end user via Amazon CloudFront offers low latency delivery through a worldwide network of edge locations.
The growing popularity of media sharing means scaling problem for site owners who face ever-increasing storage and bandwidth requirements and increased go-to-market pressure to deliver faster than the competition.
In this entry we will discuss an example of a highly available, durable and cost effective media sharing and processing platform using amazon web service.
Sharing content file involves uploading media files to online service. In this configuration, an elastic load balancer distributes incoming network traffic to upload servers, a dynamic fleet of Amazon Elastic Compute Cloud (Amazon EC2) instances. Amazon CloudWatch monitors these servers and Auto-Scaling manages them, automatically scaling EC2 capability up or down based on load. In the above diagram, a separate end-point to receive media upload was created in order to off-load this task from the website's servers.
Original uploaded files are stored in Amazon Simple Storage Service (Amazon S3) , a high available and durable storage service.
To submit a new file to be processed, upload web servers push a message into a Amazon Simple Queue Service (Amazon SQS) queue. This queue acts as a communication pipeline between file reception and file processing components.
The processing pipeline is a dedicated group of Amazon EC2 instances used to execute any kind of post processing task on the uploaded files ( eg. video transcoding, image resizing etc). To automatically adjust the needed capacity, Auto Scaling manages this group. We can use Spot instances to dynamically extend the capacity of the group and to significantly reduce the cost of file processing.
Once the processing is completed. Amazon S3 stores the output files. Original files can be stored with high durability.
Media related files can be put in a relational database like Amazon Relational Database Service (Amazon RDS) or in key-value store like Amazon SimpleDB.
A third fleet of EC2 instances is dedicated to host the website front- end of media sharing service.
Media files are distribute from Amazon S3 to end user via Amazon CloudFront offers low latency delivery through a worldwide network of edge locations.
1. Salve
ReplyDeleteInteresting piece!Great to see someone write #topic who is not a fanatic or a complete skeptic.
I enjoy reading the various AWS blogs and staying up to date with new offerings and best practices. I typically go to the root of the blog site and check the "Latest Posts" section at the bottom.
It looks like the "Latest Posts" section stopped updating about 2 weeks ago on April 20th. It would be very helpful if this could be fixed since this was very useful.
Excellent tutorials - very easy to understand with all the details. I hope you will continue to provide more such tutorials.
Merci Beaucoup,
Sain Bainuu,
ReplyDeleteThanks for highlighting this and indicating about #topic where more study and thought is necessary.
I started using this AWS Training blog for my training practice.
If I buy a Route 53 domain name, can I create an email address and use it with gmail? If so, how?
I do not want to use Workmail, or SES or an EC2 instance.
All I want is a username, password, and SMTP and whatever else is needed to link my route 53 email to gmail.
Very useful article, if I run into challenges along the way, I will share them here.
Cheers,
Morgan
Hey Brother,
ReplyDeleteSeems like I won the lottery here….This is a treasure box of blogs and your folks are like leprechauns! Phenomenal read on How to get 10 from 9!
We have a t2.micro instance (built from amzn-ami-hvm-2016.09.1.20170119-x86_64-gp2) on which a customized WordPress is deployed. Following various posts here and elsewhere, various parameters for Apache and MySQL have been set to accommodate the 'size' of the host, including the deployment of a swap partition. The site has been running without issue for weeks.
I'm currently using AWS Training USA blog for my training practice.
Please keep providing such valuable information.
Shukran,
Morgan
Salemetsiz Be,
ReplyDelete11/10!! Your blog is such a complete read. I like your approach with Design content and media sharing system. Clearly, you wrote it to make learning a cake walk for me. AWS Tutorial
I like the support for S3 in the AWS console, especially being able to apply ACLs at time of upload. However, it would be very good if the view was sortable by last modified date, as I am typically working with the last 10-15 files uploaded. AWS Training
It was cool to see your article pop up in my google search for the process yesterday. Great Guide.
Keep up the good work!
Cheers,
Radhey
Hi Bru,
ReplyDeleteSuch vivid info on the Print list items containing all characters of a given word | GeeksforGeeks! Flabbergasted! Thank you for making the read a smooth sail!
The Make Public function in S3 is perfectly working these past few months and years I should say.
Just last week, we are experiencing failure in the "Make Public" function. Once I successfully upload a folder with multiple files) then Make Public the uploaded files, the notification below shows Failure
Anyways great write up, your efforts are much appreciated.
MuchasGracias,
Morgan