🔒 Locked 🤖 Godot

[web] Amazon's AWS: EC2, EBS, S3, etc.

Started by BeanDog Jun 26, 2009 at 5:35 PM 10 replies 3.6k views

Original Post

Jun 26, 2009 at 5:35 PM

Has anyone else worked with Amazon's Web Services lately? About a month or two ago I moved LucidChart onto an EC2 instance with them. Here are some observations: You definitely need to be a system administrator to effectively use EC2 to host a traditional web site/web application. It took me about one full work day to get a basic LAMP stack up and running with all the cool features turned on, and I'm reasonably experienced in system administration in Linux. The feature set, though, is so compelling that I will never be back to a traditional shared/VPS/dedicated hosting provider. My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature. I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling. Creating new MySQL slaves is therefore a zero-pain procedure. If you've ever set up MySQL replication before, you'll understand how important that is. I lock the database, dump a SHOW MASTER STATUS to the persistent storage, lock the file system, take a snapshot, and unlock the file system and database. This takes about 1-2 seconds. I then create a new MySQL server instance from a server image I have created, attach a new EBS volume made from that snapshot, set the slave parameters based on what's in the master status file, and start up MySQL. It's pretty much up to date with the master already, and there was basically no downtime. Elastic IP addresses are what glue all this together. You can allocate up to 5 static IP addresses that you can remap instantly from one instance to another. So on major system upgrades, I just create a new instance from an image of my current instance, do whatever upgrade is necessary, and remap the production IP address to that instance. Done. Zero downtime. And I only get charged for exactly what I use! No more "500GB bandwidth included, $2/GB overages". I pay $0.17/GB with no minimums. I pay by the server-hour for processing power and by the GB/month for persistent and backup storage (again, no minimums). They recently introduced "reserved" instances, which mean you can reserve a certain instance type for 1 or 3 years (at up-front cost) and then pay dramatically-reduced rates. I'm paying $0.03/hr for my MySQL master, having paid $500 upfront to reserve that instance for three years. That works out to something like $35/mo for a 1.4Ghz Xeon with 1.7GB of RAM. One major frustration: You can't assign multiple elastic IP addresses to a single instance. So putting multiple SSL certificates on one Apache server is out (you can still get by with a UCC certificate for multiple domains). Anyway, anyone else used AWS lately?

Sander

Jun 27, 2009 at 2:27 AM

Nope. I'm pretty much the exact opposite of you. I don't like to be dependent on, or tied to anything. I run my own webserver, nameservers, jabber server, mailserver, etcetera. I even wrote my own SpamAssassin-like baysian spamfilter for my blog because I don't want to have to rely on external services like Akismet (I couldn't find any existing stand-alone comment spam filters). The only thing I need from the outside is an internet connection :-)

d000hg

Jul 6, 2009 at 4:21 PM

Sounds like it would be very cool, if I could ever get it set up. I'm an experienced developer but I hate system configuration and *nix especially.

Daerax

Jul 6, 2009 at 6:24 PM

Quote:
Original post by Sander
Nope. I'm pretty much the exact opposite of you. I don't like to be dependent on, or tied to anything. I run my own webserver, nameservers, jabber server, mailserver, etcetera. I even wrote my own SpamAssassin-like baysian spamfilter for my blog because I don't want to have to rely on external services like Akismet (I couldn't find any existing stand-alone comment spam filters). The only thing I need from the outside is an internet connection :-)

Thing is, if you are trying to run a business, especially a start up whose business is built around the whole software as a service meme you need to be able to guarantee bandwith and uptime. More importantly you want to be outsource as much work which is not your strength of focus as is possible. While managing finances, attempting to get new clients, going to meetings, doing marketing, writing the software, the option to not have to or pay people to run and manage a robust scalable data centre is a very attractive option.

@BeanDog

Thank you for the review. It would be good to see other contrasting opinions who have used this service. I have heard of and been skeptical of this cloud computing hype. However the extremely cheap overall price for the hosting you are getting with such a glowing report speaks extremely well of the service. The ability to scale both costs and resource usage with almost no effort on your part makes this option seem a clear winner. Not to mention that under this, a well engineered system will grow costs much slower than users and wasting of resources and over charging will likely be less of an issue.

I suppose cloud computing makes sense in the same vein that people no longer grow their own food or make their own clothes, people will no longer host their own data.

CodeMunkie

Jul 7, 2009 at 9:00 AM

I'm very interested in this. Thus far I have moved all of my storage to S3, but I I'm also seriously considering moving away from VPS and completely into a cloud based environment. You mention you are using EC2 as your web host. I've heard stories of EC2 instances going down, and of course there is no automatic failover. Are you using a framework such as weoceo, or did you roll your own to handle failover situations? Do you have separate EC2 instance(s) for your MySQL servers? Did you build your own AMI or use some of the off-the-shelf ones available, and if so how was the process? Sorry for the barrage of questions, but it is good to hear war stories from other guys!

Quote:
My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature.

I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling.

Can you provide any additional info or useful links on how you achieved this? This seems to be a critical component in making a cloud based server both scalable and fault tolerant, but I'm having trouble seeing how to implement.

choffstein

Jul 7, 2009 at 9:07 AM

I do remote back-ups to S3 for about ~$3 a month of all my important code and business documents. I've also made use of EC2 for massively parallel computations, and am currently trying to get the background processes of my website to dynamically scale using EC2...

I like the products. There is no way I could host a cloud for my business yet, so it is nice having EC2 at my service.

BeanDog

Jul 7, 2009 at 2:44 PM

Quote:
Original post by CodeMunkie
You mention you are using EC2 as your web host. I've heard stories of EC2 instances going down, and of course there is no automatic failover. Are you using a framework such as weoceo, or did you roll your own to handle failover situations?

Especially during my early experimentation and implementation, I frequently deeply, horribly crashed my EC2 instance (my own stupidity... locking an XFS file system, then trying to write to it). This was frustrating at the time, but it gave me a great opportunity to see how failover would work in practice. Once I understood the basic process, it was really easy to make it as automated as can be reasonable. I haven't yet set up completely automated failover, although apparently it's pretty trivial with the use of AWS's new elastic load balancers. At this point, it's me logging into AWS's web-based management console, clicking to start a new instance from my server image, SSH'ing in and running a script. It takes about a minute or two, even if I'm setting up the replacement in a different data center.

Quote:
Original post by CodeMunkie
Do you have separate EC2 instance(s) for your MySQL servers? Did you build your own AMI or use some of the off-the-shelf ones available, and if so how was the process?

I started with an off-the-shelf Fedora server image. I SSH'd in as root and installed and modified everything to be as I wanted it (e.g. moving webroot/MySQL data onto an EBS volume), then created an AMI from the server's current state. Creating an AMI, once I figured out how to do it, is actually pretty quick and painless. And the good news is that since everything in the file system that changes frequently (read: database) is on the EBS volume, I can make an updated AMI from the production server without any downtime.

Yes, I have since split my MySQL server from my web server for performance reasons.

Quote:
Original post by CodeMunkie
Quote:
My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature.

I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling.

Can you provide any additional info or useful links on how you achieved this? This seems to be a critical component in making a cloud based server both scalable and fault tolerant, but I'm having trouble seeing how to implement.

Creating EBS volumes and attaching them to instances can be done very easily using AWS's new online Management Console. Doing it automatically is also pretty easy using the command-line tools that Amazon provides for Linux (also Windows?).

It's important to use an XFS or other journaling, lockable file system to get real reliable EBS snapshots. Here's my actual EBS backup script that performs as I described:

#!/bin/bashexport EC2_HOME=/root/ec2-api-tools-1.3-36506/export JAVA_HOME=/usr/java/jre1.6.0_13/echo "SHOW MASTER STATUS;" > /root/showmasterstatus.sqlmysql -pXXXXXXXXXXX <<EOFPURGE BINARY LOGS BEFORE DATE_ADD(NOW(), INTERVAL -15 MINUTE);FLUSH TABLES WITH READ LOCK;SYSTEM mysql -pXXXXXXXXXXXX >/vol/aws/masterstatus </root/showmasterstatus.sqlSYSTEM xfs_freeze -f /volSYSTEM /root/ec2-api-tools-1.3-36506/bin/ec2-create-snapshot -K /vol/aws/private.pem -C /vol/aws/public.pem vol-45fa172cSYSTEM xfs_freeze -u /volUNLOCK TABLES;EOF

Basically, the whole script is run through the MySQL CLI: First, I purge old binary logs (just as a space-saving measure). I then FLUSH TABLES WITH READ LOCK, which makes sure everything's written to disk and the database is locked against new writes. I then dump a SHOW MASTER STATUS to the EBS volume, so that if I set up a replication slave from this snapshot, I'll have all the information I need to have the slave catch up to the master from this snapshot's state. I freeze the file system and create the snapshot (vol-45fa172c is the EBS volume's ID number), and unfreeze the file system and database.

Not rocket science, but it took me a bit to figure that one out. As for rotating those backups, here's a script I made for that purpose:

#!/usr/bin/php -q<?phpif(sizeof($argv) != 3) {  die("Usage: $argv[0] snapshot-id total-backups-to-keep\nExample: $argv[0] vol-45fa172c 100\n");}$Snapshots = array(); //Snapshot ID => Created Timestampputenv('EC2_HOME=/root/ec2-api-tools-1.3-36506/');putenv('JAVA_HOME=/usr/java/jre1.6.0_13/');$data = `/root/ec2-api-tools-1.3-36506/bin/ec2-describe-snapshots -K /vol/aws/private.pem -C /vol/aws/public.pem`;$data = split("\n", $data);foreach($data as $row) {  if(trim($row) == '') continue;  list($junk,$SnapshotId,$VolumeId,$Status,$Created,$Progress) = split("\t", $row);  if($VolumeId == $argv[1] && $Status == 'completed') {    echo "Found snapshot started at ".date('m/d/Y g:iA', strtotime($Created))."\n";    $Snapshots[$SnapshotId] = strtotime($Created);  }}$Keep = PruneBackups($Snapshots, $argv[2]);//FYI: Date/time of backups to be kept.arsort($Keep);foreach($Keep as $id => $date)  echo date('m/d/Y hA', $date)."\n";//Delete the ones not in $Keepforeach($Snapshots as $id => $date) {  if(!isset($Keep[$id])) {    echo "Kill $id - ".date('m/d/Y hA', $date)."\n";    echo `/root/ec2-api-tools-1.3-36506/bin/ec2-delete-snapshot -K /vol/aws/private.pem -C /vol/aws/public.pem $id`;  }}/*We will keep 1/4 of the total requested backups from the last 24 hours, keeping all of the most recent.We will keep 1/2 of the total requested backups from the last 30 days, one per day.We will keep 1/4 of the total requested backups from older than 30 days, one per week.Any left over will go to the newest backups available.*/function PruneBackups($Snapshots, $Total, $Now=NULL) {  if($Now === NULL) $Now = time();  $ret = $Snapshots;  $Remaining = $Total;  arsort($Snapshots);  $UsedDays = array();  $UsedWeeks = array();  foreach($Snapshots as $id => $Created) {    if($Created >= strtotime('-1 day', $Now)) {      if($Remaining > $Total * 0.75) {        $UsedDays[date('Y-m-d', $Created)] = true;        $UsedWeeks[date('Y-W', $Created)] = true;        $Remaining--;        continue;      }    }    if($Created >= strtotime('-30 days', $Now)) {      if($Remaining > $Total * 0.25 && !isset($UsedDays[date('Y-m-d', $Created)])) {        $UsedDays[date('Y-m-d', $Created)] = true;        $UsedWeeks[date('Y-W', $Created)] = true;        $Remaining--;        continue;      }    }    if($Remaining > 0 && !isset($UsedWeeks[date('Y-W', $Created)])) {      $UsedWeeks[date('Y-W', $Created)] = true;      $Remaining--;      continue;    }    unset($ret[$id]);  }  arsort($Snapshots);  foreach($Snapshots as $id => $Created) {    if(!isset($ret[$id])) {      $ret[$id] = $Created;      $Remaining--;      if($Remaining <= 0)        break;    }  }  return $ret;}?>

I take backups hourly, and I keep 96 backups of each of my two EBS volumes (one is MySQL data, other is source/other webserver stuff). This script keeps the rotation sane, as described in a comment in the script.

As for setting up MySQL on XFS on EBS on EC2 in general, here's a tutorial I roughly followed: clicky

That tutorial was crucial in getting me in the right direction, but I had to do some things slightly differently--though I couldn't really tell you now exactly what.

Disclaimer: My scripts work for me. They may or may not work for you.

Sander

Jul 7, 2009 at 5:52 PM

Quote:
Original post by Daerax
Thing is, if you are trying to run a business, especially a start up whose business is built around the whole software as a service meme you need to be able to guarantee bandwith and uptime. More importantly you want to be outsource as much work which is not your strength of focus as is possible. While managing finances, attempting to get new clients, going to meetings, doing marketing, writing the software, the option to not have to or pay people to run and manage a robust scalable data centre is a very attractive option.

Attractive, but not smart. If you're running SaaS applications then hosting is part of your core business. If something is part of your core business then you should do it yourself. If you don't have the skill to do it yourself then you shouldn't be in the SaaS business but in the application business. Here's a good article about that.

Quote:
Find the dependencies -- and eliminate them ... If it's a core business function -- do it yourself, no matter what.

CodeMunkie

Jul 8, 2009 at 11:12 AM

Great info all around! Sander, do you mean that you have your own data center? What were the challenges setting that up and what were your capital costs? Also, how many users are you serving? Any issues scaling up? Love to hear alternative approaches.

BeanDog

Jul 8, 2009 at 12:28 PM

Quote:
Original post by Sander
Attractive, but not smart. If you're running SaaS applications then hosting is part of your core business. If something is part of your core business then you should do it yourself. If you don't have the skill to do it yourself then you shouldn't be in the SaaS business but in the application business. Here's a good article about that.

Quote:
Find the dependencies -- and eliminate them ... If it's a core business function -- do it yourself, no matter what.

I completely disagree.

The core business of modern SaaS is the production of performant, scalable applications served through a web browser. The core business is not building and maintaining multiple data centers around the world, any more than the core business is developing a highly-performant and ACID-compliant database engine.

I'm not interested in recreating Oracle or MySQL. And I'm definitely not interested in maintaining physical network infrastructure. Some dependencies really are pragmatic.

In contrast, Google App Engine is an example of a horribly vendor-specific and locked-in dependency. To get the advantages of the app engine, you have to give up your choice of hosting options, and even take their database engine into account in your application's architecture.

If Amazon shut down AWS next month, I could be back up and running on Rackspace in a few days. If Google cut off their App Engine, their users are utterly screwed.

Daerax

Jul 8, 2009 at 1:16 PM

Quote:
Original post by Sander
Quote:
Original post by Daerax
Thing is, if you are trying to run a business, especially a start up whose business is built around the whole software as a service meme you need to be able to guarantee bandwith and uptime. More importantly you want to be outsource as much work which is not your strength of focus as is possible. While managing finances, attempting to get new clients, going to meetings, doing marketing, writing the software, the option to not have to or pay people to run and manage a robust scalable data centre is a very attractive option.

Attractive, but not smart. If you're running SaaS applications then hosting is part of your core business. If something is part of your core business then you should do it yourself. If you don't have the skill to do it yourself then you shouldn't be in the SaaS business but in the application business. Here's a good article about that.

Quote:
Find the dependencies -- and eliminate them ... If it's a core business function -- do it yourself, no matter what.

I do not agree with this conclusion and neither does the article. If your core competency was something where maintaining your own robust, scalable, secure, localized, fast data centres was key then clearly you offer little value by simply reselling some cloud space. But if you are not trying to build a google, yahoo or even a facebook then it makes little business sense to waste resources on this. Again, especially for start ups any capital expenditure that can be eliminated is good news. Outsourcing this also cuts many running expenses (electricity, bandwidth, employees, even time).

As well if one performed their due diligence they would be foolish to not notice the direction the market is going. Uptake of computing power is moving backwards if anything. The sale of new PCs and high end laptops has all but flattened, instead netbooks are gaining in popularity. Smartphone sales as well are increasing. People are expecting more and more that their applications will run off a browser (however sad that is) and can be bothered less and less to do local installs. This is the answer to the age old fact that any giving grandmother does not need a PC with 9210 cores to simply check her email.

Thus if you are launching a product that will be targeted at personal, small and even medium business then it makes sense to take the path of least resistance and most rewards. Thus targeting the browser. There are other benefits to this as well. You have more control over the platform your application will run, less trouble with those pesky configurations. You attack piracy by more tightly controlling distribution and use, your app will run on a wide variety of platforms depending on how thin the client is - smartphones, pcs, netbooks, readers, tablets, even tvs; plus you will be able to try new business models particularly suited to things digital - subscriptions, freemium, micropayments and all that jargon.

Seeing the way things are going and the benefits there will be little doubt that many startups (not all but enough) are going to target browsers not wanting to be sidelined. Seeing that, it makes absolutely no sense economically (resources would definitely not be allocated as efficiently as possible) or even environmentally for every Saas business to run their own data centres.

This is why everyone is making a big deal about cloud computing with trend milking industries springing up to 'advice' businesses on how to move to the clouds. how romantic. no suprise that google, amazon and microsoft are all fighting and stumbling over themselves to become market leader. they see where the future is.

Sander

Jul 9, 2009 at 2:17 AM

Quote:
Original post by CodeMunkie
Great info all around! Sander, do you mean that you have your own data center?

No. It's a co-located server, meaning that the hardware is mine. I get power and an internet pipe from the ISP. The rest is mine. I run my own nameservers for example.

Quote:
I'm not interested in recreating Oracle or MySQL. And I'm definitely not interested in maintaining physical network infrastructure. Some dependencies really are pragmatic.

No need to recreate Oracle or MySQL. But you want to control it. Database performance for example is very dependent on the exact configuration, hardware, filesystem used, drivers, etcetera. The same goes for webserver performance. When you scale up your website enough that you require multiple servers then this low level stuff becomes even more important.

@Daerax: I think you missed my point. I'm not saying SaaS isn't any good. I'm just saying that if you go the SaaS way then you will want to control the stack you're using from the hardware up so your application packs the biggest punch possible.

Maybe it's my paranoid streak, but I just don't trust the cloud. Anyway, I think I have dragged this thread off topic for long enough. Sorry BeanDog. Let's get back to discussing EC2.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.