Behold Sinners! The Apocalypse Aproacheth. No in all seriousness if you run a managed hosting company then your time is officially ‘up’. You won’t survive the coming hosting Apocalypse. Here’s why.

There are a few companies you may have heard of building large compute grids for consumption by the general public. They’re calling them their Cloud Computing products. IBM is building BlueCloud, Microsoft is building the Mesh, Amazon already has EC2, and Google has AppEngine. AppEngine is in a slightly different category than the others and the BlueCloud details are sparse, but they’re still worth mentioning. Of more immediate interest are Amazon and Microsoft’s solutions.

Microsoft is currently building their famous 300,000 server Data Center in Chicago. That’s roughly 3 times the number of servers that Google has. Microsoft has also announced several other Data center projects - each worth about $500 Million. It’s fair to say that that’s a lot of computing power, and it’s not all for MSN - Microsoft is planning on providing their platform in the cloud.

The real question is what Amazon will do when the Windows Cloud comes online. Microsoft has enough money in the bank to provide their 300,000 servers to customers for *years* without earning a single cent. That implies they can offer services at super low rates; Low enough to at least compete with Amazon’s EC2, which will support the Windows Server OS in fall 2008.

What happens with two huge cloud hosting companies get into a price war?

In the interest of self preservation they won’t make their services commodities - at least right away. But it won’t even matter. When you’re as big as Amazon, Microsoft, Google or IBM, you can afford to buy servers in such massive quantities that you could make money selling compute time for 10$ a month. The hosting space will change forever, because Amazon will eventually drop their prices by an *order of magnitude* and that has dire implications for the rest of the Mom’n'Pop hosting companies.

If thousands of companies can’t compete with Microsoft or Amazon on price, and they can’t compete in terms of convenience, then why would anyone use them? If you have to buy individual servers, or even servers by the rack, then you’re not going to get the price you need to be able to compete. You also don’t have access to the handful of specialized individuals and hardware required to make things work on such a grand scale.

The only answer is for all the smaller players to band together - to create a Federated Hosting environment, where together they can provide services that begin approaching levels of service and power that the Big 4 will offer.

Either way, we’re in an interesting period in the industry. Computing and the infrastructure of technology has become such a requirement for the economy that it will eventually become a general utility. The real question is who will be around.

Do you think it’s the end? We’re working on the answer, and your opinion is important.

I’m looking for some help answering some questions to help LayerBoom and some associated projects determine what companies and individuals really want in their own “Cloud Computing” environment.

I know the Term has been getting worn out, so to summarize, LayerBoom is providing software that helps companies build utility computing environments using existing hardware and network infrastructure. The idea is to provide software as a common platform, then provision resources across many distributed environments, and enable companies and academic environments to rent/share unused resources for extra revenue.

There’s a survey posted up at http://survey.layerboom.com

I’d appreciate your help and time!

-Trevor

(Also posted on http://trevoro.ca)

Through their Startup Essentials program, companies can get access to Sun gear and resources at heavily discounted rates. That’s great, because Sun gear is pretty much the best out there, and being competitive with other companies like Dell and HP on price will help get their technology into more Data Centers and up and coming businesses. But like all shiny objects, servers lose their luster. Even if they’re free.

In order to run a server you have to secure colocation space, pay for bandwidth, buy some switching gear and a router, and depending on your setup get UPS power. Sure you can get cheap switches to connect your high-end servers, but depending on your work load it would be like running a jet engine on bacon fat.

If you’re running more than a few servers you’ll probably need to get more than the standard ~3000W/Rack - especially if you have a SAN device. That means one full rack with extra power. Most older data centers don’t have the cooling capacity to handle todays dense and powerful systems, so your full rack will probably be half full (if you’re lucky), then you’ll have to get another rack if you want to expand. That involves more waiting. Average amount of time for a colo to provision a new rack? 4-6 weeks. You’ll also need to pay for installation, and every colo provides space on contract so you’re committed for a year. Time is money, and waiting 4-6 weeks to be able to expand your environment means you have less flexibility.

There are certain scenarios where getting colo still makes sense. If you’re running your own hosting environment *as a business* then obviously having the control over your environment is necessary. Companies with certain regulatory or security requirements will also need to stick with colo, but otherwise, why lock yourself into contracts and inflexible environments and hire extra staff to manage that operation, when you can just rent some servers en-masse and get super cheap bandwidth?

If you run the numbers you can get just as much if not more *power* for the same amount of money from companies like Joyent, Amazon, ServerBeach or Rackspace, and not have to worry about contracts. While it’s nice to have an awesome piece of kit, sometimes it isn’t worth the time or money.

Amazon calls their instance images Amazon Machine Images or AMI’s. If you want to be able to grab one of the many images from Amazon you can use download the Amazon AMI tools and AWS tools then do the following. You can download the tools here

Find and Download the AMI


$: ec2-describe-images
IMAGE   ami-cc6386a5    ubuntu-hardy-ruby/image.manifest.xml    848278689040    available       private         i386    machine
IMAGE   ami-386c8951    ubuntu-ruby-lapack/image.manifest.xml   848278689040    available       private         i386    machine
$:


Fields 3 and 4 contain important information. For this example I’m listing the images that I own. Optionally you can provide a switch that will list all Amazon images by including ‘-a’ to the end of the ec2-describe-images command.

Field 3 is the unique identifier for the AMI, and field 4 is the bucket and AMI “manifest” - or a file that describes the AMI. Because users can specify the name of the manifest, you should pay attention to this value when trying to run the next set of commands.

The AMI tools from Amazon include a utility called ‘ec2-download-bundle’. This will download the manifest file from the bucket, parse through to see what other files it needs to download, then it will reassemble the AMI image locally, and check its signature. The AMI’s are encrypted in small (usually 10 meg) chunks. The signatures for those chunks are also included in the manifest.

To download the first AMI listed above, run the following commands


mkdir 'image-to-download'
cd 'image-to-download'
ec2-download-bundle --bucket ubuntu-ruby-lapack -m image.manifest.xml --access-key $AWS_ACCESS_KEY --secret-key $AWS_SECRET --privatekey $EC2_PRIVATE_KEY


That will start downloading the bundle to your local system.

Rebuild the AMI

Now we have to unbundle the files

ec2-unbundle -m image.manifest.xml -k $EC2_PRIVATE_KEY


This will decrypt and reassemble the image from all the individual components in the list

Now you have an image named ‘image’ in your directory. You can take a look at this file by mounting it

mkdir /mnt/image
mount -t ext3 -o loop image /mnt/image
cd /mnt/image


If you’re lucky there will be copies of the kernel and perhaps the ramdisk in the /boot partition. Otherwise you’ve got to do something really tricky : You have to guess as to what kernel will work the best. Thankfully we have a good understanding of what’s required to boot one of these images.

If you’ve created an image for Xen already then chances are your kernel will work just fine, but your ramdisk might need some adjusting. A trick you can use is to chroot to the /mnt/image folder, specify which modules you want loaded and rebuild the ramdisk - then exit the chroot, copy the kernel and ramdisk out of /mnt/image and you’ll have all the components you’ll need.

I know what you’re thinking: That’s a lot of work / guessing

You’re in luck. While there are a couple sites for sharing pre-built Xen images, the community is nowhere near as large as the Parallels or VMware ‘appliance’ sites. Jailtime.org has a hanful of images but they don’t follow any sort of standard, and the disk layouts / configurations aren’t compatible with Amazon’s EC2.

LayerBoom has a Xen image that is completely compatible with Amazons AMI format, and it can run in your own environment. This means you can copy a system into Amazon from your test environment without any hassle. It also works with the Eucalyptus project, and can be booted in xVM server as well (Instructions are coming)

Download the Xen package

url: http://layerboom.com/files/xen/images/centos52-20080930.tar.gz
md5: d54a83fc22f1ec052db6ebe3c258ee45

u/l :root/password

Check it out at wiki.layerboom.com

From the article

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

Lets developers and businesses get started easily - there are no minimum fees and no commitments. You will only pay for what you actually use.
Is simple and easy to use - a single, simple API call is all that is needed to get started delivering your content.
Works seamlessly with Amazon S3 - this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
Has a global presence - we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

As Amazon continues to provide easy interfaces to rather abstract problems, it will be interesting to see how the other major players will respond. Companies like VMware and Citrix will no doubt have their own EC2′ish API, and other CDN providers will publish their own API’s, but the real question is how all of these things will integrate together.

C3 appears to be made up of different products that Citrix owns such as WANScaler, NetScaler, Xen Server, and Workflow Studio. Of course, this means you have to use each of these components to build your own cloud, and they all cost money. Not to mention the fact that there’s no way to swap in and out the different components, that I can find at least.

I’ll update more on costs and other information as Citrix gets back to me.

The vCloud will be one of the many enablers of the Federated Computing environment that people like Tim O’reilly have written about in the past. It’s really the only logical way forward for the general computing environment. As we enable people to run, manage and price these computing environments, we get closer and closer to a utility market that mirrors something like the electricity market. Different companies will be able to set prices based on how expensive or highly availble their operation / computing environment is to run, and it also gives vertical hosting companies the ability to align themselves with different tiers of hosting providers.

When this model becomes widely adopted we’ll see a parallel widespread adoption of technologies that make deploying and managing dense computing more cost effective. This means a lot more containers, and efficient data centers.

VMware is definitely on the right track, but there still needs to be an open source provider for these types of services and directories. At the end of the day the tradtional hosting providers, and even companies that claim to be “cloudy” will have to find a way to integrate into the new utility market.

Take Amazon as an example. Amazon operates their environment as a bunch of different groups, each running different services within the same company. S3, EC2, Payment Services. They’re all independent, highly scalable functions, tied together in the application we call Amazon.com.

Companies and startups are starting to break this operational model open, and putting those individual functions online for everyone. They’re building services that do something really well - or rather that do one thing really really well. They’re companies that focus on a specific function or feature and are open enough so creative people can say “I’m going to take this, this, and this - mix it in a pot and voila!”.

Do you want to build your own Twitter? Find an SMS gateway, Cloud Computing Host and XMPP service provider.

Do you want to build an interesting RSS/ATOM service? Find an RSS aggregator service and pour on some glue - see what sticks.

It’s Function as a Service world (to use an already overused description). Eventually cloud companies will realize that doing one thing really *really* well is tremendously valuable. Why does everyone have to build their own DNS service? Why does everyone have to build their own hosting system? What about Enterprise Storage, Authentication, SMS Gateways, Massively scalable XMPP services? How come I have to do that myself? Can 10,000 messages sent through a jabber server be worth a dollar? I think it can (maybe the math needs adjusting but you get my point). We’re all really just building a massive computer called the internet, only with each big trend we replace ‘The Internet’ with something else. First it was ‘The Web’, then it was ‘Web 2.0′, and now its ‘The Cloud’. The fact of the matter remains - the further along we go the more tightly knit the internet becomes, and that means that theres opportunity for programmable white label services to propel us further and faster.

Application Integration

Just like everyone thought WSDL would automagically make applications work together, people think the new ubiquitous ‘Cloud’ will make applications work together. Unless your application was designed from the beginning to talk to and operate with other cloud stacks, then you’re going to have an application integration issue on your hands. This is why behemoths like SAP exist. Everything is one universal and translatable data format, all controlled by one vendor. If you want to be able to move data, or integrate applications that run in different clouds, then you’re going to have to do it the old way. Sit down, translate the data yourself, and emulate it where you can’t. Cloud != Application / Data Babelfish

Eliminate Monitoring

Monitoring is either a dirty word or a multi-million dollar business depending on who you talk to. As a sysadmin for several high-availability environments, I can’t stress it enough. Building your application and environment to be redundant is always step number one, but it’s useless if you don’t know the state of your environment at all times - especially before and after changes. Environment monitoring is like Behaviour Driven Development (see rspec) for your application. You know what’s supposed to happen, and how things are supposed to look - and if something changes you find out about it. You still need this kind of functionality, and just putting your stuff in the cloud doesn’t make issues go away. Cloud != Perfectly Reliable Environment

Eliminate Lock-in

Oh boy I said a dirty word. If you write your application for AppEngine, then you’re using AppEngine until you refactor. Period. The AppDrop application lets you “run AppEngine on EC2″ but thats kind of masking the issue. AppDrop is just the AppEngine developer tools running in an AMI - it doesn’t do anything like emulate all the necessary infrastructure that makes AppEngine appealing in the first place and the authors acknowledge this. Don’t get me wrong - AppEngine is great. Its a highly abstracted environment for writing web applications, but it’s not appropriate for a lot of different services. If you want to be able to treat the cloud like you treat a “standard” machine -> LAMP stack then you should be aware of all these factors. Cloud != Universal Magical Computer

Eliminate Jobs

Sysadmins (and I can say this) are an odd bunch. Most of them treat their environments like children, rearing them to a specific age, but always keeping a watchful eye. Others tend to automate everything necessary in order to move on to the next best thing. The people that horde data, knowledge, useful tools, and expertise will fit right in to closed cultures, but if you want to make it in the new reality, you’re going to have to open your mind. The cloud isn’t going to make you obsolete unless you try stopping progress. If you didn’t have to worry about making system images and building back-ends all day think about all the other amazing things you could do! There are a slew of bad analogies I can use to drive this point home, but I’ll use the most cliche: Be a reed in the wind. If you’re clinging onto a job because you’re easily replaceable, then its just a matter of time. Learn a new skill. Cloud != Massive Layoffs