One of the keys to improving anything is having enough information. This has been widely discussed in environmental circles, and recent innovations such as the Kill-A-Watt and the awesome hack the Tweet-A-Watt have lead to a more widespread appreciation for just *knowing* the amount of energy your appliances, computers, and home entertainment systems are consuming.  In addition to being surprising, the reality is that all too often assumptions are made about where to focus effort to fix a particular problem - or worse, you don’t even know a problem exists. But what to do with this information? At home it’s as easy as putting your devices on a power bar - such as your home theater - and turning it off when you’re not using it. Having the data enables you to make a decision - the decision to save money because all of the sudden it’s tangible.

These kinds of details can be applied at a really big and small scales too. What if you could measure the amount of power went into making your car? The amount of energy each Google search takes? The amount of energy for every tweet? Would knowing a tweet kills a tuna make you think twice? Would it enable you to make better decisions about the products you consume? Would it allow your customers to make better decisions about their energy efficiency?

This can apply to the hosting world too. Computers currently use more energy than the entire airline industry, and that’s expected to double within the next 5 years. Data centers consume a whopping 2-3% of the power in the United States alone. Hosting companies charge flat rates for collocation, virtual servers, shared hosting, etc. Bundled into that are the charges for electricity, and the electricity required to power the cooling. Unless you’re really close to the physical infrastructure, there’s no way to measure how efficient the servers are, or how much power your server is consuming. If we could measure the amount of power a server uses then you could incorporate that into the pricing of the server, and display the information separately. As a hosting company you would be able to make better decisions about which hardware, software, etc to use. As a hosting customer, you would be able to choose locations that are more power efficient. A slew of other possibilities exist. Due to power deregulation and trading markets in many locations, what costs a dollar during the day might cost 10 cents in the middle of the night.

Data centers are built for peak capacity, but there should be an incentive for customers to adopt more energy efficient solutions. Being able to measure (in)efficiencies also means that making decisions about moving to a container might be easier to justify.

Behold Sinners! The Apocalypse Aproacheth. No in all seriousness if you run a managed hosting company then your time is officially ‘up’. You won’t survive the coming hosting Apocalypse. Here’s why.

There are a few companies you may have heard of building large compute grids for consumption by the general public. They’re calling them their Cloud Computing products. IBM is building BlueCloud, Microsoft is building the Mesh, Amazon already has EC2, and Google has AppEngine. AppEngine is in a slightly different category than the others and the BlueCloud details are sparse, but they’re still worth mentioning. Of more immediate interest are Amazon and Microsoft’s solutions.

Microsoft is currently building their famous 300,000 server Data Center in Chicago. That’s roughly 3 times the number of servers that Google has. Microsoft has also announced several other Data center projects - each worth about $500 Million. It’s fair to say that that’s a lot of computing power, and it’s not all for MSN - Microsoft is planning on providing their platform in the cloud.

The real question is what Amazon will do when the Windows Cloud comes online. Microsoft has enough money in the bank to provide their 300,000 servers to customers for *years* without earning a single cent. That implies they can offer services at super low rates; Low enough to at least compete with Amazon’s EC2, which will support the Windows Server OS in fall 2008.

What happens with two huge cloud hosting companies get into a price war?

In the interest of self preservation they won’t make their services commodities - at least right away. But it won’t even matter. When you’re as big as Amazon, Microsoft, Google or IBM, you can afford to buy servers in such massive quantities that you could make money selling compute time for 10$ a month. The hosting space will change forever, because Amazon will eventually drop their prices by an *order of magnitude* and that has dire implications for the rest of the Mom’n'Pop hosting companies.

If thousands of companies can’t compete with Microsoft or Amazon on price, and they can’t compete in terms of convenience, then why would anyone use them? If you have to buy individual servers, or even servers by the rack, then you’re not going to get the price you need to be able to compete. You also don’t have access to the handful of specialized individuals and hardware required to make things work on such a grand scale.

The only answer is for all the smaller players to band together - to create a Federated Hosting environment, where together they can provide services that begin approaching levels of service and power that the Big 4 will offer.

Either way, we’re in an interesting period in the industry. Computing and the infrastructure of technology has become such a requirement for the economy that it will eventually become a general utility. The real question is who will be around.

Do you think it’s the end? We’re working on the answer, and your opinion is important.

Hi everyone!

I’m looking for some help answering some questions to help LayerBoom and some associated projects determine what companies and individuals really want in their own “Cloud Computing” environment.

I know the Term has been getting worn out, so to summarize, LayerBoom is providing software that helps companies build utility computing environments using existing hardware and network infrastructure. The idea is to provide software as a common platform, then provision resources across many distributed environments, and enable companies and academic environments to rent/share unused resources for extra revenue.

There’s a survey posted up at http://survey.layerboom.com

I’d appreciate your help and time!

-Trevor

(Also posted on http://trevoro.ca)

There are a lot of ways for companies to bootstrap, and even more companies and partners willing to lend a helping hand if you know who to ask, and where to look. One of the companies that’s willing to help is Sun Microsystems.

Through their Startup Essentials program, companies can get access to Sun gear and resources at heavily discounted rates. That’s great, because Sun gear is pretty much the best out there, and being competitive with other companies like Dell and HP on price will help get their technology into more Data Centers and up and coming businesses. But like all shiny objects, servers lose their luster. Even if they’re free.

In order to run a server you have to secure colocation space, pay for bandwidth, buy some switching gear and a router, and depending on your setup get UPS power. Sure you can get cheap switches to connect your high-end servers, but depending on your work load it would be like running a jet engine on bacon fat.

If you’re running more than a few servers you’ll probably need to get more than the standard ~3000W/Rack - especially if you have a SAN device. That means one full rack with extra power. Most older data centers don’t have the cooling capacity to handle todays dense and powerful systems, so your full rack will probably be half full (if you’re lucky), then you’ll have to get another rack if you want to expand. That involves more waiting. Average amount of time for a colo to provision a new rack? 4-6 weeks. You’ll also need to pay for installation, and every colo provides space on contract so you’re committed for a year. Time is money, and waiting 4-6 weeks to be able to expand your environment means you have less flexibility.

There are certain scenarios where getting colo still makes sense. If you’re running your own hosting environment *as a business* then obviously having the control over your environment is necessary. Companies with certain regulatory or security requirements will also need to stick with colo, but otherwise, why lock yourself into contracts and inflexible environments and hire extra staff to manage that operation, when you can just rent some servers en-masse and get super cheap bandwidth?

If you run the numbers you can get just as much if not more *power* for the same amount of money from companies like Joyent, Amazon, ServerBeach or Rackspace, and not have to worry about contracts. While it’s nice to have an awesome piece of kit, sometimes it isn’t worth the time or money.

I’ve been working on a project that lets you quickly move systems between your private Xen implementation and Amazon’s EC2 service. There are a lot of hurdles to get this to work, and most of them are surrounding how Amazon doesn’t let you download a Kernel or Ramdisk image out of S3 unless you’re the owner. You can download someone elses image if you’ve saved it as your own but you still can’t download the kernel and ramdisk. Also, EC2 has specific requirements for how the image is built. Here’s how you can get your image out of Amazon and run it locally on your own Xen hypervisor. I will assume you are already using Amazon Web Services and have created an account. If you haven’t then sign up.

Amazon calls their instance images Amazon Machine Images or AMI’s. If you want to be able to grab one of the many images from Amazon you can use download the Amazon AMI tools and AWS tools then do the following. You can download the tools here

Find and Download the AMI


$: ec2-describe-images
IMAGE   ami-cc6386a5    ubuntu-hardy-ruby/image.manifest.xml    848278689040    available       private         i386    machine
IMAGE   ami-386c8951    ubuntu-ruby-lapack/image.manifest.xml   848278689040    available       private         i386    machine
$:


Fields 3 and 4 contain important information. For this example I’m listing the images that I own. Optionally you can provide a switch that will list all Amazon images by including ‘-a’ to the end of the ec2-describe-images command.

Field 3 is the unique identifier for the AMI, and field 4 is the bucket and AMI “manifest” - or a file that describes the AMI. Because users can specify the name of the manifest, you should pay attention to this value when trying to run the next set of commands.

The AMI tools from Amazon include a utility called ‘ec2-download-bundle’. This will download the manifest file from the bucket, parse through to see what other files it needs to download, then it will reassemble the AMI image locally, and check its signature. The AMI’s are encrypted in small (usually 10 meg) chunks. The signatures for those chunks are also included in the manifest.

To download the first AMI listed above, run the following commands


mkdir 'image-to-download'
cd 'image-to-download'
ec2-download-bundle --bucket ubuntu-ruby-lapack -m image.manifest.xml --access-key $AWS_ACCESS_KEY --secret-key $AWS_SECRET --privatekey $EC2_PRIVATE_KEY


That will start downloading the bundle to your local system.

Rebuild the AMI

Now we have to unbundle the files

ec2-unbundle -m image.manifest.xml -k $EC2_PRIVATE_KEY


This will decrypt and reassemble the image from all the individual components in the list

Now you have an image named ‘image’ in your directory. You can take a look at this file by mounting it

mkdir /mnt/image
mount -t ext3 -o loop image /mnt/image
cd /mnt/image


If you’re lucky there will be copies of the kernel and perhaps the ramdisk in the /boot partition. Otherwise you’ve got to do something really tricky : You have to guess as to what kernel will work the best. Thankfully we have a good understanding of what’s required to boot one of these images.

If you’ve created an image for Xen already then chances are your kernel will work just fine, but your ramdisk might need some adjusting. A trick you can use is to chroot to the /mnt/image folder, specify which modules you want loaded and rebuild the ramdisk - then exit the chroot, copy the kernel and ramdisk out of /mnt/image and you’ll have all the components you’ll need.

I know what you’re thinking: That’s a lot of work / guessing

You’re in luck. While there are a couple sites for sharing pre-built Xen images, the community is nowhere near as large as the Parallels or VMware ‘appliance’ sites. Jailtime.org has a hanful of images but they don’t follow any sort of standard, and the disk layouts / configurations aren’t compatible with Amazon’s EC2.

LayerBoom has a Xen image that is completely compatible with Amazons AMI format, and it can run in your own environment. This means you can copy a system into Amazon from your test environment without any hassle. It also works with the Eucalyptus project, and can be booted in xVM server as well (Instructions are coming)

Download the Xen package

url: http://layerboom.com/files/xen/images/centos52-20080930.tar.gz
md5: d54a83fc22f1ec052db6ebe3c258ee45

u/l :root/password

Wiki.layerboom.com is up for anyone interested in reading or contributing to documenting the installation and administration of the Eucalyptus Cloud Computing platform. Rather than replace the UCSB documentation, the Wiki is an attempt to supplement that resource, as well as collect common issues in one place.

Check it out at wiki.layerboom.com

Everyone knew it was coming, but this morning Amazon announced it will be providing a CDN service.
The process is the same as uploading to S3, then you simply make a call that says “Put this in the CDN”. Customers will be automatically sent to the closest For a lot of companies that are using S3 as a poor man’s CDN already this should provide some extra performance, although there aren’t any details regarding how many edge locations Amazon will be building. That will be a major comparison point between an Amazon CDN and other major delivery networks that have edges in most major POP’s around the globe.

From the article

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

Lets developers and businesses get started easily - there are no minimum fees and no commitments. You will only pay for what you actually use.
Is simple and easy to use - a single, simple API call is all that is needed to get started delivering your content.
Works seamlessly with Amazon S3 - this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
Has a global presence - we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

As Amazon continues to provide easy interfaces to rather abstract problems, it will be interesting to see how the other major players will respond. Companies like VMware and Citrix will no doubt have their own EC2′ish API, and other CDN providers will publish their own API’s, but the real question is how all of these things will integrate together.

Fresh off the heels of the VMware vCloud annoucement, Citrix has announced their own “Cloud Enabling” product titled Citrix Cloud Cente (C3). Similar to the vCloud offering, C3 will enable data centers to build their own cloud platform.You can get a copy of XenServer 5 today, but it’s unclear how the suite of tools that make up C3 will become available to the public.

C3 appears to be made up of different products that Citrix owns such as WANScaler, NetScaler, Xen Server, and Workflow Studio. Of course, this means you have to use each of these components to build your own cloud, and they all cost money. Not to mention the fact that there’s no way to swap in and out the different components, that I can find at least.

I’ll update more on costs and other information as Citrix gets back to me.

At VMWorld today in Las Vegas, VMware announed their newest initiative, vCloud “to Federate Resources Between Internal IT and External Clouds and Enable Broad Application Compatibility for Cloud Computing.”

The vCloud will be one of the many enablers of the Federated Computing environment that people like Tim O’reilly have written about in the past. It’s really the only logical way forward for the general computing environment. As we enable people to run, manage and price these computing environments, we get closer and closer to a utility market that mirrors something like the electricity market. Different companies will be able to set prices based on how expensive or highly availble their operation / computing environment is to run, and it also gives vertical hosting companies the ability to align themselves with different tiers of hosting providers.

When this model becomes widely adopted we’ll see a parallel widespread adoption of technologies that make deploying and managing dense computing more cost effective. This means a lot more containers, and efficient data centers.

VMware is definitely on the right track, but there still needs to be an open source provider for these types of services and directories. At the end of the day the tradtional hosting providers, and even companies that claim to be “cloudy” will have to find a way to integrate into the new utility market.

Websites have gone from hand-typed static pages, to massive applications with every feature under the moon. Most applications have some secret sauce that does magical things in the background - whether that be the ability to handle massive amounts of volume, reduce the barrier to entry into a market, or just keep users engaged by providing endless amounts of quick short updates.

Take Amazon as an example. Amazon operates their environment as a bunch of different groups, each running different services within the same company. S3, EC2, Payment Services. They’re all independent, highly scalable functions, tied together in the application we call Amazon.com.

Companies and startups are starting to break this operational model open, and putting those individual functions online for everyone. They’re building services that do something really well - or rather that do one thing really really well. They’re companies that focus on a specific function or feature and are open enough so creative people can say “I’m going to take this, this, and this - mix it in a pot and voila!”.

Do you want to build your own Twitter? Find an SMS gateway, Cloud Computing Host and XMPP service provider.

Do you want to build an interesting RSS/ATOM service? Find an RSS aggregator service and pour on some glue - see what sticks.

It’s Function as a Service world (to use an already overused description). Eventually cloud companies will realize that doing one thing really *really* well is tremendously valuable. Why does everyone have to build their own DNS service? Why does everyone have to build their own hosting system? What about Enterprise Storage, Authentication, SMS Gateways, Massively scalable XMPP services? How come I have to do that myself? Can 10,000 messages sent through a jabber server be worth a dollar? I think it can (maybe the math needs adjusting but you get my point). We’re all really just building a massive computer called the internet, only with each big trend we replace ‘The Internet’ with something else. First it was ‘The Web’, then it was ‘Web 2.0′, and now its ‘The Cloud’. The fact of the matter remains - the further along we go the more tightly knit the internet becomes, and that means that theres opportunity for programmable white label services to propel us further and faster.