BoingBoing was recently issued a DMCA takedown notice over hosting a copy of a widely criticized Ralph Lauren ad. They didn’t have to comply. Why? Because their servers are hosted in Canada.
Jump to 3:30
Visit msnbc.com for Breaking News, World News, and News about the Economy
While this is a pretty small example, it highlights the importance of considering the geopolitics of where you host your website, which is a component rarely considered in the hosting world.
Today the Canadian Government released a brief on the opportunities for Canada in Cloud Computing. It’s a great paper that highlights some of the benefits and strategic advantages of building large cloud computing centers in Canada. I’ll jump straight to the conclusion in the article: Canada is one of the BEST places to build out data centers and cloud computing infrastructure. The article mentions a bunch of reasons – I’ll expand on a few.
Most of the costs associated with running the 1,000’s of servers is directly associated with the price of electricity and the cost of cooling. Canada has cheap, renewable electricity & it’s colder. That means you can offer competitive services at better margins than someone running a cloud in the hot Nevada Desert. Michael Geist wrote more about it at Clean Cloud Computing.
Not only are many Canadian companies required to keep their data on native soil, the privacy and electronic documentations act means keeping information here is a really good idea.
The BC, PEI, and Quebec governments actually have the cleanest and lowest cost per KWh electricity prices in all of North America. That’s possible through the use of hydro-electric dams, which also have an extremely low carbon footprint. As stated previously, the cost of running your servers is mostly the cost of electricity.
Cheaper electricty = Competitive Cloud
One of the fastest computer networks in the world, funded in part by the Canadian government, already runs through most of Canada. We’re also right next to the American market. That means North Americans can’t really tell if their servers are in Nevada or Nunavut. From a consumers perspective, there would be no reason not to use a Canadian Cloud that’s cheap, secure, and efficient, and we would be able to export a utility that is higher margin than say, electricity.
All in all I’m really excited by this report, and I’m sure that more people will be thinking about the potential Canada has to become the world leader in cloud computing services. You can get a little more background information, and learn more about the suggested ways forward by reading the brief here “Cloud Computing and the Canadian Government“
Last week TrevorO was away on a tropical island in the Pacific collecting data for one of our ongoing (but classified) projects. We didn’t hear from him all week which we assumed meant either:
Anyway, not knowing whether he was coming back or not, Kevin (@yinkei) and I (@joshwilsdon) decided to slack off for a couple hours and cure Cancer, Parkinsons and several other annoying maladies. Hopefully he’s not too mad.
Well, ok… These diseases haven’t *quite* been cured yet, but we’re trying to help! We created a Folding @ Home team for Layerboom and we’ve been using some machines which we need to do some load testing on anyway to fold some proteins in the name of science.
A 3D model (of a heteropolymer?)If you’ve not heard about Folding @ Home before, check out their website. It’s got all kinds of interesting info. If you just want to join team Layerboom so that you can ensure the “Layerboom cures Cancer!” headline can be reprinted later in serious scientific publications such as The Onion Sci/Tech Section, you should start a client and use team 171568 when you configure it. See if you can score more points than Kevin (kevinykchan) or I (JoshW) have contributed on our own home machines.
Team Layerboom is rocketing up the charts and in under 2 weeks we’ve moved into the top 4000 for total contributions. Pretty good considering there are more than 165,000 teams and the project has been running since October 1, 2000! If you’re just curious about our team progress, check out our team page or check out these more detailed stats from free-dc.org.
P.S. Most of the work contributed by the Layerboom user so far has been done on the new ultra-powerful 8 core monster servers that will soon be powering some of our public facing stuff. All this power will be available to you soon!
One of the keys to improving anything is having enough information. This has been widely discussed in environmental circles, and recent innovations such as the Kill-A-Watt and the awesome hack the Tweet-A-Watt have lead to a more widespread appreciation for just *knowing* the amount of energy your appliances, computers, and home entertainment systems are consuming. In addition to being surprising, the reality is that all too often assumptions are made about where to focus effort to fix a particular problem – or worse, you don’t even know a problem exists. But what to do with this information? At home it’s as easy as putting your devices on a power bar – such as your home theater – and turning it off when you’re not using it. Having the data enables you to make a decision – the decision to save money because all of the sudden it’s tangible.
These kinds of details can be applied at a really big and small scales too. What if you could measure the amount of power went into making your car? The amount of energy each Google search takes? The amount of energy for every tweet? Would knowing a tweet kills a tuna make you think twice? Would it enable you to make better decisions about the products you consume? Would it allow your customers to make better decisions about their energy efficiency?
This can apply to the hosting world too. Computers currently use more energy than the entire airline industry, and that’s expected to double within the next 5 years. Data centers consume a whopping 2-3% of the power in the United States alone. Hosting companies charge flat rates for collocation, virtual servers, shared hosting, etc. Bundled into that are the charges for electricity, and the electricity required to power the cooling. Unless you’re really close to the physical infrastructure, there’s no way to measure how efficient the servers are, or how much power your server is consuming. If we could measure the amount of power a server uses then you could incorporate that into the pricing of the server, and display the information separately. As a hosting company you would be able to make better decisions about which hardware, software, etc to use. As a hosting customer, you would be able to choose locations that are more power efficient. A slew of other possibilities exist. Due to power deregulation and trading markets in many locations, what costs a dollar during the day might cost 10 cents in the middle of the night.
Data centers are built for peak capacity, but there should be an incentive for customers to adopt more energy efficient solutions. Being able to measure (in)efficiencies also means that making decisions about moving to a container might be easier to justify.
Behold Sinners! The Apocalypse Aproacheth. No in all seriousness if you run a managed hosting company then your time is officially ‘up’. You won’t survive the coming hosting Apocalypse. Here’s why.
There are a few companies you may have heard of building large compute grids for consumption by the general public. They’re calling them their Cloud Computing products. IBM is building BlueCloud, Microsoft is building the Mesh, Amazon already has EC2, and Google has AppEngine. AppEngine is in a slightly different category than the others and the BlueCloud details are sparse, but they’re still worth mentioning. Of more immediate interest are Amazon and Microsoft’s solutions.
Microsoft is currently building their famous 300,000 server Data Center in Chicago. That’s roughly 3 times the number of servers that Google has. Microsoft has also announced several other Data center projects – each worth about $500 Million. It’s fair to say that that’s a lot of computing power, and it’s not all for MSN – Microsoft is planning on providing their platform in the cloud.
The real question is what Amazon will do when the Windows Cloud comes online. Microsoft has enough money in the bank to provide their 300,000 servers to customers for *years* without earning a single cent. That implies they can offer services at super low rates; Low enough to at least compete with Amazon’s EC2, which will support the Windows Server OS in fall 2008.
What happens with two huge cloud hosting companies get into a price war?
In the interest of self preservation they won’t make their services commodities – at least right away. But it won’t even matter. When you’re as big as Amazon, Microsoft, Google or IBM, you can afford to buy servers in such massive quantities that you could make money selling compute time for 10$ a month. The hosting space will change forever, because Amazon will eventually drop their prices by an *order of magnitude* and that has dire implications for the rest of the Mom’n'Pop hosting companies.
If thousands of companies can’t compete with Microsoft or Amazon on price, and they can’t compete in terms of convenience, then why would anyone use them? If you have to buy individual servers, or even servers by the rack, then you’re not going to get the price you need to be able to compete. You also don’t have access to the handful of specialized individuals and hardware required to make things work on such a grand scale.
The only answer is for all the smaller players to band together – to create a Federated Hosting environment, where together they can provide services that begin approaching levels of service and power that the Big 4 will offer.
Either way, we’re in an interesting period in the industry. Computing and the infrastructure of technology has become such a requirement for the economy that it will eventually become a general utility. The real question is who will be around.
Do you think it’s the end? We’re working on the answer, and your opinion is important.
Hi everyone!
I’m looking for some help answering some questions to help LayerBoom and some associated projects determine what companies and individuals really want in their own “Cloud Computing” environment.
I know the Term has been getting worn out, so to summarize, LayerBoom is providing software that helps companies build utility computing environments using existing hardware and network infrastructure. The idea is to provide software as a common platform, then provision resources across many distributed environments, and enable companies and academic environments to rent/share unused resources for extra revenue.
There’s a survey posted up at http://survey.layerboom.com
I’d appreciate your help and time!
-Trevor
(Also posted on http://trevoro.ca)
There are a lot of ways for companies to bootstrap, and even more companies and partners willing to lend a helping hand if you know who to ask, and where to look. One of the companies that’s willing to help is Sun Microsystems.
Through their Startup Essentials program, companies can get access to Sun gear and resources at heavily discounted rates. That’s great, because Sun gear is pretty much the best out there, and being competitive with other companies like Dell and HP on price will help get their technology into more Data Centers and up and coming businesses. But like all shiny objects, servers lose their luster. Even if they’re free.
In order to run a server you have to secure colocation space, pay for bandwidth, buy some switching gear and a router, and depending on your setup get UPS power. Sure you can get cheap switches to connect your high-end servers, but depending on your work load it would be like running a jet engine on bacon fat.
If you’re running more than a few servers you’ll probably need to get more than the standard ~3000W/Rack – especially if you have a SAN device. That means one full rack with extra power. Most older data centers don’t have the cooling capacity to handle todays dense and powerful systems, so your full rack will probably be half full (if you’re lucky), then you’ll have to get another rack if you want to expand. That involves more waiting. Average amount of time for a colo to provision a new rack? 4-6 weeks. You’ll also need to pay for installation, and every colo provides space on contract so you’re committed for a year. Time is money, and waiting 4-6 weeks to be able to expand your environment means you have less flexibility.
There are certain scenarios where getting colo still makes sense. If you’re running your own hosting environment *as a business* then obviously having the control over your environment is necessary. Companies with certain regulatory or security requirements will also need to stick with colo, but otherwise, why lock yourself into contracts and inflexible environments and hire extra staff to manage that operation, when you can just rent some servers en-masse and get super cheap bandwidth?
If you run the numbers you can get just as much if not more *power* for the same amount of money from companies like Joyent, Amazon, ServerBeach or Rackspace, and not have to worry about contracts. While it’s nice to have an awesome piece of kit, sometimes it isn’t worth the time or money.
Comments Off
I’ve been working on a project that lets you quickly move systems between your private Xen implementation and Amazon’s EC2 service. There are a lot of hurdles to get this to work, and most of them are surrounding how Amazon doesn’t let you download a Kernel or Ramdisk image out of S3 unless you’re the owner. You can download someone elses image if you’ve saved it as your own but you still can’t download the kernel and ramdisk. Also, EC2 has specific requirements for how the image is built. Here’s how you can get your image out of Amazon and run it locally on your own Xen hypervisor. I will assume you are already using Amazon Web Services and have created an account. If you haven’t then sign up.
Amazon calls their instance images Amazon Machine Images or AMI’s. If you want to be able to grab one of the many images from Amazon you can use download the Amazon AMI tools and AWS tools then do the following. You can download the tools here
Find and Download the AMI
$: ec2-describe-images IMAGE ami-cc6386a5 ubuntu-hardy-ruby/image.manifest.xml 848278689040 available private i386 machine IMAGE ami-386c8951 ubuntu-ruby-lapack/image.manifest.xml 848278689040 available private i386 machine $:
Fields 3 and 4 contain important information. For this example I’m listing the images that I own. Optionally you can provide a switch that will list all Amazon images by including ‘-a’ to the end of the ec2-describe-images command.
Field 3 is the unique identifier for the AMI, and field 4 is the bucket and AMI “manifest” – or a file that describes the AMI. Because users can specify the name of the manifest, you should pay attention to this value when trying to run the next set of commands.
The AMI tools from Amazon include a utility called ‘ec2-download-bundle’. This will download the manifest file from the bucket, parse through to see what other files it needs to download, then it will reassemble the AMI image locally, and check its signature. The AMI’s are encrypted in small (usually 10 meg) chunks. The signatures for those chunks are also included in the manifest.
To download the first AMI listed above, run the following commands
mkdir 'image-to-download' cd 'image-to-download' ec2-download-bundle --bucket ubuntu-ruby-lapack -m image.manifest.xml --access-key $AWS_ACCESS_KEY --secret-key $AWS_SECRET --privatekey $EC2_PRIVATE_KEY
That will start downloading the bundle to your local system.
Rebuild the AMI
Now we have to unbundle the files
ec2-unbundle -m image.manifest.xml -k $EC2_PRIVATE_KEY
This will decrypt and reassemble the image from all the individual components in the list
Now you have an image named ‘image’ in your directory. You can take a look at this file by mounting it
mkdir /mnt/image mount -t ext3 -o loop image /mnt/image cd /mnt/image
If you’re lucky there will be copies of the kernel and perhaps the ramdisk in the /boot partition. Otherwise you’ve got to do something really tricky : You have to guess as to what kernel will work the best. Thankfully we have a good understanding of what’s required to boot one of these images.
If you’ve created an image for Xen already then chances are your kernel will work just fine, but your ramdisk might need some adjusting. A trick you can use is to chroot to the /mnt/image folder, specify which modules you want loaded and rebuild the ramdisk – then exit the chroot, copy the kernel and ramdisk out of /mnt/image and you’ll have all the components you’ll need.
I know what you’re thinking: That’s a lot of work / guessing
You’re in luck. While there are a couple sites for sharing pre-built Xen images, the community is nowhere near as large as the Parallels or VMware ‘appliance’ sites. Jailtime.org has a hanful of images but they don’t follow any sort of standard, and the disk layouts / configurations aren’t compatible with Amazon’s EC2.
LayerBoom has a Xen image that is completely compatible with Amazons AMI format, and it can run in your own environment. This means you can copy a system into Amazon from your test environment without any hassle. It also works with the Eucalyptus project, and can be booted in xVM server as well (Instructions are coming)
Download the Xen package
url: http://layerboom.com/files/xen/images/centos52-20080930.tar.gz
md5: d54a83fc22f1ec052db6ebe3c258ee45
u/l :root/password
Wiki.layerboom.com is up for anyone interested in reading or contributing to documenting the installation and administration of the Eucalyptus Cloud Computing platform. Rather than replace the UCSB documentation, the Wiki is an attempt to supplement that resource, as well as collect common issues in one place.
Check it out at wiki.layerboom.com
Everyone knew it was coming, but this morning Amazon announced it will be providing a CDN service.
The process is the same as uploading to S3, then you simply make a call that says “Put this in the CDN”. Customers will be automatically sent to the closest For a lot of companies that are using S3 as a poor man’s CDN already this should provide some extra performance, although there aren’t any details regarding how many edge locations Amazon will be building. That will be a major comparison point between an Amazon CDN and other major delivery networks that have edges in most major POP’s around the globe.
From the article
This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:
Lets developers and businesses get started easily – there are no minimum fees and no commitments. You will only pay for what you actually use.
Is simple and easy to use – a single, simple API call is all that is needed to get started delivering your content.
Works seamlessly with Amazon S3 – this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
Has a global presence – we use a global network of edge locations on three continents to deliver your content from the most appropriate location.
As Amazon continues to provide easy interfaces to rather abstract problems, it will be interesting to see how the other major players will respond. Companies like VMware and Citrix will no doubt have their own EC2′ish API, and other CDN providers will publish their own API’s, but the real question is how all of these things will integrate together.
