Monitoring your applications running on Amazon's EC2 platform just got easier. Today, Amazon announced the public availability of CloudWatch for EC2 with Alarms. Instead of having to poll and ask CloudWatch for the current status, you can actually have that information pushed to you instead. This is a much more efficient way to monitor you applications running in the Amazon cloud. Right now the only new alarm that is supported is for Amazon SNS, but this is a huge step forward. Before today, CloudWatch could still trigger an Auto Scaling policy, but with the addition of SNS the doors are really opening up. Using Amazon SNS, you can do almost any type of notification that you would like posting to HTTP/HTTPS, Email, JSON, or directly into SQS. This makes the service very flexible. You are able to monitor CPU Utilization, Average Disk Reads, Average Disk Writes and Network traffic in and out.
Amazon has also enabled a new monitoring tab in the EC2 management console that will allow you to look at the graphs for these elements on each of your EC2 instances. The data only goes back to the 2nd of December since it was just released today, but this will be extremely useful for a quick read on how your instances are performing.
Amazon is certainly hitting things on my wishlist this year. Now if they would only enable CloudWatch and Alarms for metrics on SQS! Hopefully this is just around the corner. You can read more about the new features and the pricing of CloudWatch on EC2 here.
Information about various cloud technologies and announcements as well as code snippets.
Showing posts with label amazon. Show all posts
Showing posts with label amazon. Show all posts
Friday, December 3, 2010
Wednesday, November 24, 2010
Downloading EXE files from S3
It is more than a little frustrating that Internet Explorer and Windows cannot do anything to match what the other major browsers and operating systems are doing, or so it seems. Internet Explorer has long been the browser that most developers hate to support. Today, I ran across something that wasn't HTML or CSS related but still caused issues on Windows using Internet Explorer.
As many of you know, I use Amazon Web Services to do much of my work and S3, Simple Storage Service, is a large part of what I work with. Today, I was made aware that someone could not download an EXE file from S3 onto their desktop. I tested it out, and it was fine for me. Classic developer response right? Turns out, it worked for me because I was using Chrome on my Mac. It didn't work quite right in Internet Explorer on Windows. The file would download, but it would not bring down the extension to the file. What I mean, is if the file was stored in S3 as "filename.exe", when it was downloaded to the Windows machine it would only be named "filename". I was encouraged by some people that I know to simply chalk it up to a security concern, and while I do not encourage anyone downloading random .exe files and installing them, I knew that this just couldn't be the case. As it turns out, Internet Explorer must be told the Content-Disposition of the file when it is an EXE or DLL. This is done in the headers when the file is requested. Luckily, S3 allows you to put metadata on a file that is passed down when the file is requested. To allow Internet Explorer to recognize and save the file with the .exe extension you will just need to add the Content-Disposition:attachment; filename=<filename.exe> header.
You can do this as metadata in S3 through the AWS console. Many of the software utilities that work with S3 also allow you to set this metadata programmatically when storing the objects.
I am used to all sorts of JavaScript, HTML, CSS problems when working with IE, but this one really threw me for a loop at first. I hope that this helps you.
Monday, January 4, 2010
Creativity for Amazon's Cloud Computing Continues
Amazon Web Services continues to move forward in innovative ways to get people to take advantage of their services. Two announcements last month enhance the EC2 service pricing and the Cloud Front delivery model. The availability of auction price instances (spot instances) in EC2 and the ability to do true Flash streaming from Cloud Front really put the squeeze on traditional hosting and edge delivery companies.
With the new Spot Instances in EC2 you can take advantage of the excess capacity that Amazon is not using at that time. Because that capacity changes as more instances are started, the price for those instances fluctuate. With good planning, this can be used to really save on cash. Over the past 20 days, the price of a Spot Instance has been less than 50% of the price of a regular priced instance much of the time. If you are considering using EC2 for some bursting tasks and quick turn around, I would look at the Spot Instances first. Because you set the price you want to pay and the price changes over time, your instance may be terminated if your price is less than the Amazon rate. For this reason, you must plan ahead when using them in a production situation. One good strategy here to lower your costs would be to purchase some Reserved Instances that will stay running and then start Spot Instances in peak situations. You can use the AWS API to see the current prices of the Spot Instances to know how to set your price at runtime. If you want to see the prices online and over time, you can use the AWS console or some other sites like Cloud Exchange. Because you cannot reserve Windows instances right now on Amazon, it makes a lot of sense to take advantage of the Spot Instances. As I write this, a small Windows instance is $0.12 an hour and you can have the same thing running as a Spot Instance for $0.042 an hour.
With the new Spot Instances in EC2 you can take advantage of the excess capacity that Amazon is not using at that time. Because that capacity changes as more instances are started, the price for those instances fluctuate. With good planning, this can be used to really save on cash. Over the past 20 days, the price of a Spot Instance has been less than 50% of the price of a regular priced instance much of the time. If you are considering using EC2 for some bursting tasks and quick turn around, I would look at the Spot Instances first. Because you set the price you want to pay and the price changes over time, your instance may be terminated if your price is less than the Amazon rate. For this reason, you must plan ahead when using them in a production situation. One good strategy here to lower your costs would be to purchase some Reserved Instances that will stay running and then start Spot Instances in peak situations. You can use the AWS API to see the current prices of the Spot Instances to know how to set your price at runtime. If you want to see the prices online and over time, you can use the AWS console or some other sites like Cloud Exchange. Because you cannot reserve Windows instances right now on Amazon, it makes a lot of sense to take advantage of the Spot Instances. As I write this, a small Windows instance is $0.12 an hour and you can have the same thing running as a Spot Instance for $0.042 an hour.
At about 1/3 the price of a standard small instance, it makes it very attractive to use the Spot Instances right now.
The announcement of streaming support from Cloud Front is also a great new addition. Delivering content to the end user from the closest data center helps improve performance. Until now, all that could be delivered was static content and progressive downloads. Streaming video from Cloud Front helps reduce your costs because only the viewed parts need to be delivered. It also does not store the file on the end user's machine. You can use Cloud Front to deliver videos in several formats, even H.264 allowing playback on the iPod and iPhone. Thank you Amazon!
Wednesday, October 14, 2009
Cloud Confusion and the DoD
Last week the Department of Defense announced a new private cloud computing environment called RACE (Rapid Access Computing Environment). This was an announcement that received a ton of attention in blogs and the circles of cloud computing advocates and naysayers. The frustrating thing about announcements and the ensuing arguments that take place afterward is that there are very few people that really understand what cloud computing is and all of the facets that are involved. The problem really stems from the over-use of the term "cloud computing" to cover what seems to be every application that runs in your browser. Just because you don't know where the actual server is located, or because it runs in your browser does not make something cloud computing. As soon as the announcement was made about RACE, the comparisons to Google came flying in. First of all, this is not even the right comparison. I am behind using cloud computing for government applications, but if we are going to compare the systems, let's at least compare them fairly. RACE is more of an IaaS cloud service and Google is both SaaS (GMail, Google Docs, etc) and PaaS (AppEngine). It seems more fairly compared to the single service of EC2 provided by Amazon. I would love to take a deep dive into RACE, but unfortunately, and as expected for security, it is only available to someone with a government card or clearance. What they do say on the website though is that provisioning after a request only takes 24 hours. Is that truly on demand computing? Call me spoiled being able to provision dozens of servers with a single web service request in 5 to 10 minutes, but that is what I expect from a system now that claims to be on-demand. This is a great direction and good start for DISA. It is very encouraging to see the activity in the government space around cloud computing. We really need to stop lumping every cloud computing service or offering under the single banner of Cloud Computing. People will continue to make their offerings sound better than others with unfair comparisons and statistics. A common language and standard is needed for fair comparison. Until then, let the spin go on!
Wednesday, July 1, 2009
North Carolina and the Amazon Associates Program
The North Carolina General Assembly has a mandate to balance the budget every year, which I can appreciate because at least it keeps us somewhat focused as a state. I guess I have a very hard time understanding how the General Assembly can try to get that money out of companies that are not even located within our state borders.
I am a member of the Amazon Associates program which will pay a small commission on products sold from simply advertising on your site. I have not made any significant amount of money from this, but there are several people in our state that do very well supplementing their income and even forming a small business out of concepts such as the Amazon Associates Program. The North Carolina General Assembly has now added to the budget that Amazon must pay the taxes on products sold through the program to North Carolina. Amazon's answer.... pull the plug and don't offer it anymore. I don't blame them. What services is North Carolina providing for them? They do not have any buildings here that our Fire Departments, Police Departments or Emergency services help protect. They don't have any cars or trucks that use our streets and highways. They do not have a physical presence here that consume any of the state provided services. Why should they pay taxes? What the General Assembly is doing is stifling small business and is really doing absolutely no good for the state. It only serves to drive companies away from our state. After passing a budget with this nonsense in it, why would Amazon even consider placing a facility here or locating a division here within North Carolina. The passage of this item buried in the middle of the budget has produced no additional revenue for North Carolina and only serves to drive business away and make its own citizens consider moving to neighboring states. Here is an excerpt from the email that I received from Amazon explaining the shut down of the program within North Carolina:
I am a member of the Amazon Associates program which will pay a small commission on products sold from simply advertising on your site. I have not made any significant amount of money from this, but there are several people in our state that do very well supplementing their income and even forming a small business out of concepts such as the Amazon Associates Program. The North Carolina General Assembly has now added to the budget that Amazon must pay the taxes on products sold through the program to North Carolina. Amazon's answer.... pull the plug and don't offer it anymore. I don't blame them. What services is North Carolina providing for them? They do not have any buildings here that our Fire Departments, Police Departments or Emergency services help protect. They don't have any cars or trucks that use our streets and highways. They do not have a physical presence here that consume any of the state provided services. Why should they pay taxes? What the General Assembly is doing is stifling small business and is really doing absolutely no good for the state. It only serves to drive companies away from our state. After passing a budget with this nonsense in it, why would Amazon even consider placing a facility here or locating a division here within North Carolina. The passage of this item buried in the middle of the budget has produced no additional revenue for North Carolina and only serves to drive business away and make its own citizens consider moving to neighboring states. Here is an excerpt from the email that I received from Amazon explaining the shut down of the program within North Carolina:
We are writing from the Amazon Associates Program to notify you that your Associates account has been closed as of June 26, 2009. This is a direct result of the unconstitutional tax collection scheme expected to be passed any day now by the North Carolina state legislature (the General Assembly) and signed by the governor. As a result, we will no longer pay any referral fees for customers referred to Amazon.com or Endless.com after June 26. We were forced to take this unfortunate action in anticipation of actual enactment because of uncertainties surrounding the legislation’s effective date.If we are going to look for ways to balance the budget, let's make sure that we actually have the ability to collect money or cut the money from many of the wasteful programs already in place. Please don't drive business and people away.
Thursday, May 21, 2009
Amazon Import/Export
After a recent talk that I gave on cloud computing, one of the attendees contacted me with some questions about the "safety" of the data and also wanted to talk about vendor lock-in. It is no secret that I am a fan of the Amazon Web Services cloud platform and so it followed that these questions all had to do with the way Amazon stores the data. While these are typical questions, today the second question became much easier to answer. Amazon Web Services announced the availability of AWS Import/Export. Quite simply, Amazon is offering its customers a very easy way to ship a disk of data to them and they will push it into S3 to your specifications or grab your data out of S3 and put it on the disk for you. This is very attractive because it can take days to actually move hundreds of GB on your office network into or out of S3 because of the simple limitations of bandwidth at the average workplace. I am not a fan of vendor lock-in and have thought long and hard how to avoid it. Any code that we write storing to and from Amazon's infrastructure is isolated enough so that we can switch it to another provider by reimplementing that single area. But, the fact still remains that it would take weeks to move all of our data and it would have to be done over an extended period of time. A strategy for this is to move data between two cloud providers instead of bringing it down locally and putting it back up. While moving over a backbone is still going to be faster, it will not be optimal. With AWS Import/Export you are one step closer. For $80.00 and $2.49 per hour, you can currently have all of the data on a disk pushed into Amazon's cloud and the export facility is coming soon. I commend Amazon for providing so many tools and conveniences.
Monday, May 18, 2009
Amazon Releases New Cloud Computing Services
One of the big draws to the cloud is its ability to scale with your application. That has become much easier with Amazon Web Services today. Early this morning the Amazon Web Services team launched three new services: CloudWatch, Auto Scaling and Elastic Load Balancing. Combining the use of these three services allows a user to configure and scale their application based upon information gathered by CloudWatch. These are important additions to the Amazon Web Services offerings because it helps take more of the coding and configuration work away from the developers and system administrators. This is a key benefit of cloud computing that AppEngine from Google and Azure from Microsoft have built in and they have kept completely transparent to the developer. While Amazon has not made it completely transparent with the release of these services, it is a great step and may be exactly the middle ground that is needed.CloudWatch allows you to monitor CPU utilization, data transfer and disk usage, request rate and traffic to your EC2 instances. Based up on the information that CloudWatch gathers, you can set triggers that will look at that data over a time period and allow you to use the Auto Scaling to automatically add or remove EC2 instances to the specific group of machines working on a particular task. Finally the Elastic Load Balancing helps you distribute the traffic coming into your application to your EC2 instances. This is a welcome addition as it accomplishes fault tolerant load balancing for us without the cost of having to setup several HAProxy instances. So, even though we incur the costs of using the new Elastic Load Balancing Service, it quickly pays for itself because we are able to remove our own load balancing configuration on EC2. I am excited to see these new services finally go beta to the public and I am looking forward to more.
Sunday, April 19, 2009
Tips on Using Amazon CloudFront
If you are serving a lot of content with an audience that is spread out over the globe, then most likely you are in need of putting your content on a CDN (Content Delivery Network). There are a host of options for you such as LimeLight, BrightCove and Akamai to name a few, but I have found that Amazon offers much of the same functionality at a better price. Moving your static content to Amazon's CloudFront can really reduce the load on your servers and certainly enables you to serve your content from a place that is closer to the request. The best thing about it is that it is very very simple to do. There are some pieces of information that can really save you alot of time and energy if you are converting your site to serve content from CloudFront. Here are some things that were important to me and that I had to address when moving our site:
Amazon CloudFront allows you to manage the content in your S3 buckets and enable them for delivery from edge locations all over the world. To enable a bucket for CloudFront, it is a simple API call to create a Distribution. Once your Distribution is provisioned, which usually takes less than 15 seconds in my experience, you will be given a Distribution url which will look something like: http://abcd1234.cloudfront.net . This Distribution URL can be simply though of as another URL to that bucket in S3. For instance, if you have a bucket in S3 named "mys3bucket" and you create a Distribution for that bucket, which returned http://abcd1234.cloudfront.net, then you could reference that bucket as http://s3.amazonaws.com/mys3bucket/ or as http://abcd1234.cloudfront.net . The only difference between the two is that the CloudFront URL will be served from the location closest to the request and the content from that distribution can be served at 1,000 Megabits per second and even more if needed and requested. For the easy creation of your S3 Buckets and CloudFront distributions, I suggest using S3Fox for Firefox. With a simple right click on your bucket through S3Fox, you can create a new Distribuion in seconds.
Beyond just offering the serving of your content from an edge location at high throughput, CloudFront distributions have another advantage that can speedup the loading of your sites pages. Using a single Distribution, you can assign up to 10 CNAMEs to it. This can really make a large difference when serving content for a page. Most browsers block and only allow 2-4 parallel connections to a single host when loading content. Creating 10 CNAME entries all pointing to the same distribution allows the browser to make 10 times the parallel connections it could using a single entry. I would suggest Steve Souders book High Performance Web Sites: Essential Knowledge for Front-End Engineers
for more tips on this.
HTTP and HTTPS Delivery
One of the drawbacks to CloudFront is that it does not support HTTPS delivery of the content. However, there is some good news here because S3 does support HTTPS. Because the content that you need to deliver over CloudFront sits in an S3 bucket, then you can also deliver that same content over HTTPS but you cannot take advantage of the edge locations. This is especially useful when you need to serve a secure site that has a bunch of images that you are already serving from CloudFront. If you try to serve those images over HTTPS, your users will get a security warning or no image at all because the browser will block it. So, when you need to serve those secure pages, simply switch your delivery URL to the S3 location instead. A word of advice here is to create your S3 bucket without using any '.'s. I know that this really messes with you if you want your URL to look like it is coming from your servers, but it will allow you to serve secure content from S3. The S3 bucket is served over HTTPS using a wildcarded certificate. This allows Amazon to serve your content using any prefix as long as it does not have any extra dots ('.'). If your bucket name is 'mys3bucket', then you can create a CNAME record that points 'mys3bucket' to mys3bucket.s3.amazonaws.com. Because the wildcarded SSL certificate is to *.s3.amazonaws.com, then you can serve your content from https://mys3bucket.s3.amazonaws.com. While this isn't the perfect solution, it is really nice for serving the same bucket content over HTTP or HTTPS.
Simple, Straight-Forward Development
I am not sure that building scalable applications is ever simple, but I am always looking for ways to make it extremely repeatative and simple for myself and other developers. Because we are using Java for most of our applications, the easiest way for me to do that was through a tag library. I included all of the logic for switching between the different CNAMES for CloudFront delivery and logic for detecting a secure connection and need to switch to straight S3 delivery right in the tag. There were some interesting details that were necessary to take full advantage of CloudFront. One was coming up with a hash scheme for a URL so that it could be cached in the browser and distributing the calls across the CNAMEs evenly. The second was to make sure that the tag was written so that it was at the request level instead of the page level. This helped ensure that even an included JSP would not cause a narrowing of the number of CNAMEs that I could use. This is the method in the tag that does most of the work
CloudFront has proven to be very useful to me. I hope with a few of these tips that you can improve the performance of your site. Please share your experience with me or any other tips that you might have!
- Speed
- Faster delivery of all content from edge locations close to the request
- Faster load time of a page in the browser
- Ability to serve content over HTTP and HTTPS
- Simple, straight-forward development
Amazon CloudFront allows you to manage the content in your S3 buckets and enable them for delivery from edge locations all over the world. To enable a bucket for CloudFront, it is a simple API call to create a Distribution. Once your Distribution is provisioned, which usually takes less than 15 seconds in my experience, you will be given a Distribution url which will look something like: http://abcd1234.cloudfront.net . This Distribution URL can be simply though of as another URL to that bucket in S3. For instance, if you have a bucket in S3 named "mys3bucket" and you create a Distribution for that bucket, which returned http://abcd1234.cloudfront.net, then you could reference that bucket as http://s3.amazonaws.com/mys3bucket/ or as http://abcd1234.cloudfront.net . The only difference between the two is that the CloudFront URL will be served from the location closest to the request and the content from that distribution can be served at 1,000 Megabits per second and even more if needed and requested. For the easy creation of your S3 Buckets and CloudFront distributions, I suggest using S3Fox for Firefox. With a simple right click on your bucket through S3Fox, you can create a new Distribuion in seconds.
Beyond just offering the serving of your content from an edge location at high throughput, CloudFront distributions have another advantage that can speedup the loading of your sites pages. Using a single Distribution, you can assign up to 10 CNAMEs to it. This can really make a large difference when serving content for a page. Most browsers block and only allow 2-4 parallel connections to a single host when loading content. Creating 10 CNAME entries all pointing to the same distribution allows the browser to make 10 times the parallel connections it could using a single entry. I would suggest Steve Souders book High Performance Web Sites: Essential Knowledge for Front-End Engineers
HTTP and HTTPS Delivery
One of the drawbacks to CloudFront is that it does not support HTTPS delivery of the content. However, there is some good news here because S3 does support HTTPS. Because the content that you need to deliver over CloudFront sits in an S3 bucket, then you can also deliver that same content over HTTPS but you cannot take advantage of the edge locations. This is especially useful when you need to serve a secure site that has a bunch of images that you are already serving from CloudFront. If you try to serve those images over HTTPS, your users will get a security warning or no image at all because the browser will block it. So, when you need to serve those secure pages, simply switch your delivery URL to the S3 location instead. A word of advice here is to create your S3 bucket without using any '.'s. I know that this really messes with you if you want your URL to look like it is coming from your servers, but it will allow you to serve secure content from S3. The S3 bucket is served over HTTPS using a wildcarded certificate. This allows Amazon to serve your content using any prefix as long as it does not have any extra dots ('.'). If your bucket name is 'mys3bucket', then you can create a CNAME record that points 'mys3bucket' to mys3bucket.s3.amazonaws.com. Because the wildcarded SSL certificate is to *.s3.amazonaws.com, then you can serve your content from https://mys3bucket.s3.amazonaws.com. While this isn't the perfect solution, it is really nice for serving the same bucket content over HTTP or HTTPS.
Simple, Straight-Forward Development
I am not sure that building scalable applications is ever simple, but I am always looking for ways to make it extremely repeatative and simple for myself and other developers. Because we are using Java for most of our applications, the easiest way for me to do that was through a tag library. I included all of the logic for switching between the different CNAMES for CloudFront delivery and logic for detecting a secure connection and need to switch to straight S3 delivery right in the tag. There were some interesting details that were necessary to take full advantage of CloudFront. One was coming up with a hash scheme for a URL so that it could be cached in the browser and distributing the calls across the CNAMEs evenly. The second was to make sure that the tag was written so that it was at the request level instead of the page level. This helped ensure that even an included JSP would not cause a narrowing of the number of CNAMEs that I could use. This is the method in the tag that does most of the work
protected String getCDNUrl() {
String result = null;
if (((PageContext) getJspContext()).getRequest().isSecure()) {
result = getCDNSecureUrl();
} else {
result = getCDNUrls().get((1 <= getCDNCount()) ? Math.abs(getFile().hashCode()) % getCDNCount() : 0);
}
return result;
}
Saturday, March 28, 2009
Playing with Eclipse and Amazon Web Services
It has almost been a week since Amazon released its AWS Eclipse plug-in, but today was the first time I had the chance to really play with it. While much of its functionality is available through the AWS Managment Console, it is really nice to have the Eclipse views right there in your development tool so that you don't have to leave it. This will enable me to monitor our servers more frequently. You almost tend to forget how much you have running in the cloud sometimes! I really wish that the plug-in would have included the ability to link multiple AWS accounts to a single Eclipse instance as I have several accounts that I need to monitor. Installing the plugin was very easy, but there was a gotcha that took me a couple of minutes to figure out. Once I had everything installed and restarted Eclipse, I immediately went to the EC2 Instances tab. I tried to right click on one of the instances and open a shell to the instance. It did not work. The red X's that you see beside the key pairs in the image below should have clued me in before I tried.
It was missing the private key pair file and therefore could not launch the shell. There was no menu item on this view that would allow me to add the key pair file and the general setup has no indication of being able to add them. To correct the problem, I had to go to the EC2 AMIs tab and attempt to launch an AMI. Once you do this, you are presented with a launch screen and a listing of all of your key pairs. If you have existing key pairs, you can right click on one of them, as shown in the screenshot below, and choose "Select Private Key File..."
This allowed me to choose my private key file and the green check marks appeared in the EC2 Instances view. This allowed me to open a shell right from within that Eclipse view. After that, I was off to the races and everything worked exactly as stated on the AWS Eclipse page.
Wednesday, February 4, 2009
Rapidly Processing Video in the Cloud
Cloud computing is more than just a buzzword to those of us that rely on it everyday. It really makes computationally expensive tasks fun to tackle and much easier to pull off. We had a very successful weekend processing a lot of video, and I thought I could take this opportunity to give an example of how the cloud computing model really saves in the end. To give you a little bit of background, we needed to rapidly process some of the videos uploaded by our users this weekend in such a way that they would not even "feel" the difference. We put Amazon Web Services to work on the task and accomplished what would have taken an estimated at 286 computing hours in 10. Using a little SimpleDB, S3 and EC2 magic, we threw the problem to a grid of 20 EC2 instances that segmented and ripped through the videos in a fraction of the time it would have taken us otherwise. What is so special about that?.. For us it was a great time saver and money saver. I have included a screenshot of the AWS Management Console when we started the processing and after it was done and the instances were terminating.
If you have jobs that would take more than a couple of hours to do, it would be well worth your time to take a dive into the Amazon Web Services model. The Management console made it very easy to start all 20 instances with a single click and take them down afterwards. We used the new ability to copy content in S3 to a working bucket and the data storage of SimpleDB to help keep our EC2 instances from double processing.
I get a lot of requests to help people understand and harness the power of the cloud, and there is so much that you can use it for that the use cases are varied for every customer. While our production systems are built to do live scaling and processing based upon load, I thought that this stand alone problem would be one of the more typical use cases to share. It really shows that an isolated problem can be solved much faster with more resources very cheaply.
If you have jobs that would take more than a couple of hours to do, it would be well worth your time to take a dive into the Amazon Web Services model. The Management console made it very easy to start all 20 instances with a single click and take them down afterwards. We used the new ability to copy content in S3 to a working bucket and the data storage of SimpleDB to help keep our EC2 instances from double processing.
I get a lot of requests to help people understand and harness the power of the cloud, and there is so much that you can use it for that the use cases are varied for every customer. While our production systems are built to do live scaling and processing based upon load, I thought that this stand alone problem would be one of the more typical use cases to share. It really shows that an isolated problem can be solved much faster with more resources very cheaply.
Friday, January 9, 2009
AWS Management Console
Amazon announced today the beta release of its management console. The use of the console, which can be found at https://console.aws.amazon.com is free and very easy to use. If you are familiar with the services offered by Amazon Web Services, you will be up and running in no time on their management console. The console currently allows you to manage your EC2 instances, Elastic IPs, Security Groups, Key Pairs and Elastic Block Stores. Launching a new instance and even viewing the console output are only a click away. This will be extremely helpful in just getting a nice consise high level view of the systems we are running in the Amazon cloud.
Tuesday, December 2, 2008
Amazon SimpleDB for FREE

Have you wanted to dive in and try some parts of the cloud, but you just were not willing to pony up any cash? Amazon is giving you that chance with their latest announcement for web services. Yesterday they opened up SimpleDB into a public beta. SimpleDB allows you to store, retrieve and query data similar to a database. The difference is scalability. SimpleDB is great for storing simple data relationships that you need to be able scale and have the ability to handle large volumes. As of yesterday, Amazon is offering 25 machine hours, 1 GB of data transfer and 1 GB of data storage for free for at least the first six months. That is more than enough runway to test out an application or to help anyone prove out a concept. It should let the average user execute over 2 million requests against SimpleDB without paying a dime. We are currently using SimpleDB for some of our prepping and internal work at DigitalChalk and it has been working very well. Here is your chance... jump on in and let me know if you need any help learning to swim.
Wednesday, November 19, 2008
Amazon CDN CloudFront
That was fast! It was just a couple of months ago that Amazon told us that they were working on a CDN. Yesterday they took another step and told the world that it has arrived. The new service is called CloudFront and it works in conjunction with S3. To get started, you simply need to have content that you have placed into a publicly accessible bucket on S3 and a couple of API calls later, you have content that accessible from the cloud in CDN style. Your S3 content is served from the closest location to the requester and is stored there for a period of time so that subsequent calls can be served quickly from that location. The great thing is that it is not limited to the Seattle or DC locations for serving the content. There are many other datacenters across the US (and the world) that are capable of serving the content helping to ensure speedy access. I can only imagine that this will have other CDN providers very nervous. They are already dropping their prices dramatically and this will squeeze them even further.
Saturday, November 15, 2008
Microsoft's Azure Sky
Well, I have waited for a while now to post on Microsoft's new cloud computing platform because I was waiting for something, well, of substance. Does anyone else but me find it humourous that they have named their cloud computing effort Azure? Makes me think of a cloudless day ironically. I have said in the past that Microsoft is late to the game, and it looks like they are going to be even later. Seems as if everything that they have to offer is still in preview mode and I have seen very few examples of working applications on the Live Services or Azure platform. I do have the need to run some applications on Windows technology right now, and I was looking forward to Microsofts entry into cloud computing to see what I might be able to make use of, but with Amazon now offering Windows 2003 servers as AMIs I am wondering if the Azure platform will even be necessary. Will Microsoft be able to compete with Amazon on price? I think I will wait and see what the price model will be and how much they really intend to support the "Coming Soon" application languages and platforms such as Eclipse and PHP.
Thursday, October 23, 2008
EC2 News More Than Expected
Wow, I was not expecting all of the announcements that happened today. I knew that EC2 was coming out of beta, that has been talked about for some time now, and we all knew that EC2 Windows boxes would be available this Fall, but I can't say that I was expecting the other news. It is exciting though for companies like DigitalChalk that rely on excellent services around cloud computing.
You can see everything that was announced today on the Amazon Web Services Blog. While I am excited about EC2 being out of beta and playing around with the new Windows instances starting at 12.5 cents per hour, the other news is even more interesting.
There is much to look forward to!
You can see everything that was announced today on the Amazon Web Services Blog. While I am excited about EC2 being out of beta and playing around with the new Windows instances starting at 12.5 cents per hour, the other news is even more interesting.
- Amazon released an SLA for EC2
- Amazon plans to release an AWS management console
- Amazon is planning to release load balancing, auto scaling and cloud monitoring services
There is much to look forward to!
Thursday, October 2, 2008
Microsoft in the Cloud
Well, we knew it was coming... Microsoft has been working on a cloud computing initiative and today... Oops! Amazon beat them to the punch. Amazon announced today that they will be putting Windows on EC2 before the end of the year. This has to be a bummer for Balmer and company over at Microsoft who have been working on an initiative that should be announced next month. I am very interested to see how Amazon is pricing this and how they deal with Microsoft licensing. I really hope that they keep to the same model where they charge by the hour instead of throwing extra requirements for a valid license key for each server you stand up. That would defeat the purpose of "infinite" scalability. I haven't seen any talk about the model yet other than a mention of the same "pay as you go pricing", but I am sure interested in the nitty gritty details. I can't say for certain that we will be using Windows on Amazon at DigitalChalk, but I will be playing around on it you can be sure just to see how it performs.
Friday, September 19, 2008
Amazon S3 as a CDN?
It is coming... Amazon announced today that they are currently working with a small group of private beta customers on a edge network solution that will allow users of S3 to deliver the content to their users with low latency and high speed making use of what they call a "global network of edge locations on three continents to deliver your content from the most appropriate location". It is well known that they are currently serving their content from data centers in Washington DC, Seattle, and in the EU. I am wondering where their other locations are to make up the global network. That will be interesting to see. We are currently serving customers around the globe and have seen good performance most of the time, even to our customers in Australia. I am hoping that this will make our service that much more effective. The press statements talks about using public buckets for the service. I am hoping that this will also be available for private buckets as well! If Amazon can keep the costs way down, they will really start to disrupt the CDN market. We are getting several vendors calling us from CDNs and the prices all seem to go lower with each call. Let's see what this does.
Thursday, August 21, 2008
Amazon Web Services adds Elastic Block Store

We have known about it for months, but very few have been able to play with it until now. Several months ago Amazon started talking about "Persistant Storage" becoming available on their platform and today it is available to everyone. It is labeled as Amazon Elastic Block Storage (EBS). At DigitalChalk we have been looking forward to this for several reasons. The most immediate benefit that I see getting from this is the ability to have a block of storage that we can put the data store on for our databases. Right now we have very frequent and bandwidth intensive backup solutions in place that snapshot our data and put it into S3. With ESB we will be able to allocate a block of storage up to a TB in size and then simply attach that to any running EC2 instance, essentially making it a SAN in the clouds. Amazon states that "each storage volume is automatically replicated within the same availability zone" helping with the backup scenerios and reliability of data. It doesn't seem to have it across data centers, but this is at least going to help prevent failure due to a single piece of hardware. They have also included the ability to create a "point-in-time snapshot of volumes" which will be extremely useful for the movement of data and quickly standing up a duplicate instance for testing of any sort. This is also going to be helpful as we continue to consult and help companies move their infrastructure to the clouds. We are finding that many we help have a hard time mentally moving from the physical box where they can just add hard drive capacity and rely on a file system. This will allow us to gradually and easily move their applications "as is" to the clouds without having to introduce S3 from the start for reliable storage. I am looking forward to using this service to improve our process at DigitalChalk. The next on the list to come out of the private beta is SimpleDB for me. We have already architected some solutions for SimpleDB and and just itching for its release.
Tuesday, July 8, 2008
DigitalChalk on Cloud Computing Podcast
It is always fun to hear a someone in a "sales" role talk technical. There are always a couple of fun laughs to be had by all, but I have to hand it to our VP of sales, Tony McCune as he did a good job today on a podcast in which he was on the hotseat.
Tony was interviewed today by John Willis in his Cloud Computing podcast. It isn't extremely technical, but it does explain some of how we are using Amazon Web Services to accomplish the architecture that we have for the DigitalChalk products. It was great to listen in on and I have to give great props to Tony for doing a good job explaining some of the technologies we are using. Check it out for yourself at http://www.johnmwillis.com/cloud-computing/cloud-cafe-podcast-7/ . On a side note, John Willis has since signed up as an instructor and you will be able to take his courses soon! As soon as he gets the first ones posted I will let you know.
Tony was interviewed today by John Willis in his Cloud Computing podcast. It isn't extremely technical, but it does explain some of how we are using Amazon Web Services to accomplish the architecture that we have for the DigitalChalk products. It was great to listen in on and I have to give great props to Tony for doing a good job explaining some of the technologies we are using. Check it out for yourself at http://www.johnmwillis.com/cloud-computing/cloud-cafe-podcast-7/ . On a side note, John Willis has since signed up as an instructor and you will be able to take his courses soon! As soon as he gets the first ones posted I will let you know.
Monday, June 23, 2008
How is the Health of Amazon Web Services
I have been spending a large amount of my time lately consulting and talking about cloud computing and specifically Amazon Web Services. This is mainly because of our infrastructure sitting completely in the cloud. One of the questions that always comes up is "How do you know everything is going ok?". Let me start with saying that it does not matter if you house your application in the cloud, in a co-location facility, in the closet or on the desk next to you... it is important that if you are supporting a production application that you know what is going on with its health. This will allow you to make informed decisions with any action you need to take BEFORE (hopefully) your customers are impacted. We have several tools that we have been using on DigitalChalk to help in the monitoring of our applications. One of our core components for monitoring is Hyperic. We have customized the Hyperic tools to allow us to monitor all of our servers soup to nuts. We have spent alot of time configuring Hyperic solutions on our EC2 instances to help us have some visibility into the AWS platform and our application. The good news is that Hyperic has just released CloudStatus into beta.

For quick updates on how AWS is performing, we have been using the RSS feeds from Amazon at http://status.aws.amazon.com/ . With CloudStatus, we really have a better view of the historical data of the services. I am looking forward to another tool in the toolbelt and I hope it will really help to improve our monitoring capabilities and streamline our diagnosis. I am really interested to see if Hyperic will come out with a dashboard that will specifically target a single AWS account's health. That would be really nice because we have seen before that even when AWS is having trouble in some areas, others are fine.

For quick updates on how AWS is performing, we have been using the RSS feeds from Amazon at http://status.aws.amazon.com/ . With CloudStatus, we really have a better view of the historical data of the services. I am looking forward to another tool in the toolbelt and I hope it will really help to improve our monitoring capabilities and streamline our diagnosis. I am really interested to see if Hyperic will come out with a dashboard that will specifically target a single AWS account's health. That would be really nice because we have seen before that even when AWS is having trouble in some areas, others are fine.
Subscribe to:
Posts (Atom)











