Amazon S3 as a Backup Solution

by Nate 23. September 2007 22:57

I've always understood the importance of backing up my workstations, but haven't always followed through with action. Luckily it has never come back to burn me too badly, although I did have one bad experience a couple of years ago when I lost a good portion of my digital music library. Well, over the last year and a half I've tried to change this by consistently backing up important files to external hard drives and taking image snapshots of all my computers using Acronis' True Image product (see My Favorite Applications and Tools post for more info on True Image). However, with more and more of my life (documents, emails, music, pictures, projects, videos, etc.) living on a computer somewhere, I've decided that I now need a more robust and fail-proof backup solution. Enter Amazon Simple Storage Service (S3).

For those who aren't familiar with S3, it is an "in-the-cloud" storage service that Amazon provides as part of its growing list of web services. These web services are built using the same technologies that Amazon.com uses, meaning that a lot of lessons have been learned and integrated into them. Developers can access these services through a variety of web service interfaces, and, using S3, can gain access to unlimited storage on Amazon's robust infrastructure. And yes, this includes the same availability and performance that users have come to expect from Amazon.com. As for backups, S3 is appealing not only because of its availability and performance, but also because files stored in S3 are encrypted and redundantly stored in multiple data centers, meaning that the likelihood of loss of files is extremely low, especially when compared to the chance that your external hard drive will fail. That said, I still don't recommend making an S3 repository your authoritative copy of your data, as it's always a good idea to have at least one copy of your data somewhere where you can access it, no matter what.

I've been keeping an eye on Amazon Web Services for quite a while now, and have recently opened up a dialog with the Amazon Web Services team about the possibility of migrating some of my organization's map cache and media into S3. I've also been playing around with integrating S3 into a couple of demo ASP.NET and Ruby on Rails applications as proofs-of-concept. As I've learned more and more about S3, the idea of using it for personal backups of important information has grown on me.

As I hinted at earlier, Amazon gives developers tools to connect to (authenticate) and transfer files to and from S3. I, however, wanted an easier, more automated solution to help with my backups. I looked at a couple of solutions, including the Firefox Organizer for Amazon S3 (which I definitely suggest using, if just for browsing the files that you're storing on S3) and Jungle Disk, a GUI for using Amazon S3 that mounts your S3 as a local drive letter (on Mac and Window machines, I'm not sure how you connect if you're using Linux, but I do know that Jungle Disk is supported on all three operating systems) and optionally automates backups. I decided on Jungle Disk, as it's easy to setup and use and acts a lot like a local hard drive. Note that there is a one-time *lifetime $20 fee for purchasing the utility. In my opinion it is well worth it. As I've said before, I'm willing to pay for a service that makes my life easier.

Jungle Disk has a ton of configuration options, but is still trivial to get setup and running. You simply download and install the utility, sign up for an S3 account, tell Jungle Disk your S3 account information, and tell it which files (or directories) you want it to backup and when. If you set it up for automatic backups, it will perform a scheduled check to see which files have changed and only backup those that have been modified, keeping the S3 bandwidth/storage costs as low as possible. Taken from the Amazon S3 pricing page:

Storage

  • $0.15 per GB-Month of storage used

Data Transfer

  • $0.10 per GB - all data transfer in
  • $0.18 per GB - first 10 TB / month data transfer out

Requests

  • $0.01 per 1,000 PUT or LIST requests
  • $0.01 per 10,000 GET and all other requests

Pretty simple and cheap, eh?

I just installed Jungle Disk and ran my first backup today. I got an average of ~600 kb/s on upload (and, yes, you can limit the upload speed and tell it to run only at certain times), and interacting with the files through Windows Explorer is just as fast as interacting with them on a local disk. In fact, as long as Jungle Disk knows where a copy of a file lives on my local machine, it will access that file automatically rather than going out to S3 to access it. If S3, however, has a newer copy of the file, it will browse to it.

Overall, so far so good. Maybe I can finally have some peace of mind about my digital files?

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Amazon S3 | Life | Utilities

1-800-GOOG-411

by Nate 24. May 2007 04:54

This one may be more important to me than it is to others, but having just moved to a new city this past September, I find myself dialing 411 more than I'd like from my mobile phone. To be honest, I dialed it quite a bit even before I moved into these new and unexplored areas. Well, Google is proposing a solution to all those extra charges! I found this one on the Google LatLong blog, but haven't tried it out yet. I'm sure I will, though, in the very near future (tomorrow, and probably the next day, and the next...).

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

And the News is...More Snow!

by Nate 31. January 2007 04:47

I'm not *too tired of it yet, although it's certainly starting to take a toll. The snow just seems to keep lurking around Colorado. We've been in a pattern for the last couple of months: it snows for a couple days, clears up for about five or six, then snows again. At my house (which is at ~9000 feet and located in a notoriously snowy area) we've seen close to ten feet of snow so far this winter and it's been so cold that none of it has gone away. And to top it all off, the wind has been so strong (they measured 120+ mph winds in Wondervu, where I live, during one wind storm earlier this month) that just about everytime I come home I have to shovel - even if it hasn't snowed in days.

I saw a report earlier today, though, that said the weather pattern is changing from El Nino to La Nina. This will hopefully mean that the ski areas get more snow while we get less. Anyway, as you can see from the image it looks like the snow isn't going away quite yet. I can't tell you how many times over the last several months I've seen that exact forecast.

Some weather records that have been set in Denver and the frontrange so far this winter:

              • 3rd snowiest December on record, with 29.4 inches from two major storms and a couple of minor snowfalls.
              • 8th coldest January on record (average temperature of 22 degrees in Denver so far).
              • 5th longest period of ground snowcover in Denver on record (over six weeks of snowcover and counting).

There's still a possibility that this could be the snowiest winter in history for Denver, although the record of 118.07 inches - set during the snowfall season of 1908-1909 - will be tough to beat. So far the city has received 58 inches of snowfall.

If you'd like to see some pics from the last couple of storms, go to my Smugmug galleries.

Listening to Mark O'Connor - Thirty-Year Retrospective...
My Nose is In Cormac McCarthy - Suttree...

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

This American Life Podcast

by Nate 27. October 2006 03:30

I caught this one from Lifehacker. Up until now, NPR's wonderfully intelligent show, This American Life, was only available on streaming connections or via Audible.com - for a fee, of course. Now the most recent show is being released weekly as a podcast. To get to archived shows, you still have to pay $.95 per show to Audible.com or iTunes.

This is great news! I wish that I had the time to sit down every week, at the same time, and listen to This American Life on my local NPR station (KUNC) - but I simply don't. Now I'll be able to listen whenever I have the time and inclination.

Link to This American Life podcasts

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

Playing in the Snow

by Nate 23. October 2006 05:27

Here at my new home we've already had three or four decent snows. There's been a good covering, probably close to ten inches, on the ground for at least two weeks now; I'm thinking that I may not see the bare ground again this year. I'll admit it - I'm definitely new to this. For the last five years or so out East we were lucky if we got a couple of three inch snows a year. Now I come out here to the mountains of Colorado and move into a cabin at 8500 feet (~UTM 466472.61 East, 4419642.16 North). I'm in for a rude awakening, right? Well, maybe. But at least I'll have fun playing in the snow.

This past weekend kicked off the year's ski season for me. I went up to Arapahoe Basin with some friends, and waited in line at the only open ski lift with fifteen queues of other skiers/boarders to get to the top of the two open slopes. Oh well, maybe I learned my lesson? Nah, probably not. I'll be there the first chance I get next year, as well. All in all it was still a great time. I got my "ski legs" back, and am itching to get onto those upper slopes.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

Banned Books

by Nate 13. September 2006 03:36

This post has nothing to do with GIS or IT, or - for that matter - anything that I usually write about. It is, however, just as (if not more) important than most topics and issues that I regularly cover.

This week is the American Library Association's "Banned Books Week". Speaking personally, many (if not most) of the books that have had the greatest impact on my life are those that are on the ALA's challenged or banned list. To me, the books that challenge my own personal convictions and make me truly examine myself as a person are those that I consider truly great. Just looking at the challenged and banned list, I can say that the following books have had a direct impact on my personal development over the last twenty or so years:

  • Aldous Huxley - A Brave New World
  • Ernest Hemingway - A Farewell to Arms
  • Ernest Hemingway - For Whom the Bell Tolls
  • Upton Sinclair - The Jungle
  • George Orwell - 1984
  • Harper Lee - To Kill A Mockingbird
  • J.D. Salinger - Catcher in the Rye
  • John Steinbeck - Of Mice and Men
  • John Steinbeck - The Grapes of Wrath
  • Joseph Conrad - Heart of Darkness
  • Joseph Heller - Catch-22
  • Ken Kesey - One Flew Over the Cuckoo's Nest
  • Truman Capote - In Cold Blood
  • Vladimir Nabokov - Lolita
  • William Golding - The Lord of the Flies
  • William Faulkner - As I Lay Dying

Other favorites:

  • Aldo Leopold - A Sand County Almanac
  • Charles Dickens - A Tale of Two Cities
  • Edward Abbey - Desert Solitaire
  • Ernest Hemingway - The Old Man and the Sea
  • Erich Maria Remarque - All Quiet on the Western Front
  • Jack Kerouac - Desolation Angels
  • Jack Kerouac - On the Road
  • Jack Kerouac - The Town and the City
  • James Agee - A Death in the Family
  • John Muir - My First Summer in the Sierras
  • Kate Chopin - The Awakening
  • Leo Tolstoy - Anna Karenina
  • Leo Tolstoy - War and Peace
  • Marcel Proust - Swan's Way
  • Mark Twain - The Innocents Abroad
  • Vladimir Nabokov - Pale Fire

I didn't mean to make the list so long, but, to be honest, I could go on and on. To reminisce like this makes me want to go out and read a great book. In addition to those listed above, I've read a number of great modern pieces of literature. Any good suggestions?

Listening to Elliot Smith - XO...

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

Smugmug

by Nate 9. September 2006 04:20
null
 
Going on a a good review from Omar Shahine, I decided to check out smugmug. To be honest, I haven't used all that many photo apps, especially not online ones. I've played around with Adobe Lightroom, Picasa, and Picajet (finally settling on Picasa because of its ease of use). The thing about these stand-alone applications, though, is that they don't really support sharing all that well. Yeah, Picasa's web albums feature is in beta now, and all of the big names in photo management software are trying to find a way to make sharing easier. But, in my opinion, they just aren't there yet.

Before I go on, I need to confess that I don't particularly like web apps. They are great for some things, but most of the time I would rather have the power of a full-blown, locally installed application. When it comes to photos, however, I just need something that will allow me to document, store, and share the photos. If I want to edit them, I'll use the GIMP.

Keeping these simple needs in mind, smugmug does everything that I want/need, and looks great doing it. The interface is intuitive and easy-to-use, and the philosophy of the company rocks. Yeah, it costs a little bit of money, but they do allow unlimited storage and the features that come with the service make the price tag look more than fair:

  • Ability to set comprehensive permissions for your photos.
  • No advertisements - seriously.
  • A really slick AJAX-style interface.
  • Integrated photo purchasing support for families, friends, and fans.
  • An intuitive interface that allows for easy upload and organization of photos.

If you're looking for a way to manage your photos and make them available to others, you should definitely check out smugmug.

Listening to My Morning Jacket - Z...

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

Zidane's Headbutt

by Nate 12. July 2006 02:52


Well, the people of France have apparently forgiven Zidane for his loss of control in the World Cup final this past Sunday. He, of course, headbutted Italian defender Marco Materazzi 20 minutes into the first overtime period and rumors are flying around as to what Materazzi said to Zidane to set him off.

One statement that really caught my eye was that of Franz Beckenbauer, former German great. He defended Zidane's actions almost immediately after the incident, saying, "Something must have been said to Zidane. He is actually a reserved and inoffensive person." While this statement is not entirely true - Zidane has lost his temper in much the same way several times in the past - it still means a lot.

The best summary that I have found of the incident and the circumstances surrounding it is on wikipedia. You can also click on the image of the headbutt above to view video.

Update: FIFA has called an investigation of the incident, and is expected to release their decision on the 20th. They have said that Zidane's World Cup Golden Ball could possibly be taken away. I'll let you all know what happens with the investigation.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Life

Powered by BlogEngine.NET 1.4.5.0
Theme by Mads Kristensen
GeoURL