Offline Disk Import and the OmNomNom Machine

Yesterday, we announced that we are expanding our offline disk import service to better serve users globally. With new disk upload centers in Switzerland, Japan and India, as well as our US center, it’s easier for people around the world to import large data sets by mailing hard drives to us rather than sending hundreds of terabytes over their slow, expensive or unreliable Internet connection.



But importing large amounts of data at scale isn’t simple. Our engineers have been working on the challenge for years. Originally, offline disk import was handled at our data centers as a way to efficiently import large amounts of data from the hard drives in our Street View cars - vehicles that capture terabytes of photographs and information about the landscape as they build a navigable, visual database of the world.



And although it’s technically challenging, the system we built for rapidly ingesting and processing these large data sources has a playful name. We call it OmNomNom. Here is one of the test OmNomNom machines that the disk import team has at its offices in Mountain View:

But rapidly ingesting and processing large data sets and making them usefully available is a bit more complex than gobbling down a cookie. As we’ve improved our ability to quickly import these drives, we dramatically reduced the time between capturing these images and making them available to users around the world.



Now, we are helping people across the world take advantage of the speed, scale and global availability of Google Cloud Storage as well as this rapid disk-upload technology. Even though it might sound like something out of Sesame Street, this is another example of how Google Cloud Platform is making the advantages of Google-sized scalable infrastructure available to you. All you need to do is send us your EncFS encrypted hard drives, and we will let you know once your encrypted bytes are imported to your designated GCS bucket. Once uploaded, we can mail your drives back to you, or if you prefer, safely and securely handle disk destruction free of charge. Check out our website for how you can be part of the Limited Preview of International Offline Disk Import.



(Oh, and for those of you who want to use Google Cloud Storage but don’t need offline disk import, you always have quick access Google Cloud Storage from the command line using gsutil).



-Posted by Benjamin Bechtolsheim, Product Marketing Manager