Cloud World

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Tuesday, 29 March 2011

High Replication Datastore: A Solid Choice for Enterprise-class Applications

Posted on 07:00 by Unknown




WebFilings delivers a revolutionary cloud-based application that enables companies to collaboratively build and file financial reports with the SEC.
Built on the Google App Engine platform
, companies of all sizes use WebFilings software to take control of their financial reporting processes. WebFilings greatly reduces the time and stress required to file financial reports with the SEC.




Enterprise Requirements

As an enterprise class application supporting many Fortune 100 customers with strict financial reporting requirements, we simply cannot afford to have maintenance outages or service disruptions. Google App Engine has proven to be a robust platform for us to build on and has allowed us to scale seamlessly to support a rapid increase in customer demand. We’re pleased to report that a recent innovation by Google has allowed us to achieve an even higher level of availability and performance.




This recent innovation is the High Replication Datastore or HR Datastore. HR Datastore allows us keep our application live even during system upgrades and other maintenance operations that Google may be performing. In addition, it delivers a new level of reliability that has given our application significantly more consistent response times and 100% availability since making the move.




Decisions... Decisions...

We were originally a bit hesitant to move to HR Datastore because it was a relatively new offering from the App Engine team. Even though the underlying technology has been around for a long time, we weren’t sure initially how the eventual consistency model might affect our application. Well, we did our homework and read the Google posts on HR Datastore, followed the App Engine news groups, pinged Googlers via the App Engine Office Hours Chat on IRC, and did some prototyping with entity groups and eventually consistent queries. From this we got a better handle on how entity groups worked to enforce strong consistency with ancestor queries and also learned of the potential performance issues of doing large parallel db.get() operations across multiple entity groups. We also reviewed our code, and made a few changes to our application where we needed to use entity groups to have strong consistency. In the end, we found the move to be much less work than we had anticipated.




The Migration

The actual migration to HR Datastore was a fairly simple process. Essentially, we just set up a a new appspot instance, and selected the High Replication Datastore option during creation. Then we migrated our data to the new instance.


During our migration process we:

  1. Created our new instance and deployed a new version of code to work with the new entity groups
  2. Scheduled our maintenance window with our customers
  3. Put our current master/slave instance into read-only mode
  4. Copied the data from our master/slave instance to our HR Datastore instance. (Due to the size of our data, this was accomplished with some help from the App Engine team)
  5. Ran a migration mapreduce to iterate over a few entities that now needed to be in entity groups. These were saved with a new parent making them part of the entity group. This enables queries to be consistent.
  6. Implemented a site alias that routes customers to the new instance, webfilings-hrd.appspot.com, from webfilngs.appspot.com. Note: This feature is not generally available so you should contact the App Engine team directly if you need similar functionality.

In short, we went through a complete release and testing cycle and got our application migrated to HR Datastore in under a month. The actual migration happened overnight when we moved a litte over 200GB of data to our new HR Datastore instance, executed our migration mapreduce, and did our verification testing in about five hours. Since moving to HR Datastore, we are wondering why we ever even thought twice. Our customers and our Customer Success Team that works with customers every day have given us feedback that the application is performing better than ever and we have had no downtime since the move.




A Solid Choice

The High Replication Datastore was designed with the needs of large enterprises in mind. HR Datastore provides maximum uptime, multiple datacenter redundancy, consistent response times, along with scalability and performance that has met the needs of our fast growing company. It does take some planning to make the move, however. You should be prepared to review your existing code and test thoroughly beforehand. We found that this effort alone had a significant payback in terms of overall application performance and reliability. Even though the average latency on HR Datastore calls can sometimes be a little bit longer, we were able to optimize and end up with an overall experience that was better for the end user.

If you have an application that has these same demands then my only question for you is “When are you moving to HR Datastore?”. If you are like us, you will be very glad you took advantage of this latest innovation from the Google team.



Posted by Dave Tucker

Director of Product Development

WebFilings (www.webfilings.com)
Email ThisBlogThis!Share to XShare to Facebook
Posted in | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • Bridging Mobile Backend as a Service to Enterprise Systems with Google App Engine and Kinvey
    The following post was contributed by Ivan Stoyanov , VP of Engineering for Kinvey, a mobile Backend as a Service provider and Google Cloud ...
  • Tutorial: Adding a cloud backend to your application with Android Studio
    Android Studio lets you easily add a cloud backend to your application, right from your IDE. A backend allows you to implement functionality...
  • 2013 Year in review: topping 100,000 requests-per-second
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • Easy Performance Profiling with Appstats
    Since App Engine debuted 2 years ago, we’ve written extensively about best practices for writing scalable apps on App Engine. We make writ...
  • TweetDeck and Google App Engine: A Match Made in the Cloud
    I'm Reza and work in London, UK for a startup called TweetDeck . Our vision is to develop the best tools to manage and filter real time ...
  • Scaling with the Kindle Fire
    Today’s blog post comes to us from Greg Bayer of Pulse , a popular news reading application for iPhone, iPad and Android devices. Pulse has ...
  • Who's at Google I/O: Mojo Helpdesk
    This post is part of Who's at Google I/O , a series of guest blog posts written by developers who are appearing in the Developer Sandbox...
  • A Day in the Cloud, new articles on scaling, and fresh open source projects for App Engine
    The latest release of Python SDK 1.2.3, which introduced the Task Queue API and integrated support for Django 1.0, may have received a lot ...
  • SendGrid gives App Engine developers a simple way of sending transactional email
    Today’s guest post is from Adam DuVander, Developer Communications Director at SendGrid. SendGrid is a cloud-based email service that deliv...
  • Qubole helps you run Hadoop on Google Compute Engine
    This guest post comes form Praveen Seluka, Software Engineer at Qubole, a leading provider of Hadoop-as-a-service.  Qubole is a leading pr...

Categories

  • 1.1.2
  • agile
  • android
  • Announcements
  • api
  • app engine
  • appengine
  • batch
  • bicycle
  • bigquery
  • canoe
  • casestudy
  • cloud
  • Cloud Datastore
  • cloud endpoints
  • cloud sql
  • cloud storage
  • cloud-storage
  • community
  • Compute Engine
  • conferences
  • customer
  • datastore
  • delete
  • developer days
  • developer-insights
  • devfests
  • django
  • email
  • entity group
  • events
  • getting started
  • google
  • googlenew
  • gps
  • green
  • Guest Blog
  • hadoop
  • html5
  • index
  • io2010
  • IO2013
  • java
  • kaazing
  • location
  • mapreduce
  • norex
  • open source
  • partner
  • payment
  • paypal
  • pipeline
  • put
  • python
  • rental
  • research project
  • solutions
  • support
  • sustainability
  • taskqueue
  • technical
  • toolkit
  • twilio
  • video
  • websockets
  • workflows

Blog Archive

  • ►  2013 (143)
    • ►  December (33)
    • ►  November (15)
    • ►  October (17)
    • ►  September (13)
    • ►  August (4)
    • ►  July (15)
    • ►  June (12)
    • ►  May (15)
    • ►  April (4)
    • ►  March (4)
    • ►  February (9)
    • ►  January (2)
  • ►  2012 (43)
    • ►  December (2)
    • ►  November (2)
    • ►  October (8)
    • ►  September (2)
    • ►  August (3)
    • ►  July (4)
    • ►  June (2)
    • ►  May (3)
    • ►  April (4)
    • ►  March (5)
    • ►  February (3)
    • ►  January (5)
  • ▼  2011 (46)
    • ►  December (3)
    • ►  November (4)
    • ►  October (4)
    • ►  September (5)
    • ►  August (3)
    • ►  July (4)
    • ►  June (3)
    • ►  May (8)
    • ►  April (2)
    • ▼  March (5)
      • Announcing App Engine 1.4.3. Release
      • High Replication Datastore: A Solid Choice for En...
      • Enomaly chooses Google App Engine for SpotCloud.com
      • Implementing Workflows on App Engine with Fantasm
      • Something old, something new: Technology meets tra...
    • ►  February (3)
    • ►  January (2)
  • ►  2010 (38)
    • ►  December (2)
    • ►  October (2)
    • ►  September (1)
    • ►  August (5)
    • ►  July (5)
    • ►  June (6)
    • ►  May (3)
    • ►  April (5)
    • ►  March (5)
    • ►  February (2)
    • ►  January (2)
  • ►  2009 (47)
    • ►  December (4)
    • ►  November (3)
    • ►  October (6)
    • ►  September (5)
    • ►  August (3)
    • ►  July (3)
    • ►  June (4)
    • ►  May (3)
    • ►  April (5)
    • ►  March (3)
    • ►  February (7)
    • ►  January (1)
  • ►  2008 (46)
    • ►  December (4)
    • ►  November (3)
    • ►  October (10)
    • ►  September (5)
    • ►  August (6)
    • ►  July (4)
    • ►  June (2)
    • ►  May (5)
    • ►  April (7)
Powered by Blogger.

About Me

Unknown
View my complete profile