Cloud World

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Tuesday, 29 March 2011

High Replication Datastore: A Solid Choice for Enterprise-class Applications

Posted on 07:00 by Unknown




WebFilings delivers a revolutionary cloud-based application that enables companies to collaboratively build and file financial reports with the SEC.
Built on the Google App Engine platform
, companies of all sizes use WebFilings software to take control of their financial reporting processes. WebFilings greatly reduces the time and stress required to file financial reports with the SEC.




Enterprise Requirements

As an enterprise class application supporting many Fortune 100 customers with strict financial reporting requirements, we simply cannot afford to have maintenance outages or service disruptions. Google App Engine has proven to be a robust platform for us to build on and has allowed us to scale seamlessly to support a rapid increase in customer demand. We’re pleased to report that a recent innovation by Google has allowed us to achieve an even higher level of availability and performance.




This recent innovation is the High Replication Datastore or HR Datastore. HR Datastore allows us keep our application live even during system upgrades and other maintenance operations that Google may be performing. In addition, it delivers a new level of reliability that has given our application significantly more consistent response times and 100% availability since making the move.




Decisions... Decisions...

We were originally a bit hesitant to move to HR Datastore because it was a relatively new offering from the App Engine team. Even though the underlying technology has been around for a long time, we weren’t sure initially how the eventual consistency model might affect our application. Well, we did our homework and read the Google posts on HR Datastore, followed the App Engine news groups, pinged Googlers via the App Engine Office Hours Chat on IRC, and did some prototyping with entity groups and eventually consistent queries. From this we got a better handle on how entity groups worked to enforce strong consistency with ancestor queries and also learned of the potential performance issues of doing large parallel db.get() operations across multiple entity groups. We also reviewed our code, and made a few changes to our application where we needed to use entity groups to have strong consistency. In the end, we found the move to be much less work than we had anticipated.




The Migration

The actual migration to HR Datastore was a fairly simple process. Essentially, we just set up a a new appspot instance, and selected the High Replication Datastore option during creation. Then we migrated our data to the new instance.


During our migration process we:

  1. Created our new instance and deployed a new version of code to work with the new entity groups
  2. Scheduled our maintenance window with our customers
  3. Put our current master/slave instance into read-only mode
  4. Copied the data from our master/slave instance to our HR Datastore instance. (Due to the size of our data, this was accomplished with some help from the App Engine team)
  5. Ran a migration mapreduce to iterate over a few entities that now needed to be in entity groups. These were saved with a new parent making them part of the entity group. This enables queries to be consistent.
  6. Implemented a site alias that routes customers to the new instance, webfilings-hrd.appspot.com, from webfilngs.appspot.com. Note: This feature is not generally available so you should contact the App Engine team directly if you need similar functionality.

In short, we went through a complete release and testing cycle and got our application migrated to HR Datastore in under a month. The actual migration happened overnight when we moved a litte over 200GB of data to our new HR Datastore instance, executed our migration mapreduce, and did our verification testing in about five hours. Since moving to HR Datastore, we are wondering why we ever even thought twice. Our customers and our Customer Success Team that works with customers every day have given us feedback that the application is performing better than ever and we have had no downtime since the move.




A Solid Choice

The High Replication Datastore was designed with the needs of large enterprises in mind. HR Datastore provides maximum uptime, multiple datacenter redundancy, consistent response times, along with scalability and performance that has met the needs of our fast growing company. It does take some planning to make the move, however. You should be prepared to review your existing code and test thoroughly beforehand. We found that this effort alone had a significant payback in terms of overall application performance and reliability. Even though the average latency on HR Datastore calls can sometimes be a little bit longer, we were able to optimize and end up with an overall experience that was better for the end user.

If you have an application that has these same demands then my only question for you is “When are you moving to HR Datastore?”. If you are like us, you will be very glad you took advantage of this latest innovation from the Google team.



Posted by Dave Tucker

Director of Product Development

WebFilings (www.webfilings.com)
Read More
Posted in | No comments

Monday, 21 March 2011

Enomaly chooses Google App Engine for SpotCloud.com

Posted on 14:26 by Unknown



SpotCloud.com is the realization of a dream originating more than 7 years ago when we first started Enomaly. SpotCloud was born from a desire to change the way cloud computing is bought and sold for the better by overcoming the inefficiencies (including large capital commitments and low utilization rates) that beset the traditional models. SpotCloud is the first multi-supplier, multi-buyer model of cloud computing, all built on Google App Engine.





SpotCloud is the first marketplace where service providers can sell their unutilized or under-utilized IaaS cloud services, and where buyers can shop competitively for these services on the basis of quality, price, and geography across a federated global pool of capacity with a single, consistent set of access and management mechanisms.



SpotCloud provides its users with the following main services:

  • VM image repository & VM image provisioning services
  • REST API for cloud service users, with management and monitoring functionality
  • REST API for cloud service providers
  • REST client to access service provider API endpoints
  • Single-point unified access & authentication to all underlying cloud services
  • Federated management portal, providing management & monitoring over all connected cloud services
  • Cloud service quality, reliability, and performance measurement instrumentation and rating
  • Runtime management service (to enable time-limited instances)
  • Billing, payment, and settlement services



Why Google App Engine?

We realized early on that a traditional data center infrastructure -- even one built on our own ECP technology -- was not a fit for a platform which needs to be global from day one. We also had a strong preference for Python, given that large parts of our existing IaaS software stack is built using it. Cost was also very important: SpotCloud needs to manage potentially very large Virtual Machines, delivering them to a globally distributed group of providers and managing them in near-real time across this extended footprint. With Google App Engine we have the power, flexibility, and global breadth of the Google infrastructure at our disposal. This was a key selling feature for us. Also the cost was practically impossible to beat.



How App Engine powers SpotCloud

As you can see from the list of services provided by SpotCloud, most of our application actually runs in the background. This is made possible by App Engine task queues and cron jobs. We need to poll for stage changes across hundreds (soon thousands) of service providers -- we need a near-real-time view of instance states, available virtual hardware profiles, cloud utilization data, etc. These things are triggered by cron schedules. Each of these tasks goes into a queue. We're able to execute these tasks to allow us to keep our view of service provider resources up to date without interrupting the front-end UI and API workflows.



Another challenge is executing tasks at a particular point in the future. Running instances deployed out to a provider through SpotCloud may expire, and renewal may or may not be allowed depending on the seller’s settings. So we need a way to terminate them if their time is up. We can use the task queue API to execute this in the future. We're able to ensure our data is consistent by adding tasks to the queue in a transactional way.



Buyers upload the appliances they want to provision to the market. So we obviously have a big storage requirement. App Engine’s data store allows us to do this in a highly performant, cost-effective manner. SpotCloud can then distribute these appliances out to providers on demand. We're able to do this because of the low-cost bandwidth Google provides.



We're big fans of Django at Enomaly, and we use it in SpotCloud. We've been able to loosely couple Django with the App Engine data store, along with other App Engine components and services like urlfetch. This means that we're not "locked-in" with App Engine. The App Engine API is structured in such a way that we're able to extend any web framework we want with App Engine components that help us scale.



SpotCloud Success

Since announcing the SpotCloud private beta at the end of last year, we’ve been overwhelmed by the amount of interest we’ve gathered, and have had large numbers of service providers from all over the globe join our beta. We now have more than 10,000 servers from 40+ countries represented in the marketplace.



Since we opened the marketplace to buyers and sellers early in 2011, the market has been off to a roaring start -- as noted by The Economist, ComputerWorld, and many others. Google App Engine has performed up to our (high!) expectations, serving an ever-increasing transaction rate in the market without a hiccup. We’re looking forward, as the market ramps up even further, to seeing if we can make App Engine break a sweat! Sign up today as a cloud capacity buyer or seller at www.spotcloud.com.



Posted by Reuven Cohen, Founder & Chief Technologist, Enomaly Inc.
Read More
Posted in | No comments

Monday, 7 March 2011

Implementing Workflows on App Engine with Fantasm

Posted on 15:35 by Unknown
This post is another entry in our ongoing series of guest posts contributed by App Engine developers. Today we partner with Jason Collins and Shawn Rusaw of VendAsta Technologies who discuss their simplistic workflow management system for Python App Engine called Fantasm. Fantasm is very simple to start using but incredibly powerful. So powerful in fact, that they couldn't fit it into one blog post. So for more detail, see the Implementing Workflows on App Engine with Fantasm article in our documentation.



Most software systems of reasonable size need to implement workflows: a series of processing steps and decision points over an artifact, typically a document. You are likely using workflows even if you don't realize it; often processing is built up iteratively over time until you're left with a complex system with lots of subtleties. These systems can mysterious and difficult to manage or extend.


A formal workflow engine can offer help in these instances. It allows developers to break the steps and decision points into manageable and testable chunks, so that processing is predictable and measurable. Workflow engines can offer visibility into system operations and metrics around execution times and failure points. They provide retry mechanisms and allow tasks to be distributed among multiple computers.


We have developed Fantasm to be just such a workflow engine. Fantasm is a Python library that allows you to specify your tasks and steps between those tasks in a YAML configuration file. It uses a finite state machine model where the tasks are states and the steps are transitions. As a developer, you implement the code that executes inside of a state and are only responsible to return an event that is used to identify the appropriate transition to the next state. It is similar to the App Engine Pipeline API but is more simplistic and serves as a user-friendly introduction to the concept of workflows in App Engine apps.


Fantasm has become the central processing tool within our organization and we hope it can be of help to yours. For a more in-depth look at Fantasm (including examples!), please see the full Implementing Workflows on App Engine with Fantasm article in the App Engine documentation.


Posted by Wesley Chun, App Engine team
Read More
Posted in api, app engine, appengine, google, pipeline, python, workflows | No comments

Wednesday, 2 March 2011

Something old, something new: Technology meets tradition for the Royal Wedding

Posted on 11:28 by Unknown

[Cross-posted from the Europe Developer Blog]


As you may have heard, the UK is gearing up for a very special wedding on April 29, when Prince William and Catherine Middleton will be married in Westminster Abbey in London.


Unlike many previous Royal Weddings, this event will have its own website – and we're honoured that St. James’s Palace has chosen to use Google’s computing infrastructure to power the site.


The site, which just went live at www.officialroyalwedding2011.org, is hosted on Google App Engine, which allows developers easily to build and host their web applications on Google's own computing infrastructure. It's a great way to run apps quickly, more securely, and at scale, which makes it ideal for such an important national occasion. The site will be regularly updated by St. James’s Palace in the run up to the wedding day.


This was a great team effort between Accenture, who built the site, the web design agency Reading Room who led on the design and creative work, and the App Engine team.


We hope it proves to be the perfect marriage of tradition and modernity as we approach the big day.

Read More
Posted in | No comments

Wednesday, 23 February 2011

Pushing Updates with the Channel API

Posted on 15:48 by Unknown

If you've been watching Best Buy closely, you already know that Best Buy is constantly trying to come up with new and creative ways to use App Engine to engage with their customers. In this guest blog post, Luke Francl, BBYOpen Developer, was kind enough to share with us Best Buy's latest App Engine project.







As part of Best Buy's Connected Store initiative, we have placed QR codes on our product information Fact Tags, in addition to the standard pricing and product descriptions already printed there. When a customer uses the Best Buy app, or any other QR code scanner, they are shown the product details for the product they have scanned, powered by the BBYOpen API or the m.bestbuy.com platform.



To track what stores and products are most popular, QR codes are also encoded with the store number. My project at Best Buy has been to analyze these scans and make new landing pages for QR codes easier to create.



Since we have the geo-location of the stores and product details from our API, it is a natural fit to display these scans on a map. We implemented an initial version of this idea, which used polling to visualize recent scans. To take our this a step further, we thought it would be exciting to use the recently launched App Engine Channel API to update our map in real-time.








Our biggest challenge was pushing the updates to multiple browsers, since we'd most certainly have more than one user at a time looking at our map. The Channel API does not currently support broadcasting a single update to many connected clients. In order to broadcast updates to multiple users, our solution was to keep a list of client IDs and send an update message to each of them.



To implement this, we decided to store the list of active channels in memcache. This solution is not ideal as there are race conditions when we modify the list of client IDs. However, it works well for our demo.



Here’s how we got it working. The code has been slightly simplified for clarity, including removing the rate limiting that we do. To play with a working demo, check out the channel-map-demo project from GitHub.



As customers in our stores scan QR codes, those scans are recorded by enqueuing a deferred. We defer all writes so we can return a response to the client as quickly as possible.



In the deferred, we call a function to push the message to all the active channels (see full source).




def push_to_channels(scan):
content = '<div class="infowindowcontent">(...) </div>' % {
'product_name': scan.product.name,
'timestamp' : scan.timestamp.strftime('%I:%M %p'),
'store_name': scan.store.name,
'state': scan.store.state,
'image': scan.product.image }

message = {'lat': scan.store.lat,
'lon': scan.store.lon,
'content': content}

channels = simplejson.loads(memcache.get('channels') or '{}')

for channel_id in channels.iterkeys():
encoded_message = simplejson.dumps(message)

channel.send_message(channel_id, encoded_message)


The message is a JSON data structure containing the latitude and longitude of the store where the scan occurred, plus a snippet of HTML to display in an InfoWindow on the map. The product information (such as name and thumbnail image) comes from our BBYOpen Products API.



Then, when a user opens up the site and requests the map page, we create a channel, add it to the serialized channels Python dictionary, stored in memcache, and pass the token back to the client (see full source).




channel_id = uuid.uuid4().hex
token = channel.create_channel(channel_id)

channels = simplejson.loads(memcache.get('channels') or '{}')

channels[channel_id] = str(datetime.now())

memcache.set('channels', simplejson.dumps(channels))


On the map page, JavaScript creates a Google Maps map and uses the token to open a channel. When the onMessage callback is called by the Channel API, a new InfoWindow is displayed on the map using the HTML content and latitude and longitude in the message (see full source).




function onMessage(message) {
var scan = JSON.parse(message.data);

var infoWindow = new google.maps.InfoWindow(
{content: scan.content,
disableAutoPan: true,
position: new google.maps.LatLng(scan.lat, scan.lon)});

infoWindow.open(map);
setTimeout(function() { infoWindow.close(); }, 10000);
};


Finally, since channels can only be open for two hours, we have a cron job that runs once an hour to remove old channels. Before deleting the client ID, a message is sent on the channel which triggers code in the JavaScript onMessage function to reload the page, thus giving it a new channel and client ID (see full source).



You can see the end result on our map, watch a video about the BBYScan project or checkout the sample channel-map-demo project and create your own Channel API based application.



Posted by Fred Sauer, App Engine Team
Read More
Posted in | No comments

Friday, 11 February 2011

App Engine 1.4.2 SDK - API Updates and Additions Edition

Posted on 12:06 by Unknown

It’s only February and we’re already at our second release for the year! Today’s SDK release, 1.4.2 focuses on improving and updating a few existing App Engine APIs.


Improved XMPP API to help applications better interact with users. Notifications are sent when users sign in and out and when their status changes, and the application can now set presence details to be returned to the user. Subscription and Presence notifications are enabled as inbound services in the application configuration.


Task Queue performance and Task Queue API improvements. First, we’ve increased the maximum rate at which tasks can be processed to 100 tasks/second. Applications can also specify the maximum number of concurrent requests allowed per queue in their queue’s configuration file. This can help you more easily manage how many resources your task queue is consuming. We’ve also added an API that allows you to programmatically delete tasks, instead of managing this manually from the Admin Console.

As always, there are more features and issue fixes such as support for JAX-WS complete with a new article on how to build SOAP enabled App Engine apps, as well as support for Django 1.2, so be sure to read the release notes for Java and Python. We’ve also updated the App Engine Roadmap with a few new projects so take a look. And if you have any feedback, please visit the App Engine Groups.


Posted by the App Engine Team

Read More
Posted in | No comments

Tuesday, 1 February 2011

Beauty On the Outside, High Replication on the Inside

Posted on 08:54 by Unknown

Today we launched Google Art Project in collaboration with 17 of the world’s most renowned museums. Google Art Project is built on top of App Engine and lets you take virtual tours of famous museums using internal Street View technology, view high resolution images of famous art work, and create personal virtual artwork collections.


When Art Project started development several months ago, the team built the application using Java and the Master/Slave Datastore. However, as their launch date approached, we released the new High Replication Datastore configuration. With a scheduled maintenance period so soon after the site’s launch, they decided to switch over to the High Replication Datastore.


Before switching, they ran a load test to set a performance baseline for comparison after the application’s data was migrated. Now that the application has launched, we wanted to share the results of the test with you as an example of what to expect after a switch to the High Replication Datastore. Below are the mean numbers for latency of different parts of the site.



Here’s a description of what each page does behind the scenes:


Homepage: This is the landing page that just serves a static webpage for site navigation. Since this page does not pull information from the datastore, the latency is stable.


Collections: Art Project lets users create individual museum collections. These load tests specifically targeted adding and deleting paintings from a user’s personal collection, as well as rendering those collections. We notice the slightly increased latency from saving and deleting entities in the datastore.


Level Maps: These pages simply performed get() calls on the datastore using entity keys. Latency on these pages is consistent across instances.


Info Spots: This handler performs the most data intensive calculations of all of the handlers. It calculates all line of sight interest points for a user’s map position in a museum gallery room, and saves the points of interest to the datastore for that location. The good news is, this calculation doesn’t have to happen for every user. Once this data has been calculated for a given spot, it can re-used for other visitors to the site.


As you can see, while there was some increased latency when switching to the High Replication datastore, the site latency is still very low. And the migration required no major code changes and no modification to the datastore structure between the two load tests.


For more information about the High Replication Datastore, see the Datastore documentation. The next scheduled maintenance period for the Master/Slave datastore is February 7th, 2011 from 5pm - 6pm PST. The High Replication datastore and Google Art Project will not need to be read-only during this period. On the High Replication Datastore, your application won’t need to be either.


Posted by the App Engine Team
Read More
Posted in | No comments
Newer Posts Older Posts Home
Subscribe to: Posts (Atom)

Popular Posts

Categories

  • 1.1.2
  • agile
  • android
  • Announcements
  • api
  • app engine
  • appengine
  • batch
  • bicycle
  • bigquery
  • canoe
  • casestudy
  • cloud
  • Cloud Datastore
  • cloud endpoints
  • cloud sql
  • cloud storage
  • cloud-storage
  • community
  • Compute Engine
  • conferences
  • customer
  • datastore
  • delete
  • developer days
  • developer-insights
  • devfests
  • django
  • email
  • entity group
  • events
  • getting started
  • google
  • googlenew
  • gps
  • green
  • Guest Blog
  • hadoop
  • html5
  • index
  • io2010
  • IO2013
  • java
  • kaazing
  • location
  • mapreduce
  • norex
  • open source
  • partner
  • payment
  • paypal
  • pipeline
  • put
  • python
  • rental
  • research project
  • solutions
  • support
  • sustainability
  • taskqueue
  • technical
  • toolkit
  • twilio
  • video
  • websockets
  • workflows

Blog Archive

  • ▼  2013 (143)
    • ▼  December (33)
      • 2013 Year in review: topping 100,000 requests-per-...
      • 2013 Year in review: making Google Compute Engine ...
      • 2013 Year in review: bringing App Engine to the PH...
      • Now Get Programmatic Access to your Billing Data W...
      • 2013 year in review: making scalability easy with ...
      • 2013 Year in review: taking Google Cloud Platform ...
      • 2013 Year in review: pushing the limits of Big Data
      • 2013 Year in review: enabling native connections f...
      • 2013 Year in review: bringing Offline Disk Import ...
      • Best practices for App Engine: memcache and eventu...
      • 2013 Year in review: giving time back to developers
      • 2013 Year in review: bringing together mobile and ...
      • Go on App Engine: tools, tests, and concurrency
      • Qubole helps you run Hadoop on Google Compute Engine
      • Alert Logic security and compliance solutions for ...
      • Outfit 7’s Talking Friends built on Google App Eng...
      • You can now deliver any-screen streaming media usi...
      • Using Google Compute Engine with open source software
      • DataTorrent offers massive-scale, real-time stream...
      • DataStax Enterprise feels right at home in Google ...
      • Why We Deployed Zencoder on Google Cloud Platform
      • Scalr and Google Compute Engine
      • Cloud9 IDE on Google Compute Engine
      • Fishlabs architects upcoming game with Compute Eng...
      • An ode to Sharkon
      • SaltStack for Google Compute Engine
      • Google Compute Engine and App Engine give Evite fr...
      • SUSE Linux Enterprise Server Now Available on Goog...
      • Google Compute Engine is now Generally Available w...
      • The new Persistent Disk - faster, cheaper and more...
      • Red Hat and Google Compute Engine – Extending the ...
      • Google Compute Engine helps Mendelics diagnose gen...
      • CoolaData digs into the “why” of online consumer b...
    • ►  November (15)
    • ►  October (17)
    • ►  September (13)
    • ►  August (4)
    • ►  July (15)
    • ►  June (12)
    • ►  May (15)
    • ►  April (4)
    • ►  March (4)
    • ►  February (9)
    • ►  January (2)
  • ►  2012 (43)
    • ►  December (2)
    • ►  November (2)
    • ►  October (8)
    • ►  September (2)
    • ►  August (3)
    • ►  July (4)
    • ►  June (2)
    • ►  May (3)
    • ►  April (4)
    • ►  March (5)
    • ►  February (3)
    • ►  January (5)
  • ►  2011 (46)
    • ►  December (3)
    • ►  November (4)
    • ►  October (4)
    • ►  September (5)
    • ►  August (3)
    • ►  July (4)
    • ►  June (3)
    • ►  May (8)
    • ►  April (2)
    • ►  March (5)
    • ►  February (3)
    • ►  January (2)
  • ►  2010 (38)
    • ►  December (2)
    • ►  October (2)
    • ►  September (1)
    • ►  August (5)
    • ►  July (5)
    • ►  June (6)
    • ►  May (3)
    • ►  April (5)
    • ►  March (5)
    • ►  February (2)
    • ►  January (2)
  • ►  2009 (47)
    • ►  December (4)
    • ►  November (3)
    • ►  October (6)
    • ►  September (5)
    • ►  August (3)
    • ►  July (3)
    • ►  June (4)
    • ►  May (3)
    • ►  April (5)
    • ►  March (3)
    • ►  February (7)
    • ►  January (1)
  • ►  2008 (46)
    • ►  December (4)
    • ►  November (3)
    • ►  October (10)
    • ►  September (5)
    • ►  August (6)
    • ►  July (4)
    • ►  June (2)
    • ►  May (5)
    • ►  April (7)
Powered by Blogger.

About Me

Unknown
View my complete profile