New! Grow your app beyond the free quotas!

Posted by Brett Slatkin, App Engine Team

We're psyched to announce that developers can now purchase additional computing resources on App Engine, enabling apps to scale beyond our free quotas. This has been our most requested improvement to App Engine and we're thrilled to deliver it, as promised.

When we released App Engine last April, it was completely free to use, but each application was limited to a set of fixed resource usage quotas. The free quotas are still there, but now you can grow beyond them.

You can now set a daily budget for your app that represents the maximum amount you're willing to pay for computing resources each day. You allocate this budget across CPU, bandwidth, storage, and email, and you pay for only what your app consumes beyond the free thresholds -- prorated up to the nearest penny. We've put together a handy screencast to help explain the process:

App Engine remains free to get started. However, along with many performance improvements over the past ten months, we've learned that we overestimated our initial free quota values. Therefore, in 90 days we will be reducing the free quota resources. We believe these new levels will continue to support a reasonably efficient application serving around 5 million page views per month, completely free.

The pricing for resources beyond those free quotas is:

• $0.10 per CPU core hour. This covers the actual CPU time an application uses to process a given request, as well as the CPU used for any Datastore usage.


• $0.10 per GB bandwidth incoming, $0.12 per GB bandwidth outgoing. This covers traffic directly to/from users, traffic between the app and any external servers accessed using the URLFetch API, and data sent via the Email API.


• $0.15 per GB of data stored by the application per month.


• $0.0001 per email recipient for emails sent by the application

Data stored in the datastore incurs additional overhead, depending on the number of indexes, as well as the number (and size) of associated properties. This overhead can be significant in some cases, and it's something that we have been underreporting up until now. Thus, you may notice an increase in the amount of data stored by your application that is listed in the Admin Console.

To decrease the impact of this change in the way we calculate storage usage, we've doubled the free storage quota to 1GB. To learn more about how our quotas work, upcoming changes to our quota levels, how they relate to billing, and overall resource consumption limitations, see the Quotas section of the docs. The Purchasing Additional Quota section has more detail about how to purchase additional computational resources for your app, and there's also a Billing FAQ. We've also made some changes to our terms of service to include language around payments, fees, and disallowing the use of multiple applications to avoid incurring fees.

As always, we welcome feedback in our discussion group, and don't forget to check out the App Engine sessions at our upcoming developer conference, Google I/O!

Read More

Back to the Future for Data Storage

Building a massive, distributed datastore which can service requests at an extremely high throughput is something that we've focused on at Google. We created something called Bigtable that underlies the datastore in App Engine. The design for Bigtable focused on scalability across a distributed system so it may operate a bit differently than databases you've worked with before, such as not supporting joins. This isn't an accident -- when you build a system that can scale to the size that Bigtable can there's no way to do a general purpose join on data sets that size and still have them be performant.

Google isn't alone in offering an non-Relational datastore to enable scaling. For example, Amazon has SimpleDB:

A traditional, clustered relational database requires a sizable upfront capital outlay, is complex to design, and often requires a DBA to maintain and administer. Amazon SimpleDB is dramatically simpler, requiring no schema, automatically indexing your data and providing a simple API for storage and access.

There are also a range of non-relational open source datastores now available such as CouchDB and Hypertable. Those are just two examples, there are many more.

While you might think this is all new, it's actually a bit of a return to the past. You see, there was a time when "RDBMS" wasn't always the answer regardless of what the question was. At the time Codd published his paper, "A Relational Model of Data for Large Shared Data Banks," there were many different approaches to datastores. It was only in the '80s that relational databases won the majority of the mindshare. Having settled on a single metaphor the industry has developed many tools and techniques to make developing on a relational database easier.

Unfortunately that majority mindshare is also a problem because while RDBMS' are useful in many situations, they are not useful in all situations. Their dominance in the mindshare means that useful alternatives aren't used, and huge amounts of time and money can be wasted trying to force non-relational problems into a relational model.

We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Michael Stonebraker pointed out in his paper, "One Size Fits All": An Idea Whose Time Has Come and Gone, that there are common datastore use cases, such as Data Warehousing and Stream Processing that are not well served by a general purpose RDBMS and that abandoning the general purpose RDBMS can give you a performance increase of one or two orders of magnitude.

It's an exciting time, and the takeaway here isn't to abandon the relational database, which is a very mature technology that works great in its domain, but instead to be willing to look outside the RDBMS box when looking for storage solutions.

Posted by Joe Gregorio, Google App Engine Team

Read More

Web App Wednesday, Mashup, Backup, and Decay

Posted by Jeff Scudder, App Engine Team

Michael Bernstein is doing something I find very interesting. Not only is he creating applications on App Engine, he creates a new one every Wednesday. His site, appropriately named Web App Wednesday, is where he is creating a new web application every Wednesday, starting first with the site itself, followed by F/OSS License Info, a single-page site that displays information about the most common Open Source software licenses. Another recent app, LinQR , is a URL shortening service and QR code generator. Good luck to Michael and we look forward to seeing what he comes up with next Wednesday.

If you're doing something cool with App Engine please let us know! We'd love to highlight it here on the blog or even have you write up an article explaining what you've done, like this new article from Omar Abdelwahed on Creating a Facebook App with Google App Engine and Best Buy Remix.

Speaking of articles, Aral Balkan has written an article on his backup and restore utility and our own Joe Gregorio published an article on a design for popularity ranking using voting and time-based decay.

Read More

The sky's (almost) the limit! "High CPU" is no more.

Posted by Pete Koomen, App Engine Team

We're very excited today to announce that we've raised limits on several App Engine operations:

• No more "High CPU Requests"! App Engine Apps were once allowed no more than 2 CPU-intensive requests per minute. We've made some adjustments to the way we handle requests, and have eliminated this limitation altogether. To learn more about how this works and the implications for your app, see our documentation.


• Response deadline raised to 30 seconds. The amount of time an App Engine app can take to respond to an incoming request has been raised from 10 to 30 seconds! There are limits on the number of simultaneous active requests an application can process at any given moment--see our docs to learn more.


• Size limits on code files, static files, and requests/responses raised to 10MB! App Engine apps can now receive requests and send responses of up to 10MB in size, and users can upload 10MB code and static files as well. Note that API requests (e.g. memcache.set(), db.put()) are still limited to 1MB in size.

These changes were inspired and driven by a great deal of developer feedback, and we're not done! Please let us know what you'd like to see next on App Engine in our Google Group!

Read More

SDK version 1.1.9 Released

Today we released version 1.1.9 of our SDK. Here's what's new in this release:

• You can now use the Python standard libraries urllib, urllib2 or httplib to make HTTP requests. This has been a frequent request on our issue tracker.


• We've been working on a set of tools that will make the process of uploading and downloading data from App Engine applications easier. Today we're excited to announce an early release of our new bulk uploading client. You can try it out here. Let us know what you think in our Google Group!


• Several updates to our datastore, including the automatic generation of single property indexes and the addition of IN and != operators to db.Query. See the Datastore API docs for more details.


• A bunch of additional bugfixes and enhancements, listed in our Release Notes.

You can download version 1.1.9 of the SDK for Windows, Mac, and Linux now on our Downloads page. As always, we spend a lot of time reading (and posting!) on our Google Group, and we welcome your feedback!

Posted by Pete Koomen, Google App Engine Team

Read More

A roadmap update!

Posted by Joe Gregorio, App Engine Team

The App Engine team has been plugging away and we're excited about some pretty big announcements in the near future. In the meantime, we decided to refresh our App Engine roadmap for the next six months with some of the great new APIs in our pipeline:

• Support for running scheduled tasks


• Task queues for performing background processing


• Ability to receive and process incoming email


• Support for sending and receiving XMPP (Jabber) messages

As always, keep in mind that development schedules are notoriously difficult to predict, and release dates may change as work progresses. We'll do our best to update this roadmap as our engineers continue development and keep you abreast of any changes!

You'll have the opportunity to discuss this roadmap (and all things App Engine) with us and your fellow developers during Google I/O, coming up in May. Check out the App Engine sessions we've already announced and don't forget to discuss in our Google Group!

Read More

Best Buy's Giftag on App Engine

Posted by Amanda Surya, Google App Engine Team

Built by Best Buy, Giftag is a gift registry add-on for FireFox and Internet Explorer that enables you to create a wish list and share it with others. The Giftag team first built a prototype of the application on another platform and went live on September 2008. About 10 weeks later, the team relaunched Giftag on App Engine just in time for Black Friday. Since then, they've been getting positive reviews from users about the product.

We're happy to share this video created by Jerry St. Sauver, Curtis Thompson, and Thomas Bombach, Jr., the development team behind Giftag. In this video, they share their thoughts on App Engine, their development experience, tips on getting started with App Engine, and finally a demo of Giftag in action.

To learn more about what the Giftag team is up to, check out the official Giftag blog.

To share your own experience using Google App Engine, tell us your story (or even your own video) in our developer forum.

Read More