Cloud World

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Thursday, 26 January 2012

My summer with the Google App Engine Team

Posted on 13:47 by Unknown



Today’s post is contributed by our Summer 2011 team intern, Chris Bunch. Chris did some great work on our Logs and MapReduce APIs and is also the first “App Engine Triple Crown” winner for developing the Experimental Logs Reader API in Python, Java and Go simultaneously.

Four years ago, I was a brand-new Ph.D. student at the University of California, Santa Barbara and when our research group (the RACELab) heard about Google App Engine, we were intrigued. We thought it presented a new model that enabled apps to scale the right way without severely constricting the types of programs users would write.


But we wanted to experiment with the core functionality of App Engine: the APIs, the scheduler, etc., and so we built AppScale, an open-source implementation of the Google App Engine APIs that allows users to deploy applications written in Python, Java, and Go to the infrastructure of their choice.

Wherever possible, we implement support for the App Engine APIs with alternative open-source technologies. We’ve added support for nine different databases, database-agnostic transactions, a REST interface that users of any programming language can communicate with (via an App Engine app), and the ability to run high performance computing programs over the whole thing and talk to it from your App Engine app. And here’s my favorite part - it all deploys automatically! You don’t need to tell it what block size you want for the distributed file system, or the size of the read buffers: we configure the necessary services automatically. Since AppScale is completely open source, if you don’t like the defaults, change them!

After creating our own system to run Google App Engine apps, I wanted to see how Google does it. Therefore, I decided to become an intern on the App Engine team and see if I could give them (and by extension, the App Engine community) something amazing over the summer. I started off with some work on the MapReduce API, making the sample app much easier to use and prettier all around. I also made a YouTube video showing how it all works and how easy it is to run MapReduce jobs over App Engine.

I then looked at a recurring question that App Engine users encounter: “How can I get my logging information for my application to answer data analytic questions?” It was an excellent problem to tackle, as we have users who want to be able to determine application-specific queries that Google Analytics or the Admin Console don’t answer. Currently users have to use appcfg to grab all their application’s data to a remote machine and run some analysis script over it.

To solve this problem, I created the Logs API, which gives applications programmatic access to their logs from within App Engine itself. Applications can use it to query small numbers of logs within a single request, and they can utilize the Pipeline, MapReduce, or Backends APIs if they have lots of logs they want to analyze. Logs contain both request-level information (e.g., the URL accessed, the HTTP response code returned) as well as logging info generated by the application (the logging module in Python, the Logger class in Java, and the logging methods that Go’s appengine package provides). The Logs API is available for use as of App Engine 1.6.1 by programmers using the Python, Java, or Go runtimes, in both the production environment and the local SDK.

I had a great time putting the Logs API together, and had a unique experience interning with the App Engine team. Programming in Python, Java, and Go on a daily basis was an exciting new challenge, and I loved it! 





Interested in interning with the App Engine team? Check out google.com/students for more information on internships.
Email ThisBlogThis!Share to XShare to Facebook
Posted in | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • SaltStack for Google Compute Engine
    SaltStack was built for the performance, reliability, security and scale delivered by Google Compute Engine . Needless to say we are very e...
  • 2013 Year in review: making Google Compute Engine Generally Available
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • Google Compute Engine and App Engine give Evite freedom to expand services anytime
    Today’s guest blog comes from Dan Mesh, Vice President of Technology at Evite, the pioneer in online invitations and social planning. Evite ...
  • Why We Deployed Zencoder on Google Cloud Platform
    Today's guest post comes from Jon Dahl, VP of Encoding Services at Brightcove. Brightcove’s Zencoder transcodes millions of video and au...
  • International Offline Disk Import now available with new disk-upload centers in Japan, India and Switzerland
    Google Cloud Platform customers have told us that transferring large data sets (in the hundreds of terabytes and beyond) can be expensive a...
  • Google Compute Engine is now Generally Available with expanded OS support, transparent maintenance, and lower prices
    Google Cloud Platform gives developers the flexibility to architect applications with both managed and unmanaged services that run on Googl...
  • 2013 Year in review: bringing App Engine to the PHP community
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • Update on Datastore Auto IDs
    In the upcoming Google App Engine 1.8.1 release, the  Datastore default auto ID policy in production will switch to scattered IDs to impro...
  • 2013 Year in review: giving time back to developers
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • DataStax Enterprise feels right at home in Google Compute Engine
    Today’s guest post comes from Martin Van Ryswyk, Vice President of Engineering at DataStax. The cloud promises many things for database user...

Categories

  • 1.1.2
  • agile
  • android
  • Announcements
  • api
  • app engine
  • appengine
  • batch
  • bicycle
  • bigquery
  • canoe
  • casestudy
  • cloud
  • Cloud Datastore
  • cloud endpoints
  • cloud sql
  • cloud storage
  • cloud-storage
  • community
  • Compute Engine
  • conferences
  • customer
  • datastore
  • delete
  • developer days
  • developer-insights
  • devfests
  • django
  • email
  • entity group
  • events
  • getting started
  • google
  • googlenew
  • gps
  • green
  • Guest Blog
  • hadoop
  • html5
  • index
  • io2010
  • IO2013
  • java
  • kaazing
  • location
  • mapreduce
  • norex
  • open source
  • partner
  • payment
  • paypal
  • pipeline
  • put
  • python
  • rental
  • research project
  • solutions
  • support
  • sustainability
  • taskqueue
  • technical
  • toolkit
  • twilio
  • video
  • websockets
  • workflows

Blog Archive

  • ►  2013 (143)
    • ►  December (33)
    • ►  November (15)
    • ►  October (17)
    • ►  September (13)
    • ►  August (4)
    • ►  July (15)
    • ►  June (12)
    • ►  May (15)
    • ►  April (4)
    • ►  March (4)
    • ►  February (9)
    • ►  January (2)
  • ▼  2012 (43)
    • ►  December (2)
    • ►  November (2)
    • ►  October (8)
    • ►  September (2)
    • ►  August (3)
    • ►  July (4)
    • ►  June (2)
    • ►  May (3)
    • ►  April (4)
    • ►  March (5)
    • ►  February (3)
    • ▼  January (5)
      • App Engine 1.6.2 Released
      • My summer with the Google App Engine Team
      • Google Cloud Storage: concurrency controls and dee...
      • Happy New Year from the App Engine team
      • Happy Birthday High Replication Datastore: 1 year,...
  • ►  2011 (46)
    • ►  December (3)
    • ►  November (4)
    • ►  October (4)
    • ►  September (5)
    • ►  August (3)
    • ►  July (4)
    • ►  June (3)
    • ►  May (8)
    • ►  April (2)
    • ►  March (5)
    • ►  February (3)
    • ►  January (2)
  • ►  2010 (38)
    • ►  December (2)
    • ►  October (2)
    • ►  September (1)
    • ►  August (5)
    • ►  July (5)
    • ►  June (6)
    • ►  May (3)
    • ►  April (5)
    • ►  March (5)
    • ►  February (2)
    • ►  January (2)
  • ►  2009 (47)
    • ►  December (4)
    • ►  November (3)
    • ►  October (6)
    • ►  September (5)
    • ►  August (3)
    • ►  July (3)
    • ►  June (4)
    • ►  May (3)
    • ►  April (5)
    • ►  March (3)
    • ►  February (7)
    • ►  January (1)
  • ►  2008 (46)
    • ►  December (4)
    • ►  November (3)
    • ►  October (10)
    • ►  September (5)
    • ►  August (6)
    • ►  July (4)
    • ►  June (2)
    • ►  May (5)
    • ►  April (7)
Powered by Blogger.

About Me

Unknown
View my complete profile