Cloud World

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Monday, 5 December 2011

Whentotweet.com - Twitter analytics for the masses

Posted on 10:00 by Unknown
Our post today comes from Stefan and Niklas of Whentotweet.com, a nifty site that recommends the best time of day to tweet based on your followers’ habits.



Twitter handles an amazing number of Tweets - over 200 million tweets are sent per day.

We saw that many Twitter users were tweeting interesting content but much of it was lost in the constant stream of tweets.

Whentotweet.com is born

While there were many tools for corporate Twitter users that performed deep analytics and provided insight into their tweets, there were none that answered the most basic question: what time of the day are my followers actually using Twitter?

And so the idea behind Whentotweet was born. In its current form, Whentotweet analyzes when your followers tweet and gives you a personalized recommendation of the best time of day to tweet to reach as many as possible.

Given the massive amount of data we needed to analyze, we knew it would be a huge engineering challenge to build what we wanted using the tools we had used previously. We also wanted to make sure we could offer at least a basic product for free. Not only did we need to process massive amounts of data - we also needed a way to do it without a second mortgage on our houses!

The Technology Used

As we went over the alternatives we started to sketch different ways of hosting our application. We had previous experience building web sites and knew that traditional cloud hosting would be expensive and difficult to manage for the kind of computing that we needed. After some quick back-of-the-envelope calculations it seemed clear that Google App Engine would give us both the kind of pricing we needed and a way to scale. We decided to write a quick test application to test our assumptions.

The test application blew our minds. Apart from proving our initial assumptions around pricing and scale we started appreciating the quick deploys. On previous projects we were used to one deploy per month. Almost immediately we shifted our schedule to one or sometimes several deploys per day to push new code to customers.

The main APIs that Whentotweet relies on are Google App Engine's task queues and Datastore. Whenever a new user requests a report it is added as a task. A typical report requires a huge number of interactions with external sites. By breaking down each external interaction into separate tasks in different queues it became easy to make sure we kept a steady rate of API calls to external sites without risking that a huge influx of users would break our API limits.

The initial task then spawns new tasks until finally one of the tasks decides that the report is complete and tweets a summary of the result and a link to a more detailed report. Whentotweet uses a "fail fast" technique so whenever any request fails, internal or external, the task terminates and puts itself back on the queue.

The Datastore saves a finished or ongoing analysis. Sometimes a single analysis will be updated several times a second by tasks as they finish and store their results.

The Result

After a few weeks of intense coding, we were ready to test our code on a small sized Twitter account with less than three hundred followers. The results came back in just a few minutes.

After verifying that everything had actually worked as well as we thought, we decided to try another account. This time one of the largest Twitter accounts on the planet: @techcrunch. Handling a Twitter account with over a million followers took the application one week. But after the analysis started, Whentotweet would quietly work in the background without us having to lift a finger.

Whentotweet got off to a better start than we imagined. During the initial launch thousands of people tested it on their Twitter accounts.

After a while blog posts appeared, recommending Whentotweet as an invaluable Twitter tool. Each post would generate a sudden huge spike in traffic. Sometimes, a blog owner would mail us and ask if we were ready for the sudden increase in traffic this would bring. But Whentotweet was built to scale and even massive sites such as Mashable.com didn't slow it down. The most amazing thing is that we didn't need to write a single extra line of code to handle these massive variations in load. Instead, as soon as we wrapped our head around the tools in the App Engine toolbox we knew that Whentotweet would easily scale. App Engine forced us to think outside the box and avoid the fallacies of traditional hosting that create bottlenecks.

Currently, over 38,000 people have tried Whentotweet and we see from the user feedback that they love it. Give it a try at: www.whentotweet.com

- Niklas Agevik (@niklas_a) and Stefan Ă…lund (@stefan_alund) of Whentotweet.com
Email ThisBlogThis!Share to XShare to Facebook
Posted in | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • SaltStack for Google Compute Engine
    SaltStack was built for the performance, reliability, security and scale delivered by Google Compute Engine . Needless to say we are very e...
  • 2013 Year in review: making Google Compute Engine Generally Available
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • Google Compute Engine and App Engine give Evite freedom to expand services anytime
    Today’s guest blog comes from Dan Mesh, Vice President of Technology at Evite, the pioneer in online invitations and social planning. Evite ...
  • Why We Deployed Zencoder on Google Cloud Platform
    Today's guest post comes from Jon Dahl, VP of Encoding Services at Brightcove. Brightcove’s Zencoder transcodes millions of video and au...
  • International Offline Disk Import now available with new disk-upload centers in Japan, India and Switzerland
    Google Cloud Platform customers have told us that transferring large data sets (in the hundreds of terabytes and beyond) can be expensive a...
  • Google Compute Engine is now Generally Available with expanded OS support, transparent maintenance, and lower prices
    Google Cloud Platform gives developers the flexibility to architect applications with both managed and unmanaged services that run on Googl...
  • 2013 Year in review: bringing App Engine to the PHP community
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • Update on Datastore Auto IDs
    In the upcoming Google App Engine 1.8.1 release, the  Datastore default auto ID policy in production will switch to scattered IDs to impro...
  • 2013 Year in review: giving time back to developers
    2013 was a busy year for Google Cloud Platform. Watch this space: each day, a different Googler who works on Cloud Platform will be sharing ...
  • DataStax Enterprise feels right at home in Google Compute Engine
    Today’s guest post comes from Martin Van Ryswyk, Vice President of Engineering at DataStax. The cloud promises many things for database user...

Categories

  • 1.1.2
  • agile
  • android
  • Announcements
  • api
  • app engine
  • appengine
  • batch
  • bicycle
  • bigquery
  • canoe
  • casestudy
  • cloud
  • Cloud Datastore
  • cloud endpoints
  • cloud sql
  • cloud storage
  • cloud-storage
  • community
  • Compute Engine
  • conferences
  • customer
  • datastore
  • delete
  • developer days
  • developer-insights
  • devfests
  • django
  • email
  • entity group
  • events
  • getting started
  • google
  • googlenew
  • gps
  • green
  • Guest Blog
  • hadoop
  • html5
  • index
  • io2010
  • IO2013
  • java
  • kaazing
  • location
  • mapreduce
  • norex
  • open source
  • partner
  • payment
  • paypal
  • pipeline
  • put
  • python
  • rental
  • research project
  • solutions
  • support
  • sustainability
  • taskqueue
  • technical
  • toolkit
  • twilio
  • video
  • websockets
  • workflows

Blog Archive

  • ►  2013 (143)
    • ►  December (33)
    • ►  November (15)
    • ►  October (17)
    • ►  September (13)
    • ►  August (4)
    • ►  July (15)
    • ►  June (12)
    • ►  May (15)
    • ►  April (4)
    • ►  March (4)
    • ►  February (9)
    • ►  January (2)
  • ►  2012 (43)
    • ►  December (2)
    • ►  November (2)
    • ►  October (8)
    • ►  September (2)
    • ►  August (3)
    • ►  July (4)
    • ►  June (2)
    • ►  May (3)
    • ►  April (4)
    • ►  March (5)
    • ►  February (3)
    • ►  January (5)
  • ▼  2011 (46)
    • ▼  December (3)
      • Simple development of App Engine apps using Cloud ...
      • App Engine 1.6.1 Released
      • Whentotweet.com - Twitter analytics for the masses
    • ►  November (4)
    • ►  October (4)
    • ►  September (5)
    • ►  August (3)
    • ►  July (4)
    • ►  June (3)
    • ►  May (8)
    • ►  April (2)
    • ►  March (5)
    • ►  February (3)
    • ►  January (2)
  • ►  2010 (38)
    • ►  December (2)
    • ►  October (2)
    • ►  September (1)
    • ►  August (5)
    • ►  July (5)
    • ►  June (6)
    • ►  May (3)
    • ►  April (5)
    • ►  March (5)
    • ►  February (2)
    • ►  January (2)
  • ►  2009 (47)
    • ►  December (4)
    • ►  November (3)
    • ►  October (6)
    • ►  September (5)
    • ►  August (3)
    • ►  July (3)
    • ►  June (4)
    • ►  May (3)
    • ►  April (5)
    • ►  March (3)
    • ►  February (7)
    • ►  January (1)
  • ►  2008 (46)
    • ►  December (4)
    • ►  November (3)
    • ►  October (10)
    • ►  September (5)
    • ►  August (6)
    • ►  July (4)
    • ►  June (2)
    • ►  May (5)
    • ►  April (7)
Powered by Blogger.

About Me

Unknown
View my complete profile