How Safari Books Online uses Google BigQuery for business intelligence

Today’s guest post is from Daniel Peter, Senior Programmer Analyst at Safari Books Online. In this post, Daniel describes how his company uses Google BigQuery to power their dashboards and analytics.



Safari Books Online is a subscription service for individuals and organizations to access a growing library of over 30,000 technology & business books and videos. Our customers browse and search the library from web browsers and mobile devices, generating powerful usage data which we can use to improve our service and increase profitability. We wanted to quickly and easily build dashboards, improve the effectiveness of our sales teams and enable ad-hoc queries to answer specific business questions. With billions of records, we found it challenging to get the answers to our questions fast enough with our existing MySQL databases.



Looking for alternative solutions to built our dashboards and enable interactive ad-hoc querying, we played with several technologies, including Hadoop. In the end, we decided to use Google BigQuery.



Here’s how we pipe data into BigQuery:







Our data starts in our CDN and server logs, gets packaged up into compressed files, and runs through our ETL server before finishing in BigQuery.



Here’s one of the dashboards we build using the data:







You can see that with the help of BigQuery, we can easily categorize our books. This dashboard shows popular books by desktop and mobile, and with BigQuery, we are able to run quick queries to dive into other usage patterns as well.



BigQuery has been very valuable for our company, and we’re just scratching the surface of what is possible.



Check out the article for more details on how we manage our import jobs, transform our data, build our dashboards, detect abuse and improve our sales team's effectiveness.



Contributed by Daniel Peter, Senior Progammer Analyst, Safari Books Online



-Posted by Ryan Boyd, Developer Advocate

Read More

Multi-Channel Chat with Twilio and Google App Engine

Today’s post comes from Kevin Whinnery, Developer Evangelist at Twilio. In this post, Kevin describes how to build a multi-channel chat application using Google App Engine and Twilio. You can follow Kevin on Twitter at @kevinwhinnery or on Google+.







Google App Engine enables developers to focus on their application’s logic by providing a scalable infrastructure and high-level APIs for persistence, file management, and other common web app needs. XMPP and Channels are among these APIs, making it ridiculously easy to write awesome real-time communications apps in the browser.



Today, we’re going to break down an example application (view it live, source code) that integrates these two App Engine services (plus SMS messaging from Twilio) in a group chat application that connects users via SMS, XMPP, and browser-based chat clients.



We won’t go through every line of code, but at a high level, this application is about receiving inbound messages and sending outbound messages. Let’s see how we do this via SMS, XMPP, and Channels.

Twilio SMS

Sending SMS text messages with the Twilio API requires signing up for a Twilio account. Once you’ve signed up for an account, you can use your account SID and auth token to make authenticated requests against the Twilio REST API. You could just use App Engine’s built-in URL fetch service to interact with the Twilio API, but our official helper library for Java makes authenticating requests and serializing data much easier, providing a POJO interface to Twilio resources and functionality. We’ll be using the Twilio helper in this example. If you’re looking for App Engine specific reference examples, our friends at Google included this reference documentation in their doc site.



In our chat application, all outbound communication and message dispatching is handled by the MultichannelChatManager class. In this application, we add subscribers to the chat room to an in-memory set. When it’s time to send out a message, we iterate over the members of this set and send out messages to all subscribers. We send out messages to SMS subscribers using the Twilio helper on line #56:

TwilioRestClient client = new TwilioRestClient("ACCOUNT_SID", "AUTH_TOKEN");

Map params = new HashMap();
params.put("Body", messageBody);
params.put("To", sub);
params.put("From", "+16122948105");

SmsFactory messageFactory = client.getAccount().getSmsFactory();

try {
Sms message = messageFactory.create(params);
System.out.println(message.getSid());
} catch (TwilioRestException e) {
e.printStackTrace();
}

To receive inbound communication, you will need to purchase a Twilio phone number or use the one given to you when you signed up for a Twilio account. You can configure this phone number to send an HTTP POST to a URL that you choose when an SMS message is received (this callback pattern is called a webhook). In this sample application, we have a Java servlet with a web.xml file configured to accept inbound SMS. In your Twilio number configuration, you would enter https://yourappname.appspot.com/sms, as below:



In the actual servlet, we handle inbound SMS messages first by looking for a “STOP” command, which will indicate that this user no longer wants to receive text messages from the app. Then, we confirm that the user is subscribed (by looking for their telephone number). Finally, we send out a message using our MultichannelChatManager class.



When Twilio sends your app the details of an SMS message with an HTTP request, it expects your app to respond with an XML format called TwiML. TwiML is a simple set of fewer than 20 XML tags that tells Twilio how to respond to inbound communication. The output of our SMS servlet is an XML (TwiML) document, which will send an SMS back to a user if they unsubscribe:

public class TwilioSmsServlet extends HttpServlet {
    // Handle Incoming SMS
 public void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException {
  try {
   TwiMLResponse twiml = new TwiMLResponse();
 
   // parse the body, looking for a command
   String smsBody = request.getParameter("Body");
   String smsFrom = request.getParameter("From");
 
   // Unsubscribe, if requested
   if (smsBody.startsWith("STOP")) {
    MultichannelChatManager.removeSub(smsFrom);
    com.twilio.sdk.verbs.Sms bye = new com.twilio.sdk.verbs.Sms("You have been unsubscribed.  Thank You!");
    twiml.append(bye);
   } else {
    // If they aren't subscribed, subscribe them
    if (!MultichannelChatManager.isSubscribed(smsFrom)) {
     MultichannelChatManager.addSub(smsFrom);
    }
    MultichannelChatManager.sendMessage(smsBody, "sms");
   }
 
   response.setContentType("application/xml");
   response.getWriter().print(twiml.toXML());
 
  } catch (Exception e) {
   e.printStackTrace();
   System.out.println(e);
  }
 }
}

App Engine XMPP Integration

App Engine provides a simple API for sending and receiving XMPP chat messages. Our chat application can receive new messages over XMPP and send them back out to all subscribed clients, similar to how our app behaves for SMS.



App Engine applications have an XMPP username associated with them by default, which takes the form of “appname@appspot.com”. The former part of the username is your unique App Engine app ID and the latter is the appspot domain that your app runs on. To send a message via XMPP to our chat app we need to send a chat message to “twiliosandbox@appspot.com” from a chat client that supports XMPP. If you used the desktop chat client Adium for Google Talk, the interaction might look something like this:



For our application to receive inbound XMPP messages, we need to configure an inbound message handler servlet in our web.xml configuration file. This webhook callback design is the same type of event mechanism used by Twilio to deliver SMS messages to our application. In this servlet, we receive an inbound POST request with information about an inbound chat message:

public class XMPPReceiverServlet extends HttpServlet {
    // Handle Incoming XMPP Chat messages
    public void doPost(HttpServletRequest req, HttpServletResponse res) throws IOException {
     XMPPService xmpp = XMPPServiceFactory.getXMPPService();
     Message msg = xmpp.parseMessage(req);
        
        // The "JID" is the unique ID of this chat client, which we use to subscribe
     JID fromJid = msg.getFromJid();
     String body = msg.getBody();
    
     // Unsubscribe, if requested
     if (body.startsWith("STOP")) {
      MultichannelChatManager.removeSub(fromJid.getId());
     } else {
      // If they aren't subscribed, subscribe them
      if (!MultichannelChatManager.isSubscribed(fromJid.getId())) {
       MultichannelChatManager.addSub(fromJid.getId());
      }
      MultichannelChatManager.sendMessage(body, "xmpp");
     }
    }
}

To send outbound messages, we use the App Engine platform APIs to send an outbound message to a specific JID, which uniquely identifies a connected XMPP client:

JID jid = new JID(sub);
Message msg = new MessageBuilder().withRecipientJids(jid).withBody(messageBody).build();
XMPPService xmpp = XMPPServiceFactory.getXMPPService();
xmpp.sendMessage(msg);

Channel API

The Channel API allows server-side push to connected clients in an App Engine application. In our chat application, we will utilize this API to push new chat messages to browser-based clients.



In order for our server to push chat messages to a browser, the client needs to be issued an ID by our server. We configure a servlet to handle issuing these IDs (and to handle incoming chat messages created by the browser in JavaScript) in web.xml. The servlet generates a unique ID for a connected client, based on the current system time:

//Generate a client token for the GAE Channel API
public void doGet(HttpServletRequest req, HttpServletResponse res) throws IOException {
    ChannelService channelService = ChannelServiceFactory.getChannelService();
 
    //The token must be based on some unique identifier for the client - I am using the current time
    //for demo purposes...
    String clientId = String.valueOf(System.currentTimeMillis());
    String token = channelService.createChannel(clientId);
 
    //Subscribe this client
    MultichannelChatManager.addSub(clientId);
 
    //Reply with the token
    res.setContentType("text/plain");
    res.getWriter().print(token);
}

In the browser, we get an ID for the current user via XHR. First, we include the Channel client JavaScript library by requesting a special URL on the App Engine server. Then we use jQuery to issue a GET request to our server to obtain a client ID:

//Get a client token to use with the channel API
$.ajax('/chat',{
    method:'GET',
    dataType:'text',
    success: function(token) {
        console.log(token);
        var channel = new goog.appengine.Channel(token);
        var socket = channel.open();
        
        //Assign our handler function to the open socket
        socket.onmessage = onMessage;
    }
});

When we get our client ID, we use that to configure the App Engine channel service in the browser for data pushed from the server. Data pushed from the server is handled in a callback function, which updates the textarea on the page. When the user enters a chat message in the browser, we issue a POST request to our ChatServlet, which uses the MultichannelChatManager class to publish a message to all connected clients. This is where we use the channel API to push data to connected web browsers:

ChannelService channelService = ChannelServiceFactory.getChannelService();
channelService.sendMessage(new ChannelMessage(sub,messageBody));

Wrapping Up

In this walkthrough, we explored three messaging APIs that work nicely on App Engine: Twilio SMS, XMPP, and Channels. Our example used Java, but all three APIs will work with Python and Go as well (Twilio has a helper library you might use for Python also).



Using platforms like Twilio and App Engine, developers can create communications applications, which previously would have required expert knowledge and infrastructure to build, in a fraction of the time. I hope you’ll be able to use these APIs to engage with your users wherever they happen to be.



Application source code is available on GitHub here.



- Contributed by Kevin Whinnery, Developer Evangelist, Twilio

Read More

Google App Engine: Hello World using Push-to-Deploy

Ever wished you could deploy to Google App Engine with the same standard tools you use to version your code? Now you can. With the Push-to-Deploy feature you can get your code onto App Engine using git and bypassing the SDK. Here is a quick example on how to clone a sample app from an existing public repository and deploy it as your own App Engine application.



Prerequisites: If you don’t have the git tool installed, get it here.



Setting up your Push-to-Deploy repository:



1. Go to the Google Cloud Console and create a project (this also creates an App Engine app) or use an existing one



2. In the App Engine Admin Console for your app (App Engine, select app, then click Application Settings on left nav bar), enable Source Push-to-deploy



3. Obtain and copy both the auth token by clicking on the link, and the repo URL that is displayed in the box:



4. In order to avoid having to type the password on each deployment, you can store the credentials locally. For this, locate your netrc file:



On Windows

Make sure there is a file named _netrc in your home directory. Also check that the HOME environment variable exists and that it points to your home directory:

$ setx HOME %USERPROFILE%

On Mac or Linux

Make sure there is a file named .netrc in your home directory.



5. Set the contents of the netrc file:

Use the auth token from step 3 (shown as <auth-token> below) and the email address from your Google account. The file should contain one line that looks like this:

machine code.google.com login <email-address> password <auth-token>

Creating/cloning an app and deploying:



6. Clone an App Engine project (example from Github)



Get to the OS prompt (Terminal on Mac, cmd on Windows), navigate to a directory where you would like to store the sample app’s source code (e.g. cd ~/Desktop), and clone the App Engine Guestbook app repository:

$ git clone https://github.com/GoogleCloudPlatform/appengine-helloworld-python.git

7. Edit the app.yaml file with the new application id

The previous step downloaded the source code for the sample Hello World app. Navigate to that folder.

$ cd appengine-helloworld-python

Open the app.yaml file and change the “application” field to the project name or app id from step 1



8. Initialize local git repo, and set up remote repo (<repo-url> obtained in step 3). In the same directory that contains the source code, and on the terminal prompt type:

$ git remote add appengine <repo-url>

The steps above initialize your local repo, add all the files to it, commit them, and set up an alias to the remote repo we will be pushing to



9. Deploy your app:

$ git push appengine master

And that’s it! The sample app is ready to serve at http://<your-app-id>.appspot.com.



Using this feature simplifies the deployment process by allowing you to use the standard git tool instead of our SDK. We hope you find this functionality useful and look forward to your comments and feedback.



-Posted by Sachin Kotwani, Product Manager

Read More

SendGrid gives App Engine developers a simple way of sending transactional email

Today’s guest post is from Adam DuVander, Developer Communications Director at SendGrid. SendGrid is a cloud-based email service that delivers email on behalf of companies to increase deliverability and improve customer communications. Integration with new or existing email systems is done via SMTP or through a REST API. In this post, Adam shows how to integrate SendGrid into Google App Engine.



Whether you’re developing apps for the web or mobile environments, you need an effective way to communicate with your customers. Building and maintaining your own SMTP infrastructure can be resource intensive and costly. SendGrid eliminates the cost and complexity of maintaining your own email infrastructure so you can focus on developing the next extraordinary app.



Google App Engine developers can easily integrate SendGrid into their applications. In the example below, I’ll show you how to use our Python library. Java and PHP developers, we have you covered with libraries, too. Any App Engine developer can sign up for SendGrid and send 25,000 emails per month for free, so get an account and follow along in the code editor of your choice.



First, copy the SendGrid Python library into your project by placing the files in a sendgrid sub-directory. When you import this library into your app you'll be able to create SendGrid instances and send mail with simple commands.



At the top of your app's .py file, import the Sendgrid library:

from sendgrid import Sendgrid
from sendgrid import Message

Now, from within your app, you can send email with the following few lines:

# make a secure connection to SendGrid
s = sendgrid.Sendgrid('', '', secure=True)
# make a message object
message = sendgrid.Message("from@mydomain.com", "message subject", "plaintext message body", 
"<strong>HTML message body</strong>")
# add a recipient
message.add_to("someone@example.com", "John Doe")
# use the Web API to send your message
s.web.send(message)

In addition to working hard to get your email to an inbox, SendGrid also provides a lot data about your emails. For example, our Event API can tell you when an email is delivered, opened, clicked or bounced, among several other events. The same way that Google App Engine is a platform that makes hosting and scaling apps easy, SendGrid simplifies your interaction with email.



Sign up for SendGrid to claim your 25,000 emails per month for free and combine the power of email and Google App Engine.



Contributed by Adam DuVander, Developer Communications Director, SendGrid



-Posted by Boyar Naito, Partner Development Manager

Read More

Create dynamic web projects in Eclipse

The latest release of the Google Plugin for Eclipse supports the creation of Dynamic Web Projects for Google App Engine. Applications created in this manner fully leverage Eclipse’s Web Tools Platform (WTP), which makes it easier to create and structure Java EE web applications and allows Google App Engine developers to benefit from advanced tooling the Eclipse Ecosystem offers for the Enterprise space.







Here are a few features enabled by the new, WTP-enabled, Plugin:

• WTP/Maven integration, enables complex projects to take advantage of the Maven headless build system and the developer-friendly Eclipse tooling.


• WTP makes it much easier for App Engine projects to use the Eclipse Database and Dali JPA tooling to do either Google Cloud Datastore or Google Cloud SQL development.


• WTP Enterprise Archive (EAR) support allows for the aggregation of multiple App Engine modules as part of an overall App Engine Application as a WTP EAR project.


• Existing Eclipse WTP projects targeting another Application Server (e.g. Tomcat, Jetty, or GlassFish, etc) can be ported to App Engine as the runtime server

To get started, head to the documentation or simply create a new Dynamic Web Project after updating to the latest version of the Google Plugin for Eclipse.



-Posted by Rajeev Dayal, Software Engineer

Read More

New in Google Cloud Storage: auto-delete, regional buckets and faster uploads

We’ve launched new features in Google Cloud Storage that make it easier to manage objects, and faster to access and upload data. With a tiny bit of upfront configuration, you can take advantage of these improvements with no changes to your application code — and we know that one thing better than improving your app is improving your app transparently!



Today we’re announcing:



Object Lifecycle Management - Configure auto-deletion policies for your objects

Regional Buckets - Granular location specifications to keep your data near your computation

gsutil - automatic parallel composite uploads - Faster uploads with gsutil



Object Lifecycle Management

Object Lifecycle Management allows you to define policies that allow Cloud Storage to automatically delete objects based on certain conditions. For example, you could configure a bucket so objects older than 365 days are deleted, or only keep the 3 most recent versions of objects in a versioned bucket. Once you have configured Lifecycle Management, the expected expiration time will be added to object metadata when possible, and all operations are logged in the access log.



Object Lifecycle Management can be used with Object Versioning to limit the number of older versions of your objects that are retained. This can help keep your apps cost-efficient while maintaining a level of protection against accidental data loss due to user application bugs or manual user errors.



Regional Buckets

Regional Buckets allow you to co-locate your Durable Reduced Availability data in the same region as your Google Compute Engine instances. Since Cloud Storage buckets and Compute Engine instances within a region share the same network fabric, this can reduce latency and increase bandwidth to your virtual machines, and may be particularly appropriate for data-intensive computations. You can still specify the less-granular United States or European datacenter locations if you'd like your data spread over multiple regions, which may be a better fit for content distribution use cases.



gsutil - Automatic Parallel Composite Uploads

Gsutil version 3.34 now automatically uploads large objects in parallel for higher throughput. Achieving maximum TCP throughput on most networks requires multiple connections, and this makes it easy and automatic. The support is built using Composite Objects. For details about temporary objects and a few caveats, see the Parallel Composite Uploads documentation. To get started, simply use 'gsutil cp' as usual. Large files are automatically uploaded in parallel.



We think there’s a little something here for everyone: If you’re managing temporary or versioned objects, running compute jobs over Cloud Storage data, or using gsutil to upload data, you’ll want to take advantage of these features right away. We hope you enjoy them!



-Posted by Brian Dorsey, Developer Programs Engineer

Read More

Dedicated memcache preview now available on Google App Engine

Yesterday we announced that dedicated memcache is in Preview. Now you can purchase in-memory data caching capacity exclusively for your application, cache more data and drive up cache hit rates. With higher cache hit rates, dedicated memcache can also reduce Datastore costs and make your application faster than ever. Dedicated memcache has been one of the top 100 features requested by App Engine customers.



With the 1.8.2 release, App Engine now has two classes of memcache service. Dedicated memcache is available in addition to the shared memcache service that has been offered on App Engine for years. No code changes are required when moving to dedicated memcache. Compared to the shared memcache service, dedicated memcache provides developers control over the cache space and performance available to an app.



Bobby Murphy at Snapchat, the rapidly growing mobile photo sharing application, said “We love dedicated memcache and it's already made a big impact on our business. Besides being able to reduce our costs substantially, our hit rates are up to the high 80s.”



By default, applications will continue to use shared memcache, and it will continue to be free. Starting today, billing enabled applications can select dedicated memcache on the App Engine admin console’s application settings page. With dedicated memcache, applications purchase a fixed capacity of RAM and operations-per-second just for that application. This gives developers the ability to plan for the needs of their applications. At this time, there is one class of dedicated memcache available:

Price	12 cents per GB per hour
Self Service Capacity	1 to 20 GB
Performance	Up to 10,000 operations per second per GB for items < 1KB

If you need more than 20GB, please contact us at cloud-accounts-team@google.com.







In addition to dedicated memcache, with the 1.8.1 release we added memcache operations per second monitoring for all memcache classes to the App Engine dashboard.

Stay tuned for more improvements to the memcache service over the coming months. As always, we’re interested in hearing where you’d like us to take caching on App Engine via the comments and the App Engine feature request tracker.



- Posted by Logan Henriquez, Product Manager

Read More