Analyzing User Sentiment in Realtime with Lexalytics Semantria

The Semantria API from Lexalytics allows you to categorize and extract text, and analyze it for sentiment and emotion. The Lexalytics block lets you process realtime data streams while they are in-motion, enabling you analyze and categorize text and run functions based on the result. For example, you can take user submissions and categorize them based on topic or content subject, crunch survey or poll results for keywords, or analyze a social media stream for user sentiment.

Sentiment Analysis with Semantria

In this article, we dive into a simple example of how to enable textual sentiment analysis with Lexalytics Semantria in a robust 1500-line PubNub JavaScript block (almost all of which is the Lexalytics JS Client including crypto support) and 85 lines of AngularJS code.

Sentiment analysis services can be useful in a variety situations, such as customer service, in combination with text translation services, marketing research, health applications and any application where you want the system to have a rough idea of how users feel. As voice and language-based techniques gain popularity, user behavior and expectations are shifting from web-based to voice-based user experiences (including text analysis) in many everyday settings.

So, what exactly is this “sentiment analysis” we speak of? In this case, Sentiment Analysis refers to taking a piece of text and trying to assign a numeric score (where positive numbers are positive sentiment, negative numbers are negative sentiment, and near-zero numbers are neutral sentiment). Sentiment has a number of issues, including incomplete context, sarcasm, slang, international languages and issues with domain-specific text analysis.

As we prepare to explore our sample AngularJS web application with sentiment analysis features, let’s check out the underlying Lexalytics Semantria API.

Lexalytics Semantria API

Automated sentiment analysis services are quite challenging to build and train on your own; they require substantial effort and engineering resources to maintain across a diverse array of application domains and user languages (not to mention immense compute resources and training sets!). In the meantime, the Lexalytics Semantria API makes it easy to enable your applications with straightforward text sentiment analysis.

Looking closer at the API, sentiment analysis is just the beginning. There are a lot of API methods available for things like entity detection, categorization, lexical chaining and more. It really is a powerful tool for distilling meaning from text. In this article, we’ll keep it simple and just implement a basic thumbs-up, thumbs-down, neutral evaluation for user-generated fragments of text.

Since you’re reading this at PubNub, we’ll presume you have a realtime application use case in mind, such as ride-hailing, messaging, IoT or other. In the sections below, we’ll dive into the sentiment analysis use case, saving other web service use cases for the future.

Obtaining your PubNub Developer Keys

The first things you’ll need before you can create a realtime application with PubNub are your publish and subscribe keys. Just in case you haven’t already, you can create an account, get your keys and be ready to use PubNub. Once you do that, the publish and subscribe keys look like UUIDs and start with “pub-c-” and “sub-c-” prefixes respectively. Keep those handy – you’ll need to plug them in when initializing the PubNub object in your HTML5 app below.

About the PubNub JavaScript SDK

PubNub plays together really well with JavaScript because the PubNub JavaScript SDK is extremely robust and has been battle-tested over the years across a huge number of mobile and backend installations. The SDK is currently on its 4th major release, which features a number of improvements such as isomorphic JavaScript, new network components, unified message/presence/status notifiers, and much more.

NOTE: for compatibility with the PubNub AngularJS SDK, our UI code will use the PubNub JavaScript v3 API syntax. We expect the AngularJS API to be v4-compatible soon. In the meantime, please stay alert when jumping between different versions of JS code!

Getting Started with Lexalytics Semantria API

The next thing you’ll need to get started with cognitive services is a Lexalytics account to take advantage of the Semantria API.

• Step 1: go to the Lexalytics Semantria signup form and sign up for a free trial.

• Step 2: make note of the API credentials (key and secret) sent to the registration email address.

Setting Up the BLOCK

With PubNub BLOCKS, it’s easy to create code to run in the network. Here’s how to make it happen:

• Step 1: go to the application instance on the PubNub Admin Portal.

• Step 2: create a new block.

• Step 3: paste in the block code from the next section and update the credentials with the Semantria credentials from the previous steps above.

• Step 4: Start the block, and test it using the “publish message” button and payload on the left-hand side of the screen. (Note: output is sent to a different channel, so you may want to have an additional browser tab open with the PubNub debug console to see messages on that channel).

That’s all it takes to create your serverless code running in the cloud!

Diving into the Code – the block

You’ll want to grab the ~1500 lines of BLOCK JavaScript and save them to a file, say, pubnub_lexalytics_sentiment.js. It’s available as a Gist on GitHub for your convenience.

We’ll skip most of the lines (since they’re boilerplate crypto and API implementation), and focus on the important parts. In this case, you should tune in around line 20.

First up, we define the output channel name. You’ll want this to match the channel the UI subscribes to.

//  PubNub
var publishChannel = "semoutput";

Next, we define the Semantria API credentials (which look like UUIDs). Make sure these are correct and active!

//  Semantria
var semantriaConsumerKey = "YOUR_KEY",         //  Put your Semantira Consumer Key here.  You may signup for a free demo at:  https://www.lexalytics.com/demo
    semantriaConsumerSecret = "YOUR_SECRET";   //  Put your Semantira Consumer Secret here.  You may signup for a free demo at:  https://www.lexalytics.com/demo
var semantriaAppName = "PubNub";

The really interesting code starts around line 1532. The overall process looks like:

• Initialize the Semantria JS API.
• Queue the docs array of strings from the incoming PubNub message into the API.
• Retrieve the analyzed documents, then publish them to the output channel.
• And that’s all! (For more details, please consult the publishResponses function).

//  Initialize Semantria
var semantria = InitSemantria(semantriaConsumerKey, semantriaConsumerSecret, semantriaAppName);
//  Submit docs to Semantria
QueueDocsInSemantria(request.message.docs)
//  Get the Semantria responses
.then(function(){
    return retrieveDocs();
})
//  Publish the responses
.then(function(responses){
    return publishResponses(responses);
})
//  Cleanup
.then(function(){
  console.log("Done.");
})
//  Catch Errors
.catch(function(err){
    console.log("Error: " + err);
});
return request.ok(); // Return a promise when you're done

All in all, even though the API has a lot of JavaScript, it doesn’t take a lot of custom code to add text sentiment analysis to your application. We like that!

OK, that said, let’s move on to the UI!

Diving into the Code – the User Interface

You’ll want to grab these 85 lines of HTML & JavaScript and save them to a file, called pubnub_lexalytics_sentiment_ui.html.

The first thing you should do after saving the code is replace two values in the JavaScript:

• YOUR_PUB_KEY: with the PubNub publish key mentioned above.
• YOUR_SUB_KEY: with the PubNub subscribe key mentioned above.

If you don’t, the UI will not be able to communicate with anything and probably clutter your console log with entirely too many errors.

For your convenience, this code is also available as a Gist on GitHub, and a Codepen as well.

Dependencies

First up, we have the JavaScript code & CSS dependencies of our application.

<!doctype html>
<html>
<head>
  <script src="https://cdn.pubnub.com/pubnub-3.15.1.min.js"></script>
  <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.6/angular.min.js"></script>
  <script src="https://cdn.pubnub.com/sdk/pubnub-angular/pubnub-angular-3.2.1.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
  <link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.2/css/bootstrap.min.css" />
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css" />
</head>
<body>

For folks who have done front-end implementation with AngularJS before, these should be the usual suspects:

• PubNub JavaScript client: to connect to our data stream integration channel.
• AngularJS: were you expecting a niftier front-end framework? Impossible!
• PubNub Angular JavaScript client: provides PubNub services in AngularJS.
• Underscore.js: we could avoid using Underscore.JS, but then our code would be less awesome.

In addition, we bring in 2 CSS features:

• Bootstrap: in this app, we use it just for vanilla UI presentation.
• Font-Awesome: we love Font Awesome because it lets us use truetype font characters instead of image-based icons. Pretty sweet!

Overall, we were pretty pleased that we could build a nifty UI with so few dependencies. And with that… on to the UI!

The User Interface

Here’s what we intend the UI to look like:

The UI is pretty straightforward – everything is inside a div tag that is managed by a single controller that we’ll set up in the AngularJS code.

<div class="container" ng-app="PubNubAngularApp" ng-controller="MyTextCtrl">
<pre>
NOTE: make sure to update the PubNub keys below with your keys,
and ensure that the sentiment analysis BLOCK is configured properly!
</pre>
<h3>MyText Sentiment Analysis</h3>

We provide a simple text input for a message to send to the PubNub channel as well as a button to perform the publish() action.

<input ng-model="toSend" />
<input type="button" ng-click="publish()" value="Send!" />

Our UI consists of a simple list of messages. We iterate over the messages in the controller scope using a trusty ng-repeat. Each message includes the original text as well as the sentiment analysis emoticon and numeric score (if applicable). The emote() function returns a Font Awesome icon, and the to_trusted() function blesses the string so that AngularJS will display it as HTML.

<ul>
  <li ng-repeat="message in messages track by $index">
    text: {{message.source_text}}
    <br />
    sentiment: <span ng-bind-html="to_trusted(emote(message.sentiment_polarity))"></span> (score: {{message.sentiment_score}})
  </li>
</ul>
</div>

And that’s it – a functioning realtime UI in just a handful of code (thanks, AngularJS)!

The AngularJS Code

Right on! Now we’re ready to dive into the AngularJS code. It’s not a ton of JavaScript, so this should hopefully be pretty straightforward.

The first lines we encounter set up our application (with a necessary dependency on the PubNub AngularJS service) and a single controller (which we dub MyTextCtrl). Both of these values correspond to the ng-app and ng-controller attributes from the preceding UI code. Note: in this application, we also use the AngularJS $sce service to produce sanitized HTML.

<script>
angular.module('PubNubAngularApp', ["pubnub.angular.service"])
.controller('MyTextCtrl', function($rootScope, $scope, $sce, Pubnub) {

Next up, we initialize a bunch of values. First is an array of message objects which starts out empty. After that, we set up the pubChannel and subChannel as the channel names where we will send and receive realtime structured data messages respectively.

NOTE: make sure this matches the channel specified by your BLOCK configuration and the BLOCK itself!

  $scope.messages     = [];
  $scope.pubChannel   = 'lexalytics-channel';
  $scope.subChannel   = 'semoutput';

We initialize the Pubnub object with our PubNub publish and subscribe keys mentioned above and set a scope variable to make sure the initialization only occurs once. NOTE: this uses the v3 API syntax.

  if (!$rootScope.initialized) {
    Pubnub.init({
      publish_key: 'YOUR_PUB_KEY',
      subscribe_key: 'YOUR_SUB_KEY',
      ssl:true
    });
    $rootScope.initialized = true;
  }

The next thing we’ll need is a realtime message callback called msgCallback; it takes care of all the realtime messages we need to handle from PubNub. In our case, we have only one scenario – an incoming message containing text fragments with sentiment analysis. The concat() operation should be in a $scope.$apply() call so that AngularJS gets the idea that a change came in asynchronously.

  var msgCallback = function(payload) {
    $scope.$apply(function() {
      $scope.messages = payload.concat($scope.messages);
    });
  };

The publish() function takes the contents of the text input, publishes it as a structured data object to the PubNub channel, and resets the text box to empty. The funny math in the id attribute is just to generate a unique ID – you could just as easily plug in a generated UUID there if you like.

  $scope.publish = function() {
    Pubnub.publish({
      channel: $scope.pubChannel,
      message: {docs:[{text:$scope.toSend,id:parseInt(Math.random(5000000) * 10000000).toString()}]}
    });
    $scope.toSend = "";
  };

We also create an emote() function that translates a message with sentiment analysis decoration into an emoticon.

$scope.emote = function(type) {
  if (type === 'positive') {
    return '<i class="fa fa-smile-o" aria-hidden="true"></i>';
  } else if (type === 'negative') {
    return '<i class="fa fa-frown-o" aria-hidden="true"></i>';
  } else {
    return '<i class="fa fa-meh-o" aria-hidden="true"></i>';
  }
};

We need a sanitize function (we’ll call it to_trusted()) so that AngularJS will trust the HTML we’re sending to the UI.

$scope.to_trusted = function(html_code) {
  return $sce.trustAsHtml(html_code);
};

Finally, in the main body of the controller, we subscribe() to the message channel (using the JavaScript v3 API syntax) and bind the events to the callback function we just created.

Pubnub.subscribe({ channel: [$scope.subChannel], message: msgCallback });

We mustn’t forget to close out the HTML tags accordingly.

});
</script>
</body>
</html>

Not too shabby for about eighty lines of HTML & JavaScript!

Additional Features

There are a couple other endpoints worth mentioning in the Semantria API.

You can find detailed API documentation Clicking here.

• Entity recognition : detection of proper nouns and disambiguating pronouns.
• Categorization : analyzes text fragments to classify them into “buckets.”
• Lexical Chaining : key text extraction, for example, to create summaries.

All in all, we found it pretty easy to get started with Sentiment Analysis using the API, and we look forward to using more of the deeper analysis features!

Conclusion

Thank you so much for joining us in the Sentiment Analysis article of our BLOCKS and web services series! Hopefully, it’s been a useful experience learning about language-enabled technologies. In future articles, we’ll dive deeper into additional web service APIs and use cases for other nifty services in realtime web applications.

Stay tuned, and please reach out anytime if you feel especially inspired or need any help!