Brendan Dawes
Analogue + Digital

Process

Using IBM Watson with Processing

Recently I've been working on a project in which I needed to analyse large amounts of text for sentiment and other such things. In the past I had used a service called Alchemy, but it then got bought by IBM to become part of their Watson services. There's many ways you can use Watson services such as Python, PHP and Node but for use within Processing I've found simple http request using CURL works great. I did however have to piece a few things together to use Watson inside Processing so I thought I'd write a post, just in case it's useful for others.

Sign-up to Watson

The first thing I needed to do was sign up to Watson via the developer site. From there I needed some service credentials for the service I wanted to use which was Natural Language Processing. Once I had these credentials I was ready to use Watson.

Http Request Library

Rune Madsen has written a great library for making http requests (GET and POST) for Processing. After I downloaded this I thought I would be good to go, but there was a little problem in that the library won't allow you to post raw JSON which I needed to interact with Watson.

Because this is Processing, which has a very active community, someone had posted on the forum about using raw JSON with the library. In the same thread a solution was posted, effectively replacing the core part of the library so JSON could now be sent.

I wasn't home and dry just yet though. This new addition didn't allow for sending user credentials as part of the request. So I went into the original http request library source code and found the user credential parts I needed and pasted them into the new code. Now I had a http library (though not strictly a library) which could do raw JSON and user authentication.

Below is the new code that I saved as PostRequest.java in the same folder as my sketch.


// Save as PostRequest.java in the same folder as your sketch
// mostly from the httprequests-for-processing library
// header support added - acd 2015-09-05
// Added user credentials - bjd

import java.io.File;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map.Entry;

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.entity.mime.MultipartEntity;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicHeader;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;

public class PostRequest {
    String url;
    ArrayList<BasicNameValuePair> nameValuePairs;
    HashMap<String, File> nameFilePairs;
    List<Header> headers;

    String content;
    String encoding;
    HttpResponse response;
    String json;

    UsernamePasswordCredentials creds;

    public PostRequest(String url) {
        this(url, "ISO-8859-1");
    }

    public PostRequest(String url, String encoding) {
        this.url = url;
        this.encoding = encoding;
        nameValuePairs = new ArrayList<BasicNameValuePair>();
        nameFilePairs = new HashMap<String, File>();
        headers = new ArrayList<Header>();
    }

    public void addUser(String user, String pwd) 
    {
        creds = new UsernamePasswordCredentials(user, pwd);
    }

    public void addData(String key, String value) {
        BasicNameValuePair nvp = new BasicNameValuePair(key, value);
        nameValuePairs.add(nvp);
    }

    public void addJson(String json) {
        this.json = json;
    }

    public void addFile(String name, File f) {
        nameFilePairs.put(name, f);
    }

    public void addFile(String name, String path) {
        File f = new File(path);
        nameFilePairs.put(name, f);
    }

    public void addHeader(String name, String value) {
        headers.add(new BasicHeader(name, value));
    }

    public void send() {
        try {
            DefaultHttpClient httpClient = new DefaultHttpClient();
            HttpPost httpPost = new HttpPost(url);
            if(creds != null){
                httpPost.addHeader(new BasicScheme().authenticate(creds, httpPost, null));              
            }

            if (nameFilePairs.isEmpty()) {
                httpPost.setEntity(new UrlEncodedFormEntity(nameValuePairs, encoding));
            } else {
                MultipartEntity mentity = new MultipartEntity();    
                Iterator<Entry<String, File>> it = nameFilePairs.entrySet().iterator();
                while (it.hasNext ()) {
                    Entry<String, File> pair =  it.next();
                    String name = (String) pair.getKey();
                    File f = (File) pair.getValue();
                    mentity.addPart(name, new FileBody(f));
                }               
                for (NameValuePair nvp : nameValuePairs) {
                    mentity.addPart(nvp.getName(), new StringBody(nvp.getValue()));
                }
                httpPost.setEntity(mentity);
            }

            // add the headers to the request
            if (!headers.isEmpty()) {
                for (Header header : headers) {
                    httpPost.addHeader(header);
                }
            }

            // add json
            if (json != null) {
                StringEntity params =new StringEntity(json);
                httpPost.addHeader("content-type", "application/x-www-form-urlencoded");
                httpPost.setEntity(params);
            }

            response = httpClient.execute( httpPost );
            HttpEntity   entity   = response.getEntity();
            this.content = EntityUtils.toString(response.getEntity());

            if ( entity != null ) EntityUtils.consume(entity);

            httpClient.getConnectionManager().shutdown();

            // Clear it out for the next time
            nameValuePairs.clear();
            nameFilePairs.clear();
        } catch( Exception e ) { 
            e.printStackTrace();
        }
    }

    /*
     ** Getters
     */
    public String getContent() {
        return this.content;
    }

    public String getHeader(String name) {
        Header header = response.getFirstHeader(name);
        if (header == null) {
            return "";
        } else {
            return header.getValue();
        }
    }
}

Constructing a Watson Request

The documentation on the Waston site shows how to put together a CURL request for the Natural Language service. It was now a matter of recreating such a request in processing.

I decided to split this into two parts: a method to construct a JSON object for any piece of text and then the actual post request.

Here's the getJSONForText method:

JSONObject getJSONForText(String txt){
  JSONObject json = new JSONObject();
  json.setString("text",txt);
  JSONObject empty = new JSONObject();
  JSONObject features = new JSONObject();
  features.setJSONObject("sentiment",empty);
  features.setJSONObject("keywords",empty);
  features.setJSONObject("emotion",empty);
  json.setJSONObject("features",features);
  return json;
}

As well as adding the text that I would like to query, I also add in the features I'd like to get in response, in this case sentiment, emotion and keywords.

This JSON then gets added to the main Watson post request via the addJson method.

Below is a simple Processing example that will query Watson to analyse a piece of text and print the results to the console. USERNAME and PASSWORD will need to be changed to the credentials you've obtained from the natural language service.


import http.requests.*;

void setup(){
  size(640,640);
  String txt = "The Quick Brown Fox Jumped Over the Lazy Dog";
  JSONObject json = getJSONForText(txt);
  PostRequest post = new PostRequest("https://gateway.watsonplatform.net/natural-language-understanding/api/v1/analyze?version=2018-03-19");
  post.addUser(USERNAME,PASSWORD);
  post.addHeader("Content-Type", "application/json");
  post.addJson(json.toString());
  post.send();
  println(post.getContent());

}

void draw(){
}

JSONObject getJSONForText(String txt){
  JSONObject json = new JSONObject();
  json.setString("text",txt);
  JSONObject empty = new JSONObject();
  JSONObject features = new JSONObject();
  features.setJSONObject("sentiment",empty);
  features.setJSONObject("keywords",empty);
  features.setJSONObject("emotion",empty);
  json.setJSONObject("features",features);
  return json;
}