Screencast: Part 1 – JavaEE 7, Mongo, REST, Mobile

This is the first part of a 3 part blog/screencast on creating an end to end application that scrape data of a source and then re exposed these data as RESTful web services.

In the first part, we will look at pulling data into our web application.

In the second part, we will expose these data as RESTful web services

In the third and final part, we will write a mobile client to consume the web services.

The technologies that we will be using will be JavaEE 7 (tutorial), Mongo database and AngularJS/Ionic on the front end.

In this first part, we will be reading RSS feed from BBC Football site (the RSS feed is right at the bottom of the page). The feed entries are saved to Mongo. The following diagram illustrates this

arch_part01 

The feed is read at regular interval. This feature is implemented using JavaEE 7’s ManagedScheduledExecutorService which is a thread pool managed by the application server.

JavaEE application servers supports the concept of connection pooling, in particular database connection pool. However this support only extends to RDBMS. So if you’re using NoSQL databases, then you’re out of luck. One solution is to implement custom resources like this particular example for Glassfish.

Luckily for us, MongoClient, the Mongo’s Java driver has an internal pool (and is thread-safe!). So we can create one instance of MongoClient and get this instance to dish out connection as and when its required. To using this instance as a singleton, we use CDI to wrap it in an ApplicationScoped object. This ApplicationScoped object ensures that only a single instance of MongoClient is created and used throughout the entire application.

The following screencast shows how we create the a JavaEE web application, read RSS feed from BBC Football website and save that into a Mongo database.

The source code can be found here.

Till next time.

Using OAuth For Your JavaEE Login

matrix_spoon_boy In my previous blog I  describe how we could add additional security layer to Java EE’s login process. The application server holds a  set of valid username/password; users are authenticated against this set of credentials.

Managing username/password is hard work; what if you are only interested about the user but not necessarily in maintaining the user. If your security requirement is not that stringent, then I could let somebody that I trust  validate that user. Once the user is validated, I can find out about the user. So you don’t maintain any username/password database. As far as you are concern there is no user.

Enter OAuth; it is a standard at allows an application to access data that a user holds in a 3rd party application. In order to access the data, the user first have to authenticated into that 3rd party application; once we pass the authentication stage, then we are authorized to access the data. The really nice thing about OAuth is that you never have to reveal your username or password to the application.

A use case might be an application would like a user’s valid email; one source of this might be from Google. So by using OAuth, the user can authorize Google to release this piece of information to the application. For more details on OAuth watch this video.

For the remainder of this blog, we will look at

  1. how to use Google’s OAuth2 provider for authentication and authorization; we’ll look at what are the information/resources we can release and how to go about doing that
  2. how to integrate/use Google’s OAuth2 as our authentication mechanism for a web application.

Register and Configure Your Application

To start using Google as our OAuth2 provider, you’ll need first do the following list of things

  1. Create a project or service account (in Google)
  2. Create a new client id
  3. Select the information/resources that your application wishes to use
  4. Optionally, configure the consent screen

Here is how everything hangs together:

  1. An user Fred, attempts to access some restricted resources/page on the JavaEE application. The JavaEE application server determines that Fred need to first login.
  2. The web application redirect Fred to  Google’s login page. Fred proceeds logins to his Google account; if Fred has already login to his account, then he will not need to relogin.
  3. After Fred has successfully login, Google OAuth2 provider will display the consent page. The consent page provides details about the web application and more importantly what resources is it trying to access, eg Fred’s email and Google+ id, his drive, etc.
  4. If Fred consents, Google now redirects the browser back to our web application

Now lets look at these steps in greater detail.

The first thing you’ll need to do is to create a project using Developer Console. This project is our JavaEE’s representation in Google’s universe. When a user gives permission to our JavaEE application to access his email, the user is actually giving permission to this project.

s0

In the example above, my project’s name is Gatekeeper and the project id is gatekeeper-sg1. Go into the project by clicking on the project’s name. On the left hand side, expand ‘API & auth’ menu item and you should see APIs, Credentials and Consent Screen.

To create client id, click on Credentials, followed by clicking on CREATE NEW CLIENT ID.

s1

Decide on the type of application you are developing. In our case its going to be a web application. Next you’ll have to configure 2 pieces of additional information associated with Web Application. The Authorized JavaScript is your application’s address in this case is the application’s domain. Since I’m writing a demo, that will be localhost with the port number. The Authorized redirect URI is the URI that Google will redirect to after the user has successfully login and have given permission for our application to access certain information.

s2

The above shows the newly created client id. We will need the Client ID, Client secret and Redirect URIs later when we write our web application.

So the next thing we need to decide is what do we want to access from a user’s Google account. Click API under APIs & auth. You will now see a list of all the resources available. For our case, we are only interested in basic information, so we’re going to enable Google+ API. Enable all the API that the application will be needing. Note that most of these API have a daily limit (for free use). We’ll look at how to use the Google+ API that we have enabled in the next section

One final, you can customize the consent screen by entering the web application’s url, image, T&C, etc under the Consent screen menu.

Exploring the API

As I’ve mentioned earlier, Google provide lots of resource that we can access. In our case we are interested in a user’s Google+ details. All of these resources are access using REST. Google provides a playground for you to experiment with these API. The playground allows you to

  • Find out what are the resources available eg. under Google+ there are different resources like email, profile, login
  • How to invoke them
  • What is the structure of the returned JSON data
  • What are the OAuth2 scopes (more on scopes later) namespaces

s3 Assume we are going to get basic detail about a user from his Google+, so let see how we do this using playground.

Open up playground. Under Step 1, look for Google+ API v1 (don’t forget to enable it in your application). Under that you will see list of information accessible. Select the information that the application will need to access; what we are saying here is that if a user eg Fred logins to our web application, then Fred will have to provide these information to the web application. The consent page will display these details. So check whatever details you want to access. As a best practice, ask for the minimum amount of information in order for your application to establish a trust relationship with the user.

The URL that you see are the scopes. Make a note of these eg. if the example on the left, I have 3 scopes: https://www.googleapis.com/auth/plus.me, https://www.googleapis.com/auth/userinfo.email and https://www.googleapis.com/auth/userinfo.profile.

Once you’ve selected all the API, click on the Authorize APIs button. This is to authorize playground application (for testing) to access these resources from your Google account.

If you’re using this the first time, then you’ll probably see Playground’s consent screen requesting permission to access to API from Step 1 above. Click on Accept to proceed. If you don’t then you cannot test those APIs.

s4 Once you’ve consented, you’ll be taken to Step 2. Nothing much to do here except to click on Exchange authorization code button which will now take you to Step 3. We are now ready to test the Google+ API. Click on List possible operations button. Depending on what you’ve request in Step1, you will now be presented with a list of URL that you can call.

s5

Select Userinfo (in our case) from the list then click on Send the request.

s6

From the response, you can see the JSON structure of the resource. So by using the playground, you can find out

  1. What is the URI for a particular to get a particular piece of information
  2. What is the HTTP method to use
  3. What is the JSON structure of the result
  4. What are the required scopes (from step 1)

Integrating with JavaEE

We now have everything we need from the OAuth provider, we will now look at how to add this to our web application.

Start by configuring your web application to use form based authentication. You should provide a login page with a button to login via Google.

<form method=”GET” action=”googleplus”>
   <button type=”submit”>Login</button>
</form>

You could also provide the option of using the traditional way of login with j_security_check.

We will now have to write a servlet that maps to googleplus. This servlet will be performing the OAuth2 setup and login. Lets look at the code first

@WebServlet("/googleplus")
public class GooglePlusServlet extends HttpServlet {

   private static final String CLIENT_ID = "client id here";
   private static final String CLIENT_SECRET = "client secret here";

   @Override
   protected void doGet(HttpServletRequest req, HttpServletResponse res)
      throws IOException, ServletException {

      //Configure
      ServiceBuilder builder= new ServiceBuilder();
      OAuthService service = builder.provider(Google2Api.class)
         .apiKey(CLIENT_ID)
         .apiSecret(CLIENT_SECRET)
         .callback("http://localhost:8080/mywebapp/oauth2callback")
         .scope("openid profile email " +
               "https://www.googleapis.com/auth/plus.login " +
               “https://www.googleapis.com/auth/plus.me”) 
         .debug(System.out)
         .build(); //Now build the call

      HttpSession sess = req.getSession();
      sess.setAttribute("oauth2Service", service);

      resp.sendRedirect(service.getAuthorizationUrl(null));
   }
}

GooglePlusServlet is the servlet that will handle the OAuth login.Google has documented the authentication exchange here. I’m however going to using the excellent Scribe library. From the ServiceBuider we lookup Google’s OAuth2 Provider class. Once we have the OAuthService, we need to set the client id, the client secret and the URI callback . All these 3 pieces of information must match the project we created earlier.

Next comes the scope information; scope is use to indicate to Google what resources are we interested in accessing. In other words what is the scope of this request. The scope in this case must always start with the string openid followed by a space delimited scope namespaces. The 2 standard namespaces are profile and email. If we just have those 2 namespace, then we are saying that we are interested in the user’s profile and email. Google supports additional scopes for its services. This is where playground comes in. If you recall, when we were exploring the API, we are also shown the scope. If we want the request to access the user’s G+ profile and circles, then add their scope namespaces to the scope() method as shown above.

Note: Scribe supports may OAuth providers out of the box. However Google is NOT one of them; the reason is found here. This gist adds support for Google. Add the Google2Api class into your project and you’re good to go.

After constructing OAuthService, save that in the session cause we’ll be using it in the next phase of the authentication.

Now its time to forward our request to Google. Get the authorization URL from service.getAuthorizationUrl() and perform a redirect to it. So what is going to happen now is

  • If you are not login to your Google account, you will now be asked to do so.
  • If this is the first time you are using the web application, you will be shown the consent screen. You can now choose to accept or reject. Once you’ve consented to the request, this permission is saved into the user’s account permissions.

Google will now perform a callback to the configured callback URL. To handle the callback, we’re going to need another servlet

@WebServlet(urlPatterns={"/oauth2callback", asyncSupported=true)
public class OAuth2CallbackServlet extends HttpServlet { @Override
   protected void doGet(HttpServletRequest req, HttpServletResponse resp)
      throws IOException, ServletException {

      //Check if the user have rejected
      String error = req.getParameter("error");
      if ((null != error) && ("access_denied".equals(error.trim())) {
         HttpSession sess = req.getSession();
         sess.invalidate();
         resp.sendRedirect(req.getContextPath());
         return;
      }

      //OK the user have consented so lets find out about the user
      AsyncContext ctx = req.startAsync();
      ctx.start(new GetUserInfo(req, resp, asyncCtx));
   }
}

public class GetUserInfo implements Runnable {
   private HttpServletRequest req;
   private HttpServletResponse resp;
   private AsyncContext asyncCtx;
   public GetUserInfo(HttpServletRequest req, HttpServletResponse resp, AsyncContext asyncCtx) {
      this.req = req;
      this.resp = resp;
      this.asyncCtx = asyncCtx;
   }

   @Override
   public void run() { 
      HttpSession sess = req.getSession();
      OAuthService serivce = (OAuthService)sess.getAttribute("oauth2Service");

      //Get the all important authorization code
      String code = req.getParameter("code");
      //Construct the access token
      Token token = service.getAccessToken(null, new Verifier(code));
      //Save the token for the duration of the session
      sess.setAttribute("token", token);

      //Perform a proxy login
      try {
         req.login("fred", "fredfred");
      } catch (ServletException e) {
         //Handle error - should not happen
      }

      //Now do something with it - get the user's G+ profile
      OAuthRequest oReq = new OAuthRequest(Verb.GET,
            "https://www.googleapis.com/oauth2/v2/userinfo");

      service.signRequest(token, oReq);
      Response oResp = oReq.send();

      //Read the result
      JsonReader reader = Json.createReader(new ByteArrayInputStream(
            oResp.getBody().getBytes()));
      JsonObject profile = reader.readObject();
      //Save the user details somewhere or associate it with
      sess.setAttribute("name", profile.getString("name"));
      sess.setAttribute("email", profile.getString("email"));
         ...
      asyncCtx.complete();
   }
}

The callback is routed to this servlet; we check if the user have rejected the request. If not we proceed to get the user’s details. I’m using an async servlet here because we’ll be making request out to Google for the user’s info and we do not want to hold on to the request processing thread.

If the user have consented to the request, then get the code from the query string. This is the most important parameter. Using the code we construct the access token. For every request to Google, we will need the access token to use the token to sign it. Constructing a new token is quite straight forward. You can also refresh an old token by passing in the token as the first parameter in getAccessToken(). Once you have gotten your token, save it in the session.

We are now ready to get the user’s profile. For your experimentations in the playground (step3), you now know which URL to use and what HTTP method to use. Construct that using OAuthRequest class. Sign it with the token, before making the call.

When the response, returns parse it using a Json parser. Since I’m using EE7, I’ll be using the Json API to perform this. Again, from playground, we know the structure of the Json response from Google. You can now save all the relevant information.

One last thing before we end. So we have successfully authenticated with Google using OAuth. But we have not login to the application. To do that, I’ve decided to create a proxy user in the authentication realm of the application server. After you have authenticated with Google, login to this proxy user so that the application server will honour all the security declarations. Just be aware that you cannot use getRemoteUser() or use getPrincipal() to get a user’s detail. A more tightly integrated approach would be to develop a custom realm (Glassfish in this example). However custom realms may not be portable.

One More Time…

There are may details and configurations so before I end lets go through the steps again

  1. Setup Project
    1. Create a project in Google (from Developer Console)
    2. From the project, create a new client id (note the client id, client secret and redirect url
    3. Enable the API that the project is going to access
  2. Find out abut the API 
    1. Go to playground
    2. Select the same API that you have enabled
    3. Note their scope, http method, URL and the response
  3. Integrate with WebApp
    1. Configure for based login
    2. Write a servlet to construct an OAuth request.
    3. Write a servlet to handle to callback
  4. Till next time…

Building Your Own Presentation Server

Trojan_Room_coffee_pot_xcoffee I gave a presentation recently on the topic of mobile enabling enterprise applications. The context was that if you have a Java EE based application tucked away somewhere in your organization how do you make that application mobile and HTML5 friendly.

As I was creating the presentation, it occurred to me that perhaps my presentation should be like a mini demo of sorts to illustrate some of the principles I was sprouting. I really liked the idea of the Keynote Remote app (which is no longer available) or something equivalent like Slideshow Remote.

So I wrote a simple presentation server that will allow me to control the presentation from my mobile phone. Instead of using Bluetooth or WIFI, my remote control will rely on the web. The following diagram shows how the overall architecture of the presentation server

prezo_2

  1. You start a presentation (the left side of the diagram above) by selecting the presentation from a browser. Yes the presentation is done on a browser like Google Presentation or Prezi.
  2. You then use your mobile phone to open the presenter’s console (right side of the diagram) to control the presentation. The console allows you to advance the slides; it also shows you speaker’s note.
  3. Participants can asked question using the comment tool. Again this is a web application. All comments are displayed on the presentation. The yellow box on the top left corner of the presentation is an example of a comment.
  4. After answering the question, the presenter can dismiss the comment by clicking on the Delete button on the presenter’s console

For the impatient, the source is available here.

Starting A Presentation

Now that we have describe the main features of the presentation server, lets see how this whole shebang works. Every presentation must be started/loaded using the following URL

http://myserver:8080/notppt/presentation/presentation.html

The presentation.html is the name of the presentation file found in the document root of a Java EE web application. The presentation file is a HTML presentation developed using Reveal.js. (more later). Presentations must be loaded from the path /presentation/; this is mapped to the following Servlet

@WebServlet("/presentation/*")
public class PresentationServlet extends HttpServlet {

   @EJB PresentationMap map; 

   protected void doGet(HttpServletRequest req, HttpServletResponse resp) {
      //Get the presentation file name
      String presoFile = req.getPathInfo();
      String uuid = ...//Generate a presentation id
      //Load the presentation file, ctx is ServletContext
      InputStream is = ctx.getResourceAsStream(presoFile);
      String baseURI = req.getScheme() + "://"
            + req.getServerName() + ":"
            + req.getServerPort() + ctx.getContextPath();
      //Read it into memory as DOM
      Document doc = Jsoup.parse(is, "UTF", baseURI);
      //Register it
      map.create(uuid, doc);
      //Load the presenation using a redirect
      resp.sendRedirect(url + "?pid=" + uuid);
   }
}

As you can see the PresentationServlet sets up the initial data structure for the presentation. It generates a unique id for it. After generating the id, it loads it into memory as a DOM-like object. The ‘DOM’ is then associated with the generated id and saved in a map which in here is represented very simplistically by a Singleton EJB. We now get the browser to load the presentation by sending the following URL back as a redirect with the presentation id (pid).

http://myserver:8080/notppt/presentation.html?pid=12345

As I was working on this, I found a really nice HTML parser call Jsoup. Jsoup is a Java HTML parser that lets you access the HTML document using CSS selectors, just like jQuery. Bye bye XPath! We use Jsoup to parses our HTML presentation; it returns returns a Document object which we will  use later to extract the presenter’s note. To learn about Jsoup, see the tutorial.

Presentation Client

The presentation is a HTML presentation developed using the Reveal.js framework. See this link on how to develop presentation with Reveal.js. To add speaker’s note into the slide, embed <div> (one for each point) inside an <aside> tag with a data-notes attribute like so

<section>
   <h1>This is my presentation</h1>
   <aside data-notes=””>
      <div>Speaker notes</div>
      <div>One div for each point</div>
   </aside>
</section>

You can look at presentation.html in the source for a complete example. Every presentation loads a jQuery script call presentation.js. The script performs the following 2 things

  1. Initializes the presentation using Reveal’s API. See the documentation here.
  2. Once Reveal have been initialized, it opens a server sent event connection back to the presentation server. The presentation uses this channel to listen to commands (eg. next slide, previous slide) and control the presentation according to the received commands.

The following is a simplified outline of the above 2 steps

$(function() {
   //Reads the presentation id from the URL
   var pid = getParameterByName("pid");
   //When Reveal has completed its initialization
   $(Reveal).on("ready", function() {
      //Create a SSE, only listen for commands of our id
      var sse = new EventSource("slide-event"); 
      $(sse).on(pid, slideControl);
   });
   ...
});
//Callback to execute when we receive commands
function slideControl(result) {
   //Read the data
   var data = JSON.parse(result.originalEvent.data);
   //If it is a movement command, then move the slide
   if (("next" === data.cmd)|| ("prev" === data.cmd))
      Reveal.slide(data.slide); {
      return;
   }
   //Else process other commands
   ...
}

You can see the entire script in web/app/presentation.js in the source.

Before proceeding any further, lets see how every piece of the system communicates with each other

prezo_3

After the presentation opens, it creates a SSE channel back to the presentation server. The presentation will only listen to commands with its presentation id; it does this by listening for specific SSE packets with its presentation id .

Presentation SSE Channel

When the presentation client displays the presentation in also creates a SSE channel back to the server to listen to commands from the presenter’s console. The SSE’s URL is /slide-event. Since JavaEE does not support SSE, I’ll use this SSE library. You can read more about the SSE library from this blog here and here. Our SSE implementation is a follows

@RequestScoped
@ServerSentEvent("/slide-event")
public class SlideNotificationChannel extends ServerSentEventHandler {
   public void publishEvent(final String id, final JsonObject obj) {
      ServerSentEventData data = new ServerSentEventData();
      data.event(id).data(obj.toString());
      //connection is inherited from ServerSentEventHandler
      connection.sendMessage(data);
   }
}

We create a SSE data, fill in the event id which is the presentation id and the command (more details of this in Presentation Control section). The command tells the presentation client to move to the next slide, return to previous slide, etc.

The SSE data that we are publishing is associated with an event name (presentation id here). This ensures that only client listening to that id will get the data. This is the line

$(sse).on(pid, slideControl);

in the presentation client (presentation.js) that registers the pid with the SSE’s event id.

Side note: this is not strictly true in this implementation as all presentation clients are connected to the same SSE channel. All presentation client will receive commands regardless of whether its for them; but because they are only listening to their own pid, they will only respond if the SSE’s event id matches their pid. Ideally the SSE channel name should be partition with the presentation id viz. clients should connect to /slide-event/<pid> instead of /slide-event. The current SSE library is not able to parameterize the SSE channel name.

Presenter Console

Lets look at the presenter console. The purpose of the presenter console is to allow the presenter to control the presentation and also to look at the speaker’s note for the current slide.

When you open the presenter’s console, you have to select which presentation you wish to control from a combo box.

The presenter’s console is build with jQuery and jQuery Mobile. The following code shows how commands are sent to the server

$(document).on("pageinit", "#main", function() {
   ...
   $("#next").on("click", function() {
      $.getJSON("control"), {
         "cmd": "next",
         "value": pid, //presentation id
      }).done(function(result) {
         //Display the speaker's note
         var notes = "";
         for (var i in result)
            notes = "* " + result.notes[i] + "\n";
         $("#notes").val(notes);
      });
      ...
   });
});

When you click on the next button (code shown above), the console will issue a command to the server using jQuery’s $.getJSON() API. The following request is made to the server to move to the next slide for presentation 123456.

http://.../control?cmd=next&value=123456

If the server manage to successfully advance the slide, the server will return the new slide’s speaker’s notes. The notes are then extracted from the result and displayed on the console. The following is an example of the response from the server

{
   "cmd": "next", "slide": 2, "totalSlides": 10,
   "notes": [ "speaker note", "First point", "Second point" ]
}

Presentation Control

Commands from the presenter’s console to the presentation server are intercepted and processed by ControlServlet; if you notice the $.getJSON() from above sends the command to control URL. This URL is mapped to ControlServlet

@WebServlet("/control")
public class ControlServlet extends HttpServlet {
   //Get the presentation map
   @EJB PresentationMap map;

   //Get the SSE channel
   @Inject @ServerSentEventContext("/slide-event")
   ServerSentEventHandlerContext<SlideNotificationChannel> slideEvents;

   @Override
   protected void doGet(HttpServletRequest req, HttpServletResponse resp) {
      //Get the command and the pid from the request
      String cmd = req.getParameter("cmd");
      ActivePresentation preso = map.get(req.getParameter("value"));
      JsonObjectBuilder builder = Json.createObjectBuilder();
      switch (cmd) {
         //Handle the next command 
         case "next":
            builder.add("cmd", "next")
               .add("slide": preso.currentSlide())
               .add("totalSlides": preso.totalSlides())
               .add("notes": preso.get())
            break;

                     …
      }

      //Send command back to the console
      Json json = builder.build();
      resp.setStatus(HttpServletResponse.SC_OK);
      resp.setContentType("application/json");
      try (PrintWriter pw = resp.getWriter()) {
         pw.println(json.toString());
      }

      //Broadcast the command to all presentations
      for (SlideNotificationChannel chan: slideEvents.getHandlers())
            chan.publishEvent(preso.getPid(), json);
      }
      ...
   }

The code for ControlServlet is quite straightforward. We start by first constructing the command object in JSON using the new JSON API from JavaEE 7. The notes attribute is the speaker notes for the current slide. The speaker notes are extracted from under <aside data-notes=””> element using Jsoup. Look at the source to see how this is done. They are returned as a JSON array.  After creating the command object, we send the object back to the presenter’s console indicating that the command have been accepted.

Now we have to inform the presentation client to execute the command, which is move the to next slide in this case. Using injection, we get the handler to all SSE channels listening on /slide-event. We then loop through all the connections and publish the event.

The publish event now goes back to the presentation client; the slideControl function is invoked (see Presentation Client section above)  and this advances the slide.

Trying Out the Demo

The source for the presentation server is available here. All the dependencies can be found under lib directory. The presentation server is written to JavaEE 7. I’ve tested in on Glassfish 4.0. Once you’ve deploy the application, point your browser to http://your_server:your_port/notppt. You will see the landing page.

The first item is to start a Start Presentation. You must have at least 1 presentation started. Click on this (open it in a new window) to start your first presentation.

Now you can select either the Control or the Comment link. The Control will take you to the presenter’s console. Try opening this on your mobile phone browser. Select a presentation to control from the combobox. Comment allows the audience to comment or ask questions on the slide.

It is very easy to take the Control and/or the Comment portion and make that into an app (Android or otherwise) by wrapping the web portion only with Cordova/Phonegap.

The presentation server currently has only 1 presentation. If you’re thinking of using it here are some suggestions.

  • Add an interface to to manage the presentations.
  • Automatically inject presentation.js into every uploaded presentation.
  • Add security (Java EE container managed).
  • Some jQuery Mobile gesture like swipe in the presenter console to control the slides instead of buttons
  • Support for other HTML presentation toolkit in particular  impress.js and deck.js

Similar Tools

It’s worth mentioning that Reveal.js do have a presentation console with speaker’s notes. There is another excellent blog on the same subject. Both of these approaches uses node.js as their server.

I also just found out that slid.es, the people behind Reveal.js, have rolled out something similar. See the following video. The idea is basically the same except that they have done it with more panache. Go check it out.

Till next time.

Using JSONP with JAX-RS

JSONP is a format used to work around the same origin policy. As the name implies it is a modification on the JSON format. Instead of returning a JSON object like so

{ "customerId": 123, "name": "fred", "email": "fred@bedrock.com" }

the JSON is return as a method call to a function like so

foo({ "customerId": 123, "name": "fred", "email": "fred@bedrock.com" })

JSONP is typically used in cross origin AJAX calls using the GET method. You can learn more about JSONP from this Stack Overflow discussion. The JSON-P website talks about why and when it is used.

Handling JSONP

We will examine JSONP by first looking at how we use it from the ever popular jQuery. To make an cross origin AJAX call with JSONP we use $.ajax() like so

$.ajax({
   type: "GET",
   url: "webresources/customer/1",
   dataType: 'jsonp' })
.done(function(result) {
   //Do something with the result
   console.log("----> " + JSON.stringify(result));
}).fail(function() {
   //Handle failure
   ...
});

Lets assume that our AJAX call is going to get a customer record. The HTTP request goes across to the server

GET /JSONPTest/webresources/customer/1?callback=jQuery210012143543711863458_1393227259817 HTTP/1.1
Accept: text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01

and the response

HTTP/1.1 200 OK
Content-Type: application/javascript
jQuery210012143543711863458_1393227259817({"name":"barney","id":1,"email":barney@gmail.com})

Notice that the callback function name is in the query string of the request.

So to write a JAX-RS resource to handle we need to

  • Handle the HTTP GET method
  • Read the callback parameter in the query string. We will use the name of the function to wrap our result
  • Produce the MIME type in the Accept header

The following JAX-RS resource should handle the JSONP request

@RequestScoped
@Path("/customer/{id}")
public class CustomerResource {

   @PersistenceContext EntityManager em;

   @GET
   @Produces("application/javascript")
   public String get(@PathParam("id") Integer id ,
         @QueryParam("callback") String callback) {

      Customer cust = em.find(Customer.class, id); 
      if (null == cust)
         throw WebApplicationException(Response.Status.NOT_FOUND);

      //Convert object to JSON
      JsonObjectBuilder builder = Json.createObjectBuilder();
      ...
      return (callback + "(" + builder.build().toString() + ")");
   }
}

The code is quite straightforward. We inject the callback query string into the handler which we subsequently use to wrap the result.

As you can see, its quite simple to handle JSONP.

JSONP as MessageBodyWriter

JAX-RS has a pair of SPI call MessageBodyReader and MessageBodyWriter that allows anyone to handle marshalling and unmarshalling of objects from HTTP to JAX-RS resources. I wrote a simple MessageBodyWriter , called JSONP4JAXRS, that simplifies the entire process (if you don’t mind adding 12K to to your project).

If you’re using JSONP4JAXRS, then the above CustomerResource can rewritten as follows

@RequestScoped
@Path("/customer/{id}")
public class CustomerResource {

   @PersistenceContext EntityManager em;

   @GET
   @Produces("application/javascript") 
      public String get(@PathParam("id") Integer id ) {

      Customer cust = em.find(Customer.class, id); 
      if (null == cust)
         throw WebApplicationException(Response.Status.NOT_FOUND);

      //Convert object to JSON
      JsonObjectBuilder builder = Json.createObjectBuilder();
      ...
      return (builder.build()); 
   }
}

No more fiddling around with callback parameter. The MessageBodyWriter will automatically pad the JsonObject with the callback.

But wait! There is more!

JSONP4JAXRS will automatically pad any return types according to the following rules:

Return Type Padding (assume fn is the name of the callback function)
JsonObject fn({ … }) where { … } is the JsonObject
JsonArray fn([ … ]) where [ … ] is the JsonArray
Any primitives including their  wrappers fn({“value”: the_value })
Any object fn({“value”: toString_value_of_object})
Iterable  fn([ … ]) The Iterable is converted to a JsonArray and the elements are converted according the the above rule

Don’t forget to wrap your Iterable (eg. List<JsonObject>) in a GenericEntity if you are returning the result via a Response.

The JSONP4JAXRS contains a class call JsonWithPadding that defines the following constants for JSONP MIME types

Constant Name MIME Type
APPLICATION_JAVASCRIPT application/javascript
APPLICATION_JSON application/json
APPLICATION_ECMASCRIPT application/ecmascript
APPLICATION_X_ECMASCRIPT application/x-ecmascript
TEXT_JAVASCRIPT text/javascript

JSONP4JAXRS works with Java EE 7 only. JSONP4JAXRS can be downloaded here. The source and binaries are in the compile NetBeans project file. The binaries is in dist directory.

Till next time.

New Beginnings Again!

exploring-calvin-and-hobbes

 

They say third time is a or should be charm! In this blog, I’ll talk about Vorpal2, a reworked version of Vorpal.

 

Excuses, excuses, excuses

The original Vorpal grew out of an experiment and the framework was implemented on top of AbstractComponent class from Tinder. It was a really easy way for someone like me with very little knowledge of XMPP to learn about XMPP and get the framework up and running. However due to the way AbstractComponent class works, I had to repeat a same piece of code in different places. There is no single choke point for the XMPP packets coming in and leaving the framework (for that I had to modify AbstractComponent which was not very appealing); this make implementing certain features extremely difficult and in some cases impossible.

The second issue was that I made the framework dependent on a custom Glassfish container call Jabberwocky. Whilst Jabberwocky is great in that it provided a very robust base to run the framework, integrated extremely well with Glassfish asadmin command and provided all sort of statistics on a Vorpal application, the down side was that it effectively ties Vorpal to Glassfish. Also Jabberwocky is build with Maven. So whenever there is a new release of Glassfish I have to figure out which dependencies have change, etc.

Finally testing the framework engine’s becomes difficult because of the 2 dependencies above.

So I’ve decided to reimplement Vorpal; the good news for me is that most of the framework is fine. I just have to rewrite how the Vorpal is loaded. I’ve removed Vorpal’s dependence on AbstractComponent and Jabberwocky; the startup is now done in ServletContainerInitializer. Its lightweight and in theory should run on any JavaEE 6 Web Profile compliant container. Glassfish will still be the best supported platform simply because this is where I develop and test Vorpal.

For the rest of this blog, I’ll talk about migrating from Vorpal to Vorpal2, packet filters which is a new feature available only in Vorpal2.

 

The Great Migration

No code change (except for ComponentManager)

I’m happy to say that there are no code changes going from Vorpal to Vorpal2 but if your application uses ComponentManager then you’ll have to rewrite that portion of it.  ComponentManager is an artefact of AbstractComponent. In the past I’ve allowed you to inject ComponentManager into your message handlers. This is no longer true; all the functions in ComponentManager can now be found in ComponentContext.

New JARs

The new Vorpal2 bundle can be found here (vorpal2). As before, the framework is split into 2 JARs

  • vorpal2.jar – the framework
  • jabberwocky_support2.jar – framework supporting files

If you have been using Vorpal, you’ll need to uninstall them. Delete jabberwocky.jar and jabberwocky_support.jar from $GLASSFISH_HOME/glassfish/modules directory.

As before, vorpal2.jar must be bundled with your application in WEB-INF/lib.

There are 2 ways you can install jabberwocky_support2.jar. You can either bundle that with your application in WEB-INF/lib or install it at your application’s CLASSPATH. For Glassfish this will be the $GLASSFISH_HOME/glassfish/modules directory.

At the writing of this blog, I’ve not fully migrated the NetBeans plugin to support Vorpal2 (hence the missing update directory from the download bundle). Hopefully I’ll have the plugins done by the next release. In the mean time, you’ll have to manage the build and package yourself. In NetBeans, add the the JARs from the bundle to your project’s library and remove the previous version of the libraries. Also remember that these JARs, or at the minimum vorpal2.jar, must be in WEB-INF/lib of the final WAR file.

Deployment descriptor

Vorpal uses a deployment file to connect to an XMPP server. The file is called xep-0114.xml. This file is found in WEB-INF directory. If you’re developing Vorpal applications with the NetBeans plugin, the template of this deployment file is in your project directory and its called xep-0114.txmlt.

The deployment file contains the following pieces of information

  • The XMPP domain, port number to connect to
  • The subdomain name we wish to use
  • The class name of the component to start – this is very specific to the frame
  • An option to specify the shared secret when connecting to the XMPP domain
  • An optional list of properties to be passed to the application

The file looks like the following

<subdomain domain=”batcomputer” port=”5275” name=”customer”>
   <component-class name=”com.kenai.jabberwock.framework.core.JabberwockyComponent”/>
   <create-subdomain shared-secret=”AlwaysRight” create=”true” cleanup=”true”/>
   <properties>
      <property name=”vorpal.component.name” value=”Customer service”/>
      <property name=”vorpal.component.description” value=”Customer service”/>
   </properties>
</subdomain>

What you need to do is the following

  • Change the component-class name from com.kenai.jabberwock.framework.core.Jabberwocky to com.kenai.jabberwocky.framework.core.XMPPComponentServer
  • Always specify the shared secret with create-domain. The create and cleanup attributes must be present but their values are ignored.

So the new deployment file xep-0114.xml looks like below

<subdomain domain=”batcomputer” port=”5275” name=”customer”>
   <component-class  name=”com.kenai.jabberwock.framework.core.XMPPComponentServer”/>
   <create-subdomain shared-secret=”AlwaysRight” create=”true” cleanup=”true”/>
   <properties>
      <property name=”vorpal.component.name” value=”Customer service”/>
      <property name=”vorpal.component.description” value=”Customer service”/>
   </properties>
</subdomain>

Recompile your application after you’ve made these changes.

New Features

Packet filtering

Packet filtering is one of the feature that I wanted to implement into Vorpal. But I could not do this easily under the old code base until now.

Packet filter are filters that are applied to incoming and outgoing XMPP packets. To implement a packet filter, first annotate your class with @PacketFilter and either implement RequestFilter or ResponseFilter (or both!). The following is an example of a request filter that filters out packets that are not send from the local domain

@PacketFilter
public class LocalEntityOnlyFilter implements RequestFilter {
    @Inject @Named(PredefinedBindings.DOMAIN) String domain;
    //Inspired by AbstractComponent.sentByLocalEntity(Packet packet)
    @Override
    public Packet incoming(final Packet in
          , ComponentContext context)
{
        final JID jid = in.getFrom();
        if (null != jid)
            return (in);
        String pktDomain = jid.getDomain();
        if (pktDomain.equals(domain)
               || pktDomain.endsWith("." + domain))
            return (in);
        return (null);
    }
}

Firstly we see that our filter is annotated with @PacketFilter and implements RequestFilter. So what this mean is that this filter will only monitor incoming packets viz. packets sent to the Vorpal application.

Filters are managed by CDI. so you can injected values into it. In this case we are injecting the domain name that the subcomponent is connected to. Filters are statesless so you should not use any CDI scopes on it.

In the filter, you can do anything to the packet; you can add stanzas, remove stanzas and change certain values in the packet. When you return a packet, this can be the original packet from the parameter or a totally new packet, Vorpal will take the packet and proceed with finding an appropriate handler. If you return a null, then Vorpal will discard the packet.

A ResponseFilter behaves the same way as a RequestFilter. When you implement ResponseFilter, you have to override the outgoing(Packet, ComponentContext) method which returns a Packet as well. If you return a null, then Vorpal will not send the packet.

If you have more that one request or response filter, Vorpal does not impose any ordering on them. Also if one of the filter rejects a packet, then that packet will not be processed by other filters. For example if you have 3 request filters; the first of these 3 filters rejects the packet, then that packet will not be passed to the other 2 to be processed.

Ordering sent packets

One side effect of this new version is you can now impose some order on how the packets are send out by returning a List in your message handler. The following code snippet illustrates this

@Body(“{body}”)
private List<ResponseContext> handle(@Named(“body”) String body) {
   …
}

handle() returns a List; so Vorpal will send out the packet according to the order of the packets in the list.

Let me know what you think of Vorpal2. If you have any issues, please post them to the forum. The new bundle can be found here.

Using Server Sent Events with Vorpal

bridge

I came across the following 2 blogs (here and here) on a CDI framework for server sent event (SSE) several months ago. I downloaded the source then tried it with Vorpal running on Glassfish 3.1.x. The result was a really long exception stack.

4 months later.

After publishing part 4 of on conversations, I decided to see if I have any luck with the SSE framework. I wasn’t really expecting it to work. Well it did. What is even more surprising is that it worked with Vorpal. 

What I’m going to do in this blog is explain how the SSE framework works and more importantly how you can use it with Vorpal. Its really easy.

Setup the push event stream

Firstly create a notification handler. The notification handler will maintain the connection between the browser and the web application. The following is a simple notification handler

@ServerSentEvents(“/notifications”)

public class NotificationHandler extends ServerSentEventHandler {

   public void sendMessage(String msg) {

      try {

         connection.sendMessage(msg);

      } catch (IOException ex) {

      }

   }

A few points to note: the notification handler must extend ServerSentEventHandler; this class is from the framework. You also have to annotate the class with @ServerSentEvents and specify the URL of the event stream. In this case it is notifications. More on this later.

Now you need to write a HTML/JSP/JSF/whatever page to invoke this event stream. Here is a snippet of the HTML page along with a snippet of Javascript that loads loads the event stream

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:h="http://java.sun.com/jsf/html">
    <h:head>
        <title>XMPP with SSE</title>
        <script src="resources/app.js"></script>
    </h:head>
    <h:body>
        <h1>Using Vorpal with SSE</h1>       
    </h:body>
</html>

The Javascript that creates the event stream is in app.js. Here is the code

var url = ‘http://’ + document.location.host

      + “/xmppsse/notifications”;

eventSource = new EventSource(url);

eventSource.onmessage = {

   document.body.innerHTML += msg + ’<br>’;

};

The important thing here is to get the event stream URL right. If our example, this is made up of the application context of our web application (in blue) and the value that you’ve specified in @ServerSentEvent annotation (in green).

 

Pushing data to the browser

We now hook the event handler to a Vorpal application. I did a screencast on developing a customer query service which allows you to use a Jabber client to query a database, which you can find here and the source here.

Lets extend this so that whenever someone performs a query, the result is also pushed to the browser using SSE.

@Message

public class MessageHandler {

   @EJB CustomerManager mgr;

   @Inject @ServerSentEventContext(“/notifications”)

        ServerSentEventHandlerContext<NotificationHandler> ctx;

   @Body(“{id}”)

   private String handle(@Named(“id”) int id) {

      //Assume we find every record and no NPE

      String record = mgr.find(id).toString();

      //Loop through all handlers

      for (NotificationHandler nh: ctx.getHandlers())

         try {

            nh.sendMessage(record);

         } catch (Exception ex) {

            nh.close();

         }

      return (record);

   }

After we got the record from CustomerManager, which uses JPA, we loop through all the registered handlers for an event stream channel and pushes the record to them. You have to get the correct handler to push to.

In our example, we are pushing to all browser registered on notifications. So we inject an instance of ServerSentEventHandlerContext with qualifying it with @ServerSentEventContext(“notifications”).

Example code

You can download the example source from here. I’ve bundled the SSE library into the download which is a NetBeans project.

Use the following steps to setup, deploy and run the example

  1. Configure an external component on your Jabber server. The example uses customer for subdomain and AlwaysRight for shared secret (Caps!). You can always change these settings. I use Openfire for the Jabber server
  2. Make sure that you’ve installed your Glassfish with Jabberwocky container.
  3. Deploy the project to Glassfish either from NetBeans or using asadmin command. It’s probably easier if you deploy if from NetBeans
  4. Open your browser and point to http://yourserver:port/xmppsse. I’ve tested with Firefox and Chrome, latest versions.
  5. Open your Jabber client, open a chat with query@customer.your_jabber_server. Send 1
  6. You should now see your reply display in your Jabber client and in your browser.

The source code for the SSE can be found here if you wish to look at it.

Part of the code (the app.js) is taken from Bhakti Mehta’s blog. Big thanks.

Till next time.

Listing Supported PubSub Features

When you perform a ServiceDiscoveryManager.getItems() on a pubsub JID (see Making Sense of Services), the server returns 2 types of information:

  1. the identity (<identity>) of the JID as defined in the Service Discovery Identities. You use the information here to determine what exactly is the JID. If you are looking for a pubsub service, then the category should contain pubsub value.
  2. the feature (<feature>) that is supported by the identity. In the case of a pubsub, these can be features like ability to create/delete nodes, persist items, etc. A list of the pubsub features can be found here.

This is typically how the packet will look like

<iq id=”SOKMA-5″ to=”fred@localhost/Smack” from=”pubsub.localhost” type=”result”>
<query xmlns=”http://jabber.org/protocol/disco#info”&gt;
<identity category=”pubsub” name=”Publish-Subscribe service” type=”service”/>
<feature var=”http://jabber.org/protocol/pubsub”/&gt;
<feature var=”http://jabber.org/protocol/pubsub#collections”/&gt;
<feature var=”http://jabber.org/protocol/pubsub#config-node”/&gt;
<feature var=”http://jabber.org/protocol/pubsub#create-and-configure”/&gt;
<feature var=”http://jabber.org/protocol/pubsub#create-nodes”/&gt;


</query>
</iq>

Although Smack allows you to retrieve the list of identities by the DiscoveryInfo.getIdentities() method, there seems to be no provision for retrieving a list of supported features associated with an identity (see thread). While there is a getFeatures() method, this however is not public. The only way seem to be explicitly asking if a feature is supported (DiscoverInfo.containsFeature(String)); for example if you want to check if a pubsub service supports user created nodes, you do the following

DiscoverInfo info = mgr.discoverInfo(“pubsub_jid@xmpp_server.org”);
if (info.containsFeature(“http://jabber.org/protocol/pubsub#create-nodes&#8221;)) {
//do something

So to get a list of the features here is a utility method

public static List<DiscoverInfo.Feature> extractFeature(String s) throws Exception {
   List<DiscoverInfo.Feature> list = new LinkedList<DiscoverInfo.Feature>();
   QName var = new QName(“var”);
   ByteArrayInputStream bis = new ByteArrayInputStream(s.getBytes());
   XMLEventReader xer = XMLInputFactory.newFactory().createXMLEventReader(bis);
   out:
   while (xer.hasNext()) {
      XMLEvent evt = xer.nextEvent();
      switch (evt.getEventType()) {
  
      case XMLStreamConstants.START_ELEMENT:
            StartElement st = evt.asStartElement();
            if (“feature”.equals(st.getName().getLocalPart()))
               list.add(new DiscoverInfo.Feature(
            st.getAttributeByName(var).getValue()));
            break;
         case XMLStreamConstants.END_DOCUMENT:
            break out;
         default:
      } //switch
   } // while
   xer.close();
   bis.close();
   return (list);
}

and the following code snippet shows how to use this utility method

DiscoverInfo info = mgr.discoverInfo(“pubsub_jid@xmpp_server.org”);
List<DiscoverInfo.Feature> featureList = extractFeature(info.getChildElementXML());