Photostream App Trilogy

I’ve started a new screencast. In the screencast, I’ll show you how to develop a photostream application. The photostream application has 2 parts

1. Allow users to upload images

2. Push the newly uploaded images (a photostream) to connected clients.

The following is a list of open source projects and technologies that I’m using to write this application

    Rather than creating one long screencast, I’ve split it into 3 parts.

The first part – upload service and and Angular client.

The second part – server notification using websocket

The third part – hybrid mobile app to take pictures and uploading them.

Here is the first part


Some points to note about

  • I’m using asynchronous Servlet to handle the file upload. This is to improve performance and to better utilize server resources by releasing the request thread when the image file is being persisted. We can resume the request once the file have been saved
  • Since this is a simple application, I’ve saved the images directly in the document root. You should use a CMS or something like a GridFS to save your images
    Till next time.

Screencast: Part 2 – JavaEE 7, Mongo, REST, Mobile

In the second part screencast, we will take the data news item stored in the Mongo database and expose that as RESTful reesources. When a request arrives it can specify the following 2 parameters in the query string

  • oid – this is the object id of the document/record. This tells the REST resource to start returning the records from the specified oid. If the oid is missing then we will start from the very first record.
  • batchSize – the number of records to return.

To allow the application server to potentially process a large number of records per request, we handle the retrieving and processing of the news items asynchronously. This is very similar in spirit to async Servlets; however unlike async Servlets, in async JAX-RS you need to provide a thread. This is where the concurrency utilities comes in.

Using @Suspended annotation, we get an instance of AsyncResponse. The AsyncResponse object is passed to a Runnable (our task) which is then dispatched to the ManagedScheduleExecutorSerivce. When the Runnable completes, it calls AsyncResponse.resume() to resume the suspended request.



The NetBeans project for part 2 is available here. You can find part 1 of this series here.

Till next time.

Integrating TOTP with Java EE’s Container Managed Security

clip_image002Edited on Monday Febuary 1 2014

After reading the following blog on time-based one time password (TOTP), I wanted to see if I could add an additional layer of security into Java EE’s  security. I installed Google Authenticator on my phone and started experimenting it with Java EE security.

I’ll describe what I did and some of the issues I found. You can get a copy of the source when you reach the end of the blog. Hopefully the journey is as pleasant as the destination 😉

The blog is broken into the following sections

  1. In the first part, I’ll give a brief intro to Google Authenticator/time based onetime password and how it works
  2. Setup JDBC realm for Glassfish application server in preparation for using it in our form based authentication
  3. Create the login and user registration with JSF Faces Flow
  4. Configure container managed security to use TOTP for login

Time-base One Time Password

So lets start with TOTP. We are all familiar with the concept of password. Password are typically static; they do not change over time; so if someone knows your password then your account is compromised.

In most operating system you can configured a password expiry policy where users are forced to change their password at regular intervals eg. 6 months. This password expiry policy gets around the above mention problem every 6 months. But the risk still exist.

TOPT takes the password expiry policy to the extreme by expiring your password at an even faster rate like say every 30 seconds! The problem here is that you cannot change your password every 30 seconds; so to make this work, TOTP uses an algorithm to change the password for us.

Very briefly this is how the TOTP works: you pass to the TOTP password generator algorithm a secret key and the current time; out pops your password; So if you call the TOTP password generator now and then again 30 seconds later (assuming that your window is 30 seconds) using the same secret key, you will get 2 completely different passwords.

To verify a password generated by TOTP algorithm, the TOTP password verification must know your secret key and the time that the TOTP password is generated. Since its a 30 second expiry period in our example, the password verifier will be able to validate the password as long as it is run inside the same 30 second window when the password is generated. All the wonderful math can be found here. TOTP is an IETF standard RFC6238.

Examples of TOTP can be found in online banking like DBS’s OTP where the account holders are issued dongles that implement the TOTP algorithm along with their unique secret keys embedded in the dongles. 

Another way of using TOTP is to augment it with your regular password as in Google’s 2-Step Verification (the focus of this blog). Google’s 2-Step Verification (G2SV) relies Google Authenticator which replaces the dongle in the above example. You have to first create an account. The Authenticator captures your TOTP details of this account by scanning a QR code. The QR code is created from the following 3 pieces of information

  1. A base 32 encoding of a 10 digit hexadecimal number. Eg the bytes from the string 8-polytope (“8-polytope”.getBytes()) will produce the following base 13 encoding: HAWXA33MPF2G64DF. This is the secret key and is unique to each user
  2. An issuer, typically the name of a company or the issuing authority eg. Yoyodyne Propulsion Systems
  3. A label, which I like to call the subject, is the user’s name or some identifier eg. johnwhorfin

All the information have to be encoded in the otpauth format. If we were to encode the values from above, we will get the following


Note that the otpauth string have to be URL safe. We can how use this otpauth to generate a QR using Google’s QR generator. You can find the query string parameters here. So to get the QR code of 350×350 pixel in size, enter the following

See the QR in its full glory here.  Of course the entire process of generating the OTP can be automated.

Clearly the secret key in which the entire TOTP scheme hinges is the weakest link. If we minimize the exposure of the secret key (like keeping it from the user, using HTTPS to generate the QR) then we can potentially make using TOPT reasonably secure. Well get to that a little later.


Glassfish JDBC Realm

Java EE security realms is a place where username, password and groups are stored. When you configure security for your application you need to specify which security realm you application will be associated with. When a user login to your application, it is from this realm that the application will get the user’s credentials, password and group id. See here for more details.

A JDBC realm is a realm where usernames, passwords, etc are stored in a RDBMS. You first create the required tables in a RDBMS; next you create a connection pool on top of it followed by a JDBC resource.

Since I’m using Glassfish and Java DB/Apache Derby, I’ll be creating and configuring JDBC realm based on these. There are lots of blogs and tutorial on creating JDBC realm for Glassfish so I won’t go into it in great detail. I’ll just list out my configuration.

The following are the table schemas, based on Apache Derby, which I’ve copied from this blog. Adapt it to whatever database you’re using.

create table logintable (
   username varchar(128) NOT NULL CONSTRAINT usernamePK primary key,
   password varchar(128) NOT NULL,
   otp_secret varchar(128) NOT NULL

create table grouptable (
   username varchar(128) NOT NULL,
   groupid varchar(128) NOT NULL,
   CONSTRAINT GROUP_PK PRIMARY KEY(username, groupid),
   CONSTRAINT USER_FK FOREIGN KEY(username) REFERENCES logintable(username)

I’ve added an extra column in logintable called otp_secret. This column will be used to store our OTP secret key.

Create a JDBC resource to the database that has the above 2 tables; see here if you do not know how. Assume that the JDBC resource name is jdbc/security. Next define a JDBC realm for Glassfish with the following configuration according to the table schemas

Property Name Property Value
JAAS Context jdbc-realm (note: this is the realm’s name)
JNDI jdbc/security
User Table logintable
User Name Column username
Password Column password
Group Table grouptable
Group Table User Name Column username (note: this is the username column in grouptable)
Group Name Column groupid
Password Encryption Algorithm none (note: enter this value)
Digest Algorithm SHA-256 (note: this is important as you’ll need to know what message digest to use)

Login Flow for TOTP

The next thing that we’re going to do is to define a flow for a user to login or to register themselves. Registration is important here because this is where we generate the TOTP secret key. The following diagram shows the authentication and registration process


  • totp-login – this is the login page and the main entry point into this flow. The page need to have the same name as the flow name to indicate that this is the starting page
  • register – if we are a new user, then we click on a button on totp-login to get us here
  • qrcode – after we have successfully registered, the qrcode page will display the QR code for our TOTP which we have to use Google Authenticator to scan. After scanning, we return to totp-login to login
  • error – the error page

totp_flowI’ll be implementing the process using JSF using the new Faces Flows; so the files are created in a directory called totp-login; this incidentally will also be the name of the flow. 

Let start with the registration (register.xhtm) page. Note all JSF pages are at its bare minimum.

<h:form id="form">
   <h:inputText id="username" value="#{newUser.username}"/>
   <h:inputSecret id="password0" value="#{newUser.password}"/>
   <h:inputSecret id="password1" value="#{newUser.passwordAgain}"/>
   <h:commandButton value="Register Me" action="#{newUser.register()}"/>

And this is the backing bean for register.xhtml

public class NewUser {
   @EJB private UserManager mgr; 
   //JavaBean properties - not shown
   public String register() {
      Map<String, String> result = OneTimePassword.generateQR(username, Authentication.ISSUER);
      mgr.register(username, password, result.get(OneTimePassword.SECRET32));
      qrUrl = result.get(OneTimePassword.OTPAUTH);
      return ("qrcode");

The code is quite simple. When the user clicks on Register Me button, we will call register(). After verify the fields, us use an EJB call UserManager to insert the username, password and the base32 encoded secret key into the JDBC realm. This corresponds to logintable above. It also assigns the user to a group called registered.

Recall that in our JDBC realm configuration, we say that we are using SHA-256 for the password digest; so in effect we don’t save the password but its digest. This hashing operation is performed by the register() method.

Before we register the new user into our database, we have to generate a secret key for it. We use generateQR(); generateQR() takes in 2 parameters, the subject and the issuer. In the example above, for the subject we will be using the user’s login name and the issuer is hardcoded to “Yoyodyne Propulsion Systems”. generateQR() returns a Map which hold a few entries. The name of the keys holds the following values

Key Name Value Description
secret32 Base32 encoded String of a randomly generated 10 byte secret
otpauth otpauth URI
url Full URL, with the otpauth, to the Google site that generates the QR code
secret Stringify 10 byte secret. Note in the demo, we only use the first 2 keys viz. secret32 and otpauth

We assign the otpauth URI to a property and move to the qrcode.xhtml page to display the QR code shown below

<h:outputText value="
   Please scan the QR code on the right using 
   your Google Authenticator App. After that you can 
   proceed to login."/>           
<h:commandButton value="Login" action="totp-login"/>
<h:graphicImage url="#{newUser.qrUrl}"/>     

A digression: I found that graphicImage cannot load images from an external source viz. if I pass it… I’ll get a broken image. Not sure if this is a bug with JSF (Majorra) or its part of the specs. Anyways, this blog pointed out a possible workaround by proxying the image via a Servlet. The otpauth is appended to /qrcode. So the url attribute in graphicImage references /qrcode/the_otpauth instead of… A Servlet is mapped to /qrcode/* to handle the image loading. See QRCodeServlet for more details.

All that remains now is the login page which is shown below

   <h:inputText id="username" value="#{authentication.username}"/>
   <h:inputSecret id="password" value="#{authentication.password}"/>
    <h:inputText id="otp" value="#{authentication.otpCode}"/>
   <h:commandButton value="New User" action="register"/>
   <h:commandButton value="Login" action="#{authentication.login()}"/>

and its corresponding backing bean

public class Authentication implements Serializable {
   //Getters and Setters not shown
   @EJB private UserManager mgr;
   public String login() {
      FacesContext ctx = FacesContext.getCurrentInstance();
      HttpServletRequest req = (HttpServletRequest)ctx.getExternalContext().getRequest();
      if (null != req.getRemoteUser())
         return ("protected/menu");
      if (null == req.getAuthType()) {
         ctx.addMessage(null, new FacesMessage("Not configured for container managed login"));
         return ("error");
      try {
         req.login(username, password);
      } catch (ServletException ex) {
         ctx.addMessage(null, new FacesMessage("Incorrect login"));
         return ("error");
      User user = mgr.getUser(username);
      if (null == user) {
         ctx.addMessage(null, new FacesMessage("Cannot find user."));
         return ("error");
      try {
         if (!OneTimePassword.checkCode(user.getOtpSecret(), otpCode, System.currentTimeMillis())) {
            ctx.addMessage(null, new FacesMessage("Incorrect login"));
            return ("error");
      } catch (InvalidKeyException | NoSuchAlgorithmException | NumberFormatException ex) {
         ctx.addMessage(null, new FacesMessage("Exception: " + ex.getMessage()));
         return ("error");
      return ("success");

The login() method takes a user name and password and will throw a ServletException if the that combination is incorrect. It will also throw ServletException under 2 other circumstances

  • If the application have not been configured for security. We check that using getAuth() method or
  • If we attempt to login a use that have been authenticated. So we check if we have the user name by calling getRemoteUser(). If this is not null, meaning that this request has been authenticated, then we redirect the request into the secure area. More on secure area later.

After successfully login, we perform the 2nd validation by checking the OTP. We pass in the secret key, which is returned when we get the User from getUser(), the OTP password that the user entered and the current time to OneTimePassword.checkCode(). checkCode() method is  this algorithm that I found here. checkCode() verifies the password using a 120 second window viz. if you give it the current time, it’ll take the current, past and future 30 second slot. If we don’t do that then the time on the OTP generator (the mobile phone that runs Google Authenticator) and the time on the server that host the web application cannot be more than 30 seconds apart.

Note that if the TOTP password is incorrect we perform a logout (logout()) before returning; the other method that you can use is HttpSession.invalidate() which performs a logout as well. The difference is that logout() will not destroy the HttpSession. One reason you might not want to invalidate the session is to keep a count of failed attempts and disable the username for 5 minutes for example.

If the OTP is correct, we exit the TOTP authentication flow by returning the success token. I’m using explicit flow definition, so the definition is as follows

<flow-definition id="totp-login">
    <flow-return id="success">

where everything under the /protected  directory requires authentication and authorization before access.


Form Based Authentication with TOTP

We’ll not turn our attention to configuring the web application with the totp-login flow described above using the JDBC realm that we have created. So if you want to secure an application in Java EE, you have to specify 3 things

  1. Authentication scheme – how are we going to collect username and password and to perform the authenticatoin
  2. Authorized role(s) – what are the application defined security roles
  3. Security constraint – what are the resources that needs to be authenticated and who is authorized to access it. The constraint ties authentication and authorization into a neat little package

Authentication Scheme

Let start the discussion by looking at authentication scheme. Java EE supports container managed security. If we’re dealing with securing web application, then common ways to authenticate a user are either basic or form-based (see here for other ways).

When you use basic authentication, you get a dialog box to request for your user name and password. If a correct combination of username and password is presented, then you will be redirected to the protected resource. Basic authentication does not give you control over the login page and the authentication process. Say you want to enter a third field, eg our TOTP field, as part of the the login, there is just no way of doing this.

Enter form based authentication. With form based authentication, you create a login and an error page in the web.xml file. The login page needs to contain j_username and j_password field; these values are POSTed to j_security_check. Form based authentication gives you control over the L&F of your login page but you’re still out of  the loop when it comes to the authentication process. Again it does not allow us to arbitrarily include any additional data into the authentication process. You can do it after you’ve passed security check but not during. Another common gripe with form based authentication is that its not very JSF friendly; you have to fallback to <form> and <input>.

So in Java EE 6, the Servlet API introduced programmatic security. At its heart, it is still form based authentication. However instead of relying of j_security_check to collect username and password, your application is now be responsible for collecting these information and performing the authentication. This opens up a lot of possibilities; you are no longer restricted to just username and password. You can include additional fields; see Authentication.login() and totp-login page above.

The following XML fragment shows form based authentication for our TOTP authentication


Note that in a typical form based login, you have 2 different views for login and for error. The container will redirect the request to the error page if you’ve login incorrectly. In our “semi” container managed security, our application controls this (see the flow above). So you don’t actually need an error page; I therefore set it to index.xhtml.

Another issue is cannot enter a JSF flow with a HTTP GET. In a typical form based login configuration, you provide the actual login page. In this scenario, we cannot have /faces/totp-login; this will not start the flow. What you need to do here is provide a view which have a POST back to start the totp-login flow; in this case index.xhtml looks like this

   <h:commandLink action="totp-login" value="Login"/>

Authorized Roles

Once you’ve setup the authentication mechanism, you now need to define a security role. Our demo application have a single role which is authenticated.


The authenticated role needs to be mapped to users and groups from the jdbc-realm. This mapping is application server specific. Glassfish uses a file called glassfish-web.xml to perform this mapping. The following XML fragment shows how we map authenticated to registered group.


Security Constraints

A security constraint defines what resources are to be protected and what roles are allowed to authorized to access it. Lets say we wish to secure everything under the /protected path. So when we try to access /faces/protected/menu, the application server will require you to be authenticated first; then check if you are authorized with the role to access that resource. In our case our security constraints looks like this


The above constraint says that what we are protecting is everything under the /faces/protected path; only those uses who are authenticated to the authenticated role will be authorized to access it.



The following are some screenshots with a running commentary of what’s happening


This is the login page. Assume that we are a new user, so we click on New User button. This brings us to the following screen


We enter the new user details. Then click on Register Me button. This will call NewUser.register(). A secret key will be generated and the appropriate QR code link is created based on the username, issuer and the secret key. The following page is displaying asking the user johnbigboote to scan that into his Google Authenticator.


After scanning in you should see the following in the Google Authenticator


The number 732803 is the OTP and the ‘pie chart’ to the right is the 30 seconds count down timer. After login you’ll be redirected to the secure area.


You can use this approach to integrate other authentication mechanism like adding CAPTCHA.

I’ve listed a few caveats in the blog. They are mostly related to JSF and not security. So let me just state one here. One of the big advantage of container managed security is that if you are going to access /faces/protected/menu resource and your are not authentication, the application server will redirect you to the login page (if we’re using form based authentication). After you have authenticated and you’re authorized to access that resource, the container will now redirect you to that resources which is /faces/protected/menu in our case.

But using programmatic authentication, you lose this ability. I have to go to a specific view after a successful login. I cannot find a way of redirecting the request back to the original resource. Let me know if you have a standard and portable way of doing this.

The source code can be found here. It’s been an extremely long blog. Didn’t start off that way. Thanks for reading.

Till next time.

Art of Conversation – Part 1

It has been a while since my last blog. A combination of work and reimagining conversation support led to this hiatus. After the last blog, I wanted to do a series on the applicational aspect of Vorpal viz. how do you go about using the various features in Vorpal to implement a custom XMPP service. However I found IQ support wanting; in particular correlating IQ request/response to be really tedious. I’ve also felt, for quite sometime now, that my initial design for conversation support which I blog about here to be extremely simplistic. So I took the opportunity to redesign these.
The good news is that if you are using previous version of Vorpal to develop your application, nothing apparent have change; everything should works as is. If you have any issues, please open an issue or post your problem in the forum. Bad news? Conversation support is quite tricky because it affects the framework greatly so if you find any NPE which wasn’t in earlier releases of Vorpal, please let me know.
Rather than writing a long blog on conversation, I’m going to break this subject into 3 or 4 blogs convering concepts, implicit and explicit conversations and interactions between these.

What are Conversations?

What is the purpose of conversation? Simply put conversation allows a Vorpal application to keep states about an interaction between 2 jabber entity. This is very similar to HTTP session support using HttpSession. In Vorpal states can be kept in ConversationContext, very similiar to HttpSession, and/or in stateful objects marked with @Conversation (think Stateful Session Bean). Every conversation is marked with a unique conversation id.
The default conversation provider gets the conversation id from the following 
  • when there is a <thread> element in <message>. This is part of the XMPP standard. See XEP-0201.
  • or by using implied conversation id support between any 2 Jabber entities when <thread> element is missing. 
All packet falling under the same conversation (viz. having the same conversation id) will be able to access to the same state information stored in ConversationContext or @Conversation message handlers.
The way implied conversation id works is by examining the to or from of a packet and figure out a thread id from there. Vorpal will first look for 
  • conversation between an entity and a chat room (ok an entity as well). It does this by looking for the following pattern ‘@conference.‘ in either the from or to. If it finds it then Vorpal can use the chat rooms JID as a conversation id
  • if the interaction is not with a chat room, then Vorpal will sort the full JID of to and from of a packet and concatenate them giving us a unique conversation id
The advantage of implied conversation id over those explicity marked with <thread> element is that implied conversation supports all 3 types of packets (message, iq and presence) whereas <thread> is only supported in message. The down side of it is that because its implied viz we compute the conversation id using information from the packet, you are really at the mercy of this algorithmn and have no control over the conversation states.

Conversation Types

Vorpal supports 2 types of conversation:
  1. implicit conversation – these are for a request response cycle; they are started automatically started automatically when you send IQ (get/set) packets and are terminated automatically when the corresponding IQ response (result/error) is received or vice versa
  2. explicit conversations – these are conversations that are started by the service by calling Conversation.begin(). Under certain situation explicit conversations are also started automatically; however you must terminate the conversation. We’ll look at this in a future blog (of this series)

Conversation Examples from the Playground

I’ll go into the nitty gritty details of conversation in my next blog. For the impatient, have a look at Eliza in playground. It contains 2 Netbeans project;
  • ElizaService is the external component that implements Eliza. I got the source here. To run it you need to define ‘arkham’ as a subdomain in your XMPP server and the shared secret is ‘ICannotSleep’. Or you can change the deployment settings
  • Arkham is a Java Eliza client
Deploy ElizaService to Glassfish with Jabberocky. To verify that Eliza has been deployed successfully run a discovery service on it. The following screenshot from Gajim shows Eliza service.
There are 2 ways of accessing Eliza. 
  1. The first is to use a standard XMPP client like Gajim. Create a chat room (muc). Then send a direct invitation to eliza@arkham.your_domain. Once Eliza has joined the chat room it will send a greeting to the root. Now you can interact with it by telling Eliza your problem. To end the therapy session, simply type quit or bye
  2. The second is to use Arkham which essentially does the same thing as 1 above. The only difference is that it will initiate a search of for the Eliza service first on the server before connecting. If it cannot find the service, it’ll notify you. Arkham uses client side Vorpal
Eliza shows how to use the client and the server to create a virtual conversation; I’ll be talking about this in greater detail in my next few blogs.
For those who want to check out these new features please download the latest from here. If you have downloaded the package prior to July 29 2012, please download it again. 
Note: I’m trying to get a hang Blogilo after I’ve decided to change from ScribeFire. So you’ll have to excuse me if there are any issues with formatting, etc.
Till next time

Screencast: Are you there?

In this screencast we will show how to use @Presence to allow a XMPP external component to respond to subscription and probe messages. This is a continuation of our series of tutorial on developing CustomerQuery component with Vorpal framework.

The source thaw was developed in the screencast is availble here.

Till next time

Screencast: Developing Discoverable XMPP Components with Vorpal

A screencast explaining how disco#info and disco#items works in XMPP. There is also a ‘live’ coding show how to use @IQ and @Query nnotation.

Comments and feedback are welcomed.

Another Screencast – Stateful Objects with Conversation

I’ve really caught the screencast bug; here is another screencast on conversation support in Vorpal. In the screencast I use Vorpal to develop an Eliza service.

Here is how it works. IM users establish a chat session with eliza@arkham.batcomputer JID (or whatever subdomain you choose to deploy it in). If the in coming messages to Eliza has <thread> element, then Vorpal can use that to establish a conversation.

To begin a ‘therapy session’ with Eliza, send to eliza@arkham.batcomputer a message with start as the message content. Vorpal will create an instance of Eliza and hold that for the duration of the conversation. When you decided to end the session, simply send it a message with quit. Vorpal will clear all the objects in the conversation’s context.

Note: the above screencast is about 16:28 mins. Not sure why Youtube thinks its 49:18 mins

I’ve blogged extensively about conversation support and the programming model in my previous posting call Conversations with Vorpal.

You can get the source code of the demo here.
Get the latest Jabberwocky bundle here.
The Eliza source used in the screencast is available from

Till next time.

%d bloggers like this: