Ad-hoc token per form implementation

As stated, the solution is simple, just include a hidden field in every form, store the same value in the session and compare the two when the form is submitted.

Although working, this fix has some disadvantages.

1. First, security is not turned-on by default. Every developer needs to be aware of the concept of CSRF and may not forget it when adding new forms. This also applies to developers that have to maintain the website in the future. They may not even know CSRF in the first place and just remove ‘that silly token’ from the form.
2. Second, it’s probably not the most elegant solution from a Separation of Concerns perspective. Now, every form needs to be aware of security and you need to perform a check in your side controller logic.

So let’s elaborate on some more mature solutions.

What are the variables?

Like always in computer science, there is no single, one-size-fits-all, perfect solution. It always depends on the context. Well, let’s give it a bit of context!

• Token freshness
  How often is a new token generated? In other words, what’s the time-to-live of a token?
• Token quality
  How random should the token be? What algorithm should you use?
• Token storage
  You’ll need to store the token in two places to compare them. First, the HTML where your forms and links reside. And somewhere else…
• Back buttons and bookmarking
  How to deal with users pressing the back button and seeing a cached web page which they submit? How to deal with bookmarks? Also, you can have Single Sign On, paving the way for deep linking into your system without re-authentication.
• Ajax
  When using Ajax, many server side requests may occur before a full page refresh happens (if any). What to do with token management?
• Punishment
  What to do with hack attempts? Do you log an error and short-circuit the server side processing? Or do you log the user off?
• Logging
  How do you log the event? What kind of information do you include in the logging?

As you can see, there are many variables and all of them may impact your implementation. Most of them even influences each other. In the next section, I’ll discuss all of them in isolation.

Token freshness

This is a major one. A choice needs to be made here. Do you generate one token when the user logs in or do you re-generate a new token each time an action is made?

While re-generating a new token with every request may be very secure, it’s not the most practical solution in reality. In practice, it doesn’t even add that much security, because the Same Origin Policy takes care of the integrity of the token. I also wouldn’t worry about brute force attacks for the token, since the user will probably be logged off long before the attacker guesses right. But when we throw back buttons, Ajax, double-clicking users, etc. into the mix, changing the token during a session has serious disadvantages.

So, unless you have really really really weak token (which will never happen unless you pick an integer between 0 and 1), I would generate the token once (@login) and reuse that value.

Token quality

First of all, we’re not talking about high end encryption stuff here. The only use of the token is turning the URL into a “random” URL. So simple tokens will do.

Important to note is that the attacker will probably never see any token, unless he logs into the system (the token is only used for secure pages). So we need something between an incremental counter and some heavy duty 1024 bit encryption.

I would say, keep it simple. Just use a Random generator or a simple AES key, if it makes your security department happy. When using something like AES, remember to encode it using Base64. It only needs to be one way encryption, since you can compare the encrypted values.

This is a simple AES token generator which also encodes the token using Base64 so it can be used in the HTML.

import java.util.Random;
 
import javax.crypto.Cipher;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
 
import org.apache.commons.codec.binary.Base64;
 
public class IntegrityTokenGenerator {
 
    public static void main(String[] args) {
        IntegrityTokenGenerator integrityTokenGenerator = new IntegrityTokenGenerator();
        System.out.println(integrityTokenGenerator.generateToken());
        System.out.println(integrityTokenGenerator.generateToken());
        System.out.println(integrityTokenGenerator.generateToken());
    }
 
    private static final String ENC_TYPE = "AES";
 
    private final SecretKey key;
    private final Cipher cipher;
    private final Random random;
 
    public IntegrityTokenGenerator() {
        try {
            KeyGenerator keyGen = KeyGenerator.getInstance(ENC_TYPE);
            key = keyGen.generateKey();
 
            cipher = Cipher.getInstance(ENC_TYPE);
            cipher.init(Cipher.ENCRYPT_MODE, key);
 
            random = new Random();
        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException("Error generating integrity token");
        }
    }
 
    public String generateToken() {
        try {
            String tokenString = "" + System.nanoTime() + "" + random.nextInt();
            byte[] tokenBytes = tokenString.getBytes("UTF-8");
            byte[] encoded = cipher.doFinal(tokenBytes);
            return new String(Base64.encodeBase64(encoded));
        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException("Error generating integrity token");
        }
    }
}

I used AES in this example, but you can use anything, like Blowfish. Performance is not an issue, because of the small token length. I’m talking about less than milliseconds. Also, encryption strength is not very important, because you only need a simple token. Remember to base64 encode binary tokens into plain text.

Also be sure to use a random value each time when you generate a token. This way, the token sequence becomes less predictable. To make it a bit more secure, include some personal data in the token, like a customer ID. The combination of data make the key even less likely to guess. Don’t put any confidential data in the token. It doesn’t add any security and only increases the risk of information leakage.

And make sure not to throw too specific exceptions. Just log them and throw a general exception. Details may never end up in the wrong hands!

One more note about the snippet. A big advantage of this approach is that you don’t have any key management. Keys may leak and key management puts another burden on the system administrators. This implementation generates a key once (when the class is instantiated) and reuses it. Java takes care of the details.

Token storage

There are many ways to keep track of the current token for the currently logged in user, but the HttpSession and Cookies are the most common. Cookies decrease the amount of state your server must manage, but for some reason, developers think cookies are unsafe.

When used for storing CSRF prevention tokens, cookies can be safe. Why? Because an attacker cannot see the contents of the cookies. It’s the browser that appends the cookies to an HTTP request, but the attacker cannot see them. A mechanism like HttpOnly makes cookies more secure, because it prevents scripts to access them. But storing security tokens on the client is inherently unsafe and is probably the prime reason not to use cookies to store the tokens. You CAN do it, but you must be very careful AND rely on the browser to do the right thing. For example, Firefox ignores HttpOnly, causing cookies to be quite unsafe.

But why not store the token in an HttpSession attribute? Generate it once, put it in the session and then compare this token with the submitted token. It has a little bit of overhead, but that will probably be nothing compared with the rest of the session data.

I prefer the HttpSession over cookies by a mile length. With session attributes, I don’t have to think about token confidentiality and with cookies I do. Easy choice.

Back buttons and bookmarking

For some reason, clients want us to build systems in web browsers that are not specifically designed for such a purpose. But it’s a fact of life, so we need to deal with it. Browsers have back/forward buttons, refresh buttons and bookmarks and allow users to navigate freely by using the address bar. If you want your users to be able to use these features, here are some notes.

Only check the CSRF prevention token on state changing actions. Users often don’t bookmark actions, but they bookmark pages. When you only check the CSRF token when the form is submitted, the user can use a bookmark to navigate to the form. The server then renders a token into the form. The user submits the form and it works. You just need to give the server a chance to render a form before the user triggers an action. Using post-redirect-get makes it even more robust, because it prevents users to bookmark the POST request.

Also, don’t re-generate tokens with each request. When you want to support back buttons, you’ll have to be aware of the possibility that the rendered page may be a cached (stale) page which contains an outdated token. So, use the same token for the duration of the session.

You might also want to consider storing the token per user in a persistent store, so that when the same user navigates to the site (after a time of inactivity, maybe days or weeks) he gets the same token, thus fixing possible bookmarking problems.

If you implement your own home brewn CSRF prevention mechanism you can redirect the user to a proper page when the integrity token is absent or invalid. This way, the user doesn’t see some strange error but a proper page instead.

Ajax

When securing your website, you also need to secure your Ajax actions. After all, an Ajax request is just the same (from the server perspective) as another request. So you need to include CSRF tokens in the Ajax requests as well.

Luckily, with most JavaScript libraries, this is easy, since they provide a generic abstraction for Ajax requests and often provide a way to append generic parameters to a request.

The implementation is really easy, just render the CSRF token to a script variable and append it to all requests, like shown below.

var globalIntegrityToken = '<%=(String)session.getAttribute("integrityToken")%>';

Yeah I know that scriptlets are ugly, but I don’t care in this example. You can use EL or whatever you like.

Appending the token to the URL is easy, for example using JQuery:

$.ajax({
  type: 'POST',
  url: '/changeEmail.do',
  data: {
    newEmail: $('emailaddress').val(),
    integrityToken: globalIntegrityToken
  }
});

Again, reusing the same token for a longer period of time makes your job a lot easier. Otherwise, if you re-generate the token per request, you need a way to update all tokens that exist in the DOM when the request is completed. This has performance impact on the client and makes the code a lot more complex. You’ll have to make sure you update all tokens, which include:

• The global variable, which is easy to update in a generic way
• All hidden fields that contain a token. If you can easily find the fields, this should also be easy
• Any tokens in querystrings. This may be a bigger challenge and you may end up using regular expressions to parse and replace the querystrings

So, when you reuse the same token for the duration of the session (or longer) Ajax will probably not be difficult.

Punishment

What do you do when you see someone fiddling with the CSRF prevention token? First of all, don’t proceed with the action the user invokes. But do you log the user off?

If you can, I would suggest making a switch to turn the logoff mechanism on and off. Turning it off makes it easier to test, but you need to turn it on in production. If you’re afraid that you (or the sysadmin) forgets to turn the mechanism on, you can skip this switch.

Just be sure your prevention mechanism doesn’t give false positives, because users will be very annoyed if they are logged off often.

Logging the user off has some other benefit. If there is a hacker messing around with your website, users will get logged off very often and complaints will come in often at your help desk. This way, you have the opportunity to detect hackers. If you let the user proceed working without notification, you will never know that something happened. Unless your sysadmins proactively scan the log files, but unfortunately I don’t see sysadmins do these kind of things very often…

Logging

How do you log the event? I would suggest creating a separate security log file so that sysadmins can react to hack attempts. It’s also useful as a proof in a lawsuit. I would log as much as possible. You can think about:

• Date/time of course and also the timezone.
• The entire HTTP request, with cookies, referrer, user agent, etc.
• The source IP address. However it may be spoofed or a proxy, at least you can see that there is a difference with the “real” user’s IP.
• The invoked action, along with it’s parameters.
• Any server side state, specific to your system.

It might be a good idea to also log the first two items when the user authenticates so you can compare the two values.

To summarize

In most cases, you’ll want to use a static CSRF prevention token for the duration of the session. It makes Ajax implementations easier and gives browser features like back buttons and bookmarks a more natural behavior. When you do, implementing a working CSRF prevention mechanism may not be a big issue anymore.

The other variables don’t have this much impact, but be sure to pick a good combination, because some variable combinations may harm each other.

Cross site request forgery (CSRF) is a type of attack, made to a web application by another web application, running in another tab or window in the same browser on the same computer. The server cannot see the difference between legitimate and malicious requests, since all HTTP requests look the same. Thus, the server (obviously) assumes a legitimate request. This gives the hacker the opportunity to impersonate the user and execute commands without the user knowing it. Or more abstract, CSRF may violate integrity.

The attack is caused by three things.

1. First, the fact that most websites completely rely on a session cookie to represent the user.
2. Second, because browsers allow websites to invoke certain cross domain requests.
3. And third, the browser appends all cookies that belong to a domain to every request to that domain.

Example

For example, you have a social networking site where you can add friends to your network. Let’s say everyone has a link on his/her personal page that can be used to add that person to your “friends” list.

The link may look like this:

<a href="/addfriend.do?userid=12345">Add friend</a>

The application uses the HttpSession (or JAAS Subject or whatever. The attack is not specific to Java) to determine the currently logged in user and its userId. The userId of the new friend is taken from the querystring. Both are combined and the relation is created. This mechanism trusts the integrity of the session, which is not a bad thing on itself.

So far so good…

Attacking the example website

But the above example has one major flaw. It assumes that a session safely represents the user and bases security checks on this assumption. For some reason, people think that the session is “safe”, just because it exists on the server. But there is one big issue with sessions. The server determines which session belongs to a certain user by a sessionId, which is stored in a cookie in the user’s browser (URL rewriting is also possible, but not common). From the server point of view, this cookie IS the user. So, when someone accesses the cookie value, this person is automatically authenticated. But that’s not the issue here. That would be a case of session hijacking (or session fixation). You can migitate the risk of session hijacking using common security mechanisms, like SSL, and by ensuring that only proper input is accepted. But we’re talking about CSRF here.

CSRF is possible, because all cookies that belong to a certain domain are sent with each request to that domain, regardless from what domain the request is invoked, even when that web page is running in a different tab or window. SSL doesn’t help you here. If a call is made using HTTPS, the cookies are still appended to the request and a valid HTTPS request is executed.

Note: The details of cookie sharing differ per browser. For example, in Internet Explorer, when you use CTRL+N to open a new window, cookies (and thus sessions) are shared between the window and its “parent”, but when a new process is created (e.g. from the Start Menu or Quick Launch), the two act as two separate browsers and cookies are not shared. This distinction is often not visible to (or known by) the end user. But there are more issues, like the annoying IE8-I-share-cookies-between-tabs-”feature”.

What can an attacker do with this knowledge? Consider the example above. If the social networking site permits HTML content on your personal page, the attacker can first create an account and then put the following HTML snippet on his personal page.

<img src="/addfriend.do?userid=[ATTACKER_USER_ID]" />

The browser sees this image tag and tries to load the image, effectively invoking the following HTTP request to the specified URL. It doesn’t matter that the URL doesn’t point to a real image, the processing on the server is being done anyway. The request is invoked from the victim’s browser, so the victim’s cookies are appended to the request.

Now, everyone who visits the attacker’s personal page becomes a friend of the attacker.

With a bit more effort, the attacker can do almost anything with the entire social networking site. SAMY is an example of an attack on MySpace which used XSS and CSRF techniques to do its thing.

Almost all major websites (like GMail) have had CSRF leaks and many major websites still have.

How about POST?

The previous example uses an image to trigger an HTTP GET request. So you might think that POST is safer. Well, opposed to what many people claim, POST is not much safer than GET (from a networking perspective they are the same). Where we use an image to trigger a GET request, we can just as easily use a (hidden) form to trigger a POST request, as shown in the next snippet:

<form id="myForm" action="/addFriend.do" method="post">
  <input type="hidden" name="userid" value="[ATTACKER_USER_ID]" />
</form>
<script type="text/javascript">
  document.getElementById("myForm").submit();
</script>

Cross Site?

The risk of CSRF may be small or big. In the above example, the attack concentrated on the same website.

This is a case of a “Stored CSRF Vulnerability” (from www.isecpartners.com/documents/XSRF_Paper.pdf), which means that the attacker uses a security hole in the application itself to invoke the HTTP methods. The attacker uses techniques like XSS as an “enabler” for their CSRF attack. This kind of attack is completely the responsibility of the website. They just need to prevent the user to enter any HTML.

But CSRF also works cross domain. In that case, we talk about “Reflected CSRF” (also from www.isecpartners.com/documents/XSRF_Paper.pdf). In this case, your website itself may be perfectly safe (meaning it doesn’t accept HTML snippets like in the example) but an attacker may still invoke HTTP requests to your site using a different site, like a forum, blog, email, IM or whatever. Effectively this means your website can be the victim of the leak in another website. Luckily, this method often fails, since the user must be logged in on the targeted site and visit the malicious other site at the same time. How often this happens depends on the type of website. I’m sure most people don’t visit many dangerous websites when doing electronic banking, but it’s not uncommon to do this when logged in on webmail, especially since these kind of websites often provide features like Single Sign On or long sessions.

But doesn’t the browser prevent this behavior?!?!?! Nope, as I’ll indicate in the following section.

Same Origin Policy

First implemented in Netscape Navigator 2.0, the Same Origin Policy is one of the main security mechanisms in modern browsers. Basically, it allows scripts that are located on the same domain to interact, while preventing interaction between scripts that are on different domains. With interaction, I mean accessing properties, invoking methods and other scripting mechanisms.

The Same Origin Policy is invoked when one of the following actions happens.

• Requesting URLs with XMLHttpRequest (XHR)
• Accessing frames and iframes
• Accessing documents
• Accessing cookies
• Accessing browser windows

The list above is not complete, but contains the major actions that are validated against the Same Origin Policy.

This means that for example, it’s not allowed to invoke cross domain XHR requests. That’s good. It adds a serious layer of security and in most cases, you won’t need cross domain XHR requests.

But the Same Origin Policy doesn’t apply to everything. For example, it’s still possible to reference images, scripts and style sheets from different domains. It’s also possible to submit forms across domains.

I’m talking about domains a lot, let’s see what I mean with a domain, using the following listing. It shows how some example URL’s relate to the following web page: http://www.domain.com/pages/homepage.html. It also shows whether the Same Origin Policy allows document retrieval, and if not, why.

So a different domain can also mean: different protocol, different port or different subdomain.

Also, be warned that there are several ways around the Same Origin Policy. For example using the Flash cross-domain.xml file. Flickr was vulnerable to this, as written here.

To summarize, the Same Origin Policy provides quite a lot of safety, but not enough to completely prevent CSRF attacks. It does provide a helping hand, as I will show in the last part.

For a more detailed discussion of the Same Origin Policy, see:
https://developer.mozilla.org/En/Same_origin_policy_for_JavaScript and
http://taossa.com/index.php/2007/02/08/same-origin-policy/ and
http://taossa.com/index.php/2007/02/17/same-origin-proposal/

HTTPOnly cookie flag

As some may know, HTTPOnly is a cookie flag that is respected by most major browsers. Some browsers only added HTTPOnly support recently, like Firefox, which only has HTTPOnly support since version 3. Some browsers, like IE, already supported the flag for ages.

It is included in the Set-Cookie Http response header, as follows:

Set-Cookie: =[; =]
[; expires=][; domain=]
[; path=][; secure][; HTTPOnly]

It is appended to the Set-Cookie value in just the same way as the Secure flag, which prevents the cookie to be sent across insecure connections (non-HTTPS).

Basically, marking a cookie as HTTPOnly, prevents any scripts to access it. HTTPOnly cookies are only accessible by the browser to include it in the requests made to the server. It is not accessible by any JavaScript (the document.cookie property).

For more information, see the OWASP HTTPOnly documentation.

Btw. The Java Servlet specification only added HttpOnly support in version 3.0, so it’s not final at the time of writing!

Fixes

The solution is actually quite simple. Just use unguessable URLs in your website. You can achieve this by including a random token in the querystring/POSTdata of “important” requests. It’s not necessary to include it in requests that don’t change any state on the server, only state modifying requests need to have the token.

Why not include it in simple GET requests? Simple, because they don’t change anything and the hacker has no way to read the information that has been sent back to the browser, because of the Same Origin Policy. As indicated before, the Same Origin Policy prevents any scripts to read content from a remote page. Using XHR is also not possible, since the Same Origin Policy prevents cross domain requests using XHR. So, thanks to the Same Origin Policy, confidentiality is ensured. But to ensure integrity, some steps need to be taken by the developer.

For integrity, you need to implement a mechanism like the following: When you render an HTML form, be sure to include a hidden field with a random integrity token. You’ll also need to put this token into the session. When the form is submitted, you’ll need to compare the submitted token with the token in the session. If they are not the same, trigger an error or log the user off. Whatever you do, don’t let the request continue executing!

Also, don’t forget to include an integrity token in state-changing GET requests and Ajax calls and check appropriately!

There are a lot of variables in a successful CSRF prevention mechanism. In the next blog, I’ll give you some guidelines so you can pick the right ones for your website.

For the people that haven’t done so already, download Firefox with the Firebug plugin. You can activate Firebug using the F12 keyword.

Firebug has a really useful “Net” tab (which must be activated because it has quite an overhead. You can use this tab for debugging (inspecting headers and stuff), but it also gives you a good indication of your performance.

To give you an example, below is a screenshot of java.sun.com.

Analyzing the problem

What can a developer learn from this output?

1. First, you can see that the server side part is quite fast. It only takes 220ms to write the entire page to the client (the first bar). They could maybe optimize a bit by flushing earlier, to have better network utilization, but this is probably the consequence of an architectural choice (I assume they’re using an MVC style approach like Struts, where you would first gather all data and then forward to a JSP page). It wouldn’t even be a big win, since the server side performance is only a small part of the total performance.
2. Second, we can see a lot of activity is happening after the initial page request. By using the handy Firebug filters, I can easily see that 3 CSS style sheets, 17 JavaScripts and 38 images are fetched. This sums up to a total page load time of almost 4 seconds. In fact, there are so many requests that they don’t even fit on my screen.
3. A third lesson we can learn from this output is that almost all requests are made sequentially. I’ve made a simple calculation. The total amount of data downloaded is 137KB and it took around 4 seconds. That’s around 35KB/s!!! As an indication, I normally download (large files) with a megabyte per second. How about bad network utilization!
4. Fourth, there are some moments where no resources are downloaded at all. This is because downloaded resources have to be parsed, executed, rendered or whatever. This also hurts parallelism. After all, the computations required to do something with a resource don’t involve network I/O, so it would be nice if the browser started downloading the next file. This would be way more efficient. Fixing this issue is not trivial in most cases, so I’m not gonna talk about it here. Steve Souders has blogged about it extensively.

Http connections in your browser

Before trying to fix the problem (which I’m not going to do, since it’s not my website), we need to know some things about how web browsers handle requests.

An important thing to know is that browsers only use a certain amount of connections in parallel to the same web server. With web server, I mean the same IP address. So if multiple domain names resolve to the same IP address I’m talking about the same web server. This is in accordance with the HTTP 1.1 protocol, which states that a single web browser instance should only use 2 connections per web server. Below is a table with some popular browsers and their default parallel connection counts, using HTTP 1.1. (when using HTTP 1.0, the numbers may differ)

Browser                # of Connections
Internet Explorer <8   2
Internet Explorer >=8  6
Firefox <3             2
Firefox >=3            6
Safari                 4

References:
http://support.microsoft.com/?scid=kb%3Ben-us%3B282402&x=8&y=8
http://blogs.zdnet.com/Burnette/?p=565
http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/

You can change the settings of your browser. For example, when using Firefox, you can use about:config to increase the amount of parallel connections. But since most visitors of your website won’t change this setting and just use a mainstream browser, you can’t rely on this setting. On my current project (online ebanking, so the end user are “normal” people, not whizkids), we have to support browsers back to IE6. This means we have to perform well in browsers that use only 2 parallel connections per web server. Also, statistics indicate that people who use old browsers often are not on high bandwith connections (or are on a corporate network).

So… we had to do some work to enhance user experience.

To wrap up: The issue is simply that too many resources are loaded. It shows up in the chatty waterfall chart.

The solution

With a bit of background knowledge, we can start looking at the solution. We need a way to increase the average download speed/decrease the loading times. As shown in the image, you can see that several small files is not very efficient, so let’s start combining them. This saves some overhead, because we have to download less files. Also, as a nice side effect, compression techniques like GZIP are more efficient on large files.

But are issues, because we developers like to create maintainable code in separate small files. And working with large project teams and version control systems is often a mess when you have large files that everyone is editing concurrently.

Luckily, the solution is easy. Just aggregate all files into one big file. On my project, I’ve used Yahoo’s YUICompressor which not only aggregates, but also minifies the scripts and stylesheets. http://developer.yahoo.com/yui/compressor/

Since we use Apache Maven 2 as our build tool, I’ve integrated compression into our build using a Maven plugin which invokes YUICompressor. I’ve used http://alchim.sourceforge.net/yuicompressor-maven-plugin/

Using the default settings, the plugin only minifies the files. The minified files get the “-min” suffix.

Below is an example of a Maven 2 configuration where the listed files are minified and aggregated into one big “all.js” file. The normal files are still present, but I’ll get into that later.

<project>
  <build>
    <plugins>
      <plugin>
        <groupId>net.sf.alchim</groupId>
        <artifactId>yuicompressor-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compress</goal>
            </goals>
          </execution>
        </executions>        
        <configuration>
          <nosuffix>true</nosuffix>
          <aggregations>
            <aggregation>
              <insertNewLine>true</insertNewLine>
              <output>${project.build.directory}/${project.build.finalName}/js/all.js</output>
              <includes>
                <include>${project.build.directory}/${project.build.finalName}/js/jquery.js</include>
                <include>${project.build.directory}/${project.build.finalName}/js/util.js</include>
                <include>${project.build.directory}/${project.build.finalName}/js/defs.js</include>
                <include>${project.build.directory}/${project.build.finalName}/js/ajax.js</include>
                <include>${project.build.directory}/${project.build.finalName}/js/handlers.js</include>
              </includes>
            </aggregation>
          </aggregations>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

As you can see, I’ve explicitly specified the files to include. You can also use wildcards. I prefer this way because now I’m 100% sure of the ordering in the final aggregation. With wildcards, on the other hand, a simple refactoring (like renaming a file) could silently break the application @runtime. By being explicit, I get an error telling me that a file is missing.

CSS can also be aggregated and minified. And I must say, YUICompressor is really impressive. For example, it doesn’t just remove all comments, but determines per occurrence if it needs to be removed. For example, if you have a Safari Commented Backslash Hack v2 (http://perishablepress.com/press/2006/08/27/css-hack-dumpster/), it will not be removed, since that would break your CSS.

Below is an example where both the JS and CSS files are aggregated into two files: “all.js” and “all.css”.

<plugin>
  <groupId>net.sf.alchim</groupId>
  <artifactId>yuicompressor-maven-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>compress</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <nosuffix>true</nosuffix>
    <aggregations>
      <aggregation>
        <insertNewLine>true</insertNewLine>
        <output>${project.build.directory}/${project.build.finalName}/js/all.js</output>
        <includes>
          <include>${project.build.directory}/${project.build.finalName}/js/jquery.js</include>
          <include>${project.build.directory}/${project.build.finalName}/js/util.js</include>
          <include>${project.build.directory}/${project.build.finalName}/js/defs.js</include>
          <include>${project.build.directory}/${project.build.finalName}/js/ajax.js</include>
          <include>${project.build.directory}/${project.build.finalName}/js/handlers.js</include>
        </includes>
      </aggregation>
      <aggregation>
        <insertNewLine>true</insertNewLine>
        <output>${project.build.directory}/${project.build.finalName}/css/all.css</output>
        <includes>
          <include>${project.build.directory}/${project.build.finalName}/css/global.css</include>
          <include>${project.build.directory}/${project.build.finalName}/css/main.css</include>
          <include>${project.build.directory}/${project.build.finalName}/css/buttons.css</include>
          <include>${project.build.directory}/${project.build.finalName}/css/menu.css</include>
          <include>${project.build.directory}/${project.build.finalName}/css/components.css</include>
        </includes>
      </aggregation>
    </aggregations>
  </configuration>
</plugin>

What about images?

Images can be aggregated too, but this is way more difficult than with scripts. You’ll have to use image sprites, but this first involves creating a sprite (you can do this with online tools if you want), but then the difficulties start, since you need some CSS tricks to “select” the image from the sprite using offsets. This is a real pixel-pain-in-the-ass, but you’ll also enter the realm of cross browser issues here.

This page is a good starting point, but beware, depending on your situation, things may become difficult.
See: http://www.alistapart.com/articles/sprites.

On the other hand, you can of course begin small. For example, creating several sprites for buttons (with hovers and sliding doors you can turn 4 images into 1), boxes and logos. Every improvement is welcome.

Conclusion

Aggregation can greatly improve the performance of your website. On my project, I’ve decreased load times (with an empty cache) from 7 seconds to less than 3 seconds, using only aggregation and script/CSS compression. Using GZIP, caching and some other tweaks, I’m currently even lower. I’m still planning to implement the image sprites.

The moral of this blog is that there is more than the server side and you have to remember, it’s the end user who experiences your application. End users really hate hickups, long load times and web pages that build up in strange ways. They will be annoyed and maybe even lose their trust in the integrity of the application. And when this happens, you’re in deep shit.

Also, remember. Performance tuning differs from other disciplines, like security. With security, there are no (or at least little) compromises. But on the other hand, you can (and should) be happy with every performance improvement as the only valid performance measurement is the happyness of the end user.

Notes

1. I didn’t remove the original files, since I want to be able to switch @runtime between aggregated files and the originals. This greatly simplifies debugging, especially in production.
2. Check performance in all browsers. As indicated, browsers have different default values for several settings and different characteristics.
3. Client side performance is often easy to test. You don’t need heavy load, like you would when doing server side performance testing. You can start with your development machine and a simple testing server is often good enough for a more thorough test.

Cross Site Scripting?

First, let me explain what Cross Site Scripting (XSS) is. XSS is a hack where malicious code (JavaScript/HTML) is injected into a trusted site. With trusted, I mean the user trusts the site, which is usually the case with e-banking websites. At least, I hope so.

A simple example is a web site which contains a search box, like the following.

<form action="/search.do" method="GET">
  <input type="text" name="searchString" value="Enter something here" />
  <input type="submit" value="Search" />
</form>

When the user submits the form, a request is made to the following URL: /search.do?searchString=[THE_USER_INPUT]

The application responds with a page, containing the results, including a summary of the search criteria, and also the search form, prefilled with the search string the user entered, like here.

<!-- Some HTML -->
 
<form action="search.jsp" method="GET">
  <input type="text" name="searchString" value="${param.searchString}" />
  <input type="submit" value="Search" />
</form>
 
<table id="results">
  <!-- The search results -->
</table>
 
<!-- The rest of the site -->

So far, so good, right?

No, wrong!!!

Suppose a hacker enters the following value into the searchString box:

"/><script>alert("hello")</script><p id="

You’ll see an alert box with the message: “Hello”. You have injected a script into your web page.

But why would anyone hack his own browser session? That’s not the danger here. The danger is that a hacker can create an URL, like

http://www.yoursite.com/search.do?searchString="/><script>alert("hello")</script><p id="

and send it to other people.

When they click on it, the script gets executed in their browser.

Of course, an alert window is harmless, but what if I inject some DOM scripting to create a form where the user must enter his or her credentials for the attacked site? Or an image to automatically trigger a “Button click” (this is actually a case of Cross Site Request Forgery)? With such a leak, I can do almost anything on your page, without you knowing.

Let’s see what happens when the login page of your bank is vulnerable to XSS. Phishing becomes easy as stepping on kittens!

As you’re probably now aware of, XSS is a great “enabler” for all kind of other hacks. And the fun part is, hackers can spot XSS flaws easily and often using automatic tools. Not scared yet? In that case, I hope you’re not working on the website where I perform my electronic payments! You should be scared for these kind of leaks.

Countermeasures

In theory, it is very easy to protect your website against XSS. Just escape all variables, according to the escaping rules that apply for the markup/script language you are about to put the variables in. Apache Commons Lang has a utility for this purpose: the class StringEscapeUtils with its method escapeHtml and some others. escapeHtml is probably the method you’ll need most, since most variables are outputted in HTML tags.

Why is this fix enough? Well, just test it out with the example above. We modify the JSP to escape the input before rendering it to the client, as shown below.

<%
  String param = request.getParameter("searchString");
  if (param == null) param = "";
  String escaped = org.apache.commons.lang.StringEscapeUtils.escapeHtml(param);
%>
<form action="search.jsp" method="GET">
  <input type="text" name="searchString" value="<%=escaped%>"/>
  <input type="submit" value="Search"/>
</form>

The difference is that I added an escaping routine here. Nothing fancy, except that the “obvious” happens. The user input is shown on the screen in exactly the same way the user has entered it.

You see, no validation or black-/whitelists are needed, which is great, since you don’t want to bother users with security.

Some caveats, which become quite annoying later on.

• Timing is essential with regards to escaping. You cannot do it on your input parameters, since you’ll then end up with HTML encoded texts in your backend systems (that was the flaw I was talking about in the introduction). You also don’t want to be bothered with HTML encoding in your Java business logic, so input filtering is not the way to go. You definitely need to filter on the output, just before you render your page back to the client.
• You may not escape a variable twice, since the user ends up with HTML codes in his/her UI. When escaping > twice, it first becomes > and after the second pass, it becomes >, which is not only wrong with regards to usability, it also opens up some interesting new security holes.
• When you persist user input unencoded, which you should, you also need to escape all variables that come from there before rendering the web page. Essentially, you just need to escape every variable that is inserted into the web page, since it may (directly or indirectly) be manipulated by a malicious user.

Implementation

So, we need a way to escape all user input just before it is rendered. What are the options?

• Custom JSP function
  Probably the easiest way to escape special characters is by implementing a JSP custom function, like the JSF “outputText” component:
  

  <h:outputText escape="true" value="<script>" />

  . This one is easy to implement. It’s just a burden to edit all 200 JSP files (and the same amount of JSP Tag files) in our project and add the function at all places where a variable is used. The second issue is that you also need to be sure you don’t escape a variable twice, since that results in strange output and, ironically, new security holes. This is especially the case with tag attributes. Do you escape in the tag or in the calling code? You’ll need a strategy there. Finally, and probably most importantly, this solution will probably become a maintenance nightmare. JSPs are cluttered with these custom functions and future maintainers may never forget one when making changes. Something more generic would be nice.

• ELResolver?
  ELResolvers are added to JSP with version 2.1. We use Tomcat 6, so that’s great. However, the ELResolver mechanism differs greatly from, for example, the JSF VariableResolver mechanism. VariableResolvers are implemented using decoration, which makes them very useful to intercept all variable access and modify the responses. ELResolvers are implemented as a CoR and the implementation makes it impossible to create a custom filtering mechanism for all variables. It is also not the right place for this kind of logic. Why? Imagine a JSP Tag file. In the calling JSP file, you use EL to pass an argument to the tag and in the JSP Tag file, you use EL to put the argument in the HTML. With an ELResolver solution, variables get escaped twice. Not good.
• Tomcat hooks, AOP, Javassist?
  There are other ways to hook into the JSP/Servlet lifecycle. Containers provide vendor specific hooks to hook into the lifecycle, but unfortunately, no Tomcat hooks to help us here. But we’re not defeated yet! We can use AOP or a bytecode library to enhance some core Tomcat classes. Of course, we’re entering the black magic room here, but if it works… Unfortunately, these are also no real options. We can for example intercept all calls that go into the EL runtime by proxying all EL calls, but this has almost all of the issues as a custom ELResolver. Darn!
• Servlet Filter?
  But what’s wrong with a plain old Servlet Filter which escapes HTML special tags afterwards? The issue with a Servlet Filter is that it can’t possibly see the difference between legitimate and injected markup. The Filter Either escapes all markup (including your own) or nothing (which renders it useless). After the JSP has been executed, you’ll end up with HTML. If there is an XSS attack, you are simply too late when you use a Servlet Filter. So, it has to do with timing. Let’s step back.
• Object graph XSS filtering!
  There are two issues with most solutions presented.
  • Timing: The solution is too early or too late, making it useless or wrong.
  • no. executions: The solution has the risk of escaping the variable more than once, making it wrong and unsafe.

  So, what’s the best point in the request processing lifecycle to do the escaping? It turns out that just before the JSP is executed is the ideal moment. It’s ideal, because you’re not too early, so you won’t notice anything in your Java code. It’s also ideal, since you have full control over execution, making it easier to execute once and only once.
  

Object graph XSS filtering

The graph walker sounds like a solution, however, it’s a difficult one to implement.

The reason is the flexibility of our MVC framework. It’s allowed to pass arbitrary object graphs to the JSP. This is an issue, since you can’t just iterate over all attributes and escape them. For example, some attributes may be collections, maps or domain objects, containing possibly unsafe strings.

So we need an object walker. What features should it have?

• Cycle detection, to prevent infinite loops.
• Ability to modify arbitrary objects, either using reflection (POJO’s) or API (Collections).

Implementation sources

A hack and slash implementation of this solution is attached. What do you guys think of it? Does it look like a workable solution?

Ps. I’m aware of the fact that I haven’t implemented any pattern or followed any best practice. It’s just a simple PoC.

Click here to download the ObjectWalker sources

In construction the architect is responsible for producing the blueprint. The architect talks to the customer in order to get a list of requirements for the building. For instance is the building used as an office-space or is it used as a family-home. Should it be a single or multi story building and how many rooms should it have. And of course the amount of money the customer is willing to spend. Geared with the list of requirements the architect uses his design skills and his technical knowledge to make a first sketch of the building. The architect discusses this sketch with his team of electrical, mechanical and structural engineers to find out if it feasible from a technical point-of-view. For instance is the construction mechanically strong enough, are the selected materials useable, etcetera. This should all result in a blueprint for a building that meets the customer’s requirements, is usable with respect to the needs of the customer and is aesthetically pleasing. Finally the constructionworkers take the blueprint and start building the construction.

In software development some similar process takes place. In short the information analist and the customer discuss about the requirements for the system. What functionality should the system offer, what is the expected load etcetera. The information analist works in close cooperation with the software architect. The software architect is responsible for making the technical decisions (which frameworks to use, what platform, how many nodes, etc.) such that the non-functional requirements (scalability, extensibility, manageability, etc.) can be met. The UML-artifacts created by the information analist and the software architect (e.g. use-case model, software architecture document, use-case realizations, etc.) are input for the developers.

One major difference between the blueprint for the construction workers and the UML-artifacts for the developers is that the blueprint has usability embedded. The architect has knowledge about how to make a construction usable to its users. When there is a requirement that a house must have 2 bathrooms the architect knows that to make them usable he could place one on the groundfloor and the other on the first floor. He could have fulfilled the requirement by putting both bathrooms on the attic but that would not have made them very useable. Very often a developers gets a requirement which basically comes down to: “the application must have a web-interface or a GUI and it should be user-friendly”. That’s it. So now it is up to the developers to figure out what “user-friendly” is.

Most developers have a technical mindset and find it hard to step into the role of the end-user. It is the end-user that decides if the application is user-friendly. He does so when the application supports the end-user to do his job without problems. So in order to design a user-friendly interface it must be clear what the application is used for and how it is used. Very often user-friendlyness is misconceived as “protect the end-user from making mistakes”. If end-users use the application every now and then to do some work it can be valid to pop-up a confirmation dialog. Expert users that use the application on a regular basis get annoyed by such dialogs.

There are currently lots of tools/frameworks that allow developers to build goodlooking user-interfaces: JavaFX, Flex, SAF, JSF, Wicket, and the list goes on. But goodlooking is not the same as user-friendly. With the tooling mentioned before we could build a very nice 3D, animated, gradient color dialog box that is very annoying (remember the Office paperclip?). So basically we have the tools to create cool and goodlooking user-interfaces but most of us lack the knowledge of how to build usable ones.

My point is that there should be a role in the softwaredevelopment team, just like a information analist or a software architect, that is responsible for designing the interaction with the end-user. Just like the architect in construction. I haven’t come across a person with such a role in the many projects I’ve worked on over the past years. Until that time it is up to us developers to design user-interfaces and for that we need to broaden our horizon and try to place ourselves on the end-user seat.

A few years ago I read “About Face. The Essentials of User Interface Design” (first edition) written by Alan Cooper which got me interested on this subject (3rd edition is the most recent version). The book describes the problems that occur when developers start designing user-interfaces and give advice on how to improve on that, all illustrated with entertaining examples. If you get into the situation where you as a developer are responsible for the design of the user-interface this book might help you do a better job.

Vincent Hartsteen

N.B.: Currently I’m reading “The Inmates are running the Asylum”, also by Alan Cooper. This book describes the role of “interaction designers”. I’ll write my findings when I’ve finished the book.

Gegeven een DomeinObject als object uit de lagere laag, en een ServiceImplementation als object uit de hogere laag. De serviceImplementation heeft een een interface methode ’someMethod()’ uit de ServiceInterface die benodigd is om de methode ‘businessMethod()’ uit het DomeinObject te implementeren (zie figuur 1, originele klasse diagram).

N.B. Het design pattern is uitgewerkt met domein objecten en service implementaties, maar dat is slechts een invulling.

—————————————————————-

—————————————————————-
Figuur 1, originele klasse diagram
—————————————————————-

Los dit als volgt op. Trek de ServiceInterface uit de hogere laag, en zet deze in de lagere laag. Maak een RequiredInterfaceFactory die een instantie van de ServiceInterface kan bevatten. Initialiseer bij het opstarten van je applicatie vanuit de bovenste laag de RequiredInterfaceFactory met de ServiceImplementation (dit kan bijvoorbeeld goed met de Spring configuratie uit Listing 1, Spring configuratie). En implementeer de businessMethod door de aanroep naar de ServiceImplementation als volgt te abstraheren:

RequiredInterfaceFactory.getInstance().getServiceInterface().someMethod();

Het klasse diagram ziet er dan uit zoals weergegeven in Figuur 2, klasse diagram.

—————————————————————-

—————————————————————-
Listing 1, Spring configuratie
—————————————————————-

—————————————————————-

—————————————————————-
Figuur 2, klasse diagram
—————————————————————-

Voordelen en nadelen

Het gebruik van dit design pattern biedt de volgende voordelen:

• Compile time dependencies zijn niet nodig
• At runtime kunnen interface methoden van bovenliggende klassen toch gebruikt worden
• Ieder object kan gebruik maken van de interface, ongeacht in welk package het zit, dus het ophalen van de interface kan in de code altijd op dezelfde manier gebeuren

Het gebruik van dit design pattern biedt de volgende nadelen:

• Je moet de RequiredInterfaceFactory injecteren met de ServiceImplementation voordat de rest van de applicatie aangesproken wordt

Neem COBOL, één van de oudste programmeertalen: de naam COBOL betekent COmmon Business Oriented Language.  Het idee was dat met COBOL business mensen zelf konden programmeren. Dat is nooit gelukt natuurlijk. COBOL wordt algemeen beschouwd als een moeilijke programmeertaal, en COBOL programmeurs, hoewel een uitstervend ras, blijven nodig om de enorme hoeveelheid COBOL programmatuur die nog altijd in gebruik is te onderhouden.

SQL, zelfde verhaal. In SQL vertel je de computer, of de database eigenlijk, wat je met de gegevens wilt, maar niet hoe. Inmiddels wordt SQL zelfs voor gemiddelde programmeurs als te moeilijk beschouwd, en zijn er Object-Relational Mappers om ze tegen de “complexiteit” van SQL te beschermen.

Volgende kandidaat: UML.  We maken diagrammen, plaatjes die gebruikers kunnen begrijpen of zelf maken, en een code-generator die de plaatjes vertaald naar programma’s, en de gebruiker is weer “in control”. Ook deze aanpak, hoewel vrij recent, is inmiddels weer uit de gratie. Zie mijn eerdere post “Wat is er nieuw aan Model-Driven Development?“. Het gebruik van DSL’s in plaats van UML helpt wel een beetje, omdat een DSL nu eenmaal eenvoudiger is. Maar de verwachting van sommigen, dat het domein specifieke karakter van een DSL betekent dat een domein expert ermee kan programmeren kunnen we op basis van de eerdere voorbeelden gerust als naïef beschouwen. Al is het maar omdat het domein van een DSL vrijwel nooit het domein van de gebruiker is, maar dat van de programmeur.

Goed, nieuwe programmeertalen of in het algemeen technische hulpmiddelen helpen dus blijkbaar niet om de kloof tussen gebruiker en programmeur te verkleinen. Laten we het helemaal anders aanpakken. We hebben architecten nodig die de gebruiker helpen om de processen van zijn organisatie in kaart te brengen, en de automatisering daar precies op te laten aansluiten. En we introduceren een Service Oriented Architecture (SOA), zodat alle applicaties makkelijk met elkaar kunnen communiceren via services, en een process engine waarmee we de processen heel direct kunnen ondersteunen door de services in de juiste volgorde aan te roepen. Alleen, SOA experts worden niet moe om te benadrukken dat een SOA niet makkelijk in te voeren is in een organisatie, dat de beoogde voordelen – snelle aanpasbaarheid van de automatisering aan de veranderende eisen van de business – pas na jaren behaald zullen worden. Tenminste, als de invoering op de juiste manier wordt gedaan, en er gezorgd is voor een goede inbedding in de organisatie.  Governance is hier het toverwoord.

Klinkt allemaal prachtig, maar ik geloof er geen woord van. Ten eerste, hoezo snelle aanpassing aan veranderende eisen, als je eerst jaren moet investeren?  Ten tweede, governance wekt bij mij de associatie van “programming by commitee”. Moet dat sneller gaan dan programmeren door één of twee programmeurs? Ten derde, nu zitten de architecten tussen de gebruiker en de ontwikkelaars, zodat de afstand juist groter geworden is. De goede niet te na gesproken natuurlijk, maar veel van deze architecten hebben weinig affiniteit met de praktische problemen waar programmeurs mee te maken hebben bij het implementeren van een SOA. Ik hoor architecten vrijwel altijd klagen dat de programmeurs zich niet aan hun richtinggevende kaders houden. Hoe zou dat toch komen? Even opvallend is dat die organisaties de er een aparte afdeling architecten op na houden meestal ook degene zijn waar het hele software ontwikkelproces vrijwel tot stilstand is gekomen.

Is er dan helemaal geen hoop? Jawel, die is er. Die projecten die radicaal agile werken, volgens eXtreme Programming, Scrum, Evo of hoe ze ook heten, met een on-site customer, behalen ook een radicaal hogere productiviteit, en bouwen wat de klant wil. Behalve dat de verhalen overtuigend klinken heb ik het ook zelf ervaren. Dat beviel zo goed, dat ik eigenlijk niet meer anders wil en misschien ook niet meer kan.

Ondanks de goede resultaten is er toch veel weerstand tegen agile methoden.  Eén veelgehoord bezwaar is, dat grote systemen niet zonder architectuur kunnen, en dat een agile werkwijze tot chaotische onbeheersbare systemen leidt. Het eerste is waar: architectuur is nodig. Het tweede is niet waar: agile leidt niet noodzakelijk tot onbeheersbare chaos. Maar agile is geen vervanging voor vakmanschap. Alleen de manier waarop de architectuur tot stand komt is helemaal anders. Architectuur is een bijprodukt van software ontwikkeling. Het ontstaat tegelijk met de software zelf, en niet vooraf.  Alleen dan is er hoop dat de architectuur als een maatpak bij de applicatie past.

Systemen zijn tegenwoordig zo ingewikkeld, dat ze niet zonder architectuur kunnen. Diezelfde complexiteit is er ook de oorzaak van, dat die architectuur niet meer helemaal vooraf bedacht kan worden. Wordt dat wel geprobeerd, dan leidt dat tot starre systemen die moeilijk veranderd kunnen worden, precies wat diezelfde architectuur probeert te vermijden. Immers, het hoeft niet meer veranderd te worden, er is toch van tevoren over nagedacht? Kortom architectuur moet, maar wel via een agile proces.

Evans beschijft eigenlijk twee soorten patterns. De eerste soort geeft een onderscheid in typen objecten en methoden. Deze patterns zijn concreet terug te zien in een domein model. Dit zijn de patterns waar we echt iets mee kunnen in de referentie architectuur. De tweede soort zijn eerder proces patterns. Deze beschrijven wat abstractere doelen en wegen naar deze doelen. Deze patterns zijn moeilijk concreet te maken in een referentie architectuur en ze komen hier verder niet meer aan bod.

Kern van een DDD domein model zijn Entities (89) en Value Objects (97). Dit zijn de klassieke OO objecten met een toestand en gedrag. Entities zijn objecten met een “identiteit”. Ze zijn niet uitwisselbaar. Ze kunnen gedurende hun levensduur veranderen, maar ze blijven uniek. Een persoon object is in de regel een Entity. Een persoon wordt ouder, zijn uiterlijk verandert, allerlei attributen wijzigen, maar een persoon blijft dezelfde identiteit. Value Objects daarentegen zijn wel uitwisselbaar. In een klantencontactsysteem is een adres niets meer dan een verzameling attributen. Een adres object kan hergebruikt worden, maar dat hoeft niet. Het maakt niet uit als in het systeem twee adres objecten geïnstantieerd zijn met hetzelfde adres. En als een persoon en zijn adres uit memory verdwijnen, dan is het wanneer de persoon opnieuw opgevraagd wordt niet belangrijk dat hetzelfde adres object teruggevonden wordt. (Voor een kadastraal systeem gaat deze redenatie uiteraard niet op…)

Onderscheid tussen Entities en Value Objects is waardevol bij het ontwerp van een applicatie. We kunnen andere ontwerpregels op deze typen objecten loslaten. Zo is het een goed idee om Value Objects immutable te maken. Wanneer een attribuut dient te wijzigen vervangen we het object gewoon door een nieuwe instantie. Fowler definieerde het onderscheid tussen Entities en Value Objects al in Patterns of Enterprise Application Architecture. Hij lijkt Value Objects echter te suggereren voor algemene, herbruikbare zaken, zoals Money. Evans gebruikt het onderscheid structureler door elk “klassiek” domein object in een van beide hokjes te duwen. Bijvoorbeeld in een facturatie systeem kunnen we facturen modelleren als Entities en factuurregels als Value Objects.

Ook geeft Evans enkele richtlijnen voor het opstellen van Entities en Value Objects:

• Gebruik Intention-Revealing Interfaces (246), waarbij namen van klassen en methoden hun effect en doel beschrijven.
• Gebruik Side-Effect-Free Functions (250), factor deze uit in Value Objects en maak er Closed Operations (268) van.
• Maak daarentegen van Entity methodes die de state wijzigen commands die geen domein klassen teruggeven. Gebruik Assertions (255) om post-condities en invarianten van deze methodes te controleren.

Services (104) zijn objecten met enkel gedrag. Ze bevatten methoden die geen logische plek hebben in een Entity of Value Object. De naam ‘Service’ is enigszins verwarrend aangezien de term ’service’ in de Referentie Architectuur en andere architecturen in een andere context gebruikt wordt. Een betere naam is wellicht ‘Domain Service’.

Entities en Value Objects zijn de objecten in het domein model die gepersisteerd worden. Factories (136) en Repositories (147) zijn de domein objecten die voor persistentie verantwoordelijk zijn. Factories creëren domein objecten en Repositories bieden zoek-ingangen. Een Factory is in feite geen afzonderlijk type object. Een Factory wordt gerealiseerd door middel van een Factory Method (GoF) op een Entity of op een al dan niet speciaal voor het doel gecreëerde Service. Door vanuit de Factory Method te delegeren naar een ‘add’ methode op een Repository bewerkstelligen we dat database logica in één type domeinobject gecentreerd wordt.

De eenheid voor persistentie is een Aggregate (125). Dit is verzameling van een of meerdere entities en value objects, waarbij één entity, de root, als aanspreekpunt fungeert. Alle niet-root objecten in de aggregate staan uitsluitend ter dienste van de root en objecten buiten de aggregate hebben uitsluitend relaties naar de aggregate root.

Verder kunnen domeinobjecten gegroepeerd worden in Modules (109), om zodoende structuur te creëren in het domeinmodel. Modules of Packages zijn standaard concepten in OO programmeertalen en modelleertechnieken.

Een Specification (224) tenslotte is een eenvoudig Value Object dat een (of meerdere) business rule controleert voor een ander object. Specifications zijn gebaseerd op predicatenlogica waardoor ze als logische condities zijn te combineren tot complexere business rules. Een Specification kan drie doelen hebben:

1. Validatie van een business rule op een object. Een toepassing hiervan is de manier waarop Mod4J met business rules in de domein DSL omgaat.
2. Een selectie criterium specificeren om een object uit een selectie te selecteren. Een toepassing hiervan is gebruik van een Specification als parameter voor een Repository.
3. Bouw van een nieuw object specificeren.

Onderstaand plaatje toont bovenstaande patterns en hun relaties:

De hierboven genoemde patterns kunnen we zonder al te grote gevolgen in de referentie architectuur opnemen ter vervanging van / uitbreiding op de huidige typen business componenten. De vertaling gaat dan als volgt:

• De huidige Referentie Architectuur kent Business Processen, welke vergelijkbaar lijken met Domain Services. Business Processen zijn echter geen onderdeel van het domein model, ze worden niet aangeroepen door domein objecten. Verder stelt de Referentie Architectuur dat elke aanroep van een domein object via een business process gaat. In feite hebben we met deze Business Processen een extra “laagje” gecreëerd dat redundant is, aangezien erboven zich een verplichte servicelaag bevindt. Niets let ons om dat laagje te schrappen en domain services alleen te definieren als ze een toegevoegde waarde hebben.
• De domein objecten zoals we ze nu kennen kunnen we onderverdelen in Entities en Value Objects.
• Aggregates worden nu wel genoemd in de implementation view als eenheid van persistentie, maar ze worden niet expliciet onderkend als modelleereenheid in de logische view.
• Specifications zijn nieuw ten opzichte van de referentie architectuur, hoewel Mod4J ze wel al toepast voor business rules.
• Repositories zitten nu in een afzonderlijke laag, de datalaag, en heten Data Access Logic component. Echter, zoals ik in een vorige blog betoogde is dat niet terecht en horen ze in de domeinlaag thuis.
• Volgens diezelfde blog zijn de Service Agents en Data Service Agents zoals we ze nu kennen dezelfde dingen en kunnen we ze onderbrengen in een (nieuwe) laag, de Integratielaag.

Mijn conclusie is dat we door toepassing van de DDD patterns de referentie architectuur consistenter en begrijpelijker kunnen maken. In het ontwerp van een applicatie komt meer structuur, terwijl het resultaat, de implementatie, nagenoeg niet wijzigt ten opzichte van de huidige referentie architectuur. Bijkomend voordeel van toepassing van DDD is dat we voor de educatie van de architectuur kunnen verwijzen naar het boek “Domain Driven Design” van Evans.

Prompt reageerde Jan-Kees van Andel: “Ik verwacht niet dat Scala verder zal groeien qua populariteit. Op de JVM Language Summit 2008 was de algemene opinie dat Scala het niet gaat worden omdat het te moeilijk is. En dat ‘te moeilijk’ komt uit de mond van hele capabele mensen (lees: JVM taalontwerpers). Zij waren vooral gecharmeerd van Clojure.”

Aangezien ik me ten doel heb gesteld om Scala dit jaar onder de aandacht te brengen bij mijn J-Tech collega’s, kan ik dat natuurlijk niet over mijn kant laten gaan

Als je het hebt over JVM taalontwerpers: de auteur van Scala, Martin Odersky, is bedenker van Java generics, en ontwikkelaar van de Java 1.2 reference compiler. Zelf maakt hij dus ook deel uit van deze groep, en hijzelf is nog allerminst klaar met Scala.

Onlangs is bekend geworden dat Java7 geen closures zal krijgen – men kon geen beslissing nemen. Eind augustus/begin september postte Jan-Kees nog hoopvol een artikel waarin de drie onderhanden voorstellen behandeld werden, maar helaas zal geen ervan het licht gaan zien in Java7. Dat op zich is al een goed argument om naar Scala te kijken: daar zitten closures rotsvast in de basis van de taal.

Natuurlijk heeft Clojure ook closures (vandaar de naam…), maar als je kijkt naar de syntax van Clojure, zie je meteen waar deze voornamelijk op gebaseerd is: LISP. En dat staat niet voor niets voor Lots of Irritating Silly Parentheses (eigenlijk LISt Processing, maar je snapt waar ik heen wil). LISP stamt uit de jaren ‘50/’60 van het vorige millennium en heeft begrijpelijkerwijs een inmiddels verouderde syntax. De Scala syntax daarentegen is grotendeels op Java gebaseerd, wat de overstap Java -> Scala mijns inziens eenvoudiger maakt.

Een ander punt is dat ik een beetje gelogen heb toen ik zei “deze nieuwe [..] programmeertaal”: Scala is al sinds 2001 in ontwikkeling, en niet door de minste wetenschappers, en is zodoende al vergaand uitontwikkeld. Clojure begon pas in 2005.

Scala’s propositie is dat het een schaalbare taal is (‘SCAlable LAnguage’). Niet alleen in de zin van performance bij gebruik op multi-core systemen, maar ook – en veel belangrijker – schaalbaar naar het type applicatie dat je er mee wilt bouwen. Dat varieert van script via desktop applicatie tot enterprise systeem. De syntax ondersteunt al deze applicaties op dezelfde manier. Dat gezegd hebbende, is Scala een rijke taal, die je – naar wens – kunt gebruiken als ‘Java on steroids’, dan wel als een bijna-pure functionele taal à la Haskell. Als je dat laatste wilt, en momenteel alleen Java beheerst, is dat inderdaad nogal een klus, en kan ik me voorstellen dat je de taal als ‘te moeilijk’ bestempelt. Maar als je kiest voor het eerste, en je je de taal langzamerhand eigen maakt, kunt je zogezegd ‘opschalen’ naar een meer functioneel gerichte programmeerstijl. Kan allemaal in Scala.

Wat betreft de functionele programmeerstijl (FP): die is echt fundamenteel anders dan de imperatieve OO programmeerstijl die voor bijvoorbeeld Java de voorkeur geniet. Dat moet je dus leren, en leren is altijd (in meer of mindere mate) moeilijk – maar om daar meteen de taal de schuld van te geven? Als je overstapt van C naar Java moet je ook een nieuwe, in het begin moeilijke, stijl aanleren, maar uiteindelijk is dat de moeite dubbel en dwars waard. Voor FP geldt dat evenzogoed. Als je echt ‘hardcore’ puur FP wilt proeven, zou je eens kunnen kijken naar Haskell of ML. Maar dat is niet voor niets, ondanks jaren taalontwikkeling, nog steeds het domein van wiskundig opgeleide academici – programmeren in die talen wordt snel heel erg abstract (hoor ik iemand ‘Monads!’ roepen?). Scala is veel meer ‘down to earth’ en bereikbaar voor de gewone stervelingen.

Mocht je nog niet overtuigd zijn dat Scala toch echt wel wat meer aandacht verdient, check dan de Wiki die ik ingericht heb (helaas alleen voor Ordinamedewerkers). Daarop vind je een powerpoint van een presentatie die ik onlangs gaf bij een architectuur intervisie meeting, een draft cookbook en een Eclipse workspace met veel kleine codevoorbeeldjes. Wel eerst even de Scala plugin installeren!

Een special meeting Scala zit in de koker. Ook kun je extra blog items van me verwachten over bijvoorbeeld Liftweb en Sweet, twee web frameworks in Scala. A propos: ik ben nog op zoek naar reviewers voor mijn kookboek! Wie meldt zich aan als proeflezer?

Hedzer Westra

Mocht je overigens meer willen weten over Spring Batch en batch processing in het algemeen? Schrijf je dan in voor Special Meeting op 18 november a.s.

Bestandsdefinitie
Als voorbeeld gebruiken we een fixed-width file afkomstig uit een legacy systeem van een fictieve online videotheek. Zoals gebruikelijk bij deze vorm van gegevensuitwisseling bestaat het bestand uit verschillende soorten records: Een header record met metadata als een bedrijfsnaam en batchnummer, een footer record met een controlegetal en uiteraard een aantal records met de daadwerkelijke informatie; in dit geval film titels.

H OrdinaVideoStore.nl 12
M Shrek II
M Lord of War
M Godfather, The
M Kungfu Panda
F 0000000000000000006

Tokenizing
Om bovenstaand bestand te kunnen parsen dient allereerst onderscheid te worden gemaakt tussen de verschillende soorten records. Aangezien dit uit het eerste karakter van elk record kan worden afgeleid gebruiken we de PrefixMatchingCompositeLineTokenizer.

<bean id="movieFileLayout"
class="org.springframework.batch.item.file.transform.PrefixMatchingCompositeLineTokenizer">
	<property name="tokenizers">
		<map>
			<entry key="H" value-ref="movieHeaderRecordLayout" />
			<entry key="M" value-ref="movieRecordLayout" />
			<entry key="F" value-ref="movieFooterRecordLayout" />
		</map>
	</property>
</bean>
 
<bean id="movieHeaderRecordLayout"
	class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
	<property name="names" value="recordtype, videostore, batchid" />
	<property name="columns" value="1,5-25,28-30" />
</bean>

Deze tokenizer geeft records op basis van het type door aan een nieuwe LineTokenizer. Zoals je in bovenstande configuratie kunt zien mapped deze tokenizer elke fixed-width kolom naar een bijbehorende kolomnaam. Hier zie je duidelijk de kracht van de configuratie mogelijkheden in Spring. Je geeft gewoon in de application context de ranges van gerelateerde kolommen op en Spring Batch doet de rest. Om deze magic mogelijk te maken heeft Spring wel wat hulp nodig in de vorm van een CustomEditorConfigurer. Deze vertaald de range definities (5-3, etc) naar betekenisvolle Range objecten.

<bean id="customEditorConfigurer"
	class="org.springframework.beans.factory.config.CustomEditorConfigurer">
	<property name="customEditors">
	  <map>
	    <entry key="org.springframework.batch.item.file.transform.Range[]">
	      <bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" />
	    </entry>
	  </map>
	</property>
</bean>

Mapping naar domein objecten
Vervolgens is het de bedoeling om de kolom/veld definities naar domein objecten te vertalen. Hiervoor heeft Spring Batch de FieldSetMapper interface geïntroduceerd. Hoewel we hiermee zelf de mapping van velden naar domein objecten kunnen verzorgen is het mogelijk om dit aan Spring Batch over te laten middels een FieldSetMapper gebaseerd op Spring’s BeanWrapper. Er moet echter ook onderscheid worden gemaakt tussen de verschillende record soorten. Hiervoor is – in tegenstelling tot de prefix-enabled tokenizer – geen standaard fieldset mapper aanwezig. Daarom schijven we zelf een eenvoudige mapper:

public class PrefixAwareMovieFieldSetMapper implements FieldSetMapper {
	private FieldSetMapper headerFieldSetMapper;
	private FieldSetMapper movieFieldSetMapper;
	private FieldSetMapper footerFieldSetMapper;
 
	public Object mapLine(FieldSet fieldSet) {
		final String recordType = fieldSet.readString("recordtype");
		if (recordType.equals("H")) {
			return headerFieldSetMapper.mapLine(fieldSet);
		}
		else if (recordType.equals("M")) {
			return movieFieldSetMapper.mapLine(fieldSet);
		}
		else if (recordType.equals("F")) {
			return footerFieldSetMapper.mapLine(fieldSet);
		}
		throw new IllegalStateException("onbekend record type");
	}
	// setters
}

Wanneer we deze FieldSetMapper configureren is de mapping van tokens naar domein objecten compleet.

<bean id="movieFileFieldSetMapper"
	class="nl.ordina.batch.PrefixAwareMovieFieldSetMapper">
	<property name="headerFieldSetMapper" ref="movieHeaderFieldSetMapper" />
	<property name="movieFieldSetMapper" ref="movieFieldSetMapper" />
	<property name="footerFieldSetMapper" ref="movieFooterFieldSetMapper" />
</bean>
 
<bean id="movieHeaderFieldSetMapper" 
   class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
	<property name="prototypeBeanName" value="headerRecord" />
</bean>
 
<bean id="headerRecord" class="nl.ordina.batch.model.MovieHeaderRecord"
scope="prototype" />

Parsen
Nu we hebben gedefinieerd welke karakters naar welke velden mappen, en welke velden naar welk domein object mapped wordt het tijd om bestanden te parsen. Hiervoor gebruiken we een FlatFileItemReader die we injecteren met de eerder gecreërde tokenizer en fieldset mapper. Geconfigureerd binnen een batch job ziet dit er als volgt uit. Wanneer deze job wordt uitgevoerd ontvangt de “DummyWriter” voor elke record in het bronbestand een overeenkomstig domein object.

<bean id="videotheekJob" parent="simpleJob">
  <property name="steps">
    <list>
      <bean id="printRecordStep" parent="simpleStep">
	<property name="itemReader">
	   <bean class="org.springframework.batch.item.file.FlatFileItemReader">
	     <property name="resource" ref="file:///videotheek/films.txt" />
	     <property name="lineTokenizer" ref="movieFileLayout" />
	     <property name="fieldSetMapper" ref="movieFileFieldSetMapper" />
	   </bean>
	</property>
	<property name="itemWriter" ref="dummyWriter" />
      </bean>
    </list>
  </property>
</bean>

Zoals je wellicht opvalt wijst de FlatFileItemReader naar een hardcoded filepath. Momenteel biedt Spring Batch nog geen mogelijkheid om dit voor meerdere bestanden configurabel te maken. Dit heb ik inmiddels gemeld via een JIRA ticket en een oplossing bijgevoegd in de vorm van een DynamicMultiResourceItemReader.

Tot slot, Spring Batch biedt m.i. behoorlijke ondersteuning om diverse soorten bestanden te parsen. Dit is met name handig in applicaties die reeds gebruik maken van het Spring Framework. Maar Spring Batch bevat méér en is zeker het overwegen waard buiten pure Spring applicaties.