The implementation of Article class was not thread safe. This was acceptable because of the assumption that the ORM library would be creating the Article instances in the requestor’s thread. However, those types of assumptions can be dangerous over the long term. Let’s pretend that the performance of the ArticleService was not good and a developer was given the task to increase its performance. To simulate this scenario, I created an implementation of the ArticleService that stored Article instances in memory (as if they were cached or if a framework like Terracotta were being used). The Article class should be thread safe for this implementation, but it wasn’t modified. We hope the test case will fail with the incorrect code. However, the test succeeded almost every time on my dual core workstation. Why was that?

The primary reason the test passed was that it was too fast. Christian warns us about the slow speed of his testing technique, but the technique is not slow. The original code being tested accessed a relational database so it was slow for that reason, not because of the testing technique. With my ArticleService implementation, the test executed in 15-30 milliseconds. My theory is that the short execution time results is what caused the test to pass almost every time. The Hotspot JIT is probably not able to compile and reorder instructions and it’s possible the threads all ran on the same CPU core. When the test did fail it seemed to be when the machine was under a little more load and the execution time was 40 ms or greater. In another experiment, I increased the size of the Article collection to 250 to 2500 and the test failed every time. For a test like this, it might be useful to measure the execute time in the test and signal an error if it runs too fast. Of course, if we didn’t know the test effectiveness was a function of execution speed we’d never think of adding the timing check.

In general, we should be very skeptical about multithread test cases. Just because they pass doesn’t necessarily mean our code is correct. A nondeterministic test that passes 99% of the time will give us false confidence. How can we be sure that our test is worthy of our confidence? Other than advanced tools like ConTest or code reviews by engineers who are experts in the Java Memory Model and multithreading, I’m not sure. It’s a difficult problem with no easy answers.

Consider this scenario. Let’s say you have created an InvoiceProcessor class that maintains a count of how many invoices have been processed. The class uses this count for logging purposes. Let’s also say the server is implemented using a Spring IOC container and a single-threaded JMS message listener container. The architect designed it this way to eliminate the need for the programmers to worry about concurrency issues. Or so he thought.

Your invoice process might look something like the following…

public class InvoiceProcessor {
    private int invoiceCount;

    public void process(Invoice invoice) {
       // ...
       invoiceCount++;
    }

    // ...

    public int getInvoiceCount() {
       return invoiceCount;
    }
}

One day a developer decides that it would be cool to use JMX to monitor the invoice count. With Spring, they just add some JMX annotations to the InvoiceProcessor and them make some simple modifications to their Spring configuration to detect their annotated MBeans. They fire up JConsole and are are delighted to see their invoice count displayed there. They can even see a graph of the invoice counts over time. Very cool. However, their delight soon fades. While testing, they notice the counts in JConsole don’t match the number of invoices in their test case. What happened? They’ve been bitten by unexpected multithreading.

Remote JMX access has created multithread access to the InvoiceProcessor. There are no memory barriers around access to the count field so there’s no way to know if the value returned by the JMX remoting thread is anywhere close to the value seen in the thread that processes invoices. Making the count field volatile is a potential solution in this scenario.

Although this example is hypothetical, I’ve actually seen a bug like this in previous versions of Hibernate’s JMX support. In general, be aware that remote JMX access will require any instrumented classes to be thread safe.

Of course, there are other ways to be bitten by unexpected multithreading. For example, you might use the InvoiceProcessor in a Swing GUI and display the count in a monitoring panel. Now the count field is being accessed from both the AWT event dispatching thread and the request processing thread. Or you might decide to schedule a Quartz job to reset the count every day at midnight. This is easily done in the Spring configuration file, but once again, your code now must now be thread safe. Other possible sources of unexpected multithreading are Java shutdown hooks or object finalizers since they run in their own threads. If you know of other common sources of unexpected multithreading, share them in the comments.

My favorite talk was Glen Smith’s overview of Grails plugins for Rich UI support, search, Ajax and more. It was a very informative and fun presentation.

The speaker’s panel discussion was one of the best I’ve seen. One of the most controversial questions was about Groovy, JRuby and the relative impedance mismatch with the Java Platform. Neil Ford of Thoughtworks initially took the position that there was no impedance mismatch with JRuby. Graeme Rocher and others listed numerous examples of where JRuby was less integrated with Java compared to Groovy. Graeme commented that although JRuby can access compiled Java classes, it is much more difficult (impossible?) for Java to use JRuby classes. Groovy classes are Java classes and so they can be used relatively transparently in Java programs.

Speaking from experience, I’ve tried to use Java libraries (QuickFIX/J) using JRuby and it has proven much more difficult than with Groovy. One issue was insufficient (and somewhat outdated) documentation about how to integrate JRuby with Java classes outside the “java.*” packages. This was a few months ago so maybe the documentation has improved. However, with Groovy I was able to use the Java libraries within a few minutes.

I agreed with a point that Neil made that having some competition in the JVM dynamic language arena helps everyone. I highly respect the work the JRuby team is doing although I believe that Groovy is a better and more pragmatic dynamic language choice for Java developers.

There’s no question that open source technology is used widely in financial applications, especially for middle components. However, there was also a discussion about how willing the financial companies are to open source their own technology (where “technology” might include algorithms and research rather than just software).

I don’t believe any company intends to sacrifice a competitive advantage by converting their technology to open source. For that reason, I doubt that we’ll see financial companies releasing algorithms or proprietary research. However, financial companies are involved in open source in a variety of roles. The user role is probably most common today, but a few financial companies have contributed to open source. One example is the Advanced Message Queuing Protocol (AMQP) effort started by JP Morgan and others. This technology benefits financial software (and software from other domains) by provides a programming language-independent and vendor-independent wire protocol for message queues. For this technology to be successful it must be widely used. An open source strategy has a clear advantage here.

Another example is the Open Financial Market Platform (OFMP) which has recently become an Eclipse Foundation project. In this case, I’m not so clear about the benefits to the company contributing the initial code. Maybe someone from the project will comment.

Tim O’Reilly closed the panel with a suggestion that open source developers consider projects that make it easier to capture and process the vast amounts of unstructured data available on the net.

import java.util.HashMap

describe "An empty", HashMap do
 before :each do
  @hash_map = HashMap.new
 end
 it "should be able to add an entry to it" do
  @hash_map.put "foo", "bar"
  @hash_map.get("foo").should == "bar"
 end
 it "should return a keyset iterator that throws an exception on next" do
  proc do
   @hash_map.key_set.iterator.next
  end.should raise_error(java.util.NoSuchElementException)
 end
end

What is the point of this example? Is there really a significant difference between this example and the following JUnit4 test class?

import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertThat;

import java.util.HashMap;
import java.util.NoSuchElementException;

import org.junit.Test;

public class HashMapTest {
    private HashMap<String,String> hashMap =
            new HashMap<String,String>();

    @Test
    public void shouldBeAbleToAddAnEntryToIt() {
        hashMap.put("foo", "bar");
        assertThat(hashMap.get("foo"), is("bar"));
    }

    @Test(expected = NoSuchElementException.class)
    public void shouldReturnAKeysetIteratorThatThrowsAnExceptionOnNext() {
        hashMap.keySet().iterator().next();
    }

}

Sure, the syntax is a bit different and the Java version is slightly more verbose with it’s access specifiers and type declarations. However, the Java version runs fast enough that you don’t need to set up a Ant server to get reasonable performance from your tests (as recommended in the article for JTestR).

I’m not questioning that scripting can be useful in unit tests. However, I’d think that Groovy’s superior Java integration would make it a better choice for this purpose than JRuby. Does anybody know of some compelling, pragmatic examples of the relative benefit of JRuby over Groovy for Java test scripting purposes?

Installation

You can use JUnitFactory through a web interface or as an Eclipse plugin. I have only used the Eclipse plugin . It was very easy to install using the JUnitFactory Eclipse update site (more info). You can also download a file if you prefer manual installation. The JUnitFactory plugin is currently only available for Eclipse, but may be available for other IDEs in the future.

Generating Tests

Warning: Carefully read the JUnitFactory FAQ before generating any tests!

I decided to generate tests for a specific class I had been modifying. When I ran the test generator, it first required me to set up a user name and password and agree that I’ve read their legal statements about security and other usage risks.

Do not generate tests for company code with this tool unless you have permission to upload your company’s proprietary source to a nonsecure remote server, possibly to be stored there and used for unspecified purposes. In some companies, you might lose your job for this type of security breach.

This wasn’t a problem for me since I was generating tests for an open source code. I set up an account and then initiated the test generation again. Since I hadn’t read the FAQ first, I was surprised to see that hundreds of files were being uploaded to their server. JUnitFactory uploads every source and class file in your Eclipse project when it generates tests. In fact, if your project depends on other projects it will also upload all the sources and classes for those projects.

When test generation begins, a JUnitFactory view is displayed in Eclipse and it shows the progress of the test generation. My first impression of the Eclipse plugins are that they are well designed. I didn’t encounter any bugs when using them.

The test generation completed after a few minutes (the source project was small, about 50-60 classes). The Eclipse view for my test generation results is shown below.

As you can see the coverage wasn’t that great. There are several reasons for this poor coverage. The number after “Done” in the status column is a count of the number of suggestions from JUnitFactory about testing problems and how to improve the test coverage. The screen shown below contains typical warnings and suggestions.

Clicking in the second column of the JUnitFactory view will display the generated test code. Being the eager person I am, I immediately ran the tests in Eclipse JUnit test runner… and they all failed. Fortunately, the situation was quickly resolved by reading the test error messages which told me that I must run the test in the Agitar-specific JUnit test runner. I do so and all the tests passed. I’m not sure why a proprietary test runner is required. It doesn’t appear to be compatible with other plugins I have installed, like the EclEmma coverage analyzer.

First Impressions

One very important fact to remember is that the tests generated by JUnitFactory are not intended to be used in the way most people use unit tests. They don’t test correct behavior, for example. They test what the code does, not what it’s supposed to do. To replace Test Driven Design (TDD) and handwritten unit tests with JUnitFactory-generated would be a huge mistake.

Agitar suggests that the tests are best used as characterization tests. A characterization test can be thought of as executable documentation of the behavior (actual, not necessarily correct, behavior) of the tested software. If you are redesigning existing (legacy) software that doesn’t have tests, a characterization test suite can help you determine if the new software behaves the same as the previous implementation.

This sounds great in theory, but there are many practical problems. For example, it may be difficult to effectively use the legacy-based characterization tests if the OO design changes in the new code. The JUnitFactory tests are at the class-level, so if even just the class name changes in the new code you’ll need to make manual modifications to the characterization tests to run them against the new code. With a highly modified OO design, you’ll probably not be able to effectively use the legacy tests at all or only with significant effort. This problem is aggravated if the legacy software has problems like undisciplined use of method visibility. The exposed methods will generally trigger unit test generation although they are really an internal implementation detail. Of course, JUnitFactory can’t do anything about that, but it’s a practical problem to consider. If you have thousands of classes with issues like these, it would be a major effort to keep the characterization test suite consistent between legacy code and new implementations.

JUnitFactory looks inside the implementation of the class when generating the tests. This can result in some tests that are arguably overly implementation-dependent and will fail even with a correct reimplementation of the code. For example, it may assert that a NPE is thrown for a null method argument. This isn’t required behavior of the class. A new implementation might handle the null argument without a NPE (by returning a empty result, for example). This can be easily corrected in any specific test suite. However, in a large legacy system there could be many of these false alarms.

Conclusions

To quote one of the JUnitFactory announcements from a few months ago:

Remember, it’s still experimental and mostly for fun so don’t expect too much or bang on it too hard.

Overall, I thought JUnitFactory was interesting but I’m not convinced it would be useful for my day-to-day work. If I weren’t required to upload the code to their server, I’d love to try it on some work-related legacy code that we are currently reimplementing. The legacy code is EJB-based (circa 2000), has few abstractions for testing, and makes heavy use of stored procedures. My guess is that JUnitFactory wouldn’t be able to help me much in this situation.

JUnitFactory does provide various types of hooks for fixing testing-related problems and improving coverage through writing extra testing support code. Based on the coverage and suggestions for the project shown earlier in the JUnitFactory view, most of the classes in that project would need custom test framework extensions to get decent coverage. This was relatively well-written code. I’m guessing that practically 100% of the classes in the legacy system I described would need custom test extensions to get acceptable test coverage. That’s a huge job. And I’m wondering if the whole approach is fatally flawed anyway.

When reimplementing a legacy system, characterization tests at the class level are generally going to be far too low level (too sensitive to behavior-preserving design changes). Another strategy is to do characterization testing closer to the system or subsystem level and verify that externally visible behavior is consistent. JUnitFactory will not help with this type of testing.

Another use for JUnitFactory is to find holes in a handwritten unit test suite. However, be prepared for studying numerous, somewhat obfuscated tests, and then doing some deep reasoning about whether what you learn represents a hole in your current test suite or not.

I’m not sure about the practical payoff, but if you are interested in automated test generation from a research perspective, definitely try out JUnitFactory.

Related Links:

• Fun With Automated Characterization Test Generation
• Characterization: Beyond Pure Unit Testing

Dave Thomas: I think agile methodologies have adopted their own venom. And to that extent, I’m pulling back, away from, the individual agile methodologies. I believe very, very strongly in agility. I’m an author of the agile manifesto and I think the underlying values of agility fit far better with my world view of how software should be built. I think some of the implementations of agile methodologies are, frankly, scarey in their kind of…

Bob Payne: dogmatic?

Dave Thomas: dogmatic, yeah that’s a good word, that’s a mild word for some it. It’s kind of like “my way or you’re not doing it properly”, a lot of blackmail, a lot of absolutes. And to me that’s not what agility is all about. So I pull back from that. I don’t want to be associated with that.

Bob Payne: In the past I’ve felt a number of pulls in different directions and I’ve started to realize that it is not necessarily the particular practice but what the resulting outcome is, the rhythm that it sets up.

This perspective sounds very similar to what I’ve been calling “agile without a name” or “effective software development” or what Tim Beck calls “pliant software development“.

P.S. I highly recommend the Agile Toolkit Podcast. It’s a great resource to hear candid discussions about agile topics.

This problem is not unique to eBay. For example, CNet News reports that “a Google engineer has warned that if the performance per watt of today’s computers doesn’t improve, the electrical costs of running them could end up far greater than the initial hardware price tag.” The Gartner group reports that IT energy costs will increase from 25-30% to 40% within 5 years. A report from Lawrence Livermore Laboratories says that data centers will be using more electricity than the 275 million televisions in the United States. The Wall Street Journal reports that one large data center can consume as much power as a small city of 30,000-40,000 people.

There are several ways to respond to these challenges. The most common approaches are to buy more efficient hardware or to place data centers near cheap and abundant power. This is reportedly one reason for Google’s interest in creating a data center in North Carolina. However, Dan Pritchett spoke of another option: writing more energy efficient software. He estimates that inefficient programming techniques may increase eBay’s energy consumption by 25-30%. This limits scalability of their data center given the limits of the power companies to provide energy, but it also results in huge amounts of wasted money for large data centers.

It’s a very interesting idea. Computationally efficient software would also generally be more energy efficient. What other techniques can we use to optimize power efficiency for “green software development”?

It was also interesting to see that XP has been adopted in about half as many organizations as Scrum. Of course, I’d guess that many of the Scrum shops are using at least some XP software development practices.

XPlanner was the fourth most widely used APM tool and the only open source product in the top 10 most used tools. (There is a factual error in the survey results that says ScrumWorks is also open source. The ScrumWorks Basic product is free but not open source). XPlanner is also the most used tool focusing specifically on XP. It surprised me a little to see that XPlanner is adopted somewhat more by large organizations than small organizations. It’s possible that this is an indication that the licensing fees for the commercial tools are high enough to cause some pain for large organizations. I know at least one large organization that chose XPlanner over VersionOne for this reason.

As the founder^[1] of the XPlanner project, I’m very happy to see it so widely used even with no marketing or sales resources.

“The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions. On the other hand, there is a significant 84 percent increase in effort to perform the tasks correctly.”

In specific contexts, like junior programmers pairing on complex tasks, pair programming results in significant increases in quality. However, in other contexts the increase in quality, if any, comes at a high cost. The study doesn’t appear to support the conclusions described in earlier XP research. Specifically, that a team could expect “a 15% increase in development cost for a 15% improvement in quality”. This earlier study used students rather than professional programmers and the strength of some of the researchers’ techniques and conclusions have been questioned.

My own personal experience is that I enjoy pair programming for some activities and in some contexts, but not in all contexts. On complex tasks, I enjoy pairing with other senior developers for collaborative problem solving. It’s not as enjoyable, but I also see benefit for the team when pairing with junior or less skilled developers for some tasks. The less skilled developer may provide the occasional useful input often I feel the collaboration is a net loss in productivity. For example, let’s say we’re working on functionality that uses several advanced technologies. If my pair is not skilled in these technologies then the pairing will probably not be a net productivity increase. You could say that it’s valuable for training the less skilled developer in these technologies. However, I’m speaking about the case where the less skilled developer has already had extensive opportunities to learn the technologies in question and has not been successful. “Training” by pair programming will generally not solve this problem, in my experience.

When developing simpler functionality, I see more benefit from testing and specifically from test-first and TDD techniques than pair programming. For the either/or types of people I generally enjoy collaboration and I’m not the type that wants to work in isolation from the team. Pair programming is only one specific form of collaboration. I’ve worked exclusively in open work areas for the last 7-8 years and worked in those environments several times before then. I like that work environment very much and although I had the option to have an office I choose to sit with the team.

I like Martin Fowler’s perspective on pair programming. He says “my view is that you should try it and the team should reflect on whether they feel they are more effective with pairing that without”. I’d also recommend that the team discuss in what contexts they feel they are more effective or not.