JUnitFactory: First Impressions
JUnitFactory is an automated unit test generation service from Agitar Software Labs. This article describes my initial experiences and impressions from using the tool to generate unit tests.
Installation
You can use JUnitFactory through a web interface or as an Eclipse plugin. I have only used the Eclipse plugin . It was very easy to install using the JUnitFactory Eclipse update site (more info). You can also download a file if you prefer manual installation. The JUnitFactory plugin is currently only available for Eclipse, but may be available for other IDEs in the future.
Generating Tests
Warning: Carefully read the JUnitFactory FAQ before generating any tests!
I decided to generate tests for a specific class I had been modifying. When I ran the test generator, it first required me to set up a user name and password and agree that I’ve read their legal statements about security and other usage risks.
Do not generate tests for company code with this tool unless you have permission to upload your company’s proprietary source to a nonsecure remote server, possibly to be stored there and used for unspecified purposes. In some companies, you might lose your job for this type of security breach.
This wasn’t a problem for me since I was generating tests for an open source code. I set up an account and then initiated the test generation again. Since I hadn’t read the FAQ first, I was surprised to see that hundreds of files were being uploaded to their server. JUnitFactory uploads every source and class file in your Eclipse project when it generates tests. In fact, if your project depends on other projects it will also upload all the sources and classes for those projects.
When test generation begins, a JUnitFactory view is displayed in Eclipse and it shows the progress of the test generation. My first impression of the Eclipse plugins are that they are well designed. I didn’t encounter any bugs when using them.
The test generation completed after a few minutes (the source project was small, about 50-60 classes). The Eclipse view for my test generation results is shown below.
As you can see the coverage wasn’t that great. There are several reasons for this poor coverage. The number after “Done” in the status column is a count of the number of suggestions from JUnitFactory about testing problems and how to improve the test coverage. The screen shown below contains typical warnings and suggestions.
Clicking in the second column of the JUnitFactory view will display the generated test code. Being the eager person I am, I immediately ran the tests in Eclipse JUnit test runner… and they all failed. Fortunately, the situation was quickly resolved by reading the test error messages which told me that I must run the test in the Agitar-specific JUnit test runner. I do so and all the tests passed. I’m not sure why a proprietary test runner is required. It doesn’t appear to be compatible with other plugins I have installed, like the EclEmma coverage analyzer.
First Impressions
One very important fact to remember is that the tests generated by JUnitFactory are not intended to be used in the way most people use unit tests. They don’t test correct behavior, for example. They test what the code does, not what it’s supposed to do. To replace Test Driven Design (TDD) and handwritten unit tests with JUnitFactory-generated would be a huge mistake.
Agitar suggests that the tests are best used as characterization tests. A characterization test can be thought of as executable documentation of the behavior (actual, not necessarily correct, behavior) of the tested software. If you are redesigning existing (legacy) software that doesn’t have tests, a characterization test suite can help you determine if the new software behaves the same as the previous implementation.
This sounds great in theory, but there are many practical problems. For example, it may be difficult to effectively use the legacy-based characterization tests if the OO design changes in the new code. The JUnitFactory tests are at the class-level, so if even just the class name changes in the new code you’ll need to make manual modifications to the characterization tests to run them against the new code. With a highly modified OO design, you’ll probably not be able to effectively use the legacy tests at all or only with significant effort. This problem is aggravated if the legacy software has problems like undisciplined use of method visibility. The exposed methods will generally trigger unit test generation although they are really an internal implementation detail. Of course, JUnitFactory can’t do anything about that, but it’s a practical problem to consider. If you have thousands of classes with issues like these, it would be a major effort to keep the characterization test suite consistent between legacy code and new implementations.
JUnitFactory looks inside the implementation of the class when generating the tests. This can result in some tests that are arguably overly implementation-dependent and will fail even with a correct reimplementation of the code. For example, it may assert that a NPE is thrown for a null method argument. This isn’t required behavior of the class. A new implementation might handle the null argument without a NPE (by returning a empty result, for example). This can be easily corrected in any specific test suite. However, in a large legacy system there could be many of these false alarms.
Conclusions
To quote one of the JUnitFactory announcements from a few months ago:
Remember, it’s still experimental and mostly for fun so don’t expect too much or bang on it too hard.
Overall, I thought JUnitFactory was interesting but I’m not convinced it would be useful for my day-to-day work. If I weren’t required to upload the code to their server, I’d love to try it on some work-related legacy code that we are currently reimplementing. The legacy code is EJB-based (circa 2000), has few abstractions for testing, and makes heavy use of stored procedures. My guess is that JUnitFactory wouldn’t be able to help me much in this situation.
JUnitFactory does provide various types of hooks for fixing testing-related problems and improving coverage through writing extra testing support code. Based on the coverage and suggestions for the project shown earlier in the JUnitFactory view, most of the classes in that project would need custom test framework extensions to get decent coverage. This was relatively well-written code. I’m guessing that practically 100% of the classes in the legacy system I described would need custom test extensions to get acceptable test coverage. That’s a huge job. And I’m wondering if the whole approach is fatally flawed anyway.
When reimplementing a legacy system, characterization tests at the class level are generally going to be far too low level (too sensitive to behavior-preserving design changes). Another strategy is to do characterization testing closer to the system or subsystem level and verify that externally visible behavior is consistent. JUnitFactory will not help with this type of testing.
Another use for JUnitFactory is to find holes in a handwritten unit test suite. However, be prepared for studying numerous, somewhat obfuscated tests, and then doing some deep reasoning about whether what you learn represents a hole in your current test suite or not.
I’m not sure about the practical payoff, but if you are interested in automated test generation from a research perspective, definitely try out JUnitFactory.
Related Links:
Alberto Savoia wrote:
Hi Steve,
Thank you for a very well written and objective review.
Making JUnit Factory freely available is a lot of fun and a very useful learning experience (both for us and the users
).
One of the challenges for us is to deciding the right set of defaults. We are trying as hard as we can to not have a bunch of options to detract from the basic simplicity of JUnit Factory.
One of the things we keep finding is that JUF rewards good coding behavior. I have a slogan at Agitar:
Ugly Code –> Ugly Tests.
Of course, we cannot yet claim: Beautiful Code –> Beautiful Tests (as in beautiful code *guarantees* beautiful tests), but we are working on it … one code pattern and test heuristic at a time.
I also keep stressing the importance of looking at JUnit Factory and automated test generations in general as: test *amplification* tools. We can do *some* magic if all you do is press a button, but the real payoff is when you write, say, 40-50 lines of test helpers (test data helpers, set-up helpers, and assertion helpers) and in return you get 400-500 lines of really good tests.
As far as uploading the entire project goes, that’s the inherited behavior from our commercial product AgitarOne. The benefit is that you pay a one-time “penalty” the first time you use it on a project but, after that, you only upload incremental changes and it’s very fast. We are looking into having a more incremental loading approach in the future.
As you reminded your readers, JUnit Factory is still very experimental, so keep experimenting and let us know how we can make it more useful.
Thanks again for taking the time to write a thoughtful review.
Alberto
Posted 08 Apr 2007 at 8:07 pm ¶
Kevin Lawrence wrote:
Great review, thanks Steve. Feedback on JUnit Factory is what we need most of all.
The low coverage is probably related to the security exceptions. JUF makes tests that will be safe to run on your client so file deletions and socket connections to random ports are disallowed.
Kevin
Posted 10 Apr 2007 at 5:08 pm ¶
Steve Bate wrote:
Hello Alberto and Kevin,
Thanks for the comments. The way JUnitFactory “studies” the code and creates tests is very interesting. I can understand why it would be a fun product to work on.
Posted 11 Apr 2007 at 1:13 am ¶
Jörg Thönnes wrote:
Hi Steve,
just found this review. I know Agitar from a very interesting presentation at the JAX-W conference in Munich, Germany in November 2006.
I found the concept appealing and a interesting complementation to the normal way to write unit tests. From this experience I support Albertos statement of “test amplification.”
On the other hand, I see your points of being to dependent on the implementation of the class. Could facades help here?
In summary, I did not try any Agitar products yet, but would love to do so if I find some time. Until then, my view stays a theoretical one…
Cheers, Jörg
Posted 18 Jul 2007 at 7:04 am ¶