NUnit

Thursday, May 3rd, 2007

NUnit 2.4.1 Release

I just released NUnit 2.4. It fixes a few issues that slipped into the 2.4 release an makes some minor improvements. I wanted to get this release out because I’ll be travelling for a few months and won’t be in a position to do further releases until July or August.

You can review the release notes for more info about the contents of this release. You can download it here.

Friday, March 16th, 2007

NUnit 2.2.10 – Final 2.2 Release

Although NUnit 2.4 is almost out the door, there were a few outstanding bug fixes to the NUnit 2.2 series hanging around, so I decided to issue a final 2.2 release. Like all the releases since NUnit 2.2.4, this is primarily a bug fix release.

See the release notes for details of what has changed. You can download it here.

Since NUnit 2.4 is coming out in a matter of days, this release is for those who plan to stay with the 2.2 series for some time. Future work will concentrate on NUnit 2.4 enhancements and on planning for NUnit 3.0.

Sunday, March 11th, 2007

NUnit 2.4 Assert Syntax – the Latest Developments

NUnit 2.4 RC2 is out now, correcting a naming conflict with several mock object frameworks that was present in RC1. You can download it here. For a full list of the extensive new features in NUnit 2.4, check out the release notes.

One of those new features is the new constraint-based design for assertions, supported by a new syntax. That syntax, in fact, was the cause of the aforementioned naming conflict, which is now resolved. As a result, RC2 makes more changes than would normally be seen in a release candidate.

I’ve already blogged quite a bit about the new syntax. I originally developed it for NUnitLite, based on design concepts in NMock2. Since there were various experiments and numerous alternatives have been discussed, I’ll describe the whole thing here from scratch, as it is being implemented in NUnit 2.4.

Design Concepts

The “new syntax” is really a second order effect. What this change is really all about is encapsulating individual assertions as objects. We call these objects “constraints” in NUnit, and they encapsulate both a test and the generation of a message.

Readers who have been following the development of NUnit over the past few years will recognize that this is not the first such encapsulation. Back in 2004, we replaced the original procedural implementation of assertions with an object-oriented approach centered around “asserter” objects. Asserters encapsulated everything about an assertion: the test to be made, any test parameters, a message and the actual value being tested.

By eliminating the actual value from the encapsulation, and providing it separately in a method call, we are able to form compound constraints and then apply them to a single actual value. This is the essence of the design, which was arrived at through a series of spikes using various encapsulations.

In addition, the new design provides a MessageWriter, which is used internally by constraints. Most users will not need to use this interface, but those creating their own constraints will want to be familiar with it. There is a clear division of responsibility between the constraint and the writer. A constraint is responsibile for defining message content but the writer is responsible for the appearance of the message. NUnit 2.4 is delivered with one writer, TextMessageWriter, but additional writers supporting formats like html will be available in a future release.

Two Models

NUnit 2.4 supports both the old model of static assert methods and the new syntax. The documentation refers to the older syntax as the “classic model” and the new one as the “constraint-based” model. The classic model has now re-implemented using constraints, so the same underlying code is used no matter which syntax you use.

Some users have expressed concern that the older syntax will eventually fade away. There’s no plan to do that, and I don’t expect it to happen for a long time, if at all. The older syntax maps directly to other frameworks. For example, CollectionAssert uses the same methods and arguments as the class of that name in the Microsoft test framework. Even though the new syntax provides added power, many users will undoubtedly prefer the compatibility that the classic model gives them.

Constraint Objects

The constraint-based model uses the Assert.That method, with an actual value being tested as it’s first argument and a constraint object as the second. The most direct, although not necessarily most convenient, way to express a constraint is to construct the object in-line. The following example uses an instance of the EqualConstraint to perform a test…

Assert.That( 2 + 2, new EqualConstraint( 4 ) );

The same assertion can be written using a helper class to construct the constraint…

Assert.That( 2 + 2, Is.EqualTo( 4 ) );

NUnit 2.4 Supports a wide range of constraints and syntax helpers:

Syntax Helper Constraint Constructor
Is.Null EqualConstraint( null )
Is.True EqualConstraint( true )
Is.False EqualConstraint( false)
Is.NaN EqualConstraint( double.NaN )
Is.Empty EmptyConstraint()
Is.Unique UniqueItemsConstraint()
Is.EqualTo( object expected ) EqualConstraint( object expected )
Is.SameAs( object expected ) SameAsConstraint( object expected )
Is.GreaterThan( IComparable expected ) GreaterThanConstraint( IComparable expected )
Is.GreaterThanOrEqualTo( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.AtLeast( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.LessThan( IComparable expected ) LessThanConstraint( IComparable expected )
Is.LessThanOrEqualTo( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.AtMost( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.TypeOf( Type expected ) ExactTypeConstraint( Type expected )
Is.InstanceOfType( Type expected ) InstanceOfypeConstraint( Type expected )
Is.AssignableFrom( Type expected ) AssignableFromConstraint( Type expected )
Is.SubsetOf( ICollection expected ) CollectionSubsetConstraint( ICollection expected )
Is.EquivalentTo( ICollection expected ) CollectionEquivalentTo( ICollection expected )
List.Contains( object expected ) CollectionContainsConstraint( object expected )
Text.Contains( string expected ) SubstringConstraint( string expected )
Text.StartsWith( string expected ) StartsWithConstraint( string expected)
Text.EndsWith( string expected ) EndsWithConstraint( string expected )
Text.Matches( string pattern ) RegexConstraint( string pattern )
Has.Property( string name, object expected ) PropertyConstraint( string name, object expected )
Has.Length( int length ) PropertyConstraint( “Length”, length )
Is.Not.Xxxx, Has.Not.Xxxx, etc. NotConstraint( Xxxx )
operator ! NotConstraint( Xxxx )
Is.All.Xxxx, Has.All.Xxxx, etc. AllItemsConstraint( Xxxx )
operator & AndConstraint( Constraint left, Constraint right )
operator | OrConstraint( Constraint left, Constraint right )

Note that Not and All are used as prefixes to any of the other constraints. The AllItemsConstraint causes the following constraint to be applied to each item in a collection, succeeding only if the constraint succeeds on every item.

Examples of use:

Assert.That( new object[] { 1, 3, 5 }, Is.SubsetOf( new object[] { 5, 4, 3, 2, 1 } );
Assert.That( new string[] { "abc", "bac", "cab" }, Has.All.Length( 3 ) );
Assert.That( 2.0d + 2.0d, Is.EqualTo( 4.0d ).Within( .000005d ) );
Assert.That( "Hello World!", Text.StartsWith( "HELLO" ).IgnoreCase );

The last two examples illustrate the use of constraint modifiers. These are properties or methods of constraints, which are used to modify their operation. They are syntactically valid on all constraints and are ignored by those constraints that do not make use of them. The following constraint modifiers are supported.

  • AsCollection is recognized by EqualConstraint and causes arrays to be compared using the underlying collections, without regard to their respective ranks or dimension lengths.
  • IgnoreCase causes any string comparisons to be case insensitive. It is recognized by EqualConstraint as well as by all the Text constraints.
  • Within( tolerance ) is recognized by EqualConstraint when both values are a floating point type. It causes the comparison to succeed when the difference between the values is less than or equal to the tolerance.

To Inherit or Not

Beginning with version 2.0, NUnit eliminated the need to use inheritance to identify test classes. That policy continues with NUnit 2.4. Test fixtures are identified by use of the TestFixture attribute and users may develop whatever hierarchy of test classes they need. In fact, most large projects develop a set of base classes for use in defining their test fixtures.

With NUnit 2.4, we are introducing a class that is intended to be used as a base for individual fixtures or for entire project test hierarchies. Our choice of a name for this class, AssertionHelper, indicates both what it is and what it is not. It provides a number of methods that are useful in making assertions. It does *not* serve to identify a class as a test fixture in and of itself.

AssertionHelper provides an Expect method, which is identical to the Assert.That method, as well as equivalent methods to those supported by the syntax helpers.
Consequently, assuming the containing class inherits from AssertionHelper, the following assertions are equivalent:

   Assert.AreEqual( 1.5, myObject.calculate(), .00005 );
   Assert.That( myObject.calculate(), Is.EqualTo( 1.5 ).Within( .00005 ) );
   Expect( myObject.calculate(), EqualTo( 1.5 ).Within( .00005 ) );

What Next?

The constraint-based design will continue to be expanded in the final release of NUnit 2.4 and beyond. Where the syntax will go is another question. The underlying constraint model is flexible enough to support a variety of syntactic layers, and the community is just getting started at trying out these ideas.

We expect that many users will want to develop their own constraints and to layer alternate syntax on top of the constraint model. The best of these new ideas will eventually be incorporated either as extensions to NUnit or as part of the framework itself.

So try out the release candidate and then try your hand at writing constraints of your own. See the NUnit documentation or source for a description of the interfaces.

Tuesday, November 28th, 2006

NUnit 2.2.9

The release process for NUnit 2.4 has taken long enough that a number of bug fixes accumulated on the NUnit 2.2 codebase. I’ve just released NUnit 2.2.9, which addreses a number of bugs.

Read the release notes…

Tuesday, November 7th, 2006

NUnit Sample Extension Correction

As pointed out by Sara Lee McLindon, the SampleFixtureExtension example in the NUnit 2.4 Beta 2 release doesn’t actually work. You can correct it by adding the following method to the SampleFixtureExtensionBuilder class.

	
protected override TestSuite MakeSuite(Type type)
{
	return new SampleFixtureExtension( type );
}

Without this fix, the builder simply creates an NUnitTestFixture and no new behavior takes place when the test is run.

Alternatively, you can download a new copy of the affected source file here.

Wednesday, October 4th, 2006

The Tester Tested : Fit and NUnit

NUnit has lots of tests of its own – unit tests that is. It even has some fairly high-level tests that don’t fit well into the normal unit-testing paradigm. But, up to now, the only acceptance tests were manual – a list of things I do before uploading a new release. Since they are manual, they don’t get run all that often and surprises happen.

I’ve been meaning to do some experimenting with writing Fit tests for NUnit, and finally got some time last weekend. You can see the results of my initial experiments here.

For each row, the Fit fixture takes a snippet of code out of the first column, compiles it using the ICodeCompiler interface, loads the resulting assembly into an AppDomain and runs the tests. In addition to verifying that the code compiles, it checks the shape of the test tree that is produced and verifies that the correct number of tests were run, skipped or ignored and if the count of successes and failures are as expected.

This particular fixture verifies that tests written by a programmer using standard NUnit syntax will behave as expected. Of course, we need other sorts of tests, including those that test the various execution options of NUnit. This seems like a good place to start though.

Charlie

Monday, October 2nd, 2006

More on Assert Syntax

In an earlier post, I presented some ideas about syntax for expressing assertions in tests. I was doing this as a part of the development of NUnitLite, with the idea of eventually putting some of the same concepts back into NUnit.

I received many useful comments. Some of them are still percolating in my brain, but others have already found there way into the implementation of NUnitLite. As a way of paying back, I thought I would tell you here how I ended up resolving some of the issues we discussed.

NUnitLite Syntax by Example

Here are some valid NUnitLite Asserts, assuming proper declaration of variables…

  Assert.That( result, Is.EqualTo( 4 ) );
  Assert.That( result, Is.Not.EqualTo( 4 ) );
  Assert.That( result, !Is.EqualTo( 4 ) );
  Assert.That( obj, Is.Type( typeof(string) ) & Is.EqualTo( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "HELLO" ).IgnoreCase );
  Assert.That( array, Is.EqualTo( new int[] { 1, 2, 3, 4 } ) );
  Assert.That( matrix, Is.EqualTo( new int[] { 1, 2, 3, 4 } ).AsCollection );
  Assert.That( collection, Contains.Item( obj ) );
  Assert.That( collection, Contains.Item( "HELLO" ).IgnoreCase );
  Assert.That( collection, Is.All.Type( string ) );
  Assert.That( collection, Is.All.Not.Null & Is.All.Type( string ) );
  Assert.That( obj, new UserConstraint(arg) );
  Assert.That( obj, XXX.UserConstraint(arg) );

Roads Taken and Not Taken

I started out trying to make the syntax more terse. However, several people pointed out that modern IDEs provide auto-completion in context, which makes the length of an expression relatively unimportant. In addition, at least one non-English speaker indicated that abbreviations like eq, gt, etc. can be confusing.

I spiked expressions like

  Assert.That( 2+2, Is.GreaterThan(3).And.LessThan(5) );

While this is definitely doable, it’s more complex and I don’t find it very readable. That could be just me, of course. In addition, while it can be implemented as a simple
operator precedence grammar, a simplistic implementation doesn’t provide type safety. That is, you can end up compiling expressions like This.And.And.That or And.That. The current implementation won’t compile such invalid sequences and Visual Studio’s Intellisense only prompts for valid continuations.

The postfix operators like IgnoreCase are what I call Modifiers. They are simply getters that modify the state of the underlying object and return this.

The two final examples illustrate how one might extend the Syntax. UserConstraint is some user-defined test that implements the IConstraint interface. It can be used ‘raw’ as in the first of the examples, or hidden behind some syntactic sugar by defining a helper class similar to NUnitLite’s Is class.

That’s where it is so far… You can download pre-release code from Codeplex. Or watch for an 0.1 release soon.

Charlie

Saturday, September 23rd, 2006

Moving to WiX

NUnit has finally gotten rid of it’s Visual Studio Install projects in favor of WiX

VS projects support only a simple, single-feature install. What’s more, if you outgrow your install project, you can’t simply migrate it to another tool. You have to start over with the new tool. It would be cool if VS could use WiX syntax as it’s internal format, but I suspect that’s not likely to happen since it would tie future development of the two together in some ways that would probably discomfit both WiX and Visual Studio. But an open source converter from the vdproj format to WiX would definitely be a cool thing!

One side effect of this change is that users of C# Express can now load and build the entire NUnit solution. I’m hoping this will gain us a few volunteers.

Jamie Cansdale – of NUnit Addin and TD.Net fame – did the initial implementation of the WiX scripts and I made additional changes and figured out how to use the the NAnt tasks so it could be made part of our standard build.

It’s unfortunate that the WiX tasks for NAnt come without documentation. By reading the source, I was able to construct the commands I listed below, which I’m posting in the hope that they will be of help to somebody.

<candle out="${wix.work.dir}/" exedir="${wix.dir}">
  <defines>
    <define name="ProductVersion" value="${package.version}" />
  </defines>
  <sources basedir="${project.install.dir}">
    <nclude name="bin.wxs" />
    <include name="nunit-gui.wxs" />
    <include name="doc.wxs" />
    <include name="tests.wxs" />
    <include name="samples.wxs" />
    <include name="NUnit.wxs" />
  </sources>
</candle>

<light exedir="${wix.dir}"
  out="${project.package.dir}/${msi.file.name}" 
  locfile="${wix.dir}/WixUI_en-us.wxl">
  <sources>
    <include name="${wix.work.dir}/NUnit.wixobj" />
    <include name="${wix.work.dir}/bin.wixobj" />
    <include name="${wix.work.dir}/nunit-gui.wixobj" />
    <include name="${wix.work.dir}/doc.wixobj" />
    <include name="${wix.work.dir}/samples.wixobj" />
    <include name="${wix.work.dir}/tests.wixobj" />
    <include name="${wix.dir}/wixui.wixlib" />
  </sources>
</light>

You’ll find our WiX scripts in the latest version of the NUnit source on SourceForge.

Charlie

Friday, September 1st, 2006

More On Syntax: Expected Exceptions

Sometimes you expect an exception to be thrown by a method. So, of course, you want a test for that. NUnit provides the ExpectedExceptionAttribute for that purpose. It has a bit of history…

First, it was a list of exception types. That’s right, in NUnit 1.x, you could specify multiple expected exceptions. At some point, somebody thought “Wait, shouldn’t you know what you’re testing for?” and it was reduced in NUnit 2.0 to a single exception type.

[Test, ExpectedException(typeof( MyException ) )]

Then there was the message. Folks wanted to check for a single exception, so a constructor argument was added that allowed you to specify the message.

[Test, ExpectedException(typeof( MyException ), "My special message" )]

But some messages are pretty long, so you might want to specify a substring…

[Test, ExpectedException(typeof( MyException ), "special", MessageMatch.Contains )]

This doesn’t even get into specifying the exception by name rather than type, using regular expressions, providing a custom user message in case of failure,… but I’m sure you get the idea. The ExpectedExceptionAttribute has been overloaded with options, but people keep asking for more, like checking the value of specific properties on a specific type of exception.

Working on NUnitLite has given me the chance to look at this with fresh eyes. The heart of the problem is that there is only so much you can express between the square brackets when you use an attribute. Properties may only be of certain types and only so much fits on the line in a readable way. On the other hand, it’s reasonably simple to check various properties of an exception in a catch block – or in any other code for that matter.

For NUnitLite, I’m trying a simpler approach. First of all, you can simply specify ExpectedException without any arguments…

[Test, ExpectedException]

That means you expect some sort of exception – but you aren’t saying what. I know that this isn’t a very good testing practice by itself, but there are uses for it, as we’ll soon see. Normally, you’ll want to specify an exception type…

[Test, ExpectedException( typeof(MyException) )]

This works slightly differently from NUnit. The test will succeed if MyException or any exception that inherits from it is thrown – NUnit requires the exact type.

Now suppose you want to verify that MyException is thrown and has an particular message and that a certain property it supports has been set to a certain value. If your test class implements the IExpectException interface, you are able to define a handler to validate the exception. Here’s the interface…

public interface IExpectException
{
        void HandleException( Exception ex );
}

Here’s an implementation to deal with the hypothetical example

public void HandleException( Exception ex )
{
        Assert.That( ex.Message, Is.EqualTo( "my special message" );
        MyException mex = (MyException) ex;
        Assert.That( mex.Param == "xxxxx" );
}

As I’ve experimented with this, I have found that it was rare for me to need more than one exception handler in a class. I can usually generalize one method sufficiently to make it work for all my needs. But, in some cases, there is a need for a different handler. In that case, you can specify the name of the handler method as an additional property of ExpectedException…

[Test, ExpectedException(Handler="AlternateHandler")]
or
[Test, ExpectedException(typeof(MyException), Handler="AlternateHandler")]

The method must exist and have the correct signature or a run-time error will be thrown.

This is as far as I’ve taken it. I can think of other ways to expand (read complicate) it, but I’m not doing that. Users can write anything they want in the handler and if something turns out to be needed repeatedly we can think about removing the duplication later.

Charlie

Wednesday, August 30th, 2006

Thinking About Assert Syntax

I’ve been using the development of NUnitLite (coming soon!) as an excuse to try out alternatives to the standard NUnit syntax for Asserts.

NUnit allows us to write things like…

Assert.AreEqual( expected, actual );
Assert.Greater( x, 5 ); Assert.Less( x, 10 );
Assert.IsTrue( x > 5 && x < 10 );

The last is instructive: if it fails, the message simply tells us that True was expected but the value was actually false. Any more intelligent message must be supplied as an argument...

Assert.IsTrue( x > 5 && x < 10, "Value of x must be < 5 and > 10" );

An alternative idea, borrowed from NMock, is to encapsulate the comparison in some sort of an object. Then you can write things like...

Assert.That( actual, new EqualsMatcher( expected ) );
Assert.That( x, new AndMatcher( 
	new (GreaterThanMatcher( 5 ), new LessThanMatcher( 10 ) ) );

Of course, these aren't too easy to type, but with the addition of some helper methods and the use of operator overloading, we can replace it with

Assert.That( actual, Is.EqualTo( expected ) );
Assert.That( x, Is.GreaterThan( 5 ) & Is.LessThan( 10 ) );

That pretty much covers what I've implemented so far. I'm not fully satisfied with the expressiveness of this syntax, though. It's clear but a bit wordy. I'm thinking of going back to a set of inherited methods to either replace or supplement the methods of the Is class. That might give me something like this...

Assert.That( actual, eq( expected ) );
Assert.That( x, gt( 5 ) & lt( 10 ) );

So what do folks think? Is it worth pursuing this syntactic approach? Or should we just go back to Assert.AreEqual and friends?

Charlie