2012-02-07

Async Unit Tests, Part 2: The Right Way

Update: The information in this blog post only applies to Visual Studio 2010. Visual Studio 2012 will support asynchronous unit tests, as long as those tests are "async Task" tests, not "async void" tests.

Last time, we looked at incorrect approaches to async unit testing. We also identified the underlying problem: that unit tests do not have an appropriate async context.

At this point, the solution should be pretty obvious: give the unit tests an async context!

It really is that easy! Why, all you have to do is write your own SynchronizationContext implementation. Keep in mind that thread-safety is paramount, because the methods under test may interact with the thread pool or other async contexts. Note that async void methods interact with SynchronizationContext in a different way than other async methods. Oh, and also remember that exceptions need special handling in some cases so their original call stack is preserved appropriately, and if you're on VS2010 you'll need to hack this in because there's no support for it on .NET 4.0.

Just kidding! Ha, ha! The good folks on the Async team have done all the hard work for you. :)

Right Way #1: The Official Approach

If you have the Async CTP installed, then check out the "My Documents\Microsoft Visual Studio Async CTP\Samples\(C# Testing) Unit Testing\AsyncTestUtilities" folder. You'll find not just one, but three async-compatible contexts, ready for you to use!

You should use GeneralThreadAffineContext unless you absolutely need another one. To use it, just copy AsyncTestUtilities.cs, CaptureAndRestorer.cs, and GeneralThreadAffineContext.cs into your test project.

Then, take each unit test and re-write it so that it has a context:

[TestMethod]
public void FourDividedByTwoIsTwo()
{
  GeneralThreadAffineContext.Run(async () =>
  {
    int result = await MyClass.Divide(4, 2);
    Assert.AreEqual(2, result);
  });
}

[TestMethod]
[ExpectedException(typeof(DivideByZeroException))]
public void DenominatorIsZeroThrowsDivideByZero()
{
  GeneralThreadAffineContext.Run(async () =>
  {
    await MyClass.Divide(4, 0);
  });
}

Our unit test methods are not async. Each one sets up an async context and passes the actual test into it as an async lambda expression. So, the actual test code can still be written with all the benefits of async/await, and the async context takes care of making sure it runs as expected:

Just as importantly, the async context ensures that tests that should fail, will fail:

[TestMethod]
public void FourDividedByTwoIsThirteen_ShouldFail()
{
  GeneralThreadAffineContext.Run(async () =>
  {
    int result = await MyClass.Divide(4, 2);
    Assert.AreEqual(13, result);
  });
}

And everyone lived happily ever after!

Well, sort of. This solution does work, but it's a bit cumbersome. Copying code files into each test project? Modifying every unit test to set up its own async context? Really?

Right Way #2: Now with Less Effort!

Boy, if only there was some way to have the MSTest framework apply the async context for us, then we could just write async unit test methods and not worry about it!

Oh yeah - there is. Visual Studio allows you to define a custom "test type." It really is that easy! Why, all you have to do is... ah, forget it. A custom "async unit test" type is already available:

Sweet.

Now you can write async unit tests (using async void):

[TestMethod]
public async void FourDividedByTwoIsTwoAsync()
{
  int result = await MyClass.Divide(4, 2);
  Assert.AreEqual(2, result);
}

[TestMethod]
[ExpectedException(typeof(DivideByZeroException))]
public async void DenominatorIsZeroThrowsDivideByZeroAsync()
{
  await MyClass.Divide(4, 0);
}

And it works:

And test failures actually fail:

[TestMethod]
public async void FourDividedByTwoIsThirteenAsync_ShouldFail()
{
  int result = await MyClass.Divide(4, 2);
  Assert.AreEqual(13, result);
}

Sniff... It's... so... beautiful...

But not quite perfect. You still have to add a NuGet package and remember to change [TestClass] to [AsyncTestClass].

Tip: You can download an Async Unit Test item type which uses [AsyncTestClass] instead of [TestClass]. This makes writing new async tests just a little bit easier, but not entirely foolproof.

Future Directions

xUnit.NET has recently released first-class support for asynchronous unit tests: in version 1.9 (2012-01-02) and newer, for any test method returning Task/Task<T>, the test framework will wait until the task completes before declaring success/failure. However, as of now, it does not support async void unit tests; this is planned for a future release.

I've been in contact with some people inside of Microsoft regarding this issue, and they said they're aware of it and are considering various options. They wouldn't give me any details, of course, but they did suggest that I would be "pleasantly surprised" when Visual Studio vNext comes out.

So, that's where we are today. Hopefully Microsoft will ship built-in async unit test support in Visual Studio vNext, and I'll be able to look back at this blog post and laugh at how fraught with peril async unit testing used to be.

9 comments:

  1. As per my comment on your previous blog, Microsoft does ship built-in async unit test support in visual studio 2012... The fact that you aren't using it properly (don't use async void, ExpectedException sucks) isn't related :-)

    ReplyDelete
    Replies
    1. This blog post was written in February of 2012. Microsoft had not released VS2012 at that time, and there's an update at the top of this post pointing out that this information only applies to VS2010. VS2012 did not support async unit tests until their RTW.

      Delete
  2. Nito.AsyncEx.UnitTests.MSTest is not strongly named. :(

    ReplyDelete
    Replies
    1. It also hasn't been needed for more than a year (when VS2012 was released). This post is just kept up for historical reasons.

      Delete
    2. Well, it's not needed for VS2012, but if your build server is still TFS2010 there's still an issue here. However, Nito.AsyncEx.UnitTests.MSTest doesn't resolve issues with TFS2010--I ended up having to rewrite the tests to use your excellent TaskEx library to wrap calls to the async code under test.

      Delete
    3. Nito.AsyncEx.UnitTests.MSTest depends on the Async CTP for VS2010. At this point it's quite hard to get the Async CTP working. I.e., if you've been keeping your build server patched with Windows Update, the Async CTP cannot be installed on it. :/

      Still, I'm glad the AsyncEx library helped! I had to do something similar (using AsyncContext) just recently because the version of NUnit that Xamarin iOS/Droid currently uses does not support async unit tests...

      Delete
  3. Hi Stephen, thanks for the great post.

    Will this solution work for Silverlight 5 async unit tests?

    Thanks,

    ReplyDelete
    Replies
    1. I have not tried it, but I believe that MSTest in VS2012 and newer will support async Task unit tests for any target framework.

      Delete