Returning Early from ASP.NET Requests

I have great reservations about writing this blog post. Pretty much everything I’m going to describe here is a bad idea and you should strongly avoid putting it into production, but there are just a few situations where this technique can be really helpful.

As I described in Async Doesn’t Change the HTTP Protocol, in the ASP.NET worldview you only get one “response” for each “request”. You can’t return early just by using an await. However, in some situations you have enough information to generate the response but the actual request processing may take some more time. That’s were today’s solution comes in.

Not Recommended

The solution in this blog post is not recommended. Before putting it into production, you need to understand why it’s not recommended.

ASP.NET executes your web site (or web application) in an AppDomain, separated from other web sites (or web applications) on the same server. There are many reasons why this AppDomain may be shut down; modern versions of IIS recycle the entire process every 29 hours by default just to keep things clean. Also, you have to take into consideration unmanaged shutdowns: hard drive failures, hurricanes, etc.

Consider what happens if you generate (and return) the response but you’re still working on the request. If you lose your AppDomain for any reason, that in-progress work is lost. The client thinks it was completed, but it really wasn’t. As long as the request is incomplete, the responsibility is on the client. When you complete the request (by sending a response), you have accepted the full responsibility of that request. If you haven’t already committed the changes, you need to be absolutely sure that they will be committed.

Proper Solutions

The correct solutions are all complicated: you need to put the additional work in a safe place, like an Azure queue, database, or persistent messaging system (Azure message bus, MSMQ, WebSphere MQ, etc). And each of those solutions brings a whole scope of additional work: setup and configuration, dead-letter queues, poison messages, etc.

But that’s the correct way to do it, because you can’t drop the ball! You store the additional work in the safe place and then return a response after the work is safely stored. Personally, I like distributed systems (like Azure queues) because it’s not just safely stored on the hard drive - it’s safely stored on six hard drives, three of which are in a different geographic location. This gives you more protection from more problems (like hard drive failures and hurricanes).

The Improper “Solution”

The unsafe way to do it is to keep the work in memory. The simple way to do this is to just toss the work into Task.Run. Unfortunately, ASP.NET has no idea if you have queued work like this, and it will feel free to take down your AppDomain when it thinks it’s idle.

The slightly safer but still unsafe way to do it is to keep the work in memory but register it with ASP.NET so that it will notify you when your AppDomain is going away. The code in this blog post uses the technique described by Phil Haack to register work with the ASP.NET runtime. It’s important to note the limitations of this approach:

• By default, you only have 30 seconds total from the time the notification goes out to the time the AppDomain is yanked out from under you.

  As noted in the comments, the ASP.NET runtime will wait an arbitrary amount of time for your background tasks to complete. Still, it’s probably best not to keep them waiting…

• You may not get notification at all. In an unmanaged shutdown (e.g., power loss), all bets are off.

Still, this approach can be useful in a limited set of scenarios, so with great reservation let’s take a look at the code.

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Hosting;
using Nito.AsyncEx;

/// <summary>
/// A type that tracks background operations and notifies ASP.NET that they are still in progress.
/// </summary>
public sealed class BackgroundTaskManager : IRegisteredObject
{
    /// <summary>
    /// A cancellation token that is set when ASP.NET is shutting down the app domain.
    /// </summary>
    private readonly CancellationTokenSource shutdown;

    /// <summary>
    /// A countdown event that is incremented each time a task is registered and decremented each time it completes. When it reaches zero, we are ready to shut down the app domain. 
    /// </summary>
    private readonly AsyncCountdownEvent count;

    /// <summary>
    /// A task that completes after <see cref="count"/> reaches zero and the object has been unregistered.
    /// </summary>
    private readonly Task done;

    private BackgroundTaskManager()
    {
        // Start the count at 1 and decrement it when ASP.NET notifies us we're shutting down.
        shutdown = new CancellationTokenSource();
        count = new AsyncCountdownEvent(1);
        shutdown.Token.Register(() => count.Signal(), useSynchronizationContext: false);

        // Register the object and unregister it when the count reaches zero.
        HostingEnvironment.RegisterObject(this);
        done = count.WaitAsync().ContinueWith(_ => HostingEnvironment.UnregisterObject(this), TaskContinuationOptions.ExecuteSynchronously);
    }

    void IRegisteredObject.Stop(bool immediate)
    {
        shutdown.Cancel();
        if (immediate)
            done.Wait();
    }

    /// <summary>
    /// Registers a task with the ASP.NET runtime.
    /// </summary>
    /// <param name="task">The task to register.</param>
    private void Register(Task task)
    {
        count.AddCount();
        task.ContinueWith(_ => count.Signal(), TaskContinuationOptions.ExecuteSynchronously);
    }

    /// <summary>
    /// The background task manager for this app domain.
    /// </summary>
    private static readonly BackgroundTaskManager instance = new BackgroundTaskManager();

    /// <summary>
    /// Gets a cancellation token that is set when ASP.NET is shutting down the app domain.
    /// </summary>
    public static CancellationToken Shutdown { get { return instance.shutdown.Token; } }
    
    /// <summary>
    /// Executes an <c>async</c> background operation, registering it with ASP.NET.
    /// </summary>
    /// <param name="operation">The background operation.</param>
    public static void Run(Func<Task> operation)
    {
        instance.Register(Task.Run(operation));
    }

    /// <summary>
    /// Executes a background operation, registering it with ASP.NET.
    /// </summary>
    /// <param name="operation">The background operation.</param>
    public static void Run(Action operation)
    {
        instance.Register(Task.Run(operation));
    }
}

BackgroundTaskManager is a singleton that keeps track of background operations. It uses an AsyncCountdownEvent from AsyncEx as a counter of background operations (plus an extra count that is decremented when ASP.NET notifies us that the AppDomain is going down).

You can queue synchronous or asynchronous work by calling Run:

BackgroundTaskManager.Run(() =>
{
    Thread.Sleep(20000);
});
BackgroundTaskManager.Run(async () =>
{
    await Task.Delay(20000);
});

BackgroundTaskManager also publishes a CancellationToken that is canceled when ASP.NET notifies us that our AppDomain is shutting down. async code can use this to abort processing (when it is safe to do so):

BackgroundTaskManager.Run(async () =>
{
    await Task.Delay(20000, BackgroundTaskManager.Shutdown);
});

One important note about background operations: exceptions are ignored! So if you want to catch errors and toss a “hail Mary” to ETW or the Event Log, you’ll need to do so with a try ... catch inside each operation. In the example above, if the AppDomain is recycled while the operation is doing the delay, the cancellation exception will be raised from the operation and then it will be ignored.

As a final reminder, do not put critical processing in a background operation like this. It works fine for the “easy” case (ASP.NET gets a gentle request to shut down the AppDomain and nicely notifies the background operations, which all complete or cancel well within the timeout window), but it can fall down if anything goes wrong (IIS is killed, or the background operations continue too long due to another process hogging the CPU, or there’s a power outage, etc).