Bugfree.dk – Ronnie Holm's blog

(This post grew out of a presentation I did on the subject. Turn to the slides for more details, including code.)

With Asp.Net, by default a page is rendered synchronously, one control at a time. The thread rendering the page comes from the thread pool within the world wide web worker process (w3wp.exe) and is assigned to the request when it comes in on the web server. Only when the page has fully rendered can the thread go back in the pool waiting for the next request.

Processing a complex page using only one thread, however, may cause the page to render slowly. The page may hold controls that require calls to web services, databases, or other external resources for it to render. Whenever the worker thread encounters such a control, it goes idle waiting for data to come back from the external resource. Only when data is received can the worker thread resume processing.

One way to speed up page processing would be for the page or control to explicitly make use of threads. Depending on the situation, this may be a cumbersome task that requires a significant amount of boiler-plate code. A simpler approach might be to turn to the Asp.Net 2.0 feature for asynchronous page processing. At its core is an extension to the page or control lifecycle.

Except for the Begin and End events, the flow is the same in both models. But while the synchronous model executes all code on a single thread, the asynchronous model makes implicit use of multiple worker threads to speed up page processing. With multiple threads, some threads still go idle waiting for input, but at least the main worker thread continues to work its way through the controls.

In the asynchronous model, we hook into the Begin and End handlers in one of the events preceding the call to Begin. Following PreRender, Asp.Net will then execute our Begin handler, where we call out to some method that carries out the long-running task. When that method exists the End handler is called and processing carries on as in the synchronous model.

To illustrate the flow and timing of the asynchronous model (turn to the slides for the complete code sample), suppose we add five controls to a page. Each control takes five seconds to render so using the synchronous model the page would take 25 seconds to render. Using the asynchronous model, and writing out trace information about which thread each method executes on, here’s what happens:

1: Page_Load: 10            9: DoWork: 8
2: Page_Load: 10           10: DoWork: 11
3: Page_Load: 10           11: Render: 11 15:16:13 15:16:18
4: Page_Load: 10           12: Render: 11 15:16:13 15:16:18
5: Page_Load: 10           13: Render: 11 15:16:13 15:16:18
6: DoWork: 4               14: Render: 11 15:16:13 15:16:18
7: DoWork: 10              15: Render: 11 15:16:13 15:16:19
8: DoWork: 9

The user control’s Page_Load and the code making up the Begin handler executes on the main worker thread. But the method carrying out the long-running task, and the code for the End handler, executes in parallel on different worker threads. Finally, after End is called, another worker thread takes over and executes the remaining parts of the control. Now page rendering has decreased from 25 to about five seconds.

All in all, little effort is required to make more efficient, and implicit, use of the thread pool, thereby decreasing the time required processing a page. Asynchronous page processing should be used judiciously, though. Optimizing away a bottleneck in the processing of a page may well lead to bottlenecks appearing elsewhere, e.g., the number of simultaneous calls made to a web service may be too much for it to handle.

For more details and examples, turn to Jeff Prosise’s article on Asynchronous Pages in Asp.Net 2.0.

Download source or watch demo.

(Other posts in this series include: Kernel space traffic shaping with Linux and User space traffic shaping with Ruby that touch exclusively on Linux issues.)

This post covers an Asp.Net MVC application (for an overview of Asp.Net MVC, the Herding Code guys recently interviewed Phil Haack) I wrote that automates the collection of information about computers and their owners on a local area network. The idea is to have the Internet gateway match the identity of each computer that wants to send traffic through it against a white list. Should the request originate from an unknown computer, the gateway redirects that computer to a web application for Internet access activation. Each user then has to create an account and provide verifiable contact information and/or associate their computer with an existing account.

With the contact information of every connected user, maintaining a network with 331 connected apartments, 371 users, and 513 privately owned computers becomes more manageable. Now you can easily email all users about general issues or individual users about specific issues, such as wireless routers exposing rogue DHCP servers or virus slowing down the network.

Without being able to contact a user, the best you can hope for is to use the MAC address to locate and disable the port on the switch to which the user is connected. But terminating network access without warning or explanation, the user has no way of knowing what hit him. Now when the user contacts you, figuring out if he’s experiencing network issues because of a closed port or something entirely different quickly becomes a challenge — and something that doesn’t scale well.

That’s where the Asp.Net MVC application comes in. To get a feel for how the software accomplishes its task, you should watch the minute and a half demo. It portrays the browsing experience of a user connecting his computer to the network for the first time and attempting to browse the web. Behind the scenes the Linux gateway uses its Netfilter capabilities to efficiently match the MAC address of each request against the white list. With no match for the computer just connected, upon browsing the web, Netfilter redirect the user’s browser to the web application. Then, every so often, using a Ruby script, the gateway queries the web server for updates to the white list and carries over the changes to Netfilter.

Database schema

The web application is based on the idea that a user is responsible for zero or more computers. Focusing on the web application, rather than the Linux part, the rest of this post highlights a few features that use and manipulate the data structures stored in a MS SQL Server Express database:

Reverse lookup of MAC

Establishing an ownership between a user and a computer is done via the MAC address of the computer. Now, it wouldn’t be particularly user friendly if the user had to go about locating the MAC address and manually typing it into a web form. Instead, the web application looks up the user’s MAC address within the ARP cache of the web server.

Because the user’s computer and the web server are both on the same local network and because they’re already exchanging data (the user is visiting a web page hosted on the server), the parties are effectively communicating through the ARP protocol of the data link layer. The data entry form is therefore able to present the user with a control that makes associating computers to the account easy. By reverse lookup, the computer currently browsing the web application is indicated by “Now logged in from here”.

Email verification

Most users don’t mind handing over basic information such as their email address. And as users are blocked from accessing the Internet, at the time of entry we only validate the email address for syntactic correctness.

A few users, however, repeatedly entered fake email addresses to gain Internet access. Consequently, we send out an email to the address, leaving open a 24 hour window for the user to follow a link. Otherwise, the software sends out a reminder email, marks the account as inactive, and redirects the user to the web application.

Validating the syntax of email addresses is done using regular expressions. As with most complex regular expressions, they’re hard to read and verify the correctness of. They do, however, stand the test against a database of hundreds of email addresses. To come full circle, though, what is needed is a way to match the domain part of the email address against the DNS MX record of the domain. Unfortunately, MX lookup isn’t build into the .Net framework.

Unhalted exception emailing error handler

Error handling deserves a post on its own. Suffice it to say that to stay on top of any application, unhalted exceptions should be logged, or in this case, emailed to a designated address. Inevitably, users of your software will use it in unanticipated ways and so a global exception handler provides invaluable insight to learn from.

(Summary part of example email. Click here for complete output.)

Download and maybe try out

The web application was developed around February 2008; about the time the Asp.Net MVC framework went into Preview 2 and had started gaining momentum. Unfortunately, developing against this early a release, now the web application doesn’t run on a computer with .Net 3.5 SP1 installed.

There’s no problem compiling the application because the Preview 2 bits are in the bin folder of the application. But running the application generates an exception stating:

   Could not load type 'System.Web.HttpContextWrapper2' from
   assembly 'System.Web.Abstractions' [located in the GAC]

That’s because the version of System.Web.Abstractions.dll that ships with .Net 3.5 SP1 no longer holds the HttpContextWrapper2 class. On a machine without .Net 3.5 SP1, however, Cassini runs the application just fine. Before running it, though, remember to create a database from the schema in Database.sql and point to that database from web.config.

Lastly, I should stress that the software is a prototype and served as a way for me to wrap my head around ASP.Net MVC and LINQ. So, the code may not win a beauty contest.

Conclusion

The activation software went online late March 2008. During the first couple of weeks I had to put in a couple of bug fixes based on what I learned from the unhalted exception emails. Since then, however, the software has served the purpose it was charged with.

The only issue that I haven’t been able to resolve is why, on rare occasions, redirecting a computer confuses its browser: if you visit a page and is redirected to the web application, for a short time after, the page keeps resolving to the web application. Most likely it’s a residual effect of the packet rewriting taking place on the gateway.

Another thing I wish I’d implemented was tracing to better understand how a couple of users got to experience a few exceptions that I don’t know how to interpret from the unhalted exception emails alone.

Download EmailingExceptionModule-1.0.zip.

In any real-world application, unhalted exceptions are an unavoidable fact of life. Such exceptions should occur relatively infrequently, but when they do some kind of supporting infrastructure better be in place to capture the when and why of the exceptions. As a developer, you should have the ability to perform a postmortem analysis on what went wrong and learn from it. Paramount to such analysis is capturing and logging sufficient information at the point of failure.

What this post covers is the thoughts and development of an Asp.Net HttpModule that captures and emails such unhalted exception information to a designated email address.

To be clear, the goal of the EmailingExceptionModule isn’t to prevent unhalted exception from occurring per se. Within code, a developer may not know how to handle an exception, and so the exception should rightfully propagate the call stack. In case no caller steps in and handles the exception, the best choice is most likely to terminate the application. It’s almost always better to be upfront with the user than conceal the possibly inconsistent state of the application.

So when does the EmailingExceptionModule come in handy then? One scenario is that of C# unchecked exceptions, where the compiler doesn’t force the caller to catch all types of exceptions thrown by the callee. Unchecked exceptions may materialize as type cast or null reference exceptions when accessing variables. What unchecked exceptions boil down to is that, given the cyclomatic complexity of some methods, it’s impractical to manually work through every conceivable path of execution ahead of time. Another scenario is problems with the runtime environment, such as out of disk space or database server down. Nonetheless, some map of the path to failure is helpful in preventing others going down that same path.

(Summary part of example email. Click here for complete output.)

With web applications, we can take advantage of the application running in a centralized environment. In the spirit of Microsoft’s Doctor Watson technology, we can register our own web application error handler and hook it into the Http request pipeline, and have the application execute our code on unhalted exceptions. In practice, such an error handler is implemented either through the Application_Error method of an application’s Global.asax or by loading an HttpModule into the application.

Hinted by the title, I went for the HttpModule. The reason may best be understood by taking a peak behind the .Net curtains: Global.asax is a filename hardwired into the framework so that whenever an application is first hit, the framework asks HttpApplicationFactory, an internal factory class, for an HttpApplication representing the application (there’s a unique one for each virtual application). As part of manufacturing the HttpApplication, the factory locates the application’s Global.asax, compiles it into a dynamically generated DLL, loads the DLL, and reflects over the Global.asax class looking for methods adhering to the convention of modulename_eventname. Methods found, such as Application_Error, are then stored in a list of MethodInfos and passed along to HttpApplication. Then, during HttpApplication initialization, events are bound to the methods within Global.asax.

That’s how Application_Error of Global.asax gets called even though the method isn’t overriding a base class implementation as is typical for the template pattern. That’s also why code within global.asax isn’t binary reusable across applications. An HttpModule, on the other hand, can be loaded into any number of applications.

In practice, implementing an HttpModule it a matter of creating a class that implements the IHttpModule interface:

   public interface IHttpModule {
      void Init(HttpApplication context);
      void Dispose();
   }

The crux of error handling with Asp.Net is the Error event of HttpApplication. The easiest way to hook up the event to a method is within the Init method of the class. In addition, we assign the application object to a field so that later we can access the most recently thrown exception through it:

   public class EmailingExceptionModule : IHttpModule {
      private HttpApplication _application;

      public void Init(HttpApplication a) {
         _application = a;
         _application.Error += OnUnhaltedException;
      }

      public void Dispose() {}

      private void OnUnhaltedException(object sender, EventArgs e) {
         Exception ex = _application.Server.GetLastError().InnerException;
         // Implementation left out for brevity. Please download source
      }
   }

The OnUnhaltedException method is where to add code that collects information about the exception and the environment, and that composes and ships the email. Depending on the environment, you may find the need to include additional information in the email. Given the source code, adding to the email is a matter of adding key/value pairs in the form of labels and values to a dictionary. Each dictionary then gets rendered as a table of the key and value columns.

To have an Asp.Net application load the EmailingExceptionModule and to configure its email related settings, add the following nodes to the application’s web.config:

   <configuration>
      <appSettings>
         <add key="EmailingExceptionModule_SenderAddress" value="sender@domain.com"/>
         <add key="EmailingExceptionModule_ReceiverAddress" value="receiver@domain.com"/>
         <add key="EmailingExceptionModule_SmtpServer" value="mail.domain.com"/>
         <add key="EmailingExceptionModule_SubjectPrefix" value="MyApp: "/>
      </appSettings>
      <system.web>
         <httpModules>
            <add name="EmailingExceptionModule" type="Holm.AspNet.EmailingExceptionModule,
                       EmailingExceptionModule, Version=1.0.0.0, Culture=neutral,
                       PublicKeyToken=13806f613f05e959"/>
         </httpModules>
      </system.web>
   </configuration>

So what’s the user experience of unhalted exceptions? Outside the confines of the EmailingExceptionModule, there’s no indication that the application took note of what happened. Depending on the user’s profile, it might be helpful to prompt for additional information at the crash point. Or perhaps turn to the Windows Live Messenger IM Control & Presence API and have the user engage in an MSN conversation with a developer.

What’s the advantage of emailing developers unhalted exceptions over storing the information in the file system or a database? I believe in keeping things simple and practical, and handing off information to the file system or a database isn’t. Most likely the information will end up gathering virtual dust on some server. The email approach, on the other hand, is a proactive, visible, and efficient means to the end of rapidly course-correcting for wrong assumptions.

Update, Sep 15: While you don’t need a PDB file to debug code through Visual Studio (that’s what <compilation debug=”true”> in web.config is for), the PDB file is required for the stack trace to contain line numbers. Since the PDB file typically has to reside next to the corresponding DLL for the .Net runtime to pick it up, the PDB file may need to be deployed to the GAC along with the DLL. Refer to the first three paragraphs of the “Debugging assemblies that live in the GAC” section of this post for how to GAC deploy PDB files.

For a centrally deployed Asp.Net application, it may be acceptable to ship and deploy the PDB file with the application. Keep in mind, though, that because the PDB file is a mapping of locations in the IL to the source file, having access to the PDB file makes it easier for someone to reverse engineer the code. It’s also possible to get at line numbers without the PDB file being deployed, but the approach is somewhat involved. Based on the mapping nature of the PDB file, the idea is to write out IL offsets as part of the exception and to post-process the offsets using a PDB file at a separate location.

Yet another alternative to shipping the PDB file is to setup a symbol server. With this nifty piece of code, the StackTrace class loads the PDB file of the symbol server, adding line numbers to the stack trace.

In this post I’ll rework a piece of code that changes the enclosing tag of a web control from a span to a div. A substitution of particular value with SharePoint, where adding web controls is a common way to extend the out of box experience. With SharePoint, and Asp.Net in general, a page is typically composed of reusable controls, each responsible for rendering its part of the page.

To showcase anti-patterns of API usage, intermediate code is included, i.e., code that may render the correct tags, although not necessarily in the true spirit of the .Net framework.

To set the stage, imagine writing a web control that inherits from ASP.Net WebControl. The goal is then to modify the behavior of the control in such a way that it (1) encloses its content in a div tag and (2) adds a CSS class for styling, and (3) can be added to a page like so:

   <MyControls:MyControl CssClass="myCssClass" runat="server" />

After making its way through the page rendering pipeline, the control tag gets transformed to:

   <div class="myCssClass">
      Hello World
   </div>

Within the control, a straight-forward way of achieving the intended output may be to hand craft the html and render it in RenderContents of WebControl. It almost goes without saying that rendering html by hand is but only a starting point when working with .Net. The next progression from there may be to override RenderBeginTag and RenderEndTag of WebControl:

   // version 1
   public class MyControl : WebControl {
      public override void RenderBeginTag(HtmlTextWriter w) {
         w.RenderBeginTag(string.Format("div class=\"{0}\"", CssClass));
      }

      public override void RenderEndTag(HtmlTextWriter w) {
         w.RenderEndTag();
      }

      protected override void RenderContents(HtmlTextWriter w) {
         w.Write("Hello World");
      }
   }

Resulting in this piece of html:

   <div class="myCssClass">
      Hello World
   </div class="myCssClass">

Clearly, RenderBeginTag and RenderEndTag of HtmlTextWriter don’t know how to separate tag from attribute. Nonetheless, the code reveals the stack based approach taken by the Render-pair of methods.

Gleaning over the documentation of HtmlTextWriter, however, I encountered the AddAttribute method for setting attributes on a tag. Inside HtmlTextWriter, what AddAttribute does is add attributes and associated values to an array that gets rendered during the next call to RenderBeginTag. So besides introducing strongly typed attributes and tags, the next version contains a modified RenderBeginTag:

   // version 2
   public override void RenderBeginTag(HtmlTextWriter w) {
      w.AddAttribute(HtmlTextWriterAttribute.Class, CssClass);
      w.RenderBeginTag(HtmlTextWriterTag.Div);
   }

Now the control renders as expected, although the code still has a certain bad smell to it. WebControl ought to know how to properly render its begin and end tag. After all it’s WebControl that exposes, among others, the CssClass property. So taking another dive into the documentation, I found the virtual, read-only TagKey property of WebControl. Strangely named as it may be, overriding TagKey makes WebControl call our override as it renders the control, adding in optional attributes, such as CssClass. Implementing the property also turns RenderBeginTag and RenderEndTag obsolete:

   // version 3
   protected override HtmlTextWriterTag TagKey {
      get {
         return HtmlTextWriterTag.Div;
      }
   }

As an added bonus, changing the enclosing tag can also be done calling the WebControl’s constructor, passing in the desired tag:

   // version  4
   public class MyControl : WebControl {
      public MyControl() : base(HtmlTextWriterTag.Div) { }

      protected override void RenderContents(HtmlTextWriter w) {
         w.Write("Hello World");
      }
   }

As it turns out, calling the constructor is how WebControl derived classes get to use span in the first place. The default WebControl constructor passes control onto another constructor that sets the value of what gets returned by the base class TagKey implementation. In essence, overriding TagKey makes it a player in the template design pattern:

   // .Net framework code
   protected WebControl() : this(HtmlTextWriterTag.Span) { }

   public WebControl(HtmlTextWriterTag tag) {
      tagKey = tag;
   }

Also worth noting is that for controls deriving from WebControl derived classes, such as CompositeControl, the constructor cannot be used to set the enclosing tag. The constructor exposed by CompositeControl and friends doesn’t call up the constructor chain passing in the tag, making TagKey the choice to go.

Finally, deep down the inhertitance chain below WebControl lives WebPart, a type of control very popular with SharePoint. Interestingly enough, WebPart defaults to using div as its enclosing tag, because walking up the chain, we come across the constructor of Panel:

   // .Net framework code
   public Panel() : base(HtmlTextWriterTag.Div) { }

So, using Panel provides another way to implement a web control that owns a distinct part of a page.