Term store overview

SharePoint organizes terms in either a taxonomy or a folksonomy. In a taxonomy, terms are organized hierarchically whereas in a folksonomy the structure is flat. A taxonomy or folksonomy is either global, making it available to all site collections within a web application that’s associated with a Managed Metadata Service, or local, making it available in one site collection only.

Multiple Managed Metadata Services may be associated with a web application or site collection, each service in turn responsible for a single term store. Within one term store, multiple groups may be created, each potentially storing multiple term sets. Finally, a term set may have terms nested up to seven levels deep:

Managed Metadata Service Application
  TermStore
    Group1
      TermSet1
        Term1
          Term1.2
            Term1.2.3
              Term1.2.3.4
                Term1.2.3.4.5
                  Term1.2.3.4.5.6
                    Term1.2.3.4.5.6.7

You manage the term store through the browser and its Term Store Management Tool (the _layouts/termstoremanager.aspx application page) or through the SharePoint API. The management tool supports importing term sets only from a special CSV file format  (_layouts/1033/ImportTermSet.csv holds a sample) with these columns:

Term Set Name
Term Set Description
LCID
Available for Tagging
Term Description
Level 1 Term
Level 2 Term
...
Level 7 Term

The CSV file is incomplete in terms of what the term store offers. It doesn’t offer columns for specifying translations and synonyms for terms. But users can use Wictor Wilén Excel sheet to maintain the basic information. It has the standard columns above and includes macros to simplify saving in the correct format and creating additional term sets.

The question remains: do you maintain compatibility with the existing CSV format so you can import term sets using the browser or a forms-based tool like the CSV Bulk Taxonomy Importer/Exporter? Or do you define your own, possibly XML based format since term sets are hierarchical by nature, with additional columns for translations and synonyms? A non-CSV based format may also be easier to process. If you want to pursue the latter approach, Ben Robb has created a PowerShell script to import his XML format.

Importing term sets using PowerShell

I decided to stick with the CSV file format and create a PowerShell script that would allow me to import a term set using the command-line:

function ImportTermSet([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName, [PSCustomObject]$termSet) {
  function ImportTerm([Microsoft.SharePoint.Taxonomy.Group]$group,
                      [Microsoft.SharePoint.Taxonomy.TermSet]$set,
                      [Microsoft.SharePoint.Taxonomy.Term]$parent,
                      [string[]]$path) {
    if ($path.Length -eq 0) {
      return
    } elseif ($group -eq $null) {
      $group = $store.Groups | where { $_.Name -eq $path[0] }
      if ($group -eq $null) {
        $group = $store.CreateGroup($path[0])
      }
    } elseif ($set -eq $null) {
      $set = $group.TermSets | where { $_.Name -eq $path[0] }
      if ($set -eq $null) {
        $set = $group.CreateTermSet($path[0])
      }
    } else {
      $node = if ($parent -eq $null) { $set } else { $parent }
      $parent = $node.Terms | where { $_.Name -eq $path[0] }
      if ($parent -eq $null) {
        $parent = $node.CreateTerm($path[0], 1033)
      }
    }

    ImportTerm $group $set $parent $path[1..($path.Length)]
  }

  function RemoveTermGroup([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName) {
    $group = $store.Groups | where { $_.Name -eq $groupName }
    if ($group -ne $null) {
      $group.TermSets | foreach { $_.Delete() }
      $group.Delete()
      $store.CommitAll()
    }
  }

  RemoveTermGroup $store $groupName
  $termSetName = $termSet[0]."Term Set Name"
  $termSet | where { $_."Level 1 Term" -ne "" } | foreach {
    $path = @($groupName, $termSetName) + @(for ($i = 1; $i -le 7; $i++) {
    $term = $_."Level $i Term"
      if ($term -eq "") {
        break
      } else {
        $term
      }
    })

    ImportTerm -path $path
  }
}

The way to use the ImportTermSet function is best illustrated with an example:

$session = Get-SPTaxonomySession -Site "http://localhost"
$store = $session.TermStores["Managed Metadata Service"]
$termSet = Import-Csv "C:\Users\Ronnie\Desktop\ImportTermSet.csv"
ImportTermSet $store "MyGroup" $termSet
$store.CommitAll()

In its current form the script ignores the Term Set Description, LCID, Available for Tagging, and Term Description columns from the CSV file. It would be possible to take these columns into account by passing them to ImportTerm as well. Instead of just removing the head of the list, you’d remove the number of items consumed by the current step before recursing.

type Proposition =
    | True
    | And of Proposition * Proposition
    | Or of Proposition * Proposition
    | Not of Proposition

In an object-oriented language like C#, this definition would be analogous to a hierarchy of classes. Proposition would be the base class of True, And, Or, and Not. Each would then have a constructor that accepts some number of arguments, e.g., the And constructor would accept a two-tuple, two propositions, making the Proposition type recursive. In C# you might even model the recursive relationship using the Composite pattern.

In F#, an instance of the Proposition type may be created using a constructor syntax that resembles that of a function call:

let p = And(Or(True, Not(True)), Not(Not(True)))

Now, because of the recursive nature of the Proposition type, instances of it form a parse tree-like structure:

                        And
                        /\
                       /  \
                      Or  Not
                      /\    \
                     /  \    \
                  True  Not  Not
                          \    \
                           \    \
                          True  True

Having a parse tree is not very useful in itself. What you need is some way to execute or evaluate it. Given the recursive nature of the grammar, it makes sense for the evaluator to be recursively defined as well. The evaluator would do pattern matching on the node type, deconstructing it into the parts needed to evaluate it. Since our language of propositions maps directly to the boolean operations of F#, it’s very simple to evaluate the parse tree:

let rec eval (p: Proposition) =
    match p with
    | True -> true
    | And(p1, p2) -> eval p1 && eval p2
    | Or (p1, p2) -> eval p1 || eval p2
    | Not(p1) -> not (eval p1)

let e = eval p // e : bool = true

The simple language of propositions doesn’t take into account operator precedence and associativity, variables, and so on. But the features of F# used here allow for more complicated languages to be expressed and evaluated directly. With C#, you’d often fallback to describing the grammar of more complex languages in Backus-Naur form and feeding it to a parser generator that would map the node types of the parse tree to classes and be able to form a valid parse tree of some input.

For the generic runner to work, each command has to fulfill the contract.

public enum ExitCode {
    Success = 0,
    Failure
};

public interface ICommand {
    string Usage { get; }
    string Description { get; }
    ExitCode Execute(string[] args);
}

I want the runner to adhere to the open/closed principle. For its behavior to be modified without altering its core delegation logic. This requires the use of reflection to retrieve and instantiate a command based on command-line arguments.

class Program {
    static IEnumerable<ICommand> GetCommands() {
        var iCommand = typeof (ICommand);
        return System.Reflection.Assembly.GetExecutingAssembly().GetTypes().ToList()
            .Where(t => iCommand.IsAssignableFrom(t) && t != iCommand)
            .Select(t => Activator.CreateInstance(t) as ICommand);
    }

    static void DisplayHelp() {
        Console.WriteLine("Console [Command] [Arg1] [Arg2] [ArgN]\n\n");
        GetCommands().ToList().ForEach(command =>
            Console.WriteLine(command.Usage + "\n" + command.Description + "\n\n"));
    }

    static int Main(string[] args) {
        if (args.Length == 0) {
            DisplayHelp();
            return (int)ExitCode.Failure;
        }

        var commandName = args[0];
        var command = GetCommands().Where(t => t.GetType().Name == commandName).SingleOrDefault();
        if (command == null)
            throw new ArgumentException(string.Format("Command '{0}' not found", commandName));

        var executeArguments = new List<string>(args);
        executeArguments.RemoveAt(0);

        var exitCode = command.Execute(executeArguments.ToArray());
        return (int)exitCode;
    }
}

A trivial example of a command that adds two numbers would be the following:

// $> GenericRunner.exe Calculator 2 3 => 2 + 3 = 5
public class Calculator : ICommand {
    public string Usage {
        get { return "Calculator [Op1] [Op2]"; }
    }

    public string Description {
        get { return "World's simplest calculator"; }
    }

    public ExitCode Execute(string[] args) {
        try {
            Console.WriteLine(
                string.Format(
                    "{0} + {1} = {2}",
                    args[0], args[1], int.Parse(args[0]) + int.Parse(args[1])));
            return ExitCode.Success;
        } catch (Exception e) {
            Console.WriteLine(e.ToString());
            return ExitCode.Failure;
        }
    }
}

Now multiple smaller assemblies can be grouped into one, with a description of all commands automatically being assembled, and without commands interfering (too much) with each other.

namespace Dk.Bugfree

open System
open System.Globalization
open Microsoft.SharePoint

type public ListAttachmentVersioningEventReceiver() =
    inherit SPItemEventReceiver()

    member private r.CustomVersion = "CustomVersion"
    member private r.ShadowLibrary = "ShadowLibrary"

    // override ItemAdded : properties:SPItemEventProperties -> unit
    override r.ItemAdded properties =
        base.ItemAdded properties
        r.SetCustomVersionLabel properties.ListItem
        r.CreateSnapshot properties

    // override ItemUpdated : properties:SPItemEventProperties -> unit
    override r.ItemUpdated properties =
        base.ItemUpdated properties
        let item = properties.ListItem

        if r.RollbackHappened item then
            r.RestoreSnapshot properties
            r.SetCustomVersionLabel item
            r.CreateSnapshot properties
        else
            r.CreateSnapshot properties
            r.SetCustomVersionLabel item

    // member private CreateSnapshot : properties:SPItemEventProperties -> unit
    member private r.CreateSnapshot properties =
        use site = properties.OpenWeb()
        let item = properties.ListItem
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let path = String.Format("Versions/{0}/{1}", item.ID, r.GetOfficialVersionLabel(item))
        let shadowFolder = r.CreateFolderPath shadowLibrary path

        item.Attachments |> Seq.cast |> Seq.iter (fun fileName ->
            let existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName)
            let newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream())
            newFile.Item.Update())

    // member private RollbackHappened : item:SPListItem -> bool
    member private r.RollbackHappened item =
        let culture = CultureInfo.InvariantCulture
        let currentVersion = Single.Parse(r.GetOfficialVersionLabel(item), culture)
        let lastVersion = Single.Parse(r.GetCustomVersionLabel(item), culture)
        currentVersion > lastVersion + 1.0f

    // member private RestoreSnapshot : properties:SPItemEventProperties -> unit
    member private r.RestoreSnapshot properties =
        let item = properties.ListItem
        let restoreVersion = r.GetCustomVersionLabel item
        r.EventFiringEnabled <- false    

        item.Attachments |> Seq.cast |> Seq.map (fun fileName -> unbox<string> fileName) |> Seq.toList
                         |> Seq.iter (fun fileName -> item.Attachments.Delete(fileName))

        use site = properties.OpenWeb()
        let path = String.Format("Versions/{0}/{1}", item.ID, restoreVersion)
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let source = r.CreateFolderPath shadowLibrary path

        source.Files |> Seq.cast |> Seq.iter (fun file ->
            let unboxedFile = unbox<SPFile> file
            item.Attachments.Add(unboxedFile.Name, unboxedFile.OpenBinary()))

        item.SystemUpdate false
        r.EventFiringEnabled <- true

    // member private CreateFolderPath : list:SPDocumentLibrary -> path:string -> SPFolder
    member private r.CreateFolderPath list path : SPFolder =
        r.CreateFolderPathRecursive list.RootFolder (path.Split [|'/'|] |> Array.toList)

    // member private CreateFolderPathRecursive : folder:SPFolder -> pathComponents:string list -> SPFolder
    member private r.CreateFolderPathRecursive folder pathComponents =
        match pathComponents with
        | [] -> folder
        | head :: tail ->
            try
                let existingFolder = folder.SubFolders.[head]
                r.CreateFolderPathRecursive existingFolder tail
            with
                :? ArgumentException ->
                    let newFolder = folder.SubFolders.Add head
                    r.CreateFolderPathRecursive newFolder tail

    // member private SetCustomVersionLabel : item:SPListItem -> unit
    member private r.SetCustomVersionLabel item =
        r.EventFiringEnabled <- false
        item.[r.CustomVersion] <- r.GetOfficialVersionLabel item
        item.SystemUpdate false
        r.EventFiringEnabled <- true  

    // member private GetCustomVersionLabel : item:SPListItem -> string
    member private r.GetCustomVersionLabel item =
        item.[r.CustomVersion] :?> string

    // member private GetOfficialVersionLabel : item:SPListItem -> string
    member private r.GetOfficialVersionLabel item =
        item.Versions.[0].VersionLabel

A couple of things to note about the F# implementation: first, it hardly specifies any types. They’re inferred by the compiler. Where type names do appear, it’s mainly because they’re required to unbox elements of an IEnumerable collection. Secondly, F# has flexible self identifiers. Methods must explicitly specify the name of the this reference in C# and use it when accessing members. Thirdly, arguments to general .NET methods are passed as a tuple value, i.e., as comma-delimited arguments surrounded by parenthesis.

This would involve adding or removing event receivers from a list as the user enables or disables versioning. The following extension method is one way to accomplish the addition-part in a type-safe manner:

// definition
public static class SPListExtensions {
    public static void RegisterEventReceiver<TReceiver>(this SPList list,
            SPEventReceiverType receiverType,
            int sequenceNumber) where TReceiver : SPItemEventReceiver {
        var assemblyName = typeof(TReceiver).Assembly.FullName;
        var className = typeof(TReceiver).FullName;

        (from SPEventReceiverDefinition definition in list.EventReceivers
         where definition.Assembly == assemblyName &&
               definition.Class == className &&
               definition.Type == receiverType
         select list.EventReceivers[definition.Id])
        .ToList()
        .ForEach(receiverToDelete => receiverToDelete.Delete());

        var receiver = list.EventReceivers.Add();
        receiver.Type = receiverType;
        receiver.Assembly = assemblyName;
        receiver.Class = className;
        receiver.SequenceNumber = sequenceNumber;
        receiver.Update();
        list.Update();
    }
}

// use
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemAdded, 10000);
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemUpdated, 10001);

Under rare circumstances the (assembly, class, type) tuple may not be unique, i.e., the same receiver may be registered multiple times, albeit with different sequence numbers. In practice I never found any use for this functionality, though, which is why I didn’t include the sequence number in the where clause above, causing all registrations matching the tuple to be removed.

Both SharePoint 2007 and 2010 support versioning for list items but not their attachments. No matter which version of a list item I look at, its attachments will always be the most recent. The attachment support seems to have been bolded on as an afterthought, resulting in behavior that’s counter-intuitive for developers as well as end-users. With SharePoint 2007 (and 2010), Microsoft suggests using a document library for proper attachment versioning. But I can’t substitute one with the other, since a list item may hold any number of attachments and an item in a document library may hold just one.

Existing solution

I counted on someone else having experienced a similar pain and come up with a workable cure. But except for Tim Ebenezer the search come up empty. Tim on the other hand has done a great job of seamlessly integrating his attachment versioning feature into SharePoint. When I activate the feature on a site, it adds a versioning menu item to the list settings page for every list on the site. Unfortunately the core versioning logic, storing attachments in a shadow library using an event receiver, isn’t particularly robust. Among other use cases, it doesn’t properly deal with a user first deleting an attachment and then, some versions later, adding an attachment with the same name.

I therefore set out to implement my own solution based on Tim’s ideas, hooking into the synchronous ItemAdding, ItemUpdating, ItemAttachmentAdding, and ItemAttachmentDeleting events and maintaining a shadow library of versions. This approach, however, quickly turned into a painful one. When the synchronous events run, nothing has yet been written to the database – at this stage a new item doesn’t even have its Id set, and merely determining the number of attachments added and how far I’ve come with the processing is tricky.

The next challenge I encountered was that event handlers cannot easily share state across multiple calls because SharePoint creates a new instance of the receiver class for every event handled. Processing multiple attachments require a counter into the array of attachments to keep track of which ones I’d copied to the shadow list. I’d have to resort to some outside-object storage, keeping in mind that the receiver might execute concurrently. But which storage should I use? Session state may have been disabled, and polluting one of the property bags stored in the content database is messy and also not thread-safe.

Overall, with the synchronous approach too much work has to go into tracking the state of the versioning process.

New solution

A synchronous solution is very hard to get right because it’s forced to work at the level of individual attachments. SharePoint doesn’t have a synchronous event that fires after all attachments have been processed. After all, why provide such an event when everything has already happened? Thinking instead in terms of the asynchronous events of ItemUpdated and ItemAdded, I have exactly what’s needed to snapshot all attachments in one batch, making versioning a lot simpler. When these events fire the item and its attachments have already been written to the database and I can focus on how to generate the snapshots — copying attachments back and forth between lists — and not worry about what the user actual did to the attachments from one version to the next.

// Prerequisites:
// 1. Create a Document Library named ShadowLibrary on the same site as the list to version
// 2. Add a row named CustomVersion of type string to the list to version
public class ListAttachmentVersioningEventReceiver : SPItemEventReceiver {
    private const string CustomVersion = "CustomVersion";
    private const string ShadowLibrary = "ShadowLibrary";

    public override void ItemAdded(SPItemEventProperties properties) {
        base.ItemUpdated(properties);
        SetCustomVersionLabel(properties.ListItem);
        CreateSnapshot(properties);
    }

    public override void ItemUpdated(SPItemEventProperties properties) {
        base.ItemUpdated(properties);

        var item = properties.ListItem;
        if (RollbackHappened(item)) {
            RestoreSnapshot(properties);
            SetCustomVersionLabel(item);
            CreateSnapshot(properties);
        }
        else {
            CreateSnapshot(properties);
            SetCustomVersionLabel(item);
        }
    }

    private void CreateSnapshot(SPItemEventProperties properties) {
        using (var site = properties.OpenWeb()) {
            var item = properties.ListItem;
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var path = string.Format("Versions/{0}/{1}", item.ID, GetOfficialVersionLabel(item));
            var shadowFolder = CreateFolderPath(shadowLibrary, path);

            foreach (string fileName in item.Attachments) {
                SPFile existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName);
                SPFile newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream());
                newFile.Item.Update();
            }
        }
    }

    private bool RollbackHappened(SPListItem item) {
        var culture = CultureInfo.InvariantCulture;
        var currentVersion = float.Parse(GetOfficialVersionLabel(item), culture);
        var lastVersion = float.Parse(GetCustomVersionLabel(item), culture);
        return currentVersion > lastVersion + 1;
    }

    private void RestoreSnapshot(SPItemEventProperties properties) {
        var item = properties.ListItem;
        var restoreVersion = GetCustomVersionLabel(item);
        EventFiringEnabled = false;

        item.Attachments.Cast<string>().ToList().ForEach(attachment => item.Attachments.Delete(attachment));
        using (var site = properties.OpenWeb()) {
            var path = string.Format("Versions/{0}/{1}", item.ID, restoreVersion);
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var source = CreateFolderPath(shadowLibrary, path);

            foreach (SPFile file in source.Files)
                item.Attachments.Add(file.Name, file.OpenBinary());
        }

        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    // can only get folder creation to work with Document Libraries
    private SPFolder CreateFolderPath(SPDocumentLibrary list, string path) {
        return CreateFolderPathRecursive(list.RootFolder, path.Split('/').ToList());
    }

    private SPFolder CreateFolderPathRecursive(SPFolder folder, IList<string> pathComponents) {
        if (pathComponents.Count == 0)
            return folder;

        SPFolder newFolder;
        try {
            newFolder = folder.SubFolders[pathComponents.First()];
        }
        catch (ArgumentException) {
            newFolder = folder.SubFolders.Add(pathComponents.First());
        }

        pathComponents.RemoveAt(0);
        return CreateFolderPathRecursive(newFolder, pathComponents);
    }

    private void SetCustomVersionLabel(SPListItem item) {
        EventFiringEnabled = false;
        item[CustomVersion] = GetOfficialVersionLabel(item);
        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    private string GetCustomVersionLabel(SPItem item) { return item[CustomVersion] as string; }
    private string GetOfficialVersionLabel(SPListItem item) { return item.Versions[0].VersionLabel; }
}

When a list item is saved, I take a snapshot of the attachments, storing them in a folder structure like {Id}/{VersionNumber}/{Attachments} in the shadow document library. When a list item is restored to a previous version, existing attachments are first deleted before the ones from the snapshot are added back in, creating a new version of the list item.

Restoring previous versions also has a counter-intuitive meaning in SharePoint. Suppose in one version of a list item, I store a key in the item’s property bag, then I’d expect the property bag values to be specific to this version. But behind the scenes restore seems to work by cloning the current version and then copying only the values of the fields from the restore version to the new one. In other words, I can’t use the item’s property bag to store version specific information, such as a version tag to detect when a restore has occurred. I also can’t use the Modified field because SharePoint sets it to the time of the restore. To carry over version information I have to create and maintain a field of my own. Hence the CustomVersion field on the list to version.

Remember that because the ItemUpdated and ItemAdded execute asynchronously, all the snapshotting logic executes on a background thread, after control has returned to the user. Should an error occur at this point, the user will never see it and the snapshot may be left in an incomplete state. On the other hand, this approach scales well and doesn’t have to be fast because no user is awaiting the result.

Lastly, there’s one place in SharePoint where the versioning abstraction leaks through. It’s in the list item version dialog which displays older versions and enables restore to any previous version. The dialog will always show the most recent attachments.

Improvements

I could use the ETag property of an SPFile object to implement a more efficient differential snapshotting algorithm that would conserve storage space. Compressing attachments before storing them in the shadow library might also be an option, although then I’d have to promote the ETag value to a shadow library field before compressing.

SPListCollection extensions

In SharePoint 2010 the TryGetList method has been added to the SPListCollection class. The method returns either an SPList instance matching the display name or null. Oftentimes, however, you want to do a lookup based on the internal name. Here’s an extension method that adheres to the semantics of TryGetList, but using the internal name. It relies on the fact that the RootFolder property of a list is actually its internal name:

// definition
public static class SPListCollectionExtensions {
    public static SPList TryGetListByInternalName(this SPListCollection lists, string internalName) {
        return (from SPList l in lists
            where l.RootFolder.Name == internalName
            select l).SingleOrDefault();
    }
}

// use
if (site.Lists.TryGetListByInternalName(internalListName) == null)
   // list not found

SPFieldCollection extensions

Using the CreateNewField method of the SPFieldCollection you can add new fields to a list. The particular annoying aspect of this method, however, is that when you want to continue working with its result, oftentimes you have to cast it to one of the SPField subclasses. But since the SPFieldType, provided as one of the arguments to CreateNewField, closely relates to the actual SPField return type, an extension method is able to do the casting. This’ll expose mismatches at compile time instead of at runtime.

All it takes is for us to map out the relation between SPField and SPFieldType:

// definition
public static class SPFieldCollectionExtensions {
    public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
            string internalName, string displayName) where TSPField : SPField {
        var spFieldToFieldType = new Dictionary<Type, SPFieldType> {
            { typeof(SPFieldDateTime), SPFieldType.DateTime },
            { typeof(SPFieldNumber), SPFieldType.Number },
            { typeof(SPFieldUser), SPFieldType.User },
            { typeof(SPFieldBoolean), SPFieldType.Boolean },
            { typeof(SPFieldMultiLineText), SPFieldType.Note },
            { typeof(SPFieldText), SPFieldType.Text }
        };

        var fieldType = spFieldToFieldType[typeof(TSPField)];
        var list = fields.List;
        var field = list.Fields[list.Fields.Add(internalName, fieldType, false)];
        field.Title = displayName;
        field.Update();
        return field as TSPField;
    }
}

// use
l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName");

Taking the CreateField extension method one step further, oftentimes you want to set properties besides internal name and display name. For that purpose I’ve defined a CreateField method that accepts an Action<TField>. This allows you to reuse common property settings across fields for brevity and consistency while at the same time maintaining strong typing.

// definition
public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
        string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPField {
    var newField = CreateField<TSPField>(fields, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
public static Action<SPFieldMultiLineText> RichTextProperties = f => {
    f.RichText = true;
    f.RichTextMode = SPRichTextMode.FullHtml;
};

l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName", f => f.Required = true);
l.Fields.CreateField(internalName, "displayName", RichTextProperties);

With the Comment field, you can leave out the type argument because the compiler infers it based on the type of the Action delegate.

Similar to CreateField, I’ve defined two additional extension methods for creating lookup fields:

// definition
public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName,
        string displayName) where TSPField : SPFieldLookup {
    var currentList = fields.List;
    var lookupList = currentList.ParentWeb.Lists.TryGetListByInternalName(lookupListName);
    var newField = currentList.Fields[currentList.Fields.AddLookup(internalName, lookupList.ID, false)];
    newField.Title = displayName;
    newField.Update();
    return newField as TSPField;
}

public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPFieldLookup {
    var newField = CreateLookup<TSPField>(fields, lookupListName, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
l.Fields.CreateLookup<SPFieldLookup>(lookupListName, internalName, displayName, f => f.AllowMultipleValues = true);

These extension methods makes using the SharePoint API more type-safe and concise, and defining lists using these methods and the template approach saves me from writing a lot of repetitive code.

New-SPWeb -url $base_url -addtoquicklaunch -template "STS#1" -name "Base"
New-SPWeb -url $base_url/Dk -addtoquicklaunch -template "STS#1" -name "Dk"
New-SPWeb -url $base_url/Uk -addtoquicklaunch -template "STS#1" -name "Uk"
New-SPWeb -url $base_url/Dk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Dk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"
New-SPWeb -url $base_url/Uk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Uk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"

$dk = Get-SPWeb $base_url/Dk
$dkBusinessUnit1 = Get-SPWeb $base_url/Dk/BusinessUnit1
$dkBusinessUnit2 = Get-SPWeb $base_url/Dk/BusinessUnit2

$locale = [System.Globalization.CultureInfo]::CreateSpecificCulture("da-DK")
$dk.Locale = $locale
$dk.Update()

$dkBusinessUnit1.Locale = $locale
$dkBusinessUnit1.Update()

$dkBusinessUnit2.Locale = $locale
$dkBusinessUnit2.Update()

How can we reduce this repetition? Well, it repeats because we’ve made the common mistake of mixing logic with data. One way to separate the two is using a table-driven approach, i.e., extract the varying parts, the data, into a table and rewrite the logic so it’s parameterized by the table. After some trial and error with PowerShell here’s the code I ended up with:

$sites = @( @("", "Base"),
            @("Dk", "Dk", "da-DK"),
            @("Uk", "Uk"),
            @("Dk/BusinessUnit1", "BusinessUnit1", "da-DK"),
            @("Dk/BusinessUnit2", "BusinessUnit2", "da-DK"),
            @("Uk/BusinessUnit1", "BusinessUnit1"),
            @("Uk/BusinessUnit2", "BusinessUnit2") )

$sites | foreach {
  $url, $title, $culture = $_
  $newSite = New-SPWeb -url "$base_url/$url" -addtoquicklaunch -template "STS#1" -name $title

  if ($culture -ne "") {
    $locale = [System.Globalization.CultureInfo]::CreateSpecificCulture($culture)
    $newSite.Locale = $locale
    $newSite.Update()
  }
}

So far so good. The un-installation part of the script, however, follows the same pattern of repetition:

Remove-SPWeb -identity $base_url/Uk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url -confirm:$false

But now that we have the table of sites at hand, we can easily make the site removal table-driven as well:

function Reverse($array) {
  $clone = $array.Clone()
  [Array]::Reverse($clone)
  $clone
}

Reverse($sites) | foreach {
  $url = $_[0]
  Remove-SPWeb -identity "$base_url/$url" -confirm:$false -ErrorAction SilentlyContinue
}

I’m sure the code could be written more elegantly, but given my working knowledge of PowerShell I’m satisfied with the result. I especially like the functional programming style.

• Use an Inversion of Control container (IoC) like Windsor to create an architecture that responds well to change, i.e., an architecture that promotes looser coupling and higher cohesion
• The ability to write unit tests against a code base is a good measure of its degree of coupling
• By having components depend on interfaces, you’re free to switch the implementation at runtime, introducing flexibility into the software
• Avoid having concrete components talk to each other. It makes it hard for the two to vary independently, e.g., adding logging later can only be done by tearing components apart and putting them back together
• At runtime the IoC container recursively composes these smaller components into an object graph, e.g., by passing concrete implementations through the constructors to satisfy the dependencies
• One alternative to using an IoC container is using a service locator. Each component’s constructor would then ask the locator for its dependent components. While this approach works, it makes testing components in isolation difficult because all dependencies are now locked away inside the service locator. Instead, by having all dependencies supplied in the constructor, unit tests can directly supply fakes implementations
• When software is only used in one environment it tends to be fairly inflexible because it’s only suited for that one purpose. As soon as you design it to be used in two or more environments it becomes more flexible. Thing of different environments as unit test, staging, production, and so on
• The idea is for components to not have to be modified depending on the where they’re running. It’s the IoC container that dynamically tie components together based on the environment
• Suppose you didn’t use a IoC container. Then ultimately the top-level object would have to pass concrete instances down the chain. The top-level object may well be the UI layer. But having the UI layer know about data access components, service components, and logging components isn’t ideal. Instead use an IoC container, which is nothing more than a factory for components
• With the Windsor IoC container (and most others), the usage pattern typically involves three stages
  • Register: tie together interfaces with concrete implementations
  • Resolve: return concrete implementations of interfaces required to satisfy the dependencies of an object
  • Release: dispose of concrete implementations
• Most people only use a fraction of the functionality of an IoC container. Instead of taking dependency of a full-blown container, you could easily roll your own IoC container that implements the core functionality in a few dozen lines of code — in an associated .NET Rocks and DnrTV episode James Kovacs elaborates on his original article
• The goal of using a container isn’t to get rid of calls to the new operator. It’s to not use the new operator for the parts of an applications where flexibility is required
• Simple forms of Aspect Oriented Programming are possible using interceptors
• Windsor supports the decorator pattern so you can have one component wrap another at runtime. This lets you implement features such as logging without actually modifying the original component. It’s an example of adhering to the open/close principle by which classes should be open for extension, but closed for modification. In other words: don’t edit the original source code to introduce new behavior. Instead use methods of composition
• Avoid configuring Windsor through XML and instead use the fluent interface — perhaps in a separate assembly so only it needs to be redeployed when the configuration changes. When a value, such as a connection string needs to be configurable, create a section in app.config and add the value there. Windsor will know how to locate the value when instantiating objects

• Enterprise service bus
  • An architectural pattern rather than a specific technology or product
  • The ethernet of SOA
• Types of service bus
  • Centralized: a broker more than a bus, e.g., BizTalk
  • Decentralized: a real service bus, e.g., NServiceBus, Mass Transit, Rhino Service Bus, MS .NET Service bus
• The current version 2.0 of NServiceBus is free whereas the binary version 2.5 will contain limitations. The limitations can be disabled by downloading and compiling the source code yourself
• NServiceBus is designed to address the fallacies of distributed computing
  • It primarily addresses the fallacies through the use of reliable one-way messaging
  • Reliable means messages are stored in a queue, such a MSMQ, which is part of all recent Windows installations. The assumption being that the queue is always present and ready to receive messages
  • One-way means messages are asynchronous and fire-and-forget. The destination service doesn’t have to be online when a message goes into its queue, which makes services temporally independent. At some point in the future the service may post a reply to the client’s queue
  • Messaging means that any action is accomplished by posting a self-contained message, containing the type of operation and its parameters, to the queue of a service
  • With NServiceBus, a class is marked with an interface to describe that it’s a type of message. Instances of the class are then serialized using a standard XML or binary serializer and stored in a queue
• NServiceBus service != traditional web service
  • An NServiceBus service is a class that implements an interface
  • Using a message queue is more reliable when the service has to make calls to other services as part of its operation. With a traditional web service, calling other services from within makes the service slower and less reliable. If only one dependent service is down it’s like with old Christmas lights wired in series: the entire stack may be down. Queues on the other hand introduce a save place, a stabilization point, to temporarily store messages
  • A service is typically configured with an input queue that identifies it to others and an error queue that messages are routed to after some number of retries. In addition, each service may be configured with a number of threads to use to parallelize message processing

• Windbg isn’t a replacement for VS, but VS doesn’t handle some advanced cases
• Windbg is a free user mode/kernel mode debugger which is part of the Debugger Tools for Windows
• Customers may not be happy installing VS to debug code in production since it installs a lot of components and requires restarts
  • Windbg requires only a simple installation once you extract the redistributable from the Debugger Tools for Windows
• Loading SOS.dll from the Microsoft .NET folder into Windbg makes it understand .NET
  • With SOS.dll loaded, you can look into the CLR and its data structures
  • Make sure to load the right SOS.dll for your runtime
  • SOS.dll is also available for the Silverlight runtime
• Debugger Markup Language support is available for version 4 of SOS.dll
  • Provides hyperlinks in the command output of SOS
• Like with the VS debugger, you can insert Debug.Break() in your demo app and have Windbg halt on it
  • Release builds can be debugged with Windbg
  • Release builds also contain symbols. Like with debug builds, symbols are stored in a separate file
  • Release builds make debugging harder because the jitter kicks in and modifies the code
  • The 64 bit calling convention of passing arguments via registers makes it harder to locate information when debugging
• Windbg generally needs symbols loaded, although it’s less important when debugging managed code
  • Use MS’ public symbol server to load symbols on demand
  • Set the _NT_SYMBOL_PATH environment variable to point to your symbols (will affect VS as well)
  • Or use the .symfix command from within Windbg
• Popular extensions to Windbg: SOSEX and Psscor2 (replacement for SOS, useful for ASP.NET debugging)
• Create dump file to analyze: use task manager or ADPlus or ProcDump from Sysinternals, which can dump based on triggers
• ADPlus collects crash dumps or hang dumps
  • Hang dumps can be captured from the same process multiple times and may be useful when debugging deadlocks or resource leaks
  • When you capture a hang dump, the process is halted for the dump period and is then restarted
• A *32 process in task manager is actually a 64 bit process when you dump it
  • WoW64.dll is involved when dumping from the task manager
  • Not what you typically want because you don’t get full access to 32 bit process information
• A .NET application is hosted within the CLR which is itself hosted within a regular Windows process
  • Looking at memory usage with the task manager therefore doesn’t tell you much about the .NET part

To set the stage, I’ve created an array of people to query.

    Person[] people = new[] { new Person { Age = 1, ...}, ... };

It’s worth noting that all arrays implicitly derive from the Array class. It’s a special class that only the compiler and the runtime may derive from. Doing so yourself is rewarded with a compiler error stating simply that the class cannot be derived from. What’s also not apparent from the definition of the Array class is that, as of .NET 2.0, it also implements IList<T>, ICollection<T>, and IEnumerable<T>, which derive from the corresponding non-generic interfaces.

    public abstract class Array :
        ICloneable, IList, ICollection,
        IEnumerable, IStructuralComparable, IStructuralEquatable

The runtime takes special care of extending the Array type with implementations of the generic interfaces based on the type of object that the array stores. That’s why the interfaces are invisible in the documentation. Any array of type T — in this case Person — will therefore implement, among others, the IEnumerable<T> interface. It’s the implementation of this interface on a class that enables LINQ to Objects to query any array or collection.

Query expression syntax

Let’s create a simple query using query expression syntax. To support it, additional keywords have been added to C# to express common query operators. Examples of what’s not supported with query expression syntax are the Sum, Take, and Skip query operators. In those cases you can combine query expression with method invocation syntax or write everything using the latter.

    var q = from p in people
            where p.Age > 25
            select p;

Method invocation syntax

To make matters transparent, let’s not rely on type inference and the var keyword for typing the result. At the same time, let’s assume the role of the compiler and translate the query expression syntax into method invocation syntax with lambda expressions. Each standard query operator, such as Where and Select, translates to a corresponding method invocation on the collection, oftentimes with a lambda expression as the argument. Both representations are semantically equivalent, but for this simple query method invocation syntax appears more complex. This is generally not the case for either representation as you can try out with a tool like LINQPad.

    IEnumerable<Person> r = people
        .Where(p => p.Age > 25)
        .Select(p => p);

The concise nature of the code is due to the compiler inferring the type arguments to Where and Select and the type of the lambda expression. In this case, because the collection stores objects of type Person, the type arguments of the generic methods as well as the type of the lambda expression is also of type Person. To make these types explicit, you can substitute real types for the generic ones in the definition of Where and Select provided later.

    IEnumerable<Person> s = people
        .Where<Person>((Person p) => p.Age > 25)
        .Select<Person, Person>((Person p) => p);

Generic delegates

With the type specifications made explicit, it’s more obvious how lambda expressions are compatible with delegates. A lambda expression is nothing more than a short-hand notation for a delegate, a type-safe pointer to a piece of code, to be passed into a method. Hence, a lambda expression can be substituted with an anonymous delegate by wrapping it in additional ceremony.

    IEnumerable<Person> t = people
        .Where<Person>(delegate(Person p) { return p.Age > 25; })
        .Select<Person, Person>(delegate(Person p) { return p; });

To make delegates type safe, their definition include the return type and the types of the arguments passed into it. This is unfortunate since the standard query operators must be able to work on any type of object. LINQ therefore relies on generic delegates in the definition of its operators. Like with other generic types, the compiler and runtime work together to generate real delegates based on the specified, or inferred, return type and types of arguments. Delegates like the ones below for Where and Select are what’s generated by the runtime.

    // delegate Boolean WhereDelegate(Person p);
    // delegate Person SelectDelegate(Person p);
    bool WhereClause(Person p) { return p.Age > 25; }
    Person SelectClause(Person p) { return p; }

    IEnumerable<Person> u = people
        .Where<Person>(WhereClause)
        .Select<Person, Person>(SelectClause);

LINQ relies on a set of generic delegates defined in the .NET framework. These delegates come in two flavors: those that return void, named Action, and those that don’t, named Func. You can use these delegates in your own code to not only parameterize methods by value but by functionality.

    delegate void Action();
    delegate void Action<T>(T obj);
    delegate void Action<T1, T2>(T1 arg1, T2 arg2);
    // up to eight arguments

    delegate TResult Func<TResult>();
    delegate TResult Func<T, TResult>(T arg);
    delegate TResult Func<T1, T2, TResult>(T1 arg1, T2 arg2);
    // up to eight arguments

Extension methods

Each of the 50 or so standard query operators is defined on the Enumerable class with the this modifier on their first IEnumerable<T> argument. This makes them extension methods to every object that implements IEnumerable<T>. From the definition of the Where operator below, it then follows that when you write "Where(p => p.Age > 25)", the compiler infers that because Where is called on a collection of type Person, p must also be of type Person. Furthermore, because the result of the comparison is a bool, the lambda expression is compatible with a delegate that accepts a type Person and returns a type bool. In other words, the matching Where has a signature like the commented one below.

The purpose of the Where operator is to filter a collection based on a predicate, the mathematical term for a function that returns true or false. The people that satisfy the predicate is therefore returned as an IEnumerable<Person>. However, the yield keyword adds an interesting twist to the return of Where and Select and other operators that return IEnumerable<T>. Instead of the operator iterating the source collection and returning every element of the result at once, the yield keyword instructs the compiler to emit a state machine so the operator can keep track of how far through the source collection it is, and return one resulting element at a time. The effect is lazy evaluation of LINQ to Objects queries.

In the example, the output of Where goes into Select whose output may goto the console using a foreach. To understand yield, think of the foreach that writes to the console as pulling on a string of objects connecting Select to Where to the source collection. Each iteration of the loop pulls the string just enough to get at the next resulting element. This may in turn require Where to iterate multiple times until the predicate is once again satisfied.

    // IEnumerable<Person> Where<Person>(
    //     IEnumerable<Person> source,
    //     Func<Person, bool> predicate)

    public static IEnumerable<TSource> Where<TSource>(
        this IEnumerable<TSource> source,
        Func<TSource, bool> predicate) {
        foreach (TSource element in source) {
            if (predicate(element))
                yield return element;
        }
    }

The Select operator is an example of one whose input type may differ from its output type. By its very nature, it projects one type onto another. It can even project onto an anonomous type, like in "Select(p => new { x = p.Name })", but then the var keyword must be used for its return type. In the example I project from Person to Person, which makes the matching signature of Select like the commented one below.

    // IEnumerable<Person> Select<Person, Person>(
    //     this IEnumerable<Person> source,
    //     Func<Person, Person> selector)

    public static IEnumerable<TResult> Select<TSource, TResult>(
        this IEnumerable<TSource> source,
        Func<TSource, TResult> selector) {
        foreach (TSource element in source)
            yield return selector(element);
    }

The ability to chain methods together to form a data pipeline is what adds real power to LINQ. Although the idea of chaining methods together is old, the traditional approach of each method returning a new or mutated instance works best for objects of your own. Imagine adding to the IEnumerable<T> interface a set of query operators. Then every collection would have to provide implementations for Where, Select, and so on. Also, it wouldn’t be possible to add new operators in vNext of .NET without breaking existing code implementing the interface.

Extension methods solve this issue by providing the necessary syntactic sugar to make chaining of methods on a collection feel like chaining on an object of your own. Behind the scenes, the compiler rewrites calls to extension methods to static methods.

    var v = Enumerable.Select(
        Enumerable.Where(people, p => p.Age > 25),
        p => p);

Note how, without the use of extension methods, operations must be specified in the opposite order in which they’re executed. This doesn’t read as nicely as the “infix” syntax when many operations are chained together.

Although Visual Studio, in tandem with the MSBuild engine, generally takes good care of compilation, I rarely want to rely on it solely. Instead, I’d prefer that any developer task be easily run from the command-line. This ensures that no magic is going on, that the task is kept simple and flexibility, and that it doesn’t rely on Visual Studio to work. The challenge, however, is which of the many tools available to pick. It must be one that’s flexible enough to meet most requirements with relative ease or the automation will likely not be a valid documentation medium in itself.

Why not to use Windows batch files

As a general example, consider the deployment of a SharePoint 2007 solution. With SharePoint a good deal more than compiling is needed to bring new functionality into a SharePoint installation. Whereas Visual Studio and MSBuild may still compile the code and WSPBuilder create the WSP installation package, both from within Visual Studio and from the command-line, getting everything setup cries for automation, even locally. A common approach (possibly inspired by popular SharePoint 2007 literature) is that of the batch file applying various operations to a WSP. For the sake of brevity, I’ve kept the example short, but imagine extending it with a check-out source code task, a compilation task, a WSPBuilder task, and a feature deactivation and activation task, and possibly a warm-up task. Add to this a master script that orchestrate the whole thing. In the end, I’d end up with scripts that become increasingly harder to maintain as they grow in number and size.

    @set STSADM="c:\program files\common files\microsoft shared\web server extensions\12\bin\stsadm"

    if "%1" == "uninstall" goto uninstall
    if "%1" == "install" goto install
    if "%1" == "" goto reinstall

    :uninstall
        %STSADM% -o retractsolution -name Foo.wsp -immediate
        %STSADM% -o execadmsvcjobs
        %STSADM% -o deletesolution -name Foo.wsp -override
        goto end

    :install
        %STSADM% -o addsolution -filename Foo.wsp
        %STSADM% -o deploysolution -name Foo.wsp -immediate -allowGacDeployment -force
        %STSADM% -o execadmsvcjobs
        goto end

    :reinstall
        call Foo uninstall
        call Foo install
        goto end

    :end

Still, because of the ubiquitous nature of command.com and cmd.exe, the batch file interpreters, batch files are everywhere. Regardless that the technology is a left-over from the days of MS DOS and haven’t evolved much since. Not only are the branching and looping constructs limited, so are the available commands. Suppose I want to find out if a command was indeed successful. This turns out to be really hard when all I have to work with is the errorlevel of the most recently executed command. Assuming the command adheres to the errorlevel convention, for the script to fail as early and as close to the real error as possible, I’d have to inspect the property after each command, causing the batch file to grow quite verbose. Sadly, batch files lack the equivalent of set -o errexit of Bash, where the interpreter checks for success after each command and aborts immediately on error. Relying solely on the errorlevel is oftentimes insufficient anyway. To determine success, I’d typically have to parse actual command output or inspect some system property by downloading additional commands or building my own.

Essential vs. incidental requirements

Unless I plan to sit idle and stare at the screen and be a human error detector while the batch file run, I think it’s safe to say that it’s mostly unsuitable for developer automation. Hence in late 2005 I rolled my own automation tool in Python. With Python or Ruby or a similar dynamic language acting as the glue that ties everything together, almost any task can be automated in a robust way. Of course, I could also automate with a static language like C# (it’s surprisingly common). But for script-like tasks, I don’t particularly fancy the long cycle of editing source code in Visual Studio, compiling it, deploying it, and having a hard time debugging it in environment without Visual Studio. A dynamic language, on the other hand, short-circuits the development cycle and allows for interactive programming through a REPL.

With a dynamic language that interacts with .NET, such as IronPython, IronRuby, or Powershell, possibly with supporting DSLs like Paver, Rake, or psake on top, the need for writing custom commands to interact with the operating system or the application almost vanishes. The question, then, is which of the dynamic languages to go with when at their technical core they’re so much alike. Besides sharing the concept of a REPL, the notion of a tuple, a list, a map, and operations on each are baked into their syntax, making code quite terse. It even makes it convenient to express any configuration in the language itself and not in XML where I’d first have to come up with a schema, and then create an instance of it before parsing it. On Windows, however, Powershell is gradually becoming the next ubiquitous interpreter, with applications shipping with cmdlets, whereas IronPython or IronRuby is a separate download.

Conclusion

No matter the tool, what I end up doing is defining tasks, form dependencies between tasks, and have the tool execute tasks in an order that satisfies their dependencies. As the tool traverses the dependency graph and executes tasks, it’s up to each task to detect success, and up to the tool to report on progress. A good tool is therefore characterized by the ease with which I can express these things, the available language constructs, the ease of debugging, and the tool’s ability to converse in foreign languages.

The book tells the story of 15 prominent coders by asking each coder roughly the same questions. After covering their initial exposure to programming, it moves on to how they design and debug programs and whether or not they use literate programming or design by contract or more formal proof techniques. In my opinion, each interview varies too little and is backed by too many anecdotes. I’d hoped for more universal wisdom and advice that would transcend time and technologies.

Perhaps not too surprisingly, during the interviews you learn how many of the coders got into the field entirely by chance. Simon Peyton Jones, a creator of Haskell, was drawn to electrical and software engineering because he found math too hard and competitive. Others were lucky enough that their school happened to have a mainframe sitting in the basement for them to tinker with. On the whole, computers simply sparked an early and lifelong passion.

Another common coder’s trait seems to be that of code reading. As an individual developer, you should improve by studying real code of every quality. As a team of developers, you should improve by shared code reviews of what you’re building. Like no other technique, studying and reviewing code will help share knowledge and improve general developer skills. Yet, the typical organization keeps to in-house presentations on the vNext features of some product and calls it knowledge sharing for developers.

I ought to do more a la Scott Hanselman’s weekly source code. When taking on C++ and MFC in the late nineties, I remember reading the MFC sources and admiring how well it applied design patterns and design by contract to the C-based Windows API. That code base alone had a profound impact on my understanding of object-orientation and how to write code.

Why traditional dependency-breaking techniques come short

After watching a screencast on how to use Moles to unit test LINQ to SQL without hitting the database, I thought it would be interesting to do the same with TypeMock. But first, let’s make sure we understand why traditional dependency-breaking techniques come short in testing LINQ to SQL. Assuming we want to put a repository under test, our goal is to mock how it accesses the database. Here’s a simple implementation of a repository that queries the Employee table of the AdventureWorks database:

    public class EmployeeRepository {
        public List<Employee> GetEmployeesByHireDate(DateTime start, DateTime end) {
            using (var ctx = new AdventureWorksDataContext())
                return (from e in ctx.Employees
                        where e.HireDate >= start && e.HireDate <= end
                        select e).ToList();
        }
    }

All calls to the database are routed through the AdventureWorksDataContext generated by Visual Studio. To mock access to the database, we therefore have to mock part of the data context. Easier said than done, though, for the context doesn’t expose an interface that a fake can implement. In addition, the tables are accessed through properties on the context that return a type of Table<TEntity>. Unfortunately, the constructor of Table<TEntity> is internal and the class itself is sealed, eliminating the hope of instantiating or subclassing the type by traditional means:

    public sealed class Table<TEntity> : IQueryProvider,
            ITable, IListSource, ITable<TEntity>, IQueryable<TEntity>,
            IEnumerable<TEntity>, IQueryable, IEnumerable
            where TEntity : class {
        internal Table(DataContext context, MetaTable metaTable) {
            ...
        }
    }

For an example of how the data context itself creates an instance of Table<TEntity>, take a look at the Employees property on the AdventureWorksDataContext. It relies on the GetTable<Employee> method on the DataContext class to create an instance of Table<Employee>. Despite its constructors being internal, the GetTable<TEntity> method has no trouble constructing an instance of the Table<TEntity> type, as they both reside in the System.Data.Linq assembly:

    public partial class AdventureWorksDataContext : DataContext {
        public Table<Employee> Employees {
            get {
                return GetTable<Employee>();
            }
        }
    }

How to break the unbreakable

The design of LINQ to SQL leaves us short of a traditional testing seam, as Michael Feathers would phrase it; a place at which we can alter the behavior of a program without editing in that place. This explains why, with LINQ to SQL, traditionally we’ve had to test against a real database with all its constraints, making our tests brittle, slow, and painful to write and maintain. With the new breed of mocking frameworks the issues of not being able to subclass or not being able to call an internal constructor go away (and new issues take their place). Regardless, here’s how to write a unit test for the CustomerRepository that doesn’t hit the database:

    [TestClass]
    public class CustomerRepositoryTest {
        private EmployeeRepository _repository;

        [TestInitialize]
        public void Initialize() {
            _repository = new EmployeeRepository();

            var fakeEmployees = new List<Employee> {
                new Employee {EmployeeID = 1, HireDate = new DateTime(2004, 12, 1)},
                new Employee {EmployeeID = 2, HireDate = new DateTime(2006, 7, 1)},
                new Employee {EmployeeID = 3, HireDate = new DateTime(2009, 3, 1)}
            }.AsQueryable();

            var fakeDataContext = Isolate.Fake.Instance<AdventureWorksDataContext>();
            Isolate.Swap.NextInstance<AdventureWorksDataContext>().With(fakeDataContext);

            // var fakeEmployeeTable = Isolate.Fake.Instance<Table<Employee>>();
            // Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturn(fakeEmployeeTable);
            // Isolate.WhenCalled(() => fakeEmployeeTable).WillReturnCollectionValuesOf(fakeEmployees);
            // or by transitivity
            Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturnCollectionValuesOf(fakeEmployees);
        }

        [TestMethod]
        public void GetEmployeesByHireDate_should_return_hires_from_2008_until_present() {
            var employees = _repository.GetEmployeesByHireDate(new DateTime(2008, 1, 1), DateTime.Now);
            Assert.AreEqual(1, employees.Count());
            Assert.AreEqual(3, employees[0].EmployeeID);
        }
    }

The test method itself looks exactly as if we’d been testing against a real database. The difference lies in the Initialize method, where we setup the fake data context and database contents. We instruct TypeMock to return the fake context in place of the real one inside EmployeeRepository. And whenever someone calls the Employees property on the fake context, we have TypeMock intercept the call and return a fake collection of type IQueryable<Employee>. We could’ve returned an instance of Table<Employee>, which implements IQueryable<Employee>, but in this case returning the collection is simpler and sufficient. Had we had more methods on our repository, we likely would’ve added additional rows to the Employee table and populated more of its columns.

I found the given-expect pattern useful in testing a piece of code that I was working on this week. I was refactoring and adding tests around an ASP.NET control adapter that makes SharePoint 2007 pages more XHTML compliant. I wanted to reuse the transformations outside the control adapter and hence ended up moving the transformation logic to a new class. It accepts possibly malformed HTML and relies on heuristics of the HTML Agility Pack to build a DOM off of it. I can then query the DOM, looking for known violations, and patch them before returning XHTML to the caller.

    public class HtmlToXHtmlTransformer {
        private readonly HtmlDocument _document;

        public HtmlToXHtmlTransformer(string html) {
            _document = new HtmlDocument();
            _document.DetectEncoding(new StringReader(html));
            _document.LoadHtml(html);
        }

        private void Transform(string xpath, Action<HtmlNode> nodeMatch) {
            var nodes = _document.DocumentNode.SelectNodes(xpath);
            if (nodes != null)
                foreach (var node in nodes)
                    nodeMatch.Invoke(node);
        }

        private void FixDuplicateBorderAttributeOnSPGridViewControl() {
            Transform("//table[count(@border)=2]", node => node.Attributes.Remove("border"));
        }

        public string Transform() {
            FixDuplicateBorderAttributeOnSPGridViewControl();
            _document.OptionWriteEmptyNodes = true;
            return _document.DocumentNode.WriteTo();
        }
    }

The complete HtmlToXHtmlTransformer collects a dozen transformations. Its Transform method is what we want to call with various HTML fragments to verify that they come out as XHTML. For this purpose, we might do the tests as Visual Studio data-driven tests that read their input and output from a text file. But in most cases I prefer traditional tests, so I can describe the purpose of a test with a descriptive method name and possibly a comment.

    [TestClass]
    public class HtmlToXHtmlTransformerTest {
        private string _result;

        [TestMethod]
        public void Must_selfclose_nodes_when_allowed() {
            Given("<br>");
            Expect("<br />");
        }

        [TestMethod]
        public void Must_remove_duplicate_border_on_SPGridView_control {
            Given(@"<table border=""0"" border=""0""></table>");
            Expect(@"<table border=""0""></table>");
        }

        private void Expect(string xhtml) {
            Assert.AreEqual(xhtml, _result);
        }

        private void Given(string html) {
            var transformer = new HtmlToXHtmlTransformer(html);
            _result = transformer.Transform();
        }
    }

I particularly like the clarity of the given-expect pattern and find that for a reasonable number of tests it’s a viable alternative to data-driven test. I do, however, recognize the value of data-driven tests in situations where a non-developer wants to test a class. Though at the unit test level I’ve never experienced this. It’s more characteristic of FitNesse for acceptance testing. However you unit test, just make sure your tests run with a minimum of effort on your part and that they run fast.

Code equals data

Configuring the editor is done by placing Emacs Lisp code in the .emacs file. Whenever Emacs starts, it looks for this file in the directory specified by the HOME environment variable and executes its content. Unlike the XML configuration files of .NET and Java, with Emacs there’s no intermediate configuration format. There’s no extra language to learn and no limitation in the expressiveness of the configuration language. With this interchangeability of code and data, every behavior of Emacs can be hooked into and tailored and every global variable redefined.

Starting Emacs from the Windows taskbar

On my machine, I like everything Emacs-related to reside in the directory that I extracted Emacs to, yet I want to be able to start Emacs from the taskbar. I do this by pinning cmd.exe to the taskbar and modifying its destination so it defines HOME for this instance before spawning Emacs. This way the HOME of Emacs doesn’t conflict with the HOME of other applications.

    cmd.exe /C "set HOME=...\emacs-23.1&&...\emacs-23.1\bin\runemacs.exe"

My .emacs isn’t too esoteric. Nevertheless, here it is for posterity. Reading on, keep in mind that unless otherwise noted, all referenced packages come with Emacs 23.1.

General settings

I start off redefining a couple of global variables. Much functionality relies on the user name and mail address to be known in advance. I then remove the toolbar at the top of the Emacs window, because I use menus or keyboard shortcuts anyway. Next I disable the splash screen on startup, and redefine a few global variables governing Emacs’ scrolling behavior. By default scrolling isn’t line by line, but chuck by chuck of text. I find this behavior confusing.

    (setq user-full-name "Ronnie Holm")
    (setq user-mail-address "foo@bar.baz")

    (tool-bar-mode nil)                            ;; don't show the toolbar

    (setq inhibit-startup-message t                ;; don't show ...
          inhibit-startup-echo-area-message t)     ;; ... startup messages

    (setq scroll-margin 1                          ;; do smooth scrolling
          scroll-conservatively 100000
          scroll-up-aggressively 0.01
          scroll-down-aggressively 0.01)

Next I turn on automatic syntax highlighting and commands specific to known file types. Following that I turn the blinking square cursor into a solid and less distracting one. Also, I instruct Emacs not to retain backups of files that I edit or save. I don’t discount the value of backups, but I also don’t like having to remove backup and auto-save files when they clutter my folders. Instead  Emacs should move these files to the Windows Recycle Bin. Lastly, I enable iswitchb-mode, which makes switching buffers easier by providing on-the-fly suggestions.

    (global-font-lock-mode t)                      ;; always do syntax highlighting
    (blink-cursor-mode 0)                          ;; don't blink cursor
    (setq backup-inhibited t)                      ;; no backup files
    (setq delete-by-moving-to-trash t)             ;; delete moves to recycle bin
    (iswitchb-mode t)                              ;; easy buffer switching

The last of the general settings involve modifications to the modeline at the bottom of the Emacs window. By default it displays the buffer name, the line number, and the modes Emacs is in. I extend the modeline with the column cursor position and the size of the current buffer.

    (column-number-mode t)                         ;; show column numbers
    (size-indication-mode t)                       ;; show file size

Package settings

Oftentimes I find myself editing the same set of files every day. So having the cursor move to where I left it editing the file the last time is handy. This is done by save-place mode. Behind the scenes it stores a file name/offset mapping in a file below HOME.

    (setq save-place-file "~/.emacs.d/saveplace")  ;; location of saveplace file
    (setq-default save-place t)                    ;; activate for all buffer
    (require 'saveplace)

As hinted by Getting organized with Emacs Org-mode, I very much enjoy using Org-mode for keeping notes, maintaining lists, and much more. The next section of my .emacs is therefore dedicated to customizing org-mode. Firstly, to make documents easier to read, instead of displaying all the asterisks that designate an indentation level, only display the last one. Secondly, align each indentation level by always adding two asterisks on indentation. Thirdly, when clocking time, have Emacs remember what I was doing and for how long across restarts. But remove clocked time less than one minute.

    (setq org-hide-leading-stars t)                ;; hide but one star in outline
    (setq org-add-levels-only t)                   ;; align items nicely
    (setq org-clock-persist t)                     ;; keep track of time ...
    (org-clock-persistence-insinuate)              ;; ... across sessions
    (setq org-clock-out-remove-zero-time-clocks t) ;; remove 0-duration clocked

The default Emacs color theme looks kind of dull. So I downloaded the color-theme package and selected a more aesthetically pleasing theme. After downloading color-theme, place its Lisp files in ~/site-lisp. That’s one of the places where Emacs will automatically look for packages. The 300 or so packages that come with Emacs are in the ~/lisp folder.

    (require 'color-theme)                         ;; use a color theme
    (color-theme-initialize)
    (color-theme-arjen)

My last additions to .emacs involve Haskell. Emacs doesn’t support Haskell out-of-the-box. But a haskell-mode is available for download. It turns Emacs into a decent Haskell development environment. Besides syntax highlighting and code formatting capabilities, it adds to Emacs commands for interacting with the Haskell compiler and Haskell interactive.

    (load "~/site-lisp/haskell-mode-2.6.4/haskell-site-file")
    (add-hook 'haskell-mode-hook 'turn-on-haskell-doc-mode)
    (add-hook 'haskell-mode-hook 'turn-on-haskell-indentation)

This concludes my .emacs. Quite a few of these settings I picked up reading the emacs-fu blog.

Here’re the talks that I attended, all recorded and available with slides.

SharePoint 2010 Planning and Best Practice Approaches to Upgrade

Upgrading to SharePoint 2010, you should first establish a copy of your existing environment on the latest patch level. Then run the stsadm command with the new PreUpgradeChecker option. Make sure your environment has no unused features, site templates, and so on, and identify everything that has changed from out-of-the-box by running a tool like WinMerge on the 12 hive. Also, assert that no pages has been ghosted, and don’t attempt to upgrade your SharePoint 2007 master page. Start with the 2010 ones and add in your 2007 modifications. SharePoint 2010 ships with both 2007 and 2010 master pages, but SharePoint 2007 and 2010 isn’t binary compatible.

Make sure all hardware requirements are satisfied: 8 GB RAM at the minimum on the Windows 2008 servers running SharePoint and MS SQL Server. On the client you need Internet Explorer 7 or 8 or Firefox 3.x to author content. Internet Explorer 6 isn’t supported. Not even with a degraded reader experience. The SharePoint 2010 development environment should be setup with Visual Studio 2010 on Windows Vista or Windows 7 in order to compile, debug, build, and deploy right out of Visual Studio 2010.

Understanding, using, and customizing PowerShell for a SharePoint 2010 Environment

This talk started with some PowerShell history and basics. PowerShell is made up of a shell that executes commands and an integrated scripting environment for development. .NET compiled code, called cmdlets, are the executable units combined using the PowerShell pipeline. The PowerShell language itself is dynamic and loosely typed, so being able to express yourself tops performance. Scripts may be compiled and deployed as a SnapIn, consisting of compiled DLLs and registry updates, or as a module using xcopy deployment.

SharePoint 2010 ships with 650 cmdlets, and its own base class for writing additional SharePoint cmdlets.

Design and Manage Site Collections in SharePoint 2010

By default every single site collection within a web application goes into one rapidly growing database (think about users migrating from file shares to site collections). Such a large database takes time to backup and restore. Hence, with SharePoint 2010, consider partitioning site collections into their own databases from the beginning. After a database is created, map it to one and only one site collection. Then, instead of putting a size limit on the web application, put it on the site collection itself.

In SharePoint 2007 you had to create multiple Shared Service Providers (SSP) if site collections had different needs or you didn’t want a site collection to access parts of an existing SSP. With SharePoint 2010, application services can now be shared between site collections.

With sandbox solutions, the solution is deployed to the database hosting the site, rather than to front-end servers. This makes it easier to deploy code, but the code is limited in what it can do, and you may have to set resource quotas across the site collection.

SharePoint 2010, Getting Ready

This talk provided an overview of SharePoint 2010. WSS 3.0 becomes SharePoint Foundation 2010 and MOSS 2007 becomes SharePoint Server 2010. Business Connectivity Services (BCS) are no longer part of the server edition, but has been moved to Foundation. A lot more services have been added, like state service, usage and health service, and Visio graphics service. Generally, with all the improvements to deployment and upgrade with PowerShell and the addition of multi-tenancy for the SSP, SharePoint 2010 is better in a hosting scenario.

The web user interface has been made more Office-like with the addition of the ribbon and the use of AJAX. In document libraries you can create document sets as a way to relate documents to each other and have versioning and workflow follow every document in the set. Through the BCS, you can expose external data as native SharePoint lists with full CRUD capabilities.

SharePoint 2010 Development Tips and Tricks

The focus of the first part this talk is on the SharePoint 2010 Foundation features. Instead of developing against the web services or the object model, you can use the new managed client object model. This way, you can work against SharePoint from .NET code, from a Silverlight application, or from JavaScript. Behind the scenes, the client object model works against a WCF service running on the SharePoint server, which talks to the server object model on the server, and returns JSON to the client.

The second part of this talk is about sandbox solutions. This feature provides site collection users with the ability to upload and deploy WSPs. Finally, the third part of the talk is about the fluid integration model, which solves the problem of accessing cross-domain resources, e.g., having a Silverlight application access information on a SharePoint server on a different domain.

Installing and Configuring SharePoint Server 2010 in a Virtual Environment

This talk recommends that everyone read the MS whitepaper on Virtualization of Microsoft SharePoint Products and Technologies. Configuring a development environment, the dilemma is weather to create one virtual machine hosting everything or one virtual machine for the MS SQL Server, the Active Directory, and the SharePoint server each. The presenter suggests a middle ground. Still, it’s important to create a number of service accounts and use them in your development environment rather than running everything as administrator.

The SharePoint 2010 installer comes with a prerequisite installer that goes onto the web and downloads required software. After that, you can run the SharePoint server installer. Here you can choose between Standalone and Server farm, but always pick server farm. Standalone should only be used for demo purposes.

Last week at work I did a 45 minutes presentation on how to get started with SharePoint development. Instead of the usual presentation on the technical merits of SharePoint, my focus was more on the change in mindset required coming from ASP.NET. For even though ASP.NET is a cornerstone of SharePoint, SharePoint is so much more. Not to mention different.

History makes a difference

SharePoint is a long-evolving product, dating back to the late nineties. From its inception and up till now new technologies have replaced old ones, APIs have emerged and died, product teams have merged, and Microsoft have bought and integrated other companies’ products into SharePoint. At the same time SharePoint has become the fastest growing server product in Microsoft history. All of this inevitably leaves its marks on the APIs, which aren’t as consistent and defect free as we’re used to with the .NET framework.

Provisioning is the magic sauce

What truly makes SharePoint development different from ASP.NET is the notion of provisioning. With ASP.NET you create a bunch of ASPX files, organize them in a static file structure, and deploy them to a server. With SharePoint, the structure emerges over time as SharePoint or the user creates instances of templates: sites from site templates, lists from list templates, pages from page templates, and so forth. From a developer’s perspective the challenge isn’t the instance creation in itself. It’s that SharePoint encourages users to not only dynamically evolve the structure, but also to modify the instances, severing their definition from the template. While it empowers users, it also makes it challenging to programmatically evolve the structure.

Developing with SharePoint is about creating templates and components for SharePoint or the user to compose a site of. To make this composition work, SharePoint imposes constraints on what can be done and how. While any constraint may at times be perceived as a hindrance, accepting it is what guarantees that a components will play nicely with others. These pieces of functionality are then packaged as features and added to a WSP installation file for deployment to SharePoint. Features may then be activated, making their functionality available to a site. Or deactivated and the WSP uninstalled.

Why learn SharePoint

By the end of the day, no doubt SharePoint is more challenging to work with than ASP.NET. Not acknowledging its history makes many ASP.NET developers so frustrated that they’ll never again want to work with SharePoint. Coming from ASP.NET I can understand why. At times SharePoint isn’t as smooth and evolved as ASP.NET. But knowing what to expect takes the worst of the frustration. And once you learn to leverage the power of SharePoint, you can build applications without writing a lot of plumping code. To a large extend the architecture is given, leaving you with the opportunity to fill in the blanks.

Maybe it’s that the SharePoint API is hard to use. Maybe it’s that coding against SharePoint is about making smaller additions here and there. Maybe it’s that the Patterns & Practices SharePoint Guidance isn’t widely known. Whatever the reason, developing for SharePoint requires equal attention to the separation of presentation, business, and data access code. Hence, starting with data access, we may want to create a repository and route queries through it (the SharePoint Guidance outlines a more sophisticated implementation than the one below):

    [TestClass]
    public class EmployeesRepositoryTest {
        private SPSite _siteCollection;
        private SPWeb _site;
        private EmployeesRepository _repository;

        private readonly Employee _duffyDuck = new Employee {
            Id = 1000, Name = "Duffy Duck", HireDate = new DateTime(2009, 12, 1),
            Remarks = "Looks like a duck, quacks like a duck, probably is a duck"
        };
        private readonly Employee _porkyPig = new Employee {
            Id = 1001, Name = "Porky Pig", HireDate = new DateTime(2010, 2, 1)
        };
        private readonly Employee _sylvesterTheCat = new Employee {
            Id = 1002, Name = "Sylvester the Cat", HireDate = new DateTime(2010, 3, 1)
        };
        private readonly Employee _bugsBunny = new Employee {
            Id = 1100, Name = "Bugs Bunny", HireDate = new DateTime(2010, 1, 1)
        };

        public void AddEmployees() {
            _repository.AddEmployee(_site, _duffyDuck);
            _repository.AddEmployee(_site, _porkyPig);
            _repository.AddEmployee(_site, _sylvesterTheCat);
        }

        public void ClearEmployees() {
            var definition = new EmployeesDefinition();
            var employees = _site.Lists[definition.ListName];
            while (employees.Items.Count > 0)
                employees.Items.Delete(0);
        }

        [TestInitialize]
        public void Initialize() {
            _siteCollection = new SPSite("http://localhost");
            _site = _siteCollection.OpenWeb("/");
            _repository = new EmployeesRepository();
            ClearEmployees();
            AddEmployees();
        }

        [TestCleanup]
        public void Cleanup() {
            _site.Dispose();
            _siteCollection.Dispose();
        }

        [TestMethod]
        public void AddEmployee_should_add_valid_employee() {
            _repository.AddEmployee(_site, _bugsBunny);
            var e = _repository.GetEmployeeById(_site, _bugsBunny.Id);
            Assert.AreEqual(_bugsBunny.Id, e.Id);
            Assert.AreEqual(_bugsBunny.Name, e.Name);
            Assert.AreEqual(_bugsBunny.HireDate, e.HireDate);
            Assert.AreEqual(_bugsBunny.Remarks, e.Remarks);
        }

        [TestMethod]
        public void GetEmployeesHiredBetween_should_return_2010_hires() {
            var from = new DateTime(2010, 1, 1);
            var to = new DateTime(2010, 12, 31);
            var employees = _repository.GetEmployeesHiredBetween(_site, from, to);
            Assert.AreEqual(2, employees.Count);
            Assert.AreEqual(_porkyPig.Id, employees[0].Id);
            Assert.AreEqual(_sylvesterTheCat.Id, employees[1].Id);
        }
    }

In the words of Martin Fowler, here’s the essence of the Repository pattern:

A Repository mediates between the domain and data mapping layers, acting like an in-memory domain object collection. Client objects construct query specifications declaratively and submit them to Repository for satisfaction. Objects can be added to and removed from the Repository, as they can from a simple collection of objects, and the mapping code encapsulated by the Repository will carry out the appropriate operations behind the scenes. Conceptually, a Repository encapsulates the set of objects persisted in a data store and the operations performed over them, providing a more object-oriented view of the persistence layer.

In SharePoint terms, business logic should query a repository which in turn queries a SharePoint list. Within the repository, the weakly typed items returned are then mapped to strongly typed data transfer objects, which are returned to the business layer.

With this approach to data access comes a number of advantages: (1) duplicate data access code is eliminated. Only within the repository do we setup the query and transform the weakly typed SPListItemCollection into strongly typed data transfer objects. (2) The list definition classes introduced in SharePoint list definition using the Template pattern may be used to construct CAML queries from strongly typed field names. Lastly, (3) accessing data through a repository makes it easier to mock the data access part of the application and to integration test that part.

The code below is a simple, yet usable, implementation of the Repository pattern. The idea is to have all repositories inherit from a common base class. Its purpose is to wrap the querying of a list and to log what’s going on. Because it’s a base class, it shouldn’t know how to transform the weakly typed result into strongly data transfer objects or which CRUD operations a particular repository supports:

    public abstract class ListRepository {
        protected ListDefinition Definition { get; set; }
        protected SPListItemCollection Result { get; set; }

        protected void Query(SPWeb site, string caml) {
            AssertValidSite(site);
            AssertValidCaml(caml);
            AssertListExistence(site);
            AssertListDefinitionSetBySubclass();

            var watch = new Stopwatch();
            var list = site.Lists[Definition.ListName];
            var query = new SPQuery {Query = caml};
            Debug.WriteLine(
                string.Format("About to run query against list '{0}': {1}", list, caml));
            watch.Start();
            Result = list.GetItems(query);
            watch.Stop();
            Debug.WriteLine(
                string.Format("Query against '{0}' returned {1} rows in {2} ms",
                              list, Result.Count, watch.ElapsedMilliseconds));
        }

        protected void AssertListExistence(SPWeb site) {
            if (!ListDefinition.ListExists(site, Definition.ListName))
                throw new ArgumentException(
                    string.Format("No '{0}' list on site '{1}'", Definition.ListName, site.Url));
        }

        protected void AssertListDefinitionSetBySubclass() {
            if (Definition == null)
                throw new NullReferenceException(
                    string.Format(
                        "Sublcass must set Definition property prior querying '{0}' list",
                        Definition.ListName));
        }

        protected void AssertValidCaml(string query) {
            if (string.IsNullOrEmpty(query))
                throw new NullReferenceException("Query must not be null or empty");
        }

        protected void AssertValidSite(SPWeb site) {
            if (site == null)
                throw new NullReferenceException("Site must not be null");
        }
    }

Each ListRepository connects to a corresponding ListDefinition, holding the name of the list to query and its strongly typed field names. It’s the responsibility of a concrete repository to set the Definition property prior to doing any querying. After running a query, the Result property holds the weakly typed result, which a concrete repository can then transform into data transfer objects to be passed to the business layer.

As an example, add to the concrete EmployeeRepository any CRUD method you see necessary to fulfill the business requirements:

    public class EmployeesRepository : ListRepository {
        public EmployeesRepository() {
            Definition = new EmployeesDefinition();
        }

        public void AddEmployee(SPWeb site, Employee e) {
            var list = site.Lists[Definition.ListName];
            var item = list.Items.Add();
            item[EmployeesDefinition.EmployeeId] = e.Id;
            item[EmployeesDefinition.Name] = e.Name;
            item[EmployeesDefinition.HireDate] = e.HireDate;
            item[EmployeesDefinition.Remarks] = e.Remarks;
            item.Update();
        }

        public Employee GetEmployeeById(SPWeb site, int id) {
            var caml =
                string.Format(@"
                      <Where>
                        <Eq>
                          <FieldRef Name=""{0}"" />
                          <Value Type=""Integer"">{1}</Value>
                        </Eq>
                      </Where>",
                    EmployeesDefinition.EmployeeId, id);
            Query(site, caml);

            IList<Employee> employees = Map(Result);
            if (employees.Count == 0)
                throw new ArgumentException(string.Format("No employee with id = {0} exists", id));
            return employees[0];
        }

        public ReadOnlyCollection<Employee> GetEmployeesHiredBetween(SPWeb site, DateTime from, DateTime to) {
            var caml =
                string.Format(@"
                      <Where>
                        <And>
                            <Geq>
                              <FieldRef Name=""{0}"" />
                              <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{1}</Value>
                            </Geq>
                            <Leq>
                              <FieldRef Name=""{0}"" />
                              <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{2}</Value>
                            </Leq>
                        </And>
                      </Where>",
                    EmployeesDefinition.HireDate,
                    SPUtility.CreateISO8601DateTimeFromSystemDateTime(from),
                    SPUtility.CreateISO8601DateTimeFromSystemDateTime(to));
            Query(site, caml);
            return new ReadOnlyCollection<Employee>(Map(Result));
        }

        protected IList<Employee> Map(SPListItemCollection items) {
            var employees = new List<Employee>();
            foreach (SPItem item in items) {
                var e = new Employee {
                                Id = (int)item[EmployeesDefinition.EmployeeId],
                                Name = (string)item[EmployeesDefinition.Name],
                                HireDate = (DateTime)item[EmployeesDefinition.HireDate],
                                Remarks = (string)item[EmployeesDefinition.Remarks]
                            };
                employees.Add(e);
            }
            return employees;
        }
    }

Calling the GetEmployeeById or GetEmployeesHiredBetween methods, the caller is required to pass in an SPWeb instance pointing to the the site holding the list to query. Outside the SharePoint context, you have to manually create this instance, like with the integration tests above. But within the SharePoint context, callers are likely to just pass in SPContext.Current.Web.

The above repository implementation deliberately ignores any issue of caching. If you find the need for it, however, you can replace SPList.GetItem with PortalSiteMapProvider.GetCachedListItemsByQuery. The advantage of using the PortalSiteMapProvider over Asp.Net caching of the result is that the provider takes care of invalidating cache entries when items are modified. The disadvantage is that the provider is part of the SharePoint publishing API, which isn’t part of WSS 3.0. In addition, it’s only available from code running within SharePoint, and likely requires cache settings to be tweaked.

Another, non-trivial, improvement would include the use of the Unit of Work pattern. A commonly used pattern in ORMs because it offers a way to “keep track of everything you do during a business transaction that can affect the database. When you’re done, it figures out everything that needs to be done to alter the database as a result of your work". Like with the DataContext class in LINQ to SQL or the DataSet class in ADO.NET, it could add transaction support to SharePoint lists.

Books on SharePoint often show how to create lists from code by calling the SharePoint API from directly within a feature receiver. This receiver would then contain in-place string literals for column names, create the columns, update the default view, and so on. While such an approach provides for an nice demonstration of the SharePoint API, it tends to carry over into production code.

My take on feature receivers, however, is that they should contain the least possible amount of code. Staying true to object-orientation and separation of concerns, list definition and creation should instead consist of a common set of classes and methods to be called from the receiver:

    public override void FeatureActivated(SPFeatureReceiverProperties p) {
        using (var site = p.Feature.Parent as SPWeb) {
            var employees = new EmployeesDefinition();
            if (!ListDefinition.ListExists(site, employees.ListName))
                employees.CreateOnSite(site);
        }
    }

The first area to improve on is that of string literal field names getting duplicated across the code base. String literals make it impossible for the compiler to enforce a consistent naming of columns. As column names are used in defining the list, in forming CAML queries against it, and in accessing the result, misspelled column names is a common source of runtime errors. The second area to improve on is that of duplication of code for doing common list operations, like checking for the existence of the list or removing its title column. Lastly, we want to get rid of duplicate control logic, i.e., verifying preconditions and calling the same series of methods with every list definition.

One approach to improving on these areas is to start by defining a base class for list definitions. The base class combines helper methods, working on lists, with the Template pattern for defining the common steps of list creation:

    public abstract class ListDefinition {
        public string ListName;
        public string ListDescription;
        protected SPList List;

        public void CreateOnSite(SPWeb site) {
            if (string.IsNullOrEmpty(ListName))
                throw new ArgumentException("Expected not null or not empty list name member");
            if (site == null) throw new NullReferenceException("Expected valid site");
            CreateList(site);
            if (List == null) throw new NullReferenceException("Expected list member not null");
            CreateFields(List);
            UpdateDefaultView(List);
            SetAdditionalProperties(List);
        }

        protected abstract void CreateList(SPWeb site);
        protected virtual void CreateFields(SPList l) {}
        protected virtual void UpdateDefaultView(SPList l) {}
        protected virtual void SetAdditionalProperties(SPList l) {}

        public static void HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(SPList l) {
            var title = l.Fields.GetField("Title");
            title.Required = false;
            title.Hidden = true;
            title.Update();
        }

        public static bool ListExists(SPWeb site, string listName) {
            if (string.IsNullOrEmpty(listName))
                throw new ArgumentException("ListNameMustNotBeNullOrEmpty", listName);
            SPList list = null;
            try {
                list = site.Lists[listName];
            }
            catch (ArgumentException) {
                // list not found
            }
            return list != null ? true : false;
        }
    }

At the core of the Template pattern is the template method. It defines the steps of an algorithm through a series of method calls. Each either abstract or virtual depending on if the step is mandatory or voluntary in the algorithm. Subclasses then specify the behavior of individual steps by implementing or overriding methods. In case of the ListDefinition class, CreateOnSite is our template method. It ensures that context, in the form of an SPList instance, is passed along to each step in the algorithm.

Each subclass defines the specific characteristics of a list. It defines constants for column names and their data types, the views and which column to index, and so on. This makes for a concise, easy to read, and consistent list definition:

    public class EmployeesDefinition : ListDefinition {
        public const string EmployeeId = "EmployeeId";
        public const string Name = "Name";
        public const string HireDate = "HireDate";
        public const string Remarks = "Remarks";

        public EmployeesDefinition() {
            ListName = "AcmeEmployees";
            ListDescription = "Employees at Acme Corp.";
        }

        protected override void CreateList(SPWeb site) {
            var guid = site.Lists.Add(ListName, ListDescription, SPListTemplateType.GenericList);
            List = site.Lists[guid];
        }

        protected override void CreateFields(SPList l) {
            l.Fields.Add(EmployeeId, SPFieldType.Integer, true);
            l.Fields.Add(Name, SPFieldType.Text, true);
            l.Fields.Add(HireDate, SPFieldType.DateTime, true);
            var remarks = (SPFieldMultiLineText) l.Fields[l.Fields.Add(Remarks, SPFieldType.Note, false)];
            remarks.RichText = true;
            remarks.RichTextMode = SPRichTextMode.FullHtml;
            remarks.Update();
            l.Update();
        }

        protected override void UpdateDefaultView(SPList l) {
            var v = l.DefaultView;
            v.ViewFields.Delete("LinkTitle");
            v.ViewFields.Add(EmployeeId);
            v.ViewFields.Add(Name);
            v.ViewFields.Add(HireDate);
            v.ViewFields.Add(Remarks);
            v.Update();
        }

        protected override void SetAdditionalProperties(SPList l) {
            HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(l);
            l.Fields[HireDate].Indexed = true;
            l.Update();
        }
    }

Since the string literals have now become public constants, we get IntelliSense and compile-time checking referencing them. In addition, by defining common operations on the base class and using the Template pattern, we reduce the amount of code that would otherwise have to go into each definition. Finally, the base class asserts, on behalf of all subclasses, that the list name is set before attempting to create the list and that the SPList field is set after calling the CreateList method.

My goals for the coming year are:

• Focus more on architecture and design. Not in the architecture astronaut connotation. But as in becoming more aware of how to build better software. I want to apply the SOLID principles more consciously and re-read classics such as Design Patterns: Elements of Reusable Object-Oriented Software and Patterns of Enterprise Application Architecture.
• Continue exploring LINQ. It’s a fascinating technology that’s going to get only more pervasive in years to come. Truly understanding LINQ is key to getting into the habit of applying a more functional and succinct style to my own code. To get started on this endeavor I’m going to read and apply LINQ in Action.
• Learn Entity Framework. Using an object-relational mapper should be second-nature to any developer. In the past the light-weight LINQ to SQL has served me well. Still, I want to step up and learn how Entity Framework deals with the impedance mismatch. I want to write an application that uses Entity Framework.
• Become better at browser technologies. JavaScript and jQuery are high on the list, but I want to improve on HTML and CSS as well. My goal isn’t to become an HTML or CSS Jedi. I simply want to have enough knowledge to avoid making rookie mistakes. The technologies all come together in jQuery, making Learning jQuery a good starting point. Besides, I want to use a tool like Firebug to learn from others.
• Be on top of SharePoint 2010. SharePoint 2007 and 2010 is what I want to work with daily. That doesn’t imply me not keeping up to speed with the rest of .NET. Just that SharePoint will act as my gateway to other technologies. The great thing about SharePoint is that development is often split between working inside and outside SharePoint.
• Become more involved in higher-level development decisions. By the end of the day, developers don’t deliver software, they deliver value to stakeholders. Writing code is but a means to that end. I want to be involved in all steps from conception to completion.

To me these points strike a good balance between technical and non-technical goals and between short and long term goals. However, by the end of the day, building great software is about being attentive to the entire stack and its surroundings.

So how did the year go?

• Changed jobs to one that gravitates more around larger projects. The rationale being that larger projects come with larger engineering challenges. I spend six months on an Asp.Net project followed by three on a SharePoint project. For various reasons, I wasn’t as involved in higher-level development decisions as I would’ve liked to.
• Did two presentations. One on Unit testing and mocking for the Asp.Net project that I was working on at the time. The other presentation was on Parallel page rendering with Asp.Net and SharePoint for my current project. None involving rocket science, but I find so few are familiar with the techniques.
• Made my way through three certifications. The 70-536: MS .NET Framework, Application Development Foundation, 70-562: MS .NET Framework 3.5, ASP.NET Application Development, and Prince2 Foundation. I’ve mixed feelings with regard to certifications, though. Certainly, I learned valuable lessons studying for each one. Yet, given the same amount of time, I’m inclined to think I would’ve learned more studying books, blogs, listening to various podcasts, and writing.
• Attended the Jaoo conference on software engineering. It was my first time ever attending a conference and it sure was enlightening. I intentionally went for a conference not tied to a specific vendor because I felt I’d learn more that way. My issue with Microsoft conferences like Tech-Ed or PDC is their focus on the latest technology buzz. Buzz that quickly becomes obsolete or is already heavily exposed on the web. Not to mention that most sessions are available online afterward.
• Learned functional programming. My focus on new technology has mostly been on various aspects of functional programming. A subject that’s interested me for years, but one that’s only recently started to permeate C#, my language of choice. I learned a great deal about F# when I generated fractal terrains and about XSL when I had to jump through functional loops and about functional programming in general.
• Wrote 16 blog posts (25 pages, 10,000 words). Compared to previous years it’s a record. Naturally, not all posts were equally good. Nevertheless I enjoyed learning about the subjects and feel they helped advance my writing skills. The posts on Basic unit testing guidelines and Why not to comment code did manage to gain traction on Reddit, though.

All things considered, I’m pleased with 2009. The only thing I regret is not being more involved in higher-level development decisions. I like coding, but it’s a means to an end. Being involved with a task from start to finish is what ignites my fire. I must push harder on that in 2010.

I started out confident that printing would be easy to fix. But after Googling and playing around with JavaScript for a while, frustration, fueled by the number of non-working hits I’d stumbled on, started to grow on me. Suppose you start out with an HTML document hosting an iframe and a few lines of JavaScript.

  <html>
    <head>
      <title>IFrame printing example</title>
      <script type="text/javascript">
        function myPrint() {
          var browser = navigator.appName;
          if (browser == "Microsoft Internet Explorer") {
            window.frames["frame"].focus();
            window.frames.print();
          }
          else if (browser == "Netscape")
            alert("Printing not supported in Firefox");
        }
      </script>
    </head>
    <body>
      <iframe id="frame" name="frame" width="100"
              height="100" src="http://google.com">
      </iframe>
      <a href="#" onclick="javascript:myPrint();">Print</a>
    </body>
  </html>

A number of issues arise when printing the iframe. First you need to consider if you’re always hosting the main document on the same domain as the iframe’s source. If so, printing only the iframe with Internet Explorer works like a charm. If, on the other hand, the main document and the iframe’s source reside on different domains, it triggers Internet Explorer to displays a bar across the top of the window.

Now, to print the iframe’s content, you must allow blocked content. Only then can JavaScript access the iframe’s object model. But at the same time, you make yourself vulnerable to cross-site scripting attacks.

As far as Firefox support goes, you can issue the same API calls as for Internet Explorer. But Firefox will always print the main page, including only the visible parts of the iframe. To my knowledge there’s no way to make Firefox print the iframe’s content as seamlessly as Internet Explorer. One workaround, though, would be to open a new window, grab a reference to the iframe element, and move its contents to the new window before printing it. For cross-domain content, however, Firefox responds with a “permission denied” error when accessing the iframe element. And there’s no option to allow blocked content. Another workaround may therefore be to use Ajax to retrieve the source of the iframe for any domain and insert it into the new window. That, however, wouldn’t work in my case since the iframe may contain a form that the user may have started filling out before hitting print.

As a last resort, I considered writing a proxy to retrieve cross-domain hosted content. This would make Internet Explorer never display the bar and allow Firefox to always open a new window with the iframe’s content loaded, even if the user started filling out a form first. But this approach sort of defies the purpose of the browser bar and the permission denied error. Unless you absolutely trust your iframe’s source, you’re exposing your users without their consent.

In the end I went with the simplest and most secure solution of not supporting printing at all.

With Asp.Net, by default a page is rendered synchronously, one control at a time. The thread rendering the page comes from the thread pool within the world wide web worker process (w3wp.exe) and is assigned to the request when it comes in on the web server. Only when the page has fully rendered can the thread go back in the pool waiting for the next request.

Processing a complex page using only one thread, however, may cause the page to render slowly. The page may hold controls that require calls to web services, databases, or other external resources for it to render. Whenever the worker thread encounters such a control, it goes idle waiting for data to come back from the external resource. Only when data is received can the worker thread resume processing.

One way to speed up page processing would be for the page or control to explicitly make use of threads. Depending on the situation, this may be a cumbersome task that requires a significant amount of boiler-plate code. A simpler approach might be to turn to the Asp.Net 2.0 feature for asynchronous page processing. At its core is an extension to the page or control lifecycle.

Except for the Begin and End events, the flow is the same in both models. But while the synchronous model executes all code on a single thread, the asynchronous model makes implicit use of multiple worker threads to speed up page processing. With multiple threads, some threads still go idle waiting for input, but at least the main worker thread continues to work its way through the controls.

In the asynchronous model, we hook into the Begin and End handlers in one of the events preceding the call to Begin. Following PreRender, Asp.Net will then execute our Begin handler, where we call out to some method that carries out the long-running task. When that method exists the End handler is called and processing carries on as in the synchronous model.

To illustrate the flow and timing of the asynchronous model (turn to the slides for the complete code sample), suppose we add five controls to a page. Each control takes five seconds to render so using the synchronous model the page would take 25 seconds to render. Using the asynchronous model, and writing out trace information about which thread each method executes on, here’s what happens:

1: Page_Load: 10            9: DoWork: 8
2: Page_Load: 10           10: DoWork: 11
3: Page_Load: 10           11: Render: 11 15:16:13 15:16:18
4: Page_Load: 10           12: Render: 11 15:16:13 15:16:18
5: Page_Load: 10           13: Render: 11 15:16:13 15:16:18
6: DoWork: 4               14: Render: 11 15:16:13 15:16:18
7: DoWork: 10              15: Render: 11 15:16:13 15:16:19
8: DoWork: 9

The user control’s Page_Load and the code making up the Begin handler executes on the main worker thread. But the method carrying out the long-running task, and the code for the End handler, executes in parallel on different worker threads. Finally, after End is called, another worker thread takes over and executes the remaining parts of the control. Now page rendering has decreased from 25 to about five seconds.

All in all, little effort is required to make more efficient, and implicit, use of the thread pool, thereby decreasing the time required processing a page. Asynchronous page processing should be used judiciously, though. Optimizing away a bottleneck in the processing of a page may well lead to bottlenecks appearing elsewhere, e.g., the number of simultaneous calls made to a web service may be too much for it to handle.

For more details and examples, turn to Jeff Prosise‘s article on Asynchronous Pages in Asp.Net 2.0.

In a nutshell Org-mode is a plug-in for Emacs that makes taking notes, maintaining lists, clocking time, and many other tasks more straightforward. The mode works by adding features on top of Emacs for editing plain text files. Features that enrich the interaction with the underlying text buffer by parsing and adding (meta-)information to it.

With Org-mode you group items into a hierarchical structure. A path to the item to work on is then formed by unfolding items in a way analogous to how you navigate the file system with Explorer.

To efficiently work with items, Org-mode supports moving items around the hierarchy. Items may also be tagged (buy, work, phone, etc.) or associated with a state (done, waiting, started, etc.). Additional meta-information may include a deadline, an estimate, or time spent working on a task. The clocking of time is accomplished by Org-mode adding a timestamp to the item. On completion, another timestamp and the elapsed time is calculated and added.

Below is an example of an expanded and a collapsed item containing clocked times and estimates.

With a few keystrokes, Org-mode is able to correlate estimates with time spent on an item-by-item basis. This truly short-circuits the estimation feedback cycle. Imagine doing this on a file containing a large hierarchy of items and the transparency it brings.

If you want to keep track of billable hours, it may make sense to clock all your time using Org-mode and generate timesheets from the clocking information. Especially since the billing context is automatically provided by the item storing the timestamp, and since you’re free to organize the file in any way you see fit. Few other tools manage to compete with the flexibility of a plain text buffer.

This post only scratches the surface of Org-mode. To get a sense of the possibilities, the manual amounts to 184 pages. However, for a quick introduction I encourage you to watch the Google Tech Talk on Org-mode.

From Oct 5 to Oct 7, I was fortunate enough to attend the Jaoo 2009 conference on software engineering in Aarhus. It was a great conference with three days of insightful talks. The talks I attended were:

Monday

• Keynote: Scaling Up Agility: The Architected Agile Approach/Barry Boehm (slides)
• Working Effectively with Legacy Code 2/Michael Feathers (slides)
• Evolving the Key/Value Programming Model to a Higher Level/Billy Newport (slides)
• Techniques That Still Work no Matter How Hard We Try to Forget Them/Keith Braithwaite (slides)
• Making use of Patterns/Martin Fowler
• xUnit Test Patterns – Refactoring Test Code to Improve ROI/Gerard Meszaros (slides)

Tuesday

• Keynote: Classes, Jim, But Not as We Know Them/Simon Peyton-Jones (slides)
• Agile Modeling and Documentation “Best” Practices/Scott Ambler (slides)
• Actors, and the Forgotten Art of Modeling Concurrent Systems/Dave Thomas, Kresten Krab Thorup (slides)
• Buy it, Build it, Download It, or Browse It? Achieving Effectiveness with Enterprise Applications/Michael T. Nygard (slides)
• Hydras and Hypermedia/Ian Robinson (slides)
  
• Maximum Value, Maximum Speed through Lean Thinking – How Business and IT can Collaborate/Dave Thomas

Wednesday

• Keynote: C++, Java and .NET – Lessons Learned from the Internet Age, and What it Means for the Cloud and Emerging Languages/Cameron Purdy (slides)
• Guiding Your Personal Life — “Plan-Driven” or “Agile”/Linda Rising (slides)
• Object Relational Mapping += 2: More than just data <-> object/Oren Eini
• Deliberate Practice in Software Development/Mary Poppendieck (slides)
• Umbraco: An Open Source Web CMS that Devs Will Like … Really? (slides)
• Delivering Quantified Stakeholder Values with Agile – A Case study from Bring/Kai Gilb (slides)

As a supplement to the official homepage, the Twitter feed provided a great source of information for attendees to learn about the presentations and reactions. Not being on Twitter meant missing out on half the conference.

For other reactions, listen to (in Danish) Daniel Frost and Brian Rasmussen discussing day one and two. On day three Daniel interviewed Mads Torgersen of the C# core team.

I was just listening to the Why comments are evil episode of Deep Fried Bytes. The views put forward by the guest, Corey Haines, deeply resonate with me because I believe they’re what every developer should strive for. I’m not advocating that code should be entirely free of comments, but rather that most comments do more harm than good.

Below I’ve listed the main points of view.

• It’s important to differentiate between code comments and API documentation. For API documentation, don’t auto-generate. Especially don’t go with tools like GhostDoc. Take the time to document properly. Otherwise, you’re more likely to confuse the developer trying to make sense of your code.
• Stop focusing on having the computer understand you and start focusing on the person that comes after you, e.g., while there’s some overhead to method calls, unless the profiler tells you otherwise, don’t hesitate to break down a method into as many smaller methods as it takes to make it more approachable.
• Every time you’re tempted to comment, take the time to think if you can make the code tell its own story, e.g., instead of some branching condition, extract the condition into a method to emphasize the story of why over how.
• Developers comment to explain a confusing part of their code rather than take the time to refactor it by extracting a method or naming a variable properly. Inevitably the code gets modified and the comment rots.
• Parts of a method may be complicated so a comment is added. Saying part of a method is complicated is in itself a sign that it’s doing too much.
• The prevailing culture among developers is to focus on what it takes to get code working. It’s a battle of short-term vs. long-term maintainability and short-term almost always comes out as the winner. Writing code, ask yourself if you’re able to immediately understand the code a month from now.

The opinions put forward in the episode aren’t new. Still, I find them well worth iterating because as developers we tend to not pay enough attention to them. We tend to focus too little on good software craftsmanship.

Like with any habit, every so often you should take the time to review it. I find TimeSnapper (the free edition) to be an invaluable tool for reviewing computer habits. It captures an image of the desktop every few seconds and is able to create a movie from the images. That way, a partial life log (mine is currently 393 days) is automatically created to help you remember what you did yesterday and the day before.

There’s nothing like reviewing a few of weeks of captured images to make habits stand out. I found that I’ve a tendency to re-visit the same maybe ten sites all too often. I also have difficulty single-tasking. Listening to a podcast or watching a movie, I’m often browsing the web instead of paying attention to the podcast or skipping it altogether. I may even have several media players launched at once, although only one playing.

Most of my distractions can be attributed to me always being online. I need Internet access for work, so turning it off altogether isn’t an alternative. So, not unlike Paul Graham, on several occasions, I’ve made myself adhere to a set of rules for using the Internet. And every time they’ve worked — for a short while.

In Disconnecting Distraction, Paul takes these rules one step further and argues that using two computers may be a better solution: one computer with Internet access and one without for doing real work. Ideally, the computers should be placed some distance apart to make you psychically get up to go on the Internet.

For a week I’ve given the two-computer approach a try. It was hard initially, but I’ve come to appreciate it. Forming permanent habits takes time. Thus, I’ll continue the experiment for a couple of weeks. Perhaps in time I’ll even learn to turn off instant messaging and only do mail once or twice a day. Or I may end up spending most of the time in front of the connected computer.

Then again, there’s more to life than work.

Not logging implies that debugging the application outside the development environment is largely based on guesswork. That’s interesting given that most applications spend the majority of their life running outside the development environment.

Life in the sandbox

I suspect the absence or low quality of log statements can be, at least partially, explained by developers not finding them useful during development. The debugger available there is generally inferior to the print-lining approach of logging. That said, I’d argue that many developers don’t concern themselves enough with how their code behaves in a production environment. Rather than investing a small amount of time in adding log statements up-front, their focus is solely on satisfying functional customer requirements.

A related case could be made for exception handling. For instance, rather than asserting the validity of an argument up-front and throwing an appropriate exception, the code may later fail with a NullReferenceException. Not throwing the exception up-front, close to where the actual error occurred, one will have a hard time tracking down the issue. Especially since a stack trace with symbols and line numbers is rarely available in a production environment.

For someone having to resolve the issue, almost any clue as to the cause would be most welcomed. Without a clue that someone would walk around in the dark in the hope of stumbling into a solution.

End users aren’t the only users

As software developers we have to acknowledge that end users aren’t the only users of our software. System administrators, developers, and the like should also be factored in during development. Especially since the cost of instrumenting code at key places, early on, doesn’t add significantly to the overall development cost. Focusing on what can go wrong and communicating it well saves time, money, and frustration down the line. And as a bonus you might create better self-documenting code.

That said it’s important to strike a balance between logging too little and too much information. A log should be detailed enough to reveal patterns of use. Ideally, the log should read like a story with short sentences detailing what’s about to happen, what did or didn’t happen, and possibly why.

I’m not suggesting that we turn logging into a runtime version of literate programming. But simply that sufficient contextual information be provided, i.e., instead of logging “URL = ‘bugfree.dk’”, consider the more elaborate “Retrieving URL for indexing: ‘http://bugfree.dk’”.

Keep your statements clean

On the other hand, log statements shouldn’t be immune to the DRY principle. So make sure to have the logging framework automatically capture common bits of contextual information, such as the date and time of the entry and the class and method within which the log statement originated. On a similar note, don’t explicitly log when a method is entered or existed. Instead, treat such a case as a cross-cutting concern and take an Aspect Oriented approach to logging it.

Lastly, not every method requires (extensive) logging. Figuring out when and what to log is key. Debugging the application, fixing bugs, and experience should provide guidance into what’s useful to log.

Update, Aug 5, 2009: This post grew out of a couple of presentations that I did on the subject back in May and June. Here are my slides from The Unit testing and mocking presentation. Among other things, they contain a few C# samples elaborating on some of the points below.

This post is born of the need to formalize a set of guidelines on how to write and organize tests. In the past I couldn’t help feeling that with the lack of guidelines I had to start over explaining my views on every new project. With no guidelines the tests written quickly grew unmaintainable, giving unit testing a bad name.

Why unit test

Unit testing done right helps build confidence in the code base and drives forward development. Both in terms of forcing one to reflect on the requirements by authoring tests, but also by providing a foundation for developing more modular and testable code.

Given the cyclomatic complexity of even simple methods, however, not every path through a method is worth exercising with a test. Instead, focus on writing representative unit tests for good and bad scenarios.

Unit test != integration test

For object-oriented code a unit test is one that exercises a class in isolation, without the code under test relying on other classes to carry out its operation. Similarly, a unit test shouldn’t rely on the presence of a database or a key/value pair in a configuration file for it to run. If it did, it would be an integration test. Not that there’s anything wrong with integration tests. They just take on more dependencies and hence tend to be more brittle. And so they cause more false positives because some dependent part isn’t properly configured. The key point, though, is not to confuse unit testing with integration testing.

Typically, an object delegates part of its operation to other objects. Writing modular and testable code therefore involves the application of design patterns, such as Inversion of Control (IoC). With IoC a test can inject (fake) components into the object under test. The fake components share their interfaces with the real ones, but the test controls how they interact with the object under test and hence how the object under test behaves.

Keep up quality

As tests are written it’s vital for the understandability and maintainability of the test suite to keep them small and focused. As a rule of thumb a test should amount to no more than 10-15 lines of code. Longer tests are indicative of too much functionality being tested at once or that code common to multiple tests should be refactored into helper methods.

In terms of quality, the code comprising the tests should be of production code quality, i.e., the code must be kept clean and refactored as the need arises. Otherwise, tests will start to emit the classic code smells. In the longer run code smells lead to tests that are not maintainable and for the time that went into writing them to be wasted.

Organizing test code

Assuming the use of Visual Studio (VS), for each VS project with code that one wants to test, create matching projects that host the various kinds of tests, i.e., for the Acme.Intranet.Search project, create the following test related projects:

    Acme.Intranet.Search.Common
    Acme.Intranet.Search.UnitTest
    Acme.Intranet.Search.IntegrationTest

The Acme.Intranet.Common project is optional and may include code that is shared between test projects, such as custom assertions. As for the Acme.Intranet.Search.UnitTest project it should be fairly self-contained. One should be able to move the common assembly, the test assembly, and the business code assembly to another machine and have the tests execute there without further setup. Should a test rely on, say, a data file with test data, then include the file as an embedded resource within the test assembly.

Finally, within each test project, a class should be created for each class under test. In addition, the directory structure should match the namespace structure of the class under test, e.g., suppose the fully qualified name of a class is Acme.Intranet.Search.Business.Crawler, then create the following directory structure within the test assembly:

    Acme.Intranet.Search.UnitTest
        Acme
            Intranet
                Search
                    Buesiness
                        CrawlerTest.cs

While it’s important to write tests, it’s even more important to know where to put and find tests for a given functional area.

Naming tests

As far as naming and structure goes, a test should look something along these lines:

    [TestClass]
    public class SomeClassTest {
        [TestMethod]
        public void SomeMethod_should_set_error_message_when_no_                    connection_string_is_configured() {
            // arrange
            // act
            // assert
        }
    }

The test should generally start with the name of the method or property being tested, followed by the word "should" followed by the successful outcome in a descriptive form. Because names of tests tend to be longer than those of regular methods, underscores are used for ease of readability. In addition, the body of most tests should be composed of three parts: (1) the arrange part that sets up the object under test and possibly injects fake dependencies into it. (2) The act part then exercises the method under test, and finally (3) the assert part that verifies that expected state and/or behavioral changes did indeed take place.

Gathering metrics

Whenever a build is kicked of (on a build server) it should exercise all tests. Should a test fail, it should cause the entire build to fail, stressing the importance of keeping tests green at all times. Furthermore, a build report should include basic metrics such as code coverage, number of tests run, time spend running the tests, and so forth.

Keep in mind, though, that a high degree of code coverage isn’t a goal in itself. Instead, focus on writing solid, focused, and representative tests that eventually drive up code coverage.

Tooling

As far as .Net and tooling goes, MSTest or NUnit should be used in concert with TypeMock or Rhino Mocks. The use of TypeMock may be preferred over Rhino Mocks because of its unique approach to mocking. TypeMock doesn’t create fake objects by emitting MSIL and dynamically loading a runtime-generated assembly into the test runner. Instead, TypeMock hooks into the CLR APIs and intercepts calls as the unit test executes. From a coding point of view the TypeMock approach may not change much, but from a functionality point of view it enables the testing of legacy code or new code not written with IoC in mind.

    for (int i = 0; i < 5; i++)
        // do something

No such for-construct exists in XSL. In fact there’re no looping constructs of any kind in XSL, except for the one used to iterate nodes. However, since XSL and C# are both Turing complete languages, and since anything that can be expressed in one Turing complete language can be expressed in another, it follows that it must be possible to express a loop with a counter in XSL.

But as Alan Parlis puts it in Epigrams on Programming:

Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy.

Although XSL and C# are both Turing complete languages, XSL is a functional programming language and therefore belongs to a strain of languages that forgo build-in looping constructs. Instead, looping is accomplished through recursion.

We can turn the iterative C# loop into a recursive one like so:

    void Loop(int i) {
        if (i > 0) {
            // do something
            Loop(i - 1);
        }
    }

and transform the recursive solution into an XSL template:

    <xsl:template name="Loop">
        <xsl:param name="i" select="5"/>
        <xsl:if test="$i > 0">
            <!-- do something -->
            <xsl:call-template name="Loop">
                <xsl:with-param name="i" select="$i - 1"/>
            </xsl:call-template>
        </xsl:if>
    </xsl:template>

Okay, but what about nested loops then:

    for (int i = 0; i < 5; i++)
        for (int j = 0; j < 5; j++)
            // do something

Giving it some thought, the nested loops (of any nesting level) can be expressed using recursion following a pattern of nested if-statements:

    void NestedLoops(int i, int j, int k) {
        if (i > 0) {
            // do something
            if (j == 1)
                NestedLoops(i - 1, k, k);
            else
                NestedLoops(i, j - 1, k);
        }
    }

Again, performing a one-to-one transformation to XSL:

    <xsl:template name="NestedLoops">
        <xsl:param name="i"/>
        <xsl:param name="j"/>
        <xsl:param name="k"/>
        <xsl:if test="$i > 0">
            <!-- do something -->
            <xsl:choose>
                <xsl:when test="$j = 1">
                    <xsl:call-template name="NestedLoops">
                        <xsl:with-param name="i" select="$i - 1"/>
                        <xsl:with-param name="j" select="$k"/>
                        <xsl:with-param name="k" select="$k"/>
                    </xsl:call-template>
                </xsl:when>
                <xsl:otherwise>
                    <xsl:call-template name="NestedLoops">
                        <xsl:with-param name="i" select="$i"/>
                        <xsl:with-param name="j" select="$j - 1"/>
                        <xsl:with-param name="k" select="$k"/>
                    </xsl:call-template>
                </xsl:otherwise>
            </xsl:choose>
        </xsl:if>
    </xsl:template>

Wordy, but seeing through the angle brackets I find solution quite elegant.

Consider the state of computer-based word processing in 1977 when WordPerfect was conceived of. Afraid that word processors would interfere with so-called important applications, companies did word processing on dedicated mini-computers and mainframes. These machines were operated by a word processing department of employees that typed up documents. It wasn’t until the IBM PC that corporate users began typing documents themselves.

The early years

Pete’s way into WordPerfect — and his initial exposure to computers — was by way of his brother-in-law, Bruce Bastian. Following Bruce’s computer science degree in 1978, Bruce joined Alan Ashton to realize Alan’s vision of a word processor for a Data General mini-computer. It was then, in 1979, that Pete joined the two as office manager. By 1992, when forced to leave WordPerfect, he’d turned vice president of marketing.

Alan’s unique ideas involved crafting a word processor that didn’t require entering codes to format and layout documents. Compared to HTML authoring of today, contemporary word processors required the author to mix text and codes. Like tags in HTML, the codes described properties of the document, such as margins, line feeds, and emphasized text, and were specific to each product. Alan’s word processor, although character-oriented by nature, was to display the underlying codes using textual effects, creating a rough WYSIWYG experience.

Other innovative features included the ability to scroll through a document line by line, rather than page by page, and non-modal editing. Contemporary word processors required the author to switch between command and text modes the way the Vi family of editors still do. Without menus, modes made it easier to separate commands from text. Nonetheless, WordPerfect was modeless and hence characters could be typed in anywhere and anytime.

In the late seventies, word processors (and operating systems) were largely written in assembler and hence required rewriting for each hardware platform. Up until the IBM PC version, assembler also dominated WordPerfect, which made it a challenge to maintain considering the plethora of supported platforms. Migrating to higher-level languages had been considered, but was discarded on the premise that it would result in slower applications with bigger memory footprints. On that note, it appears we haven’t learned as we keep rewriting the same fundamental applications and frameworks. Not in assembler, but in higher-level languages.

Microsoft catching up

The 1981 introduction of the IBM PC pawed the way for a shift in the way software was developed and marketed. For vendors, the common architecture made lock-ins harder, yet provided access to a size of user base never before seen. WordPerfect was quick to seize the opportunity and released WordPerfect for the IBM PC in 1982.

Although the software marked for the IBM PC was dominated by Microsoft, for years Word (and other word processors) couldn’t keep up with the features and sales of WordPerfect. Two technologies, however, enabled Microsoft to overtake WordPerfect: laser printers and graphical user interfaces. For years WordPerfect had supported a few laser printers and had only a text-based interface. By 1987, though, it had become clear that graphical user interfaces was the future. It just wasn’t obvious weather Windows, of which version 2 had just been released or OS/2, whose development was announced that year, or some other vendor would end up dominating the market.

The final years

Moving from text to graphics posed a challenge for WordPerfect. Under DOS, the operating system provided only rudimentary services for drawing on the screen. For proportional font and image rendering, WordPerfect would itself have to take on rendering or use the services of a graphical user interface. But since it wasn’t clear whether Windows, OS/2, or some third environment were to dominate the future, WordPerfect decided to only augment the text-based interface with a graphical preview page. At the same time Microsoft released its first version of Word for Windows 2.0, relying on Windows for proportional font rendering, image display, and decent laser printer support.

In light of present knowledge, it’s all too easy to pass judgment on past decisions. Nevertheless, it appears that the inability of WordPerfect to keep their eyes on the rolling ball had caused Microsoft to take over by the early nineties. Instead of focusing on WordPerfect for Windows, substantial resources went into developing versions of WordPerfect for other platforms, developing a spreadsheet and a database application, and many more.

For years the availability of WordPerfect on almost any conceivable platform had supported the growth of the company. Now, the inability to cut losses and retain focus dragged WordPerfect into oblivion.

Non-authoritative DHCP servers on the network have posed a challenge for years. At first, I attributed the DHCP servers to badly configured Windows machines. However, the number of DHCP servers suggested that another explanation might be closer to the truth. Users accidentally installing DHCP servers are just not that common, and inspecting a couple of machines in question they were acquitted. What else could be the source of the DHCP traffic, then? Common to the machines were that wireless routers sat between them and the LAN.

After much investigation, it turns out that a lot, if not all, wireless routers expose their DHCP server on all ports of the router, when it’s only necessary to expose it on inward-facing ports. I guess vendors don’t bother limiting the exposure because generally it’s of no concern to consumers. ISPs provide the wireless router with the necessary configuration parameters (IP address, subnet mask, default gateway, DNS servers, and so on) in addition to blocking DHCP traffic originating from the wireless router.

On our network — equipped with eight aging 48-port Cisco Catalyst 3500 XL switches — DHCP traffic cannot be prevented from escaping the wireless router and making its way into the switches and onto LAN. The configuration parameters of rogue DHCP servers, intended for internal use only, are transmitted to external DHCP clients. Clients for which the supplied gateway, DNS server, and so on are only internally available. The DHCP client, unable to discern the rogue servers from the authoritative one, passes the parameters on to the network stack, setting itself up for network disconnectivity.

The essence of the issue is that whenever a client requests configuration parameters, it, by definition of the DHCP protocol, broadcasts a message to be picked up by any DHCP server on the network. The client then awaits an offering of configuration parameters from one or more DHCP servers. Then, still according to specification, the DHCP client may use any strategy to accept one of several offerings. I can only guess as to how Windows makes its decision, but randomness and first responder probably play a key role. In any event, Windows may keep accepting offerings from rogue DHCP servers regardless of the number of times I command the IP configuration to “ipconfig /release” and “ipconfig /renew”.

How to counteract rogue DHCP servers? There’re only a limited set of options: hard-coding the configuration parameters of individual machines, establishing VLANs so computers become invisible to each other, or replacing the Cisco Catalyst 3500 switches with contemporary switches that are able to suppress DHCP traffic travelling the wrong way.

The first step along the way is the maxDisplacements function that, given a number of splits of line segments, returns a list of maximum displacements. Here list comprehension comes in handy as a way to generate maximum displacements where the next is half of that of the previous, starting with 0.5:

    let maxDisplacements n = [for i in 1 .. n -> 2.0**(float)(-i)]

Next is the composeCoordinates function whose job it is to infer the x coordinates from the y coordinates. When splitting line segments, as defined later by the split function, only y coordinates need to be dealt with. But for displaying the result as a list of points, e.g., as input to a graphing program, composeCoordinates generates the (x, y) coordinate pairs given a list of y coordinates. Again list comprehension is used to uniformly distribute the x coordinates in the range 0 to 1. Next, the x and y coordinates are fused using the build-in zip function, i.e., [x1; x2] and [y1; y2] become [(x1, y1); (x2, y2)].

    let composeCoordinates (ys : list<float>) =
        let dx = 1.0 / (float)(ys.Length - 1)
        let xs = [for x in 0 .. ys.Length - 1 -> (float)x * dx]
        List.zip xs ys

Now for the crux of the fractal generation process: the split function. As is customary with functional languages, looping is done using recursion and split is no exception. Passing in a list of y coordinates, a maximum displacement as generated by maxDisplacements, and a random number generator, split is defined as:

    let rec split (ys : list<float>) displacement (random : Random) =
        match ys with
        | a :: b :: tail ->
            let m = b - a / 2.0
            let r = random.NextDouble() * displacement
            a :: a + m + r :: split (b :: tail) displacement random
        | a -> a

Using pattern matching, an if-then-else construct on steroids, split examines its input of y coordinates to determine if the inductive or the base case of the recursive implementation needs to be invoked. The first pattern matches a list starting with elements a and b following by zero or more elements. In case this pattern matches, a new list is returned. It’s composed of the original a element, a new displaced element, the original b element, and split applied to the original list with its first element removed. Removing one element at a time, at some point only one element is left in the list in which case the base case is invoked. When this happens, all the original line segments have been split in two.

Finally, the main function ties together the previous ones. Given an initial line segment, the number of times to apply split to it, and a random number generator, the actual splitting is done using the build-in fold_left function. fold_left is an example of fun with folds and a function that is itself recursively defined to make looping implicit.

    let main() =
        let initial = [0.0; 0.0]
        let splits = 8
        let r = new Random()
        let ys = List.fold_left (fun s d -> split s d r) initial (maxDisplacements splits)
        print_any (composeCoordinates ys)

While initially hard to grog, the use of fold_left becomes clearer when applied to a list of integers to compute their product. Of the three arguments to fold_left the first is a function defined by a lambda expression (here the predefined multiplication operator is also applicable but less explicit). The second argument is the initial value of an accumulator. And finally, the third argument is the list to work on.

    List.fold_left (fun x y -> x * y) 1 [1;2;3]
    List.fold_left (*) 1 [1;2;3]

What fold_left does is recursively call itself passing along the first argument as is. The second argument is the new value of the accumulator, computed by passing to the function of the first argument the original value of the accumulator and the first element of the list. That way it’s up to the function passed as the first argument to decide what operation to apply to the elements in the list. The third argument is the original list with the first element removed. The recursive chain of calls with their arguments then takes on this form:

    List.fold_left (fun 1 1 -> 1 * 1) 1 [1;2;3]
    List.fold_left (fun 1 2 -> 1 * 2) 1 [2;3]
    List.fold_left (fun 2 3 -> 2 * 3) 2 [3]

Within the main function the first argument to fold_left is a lambda expression that applies the split function to a list. The second argument, the accumulator, isn’t a scalar like with the product example, but a list of line segments to be passed to split. Finally, the third argument is the maximum displacement list. And so for the original line segment to be split n times, n maximum displacements must be provided by the maxDisplacements function.

What’s so fascinating about the Mandelbrot set, and other fractals, is that the often simple equations that define them are able to give birth to such complex creations.

In general, a fractal is defined as:

“[…] ‘a rough or fragmented geometric shape that can be split into parts, each of which is (at least approximately) a reduced-size copy of the whole,’ a property called self-similarity. A fractal often has the following features: it has a fine structure at arbitrarily small scales; it is too irregular to be easily described in traditional Euclidean geometric language; it is self-similar (at least approximately or stochastically); [… and] it has a simple and recursive definition. […] Natural objects that approximate fractals to a degree include clouds, mountain ranges, lightning bolts, coastlines, and snowflakes.”

The Mandelbrot set is an example of a fractal whose definition contains no stochastic element, i.e., the fractal looks the same every time it’s generated. Recently, though, I came across a simple algorithm that claimed to generate fractal terrains by adding randomness to the equation. So, given my history with fractals, I wanted to play with the algorithm that’s based on progressive refinement through midpoint displacement in one dimension:

    Start with a single horizontal line segment
    Repeat for a sufficiently large number of times
        Repeat over each line segment
            Find the midpoint of the line segment
            Displace the midpoint in y by a random amount
        Reduce the range for random numbers

Here’s my implementation of the algorithm in C#:

    class Program {
        static void Main() {
            var ys = new List<double>(new double[] { 0.0, 0.0 } );
            double displacement = 1.0;
            Random random = new Random();

            for (int i = 0; i < 8; i++)
                ys = Split(ys, displacement *= 0.5, random);
            ComposeCoordinatePairs(ys);
        }

        static List<double> Split(List<double> ys, double displacement, Random random) {
            if (ys.Count < 2)
                throw new ArgumentException(">= 2 coordinates required");
            var r = new List<double>();
            for (int i = 0; i < ys.Count - 1; i++) {
                double dy = (ys[i + 1] - ys[i]) / 2.0;
                double d = random.NextDouble() * displacement;
                r.Add(ys[i]);
                r.Add(ys[i] + dy + d);
            }
            r.Add(ys.Last());
            return r;
        }

        static void ComposeCoordinatePairs(List<double> ys) {
            double dx = 1.0 / (ys.Count - 1);
            for (int i = 0; i < ys.Count; i++)
                Console.WriteLine("{0:0.000} {1:0.000}", i * dx, ys[i]);
        }

Splitting line segments, the C# code doesn't store the x-component of the coordinate pairs because it can be inferred from the number of y-components and the fact that x-components go from 0 to 1, both inclusive. Hence, starting with the line segment (0,0)-(1,0), the code splits it by calling the Split method passing in a list of y-components of the line segment. After the first iteration the single line segment becomes two, then four, then eight, and so on until the line segments have undergone eight iterations of splitting. Generally, after the i'th iteration the number of line segments is 2^i and so after eight iterations we end with 256 line segments.

As prescribed by the algorithm, merely splitting line segments doesn’t give rise to a mountain. The mountain emerges because each time a line segment is split in two, a random displacement is added to the y-component at the center of the line segment. Assigning an upper limit to the maximum displacement within an iteration is what defines the roughness of the generated terrain. For the above implementation, the maximum displacement is defined as half of that of the previous iteration starting with 0.5. Thus, the first couple of iterations roughly define the terrain and additional iterations fill in the details.

In animated form the transformations from two to 256 line segments looks like so:

Among the practical applications of such a random fractal terrain generator are computer games. Not necessarily for generating 2D ridgelines, but for generating entire 3D terrains or even clouds. It works by generalizing 1D midpoint displacement to 2D in the form of the diamond-square algorithm. Then a cloud, for instance, can be derived from the 3D terrain by applying a height map to it; a height map in which different heights get assigned different colors.

The reason for bringing up the linked file feature in conjunction with assembly signing is that to me the feature seems particularly well-suited for linking one and only key file into multiple Visual Studio projects. Yet people seem to be unaware of the feature and so common approaches to signing is to either create a new key file for each assembly — most likely because Visual Studio makes it so easy to create a new key file from the Project Settings dialog — or signing assemblies by creating one key file and then copy/paste it to other projects in the solution.

Below is an example from the GAC showcasing how assemblies related to Pex naturally go together and are therefore signed using the same key (they share 76a274db078248c8, their public key token). The same goes for assemblies relating to other Microsoft products, such as the ones comprising a particular version of the .Net framework (because of the side-by-side execution, different versions of .Net framework assemblies are signed using different keys). So why not apply the same principle to what you deploy to the GAC?

While using the same key (copy or not) makes it possible to differentiate your assemblies from everyone else and for your four-part assembly name in web.config and spread around the code to be uniform — SharePoint developers often need to provide the four-part name for their assemblies to other parts of the solution — copying the key around is still a violation of the DRY principle and as such is indicative of a code smell. More importantly, using the link feature of Visual Studio, there’s no reason to keep the bad smell around.

Assuming you’ve already created a solution (in Visual Studio 2008) and added a project to it, the way to share a key file is to first create one by right clicking a project in Solution Explorer and selecting Properties. Now click the Signing tab and check the Sign the assembly check box. The dropdown lets you create a new key file by selecting New and filling out the information in the Create Name Key window. Alternatively, a key file is created using sn.exe, the strong name utility that comes with the Microsoft Windows SDK. On my machine, sn.exe is located in C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin and is invoked like so:

    > sn -k MySharedKeyPair.snk

How the key file is created isn’t particularly relevant to the rest of this post. Just make sure to place the key file in a common location, e.g., the top-level directory holding the Visual Studio solution file. Now go to Solution Explorer, right click on your project, and select Add existing Item. Then locate the key file within the solution directory and make sure to click the arrow on the Add button and select Add as Link. Note how the file is added to the project and marked by an arrow indicating it’s a link:

Returning to the Signing tab and selecting the dropdown, it now includes the linked key file. Don’t mind the absolute path; once you select MySharedKeyPair.snk, Visual Studio converts the path to a relative one (close and reopen the Signing tab and see for yourself). Also note that if you select Browse, the key file you select is copied to the project folder and added to the project as a regular file.

Another way to associate a key file with an assembly is through the AssemblyKeyFile attribute. The attribute instructs the compiler of where to look for the key file and, although not used by the runtime, the attribute is included in the assembly along with any other attribute for everyone to see.

Using AssemblyKeyFile is probably the simplest approach to sharing a key file between projects, but now compiling the assembly results in a compiler warning as displayed above. The use of the AssemblyKeyFile attribute has been deprecated by Microsoft because it may lead to you inadvertently disclosing sensitive information through the embedded path and because working with a relative path may confuse users and the build system (the relative path is retained in the assembly, though). However, it appears AssemblyKeyFile is still the preferred way for MS to sign their assemblies. Inspecting a couple of .Net 3.5 assemblies with Reflector, here’s what their AssemblyKeyFile looks like:

    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\EcmaPublicKey.snk")]
    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\35MSSharedLib1024.snk")]
    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\FinalPublicKey.snk")]
    …

To come full circle, the question of how Visual Studio, or rather MSBuild, actually signs an assembly using the information provided in the Properties dialog (real file or link) remains to be answered. Since a Visual Studio project file acts as input to MSBuild, we can see what’s going on under the hood of MSBuild by invoking a compile from the command line. On my machine msbuild.exe is located in C:\Windows\Microsoft.NET\Framework\v3.5 and assuming you have a console window open with its current directory set to the directory of your project file, msbuild is invoked like so:

    > msbuild /verbosity:diagnostic > log.txt

Bring up log.txt in your favorite editor and eventually you’ll come across where csc.exe, the C# compiler, is invoked:

As you might have guessed, MSBuild carries over the location of the key file by passing the /keyfile argument to csc.exe.

In order to test code whose branching behavior depends on a value read from a configuration file, the test should provide various inputs and assert that the code under test acts accordingly. In addition, the test should run in milliseconds to encourage frequent runs and it should be self-contained, meaning no setup should be required before running the test. These requirements are best met with the web part running in a controlled environment. Yet, as far as the code under test goes, it should behave as if part of an Asp.Net page.

For the sake of brevity, here’s my web part with the code to initialize the data context:

    public class MyWebPart : WebPart {
        MyDataContext _db;    /* LINQ database context */
        string _error;        /* Early stage error message */    	

        public MyWebPart() {
            Initialize();
        }

        private void Initialize() {
            string c = GetMyConnectionString();
            if (c == null)
                 _error = "Unable to do something";
            else
                _db = new MyDataContext(c);
        }

        private string GetMyConnectionString() {
            System.Configuration.ConnectionStringSettings c =
                System.Configuration.ConfigurationManager.
                    ConnectionStrings["MyConnection"];
            return c == null ? null : c.ConnectionString;
        }

        protected override void Render(HtmlTextWriter w) {
            // display error message if set
        }
    }

Notice how the constructor calls out to an Initialize method. Otherwise code within the constructor would fire before I’d have a chance to setup a controlled environment around the web part and I wouldn’t be able to test it. Turning the attention to Initialize itself, it’s made up of only a simple if statement. The idea is that if I can’t mock something as simple as reading a value from a configuration file, most likely I can’t mock something more involved either. On the other hand, for a developer already familiar with TypeMock and test first development, the code above may be born of the need to satisfy a failing test.

I want to test Initialize by controlling how GetMyConnectionString gets to return the connection string. While running in an Asp.Net context, GetMyConnectionString would attempt to read the string from web.config. Within my controlled environment, however, I want to inject into the web part an alternate implementation that returns a value of my choosing. That way I can control the path through Initialize and inspecting the state of the _error or _db fields, I’m able to determine if the code behaved as intended.

Using the method chaining DSL of TypeMock, the test takes on this form (of course, having more tests, I’d move shared code into helper methods decreasing the test to code ratio):

    [TestClass, ClearMocks]
    public class MyWebPartTest {
        [TestMethod, Isolated, VerifyMocks]
        public void Should_set_error_message_when_no_
                    connection_string_is_configured() {
            // arrange
            MyWebPart fake = Isolate.Fake.Instance<MyWebPart>();
            Isolate.NonPublic.WhenCalled(fake,
                "Initialize").CallOriginal();
            Isolate.NonPublic.WhenCalled(fake,
                "GetMyConnectionString").WillReturn(null);
            PrivateObject handle = new PrivateObject(fake);

            // act
            handle.Invoke("Initialize", null); 

            // assert
            Isolate.Verify.NonPublic.WasCalled(fake,
                "GetMyConnectionString");
            string error = (string)handle.GetField("_error");
            MyDataContext ctx = (MyDataContext)handle.GetField("_db");
            Assert.AreEqual("Unable to do something", error);
            Assert.AreEqual(null, ctx);
        }
    }

There’re three parts to tests using TypeMock: (1) the arrange part sets up expectations of what’s going to happen when a method, a property, or some other part of the code under test is invoked; (2) the act part carries out the actual interaction with the object under test, turning to PrivateObject to invoke the Initialize method; and finally (3) the assert part verifies that what was supposed to happen did happen. Worth noting is that this test asserts both state and behavior, i.e., as part of executing the code under test, _error or _db should’ve been set to an appropriate state and GetMyConnectionString should’ve been called once and only once.

This concludes my example, which I believe illustrates the basics of testing and mocking using MS Test and TypeMock. Comparing and contrasting TypeMock to other popular mocking frameworks, TypeMock has the distinct feature that it can mock almost anything, with or without using interfaces. Some argue that not enforcing the use of interfaces is also the disadvantage of TypeMock, because for code to be truly testable, it should be modularized using the Inversion of Control principle. However, with legacy code, such as the .Net framework or the SharePoint API, to me TypeMock appears to bridge the two worlds nicely, leaving the developer in charge.

To learn more about the basic concepts of mocking, Inversion of Control, and Dependency Injection (containers), I’d recommend listening to a couple of Alt.Net podcasts: Dependency Injection and Inversion of Control and More Dependency Injection and Inversion of Control.

(Other posts in this series include: Kernel space traffic shaping with Linux and User space traffic shaping with Ruby that touch exclusively on Linux issues.)

This post covers an Asp.Net MVC application (for an overview of Asp.Net MVC, the Herding Code guys recently interviewed Phil Haack) I wrote that automates the collection of information about computers and their owners on a local area network. The idea is to have the Internet gateway match the identity of each computer that wants to send traffic through it against a white list. Should the request originate from an unknown computer, the gateway redirects that computer to a web application for Internet access activation. Each user then has to create an account and provide verifiable contact information and/or associate their computer with an existing account.

With the contact information of every connected user, maintaining a network with 331 connected apartments, 371 users, and 513 privately owned computers becomes more manageable. Now you can easily email all users about general issues or individual users about specific issues, such as wireless routers exposing rogue DHCP servers or virus slowing down the network.

Without being able to contact a user, the best you can hope for is to use the MAC address to locate and disable the port on the switch to which the user is connected. But terminating network access without warning or explanation, the user has no way of knowing what hit him. Now when the user contacts you, figuring out if he’s experiencing network issues because of a closed port or something entirely different quickly becomes a challenge — and something that doesn’t scale well.

That’s where the Asp.Net MVC application comes in. To get a feel for how the software accomplishes its task, you should watch the minute and a half demo. It portrays the browsing experience of a user connecting his computer to the network for the first time and attempting to browse the web. Behind the scenes the Linux gateway uses its Netfilter capabilities to efficiently match the MAC address of each request against the white list. With no match for the computer just connected, upon browsing the web, Netfilter redirect the user’s browser to the web application. Then, every so often, using a Ruby script, the gateway queries the web server for updates to the white list and carries over the changes to Netfilter.

Database schema

The web application is based on the idea that a user is responsible for zero or more computers. Focusing on the web application, rather than the Linux part, the rest of this post highlights a few features that use and manipulate the data structures stored in a MS SQL Server Express database:

Reverse lookup of MAC

Establishing an ownership between a user and a computer is done via the MAC address of the computer. Now, it wouldn’t be particularly user friendly if the user had to go about locating the MAC address and manually typing it into a web form. Instead, the web application looks up the user’s MAC address within the ARP cache of the web server.

Because the user’s computer and the web server are both on the same local network and because they’re already exchanging data (the user is visiting a web page hosted on the server), the parties are effectively communicating through the ARP protocol of the data link layer. The data entry form is therefore able to present the user with a control that makes associating computers to the account easy. By reverse lookup, the computer currently browsing the web application is indicated by “Now logged in from here”.

Email verification

Most users don’t mind handing over basic information such as their email address. And as users are blocked from accessing the Internet, at the time of entry we only validate the email address for syntactic correctness.

A few users, however, repeatedly entered fake email addresses to gain Internet access. Consequently, we send out an email to the address, leaving open a 24 hour window for the user to follow a link. Otherwise, the software sends out a reminder email, marks the account as inactive, and redirects the user to the web application.

Validating the syntax of email addresses is done using regular expressions. As with most complex regular expressions, they’re hard to read and verify the correctness of. They do, however, stand the test against a database of hundreds of email addresses. To come full circle, though, what is needed is a way to match the domain part of the email address against the DNS MX record of the domain. Unfortunately, MX lookup isn’t build into the .Net framework.

Unhalted exception emailing error handler

Error handling deserves a post on its own. Suffice it to say that to stay on top of any application, unhalted exceptions should be logged, or in this case, emailed to a designated address. Inevitably, users of your software will use it in unanticipated ways and so a global exception handler provides invaluable insight to learn from.

(Summary part of example email. Click here for complete output.)

Download and maybe try out

The web application was developed around February 2008; about the time the Asp.Net MVC framework went into Preview 2 and had started gaining momentum. Unfortunately, developing against this early a release, now the web application doesn’t run on a computer with .Net 3.5 SP1 installed.

There’s no problem compiling the application because the Preview 2 bits are in the bin folder of the application. But running the application generates an exception stating:

   Could not load type 'System.Web.HttpContextWrapper2' from
   assembly 'System.Web.Abstractions' [located in the GAC]

That’s because the version of System.Web.Abstractions.dll that ships with .Net 3.5 SP1 no longer holds the HttpContextWrapper2 class. On a machine without .Net 3.5 SP1, however, Cassini runs the application just fine. Before running it, though, remember to create a database from the schema in Database.sql and point to that database from web.config.

Lastly, I should stress that the software is a prototype and served as a way for me to wrap my head around ASP.Net MVC and LINQ. So, the code may not win a beauty contest.

Conclusion

The activation software went online late March 2008. During the first couple of weeks I had to put in a couple of bug fixes based on what I learned from the unhalted exception emails. Since then, however, the software has served the purpose it was charged with.

The only issue that I haven’t been able to resolve is why, on rare occasions, redirecting a computer confuses its browser: if you visit a page and is redirected to the web application, for a short time after, the page keeps resolving to the web application. Most likely it’s a residual effect of the packet rewriting taking place on the gateway.

Another thing I wish I’d implemented was tracing to better understand how a couple of users got to experience a few exceptions that I don’t know how to interpret from the unhalted exception emails alone.

In any real-world application, unhalted exceptions are an unavoidable fact of life. Such exceptions should occur relatively infrequently, but when they do some kind of supporting infrastructure better be in place to capture the when and why of the exceptions. As a developer, you should have the ability to perform a postmortem analysis on what went wrong and learn from it. Paramount to such analysis is capturing and logging sufficient information at the point of failure.

What this post covers is the thoughts and development of an Asp.Net HttpModule that captures and emails such unhalted exception information to a designated email address.

To be clear, the goal of the EmailingExceptionModule isn’t to prevent unhalted exception from occurring per se. Within code, a developer may not know how to handle an exception, and so the exception should rightfully propagate the call stack. In case no caller steps in and handles the exception, the best choice is most likely to terminate the application. It’s almost always better to be upfront with the user than conceal the possibly inconsistent state of the application.

So when does the EmailingExceptionModule come in handy then? One scenario is that of C# unchecked exceptions, where the compiler doesn’t force the caller to catch all types of exceptions thrown by the callee. Unchecked exceptions may materialize as type cast or null reference exceptions when accessing variables. What unchecked exceptions boil down to is that, given the cyclomatic complexity of some methods, it’s impractical to manually work through every conceivable path of execution ahead of time. Another scenario is problems with the runtime environment, such as out of disk space or database server down. Nonetheless, some map of the path to failure is helpful in preventing others going down that same path.

(Summary part of example email. Click here for complete output.)

With web applications, we can take advantage of the application running in a centralized environment. In the spirit of Microsoft’s Doctor Watson technology, we can register our own web application error handler and hook it into the Http request pipeline, and have the application execute our code on unhalted exceptions. In practice, such an error handler is implemented either through the Application_Error method of an application’s Global.asax or by loading an HttpModule into the application.

Hinted by the title, I went for the HttpModule. The reason may best be understood by taking a peak behind the .Net curtains: Global.asax is a filename hardwired into the framework so that whenever an application is first hit, the framework asks HttpApplicationFactory, an internal factory class, for an HttpApplication representing the application (there’s a unique one for each virtual application). As part of manufacturing the HttpApplication, the factory locates the application’s Global.asax, compiles it into a dynamically generated DLL, loads the DLL, and reflects over the Global.asax class looking for methods adhering to the convention of modulename_eventname. Methods found, such as Application_Error, are then stored in a list of MethodInfos and passed along to HttpApplication. Then, during HttpApplication initialization, events are bound to the methods within Global.asax.

That’s how Application_Error of Global.asax gets called even though the method isn’t overriding a base class implementation as is typical for the template pattern. That’s also why code within global.asax isn’t binary reusable across applications. An HttpModule, on the other hand, can be loaded into any number of applications.

In practice, implementing an HttpModule it a matter of creating a class that implements the IHttpModule interface:

   public interface IHttpModule {
      void Init(HttpApplication context);
      void Dispose();
   }

The crux of error handling with Asp.Net is the Error event of HttpApplication. The easiest way to hook up the event to a method is within the Init method of the class. In addition, we assign the application object to a field so that later we can access the most recently thrown exception through it:

   public class EmailingExceptionModule : IHttpModule {
      private HttpApplication _application;

      public void Init(HttpApplication a) {
         _application = a;
         _application.Error += OnUnhaltedException;
      }

      public void Dispose() {}

      private void OnUnhaltedException(object sender, EventArgs e) {
         Exception ex = _application.Server.GetLastError().InnerException;
         // Implementation left out for brevity. Please download source
      }
   }

The OnUnhaltedException method is where to add code that collects information about the exception and the environment, and that composes and ships the email. Depending on the environment, you may find the need to include additional information in the email. Given the source code, adding to the email is a matter of adding key/value pairs in the form of labels and values to a dictionary. Each dictionary then gets rendered as a table of the key and value columns.

To have an Asp.Net application load the EmailingExceptionModule and to configure its email related settings, add the following nodes to the application’s web.config:

   <configuration>
      <appSettings>
         <add key="EmailingExceptionModule_SenderAddress" value="sender@domain.com"/>
         <add key="EmailingExceptionModule_ReceiverAddress" value="receiver@domain.com"/>
         <add key="EmailingExceptionModule_SmtpServer" value="mail.domain.com"/>
         <add key="EmailingExceptionModule_SubjectPrefix" value="MyApp: "/>
      </appSettings>
      <system.web>
         <httpModules>
            <add name="EmailingExceptionModule" type="Holm.AspNet.EmailingExceptionModule,
                       EmailingExceptionModule, Version=1.0.0.0, Culture=neutral,
                       PublicKeyToken=13806f613f05e959"/>
         </httpModules>
      </system.web>
   </configuration>

So what’s the user experience of unhalted exceptions? Outside the confines of the EmailingExceptionModule, there’s no indication that the application took note of what happened. Depending on the user’s profile, it might be helpful to prompt for additional information at the crash point. Or perhaps turn to the Windows Live Messenger IM Control & Presence API and have the user engage in an MSN conversation with a developer.

What’s the advantage of emailing developers unhalted exceptions over storing the information in the file system or a database? I believe in keeping things simple and practical, and handing off information to the file system or a database isn’t. Most likely the information will end up gathering virtual dust on some server. The email approach, on the other hand, is a proactive, visible, and efficient means to the end of rapidly course-correcting for wrong assumptions.

Update, Sep 15: While you don’t need a PDB file to debug code through Visual Studio (that’s what <compilation debug=”true”> in web.config is for), the PDB file is required for the stack trace to contain line numbers. Since the PDB file typically has to reside next to the corresponding DLL for the .Net runtime to pick it up, the PDB file may need to be deployed to the GAC along with the DLL. Refer to the first three paragraphs of the “Debugging assemblies that live in the GAC” section of this post for how to GAC deploy PDB files.

For a centrally deployed Asp.Net application, it may be acceptable to ship and deploy the PDB file with the application. Keep in mind, though, that because the PDB file is a mapping of locations in the IL to the source file, having access to the PDB file makes it easier for someone to reverse engineer the code. It’s also possible to get at line numbers without the PDB file being deployed, but the approach is somewhat involved. Based on the mapping nature of the PDB file, the idea is to write out IL offsets as part of the exception and to post-process the offsets using a PDB file at a separate location.

Yet another alternative to shipping the PDB file is to setup a symbol server. With this nifty piece of code, the StackTrace class loads the PDB file of the symbol server, adding line numbers to the stack trace.