Term store overview

SharePoint organizes terms in either a taxonomy or a folksonomy. In a taxonomy, terms are organized hierarchically whereas in a folksonomy the structure is flat. A taxonomy or folksonomy is either global, making it available to all site collections within a web application that’s associated with a Managed Metadata Service, or local, making it available in one site collection only.

Multiple Managed Metadata Services may be associated with a web application or site collection, each service in turn responsible for a single term store. Within one term store, multiple groups may be created, each potentially storing multiple term sets. Finally, a term set may have terms nested up to seven levels deep:

Managed Metadata Service Application
  TermStore
    Group1
      TermSet1
        Term1
          Term1.2
            Term1.2.3
              Term1.2.3.4
                Term1.2.3.4.5
                  Term1.2.3.4.5.6
                    Term1.2.3.4.5.6.7

You manage the term store through the browser and its Term Store Management Tool (the _layouts/termstoremanager.aspx application page) or through the SharePoint API. The management tool supports importing term sets only from a special CSV file format  (_layouts/1033/ImportTermSet.csv holds a sample) with these columns:

Term Set Name
Term Set Description
LCID
Available for Tagging
Term Description
Level 1 Term
Level 2 Term
...
Level 7 Term

The CSV file is incomplete in terms of what the term store offers. It doesn’t offer columns for specifying translations and synonyms for terms. But users can use Wictor Wilén Excel sheet to maintain the basic information. It has the standard columns above and includes macros to simplify saving in the correct format and creating additional term sets.

The question remains: do you maintain compatibility with the existing CSV format so you can import term sets using the browser or a forms-based tool like the CSV Bulk Taxonomy Importer/Exporter? Or do you define your own, possibly XML based format since term sets are hierarchical by nature, with additional columns for translations and synonyms? A non-CSV based format may also be easier to process. If you want to pursue the latter approach, Ben Robb has created a PowerShell script to import his XML format.

Importing term sets using PowerShell

I decided to stick with the CSV file format and create a PowerShell script that would allow me to import a term set using the command-line:

function ImportTermSet([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName, [PSCustomObject]$termSet) {
  function ImportTerm([Microsoft.SharePoint.Taxonomy.Group]$group,
                      [Microsoft.SharePoint.Taxonomy.TermSet]$set,
                      [Microsoft.SharePoint.Taxonomy.Term]$parent,
                      [string[]]$path) {
    if ($path.Length -eq 0) {
      return
    } elseif ($group -eq $null) {
      $group = $store.Groups | where { $_.Name -eq $path[0] }
      if ($group -eq $null) {
        $group = $store.CreateGroup($path[0])
      }
    } elseif ($set -eq $null) {
      $set = $group.TermSets | where { $_.Name -eq $path[0] }
      if ($set -eq $null) {
        $set = $group.CreateTermSet($path[0])
      }
    } else {
      $node = if ($parent -eq $null) { $set } else { $parent }
      $parent = $node.Terms | where { $_.Name -eq $path[0] }
      if ($parent -eq $null) {
        $parent = $node.CreateTerm($path[0], 1033)
      }
    }

    ImportTerm $group $set $parent $path[1..($path.Length)]
  }

  function RemoveTermGroup([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName) {
    $group = $store.Groups | where { $_.Name -eq $groupName }
    if ($group -ne $null) {
      $group.TermSets | foreach { $_.Delete() }
      $group.Delete()
      $store.CommitAll()
    }
  }

  RemoveTermGroup $store $groupName
  $termSetName = $termSet[0]."Term Set Name"
  $termSet | where { $_."Level 1 Term" -ne "" } | foreach {
    $path = @($groupName, $termSetName) + @(for ($i = 1; $i -le 7; $i++) {
    $term = $_."Level $i Term"
      if ($term -eq "") {
        break
      } else {
        $term
      }
    })

    ImportTerm -path $path
  }
}

The way to use the ImportTermSet function is best illustrated with an example:

$session = Get-SPTaxonomySession -Site "http://localhost"
$store = $session.TermStores["Managed Metadata Service"]
$termSet = Import-Csv "C:\Users\Ronnie\Desktop\ImportTermSet.csv"
ImportTermSet $store "MyGroup" $termSet
$store.CommitAll()

In its current form the script ignores the Term Set Description, LCID, Available for Tagging, and Term Description columns from the CSV file. It would be possible to take these columns into account by passing them to ImportTerm as well. Instead of just removing the head of the list, you’d remove the number of items consumed by the current step before recursing.

For the generic runner to work, each command has to fulfill the contract.

public enum ExitCode {
    Success = 0,
    Failure
};

public interface ICommand {
    string Usage { get; }
    string Description { get; }
    ExitCode Execute(string[] args);
}

I want the runner to adhere to the open/closed principle. For its behavior to be modified without altering its core delegation logic. This requires the use of reflection to retrieve and instantiate a command based on command-line arguments.

class Program {
    static IEnumerable<ICommand> GetCommands() {
        var iCommand = typeof (ICommand);
        return System.Reflection.Assembly.GetExecutingAssembly().GetTypes().ToList()
            .Where(t => iCommand.IsAssignableFrom(t) && t != iCommand)
            .Select(t => Activator.CreateInstance(t) as ICommand);
    }

    static void DisplayHelp() {
        Console.WriteLine("Console [Command] [Arg1] [Arg2] [ArgN]\n\n");
        GetCommands().ToList().ForEach(command =>
            Console.WriteLine(command.Usage + "\n" + command.Description + "\n\n"));
    }

    static int Main(string[] args) {
        if (args.Length == 0) {
            DisplayHelp();
            return (int)ExitCode.Failure;
        }

        var commandName = args[0];
        var command = GetCommands().Where(t => t.GetType().Name == commandName).SingleOrDefault();
        if (command == null)
            throw new ArgumentException(string.Format("Command '{0}' not found", commandName));

        var executeArguments = new List<string>(args);
        executeArguments.RemoveAt(0);

        var exitCode = command.Execute(executeArguments.ToArray());
        return (int)exitCode;
    }
}

A trivial example of a command that adds two numbers would be the following:

// $> GenericRunner.exe Calculator 2 3 => 2 + 3 = 5
public class Calculator : ICommand {
    public string Usage {
        get { return "Calculator [Op1] [Op2]"; }
    }

    public string Description {
        get { return "World's simplest calculator"; }
    }

    public ExitCode Execute(string[] args) {
        try {
            Console.WriteLine(
                string.Format(
                    "{0} + {1} = {2}",
                    args[0], args[1], int.Parse(args[0]) + int.Parse(args[1])));
            return ExitCode.Success;
        } catch (Exception e) {
            Console.WriteLine(e.ToString());
            return ExitCode.Failure;
        }
    }
}

Now multiple smaller assemblies can be grouped into one, with a description of all commands automatically being assembled, and without commands interfering (too much) with each other.

namespace Dk.Bugfree

open System
open System.Globalization
open Microsoft.SharePoint

type public ListAttachmentVersioningEventReceiver() =
    inherit SPItemEventReceiver()

    member private r.CustomVersion = "CustomVersion"
    member private r.ShadowLibrary = "ShadowLibrary"

    // override ItemAdded : properties:SPItemEventProperties -> unit
    override r.ItemAdded properties =
        base.ItemAdded properties
        r.SetCustomVersionLabel properties.ListItem
        r.CreateSnapshot properties

    // override ItemUpdated : properties:SPItemEventProperties -> unit
    override r.ItemUpdated properties =
        base.ItemUpdated properties
        let item = properties.ListItem

        if r.RollbackHappened item then
            r.RestoreSnapshot properties
            r.SetCustomVersionLabel item
            r.CreateSnapshot properties
        else
            r.CreateSnapshot properties
            r.SetCustomVersionLabel item

    // member private CreateSnapshot : properties:SPItemEventProperties -> unit
    member private r.CreateSnapshot properties =
        use site = properties.OpenWeb()
        let item = properties.ListItem
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let path = String.Format("Versions/{0}/{1}", item.ID, r.GetOfficialVersionLabel(item))
        let shadowFolder = r.CreateFolderPath shadowLibrary path

        item.Attachments |> Seq.cast |> Seq.iter (fun fileName ->
            let existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName)
            let newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream())
            newFile.Item.Update())

    // member private RollbackHappened : item:SPListItem -> bool
    member private r.RollbackHappened item =
        let culture = CultureInfo.InvariantCulture
        let currentVersion = Single.Parse(r.GetOfficialVersionLabel(item), culture)
        let lastVersion = Single.Parse(r.GetCustomVersionLabel(item), culture)
        currentVersion > lastVersion + 1.0f

    // member private RestoreSnapshot : properties:SPItemEventProperties -> unit
    member private r.RestoreSnapshot properties =
        let item = properties.ListItem
        let restoreVersion = r.GetCustomVersionLabel item
        r.EventFiringEnabled <- false    

        item.Attachments |> Seq.cast |> Seq.map (fun fileName -> unbox<string> fileName) |> Seq.toList
                         |> Seq.iter (fun fileName -> item.Attachments.Delete(fileName))

        use site = properties.OpenWeb()
        let path = String.Format("Versions/{0}/{1}", item.ID, restoreVersion)
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let source = r.CreateFolderPath shadowLibrary path

        source.Files |> Seq.cast |> Seq.iter (fun file ->
            let unboxedFile = unbox<SPFile> file
            item.Attachments.Add(unboxedFile.Name, unboxedFile.OpenBinary()))

        item.SystemUpdate false
        r.EventFiringEnabled <- true

    // member private CreateFolderPath : list:SPDocumentLibrary -> path:string -> SPFolder
    member private r.CreateFolderPath list path : SPFolder =
        r.CreateFolderPathRecursive list.RootFolder (path.Split [|'/'|] |> Array.toList)

    // member private CreateFolderPathRecursive : folder:SPFolder -> pathComponents:string list -> SPFolder
    member private r.CreateFolderPathRecursive folder pathComponents =
        match pathComponents with
        | [] -> folder
        | head :: tail ->
            try
                let existingFolder = folder.SubFolders.[head]
                r.CreateFolderPathRecursive existingFolder tail
            with
                :? ArgumentException ->
                    let newFolder = folder.SubFolders.Add head
                    r.CreateFolderPathRecursive newFolder tail

    // member private SetCustomVersionLabel : item:SPListItem -> unit
    member private r.SetCustomVersionLabel item =
        r.EventFiringEnabled <- false
        item.[r.CustomVersion] <- r.GetOfficialVersionLabel item
        item.SystemUpdate false
        r.EventFiringEnabled <- true  

    // member private GetCustomVersionLabel : item:SPListItem -> string
    member private r.GetCustomVersionLabel item =
        item.[r.CustomVersion] :?> string

    // member private GetOfficialVersionLabel : item:SPListItem -> string
    member private r.GetOfficialVersionLabel item =
        item.Versions.[0].VersionLabel

A couple of things to note about the F# implementation: first, it hardly specifies any types. They’re inferred by the compiler. Where type names do appear, it’s mainly because they’re required to unbox elements of an IEnumerable collection. Secondly, F# has flexible self identifiers. Methods must explicitly specify the name of the this reference in C# and use it when accessing members. Thirdly, arguments to general .NET methods are passed as a tuple value, i.e., as comma-delimited arguments surrounded by parenthesis.

This would involve adding or removing event receivers from a list as the user enables or disables versioning. The following extension method is one way to accomplish the addition-part in a type-safe manner:

// definition
public static class SPListExtensions {
    public static void RegisterEventReceiver<TReceiver>(this SPList list,
            SPEventReceiverType receiverType,
            int sequenceNumber) where TReceiver : SPItemEventReceiver {
        var assemblyName = typeof(TReceiver).Assembly.FullName;
        var className = typeof(TReceiver).FullName;

        (from SPEventReceiverDefinition definition in list.EventReceivers
         where definition.Assembly == assemblyName &&
               definition.Class == className &&
               definition.Type == receiverType
         select list.EventReceivers[definition.Id])
        .ToList()
        .ForEach(receiverToDelete => receiverToDelete.Delete());

        var receiver = list.EventReceivers.Add();
        receiver.Type = receiverType;
        receiver.Assembly = assemblyName;
        receiver.Class = className;
        receiver.SequenceNumber = sequenceNumber;
        receiver.Update();
        list.Update();
    }
}

// use
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemAdded, 10000);
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemUpdated, 10001);

Under rare circumstances the (assembly, class, type) tuple may not be unique, i.e., the same receiver may be registered multiple times, albeit with different sequence numbers. In practice I never found any use for this functionality, though, which is why I didn’t include the sequence number in the where clause above, causing all registrations matching the tuple to be removed.

Both SharePoint 2007 and 2010 support versioning for list items but not their attachments. No matter which version of a list item I look at, its attachments will always be the most recent. The attachment support seems to have been bolded on as an afterthought, resulting in behavior that’s counter-intuitive for developers as well as end-users. With SharePoint 2007 (and 2010), Microsoft suggests using a document library for proper attachment versioning. But I can’t substitute one with the other, since a list item may hold any number of attachments and an item in a document library may hold just one.

Existing solution

I counted on someone else having experienced a similar pain and come up with a workable cure. But except for Tim Ebenezer the search come up empty. Tim on the other hand has done a great job of seamlessly integrating his attachment versioning feature into SharePoint. When I activate the feature on a site, it adds a versioning menu item to the list settings page for every list on the site. Unfortunately the core versioning logic, storing attachments in a shadow library using an event receiver, isn’t particularly robust. Among other use cases, it doesn’t properly deal with a user first deleting an attachment and then, some versions later, adding an attachment with the same name.

I therefore set out to implement my own solution based on Tim’s ideas, hooking into the synchronous ItemAdding, ItemUpdating, ItemAttachmentAdding, and ItemAttachmentDeleting events and maintaining a shadow library of versions. This approach, however, quickly turned into a painful one. When the synchronous events run, nothing has yet been written to the database – at this stage a new item doesn’t even have its Id set, and merely determining the number of attachments added and how far I’ve come with the processing is tricky.

The next challenge I encountered was that event handlers cannot easily share state across multiple calls because SharePoint creates a new instance of the receiver class for every event handled. Processing multiple attachments require a counter into the array of attachments to keep track of which ones I’d copied to the shadow list. I’d have to resort to some outside-object storage, keeping in mind that the receiver might execute concurrently. But which storage should I use? Session state may have been disabled, and polluting one of the property bags stored in the content database is messy and also not thread-safe.

Overall, with the synchronous approach too much work has to go into tracking the state of the versioning process.

New solution

A synchronous solution is very hard to get right because it’s forced to work at the level of individual attachments. SharePoint doesn’t have a synchronous event that fires after all attachments have been processed. After all, why provide such an event when everything has already happened? Thinking instead in terms of the asynchronous events of ItemUpdated and ItemAdded, I have exactly what’s needed to snapshot all attachments in one batch, making versioning a lot simpler. When these events fire the item and its attachments have already been written to the database and I can focus on how to generate the snapshots — copying attachments back and forth between lists — and not worry about what the user actual did to the attachments from one version to the next.

// Prerequisites:
// 1. Create a Document Library named ShadowLibrary on the same site as the list to version
// 2. Add a row named CustomVersion of type string to the list to version
public class ListAttachmentVersioningEventReceiver : SPItemEventReceiver {
    private const string CustomVersion = "CustomVersion";
    private const string ShadowLibrary = "ShadowLibrary";

    public override void ItemAdded(SPItemEventProperties properties) {
        base.ItemUpdated(properties);
        SetCustomVersionLabel(properties.ListItem);
        CreateSnapshot(properties);
    }

    public override void ItemUpdated(SPItemEventProperties properties) {
        base.ItemUpdated(properties);

        var item = properties.ListItem;
        if (RollbackHappened(item)) {
            RestoreSnapshot(properties);
            SetCustomVersionLabel(item);
            CreateSnapshot(properties);
        }
        else {
            CreateSnapshot(properties);
            SetCustomVersionLabel(item);
        }
    }

    private void CreateSnapshot(SPItemEventProperties properties) {
        using (var site = properties.OpenWeb()) {
            var item = properties.ListItem;
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var path = string.Format("Versions/{0}/{1}", item.ID, GetOfficialVersionLabel(item));
            var shadowFolder = CreateFolderPath(shadowLibrary, path);

            foreach (string fileName in item.Attachments) {
                SPFile existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName);
                SPFile newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream());
                newFile.Item.Update();
            }
        }
    }

    private bool RollbackHappened(SPListItem item) {
        var culture = CultureInfo.InvariantCulture;
        var currentVersion = float.Parse(GetOfficialVersionLabel(item), culture);
        var lastVersion = float.Parse(GetCustomVersionLabel(item), culture);
        return currentVersion > lastVersion + 1;
    }

    private void RestoreSnapshot(SPItemEventProperties properties) {
        var item = properties.ListItem;
        var restoreVersion = GetCustomVersionLabel(item);
        EventFiringEnabled = false;

        item.Attachments.Cast<string>().ToList().ForEach(attachment => item.Attachments.Delete(attachment));
        using (var site = properties.OpenWeb()) {
            var path = string.Format("Versions/{0}/{1}", item.ID, restoreVersion);
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var source = CreateFolderPath(shadowLibrary, path);

            foreach (SPFile file in source.Files)
                item.Attachments.Add(file.Name, file.OpenBinary());
        }

        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    // can only get folder creation to work with Document Libraries
    private SPFolder CreateFolderPath(SPDocumentLibrary list, string path) {
        return CreateFolderPathRecursive(list.RootFolder, path.Split('/').ToList());
    }

    private SPFolder CreateFolderPathRecursive(SPFolder folder, IList<string> pathComponents) {
        if (pathComponents.Count == 0)
            return folder;

        SPFolder newFolder;
        try {
            newFolder = folder.SubFolders[pathComponents.First()];
        }
        catch (ArgumentException) {
            newFolder = folder.SubFolders.Add(pathComponents.First());
        }

        pathComponents.RemoveAt(0);
        return CreateFolderPathRecursive(newFolder, pathComponents);
    }

    private void SetCustomVersionLabel(SPListItem item) {
        EventFiringEnabled = false;
        item[CustomVersion] = GetOfficialVersionLabel(item);
        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    private string GetCustomVersionLabel(SPItem item) { return item[CustomVersion] as string; }
    private string GetOfficialVersionLabel(SPListItem item) { return item.Versions[0].VersionLabel; }
}

When a list item is saved, I take a snapshot of the attachments, storing them in a folder structure like {Id}/{VersionNumber}/{Attachments} in the shadow document library. When a list item is restored to a previous version, existing attachments are first deleted before the ones from the snapshot are added back in, creating a new version of the list item.

Restoring previous versions also has a counter-intuitive meaning in SharePoint. Suppose in one version of a list item, I store a key in the item’s property bag, then I’d expect the property bag values to be specific to this version. But behind the scenes restore seems to work by cloning the current version and then copying only the values of the fields from the restore version to the new one. In other words, I can’t use the item’s property bag to store version specific information, such as a version tag to detect when a restore has occurred. I also can’t use the Modified field because SharePoint sets it to the time of the restore. To carry over version information I have to create and maintain a field of my own. Hence the CustomVersion field on the list to version.

Remember that because the ItemUpdated and ItemAdded execute asynchronously, all the snapshotting logic executes on a background thread, after control has returned to the user. Should an error occur at this point, the user will never see it and the snapshot may be left in an incomplete state. On the other hand, this approach scales well and doesn’t have to be fast because no user is awaiting the result.

Lastly, there’s one place in SharePoint where the versioning abstraction leaks through. It’s in the list item version dialog which displays older versions and enables restore to any previous version. The dialog will always show the most recent attachments.

Improvements

I could use the ETag property of an SPFile object to implement a more efficient differential snapshotting algorithm that would conserve storage space. Compressing attachments before storing them in the shadow library might also be an option, although then I’d have to promote the ETag value to a shadow library field before compressing.

SPListCollection extensions

In SharePoint 2010 the TryGetList method has been added to the SPListCollection class. The method returns either an SPList instance matching the display name or null. Oftentimes, however, you want to do a lookup based on the internal name. Here’s an extension method that adheres to the semantics of TryGetList, but using the internal name. It relies on the fact that the RootFolder property of a list is actually its internal name:

// definition
public static class SPListCollectionExtensions {
    public static SPList TryGetListByInternalName(this SPListCollection lists, string internalName) {
        return (from SPList l in lists
            where l.RootFolder.Name == internalName
            select l).SingleOrDefault();
    }
}

// use
if (site.Lists.TryGetListByInternalName(internalListName) == null)
   // list not found

SPFieldCollection extensions

Using the CreateNewField method of the SPFieldCollection you can add new fields to a list. The particular annoying aspect of this method, however, is that when you want to continue working with its result, oftentimes you have to cast it to one of the SPField subclasses. But since the SPFieldType, provided as one of the arguments to CreateNewField, closely relates to the actual SPField return type, an extension method is able to do the casting. This’ll expose mismatches at compile time instead of at runtime.

All it takes is for us to map out the relation between SPField and SPFieldType:

// definition
public static class SPFieldCollectionExtensions {
    public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
            string internalName, string displayName) where TSPField : SPField {
        var spFieldToFieldType = new Dictionary<Type, SPFieldType> {
            { typeof(SPFieldDateTime), SPFieldType.DateTime },
            { typeof(SPFieldNumber), SPFieldType.Number },
            { typeof(SPFieldUser), SPFieldType.User },
            { typeof(SPFieldBoolean), SPFieldType.Boolean },
            { typeof(SPFieldMultiLineText), SPFieldType.Note },
            { typeof(SPFieldText), SPFieldType.Text }
        };

        var fieldType = spFieldToFieldType[typeof(TSPField)];
        var list = fields.List;
        var field = list.Fields[list.Fields.Add(internalName, fieldType, false)];
        field.Title = displayName;
        field.Update();
        return field as TSPField;
    }
}

// use
l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName");

Taking the CreateField extension method one step further, oftentimes you want to set properties besides internal name and display name. For that purpose I’ve defined a CreateField method that accepts an Action<TField>. This allows you to reuse common property settings across fields for brevity and consistency while at the same time maintaining strong typing.

// definition
public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
        string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPField {
    var newField = CreateField<TSPField>(fields, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
public static Action<SPFieldMultiLineText> RichTextProperties = f => {
    f.RichText = true;
    f.RichTextMode = SPRichTextMode.FullHtml;
};

l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName", f => f.Required = true);
l.Fields.CreateField(internalName, "displayName", RichTextProperties);

With the Comment field, you can leave out the type argument because the compiler infers it based on the type of the Action delegate.

Similar to CreateField, I’ve defined two additional extension methods for creating lookup fields:

// definition
public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName,
        string displayName) where TSPField : SPFieldLookup {
    var currentList = fields.List;
    var lookupList = currentList.ParentWeb.Lists.TryGetListByInternalName(lookupListName);
    var newField = currentList.Fields[currentList.Fields.AddLookup(internalName, lookupList.ID, false)];
    newField.Title = displayName;
    newField.Update();
    return newField as TSPField;
}

public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPFieldLookup {
    var newField = CreateLookup<TSPField>(fields, lookupListName, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
l.Fields.CreateLookup<SPFieldLookup>(lookupListName, internalName, displayName, f => f.AllowMultipleValues = true);

These extension methods makes using the SharePoint API more type-safe and concise, and defining lists using these methods and the template approach saves me from writing a lot of repetitive code.

New-SPWeb -url $base_url -addtoquicklaunch -template "STS#1" -name "Base"
New-SPWeb -url $base_url/Dk -addtoquicklaunch -template "STS#1" -name "Dk"
New-SPWeb -url $base_url/Uk -addtoquicklaunch -template "STS#1" -name "Uk"
New-SPWeb -url $base_url/Dk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Dk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"
New-SPWeb -url $base_url/Uk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Uk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"

$dk = Get-SPWeb $base_url/Dk
$dkBusinessUnit1 = Get-SPWeb $base_url/Dk/BusinessUnit1
$dkBusinessUnit2 = Get-SPWeb $base_url/Dk/BusinessUnit2

$locale = [System.Globalization.CultureInfo]::CreateSpecificCulture("da-DK")
$dk.Locale = $locale
$dk.Update()

$dkBusinessUnit1.Locale = $locale
$dkBusinessUnit1.Update()

$dkBusinessUnit2.Locale = $locale
$dkBusinessUnit2.Update()

How can we reduce this repetition? Well, it repeats because we’ve made the common mistake of mixing logic with data. One way to separate the two is using a table-driven approach, i.e., extract the varying parts, the data, into a table and rewrite the logic so it’s parameterized by the table. After some trial and error with PowerShell here’s the code I ended up with:

$sites = @( @("", "Base"),
            @("Dk", "Dk", "da-DK"),
            @("Uk", "Uk"),
            @("Dk/BusinessUnit1", "BusinessUnit1", "da-DK"),
            @("Dk/BusinessUnit2", "BusinessUnit2", "da-DK"),
            @("Uk/BusinessUnit1", "BusinessUnit1"),
            @("Uk/BusinessUnit2", "BusinessUnit2") )

$sites | foreach {
  $url, $title, $culture = $_
  $newSite = New-SPWeb -url "$base_url/$url" -addtoquicklaunch -template "STS#1" -name $title

  if ($culture -ne "") {
    $locale = [System.Globalization.CultureInfo]::CreateSpecificCulture($culture)
    $newSite.Locale = $locale
    $newSite.Update()
  }
}

So far so good. The un-installation part of the script, however, follows the same pattern of repetition:

Remove-SPWeb -identity $base_url/Uk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url -confirm:$false

But now that we have the table of sites at hand, we can easily make the site removal table-driven as well:

function Reverse($array) {
  $clone = $array.Clone()
  [Array]::Reverse($clone)
  $clone
}

Reverse($sites) | foreach {
  $url = $_[0]
  Remove-SPWeb -identity "$base_url/$url" -confirm:$false -ErrorAction SilentlyContinue
}

I’m sure the code could be written more elegantly, but given my working knowledge of PowerShell I’m satisfied with the result. I especially like the functional programming style.

Although Visual Studio, in tandem with the MSBuild engine, generally takes good care of compilation, I rarely want to rely on it solely. Instead, I’d prefer that any developer task be easily run from the command-line. This ensures that no magic is going on, that the task is kept simple and flexibility, and that it doesn’t rely on Visual Studio to work. The challenge, however, is which of the many tools available to pick. It must be one that’s flexible enough to meet most requirements with relative ease or the automation will likely not be a valid documentation medium in itself.

Why not to use Windows batch files

As a general example, consider the deployment of a SharePoint 2007 solution. With SharePoint a good deal more than compiling is needed to bring new functionality into a SharePoint installation. Whereas Visual Studio and MSBuild may still compile the code and WSPBuilder create the WSP installation package, both from within Visual Studio and from the command-line, getting everything setup cries for automation, even locally. A common approach (possibly inspired by popular SharePoint 2007 literature) is that of the batch file applying various operations to a WSP. For the sake of brevity, I’ve kept the example short, but imagine extending it with a check-out source code task, a compilation task, a WSPBuilder task, and a feature deactivation and activation task, and possibly a warm-up task. Add to this a master script that orchestrate the whole thing. In the end, I’d end up with scripts that become increasingly harder to maintain as they grow in number and size.

    @set STSADM="c:\program files\common files\microsoft shared\web server extensions\12\bin\stsadm"

    if "%1" == "uninstall" goto uninstall
    if "%1" == "install" goto install
    if "%1" == "" goto reinstall

    :uninstall
        %STSADM% -o retractsolution -name Foo.wsp -immediate
        %STSADM% -o execadmsvcjobs
        %STSADM% -o deletesolution -name Foo.wsp -override
        goto end

    :install
        %STSADM% -o addsolution -filename Foo.wsp
        %STSADM% -o deploysolution -name Foo.wsp -immediate -allowGacDeployment -force
        %STSADM% -o execadmsvcjobs
        goto end

    :reinstall
        call Foo uninstall
        call Foo install
        goto end

    :end

Still, because of the ubiquitous nature of command.com and cmd.exe, the batch file interpreters, batch files are everywhere. Regardless that the technology is a left-over from the days of MS DOS and haven’t evolved much since. Not only are the branching and looping constructs limited, so are the available commands. Suppose I want to find out if a command was indeed successful. This turns out to be really hard when all I have to work with is the errorlevel of the most recently executed command. Assuming the command adheres to the errorlevel convention, for the script to fail as early and as close to the real error as possible, I’d have to inspect the property after each command, causing the batch file to grow quite verbose. Sadly, batch files lack the equivalent of set -o errexit of Bash, where the interpreter checks for success after each command and aborts immediately on error. Relying solely on the errorlevel is oftentimes insufficient anyway. To determine success, I’d typically have to parse actual command output or inspect some system property by downloading additional commands or building my own.

Essential vs. incidental requirements

Unless I plan to sit idle and stare at the screen and be a human error detector while the batch file run, I think it’s safe to say that it’s mostly unsuitable for developer automation. Hence in late 2005 I rolled my own automation tool in Python. With Python or Ruby or a similar dynamic language acting as the glue that ties everything together, almost any task can be automated in a robust way. Of course, I could also automate with a static language like C# (it’s surprisingly common). But for script-like tasks, I don’t particularly fancy the long cycle of editing source code in Visual Studio, compiling it, deploying it, and having a hard time debugging it in environment without Visual Studio. A dynamic language, on the other hand, short-circuits the development cycle and allows for interactive programming through a REPL.

With a dynamic language that interacts with .NET, such as IronPython, IronRuby, or Powershell, possibly with supporting DSLs like Paver, Rake, or psake on top, the need for writing custom commands to interact with the operating system or the application almost vanishes. The question, then, is which of the dynamic languages to go with when at their technical core they’re so much alike. Besides sharing the concept of a REPL, the notion of a tuple, a list, a map, and operations on each are baked into their syntax, making code quite terse. It even makes it convenient to express any configuration in the language itself and not in XML where I’d first have to come up with a schema, and then create an instance of it before parsing it. On Windows, however, Powershell is gradually becoming the next ubiquitous interpreter, with applications shipping with cmdlets, whereas IronPython or IronRuby is a separate download.

Conclusion

No matter the tool, what I end up doing is defining tasks, form dependencies between tasks, and have the tool execute tasks in an order that satisfies their dependencies. As the tool traverses the dependency graph and executes tasks, it’s up to each task to detect success, and up to the tool to report on progress. A good tool is therefore characterized by the ease with which I can express these things, the available language constructs, the ease of debugging, and the tool’s ability to converse in foreign languages.

I found the given-expect pattern useful in testing a piece of code that I was working on this week. I was refactoring and adding tests around an ASP.NET control adapter that makes SharePoint 2007 pages more XHTML compliant. I wanted to reuse the transformations outside the control adapter and hence ended up moving the transformation logic to a new class. It accepts possibly malformed HTML and relies on heuristics of the HTML Agility Pack to build a DOM off of it. I can then query the DOM, looking for known violations, and patch them before returning XHTML to the caller.

    public class HtmlToXHtmlTransformer {
        private readonly HtmlDocument _document;

        public HtmlToXHtmlTransformer(string html) {
            _document = new HtmlDocument();
            _document.DetectEncoding(new StringReader(html));
            _document.LoadHtml(html);
        }

        private void Transform(string xpath, Action<HtmlNode> nodeMatch) {
            var nodes = _document.DocumentNode.SelectNodes(xpath);
            if (nodes != null)
                foreach (var node in nodes)
                    nodeMatch.Invoke(node);
        }

        private void FixDuplicateBorderAttributeOnSPGridViewControl() {
            Transform("//table[count(@border)=2]", node => node.Attributes.Remove("border"));
        }

        public string Transform() {
            FixDuplicateBorderAttributeOnSPGridViewControl();
            _document.OptionWriteEmptyNodes = true;
            return _document.DocumentNode.WriteTo();
        }
    }

The complete HtmlToXHtmlTransformer collects a dozen transformations. Its Transform method is what we want to call with various HTML fragments to verify that they come out as XHTML. For this purpose, we might do the tests as Visual Studio data-driven tests that read their input and output from a text file. But in most cases I prefer traditional tests, so I can describe the purpose of a test with a descriptive method name and possibly a comment.

    [TestClass]
    public class HtmlToXHtmlTransformerTest {
        private string _result;

        [TestMethod]
        public void Must_selfclose_nodes_when_allowed() {
            Given("<br>");
            Expect("<br />");
        }

        [TestMethod]
        public void Must_remove_duplicate_border_on_SPGridView_control {
            Given(@"<table border=""0"" border=""0""></table>");
            Expect(@"<table border=""0""></table>");
        }

        private void Expect(string xhtml) {
            Assert.AreEqual(xhtml, _result);
        }

        private void Given(string html) {
            var transformer = new HtmlToXHtmlTransformer(html);
            _result = transformer.Transform();
        }
    }

I particularly like the clarity of the given-expect pattern and find that for a reasonable number of tests it’s a viable alternative to data-driven test. I do, however, recognize the value of data-driven tests in situations where a non-developer wants to test a class. Though at the unit test level I’ve never experienced this. It’s more characteristic of FitNesse for acceptance testing. However you unit test, just make sure your tests run with a minimum of effort on your part and that they run fast.

Here’re the talks that I attended, all recorded and available with slides.

SharePoint 2010 Planning and Best Practice Approaches to Upgrade

Upgrading to SharePoint 2010, you should first establish a copy of your existing environment on the latest patch level. Then run the stsadm command with the new PreUpgradeChecker option. Make sure your environment has no unused features, site templates, and so on, and identify everything that has changed from out-of-the-box by running a tool like WinMerge on the 12 hive. Also, assert that no pages has been ghosted, and don’t attempt to upgrade your SharePoint 2007 master page. Start with the 2010 ones and add in your 2007 modifications. SharePoint 2010 ships with both 2007 and 2010 master pages, but SharePoint 2007 and 2010 isn’t binary compatible.

Make sure all hardware requirements are satisfied: 8 GB RAM at the minimum on the Windows 2008 servers running SharePoint and MS SQL Server. On the client you need Internet Explorer 7 or 8 or Firefox 3.x to author content. Internet Explorer 6 isn’t supported. Not even with a degraded reader experience. The SharePoint 2010 development environment should be setup with Visual Studio 2010 on Windows Vista or Windows 7 in order to compile, debug, build, and deploy right out of Visual Studio 2010.

Understanding, using, and customizing PowerShell for a SharePoint 2010 Environment

This talk started with some PowerShell history and basics. PowerShell is made up of a shell that executes commands and an integrated scripting environment for development. .NET compiled code, called cmdlets, are the executable units combined using the PowerShell pipeline. The PowerShell language itself is dynamic and loosely typed, so being able to express yourself tops performance. Scripts may be compiled and deployed as a SnapIn, consisting of compiled DLLs and registry updates, or as a module using xcopy deployment.

SharePoint 2010 ships with 650 cmdlets, and its own base class for writing additional SharePoint cmdlets.

Design and Manage Site Collections in SharePoint 2010

By default every single site collection within a web application goes into one rapidly growing database (think about users migrating from file shares to site collections). Such a large database takes time to backup and restore. Hence, with SharePoint 2010, consider partitioning site collections into their own databases from the beginning. After a database is created, map it to one and only one site collection. Then, instead of putting a size limit on the web application, put it on the site collection itself.

In SharePoint 2007 you had to create multiple Shared Service Providers (SSP) if site collections had different needs or you didn’t want a site collection to access parts of an existing SSP. With SharePoint 2010, application services can now be shared between site collections.

With sandbox solutions, the solution is deployed to the database hosting the site, rather than to front-end servers. This makes it easier to deploy code, but the code is limited in what it can do, and you may have to set resource quotas across the site collection.

SharePoint 2010, Getting Ready

This talk provided an overview of SharePoint 2010. WSS 3.0 becomes SharePoint Foundation 2010 and MOSS 2007 becomes SharePoint Server 2010. Business Connectivity Services (BCS) are no longer part of the server edition, but has been moved to Foundation. A lot more services have been added, like state service, usage and health service, and Visio graphics service. Generally, with all the improvements to deployment and upgrade with PowerShell and the addition of multi-tenancy for the SSP, SharePoint 2010 is better in a hosting scenario.

The web user interface has been made more Office-like with the addition of the ribbon and the use of AJAX. In document libraries you can create document sets as a way to relate documents to each other and have versioning and workflow follow every document in the set. Through the BCS, you can expose external data as native SharePoint lists with full CRUD capabilities.

SharePoint 2010 Development Tips and Tricks

The focus of the first part this talk is on the SharePoint 2010 Foundation features. Instead of developing against the web services or the object model, you can use the new managed client object model. This way, you can work against SharePoint from .NET code, from a Silverlight application, or from JavaScript. Behind the scenes, the client object model works against a WCF service running on the SharePoint server, which talks to the server object model on the server, and returns JSON to the client.

The second part of this talk is about sandbox solutions. This feature provides site collection users with the ability to upload and deploy WSPs. Finally, the third part of the talk is about the fluid integration model, which solves the problem of accessing cross-domain resources, e.g., having a Silverlight application access information on a SharePoint server on a different domain.

Installing and Configuring SharePoint Server 2010 in a Virtual Environment

This talk recommends that everyone read the MS whitepaper on Virtualization of Microsoft SharePoint Products and Technologies. Configuring a development environment, the dilemma is weather to create one virtual machine hosting everything or one virtual machine for the MS SQL Server, the Active Directory, and the SharePoint server each. The presenter suggests a middle ground. Still, it’s important to create a number of service accounts and use them in your development environment rather than running everything as administrator.

The SharePoint 2010 installer comes with a prerequisite installer that goes onto the web and downloads required software. After that, you can run the SharePoint server installer. Here you can choose between Standalone and Server farm, but always pick server farm. Standalone should only be used for demo purposes.

Last week at work I did a 45 minutes presentation on how to get started with SharePoint development. Instead of the usual presentation on the technical merits of SharePoint, my focus was more on the change in mindset required coming from ASP.NET. For even though ASP.NET is a cornerstone of SharePoint, SharePoint is so much more. Not to mention different.

History makes a difference

SharePoint is a long-evolving product, dating back to the late nineties. From its inception and up till now new technologies have replaced old ones, APIs have emerged and died, product teams have merged, and Microsoft have bought and integrated other companies’ products into SharePoint. At the same time SharePoint has become the fastest growing server product in Microsoft history. All of this inevitably leaves its marks on the APIs, which aren’t as consistent and defect free as we’re used to with the .NET framework.

Provisioning is the magic sauce

What truly makes SharePoint development different from ASP.NET is the notion of provisioning. With ASP.NET you create a bunch of ASPX files, organize them in a static file structure, and deploy them to a server. With SharePoint, the structure emerges over time as SharePoint or the user creates instances of templates: sites from site templates, lists from list templates, pages from page templates, and so forth. From a developer’s perspective the challenge isn’t the instance creation in itself. It’s that SharePoint encourages users to not only dynamically evolve the structure, but also to modify the instances, severing their definition from the template. While it empowers users, it also makes it challenging to programmatically evolve the structure.

Developing with SharePoint is about creating templates and components for SharePoint or the user to compose a site of. To make this composition work, SharePoint imposes constraints on what can be done and how. While any constraint may at times be perceived as a hindrance, accepting it is what guarantees that a components will play nicely with others. These pieces of functionality are then packaged as features and added to a WSP installation file for deployment to SharePoint. Features may then be activated, making their functionality available to a site. Or deactivated and the WSP uninstalled.

Why learn SharePoint

By the end of the day, no doubt SharePoint is more challenging to work with than ASP.NET. Not acknowledging its history makes many ASP.NET developers so frustrated that they’ll never again want to work with SharePoint. Coming from ASP.NET I can understand why. At times SharePoint isn’t as smooth and evolved as ASP.NET. But knowing what to expect takes the worst of the frustration. And once you learn to leverage the power of SharePoint, you can build applications without writing a lot of plumping code. To a large extend the architecture is given, leaving you with the opportunity to fill in the blanks.

Maybe it’s that the SharePoint API is hard to use. Maybe it’s that coding against SharePoint is about making smaller additions here and there. Maybe it’s that the Patterns & Practices SharePoint Guidance isn’t widely known. Whatever the reason, developing for SharePoint requires equal attention to the separation of presentation, business, and data access code. Hence, starting with data access, we may want to create a repository and route queries through it (the SharePoint Guidance outlines a more sophisticated implementation than the one below):

    [TestClass]
    public class EmployeesRepositoryTest {
        private SPSite _siteCollection;
        private SPWeb _site;
        private EmployeesRepository _repository;

        private readonly Employee _duffyDuck = new Employee {
            Id = 1000, Name = "Duffy Duck", HireDate = new DateTime(2009, 12, 1),
            Remarks = "Looks like a duck, quacks like a duck, probably is a duck"
        };
        private readonly Employee _porkyPig = new Employee {
            Id = 1001, Name = "Porky Pig", HireDate = new DateTime(2010, 2, 1)
        };
        private readonly Employee _sylvesterTheCat = new Employee {
            Id = 1002, Name = "Sylvester the Cat", HireDate = new DateTime(2010, 3, 1)
        };
        private readonly Employee _bugsBunny = new Employee {
            Id = 1100, Name = "Bugs Bunny", HireDate = new DateTime(2010, 1, 1)
        };

        public void AddEmployees() {
            _repository.AddEmployee(_site, _duffyDuck);
            _repository.AddEmployee(_site, _porkyPig);
            _repository.AddEmployee(_site, _sylvesterTheCat);
        }

        public void ClearEmployees() {
            var definition = new EmployeesDefinition();
            var employees = _site.Lists[definition.ListName];
            while (employees.Items.Count > 0)
                employees.Items.Delete(0);
        }

        [TestInitialize]
        public void Initialize() {
            _siteCollection = new SPSite("http://localhost");
            _site = _siteCollection.OpenWeb("/");
            _repository = new EmployeesRepository();
            ClearEmployees();
            AddEmployees();
        }

        [TestCleanup]
        public void Cleanup() {
            _site.Dispose();
            _siteCollection.Dispose();
        }

        [TestMethod]
        public void AddEmployee_should_add_valid_employee() {
            _repository.AddEmployee(_site, _bugsBunny);
            var e = _repository.GetEmployeeById(_site, _bugsBunny.Id);
            Assert.AreEqual(_bugsBunny.Id, e.Id);
            Assert.AreEqual(_bugsBunny.Name, e.Name);
            Assert.AreEqual(_bugsBunny.HireDate, e.HireDate);
            Assert.AreEqual(_bugsBunny.Remarks, e.Remarks);
        }

        [TestMethod]
        public void GetEmployeesHiredBetween_should_return_2010_hires() {
            var from = new DateTime(2010, 1, 1);
            var to = new DateTime(2010, 12, 31);
            var employees = _repository.GetEmployeesHiredBetween(_site, from, to);
            Assert.AreEqual(2, employees.Count);
            Assert.AreEqual(_porkyPig.Id, employees[0].Id);
            Assert.AreEqual(_sylvesterTheCat.Id, employees[1].Id);
        }
    }

In the words of Martin Fowler, here’s the essence of the Repository pattern:

A Repository mediates between the domain and data mapping layers, acting like an in-memory domain object collection. Client objects construct query specifications declaratively and submit them to Repository for satisfaction. Objects can be added to and removed from the Repository, as they can from a simple collection of objects, and the mapping code encapsulated by the Repository will carry out the appropriate operations behind the scenes. Conceptually, a Repository encapsulates the set of objects persisted in a data store and the operations performed over them, providing a more object-oriented view of the persistence layer.

In SharePoint terms, business logic should query a repository which in turn queries a SharePoint list. Within the repository, the weakly typed items returned are then mapped to strongly typed data transfer objects, which are returned to the business layer.

With this approach to data access comes a number of advantages: (1) duplicate data access code is eliminated. Only within the repository do we setup the query and transform the weakly typed SPListItemCollection into strongly typed data transfer objects. (2) The list definition classes introduced in SharePoint list definition using the Template pattern may be used to construct CAML queries from strongly typed field names. Lastly, (3) accessing data through a repository makes it easier to mock the data access part of the application and to integration test that part.

The code below is a simple, yet usable, implementation of the Repository pattern. The idea is to have all repositories inherit from a common base class. Its purpose is to wrap the querying of a list and to log what’s going on. Because it’s a base class, it shouldn’t know how to transform the weakly typed result into strongly data transfer objects or which CRUD operations a particular repository supports:

    public abstract class ListRepository {
        protected ListDefinition Definition { get; set; }
        protected SPListItemCollection Result { get; set; }

        protected void Query(SPWeb site, string caml) {
            AssertValidSite(site);
            AssertValidCaml(caml);
            AssertListExistence(site);
            AssertListDefinitionSetBySubclass();

            var watch = new Stopwatch();
            var list = site.Lists[Definition.ListName];
            var query = new SPQuery {Query = caml};
            Debug.WriteLine(
                string.Format("About to run query against list '{0}': {1}", list, caml));
            watch.Start();
            Result = list.GetItems(query);
            watch.Stop();
            Debug.WriteLine(
                string.Format("Query against '{0}' returned {1} rows in {2} ms",
                              list, Result.Count, watch.ElapsedMilliseconds));
        }

        protected void AssertListExistence(SPWeb site) {
            if (!ListDefinition.ListExists(site, Definition.ListName))
                throw new ArgumentException(
                    string.Format("No '{0}' list on site '{1}'", Definition.ListName, site.Url));
        }

        protected void AssertListDefinitionSetBySubclass() {
            if (Definition == null)
                throw new NullReferenceException(
                    string.Format(
                        "Sublcass must set Definition property prior querying '{0}' list",
                        Definition.ListName));
        }

        protected void AssertValidCaml(string query) {
            if (string.IsNullOrEmpty(query))
                throw new NullReferenceException("Query must not be null or empty");
        }

        protected void AssertValidSite(SPWeb site) {
            if (site == null)
                throw new NullReferenceException("Site must not be null");
        }
    }

Each ListRepository connects to a corresponding ListDefinition, holding the name of the list to query and its strongly typed field names. It’s the responsibility of a concrete repository to set the Definition property prior to doing any querying. After running a query, the Result property holds the weakly typed result, which a concrete repository can then transform into data transfer objects to be passed to the business layer.

As an example, add to the concrete EmployeeRepository any CRUD method you see necessary to fulfill the business requirements:

    public class EmployeesRepository : ListRepository {
        public EmployeesRepository() {
            Definition = new EmployeesDefinition();
        }

        public void AddEmployee(SPWeb site, Employee e) {
            var list = site.Lists[Definition.ListName];
            var item = list.Items.Add();
            item[EmployeesDefinition.EmployeeId] = e.Id;
            item[EmployeesDefinition.Name] = e.Name;
            item[EmployeesDefinition.HireDate] = e.HireDate;
            item[EmployeesDefinition.Remarks] = e.Remarks;
            item.Update();
        }

        public Employee GetEmployeeById(SPWeb site, int id) {
            var caml =
                string.Format(@"
                      <Where>
                        <Eq>
                          <FieldRef Name=""{0}"" />
                          <Value Type=""Integer"">{1}</Value>
                        </Eq>
                      </Where>",
                    EmployeesDefinition.EmployeeId, id);
            Query(site, caml);

            IList<Employee> employees = Map(Result);
            if (employees.Count == 0)
                throw new ArgumentException(string.Format("No employee with id = {0} exists", id));
            return employees[0];
        }

        public ReadOnlyCollection<Employee> GetEmployeesHiredBetween(SPWeb site, DateTime from, DateTime to) {
            var caml =
                string.Format(@"
                      <Where>
                        <And>
                            <Geq>
                              <FieldRef Name=""{0}"" />
                              <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{1}</Value>
                            </Geq>
                            <Leq>
                              <FieldRef Name=""{0}"" />
                              <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{2}</Value>
                            </Leq>
                        </And>
                      </Where>",
                    EmployeesDefinition.HireDate,
                    SPUtility.CreateISO8601DateTimeFromSystemDateTime(from),
                    SPUtility.CreateISO8601DateTimeFromSystemDateTime(to));
            Query(site, caml);
            return new ReadOnlyCollection<Employee>(Map(Result));
        }

        protected IList<Employee> Map(SPListItemCollection items) {
            var employees = new List<Employee>();
            foreach (SPItem item in items) {
                var e = new Employee {
                                Id = (int)item[EmployeesDefinition.EmployeeId],
                                Name = (string)item[EmployeesDefinition.Name],
                                HireDate = (DateTime)item[EmployeesDefinition.HireDate],
                                Remarks = (string)item[EmployeesDefinition.Remarks]
                            };
                employees.Add(e);
            }
            return employees;
        }
    }

Calling the GetEmployeeById or GetEmployeesHiredBetween methods, the caller is required to pass in an SPWeb instance pointing to the the site holding the list to query. Outside the SharePoint context, you have to manually create this instance, like with the integration tests above. But within the SharePoint context, callers are likely to just pass in SPContext.Current.Web.

The above repository implementation deliberately ignores any issue of caching. If you find the need for it, however, you can replace SPList.GetItem with PortalSiteMapProvider.GetCachedListItemsByQuery. The advantage of using the PortalSiteMapProvider over Asp.Net caching of the result is that the provider takes care of invalidating cache entries when items are modified. The disadvantage is that the provider is part of the SharePoint publishing API, which isn’t part of WSS 3.0. In addition, it’s only available from code running within SharePoint, and likely requires cache settings to be tweaked.

Another, non-trivial, improvement would include the use of the Unit of Work pattern. A commonly used pattern in ORMs because it offers a way to “keep track of everything you do during a business transaction that can affect the database. When you’re done, it figures out everything that needs to be done to alter the database as a result of your work". Like with the DataContext class in LINQ to SQL or the DataSet class in ADO.NET, it could add transaction support to SharePoint lists.

Books on SharePoint often show how to create lists from code by calling the SharePoint API from directly within a feature receiver. This receiver would then contain in-place string literals for column names, create the columns, update the default view, and so on. While such an approach provides for an nice demonstration of the SharePoint API, it tends to carry over into production code.

My take on feature receivers, however, is that they should contain the least possible amount of code. Staying true to object-orientation and separation of concerns, list definition and creation should instead consist of a common set of classes and methods to be called from the receiver:

    public override void FeatureActivated(SPFeatureReceiverProperties p) {
        using (var site = p.Feature.Parent as SPWeb) {
            var employees = new EmployeesDefinition();
            if (!ListDefinition.ListExists(site, employees.ListName))
                employees.CreateOnSite(site);
        }
    }

The first area to improve on is that of string literal field names getting duplicated across the code base. String literals make it impossible for the compiler to enforce a consistent naming of columns. As column names are used in defining the list, in forming CAML queries against it, and in accessing the result, misspelled column names is a common source of runtime errors. The second area to improve on is that of duplication of code for doing common list operations, like checking for the existence of the list or removing its title column. Lastly, we want to get rid of duplicate control logic, i.e., verifying preconditions and calling the same series of methods with every list definition.

One approach to improving on these areas is to start by defining a base class for list definitions. The base class combines helper methods, working on lists, with the Template pattern for defining the common steps of list creation:

    public abstract class ListDefinition {
        public string ListName;
        public string ListDescription;
        protected SPList List;

        public void CreateOnSite(SPWeb site) {
            if (string.IsNullOrEmpty(ListName))
                throw new ArgumentException("Expected not null or not empty list name member");
            if (site == null) throw new NullReferenceException("Expected valid site");
            CreateList(site);
            if (List == null) throw new NullReferenceException("Expected list member not null");
            CreateFields(List);
            UpdateDefaultView(List);
            SetAdditionalProperties(List);
        }

        protected abstract void CreateList(SPWeb site);
        protected virtual void CreateFields(SPList l) {}
        protected virtual void UpdateDefaultView(SPList l) {}
        protected virtual void SetAdditionalProperties(SPList l) {}

        public static void HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(SPList l) {
            var title = l.Fields.GetField("Title");
            title.Required = false;
            title.Hidden = true;
            title.Update();
        }

        public static bool ListExists(SPWeb site, string listName) {
            if (string.IsNullOrEmpty(listName))
                throw new ArgumentException("ListNameMustNotBeNullOrEmpty", listName);
            SPList list = null;
            try {
                list = site.Lists[listName];
            }
            catch (ArgumentException) {
                // list not found
            }
            return list != null ? true : false;
        }
    }

At the core of the Template pattern is the template method. It defines the steps of an algorithm through a series of method calls. Each either abstract or virtual depending on if the step is mandatory or voluntary in the algorithm. Subclasses then specify the behavior of individual steps by implementing or overriding methods. In case of the ListDefinition class, CreateOnSite is our template method. It ensures that context, in the form of an SPList instance, is passed along to each step in the algorithm.

Each subclass defines the specific characteristics of a list. It defines constants for column names and their data types, the views and which column to index, and so on. This makes for a concise, easy to read, and consistent list definition:

    public class EmployeesDefinition : ListDefinition {
        public const string EmployeeId = "EmployeeId";
        public const string Name = "Name";
        public const string HireDate = "HireDate";
        public const string Remarks = "Remarks";

        public EmployeesDefinition() {
            ListName = "AcmeEmployees";
            ListDescription = "Employees at Acme Corp.";
        }

        protected override void CreateList(SPWeb site) {
            var guid = site.Lists.Add(ListName, ListDescription, SPListTemplateType.GenericList);
            List = site.Lists[guid];
        }

        protected override void CreateFields(SPList l) {
            l.Fields.Add(EmployeeId, SPFieldType.Integer, true);
            l.Fields.Add(Name, SPFieldType.Text, true);
            l.Fields.Add(HireDate, SPFieldType.DateTime, true);
            var remarks = (SPFieldMultiLineText) l.Fields[l.Fields.Add(Remarks, SPFieldType.Note, false)];
            remarks.RichText = true;
            remarks.RichTextMode = SPRichTextMode.FullHtml;
            remarks.Update();
            l.Update();
        }

        protected override void UpdateDefaultView(SPList l) {
            var v = l.DefaultView;
            v.ViewFields.Delete("LinkTitle");
            v.ViewFields.Add(EmployeeId);
            v.ViewFields.Add(Name);
            v.ViewFields.Add(HireDate);
            v.ViewFields.Add(Remarks);
            v.Update();
        }

        protected override void SetAdditionalProperties(SPList l) {
            HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(l);
            l.Fields[HireDate].Indexed = true;
            l.Update();
        }
    }

Since the string literals have now become public constants, we get IntelliSense and compile-time checking referencing them. In addition, by defining common operations on the base class and using the Template pattern, we reduce the amount of code that would otherwise have to go into each definition. Finally, the base class asserts, on behalf of all subclasses, that the list name is set before attempting to create the list and that the SPList field is set after calling the CreateList method.

The reason for bringing up the linked file feature in conjunction with assembly signing is that to me the feature seems particularly well-suited for linking one and only key file into multiple Visual Studio projects. Yet people seem to be unaware of the feature and so common approaches to signing is to either create a new key file for each assembly — most likely because Visual Studio makes it so easy to create a new key file from the Project Settings dialog — or signing assemblies by creating one key file and then copy/paste it to other projects in the solution.

Below is an example from the GAC showcasing how assemblies related to Pex naturally go together and are therefore signed using the same key (they share 76a274db078248c8, their public key token). The same goes for assemblies relating to other Microsoft products, such as the ones comprising a particular version of the .Net framework (because of the side-by-side execution, different versions of .Net framework assemblies are signed using different keys). So why not apply the same principle to what you deploy to the GAC?

While using the same key (copy or not) makes it possible to differentiate your assemblies from everyone else and for your four-part assembly name in web.config and spread around the code to be uniform — SharePoint developers often need to provide the four-part name for their assemblies to other parts of the solution — copying the key around is still a violation of the DRY principle and as such is indicative of a code smell. More importantly, using the link feature of Visual Studio, there’s no reason to keep the bad smell around.

Assuming you’ve already created a solution (in Visual Studio 2008) and added a project to it, the way to share a key file is to first create one by right clicking a project in Solution Explorer and selecting Properties. Now click the Signing tab and check the Sign the assembly check box. The dropdown lets you create a new key file by selecting New and filling out the information in the Create Name Key window. Alternatively, a key file is created using sn.exe, the strong name utility that comes with the Microsoft Windows SDK. On my machine, sn.exe is located in C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin and is invoked like so:

    > sn -k MySharedKeyPair.snk

How the key file is created isn’t particularly relevant to the rest of this post. Just make sure to place the key file in a common location, e.g., the top-level directory holding the Visual Studio solution file. Now go to Solution Explorer, right click on your project, and select Add existing Item. Then locate the key file within the solution directory and make sure to click the arrow on the Add button and select Add as Link. Note how the file is added to the project and marked by an arrow indicating it’s a link:

Returning to the Signing tab and selecting the dropdown, it now includes the linked key file. Don’t mind the absolute path; once you select MySharedKeyPair.snk, Visual Studio converts the path to a relative one (close and reopen the Signing tab and see for yourself). Also note that if you select Browse, the key file you select is copied to the project folder and added to the project as a regular file.

Another way to associate a key file with an assembly is through the AssemblyKeyFile attribute. The attribute instructs the compiler of where to look for the key file and, although not used by the runtime, the attribute is included in the assembly along with any other attribute for everyone to see.

Using AssemblyKeyFile is probably the simplest approach to sharing a key file between projects, but now compiling the assembly results in a compiler warning as displayed above. The use of the AssemblyKeyFile attribute has been deprecated by Microsoft because it may lead to you inadvertently disclosing sensitive information through the embedded path and because working with a relative path may confuse users and the build system (the relative path is retained in the assembly, though). However, it appears AssemblyKeyFile is still the preferred way for MS to sign their assemblies. Inspecting a couple of .Net 3.5 assemblies with Reflector, here’s what their AssemblyKeyFile looks like:

    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\EcmaPublicKey.snk")]
    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\35MSSharedLib1024.snk")]
    [assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\FinalPublicKey.snk")]
    …

To come full circle, the question of how Visual Studio, or rather MSBuild, actually signs an assembly using the information provided in the Properties dialog (real file or link) remains to be answered. Since a Visual Studio project file acts as input to MSBuild, we can see what’s going on under the hood of MSBuild by invoking a compile from the command line. On my machine msbuild.exe is located in C:\Windows\Microsoft.NET\Framework\v3.5 and assuming you have a console window open with its current directory set to the directory of your project file, msbuild is invoked like so:

    > msbuild /verbosity:diagnostic > log.txt

Bring up log.txt in your favorite editor and eventually you’ll come across where csc.exe, the C# compiler, is invoked:

As you might have guessed, MSBuild carries over the location of the key file by passing the /keyfile argument to csc.exe.

In order to test code whose branching behavior depends on a value read from a configuration file, the test should provide various inputs and assert that the code under test acts accordingly. In addition, the test should run in milliseconds to encourage frequent runs and it should be self-contained, meaning no setup should be required before running the test. These requirements are best met with the web part running in a controlled environment. Yet, as far as the code under test goes, it should behave as if part of an Asp.Net page.

For the sake of brevity, here’s my web part with the code to initialize the data context:

    public class MyWebPart : WebPart {
        MyDataContext _db;    /* LINQ database context */
        string _error;        /* Early stage error message */    	

        public MyWebPart() {
            Initialize();
        }

        private void Initialize() {
            string c = GetMyConnectionString();
            if (c == null)
                 _error = "Unable to do something";
            else
                _db = new MyDataContext(c);
        }

        private string GetMyConnectionString() {
            System.Configuration.ConnectionStringSettings c =
                System.Configuration.ConfigurationManager.
                    ConnectionStrings["MyConnection"];
            return c == null ? null : c.ConnectionString;
        }

        protected override void Render(HtmlTextWriter w) {
            // display error message if set
        }
    }

Notice how the constructor calls out to an Initialize method. Otherwise code within the constructor would fire before I’d have a chance to setup a controlled environment around the web part and I wouldn’t be able to test it. Turning the attention to Initialize itself, it’s made up of only a simple if statement. The idea is that if I can’t mock something as simple as reading a value from a configuration file, most likely I can’t mock something more involved either. On the other hand, for a developer already familiar with TypeMock and test first development, the code above may be born of the need to satisfy a failing test.

I want to test Initialize by controlling how GetMyConnectionString gets to return the connection string. While running in an Asp.Net context, GetMyConnectionString would attempt to read the string from web.config. Within my controlled environment, however, I want to inject into the web part an alternate implementation that returns a value of my choosing. That way I can control the path through Initialize and inspecting the state of the _error or _db fields, I’m able to determine if the code behaved as intended.

Using the method chaining DSL of TypeMock, the test takes on this form (of course, having more tests, I’d move shared code into helper methods decreasing the test to code ratio):

    [TestClass, ClearMocks]
    public class MyWebPartTest {
        [TestMethod, Isolated, VerifyMocks]
        public void Should_set_error_message_when_no_
                    connection_string_is_configured() {
            // arrange
            MyWebPart fake = Isolate.Fake.Instance<MyWebPart>();
            Isolate.NonPublic.WhenCalled(fake,
                "Initialize").CallOriginal();
            Isolate.NonPublic.WhenCalled(fake,
                "GetMyConnectionString").WillReturn(null);
            PrivateObject handle = new PrivateObject(fake);

            // act
            handle.Invoke("Initialize", null); 

            // assert
            Isolate.Verify.NonPublic.WasCalled(fake,
                "GetMyConnectionString");
            string error = (string)handle.GetField("_error");
            MyDataContext ctx = (MyDataContext)handle.GetField("_db");
            Assert.AreEqual("Unable to do something", error);
            Assert.AreEqual(null, ctx);
        }
    }

There’re three parts to tests using TypeMock: (1) the arrange part sets up expectations of what’s going to happen when a method, a property, or some other part of the code under test is invoked; (2) the act part carries out the actual interaction with the object under test, turning to PrivateObject to invoke the Initialize method; and finally (3) the assert part verifies that what was supposed to happen did happen. Worth noting is that this test asserts both state and behavior, i.e., as part of executing the code under test, _error or _db should’ve been set to an appropriate state and GetMyConnectionString should’ve been called once and only once.

This concludes my example, which I believe illustrates the basics of testing and mocking using MS Test and TypeMock. Comparing and contrasting TypeMock to other popular mocking frameworks, TypeMock has the distinct feature that it can mock almost anything, with or without using interfaces. Some argue that not enforcing the use of interfaces is also the disadvantage of TypeMock, because for code to be truly testable, it should be modularized using the Inversion of Control principle. However, with legacy code, such as the .Net framework or the SharePoint API, to me TypeMock appears to bridge the two worlds nicely, leaving the developer in charge.

To learn more about the basic concepts of mocking, Inversion of Control, and Dependency Injection (containers), I’d recommend listening to a couple of Alt.Net podcasts: Dependency Injection and Inversion of Control and More Dependency Injection and Inversion of Control.

In any real-world application, unhalted exceptions are an unavoidable fact of life. Such exceptions should occur relatively infrequently, but when they do some kind of supporting infrastructure better be in place to capture the when and why of the exceptions. As a developer, you should have the ability to perform a postmortem analysis on what went wrong and learn from it. Paramount to such analysis is capturing and logging sufficient information at the point of failure.

What this post covers is the thoughts and development of an Asp.Net HttpModule that captures and emails such unhalted exception information to a designated email address.

To be clear, the goal of the EmailingExceptionModule isn’t to prevent unhalted exception from occurring per se. Within code, a developer may not know how to handle an exception, and so the exception should rightfully propagate the call stack. In case no caller steps in and handles the exception, the best choice is most likely to terminate the application. It’s almost always better to be upfront with the user than conceal the possibly inconsistent state of the application.

So when does the EmailingExceptionModule come in handy then? One scenario is that of C# unchecked exceptions, where the compiler doesn’t force the caller to catch all types of exceptions thrown by the callee. Unchecked exceptions may materialize as type cast or null reference exceptions when accessing variables. What unchecked exceptions boil down to is that, given the cyclomatic complexity of some methods, it’s impractical to manually work through every conceivable path of execution ahead of time. Another scenario is problems with the runtime environment, such as out of disk space or database server down. Nonetheless, some map of the path to failure is helpful in preventing others going down that same path.

(Summary part of example email. Click here for complete output.)

With web applications, we can take advantage of the application running in a centralized environment. In the spirit of Microsoft’s Doctor Watson technology, we can register our own web application error handler and hook it into the Http request pipeline, and have the application execute our code on unhalted exceptions. In practice, such an error handler is implemented either through the Application_Error method of an application’s Global.asax or by loading an HttpModule into the application.

Hinted by the title, I went for the HttpModule. The reason may best be understood by taking a peak behind the .Net curtains: Global.asax is a filename hardwired into the framework so that whenever an application is first hit, the framework asks HttpApplicationFactory, an internal factory class, for an HttpApplication representing the application (there’s a unique one for each virtual application). As part of manufacturing the HttpApplication, the factory locates the application’s Global.asax, compiles it into a dynamically generated DLL, loads the DLL, and reflects over the Global.asax class looking for methods adhering to the convention of modulename_eventname. Methods found, such as Application_Error, are then stored in a list of MethodInfos and passed along to HttpApplication. Then, during HttpApplication initialization, events are bound to the methods within Global.asax.

That’s how Application_Error of Global.asax gets called even though the method isn’t overriding a base class implementation as is typical for the template pattern. That’s also why code within global.asax isn’t binary reusable across applications. An HttpModule, on the other hand, can be loaded into any number of applications.

In practice, implementing an HttpModule it a matter of creating a class that implements the IHttpModule interface:

   public interface IHttpModule {
      void Init(HttpApplication context);
      void Dispose();
   }

The crux of error handling with Asp.Net is the Error event of HttpApplication. The easiest way to hook up the event to a method is within the Init method of the class. In addition, we assign the application object to a field so that later we can access the most recently thrown exception through it:

   public class EmailingExceptionModule : IHttpModule {
      private HttpApplication _application;

      public void Init(HttpApplication a) {
         _application = a;
         _application.Error += OnUnhaltedException;
      }

      public void Dispose() {}

      private void OnUnhaltedException(object sender, EventArgs e) {
         Exception ex = _application.Server.GetLastError().InnerException;
         // Implementation left out for brevity. Please download source
      }
   }

The OnUnhaltedException method is where to add code that collects information about the exception and the environment, and that composes and ships the email. Depending on the environment, you may find the need to include additional information in the email. Given the source code, adding to the email is a matter of adding key/value pairs in the form of labels and values to a dictionary. Each dictionary then gets rendered as a table of the key and value columns.

To have an Asp.Net application load the EmailingExceptionModule and to configure its email related settings, add the following nodes to the application’s web.config:

   <configuration>
      <appSettings>
         <add key="EmailingExceptionModule_SenderAddress" value="sender@domain.com"/>
         <add key="EmailingExceptionModule_ReceiverAddress" value="receiver@domain.com"/>
         <add key="EmailingExceptionModule_SmtpServer" value="mail.domain.com"/>
         <add key="EmailingExceptionModule_SubjectPrefix" value="MyApp: "/>
      </appSettings>
      <system.web>
         <httpModules>
            <add name="EmailingExceptionModule" type="Holm.AspNet.EmailingExceptionModule,
                       EmailingExceptionModule, Version=1.0.0.0, Culture=neutral,
                       PublicKeyToken=13806f613f05e959"/>
         </httpModules>
      </system.web>
   </configuration>

So what’s the user experience of unhalted exceptions? Outside the confines of the EmailingExceptionModule, there’s no indication that the application took note of what happened. Depending on the user’s profile, it might be helpful to prompt for additional information at the crash point. Or perhaps turn to the Windows Live Messenger IM Control & Presence API and have the user engage in an MSN conversation with a developer.

What’s the advantage of emailing developers unhalted exceptions over storing the information in the file system or a database? I believe in keeping things simple and practical, and handing off information to the file system or a database isn’t. Most likely the information will end up gathering virtual dust on some server. The email approach, on the other hand, is a proactive, visible, and efficient means to the end of rapidly course-correcting for wrong assumptions.

Update, Sep 15: While you don’t need a PDB file to debug code through Visual Studio (that’s what <compilation debug=”true”> in web.config is for), the PDB file is required for the stack trace to contain line numbers. Since the PDB file typically has to reside next to the corresponding DLL for the .Net runtime to pick it up, the PDB file may need to be deployed to the GAC along with the DLL. Refer to the first three paragraphs of the “Debugging assemblies that live in the GAC” section of this post for how to GAC deploy PDB files.

For a centrally deployed Asp.Net application, it may be acceptable to ship and deploy the PDB file with the application. Keep in mind, though, that because the PDB file is a mapping of locations in the IL to the source file, having access to the PDB file makes it easier for someone to reverse engineer the code. It’s also possible to get at line numbers without the PDB file being deployed, but the approach is somewhat involved. Based on the mapping nature of the PDB file, the idea is to write out IL offsets as part of the exception and to post-process the offsets using a PDB file at a separate location.

Yet another alternative to shipping the PDB file is to setup a symbol server. With this nifty piece of code, the StackTrace class loads the PDB file of the symbol server, adding line numbers to the stack trace.

To showcase anti-patterns of API usage, intermediate code is included, i.e., code that may render the correct tags, although not necessarily in the true spirit of the .Net framework.

To set the stage, imagine writing a web control that inherits from ASP.Net WebControl. The goal is then to modify the behavior of the control in such a way that it (1) encloses its content in a div tag and (2) adds a CSS class for styling, and (3) can be added to a page like so:

   <MyControls:MyControl CssClass="myCssClass" runat="server" />

After making its way through the page rendering pipeline, the control tag gets transformed to:

   <div class="myCssClass">
      Hello World
   </div>

Within the control, a straight-forward way of achieving the intended output may be to hand craft the html and render it in RenderContents of WebControl. It almost goes without saying that rendering html by hand is but only a starting point when working with .Net. The next progression from there may be to override RenderBeginTag and RenderEndTag of WebControl:

   // version 1
   public class MyControl : WebControl {
      public override void RenderBeginTag(HtmlTextWriter w) {
         w.RenderBeginTag(string.Format("div class=\"{0}\"", CssClass));
      }

      public override void RenderEndTag(HtmlTextWriter w) {
         w.RenderEndTag();
      }

      protected override void RenderContents(HtmlTextWriter w) {
         w.Write("Hello World");
      }
   }

Resulting in this piece of html:

   <div class="myCssClass">
      Hello World
   </div class="myCssClass">

Clearly, RenderBeginTag and RenderEndTag of HtmlTextWriter don’t know how to separate tag from attribute. Nonetheless, the code reveals the stack based approach taken by the Render-pair of methods.

Gleaning over the documentation of HtmlTextWriter, however, I encountered the AddAttribute method for setting attributes on a tag. Inside HtmlTextWriter, what AddAttribute does is add attributes and associated values to an array that gets rendered during the next call to RenderBeginTag. So besides introducing strongly typed attributes and tags, the next version contains a modified RenderBeginTag:

   // version 2
   public override void RenderBeginTag(HtmlTextWriter w) {
      w.AddAttribute(HtmlTextWriterAttribute.Class, CssClass);
      w.RenderBeginTag(HtmlTextWriterTag.Div);
   }

Now the control renders as expected, although the code still has a certain bad smell to it. WebControl ought to know how to properly render its begin and end tag. After all it’s WebControl that exposes, among others, the CssClass property. So taking another dive into the documentation, I found the virtual, read-only TagKey property of WebControl. Strangely named as it may be, overriding TagKey makes WebControl call our override as it renders the control, adding in optional attributes, such as CssClass. Implementing the property also turns RenderBeginTag and RenderEndTag obsolete:

   // version 3
   protected override HtmlTextWriterTag TagKey {
      get {
         return HtmlTextWriterTag.Div;
      }
   }

As an added bonus, changing the enclosing tag can also be done calling the WebControl’s constructor, passing in the desired tag:

   // version  4
   public class MyControl : WebControl {
      public MyControl() : base(HtmlTextWriterTag.Div) { }

      protected override void RenderContents(HtmlTextWriter w) {
         w.Write("Hello World");
      }
   }

As it turns out, calling the constructor is how WebControl derived classes get to use span in the first place. The default WebControl constructor passes control onto another constructor that sets the value of what gets returned by the base class TagKey implementation. In essence, overriding TagKey makes it a player in the template design pattern:

   // .Net framework code
   protected WebControl() : this(HtmlTextWriterTag.Span) { }

   public WebControl(HtmlTextWriterTag tag) {
      tagKey = tag;
   }

Also worth noting is that for controls deriving from WebControl derived classes, such as CompositeControl, the constructor cannot be used to set the enclosing tag. The constructor exposed by CompositeControl and friends doesn’t call up the constructor chain passing in the tag, making TagKey the choice to go.

Finally, deep down the inhertitance chain below WebControl lives WebPart, a type of control very popular with SharePoint. Interestingly enough, WebPart defaults to using div as its enclosing tag, because walking up the chain, we come across the constructor of Panel:

   // .Net framework code
   public Panel() : base(HtmlTextWriterTag.Div) { }

So, using Panel provides another way to implement a web control that owns a distinct part of a page.

To decide on the items to display, an initial approach might be to go through the list one item at a time, looking for matching tags. But working with large lists in SharePoint this approach is not recommended. Alternatively, the predicate against which each item in the list is evaluated could take on the form of a CAML query. Looking for items matching a single tag is then easily expressed:

   <Where>
      <Eq>
         <FieldRef Name='MyField' />
         <Value Type='LookupMulti'>tag1</Value>
      </Eq>
   </Where>

Next, consider a query with two tags OR’ed together. Although more verbose, the query is still fairly straightforward to compose on the fly:

   <Where>
      <Or>
         <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag1</Value>
         </Eq>
         <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag2</Value>
         </Eq>
      </Or>
   </Where>

However, for CAML queries where more than two tags are OR’ed together, the binary nature of the OR operator works against us. Where real programming languages have their parsers automatically transform an expression like (a || b || c) into ((a || b) || c), the CAML query parser applies no such transform — In a sense CAML is more like an XML serialized, abstract syntax tree than a query language intended for direct use:

    <Where>
      <Or>
         <Or>
            <Eq>
               <FieldRef Name='MyField' />
               <Value Type='LookupMulti'>tag1</Value>
            </Eq>
            <Eq>
               <FieldRef Name='MyField' />
               <Value Type='LookupMulti'>tag2</Value>
            </Eq>
         </Or>
         <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag3</Value>
         </Eq>
      </Or>
   </Where>

So how do we go about composing a query that OR together n tags (not to mention a mix of ORs and ANDs)? One approach might be using the U2U CAML Query Builder. The tool delivers a UI for formulating queries, but also exposes its CAML query builder API as a .Net assembly. Unfortunately, documentation on how to use the builder is sparse. Nonetheless, I went for the case of composing a query that OR together three tags:

   string ComposeCamlQuery() {
      Builder b = new Builder(CamlTypes.GetListItems);
      b.AddViewField("Title");
      b.AddViewField("MyField");

      bool addCombinerNode;
      foreach (string tag in new string[] { "tag1", "tag2", "tag3" })
         b.AddWhereField("MyField", tag, "LookupMulti",
                         "Or", out addCombinerNode);
      return b.CamlDocument.InnerXml;
   }

The outcome is a rather strange looking CAML query with extra <And></And> and missing <Eq></Eq> tags. Funny thing is that this type of the query can be correctly composed through the UI, so most likely I’m not using the API correctly:

   <Where>
      <And>
         <And>
            <Or>
               <FieldRef Name="MyField" />
               <Value Type="LookupMulti">tag1</Value>
            </Or>
            <Or>
               <FieldRef Name="MyField" />
               <Value Type="LookupMulti">tag2</Value>
            </Or>
         </And>
         <Or>
            <FieldRef Name="MyField" />
            <Value Type="LookupMulti">tag3</Value>
         </Or>
      </And>
   </Where>

I eventually abandoned the idea of using the CAML Query Builder API. Instead, a colleague pointed me to Waldek Mastykarz’s post on generating dynamic CAML queries. As long as all tags are to be either AND’ed or OR’ed together, Mastykarz provides an elegant solution, which I modified to be recursive, more generic, and read like an induction proof (at a negligible performance cost):

   ComposeCamlQuery(new[] { "tag1", "tag2", "tag3" },
                    "Or",
                    "<Where>{0}</Where>",
                    @"<Eq>
                       <FieldRef Name='MyField' />
                       <Value Type='LookupMulti'>{0}</Value>
                      </Eq>");

   string ComposeCamlQuery(IList<string> ops, string relOp,
                           string query, string leaf) {
      return ops.Count == 1
         ? string.Format(query,
                         string.Format(leaf, ops[0]))
         : ComposeCamlQuery(ops.Skip(1).ToList(),
                            relOp,
                            string.Format(query,
                               string.Format("<{0}>{1}{{0}}</{0}>",
                               relOp,
                               string.Format(leaf, ops[0]))),
                            leaf);
   }

Obviously, more complex code could be written for a mix of AND’s and OR’s. Before doing so, however, you probably want to evaluate other approaches for filtering list items:

1. Formulate a simpler, too general query and post-process the result
2. Formulate smaller queries and optionally merge the results and do post-processing
3. Use the SharePoint Search API

I also considered such alternatives as Linq to SharePoint or Caml.net, but although the first holds a lot of promise, it isn’t ready for prime time. As for the latter, its focus is more on composing type-safe queries than dynamic ones, so it’s more of a supplement to the first two alternatives.

As for SharePoint Search, the downside is that search has to be configured and that the list must have been indexed for the outcome to be accurate. Depending on the frequency with which the list is modified, the time between incremental crawls, and the context in which the results are displayed, the search approach may or may not be the solution you’re looking for.

Update, July 19: Added recursive code solution.

This book obviously targets SharePoint administrators. But after finishing the book, I believe it’s a must read for SharePoint developers as well. As such, I’d argue that a developer should know what and how SharePoint is capable of, starting with the point and click approach of this book and then progressing towards code. And by capable I’m referring to all parts of SharePoint, including the Central Administration and the Shared Services Provider.

Knowing the capabilities of point and click comes in handy when discussing requirements. It often lays the groundwork for a prototype and helps you figure out where to extend SharePoint to fulfill the business needs. In addition, knowing what SharePoint can do on a general level makes estimates more reliable. Learning about administrative tasks also has the added benefit of establishing a common terminology between a developer and an administrator.

All in all, the SharePoint Administrator’s Companion does a solid job of providing an overview of the capabilities of the SharePoint platform, demonstrated mostly through point and click. I’d therefore advocate that any developer should know the content of this book by heart, and use it as a starting point before moving on to more code-centric books such as the Inside Microsoft Windows SharePoint Services 3.0 and the Inside Microsoft Office SharePoint Service 2007.

In preparation for the exam I found the Inside Microsoft Windows SharePoint Services 3.0 book and the Windows SharePoint Services 3.0 SDK to be the best study materials. In addition, I looked to others for tips and tricks.

I found the SDK to be supprisingly thorough, compared to the book, which doesn’t cover the entire curriculum. The book alone does get one enough points to pass the exam, though.

Overall the exam questions could benefit from a review for verbosity and clear wording. Also, one question was about the RegisterHiddenField method and the RequestFromAdminPort property, which MSDN marks as obsolete. Another question was about adding a column to a content type and pushing the change to a list based on the content type. On that question I could’ve sworn that two of the possible answers were correct.

Nonetheless, learning SharePoint with a specific goal and deadline in mind serves as a good motivator.

Unfortunately, using the web interface and SharePoint Designer doesn’t scale well. With the drag and drop tools there is just no real support for deploying your portal to another server, and hence to unify the individual contributions of team members.

Therefore, the current way to go for any real SharePoint development is to work with the 12 Hive directly. Named after Office 12 (Office 2007), the 12 Hive is a hierarchy of xml files residing in “Program Files\Common Files\Microsoft Shared\web server extensions\12″ that controls almost everything about how SharePoint and your portal behaves.

Using Visual Studio (VS), you’d create a 12 Hive parallel file hierarchy of new folders and files and work within VS as usual. Like with any other project, you’d use version control, continuous integration, and so on, on xml and .Net code. With VS, deployment (to your development box) would then involve checking out xml and code from source control and compiling the code.

Then, using a build script, you’d xcopy the xml and binary into the 12 Hive and GAC, respectively, and possibly run various stsadm commands to activate your features. Alternatively, you’d create a SharePoint solution package that pretty much does the same thing when invoked by SharePoint.