Bugfree.dk – Ronnie Holm's blog

One aspect of many functional languages that’s always fascinated me is the ease with which you can express a recursive structure, such as a language, and have it evaluated. Suppose you want to define the grammar of a DSL of propositions, in F# you could do so using the discriminated union type:

type Proposition =
    | True
    | And of Proposition * Proposition
    | Or of Proposition * Proposition
    | Not of Proposition

In an object-oriented language like C#, this definition would be analogous to a hierarchy of classes. Proposition would be the base class of True, And, Or, and Not. Each would then have a constructor that accepts some number of arguments, e.g., the And constructor would accept a two-tuple, two propositions, making the Proposition type recursive. In C# you might even model the recursive relationship using the Composite pattern.

In F#, an instance of the Proposition type may be created using a constructor syntax that resembles that of a function call:

let p = And(Or(True, Not(True)), Not(Not(True)))

Now, because of the recursive nature of the Proposition type, instances of it form a parse tree-like structure:

                        And
                        /\
                       /  \
                      Or  Not
                      /\    \
                     /  \    \
                  True  Not  Not
                          \    \
                           \    \
                          True  True

Having a parse tree is not very useful in itself. What you need is some way to execute or evaluate it. Given the recursive nature of the grammar, it makes sense for the evaluator to be recursively defined as well. The evaluator would do pattern matching on the node type, deconstructing it into the parts needed to evaluate it. Since our language of propositions maps directly to the boolean operations of F#, it’s very simple to evaluate the parse tree:

let rec eval (p: Proposition) =
    match p with
    | True -> true
    | And(p1, p2) -> eval p1 && eval p2
    | Or (p1, p2) -> eval p1 || eval p2
    | Not(p1) -> not (eval p1)

let e = eval p // e : bool = true

The simple language of propositions doesn’t take into account operator precedence and associativity, variables, and so on. But the features of F# used here allow for more complicated languages to be expressed and evaluated directly. With C#, you’d often fallback to describing the grammar of more complex languages in Backus-Naur form and feeding it to a parser generator that would map the node types of the parse tree to classes and be able to form a valid parse tree of some input.

Most enterprise projects have one or more console applications for utility tasks such as cleaning up or importing data into the database. These utilities tend to be project-specific and small in terms of code size, and instead of several smaller assemblies, it makes sense to combine these into a single assembly. The generic runner would read the utility, called the command, and arguments from the command-line and use the command pattern to create and execute it.

For the generic runner to work, each command has to fulfill the contract.

public enum ExitCode {
    Success = 0,
    Failure
};

public interface ICommand {
    string Usage { get; }
    string Description { get; }
    ExitCode Execute(string[] args);
}

I want the runner to adhere to the open/closed principle. For its behavior to be modified without altering its core delegation logic. This requires the use of reflection to retrieve and instantiate a command based on command-line arguments.

class Program {
    static IEnumerable<ICommand> GetCommands() {
        var iCommand = typeof (ICommand);
        return System.Reflection.Assembly.GetExecutingAssembly().GetTypes().ToList()
            .Where(t => iCommand.IsAssignableFrom(t) && t != iCommand)
            .Select(t => Activator.CreateInstance(t) as ICommand);
    }

    static void DisplayHelp() {
        Console.WriteLine("Console [Command] [Arg1] [Arg2] [ArgN]\n\n");
        GetCommands().ToList().ForEach(command =>
            Console.WriteLine(command.Usage + "\n" + command.Description + "\n\n"));
    }

    static int Main(string[] args) {
        if (args.Length == 0) {
            DisplayHelp();
            return (int)ExitCode.Failure;
        }

        var commandName = args[0];
        var command = GetCommands().Where(t => t.GetType().Name == commandName).SingleOrDefault();
        if (command == null)
            throw new ArgumentException(string.Format("Command '{0}' not found", commandName));

        var executeArguments = new List<string>(args);
        executeArguments.RemoveAt(0);

        var exitCode = command.Execute(executeArguments.ToArray());
        return (int)exitCode;
    }
}

A trivial example of a command that adds two numbers would be the following:

// $> GenericRunner.exe Calculator 2 3 => 2 + 3 = 5
public class Calculator : ICommand {
    public string Usage {
        get { return "Calculator [Op1] [Op2]"; }
    }

    public string Description {
        get { return "World's simplest calculator"; }
    }

    public ExitCode Execute(string[] args) {
        try {
            Console.WriteLine(
                string.Format(
                    "{0} + {1} = {2}",
                    args[0], args[1], int.Parse(args[0]) + int.Parse(args[1])));
            return ExitCode.Success;
        } catch (Exception e) {
            Console.WriteLine(e.ToString());
            return ExitCode.Failure;
        }
    }
}

Now multiple smaller assemblies can be grouped into one, with a description of all commands automatically being assembled, and without commands interfering (too much) with each other.

It’s been a while since I last took a serious look at F#. Back then I did a simple random fractal terrain generator which, even though the algorithm is simple, I found challenging to do. Nevertheless, functional programming is just one of those areas that I keep returning to. This time around I want to use the event receiver for versioning attachments in SharePoint lists to get familiar with object-oriented F#. Of course, translating a C# class to an F# class, the result will look like C# with different syntax and better type inference. The point here is to use classes in F# as a way to expose functionality to other .NET languages. In a real-world F# application the core logic would likely not be object-oriented.

namespace Dk.Bugfree

open System
open System.Globalization
open Microsoft.SharePoint

type public ListAttachmentVersioningEventReceiver() =
    inherit SPItemEventReceiver()

    member private r.CustomVersion = "CustomVersion"
    member private r.ShadowLibrary = "ShadowLibrary"

    // override ItemAdded : properties:SPItemEventProperties -> unit
    override r.ItemAdded properties =
        base.ItemAdded properties
        r.SetCustomVersionLabel properties.ListItem
        r.CreateSnapshot properties

    // override ItemUpdated : properties:SPItemEventProperties -> unit
    override r.ItemUpdated properties =
        base.ItemUpdated properties
        let item = properties.ListItem

        if r.RollbackHappened item then
            r.RestoreSnapshot properties
            r.SetCustomVersionLabel item
            r.CreateSnapshot properties
        else
            r.CreateSnapshot properties
            r.SetCustomVersionLabel item

    // member private CreateSnapshot : properties:SPItemEventProperties -> unit
    member private r.CreateSnapshot properties =
        use site = properties.OpenWeb()
        let item = properties.ListItem
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let path = String.Format("Versions/{0}/{1}", item.ID, r.GetOfficialVersionLabel(item))
        let shadowFolder = r.CreateFolderPath shadowLibrary path

        item.Attachments |> Seq.cast |> Seq.iter (fun fileName ->
            let existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName)
            let newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream())
            newFile.Item.Update())

    // member private RollbackHappened : item:SPListItem -> bool
    member private r.RollbackHappened item =
        let culture = CultureInfo.InvariantCulture
        let currentVersion = Single.Parse(r.GetOfficialVersionLabel(item), culture)
        let lastVersion = Single.Parse(r.GetCustomVersionLabel(item), culture)
        currentVersion > lastVersion + 1.0f

    // member private RestoreSnapshot : properties:SPItemEventProperties -> unit
    member private r.RestoreSnapshot properties =
        let item = properties.ListItem
        let restoreVersion = r.GetCustomVersionLabel item
        r.EventFiringEnabled <- false    

        item.Attachments |> Seq.cast |> Seq.map (fun fileName -> unbox<string> fileName) |> Seq.toList
                         |> Seq.iter (fun fileName -> item.Attachments.Delete(fileName))

        use site = properties.OpenWeb()
        let path = String.Format("Versions/{0}/{1}", item.ID, restoreVersion)
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let source = r.CreateFolderPath shadowLibrary path

        source.Files |> Seq.cast |> Seq.iter (fun file ->
            let unboxedFile = unbox<SPFile> file
            item.Attachments.Add(unboxedFile.Name, unboxedFile.OpenBinary()))

        item.SystemUpdate false
        r.EventFiringEnabled <- true

    // member private CreateFolderPath : list:SPDocumentLibrary -> path:string -> SPFolder
    member private r.CreateFolderPath list path : SPFolder =
        r.CreateFolderPathRecursive list.RootFolder (path.Split [|'/'|] |> Array.toList)

    // member private CreateFolderPathRecursive : folder:SPFolder -> pathComponents:string list -> SPFolder
    member private r.CreateFolderPathRecursive folder pathComponents =
        match pathComponents with
        | [] -> folder
        | head :: tail ->
            try
                let existingFolder = folder.SubFolders.[head]
                r.CreateFolderPathRecursive existingFolder tail
            with
                :? ArgumentException ->
                    let newFolder = folder.SubFolders.Add head
                    r.CreateFolderPathRecursive newFolder tail

    // member private SetCustomVersionLabel : item:SPListItem -> unit
    member private r.SetCustomVersionLabel item =
        r.EventFiringEnabled <- false
        item.[r.CustomVersion] <- r.GetOfficialVersionLabel item
        item.SystemUpdate false
        r.EventFiringEnabled <- true  

    // member private GetCustomVersionLabel : item:SPListItem -> string
    member private r.GetCustomVersionLabel item =
        item.[r.CustomVersion] :?> string

    // member private GetOfficialVersionLabel : item:SPListItem -> string
    member private r.GetOfficialVersionLabel item =
        item.Versions.[0].VersionLabel

A couple of things to note about the F# implementation: first, it hardly specifies any types. They’re inferred by the compiler. Where type names do appear, it’s mainly because they’re required to unbox elements of an IEnumerable collection. Secondly, F# has flexible self identifiers. Methods must explicitly specify the name of the this reference in C# and use it when accessing members. Thirdly, arguments to general .NET methods are passed as a tuple value, i.e., as comma-delimited arguments surrounded by parenthesis.

In versioning attachments in a SharePoint list using snapshotting, an event receiver was responsible for the heavy lifting. To enable versioning of a list, I could therefore have associated the receiver with a list by adding the usual registration XML to a feature. But versioning is a truly reusable building block that shouldn’t be restricted to lists that are known when the feature is created. A better solution would be to extend the SharePoint list settings page for all lists on a site on which the versioning feature is enabled. The user may then activate or deactivate attachment versioning on the fly.

This would involve adding or removing event receivers from a list as the user enables or disables versioning. The following extension method is one way to accomplish the addition-part in a type-safe manner:

// definition
public static class SPListExtensions {
    public static void RegisterEventReceiver<TReceiver>(this SPList list,
            SPEventReceiverType receiverType,
            int sequenceNumber) where TReceiver : SPItemEventReceiver {
        var assemblyName = typeof(TReceiver).Assembly.FullName;
        var className = typeof(TReceiver).FullName;

        (from SPEventReceiverDefinition definition in list.EventReceivers
         where definition.Assembly == assemblyName &&
               definition.Class == className &&
               definition.Type == receiverType
         select list.EventReceivers[definition.Id])
        .ToList()
        .ForEach(receiverToDelete => receiverToDelete.Delete());

        var receiver = list.EventReceivers.Add();
        receiver.Type = receiverType;
        receiver.Assembly = assemblyName;
        receiver.Class = className;
        receiver.SequenceNumber = sequenceNumber;
        receiver.Update();
        list.Update();
    }
}

// use
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemAdded, 10000);
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemUpdated, 10001);

Under rare circumstances the (assembly, class, type) tuple may not be unique, i.e., the same receiver may be registered multiple times, albeit with different sequence numbers. In practice I never found any use for this functionality, though, which is why I didn’t include the sequence number in the where clause above, causing all registrations matching the tuple to be removed.

A quick word on organizing extension methods: I usually collect them in an Extensions folder, appending Extensions to the name of class being extended and keeping with the one class per file convention. For brevity I’ve left out the using and the namespace part below.

SPListCollection extensions

In SharePoint 2010 the TryGetList method has been added to the SPListCollection class. The method returns either an SPList instance matching the display name or null. Oftentimes, however, you want to do a lookup based on the internal name. Here’s an extension method that adheres to the semantics of TryGetList, but using the internal name. It relies on the fact that the RootFolder property of a list is actually its internal name:

// definition
public static class SPListCollectionExtensions {
    public static SPList TryGetListByInternalName(this SPListCollection lists, string internalName) {
        return (from SPList l in lists
            where l.RootFolder.Name == internalName
            select l).SingleOrDefault();
    }
}

// use
if (site.Lists.TryGetListByInternalName(internalListName) == null)
   // list not found

SPFieldCollection extensions

Using the CreateNewField method of the SPFieldCollection you can add new fields to a list. The particular annoying aspect of this method, however, is that when you want to continue working with its result, oftentimes you have to cast it to one of the SPField subclasses. But since the SPFieldType, provided as one of the arguments to CreateNewField, closely relates to the actual SPField return type, an extension method is able to do the casting. This’ll expose mismatches at compile time instead of at runtime.

All it takes is for us to map out the relation between SPField and SPFieldType:

// definition
public static class SPFieldCollectionExtensions {
    public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
            string internalName, string displayName) where TSPField : SPField {
        var spFieldToFieldType = new Dictionary<Type, SPFieldType> {
            { typeof(SPFieldDateTime), SPFieldType.DateTime },
            { typeof(SPFieldNumber), SPFieldType.Number },
            { typeof(SPFieldUser), SPFieldType.User },
            { typeof(SPFieldBoolean), SPFieldType.Boolean },
            { typeof(SPFieldMultiLineText), SPFieldType.Note },
            { typeof(SPFieldText), SPFieldType.Text }
        };

        var fieldType = spFieldToFieldType[typeof(TSPField)];
        var list = fields.List;
        var field = list.Fields[list.Fields.Add(internalName, fieldType, false)];
        field.Title = displayName;
        field.Update();
        return field as TSPField;
    }
}

// use
l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName");

Taking the CreateField extension method one step further, oftentimes you want to set properties besides internal name and display name. For that purpose I’ve defined a CreateField method that accepts an Action<TField>. This allows you to reuse common property settings across fields for brevity and consistency while at the same time maintaining strong typing.

// definition
public static TSPField CreateField<TSPField>(this SPFieldCollection fields,
        string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPField {
    var newField = CreateField<TSPField>(fields, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
public static Action<SPFieldMultiLineText> RichTextProperties = f => {
    f.RichText = true;
    f.RichTextMode = SPRichTextMode.FullHtml;
};

l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName", f => f.Required = true);
l.Fields.CreateField(internalName, "displayName", RichTextProperties);

With the Comment field, you can leave out the type argument because the compiler infers it based on the type of the Action delegate.

Similar to CreateField, I’ve defined two additional extension methods for creating lookup fields:

// definition
public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName,
        string displayName) where TSPField : SPFieldLookup {
    var currentList = fields.List;
    var lookupList = currentList.ParentWeb.Lists.TryGetListByInternalName(lookupListName);
    var newField = currentList.Fields[currentList.Fields.AddLookup(internalName, lookupList.ID, false)];
    newField.Title = displayName;
    newField.Update();
    return newField as TSPField;
}

public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields,
        string lookupListName, string internalName, string displayName,
        Action<TSPField> setAdditionalProperties) where TSPField : SPFieldLookup {
    var newField = CreateLookup<TSPField>(fields, lookupListName, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
l.Fields.CreateLookup<SPFieldLookup>(lookupListName, internalName, displayName, f => f.AllowMultipleValues = true);

These extension methods makes using the SharePoint API more type-safe and concise, and defining lists using these methods and the template approach saves me from writing a lot of repetitive code.

While cleaning up some PowerShell installations scripts that I wrote some time ago, I come across the following piece of code (modified slightly for anonymity) for creating a hierarchy of SharePoint sites and setting the locale for a subset of the sites. It works but it sure doesn’t adhere to the DRY principle.

New-SPWeb -url $base_url -addtoquicklaunch -template "STS#1" -name "Base"
New-SPWeb -url $base_url/Dk -addtoquicklaunch -template "STS#1" -name "Dk"
New-SPWeb -url $base_url/Uk -addtoquicklaunch -template "STS#1" -name "Uk"
New-SPWeb -url $base_url/Dk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Dk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"
New-SPWeb -url $base_url/Uk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Uk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"

$dk = Get-SPWeb $base_url/Dk
$dkBusinessUnit1 = Get-SPWeb $base_url/Dk/BusinessUnit1
$dkBusinessUnit2 = Get-SPWeb $base_url/Dk/BusinessUnit2

$locale = [System.Globalization.CultureInfo]::CreateSpecificCulture("da-DK")
$dk.Locale = $locale
$dk.Update()

$dkBusinessUnit1.Locale = $locale
$dkBusinessUnit1.Update()

$dkBusinessUnit2.Locale = $locale
$dkBusinessUnit2.Update()

How can we reduce this repetition? Well, it repeats because we’ve made the common mistake of mixing logic with data. One way to separate the two is using a table-driven approach, i.e., extract the varying parts, the data, into a table and rewrite the logic so it’s parameterized by the table. After some trial and error with PowerShell here’s the code I ended up with:

$sites = @( @("", "Base"),
            @("Dk", "Dk", "da-DK"),
            @("Uk", "Uk"),
            @("Dk/BusinessUnit1", "BusinessUnit1", "da-DK"),
            @("Dk/BusinessUnit2", "BusinessUnit2", "da-DK"),
            @("Uk/BusinessUnit1", "BusinessUnit1"),
            @("Uk/BusinessUnit2", "BusinessUnit2") )

$sites | foreach {
  $url, $title, $culture = $_
  $newSite = New-SPWeb -url "$base_url/$url" -addtoquicklaunch -template "STS#1" -name $title

  if ($culture -ne "") {
    $locale = [System.Globalization.CultureInfo]::CreateSpecificCulture($culture)
    $newSite.Locale = $locale
    $newSite.Update()
  }
}

So far so good. The un-installation part of the script, however, follows the same pattern of repetition:

Remove-SPWeb -identity $base_url/Uk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url -confirm:$false

But now that we have the table of sites at hand, we can easily make the site removal table-driven as well:

function Reverse($array) {
  $clone = $array.Clone()
  [Array]::Reverse($clone)
  $clone
}

Reverse($sites) | foreach {
  $url = $_[0]
  Remove-SPWeb -identity "$base_url/$url" -confirm:$false -ErrorAction SilentlyContinue
}

I’m sure the code could be written more elegantly, but given my working knowledge of PowerShell I’m satisfied with the result. I especially like the functional programming style.

My notes from a talk on Windsor by Mogens Heller Grabe which I attended this week. Slides and code are available through Dropbox, but I don’t know for how long.

• Use an Inversion of Control container (IoC) like Windsor to create an architecture that responds well to change, i.e., an architecture that promotes looser coupling and higher cohesion
• The ability to write unit tests against a code base is a good measure of its degree of coupling
• By having components depend on interfaces, you’re free to switch the implementation at runtime, introducing flexibility into the software
• Avoid having concrete components talk to each other. It makes it hard for the two to vary independently, e.g., adding logging later can only be done by tearing components apart and putting them back together
• At runtime the IoC container recursively composes these smaller components into an object graph, e.g., by passing concrete implementations through the constructors to satisfy the dependencies
• One alternative to using an IoC container is using a service locator. Each component’s constructor would then ask the locator for its dependent components. While this approach works, it makes testing components in isolation difficult because all dependencies are now locked away inside the service locator. Instead, by having all dependencies supplied in the constructor, unit tests can directly supply fakes implementations
• When software is only used in one environment it tends to be fairly inflexible because it’s only suited for that one purpose. As soon as you design it to be used in two or more environments it becomes more flexible. Thing of different environments as unit test, staging, production, and so on
• The idea is for components to not have to be modified depending on the where they’re running. It’s the IoC container that dynamically tie components together based on the environment
• Suppose you didn’t use a IoC container. Then ultimately the top-level object would have to pass concrete instances down the chain. The top-level object may well be the UI layer. But having the UI layer know about data access components, service components, and logging components isn’t ideal. Instead use an IoC container, which is nothing more than a factory for components
• With the Windsor IoC container (and most others), the usage pattern typically involves three stages
  • Register: tie together interfaces with concrete implementations
  • Resolve: return concrete implementations of interfaces required to satisfy the dependencies of an object
  • Release: dispose of concrete implementations
• Most people only use a fraction of the functionality of an IoC container. Instead of taking dependency of a full-blown container, you could easily roll your own IoC container that implements the core functionality in a few dozen lines of code — in an associated .NET Rocks and DnrTV episode James Kovacs elaborates on his original article
• The goal of using a container isn’t to get rid of calls to the new operator. It’s to not use the new operator for the parts of an applications where flexibility is required
• Simple forms of Aspect Oriented Programming are possible using interceptors
• Windsor supports the decorator pattern so you can have one component wrap another at runtime. This lets you implement features such as logging without actually modifying the original component. It’s an example of adhering to the open/close principle by which classes should be open for extension, but closed for modification. In other words: don’t edit the original source code to introduce new behavior. Instead use methods of composition
• Avoid configuring Windsor through XML and instead use the fluent interface — perhaps in a separate assembly so only it needs to be redeployed when the configuration changes. When a value, such as a connection string needs to be configurable, create a section in app.config and add the value there. Windsor will know how to locate the value when instantiating objects

My notes from a talk on NServiceBus by Mogens Heller Grabe which I attended this week. Slides and code are available through Dropbox, but I don’t know for how long.

• Enterprise service bus
  • An architectural pattern rather than a specific technology or product
  • The ethernet of SOA
• Types of service bus
  • Centralized: a broker more than a bus, e.g., BizTalk
  • Decentralized: a real service bus, e.g., NServiceBus, Mass Transit, Rhino Service Bus, MS .NET Service bus
• The current version 2.0 of NServiceBus is free whereas the binary version 2.5 will contain limitations. The limitations can be disabled by downloading and compiling the source code yourself
• NServiceBus is designed to address the fallacies of distributed computing
  • It primarily addresses the fallacies through the use of reliable one-way messaging
  • Reliable means messages are stored in a queue, such a MSMQ, which is part of all recent Windows installations. The assumption being that the queue is always present and ready to receive messages
  • One-way means messages are asynchronous and fire-and-forget. The destination service doesn’t have to be online when a message goes into its queue, which makes services temporally independent. At some point in the future the service may post a reply to the client’s queue
  • Messaging means that any action is accomplished by posting a self-contained message, containing the type of operation and its parameters, to the queue of a service
  • With NServiceBus, a class is marked with an interface to describe that it’s a type of message. Instances of the class are then serialized using a standard XML or binary serializer and stored in a queue
• NServiceBus service != traditional web service
  • An NServiceBus service is a class that implements an interface
  • Using a message queue is more reliable when the service has to make calls to other services as part of its operation. With a traditional web service, calling other services from within makes the service slower and less reliable. If only one dependent service is down it’s like with old Christmas lights wired in series: the entire stack may be down. Queues on the other hand introduce a save place, a stabilization point, to temporarily store messages
  • A service is typically configured with an input queue that identifies it to others and an error queue that messages are routed to after some number of retries. In addition, each service may be configured with a number of threads to use to parallelize message processing

My notes from a talk on Windbg by Brian Rasmussen which I attended this week. The talk was recorded and parts one and two are available though Channel 9.

• Windbg isn’t a replacement for VS, but VS doesn’t handle some advanced cases
• Windbg is a free user mode/kernel mode debugger which is part of the Debugger Tools for Windows
• Customers may not be happy installing VS to debug code in production since it installs a lot of components and requires restarts
  • Windbg requires only a simple installation once you extract the redistributable from the Debugger Tools for Windows
• Loading SOS.dll from the Microsoft .NET folder into Windbg makes it understand .NET
  • With SOS.dll loaded, you can look into the CLR and its data structures
  • Make sure to load the right SOS.dll for your runtime
  • SOS.dll is also available for the Silverlight runtime
• Debugger Markup Language support is available for version 4 of SOS.dll
  • Provides hyperlinks in the command output of SOS
• Like with the VS debugger, you can insert Debug.Break() in your demo app and have Windbg halt on it
  • Release builds can be debugged with Windbg
  • Release builds also contain symbols. Like with debug builds, symbols are stored in a separate file
  • Release builds make debugging harder because the jitter kicks in and modifies the code
  • The 64 bit calling convention of passing arguments via registers makes it harder to locate information when debugging
• Windbg generally needs symbols loaded, although it’s less important when debugging managed code
  • Use MS’ public symbol server to load symbols on demand
  • Set the _NT_SYMBOL_PATH environment variable to point to your symbols (will affect VS as well)
  • Or use the .symfix command from within Windbg
• Popular extensions to Windbg: SOSEX and Psscor2 (replacement for SOS, useful for ASP.NET debugging)
• Create dump file to analyze: use task manager or ADPlus or ProcDump from Sysinternals, which can dump based on triggers
• ADPlus collects crash dumps or hang dumps
  • Hang dumps can be captured from the same process multiple times and may be useful when debugging deadlocks or resource leaks
  • When you capture a hang dump, the process is halted for the dump period and is then restarted
• A *32 process in task manager is actually a 64 bit process when you dump it
  • WoW64.dll is involved when dumping from the task manager
  • Not what you typically want because you don’t get full access to 32 bit process information
• A .NET application is hosted within the CLR which is itself hosted within a regular Windows process
  • Looking at memory usage with the task manager therefore doesn’t tell you much about the .NET part

Developer automation is a subject that I’ve always felt passionate about. Unit testing may be the most common example, but other tasks may include check-out of source code, build, deploy, setup, and warm-up of an application. I may even want one system rule them all and have the same automation drive continuous integration. Whatever the use, to fully reap the benefits of automation, a developer should master at least one automation tool the same way he masters other developer tools. This and later posts capture my experiences with a few such automation tools centered around Windows and .NET, starting with why the ubiquitous batch file is best avoided and how to characterize better solutions in terms of it.

Although Visual Studio, in tandem with the MSBuild engine, generally takes good care of compilation, I rarely want to rely on it solely. Instead, I’d prefer that any developer task be easily run from the command-line. This ensures that no magic is going on, that the task is kept simple and flexibility, and that it doesn’t rely on Visual Studio to work. The challenge, however, is which of the many tools available to pick. It must be one that’s flexible enough to meet most requirements with relative ease or the automation will likely not be a valid documentation medium in itself.

Why not to use Windows batch files

As a general example, consider the deployment of a SharePoint 2007 solution. With SharePoint a good deal more than compiling is needed to bring new functionality into a SharePoint installation. Whereas Visual Studio and MSBuild may still compile the code and WSPBuilder create the WSP installation package, both from within Visual Studio and from the command-line, getting everything setup cries for automation, even locally. A common approach (possibly inspired by popular SharePoint 2007 literature) is that of the batch file applying various operations to a WSP. For the sake of brevity, I’ve kept the example short, but imagine extending it with a check-out source code task, a compilation task, a WSPBuilder task, and a feature deactivation and activation task, and possibly a warm-up task. Add to this a master script that orchestrate the whole thing. In the end, I’d end up with scripts that become increasingly harder to maintain as they grow in number and size.

    @set STSADM="c:\program files\common files\microsoft shared\web server extensions\12\bin\stsadm"

    if "%1" == "uninstall" goto uninstall
    if "%1" == "install" goto install
    if "%1" == "" goto reinstall

    :uninstall
        %STSADM% -o retractsolution -name Foo.wsp -immediate
        %STSADM% -o execadmsvcjobs
        %STSADM% -o deletesolution -name Foo.wsp -override
        goto end

    :install
        %STSADM% -o addsolution -filename Foo.wsp
        %STSADM% -o deploysolution -name Foo.wsp -immediate -allowGacDeployment -force
        %STSADM% -o execadmsvcjobs
        goto end

    :reinstall
        call Foo uninstall
        call Foo install
        goto end

    :end

Still, because of the ubiquitous nature of command.com and cmd.exe, the batch file interpreters, batch files are everywhere. Regardless that the technology is a left-over from the days of MS DOS and haven’t evolved much since. Not only are the branching and looping constructs limited, so are the available commands. Suppose I want to find out if a command was indeed successful. This turns out to be really hard when all I have to work with is the errorlevel of the most recently executed command. Assuming the command adheres to the errorlevel convention, for the script to fail as early and as close to the real error as possible, I’d have to inspect the property after each command, causing the batch file to grow quite verbose. Sadly, batch files lack the equivalent of set -o errexit of Bash, where the interpreter checks for success after each command and aborts immediately on error. Relying solely on the errorlevel is oftentimes insufficient anyway. To determine success, I’d typically have to parse actual command output or inspect some system property by downloading additional commands or building my own.

Essential vs. incidental requirements

Unless I plan to sit idle and stare at the screen and be a human error detector while the batch file run, I think it’s safe to say that it’s mostly unsuitable for developer automation. Hence in late 2005 I rolled my own automation tool in Python. With Python or Ruby or a similar dynamic language acting as the glue that ties everything together, almost any task can be automated in a robust way. Of course, I could also automate with a static language like C# (it’s surprisingly common). But for script-like tasks, I don’t particularly fancy the long cycle of editing source code in Visual Studio, compiling it, deploying it, and having a hard time debugging it in environment without Visual Studio. A dynamic language, on the other hand, short-circuits the development cycle and allows for interactive programming through a REPL.

With a dynamic language that interacts with .NET, such as IronPython, IronRuby, or Powershell, possibly with supporting DSLs like Paver, Rake, or psake on top, the need for writing custom commands to interact with the operating system or the application almost vanishes. The question, then, is which of the dynamic languages to go with when at their technical core they’re so much alike. Besides sharing the concept of a REPL, the notion of a tuple, a list, a map, and operations on each are baked into their syntax, making code quite terse. It even makes it convenient to express any configuration in the language itself and not in XML where I’d first have to come up with a schema, and then create an instance of it before parsing it. On Windows, however, Powershell is gradually becoming the next ubiquitous interpreter, with applications shipping with cmdlets, whereas IronPython or IronRuby is a separate download.

Conclusion

No matter the tool, what I end up doing is defining tasks, form dependencies between tasks, and have the tool execute tasks in an order that satisfies their dependencies. As the tool traverses the dependency graph and executes tasks, it’s up to each task to detect success, and up to the tool to report on progress. A good tool is therefore characterized by the ease with which I can express these things, the available language constructs, the ease of debugging, and the tool’s ability to converse in foreign languages.