Part 4 – Strongly typed SharePoint list operations using the repository pattern in F#
Part 3 – Bringing together repository and tree structure in business layer
Part 2 – Creating and working with a custom tree data structure in C#
Part 1 – Strongly typed SharePoint list operations using the repository pattern

As an experiment, I wanted to re-implement the SharePoint repository pattern from Part 1 in F#.

open System
open System.Xml.Linq
open Microsoft.SharePoint
open Microsoft.SharePoint.Taxonomy

type LegalDocument =
    { Title: string
      ApprovalStatus: string
      Locations: seq<string>
      Level: int
      ProcessHierarchy: Map<Guid, string> 
      DocumentId: string
      ParentDocumentId: string
      Url: Uri }

type LegalDocumentRepository() =
    let LinkFilename = "LinkFilename"
    let FileLeafRef = "FileLeafRef"
    let ModerationStatus = "_ModerationStatus"
    let Location = "Location_Legal"
    let ProcessHierarchyLevel = "ProcessHierarchyLevel"
    let ProcessHierarchyLegal = "ProcessHierarchyLegal"
    let DocumentId = "_dlc_DocId"
    let ParentDocumentId = "Parent_x0020_Document_x0020_Id"

    let xn s = XName.Get s

    member __.Query (list: SPList) (caml: XElement): SPListItemCollection =
        let query = 
            SPQuery(
                ViewFields = ([LinkFilename; 
                               ModerationStatus; 
                               Location; 
                               ProcessHierarchyLevel; 
                               ProcessHierarchyLegal; 
                               DocumentId; 
                               ParentDocumentId] |> List.fold (fun acc fld -> 
                                   acc + XElement(xn "FieldRef", XAttribute(xn "Name", xn fld)).ToString()) ""),
                Query = caml.ToString(),
                ViewAttributes =  @"Scope=""RecursiveAll""")
        list.GetItems query

    member this.GetApprovedDocuments (list: SPList) =
        let caml = 
            XElement(xn "Where",
                XElement(xn "Eq",
                    XElement(xn "FieldRef", XAttribute(xn "Name", ModerationStatus)),
                    XElement(xn "Value", XAttribute(xn "Type", xn "ModStat"), "Approved")))
        this.Query list caml |> this.Map

    member __.Map items =
        let levelMapper o =
            let s = o |> string
            match (s |> String.IsNullOrEmpty) with
            | true -> 0
            | false -> 
                let v = s.Split [|':'|]
                match v.Length with
                | 0 -> 0
                | _ -> v.[0] |> int

        let locationMapper (o: obj) = 
            o :?> TaxonomyFieldValueCollection |> Seq.map (fun f -> f.Label)

        let processHierarchyMapper (o: obj) =
            let t = o :?> TaxonomyFieldValueCollection
            t |> Seq.map (fun f -> Guid(f.TermGuid), f.Label)
              |> Map.ofSeq

        let parentDocumentIdMapper o =
            let s = o |> string
            match (String.IsNullOrEmpty s) with
            | true -> ""
            | false -> s.Trim()

        let urlPrefix = items.List.ParentWeb.Url + "/"
        items |> Seq.cast<SPListItem> 
              |> Seq.map (fun i ->
                    { Title = i.[LinkFilename] |> string
                      ApprovalStatus =  i.[ModerationStatus] |> string
                      Locations = locationMapper i.[Location]
                      Level = levelMapper i.[ProcessHierarchyLevel]
                      ProcessHierarchy = processHierarchyMapper i.[ProcessHierarchyLegal] 
                      DocumentId = i.[DocumentId] |> string 
                      ParentDocumentId = parentDocumentIdMapper i.[ParentDocumentId]
                      Url = Uri (urlPrefix + i.Url) })

[<EntryPoint>]
let main args =
    use siteCollection = new SPSite "http://intranet"
    use site = siteCollection.OpenWeb "Legal"
    let list = site.Lists.["LegalDocuments"]
    let repository = LegalDocumentRepository()
    let documents = repository.GetApprovedDocuments list

    documents |> Seq.iter (printfn "%A")
    0

In C#, the cast from string to LINQ to XML type is implicit. In F#, casts are generally explicit so we require the small xn helper function. Using a record type over a regular LegalDocument class has the benefit that all members must be explicitly set on creation or the compiler will issue an error. This comes in handy when modifying LegalDocument during the application life-cycle as the compiler will make sure we update all places where it’s used.

Part 4 – Strongly typed SharePoint list operations using the repository pattern in F#
Part 3 – Bringing together repository and tree structure in business layer
Part 2 – Creating and working with a custom tree data structure in C#
Part 1 – Strongly typed SharePoint list operations using the repository pattern

Now that we’ve created the data access layer in the form of a repository to query a SharePoint list and created a supporting tree data structure to re-organize the data returned from the repository, it’s time to combine the two in the business layer. Because we’re using the repository to abstract away storage details and the tree structure, as opposed to XML or tree view control APIs, to structure data, the business layer has become easier to reason about. In fact, the business layer can, or perhaps should, be developed as a console application or using a unit testing framework to ensure presentation independence and to work around the very long SharePoint deployment and feedback cycle.

To make the code easier to follow, let’s assume the LegalDocumentRepository.GetDocuments method brings back this set of legal documents from the SharePoint list:

new LegalDocument { Title = "node-1", DocumentId = "node-1", ParentDocumentId = "root", Level = 2 },
new LegalDocument { Title = "node-1.1", DocumentId = "node-1.1", ParentDocumentId = "node-1", Level = 3 },
new LegalDocument { Title = "node-1.1.1", DocumentId = "node-1.1.1", ParentDocumentId = "node-1.1", Level = 4 },
new LegalDocument { Title = "node-1.2", DocumentId = "node-1.2", ParentDocumentId = "node-1", Level = 3 },
new LegalDocument { Title = "node-2", DocumentId = "node-2", ParentDocumentId = "root", Level = 2 },

The goal is to transform this set of documents into a tree by tying documents together using the DocumentId/ParentDocumentId properties. In doing so, we assume the parent of a document must always be at a lower Level than the child. Thus, building the tree is done by building lower levels first.

Here’s the business layer class that calls out to the repository and transforms the tabular result into the tree structure:

public class DocumentBrowser {
    public Tree<LegalDocument> GetDocuments(SPList l) {
        var repository = new LegalDocumentRepository();
        var documents = repository.GetDocuments(l);

        // in the real world, the root LegalDocument instance would also be retrieved from the list.
        var root =
            new Tree<LegalDocument>(new LegalDocument {
                Title = "root",
                DocumentId = "root",
                ParentDocumentId = "",
                Level = 1
            });

        return BuildTree(documents, root);           
    }

    public Tree<T> BuildTree<T>(IEnumerable<T> documents, Tree<T> tree) where T : LegalDocument {
        documents
            .OrderBy(d => d.Level)
            .ToList()
            .ForEach(d => {
                var p = tree.Find(pd => pd.DocumentId == d.ParentDocumentId);
                if (p != null)
                    p.Children.Add(new Tree<T>(d));
            });

        return tree;
    }
}

A console application that exercises the business layer looks like so:

public static void Main() {
    using (var siteCollection = new SPSite("http://intranet")) {
        using (var site = siteCollection.OpenWeb("Legal")) {
            var list = site.Lists["LegalDocuments"];
            var documentBrowser = new DocumentBrowser();
            var tree = documentBrowser.GetDocuments(list);
            System.Console.WriteLine(tree);

            /*
             root
               node-1
                 node-1.1
                   node-1.1.1
                 node-1.2
               node-2
            */
        }
    }
}

Ultimately, the layering introduces more classes and code, but the code becomes more abstract, less cluttered, easier to read, and faster to debug.

Part 4 – Strongly typed SharePoint list operations using the repository pattern in F#
Part 3 – Bringing together repository and tree structure in business layer
Part 2 – Creating and working with a custom tree data structure in C#
Part 1 – Strongly typed SharePoint list operations using the repository pattern

I’d claim the tree data structure is underused in general application development. Sure, XML or tree view control APIs are generally tree-based underneath, but how often does an application use a tree structure in its business layer? Maybe it’s because trees are initially a little hard to grasp given their recursive nature or maybe it’s because we as developers are conditioned to approach problems from a more imperative angle.

It wasn’t until recently, when I had to transform a tabular data structure into a tree data structure, that I created my own tree in C#. Here’s the code I ended up with for constructing a tree, doing a depth-first search for a node, and printing the tree:

public class Tree<T> {
    public T Value;
    public IList<Tree<T>> Children;

    public Tree(T value) {
        Value = value;
        Children = new List<Tree<T>>();
    }

    public Tree<T> Find(Func<T, bool> predicate) {
        if (predicate(Value))
            return this;
           
        return Children
            .Select(c => c.Find(predicate))
            .FirstOrDefault(c => c != null);
    }

    public override string ToString() {
        return ToStringRecursive(this, 0);
    }

    private string ToStringRecursive(Tree<T> tree, int indentation) {
        var seed = (new String(' ', indentation)) + tree.Value + Environment.NewLine;
        return tree
            .Children
            .Aggregate(seed, (current, next) => current + 
                ToStringRecursive(next, indentation + 2));
    }
}

Constructing a tree of strings is done as follows:

var root = new Tree<string>("root") {
    Children = new List<Tree<string>> {
        new Tree<string>("node-1") {
            Children = new List<Tree<string>> {
                new Tree<string>("node-1.1") {
                    Children = new List<Tree<string>> {
                        new Tree<string>("node-1.1.1")  
                    }
                },
                new Tree<string>("node-1.2")
            }
        },
        new Tree<string>("node-2")
    }
};

Now you can query on the properties of the type of tree node:

System.Console.WriteLine(root);
/*
root
  node-1
    node-1.1
      node-1.1.1
    node-1.2
  node-2
*/

var node1 = root.Find(n => n == "node-1");
var node2 = node1.Find(n => n == "node-1.1.1");

System.Console.WriteLine(node1);
/*
node-1
  node-1.1
    node-1.1.1
  node-1.2
*/

System.Console.WriteLine(node2);
/*
node-1.1.1            
*/

Strictly speaking, the traversal logic doesn’t belong to the tree class itself, but in a separate visitor class. Maybe at a later point, you want to add different traversal algorithms or some other code needs to iterate the tree to transform it. The visitor algorithms could be reused, calling out to a delegate for the specific details of the traversal.

Part 4 – Strongly typed SharePoint list operations using the repository pattern in F#
Part 3 – Bringing together repository and tree structure in business layer
Part 2 – Creating and working with a custom tree data structure in C#
Part 1 – Strongly typed SharePoint list operations using the repository pattern

I first wrote about accessing SharePoint lists using the repository pattern about three years ago. Since then I’ve reused that approach, with a few improvements, on almost every SharePoint project I’ve worked on. The improvements include generating CAML queries using LINQ to XML, taking a more functional approach to mapping from loosely typed to strongly typed data, passing in more specific arguments, not tying the repository to a definition class, and removing assertions to focus more on validating user rather than developer input.

Depicted graphically, here’s what I want my repository to do, albeit in a less sophisticated way than the SharePoint guidance by the Patterns & Practices team:

As an example, suppose I have a SharePoint list containing the internal fields below. Accessing field values through the SPListItem.Item property yields results of type object, which may then be cast to the type of the input type column. For some fields, such as Location_Legal and Url, I not only want to project it onto a differently named field but also a different type as indicted by the Output type column. The type may be changed to one that is more precise and/or easier to work with, given the client’s requirements. As a final note, observe how the internal names are inconsistently named. The repository can help bring some order to the naming and type chaos:

Here’s an example of a simple repository with read operations only. It’s easily extended with create, update, and delete operations. Don’t pay too much attention to the concrete fields. They’re meant as  examples:

public class LegalDocument {
    public string Title;
    public string ApprovalStatus;
    public IList<string> Locations;
    public int Level;
    public Uri Url;
    public IDictionary<Guid, string> ProcessHierarchy;
    public string DocumentId;
    public string ParentDocumentId;
}

public class LegalDocumentRepository {
    const string LinkFilename = "LinkFilename";
    const string FileLeafRef = "FileLeafRef";
    const string ModerationStatus = "_ModerationStatus";
    const string Location = "Location_Legal";
    const string ProcessHierarchyLevel = "ProcessHierarchyLevel";
    const string ProcessHierarchyLegal = "ProcessHierarchyLegal";
    const string DocumentId = "_dlc_DocId";
    const string ParentDocumentId = "Parent_x0020_Document_x0020_Id";

    protected SPListItemCollection Query(SPList list, XElement caml) {
        var query = new SPQuery { 
            Query = caml.ToString(),
            ViewFields = new[] { 
                LinkFilename, 
                ModerationStatus, 
                Location,
                ProcessHierarchyLevel, 
                ProcessHierarchyLegal,
                DocumentId,
                ParentDocumentId }
                    .Aggregate("", (acc, fld) =>
                        acc + (new XElement("FieldRef", new XAttribute("Name", fld)).ToString())),
            ViewAttributes = @"Scope=""RecursiveAll"""
        };

        var result = list.GetItems(query);      
        return result;
    }

    public LegalDocument GetByFilename(SPList list, string filename) {
        var caml =
            new XElement("Where",
                new XElement("Eq",
                    new XElement("FieldRef", new XAttribute("Name", FileLeafRef)),
                    new XElement("Value", new XAttribute("Type", "Text"), filename)));
        var result = Query(list, caml);
        return Map(result).Single();
    }

    public IEnumerable<LegalDocument> GetApprovedDocuments(SPList list) {
        var caml =
            new XElement("Where",
                new XElement("Eq",
                    new XElement("FieldRef", new XAttribute("Name", ModerationStatus)),
                    new XElement("Value", new XAttribute("Type", "ModStat"), "Approved")));
        var result = Query(list, caml);
        return Map(result);
    }

    private IEnumerable<LegalDocument> Map(SPListItemCollection items) {
        Func<object, int> levelMapper = o => {
            var s = (string) o;
            if (string.IsNullOrEmpty(s))
                return 0;
            var v = s.Split(new[] { ':' });
            return v.Length == 0 ? 0 : int.Parse(v[0]);
        };

        Func<object, IList<string>> locationMapper = o => {
            var t = (TaxonomyFieldValueCollection) o;
            return t.Select(f => f.Label).ToList();
        };

        Func<object, IDictionary<Guid, string>> processHierarchyMapper = o => {
            var t = (TaxonomyFieldValueCollection) o;
            var kvp = new Dictionary<Guid, string>();
            t.ForEach(f => kvp.Add(new Guid(f.TermGuid), f.Label));
            return kvp;
        };

        Func<object, string> parentDocumentId = o => {
            var s = (string)o;
            return string.IsNullOrEmpty(s) ? "" : s.Trim();
        };

        var urlPrefix = items.List.ParentWeb.Url + "/";
        var documents = 
            from SPListItem i in items
            select new LegalDocument {
                Title = (string) i[LinkFilename], 
                ApprovalStatus = (string) i[ModerationStatus], 
                Locations = locationMapper(i[Location]), 
                Level = levelMapper(i[ProcessHierarchyLevel]), 
                ProcessHierarchy = processHierarchyMapper(i[ProcessHierarchyLegal]), 
                DocumentId = (string) i[DocumentId], 
                ParentDocumentId = parentDocumentId(i[ParentDocumentId]),
                Url = new Uri(urlPrefix + i.Url)
            };

        return documents;
    }
}

The repository code has a number of interesting points to be aware of:

• Only include the fields required by the client in the CAML query. Extra fields will have a negative impact on mapping performance and creates a high coupling between the repository and the SharePoint list. This coupling may prevent future updates to fields without updating and re-deploying the repository as well.
• Calling SPList.GetItems() only takes a few milliseconds. The expensive part is materializing the result collection. Replacing the call with one to SPList.GetDataTable() doesn’t change anything, except that now the repository’s Query method, rather than the Map method, is where the majority of the execution time is spent.
• Each of the repository access methods, such as GetApprovedDocuments, accepts a SPList instance. The calling code must take of getting hold of an appropriate SPList instance. In case a repository method require access to the list’s parent SPWeb or SPSite instance, they’re available through the SPList instance.
• The LINQ to XML code is more verbose than the equivalent string representation of a CAML query, but prevents certain types of syntax errors from creeping into the query. Complex structures, strings in this case, should generally be constructed by a builder that encapsulates the details of the structure.
• Whether to create many smaller repositories or a large one that combines all the operations on a list is a matter of taste and general application architecture. Maybe the architecture consists of multiple bounded context in which case multiple repositories would make sense.
• The mapping within the Map method is lazy. Only when the client enumerates the collection, or calls ToList() on it to force immediate query evaluation, does the mapping of each item take place. Depending on how you plan to use the repository, it may or may not make sense to return an lazily evaluated collection.
• Unit testing code that makes use of a repository becomes simpler. The repository can implement an interface through which your clients access it, enabling you to inject a fake repository into your business logic. Alternatively, your mocking framework may be capable of mocking a concrete class.
• LINQ to SharePoint could be used as an alternative to writing CAML queries by hand. I’ve refrained from using it because I think it adds unnecessary complexity and no value to the solution.

With this simple repository abstraction, I’ve created a data access layer that the business layer can call into and get back objects and collections of objects almost as if they’re in-memory objects, hiding away most of the SharePoint data access intricacies.

Suppose a site collection needs to have in-place records management disabled entirely. Then the first step would be to disable the site collection-scoped In Place Records Management feature:

disable-spfeature -identity InPlaceRecords -url $site_collection_url

However, what actually triggers the declaration of a record is the retention policy on a document library or its containing content type. Thus, removing the retention policy prevents future items from becoming records. On the document library or its containing content type, goto the Information management policy settings and disable the retention policy:

Existing records will remain unaffected by the feature deactivation and the disabling of the retention policy. We need to enumerate the document library and undeclare each exising record:

function UndeclareRecords($siteUrl, $libraryName) {
  function IsRecord($item) {
    [Microsoft.Office.RecordsManagement.RecordsRepository.Records]::IsRecord($item)
  }

  function EnsureCheckedIn($item) {
    if ($item.File.CheckOutType -ne [Microsoft.SharePoint.SPFile+SPCheckOutType]::None `
        -and (-not [Microsoft.Office.RecordsManagement.RecordsRepository.Records]::IsLocked)) {
      write-host "Checking in file: $($item.Url)"
      $file.CheckIn("Checked in by script prior to undeclaring as record", `
                    [Microsoft.SharePoint.SPCheckinType]::MinorCheckIn)
    }
  }

  function UndeclareRecord($item) {
    write-host "Undeclaring record: $($item.Url)"
    [Microsoft.Office.RecordsManagement.RecordsRepository.Records]::UndeclareItemAsRecord($item) 
  }

  $site = Get-SPWeb $siteUrl
  $list = $site.Lists[$libraryName]
  $list.Items | foreach { 
    if (IsRecord $_) {
      EnsureCheckedIn $_
      UndeclareRecord $_
    }
  }

  $site.Dispose()
}

Now every item in the document library is back to being a regular and modifiable item.

1. Favor readers over writers (10:20). There’re systematic differences in opinion between those who spent their time reading code and those who spent it writing code. Whenever there’s a difference in opinion between these two groups, the readers are always right and the writers are always wrong. The readers will always push in the direction of clarity and simplicity and the ability to change behavior easily. At least if you’re building software that’s going to last.

2. Create uniform interfaces (12:15). Always create an interface to your code to make it easier on the reader. When you build interfaces, you should have standards that apply uniformly across your code base to build solid expectations. Those who use your code should know what to provide and what to expect when interfacing with your codebase.

3. Make illegal states unrepresentable (18:03). Use the type system as a tool to enforce invariants on the code you write. Choose your data types such that states that are illegal don’t show up as legal states in the program. Take this code representing various connection information as an example. It keeps track of relevant information in a fairly readable manner:

type connection_state =
  | Connecting
  | Connected
  | Disconnected

type connection_info = {
    state:             connection_state
    server:            IPAddress
    last_ping_time:    DateTime option
    last_ping_id:      int option
    session_id:        string option
    when_initiated:    DateTime option
    when_disconnected: DateTime option
}

On the surface these types look reasonable, but there’re some tricky invariants that need to hold about the data. For instance, if you have a last_ping_time, you should probably also have a last_ping_id and vice versa. And the session_id and when_initiated probably only makes sense when you’re connected. Similarly, when_disconnected only makes sense if and when you’ve been disconnected.

The key is that there’s nothing about the types that help you enforce all these invariants. A better approach would be to refactor the connection_info into a series of types where the invariants would be inherent in the types themselves rather than being implicit in the logic surrounding the types:

type connecting = { when_initiated: DateTime }
type connected = { last_ping: (DateTime * int) option
                   session_id: string }
type disconnected = { when_connected: DateTime }

type connection_state =
  | Connecting of connecting
  | Connected of connected
  | Disconnected of disconnected

type connection_info = {
    state: connection_state
    server: IPAddress
}

server remains in connection_info because it applies to any of the states. The other information have been grouped together with the state it related to. The different connection_states are no longer merely a simple enumerated type but each of the different tags have content. Note also how the last_ping is now both the last_ping_time and last_ping_id. Either both are present, and grouped together, or not.

4. Code for exhaustiveness (28:33). This one is closely related to making illegal states unrepresentable in that you should write your code aiming at exhaustiveness guarantees. For instance, when you have a match statement, the compiler will warn you if the match is not exhaustive. The key benefit is as a refactoring tool because it guides changes in the code base. Don’t use the match all (_ –> …) because it means that if you expand on the discriminated union the compiler will not warn you.

5. Open few modules (34:08). When you open a module, it makes your code a bit shorter, and that’s great for the guy who wrote the code, but not the guy reading it. Now you can no longer just look at the code and tell where the value came from. In F#, with Visual Studio integration and IntelliSense, this is less of a problem. But the key advice is to respect the cognitive limitation of the people reading the code. If you want people to remember something, make them remember only for a short period of time.

6. Make common errors obvious (38:10). Use exceptions for exceptional conditions is what people often tell you. But whether something is a common case depends on context. For instance, in one context it’s an error if an element isn’t in a list, whereas in others it’s perfectly acceptable. For your API it may depend on the caller if a case is exceptional or not. To better support the caller, you could create two versions depending on how you want to communicate an error:

val hd : 'T list -> 'T option
val hd_exn : 'T list -> 'T

Now you can tell from the name of the function weather it throws an exception or not. For people reading the code it makes it easier to understand the error behavior of the code.

7. Avoid boilerplate (40:52). Avoid repeating the same code, or almost the same code, in multiple places. Boilerplate appears, in general, because people have a cut and paste template they use to do almost the same thing in multiple spots and because their language isn’t good enough to encode what they want to do in a clean way. You want to get rid of boilerplate because the structure you’re repeating is there for a reason, and your code evolves. At that point you’re not going to remember all the places where the repetition shows up. It also goes back to readability. It’s hard to convince people to read code if it’s dull. And nothing is duller than boilerplate, even though the code is critical.

Interestingly, reducing boilerplate doesn’t always make code less verbose. The goal isn’t to make the code shorter but to separate out which parts of the code is the same and which parts vary.

8. Avoid complex type hackery (45:30). The enemy of good code, of correctness, is not dynamic guarantees, properties about the code that cannot be proved at compile time, but complexity. Refrain from making your code more complex (using phantom types, for instance), just so you can have the type system verify additional properties of the code.

9. Don’t be puritanical about purity (47:10). Avoiding side-effect is generally worth striving for, because code without side-effects tends to be easier to reason about. But sometimes it’s also just easier to code with side-effects. There may even be performance reasons for allowing side-effects so don’t be embarrassed about it. In reality, programs don’t just compute things, they do things like write files, send messages, and so on. All this doing involves side-effects. Again, remember that the enemy of correctness is complexity. Don’t jump through hoops to make your code too complex just to make it pure.

Record types are limited by the fact that they have a single schema. In most applications, however, you want to store different shapes of data depending on different cases. In C#, we might do so using a class hierarchy with a base class keeping common state and subclasses keeping their own.

F# models a similar concept of hierarchical types using the discriminated union type. For instance, this discriminated union represents the result of a function that might succeed or might fail:

type Errorable<'a> =
  | Success of 'a
  | Error of string

If you disassemble the generated IL, the result is a class hierarchy. It has compare, equals, and hash code methods, like record types, but they’re placed on the base class. Factory methods are provided to create instances of the closed set of subtypes, and predicate methods are provided to check if an instance is of some type. In addition, each subtype has a constructor generated for it, accepting the argument following the “of” keyword, called the value constructor. Like record types, discriminated union types are immutable, which is why they’ve getter properties but no setters:

[Serializable, CompilationMapping(SourceConstructFlags.SumType)]
public abstract class Errorable<a> : IEquatable<Errorable<a>>, IStructuralEquatable,
                      IComparable<Errorable<a>>, IComparable, IStructuralComparable {
    // Methods
    internal Errorable();
    public sealed override int CompareTo(Errorable<a> obj);
    public sealed override int CompareTo(object obj);
    public sealed override int CompareTo(object obj, IComparer comp);
    public sealed override bool Equals(Errorable<a> obj);
    public sealed override bool Equals(object obj);
    public sealed override bool Equals(object obj, IEqualityComparer comp);
    public sealed override int GetHashCode();
    public sealed override int GetHashCode(IEqualityComparer comp);
    public static Errorable<a> NewError(string item);
    public static Errorable<a> NewSuccess(a item);

    // Properties
    public bool IsError { get; }
    public bool IsSuccess { get; }
    public int Tag { get; }

    // Nested Types
    [Serializable]
    public class Error : Errorable<a> {
        internal readonly string item;
        internal Error(string item);
        public string Item { get; }
    }

    [Serializable]
    public class Success : Errorable<a> {
        internal readonly a item;
        internal Success(a item);
        public a Item { get; }
    }

    public static class Tags {
        public const int Error = 1;
        public const int Success = 0;
    }
}

In an object-oriented language, you typically operate on a class hierarchy through virtual methods on the classes. Say you want to display the Errorable type, then you’d implement a virtual method on the Error and Success subclasses with behavior specific to the each type. In a functional language, the organizing principle is by function. You write one display function that use pattern matching to dispatch on the different cases:

let display x =
    match x with
    | Error e -> "Oh no! " + e
    | Success a -> a.ToString()

The variable names — e and a — get bound to the content of the case at hand. For the Error case, e is the error message, while for the success case a is the success value, whatever generic type it is.

Provided you leave out the match all case (_ –> …), a powerful feature of discriminated unions and pattern matching over classes and virtual methods is that the compiler is able to enforce that no cases have been forgotten, that no redundant cases exist, and that there’re no impossible cases. In C#, you’d encounter a missing case at runtime only if you remember to put in a catch all else or default case statement.

Suppose you want to store information about a person in F#, then the build-in record type provides a nice way of grouping related data together:

type Person = { Name: string; Age: int }

Decompiling the generated IL and coming at it from a C# angle provides for an interesting exercise. It not only shows how terse F# can be, but also how easy it is to consume F# types from other .NET languages (decompiled IL is method signatures only, not actual implementation):

[Serializable, CompilationMapping(SourceConstructFlags.RecordType)]
public sealed class Person : IEquatable<Person>, IStructuralEquatable, IComparable<Person>, 
                             IComparable, IStructuralComparable {
    // Fields
    [DebuggerBrowsable(DebuggerBrowsableState.Never)] internal int Age@;
    [DebuggerBrowsable(DebuggerBrowsableState.Never)] internal string Name@;

    // Methods
    public Person(string name, int age);
    [CompilerGenerated] public sealed override int CompareTo(Person obj);
    [CompilerGenerated] public sealed override int CompareTo(object obj);
    [CompilerGenerated] public sealed override int CompareTo(object obj, IComparer comp);
    [CompilerGenerated] public sealed override bool Equals(Person obj);
    [CompilerGenerated] public sealed override bool Equals(object obj);
    [CompilerGenerated] public sealed override bool Equals(object obj, IEqualityComparer comp);
    [CompilerGenerated] public sealed override int GetHashCode();
    [CompilerGenerated] public sealed override int GetHashCode(IEqualityComparer comp);

    // Properties
    [CompilationMapping(SourceConstructFlags.Field, 1)] public int Age { get; }
    [CompilationMapping(SourceConstructFlags.Field, 0)] public string Name { get; }
}

Observe how a public constructor has been generated, accepting arguments in the order in which the record was defined, and how only getter properties for Name and Age exist. That’s because the record type is immutable. The value of Person can only be set once using the constructor. Finally, compare, equals, and hash code methods for doing value comparisons, implementing all the proper interfaces, have been compiler-generated as well.

Suppose you have a function divisiblyBy and a list of numbers. Then you can apply the filter function to get a list of even numbers:

// val divisibleBy : int -> int -> bool
let divisibleBy factor value = value % factor = 0
let evenNums = List.filter (fun x -> divisibleBy 2 x) [1..10]

This lambda syntax is a bit noisy, but can oftentimes be transformed into a shorthand lambda syntax. When the argument passed into the lambda — in this case x — is the last argument to the function within the lambda body, a bit of compiler trickery combined with the use of partial function application can help simplify the expression. Combining partial function application with function composition even means we don’t have to explicitly create a lot of smaller functions, such as one that negates another function:

let evenNums = List.filter (divisibleBy 2) [1..10]       // partial function application
let oddNums = List.filter (not << divisibleBy 2) [1..10] // and function composition

Of course, the concept of partial function application isn’t tied to lambda expressions:

let divisibleByTwo = divisibleBy 2           // val divisibleByTwo : (int -> bool)
let notDivisibleByTwo = not << divisibleBy 2 // val notDivisibleByTwo : (int -> bool)
let a = divisibleByTwo 4                     // val a : bool = true
let b = notDivisibleByTwo 4                  // val b : bool = false

The partial application of divisibleBy results in a new function of one less argument, since we provided a value for the factor argument. As you can tell by their signatures, divisibleByTwo and notDivisibleByTwo are functions that take one argument, the value, and return a bool. The compiler knows that divisibleByTwo and notDivisibleByTwo take one and only argument for them to fully evaluate and that filter takes as argument a function that goes from an argument of type ‘T — in this case int — to bool.

val List.filter : ('T -> bool) -> 'T list -> 'T list

Now all types align and we’ve simplified the code a little.

PS: Partial function application is also possible in C# if you define functions using one of the Func<T> or Action<T> delegates.

The first thing I learned is how the F# list type, which is a linked list and not List<T>, is immutable for reasons of safety and efficiency. Once you create a list, you can never again modify this exact instance. If you do, what you’ll get back is a new list. Think of the safety in terms of threads: if one thread, inadvertently or not, modifies the list, it’ll just get back a new list, shielding any other thread from the change.

This copy-on-modification behavior may seem inefficient at first. And it would be if it weren’t for F#’s ability to share part of a list:

let smallOddPrimes = [3; 5; 7]
let smallPrimes = 2 :: smallOddPrimes // [2; 3; 5; 7]

When you create the smallPrimes list, F# need only create one additional cell containing the even prime. The next pointer of this cell will be a reference to the smallOddPrimes list:

Object.ReferenceEquals(smallPrimes.Tail, smallOddPrimes) // true

The tail of the smallPrimes list — the possibly empty list comprised of everything but the first element — points back to smallOddPrimes list, saving both processing time and memory while retaining thread-safety.

Before diving into the details of how PowerShell can be used to build and deploy a SharePoint solution, I quickly want to cover some of the downsides as well: the PowerShell environment and the SharePoint cmdlets add additional plumping on top of something that’s already very complex. If you’re already well-versed in the SharePoint .NET APIs, the added (accidental) complexity may not be worth it. The good-old console application in which you’re already familiar with the syntax, semantics, and debugging procedures hasn’t been obsoleted by PowerShell.

Getting started with PSake

Nothing prevents you from automating using plain PowerShell, but leaning on an open source module like PSake will relieve you from writing tedious plumping code. PSake is an internal DSL for defining, among other things, tasks and interdependencies between these. It then determines a proper task execution order and does error detection while it executes each task. PSake also comes with a handy external command-wrapper to be used within tasks which determines failure based on the exit code of the external process.

To get started with PSake, all you need is to go to the PSake Github repository and download psake.ps1 and psake.psm1 to a local folder. I usually place these in the root of my source folder, next to the solution file. psake.ps1 is a wrapper that enables you to open a PowerShell console and execute a PSake task without first having to load the PSake module. The wrapper will simply load the module and pass along any command-line arguments to PSake.

Keep in mind that PSake isn’t designed with nested scripts in mind. It’s possible to have a master PSake script call out to child PSake scripts, but an error in a child will not get propagated to the master, causing the master to continue on error. Tasks should therefore be collected in a single file, possibly named default.ps1, which is what PSake looks for by default.

PSake script organization

The script organization outlined in this section is backed by snippet-like examples from one of my build files. Don’t focus on the PowerShell specifics in these examples, but instead on the overall organization and purpose of each section.

Every one of my PSake scripts start out with global definitions at the top. These are used throughout the script and conceptually serve as common hidden arguments to the tasks. PSake also comes with a Properties block for making variables available to tasks, but I don’t think it adds significant value over plain script definitions.

$base_dir = resolve-path .
$sln_file = "$base_dir\$acme.sln"
$build_configuration = "debug"
$acme = "Acme.Intranet"
$msbuild_exe = "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe"
$wsp_projs = @("Branding", "Records")
$other_projs = @("Console")
$current_user = "$env:userdomain\$env:username"
$web_application_name = "Acme"
...

Next come any PowerShell function used by other functions or tasks. These functions will usually extend the capabilities of existing cmdlets by composition or/and calling out to .NET APIs directly. I strive to make these function build script-independent by parameterization. To limit side effects and simplify debugging, the functions should rely solely on the arguments being passed, and refrain from accessing any global variable.

Here I load the Microsoft.SharePoint.dll but the types of any .NET assembly can be made available to the script this way.

[Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint") | out-null

function RemoveBuildFolders([string]$source_dir) { 
  ("bin", "obj", "pkg", "pkgobj") | foreach {
    $path = "$source_dir\$_"
    if (test-path $path) {
      remove-item $path -confirm:$false -recurse -force
    }
  }
}

function IsUserAdministrator() { ... }
function ImportTermSetFromFile([string]$filePath, [string]$groupName) { ... }
function WaitForTimerJobToFinish([string]$solutionFileName) { ... }
...

For the script to use any of the SharePoint cmdlets, you must first load the SharePoint PowerShell snapin. The loading of this snapin is actually what differentiates the SharePoint 2010 Management Shell from the regular PowerShell — the former launches PowerShell and executes a script that loads the snapin. For some reason, however, it’s necessary to unload/load the snapin every time the script executes or the SharePoint cmdlets will only be available the first time around. Hence this code needs to be placed outside of any function, causing it to execute immediately.

$name = "Microsoft.SharePoint.PowerShell"
$snapin = get-pssnapin | where { $_.name -eq $name }
if ($snapin -ne $null) {
  remove-pssnapin $name
}
add-pssnapin $name

Now we’re ready to define the actual tasks. Depending on your SharePoint solution, you may want to distribute the responsibility among the tasks differently, but build and deployment will likely assume this overall form:

task Install -depends Clean, Compile, CreateWebApplication, CreateSiteCollection,
                      ActivateSiteCollectionFeatures, CreateSites, PreInstall,
                      InstallWsps, ActivateSiteFeatures, PostInstall

task Uninstall -depends DeactivateSiteFeatures, RemoveSites, DeactivateSiteCollectionFeatures,
                        RemoveSiteCollection, UninstallWsps, RemoveWebApplication

The Compile task, for instance, calls out to MSBuild just like Visual Studio does. This has the nice benefit that the build script will automatically use build settings defined in Visual Studio as they’re stored in the MSBuild project file. Note also how WSPs are created by invoking the package target of a SharePoint project file. Again, all the package specific settings of Visual Studio will carry over seamlessly.

task Compile {
  exec { & "$msbuild_exe" "/p:configuration=$build_configuration" "$sln_file" }
  $wsp_projs | foreach {
    $proj_file = "$base_dir\$acme.$_\$acme.$_.csproj"
    exec { & "$msbuild_exe" "/p:configuration=$build_configuration" "$proj_file" "/t:package" }
  }
}

The ActivateSiteCollectionFeatures task illustrates the SharePoint cmdlets in action. It also shows how to spawn a new PowerShell console to avoid a “The trial period for this product has expired” error when activating the PublishingSite feature. This error seems to be related to stale state (a bug) within the current PowerShell session as I’m not running a trial version of SharePoint. On a similar note, one of my scripts activate the Nintex Workflow 2010 features. This operation will fail if you use the enable-spfeature cmdlet instead of stsadm, alluding to another bug in the PowerShell/SharePoint integration.

task ActivateSiteCollectionFeatures {
  install-spfeature PublishingSite
  
  # work around the 'The trial period for this product has expired.' error
  exec { powershell -command "add-pssnapin microsoft.sharepoint.powershell; 
                              enable-spfeature -identity PublishingSite -url $site_collection_url" } 

  enable-spfeature -identity OfficeWebApps -url $site_collection_url
  enable-spfeature -identity PremiumSite -url $site_collection_url
  ...
}

Assuming a complete script, here’s how you’d invoke the Install task in a script named default.ps1.

.\psake.ps1 .\default.ps1 Install

This’ll execute the Install task, recursively executing its dependencies.

Observations

Automating SharePoint build and deployment is challenging. At times SharePoint will exhibit non-deterministic behavior, succeeding or failing by random. For instance, even though you explicitly deactivate and uninstall a feature, during installation the script may fail telling you the feature is already activated. You could add a “continue on error” flag to problematic cmdlets, but this would also prevent error detection, and you’ll likely end up with cascade failures later or functionality you can’t trust being correctly build and deployed. Adding error recovery and correction code at select places is probably a better choice.

Debugging a large PowerShell script can be time consuming. With PowerGUI or a similar environment you’re able to set breakpoints, but simple print-lining and writing out the value of $errors[0] will usually suffice. The $errors collection is a circular buffer of the most recent script errors encountered. You can also modify the trace level within the special TaskSetup and TaskTearDown tasks that execute before and after any of your tasks.

TaskSetup { Set-PSDebug -trace 1 }
TaskTearDown { Set-PSDebug -trace 0 }

This’ll emit a lot of trace output which may also be useful to include in a build log.

Conclusion

I think there’s some merit to the argument of not relying solely on PowerShell for complex SharePoint build and deployment. Don’t get me wrong, PowerShell is a very nice scripting language and there’s very little PowerShell can’t do from a purely technical perspective. The dynamic nature of PowerShell, however, makes it less suited for longer and complex scripts.

My latest SharePoint build and deployment script is approaching 700 lines of PowerShell code and contains some fairly complex functions that should probably have been written in a non-scripting language and exposed to PowerShell through something like a generic command-line runner. You could also have created custom cmdlets, but why tie your code unnecessarily to PowerShell?

For additional inspiration on how to organize your build scripts, a number of prominent open source projects use PSake, most notably NServiceBus and RavenDB.

Here I provide an F# implementation of a Powershell script for populating the SharePoint 2010 term store with a term set. Each term set is stored in a CSV file (whose format is defined by Microsoft) that drives the calls made to the SharePoint .NET APIs to import it:

|   |       0       |       5      |       6      | ... |      11      |
|---|---------------|--------------|--------------|-----|--------------|
| 0 | Term Set Name | Level 1 Term | Level 2 Term | ... | Level 7 Term |
| 1 | TermSetName   |              |              | ... |              |
| 2 |               | Term1        |              | ... |              |
| 3 |               | Term2        | Term2.2      | ... |              |

Once the term set has been imported, this is what the hierarchy looks like from within SharePoint:

First the function signatures:

// Program.fsi
module Bugfree.SharePoint.TermSetImporter

open Microsoft.SharePoint.Taxonomy

val importCsv : string -> seq<string[]>
val removeTermGroup : TermStore -> string -> unit
val getOrCreate : 'b -> seq<'a> -> ('a -> bool) -> ('b -> 'a) -> 'a
val getOrCreateGroup : string -> TermStore -> Group
val getOrCreateSet : string -> Group -> TermSet
val getOrCreateTerm : string -> TermSetItem -> Term
val importTerm : TermSetItem -> string list -> TermSetItem
val importTermSet : TermStore -> string -> seq<string[]> -> unit 
val main : string[] -> int

Then the implementation:

// Program.fs
module Bugfree.SharePoint.TermSetImporter

open System.IO
open System.Collections.Generic
open Microsoft.SharePoint
open Microsoft.SharePoint.Taxonomy

let importCsv path =
    seq { use sr = File.OpenText(path)
          while not sr.EndOfStream do                  
            let line = sr.ReadLine()                  
            let tokens = line.Split [|','|]                  
            yield tokens }

let removeTermGroup (store : TermStore) name =
    store.Groups |> Seq.filter(fun g -> g.Name = name) 
                 |> Seq.iter(fun g -> g.TermSets |> Seq.iter(fun t -> t.Delete())
                                      g.Delete()
                                      store.CommitAll())

let getOrCreate name children predicate create =   
    match (Seq.tryFind predicate children) with
    | Some c -> c
    | None -> create name

let getOrCreateGroup name (store : TermStore) =
    getOrCreate name store.Groups (fun g -> g.Name = name) 
                                  (fun name -> store.CreateGroup(name))

let getOrCreateSet name (group : Group) =
    getOrCreate name group.TermSets (fun s -> s.Name = name) 
                                    (fun name -> group.CreateTermSet(name))

let getOrCreateTerm name (item : TermSetItem) =
    getOrCreate name item.Terms (fun t -> t.Name = name) 
                                (fun name -> item.CreateTerm(name, 1033))

let rec importTerm (parent : TermSetItem) levels =       
    match levels with
    | [] -> parent
    | head::tail -> let t = getOrCreateTerm head parent
                    importTerm t tail
                               
let importTermSet (store : TermStore) groupName (rows : seq<string[]>) =  
    let termSetName = (rows |> Seq.nth 1).[0].Replace("\"", "")
    let termSet = getOrCreateGroup groupName store |> getOrCreateSet termSetName
    
    rows |> Seq.skip 2 
         |> Seq.iter(fun r -> r.[5..] |> Array.filter(fun i -> i <> "") 
                                      |> Array.map(fun i -> i.Replace("\"", ""))
                                      |> Array.toList
                                      |> importTerm termSet
                                      |> ignore)
    store.CommitAll()

[<EntryPoint>]
let main args =
    let siteCollection = new SPSite("http://sp2010")
    let session = new TaxonomySession(siteCollection)
    let store = session.TermStores.["Managed Metadata Service"]    
    let rows = importCsv "C:\Test.csv"
    let groupName = "MyGroup"
    removeTermGroup store groupName
    importTermSet store groupName rows
    0

Maybe it’s implementing the population algorithm twice, but to me the F# implementation seems the easiest one to implement and follow. Not because PowerShell is a bad language, but because F#’s static typing and type inferencing makes it better suited for implementing something of this relative complexity.

• rss
  • channel
    • title
    • link
    • description
    • …
    • item
      • title
      • link
      • pubdate
      • enclosure
        • url
        • …
      • …
    • …

Assuming the feed points to a blog, the initial channel tags would hold general information about the blog and each item would represent a blog post. Files would be associated with posts through the enclosure tag and this is how you’d download episodes. Using PowerShell to parse the XML file, it’s fairly straightforward to download either the most recent enclosures or all episodes.

function Get-RssEnclosures([String]$rssUrl, 
                           [DateTime]$moreRecentThan, 
                           [String]$destinationFolder) {
  # directory where WebClient will download files to
  [Environment]::CurrentDirectory = $destinationFolder 

  $feed = [xml](new-object Net.WebClient).DownloadString($rssUrl)
  $feed.rss.channel.item | foreach {
    $enclosureUrl = $_.enclosure.url    
    
    # some feeds I download contain empty enclosure urls
    if ($enclosureUrl -ne "") {
      $enclosureUrl = new-object Uri($enclosureUrl)
      $filename = $enclosureUrl.Segments[-1]
      
      # some items I download have non-parsable 'EDT' as part of pubdate 
      $pubDate = [DateTime]::Parse($_.pubdate.Replace("EDT", ""))

      if (($pubDate -ge $moreRecentThan) -and 
          (-not (test-path (join-path $destinationFolder $fileName)))) {
        (new-object Net.WebClient).DownloadFile($enclosureUrl, $filename)
      }
    }
  } 
}

@("http://feeds.feedburner.com/HanselminutesCompleteMP3",
  "http://www.pwop.com/feed.aspx?show=dotnetrocks&filetype=master",
  "http://feeds.feedburner.com/DnrtvWmv",
  "http://thisdeveloperslife.com/rss",
  "http://feeds.feedburner.com/herdingcode",
  "http://feeds.feedburner.com/anugcast") | foreach {
  Get-RssEnclosures $_ ([DateTime]::Now.AddDays(-7)) "E:\Podcasts"
}

The nice thing about PowerShell is that you can dot your way through the XML hierarchy and use regular .NET classes for date parsing and file download. Now you can setup this script to execute at regular intervals and go to the destination folder whenever you feel like listening to a podcast.

At this point, I should probably point out that some of the pain I encountered may well be due to my lack of experience with the framework and tooling. In this case, feel free to drop me a note. I would also like to point out that my aim here is to highlight the significant issues I encountered. There’re already a lot of posts out there iterating the positive sides of BCS, but as a developer I’m equally, if not more, concerned with its limitations.

The SharePoint Designer 2010 experience

With the aid of SharePoint Designer 2010 (SPD), it’s a breeze to define an external content type to match the response of a web service and create an external list based on this content type. The ugly truth, though, is that although the external list will visually appear much like a standard SharePoint list, in terms of functionality it’s significantly impaired (see also Inside Microsoft SharePoint 2010, page 493). In short, while BCS and SPD enables you get off the ground quickly, if you don’t understand their limitations, you may have incurred a significant technical debt from the get-go.

Behind the scenes, what SPD does is maintain an XML-based metadata model, mapping a set of method stereotypes, such as Finder, SpecificFinder, Insert, Update, and Delete to actual methods on the web service. For BCS to work, these methods must satisfy a rigid set of requirements concerning their signature and behavior. When SharePoint, through the BCS runtime, needs to interact with the external system, BCS will look up the specific method to call within the model and process the result accordingly.

Interestingly, while SPD may be a good starting point for creating the model, the model is actually richer than SPD is able to express. Oftentimes you need to go beyond SPD and export the model, edit the XML file by hand, and then import it back into SharePoint. Your modifications will mostly be left alone by SPD, though not always. Take field ordering within a content type for example: whenever you edit the content type, the order seems to get reset to the order in which .NET reflects over the DataMembers of the DataContract within the generated web service proxy. This in turn affects the order in which form fields appear on the generated display, new, and edit forms. The DataMember attribute actually has an Order property but since it has no effect on the generated WSDL, SPD can’t possibly take it into account. The net effect is that you’ll oftentimes have to go edit the XML file by hand or script the it.

When creating an external list based on a content types, the display, edit, and new forms are generated from the model. But how are future changes to the model propagated to existing forms? It turns out some are carried over automatically (see Professional Business Connectivity Services in SharePoint 2010, page 55), whereas others will require the list to be re-generated. Not the ideal choice if your end-users made any modifications that needs to be preserved. It seems difficult, bordering impossible, to modify the generated ASPX forms, except through hacks with JavaScript, in which case InfoPath is probably the better choice. I wouldn’t want to use InfoPath for a 20 lists solution, though. Using InfoPath on this scale quickly becomes tedious, error-phone, and not maintainable in the long run.

In my opinion, the process of defining content types with SPD doesn’t scale. Once you go beyond a few content types, working with SPD becomes tedious. When your data source is a web service, the list of methods on it required just to make CRUD operations work makes the contract quite large. Up to more than 100 methods for my 20 content types, and except for the stereotypical Finder method, none of these support bulk operations. As SPD has to retrieve the contract and generate a proxy from it, not to mention manipulate the model stored within the BCS Service Application, the UI isn’t particularly responsive. The SPD UI itself seems to have been optimized for working with only a few read-only external content types within a solution. You can’t specify field ordering or which field control to use for rendering or editing a field, and it’s sometimes hard to locate your web method amongst the 100+ candidates.

The SharePoint end-user experience

External content types are made up of fields whose types are a subset of the primitive types of .NET. For each such .NET type, a rudimentary mapping to SharePoint’s field controls exists, describing a field’s visual appearance. Fields of type string for instance will, on the new and edit forms, render as a single-line text field. With regular lists, using the browser interface, you can easily turn a string field into a multi-line text area or a Rich Text field, but not so with external lists. At least not unless your forms are InfoPath-based. For ASPX forms, the .NET DateTime type, on the other hand, will render as the standard SharePoint date/time picker, allowing you to set date and time, but not easily only the date.

What I’m particularly missing is the People Picker. Within my database a number of fields store a domain\username provided by the user. These fields are of type string within the database and web service, so SPD will render these as single-line text field without the usual validator button. BCS does offer the ability to render custom SPFields by hand-tweaking the model (see Professional Business Connectivity Services in SharePoint 2010, page 174) and you may be able to use this approach to get the People Picker to render. Another alternative would be to alter the visual appearance and logic of the generated forms with JavaScript, although this seems like a brittle hack.

While on the subject of picker controls, the external item picker is used to setup one-to-many associations between external lists. To make the picker show up, you need to add of a lookup method to your web service and provide a stereotypical mapping to it. I mention this picker because it makes SharePoint display the associated item’s friendlier title instead of its foreign key identifier on the new and edit forms. Back on the list view, however, SharePoint will keep displaying the foreign key identifier, forcing end-users to have to lookup the meaning of it themselves. Moving on to many-to-many associations, they’re simply not supported by the user interface. You’d have to build a custom control for setting up the association or directly expose the junction table to end-users. But remember how list views can’t display the title of foreign items? The list view will end up showing rows of foreign key identifiers.

In my opinion, SharePoint does a decent job at presenting external data in read-only form but is severely lacking when it comes to editing. Creating a decent data entry experience is too much work compared to using an established technology such as ASP.NET. Sure, you could go with InfoPath, but it comes with its own set of peculiarities.

Conclusion

My 20 external content types turn into a metadata model of roughly 8,000 lines of XML with a huge blob at the top, storing the MSIL of the service proxy generated by SPD. You can think of this model as an XML representation of 20 vtables, enriched with attributes to support form generation. All in all, a very complex setup for what it actually brings to the table. As a technology, I believe BCS to be too immature for intermediate to complex web service-based, read/write solutions — and I don’t want to create a code-based model using the BCS templates that ship with SharePoint for Visual Studio. In this case I’d rather go with an ASP.NET solution and embed it into SharePoint. I’d probably also skip the web service altogether and use a data access assembly instead.

But for the sake of the argument, let’s say BCS is still your tool of choice and that your solution warrants the creation of a significant number of external content types/external lists/forms. Then you should probably roll your own tools to either create the XML of the model directly or use the BCS object model to create the model and then export it. This way, you’ll have more control over field ordering and the like and be less dependent on SPD.

I considered a few strategies for how to implement these requirements:

1. Use one table and clone rows before change. Every time a record is modified, a copy of it is made beforehand. I use the Active attribute of a record to denote which records are active vs. which make up the history. This approach, however, requires that you make the primary key a combination of Id and VersionId columns. Likewise, for relationships between tables, you must make the foreign key be a combination of ForeignId and ForeignVersionId. Referential integrity now becomes relatively straightforward to maintain, but the table may grow large and may likely contain a lot redundant information.
2. Create one history table for each source table. A variation of #1 where each source table would have en corresponding history table with a close to identical schema, but with its own primary key. While this ensures that the source table only contains current rows, it effectively doubles the number of tables. Not to mention that every structural change to the source table would have to be propagated to the history table.
3. Create a generic and conceptually separate history database. This approach consists of a history table serving any number of source tables. Conceptually, it’s a audit trail database of its own that the rest of the database need not know about. Each row in this history table would hold before and after values for changed columns only. You may even take it so far as to start with an empty source and a some history and replay every change made to the source up to some point in time. It would require special care to be taken for non-additive schema changes.

I decided to go with the third option. To make it concrete, say I have a Contacts table. Besides the primary key, I store the name of the contact and possibly a comment:

CREATE TABLE [dbo].[Contacts](
  [Id] [int] IDENTITY(1,1) NOT NULL,
  [Name] [nvarchar](255) NOT NULL,
  [Comment] [nvarchar](max) NULL,
  [CreatedBy] [nvarchar](30) NOT NULL,
  [CreatedAt] [datetime] NOT NULL,
  [ModifiedBy] [nvarchar](30) NOT NULL,
  [ModifiedAt] [datetime] NOT NULL,
  [UniqueRowGuid] [uniqueidentifier] NOT NULL,
  [Active] [bit] NOT NULL)

The remaining fields are system fields that I add to almost every table in the database. You could argue that the created and modified columns should really be part of the history table, but I prefer for the source table to be self-contained. UniqueRowGuid provides a unique identity of any row in any table and is useful for creating weak relationships between tables in the same database or across databases.

If it weren’t for the fact that a primary key of type Guid makes for bad clustered index performance, I could’ve made the primary key in every table be the UniqueRowGuid — MS SQL Server even has a special type of sequential Guid for this very purpose. But I stuck with the traditional integer-based key. The Active column is used to denote if a row has been deleted. I want to keep every row for some period of time, even though to the user it appears to have been deleted.

Jumping straight to the history schema, here’s what I decided to store for each change.

CREATE TABLE [dbo].[History](
  [Id] [int] IDENTITY(1,1) NOT NULL,
  [ChangeCorrelationGuid] [uniqueidentifier] NOT NULL,
  [UniqueRowGuid] [uniqueidentifier] NOT NULL,
  [ColumnName] [nvarchar](255) NOT NULL,
  [BeforeValue] [nvarchar](max) NOT NULL,
  [AfterValue] [nvarchar](max) NOT NULL,
  [CreatedAt] [datetime] NOT NULL,
  [Comment] [nvarchar](max) NULL)

The ChangeCorrelationGuid allows for a grouping of related history entries. Suppose in one transaction you modified both the Name and Comment of a contact. Using this ChangeCorrelationGuid, you effectively create different versions of the row. Next comes the UniqueRowGuid which contains the row-value from the source table, resulting in a loose coupling between History and Contacts. Think of it as identifying the source row whereas ColumnName identifying the source column.

Strictly speaking, I don’t need to store both the BeforeValue and an AfterValue, since the AfterValue of one change becomes the Before version of the next. But this piece of controlled redundency makes querying for the summary easier and less costly.

One of the new features of SharePoint 2010 is the ability to tag content using terms from a term store. The out of the box tooling for importing, exporting, and maintaining term stores is somewhat limited though. That’s why I created a PowerShell script to import term sets into the term store.

Term store overview

SharePoint organizes terms in either a taxonomy or a folksonomy. In a taxonomy, terms are organized hierarchically whereas in a folksonomy the structure is flat. A taxonomy or folksonomy is either global, making it available to all site collections within a web application that’s associated with a Managed Metadata Service, or local, making it available in one site collection only.

Multiple Managed Metadata Services may be associated with a web application or site collection, each service in turn responsible for a single term store. Within one term store, multiple groups may be created, each potentially storing multiple term sets. Finally, a term set may have terms nested up to seven levels deep:

Managed Metadata Service Application
  TermStore
    Group1
      TermSet1
        Term1
          Term1.2
            Term1.2.3
              Term1.2.3.4
                Term1.2.3.4.5
                  Term1.2.3.4.5.6
                    Term1.2.3.4.5.6.7

You manage the term store through the browser and its Term Store Management Tool (the _layouts/termstoremanager.aspx application page) or through the SharePoint API. The management tool supports importing term sets only from a special CSV file format  (_layouts/1033/ImportTermSet.csv holds a sample) with these columns:

Term Set Name
Term Set Description
LCID
Available for Tagging
Term Description
Level 1 Term
Level 2 Term
...
Level 7 Term

The CSV file is incomplete in terms of what the term store offers. It doesn’t offer columns for specifying translations and synonyms for terms. But users can use Wictor Wilén Excel sheet to maintain the basic information. It has the standard columns above and includes macros to simplify saving in the correct format and creating additional term sets.

The question remains: do you maintain compatibility with the existing CSV format so you can import term sets using the browser or a forms-based tool like the CSV Bulk Taxonomy Importer/Exporter? Or do you define your own, possibly XML based format since term sets are hierarchical by nature, with additional columns for translations and synonyms? A non-CSV based format may also be easier to process. If you want to pursue the latter approach, Ben Robb has created a PowerShell script to import his XML format.

Importing term sets using PowerShell

I decided to stick with the CSV file format and create a PowerShell script that would allow me to import a term set using the command-line:

function ImportTermSet([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName, [PSCustomObject]$termSet) {  
  function ImportTerm([Microsoft.SharePoint.Taxonomy.Group]$group, 
                      [Microsoft.SharePoint.Taxonomy.TermSet]$set, 
                      [Microsoft.SharePoint.Taxonomy.Term]$parent, 
                      [string[]]$path) {       	
    if ($path.Length -eq 0) {
      return
    } elseif ($group -eq $null) {
      $group = $store.Groups | where { $_.Name -eq $path[0] }
      if ($group -eq $null) {
        $group = $store.CreateGroup($path[0])
      }
    } elseif ($set -eq $null) {
      $set = $group.TermSets | where { $_.Name -eq $path[0] }
      if ($set -eq $null) {
        $set = $group.CreateTermSet($path[0])
      }
    } else {
      $node = if ($parent -eq $null) { $set } else { $parent }
      $parent = $node.Terms | where { $_.Name -eq $path[0] }  	   
      if ($parent -eq $null) {
        $parent = $node.CreateTerm($path[0], 1033)
      } 
    }
    
    ImportTerm $group $set $parent $path[1..($path.Length)]					
  }
  
  function RemoveTermGroup([Microsoft.SharePoint.Taxonomy.TermStore]$store, [string]$groupName) {
    $group = $store.Groups | where { $_.Name -eq $groupName }
    if ($group -ne $null) {
      $group.TermSets | foreach { $_.Delete() }
      $group.Delete()
      $store.CommitAll()
    }
  }
  
  RemoveTermGroup $store $groupName
  $termSetName = $termSet[0]."Term Set Name"    
  $termSet | where { $_."Level 1 Term" -ne "" } | foreach {
    $path = @($groupName, $termSetName) + @(for ($i = 1; $i -le 7; $i++) { 
      $term = $_."Level $i Term"
        if ($term -eq "") {
          break
        } else {
          $term
        }
      }
    )
	
    ImportTerm -path $path
  }
}

The way to use the ImportTermSet function is best illustrated with an example:

$session = Get-SPTaxonomySession -Site "http://localhost"
$store = $session.TermStores["Managed Metadata Service"]   
$termSet = Import-Csv "C:\Users\Ronnie\Desktop\ImportTermSet.csv"
ImportTermSet $store "MyGroup" $termSet
$store.CommitAll()

In its current form the script ignores the Term Set Description, LCID, Available for Tagging, and Term Description columns from the CSV file. It would be possible to take these columns into account by passing them to ImportTerm as well. Instead of just removing the head of the list, you’d remove the number of items consumed by the current step before recursing.

type Proposition =
    | True
    | And of Proposition * Proposition
    | Or of Proposition * Proposition
    | Not of Proposition

In an object-oriented language like C#, this definition would be analogous to a hierarchy of classes. Proposition would be the base class of True, And, Or, and Not. Each would then have a constructor that accepts some number of arguments, e.g., the And constructor would accept a two-tuple, two propositions, making the Proposition type recursive. In C# you might even model the recursive relationship using the Composite pattern.

In F#, an instance of the Proposition type may be created using a constructor syntax that resembles that of a function call:

let p = And(Or(True, Not(True)), Not(Not(True)))

Now, because of the recursive nature of the Proposition type, instances of it form a parse tree-like structure:

                        And
                        /\
                       /  \
                      Or  Not 
                      /\    \
                     /  \    \
                  True  Not  Not
                          \    \
                           \    \ 
                          True  True

Having a parse tree is not very useful in itself. What you need is some way to execute or evaluate it. Given the recursive nature of the grammar, it makes sense for the evaluator to be recursively defined as well. The evaluator would do pattern matching on the node type, deconstructing it into the parts needed to evaluate it. Since our language of propositions maps directly to the boolean operations of F#, it’s very simple to evaluate the parse tree:

let rec eval (p: Proposition) =
    match p with
    | True -> true
    | And(p1, p2) -> eval p1 && eval p2
    | Or (p1, p2) -> eval p1 || eval p2
    | Not(p1) -> not (eval p1)

let e = eval p // e : bool = true

The features of F# used here allow for more complicated languages to be expressed and evaluated directly without the need for any Backus-Naur form and parser generator.

PS: I’ve encountered this gist implementing a nice progression of the same idea with expressions and variables.

For the generic runner to work, each command has to fulfill the contract.

public enum ExitCode {
    Success = 0,
    Failure
};

public interface ICommand {
    string Usage { get; }
    string Description { get; }
    ExitCode Execute(string[] args);
}

I want the runner to adhere to the open/closed principle. For its behavior to be modified without altering its core delegation logic. This requires the use of reflection to retrieve and instantiate a command based on command-line arguments.

class Program {
    static IEnumerable<ICommand> GetCommands() {
        var iCommand = typeof (ICommand);
        return System.Reflection.Assembly.GetExecutingAssembly().GetTypes().ToList()
            .Where(t => iCommand.IsAssignableFrom(t) && t != iCommand)
            .Select(t => Activator.CreateInstance(t) as ICommand);
    }

    static void DisplayHelp() {
        Console.WriteLine("Console [Command] [Arg1] [Arg2] [ArgN]\n\n");
        GetCommands().ToList().ForEach(command => 
            Console.WriteLine(command.Usage + "\n" + command.Description + "\n\n"));
    }

    static int Main(string[] args) {
        if (args.Length == 0) {
            DisplayHelp();
            return (int)ExitCode.Failure;
        }

        var commandName = args[0];
        var command = GetCommands().Where(t => t.GetType().Name == commandName).SingleOrDefault();
        if (command == null)
            throw new ArgumentException(string.Format("Command '{0}' not found", commandName));

        var executeArguments = new List<string>(args);
        executeArguments.RemoveAt(0);

        var exitCode = command.Execute(executeArguments.ToArray());
        return (int)exitCode;
    }
}

A trivial example of a command that adds two numbers would be the following:

// $> GenericRunner.exe Calculator 2 3 => 2 + 3 = 5
public class Calculator : ICommand {
    public string Usage {
        get { return "Calculator [Op1] [Op2]"; }
    }

    public string Description {
        get { return "World's simplest calculator"; }
    }
        
    public ExitCode Execute(string[] args) {
        try {
            Console.WriteLine(
                string.Format(
                    "{0} + {1} = {2}",
                    args[0], args[1], int.Parse(args[0]) + int.Parse(args[1])));
            return ExitCode.Success;
        } catch (Exception e) {
            Console.WriteLine(e.ToString());
            return ExitCode.Failure;
        }
    }
}

Now multiple smaller assemblies can be grouped into one, with a description of all commands automatically being assembled, and without commands interfering (too much) with each other.

namespace Dk.Bugfree

open System
open System.Globalization
open Microsoft.SharePoint

type public ListAttachmentVersioningEventReceiver() =
    inherit SPItemEventReceiver()
    
    member private r.CustomVersion = "CustomVersion"
    member private r.ShadowLibrary = "ShadowLibrary"

    // override ItemAdded : properties:SPItemEventProperties -> unit
    override r.ItemAdded properties =
        base.ItemAdded properties
        r.SetCustomVersionLabel properties.ListItem
        r.CreateSnapshot properties

    // override ItemUpdated : properties:SPItemEventProperties -> unit
    override r.ItemUpdated properties =
        base.ItemUpdated properties
        let item = properties.ListItem

        if r.RollbackHappened item then
            r.RestoreSnapshot properties
            r.SetCustomVersionLabel item
            r.CreateSnapshot properties
        else
            r.CreateSnapshot properties
            r.SetCustomVersionLabel item

    // member private CreateSnapshot : properties:SPItemEventProperties -> unit
    member private r.CreateSnapshot properties =
        use site = properties.OpenWeb()
        let item = properties.ListItem
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let path = String.Format("Versions/{0}/{1}", item.ID, r.GetOfficialVersionLabel(item))
        let shadowFolder = r.CreateFolderPath shadowLibrary path
        
        item.Attachments |> Seq.cast |> Seq.iter (fun fileName ->            
            let existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName)
            let newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream())
            newFile.Item.Update())
    
    // member private RollbackHappened : item:SPListItem -> bool
    member private r.RollbackHappened item =
        let culture = CultureInfo.InvariantCulture
        let currentVersion = Single.Parse(r.GetOfficialVersionLabel(item), culture)
        let lastVersion = Single.Parse(r.GetCustomVersionLabel(item), culture)
        currentVersion > lastVersion + 1.0f

    // member private RestoreSnapshot : properties:SPItemEventProperties -> unit
    member private r.RestoreSnapshot properties =
        let item = properties.ListItem
        let restoreVersion = r.GetCustomVersionLabel item
        r.EventFiringEnabled <- false    

        item.Attachments |> Seq.cast |> Seq.map (fun fileName -> unbox<string> fileName) |> Seq.toList
                         |> Seq.iter (fun fileName -> item.Attachments.Delete(fileName))
        
        use site = properties.OpenWeb()
        let path = String.Format("Versions/{0}/{1}", item.ID, restoreVersion)
        let shadowLibrary = site.Lists.[r.ShadowLibrary] :?> SPDocumentLibrary
        let source = r.CreateFolderPath shadowLibrary path

        source.Files |> Seq.cast |> Seq.iter (fun file ->
            let unboxedFile = unbox<SPFile> file
            item.Attachments.Add(unboxedFile.Name, unboxedFile.OpenBinary()))

        item.SystemUpdate false
        r.EventFiringEnabled <- true

    // member private CreateFolderPath : list:SPDocumentLibrary -> path:string -> SPFolder
    member private r.CreateFolderPath list path : SPFolder =
        r.CreateFolderPathRecursive list.RootFolder (path.Split [|'/'|] |> Array.toList)

    // member private CreateFolderPathRecursive : folder:SPFolder -> pathComponents:string list -> SPFolder
    member private r.CreateFolderPathRecursive folder pathComponents =                
        match pathComponents with
        | [] -> folder
        | head :: tail -> 
            try
                let existingFolder = folder.SubFolders.[head]
                r.CreateFolderPathRecursive existingFolder tail
            with
                :? ArgumentException ->
                    let newFolder = folder.SubFolders.Add head
                    r.CreateFolderPathRecursive newFolder tail

    // member private SetCustomVersionLabel : item:SPListItem -> unit
    member private r.SetCustomVersionLabel item =
        r.EventFiringEnabled <- false
        item.[r.CustomVersion] <- r.GetOfficialVersionLabel item
        item.SystemUpdate false
        r.EventFiringEnabled <- true  

    // member private GetCustomVersionLabel : item:SPListItem -> string
    member private r.GetCustomVersionLabel item =
        item.[r.CustomVersion] :?> string

    // member private GetOfficialVersionLabel : item:SPListItem -> string
    member private r.GetOfficialVersionLabel item =
        item.Versions.[0].VersionLabel

A couple of things to note about the F# implementation: first, it hardly specifies any types. They’re inferred by the compiler. Where type names do appear, it’s mainly because they’re required to unbox elements of an IEnumerable collection. Secondly, F# has flexible self identifiers. Methods must explicitly specify the name of the this reference in C# and use it when accessing members. Thirdly, arguments to general .NET methods are passed as a tuple value, i.e., as comma-delimited arguments surrounded by parenthesis.

This would involve adding or removing event receivers from a list as the user enables or disables versioning. The following extension method is one way to accomplish the addition-part in a type-safe manner:

// definition 
public static class SPListExtensions {
    public static void RegisterEventReceiver<TReceiver>(this SPList list, 
            SPEventReceiverType receiverType, 
            int sequenceNumber) where TReceiver : SPItemEventReceiver {
        var assemblyName = typeof(TReceiver).Assembly.FullName;
        var className = typeof(TReceiver).FullName;

        (from SPEventReceiverDefinition definition in list.EventReceivers
         where definition.Assembly == assemblyName &&
               definition.Class == className &&
               definition.Type == receiverType
         select list.EventReceivers[definition.Id])
        .ToList()
        .ForEach(receiverToDelete => receiverToDelete.Delete());

        var receiver = list.EventReceivers.Add();
        receiver.Type = receiverType;
        receiver.Assembly = assemblyName;
        receiver.Class = className;
        receiver.SequenceNumber = sequenceNumber;
        receiver.Update();
        list.Update();
    }
}

// use
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemAdded, 10000);
list.RegisterEventReceiver<ListAttachmentVersioningEventReceiver>(
    SPEventReceiverType.ItemUpdated, 10001);

Under rare circumstances the (assembly, class, type) tuple may not be unique, i.e., the same receiver may be registered multiple times, albeit with different sequence numbers. In practice I never found any use for this functionality, though, which is why I didn’t include the sequence number in the where clause above, causing all registrations matching the tuple to be removed.

Both SharePoint 2007 and 2010 support versioning for list items but not their attachments. No matter which version of a list item I look at, its attachments will always be the most recent. The attachment support seems to have been bolded on as an afterthought, resulting in behavior that’s counter-intuitive for developers as well as end-users. With SharePoint 2007 (and 2010), Microsoft suggests using a document library for proper attachment versioning. But I can’t substitute one with the other, since a list item may hold any number of attachments and an item in a document library may hold just one.

Existing solution

I counted on someone else having experienced a similar pain and come up with a workable cure. But except for Tim Ebenezer the search come up empty. Tim on the other hand has done a great job of seamlessly integrating his attachment versioning feature into SharePoint. When I activate the feature on a site, it adds a versioning menu item to the list settings page for every list on the site. Unfortunately the core versioning logic, storing attachments in a shadow library using an event receiver, isn’t particularly robust. Among other use cases, it doesn’t properly deal with a user first deleting an attachment and then, some versions later, adding an attachment with the same name.

I therefore set out to implement my own solution based on Tim’s ideas, hooking into the synchronous ItemAdding, ItemUpdating, ItemAttachmentAdding, and ItemAttachmentDeleting events and maintaining a shadow library of versions. This approach, however, quickly turned into a painful one. When the synchronous events run, nothing has yet been written to the database – at this stage a new item doesn’t even have its Id set, and merely determining the number of attachments added and how far I’ve come with the processing is tricky.

The next challenge I encountered was that event handlers cannot easily share state across multiple calls because SharePoint creates a new instance of the receiver class for every event handled. Processing multiple attachments require a counter into the array of attachments to keep track of which ones I’d copied to the shadow list. I’d have to resort to some outside-object storage, keeping in mind that the receiver might execute concurrently. But which storage should I use? Session state may have been disabled, and polluting one of the property bags stored in the content database is messy and also not thread-safe.

Overall, with the synchronous approach too much work has to go into tracking the state of the versioning process.

New solution

A synchronous solution is very hard to get right because it’s forced to work at the level of individual attachments. SharePoint doesn’t have a synchronous event that fires after all attachments have been processed. After all, why provide such an event when everything has already happened? Thinking instead in terms of the asynchronous events of ItemUpdated and ItemAdded, I have exactly what’s needed to snapshot all attachments in one batch, making versioning a lot simpler. When these events fire the item and its attachments have already been written to the database and I can focus on how to generate the snapshots — copying attachments back and forth between lists — and not worry about what the user actual did to the attachments from one version to the next.

// Prerequisites:
// 1. Create a Document Library named ShadowLibrary on the same site as the list to version
// 2. Add a row named CustomVersion of type string to the list to version
public class ListAttachmentVersioningEventReceiver : SPItemEventReceiver {
    private const string CustomVersion = "CustomVersion";
    private const string ShadowLibrary = "ShadowLibrary";

    public override void ItemAdded(SPItemEventProperties properties) {
        base.ItemUpdated(properties);
        SetCustomVersionLabel(properties.ListItem);
        CreateSnapshot(properties);
    }

    public override void ItemUpdated(SPItemEventProperties properties) {
        base.ItemUpdated(properties);

        var item = properties.ListItem;
        if (RollbackHappened(item)) {
            RestoreSnapshot(properties);
            SetCustomVersionLabel(item);
            CreateSnapshot(properties);
        }
        else {
            CreateSnapshot(properties);
            SetCustomVersionLabel(item);
        }
    }

    private void CreateSnapshot(SPItemEventProperties properties) {
        using (var site = properties.OpenWeb()) {
            var item = properties.ListItem;
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var path = string.Format("Versions/{0}/{1}", item.ID, GetOfficialVersionLabel(item));
            var shadowFolder = CreateFolderPath(shadowLibrary, path);

            foreach (string fileName in item.Attachments) {
                SPFile existingFile = item.ParentList.ParentWeb.GetFile(item.Attachments.UrlPrefix + fileName);
                SPFile newFile = shadowFolder.Files.Add(fileName, existingFile.OpenBinaryStream());
                newFile.Item.Update();                    
            }
        }
    }

    private bool RollbackHappened(SPListItem item) {
        var culture = CultureInfo.InvariantCulture;
        var currentVersion = float.Parse(GetOfficialVersionLabel(item), culture);
        var lastVersion = float.Parse(GetCustomVersionLabel(item), culture);
        return currentVersion > lastVersion + 1;
    }

    private void RestoreSnapshot(SPItemEventProperties properties) {
        var item = properties.ListItem;
        var restoreVersion = GetCustomVersionLabel(item);
        EventFiringEnabled = false;

        item.Attachments.Cast<string>().ToList().ForEach(attachment => item.Attachments.Delete(attachment));
        using (var site = properties.OpenWeb()) {
            var path = string.Format("Versions/{0}/{1}", item.ID, restoreVersion);
            var shadowLibrary = site.Lists[ShadowLibrary] as SPDocumentLibrary;
            var source = CreateFolderPath(shadowLibrary, path);

            foreach (SPFile file in source.Files)
                item.Attachments.Add(file.Name, file.OpenBinary());
        }

        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    // can only get folder creation to work with Document Libraries
    private SPFolder CreateFolderPath(SPDocumentLibrary list, string path) {
        return CreateFolderPathRecursive(list.RootFolder, path.Split('/').ToList());
    }

    private SPFolder CreateFolderPathRecursive(SPFolder folder, IList<string> pathComponents) {
        if (pathComponents.Count == 0)
            return folder;

        SPFolder newFolder;
        try {
            newFolder = folder.SubFolders[pathComponents.First()];
        }
        catch (ArgumentException) {
            newFolder = folder.SubFolders.Add(pathComponents.First());
        }

        pathComponents.RemoveAt(0);
        return CreateFolderPathRecursive(newFolder, pathComponents);
    }

    private void SetCustomVersionLabel(SPListItem item) {
        EventFiringEnabled = false;
        item[CustomVersion] = GetOfficialVersionLabel(item);
        item.SystemUpdate(false);
        EventFiringEnabled = true;
    }

    private string GetCustomVersionLabel(SPItem item) { return item[CustomVersion] as string; }
    private string GetOfficialVersionLabel(SPListItem item) { return item.Versions[0].VersionLabel; }
}

When a list item is saved, I take a snapshot of the attachments, storing them in a folder structure like {Id}/{VersionNumber}/{Attachments} in the shadow document library. When a list item is restored to a previous version, existing attachments are first deleted before the ones from the snapshot are added back in, creating a new version of the list item.

Restoring previous versions also has a counter-intuitive meaning in SharePoint. Suppose in one version of a list item, I store a key in the item’s property bag, then I’d expect the property bag values to be specific to this version. But behind the scenes restore seems to work by cloning the current version and then copying only the values of the fields from the restore version to the new one. In other words, I can’t use the item’s property bag to store version specific information, such as a version tag to detect when a restore has occurred. I also can’t use the Modified field because SharePoint sets it to the time of the restore. To carry over version information I have to create and maintain a field of my own. Hence the CustomVersion field on the list to version.

Remember that because the ItemUpdated and ItemAdded execute asynchronously, all the snapshotting logic executes on a background thread, after control has returned to the user. Should an error occur at this point, the user will never see it and the snapshot may be left in an incomplete state. On the other hand, this approach scales well and doesn’t have to be fast because no user is awaiting the result.

Lastly, there’s one place in SharePoint where the versioning abstraction leaks through. It’s in the list item version dialog which displays older versions and enables restore to any previous version. The dialog will always show the most recent attachments.

Improvements

I could use the ETag property of an SPFile object to implement a more efficient differential snapshotting algorithm that would conserve storage space. Compressing attachments before storing them in the shadow library might also be an option, although then I’d have to promote the ETag value to a shadow library field before compressing.

SPListCollection extensions

In SharePoint 2010 the TryGetList method has been added to the SPListCollection class. The method returns either an SPList instance matching the display name or null. Oftentimes, however, you want to do a lookup based on the internal name. Here’s an extension method that adheres to the semantics of TryGetList, but using the internal name. It relies on the fact that the RootFolder property of a list is actually its internal name:

// definition
public static class SPListCollectionExtensions {
    public static SPList TryGetListByInternalName(this SPListCollection lists, string internalName) {
        return (from SPList l in lists
            where l.RootFolder.Name == internalName
            select l).SingleOrDefault();
    }
}

// use
if (site.Lists.TryGetListByInternalName(internalListName) == null)
   // list not found

SPFieldCollection extensions

Using the CreateNewField method of the SPFieldCollection you can add new fields to a list. The particular annoying aspect of this method, however, is that when you want to continue working with its result, oftentimes you have to cast it to one of the SPField subclasses. But since the SPFieldType, provided as one of the arguments to CreateNewField, closely relates to the actual SPField return type, an extension method is able to do the casting. This’ll expose mismatches at compile time instead of at runtime.

All it takes is for us to map out the relation between SPField and SPFieldType:

// definition    
public static class SPFieldCollectionExtensions {
    public static TSPField CreateField<TSPField>(this SPFieldCollection fields, 
            string internalName, string displayName) where TSPField : SPField {
        var spFieldToFieldType = new Dictionary<Type, SPFieldType> {
            { typeof(SPFieldDateTime), SPFieldType.DateTime },
            { typeof(SPFieldNumber), SPFieldType.Number },
            { typeof(SPFieldUser), SPFieldType.User },
            { typeof(SPFieldBoolean), SPFieldType.Boolean },
            { typeof(SPFieldMultiLineText), SPFieldType.Note },
            { typeof(SPFieldText), SPFieldType.Text }
        };

        var fieldType = spFieldToFieldType[typeof(TSPField)]; 
        var list = fields.List;
        var field = list.Fields[list.Fields.Add(internalName, fieldType, false)];
        field.Title = displayName;
        field.Update();
        return field as TSPField;
    }
}

// use
l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName");

Taking the CreateField extension method one step further, oftentimes you want to set properties besides internal name and display name. For that purpose I’ve defined a CreateField method that accepts an Action<TField>. This allows you to reuse common property settings across fields for brevity and consistency while at the same time maintaining strong typing.

// definition
public static TSPField CreateField<TSPField>(this SPFieldCollection fields, 
        string internalName, string displayName, 
        Action<TSPField> setAdditionalProperties) where TSPField : SPField {
    var newField = CreateField<TSPField>(fields, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
public static Action<SPFieldMultiLineText> RichTextProperties = f => {
    f.RichText = true;
    f.RichTextMode = SPRichTextMode.FullHtml;
};

l.Fields.CreateField<SPFieldBoolean>(internalName, "displayName", f => f.Required = true);
l.Fields.CreateField(internalName, "displayName", RichTextProperties);

With the Comment field, you can leave out the type argument because the compiler infers it based on the type of the Action delegate.

Similar to CreateField, I’ve defined two additional extension methods for creating lookup fields:

// definition
public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields, 
        string lookupListName, string internalName, 
        string displayName) where TSPField : SPFieldLookup {
    var currentList = fields.List;
    var lookupList = currentList.ParentWeb.Lists.TryGetListByInternalName(lookupListName);
    var newField = currentList.Fields[currentList.Fields.AddLookup(internalName, lookupList.ID, false)];
    newField.Title = displayName;
    newField.Update();
    return newField as TSPField;
}

public static TSPField CreateLookup<TSPField>(this SPFieldCollection fields, 
        string lookupListName, string internalName, string displayName, 
        Action<TSPField> setAdditionalProperties) where TSPField : SPFieldLookup {
    var newField = CreateLookup<TSPField>(fields, lookupListName, internalName, displayName);
    setAdditionalProperties(newField);
    newField.Update();
    return newField;
}

// use
l.Fields.CreateLookup<SPFieldLookup>(lookupListName, internalName, displayName, f => f.AllowMultipleValues = true);

These extension methods makes using the SharePoint API more type-safe and concise, and defining lists using these methods and the template approach saves me from writing a lot of repetitive code.

New-SPWeb -url $base_url -addtoquicklaunch -template "STS#1" -name "Base"
New-SPWeb -url $base_url/Dk -addtoquicklaunch -template "STS#1" -name "Dk"
New-SPWeb -url $base_url/Uk -addtoquicklaunch -template "STS#1" -name "Uk"
New-SPWeb -url $base_url/Dk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"  
New-SPWeb -url $base_url/Dk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"
New-SPWeb -url $base_url/Uk/BusinessUnit1 -addtoquicklaunch -template "STS#1" -name "BusinessUnit1"
New-SPWeb -url $base_url/Uk/BusinessUnit2 -addtoquicklaunch -template "STS#1" -name "BusinessUnit2"

$dk = Get-SPWeb $base_url/Dk
$dkBusinessUnit1 = Get-SPWeb $base_url/Dk/BusinessUnit1
$dkBusinessUnit2 = Get-SPWeb $base_url/Dk/BusinessUnit2

$locale = [System.Globalization.CultureInfo]::CreateSpecificCulture("da-DK")
$dk.Locale = $locale
$dk.Update()

$dkBusinessUnit1.Locale = $locale
$dkBusinessUnit1.Update()
  
$dkBusinessUnit2.Locale = $locale
$dkBusinessUnit2.Update()

How can we reduce this repetition? Well, it repeats because we’ve made the common mistake of mixing logic with data. One way to separate the two is using a table-driven approach, i.e., extract the varying parts, the data, into a table and rewrite the logic so it’s parameterized by the table. After some trial and error with PowerShell here’s the code I ended up with:

$sites = @( @("", "Base"), 
            @("Dk", "Dk", "da-DK"),
            @("Uk", "Uk"),
            @("Dk/BusinessUnit1", "BusinessUnit1", "da-DK"),
            @("Dk/BusinessUnit2", "BusinessUnit2", "da-DK"),
            @("Uk/BusinessUnit1", "BusinessUnit1"),
            @("Uk/BusinessUnit2", "BusinessUnit2") )

$sites | foreach {
  $url, $title, $culture = $_
  $newSite = New-SPWeb -url "$base_url/$url" -addtoquicklaunch -template "STS#1" -name $title
    
  if ($culture -ne "") {
    $locale = [System.Globalization.CultureInfo]::CreateSpecificCulture($culture)
    $newSite.Locale = $locale
    $newSite.Update()
  }
}

So far so good. The un-installation part of the script, however, follows the same pattern of repetition:

Remove-SPWeb -identity $base_url/Uk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit2 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk/BusinessUnit1 -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Dk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url/Uk -confirm:$false -ErrorAction SilentlyContinue
Remove-SPWeb -identity $base_url -confirm:$false

But now that we have the table of sites at hand, we can easily make the site removal table-driven as well:

function Reverse($array) {
  $clone = $array.Clone()
  [Array]::Reverse($clone)
  $clone
}

Reverse($sites) | foreach {
  $url = $_[0]  
  Remove-SPWeb -identity "$base_url/$url" -confirm:$false -ErrorAction SilentlyContinue
}

I’m sure the code could be written more elegantly, but given my working knowledge of PowerShell I’m satisfied with the result. I especially like the functional programming style.

• Use an Inversion of Control container (IoC) like Windsor to create an architecture that responds well to change, i.e., an architecture that promotes looser coupling and higher cohesion
• The ability to write unit tests against a code base is a good measure of its degree of coupling
• By having components depend on interfaces, you’re free to switch the implementation at runtime, introducing flexibility into the software
• Avoid having concrete components talk to each other. It makes it hard for the two to vary independently, e.g., adding logging later can only be done by tearing components apart and putting them back together
• At runtime the IoC container recursively composes these smaller components into an object graph, e.g., by passing concrete implementations through the constructors to satisfy the dependencies
• One alternative to using an IoC container is using a service locator. Each component’s constructor would then ask the locator for its dependent components. While this approach works, it makes testing components in isolation difficult because all dependencies are now locked away inside the service locator. Instead, by having all dependencies supplied in the constructor, unit tests can directly supply fakes implementations
• When software is only used in one environment it tends to be fairly inflexible because it’s only suited for that one purpose. As soon as you design it to be used in two or more environments it becomes more flexible. Thing of different environments as unit test, staging, production, and so on
• The idea is for components to not have to be modified depending on the where they’re running. It’s the IoC container that dynamically tie components together based on the environment
• Suppose you didn’t use a IoC container. Then ultimately the top-level object would have to pass concrete instances down the chain. The top-level object may well be the UI layer. But having the UI layer know about data access components, service components, and logging components isn’t ideal. Instead use an IoC container, which is nothing more than a factory for components
• With the Windsor IoC container (and most others), the usage pattern typically involves three stages
  • Register: tie together interfaces with concrete implementations
  • Resolve: return concrete implementations of interfaces required to satisfy the dependencies of an object
  • Release: dispose of concrete implementations
• Most people only use a fraction of the functionality of an IoC container. Instead of taking dependency of a full-blown container, you could easily roll your own IoC container that implements the core functionality in a few dozen lines of code — in an associated .NET Rocks and DnrTV episode James Kovacs elaborates on his original article
• The goal of using a container isn’t to get rid of calls to the new operator. It’s to not use the new operator for the parts of an applications where flexibility is required
• Simple forms of Aspect Oriented Programming are possible using interceptors
• Windsor supports the decorator pattern so you can have one component wrap another at runtime. This lets you implement features such as logging without actually modifying the original component. It’s an example of adhering to the open/close principle by which classes should be open for extension, but closed for modification. In other words: don’t edit the original source code to introduce new behavior. Instead use methods of composition
• Avoid configuring Windsor through XML and instead use the fluent interface — perhaps in a separate assembly so only it needs to be redeployed when the configuration changes. When a value, such as a connection string needs to be configurable, create a section in app.config and add the value there. Windsor will know how to locate the value when instantiating objects

• Enterprise service bus
  • An architectural pattern rather than a specific technology or product
  • The ethernet of SOA
• Types of service bus
  • Centralized: a broker more than a bus, e.g., BizTalk
  • Decentralized: a real service bus, e.g., NServiceBus, Mass Transit, Rhino Service Bus, MS .NET Service bus
• The current version 2.0 of NServiceBus is free whereas the binary version 2.5 will contain limitations. The limitations can be disabled by downloading and compiling the source code yourself
• NServiceBus is designed to address the fallacies of distributed computing
  • It primarily addresses the fallacies through the use of reliable one-way messaging
  • Reliable means messages are stored in a queue, such a MSMQ, which is part of all recent Windows installations. The assumption being that the queue is always present and ready to receive messages
  • One-way means messages are asynchronous and fire-and-forget. The destination service doesn’t have to be online when a message goes into its queue, which makes services temporally independent. At some point in the future the service may post a reply to the client’s queue
  • Messaging means that any action is accomplished by posting a self-contained message, containing the type of operation and its parameters, to the queue of a service
  • With NServiceBus, a class is marked with an interface to describe that it’s a type of message. Instances of the class are then serialized using a standard XML or binary serializer and stored in a queue
• NServiceBus service != traditional web service
  • An NServiceBus service is a class that implements an interface
  • Using a message queue is more reliable when the service has to make calls to other services as part of its operation. With a traditional web service, calling other services from within makes the service slower and less reliable. If only one dependent service is down it’s like with old Christmas lights wired in series: the entire stack may be down. Queues on the other hand introduce a save place, a stabilization point, to temporarily store messages
  • A service is typically configured with an input queue that identifies it to others and an error queue that messages are routed to after some number of retries. In addition, each service may be configured with a number of threads to use to parallelize message processing

• Windbg isn’t a replacement for VS, but VS doesn’t handle some advanced cases
• Windbg is a free user mode/kernel mode debugger which is part of the Debugger Tools for Windows
• Customers may not be happy installing VS to debug code in production since it installs a lot of components and requires restarts
  • Windbg requires only a simple installation once you extract the redistributable from the Debugger Tools for Windows
• Loading SOS.dll from the Microsoft .NET folder into Windbg makes it understand .NET
  • With SOS.dll loaded, you can look into the CLR and its data structures
  • Make sure to load the right SOS.dll for your runtime
  • SOS.dll is also available for the Silverlight runtime
• Debugger Markup Language support is available for version 4 of SOS.dll
  • Provides hyperlinks in the command output of SOS
• Like with the VS debugger, you can insert Debug.Break() in your demo app and have Windbg halt on it
  • Release builds can be debugged with Windbg
  • Release builds also contain symbols. Like with debug builds, symbols are stored in a separate file
  • Release builds make debugging harder because the jitter kicks in and modifies the code
  • The 64 bit calling convention of passing arguments via registers makes it harder to locate information when debugging
• Windbg generally needs symbols loaded, although it’s less important when debugging managed code
  • Use MS’ public symbol server to load symbols on demand
  • Set the _NT_SYMBOL_PATH environment variable to point to your symbols (will affect VS as well)
  • Or use the .symfix command from within Windbg
• Popular extensions to Windbg: SOSEX and Psscor2 (replacement for SOS, useful for ASP.NET debugging)
• Create dump file to analyze: use task manager or ADPlus or ProcDump from Sysinternals, which can dump based on triggers
• ADPlus collects crash dumps or hang dumps
  • Hang dumps can be captured from the same process multiple times and may be useful when debugging deadlocks or resource leaks
  • When you capture a hang dump, the process is halted for the dump period and is then restarted
• A *32 process in task manager is actually a 64 bit process when you dump it
  • WoW64.dll is involved when dumping from the task manager
  • Not what you typically want because you don’t get full access to 32 bit process information
• A .NET application is hosted within the CLR which is itself hosted within a regular Windows process
  • Looking at memory usage with the task manager therefore doesn’t tell you much about the .NET part

To set the stage, I’ve created an array of people to query.

Person[] people = new[] { new Person { Age = 1, ...}, ... };

It’s worth noting that all arrays implicitly derive from the Array class. It’s a special class that only the compiler and the runtime may derive from. Doing so yourself is rewarded with a compiler error stating simply that the class cannot be derived from. What’s also not apparent from the definition of the Array class is that, as of .NET 2.0, it also implements IList<T>, ICollection<T>, and IEnumerable<T>, which derive from the corresponding non-generic interfaces.

public abstract class Array :
    ICloneable, IList, ICollection,
    IEnumerable, IStructuralComparable, IStructuralEquatable

The runtime takes special care of extending the Array type with implementations of the generic interfaces based on the type of object that the array stores. That’s why the interfaces are invisible in the documentation. Any array of type T — in this case Person — will therefore implement, among others, the IEnumerable<T> interface. It’s the implementation of this interface on a class that enables LINQ to Objects to query any array or collection.

Query expression syntax

Let’s create a simple query using query expression syntax. To support it, additional keywords have been added to C# to express common query operators. Examples of what’s not supported with query expression syntax are the Sum, Take, and Skip query operators. In those cases you can combine query expression with method invocation syntax or write everything using the latter.

var q = from p in people
        where p.Age > 25
        select p;

Method invocation syntax

To make matters transparent, let’s not rely on type inference and the var keyword for typing the result. At the same time, let’s assume the role of the compiler and translate the query expression syntax into method invocation syntax with lambda expressions. Each standard query operator, such as Where and Select, translates to a corresponding method invocation on the collection, oftentimes with a lambda expression as the argument. Both representations are semantically equivalent, but for this simple query method invocation syntax appears more complex. This is generally not the case for either representation as you can try out with a tool like LINQPad.

IEnumerable<Person> r = people
    .Where(p => p.Age > 25)
    .Select(p => p);

The concise nature of the code is due to the compiler inferring the type arguments to Where and Select and the type of the lambda expression. In this case, because the collection stores objects of type Person, the type arguments of the generic methods as well as the type of the lambda expression is also of type Person. To make these types explicit, you can substitute real types for the generic ones in the definition of Where and Select provided later.

IEnumerable<Person> s = people
    .Where<Person>((Person p) => p.Age > 25)
    .Select<Person, Person>((Person p) => p);

Generic delegates

With the type specifications made explicit, it’s more obvious how lambda expressions are compatible with delegates. A lambda expression is nothing more than a short-hand notation for a delegate, a type-safe pointer to a piece of code, to be passed into a method. Hence, a lambda expression can be substituted with an anonymous delegate by wrapping it in additional ceremony.

IEnumerable<Person> t = people
    .Where<Person>(delegate(Person p) { return p.Age > 25; })
    .Select<Person, Person>(delegate(Person p) { return p; });

To make delegates type safe, their definition include the return type and the types of the arguments passed into it. This is unfortunate since the standard query operators must be able to work on any type of object. LINQ therefore relies on generic delegates in the definition of its operators. Like with other generic types, the compiler and runtime work together to generate real delegates based on the specified, or inferred, return type and types of arguments. Delegates like the ones below for Where and Select are what’s generated by the runtime.

// delegate Boolean WhereDelegate(Person p);
// delegate Person SelectDelegate(Person p);
bool WhereClause(Person p) { return p.Age > 25; }
Person SelectClause(Person p) { return p; }

IEnumerable<Person> u = people
    .Where<Person>(WhereClause)
    .Select<Person, Person>(SelectClause);

LINQ relies on a set of generic delegates defined in the .NET framework. These delegates come in two flavors: those that return void, named Action, and those that don’t, named Func. You can use these delegates in your own code to not only parameterize methods by value but by functionality.

delegate void Action();
delegate void Action<T>(T obj);
delegate void Action<T1, T2>(T1 arg1, T2 arg2);
// up to eight arguments

delegate TResult Func<TResult>();
delegate TResult Func<T, TResult>(T arg);
delegate TResult Func<T1, T2, TResult>(T1 arg1, T2 arg2);
// up to eight arguments

Extension methods

Each of the 50 or so standard query operators is defined on the Enumerable class with the this modifier on their first IEnumerable<T> argument. This makes them extension methods to every object that implements IEnumerable<T>. From the definition of the Where operator below, it then follows that when you write "Where(p => p.Age > 25)", the compiler infers that because Where is called on a collection of type Person, p must also be of type Person. Furthermore, because the result of the comparison is a bool, the lambda expression is compatible with a delegate that accepts a type Person and returns a type bool. In other words, the matching Where has a signature like the commented one below.

The purpose of the Where operator is to filter a collection based on a predicate, the mathematical term for a function that returns true or false. The people that satisfy the predicate is therefore returned as an IEnumerable<Person>. However, the yield keyword adds an interesting twist to the return of Where and Select and other operators that return IEnumerable<T>. Instead of the operator iterating the source collection and returning every element of the result at once, the yield keyword instructs the compiler to emit a state machine so the operator can keep track of how far through the source collection it is, and return one resulting element at a time. The effect is lazy evaluation of LINQ to Objects queries.

In the example, the output of Where goes into Select whose output may goto the console using a foreach. To understand yield, think of the foreach that writes to the console as pulling on a string of objects connecting Select to Where to the source collection. Each iteration of the loop pulls the string just enough to get at the next resulting element. This may in turn require Where to iterate multiple times until the predicate is once again satisfied.

// IEnumerable<Person> Where<Person>(
//     IEnumerable<Person> source,
//     Func<Person, bool> predicate)

public static IEnumerable<TSource> Where<TSource>(
    this IEnumerable<TSource> source,
    Func<TSource, bool> predicate) {
    foreach (TSource element in source) {
        if (predicate(element))
            yield return element;
    }
}

The Select operator is an example of one whose input type may differ from its output type. By its very nature, it projects one type onto another. It can even project onto an anonomous type, like in "Select(p => new { x = p.Name })", but then the var keyword must be used for its return type. In the example I project from Person to Person, which makes the matching signature of Select like the commented one below.

// IEnumerable<Person> Select<Person, Person>(
//     this IEnumerable<Person> source,
//     Func<Person, Person> selector)

public static IEnumerable<TResult> Select<TSource, TResult>(
    this IEnumerable<TSource> source,
    Func<TSource, TResult> selector) {
    foreach (TSource element in source)
        yield return selector(element);
}

The ability to chain methods together to form a data pipeline is what adds real power to LINQ. Although the idea of chaining methods together is old, the traditional approach of each method returning a new or mutated instance works best for objects of your own. Imagine adding to the IEnumerable<T> interface a set of query operators. Then every collection would have to provide implementations for Where, Select, and so on. Also, it wouldn’t be possible to add new operators in vNext of .NET without breaking existing code implementing the interface.

Extension methods solve this issue by providing the necessary syntactic sugar to make chaining of methods on a collection feel like chaining on an object of your own. Behind the scenes, the compiler rewrites calls to extension methods to static methods.

var v = Enumerable.Select(
    Enumerable.Where(people, p => p.Age > 25),
    p => p);

Note how, without the use of extension methods, operations must be specified in the opposite order in which they’re executed. This doesn’t read as nicely as the “infix” syntax when many operations are chained together.

Although Visual Studio, in tandem with the MSBuild engine, generally takes good care of compilation, I rarely want to rely on it solely. Instead, I’d prefer that any developer task be easily run from the command-line. This ensures that no magic is going on, that the task is kept simple and flexibility, and that it doesn’t rely on Visual Studio to work. The challenge, however, is which of the many tools available to pick. It must be one that’s flexible enough to meet most requirements with relative ease or the automation will likely not be a valid documentation medium in itself.

Why not to use Windows batch files

As a general example, consider the deployment of a SharePoint 2007 solution. With SharePoint a good deal more than compiling is needed to bring new functionality into a SharePoint installation. Whereas Visual Studio and MSBuild may still compile the code and WSPBuilder create the WSP installation package, both from within Visual Studio and from the command-line, getting everything setup cries for automation, even locally. A common approach (possibly inspired by popular SharePoint 2007 literature) is that of the batch file applying various operations to a WSP. For the sake of brevity, I’ve kept the example short, but imagine extending it with a check-out source code task, a compilation task, a WSPBuilder task, and a feature deactivation and activation task, and possibly a warm-up task. Add to this a master script that orchestrate the whole thing. In the end, I’d end up with scripts that become increasingly harder to maintain as they grow in number and size.

@set STSADM="c:\program files\common files\microsoft shared\web server extensions\12\bin\stsadm"

if "%1" == "uninstall" goto uninstall
if "%1" == "install" goto install
if "%1" == "" goto reinstall

:uninstall
    %STSADM% -o retractsolution -name Foo.wsp -immediate
    %STSADM% -o execadmsvcjobs
    %STSADM% -o deletesolution -name Foo.wsp -override
    goto end

:install  
    %STSADM% -o addsolution -filename Foo.wsp
    %STSADM% -o deploysolution -name Foo.wsp -immediate -allowGacDeployment -force
    %STSADM% -o execadmsvcjobs
    goto end

:reinstall
    call Foo uninstall
    call Foo install
    goto end
	
:end

Still, because of the ubiquitous nature of command.com and cmd.exe, the batch file interpreters, batch files are everywhere. Regardless that the technology is a left-over from the days of MS DOS and haven’t evolved much since. Not only are the branching and looping constructs limited, so are the available commands. Suppose I want to find out if a command was indeed successful. This turns out to be really hard when all I have to work with is the errorlevel of the most recently executed command. Assuming the command adheres to the errorlevel convention, for the script to fail as early and as close to the real error as possible, I’d have to inspect the property after each command, causing the batch file to grow quite verbose. Sadly, batch files lack the equivalent of set -o errexit of Bash, where the interpreter checks for success after each command and aborts immediately on error. Relying solely on the errorlevel is oftentimes insufficient anyway. To determine success, I’d typically have to parse actual command output or inspect some system property by downloading additional commands or building my own.

Essential vs. incidental requirements

Unless I plan to sit idle and stare at the screen and be a human error detector while the batch file run, I think it’s safe to say that it’s mostly unsuitable for developer automation. Hence in late 2005 I rolled my own automation tool in Python. With Python or Ruby or a similar dynamic language acting as the glue that ties everything together, almost any task can be automated in a robust way. Of course, I could also automate with a static language like C# (it’s surprisingly common). But for script-like tasks, I don’t particularly fancy the long cycle of editing source code in Visual Studio, compiling it, deploying it, and having a hard time debugging it in environment without Visual Studio. A dynamic language, on the other hand, short-circuits the development cycle and allows for interactive programming through a REPL.

With a dynamic language that interacts with .NET, such as IronPython, IronRuby, or Powershell, possibly with supporting DSLs like Paver, Rake, or psake on top, the need for writing custom commands to interact with the operating system or the application almost vanishes. The question, then, is which of the dynamic languages to go with when at their technical core they’re so much alike. Besides sharing the concept of a REPL, the notion of a tuple, a list, a map, and operations on each are baked into their syntax, making code quite terse. It even makes it convenient to express any configuration in the language itself and not in XML where I’d first have to come up with a schema, and then create an instance of it before parsing it. On Windows, however, Powershell is gradually becoming the next ubiquitous interpreter, with applications shipping with cmdlets, whereas IronPython or IronRuby is a separate download.

Conclusion

No matter the tool, what I end up doing is defining tasks, form dependencies between tasks, and have the tool execute tasks in an order that satisfies their dependencies. As the tool traverses the dependency graph and executes tasks, it’s up to each task to detect success, and up to the tool to report on progress. A good tool is therefore characterized by the ease with which I can express these things, the available language constructs, the ease of debugging, and the tool’s ability to converse in foreign languages.

The book tells the story of 15 prominent coders by asking each coder roughly the same questions. After covering their initial exposure to programming, it moves on to how they design and debug programs and whether or not they use literate programming or design by contract or more formal proof techniques. In my opinion, each interview varies too little and is backed by too many anecdotes. I’d hoped for more universal wisdom and advice that would transcend time and technologies.

Perhaps not too surprisingly, during the interviews you learn how many of the coders got into the field entirely by chance. Simon Peyton Jones, a creator of Haskell, was drawn to electrical and software engineering because he found math too hard and competitive. Others were lucky enough that their school happened to have a mainframe sitting in the basement for them to tinker with. On the whole, computers simply sparked an early and lifelong passion.

Another common coder’s trait seems to be that of code reading. As an individual developer, you should improve by studying real code of every quality. As a team of developers, you should improve by shared code reviews of what you’re building. Like no other technique, studying and reviewing code will help share knowledge and improve general developer skills. Yet, the typical organization keeps to in-house presentations on the vNext features of some product and calls it knowledge sharing for developers.

I ought to do more a la Scott Hanselman’s weekly source code. When taking on C++ and MFC in the late nineties, I remember reading the MFC sources and admiring how well it applied design patterns and design by contract to the C-based Windows API. That code base alone had a profound impact on my understanding of object-orientation and how to write code.

Why traditional dependency-breaking techniques come short

After watching a screencast on how to use Moles to unit test LINQ to SQL without hitting the database, I thought it would be interesting to do the same with TypeMock. But first, let’s make sure we understand why traditional dependency-breaking techniques come short in testing LINQ to SQL. Assuming we want to put a repository under test, our goal is to mock how it accesses the database. Here’s a simple implementation of a repository that queries the Employee table of the AdventureWorks database:

public class EmployeeRepository {
    public List<Employee> GetEmployeesByHireDate(DateTime start, DateTime end) {
        using (var ctx = new AdventureWorksDataContext())
            return (from e in ctx.Employees
                    where e.HireDate >= start && e.HireDate <= end
                    select e).ToList();
    }
}

All calls to the database are routed through the AdventureWorksDataContext generated by Visual Studio. To mock access to the database, we therefore have to mock part of the data context. Easier said than done, though, for the context doesn’t expose an interface that a fake can implement. In addition, the tables are accessed through properties on the context that return a type of Table<TEntity>. Unfortunately, the constructor of Table<TEntity> is internal and the class itself is sealed, eliminating the hope of instantiating or subclassing the type by traditional means:

public sealed class Table<TEntity> : IQueryProvider, 
        ITable, IListSource, ITable<TEntity>, IQueryable<TEntity>, 
        IEnumerable<TEntity>, IQueryable, IEnumerable
        where TEntity : class {
    internal Table(DataContext context, MetaTable metaTable) {
        ...
    }
}

For an example of how the data context itself creates an instance of Table<TEntity>, take a look at the Employees property on the AdventureWorksDataContext. It relies on the GetTable<Employee> method on the DataContext class to create an instance of Table<Employee>. Despite its constructors being internal, the GetTable<TEntity> method has no trouble constructing an instance of the Table<TEntity> type, as they both reside in the System.Data.Linq assembly:

public partial class AdventureWorksDataContext : DataContext {
    public Table<Employee> Employees {
        get {
            return GetTable<Employee>();
        }
    }
}

How to break the unbreakable

The design of LINQ to SQL leaves us short of a traditional testing seam, as Michael Feathers would phrase it; a place at which we can alter the behavior of a program without editing in that place. This explains why, with LINQ to SQL, traditionally we’ve had to test against a real database with all its constraints, making our tests brittle, slow, and painful to write and maintain. With the new breed of mocking frameworks the issues of not being able to subclass or not being able to call an internal constructor go away (and new issues take their place). Regardless, here’s how to write a unit test for the CustomerRepository that doesn’t hit the database:

[TestClass]
public class CustomerRepositoryTest {
    private EmployeeRepository _repository;

    [TestInitialize]
    public void Initialize() {
        _repository = new EmployeeRepository();

        var fakeEmployees = new List<Employee> {
            new Employee {EmployeeID = 1, HireDate = new DateTime(2004, 12, 1)},
            new Employee {EmployeeID = 2, HireDate = new DateTime(2006, 7, 1)},
            new Employee {EmployeeID = 3, HireDate = new DateTime(2009, 3, 1)}
        }.AsQueryable();

        var fakeDataContext = Isolate.Fake.Instance<AdventureWorksDataContext>();
        Isolate.Swap.NextInstance<AdventureWorksDataContext>().With(fakeDataContext);

        // var fakeEmployeeTable = Isolate.Fake.Instance<Table<Employee>>();
        // Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturn(fakeEmployeeTable);
        // Isolate.WhenCalled(() => fakeEmployeeTable).WillReturnCollectionValuesOf(fakeEmployees);
        // or by transitivity
        Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturnCollectionValuesOf(fakeEmployees);
    }
        
    [TestMethod]
    public void GetEmployeesByHireDate_should_return_hires_from_2008_until_present() {
        var employees = _repository.GetEmployeesByHireDate(new DateTime(2008, 1, 1), DateTime.Now);
        Assert.AreEqual(1, employees.Count());
        Assert.AreEqual(3, employees[0].EmployeeID);
    }
}

The test method itself looks exactly as if we’d been testing against a real database. The difference lies in the Initialize method, where we setup the fake data context and database contents. We instruct TypeMock to return the fake context in place of the real one inside EmployeeRepository. And whenever someone calls the Employees property on the fake context, we have TypeMock intercept the call and return a fake collection of type IQueryable<Employee>. We could’ve returned an instance of Table<Employee>, which implements IQueryable<Employee>, but in this case returning the collection is simpler and sufficient. Had we had more methods on our repository, we likely would’ve added additional rows to the Employee table and populated more of its columns.

I found the given-expect pattern useful in testing a piece of code that I was working on this week. I was refactoring and adding tests around an ASP.NET control adapter that makes SharePoint 2007 pages more XHTML compliant. I wanted to reuse the transformations outside the control adapter and hence ended up moving the transformation logic to a new class. It accepts possibly malformed HTML and relies on heuristics of the HTML Agility Pack to build a DOM off of it. I can then query the DOM, looking for known violations, and patch them before returning XHTML to the caller.

public class HtmlToXHtmlTransformer {
    private readonly HtmlDocument _document;

    public HtmlToXHtmlTransformer(string html) {
        _document = new HtmlDocument();
        _document.DetectEncoding(new StringReader(html));
        _document.LoadHtml(html);
    }

    private void Transform(string xpath, Action<HtmlNode> nodeMatch) {
        var nodes = _document.DocumentNode.SelectNodes(xpath);
        if (nodes != null)
            foreach (var node in nodes)
                nodeMatch.Invoke(node);
    }

    private void FixDuplicateBorderAttributeOnSPGridViewControl() {
        Transform("//table[count(@border)=2]", node => node.Attributes.Remove("border"));
    }

    public string Transform() {
        FixDuplicateBorderAttributeOnSPGridViewControl();
        _document.OptionWriteEmptyNodes = true;
        return _document.DocumentNode.WriteTo(); 
    }        
}

The complete HtmlToXHtmlTransformer collects a dozen transformations. Its Transform method is what we want to call with various HTML fragments to verify that they come out as XHTML. For this purpose, we might do the tests as Visual Studio data-driven tests that read their input and output from a text file. But in most cases I prefer traditional tests, so I can describe the purpose of a test with a descriptive method name and possibly a comment.

     
[TestClass]
public class HtmlToXHtmlTransformerTest {
    private string _result;

    [TestMethod]
    public void Must_selfclose_nodes_when_allowed() {
        Given("<br>");
        Expect("<br />");
    }

    [TestMethod]
    public void Must_remove_duplicate_border_on_SPGridView_control {
        Given(@"<table border=""0"" border=""0""></table>");
        Expect(@"<table border=""0""></table>");
    }

    private void Expect(string xhtml) {
        Assert.AreEqual(xhtml, _result);
    }

    private void Given(string html) {
        var transformer = new HtmlToXHtmlTransformer(html);
        _result = transformer.Transform();
    }
}

I particularly like the clarity of the given-expect pattern and find that for a reasonable number of tests it’s a viable alternative to data-driven test. I do, however, recognize the value of data-driven tests in situations where a non-developer wants to test a class. Though at the unit test level I’ve never experienced this. It’s more characteristic of FitNesse for acceptance testing. However you unit test, just make sure your tests run with a minimum of effort on your part and that they run fast.

Code equals data

Configuring the editor is done by placing Emacs Lisp code in the .emacs file. Whenever Emacs starts, it looks for this file in the directory specified by the HOME environment variable and executes its content. Unlike the XML configuration files of .NET and Java, with Emacs there’s no intermediate configuration format. There’s no extra language to learn and no limitation in the expressiveness of the configuration language. With this interchangeability of code and data, every behavior of Emacs can be hooked into and tailored and every global variable redefined.

Starting Emacs from the Windows taskbar

On my machine, I like everything Emacs-related to reside in the directory that I extracted Emacs to, yet I want to be able to start Emacs from the taskbar. I do this by pinning cmd.exe to the taskbar and modifying its destination so it defines HOME for this instance before spawning Emacs. This way the HOME of Emacs doesn’t conflict with the HOME of other applications.

    cmd.exe /C "set HOME=...\emacs-23.1&&...\emacs-23.1\bin\runemacs.exe"

My .emacs isn’t too esoteric. Nevertheless, here it is for posterity. Reading on, keep in mind that unless otherwise noted, all referenced packages come with Emacs 23.1.

General settings

I start off redefining a couple of global variables. Much functionality relies on the user name and mail address to be known in advance. I then remove the toolbar at the top of the Emacs window, because I use menus or keyboard shortcuts anyway. Next I disable the splash screen on startup, and redefine a few global variables governing Emacs’ scrolling behavior. By default scrolling isn’t line by line, but chuck by chuck of text. I find this behavior confusing.

(setq user-full-name "Ronnie Holm")
(setq user-mail-address "foo@bar.baz")

(tool-bar-mode nil)                            ;; don't show the toolbar

(setq inhibit-startup-message t                ;; don't show ...
      inhibit-startup-echo-area-message t)     ;; ... startup messages

(setq scroll-margin 1                          ;; do smooth scrolling
      scroll-conservatively 100000
      scroll-up-aggressively 0.01
      scroll-down-aggressively 0.01)

Next I turn on automatic syntax highlighting and commands specific to known file types. Following that I turn the blinking square cursor into a solid and less distracting one. Also, I instruct Emacs not to retain backups of files that I edit or save. I don’t discount the value of backups, but I also don’t like having to remove backup and auto-save files when they clutter my folders. Instead  Emacs should move these files to the Windows Recycle Bin. Lastly, I enable iswitchb-mode, which makes switching buffers easier by providing on-the-fly suggestions.

(global-font-lock-mode t)                      ;; always do syntax highlighting
(blink-cursor-mode 0)                          ;; don't blink cursor
(setq backup-inhibited t)                      ;; no backup files
(setq delete-by-moving-to-trash t)             ;; delete moves to recycle bin
(iswitchb-mode t)                              ;; easy buffer switching

The last of the general settings involve modifications to the modeline at the bottom of the Emacs window. By default it displays the buffer name, the line number, and the modes Emacs is in. I extend the modeline with the column cursor position and the size of the current buffer.

(column-number-mode t)                         ;; show column numbers
(size-indication-mode t)                       ;; show file size

Package settings

Oftentimes I find myself editing the same set of files every day. So having the cursor move to where I left it editing the file the last time is handy. This is done by save-place mode. Behind the scenes it stores a file name/offset mapping in a file below HOME.

(setq save-place-file "~/.emacs.d/saveplace")  ;; location of saveplace file
(setq-default save-place t)                    ;; activate for all buffer
(require 'saveplace)

As hinted by Getting organized with Emacs Org-mode, I very much enjoy using Org-mode for keeping notes, maintaining lists, and much more. The next section of my .emacs is therefore dedicated to customizing org-mode. Firstly, to make documents easier to read, instead of displaying all the asterisks that designate an indentation level, only display the last one. Secondly, align each indentation level by always adding two asterisks on indentation. Thirdly, when clocking time, have Emacs remember what I was doing and for how long across restarts. But remove clocked time less than one minute.

(setq org-hide-leading-stars t)                ;; hide but one star in outline
(setq org-add-levels-only t)                   ;; align items nicely
(setq org-clock-persist t)                     ;; keep track of time ...
(org-clock-persistence-insinuate)              ;; ... across sessions
(setq org-clock-out-remove-zero-time-clocks t) ;; remove 0-duration clocked

The default Emacs color theme looks kind of dull. So I downloaded the color-theme package and selected a more aesthetically pleasing theme. After downloading color-theme, place its Lisp files in ~/site-lisp. That’s one of the places where Emacs will automatically look for packages. The 300 or so packages that come with Emacs are in the ~/lisp folder.

(require 'color-theme)                         ;; use a color theme
(color-theme-initialize)
(color-theme-arjen)

My last additions to .emacs involve Haskell. Emacs doesn’t support Haskell out-of-the-box. But a haskell-mode is available for download. It turns Emacs into a decent Haskell development environment. Besides syntax highlighting and code formatting capabilities, it adds to Emacs commands for interacting with the Haskell compiler and Haskell interactive.

(load "~/site-lisp/haskell-mode-2.6.4/haskell-site-file")
(add-hook 'haskell-mode-hook 'turn-on-haskell-doc-mode)
(add-hook 'haskell-mode-hook 'turn-on-haskell-indentation)

This concludes my .emacs. Quite a few of these settings I picked up reading the emacs-fu blog.

Here’re the talks that I attended, all recorded and available with slides.

SharePoint 2010 Planning and Best Practice Approaches to Upgrade

Upgrading to SharePoint 2010, you should first establish a copy of your existing environment on the latest patch level. Then run the stsadm command with the new PreUpgradeChecker option. Make sure your environment has no unused features, site templates, and so on, and identify everything that has changed from out-of-the-box by running a tool like WinMerge on the 12 hive. Also, assert that no pages has been ghosted, and don’t attempt to upgrade your SharePoint 2007 master page. Start with the 2010 ones and add in your 2007 modifications. SharePoint 2010 ships with both 2007 and 2010 master pages, but SharePoint 2007 and 2010 isn’t binary compatible.

Make sure all hardware requirements are satisfied: 8 GB RAM at the minimum on the Windows 2008 servers running SharePoint and MS SQL Server. On the client you need Internet Explorer 7 or 8 or Firefox 3.x to author content. Internet Explorer 6 isn’t supported. Not even with a degraded reader experience. The SharePoint 2010 development environment should be setup with Visual Studio 2010 on Windows Vista or Windows 7 in order to compile, debug, build, and deploy right out of Visual Studio 2010.

Understanding, using, and customizing PowerShell for a SharePoint 2010 Environment

This talk started with some PowerShell history and basics. PowerShell is made up of a shell that executes commands and an integrated scripting environment for development. .NET compiled code, called cmdlets, are the executable units combined using the PowerShell pipeline. The PowerShell language itself is dynamic and loosely typed, so being able to express yourself tops performance. Scripts may be compiled and deployed as a SnapIn, consisting of compiled DLLs and registry updates, or as a module using xcopy deployment.

SharePoint 2010 ships with 650 cmdlets, and its own base class for writing additional SharePoint cmdlets.

Design and Manage Site Collections in SharePoint 2010

By default every single site collection within a web application goes into one rapidly growing database (think about users migrating from file shares to site collections). Such a large database takes time to backup and restore. Hence, with SharePoint 2010, consider partitioning site collections into their own databases from the beginning. After a database is created, map it to one and only one site collection. Then, instead of putting a size limit on the web application, put it on the site collection itself.

In SharePoint 2007 you had to create multiple Shared Service Providers (SSP) if site collections had different needs or you didn’t want a site collection to access parts of an existing SSP. With SharePoint 2010, application services can now be shared between site collections.

With sandbox solutions, the solution is deployed to the database hosting the site, rather than to front-end servers. This makes it easier to deploy code, but the code is limited in what it can do, and you may have to set resource quotas across the site collection.

SharePoint 2010, Getting Ready

This talk provided an overview of SharePoint 2010. WSS 3.0 becomes SharePoint Foundation 2010 and MOSS 2007 becomes SharePoint Server 2010. Business Connectivity Services (BCS) are no longer part of the server edition, but has been moved to Foundation. A lot more services have been added, like state service, usage and health service, and Visio graphics service. Generally, with all the improvements to deployment and upgrade with PowerShell and the addition of multi-tenancy for the SSP, SharePoint 2010 is better in a hosting scenario.

The web user interface has been made more Office-like with the addition of the ribbon and the use of AJAX. In document libraries you can create document sets as a way to relate documents to each other and have versioning and workflow follow every document in the set. Through the BCS, you can expose external data as native SharePoint lists with full CRUD capabilities.

SharePoint 2010 Development Tips and Tricks

The focus of the first part this talk is on the SharePoint 2010 Foundation features. Instead of developing against the web services or the object model, you can use the new managed client object model. This way, you can work against SharePoint from .NET code, from a Silverlight application, or from JavaScript. Behind the scenes, the client object model works against a WCF service running on the SharePoint server, which talks to the server object model on the server, and returns JSON to the client.

The second part of this talk is about sandbox solutions. This feature provides site collection users with the ability to upload and deploy WSPs. Finally, the third part of the talk is about the fluid integration model, which solves the problem of accessing cross-domain resources, e.g., having a Silverlight application access information on a SharePoint server on a different domain.

Installing and Configuring SharePoint Server 2010 in a Virtual Environment

This talk recommends that everyone read the MS whitepaper on Virtualization of Microsoft SharePoint Products and Technologies. Configuring a development environment, the dilemma is weather to create one virtual machine hosting everything or one virtual machine for the MS SQL Server, the Active Directory, and the SharePoint server each. The presenter suggests a middle ground. Still, it’s important to create a number of service accounts and use them in your development environment rather than running everything as administrator.

The SharePoint 2010 installer comes with a prerequisite installer that goes onto the web and downloads required software. After that, you can run the SharePoint server installer. Here you can choose between Standalone and Server farm, but always pick server farm. Standalone should only be used for demo purposes.

Last week at work I did a 45 minutes presentation on how to get started with SharePoint development. Instead of the usual presentation on the technical merits of SharePoint, my focus was more on the change in mindset required coming from ASP.NET. For even though ASP.NET is a cornerstone of SharePoint, SharePoint is so much more. Not to mention different.

History makes a difference

SharePoint is a long-evolving product, dating back to the late nineties. From its inception and up till now new technologies have replaced old ones, APIs have emerged and died, product teams have merged, and Microsoft have bought and integrated other companies’ products into SharePoint. At the same time SharePoint has become the fastest growing server product in Microsoft history. All of this inevitably leaves its marks on the APIs, which aren’t as consistent and defect free as we’re used to with the .NET framework.

Provisioning is the magic sauce

What truly makes SharePoint development different from ASP.NET is the notion of provisioning. With ASP.NET you create a bunch of ASPX files, organize them in a static file structure, and deploy them to a server. With SharePoint, the structure emerges over time as SharePoint or the user creates instances of templates: sites from site templates, lists from list templates, pages from page templates, and so forth. From a developer’s perspective the challenge isn’t the instance creation in itself. It’s that SharePoint encourages users to not only dynamically evolve the structure, but also to modify the instances, severing their definition from the template. While it empowers users, it also makes it challenging to programmatically evolve the structure.

Developing with SharePoint is about creating templates and components for SharePoint or the user to compose a site of. To make this composition work, SharePoint imposes constraints on what can be done and how. While any constraint may at times be perceived as a hindrance, accepting it is what guarantees that a components will play nicely with others. These pieces of functionality are then packaged as features and added to a WSP installation file for deployment to SharePoint. Features may then be activated, making their functionality available to a site. Or deactivated and the WSP uninstalled.

Why learn SharePoint

By the end of the day, no doubt SharePoint is more challenging to work with than ASP.NET. Not acknowledging its history makes many ASP.NET developers so frustrated that they’ll never again want to work with SharePoint. Coming from ASP.NET I can understand why. At times SharePoint isn’t as smooth and evolved as ASP.NET. But knowing what to expect takes the worst of the frustration. And once you learn to leverage the power of SharePoint, you can build applications without writing a lot of plumping code. To a large extend the architecture is given, leaving you with the opportunity to fill in the blanks.

Maybe it’s that the SharePoint API is hard to use. Maybe it’s that coding against SharePoint is about making smaller additions here and there. Maybe it’s that the Patterns & Practices SharePoint Guidance isn’t widely known. Whatever the reason, developing for SharePoint requires equal attention to the separation of presentation, business, and data access code. Hence, starting with data access, we may want to create a repository and route queries through it (the SharePoint Guidance outlines a more sophisticated implementation than the one below):

[TestClass]
public class EmployeesRepositoryTest {
    private SPSite _siteCollection;
    private SPWeb _site;
    private EmployeesRepository _repository;

    private readonly Employee _duffyDuck = new Employee {
        Id = 1000, Name = "Duffy Duck", HireDate = new DateTime(2009, 12, 1),
        Remarks = "Looks like a duck, quacks like a duck, probably is a duck"
    };
    private readonly Employee _porkyPig = new Employee {
        Id = 1001, Name = "Porky Pig", HireDate = new DateTime(2010, 2, 1)
    };
    private readonly Employee _sylvesterTheCat = new Employee {
        Id = 1002, Name = "Sylvester the Cat", HireDate = new DateTime(2010, 3, 1)
    };
    private readonly Employee _bugsBunny = new Employee {
        Id = 1100, Name = "Bugs Bunny", HireDate = new DateTime(2010, 1, 1)
    };

    public void AddEmployees() {
        _repository.AddEmployee(_site, _duffyDuck);
        _repository.AddEmployee(_site, _porkyPig);
        _repository.AddEmployee(_site, _sylvesterTheCat);
    }

    public void ClearEmployees() {
        var definition = new EmployeesDefinition();
        var employees = _site.Lists[definition.ListName];
        while (employees.Items.Count > 0)
            employees.Items.Delete(0);
    }

    [TestInitialize]
    public void Initialize() {
        _siteCollection = new SPSite("http://localhost");
        _site = _siteCollection.OpenWeb("/");
        _repository = new EmployeesRepository();
        ClearEmployees();
        AddEmployees();
    }

    [TestCleanup]
    public void Cleanup() {
        _site.Dispose();
        _siteCollection.Dispose();
    }

    [TestMethod]
    public void AddEmployee_should_add_valid_employee() {
        _repository.AddEmployee(_site, _bugsBunny);
        var e = _repository.GetEmployeeById(_site, _bugsBunny.Id);
        Assert.AreEqual(_bugsBunny.Id, e.Id);
        Assert.AreEqual(_bugsBunny.Name, e.Name);
        Assert.AreEqual(_bugsBunny.HireDate, e.HireDate);
        Assert.AreEqual(_bugsBunny.Remarks, e.Remarks);
    }

    [TestMethod]
    public void GetEmployeesHiredBetween_should_return_2010_hires() {
        var from = new DateTime(2010, 1, 1);
        var to = new DateTime(2010, 12, 31);
        var employees = _repository.GetEmployeesHiredBetween(_site, from, to);
        Assert.AreEqual(2, employees.Count);
        Assert.AreEqual(_porkyPig.Id, employees[0].Id);
        Assert.AreEqual(_sylvesterTheCat.Id, employees[1].Id);
    }
}

In the words of Martin Fowler, here’s the essence of the Repository pattern:

A Repository mediates between the domain and data mapping layers, acting like an in-memory domain object collection. Client objects construct query specifications declaratively and submit them to Repository for satisfaction. Objects can be added to and removed from the Repository, as they can from a simple collection of objects, and the mapping code encapsulated by the Repository will carry out the appropriate operations behind the scenes. Conceptually, a Repository encapsulates the set of objects persisted in a data store and the operations performed over them, providing a more object-oriented view of the persistence layer.

In SharePoint terms, business logic should query a repository which in turn queries a SharePoint list. Within the repository, the weakly typed items returned are then mapped to strongly typed data transfer objects, which are returned to the business layer.

With this approach to data access comes a number of advantages: (1) duplicate data access code is eliminated. Only within the repository do we setup the query and transform the weakly typed SPListItemCollection into strongly typed data transfer objects. (2) The list definition classes introduced in SharePoint list definition using the Template pattern may be used to construct CAML queries from strongly typed field names. Lastly, (3) accessing data through a repository makes it easier to mock the data access part of the application and to integration test that part.

The code below is a simple, yet usable, implementation of the Repository pattern. The idea is to have all repositories inherit from a common base class. Its purpose is to wrap the querying of a list and to log what’s going on. Because it’s a base class, it shouldn’t know how to transform the weakly typed result into strongly data transfer objects or which CRUD operations a particular repository supports:

public abstract class ListRepository {
    protected ListDefinition Definition { get; set; }
    protected SPListItemCollection Result { get; set; }

    protected void Query(SPWeb site, string caml) {
        AssertValidSite(site);
        AssertValidCaml(caml);
        AssertListExistence(site);
        AssertListDefinitionSetBySubclass();

        var watch = new Stopwatch();
        var list = site.Lists[Definition.ListName];
        var query = new SPQuery {Query = caml};
        Debug.WriteLine(
            string.Format("About to run query against list '{0}': {1}", list, caml));
        watch.Start();
        Result = list.GetItems(query);
        watch.Stop();
        Debug.WriteLine(
            string.Format("Query against '{0}' returned {1} rows in {2} ms", 
                          list, Result.Count, watch.ElapsedMilliseconds));
    }

    protected void AssertListExistence(SPWeb site) {
        if (!ListDefinition.ListExists(site, Definition.ListName))
            throw new ArgumentException(
                string.Format("No '{0}' list on site '{1}'", Definition.ListName, site.Url));
    }

    protected void AssertListDefinitionSetBySubclass() {
        if (Definition == null)
            throw new NullReferenceException(
                string.Format(
                    "Subclass must set Definition property prior querying '{0}' list", 
                    Definition.ListName));
    }

    protected void AssertValidCaml(string query) {
        if (string.IsNullOrEmpty(query))
            throw new NullReferenceException("Query must not be null or empty");
    }

    protected void AssertValidSite(SPWeb site) {
        if (site == null)
            throw new NullReferenceException("Site must not be null");
    }
}

Each ListRepository connects to a corresponding ListDefinition, holding the name of the list to query and its strongly typed field names. It’s the responsibility of a concrete repository to set the Definition property prior to doing any querying. After running a query, the Result property holds the weakly typed result, which a concrete repository can then transform into data transfer objects to be passed to the business layer.

As an example, add to the concrete EmployeeRepository any CRUD method you see necessary to fulfill the business requirements:

public class EmployeesRepository : ListRepository {
    public EmployeesRepository() {
        Definition = new EmployeesDefinition();
    }

    public void AddEmployee(SPWeb site, Employee e) {
        var list = site.Lists[Definition.ListName];
        var item = list.Items.Add();
        item[EmployeesDefinition.EmployeeId] = e.Id;
        item[EmployeesDefinition.Name] = e.Name;
        item[EmployeesDefinition.HireDate] = e.HireDate;
        item[EmployeesDefinition.Remarks] = e.Remarks;
        item.Update();
    }

    public Employee GetEmployeeById(SPWeb site, int id) {
        var caml =
            string.Format(@"
                  <Where>
                    <Eq>
                      <FieldRef Name=""{0}"" />
                      <Value Type=""Integer"">{1}</Value>
                    </Eq>
                  </Where>",
                EmployeesDefinition.EmployeeId, id);
        Query(site, caml);

        IList<Employee> employees = Map(Result);
        if (employees.Count == 0)
            throw new ArgumentException(string.Format("No employee with id = {0} exists", id));
        return employees[0];
    }

    public ReadOnlyCollection<Employee> GetEmployeesHiredBetween(SPWeb site, DateTime from, DateTime to) {
        var caml =
            string.Format(@"
                  <Where>
                    <And>
                        <Geq>
                          <FieldRef Name=""{0}"" />
                          <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{1}</Value>
                        </Geq>
                        <Leq>
                          <FieldRef Name=""{0}"" />
                          <Value IncludeTimeValue=""TRUE"" Type=""DateTime"">{2}</Value>
                        </Leq>
                    </And>
                  </Where>",             
                EmployeesDefinition.HireDate, 
                SPUtility.CreateISO8601DateTimeFromSystemDateTime(from),
                SPUtility.CreateISO8601DateTimeFromSystemDateTime(to));
        Query(site, caml);
        return new ReadOnlyCollection<Employee>(Map(Result));
    }

    protected IList<Employee> Map(SPListItemCollection items) {
        var employees = new List<Employee>();
        foreach (SPItem item in items) {
            var e = new Employee {
                            Id = (int)item[EmployeesDefinition.EmployeeId],
                            Name = (string)item[EmployeesDefinition.Name],
                            HireDate = (DateTime)item[EmployeesDefinition.HireDate],
                            Remarks = (string)item[EmployeesDefinition.Remarks]
                        };
            employees.Add(e);
        }
        return employees;
    }
}

Calling the GetEmployeeById or GetEmployeesHiredBetween methods, the caller is required to pass in an SPWeb instance pointing to the the site holding the list to query. Outside the SharePoint context, you have to manually create this instance, like with the integration tests above. But within the SharePoint context, callers are likely to just pass in SPContext.Current.Web.

The above repository implementation deliberately ignores any issue of caching. If you find the need for it, however, you can replace SPList.GetItem with PortalSiteMapProvider.GetCachedListItemsByQuery. The advantage of using the PortalSiteMapProvider over Asp.Net caching of the result is that the provider takes care of invalidating cache entries when items are modified. The disadvantage is that the provider is part of the SharePoint publishing API, which isn’t part of WSS 3.0. In addition, it’s only available from code running within SharePoint, and likely requires cache settings to be tweaked.

Another, non-trivial, improvement would include the use of the Unit of Work pattern. A commonly used pattern in ORMs because it offers a way to “keep track of everything you do during a business transaction that can affect the database. When you’re done, it figures out everything that needs to be done to alter the database as a result of your work". Like with the DataContext class in LINQ to SQL or the DataSet class in ADO.NET, it could add transaction support to SharePoint lists.

Books on SharePoint often show how to create lists from code by calling the SharePoint API from directly within a feature receiver. This receiver would then contain in-place string literals for column names, create the columns, update the default view, and so on. While such an approach provides for an nice demonstration of the SharePoint API, it tends to carry over into production code.

My take on feature receivers, however, is that they should contain the least possible amount of code. Staying true to object-orientation and separation of concerns, list definition and creation should instead consist of a common set of classes and methods to be called from the receiver:

public override void FeatureActivated(SPFeatureReceiverProperties p) {
    using (var site = p.Feature.Parent as SPWeb) {
        var employees = new EmployeesDefinition();
        if (!ListDefinition.ListExists(site, employees.ListName)) 
            employees.CreateOnSite(site);            
    }
}

The first area to improve on is that of string literal field names getting duplicated across the code base. String literals make it impossible for the compiler to enforce a consistent naming of columns. As column names are used in defining the list, in forming CAML queries against it, and in accessing the result, misspelled column names is a common source of runtime errors. The second area to improve on is that of duplication of code for doing common list operations, like checking for the existence of the list or removing its title column. Lastly, we want to get rid of duplicate control logic, i.e., verifying preconditions and calling the same series of methods with every list definition.

One approach to improving on these areas is to start by defining a base class for list definitions. The base class combines helper methods, working on lists, with the Template pattern for defining the common steps of list creation:

public abstract class ListDefinition {
    public string ListName;
    public string ListDescription;
    protected SPList List;

    public void CreateOnSite(SPWeb site) {
        if (string.IsNullOrEmpty(ListName))
            throw new ArgumentException("Expected not null or not empty list name member");
        if (site == null) throw new NullReferenceException("Expected valid site");
        CreateList(site);
        if (List == null) throw new NullReferenceException("Expected list member not null");
        CreateFields(List);
        UpdateDefaultView(List);
        SetAdditionalProperties(List);
    }

    protected abstract void CreateList(SPWeb site);
    protected virtual void CreateFields(SPList l) {}
    protected virtual void UpdateDefaultView(SPList l) {}
    protected virtual void SetAdditionalProperties(SPList l) {}

    public static void HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(SPList l) {
        var title = l.Fields.GetField("Title");
        title.Required = false;
        title.Hidden = true;
        title.Update();
    }

    public static bool ListExists(SPWeb site, string listName) {
        if (string.IsNullOrEmpty(listName))
            throw new ArgumentException("ListNameMustNotBeNullOrEmpty", listName);
        SPList list = null;
        try {
            list = site.Lists[listName];
        }
        catch (ArgumentException) {
            // list not found            
        }
        return list != null ? true : false;
    }
}

At the core of the Template pattern is the template method. It defines the steps of an algorithm through a series of method calls. Each either abstract or virtual depending on if the step is mandatory or voluntary in the algorithm. Subclasses then specify the behavior of individual steps by implementing or overriding methods. In case of the ListDefinition class, CreateOnSite is our template method. It ensures that context, in the form of an SPList instance, is passed along to each step in the algorithm.

Each subclass defines the specific characteristics of a list. It defines constants for column names and their data types, the views and which column to index, and so on. This makes for a concise, easy to read, and consistent list definition:

public class EmployeesDefinition : ListDefinition {
    public const string EmployeeId = "EmployeeId";
    public const string Name = "Name";
    public const string HireDate = "HireDate";
    public const string Remarks = "Remarks";

    public EmployeesDefinition() {
        ListName = "AcmeEmployees";
        ListDescription = "Employees at Acme Corp.";
    }

    protected override void CreateList(SPWeb site) {
        var guid = site.Lists.Add(ListName, ListDescription, SPListTemplateType.GenericList);
        List = site.Lists[guid];
    }

    protected override void CreateFields(SPList l) {
        l.Fields.Add(EmployeeId, SPFieldType.Integer, true);
        l.Fields.Add(Name, SPFieldType.Text, true);
        l.Fields.Add(HireDate, SPFieldType.DateTime, true);
        var remarks = (SPFieldMultiLineText) l.Fields[l.Fields.Add(Remarks, SPFieldType.Note, false)];
        remarks.RichText = true;
        remarks.RichTextMode = SPRichTextMode.FullHtml;
        remarks.Update();
        l.Update();
    }

    protected override void UpdateDefaultView(SPList l) {
        var v = l.DefaultView;
        v.ViewFields.Delete("LinkTitle");
        v.ViewFields.Add(EmployeeId);
        v.ViewFields.Add(Name);
        v.ViewFields.Add(HireDate);
        v.ViewFields.Add(Remarks);
        v.Update();
    }

    protected override void SetAdditionalProperties(SPList l) {
        HideAndMakeTitleFieldNotRequiredOnItemNewAndEditPage(l);
        l.Fields[HireDate].Indexed = true;
        l.Update();
    }
}

Since the string literals have now become public constants, we get IntelliSense and compile-time checking referencing them. In addition, by defining common operations on the base class and using the Template pattern, we reduce the amount of code that would otherwise have to go into each definition. Finally, the base class asserts, on behalf of all subclasses, that the list name is set before attempting to create the list and that the SPList field is set after calling the CreateList method.

My goals for the coming year are:

• Focus more on architecture and design. Not in the architecture astronaut connotation. But as in becoming more aware of how to build better software. I want to apply the SOLID principles more consciously and re-read classics such as Design Patterns: Elements of Reusable Object-Oriented Software and Patterns of Enterprise Application Architecture.
• Continue exploring LINQ. It’s a fascinating technology that’s going to get only more pervasive in years to come. Truly understanding LINQ is key to getting into the habit of applying a more functional and succinct style to my own code. To get started on this endeavor I’m going to read and apply LINQ in Action.
• Learn Entity Framework. Using an object-relational mapper should be second-nature to any developer. In the past the light-weight LINQ to SQL has served me well. Still, I want to step up and learn how Entity Framework deals with the impedance mismatch. I want to write an application that uses Entity Framework.
• Become better at browser technologies. JavaScript and jQuery are high on the list, but I want to improve on HTML and CSS as well. My goal isn’t to become an HTML or CSS Jedi. I simply want to have enough knowledge to avoid making rookie mistakes. The technologies all come together in jQuery, making Learning jQuery a good starting point. Besides, I want to use a tool like Firebug to learn from others.
• Be on top of SharePoint 2010. SharePoint 2007 and 2010 is what I want to work with daily. That doesn’t imply me not keeping up to speed with the rest of .NET. Just that SharePoint will act as my gateway to other technologies. The great thing about SharePoint is that development is often split between working inside and outside SharePoint.
• Become more involved in higher-level development decisions. By the end of the day, developers don’t deliver software, they deliver value to stakeholders. Writing code is but a means to that end. I want to be involved in all steps from conception to completion.

To me these points strike a good balance between technical and non-technical goals and between short and long term goals. However, by the end of the day, building great software is about being attentive to the entire stack and its surroundings.

So how did the year go?

• Changed jobs to one that gravitates more around larger projects. The rationale being that larger projects come with larger engineering challenges. I spend six months on an Asp.Net project followed by three on a SharePoint project. For various reasons, I wasn’t as involved in higher-level development decisions as I would’ve liked to.
• Did two presentations. One on Unit testing and mocking for the Asp.Net project that I was working on at the time. The other presentation was on Parallel page rendering with Asp.Net and SharePoint for my current project. None involving rocket science, but I find so few are familiar with the techniques.
• Made my way through three certifications. The 70-536: MS .NET Framework, Application Development Foundation, 70-562: MS .NET Framework 3.5, ASP.NET Application Development, and Prince2 Foundation. I’ve mixed feelings with regard to certifications, though. Certainly, I learned valuable lessons studying for each one. Yet, given the same amount of time, I’m inclined to think I would’ve learned more studying books, blogs, listening to various podcasts, and writing.
• Attended the Jaoo conference on software engineering. It was my first time ever attending a conference and it sure was enlightening. I intentionally went for a conference not tied to a specific vendor because I felt I’d learn more that way. My issue with Microsoft conferences like Tech-Ed or PDC is their focus on the latest technology buzz. Buzz that quickly becomes obsolete or is already heavily exposed on the web. Not to mention that most sessions are available online afterward.
• Learned functional programming. My focus on new technology has mostly been on various aspects of functional programming. A subject that’s interested me for years, but one that’s only recently started to permeate C#, my language of choice. I learned a great deal about F# when I generated fractal terrains and about XSL when I had to jump through functional loops and about functional programming in general.
• Wrote 16 blog posts (25 pages, 10,000 words). Compared to previous years it’s a record. Naturally, not all posts were equally good. Nevertheless I enjoyed learning about the subjects and feel they helped advance my writing skills. The posts on Basic unit testing guidelines and Why not to comment code did manage to gain traction on Reddit, though.

All things considered, I’m pleased with 2009. The only thing I regret is not being more involved in higher-level development decisions. I like coding, but it’s a means to an end. Being involved with a task from start to finish is what ignites my fire. I must push harder on that in 2010.

I started out confident that printing would be easy to fix. But after Googling and playing around with JavaScript for a while, frustration, fueled by the number of non-working hits I’d stumbled on, started to grow on me. Suppose you start out with an HTML document hosting an iframe and a few lines of JavaScript.

<html>
    <head>
        <title>IFrame printing example</title>
        <script type="text/javascript">
            function myPrint() {
                var browser = navigator.appName;
                if (browser == "Microsoft Internet Explorer") {
                    window.frames["frame"].focus();
                    window.frames.print();
                }
                else if (browser == "Netscape")
                    alert("Printing not supported in Firefox");
            }
        </script>
    </head>
    <body>
        <iframe id="frame" name="frame" width="100"
            height="100" src="http://google.com">
        </iframe>
        <a href="#" onclick="javascript:myPrint();">Print</a>
    </body>
</html>

A number of issues arise when printing the iframe. First you need to consider if you’re always hosting the main document on the same domain as the iframe’s source. If so, printing only the iframe with Internet Explorer works like a charm. If, on the other hand, the main document and the iframe’s source reside on different domains, it triggers Internet Explorer to displays a bar across the top of the window.

Now, to print the iframe’s content, you must allow blocked content. Only then can JavaScript access the iframe’s object model. But at the same time, you make yourself vulnerable to cross-site scripting attacks.

As far as Firefox support goes, you can issue the same API calls as for Internet Explorer. But Firefox will always print the main page, including only the visible parts of the iframe. To my knowledge there’s no way to make Firefox print the iframe’s content as seamlessly as Internet Explorer. One workaround, though, would be to open a new window, grab a reference to the iframe element, and move its contents to the new window before printing it. For cross-domain content, however, Firefox responds with a “permission denied” error when accessing the iframe element. And there’s no option to allow blocked content. Another workaround may therefore be to use Ajax to retrieve the source of the iframe for any domain and insert it into the new window. That, however, wouldn’t work in my case since the iframe may contain a form that the user may have started filling out before hitting print.

As a last resort, I considered writing a proxy to retrieve cross-domain hosted content. This would make Internet Explorer never display the bar and allow Firefox to always open a new window with the iframe’s content loaded, even if the user started filling out a form first. But this approach sort of defies the purpose of the browser bar and the permission denied error. Unless you absolutely trust your iframe’s source, you’re exposing your users without their consent.

In the end I went with the simplest and most secure solution of not supporting printing at all.

With Asp.Net, by default a page is rendered synchronously, one control at a time. The thread rendering the page comes from the thread pool within the world wide web worker process (w3wp.exe) and is assigned to the request when it comes in on the web server. Only when the page has fully rendered can the thread go back in the pool waiting for the next request.

Processing a complex page using only one thread, however, may cause the page to render slowly. The page may hold controls that require calls to web services, databases, or other external resources for it to render. Whenever the worker thread encounters such a control, it goes idle waiting for data to come back from the external resource. Only when data is received can the worker thread resume processing.

One way to speed up page processing would be for the page or control to explicitly make use of threads. Depending on the situation, this may be a cumbersome task that requires a significant amount of boiler-plate code. A simpler approach might be to turn to the Asp.Net 2.0 feature for asynchronous page processing. At its core is an extension to the page or control lifecycle.

Except for the Begin and End events, the flow is the same in both models. But while the synchronous model executes all code on a single thread, the asynchronous model makes implicit use of multiple worker threads to speed up page processing. With multiple threads, some threads still go idle waiting for input, but at least the main worker thread continues to work its way through the controls.

In the asynchronous model, we hook into the Begin and End handlers in one of the events preceding the call to Begin. Following PreRender, Asp.Net will then execute our Begin handler, where we call out to some method that carries out the long-running task. When that method exists the End handler is called and processing carries on as in the synchronous model.

To illustrate the flow and timing of the asynchronous model (turn to the slides for the complete code sample), suppose we add five controls to a page. Each control takes five seconds to render so using the synchronous model the page would take 25 seconds to render. Using the asynchronous model, and writing out trace information about which thread each method executes on, here’s what happens:

1: Page_Load: 10            9: DoWork: 8
2: Page_Load: 10           10: DoWork: 11
3: Page_Load: 10           11: Render: 11 15:16:13 15:16:18
4: Page_Load: 10           12: Render: 11 15:16:13 15:16:18
5: Page_Load: 10           13: Render: 11 15:16:13 15:16:18
6: DoWork: 4               14: Render: 11 15:16:13 15:16:18
7: DoWork: 10              15: Render: 11 15:16:13 15:16:19 
8: DoWork: 9

The user control’s Page_Load and the code making up the Begin handler executes on the main worker thread. But the method carrying out the long-running task, and the code for the End handler, executes in parallel on different worker threads. Finally, after End is called, another worker thread takes over and executes the remaining parts of the control. Now page rendering has decreased from 25 to about five seconds.

All in all, little effort is required to make more efficient, and implicit, use of the thread pool, thereby decreasing the time required processing a page. Asynchronous page processing should be used judiciously, though. Optimizing away a bottleneck in the processing of a page may well lead to bottlenecks appearing elsewhere, e.g., the number of simultaneous calls made to a web service may be too much for it to handle.

For more details and examples, turn to Jeff Prosise‘s article on Asynchronous Pages in Asp.Net 2.0.

In a nutshell Org-mode is a plug-in for Emacs that makes taking notes, maintaining lists, clocking time, and many other tasks more straightforward. The mode works by adding features on top of Emacs for editing plain text files. Features that enrich the interaction with the underlying text buffer by parsing and adding (meta-)information to it.

With Org-mode you group items into a hierarchical structure. A path to the item to work on is then formed by unfolding items in a way analogous to how you navigate the file system with Explorer.

To efficiently work with items, Org-mode supports moving items around the hierarchy. Items may also be tagged (buy, work, phone, etc.) or associated with a state (done, waiting, started, etc.). Additional meta-information may include a deadline, an estimate, or time spent working on a task. The clocking of time is accomplished by Org-mode adding a timestamp to the item. On completion, another timestamp and the elapsed time is calculated and added.

Below is an example of an expanded and a collapsed item containing clocked times and estimates.

With a few keystrokes, Org-mode is able to correlate estimates with time spent on an item-by-item basis. This truly short-circuits the estimation feedback cycle. Imagine doing this on a file containing a large hierarchy of items and the transparency it brings.

If you want to keep track of billable hours, it may make sense to clock all your time using Org-mode and generate timesheets from the clocking information. Especially since the billing context is automatically provided by the item storing the timestamp, and since you’re free to organize the file in any way you see fit. Few other tools manage to compete with the flexibility of a plain text buffer.

This post only scratches the surface of Org-mode. To get a sense of the possibilities, the manual amounts to 184 pages. However, for a quick introduction I encourage you to watch the Google Tech Talk on Org-mode.

From Oct 5 to Oct 7, I was fortunate enough to attend the Jaoo 2009 conference on software engineering in Aarhus. It was a great conference with three days of insightful talks. The talks I attended were:

Monday

• Keynote: Scaling Up Agility: The Architected Agile Approach/Barry Boehm (slides)
• Working Effectively with Legacy Code 2/Michael Feathers (slides)
• Evolving the Key/Value Programming Model to a Higher Level/Billy Newport (slides)
• Techniques That Still Work no Matter How Hard We Try to Forget Them/Keith Braithwaite (slides)
• Making use of Patterns/Martin Fowler
• xUnit Test Patterns – Refactoring Test Code to Improve ROI/Gerard Meszaros (slides)

Tuesday

• Keynote: Classes, Jim, But Not as We Know Them/Simon Peyton-Jones (slides)
• Agile Modeling and Documentation “Best” Practices/Scott Ambler (slides)
• Actors, and the Forgotten Art of Modeling Concurrent Systems/Dave Thomas, Kresten Krab Thorup (slides)
• Buy it, Build it, Download It, or Browse It? Achieving Effectiveness with Enterprise Applications/Michael T. Nygard (slides)
• Hydras and Hypermedia/Ian Robinson (slides)
  
• Maximum Value, Maximum Speed through Lean Thinking – How Business and IT can Collaborate/Dave Thomas

Wednesday

• Keynote: C++, Java and .NET – Lessons Learned from the Internet Age, and What it Means for the Cloud and Emerging Languages/Cameron Purdy (slides)
• Guiding Your Personal Life — “Plan-Driven” or “Agile”/Linda Rising (slides)
• Object Relational Mapping += 2: More than just data <-> object/Oren Eini
• Deliberate Practice in Software Development/Mary Poppendieck (slides)
• Umbraco: An Open Source Web CMS that Devs Will Like … Really? (slides)
• Delivering Quantified Stakeholder Values with Agile – A Case study from Bring/Kai Gilb (slides)

As a supplement to the official homepage, the Twitter feed provided a great source of information for attendees to learn about the presentations and reactions. Not being on Twitter meant missing out on half the conference.

For other reactions, listen to (in Danish) Daniel Frost and Brian Rasmussen discussing day one and two. On day three Daniel interviewed Mads Torgersen of the C# core team.

I was just listening to the Why comments are evil episode of Deep Fried Bytes. The views put forward by the guest, Corey Haines, deeply resonate with me because I believe they’re what every developer should strive for. I’m not advocating that code should be entirely free of comments, but rather that most comments do more harm than good.

Below I’ve listed the main points of view.

• It’s important to differentiate between code comments and API documentation. For API documentation, don’t auto-generate. Especially don’t go with tools like GhostDoc. Take the time to document properly. Otherwise, you’re more likely to confuse the developer trying to make sense of your code.
• Stop focusing on having the computer understand you and start focusing on the person that comes after you, e.g., while there’s some overhead to method calls, unless the profiler tells you otherwise, don’t hesitate to break down a method into as many smaller methods as it takes to make it more approachable.
• Every time you’re tempted to comment, take the time to think if you can make the code tell its own story, e.g., instead of some branching condition, extract the condition into a method to emphasize the story of why over how.
• Developers comment to explain a confusing part of their code rather than take the time to refactor it by extracting a method or naming a variable properly. Inevitably the code gets modified and the comment rots.
• Parts of a method may be complicated so a comment is added. Saying part of a method is complicated is in itself a sign that it’s doing too much.
• The prevailing culture among developers is to focus on what it takes to get code working. It’s a battle of short-term vs. long-term maintainability and short-term almost always comes out as the winner. Writing code, ask yourself if you’re able to immediately understand the code a month from now.

The opinions put forward in the episode aren’t new. Still, I find them well worth iterating because as developers we tend to not pay enough attention to them. We tend to focus too little on good software craftsmanship.

Like with any habit, every so often you should take the time to review it. I find TimeSnapper (the free edition) to be an invaluable tool for reviewing computer habits. It captures an image of the desktop every few seconds and is able to create a movie from the images. That way, a partial life log (mine is currently 393 days) is automatically created to help you remember what you did yesterday and the day before.

There’s nothing like reviewing a few of weeks of captured images to make habits stand out. I found that I’ve a tendency to re-visit the same maybe ten sites all too often. I also have difficulty single-tasking. Listening to a podcast or watching a movie, I’m often browsing the web instead of paying attention to the podcast or skipping it altogether. I may even have several media players launched at once, although only one playing.

Most of my distractions can be attributed to me always being online. I need Internet access for work, so turning it off altogether isn’t an alternative. So, not unlike Paul Graham, on several occasions, I’ve made myself adhere to a set of rules for using the Internet. And every time they’ve worked — for a short while.

In Disconnecting Distraction, Paul takes these rules one step further and argues that using two computers may be a better solution: one computer with Internet access and one without for doing real work. Ideally, the computers should be placed some distance apart to make you psychically get up to go on the Internet.

For a week I’ve given the two-computer approach a try. It was hard initially, but I’ve come to appreciate it. Forming permanent habits takes time. Thus, I’ll continue the experiment for a couple of weeks. Perhaps in time I’ll even learn to turn off instant messaging and only do mail once or twice a day. Or I may end up spending most of the time in front of the connected computer.

Then again, there’s more to life than work.

Not logging implies that debugging the application outside the development environment is largely based on guesswork. That’s interesting given that most applications spend the majority of their life running outside the development environment.

Life in the sandbox

I suspect the absence or low quality of log statements can be, at least partially, explained by developers not finding them useful during development. The debugger available there is generally inferior to the print-lining approach of logging. That said, I’d argue that many developers don’t concern themselves enough with how their code behaves in a production environment. Rather than investing a small amount of time in adding log statements up-front, their focus is solely on satisfying functional customer requirements.

A related case could be made for exception handling. For instance, rather than asserting the validity of an argument up-front and throwing an appropriate exception, the code may later fail with a NullReferenceException. Not throwing the exception up-front, close to where the actual error occurred, one will have a hard time tracking down the issue. Especially since a stack trace with symbols and line numbers is rarely available in a production environment.

For someone having to resolve the issue, almost any clue as to the cause would be most welcomed. Without a clue that someone would walk around in the dark in the hope of stumbling into a solution.

End users aren’t the only users

As software developers we have to acknowledge that end users aren’t the only users of our software. System administrators, developers, and the like should also be factored in during development. Especially since the cost of instrumenting code at key places, early on, doesn’t add significantly to the overall development cost. Focusing on what can go wrong and communicating it well saves time, money, and frustration down the line. And as a bonus you might create better self-documenting code.

That said it’s important to strike a balance between logging too little and too much information. A log should be detailed enough to reveal patterns of use. Ideally, the log should read like a story with short sentences detailing what’s about to happen, what did or didn’t happen, and possibly why.

I’m not suggesting that we turn logging into a runtime version of literate programming. But simply that sufficient contextual information be provided, i.e., instead of logging “URL = ‘bugfree.dk’”, consider the more elaborate “Retrieving URL for indexing: ‘http://bugfree.dk’”.

Keep your statements clean

On the other hand, log statements shouldn’t be immune to the DRY principle. So make sure to have the logging framework automatically capture common bits of contextual information, such as the date and time of the entry and the class and method within which the log statement originated. On a similar note, don’t explicitly log when a method is entered or existed. Instead, treat such a case as a cross-cutting concern and take an Aspect Oriented approach to logging it.

Lastly, not every method requires (extensive) logging. Figuring out when and what to log is key. Debugging the application, fixing bugs, and experience should provide guidance into what’s useful to log.

Update, Aug 5, 2009: This post grew out of a couple of presentations that I did on the subject back in May and June. Here are my slides from The Unit testing and mocking presentation. Among other things, they contain a few C# samples elaborating on some of the points below.

This post is born of the need to formalize a set of guidelines on how to write and organize tests. In the past I couldn’t help feeling that with the lack of guidelines I had to start over explaining my views on every new project. With no guidelines the tests written quickly grew unmaintainable, giving unit testing a bad name.

Why unit test

Unit testing done right helps build confidence in the code base and drives forward development. Both in terms of forcing one to reflect on the requirements by authoring tests, but also by providing a foundation for developing more modular and testable code.

Given the cyclomatic complexity of even simple methods, however, not every path through a method is worth exercising with a test. Instead, focus on writing representative unit tests for good and bad scenarios.

Unit test != integration test

For object-oriented code a unit test is one that exercises a class in isolation, without the code under test relying on other classes to carry out its operation. Similarly, a unit test shouldn’t rely on the presence of a database or a key/value pair in a configuration file for it to run. If it did, it would be an integration test. Not that there’s anything wrong with integration tests. They just take on more dependencies and hence tend to be more brittle. And so they cause more false positives because some dependent part isn’t properly configured. The key point, though, is not to confuse unit testing with integration testing.

Typically, an object delegates part of its operation to other objects. Writing modular and testable code therefore involves the application of design patterns, such as Inversion of Control (IoC). With IoC a test can inject (fake) components into the object under test. The fake components share their interfaces with the real ones, but the test controls how they interact with the object under test and hence how the object under test behaves.

Keep up quality

As tests are written it’s vital for the understandability and maintainability of the test suite to keep them small and focused. As a rule of thumb a test should amount to no more than 10-15 lines of code. Longer tests are indicative of too much functionality being tested at once or that code common to multiple tests should be refactored into helper methods.

In terms of quality, the code comprising the tests should be of production code quality, i.e., the code must be kept clean and refactored as the need arises. Otherwise, tests will start to emit the classic code smells. In the longer run code smells lead to tests that are not maintainable and for the time that went into writing them to be wasted.

Organizing test code

Assuming the use of Visual Studio (VS), for each VS project with code that one wants to test, create matching projects that host the various kinds of tests, i.e., for the Acme.Intranet.Search project, create the following test related projects:

Acme.Intranet.Search.Common
Acme.Intranet.Search.UnitTest
Acme.Intranet.Search.IntegrationTest

The Acme.Intranet.Common project is optional and may include code that is shared between test projects, such as custom assertions. As for the Acme.Intranet.Search.UnitTest project it should be fairly self-contained. One should be able to move the common assembly, the test assembly, and the business code assembly to another machine and have the tests execute there without further setup. Should a test rely on, say, a data file with test data, then include the file as an embedded resource within the test assembly.

Finally, within each test project, a class should be created for each class under test. In addition, the directory structure should match the namespace structure of the class under test, e.g., suppose the fully qualified name of a class is Acme.Intranet.Search.Business.Crawler, then create the following directory structure within the test assembly:

Acme.Intranet.Search.UnitTest
    Acme
        Intranet
            Search
                Buesiness
                    CrawlerTest.cs

While it’s important to write tests, it’s even more important to know where to put and find tests for a given functional area.

Naming tests

As far as naming and structure goes, a test should look something along these lines:

[TestClass]
public class SomeClassTest {
    [TestMethod]
    public void SomeMethod_should_set_error_message_when_no_connection_string_is_configured() {
        // arrange
        // act
        // assert
    }
}

The test should generally start with the name of the method or property being tested, followed by the word "should" followed by the successful outcome in a descriptive form. Because names of tests tend to be longer than those of regular methods, underscores are used for ease of readability. In addition, the body of most tests should be composed of three parts: (1) the arrange part that sets up the object under test and possibly injects fake dependencies into it. (2) The act part then exercises the method under test, and finally (3) the assert part that verifies that expected state and/or behavioral changes did indeed take place.

Gathering metrics

Whenever a build is kicked of (on a build server) it should exercise all tests. Should a test fail, it should cause the entire build to fail, stressing the importance of keeping tests green at all times. Furthermore, a build report should include basic metrics such as code coverage, number of tests run, time spend running the tests, and so forth.

Keep in mind, though, that a high degree of code coverage isn’t a goal in itself. Instead, focus on writing solid, focused, and representative tests that eventually drive up code coverage.

Tooling

As far as .Net and tooling goes, MSTest or NUnit should be used in concert with TypeMock or Rhino Mocks. The use of TypeMock may be preferred over Rhino Mocks because of its unique approach to mocking. TypeMock doesn’t create fake objects by emitting MSIL and dynamically loading a runtime-generated assembly into the test runner. Instead, TypeMock hooks into the CLR APIs and intercepts calls as the unit test executes. From a coding point of view the TypeMock approach may not change much, but from a functionality point of view it enables the testing of legacy code or new code not written with IoC in mind.

for (int i = 0; i < 5; i++)
    // do something

No such for-construct exists in XSL. In fact there’re no looping constructs of any kind in XSL, except for the one used to iterate nodes. However, since XSL and C# are both Turing complete languages, and since anything that can be expressed in one Turing complete language can be expressed in another, it follows that it must be possible to express a loop with a counter in XSL.

But as Alan Parlis puts it in Epigrams on Programming:

Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy.

Although XSL and C# are both Turing complete languages, XSL is a functional programming language and therefore belongs to a strain of languages that forgo build-in looping constructs. Instead, looping is accomplished through recursion.

We can turn the iterative C# loop into a recursive one like so:

void Loop(int i) {
    if (i > 0) {
        // do something
        Loop(i - 1);
    }
}  

and transform the recursive solution into an XSL template:

<xsl:template name="Loop">
    <xsl:param name="i" select="5"/>
    <xsl:if test="$i > 0">
        <!-- do something -->
        <xsl:call-template name="Loop">
            <xsl:with-param name="i" select="$i - 1"/>
        </xsl:call-template>
    </xsl:if>
</xsl:template>

Okay, but what about nested loops then:

for (int i = 0; i < 5; i++)
    for (int j = 0; j < 5; j++)
        // do something

Giving it some thought, the nested loops (of any nesting level) can be expressed using recursion following a pattern of nested if-statements:

void NestedLoops(int i, int j, int k) {
    if (i > 0) {
        // do something
        if (j == 1)
            NestedLoops(i - 1, k, k);
        else
            NestedLoops(i, j - 1, k);
    }
}

Again, performing a one-to-one transformation to XSL:

<xsl:template name="NestedLoops">
    <xsl:param name="i"/>
    <xsl:param name="j"/>
    <xsl:param name="k"/>
    <xsl:if test="$i > 0">
        <!-- do something -->
        <xsl:choose>
            <xsl:when test="$j = 1">
                <xsl:call-template name="NestedLoops">
                    <xsl:with-param name="i" select="$i - 1"/>
                    <xsl:with-param name="j" select="$k"/>
                    <xsl:with-param name="k" select="$k"/>
                </xsl:call-template>
            </xsl:when>
            <xsl:otherwise>
                <xsl:call-template name="NestedLoops">
                    <xsl:with-param name="i" select="$i"/>
                    <xsl:with-param name="j" select="$j - 1"/>
                    <xsl:with-param name="k" select="$k"/>
                </xsl:call-template>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:if>    
</xsl:template>

Wordy, but seeing through the angle brackets I find solution quite elegant.

Consider the state of computer-based word processing in 1977 when WordPerfect was conceived of. Afraid that word processors would interfere with so-called important applications, companies did word processing on dedicated mini-computers and mainframes. These machines were operated by a word processing department of employees that typed up documents. It wasn’t until the IBM PC that corporate users began typing documents themselves.

The early years

Pete’s way into WordPerfect — and his initial exposure to computers — was by way of his brother-in-law, Bruce Bastian. Following Bruce’s computer science degree in 1978, Bruce joined Alan Ashton to realize Alan’s vision of a word processor for a Data General mini-computer. It was then, in 1979, that Pete joined the two as office manager. By 1992, when forced to leave WordPerfect, he’d turned vice president of marketing.

Alan’s unique ideas involved crafting a word processor that didn’t require entering codes to format and layout documents. Compared to HTML authoring of today, contemporary word processors required the author to mix text and codes. Like tags in HTML, the codes described properties of the document, such as margins, line feeds, and emphasized text, and were specific to each product. Alan’s word processor, although character-oriented by nature, was to display the underlying codes using textual effects, creating a rough WYSIWYG experience.

Other innovative features included the ability to scroll through a document line by line, rather than page by page, and non-modal editing. Contemporary word processors required the author to switch between command and text modes the way the Vi family of editors still do. Without menus, modes made it easier to separate commands from text. Nonetheless, WordPerfect was modeless and hence characters could be typed in anywhere and anytime.

In the late seventies, word processors (and operating systems) were largely written in assembler and hence required rewriting for each hardware platform. Up until the IBM PC version, assembler also dominated WordPerfect, which made it a challenge to maintain considering the plethora of supported platforms. Migrating to higher-level languages had been considered, but was discarded on the premise that it would result in slower applications with bigger memory footprints. On that note, it appears we haven’t learned as we keep rewriting the same fundamental applications and frameworks. Not in assembler, but in higher-level languages.

Microsoft catching up

The 1981 introduction of the IBM PC pawed the way for a shift in the way software was developed and marketed. For vendors, the common architecture made lock-ins harder, yet provided access to a size of user base never before seen. WordPerfect was quick to seize the opportunity and released WordPerfect for the IBM PC in 1982.

Although the software marked for the IBM PC was dominated by Microsoft, for years Word (and other word processors) couldn’t keep up with the features and sales of WordPerfect. Two technologies, however, enabled Microsoft to overtake WordPerfect: laser printers and graphical user interfaces. For years WordPerfect had supported a few laser printers and had only a text-based interface. By 1987, though, it had become clear that graphical user interfaces was the future. It just wasn’t obvious weather Windows, of which version 2 had just been released or OS/2, whose development was announced that year, or some other vendor would end up dominating the market.

The final years

Moving from text to graphics posed a challenge for WordPerfect. Under DOS, the operating system provided only rudimentary services for drawing on the screen. For proportional font and image rendering, WordPerfect would itself have to take on rendering or use the services of a graphical user interface. But since it wasn’t clear whether Windows, OS/2, or some third environment were to dominate the future, WordPerfect decided to only augment the text-based interface with a graphical preview page. At the same time Microsoft released its first version of Word for Windows 2.0, relying on Windows for proportional font rendering, image display, and decent laser printer support.

In light of present knowledge, it’s all too easy to pass judgment on past decisions. Nevertheless, it appears that the inability of WordPerfect to keep their eyes on the rolling ball had caused Microsoft to take over by the early nineties. Instead of focusing on WordPerfect for Windows, substantial resources went into developing versions of WordPerfect for other platforms, developing a spreadsheet and a database application, and many more.

For years the availability of WordPerfect on almost any conceivable platform had supported the growth of the company. Now, the inability to cut losses and retain focus dragged WordPerfect into oblivion.

Non-authoritative DHCP servers on the network have posed a challenge for years. At first, I attributed the DHCP servers to badly configured Windows machines. However, the number of DHCP servers suggested that another explanation might be closer to the truth. Users accidentally installing DHCP servers are just not that common, and inspecting a couple of machines in question they were acquitted. What else could be the source of the DHCP traffic, then? Common to the machines were that wireless routers sat between them and the LAN.

After much investigation, it turns out that a lot, if not all, wireless routers expose their DHCP server on all ports of the router, when it’s only necessary to expose it on inward-facing ports. I guess vendors don’t bother limiting the exposure because generally it’s of no concern to consumers. ISPs provide the wireless router with the necessary configuration parameters (IP address, subnet mask, default gateway, DNS servers, and so on) in addition to blocking DHCP traffic originating from the wireless router.

On our network — equipped with eight aging 48-port Cisco Catalyst 3500 XL switches — DHCP traffic cannot be prevented from escaping the wireless router and making its way into the switches and onto LAN. The configuration parameters of rogue DHCP servers, intended for internal use only, are transmitted to external DHCP clients. Clients for which the supplied gateway, DNS server, and so on are only internally available. The DHCP client, unable to discern the rogue servers from the authoritative one, passes the parameters on to the network stack, setting itself up for network disconnectivity.

The essence of the issue is that whenever a client requests configuration parameters, it, by definition of the DHCP protocol, broadcasts a message to be picked up by any DHCP server on the network. The client then awaits an offering of configuration parameters from one or more DHCP servers. Then, still according to specification, the DHCP client may use any strategy to accept one of several offerings. I can only guess as to how Windows makes its decision, but randomness and first responder probably play a key role. In any event, Windows may keep accepting offerings from rogue DHCP servers regardless of the number of times I command the IP configuration to “ipconfig /release” and “ipconfig /renew”.

How to counteract rogue DHCP servers? There’re only a limited set of options: hard-coding the configuration parameters of individual machines, establishing VLANs so computers become invisible to each other, or replacing the Cisco Catalyst 3500 switches with contemporary switches that are able to suppress DHCP traffic travelling the wrong way.

The first step along the way is the maxDisplacements function that, given a number of splits of line segments, returns a list of maximum displacements. Here list comprehension comes in handy as a way to generate maximum displacements where the next is half of that of the previous, starting with 0.5:

let maxDisplacements n = [for i in 1 .. n -> 2.0**(float)(-i)]

Next is the composeCoordinates function whose job it is to infer the x coordinates from the y coordinates. When splitting line segments, as defined later by the split function, only y coordinates need to be dealt with. But for displaying the result as a list of points, e.g., as input to a graphing program, composeCoordinates generates the (x, y) coordinate pairs given a list of y coordinates. Again list comprehension is used to uniformly distribute the x coordinates in the range 0 to 1. Next, the x and y coordinates are fused using the build-in zip function, i.e., [x1; x2] and [y1; y2] become [(x1, y1); (x2, y2)].

let composeCoordinates (ys : list<float>) =
    let dx = 1.0 / (float)(ys.Length - 1)
    let xs = [for x in 0 .. ys.Length - 1 -> (float)x * dx]
    List.zip xs ys

Now for the crux of the fractal generation process: the split function. As is customary with functional languages, looping is done using recursion and split is no exception. Passing in a list of y coordinates, a maximum displacement as generated by maxDisplacements, and a random number generator, split is defined as:

let rec split (ys : list<float>) displacement (random : Random) =
    match ys with
    | a :: b :: tail -> 
        let m = b - a / 2.0
        let r = random.NextDouble() * displacement
        a :: a + m + r :: split (b :: tail) displacement random
    | a -> a

Using pattern matching, an if-then-else construct on steroids, split examines its input of y coordinates to determine if the inductive or the base case of the recursive implementation needs to be invoked. The first pattern matches a list starting with elements a and b following by zero or more elements. In case this pattern matches, a new list is returned. It’s composed of the original a element, a new displaced element, the original b element, and split applied to the original list with its first element removed. Removing one element at a time, at some point only one element is left in the list in which case the base case is invoked. When this happens, all the original line segments have been split in two.

Finally, the main function ties together the previous ones. Given an initial line segment, the number of times to apply split to it, and a random number generator, the actual splitting is done using the build-in fold_left function. fold_left is an example of fun with folds and a function that is itself recursively defined to make looping implicit.

let main() =
    let initial = [0.0; 0.0]
    let splits = 8
    let r = new Random()
    let ys = List.fold_left (fun s d -> split s d r) initial (maxDisplacements splits) 
    print_any (composeCoordinates ys)

While initially hard to grog, the use of fold_left becomes clearer when applied to a list of integers to compute their product. Of the three arguments to fold_left the first is a function defined by a lambda expression (here the predefined multiplication operator is also applicable but less explicit). The second argument is the initial value of an accumulator. And finally, the third argument is the list to work on.

List.fold_left (fun x y -> x * y) 1 [1;2;3]
List.fold_left (*) 1 [1;2;3]

What fold_left does is recursively call itself passing along the first argument as is. The second argument is the new value of the accumulator, computed by passing to the function of the first argument the original value of the accumulator and the first element of the list. That way it’s up to the function passed as the first argument to decide what operation to apply to the elements in the list. The third argument is the original list with the first element removed. The recursive chain of calls with their arguments then takes on this form:

List.fold_left (fun 1 1 -> 1 * 1) 1 [1;2;3]
List.fold_left (fun 1 2 -> 1 * 2) 1 [2;3]
List.fold_left (fun 2 3 -> 2 * 3) 2 [3]

Within the main function the first argument to fold_left is a lambda expression that applies the split function to a list. The second argument, the accumulator, isn’t a scalar like with the product example, but a list of line segments to be passed to split. Finally, the third argument is the maximum displacement list. And so for the original line segment to be split n times, n maximum displacements must be provided by the maxDisplacements function.

What’s so fascinating about the Mandelbrot set, and other fractals, is that the often simple equations that define them are able to give birth to such complex creations.

In general, a fractal is defined as:

“[…] ‘a rough or fragmented geometric shape that can be split into parts, each of which is (at least approximately) a reduced-size copy of the whole,’ a property called self-similarity. A fractal often has the following features: it has a fine structure at arbitrarily small scales; it is too irregular to be easily described in traditional Euclidean geometric language; it is self-similar (at least approximately or stochastically); [… and] it has a simple and recursive definition. […] Natural objects that approximate fractals to a degree include clouds, mountain ranges, lightning bolts, coastlines, and snowflakes.”

The Mandelbrot set is an example of a fractal whose definition contains no stochastic element, i.e., the fractal looks the same every time it’s generated. Recently, though, I came across a simple algorithm that claimed to generate fractal terrains by adding randomness to the equation. So, given my history with fractals, I wanted to play with the algorithm that’s based on progressive refinement through midpoint displacement in one dimension:

Start with a single horizontal line segment
Repeat for a sufficiently large number of times
    Repeat over each line segment
        Find the midpoint of the line segment
        Displace the midpoint in y by a random amount
    Reduce the range for random numbers

Here’s my implementation of the algorithm in C#:

using System;
using System.Collections.Generic;

namespace FractalTerrain {
    class Point {
        public double x;
        public double y;
    }

    class Program {
        static void Main(string[] args) {
            int iterations = 8;
            double roughness = 0.5;
            double displacementFactor = 1.0;
            Random generator = new Random();
            double[] ys = { 0.0, 0.0 };

            for (int i = 0; i < iterations; i++) {
                ys = Split(ys, displacementFactor, generator);
                displacementFactor *= roughness;
            }

            Point[] points = ComposeCoordinatePairs(ys);
            foreach (Point p in points) {
                Console.WriteLine("{0:0.000} {1:0.000}", p.x, p.y);
            }
        }

        static double[] Split(double[] ys, double displacementFactor, Random generator) {
            double[] newYs = new double[2 * ys.Length - 1];

            for (int i = 0; i < ys.Length - 1; i++) {
                double midpoint = (ys[i] + ys[i + 1]) / 2.0;
                double displacement = generator.NextDouble() * displacementFactor;
                newYs[2 * i] = ys[i];
                newYs[2 * i + 1] = midpoint + displacement;
            }

            return newYs;
        }

        static Point[] ComposeCoordinatePairs(double[] ys) {
            double increment = 1.0 / (ys.Length - 1);
            Point[] points = new Point[ys.Length];

            for (int i = 0; i < ys.Length; i++) {
                points[i] = new Point();
                points[i].x = increment * i;
                points[i].y = ys[i];
            }

            return points;
        }
    }
}

Splitting line segments, the C# code doesn't store the x-component of the coordinate pairs because it can be inferred from the number of y-components and the fact that x-components go from 0 to 1, both inclusive. Hence, starting with the line segment (0,0)-(1,0), the code splits it by calling the Split method passing in a list of y-components of the line segment. After the first iteration the single line segment becomes two, then four, then eight, and so on until the line segments have undergone eight iterations of splitting. Generally, after the i'th iteration the number of line segments is 2^i and so after eight iterations we end with 256 line segments.

As prescribed by the algorithm, merely splitting line segments doesn’t give rise to a mountain. The mountain emerges because each time a line segment is split in two, a random displacement is added to the y-component at the center of the line segment. Assigning an upper limit to the maximum displacement within an iteration is what defines the roughness of the generated terrain. For the above implementation, the maximum displacement is defined as half of that of the previous iteration starting with 0.5. Thus, the first couple of iterations roughly define the terrain and additional iterations fill in the details.

In animated form the transformations from two to 256 line segments looks like so:

Among the practical applications of such a random fractal terrain generator are computer games. Not necessarily for generating 2D ridgelines, but for generating entire 3D terrains or even clouds. It works by generalizing 1D midpoint displacement to 2D in the form of the diamond-square algorithm. Then a cloud, for instance, can be derived from the 3D terrain by applying a height map to it; a height map in which different heights get assigned different colors.

The reason for bringing up the linked file feature in conjunction with assembly signing is that to me the feature seems particularly well-suited for linking one and only key file into multiple Visual Studio projects. Yet people seem to be unaware of the feature and so common approaches to signing is to either create a new key file for each assembly — most likely because Visual Studio makes it so easy to create a new key file from the Project Settings dialog — or signing assemblies by creating one key file and then copy/paste it to other projects in the solution.

Below is an example from the GAC showcasing how assemblies related to Pex naturally go together and are therefore signed using the same key (they share 76a274db078248c8, their public key token). The same goes for assemblies relating to other Microsoft products, such as the ones comprising a particular version of the .Net framework (because of the side-by-side execution, different versions of .Net framework assemblies are signed using different keys). So why not apply the same principle to what you deploy to the GAC?

While using the same key (copy or not) makes it possible to differentiate your assemblies from everyone else and for your four-part assembly name in web.config and spread around the code to be uniform — SharePoint developers often need to provide the four-part name for their assemblies to other parts of the solution — copying the key around is still a violation of the DRY principle and as such is indicative of a code smell. More importantly, using the link feature of Visual Studio, there’s no reason to keep the bad smell around.

Assuming you’ve already created a solution (in Visual Studio 2008) and added a project to it, the way to share a key file is to first create one by right clicking a project in Solution Explorer and selecting Properties. Now click the Signing tab and check the Sign the assembly check box. The dropdown lets you create a new key file by selecting New and filling out the information in the Create Name Key window. Alternatively, a key file is created using sn.exe, the strong name utility that comes with the Microsoft Windows SDK. On my machine, sn.exe is located in C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin and is invoked like so:

% sn -k MySharedKeyPair.snk

How the key file is created isn’t particularly relevant to the rest of this post. Just make sure to place the key file in a common location, e.g., the top-level directory holding the Visual Studio solution file. Now go to Solution Explorer, right click on your project, and select Add existing Item. Then locate the key file within the solution directory and make sure to click the arrow on the Add button and select Add as Link. Note how the file is added to the project and marked by an arrow indicating it’s a link:

Returning to the Signing tab and selecting the dropdown, it now includes the linked key file. Don’t mind the absolute path; once you select MySharedKeyPair.snk, Visual Studio converts the path to a relative one (close and reopen the Signing tab and see for yourself). Also note that if you select Browse, the key file you select is copied to the project folder and added to the project as a regular file.

Another way to associate a key file with an assembly is through the AssemblyKeyFile attribute. The attribute instructs the compiler of where to look for the key file and, although not used by the runtime, the attribute is included in the assembly along with any other attribute for everyone to see.

Using AssemblyKeyFile is probably the simplest approach to sharing a key file between projects, but now compiling the assembly results in a compiler warning as displayed above. The use of the AssemblyKeyFile attribute has been deprecated by Microsoft because it may lead to you inadvertently disclosing sensitive information through the embedded path and because working with a relative path may confuse users and the build system (the relative path is retained in the assembly, though). However, it appears AssemblyKeyFile is still the preferred way for MS to sign their assemblies. Inspecting a couple of .Net 3.5 assemblies with Reflector, here’s what their AssemblyKeyFile looks like:

[assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\EcmaPublicKey.snk")]
[assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\35MSSharedLib1024.snk")]
[assembly: AssemblyKeyFile(@"f:\\dd\\tools\\devdiv\\FinalPublicKey.snk")]
...

To come full circle, the question of how Visual Studio, or rather MSBuild, actually signs an assembly using the information provided in the Properties dialog (real file or link) remains to be answered. Since a Visual Studio project file acts as input to MSBuild, we can see what’s going on under the hood of MSBuild by invoking a compile from the command line. On my machine msbuild.exe is located in C:\Windows\Microsoft.NET\Framework\v3.5 and assuming you have a console window open with its current directory set to the directory of your project file, msbuild is invoked like so:

% msbuild /verbosity:diagnostic > log.txt

Bring up log.txt in your favorite editor and eventually you’ll come across where csc.exe, the C# compiler, is invoked:

As you might have guessed, MSBuild carries over the location of the key file by passing the /keyfile argument to csc.exe.

In order to test code whose branching behavior depends on a value read from a configuration file, the test should provide various inputs and assert that the code under test acts accordingly. In addition, the test should run in milliseconds to encourage frequent runs and it should be self-contained, meaning no setup should be required before running the test. These requirements are best met with the web part running in a controlled environment. Yet, as far as the code under test goes, it should behave as if part of an Asp.Net page.

For the sake of brevity, here’s my web part with the code to initialize the data context:

public class MyWebPart : WebPart {
    MyDataContext _db;    /* LINQ database context */
    string _error;        /* Early stage error message */    	

    public MyWebPart() {
        Initialize();
    }

    private void Initialize() {
        string c = GetMyConnectionString();
        if (c == null)
            _error = "Unable to do something";
        else
            _db = new MyDataContext(c);
    }

    private string GetMyConnectionString() {
        System.Configuration.ConnectionStringSettings c = 
            System.Configuration.ConfigurationManager.
                ConnectionStrings["MyConnection"];
        return c == null ? null : c.ConnectionString;
    }
        
    protected override void Render(HtmlTextWriter w) {
        // display error message if set
    }
}

Notice how the constructor calls out to an Initialize method. Otherwise code within the constructor would fire before I’d have a chance to setup a controlled environment around the web part and I wouldn’t be able to test it. Turning the attention to Initialize itself, it’s made up of only a simple if statement. The idea is that if I can’t mock something as simple as reading a value from a configuration file, most likely I can’t mock something more involved either. On the other hand, for a developer already familiar with TypeMock and test first development, the code above may be born of the need to satisfy a failing test.

I want to test Initialize by controlling how GetMyConnectionString gets to return the connection string. While running in an Asp.Net context, GetMyConnectionString would attempt to read the string from web.config. Within my controlled environment, however, I want to inject into the web part an alternate implementation that returns a value of my choosing. That way I can control the path through Initialize and inspecting the state of the _error or _db fields, I’m able to determine if the code behaved as intended.

Using the method chaining DSL of TypeMock, the test takes on this form (of course, having more tests, I’d move shared code into helper methods decreasing the test to code ratio):

[TestClass, ClearMocks]
public class MyWebPartTest {
    [TestMethod, Isolated, VerifyMocks]
    public void Should_set_error_message_when_no_
                connection_string_is_configured() {
        // arrange
        MyWebPart fake = Isolate.Fake.Instance<MyWebPart>();
        Isolate.NonPublic.WhenCalled(fake, 
            "Initialize").CallOriginal();
        Isolate.NonPublic.WhenCalled(fake, 
            "GetMyConnectionString").WillReturn(null);
        PrivateObject handle = new PrivateObject(fake);

        // act
        handle.Invoke("Initialize", null); 

        // assert
        Isolate.Verify.NonPublic.WasCalled(fake, 
            "GetMyConnectionString");
        string error = (string)handle.GetField("_error");
        MyDataContext ctx = (MyDataContext)handle.GetField("_db");
        Assert.AreEqual("Unable to do something", error);
        Assert.AreEqual(null, ctx);
    }
}

There’re three parts to tests using TypeMock: (1) the arrange part sets up expectations of what’s going to happen when a method, a property, or some other part of the code under test is invoked; (2) the act part carries out the actual interaction with the object under test, turning to PrivateObject to invoke the Initialize method; and finally (3) the assert part verifies that what was supposed to happen did happen. Worth noting is that this test asserts both state and behavior, i.e., as part of executing the code under test, _error or _db should’ve been set to an appropriate state and GetMyConnectionString should’ve been called once and only once.

This concludes my example, which I believe illustrates the basics of testing and mocking using MS Test and TypeMock. Comparing and contrasting TypeMock to other popular mocking frameworks, TypeMock has the distinct feature that it can mock almost anything, with or without using interfaces. Some argue that not enforcing the use of interfaces is also the disadvantage of TypeMock, because for code to be truly testable, it should be modularized using the Inversion of Control principle. However, with legacy code, such as the .Net framework or the SharePoint API, to me TypeMock appears to bridge the two worlds nicely, leaving the developer in charge.

To learn more about the basic concepts of mocking, Inversion of Control, and Dependency Injection (containers), I’d recommend listening to a couple of Alt.Net podcasts: Dependency Injection and Inversion of Control and More Dependency Injection and Inversion of Control.

(Other posts in this series include: Kernel space traffic shaping with Linux and User space traffic shaping with Ruby that touch exclusively on Linux issues.)

This post covers an Asp.Net MVC application (for an overview of Asp.Net MVC, the Herding Code guys recently interviewed Phil Haack) I wrote that automates the collection of information about computers and their owners on a local area network. The idea is to have the Internet gateway match the identity of each computer that wants to send traffic through it against a white list. Should the request originate from an unknown computer, the gateway redirects that computer to a web application for Internet access activation. Each user then has to create an account and provide verifiable contact information and/or associate their computer with an existing account.

With the contact information of every connected user, maintaining a network with 331 connected apartments, 371 users, and 513 privately owned computers becomes more manageable. Now you can easily email all users about general issues or individual users about specific issues, such as wireless routers exposing rogue DHCP servers or virus slowing down the network.

Without being able to contact a user, the best you can hope for is to use the MAC address to locate and disable the port on the switch to which the user is connected. But terminating network access without warning or explanation, the user has no way of knowing what hit him. Now when the user contacts you, figuring out if he’s experiencing network issues because of a closed port or something entirely different quickly becomes a challenge — and something that doesn’t scale well.

That’s where the Asp.Net MVC application comes in. To get a feel for how the software accomplishes its task, you should watch the minute and a half demo. It portrays the browsing experience of a user connecting his computer to the network for the first time and attempting to browse the web. Behind the scenes the Linux gateway uses its Netfilter capabilities to efficiently match the MAC address of each request against the white list. With no match for the computer just connected, upon browsing the web, Netfilter redirect the user’s browser to the web application. Then, every so often, using a Ruby script, the gateway queries the web server for updates to the white list and carries over the changes to Netfilter.

Database schema

The web application is based on the idea that a user is responsible for zero or more computers. Focusing on the web application, rather than the Linux part, the rest of this post highlights a few features that use and manipulate the data structures stored in a MS SQL Server Express database:

Reverse lookup of MAC

Establishing an ownership between a user and a computer is done via the MAC address of the computer. Now, it wouldn’t be particularly user friendly if the user had to go about locating the MAC address and manually typing it into a web form. Instead, the web application looks up the user’s MAC address within the ARP cache of the web server.

Because the user’s computer and the web server are both on the same local network and because they’re already exchanging data (the user is visiting a web page hosted on the server), the parties are effectively communicating through the ARP protocol of the data link layer. The data entry form is therefore able to present the user with a control that makes associating computers to the account easy. By reverse lookup, the computer currently browsing the web application is indicated by “Now logged in from here”.

Email verification

Most users don’t mind handing over basic information such as their email address. And as users are blocked from accessing the Internet, at the time of entry we only validate the email address for syntactic correctness.

A few users, however, repeatedly entered fake email addresses to gain Internet access. Consequently, we send out an email to the address, leaving open a 24 hour window for the user to follow a link. Otherwise, the software sends out a reminder email, marks the account as inactive, and redirects the user to the web application.

Validating the syntax of email addresses is done using regular expressions. As with most complex regular expressions, they’re hard to read and verify the correctness of. They do, however, stand the test against a database of hundreds of email addresses. To come full circle, though, what is needed is a way to match the domain part of the email address against the DNS MX record of the domain. Unfortunately, MX lookup isn’t build into the .Net framework.

Unhalted exception emailing error handler

Error handling deserves a post on its own. Suffice it to say that to stay on top of any application, unhalted exceptions should be logged, or in this case, emailed to a designated address. Inevitably, users of your software will use it in unanticipated ways and so a global exception handler provides invaluable insight to learn from.

(Summary part of example email. Click here for complete output.)

Download and maybe try out

The web application was developed around February 2008; about the time the Asp.Net MVC framework went into Preview 2 and had started gaining momentum. Unfortunately, developing against this early a release, now the web application doesn’t run on a computer with .Net 3.5 SP1 installed.

There’s no problem compiling the application because the Preview 2 bits are in the bin folder of the application. But running the application generates an exception stating:

Could not load type 'System.Web.HttpContextWrapper2' from 
assembly 'System.Web.Abstractions' [located in the GAC]

That’s because the version of System.Web.Abstractions.dll that ships with .Net 3.5 SP1 no longer holds the HttpContextWrapper2 class. On a machine without .Net 3.5 SP1, however, Cassini runs the application just fine. Before running it, though, remember to create a database from the schema in Database.sql and point to that database from web.config.

Lastly, I should stress that the software is a prototype and served as a way for me to wrap my head around ASP.Net MVC and LINQ. So, the code may not win a beauty contest.

Conclusion

The activation software went online late March 2008. During the first couple of weeks I had to put in a couple of bug fixes based on what I learned from the unhalted exception emails. Since then, however, the software has served the purpose it was charged with.

The only issue that I haven’t been able to resolve is why, on rare occasions, redirecting a computer confuses its browser: if you visit a page and is redirected to the web application, for a short time after, the page keeps resolving to the web application. Most likely it’s a residual effect of the packet rewriting taking place on the gateway.

Another thing I wish I’d implemented was tracing to better understand how a couple of users got to experience a few exceptions that I don’t know how to interpret from the unhalted exception emails alone.

In any real-world application, unhalted exceptions are an unavoidable fact of life. Such exceptions should occur relatively infrequently, but when they do some kind of supporting infrastructure better be in place to capture the when and why of the exceptions. As a developer, you should have the ability to perform a postmortem analysis on what went wrong and learn from it. Paramount to such analysis is capturing and logging sufficient information at the point of failure.

What this post covers is the thoughts and development of an Asp.Net HttpModule that captures and emails such unhalted exception information to a designated email address.

To be clear, the goal of the EmailingExceptionModule isn’t to prevent unhalted exception from occurring per se. Within code, a developer may not know how to handle an exception, and so the exception should rightfully propagate the call stack. In case no caller steps in and handles the exception, the best choice is most likely to terminate the application. It’s almost always better to be upfront with the user than conceal the possibly inconsistent state of the application.

So when does the EmailingExceptionModule come in handy then? One scenario is that of C# unchecked exceptions, where the compiler doesn’t force the caller to catch all types of exceptions thrown by the callee. Unchecked exceptions may materialize as type cast or null reference exceptions when accessing variables. What unchecked exceptions boil down to is that, given the cyclomatic complexity of some methods, it’s impractical to manually work through every conceivable path of execution ahead of time. Another scenario is problems with the runtime environment, such as out of disk space or database server down. Nonetheless, some map of the path to failure is helpful in preventing others going down that same path.

(Summary part of example email. Click here for complete output.)

With web applications, we can take advantage of the application running in a centralized environment. In the spirit of Microsoft’s Doctor Watson technology, we can register our own web application error handler and hook it into the Http request pipeline, and have the application execute our code on unhalted exceptions. In practice, such an error handler is implemented either through the Application_Error method of an application’s Global.asax or by loading an HttpModule into the application.

Hinted by the title, I went for the HttpModule. The reason may best be understood by taking a peak behind the .Net curtains: Global.asax is a filename hardwired into the framework so that whenever an application is first hit, the framework asks HttpApplicationFactory, an internal factory class, for an HttpApplication representing the application (there’s a unique one for each virtual application). As part of manufacturing the HttpApplication, the factory locates the application’s Global.asax, compiles it into a dynamically generated DLL, loads the DLL, and reflects over the Global.asax class looking for methods adhering to the convention of modulename_eventname. Methods found, such as Application_Error, are then stored in a list of MethodInfos and passed along to HttpApplication. Then, during HttpApplication initialization, events are bound to the methods within Global.asax.

That’s how Application_Error of Global.asax gets called even though the method isn’t overriding a base class implementation as is typical for the template pattern. That’s also why code within global.asax isn’t binary reusable across applications. An HttpModule, on the other hand, can be loaded into any number of applications.

In practice, implementing an HttpModule it a matter of creating a class that implements the IHttpModule interface:

public interface IHttpModule {
    void Init(HttpApplication context);
    void Dispose();
}

The crux of error handling with Asp.Net is the Error event of HttpApplication. The easiest way to hook up the event to a method is within the Init method of the class. In addition, we assign the application object to a field so that later we can access the most recently thrown exception through it:

public class EmailingExceptionModule : IHttpModule {
    private HttpApplication _application;

    public void Init(HttpApplication a) {
        _application = a;
        _application.Error += OnUnhaltedException;
    }

   public void Dispose() {}
   
   private void OnUnhaltedException(object sender, EventArgs e) {
       Exception ex = _application.Server.GetLastError().InnerException;
       // Implementation left out for brevity. Please download source 
   }
}

The OnUnhaltedException method is where to add code that collects information about the exception and the environment, and that composes and ships the email. Depending on the environment, you may find the need to include additional information in the email. Given the source code, adding to the email is a matter of adding key/value pairs in the form of labels and values to a dictionary. Each dictionary then gets rendered as a table of the key and value columns.

To have an Asp.Net application load the EmailingExceptionModule and to configure its email related settings, add the following nodes to the application’s web.config:

<configuration>
    <appSettings>
        <add key="EmailingExceptionModule_SenderAddress" value="sender@domain.com"/>
        <add key="EmailingExceptionModule_ReceiverAddress" value="receiver@domain.com"/>
        <add key="EmailingExceptionModule_SmtpServer" value="mail.domain.com"/>
        <add key="EmailingExceptionModule_SubjectPrefix" value="MyApp: "/>
    </appSettings>
    <system.web>
        <httpModules>
            <add name="EmailingExceptionModule" type="Holm.AspNet.EmailingExceptionModule,
                    EmailingExceptionModule, Version=1.0.0.0, Culture=neutral, 
                    PublicKeyToken=13806f613f05e959"/>
        </httpModules>
    </system.web>                                    
</configuration>

So what’s the user experience of unhalted exceptions? Outside the confines of the EmailingExceptionModule, there’s no indication that the application took note of what happened. Depending on the user’s profile, it might be helpful to prompt for additional information at the crash point. Or perhaps turn to the Windows Live Messenger IM Control & Presence API and have the user engage in an MSN conversation with a developer.

What’s the advantage of emailing developers unhalted exceptions over storing the information in the file system or a database? I believe in keeping things simple and practical, and handing off information to the file system or a database isn’t. Most likely the information will end up gathering virtual dust on some server. The email approach, on the other hand, is a proactive, visible, and efficient means to the end of rapidly course-correcting for wrong assumptions.

Update, Sep 15: While you don’t need a PDB file to debug code through Visual Studio (that’s what <compilation debug=”true”> in web.config is for), the PDB file is required for the stack trace to contain line numbers. Since the PDB file typically has to reside next to the corresponding DLL for the .Net runtime to pick it up, the PDB file may need to be deployed to the GAC along with the DLL. Refer to the first three paragraphs of the “Debugging assemblies that live in the GAC” section of this post for how to GAC deploy PDB files.

For a centrally deployed Asp.Net application, it may be acceptable to ship and deploy the PDB file with the application. Keep in mind, though, that because the PDB file is a mapping of locations in the IL to the source file, having access to the PDB file makes it easier for someone to reverse engineer the code. It’s also possible to get at line numbers without the PDB file being deployed, but the approach is somewhat involved. Based on the mapping nature of the PDB file, the idea is to write out IL offsets as part of the exception and to post-process the offsets using a PDB file at a separate location.

Yet another alternative to shipping the PDB file is to setup a symbol server. With this nifty piece of code, the StackTrace class loads the PDB file of the symbol server, adding line numbers to the stack trace.

To showcase anti-patterns of API usage, intermediate code is included, i.e., code that may render the correct tags, although not necessarily in the true spirit of the .Net framework.

To set the stage, imagine writing a web control that inherits from ASP.Net WebControl. The goal is then to modify the behavior of the control in such a way that it (1) encloses its content in a div tag and (2) adds a CSS class for styling, and (3) can be added to a page like so:

<MyControls:MyControl CssClass="myCssClass" runat="server" />

After making its way through the page rendering pipeline, the control tag gets transformed to:

<div class="myCssClass">
    Hello World
</div>

Within the control, a straight-forward way of achieving the intended output may be to hand craft the html and render it in RenderContents of WebControl. It almost goes without saying that rendering html by hand is but only a starting point when working with .Net. The next progression from there may be to override RenderBeginTag and RenderEndTag of WebControl:

// version 1
public class MyControl : WebControl {
    public override void RenderBeginTag(HtmlTextWriter w) {
        w.RenderBeginTag(string.Format("div class=\"{0}\"", CssClass));
    }
    
    public override void RenderEndTag(HtmlTextWriter w) {
        w.RenderEndTag();
    }

    protected override void RenderContents(HtmlTextWriter w) {
        w.Write("Hello World");
    }
}

Resulting in this piece of html:

<div class="myCssClass">
    Hello World
</div class="myCssClass">

Clearly, RenderBeginTag and RenderEndTag of HtmlTextWriter don’t know how to separate tag from attribute. Nonetheless, the code reveals the stack based approach taken by the Render-pair of methods.

Gleaning over the documentation of HtmlTextWriter, however, I encountered the AddAttribute method for setting attributes on a tag. Inside HtmlTextWriter, what AddAttribute does is add attributes and associated values to an array that gets rendered during the next call to RenderBeginTag. So besides introducing strongly typed attributes and tags, the next version contains a modified RenderBeginTag:

// version 2
public override void RenderBeginTag(HtmlTextWriter w) {
    w.AddAttribute(HtmlTextWriterAttribute.Class, CssClass);
    w.RenderBeginTag(HtmlTextWriterTag.Div);
}

Now the control renders as expected, although the code still has a certain bad smell to it. WebControl ought to know how to properly render its begin and end tag. After all it’s WebControl that exposes, among others, the CssClass property. So taking another dive into the documentation, I found the virtual, read-only TagKey property of WebControl. Strangely named as it may be, overriding TagKey makes WebControl call our override as it renders the control, adding in optional attributes, such as CssClass. Implementing the property also turns RenderBeginTag and RenderEndTag obsolete:

// version 3
protected override HtmlTextWriterTag TagKey {
    get {
        return HtmlTextWriterTag.Div;
    }
}

As an added bonus, changing the enclosing tag can also be done calling the WebControl’s constructor, passing in the desired tag:

// version  4
public class MyControl : WebControl {
    public MyControl() : base(HtmlTextWriterTag.Div) { }

    protected override void RenderContents(HtmlTextWriter w) {
        w.Write("Hello World");
    }
}

As it turns out, calling the constructor is how WebControl derived classes get to use span in the first place. The default WebControl constructor passes control onto another constructor that sets the value of what gets returned by the base class TagKey implementation. In essence, overriding TagKey makes it a player in the template design pattern:

// .Net framework code
protected WebControl() : this(HtmlTextWriterTag.Span) { }

public WebControl(HtmlTextWriterTag tag) {
    tagKey = tag;
}

Also worth noting is that for controls deriving from WebControl derived classes, such as CompositeControl, the constructor cannot be used to set the enclosing tag. The constructor exposed by CompositeControl and friends doesn’t call up the constructor chain passing in the tag, making TagKey the choice to go.

Finally, deep down the inhertitance chain below WebControl lives WebPart, a type of control very popular with SharePoint. Interestingly enough, WebPart defaults to using div as its enclosing tag, because walking up the chain, we come across the constructor of Panel:

// .Net framework code
public Panel() : base(HtmlTextWriterTag.Div) { }

So, using Panel provides another way to implement a web control that owns a distinct part of a page.

To decide on the items to display, an initial approach might be to go through the list one item at a time, looking for matching tags. But working with large lists in SharePoint this approach is not recommended. Alternatively, the predicate against which each item in the list is evaluated could take on the form of a CAML query. Looking for items matching a single tag is then easily expressed:

<Where>
    <Eq>
        <FieldRef Name='MyField' />
        <Value Type='LookupMulti'>tag1</Value>
    </Eq>
</Where>

Next, consider a query with two tags OR’ed together. Although more verbose, the query is still fairly straightforward to compose on the fly:

<Where>
    <Or>
        <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag1</Value>
        </Eq>
        <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag2</Value>
        </Eq>
    </Or>
</Where>

However, for CAML queries where more than two tags are OR’ed together, the binary nature of the OR operator works against us. Where real programming languages have their parsers automatically transform an expression like (a || b || c) into ((a || b) || c), the CAML query parser applies no such transform — In a sense CAML is more like an XML serialized, abstract syntax tree than a query language intended for direct use:

<Where>
    <Or>
        <Or>
            <Eq>
                <FieldRef Name='MyField' />
                <Value Type='LookupMulti'>tag1</Value>
            </Eq>
            <Eq>
                <FieldRef Name='MyField' />
                <Value Type='LookupMulti'>tag2</Value>
            </Eq>
        </Or>
        <Eq>
            <FieldRef Name='MyField' />
            <Value Type='LookupMulti'>tag3</Value>
        </Eq>
    </Or>
</Where>

So how do we go about composing a query that OR together n tags (not to mention a mix of ORs and ANDs)? One approach might be using the U2U CAML Query Builder. The tool delivers a UI for formulating queries, but also exposes its CAML query builder API as a .Net assembly. Unfortunately, documentation on how to use the builder is sparse. Nonetheless, I went for the case of composing a query that OR together three tags:

string ComposeCamlQuery() {
    Builder b = new Builder(CamlTypes.GetListItems);
    b.AddViewField("Title");
    b.AddViewField("MyField");

    bool addCombinerNode;
    foreach (string tag in new string[] { "tag1", "tag2", "tag3" })
        b.AddWhereField("MyField", tag, "LookupMulti", 
                        "Or", out addCombinerNode);
    return b.CamlDocument.InnerXml;
}

The outcome is a rather strange looking CAML query with extra <And></And> and missing <Eq></Eq> tags. Funny thing is that this type of the query can be correctly composed through the UI, so most likely I’m not using the API correctly:

<Where>
    <And>
        <And>
            <Or>
                <FieldRef Name="MyField" /> 
                <Value Type="LookupMulti">tag1</Value> 
            </Or>
            <Or>
                <FieldRef Name="MyField" /> 
                <Value Type="LookupMulti">tag2</Value> 
            </Or>
        </And>
        <Or>
            <FieldRef Name="MyField" /> 
            <Value Type="LookupMulti">tag3</Value> 
        </Or>
    </And>
</Where>

I eventually abandoned the idea of using the CAML Query Builder API. Instead, a colleague pointed me to Waldek Mastykarz’s post on generating dynamic CAML queries. As long as all tags are to be either AND’ed or OR’ed together, Mastykarz provides an elegant solution, which I modified to be recursive, more generic, and read like an induction proof (at a negligible performance cost):

ComposeCamlQuery(new[] { "tag1", "tag2", "tag3" }, 
    "Or", 
    "<Where>{0}</Where>",
    @"<Eq>
        <FieldRef Name='MyField' />
        <Value Type='LookupMulti'>{0}</Value>
     </Eq>");

string ComposeCamlQuery(IList<string> ops, string relOp, 
                        string query, string leaf) {
    return ops.Count == 1
        ? string.Format(query, string.Format(leaf, ops[0]))
        : ComposeCamlQuery(ops.Skip(1).ToList(),
            relOp, 
            string.Format(query, 
                string.Format("<{0}>{1}{{0}}</{0}>", 
                    relOp, 
                    string.Format(leaf, ops[0]))), 
                    leaf);
}

Obviously, more complex code could be written for a mix of AND’s and OR’s. Before doing so, however, you probably want to evaluate other approaches for filtering list items:

1. Formulate a simpler, too general query and post-process the result
2. Formulate smaller queries and optionally merge the results and do post-processing
3. Use the SharePoint Search API

I also considered such alternatives as Linq to SharePoint or Caml.net, but although the first holds a lot of promise, it isn’t ready for prime time. As for the latter, its focus is more on composing type-safe queries than dynamic ones, so it’s more of a supplement to the first two alternatives.

As for SharePoint Search, the downside is that search has to be configured and that the list must have been indexed for the outcome to be accurate. Depending on the frequency with which the list is modified, the time between incremental crawls, and the context in which the results are displayed, the search approach may or may not be the solution you’re looking for.

Update, July 19: Added recursive code solution.

Researching the issue I came across LINQPad (see LINQPad in action). The tool is a generic testbed for molding expressions or statements (LINQ being one such class) before embedding them into code. Also, because LINQPad is written by one of the authors of C# in a Nutshell, the tool comes with a lot of samples from the book.

On the implementation side, LINQPad works by emitting MSIL behind the scenes, conceptually similar to the DataContext classes generated by Visual Studio when using the database design surface.

As an interesting side note, LINQPad can also show the query after it has been transformed into lambdas, anonymous types, and the like:

Users
    .Join (
        Macs,
        u => u.UserId,
        m => m.UserId,
            (u, m) =>
                 new
                 {
                     u = u,
                     m = m
                 }
    )
    .Where (temp0 => (temp0.u.Created < DateTime.Now.AddHours (-24)))
    .Select (
        temp0 =>
            new
            {
                UserId = temp0.u.UserId,
                UserCreated = temp0.u.Created,
                Mac = temp0.m.Mac,
                Active = temp0.m.Active
            }
    )
    .Take (15)

Keep in mind how the .Net 3.5 framework and the .Net 3.5 compiler play together to ultimately transform the LINQ query into MSIL. What the framework does is transform the LINQ query into the above expression by applying a transform to the abstract syntax tree of the LINQ query. The .Net 3.5 compiler then transforms the lambdas, the anonymous types, and the like into .Net 2.0 MSIL (there’s no such thing as a .Net 3.5 CLR. For an overview of how the various versions of the framework, the compiler, and the CLR relate to each other, see Scott Hanselman’s blog entry).

But back to LINQPad. The only downside that I’ve encountered is that there’s no IntelliSense support in the current 1.24 release. However, even without IntelliSense support LINQPad is still a very valuable tool.

This book obviously targets SharePoint administrators. But after finishing the book, I believe it’s a must read for SharePoint developers as well. As such, I’d argue that a developer should know what and how SharePoint is capable of, starting with the point and click approach of this book and then progressing towards code. And by capable I’m referring to all parts of SharePoint, including the Central Administration and the Shared Services Provider.

Knowing the capabilities of point and click comes in handy when discussing requirements. It often lays the groundwork for a prototype and helps you figure out where to extend SharePoint to fulfill the business needs. In addition, knowing what SharePoint can do on a general level makes estimates more reliable. Learning about administrative tasks also has the added benefit of establishing a common terminology between a developer and an administrator.

All in all, the SharePoint Administrator’s Companion does a solid job of providing an overview of the capabilities of the SharePoint platform, demonstrated mostly through point and click. I’d therefore advocate that any developer should know the content of this book by heart, and use it as a starting point before moving on to more code-centric books such as the Inside Microsoft Windows SharePoint Services 3.0 and the Inside Microsoft Office SharePoint Service 2007.

In my Kernel space traffic shaping with Linux post, I came to the conclusion that none of the traffic shaping algorithms within the Linux kernel was suitable for my needs. The effects of running the traffic shaping algorithms were too hard to quantify and coming up with the right set of parameters to go with each algorithm was challenging.

So I decided to come up with my own shaping algorithm, running in user space because it’s simpler working from there. I also wanted to use Ruby to learn the language and because of Ruby’s good properties as an integration platform. Lastly, I wanted the shaper to perform deferred shaping based on the traffic patterns observed over a period of hours or days rather than the more or less immediate shaping carried out by the kernel-based algorithms.

Before getting into the details, I should mention that these concepts have been successfully applied to shaping traffic on our 100/100 mbps Internet connection, shared by some 330 apartments, for well over a year. We’re no longer experiencing issues with network congestion and now that we use a payload agnostic shaper, we no longer have to combat the ever more sophisticated attempts of P2P software to camouflage its traffic.

Architectural overview

At an overview level the shaper is composed into a number of subsystems. At the top are the WRR scheduler, the ARP cache, and the RRD database system that retrieve and store metadata information about computers and their traffic.

Based on input from these subsystems, the shaper’s decision engine evaluates each computer’s bandwidth usage against a set of rules. Should at least one rule be violated, e.g., too much traffic over some defined period of time, the shaper calls out to another subsystem that determines how to handle the violation. In this case Netfilter is called upon to take action, blocking the computer from accessing the Internet and redirecting it to an information page.

Weighted Round Robin scheduler

Within the shaper, kernel and user space form a symbiosis through the WRR scheduler. As part of WRR’s inner workings, the scheduler counts the bytes transmitted on a per IP basis. So although we don’t use WRR for shaping, per say, we do use it to track the byte counters of each computer.

Getting WRR to reveal this information is done through the tc (for traffic control) command. As outlined in the Kernel space traffic shaping with Linux post, only outgoing traffic can be shaped by WRR (any algorithm really). Hence, tc is called once for eth0 and once for eth1, and parsing the result, we know have much traffic has entered and exited each computer between this and the previous call to tc.

For each computer the output has the form below. Of particular interest are the address and the bytes fields:

% tc class show dev eth0
class wrr 8001:1fb parent 8001:
   (address: 192.168.1.231)
   (total weight: 0.872749) (current position: 4) (counters: 1 2 : 3 4)
   Pars 1: (weight 0.872749) (decr: 1e-10) (incr: 7.5e-11) (min: 0.1) (max: 1)
   Pars 2: (weight 1) (decr: 0) (incr: 0) (min: 1) (max: 1)
   bytes: 4546184) (packets: 55373)
   ...

The address is dynamically assigned through DHCP and is therefore subject to change. Also, the byte counters aren’t retained across restarts, so we need to draw on additional subsystems to align the WRR output with a computer’s unique identity across time.

Address Resolution Protocol

In a DHCP based environment the IP address of a machine may change over time. So to ensure that traffic is always attributed to the correct physical machine, the WRR byte counters aren’t tied directly to the IP address when stored. Instead, we use the ARP cache to look up the corresponding MAC address, which is assumed to be static.

This lookup is done by maintaining an in-memory set of IP/MAC mappings, populated by parsing the output of the ip command:

% ip neigh
192.168.1.231 dev eth1 lladdr 00:50:8d:68:50:75 REACHABLE
192.168.3.117 dev eth1 lladdr 00:11:d8:8f:0e:3b REACHABLE
...

Thus, combining WRR with ARP, the shaper is able to associate byte counters with the MAC addresses of LAN-connected computers. Restarting the machine, the shaper, or Netfilter, however, will still cause traffic shaping data accumulated over time to be lost.

Round Robin Databases

We could’ve opted for persistence to a text file, a hierarchical XML structure, or a relational database, but time series data doesn’t lend itself well to these traditional approaches — at least not without preprocessing. Because with time series data, such as the 32 bit integer byte counters for each computer, counter-wrap is a frequent event (occurs every 4GB of transferred data). And restarting the machine, the shaper, or Netfilter is also a common event that’ll most likely cause an outlier to be recorded because all counters are reset.

Logic for making sure these events doesn’t pollute our database with erroneous measurements are part of the defining characteristics of a time series database system. In addition, querying data, such as summing within a period of time and making sure the sum isn’t affected by the above events, is what a time series database is good at.

On Linux-based based systems, Tobias Oetiker’s Round Robin Database tools are the de facto tools for storing, querying, and graphing time series data, and therefore the ones used by the shaper.

The idea is that, using the RRD tools, each computer gets its own database, named after the corresponding MAC address, describing its traffic over time. So querying the database of each computer can tell us how much data was transferred and received over some period of time. Using the RRD tools for this task eliminating the need on our behalf to deal with outliers, missing values, counter-wraps, and so forth. All the shaper has to do is record the value of the counters at regular intervals and RRD makes sure data is consistent within the database.

Decision engine

To put the shaper online, it’s run from a Bash script containing an infinite loop that (1) reads and parses the WRR output, (2) reads and parses the ARP cache entries, (3) writes the byte counters to the RRD databases, (4) uses the RRD querying tool to sum the data based on the rules specified, possibly causing a violation event to fire, and finally (5) goto sleep for some period of time before starting all over.

Whenever a rule is violated, the action taken may be whatever can be expressed though Ruby code or through a callout. This may involve sending an email to the network administrator or disabling Internet access for the computer in question.

Within our configuration, we defined a set of rules that state that within a four hour sliding window a computer is allowed to upload no more than 5GB and download 10GB of data. Similarly, during a seven day sliding window, a computer is allowed to upload no more than 30GB and download 60GB of data (the exact quotas and periods obviously depend on the network capacity, users online, their usage patterns, and so forth).

Upon violation of a rule, we employ Netfilter to redirect a computer to an information page stating that the computer is blocked from accessing the Internet in conjunction with the sum of the four hour and seven day totals.

Conclusion

Observing the proof of concept shaper in action, the biggest problem seems to be that a few users modify their MAC address to get a bigger piece of the bandwidth pie. If we wanted, though, changing the MAC can be counteracted by introducing another layer, and instead tie traffic to the port on the switch to which the computer is connected.

As far as the available source code goes, it should probably only be used as a starting point for building your own system.

In preparation for the exam I found the Inside Microsoft Windows SharePoint Services 3.0 book and the Windows SharePoint Services 3.0 SDK to be the best study materials. In addition, I looked to others for tips and tricks.

I found the SDK to be supprisingly thorough, compared to the book, which doesn’t cover the entire curriculum. The book alone does get one enough points to pass the exam, though.

Overall the exam questions could benefit from a review for verbosity and clear wording. Also, one question was about the RegisterHiddenField method and the RequestFromAdminPort property, which MSDN marks as obsolete. Another question was about adding a column to a content type and pushing the change to a list based on the content type. On that question I could’ve sworn that two of the possible answers were correct.

Nonetheless, learning SharePoint with a specific goal and deadline in mind serves as a good motivator.

With this many users, and even more computers, access to the Internet through our 30/30 mbit/s connection quickly turned bandwidth into a scarce resource. Not so much because users surf the web or check their email, but because P2P clients are in common use. And with this kind of software no amount of bandwidth can really satisfy its need, calling for a way to distribute bandwidth between users or computers that is fairer than the first come, first served one.

For some time we counteracted the effect of P2P clients on our bandwidth by using ipp2p on top of Netfilter. Unfortunately, popular P2P clients are able to sneak below the ipp2p radar using HTTP as their transport and obfuscating their traffic.

Thus, we turned our attention to the queuing disciplines of the Linux Advanced Routing & Traffic Control Howto. The idea behind a queuing discipline, or qdisc for short, is to apply some processing on the queue of packets in the kernel waiting to be sent (either from the network interface on the local area network side to the network interface on the Internet side or vice versa). Generally speaking, processing involves moving around packets in the sent queue to allow for some packets to go out on the wire before others, based on the algorithm of the qdisc.

The net effect is that users whose packets get moved to the front of the queue will experience a lower latency, higher bandwidth connection. Conversely, owners of packets in the back of the queue may have their packets delayed to the point where their TCP/IP stack is forced to decrease the speed with which data is sent, simply because the receiver reports that not all packets arrived on schedule.

Experimenting with the simpler qdiscs, such as Token Bucket Filter (TBF), Stochastic Fairness Queuing (SFQ), and Class Based Queuing (CBQ), we found quantifying their effects on the bandwidth consumption hard. Partly because these qdiscs aren’t intended for shaping individual computers, but rather a group of computers sharing some network usage characteristic. Sure, SFQ shapes each connection, but a computer may have any number of open connections, so shaping each connection independently is no good at limiting P2P traffic.

Therefore, we turned our attention to the Weighted Round Robin qdisc (which requires kernel patching and compilation). As opposed to the other qdiscs, WRR has the ability to shape traffic from individual computers by creating a CBQ for each one. Applying WRR to the sent queue of each network interface, WRR will assign an inbound and an outbound weight to each computer. Furthermore, the weight is adjusted as a function of the amount of data transmitted within some quantum of time. Then, when the demand for bandwidth exceeds what’s available, the computer with the highest weight gets to go first.

On paper this is a great idea, but like with the other qdiscs, we found it hard to balance the various parameters and measure the net effect.

In conclusion, a great deal of time went into experimenting with the various qdiscs, even in combination with ipp2p. But eventually we decided that none of the qdiscs were up to solving our network congestion problem. The research effort wasn’t all in vain, though, because we found that WRR provides us with a cost effective way of counting incoming and outgoing bytes on a computer by computer basis. Thus, in a way all this helped crystallize the idea of building a payload agnostic, user space traffic shaper in Ruby.

As part of my use of the e107 open source contents management system, I want to purge the user accounts that’ve not logged in recently. This is something not supported out of the box, but by going through MySQL and accessing the e107_user table directly, the date of last login is accessible to anyone.

I’ve therefore written a small command-line tool in C# that makes use of the e107_user information to sent inactive users a warning email that their accounts are about to be purged (or whatever you want to happen).

The tool may be run in one of two modes, determined by a command-line switch.

• GenerateWarningMails: emails users that have not logged in during the last WarningPeriod days.
• GeneratePurgeMailForAdministrator: emails site administrator a list of users that have not logged in during the last PurgePeriod days.

Before running E107InactiveUserPurger, make sure to go to the Debug directory and adjust the settings in E107InactiveUserPurger.exe.xml to match your environment. You might also want to adjust the WarningMailTemplate.txt, which serves as a template for generating the body of warning emails.

Note that within the WarningMailTemplate.txt file, you’re free to add placeholders for fields within the e107_user table. Thus, if you want to include the login name of a user within the warning email, you’d write $user_loginname$.

Now you might run E107InactiveUserPurger as a scheduled task. Keep in mind, though, that the tool doesn’t actually purge the user accounts. Therefore, running it too often with the GenerateWarningMails switch may result in users receiving more than one warning email.

Unfortunately, using the web interface and SharePoint Designer doesn’t scale well. With the drag and drop tools there is just no real support for deploying your portal to another server, and hence to unify the individual contributions of team members.

Therefore, the current way to go for any real SharePoint development is to work with the 12 Hive directly. Named after Office 12 (Office 2007), the 12 Hive is a hierarchy of xml files residing in “Program Files\Common Files\Microsoft Shared\web server extensions\12″ that controls almost everything about how SharePoint and your portal behaves.

Using Visual Studio (VS), you’d create a 12 Hive parallel file hierarchy of new folders and files and work within VS as usual. Like with any other project, you’d use version control, continuous integration, and so on, on xml and .Net code. With VS, deployment (to your development box) would then involve checking out xml and code from source control and compiling the code.

Then, using a build script, you’d xcopy the xml and binary into the 12 Hive and GAC, respectively, and possibly run various stsadm commands to activate your features. Alternatively, you’d create a SharePoint solution package that pretty much does the same thing when invoked by SharePoint.

In summary, Moore’s law enables game designers to use real time physics in games to accurately simulate something as complicated as a race car. Simulating an airplane has been possible for well over twenty years, but that’s only because the airplane moves smoothly through the air and is composed of simple components, such as a wing and a fuselage, which translates into few and simple equations to solve for each frame.

A race car, on the other hand, has four wheels touching the ground and connected to the car through some kind of suspension. Hence each wheel has to be accounted for independently. In addition, the clutch, gear box, and aerodynamics of the car, not to mention the terrain, have a profound impact on the physics model, calling for a more involved set of equations to solve.

As computes have become fast enough to match the frame rate given this challenging set of equations, race car games have become as realistic as flight simulators. With more horsepower, however, comes a need for even more realism. So even though computers become faster/fatter at an exponential rate, simplifying the model is, and probably always will be, worth striving for.

This is where Rigs of Rods enters the video at time index 00:56:10. The game takes the opposite approach of using advanced equations, with the risk of introducing divergence or singularities into the model when, say, the denominator of a fraction approaches zero. Instead the game uses simple/simpler equations, such as harmonic oscillators, as a mean of modeling airplanes, cars, trucks, etc. using sticks and stones (or rigs of rods, hence the name).


The sticks and stones approach works by assigning mass to the stones and connecting them using the sticks, and then adding textures on top. This more general, but also more computationally intensive, model creates a very realistic experience.

I’ve been playing the game for a couple of days now, and although it initially appears simple it’s surprisingly entertaining. A lot of time goes into exploring the scenery using the various modes of transportation.

It’s a simple service where you enter keywords for Google to search. Then Google will monitor the search results and send you an email whenever new hits appear in the search result.

Time will tell if it’s truly useful to have your own personal intelligence service at your hands. So far I receive alerts about the company I work for, the region where I live, myself, and so on.

Update, May 29: In my experience there are too many ads at the top of each email I receive. Also, Google sends me a daily email for each alert even though there are no new search results.

Thus, bulking alerts would be a nice option. Or perhaps you could write something better by coding against the Google APIs.

Of course, all that power is there to be tapped primarily by games. However, with the firmware upgrade to version 1.6, Sony added a menu item for easily joining the Folding@home distributed computing project: a project that does molecular simulation across hundreds of thousands of computers in the hope of identifying the underlying causes of Alzheimer’s, Parkinson’s, and cancer, etc.

Assuming the PS3 has network access (it comes with build-in Wi-Fi, by the way), selecting the menu item causes the console to go download the Folding@home client and start processing. That way, when you’re not using the console for gaming, you can donate the spare cycles to something useful. At least to me, participating in the Folding@home project is more meaningful than, say, aiding the search for extraterrestrial intelligence.

According to the Folding@home FAQ, the PS3 has been benchmarked at 10x the speed of a regular computer. The downside, however, is that “… the PS3 [client] takes the middle ground between GPU’s (extreme speed, but at limited types of WU’s [work units]) and CPU’s (less speed, but more flexibility in types of WU’s)”. Perhaps because the processor “… lack most of the general-purpose features that you normally expect in a processor”, as described here.

Another interesting aspect of the PS3 is the support for running other operating systems, particularly Linux. From the menu, you can repartition the standard 60 GB SATA hard disk. The operating system of the PS3 is run from internal memory for faster boot times and it’s only using the disk for storing game state and other data not vital for booting the PS3 kernel. Thus, installing Linux, you shouldn’t have to worry about accidentally wiping out the factory operating system.

Installing Linux on the thing is still something left to try. But it seems like buying a USB mouse and keyboard could transform the PS3 into a regular desktop computer, in turn making the price more digestible.

Update, September: I successfully downloaded and installed Yellow Dog Linux on a PS3. Following the installation guidelines it went surprisingly smooth. After installing, you can add the Fedora software archives to the list of possible installation sources for Yum. This’ll make even more software available without having to compile it yourself. Personally, the most used application is the VLC media player, which, at a mare double click, is able to play just about any video format in full screen mode.

The steps you have to go through are:

1. Locate vmware-vdiskmanager.exe which, on my machine, is in C:\Program Files\VMware\VMware Workstation.
2. Using the examples here together with the /? switch, the program can be invokes like:

   vmware-vdiskmanager.exe -x 15GB "path to image.vmdk"
   

   to make the new image 15 GB of size. During the enlargement process the tool creates a temporary copy the image, so you must have sufficient space on your host environment for this. Also note that this example doesn’t preallocate disk space. What this means is that although the size is logically increased to 15 GB, the image doesn’t initially grow larger on disk.

3. Now the image can contain 15 GB of data, but the guest operation system doesn’t automatically pick up the change in disk size. If you boot Windows, it’ll tell you the disk size is 10 GB, the size the disk had when Windows was installed on it. Therefore, we need to use PartitionMagic or simular tool to make Windows adjust as discribed here.

All in all the process took roughly 80 minutes on my laptop. Unfortunately, it didn’t make Visual Studio 2005 run any faster than on the W2k3 server running out of disk space. I guess the dramatic slowdown is just a necessary evil of running Visual Studio 2005 in a virtualized environment on a laptop as the development environment is not the fastest running on native hardware either.

runat.exe turned out to be quite useful when I had to do some work with MS SQL 2005 Server, Analysis Service. The thing with Analysis Service is that it’s Windows authentication only, whereas MS SQL 2005 Server supports mixed mode authentication.

My only option for getting in touch with Analysis Service was using SQL Server Management Studio by impersonating an existing Windows user, who is allowed access to Analysis Service (I didn’t have the option of creating a new account on the machine running Analysis Service and the machine wasn’t allowed to trust the domain my desktop computer was on (it’s a tough world out there)).

So, using runat.exe, I ran the command below (on one line, from a shortcut), in effect causing the display of a prompt asking for the password for my_impersonated_user. Upon entering the password sqlwb.exe starts up as if it was run from the local my_impersonated_user account.

C:\\Windows\\System32\\runas.exe /user:my_impersonated_user 
"c:\\Program Files\\Microsoft SQL Server\\90\\Tools\\
Binn\\VSShell\\Common7\\IDE\\sqlwb.exe"

For the command to succeed my_impersonated_user have to be a user on the local machine with a username and password identical to the user that is allowed access to MS SQL 2005 Server/Analysis service.

Now Windows lets you logon to Analysis Service even if your local machine is on a different domain than the MS SQL 2005 Server.

What to do about those people? Force them to read up on Adam Cogan’s rules to better email, of course.

My favorite one is probably rule #2, the one about moving email from your inbox to a special folder when you’ve replied to or otherwise handled it. This way, items in your inbox can be considered a task list.

Also, #43 about not sending email immediately because you’ll often think of something you missed is a good one. Unfortunately, delayed sending isn’t supported by most public web mail providers.

Update: July 26: Inspired by David Allen’s Getting Things Done, this Inbox Zero Google Tech Talk video details a slightly different approach for managing email. Also, the Todo.txt site, co-created by Lifehacker Gina Trapani, argues that using your inbox as a todo list is a bad habit and proposes a simpler solution.

All I need now is a project where I can actually apply what I’ve learned.

So as an exercise I’ve been thinking about constructing a DSL for describing accounting transactions (debits and credits) and describing reports (income vs. expense and account summaries).

For a couple of years, I’ve been using Gnucash, a double entry accounting system, for simple personal accounting. It works great except it’s Linux only, which is not ideal since I’ve been running Windows on my desktop lately. So every time I need to do accounting stuff, I’ve got to fire up VMware.

Instead it would be fun to literally do double entry accounting in Ruby, by expressing each transaction and report in terms of the DSL.

This way my accounting information can all be recorded using the DSL and when entering the information, you can make use of the arithmetic features and function of Ruby itself, e.g., if an entry must be the sum of several values, you can enter each value and have Ruby do the summing.

Among the good wiki engines out there (most notably MediaWiki, MoinMoin, and FlexWiki), I went with FlexWiki, because it being asp.net based means that all the software required for a basic configuration ships with Windows. In addition, FlexWiki’s provider-based storage model lets you store pages in an MS SQL server, if you want to.

My incentive for going the wiki way was that I expected my documents to become more structured when making use of the typical wiki-like features …

• Documents are automatically linked together.
  I found my pages to be fairly independent, so hyperlinks are almost never present.
• All modifications are recoded.
  Versioning isn’t really that big an issue, because I usually add contents to the end of a page rather than changing existing contents, and for pages such as a shopping list, I don’t really need versioning.
• More sophisticating layout capabilities.
  Although I occasionally created a table and used colors, generally I can do with simple formatting. For those few cases where I can’t, a Word document is in place, since it’s equally well indexed by MS or Google desktop search engines.

… but most of the time, I didn’t make use of the features FlexWiki provided over text files. So, if you also factor in the disadvantages …

• Extra infrastructure required.
  Web server, and possibly an SQL server, is a must-have-installed. Also, you probably need to figure out how to configure the web server so you don’t share your live with the world.
• Browser required for page edit.
  After launching the browser, you must navigate to the wiki page, which may add considerable overhead for quick edits as opposed to simply double clicking a text file from within Explorer.
• Lacking WYSIWYG support.
  With FlexWiki, editing a page is a two-step process. First you write content in a markup language. Then you click “Save” and the browser switches to display view, where your markup is converted to html. Now, if the result is not what you expected, you have to repeat the cycle.
• Build-in search less powerful than desktop search engines.
  If you use the file system for storing wiki pages then the pages are also searchable by desktop search engines. However, if you opt for MS SQL storage, you have to rely on the build-in search capabilities of the wiki to find pages.
• Backup is more involved.
  Backing up and restoring a database is more complicated than copying a directory.

… I’ve decided that the wiki doesn’t cut it for me: simply because there is less overhead associated with editing text or even Word documents and because with desktop search engines I retain most of the properties I need.

This doesn’t mean that a wikis aren’t suited for maintaining a corpus of personal information; just that the corpus should serve a different purpose to make better use of wiki features.

Now, how often do you crank up your playback speed?

I’ve been conducting some un-scientific experiments with increasing the playback speed with Media Player (only supports 1.5x) and my mp3-player (supports variable speed).

My conclusion is that listening to or viewing a podcast at about 1.5x the original speed is really the optimal trade-off between time consumption, not stressing the brain, and still be able to comprehend the contents.

And I save 15 minutes of every hour consuming podcasts.

Just make sure you don’t start speaking 1.5x faster because you listen to too many podcasts.

Update, August 6, 2007: Steve Pavlina describes many of the same thoughts, although he’s an advocate of a much higher speed. Steve’s blog entry mentions that “… in the latest version of Windows Media Player, click the arrow below “Now Playing.” Then click Enhancements -> Show Play Speed Settings” as a way to bring up a speed slider.

For team development a way is needed to ensure that team members doesn’t inadvertently break the shared code, e.g., a developer forgets to commit all files related to a change, which is something not easily fixable by other developers.

Also, the team might want to run post-compile tasks, such as unit tests, on a dedicated machine to avoid the works-on-my-machine syndrome.

Ideally, the project’s code should always go from one stable state to another so that a developer doesn’t get broken code the next time he synchronizes with source control.

One way to achieve this is by means of continuous integration of changes into the code. Either by rolling your own build script that monitors source control for changes to trigger a build and inform developers of the result or by leveraging an existing continuous integration solution, such as CruiseControl.net (ccnet): a continuous integration server, an asp.net based front-end, and a desktop client.

Although CruiseControl (both the Java and .Net incarnation) appears to be a popular choice among developers, I find it somewhat immature: it violates the dry principle in its projects configuration file and, worst of all, it doesn’t support build queues to ensure that only one of several projects are actually building at once.

It’s not a problem for ccnet itself to perform parallel continuous integration. However, the build chain is no stronger than the weakest link, which in my case is devenv.exe, the Visual Studio executable used by ccnet to build the application (by running devenv.exe from the command line with a couple of arguments, you can leverage the build system within Visual Studio from a script).

The downside to using devenv.exe, however, is that all running instances share a temporary build folder for compiler output. Thus, when ccnet does continuous integration of branches in parallel, the instances of devenv.exe spawned by ccnet compete for write access to identically named files, namely the executables generated by the compiler.

Not surprisingly, this results in build failures for one or more ccnet projects, although there is nothing wrong with the code.

In my experience from the trenches this happens too often to be ignored for any serious use of ccnet: It’s just too common for a developer to cross-commit a bugfix to several branches or for independent developers to commit to their working branches in between when ccnet checks the branch for updates.

Having these false negatives make the team members not trust ccnet and not take proper action whenever they see a build failure, thus rendering continuous integration useless.

One solution to the above file locking problem is to serialize the builds manually by building branches at predefined intervals, thus interleaving the builds so that they shouldn’t be able to overlap. As a result the developer may have to wait a fair amount of time to be sure his latest commit doesn’t breaks anything. Also, if a build has not finished when the next is scheduled to start, it may cause a cascade of broken builds to occur until the interleaving is properly realigned.

Another solution is to head over to Jay Flowers. He has branched off the official 1.0 version of ccnet and created a ccnet that allows for a lot more constraints than just serializing builds. It happens at the expense of running the non-official, not-recent version of ccnet, though.

My solution to the parallel build problem was to write a front-end for ccnet that implements build queues without the knowledge of ccnet, i.e., by means of checking for changes on a configurable number of branches. If a change is detected, the branch is enqueued. Then when all branches are checked, the tool force builds each branch, monitoring the state of the ccnet server to ensure that only one branch is building at a time.

With this solution you integrate the nice features of ccnet, such as the nice webgui for displaying what went wrong with a build and the systray application that goes green, yellow, or red depending on the state of your builds, with rapid feedback in a way that is compatible with Visual Studio’s integrated build engine.

The biggest downside, however, is that you can no longer see the changes that triggered a build from the web gui as ccnet is no longer in charge of cvs operations.

The tool is roughly 100 lines of IronPython code and works by spawning cvs.exe, synchronizing a number of working copies with the cvs repository, and in the process monitors the output of cvs.exe for changes, additions, and removals of files to the working copies.

The state of CruiseControl is determined by screen scraping the webgui and builds are forced by simulating a click on the “build” button for the ccnet project in the webgui for which a change was detected.

The url of the ccnet webgui, the path to the cvs.exe, and mappings between the path on disk where each working copy resides and the corresponding name of the ccnet project are stored in the tools configuration file, which is itself Python code loaded into the tool on startup.

Using Python as a language for your configuration file, you don’t have to invent your own language or use verbose xml, which requires you to write code for parsing it.

PS: I recently stumbled across the Sequential Task Plug In that I want to try out sometime.

Update, April 16: It has come to my attention that there is a way to minimize duplication in the configuration file. It involves “… users to set up DTD entities within the CCNet configuration” as mentionen here. I haven’t tried it, though.

Update, August 10: CruiseControl.net 1.3 now comes with a build-in queue feature.

At some point in the past HEAD branched into B1, which subsequently branched into B1.1. Then development on B1 completed and it was merged back into HEAD so a release branch B2 could be created. B1.1, however, kept living its separate life for a while, because the features added aren’t to be released until with release branch B5. This also goes for the changes made to B3, so it makes sense for B1.1 to be merged into B3.

As a result, I just spend a few hours on tracking down a merge issue resulting from this merge, because a change made in B1 wasn’t present in B3 after B1.1 was merged into this branch. I finally figured out why: suppose a line is removed in B1 before the spin-off of B1.1. After that the very same line is added back in. Now when B1.1 is merged into B3, the line is removed again. There is no merge conflict at this point, because from CVS’ point of view it’s a clean merge.

It seems obvious what happened when looking at the diagram, but when you’re working with a large repository with lots of modified files and you’re merging code not written by yourself, this sort of indirect change propagation may inevitably bite you as well.

• PageDefrag
  

  PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your page file and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files [...].

  Has the ability to run at the next or every boot (that’s how it gets exclusive access to the system files). Running it every time I boot typically adds 10-20 seconds to my boot time, but I guess it depends on how often you boot, the amount of physical memory, and what kind of applications you run.

• Contig
  

  Contig is a single-file defragmenter that attempts to make files contiguous on disk. It’s perfect for quickly optimizing files that are continuously becoming fragmented, or that you want to ensure are in as few fragments as possible.

  Also has a recursive switch, so you can specify C:\ as the starting point and it’ll defragment its way through your entire drive.

• ShareEnum
  

  When you run ShareEnum it uses NetBIOS enumeration to scan all the computers within the domains accessible to it, showing file and print shares and their security settings.

  A worthy replacement for Network Neighborhood, although it sometimes is not able to find any shares, although I know there’s quite a few out there.

• Autoruns
  

  This utility, which has the most comprehensive knowledge of auto-starting locations of any startup monitor, shows you what programs are configured to run during system boot or login, and shows you the entries in the order Windows processes them. These programs include ones in your startup folder, Run, RunOnce, and other Registry keys.

  On my computer, it takes longer for Windows to become responsive after log on than it takes Windows to boot and display the log on screen, because Windows loads a ton of software, fills up the systray, and starts various services I really don’t need.

  Services such as Remote Registry for accessing the registry across the network, Windows Image Acquisition for popping up a dialog box when I scan an image on the scanner I don’t have, or Windows Zero Configuration for managing the WLAN access I never use.

  Those services are all on by default and can be disabled using AutoRuns (or using Control panel => Administrative tools => Services).

• Process Explorer
  

  The Process Explorer display consists of two sub-windows. The top window always shows a list of the currently active processes, including the names of their owning accounts, whereas the information displayed in the bottom window depends on the mode that Process Explorer is in: if it is in handle mode you’ll see the handles that the process selected in the top window has opened; if Process Explorer is in DLL mode you’ll see the DLLs and memory-mapped files that the process has loaded. Process Explorer also has a powerful search capability that will quickly show you which processes have particular handles opened or DLLs loaded.

  The quote speaks for itself.

There’s a lot more tools where these came from. Among the ones I use less frequently are FileMon and Regmon for real-time monitoring of file system and registry access, respectively. Also, I use TCPView as a more user friendly replacement for the netstat command.

Finally, take a look at dnrTV, where Scott Hanselman recently showed off some of these tools.

Typically, you’d target your frustration at the debugger and insert a breakpoint around the problematic part of your code to extract the SQL and copy it into the Query Analyzer for investigation.

Unfortunately, pasting the SQL into Query Analyzer causes the statement to appear on one line entirely. You then find yourself inserting new line characters for readability and for making the error message more precise.

Transforming SQL this way is tedious, boring, and crying for automation. Fortunately, Enterprise Manager responds well to these cries. Just drill down to a table view of your database, right click on a table, and select “Open table” followed by “Query”.

Now paste your extracted SQL into the text portion of the Query by Example window and hit “Run”. Although the query might not actually execute, it’s transformed into a more readable one.

Obviously, you could start and stay in the QBE window. Personally, though, I dislike QBE as a working environtment beyond for layout beautification, so I usually copy the transformed query into Query Analyzer.

By the way, the transform also comes in handy if you want to make sure your query doesn’t contain any camouflaged expensive joins, e.g., for a query with inner joins done using where clauses, QBE transforms it into explicit inner joins.

It stalls at “Windows is loading files …”. Apparently the subsequent switch into graphics mode causes Vista to hang, because Vista all the sudden doesn’t properly support the emulated video hardware of VMWare.

That’s odd, because I previously installed Beta 2 on VMWare without any sign of trouble (could it be that MS wants people to try out RC1 on their free Virtual PC? By the way, VMWare Server is free too).

Luckily, Joel on Software provides a solution to the VMWare issue, by adding

svga.maxWidth = "640" 
svga.maxHeight = "480"

to the configuration file for your virtual machine prior to booting the Vista DVD. It’ll force the installer to run in SVGA mode. Then when the installation is complete and you’ve installed the VMWare tools, remove the above lines and Vista shines.

My intention was to run Vista in VMWare on a secondary computer and connect to it via Remote Desktop. For some yet to be uncovered reason, however, I wasn’t able to connect from my WinXP machine. Perhaps the host OS blocked the incoming request or perhaps Vista just wasn’t properly configured.

Anyway, I decided to run Vista on the bare hardware instead, so I fired up PartitionMagic. Unfortunately some guy accidentally cut power while PartitionMagic was doing its shrink-existing-partition-to-create-space-for-an-additional-one magic. After that the existing OS chickened out and bluescreened on me, so out everything went and in Vista came.

Thus, now I utilize a two-monitor setup, running Vista on a native 2.6GHz processor with 2GB of memory on one monitor and WinXP on the other. Unfortunately, the Vista machine has a crappy video card, so no fancy, transparent UI for me (unless I find a workaround).

As a final note, I’m using Synergy to seamlessly share one mouse and keyboard between the physical machines as if it was one machine with dual head video output.

Way cool!

This book is essential in preparing for the 70-536 certification exam. It’s the first of two exams one has to pass to become a MCTS (Microsoft Certified Technology Specialist), specializing in either web or Windows application development.

I’d rather have used Amit Kalanis book, but its shipping date just continues to get pushed.

• Open Notepad
• Type in “this app can break” (without the quotes)
• Save file
• Close Notepad
• Re-open saved file in Notepad

Now Notepad displays funny looking foreign characters.

This is caused by a “bug” in the Win32 function IsTextUnicode, which Notepad calls to heuristically determine if a file is Unicode encoded.

For more in-depth info click here.

The following code benchmarks each method, running it 500 times in a row. Furthermore, to ensure that the results are independent of the length of the list, I test each function on lists of sizes 1000 to 30000, in 100 increments, looking for a clear growth patterns (but found none).

from time import time
def scale_using_recursion(s, l):
    if len(l) == 0:
        return []
    else:
        return [(l[0] * s)] + scale_using_recursion(s, l[1:])

def scale_using_loop(s, l):
    r = []
    for e in l:
        r.append(e * s)
    return r

def scale_using_map(s, l):
    return map(lambda e: e * s, l)

def scale_using_list_comprehension(s, l):
    return [e * s for e in l]

def run(func, iterations, list_size):
    l = [e for e in range(list_size)]
    t = time()
    [func(10, l) for i in range(iterations)]
    return (time() - t) / iterations

def time_stuff():
    iterations = 500

    for list_size in range(1000, 30000, 100):
        t1 = run(scale_using_list_comprehension, iterations, list_size)
        t2 = run(scale_using_map, iterations, list_size)
        t3 = run(scale_using_loop, iterations, list_size)
        print("%s %s %s %s") % (list_size, t1/t1, t2/t1, t3/t1)

if __name__ == "__main__":
    time_stuff()

The benchmark was performed under MS Windows, running Python 2.4.2. To make the results independent of the speed of the hardware, I compare each scaling method to the winner of the race, the list comprehension approach.

scale_using_list_comprehension:  1.0
scale_using_loop:                1.7
scale_using_map:                 2.1
(scale_using_recursion:         76.0)

Not surprisingly, the recursive approach is slow and infeasible for real use (and only tested on a list size of 500 to get an indication of relative performance). Generally, recursive functions are of limited use in Python because the interpreter doesn’t optimize for tail recursion; a technique commonly used in functional languages that causes a tail recursion to get transformed into an iteration, making a recursive and an interative approach equally fast.

Lacking this features, Python blows up with a “RuntimeError: maximum recursion depth exceeded”, when it reaches the default recursion depth of 500. We could increase the number for certain situations, of course, but we would never truly be in safe range of the RuntimeError, and performing the way it does, it doesn’t really matter.

I was surprised to learn that map and for loops are roughly 70% and 110% slower, respectively, than list comprehension. I can’t really put my finger on what makes them so slow.

Although this isn’t the most scientific of studies, it does indicate that list comprehension is the way to go, if possible. Luckily, in most cases, list comprehension is also the shortest, the most elegant, and the most readable of the approaches.

I’ll use a real-world build tool I’ve written as the driving example. Typical tasks for such a tool would include, but not be limited to, checking out sources from cvs, compiling the sources, and running NUnit against a number of generated targets.

Each such task may be defined as a function, taking any number of arguments and returning a list of the form:

[status, [(command, output), (...)]]

where status indicates binary success, command is the command run to complete the task and output is the output (stdout/stderr) of running the command. There may be more than one (command, output) tuple inside the sublist, if the task requires more than one command to complete.

In FP the list and tuple are fundamental and very powerful concepts, because they can easily be used to define complex structures for you to use when passing data around. They also facilitate easy processing using list comprehension, the functions build into the language, or functions of your own, providing elegant syntax for common operations, such as insertion, deletion, searching, slicing, etc.

These lists and tuples are more dynamic in nature than their array and struct counterparts in, say, C#. In fact, rather than creating user defined types, a list, or a tuple, or a combination often suffices, eliminating the need to write a lot of boilerplate code.

Now, to provide an example where Python’s FP support really kicks in with the build tool, is in the implementation of the code responsible for running NUnit against a number of targets:

def NUnit(nunitconsole, projectsOrAssemblies):
    results = [Run("\"" + nunitconsole + "\" " + source) 
              for source in projectsOrAssemblies]
    commands = [command[1][0] for command in results]
    outputs = [output[1][1] for output in results]
    failures = Sigma(re.findall("Failures: (\d+)", str(outputs)))
    return [not bool(int(failures)), zip(commands, outputs)]

This function relies heavily on the use of lists and list comprehension, rather than imperative style loops (which, of course, is also possible in Python). List comprehension is a construct that allows you to easily create a new list as a result of some processing on the elements of another.

A highlevel walk-through of the function: first the NUnit executable is invoked one target at a time. The Run function itself returns a list of the form [exitcode, (command, output)]; hence with n targets to test, the commands and outputs variables end up containing n commands and their output. The Failures variable counts the total number of failures accumulated across all targets, extracted as a list of strings matching the regular expression and summarized by the Sigma function.

Python doesn’t allow summing over a list of strings using its build-in Sum function. However, you can easily implement a Sum function that does in something along the lines of:

def Sigma(list): 
    return reduce(lambda a, b: int(a) + int(b), list)

where reduce is a function present in most, if not all, FP languages, which reduces a list of values to a single value by applying a user-defined operation to pairwise values of the list.

Finally, the zip function is a build-in function, merging the elements of two lists into one.

Another important construct in FP is the lambda notation. It enables you to create an anonymous function, a function without an explicitly name, leading us to another key characteristic of FP: functions are treated as first-class citizens, in the sense that they may be passed around as just another variable, e.g., you can create a list of functions with parameters and then do the actual invocation at a later stage.

Using deferred invocation, the build tool manages the overall build process; in a config file, similar to a makefile, a list of lambdas are defined, each defining a function that executes a task. This provides us with a generic way of annotating a task and for each task to be run by a central runner, which takes care of logging description, command, and outcome to a html report. The prefs used as parameters to each task is an associated array of associated arrays of … well as many levels as you require.

In effect the config file defines a domain specific language, using regular Python syntax, which can easily be loaded into what corresponds to the make utility by an import statement:

prefs = {
    "cvs" :
    {
        "cvs" : "C:\\Program Files\\cvsnt\\cvs.exe", 
        "workingDirectory" : "C:\\Inetpub\\wwwroot",
        "root" : ":pserver:foo:bar@1.2.3.4:/foobar",
        "module" : "baz",
        "revision" : "HEAD"    
    },
    "nunit" :  
    {
        "nunitconsole" : "C:\\...\\nunit-console.exe",
        "projectsOrAssemblies" : [C:\\Inetpub\\...\\foobar.nunit" ]
    },
    ...
}

tasks = [ ("Checkout Foobar from cvs",
    lambda: Task.CvsCheckout(prefs["cvs"]["cvs"],
                             prefs["cvs"]["workingDirectory"],
                             prefs["cvs"]["root"],
                             prefs["cvs"]["module"],
                             prefs["cvs"]["revision"])),
    ...,
    ("Running NUnit",
    lambda: Task.NUnit(prefs["nunit"]["nunitconsole"],
                       prefs["nunit"]["projectsOrAssemblies"]))
]

This example, using the basic concepts of FP such as lists, list comprehension, tuples, build-in functions, and nice syntax, hopefully provided you with a glimpse of how a lot can be achieved in relatively few lines of Python code. Also, if you really want to explore FP from a Python perspective, I suggest you take a look at the Xoltar Toolkit (unfortunately, though, it isn’t under active development), described in David Mertz’s columns More Functional Programming in Python and Even More Functional Programming in Python

Bottom line is that the use of FP style in your programs, although it takes time getting use to, leads to fewer lines of code, which leads to fewer places where things can go wrong.

This is not to say that your entire program should be written using the FP style (as is the case with Haskell). However, learning the basic concepts, and how Python exposes these to the developer, you’ll know which patterns to look for in your code that may be better solved using FP constructs.