Why traditional dependency-breaking techniques come short

After watching a screencast on how to use Moles to unit test LINQ to SQL without hitting the database, I thought it would be interesting to do the same with TypeMock. But first, let’s make sure we understand why traditional dependency-breaking techniques come short in testing LINQ to SQL. Assuming we want to put a repository under test, our goal is to mock how it accesses the database. Here’s a simple implementation of a repository that queries the Employee table of the AdventureWorks database:

    public class EmployeeRepository {
        public List<Employee> GetEmployeesByHireDate(DateTime start, DateTime end) {
            using (var ctx = new AdventureWorksDataContext())
                return (from e in ctx.Employees
                        where e.HireDate >= start && e.HireDate <= end
                        select e).ToList();
        }
    }

All calls to the database are routed through the AdventureWorksDataContext generated by Visual Studio. To mock access to the database, we therefore have to mock part of the data context. Easier said than done, though, for the context doesn’t expose an interface that a fake can implement. In addition, the tables are accessed through properties on the context that return a type of Table<TEntity>. Unfortunately, the constructor of Table<TEntity> is internal and the class itself is sealed, eliminating the hope of instantiating or subclassing the type by traditional means:

    public sealed class Table<TEntity> : IQueryProvider,
            ITable, IListSource, ITable<TEntity>, IQueryable<TEntity>,
            IEnumerable<TEntity>, IQueryable, IEnumerable
            where TEntity : class {
        internal Table(DataContext context, MetaTable metaTable) {
            ...
        }
    }

For an example of how the data context itself creates an instance of Table<TEntity>, take a look at the Employees property on the AdventureWorksDataContext. It relies on the GetTable<Employee> method on the DataContext class to create an instance of Table<Employee>. Despite its constructors being internal, the GetTable<TEntity> method has no trouble constructing an instance of the Table<TEntity> type, as they both reside in the System.Data.Linq assembly:

    public partial class AdventureWorksDataContext : DataContext {
        public Table<Employee> Employees {
            get {
                return GetTable<Employee>();
            }
        }
    }

How to break the unbreakable

The design of LINQ to SQL leaves us short of a traditional testing seam, as Michael Feathers would phrase it; a place at which we can alter the behavior of a program without editing in that place. This explains why, with LINQ to SQL, traditionally we’ve had to test against a real database with all its constraints, making our tests brittle, slow, and painful to write and maintain. With the new breed of mocking frameworks the issues of not being able to subclass or not being able to call an internal constructor go away (and new issues take their place). Regardless, here’s how to write a unit test for the CustomerRepository that doesn’t hit the database:

    [TestClass]
    public class CustomerRepositoryTest {
        private EmployeeRepository _repository;

        [TestInitialize]
        public void Initialize() {
            _repository = new EmployeeRepository();

            var fakeEmployees = new List<Employee> {
                new Employee {EmployeeID = 1, HireDate = new DateTime(2004, 12, 1)},
                new Employee {EmployeeID = 2, HireDate = new DateTime(2006, 7, 1)},
                new Employee {EmployeeID = 3, HireDate = new DateTime(2009, 3, 1)}
            }.AsQueryable();

            var fakeDataContext = Isolate.Fake.Instance<AdventureWorksDataContext>();
            Isolate.Swap.NextInstance<AdventureWorksDataContext>().With(fakeDataContext);

            // var fakeEmployeeTable = Isolate.Fake.Instance<Table<Employee>>();
            // Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturn(fakeEmployeeTable);
            // Isolate.WhenCalled(() => fakeEmployeeTable).WillReturnCollectionValuesOf(fakeEmployees);
            // or by transitivity
            Isolate.WhenCalled(() => fakeDataContext.Employees).WillReturnCollectionValuesOf(fakeEmployees);
        }

        [TestMethod]
        public void GetEmployeesByHireDate_should_return_hires_from_2008_until_present() {
            var employees = _repository.GetEmployeesByHireDate(new DateTime(2008, 1, 1), DateTime.Now);
            Assert.AreEqual(1, employees.Count());
            Assert.AreEqual(3, employees[0].EmployeeID);
        }
    }

The test method itself looks exactly as if we’d been testing against a real database. The difference lies in the Initialize method, where we setup the fake data context and database contents. We instruct TypeMock to return the fake context in place of the real one inside EmployeeRepository. And whenever someone calls the Employees property on the fake context, we have TypeMock intercept the call and return a fake collection of type IQueryable<Employee>. We could’ve returned an instance of Table<Employee>, which implements IQueryable<Employee>, but in this case returning the collection is simpler and sufficient. Had we had more methods on our repository, we likely would’ve added additional rows to the Employee table and populated more of its columns.

In a nutshell Org-mode is a plug-in for Emacs that makes taking notes, maintaining lists, clocking time, and many other tasks more straightforward. The mode works by adding features on top of Emacs for editing plain text files. Features that enrich the interaction with the underlying text buffer by parsing and adding (meta-)information to it.

With Org-mode you group items into a hierarchical structure. A path to the item to work on is then formed by unfolding items in a way analogous to how you navigate the file system with Explorer.

To efficiently work with items, Org-mode supports moving items around the hierarchy. Items may also be tagged (buy, work, phone, etc.) or associated with a state (done, waiting, started, etc.). Additional meta-information may include a deadline, an estimate, or time spent working on a task. The clocking of time is accomplished by Org-mode adding a timestamp to the item. On completion, another timestamp and the elapsed time is calculated and added.

Below is an example of an expanded and a collapsed item containing clocked times and estimates.

With a few keystrokes, Org-mode is able to correlate estimates with time spent on an item-by-item basis. This truly short-circuits the estimation feedback cycle. Imagine doing this on a file containing a large hierarchy of items and the transparency it brings.

If you want to keep track of billable hours, it may make sense to clock all your time using Org-mode and generate timesheets from the clocking information. Especially since the billing context is automatically provided by the item storing the timestamp, and since you’re free to organize the file in any way you see fit. Few other tools manage to compete with the flexibility of a plain text buffer.

This post only scratches the surface of Org-mode. To get a sense of the possibilities, the manual amounts to 184 pages. However, for a quick introduction I encourage you to watch the Google Tech Talk on Org-mode.

Like with any habit, every so often you should take the time to review it. I find TimeSnapper (the free edition) to be an invaluable tool for reviewing computer habits. It captures an image of the desktop every few seconds and is able to create a movie from the images. That way, a partial life log (mine is currently 393 days) is automatically created to help you remember what you did yesterday and the day before.

There’s nothing like reviewing a few of weeks of captured images to make habits stand out. I found that I’ve a tendency to re-visit the same maybe ten sites all too often. I also have difficulty single-tasking. Listening to a podcast or watching a movie, I’m often browsing the web instead of paying attention to the podcast or skipping it altogether. I may even have several media players launched at once, although only one playing.

Most of my distractions can be attributed to me always being online. I need Internet access for work, so turning it off altogether isn’t an alternative. So, not unlike Paul Graham, on several occasions, I’ve made myself adhere to a set of rules for using the Internet. And every time they’ve worked — for a short while.

In Disconnecting Distraction, Paul takes these rules one step further and argues that using two computers may be a better solution: one computer with Internet access and one without for doing real work. Ideally, the computers should be placed some distance apart to make you psychically get up to go on the Internet.

For a week I’ve given the two-computer approach a try. It was hard initially, but I’ve come to appreciate it. Forming permanent habits takes time. Thus, I’ll continue the experiment for a couple of weeks. Perhaps in time I’ll even learn to turn off instant messaging and only do mail once or twice a day. Or I may end up spending most of the time in front of the connected computer.

Then again, there’s more to life than work.

Researching the issue I came across LINQPad (see LINQPad in action). The tool is a generic testbed for molding expressions or statements (LINQ being one such class) before embedding them into code. Also, because LINQPad is written by one of the authors of C# in a Nutshell, the tool comes with a lot of samples from the book.

On the implementation side, LINQPad works by emitting MSIL behind the scenes, conceptually similar to the DataContext classes generated by Visual Studio when using the database design surface.

As an interesting side note, LINQPad can also show the query after it has been transformed into lambdas, anonymous types, and the like:

   Users
      .Join (
         Macs,
         u => u.UserId,
         m => m.UserId,
         (u, m) =>
            new
            {
               u = u,
               m = m
            }
      )
      .Where (temp0 => (temp0.u.Created < DateTime.Now.AddHours (-24)))
      .Select (
         temp0 =>
            new
            {
               UserId = temp0.u.UserId,
               UserCreated = temp0.u.Created,
               Mac = temp0.m.Mac,
               Active = temp0.m.Active
            }
      )
      .Take (15)

Keep in mind how the .Net 3.5 framework and the .Net 3.5 compiler play together to ultimately transform the LINQ query into MSIL. What the framework does is transform the LINQ query into the above expression by applying a transform to the abstract syntax tree of the LINQ query. The .Net 3.5 compiler then transforms the lambdas, the anonymous types, and the like into .Net 2.0 MSIL (there’s no such thing as a .Net 3.5 CLR. For an overview of how the various versions of the framework, the compiler, and the CLR relate to each other, see Scott Hanselman’s blog entry).

But back to LINQPad. The only downside that I’ve encountered is that there’s no IntelliSense support in the current 1.24 release. However, even without IntelliSense support LINQPad is still a very valuable tool.

As part of my use of the e107 open source contents management system, I want to purge the user accounts that’ve not logged in recently. This is something not supported out of the box, but by going through MySQL and accessing the e107_user table directly, the date of last login is accessible to anyone.

I’ve therefore written a small command-line tool in C# that makes use of the e107_user information to sent inactive users a warning email that their accounts are about to be purged (or whatever you want to happen).

The tool may be run in one of two modes, determined by a command-line switch.

• GenerateWarningMails: emails users that have not logged in during the last WarningPeriod days.
• GeneratePurgeMailForAdministrator: emails site administrator a list of users that have not logged in during the last PurgePeriod days.

Before running E107InactiveUserPurger, make sure to go to the Debug directory and adjust the settings in E107InactiveUserPurger.exe.xml to match your environment. You might also want to adjust the WarningMailTemplate.txt, which serves as a template for generating the body of warning emails.

Note that within the WarningMailTemplate.txt file, you’re free to add placeholders for fields within the e107_user table. Thus, if you want to include the login name of a user within the warning email, you’d write $user_loginname$.

Now you might run E107InactiveUserPurger as a scheduled task. Keep in mind, though, that the tool doesn’t actually purge the user accounts. Therefore, running it too often with the GenerateWarningMails switch may result in users receiving more than one warning email.

It’s a simple service where you enter keywords for Google to search. Then Google will monitor the search results and send you an email whenever new hits appear in the search result.

Time will tell if it’s truly useful to have your own personal intelligence service at your hands. So far I receive alerts about the company I work for, the region where I live, myself, and so on.

Update, May 29: In my experience there are too many ads at the top of each email I receive. Also, Google sends me a daily email for each alert even though there are no new search results.

Thus, bulking alerts would be a nice option. Or perhaps you could write something better by coding against the Google APIs.

runat.exe turned out to be quite useful when I had to do some work with MS SQL 2005 Server, Analysis Service. The thing with Analysis Service is that it’s Windows authentication only, whereas MS SQL 2005 Server supports mixed mode authentication.

My only option for getting in touch with Analysis Service was using SQL Server Management Studio by impersonating an existing Windows user, who is allowed access to Analysis Service (I didn’t have the option of creating a new account on the machine running Analysis Service and the machine wasn’t allowed to trust the domain my desktop computer was on (it’s a tough world out there)).

So, using runat.exe, I ran the command below (on one line, from a shortcut), in effect causing the display of a prompt asking for the password for my_impersonated_user. Upon entering the password sqlwb.exe starts up as if it was run from the local my_impersonated_user account.

C:\\Windows\\System32\\runas.exe /user:my_impersonated_user
"c:\\Program Files\\Microsoft SQL Server\\90\\Tools\\
Binn\\VSShell\\Common7\\IDE\\sqlwb.exe"

For the command to succeed my_impersonated_user have to be a user on the local machine with a username and password identical to the user that is allowed access to MS SQL 2005 Server/Analysis service.

Now Windows lets you logon to Analysis Service even if your local machine is on a different domain than the MS SQL 2005 Server.

Among the good wiki engines out there (most notably MediaWiki, MoinMoin, and FlexWiki), I went with FlexWiki, because it being asp.net based means that all the software required for a basic configuration ships with Windows. In addition, FlexWiki’s provider-based storage model lets you store pages in an MS SQL server, if you want to.

My incentive for going the wiki way was that I expected my documents to become more structured when making use of the typical wiki-like features …

• Documents are automatically linked together.
  I found my pages to be fairly independent, so hyperlinks are almost never present.
• All modifications are recoded.
  Versioning isn’t really that big an issue, because I usually add contents to the end of a page rather than changing existing contents, and for pages such as a shopping list, I don’t really need versioning.
• More sophisticating layout capabilities.
  Although I occasionally created a table and used colors, generally I can do with simple formatting. For those few cases where I can’t, a Word document is in place, since it’s equally well indexed by MS or Google desktop search engines.

… but most of the time, I didn’t make use of the features FlexWiki provided over text files. So, if you also factor in the disadvantages …

• Extra infrastructure required.
  Web server, and possibly an SQL server, is a must-have-installed. Also, you probably need to figure out how to configure the web server so you don’t share your live with the world.
• Browser required for page edit.
  After launching the browser, you must navigate to the wiki page, which may add considerable overhead for quick edits as opposed to simply double clicking a text file from within Explorer.
• Lacking WYSIWYG support.
  With FlexWiki, editing a page is a two-step process. First you write content in a markup language. Then you click “Save” and the browser switches to display view, where your markup is converted to html. Now, if the result is not what you expected, you have to repeat the cycle.
• Build-in search less powerful than desktop search engines.
  If you use the file system for storing wiki pages then the pages are also searchable by desktop search engines. However, if you opt for MS SQL storage, you have to rely on the build-in search capabilities of the wiki to find pages.
• Backup is more involved.
  Backing up and restoring a database is more complicated than copying a directory.

… I’ve decided that the wiki doesn’t cut it for me: simply because there is less overhead associated with editing text or even Word documents and because with desktop search engines I retain most of the properties I need.

This doesn’t mean that a wikis aren’t suited for maintaining a corpus of personal information; just that the corpus should serve a different purpose to make better use of wiki features.

For team development a way is needed to ensure that team members doesn’t inadvertently break the shared code, e.g., a developer forgets to commit all files related to a change, which is something not easily fixable by other developers.

Also, the team might want to run post-compile tasks, such as unit tests, on a dedicated machine to avoid the works-on-my-machine syndrome.

Ideally, the project’s code should always go from one stable state to another so that a developer doesn’t get broken code the next time he synchronizes with source control.

One way to achieve this is by means of continuous integration of changes into the code. Either by rolling your own build script that monitors source control for changes to trigger a build and inform developers of the result or by leveraging an existing continuous integration solution, such as CruiseControl.net (ccnet): a continuous integration server, an asp.net based front-end, and a desktop client.

Although CruiseControl (both the Java and .Net incarnation) appears to be a popular choice among developers, I find it somewhat immature: it violates the dry principle in its projects configuration file and, worst of all, it doesn’t support build queues to ensure that only one of several projects are actually building at once.

It’s not a problem for ccnet itself to perform parallel continuous integration. However, the build chain is no stronger than the weakest link, which in my case is devenv.exe, the Visual Studio executable used by ccnet to build the application (by running devenv.exe from the command line with a couple of arguments, you can leverage the build system within Visual Studio from a script).

The downside to using devenv.exe, however, is that all running instances share a temporary build folder for compiler output. Thus, when ccnet does continuous integration of branches in parallel, the instances of devenv.exe spawned by ccnet compete for write access to identically named files, namely the executables generated by the compiler.

Not surprisingly, this results in build failures for one or more ccnet projects, although there is nothing wrong with the code.

In my experience from the trenches this happens too often to be ignored for any serious use of ccnet: It’s just too common for a developer to cross-commit a bugfix to several branches or for independent developers to commit to their working branches in between when ccnet checks the branch for updates.

Having these false negatives make the team members not trust ccnet and not take proper action whenever they see a build failure, thus rendering continuous integration useless.

One solution to the above file locking problem is to serialize the builds manually by building branches at predefined intervals, thus interleaving the builds so that they shouldn’t be able to overlap. As a result the developer may have to wait a fair amount of time to be sure his latest commit doesn’t breaks anything. Also, if a build has not finished when the next is scheduled to start, it may cause a cascade of broken builds to occur until the interleaving is properly realigned.

Another solution is to head over to Jay Flowers. He has branched off the official 1.0 version of ccnet and created a ccnet that allows for a lot more constraints than just serializing builds. It happens at the expense of running the non-official, not-recent version of ccnet, though.

My solution to the parallel build problem was to write a front-end for ccnet that implements build queues without the knowledge of ccnet, i.e., by means of checking for changes on a configurable number of branches. If a change is detected, the branch is enqueued. Then when all branches are checked, the tool force builds each branch, monitoring the state of the ccnet server to ensure that only one branch is building at a time.

With this solution you integrate the nice features of ccnet, such as the nice webgui for displaying what went wrong with a build and the systray application that goes green, yellow, or red depending on the state of your builds, with rapid feedback in a way that is compatible with Visual Studio’s integrated build engine.

The biggest downside, however, is that you can no longer see the changes that triggered a build from the web gui as ccnet is no longer in charge of cvs operations.

The tool is roughly 100 lines of IronPython code and works by spawning cvs.exe, synchronizing a number of working copies with the cvs repository, and in the process monitors the output of cvs.exe for changes, additions, and removals of files to the working copies.

The state of CruiseControl is determined by screen scraping the webgui and builds are forced by simulating a click on the “build” button for the ccnet project in the webgui for which a change was detected.

The url of the ccnet webgui, the path to the cvs.exe, and mappings between the path on disk where each working copy resides and the corresponding name of the ccnet project are stored in the tools configuration file, which is itself Python code loaded into the tool on startup.

Using Python as a language for your configuration file, you don’t have to invent your own language or use verbose xml, which requires you to write code for parsing it.

PS: I recently stumbled across the Sequential Task Plug In that I want to try out sometime.

Update, April 16: It has come to my attention that there is a way to minimize duplication in the configuration file. It involves “… users to set up DTD entities within the CCNet configuration” as mentionen here. I haven’t tried it, though.

Update, August 10: CruiseControl.net 1.3 now comes with a build-in queue feature.

At some point in the past HEAD branched into B1, which subsequently branched into B1.1. Then development on B1 completed and it was merged back into HEAD so a release branch B2 could be created. B1.1, however, kept living its separate life for a while, because the features added aren’t to be released until with release branch B5. This also goes for the changes made to B3, so it makes sense for B1.1 to be merged into B3.

As a result, I just spend a few hours on tracking down a merge issue resulting from this merge, because a change made in B1 wasn’t present in B3 after B1.1 was merged into this branch. I finally figured out why: suppose a line is removed in B1 before the spin-off of B1.1. After that the very same line is added back in. Now when B1.1 is merged into B3, the line is removed again. There is no merge conflict at this point, because from CVS’ point of view it’s a clean merge.

It seems obvious what happened when looking at the diagram, but when you’re working with a large repository with lots of modified files and you’re merging code not written by yourself, this sort of indirect change propagation may inevitably bite you as well.

• PageDefrag
  

  PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your page file and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files [...].

  Has the ability to run at the next or every boot (that’s how it gets exclusive access to the system files). Running it every time I boot typically adds 10-20 seconds to my boot time, but I guess it depends on how often you boot, the amount of physical memory, and what kind of applications you run.

• Contig
  

  Contig is a single-file defragmenter that attempts to make files contiguous on disk. It’s perfect for quickly optimizing files that are continuously becoming fragmented, or that you want to ensure are in as few fragments as possible.

  Also has a recursive switch, so you can specify C:\ as the starting point and it’ll defragment its way through your entire drive.

• ShareEnum
  

  When you run ShareEnum it uses NetBIOS enumeration to scan all the computers within the domains accessible to it, showing file and print shares and their security settings.

  A worthy replacement for Network Neighborhood, although it sometimes is not able to find any shares, although I know there’s quite a few out there.

• Autoruns
  

  This utility, which has the most comprehensive knowledge of auto-starting locations of any startup monitor, shows you what programs are configured to run during system boot or login, and shows you the entries in the order Windows processes them. These programs include ones in your startup folder, Run, RunOnce, and other Registry keys.

  On my computer, it takes longer for Windows to become responsive after log on than it takes Windows to boot and display the log on screen, because Windows loads a ton of software, fills up the systray, and starts various services I really don’t need.

  Services such as Remote Registry for accessing the registry across the network, Windows Image Acquisition for popping up a dialog box when I scan an image on the scanner I don’t have, or Windows Zero Configuration for managing the WLAN access I never use.

  Those services are all on by default and can be disabled using AutoRuns (or using Control panel => Administrative tools => Services).

• Process Explorer
  

  The Process Explorer display consists of two sub-windows. The top window always shows a list of the currently active processes, including the names of their owning accounts, whereas the information displayed in the bottom window depends on the mode that Process Explorer is in: if it is in handle mode you’ll see the handles that the process selected in the top window has opened; if Process Explorer is in DLL mode you’ll see the DLLs and memory-mapped files that the process has loaded. Process Explorer also has a powerful search capability that will quickly show you which processes have particular handles opened or DLLs loaded.

  The quote speaks for itself.

There’s a lot more tools where these came from. Among the ones I use less frequently are FileMon and Regmon for real-time monitoring of file system and registry access, respectively. Also, I use TCPView as a more user friendly replacement for the netstat command.

Finally, take a look at dnrTV, where Scott Hanselman recently showed off some of these tools.

Typically, you’d target your frustration at the debugger and insert a breakpoint around the problematic part of your code to extract the SQL and copy it into the Query Analyzer for investigation.

Unfortunately, pasting the SQL into Query Analyzer causes the statement to appear on one line entirely. You then find yourself inserting new line characters for readability and for making the error message more precise.

Transforming SQL this way is tedious, boring, and crying for automation. Fortunately, Enterprise Manager responds well to these cries. Just drill down to a table view of your database, right click on a table, and select “Open table” followed by “Query”.

Now paste your extracted SQL into the text portion of the Query by Example window and hit “Run”. Although the query might not actually execute, it’s transformed into a more readable one.

Obviously, you could start and stay in the QBE window. Personally, though, I dislike QBE as a working environtment beyond for layout beautification, so I usually copy the transformed query into Query Analyzer.

By the way, the transform also comes in handy if you want to make sure your query doesn’t contain any camouflaged expensive joins, e.g., for a query with inner joins done using where clauses, QBE transforms it into explicit inner joins.

It stalls at “Windows is loading files …”. Apparently the subsequent switch into graphics mode causes Vista to hang, because Vista all the sudden doesn’t properly support the emulated video hardware of VMWare.

That’s odd, because I previously installed Beta 2 on VMWare without any sign of trouble (could it be that MS wants people to try out RC1 on their free Virtual PC? By the way, VMWare Server is free too).

Luckily, Joel on Software provides a solution to the VMWare issue, by adding

    svga.maxWidth = "640"
    svga.maxHeight = "480"

to the configuration file for your virtual machine prior to booting the Vista DVD. It’ll force the installer to run in SVGA mode. Then when the installation is complete and you’ve installed the VMWare tools, remove the above lines and Vista shines.

My intention was to run Vista in VMWare on a secondary computer and connect to it via Remote Desktop. For some yet to be uncovered reason, however, I wasn’t able to connect from my WinXP machine. Perhaps the host OS blocked the incoming request or perhaps Vista just wasn’t properly configured.

Anyway, I decided to run Vista on the bare hardware instead, so I fired up PartitionMagic. Unfortunately some guy accidentally cut power while PartitionMagic was doing its shrink-existing-partition-to-create-space-for-an-additional-one magic. After that the existing OS chickened out and bluescreened on me, so out everything went and in Vista came.

Thus, now I utilize a two-monitor setup, running Vista on a native 2.6GHz processor with 2GB of memory on one monitor and WinXP on the other. Unfortunately, the Vista machine has a crappy video card, so no fancy, transparent UI for me (unless I find a workaround).

As a final note, I’m using Synergy to seamlessly share one mouse and keyboard between the physical machines as if it was one machine with dual head video output.

Way cool!

I’ll use a real-world build tool I’ve written as the driving example. Typical tasks for such a tool would include, but not be limited to, checking out sources from cvs, compiling the sources, and running NUnit against a number of generated targets.

Each such task may be defined as a function, taking any number of arguments and returning a list of the form:

[status, [(command, output), (...)]]

where status indicates binary success, command is the command run to complete the task and output is the output (stdout/stderr) of running the command. There may be more than one (command, output) tuple inside the sublist, if the task requires more than one command to complete.

In FP the list and tuple are fundamental and very powerful concepts, because they can easily be used to define complex structures for you to use when passing data around. They also facilitate easy processing using list comprehension, the functions build into the language, or functions of your own, providing elegant syntax for common operations, such as insertion, deletion, searching, slicing, etc.

These lists and tuples are more dynamic in nature than their array and struct counterparts in, say, C#. In fact, rather than creating user defined types, a list, or a tuple, or a combination often suffices, eliminating the need to write a lot of boilerplate code.

Now, to provide an example where Python’s FP support really kicks in with the build tool, is in the implementation of the code responsible for running NUnit against a number of targets:

def NUnit(nunitconsole, projectsOrAssemblies):
    results = [Run("\"" + nunitconsole + "\" " + source)
              for source in projectsOrAssemblies]
    commands = [command[1][0] for command in results]
    outputs = [output[1][1] for output in results]
    failures = Sigma(re.findall("Failures: (\d+)", str(outputs)))
    return [not bool(int(failures)), zip(commands, outputs)]

This function relies heavily on the use of lists and list comprehension, rather than imperative style loops (which, of course, is also possible in Python). List comprehension is a construct that allows you to easily create a new list as a result of some processing on the elements of another.

A highlevel walk-through of the function: first the NUnit executable is invoked one target at a time. The Run function itself returns a list of the form [exitcode, (command, output)]; hence with n targets to test, the commands and outputs variables end up containing n commands and their output. The Failures variable counts the total number of failures accumulated across all targets, extracted as a list of strings matching the regular expression and summarized by the Sigma function.

Python doesn’t allow summing over a list of strings using its build-in Sum function. However, you can easily implement a Sum function that does in something along the lines of:

def Sigma(list):
    return reduce(lambda a, b: int(a) + int(b), list)

where reduce is a function present in most, if not all, FP languages, which reduces a list of values to a single value by applying a user-defined operation to pairwise values of the list.

Finally, the zip function is a build-in function, merging the elements of two lists into one.

Another important construct in FP is the lambda notation. It enables you to create an anonymous function, a function without an explicitly name, leading us to another key characteristic of FP: functions are treated as first-class citizens, in the sense that they may be passed around as just another variable, e.g., you can create a list of functions with parameters and then do the actual invocation at a later stage.

Using deferred invocation, the build tool manages the overall build process; in a config file, similar to a makefile, a list of lambdas are defined, each defining a function that executes a task. This provides us with a generic way of annotating a task and for each task to be run by a central runner, which takes care of logging description, command, and outcome to a html report. The prefs used as parameters to each task is an associated array of associated arrays of … well as many levels as you require.

In effect the config file defines a domain specific language, using regular Python syntax, which can easily be loaded into what corresponds to the make utility by an import statement:

prefs = {
          "cvs" :
          {
            "cvs" : "C:\\Program Files\\cvsnt\\cvs.exe",
            "workingDirectory" : "C:\\Inetpub\\wwwroot",
            "root" : ":pserver:foo:bar@1.2.3.4:/foobar",
            "module" : "baz",
            "revision" : "HEAD"
          },
          "nunit" :
          {
            "nunitconsole" : "C:\\...\\nunit-console.exe",
            "projectsOrAssemblies" : [C:\\Inetpub\\...\\foobar.nunit" ]
          },
          ...
        }

tasks = [ ("Checkout Foobar from cvs",
           lambda: Task.CvsCheckout(prefs["cvs"]["cvs"],
                                    prefs["cvs"]["workingDirectory"],
                                    prefs["cvs"]["root"],
                                    prefs["cvs"]["module"],
                                    prefs["cvs"]["revision"])),
          ...,
          ("Running NUnit",
           lambda: Task.NUnit(prefs["nunit"]["nunitconsole"],
                              prefs["nunit"]["projectsOrAssemblies"]))
        ]

This example, using the basic concepts of FP such as lists, list comprehension, tuples, build-in functions, and nice syntax, hopefully provided you with a glimpse of how a lot can be achieved in relatively few lines of Python code. Also, if you really want to explore FP from a Python perspective, I suggest you take a look at the Xoltar Toolkit (unfortunately, though, it isn’t under active development), described in David Mertz’s columns More Functional Programming in Python and Even More Functional Programming in Python

Bottom line is that the use of FP style in your programs, although it takes time getting use to, leads to fewer lines of code, which leads to fewer places where things can go wrong.

This is not to say that your entire program should be written using the FP style (as is the case with Haskell). However, learning the basic concepts, and how Python exposes these to the developer, you’ll know which patterns to look for in your code that may be better solved using FP constructs.