For anyone following this blog I have been doing most of my posts here www.grahambrooks.com/blog

Programming in the small

A personal quick reminder of these excellent blog entries.

http://ivan.truemesh.com/archives/cat_programming_in_the_small.html

John Wooden: Coaching for people, not points

I came across this really interesting talk by John Wooden about his coaching career and the measurement of success. Although the focus is basketball it is remarkable how many thoughts have relevance to other walks and works of life.

"Success is peace of mind attained only through self-satisfaction in knowing that you made the effort to do the best of which you are capable."

View the talk on TED

Think about change

Over the last few years I have noticed a growing trend in software developers to constrain their thinking to static models.

Static thinking is a great way of envisioning a system and we have many tools to support modelling systems in this way - class, deployment, network diagrams all fall into this category. But all these models are about a single state of the system - either historical, real (current) or imaginary (future).

However many projects involve more difficult thinking. Typically these difficulties involve moving from one state to another and the most complex of these is migrating from one version of a system to another.

It is at this point that I struggle to find an effective model that captures this transition. How do we model and capture the upgrade of a database, application and infrastructure requirements. And further these transitions occur at different times and are typically not instantaneous which can mean that the service is not available. How do we model this sequence of events so we can understand the effect of our evolving software design on the rolling out the new software.

Quite often these rollout plans are written down (impact analysis, rollout network models, sequence diagrams, flow charts) and held in the minds of the project team. But these models are not easily tested. Organisations with a significant investment in their live environment often struggle to replicate that environment to allow the model and plans to be tested before hitting the production environment.

I think finding an effective tool to model and execute software updates will be one of the key challenges for this decade - as it has for the last two.

Macs on trains

On my current project I am travelling more that usual, mostly by train between London and the North West. The journey time is pretty manageable and the service (so far this year) has been pretty good leaving and arriving on time.

The thing that struck me today was the number of Macs in coach B. Of the 10 computers in used well over half were macs. I remember some time ago that macs were a rarity for train travellers but no more. Ok most of them were MacBooks and not the shiny new aluminium ones but the trend was quite startling.

The Designful Company

The Designful Company by Marty Neumeier

I have to confess that when it comes to reading books related to work I have a short attention span. For technical books I tend to skim through the text and examples until I get a 'feel' for the content. Most of the time this is sufficient, after all if I need more details I can come back to the book and dig into them. The most important thing for me is to have a high level view of the technology and to know were to get more information.

So when it came to reading 'The Designful Company' by Marty Neumeier I kicked off with my usual skimming approach - but quickly found myself changing tack and putting my novel hat on - reading every word!

Quoting Thomas Aquinas "Ad pulcritudenum tria requiruntur integritas, consonantia, claritas". Design requires 3 qualities

• integrity


• harmoney


• radiance

Reading this I was struck by the thought that these are the things I am looking for in the architecture and design of a software system.

In fact many of the ideas presented by the book (intended for a wide variety of companies) resonate with current agile software development trends.

The bit on agility (p21) brought a smile to my face

'The Designful Company' is an easy thought provoking read. Presenting a strong argument for corporate adoption of design to drive growth in the 21 century.

Given my reading habbits it also helps that the book is small and layed out well with large print.

It may be just me but I am seeing a convergence in thinking across many different disciplines - or maybe software development practitioners are still learning from others.

SCMRSS reaches Alpha 1 release

SCMRSS is a simple web application that turns Source Control events into an RSS feed. Written in Ruby using the Ramaze web framework. Once configured the web server polls the source control repository for changes and when found delivers those changes as a simple RSS feed.

This is the first alpha release so I would welcome any and all feedback either here or as issues at RubyForge

Remote Pair Programming setup

For a few years now I have been looking for a solution that allows two or more people to share an editing session who are not sat side by side.

Each OS has its own preferred way of sharing a desctop/workspace for collaborative working. Some work cross platform (e.g. VNC) and some work via the internet (e.g. GoToMyPC). But for much of my daily work I want a nice fast way of sharing the code that I am working on with a collegue. One or both of us are likely to be behind firewalls, proxies and all sort of important security that makes collaborative remote working so very very hard.

On my current project we are using Eclipse as our development IDE, a local IRC server for ad-hoc team communication and point to point instant messaging. Backed up by Skype for person to person video and occasionaly more traditional email and phone for that personal touch.

All in all this is working fiine and I have to admit I had forgotten how useful IRC is in comparison to IM when it comes to team working. Just having everyone aware of the conversations that are taking place can be a real boon.

I had some spare time last evening and decided to see if I could track down a viable solution to the remote pair programming. Confining my requirements to either complete desktop sharing or Eclipse based paring helped quite a bit because the other added complication is that we have a mixed OS development team including Windows, Linux and Mac OS X. The number of OSs is likely to settle down to just two but at the moment we have quite a mixed bag - which is actually quite refreshing.

So back to the remote pairing issue. I quickly realised that sharing the editing session is likely to be sufficient to our needs and that other tools could provide text, voice and video quite effectively and did not need to be replaced.

My first efforts involved the Eclipse Communication Framework (ECF). I had played about with much earlier versions and concluded that it was a little fussy and difficult to work with for my tastes but decided to give it a go. Working behind a proxy with limited ports meant setting up and running a local server.

I can see a lot of promise in the ECF but it just feels far to heavy weight. I might have made some errors but I could not get it to work, connecting to the server seemed ok (although there were a lot of stack backtraces on startup - now I really cant understand why people still dump these things to the console with the idea that users - even developer users will have a hope of understanding what went wrong. With 3 pages of small text flying past it is just too easy to spot the line that tells you what when wrong. So after an unhappy hour or two trying to get two instances of Eclipse to open a shared editing setting I gave up and went back to searching for alternative.

After a little bit more searching I cam up with XPairtise and open source project that seemed to do exactly what I now wanted - share an editing session. Unusually for an open source project the documentation is pretty good although it was early in the morning and I almost missed that there are two downloads; one for the eclipse plugin and the other for the server.

The setup which I eventually came up with involved the XParitise server and Eclipse running natively on my Mac and a Ubuntu VM mimicing a remote pair. The server is nice and quiet just reporting that it is up and running - a refreshing change. After setting up accounts through the Eclipse preferences pane (a little quirky on the UI the first time around) but it was heart warming to receive the 'account created' message.

Getting the shared editing session to work took some time. First when creating a shared workspace all the files from the project are shipped up to the server. When joining the shared workspace again the project files are brought down so take heed of the backup dialog or work in a different Eclipse workspace for shared working.

It took at least 2 Eclipse restarts to get the shared editing to work and there is a note on the XPairtise site about using the eclipse -clean option to refresh all the plug-ins

But after this initial setup headache I have a configuration that will allow remote pair programming.

Result!!

Update: I have just rerun the setup within a distributed team (2 locations) and after a bit of a lag in synchronising the project contents everything worked fine with 3 concurrent users (Driver, Navigator and Spectator).

Setting Windows Shell Font

I keep loosing track of this little tidbit

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Console\TrueTypeFont" /v 00 /d Consolas

logoff

The full article can be found here http://blogs.msdn.com/ie/archive/2008/04/22/give-your-eyes-a-treat.aspx

Timewarp

About a month ago I was asked to look at an application that has been
under active development and support for quite some time. Written in C
and C++ (my old stomping grounds for 20 years) I was really looking
forward to getting stuck into the code.

What I was not expecting was the lack of tool support. I sort of knew
that refatoring tools, tests and code coverage would be a bit of a
challenge but I did not think that it would be the challenge that it
turned out to be.

I have gotten used to being able to download tools, try them out and
if appropriate buy them and add them to my toolbox. Ideally there
would be an open source tool I could use instead to get the job
done. Moving back into the C++ world also took me back in time to an
era where evaluation downloads are not available. Instead I would have
to register my interest and someone would be in touch to talk to me
about the wonders of their application and to perhaps run the
application over my source code. Well first I tend to get about a bit
and at the time I was out of my home country and well away from the
tool vendors timezone. I was up against the clock in putting together
some initial findings and recommendations, and lastly it was not my
source code so I could not offer it up to someone else to go poking
around in anyway.

Strike one - keep on searching for someone with a little less of a
protectionist attitude.

Unfortunately this seemed to become a theme. Other suppliers had no
evaluations and insisted on money up front and land shipping. In a way
they may have been doing me a favour, if the software is so complex
that it needs to be demonstrated and the licencing enforcement so
strong that it becomes difficult to use (I have been hit before by
node locked liceses to machines that decide to die) then maybe this
approach has just saved me a lot of heart ache and pain.

On the plus side there are tools out in 'open source' land that can
help and there *are* a few companies that have moved with the times
and provide a more open licensing scheme. When I have time I will list
out the toolkit that I came up with.

Too many VMs

As part of my work life I regularly need to run windows development software on my Mac. Typically for a client engagement. This is a really convenient setup. I can take a snapshot before I start work on something new and then at the end roll back to the original state ready to work on the next project.

One of the downsides I have encountered recently is the need to keep all these machines up to date with patches and other software updates that seem to be part of everyday life.

Don't get me wrong I think its great that we have this steady stream of updates to make things better but when you start multiplying this up for the VMs I have on my machine the effort to keep them up to date starts to be become a bit of a pain.

I think it is time to start rationalising to a smaller number - just so I can keep them up to date.

Quietly Does it

I have had my eye on a pair of Bose headphones for some time but never managed to justify the cost. And to be honest I am not sure that I ever will. Except that they are truly fabulous and on my first tip with them on a plane left me feeling relaxed and comfortable. The noise onboard a plane seem to raise my tension levels but removing that noise really helped me enjoy the flight. I was told when I bought them that the batteries were fully charged and would each last 10 hours. I can only testify to 8 on one battery with almost continuous use. I will confess that I have tried a number of different headphones, both in and out of ear, with and without noise cancelling but nothing comes close to the peace that these phones bring with them. So maybe I have justified them :)

Toronto - Sunday

Today is my first day in Toronto, preparing for Agile 2008. After a surprisingly smooth and peaceful crossing from the UK, and an even smoother journey to downtown Toronto I find myself preparing some slides for my short experience report 'TeamPace - Keeping the build times down' on Thursday. I am staying at the Hilton just across the street from the conference at the Sheraton Centre for an early morning start.

Code is Poetry

On a recent visit to the Wordpress sight I was struck by their tag line "Code is poetry". I have been searching for some time for a phrase that would sum up how I feel about well crafted code; code that is easy to understand, read and generally feels 'just right'.

I am not completely convinced that code is poetry - but it comes pretty close.

Agile2008

Next week I will be in Toronto for Agile 2008 to present a short experience report, catch up with friends and immerse myself in all things Agile :)

Trains and WiFi

I am just returning from Leeds to London by train with power and free Wireless internet. Ok it is not the fastest link but works well - even for blogging

Keyboard touching

I am definitely with Jeff Atwood on when it is appropriate to touch monitors - NEVER.

Further perf ruby, python C++ file reading

Following on from the log files article I decided to do some basic perf checks of ruby and python reading text files. The results were a little disapointing - performance was roughly the same, so my ruby log file reading optimisation was complete rot.

Further experimentation required.

ARGV.each do | param | cc = 0 File.new(param, 'r').each_line do |line| cc += line.size end puts "File has #{cc} characters" end Processing /Users/gcb/work/log-analysis/cc.rb ... created /Users/gcb/work/log-analysis/cc.rb.html	Realy simple script - and probably the most obvious - add up the length of all the lines in the file.	File has 1673435763 characters real 0m56.035s user 0m33.873s sys 0m3.609s
ARGV.each do | param | cc = 0 i = File.open(param, "r") begin line = i.readline() until line.nil? cc += line.size line = i.readline() end rescue Exception => e ensure i.close end puts "File has #{cc} characters" end Processing /Users/gcb/work/log-analysis/cc1.rb ... created /Users/gcb/work/log-analysis/cc1.rb.html	Based on previoud observations this one uses the realine method from the IO library but did not affect the performance.	File has 1673435763 characters real 0m55.569s user 0m35.506s sys 0m3.451s
import sys cc = 0 source = open(sys.argv[1]) for line in source: cc += len(line) source.close() print 'file has ', cc, ' characters' Processing /Users/gcb/work/log-analysis/cc.py ... created /Users/gcb/work/log-analysis/cc.py.html	As a benchmark a simple python scrpt - again adding up all the line lengths in the file.	file has 1673435763 characters real 0m53.462s user 0m23.147s sys 0m3.781s
#include <stdio.h> int main(int argc, char** argv) { int count = 0; FILE* f = fopen(argv[1], "r"); while (getc(f)) count++; printf("File has %d characters\n", count); } Processing /Users/gcb/work/log-analysis/cc.cpp ... created /Users/gcb/work/log-analysis/cc.cpp.html	Baseline written in C++	File has 1673392372 characters real 0m53.167s user 0m31.473s sys 0m3.094s
#include <stdio.h> int main(int argc, char** argv) { int count = 0; FILE* f = fopen(argv[1], "r"); char buffer[512]; int read = fread(buffer, 1, 512, f); while (read > 0) { count += read; read = fread(buffer, 1, 512, f); } printf("File has %d characters\n", count); } Processing /Users/gcb/work/log-analysis/cc1.cpp ... created /Users/gcb/work/log-analysis/cc1.cpp.html	A (poor) buffered version of the baseline written in C++	File has 1673435763 characters real 0m52.425s user 0m1.526s sys 0m4.473s

Blogging Code

I quite often find myself blogging about program source code, that code is typically stored in source files which I then run through a pretty printer (something like source-highlight). Combining everyting together means some copy and pasting - not the most repeatable process and quite often the code and article evolve together - so I end up copying and pasting quite often.

So I came up with mashup. A small ruby program to process html files and handle include directives to do inline include of another file and for this purpose the results of a process

The following source was include with

<x:include value="source-highlight -o STDOUT ~/projects/mashup/mashup"/>

By running:

mashup blogging-code.html > blogging-code-publish.html

#!/usr/bin/ruby

ARGV.each do |arg|
  contents = File.new(arg).read()

  contents.sub!(/<s:include\s+value="([^"]*)"\s*\/>/) do |match|
    replacement = File.new($1).read()
    replacement.gsub!(/.*<body>/m, '')
    replacement.gsub!(/<\/body>.*/m, '')
    replacement
  end

  contents.sub!(/<x:include\s+value="([^"]*)"\s*\/>/) do |match|
    replacement = `#{$1}`
    replacement.gsub!(/.*<body>/m, '')
    replacement.gsub!(/<\/body>.*/m, '')
    replacement
  end

  puts contents
end

Log files

Log files are one of those must have things for any web application. It is just so hard to predict all of the possible ways users are going to interact with the site that gathering post live information about application behaviour is essential. It does however produce quite a lot of data.

On my current project, after a fairly significant release we resolved to check the log files to see if there were any unexpected incidents that required fixes or web content changes. The log files ran to several gigabytes containing entries not only from the application but also from a very noisy subsystems. After a quick look it became evident that it would not be effective just scanning through the log files but that some cleaning or automated analysis was required.

On the initial scan we noticed that there were some Java stack traces being repeated so something that could capture the distinct stack traces and then list the errors that caused them. In this was we could look at the general issues based on priority (number of occurrences).

Our first effort was to write a fairly simple ruby script using a hash map keyed on the text of the stack trace. Each map entry contained an array of error lines from the log files. We kicked the script off after testing with a small log file and went to lunch.

When we came back the script was still running. Sometime later it ran out of memory - not ideal.

It has been sometime since I have written any C++. Most of my work these days involves Java, C# and a little bit of Ruby for work around the codebase so it took a little while for my C++ brain to kick in. A colleague also took up the challenge by writing a solution in Python.

The results were quite startling with the Python script performing almost as well as the C++ at around 500,000 lines per second.

#include <iostream>
    #include <fstream>
    #include <sstream>
    #include <string>
    #include <map>
    #include <vector>

    using namespace std;

    class progress {
       int count;
    public:
       progress() {
          count = 0;
       }
       void ping()
       {
          cerr << "\b" << "|/-\\"[count++%4];
       }
    };

    typedef map<string, vector<string>*> error_map;

    void print_errors(error_map& errors)
    {
       for(error_map::iterator iter = errors.begin(); iter != errors.end(); iter++ ) {
          vector<string>* reports = iter->second;
          if (reports->size() > 1)
          {
             string first = (*reports)[0];
             cout << "\n\n\n";
             cout << reports->size() << " instances of\n";
             cout << "FIRST instance " << first << "\n";
             if (reports->size() > 1)
                cout << "LAST  instance " << (*reports)[reports->size() -1] << "\n";

             cout << iter->first;
          }
       }
    }

    void process_file(char* filename, error_map& errors)
    {
       progress p;
       cerr << "\bProcessing " << filename << "\n";

       ifstream file(filename);
       string line;
       string pending_error;
       string pending_stack;
       int stack_lines = 0;
       int line_number = 1;
       bool skipping = false;

       while (getline(file, line))
       {
          if (line_number % 100000 == 0)
             p.ping();

          line_number++;

          if (line[0] == '#' || line.find("Notice") != string::npos)
          {
             if (pending_stack.size() != 0)
             {
                // Process stack trace
                if (errors[pending_stack] == NULL)
                   errors[pending_stack] = new vector<string>();

                errors[pending_stack]->insert(errors[pending_stack]->end(), pending_error);
                pending_stack.clear();
             }
             // Ignore lines from systems we are not interested in
             skipping = line.find("ignore-one") != string::npos
             || line.find("ignore-two") != string::npos
             || line.find("ignore-three") != string::npos
             || line.find("INFO") != string::npos
             || line.find("WARN") != string::npos;

             if (!skipping)
                pending_error = line;

             stack_lines = 0;
          }
          else
          {
             if (!skipping && stack_lines < 20)
             {
                pending_stack.append(line);
                pending_stack.append("\n");
             }
             stack_lines++;
          }
       }
    }


    int main (int argc, char * const argv[]) {
       error_map* e = new map<string, vector<string>*>();
       error_map& errors = *e;

       for (int i = 1; i < argc; i++)
       {
          process_file(argv[i], errors);
       }

       print_errors(errors);  
       return 0;
    }
    

---

The Python solution is a little shorter however

import os, sys

    def is_valid(item):
        for token in ['ignore-one', 'ignore-two', 'ignore-tree', 'NavigationLink instance']:
            if token in item:
                return False
        return True

    directory = sys.argv[1]

    errors = {}

    for filename in os.listdir(directory):
        last_error = ''
        last_stack = ''
        stack_count = 0

        file_path = os.path.join(directory, filename)
        print 'Processing', file_path

        source = open(file_path)

        for line in source:
            if line.startswith('#') or ('<Notice>' in line):
                if is_valid(last_stack) and is_valid(last_error):
                    errors.setdefault(last_stack, []).append(last_error)
                last_error = line
                last_stack = ''
                stack_count = 0
            else:
                stack_count += 1
                if stack_count <= 20:
                    last_stack += line

        source.close()

    print 'Writing report to grok.txt'

    out_file = open('grok.txt', 'w')

    for stack, error_list in errors.iteritems():
        if (len(error_list) > 1) and (len(stack.strip()) > 0):
            out_file.write('Found %d items like: %s' % (len(error_list), error_list[0]))
            out_file.write(stack)
            out_file.write('\n\n')
            out_file.write('---------------------------------------------------------');
            out_file.write('\n\n')

    out_file.close()
    

Both solutions limited the number of lines in the stack trace used for the key to 20. This was fine for non-reflected methods.