Data Transformation Wizardry with RhinoETL


I recently gave a talk for the Christchurch .net User Group at the Epic centre ( I spoke about what ETL is and went through a number of worked examples showing how to apply RhinoETL.

I think the Data Anonymiser example was particularly well-received.

I have put the slide deck and the example source code up here for everyone. Please let me know if you find this useful.


50 Shades of Grey

This is my (somewhat overliteral) interpretation, implemented in HTML 5 using Canvas.
I thought I'd try out some of the new HTML 5 features and have a bit of fun at the same time.
Who said HTML 5 wasn't sexy!
Your browser doesn't support the canvas element
Javascript for above:
function initiate() {
    var elem = document.getElementById('canvas');
    canvas = elem.getContext('2d');

window.addEventListener("load", initiate, false);

function draw50ShadedBoxes() {

    var boxesdrawn=0;
    var x = 1;
    var y = 1;

    var width = 50;
    var height = 50;
    var separation=75;

    var greyness = 10;
    var col = "hsl(10, 50%, " + greyness + "%)";

    for (y; y <= 8; y += 1) {
        for (x; x <= 7; x += 1) {

            if (boxesdrawn == 50) {

            var x1 = x * separation;
            var y1 = y * separation;

            greyness += 1;
            col = "hsl(10, 0%, " + greyness + "%)";

            canvas.fillStyle = col;
            canvas.fillRect(x1, y1, width, height);

            boxesdrawn += 1;

WDCNZ - Thoughts on Douglas Crockford's talk: "Programming Style and Your Brain"

I attended WDC2012 last week and had a great time. Douglas Crockford's presentation was probably the highlight for me. These are my thoughts on one particular meme from his talk - 'The Abyss' ... 

Douglas described this as the time that developers spend doing non-productive work when trying to debug/identify an issue or bug. Due to the strange psychology of developers, we are so used to being in 'The Abyss' that we hardly even notice going into it, just the feeling of euphoria from escaping and fixing the problem. 


Being unaware of 'The Abyss' causes a kind of blindness preventing developers from learning from the mistakes. We just fix the problem and move on. However, programs fail due to predictable bugs caused by common and predictable programming practices.  


Douglas recommends that we take time to reflect deeply on our experiences in 'The Abyss' and learn (This sounds obvious, but I'm so busy I don't have time!). Through this reflection, Douglas has observated that simple changes to one's programming style make it possible to eliminate entire classes of bugs. 

An example given by Douglas was: never use ++i / or i++ as it is likely that sometimes you (or someone maintaining the code in the future) are going to get caught out by whether it's pre/post increment. Instead just use i+=1 and the issue never arises. 

So I'm going to try the following if I find myself stuck in the 'Abyss':

  • Keep a log of time spent in the Abyss.
  • Document what I did to get out.
  • Regularly reflect upon how I could have avoided getting there in the first place.
  • Use my list to conduct a personal code review before committing changes.



Recently I spent ages trying to work out why Windows authentication wasn't working with a recently deployed website. It turns out that because I did the deployment by hand, I'd forgotten to check an option in IIS. 

Learning outcome:

Deploying websites by hand is dumb, it's time consuming and error prone. Invest in automation and you can be sure it's done right. 

Thanks Douglas!

Thoughts about multiple screen interfaces.

I went to the .net User's group meeting today, where the discussion was about MVC.

Afterwards I had a chat with a few folks about how Microsoft was doing with Windows 8 (after trying the developer preview).

My thoughts:

  1. There seems to be a disconnect between Metro style apps and 'Standard' Windows 
  2. Why do they need to be 'baked' into the OS - couldn't we get something just as rich over the web?
  3. MS is betting heavily on the future of PC input being "Touch".
Now the "Touch" interface thing got us talking about how we could see it working in the future:
You could have 2 screens (they are getting cheaper all the time):
  1. one for (horizontal) for navigation/display/input  (replacing keyboard and mouse)
  2. another (vertical) for display.
 The idea being that it's a lot less strain to drive things horizontallly. Additionally, with 2 screens, your hands aren't going to constantly be obscuring the areas of interest.
A really cool thing would be if the input screen was flexible. This would mean when keyboard input was required, a keyboard imprint could be raised from behind the screen, to create a keyboard with tactile feedback from the screen.
Another similar idea exists for the eBook format. Here you could have a book shaped device with 2 screens, one each side. To go to the next page, you turn it around, (right edge first). Previous page, turn around (left edge first). The 'next' page could be pre-rendered as soon as you turn (effectively double-buffering), this means the page turn speed is a quick as you can turn the device. I think this would be quite a natural way of reading, with NO buttons or pressing of anything.

Sound not working on my PC

I've got Cambridge Sound works analog speakers (DT2200) 5.1 system from about 2001 still going strong (about the only thing about my PC that hasn't changed). The mini speakers and subwoofer are really great, I use 3 of them L,R,C and don't bother with rear speakers as it's quite a lot of wires.

Anyway, just now the sound stopped working. I've just spent an hour trying to workout why. I went through all the obvious stuff like:  

  • do the speakers have power
  • are all the cables plugged in
  • check the volume isn't muted
  • check the volume settings aren't too low

Then I tried a few more esoteric solutions:

  • Windows 7: Troubleshoot audio playback wizard
  • Updating my Realtek Audio drivers
  • Swapping out audio cables and speakers

No luck. Just as I was about to give up, I realised:

The volume control dial on my speakers also has an adjustment for Front/Rear speaker balance. A certain small child had obviously 'played' with this while I was away, and adjusted it so that the balance was 100% rear speakers (which aren't plugged in).

So my suggestion to Microsoft is they add a step to their troubleshooting wizard:

- Have you checked that small children haven't fiddled with the front/rear speaker balance setting

Anyway, I thought I'd post this in case it helps anyone else with the same problem.

Factorial Bug Hunt

I was playing around with TDD again. It highlighted to me just how bad a very simple and correct looking function can be.

In maths, Factorial(n) is written n!, so Factorial(10) is the same as 10!.

1! =1
2! =1*2=2
3! =1*2*3=6
n! = n * n-1 * n-2 * ... * 1

I wrote this function without TDD to calculate the nth Factorial number recursively:

public static long FactorialRecursive(int index)
   if (index==1)
      return 1;
   return index * FactorialRecursive(--index);

I then rewrote the function using TDD. I discovered 4 quite serious bugs in the above code, see if you can find them (answers in my next post, although I expect there will be others that I've missed too). I also tried a radically different approach to writing the function, how would you write it?

  • The benefit of unit-testing is that seemingly correct code should get smoked out.
  • The benefit of TDD is you get a better design, and just enough tests to prove it is correct.

Amazon EC2 - Short Term Use Tip

I just recently left one of my EC2 compute instances running for a week without meaning to - this gave me an extra $20 bill. Not the biggest in the world, but still annoying if you don't need to pay for it.

For my automated build server, I only need EC2 instances a couple of hours at a time never more than a day. So as a precaution:

  1. add a scheduled task running daily to automatically shut it down at 3am. (The command 'shutdown /s' can be run to achieve this on a Windows Server 2008 box)
This way, if I forget to shut down an instance it won't run for more than 24 hours before it shuts itself down :)

Version Control your CV!

I've just been updating my CV with my latest experience. I noticed that I've got a lot of different versions of it kicking around at different dates and basically all these files were becoming a bit of a pain. 

I've been using Mercurial (and FogCreek's Kiln offering hosted Mercurial) for Version Control lately (on personal and work projects) and it struck me as a good idea to put my CV into it's own mini repository. Having a local copy cloned to the cloud means it's backed up too.

I've got a plain text version which allows me to see the changes between versions.

E.on in the doghouse for Data Overkill

I just recently received a free home energy monitor pack from E.on. This is a great deal, they send you a free box with a transmitter that attaches to your mains fuse box and a wireless monitor device that you put somewhere handy in your house. This lets you easily see at a glance how many watts you are using at any one time and how many KwH you have consumed in the past day / week / month.

So great, we can all use these boxes to monitor how much energy we are using and can find easy ways to cut down and save the planet. 

The wireless monitor box comes with a USB lead and cd to install PC based client software. This is to enable you to download data from the wireless monitor/collector via USB into your PC, enabling you to do more sophisticated analysis of your energy usage. 

The downside is that each client install requires a SQL Server 2005 Express install. This is in my opinion slightly overkill for logging a bit of data about your energy usage!

Chess: Blackpool 2007 Best Game Prize

Here is a game that I played back in 2007 and won the best game prize of the tournament.

After mishandling the opening as black, I managed to gain the initiative with my 14th move. Black's 18th move wins the game with an unexpected move.


Get Microsoft Silverlight