I’ve not written for a few weeks because we’ve been busy moving house.

In an attempt to turn my brain away from the lure of unpacking more boxes and moving more furniture around, here I am in front of the keyboard for a short while.

Dealing with boxes for a number of weeks has reminded me of a few concepts in software development: notably the power and pitfalls of abstraction, and the effects that deadline pressure can have on technical debt and efficiency.

Yes, there’s a metaphor in everything if you look hard enough.

Abstraction

One of the most powerful tools that we have in software is abstraction. Designing data structures, classes, interfaces and APIs well can have downstream effects on efficiency that are in the orders of magnitude: people want to use tools and systems that are understandable and easily composable.

Good abstraction, applied at the right time, prevents technical debt. Designing the right primitives in your system can be the difference between others wanting to build upon it rather than wanting to rewrite it as soon as the business logic starts to get more complicated.

When you’re moving house, boxes are a great abstraction to deal with, since you don’t need to worry about all of the stuff that’s inside of them, and nor does the removal company.

The complexity of a house’s worth of items of different sizes are abstracted away into uniform objects that are easy to move around and stack in a way that the individual items cannot. Tessellation? No problem.

The idea of boxes and a removal van to take them somewhere is a simple concept: we abstract away all of our items, label them correctly, and then they are put back in a similar location at the other end.

But, as with most projects, it might be simple, but it’s not straightforward.

Technical debt

No matter how hard you try in any project, technical debt is going to happen. Here’s one of my favourite cartoons from MonkeyUser:

It always happens.

Sometimes you design something based on incorrect assumptions, or you end up with not enough time to implement something properly so that in the future it is extensible and easy to work with. Perhaps, like the cartoon, you just go too fast and paint yourself into a corner.

We started packing months before the moving date, as we are both working full time jobs as well as juggling all of the usual things that come out of a modern family life.

The initial phase of packing is usually the easiest, as you can tackle all of the areas that have the least dependencies on day-to-day life: emptying those drawers that haven’t been touched for a year, packing all of the winter coats while it’s summer outside, and deciding that you can get by without using that slow cooker for a while.

Easy.

We also started in a well-organised manner. We had pre-made stickers for labelling all of the boxes, and we used a system design of grouping items by size and by intended location (kitchen, bedroom, bathroom, etc.). So, we certainly started out aiming to avoid any technical debt here, but like every software project, other constraints and project-related phenomena begin to get in the way.

Packing to move house, from experience, seems to be a manifestation of Parkinson’s Law, where the work expands to fill all of the time available for its completion. This may sound like the last software project that you did, and it will probably be like the next one that you do as well.

When it came to packing, a combination of the difficulty of trying to estimate the amount of time that it would take (can we take a day off this weekend, or not?), plus the increasing dependencies as the packing went on (e.g. you can’t pack beyond the minimal essential daily items for each room, and everything packed increasingly uses up space needed for more packing and organising), means that lots of stuff just has to happen right at to the last minute, which is where your design starts to go out of the window and you introduce spoonfuls of technical debt.

Deadline crunch

In the crunch period in the final few days leading up to moving, you usually realise the following things:

Everything you haven’t packed yet is awkward in some way. Either you have the remaining items that didn’t fit nicely into boxes, or you have random bits from every room that don’t have clear logical box groupings, or they are breakable, difficult to manage, or just annoying. For example, should all cables be kept together as cables, or should they be taped to the item they plug into? What about all of the odds and sods that never got sorted out last time?
You haven’t got enough time to sort it out properly. Because the removal date is fixed, you know that there isn’t really anything that you can do. You can’t unpack all of the previous boxes because the resulting refactoring is too big and you don’t even know how you’d do it, let alone getting it done in time.
You start making shortcuts and hacks to get the project over the line. This is where you end up with boxes marked “stuff” or “decor” or “misc”, or where you begin labelling all of the random boxes with the same room destination because you feel that in the future you’ll be able to sort it out easily if all of your random boxes are in the same place.

Even though you had a great plan, and you had plenty of time, inevitably it was too hard to estimate the real complexities until you got going, and the consistent marching of time meant that it was too difficult to keep the same design and implementation purity as you went along.

Does this look familiar?

Testing in production

Even if you’ve managed to keep your well designed boxes to say, 80% of the total and your boxes of miscellaneous junk to around 20% (hello, Pareto), using a removals company is like the DevOps divide.

A removals company who you haven’t necessarily met before, will turn up with their truck and people on the day, and you haven’t had the ability to do any load testing or a dry run. You’ve assembled all of your containers and labelled them, but what happens next is out of your hands.

On the day of the move, just minutes after you seal the last box, the truck turns up and a squadron of operators start loading them based on different criteria that you used to pack, because you weren’t aware of their requirements, and they weren’t aware of yours.

Here, the truck is organised in such a way that it needs all of the heavy and bulky furniture to be loaded first, because of the way that it is stacked and tethered to the sides. Then, the boxes are moved and stacked via an algorithm that was not part of your design. Because, of course, they’ve not seen your design, nor is it important to them.

Then everything travels to the destination. If you’re lucky, it’ll all get there in one piece. Then begins unloading, where you need to map all of the items to new locations in the new home.

At the other end, not only is everyone tired, but the want to do everything perfectly is overridden with the want to just get the damn move done. You soon begin focussing on the end goal (i.e. finishing shipping) rather than the quality of the process.

This manifests in a number of wonky decisions.

For example, the mapping between the initial environment and the final environment isn’t always clear, so your labelling doesn’t necessarily translate across: which is the master bedroom versus the second bedroom? What exactly does “family area” mean? Depending on the individual, things may end up in different places. And I’m not just talking about the removals folks—people in your own family might not even agree on how to interpret the plan.

At the end of the day, when the door closes and you survey the space, you’re just glad that everything is there and you’ve managed to ship it all. Yes, it’s full of issues and a bit of a mess, but you’ll sort them out later.

You consider the rest of the work to be on the backlog.

Technical debt realised

The next day, the technical debt that was incurred in the system starts to come back to haunt you.

Walking around the different spaces that are filled with boxes, you can spot the ones that should have been stacked elsewhere instead, either through your own poor labelling predictions or from how you were feeling towards the end of the day. As you see the last 5% of boxes rolling off of the truck, you become more motivated to “just stick it there” rather than ensure that it’s in exactly the right place.

You also can’t help but notice the giant stack of boxes in one location that are all labelled ambiguously. Ah yes: your random pile!

You seem to have way more “decor”, “cables”, “electricals” and “miscellaneous” items than you can ever have imagined. And what was in them again? The issue is that you’re going to have to open all of those boxes and unpack them in order to discover where they need to go, and then repack them again to take them there.

The joy of technical debt. What a nightmare.

It’s all the same

There are so many parallels to software projects when you move house:

Estimation is hard, and the managers don’t know the details. When we knew we were moving, we had a sales manager from the removals company spend five minutes looking around our house and then came up with an estimate of the time and number of people it would take to do the move. They couldn’t see the place we were moving into, so went by our verbal description. We also roughly estimated how long we thought it would take to pack, and we were pretty far off in our estimates, regardless of being quite experienced at this now. The only time you have zero uncertainty in a project is when you’ve done it, which is obviously too late in the process to help inform an estimation.
You’re always fighting against Parkinson’s Law. Even if you had a year to pack to move house, dependencies mean that you can’t finish until right up until the last moment. Also, if you have more time than you think, you end up increasing the time that you spend on the design and implementation, often to the point of diminishing returns. It doesn’t matter that the contents of the box are expertly packed and grouped if the deadline crunch meant that you forgot to label the outside correctly.
If the people shipping your project aren’t the ones that designed it, you better hope that you have clear documentation. When you move house, the removals staff haven’t seen your old or new home before, nor have they seen any of your stuff. If you’re unable to provide them with clear documentation and playbooks, don’t be surprised if your things all end up in the wrong place, even if you know exactly where it should all go in your mind.
Deadline crunch will make you do terrible things. That last 24 hours causes countless boxes of “miscellaneous” to appear, and the amount of time that it takes to deal with them after the shipping date is exponentially higher than the amount of time required to deal with the boxes that were well-planned.
Technical debt is inevitable. In any complex process that is hard to estimate, it’s inevitable that there will be subpar decisions made, and it is never through malice. Either it comes through lack of information, understanding, or poor interpretation. Going into the process expecting that everything is going to be perfect on the day isn’t just the wrong thing to do, it can be harmful. That’s because…
You need time after shipping to clear everything up. After the deadline, you’re going to need time to get everything back in order. Things need to be tidied up, moved around, removed, or reconsidered. And that’s just a normal part of the process, rather than feeling that it’s as a result of failing to do something earlier on that you couldn’t have predicted.

Is it any surprise that we struggle to estimate and deliver software projects reliably?

Anyway, back to unpacking I go. I consider this to be the clear up time after hitting the deadline.

The Engineering Manager

Moving house is like building software