(17th October 2013)

Going Back to Basics - Overcomplicating Estimating, Analysis and Deadlines

If there is one thing that frustrates me more than anything, it is how things are made more complicated than they need to be. This is not just in the world of software engineering, but everything around us. It could be anything from the process of applying for a mortgage to opening food packaging.

My latest gripe is around the process of estimating and analyzing software engineering projects and calculating deadlines. Having worked for a number of large global organizations that employ extremely talented people, they all seem to have the same pitfalls.

Business and Product owners usually ask the same question for every project. "How much?" and "How soon?" Yet with this in mind projects always seem to be overcomplicated and delivered late, estimating is hugely inaccurate and communication with the Business and Product owners is always an issue.

It seems that as software engineers, we think logically and this can result in looking at things too deeply, disregarding the basics. It is almost as if we think up a complex algorithm or process to solve an issue before we really know what the problem is. This mentality quite often crosses over into simple day-to-day tasks and we become oblivious to what we are actually trying to achieve. This escalates when like-minded people work on these processes together and things can escalate out of control.

What fascinates me is that other industries, with little or no formal training either don't have the same issues or solve them relatively easily, and manage to work efficiently and effectively.

For example, a car mechanic or a garage owner, probably doesn't understand Agile methodologies and the concepts of Scrum, Kanban or Fibonacci's theory for sizing work, but they still utilize the same set of principles, probably without even knowing. I really don't think my local garage have a daily stand up meeting informing the other mechanics of what they worked on the previous day and what they will be working on for the day ahead. I can't imagine them sitting around a table on a Friday afternoon having a retrospective meeting and analyzing burn down charts. I am pretty sure that in your bog standard local garage, this just doesn't happen.

The auto repairs industry has been happily working for a long time using an effective process for breaking down work, estimating and invoicing, so why is it so different in software engineering. The only variable in software engineering is that we use a different measurement as a calculation for cost. For the purpose of this article, we will use story points.

To understand some of the frustrations Product and Business owners experience with delayed projects, we can use the same comparison. When a software project is estimated and sized a cost is attributed to it in the form of story points. If there is a delay in the project, this will come at a cost in the form of additional points.

So lets take this scenario. I take my car into a garage for some repairs, and am given an estimate of £50 for the work and am told my car will be ready by the end of the day. On arriving to collect my car, I am presented with an invoice for £500. Not a happy bunny!

Exactly the same scenario applies to software engineering projects. If I give an estimate of 3 points to the Business or Product owner, and when I deliver the work, it actually comes in at 15 points without informing them, there will be some angry Business or Product owners.

In both scenarios, if you knew the cost would be a lot higher than expected, you would have had the option not to proceed with the work and to spend you money/points elsewhere.

To the same point, if I am told my car will be ready by the end of the day, but when that time approaches I am then informed that it will take 1 week to fix, again, I would not be happy. When deadlines are pushed out for projects this has the same effect with the Business or Product owner.

Sometimes there are underlying issues that cannot be immediately detected and will increase the cost of delivering your project or fixing your car. It is possible that once the mechanic has removed the engine, they then find other problems that were unexpected or not visible before. This also happens in software engineering projects. At this point effective communication is paramount.

When an unforeseen problem occurs with my car, I get a phone call from the garage telling me about the issue, how much extra it will cost me, and how much longer it will take. We need to do the same with our software engineering projects. If any unforeseen plumbing issues within the code base are found during our analysis or in the development process, this needs to be flagged to the Business or Product owners immediately, as the added cost created by the issue might mean that they can no longer afford to continue with the project, or they may want to spend the points elsewhere.

There is no such thing as bad news in these scenarios, just the process of passing information. I have always gone by the rule that Business and Product owners will be more appreciative of finding out sooner about issues with a project rather than being informed when it is too late to do anything about it.

When it comes down to breaking out tasks for our projects into testable and releasable pieces, again we need to take the same mentality as the car mechanics. When you view your estimate and invoice for the work on your car, it is usually itemized with the different parts that have been completed and the cost attached to it. This should be the same process when breaking out project tasks. They should be broken into testable and releasable pieces that are of a suitable size to be worked on. Itemizing work will also give the Business and Product owners a good view of where costs come from. If I were given a large bill for the repair of my car with no breakdown explanation, I would most certainly want to know how it came to that price and where the costs came from.

Another problem that arises is how Business and Product owners translate cost, in this case points, into time and deadlines. This is something that most people are guilty of; and maybe isn't the jurisdiction of that person to make comment or assumption.

It would be very easy for me to assume that the cost of my car repair being £20 means that it is a small job and that I will have my car back within a couple of hours. Although cost can sometimes be related to time and deadlines, this can be a very dangerous assumption to make. What I might have not comprehended is that although the cost is low, my car may not be the main priority. It could also be possible that the job is simple, but work involved may take more time as there is only one mechanic available to work on my car when possibly 3 may be required. Resourcing can always be an issue when it comes to the time required to deliver a project.

It is also very common for a Business or Product owner to assume the size of a project or piece of work. This is also dangerous without fully understanding the mechanics of what is underneath the hood. I am not qualified and do not have the knowledge to make an assumption on the size of work to repair a misfiring engine in my car and therefore have little justification to force this upon the mechanic working on my car. Even if I had an understanding of cars, it is more than likely that the vehicle is different from where my knowledge lies, which means the process of fixing it might different. Software engineering is exactly the same. If a Business or Product owner has an understanding of the software being used, it is possible that what is involved to deliver the task could be different in each case.

So we get to the stage when everything is agreed with all parties, we have our 'finger in the air' estimates and have tasks in the sprint/iteration; we then have to go through the farce that is the analysis phase. I don't think there is an engineer in the world that likes analyzing the work they are due to undertake, at least I haven't met one yet.

My bugbears with analysis of tasks are the amount of time it takes and the varying depth that engineers go into. Also, in many environments, at this stage the engineers are left alone to fend for themselves, whereas I see this as a more collaborative process with the Business or Project owner and any quality assurance personnel.

There is nothing more frustrating than spending 4 hours analyzing a task that only takes half an hour to complete. For many engineers working on larger scale projects, it can seem that you spend most of your time updating work flow systems and writing documentation that it seems nobody is ever going to read, just so that everything fits into the Agile process.

For a large-scale project that could have 10 tasks, probably 40% of the estimate could be attributed to analysis. If my local garage had to work on 10 items to fix my car, I doubt they would analyze each piece of work only when they have finished the previous item. They would take the job as a whole and analyze what needs to be done before starting work. So, why do we work in a way that makes us break each task down to its smallest element and treat it as it's own project.

From my view, the whole project should be analyzed after it has been defined with everybody involved, as a group. This includes Business or Product owners, Engineers, Quality Assurance, Architects and Project Managers. You then have a list of tasks, which individually do not need their own analysis. This will then allow engineers to do what they like doing the most and write the code.

Of course, this does not solve the problems of finding unexpected issues during the development phase of a task, but it doesn't matter how in depth you go with your analysis, this is still a risk when working on complex web sites as part of a team.

My final point is about the Agile process as a whole and if one solution fits all situations. Having worked as a Developer and Manager using both Scrum and Kanban, both have their ups and downs but whatever process you work with I always find occasions when engineering teams follow process for the sake of process.

Being an advocate for quality, this is where my main frustrations lie with Agile methodologies. The process seems focused more on delivery over quality, this is more so when working with Kanban's 'just in time' approach, especially on larger scale projects.

The Agile principle of 'early and continuous delivery of valuable software' definitely gives early visibility. What is dangerous, though, is that focusing on continuous delivery has the effect of creating a growing defect backlog while engineers work to put something in front of the customer. The small, cross-functional teams can lead to poor documentation, lack of focus on architecture, a messy codebase and other non-functional requirements (e.g. maintenance). In most Agile projects I've been involved with, the defect list has grown faster than it can be dealt with.

The process promises solutions it cannot always deliver. It can promote sloppy or incomplete requirements, can hide the true cost of development and can prevent effective management. Contrary to what we're told to expect, this leads to long-running projects, dissatisfied Business or Product owners and an overall development ineffectiveness.

If the garage mechanics adopted some of these Agile principles for a large job they could end up returning my car with additional broken parts and no indication of when they could be fixed and at what cost. All of this in an attempt to cover up cost and keep to deadlines. At any reputable garage, this just would not happen, so why should developers become cowboys of the engineering world.

I am not saying that Agile methodologies are wrong, but that sometimes we need to go back to basics, taking away much of the complexity that we don't even realize we add. These processes have of course had a large amount of time, money and research go into them and by utilizing them to their full potential you can encourage much more effective engineering. But by looking at a project as a whole, a clear decision can be made on how best to manage the project. There is no rule about having to use the same process or methodology for all projects.

I suggest we think more like our local garage - take a step back to review the bigger picture, simplify and communicate regarding cost throughout the whole process.

View this article on Ezine