Planning

Compromise sticks and spreads

Nobody wants to write poor code or dig through it, yet, somehow, it happens. It all begins with a requirement or a prototype that needs to be developed fast in order to obtain a validation. Management and designers like it, it hits the deadline. You are congratulated for your success.  Up until now, everything is as it should be. This is how prototypes should be developed.

Then a new requirement comes. "Wait! I've only created a quick hack!" But the prototype works and you've already been congratulated and the second deadline is coming fast. You promise yourself that you are going to fix it as soon as possible, but the very day that you check-in, you and other people start adding layers after layers over it and change never happens. It could be that the feature is perfectly engineered and polished at first but, as new code is added on top of it, the original is never truly reworked, so it slowly starts to rot.

The project becomes more and more difficult to maintain and grow. Morale gets down, quality gets down, and productivity gets down, self esteem gets down, will for self improvement gets down. Some people even leave their jobs. Even if some teams actually start by cleaning up their previous mess and bring again the code to a better state, it is a onetime process, that happens only once in a few months or years. Being a one time, massive scale endeavor, the result has high chances to only end up as a different mess.

When one makes a compromise, it sticks. And then, another compromise is needed to cover the previous one. Debts start to gather in and, in the end; the debt is so high that one realizes that he/she will never ever cover it. And the process keeps on going and accumulates more and more debts, until everything becomes so expensive in terms of money and people that we need to throw everything away and start again. 

Why?

I think it all has some simple root causes:

• People develop in isolation. They don't talk so they don't have the chance to unite for quality.
• No code reviews.
• No frequent informal verbal exchanges.
• Rigid planning, commitments to feature lists instead of quality.
• Bugs are not fixed immediately as they appear and, instead, are postponed to "debug periods", and often scheduled only at the end of the project.
• Interruptions of any kind.

Lack of communication and rigid planning make prototyping lose its core purpose, as it is perceived as feature complete when it is merely a quick proof of concept. Developing in isolation makes it difficult for people to say "STOP, we need to change this" and management never truly gets to the bottom of problems to fix them.

Do we really need to start from scratch?

A very good friend said to me yesterday that, in nature, something has to die in order for a species to evolve. But how does this translate to software development? Does this mean that we need to start from scratch in order to evolve? Always?

I don't think so or, at least, it should not happen unless a major technological breakthrough occurs that renders everything else obsolete. I believe that software needs only to be released in order to evolve. It needs clean, frequent releases and then it needs mutations. We, programmers, call them "refactorings". I will stress this again: in order to evolve, software needs to be cleaned up before the release - no debts to the next iteration. And we need a culture that embraces change and mutations. 

Beside internal clean-up, these releases should also reach their customers (editorial teams, designers, beta testers) in order to get their feedback and adjust the feature list for the next iteration. A release without customers is tough to justify (close to pointless) and it's a pity to put in so much effort and not take the opportunity to do some user testing. After all, evolution needs feedback in order not to generate monstrosities. 

To sum up, a healthy development process should accommodate short term iterations consisting of:

• Detailed planning based on a roughly detailed feature list
• Actual implementation; communication among developers, peer reviews
• Debugging and stabilization - no debts for the future, all bugs are fixed
• Sign-off by QA
• Release to customers for feedback
• Communicate status, negotiate features and deadlines, incorporate feedback, and discuss how we can improve the process so that the next iteration gets better.

Managers don't manage people

Another thing I realized, no matter how shocking it sounds, is that managers don't manage people. They manage processes. People buy in and start adding value or they don't. You can assign tasks but you can't really force anyone to complete them. Therefore, the notion of "motivating an employee" makes no sense, really.  He/she is either motivated or not. What you can do, however, is create an environment that appeals to the employees, so they get excited about their job and become productive. And this is the really tricky part. 

Quality from the user's perspective

Indeed, the only thing that matters to sales is how the product is perceived by its customer. Nothing else. Code doesn't matter, as it is invisible. Code only matters to the production people. If it is crappy, it ruins the life of the men and women involved in its production. It turns them away from the product, productivity drops, conflict sets in, and motivation goes away. Yet this does not impact sales directly; it impacts everything else. 

Getting back to process - Planning

I'll make a small detour from the quality issues to examine planning, as the core activity that drives project to completion. In the following paragraphs, I will present the normal planning method with its advantages and disadvantages and how it can be applied. After understanding planning, I will get back and show how to transition to methods that are more appropriate to ensuring quality in software.

Everything in a project revolves around plans and planning. Planning is a continuous process, plans change as requirements change, conditions change and understanding grows. Visibility and vision is based on plans. Communication with stakeholders is based on the same plans; resources are requested just the same. Commitments are based on plans. Monitoring and controlling is based on plans. Strategies are based on plans. Planning and plans - everywhere. Yet some managers either don't take planning seriously or they don't iterate on already made plans to adjust them. Why? 

The answer is quite simple and straight forward (order of bullets is irrelevant):

• In the early stages of the project making informed plans is really hard and it seems that we can do better off without them. After all, it is easy to get people to do something tangible now.
• Some engineers and designers are reluctant to planning as well (especially if they are not used with the process) and managers don't want to deal with this kind of issues up front.
• In the late stages of the project we are so busy putting up fires, escalating issues, requesting resources, fixing and dispatching bugs that we can't stop to plan. 
• Superficial planning is hard to spot therefore allow your plans to be challenged by the stakeholders and the team.
• Planning requires going back to the same document, ripping it off and doing it again many times. It's difficult to change everything when you worked so hard on it. Over and over again.
• Some feel that planning is a waste of time: things seem to work without plans at first.
• Some managers (former star employees) are so used to actively solve issues that have      visible impact on the project that, when they are required to stop and think about the future, they feel it is useless. Why? Because to them plans seem to be only coloured excel sheets, without a tangible impact on the outcome of their project.
• Planning is a costly activity and some organizations or teams may naively try to eliminate it as waste. Planning is costly because it involves not only the manager, but the whole team or, at least, the most senior part of the team.
• If not properly communicated, a plan means a commitment which seems hard to make and changed later.

How to plan? 

Here it is: http://www.projectsmart.co.uk/project-planning-step-by-step.html. Easy, isn't it? :)

For starters, I will assume that we already have some visibility on the sales volume, on the budget and on the release date. Let's assume the mandate sounds something like: "We need a project that will generate roughly X units sold and that will be released around the Y date. It should cost somewhere around Z monetary units." I will assume that there is historical information in the company about this kind of project and that the constraints are reasonable. Also, probably most important of all, we have some good visibility on the project deliverables and core features - that means that we are well advanced through Step 1 (Project Goals) and Step 2 (Project Deliverables) described in the link above.

Top-down approach - project schedule:

1) We know the release date and the budget, therefore we can establish quite easily the major milestones and the resource ramp-up (based on historical information, common sense and available technology).

2) Start drilling down and get a list of raw features and core activities for each milestone. 

For preproduction (research phase) we allow room for exploration, namely we select few core features (technical and design breakthroughs) that we are going to develop further.

For production, based on the already made list of features and the vision that we have at this point, we create a macro Gantt chart that displays all the major activities, with resources assigned and time frame. This estimation takes into consideration the following: rough duration, rough vision and feature value in the overall economy of the project.

Then, when the actual implementation is soon to be started, PM, design and engineers start refining the implementation so that it fits the allocated budget.

In order to ensure quality, buffers are embedded either inside the activities or/and special periods are allocated during the entire process for stabilization and debugging.

Ideally, the plan should fit on a single computer screen or whiteboard, so that the PM and all team members have clear visibility on where they are and what comes next, by a single look. Also, this coarse granularity allows flexibility for later issues. And let's don't forget that, after a certain level of detail, the amount or work required to reschedule and re-plan can be quite a challenge. 

Advantages of the method:

• Visibility - great communication tool. Very easy to understand even by an outsider.
• Not very hard to develop (it is all based on honest estimates, seen from the top) - yes,      the planning itself should be seen as a separate project conducted by the project manager and have all the team and stakeholders involved. Again, keep it simple.
• Accurate in terms of feature relevance - more important features are given more time      and more resources in the total economy of the project.
• The plan is small and easy to track and change.
• Mitigates some of the pitfalls of the pure waterfall method, as it allows room for changes down the stream - features are not detailed until close to their implementation.
• More important features are put in first.
• Dependencies are easy to spot and taken into consideration.
• Resources are easily leveled and their need understood.

Possible pitfalls:

• Distribution of effort in front of the deadlines (tendency):

Rough distribution of effort over time

(Effort increases substantially as the deadline approaches) 

• The plan is long term. Even if at one point the team commits honestly to respect the schedule, this commitment erodes in time, especially after problems arise. If the process for re-planning does not involve the team again, some people will feel the plan was sloppy from the first place, some will remain committed to the previous iteration while others will feel that the project is not firmly lead - after all, new requirements are added or tasks are exceeding their allocated time. Even if everything works fine, still the commitment erodes and some will try to break the boundaries. Commitment needs to be reaffirmed from time to time.
• People may feel that there is time. Initially the requirements are fuzzy and the tendency is to underestimate them.
• People may feel that there is room for sloppiness because of the buffers (especially if      the plan is not that detailed). Project is long, we have all the time in the world, and we can afford to be less disciplined. Milestones may be treated with discontent and some may even feel that they are added there only because it's good to have milestones. Again, because of the long time, people feel that there is no real pressure in the beginning. As initial task allocations are exceeded, a sense of poor quality spreads (broken windows).
• If iterations are added to plan, because we know how many iterations will be, the first ones may be treated superficially. 
• Some creative people will invoke the time allocated for tuning and debug at the end to over saturate the project with features. They will always claim there is time for debug, to favour putting features in - (http://blog.alexandrugris.ro/2010/08/software-quality-1.html). Beside the risk at which the project is exposed, this also creates tension in the team.
• It is harder to enforce quality bars. Outside debugging periods, bugs tend to creep in (after all, why are the debug periods scheduled unless to fix bugs?!) and, because it is hard to break the time commitments advertised to all stakeholders, project management may feel that the quality is good enough for the time being and that it can be fixed later. Just the same, it is hard to drop features if they were initially incorporated into the plan and some people grew attachment to them.
• Even though we try to retrofit the design and implementation on the initial estimate, many times it shows that the original estimate it just too short to be useful.
• Feature value changes during development. 
• Historical records show that, initially, most plans have much more features scheduled      then available in the release version. As project matures naturally, the number of features decreases dramatically and focus shifts to bringing core elements to perfection. Rigid plans are an obstacle in front of this natural selection process.
• New requirements will appear and we need to plan for them, even if they are uncertain in the beginning.

Some hints to apply (which worked for me):

• Clearly state that the plan is “tentative” and that it is subject to modifications.
• Allow room for iterations
• Commit only to what you know you can respect
• Estimate risks on each element
• Use it to derive a cut list when delays occur and estimate impact of delays
• Detail as you advance through the plan
• Release often to customers. Plan for intermediate releases to be used as benchmarks against the plan.