Sunday, December 5, 2010

Christmas Charity

Casuta cu copii | Promote your Page too


An initiative by two of my dearest friends: Razvan and Corina.

It is about bringing gifts (footwear and winter clothes) to poor children in an orphanage in Bucharest.

Charity really happens

Saturday, September 25, 2010

Planning

Compromise sticks and spreads

Nobody wants to write poor code or dig through it, yet, somehow, it happens. It all begins with a requirement or a prototype that needs to be developed fast in order to obtain a validation. Management and designers like it, it hits the deadline. You are congratulated for your success.  Up until now, everything is as it should be. This is how prototypes should be developed.

Then a new requirement comes. "Wait! I've only created a quick hack!" But the prototype works and you've already been congratulated and the second deadline is coming fast. You promise yourself that you are going to fix it as soon as possible, but the very day that you check-in, you and other people start adding layers after layers over it and change never happens. It could be that the feature is perfectly engineered and polished at first but, as new code is added on top of it, the original is never truly reworked, so it slowly starts to rot.

The project becomes more and more difficult to maintain and grow. Morale gets down, quality gets down, and productivity gets down, self esteem gets down, will for self improvement gets down. Some people even leave their jobs. Even if some teams actually start by cleaning up their previous mess and bring again the code to a better state, it is a onetime process, that happens only once in a few months or years. Being a one time, massive scale endeavor, the result has high chances to only end up as a different mess.

When one makes a compromise, it sticks. And then, another compromise is needed to cover the previous one. Debts start to gather in and, in the end; the debt is so high that one realizes that he/she will never ever cover it. And the process keeps on going and accumulates more and more debts, until everything becomes so expensive in terms of money and people that we need to throw everything away and start again. 

Why?

I think it all has some simple root causes:
  • People develop in isolation. They don't talk so they don't have the chance to unite for quality.
  • No code reviews.
  • No frequent informal verbal exchanges.
  • Rigid planning, commitments to feature lists instead of quality.
  • Bugs are not fixed immediately as they appear and, instead, are postponed to "debug periods", and often scheduled only at the end of the project.
  • Interruptions of any kind.

Lack of communication and rigid planning make prototyping lose its core purpose, as it is perceived as feature complete when it is merely a quick proof of concept. Developing in isolation makes it difficult for people to say "STOP, we need to change this" and management never truly gets to the bottom of problems to fix them.

Do we really need to start from scratch?

A very good friend said to me yesterday that, in nature, something has to die in order for a species to evolve. But how does this translate to software development? Does this mean that we need to start from scratch in order to evolve? Always?

I don't think so or, at least, it should not happen unless a major technological breakthrough occurs that renders everything else obsolete. I believe that software needs only to be released in order to evolve. It needs clean, frequent releases and then it needs mutations. We, programmers, call them "refactorings". I will stress this again: in order to evolve, software needs to be cleaned up before the release - no debts to the next iteration. And we need a culture that embraces change and mutations. 

Beside internal clean-up, these releases should also reach their customers (editorial teams, designers, beta testers) in order to get their feedback and adjust the feature list for the next iteration. A release without customers is tough to justify (close to pointless) and it's a pity to put in so much effort and not take the opportunity to do some user testing. After all, evolution needs feedback in order not to generate monstrosities. 

To sum up, a healthy development process should accommodate short term iterations consisting of:
  • Detailed planning based on a roughly detailed feature list
  • Actual implementation; communication among developers, peer reviews
  • Debugging and stabilization - no debts for the future, all bugs are fixed
  • Sign-off by QA
  • Release to customers for feedback
  • Communicate status, negotiate features and deadlines, incorporate feedback, and discuss how we can improve the process so that the next iteration gets better.
Managers don't manage people

Another thing I realized, no matter how shocking it sounds, is that managers don't manage people. They manage processes. People buy in and start adding value or they don't. You can assign tasks but you can't really force anyone to complete them. Therefore, the notion of "motivating an employee" makes no sense, really.  He/she is either motivated or not. What you can do, however, is create an environment that appeals to the employees, so they get excited about their job and become productive. And this is the really tricky part. 

Quality from the user's perspective

Indeed, the only thing that matters to sales is how the product is perceived by its customer. Nothing else. Code doesn't matter, as it is invisible. Code only matters to the production people. If it is crappy, it ruins the life of the men and women involved in its production. It turns them away from the product, productivity drops, conflict sets in, and motivation goes away. Yet this does not impact sales directly; it impacts everything else. 

Getting back to process - Planning

I'll make a small detour from the quality issues to examine planning, as the core activity that drives project to completion. In the following paragraphs, I will present the normal planning method with its advantages and disadvantages and how it can be applied. After understanding planning, I will get back and show how to transition to methods that are more appropriate to ensuring quality in software.

Everything in a project revolves around plans and planning. Planning is a continuous process, plans change as requirements change, conditions change and understanding grows. Visibility and vision is based on plans. Communication with stakeholders is based on the same plans; resources are requested just the same. Commitments are based on plans. Monitoring and controlling is based on plans. Strategies are based on plans. Planning and plans - everywhere. Yet some managers either don't take planning seriously or they don't iterate on already made plans to adjust them. Why? 

The answer is quite simple and straight forward (order of bullets is irrelevant):
  • In the early stages of the project making informed plans is really hard and it seems that we can do better off without them. After all, it is easy to get people to do something tangible now.
  • Some engineers and designers are reluctant to planning as well (especially if they are not used with the process) and managers don't want to deal with this kind of issues up front.
  • In the late stages of the project we are so busy putting up fires, escalating issues, requesting resources, fixing and dispatching bugs that we can't stop to plan. 
  • Superficial planning is hard to spot therefore allow your plans to be challenged by the stakeholders and the team.
  • Planning requires going back to the same document, ripping it off and doing it again many times. It's difficult to change everything when you worked so hard on it. Over and over again.
  • Some feel that planning is a waste of time: things seem to work without plans at first.
  • Some managers (former star employees) are so used to actively solve issues that have      visible impact on the project that, when they are required to stop and think about the future, they feel it is useless. Why? Because to them plans seem to be only coloured excel sheets, without a tangible impact on the outcome of their project.
  • Planning is a costly activity and some organizations or teams may naively try to eliminate it as waste. Planning is costly because it involves not only the manager, but the whole team or, at least, the most senior part of the team.
  • If not properly communicated, a plan means a commitment which seems hard to make and changed later.

How to plan? 


For starters, I will assume that we already have some visibility on the sales volume, on the budget and on the release date. Let's assume the mandate sounds something like: "We need a project that will generate roughly X units sold and that will be released around the Y date. It should cost somewhere around Z monetary units." I will assume that there is historical information in the company about this kind of project and that the constraints are reasonable. Also, probably most important of all, we have some good visibility on the project deliverables and core features - that means that we are well advanced through Step 1 (Project Goals) and Step 2 (Project Deliverables) described in the link above.

Top-down approach - project schedule:

1) We know the release date and the budget, therefore we can establish quite easily the major milestones and the resource ramp-up (based on historical information, common sense and available technology).

2) Start drilling down and get a list of raw features and core activities for each milestone. 

For preproduction (research phase) we allow room for exploration, namely we select few core features (technical and design breakthroughs) that we are going to develop further.
For production, based on the already made list of features and the vision that we have at this point, we create a macro Gantt chart that displays all the major activities, with resources assigned and time frame. This estimation takes into consideration the following: rough duration, rough vision and feature value in the overall economy of the project.
Then, when the actual implementation is soon to be started, PM, design and engineers start refining the implementation so that it fits the allocated budget.
In order to ensure quality, buffers are embedded either inside the activities or/and special periods are allocated during the entire process for stabilization and debugging.
Ideally, the plan should fit on a single computer screen or whiteboard, so that the PM and all team members have clear visibility on where they are and what comes next, by a single look. Also, this coarse granularity allows flexibility for later issues. And let's don't forget that, after a certain level of detail, the amount or work required to reschedule and re-plan can be quite a challenge. 

Advantages of the method:

  • Visibility - great communication tool. Very easy to understand even by an outsider.
  • Not very hard to develop (it is all based on honest estimates, seen from the top) - yes,      the planning itself should be seen as a separate project conducted by the project manager and have all the team and stakeholders involved. Again, keep it simple.
  • Accurate in terms of feature relevance - more important features are given more time      and more resources in the total economy of the project.
  • The plan is small and easy to track and change.
  • Mitigates some of the pitfalls of the pure waterfall method, as it allows room for changes down the stream - features are not detailed until close to their implementation.
  • More important features are put in first.
  • Dependencies are easy to spot and taken into consideration.
  • Resources are easily leveled and their need understood.

Possible pitfalls:

  • Distribution of effort in front of the deadlines (tendency):

Rough distribution of effort over time
(Effort increases substantially as the deadline approaches) 

  • The plan is long term. Even if at one point the team commits honestly to respect the schedule, this commitment erodes in time, especially after problems arise. If the process for re-planning does not involve the team again, some people will feel the plan was sloppy from the first place, some will remain committed to the previous iteration while others will feel that the project is not firmly lead - after all, new requirements are added or tasks are exceeding their allocated time. Even if everything works fine, still the commitment erodes and some will try to break the boundaries. Commitment needs to be reaffirmed from time to time.
  • People may feel that there is time. Initially the requirements are fuzzy and the tendency is to underestimate them.
  • People may feel that there is room for sloppiness because of the buffers (especially if      the plan is not that detailed). Project is long, we have all the time in the world, and we can afford to be less disciplined. Milestones may be treated with discontent and some may even feel that they are added there only because it's good to have milestones. Again, because of the long time, people feel that there is no real pressure in the beginning. As initial task allocations are exceeded, a sense of poor quality spreads (broken windows).
  • If iterations are added to plan, because we know how many iterations will be, the first ones may be treated superficially. 
  • Some creative people will invoke the time allocated for tuning and debug at the end to over saturate the project with features. They will always claim there is time for debug, to favour putting features in - (http://blog.alexandrugris.ro/2010/08/software-quality-1.html). Beside the risk at which the project is exposed, this also creates tension in the team.
  • It is harder to enforce quality bars. Outside debugging periods, bugs tend to creep in (after all, why are the debug periods scheduled unless to fix bugs?!) and, because it is hard to break the time commitments advertised to all stakeholders, project management may feel that the quality is good enough for the time being and that it can be fixed later. Just the same, it is hard to drop features if they were initially incorporated into the plan and some people grew attachment to them.
  • Even though we try to retrofit the design and implementation on the initial estimate, many times it shows that the original estimate it just too short to be useful.
  • Feature value changes during development. 
  • Historical records show that, initially, most plans have much more features scheduled      then available in the release version. As project matures naturally, the number of features decreases dramatically and focus shifts to bringing core elements to perfection. Rigid plans are an obstacle in front of this natural selection process.
  • New requirements will appear and we need to plan for them, even if they are uncertain in the beginning.


Some hints to apply (which worked for me):
  • Clearly state that the plan is “tentative” and that it is subject to modifications.
  • Allow room for iterations
  • Commit only to what you know you can respect
  • Estimate risks on each element
  • Use it to derive a cut list when delays occur and estimate impact of delays
  • Detail as you advance through the plan
  • Release often to customers. Plan for intermediate releases to be used as benchmarks against the plan.


Saturday, August 28, 2010

Software Quality (4)

In my previous post, I started a quick pass through the most common methods for ensuring quality at the lowest level: the day to day production activities. My assumption is that, if we are able to maintain a high level of quality throughout the entire development process, we will end up with a better, easier to maintain product, sustainable over several generations, and, as a side effect, a happier team.

We need to find cost-effective, lightweight processes, that maximize ROI. These processes should, ideally, not increase the number of collateral activities required, but, instead, should provide some easy to implement practices, acknowledged by everyone as valuable. Only by acquiring consensus, coupled with self discipline, they will be maintained when the pressure sets in. I am talking about a way of working that  should ensure:

  • Better understanding of the quality level; clear visibility for everyone of what has been correctly done and what are the current issues
  • Predictable results
  • Increased return of investment (see footnote 1) 

A healthy development process should also have positive side effects:

  • Shorter develop-prototype-test cycles (2)
  • Better spreading of knowledge throughout the team
  • Increased ownership and pride
  • Friendlier environment
  • A better distribution of effort throughout the entire length of the project

Some practices are team-wise, while others are linked to self management and can be employed even without a team-spread acknowledged process (3).

Working outside the product code - a workbench for rapid iterations:

Few things are as annoying as frequent interruptions or uncovering poor quality work that you have to deal with - like digging through messy code that has side-effects. These usually happen in a large code base, that has high coupling between components or that starts very slow. These two symptoms work hand in hand and, I think, are the biggest two impediments to constant re-factoring and team morale (starting slow leads to fewer runs which means less testing and fewer recompilations, which, in turn, means that programmers won't spend time improving existing code and will only add new layers on top of it). On the other hand, small applications have a very short modify-run-test cycle, that allows them to mature quickly and with fewer bugs. However, this advantage is usually lost in time, as the code and data gets bigger and bigger.

Is it possible to maintain the advantages of extremely short iterations over time? One solution that can often be applied is to isolate development from the main product, something that can be done easier especially when coding a new feature from scratch. The main goal would be to eliminate as much as possible the interruptions, by initially constructing a sandbox that (9):

  • Compiles and starts fast
  • Allows to write dirty modifications and test them incrementally
  • Allows to re-factor constantly what has been written, without impacting the rest of the code
  • Allows to focus only on new code, without worrying about other changes

Such a sandbox should allow very short prototype-compile-start-test-modify cycles. Even one at 2-3 lines of code modified - priceless! A programmer could hack test cases directly into the dev-code and run them tens of times a day to see the effects of his latest additions. He could easily re-factor everything until he is fully satisfied with the results (4):

  • Code that gets fast prototype-improved cycles matures faster.
  • When integrated, it has very few bugs because it was extensively tested and went through. 
  • Keeping the development sandbox in sync with future additions permits to quickly check future bugs and ensure consistency in time.
  • The amount of dependencies to existing system is kept to a minimum. The code becomes self-contained, reusable.
  • Having more than one client for your code (5) requires that boundaries are well cut an clear. This helps maintain the code clean, as artificial dependencies cannot be created. 

Sometimes it is not possible to work outside the main product. Many times, the code has been developed directly inside the main branch and, therefore, the amount of dependencies to other areas is so big that they cannot be decoupled. In the rush of development, it is very easy to create artificial links and not maintain clean cuts between modules, just because there is no system in place to enforce these boundaries. After all, it compiles! (6)

When developing outside the main branch, this kind of borders are naturally enforced by the development system, thus they cannot be crossed easily. However, if the development sandbox is not kept in sync with future additions and boundaries are not enforced further, dependencies will start to spread and the system will start to decay (7).

Test Oriented Development

In my previous paragraphs, I basically described an approach that resembles somehow to test driven development (TDD). It allows programmers to develop their code in separate applications (or test cases), iterate fast to get results fast and then integrate when ready. If these test cases are run and maintained throughout development, the benefits could be even higher as they require that the code is written in a modular fashion, they serve as test-bed for new features and bug-hunting and, equally important, they can be used as an entry point to understand functionality. 

I will not get into more details regarding TDD, as I have not practiced it first-hand beyond what I've described above, but I think that great attention should be put into integrating automated testing during the production phase and, equally important, to find a framework and a state of mind within the team that fosters quality at this level, even in times of great pressure. Sometimes it may seem like a chore that tests need to be maintained as well but, at least from my perspective, it is easier and cheaper to maintain tests or develop functionality outside the main product then to dive into unknown, hacky code, that has side effects and unexpected links to other obscure areas. 

Some may argue that this kind of test oriented development is not suited for all sorts of tasks and I agree. From what I've seen, however, I believe that greenfield development can be isolated from the main branch at least in its initial phases (be it UI or network or anything else). And, although the initial phases have passed, why not try to keep this isolation further and also maintain the sandbox in a working state? It can be used as a priceless testbed for future development and regression testing.

Other disciplines need fewer interruptions and rapid prototyping as well:

In order to spread the benefits of rapid iterations, the team should foster the development of such tools that require that the designers, programmers, artists restart the game (or the product at large) as few times as possible as fast as possible. The best development tool for a designer or for an artist is that one that allows him to see his work directly inside the engine, live, as he performs his changes. This would allow him to prototype as much as possible, ideally continuously.  It is essentially the same need that programmers have regarding their code. Example of such tools include editors that have accelerated, game-identical, simulation capabilities, live connections to 3DS Max, tuning tools that connect to the running game, live editors. The shorter the produce-export-test cycle is, the more time is left for iterations and more time the team has to creatively play and improve the product.

On the other hand, a lot of bugs appear because, during frequent changes, something is lost on the way. Very useful are the tools that automatically validate integrity before the asset gets into the game. The more checks the better and, therefore, such tools should be developed and spread to other teams as well, reducing implementation cost through sharing of technology and knowledge.

Similar processes to TDD should be implemented in the art and design departments, to ensure fewer bugs at release time, less overtime and a clearer picture of where the project stands. Activities that support high quality also help spread the knowledge and reuse of existing and well tested modules. Code / asset reviews, tools that verify content, tests, using 3rd party APIs, creating sand-boxes, may seem, sometimes, as to provide an additional burden, but in the end, if applied wisely, should decrease effort, decrease costs and unleash creativity.

Sum-up

The more iterations the better the product has the chances to be. The lowest level at which a developer can iterate is the source code or asset production. When people iterate very fast, they can try new things, have less fear of failure and their work product is better tested and crafted. Even more, the productivity loss and frustration associated with going in and out of flow is diminished. Coupled with discipline and peer reviews, constant iteration and continuous testing is key to attaining high quality, both during day-to-day activities and, at a higher level, feature and product-wise.

Implementation

Although sometimes difficult to see from the developer's seat, a healthy development process has the tendency to accumulate value as it rolls on. I assimilate positive practices to a snow-ball: they are difficult to establish in the first place but they grow in value faster and faster as the team acquires more and more success stories (8).

Some practices will not be seen as valuable by all the team from the very beginning. As such, they need to be implemented in less stressful times, when people are more prepared to try new things. Their value is usually seen in time, after they have become established methods, adapted to each organization. Feedback from the team is very important, as it removes implementation barriers and increases commitment to respecting the new ways. I think that, when implementing such changes in methodology, to understand that there isn't a-size-that-fits-everyone solution. Therefore, management and people should invest time and thought into finding the right way of adapting industry proven practices to their line of work, through collaboration, based on common goals. (Our Iceberg Is Melting)

In the following post I will depart from the code level perspective, and reach into some  higher level management frameworks:
  • Scrum and quality management (two previous posts here)
  • A process for quality management, that includes test driven development, white-box testing and black-box testing (mixing developers with testers and making testers part of the development team).
  • Case study: radical approach: one day per week, developer self managed time
  • Lean thinking
  • The importance of having all the team experience the product first hand, as its first beneficiaries (play the game!)

Footnotes:

(1) The production cost should decrease, both on the short run and on the lung run, by diminishing the need for suffocating debug periods right before the deadlines and turnover. Even more, remember that 80% of programming time is lost into digging and trying to understand what their predecessors have done. Decreasing this 80% should definitely reduce project costs or allow for more improvements.

(2) Actually, a well implemented quality strategy should have the side effect of prototype-ability. That is, a product that allows new changes to be integrated, tested and removed quickly, without major side-effects. 

To achieve that, the number of extra activities that are needed to prepare the prototype should be decreased to a minimum. That can be done only if the code is well understood by everyone: concise, clean and modular. I will not get into the details of what clean code is, but I'd like to recommend two books on the matter: one is "The Pragmatic Programmer" and the other one is "Clean Code". Why clean code? Because no stakeholder is interested in productivity drops or having a product that has sluggish performance or frequent crashes.

(3)  One practice I am not talking here about is pair programming.  Some of the most rewarding moments I had in my professional life (and also moments of intense knowledge transfer) are related to pair programming experiences I had with more experienced engineers. I believe it should be encouraged as a good practice inside a team - formally or informally.

(4) Just to give some examples I've worked on, that used this kind of approach: 

The character animation system from SH4 (together with a co-worker and friend): developed as a DLL loaded by a basic engine viewer. Start-up times? Few seconds. We could easily export animations from 3DS Max to the engine and test our system tens of times a day. 

The pathfinding algorithm from SH4 AddOn: I've started hacking a C# GUI app that allowed me to visually check paths on a simple world map generated from game-data, without having to start the game. Again, compilation and start-up times? Seconds.

The AIFramework dll from SH5: hacked as a Python prototype, then coded it in C++ as a dll, loaded by a console application that hosted the test-cases. Restart time? Again seconds.

(5) At least, you have the main product and the application that was used to develop that code.  

(6) Thus, even for simple changes like tuning a hard-coded parameter, the programmer suffers a compilation and a full restart, which is a very lengthy process. Sometimes, the boundaries are so poorly cut that even a slight change could trigger a major recompilation. When this cases occur, the only solution is to locally re-factor every time a problem is encountered, thus slowly improving code over time. 

(7) Adding scripting support to a project, for instance, could, potentially, bring this kind of benefit as well. Since the boundary is naturally enforced by the compiler, the developer is forced to pay more attention to how he organizes his code.  Also, well implemented scripting allows code modules to be developed and tested in an already started application, reducing the compile-start-test-modify overhead.

(8) The not-invented-here syndrome should be avoided at any cost. Good tools are expensive to build and they may not be justified financially if developed for a single project. However, the more sharing goes on and the more projects benefit from valuable technologies and practices, the ratio between initial investment and its returns becomes more and more favorable. Initial development cost is covered by multiple teams and, as sharing is increased, technologies  converge and become more and more useful and easier to use. Again, the snowball effect.

(9) This is not conflicting with the continuous integration principle.The idea is to build a sandbox that allows fast prototyping and short modification times for (new) features, integrate when the feature is almost complete and then keep this sandbox in sync with next changes (an example: a new data import / export pipeline should be developed separately while the previous system continues to work and commit when ready and well tested). It does not mean working on an old branch or separated from the rest of the team. By contrary, changes should be presented continuously and designers and testers asked to validate functionality on a frequent basis.