Software Quality (3)

In my previous two posts (here and here), I've briefly touched some hot subjects:

• Quality from the user perspective and how important it is to sales. I've presented  some types of issues that hinder smooth customer experience.
• Internal quality as a requirement for sustainable product development. I've asserted that customer-perceived quality cannot be sustained for long on a shaky ground.
• We've seen a nightmare scenario that could happen if a well defined and rigorously followed quality management process is not in place. I've showed how this could lead to excessive overtime, slowed-down development, higher costs, lower morale, a bill many times paid by future development.
• In the footnotes, I've showed that quality is a focus of all true professionals and how true professionals openly discuss and defend their principles.

In this post I will discuss (self) discipline as a basis for building great products and productive environments. I will also describe some practices that the software industry at large uses to attain sustainable quality, thus increasing employee self-esteem and customer happiness.

As software applications become more and more pervasive, penetrating deeply into our lives, so is the increased concern for high reliability and excellent experience. Software market is a very mature place, with a lot of competitors striving to sell roughly similar products to increasingly demanding customers. Concerns with quality began few decades ago, when human life was put for the first time in the hands of software driven machines: think airplanes, rockets, cars, only to name a few. Since then, the talk about reliability has moved to desktop applications, servers, phones - basically, everything you can think of. Of course, the processes needed to ensure bug-free software-driven life support systems are different then those needed to put rapidly on the market a state-of-the-art entertainment application, but they all have something in common: attention to detail, discipline and commitment to excellence.

Discipline:

Maintaining quality is a long-running, tedious process, that demands strength to resist the temptation to bend the rules to finish your work faster. It is very difficult to achieve, as the pressure is high from all sides and the rationale behind sticking to processes is not always apparent and understood by everyone. Even more, the results of applying quality management principles cannot be predictably foreseen, as, if everything goes smooth, no problem arise and people may feel that they spent more work and more money on something that cannot be actually measured. Disaster strikes only if quality is not managed properly, but it strikes later, after glorious results have been shown off and, of course, then, something else can be found to blame. (I use the word "managed" but I don't refer only to managers. Indeed, they should create a framework for people to have courage to defend and encourage the quality of their work but everyone should be held accountable for his/her deeds.)

Discipline manifests in three ways:

• Sticking to tedious processes that, at times, seem only to make the development harder.
• Taking time to challenge our work and the processes even in times of crisis (Kaizen). 
• Sticking to your principles and processes even when external pressure occurs.
• Working constantly, planning and following plans throughout the entire duration of the project. Indeed, it seems very hard to focus after a difficult ending and when the deadline is two-three years ahead, when requirements will definitely change and people fear their effort will be in vain. It's difficult to explain and find the heart to undertake hard work from day one, yet this is precisely why many projects are delayed, quality is decreased and frustration accumulates. It is very tough, requires thinking ahead, a strong heart and discipline.

In the decades since software management established itself as a full fledged science, processes matured and many companies found ways to ensure enduring quality of their products. More and more, developers become aware of the impact their production discipline has and start to take pride in the sustaining quality of their work, instead of their quick hacks to a local solution. As success stories hit the headlines, we start asking ourselves how they did it.

In the next paragraphs I will discuss mostly from the programmer's perspective, but the same rules can be extrapolated to both design and art and even to project management itself.

Where does the programming time go?

Here is an astonishing fact, that every developer can confirm: each programmer spends up to 80-90% of his/her time not writing code, but trying to understand what he/she or his/her predecessors have done before. Many times he/she fails to understand all facets and introduce subtle bugs.

Digging through sources is, most of the time, a tedious and not rewarding activity. Thus, optimizing by 50% the time spend searching is something that could radically improve life of programmers and has visible results on the budget. Yet this is hard to recognize, because, after spending so much time understanding what others have done, programmers feel the pressure to quickly get out of there by hacking their way to a solution to report a success to their managers (see also The Broken Window Effect). In a culture where re-factoring is not understood, it is difficult to explain why we need the extra time. After all, the results may be even worse and more spaghetti layers added to the already overwhelming complexity. I can even say that full re-engineering of a poorly written module that has thousands of lines of code is so difficult, that we'd better not touch it. Is there an escape? I think so.

Testing and re-factoring:

Some time ago, I read a paper from Microsoft that said that there are only two effective ways to increase quality of code. One is peer code reviews and the other is permanent testing. Many procedures have been created to make room for these and, as time passes, more and more companies embrace test driven development (TDD), pair programming, scrum and other agile methodologies. Even in companies known for not applying agile principles, code reviews executed by peers or by an external audit committee are performed. The outcome of these practices is constant re-factoring. In a word, the code is not left to rot. It is constantly updated and adjusted to the latest specifications.

Is there a point to undertake massive re-factorings at once? I'd be very cautious about that. The risk of  getting a bigger mess or exchange one mess for another is extremely high. However, I am strongly advocating constant, small, incremental updates. When a programmer finds something that starts to rot, he must update it on the spot, not hack something else on top of it. This constant improvement will not change the software into something better over night, but will cultivate a sense of pride and a mentality of constant improvement, shifting the focus from hacking. After all, constant improvement is the second most important trait of real professionals after sharing knowledge with their peers. (Anyway, the two are very much linked and roughly two facets of the same trait).

Code reviews:

The vehicle to encourage re-factoring and sharing is peer code reviews. Two or more developers gather in front of a computer and the programmer just about to commit his/her sources explains to the rest what he has done and why he took the decisions he took. Then the audience suggests a number of changes and the code is committed after the they have been made.

It is very important to understand the benefits of code reviews and the programmer not to take it as he is verified. He is not. He just shares to the world his solution and spreads his knowledge. It is a time of joy, socialization and pride. He is challenged on points he might not have thought of. He is challenged to respect coding standards and, by verbalizing his solution while going once again through his work, he, many times, discovers bugs he might not easily find otherwise. (I've experienced myself code reviews and I was surprised to uncover bugs I've introduced, bugs that, otherwise, would have haunted me later when found by QA).

Code review is very powerful but, unfortunately, some people are stopped by their egos to practice it. In the professional world, such egos should not surface but, since ego is a human trait, it should be taken into consideration. Code reviews are not places for showing off nor evaluation sessions. They are fun gatherings since they basically are just a bunch of professionals discussing what they do and love the most. They also create personal bonds and disseminate knowledge.

Regarding the time investment, usually a code review lasts for something between 10 to 30 minutes, when conducted by 2-3 programmers. At an average of 15 minutes, that means 3 * 15 = 45 minutes, 3/4 man-hours. Consider that, if you find a bug in 2-3 sessions, finding it, testing it, fixing it, regressing it is, by far, more expensive, without considering the other benefits (clean-up, knowledge transfer, bonding, which are harder to measure)

Sum-up:

Up until now, we've seen how work discipline is important, what are the most difficult aspects to creating a disciplined working environment but how it positively impacts production. We've also touched re-factoring code, bringing it up to date and code reviews. In the next chapters, I will talk about other vehicles to ensure enduring quality of our products, starting from the base level - the production itself:

• Working outside the main branch
• The benefits of test driven development (two clients for the same code); white/black box testing
• Short compilation and start-up times
• Prototype-and-improve cycles
• Scrum and quality management
• A process for quality management
• Case study: radical approach: one day per week, developer self managed time
• Lean thinking

GO TO NEXT CHAPTER

Software Quality (2)

In my previous post, I talked about software quality from the user's perspective. In the final paragraph, I briefly exposed my view on the general framework to achieve such quality - the well known KISS (keep it small and simple) approach to software development.

General Process (statement):

Managing quality on the grand scale is best achieved through:

 1) start with a small set of features
 2) make sure they form a consistent core
 3) implement, adapt, test, polish, bring to perfection.
 4) incrementally add new elements; perceived quality comes first (usability, accessibility, wow moments, visual polish); experience is the product.
 5) resume from 2, redesign and re-implement as needed. change is good and healthy
 6) when the time, the budget and the quality is up-to-par, deliver
 7) offer support, talk.

The beautiful part is that quality management scales down from the whole to each of the components, to the very last line of code, design document or model. In a word, you can't cheat quality. You can't have high perceived quality on a rotten core. Eventually, it could happen for one product, but at the expense of future development and constantly increasing development costs. True professionals have a good and pragmatic understanding of quality and take great pride in applying these principles throughout their work. They stick to their guns to defend quality in all its dimensions when challenged* (see footnotes).

In my future posts, I will drill down and see how we can deliver a high quality feature, then extrapolate to the whole project in the form of a proposed process. But, before that, I will refer to a possible scenario, that can happen when quality management is not seriously considered from the early stages of product development - excessive overtime due to not meeting quality standards**.

The problem with overtime due to quality issues:

1) Faulty or no quality management procedure leads to:
       a) Product looks better on paper than in reality (everything appears as done yet the project has an unknown quality status)
       b) Lack of visibility and a clear estimate of the remaining work

2) Because of the overly optimistic view on the project status, new features creep in. In fact, it is very difficult to freeze development because no solid argument can be given. The papers look good, the project seems to be working and roughly healthy. It is this moment, when the game starts to shape-up, that the designers and artists push to add more and more features to the game, without considering very much the possible inconsistencies***. After all, we still have time to tune an polish and then tune again, right? Not quite!

3) Close to the deadline, as the team moves slowly from features to bug fixing, more and more bugs are discovered. This time usually coincides with the moment when the test team is ramped-up and the product is more and more benchmark-ed against quality standards. Slowly, the real picture creeps in.

4) A vicious circle becomes apparent: more resources are added to the end of the project, effort on integrating new people is unknown and not measured, programmers starts hacking through the code, new and reopened issues rise up, more overtime is needed to fix them, yet tired people make new mistakes and the morale goes down. Although the rate of fixing bugs can be positive and on a very good trend, the hidden code quality is decreasing fast, at the expense of future development****.

An even worse consequence can happen: at a certain point, all development is frozen or heavily slowed down for days or weeks because the build is unplayable and a large part of the team can't work.

Footnotes:


* Through civilized communication, active listening and understanding of all opinions. Professionals think in terms of benefit for the customer, cost, industry standards, future investments, return of investment. They don't think of themselves "I am a professional therefore I know what quality means". On the contrary, they are willing to challenge their current perceptions and improve them continuously. Professionalism means modesty and openness to dialogue.


** Overtime can also happen due to the desire to meet a certain market opportunity when the team willingly commits to some seemingly impossible goals, but I will not refer here to this kind of enthusiasm. Most commonly, however, it is a mixture of quality issues, excitement, not finding what is fun soon enough to fit the budget (in game industry) or other scope management issues, all to various degrees. It can be light - few extra hours, maybe a weekend or two once in a while, or it can be worse.


*** It is normal for this to happen because, now, they finally have live feedback on their creative effort. These guys are very passionate and take a lot of pride in their work.


**** As some of the main ingredients of attaining quality are professionalism, excitement and commitment from all the people, excessive overtime is an enemy of quality. While based on previous records we can estimate the needs for overtime in terms of budget buffers, it is important that:

•  managers and teams proactively try to estimate and find ways to diminish the need for long hours in the office and keep the project on track early, since the conception phase.
•  stabilization and debug periods are planned throughout the course of the project, in order to maintain the build as close as possible to "release quality" and have a good visibility on how it is doing. 
• a process is in place to ensure that quality from the user's perspective is attained on solid grounds and that the project is not rotting inside - quality is sustainable on the long run.

While quality and deadlines are the responsibility of the manager, the whole team should actively participate to meet them. After all, it is a sign of professionalism of all sides involved.


GO TO NEXT CHAPTER

Software Quality (1)

Foreword:

I start this thread to lay down some thoughts that haunted me during the past period. I would like to convince the reader that quality matters very much, that one cannot obtain customer satisfaction  unless industry-proven practices are employed throughout the entire course of the project and that failing to implement solid quality management procedures can have very bad consequences. These consequences are reflected in development costs, sales, trust and can span across multiple generations of the same product. In the end, I'd like the reader to retain the idea that focus on quality is the responsibility of all team members. Some hints will be given on how teams achieve excellence in their work, to their own benefit and that of the customer. I write these posts based on my own experiences, articles, books, discussions with friends who also work in the software industry throughout the world, and some projections I made for the future.

Customer Perceived Quality:

Every (software) project has to maintain an optimum balance between quality, cost and time. Provided that we keep cost and time fixed (which is the case for many projects) we have only one degree of freedom left. We need to understand what "quality" is, in order to know how to properly handle it.

To discuss about quality from the user perspective is to discuss about project scope and the number of issues the customers encounter (including documentation). On the other hand, quality is a general term that encompasses also: internal knowledge (code, processes, graphical assets), internal and external communication, management and the sense of self fulfillment of all stakeholders (team, all parties). On a broader sense I would like to link quality to professionalism and success, excluding the financial part* (see footnotes).

In this post I will refer to customer perceived quality:

• Project scope: arguably, but a project that has a higher number of features should provide more usage scenarios and therefore should have a higher (potential) value for the customer (personally I like simple stuff but that is another story).
• Broken features ("hard" bugs): features that don't function the way they were obviously designed to function (like crashes or a button that does nothing when pressed)
• Design errors ("soft" bugs): features that function according to design, yet they are hard to use, have uncertain value, don't really satisfy user needs, are inconsistent, etc.
• Perceived quality - degree of polish, nice touches, attention to detail, wow moments and, very important, smooth performance
• Product documentation - should be large enough to cover all aspects, yet concise and easy to read by the target audience - I will not cover documentation here, as it is a vast subject.

If the bare minimum functionality is not implemented, the project is no good. Imagine a submarine without a periscope: it's inconceivable and useless. On the other hand, a submarine crew that has fewer animations than initially planned could be considered acceptable, as long as it is consistent and the number is above a bare minimum to provide some immersion and give a clue of what the crew is doing. In this case, people have a fuzzy understanding of what the optimum number of animations is (of course, the more the better, but more providing diminishing returns) and less animations are not true barriers to functionality (personally, I would choose to have less functions but highly polished. Given the animation example, I would think very carefully if I want this feature at all if I cannot bring it close to perfection).

The key for "hard bugs" is to obtain a low reproduction rate, no game breakers and no fully broken functionality. There is a threshold that must not be passed but, a random crash once in a while (very rare!) is acceptable, as long as the load times are short and no (significant) progress is lost. Also, this kind of bugs are easily caught by the test teams, so they are usually fixed before the release. The least damaging are bugs like those in the "flickering textures far away when the light falls from a certain angle" class. They don't disturb too much, don't affect progress in any way and, as long as they happen randomly, could be considered almost no issue.

Design deficiencies are: inconsistencies, lack of usability, lack of accessibility, crappy functionality that no one understands.  Some design errors are quick hacks for a deeper (technical / concept / budget) problem. Everybody knows about them, nobody is proud of them. The second category of design deficiencies is what I call "the elitist design", which is "good core but misunderstood and poorly explained" (at large or by some less experienced players). On the opposite side, comes the "dumb-ed down symptom". Generally speaking, a large project scope fosters design inconsistencies. Some design errors are hard to spot and require extensive play / usability tests. The problem is that the team knows the product so well, that they become blind to this kind of issues. Therefore, external help is needed.

As long as the above deficiencies are kept to an acceptable level and the product has optimum performance in terms of speed and smoothness,  small enhancements like flooding inside the submarine, washed periscope lenses, provide real value and increase the perceived quality. On the other hand, easily observable visual or audio bugs, although may not deeply hinder game-play mechanics, dramatically decrease the sensation of quality. Very important, a visible and firm support policy is a certain way to improve perception from our customers.

To conclude, my feeling is that, in order to maximize the quality of a software product, the best approach is to "stay small and continuously polish". This leads to a product that has fewer but better polished features. Keeping the number of elements low maintains a higher degree of manageability and saves time for iterations.

Quality is very expensive yet, today, quality sells. Due to increased market competition, customers are having higher-than-ever quality demands and not meeting them is certainly affecting sales both on the short and on the long run - through tainted image and brand. Due to the ubiquity of the Internet, customers easily create to themselves an image of the project before purchase and this image sticks**. Web pages, reviews remain for years.

As the marketing people say, experience is the product so a lot of effort has to be put into the perceived quality: visual / audio polish, accessibility, beauty, usability, minimum workload for the user. Starting from a small core of must-haves, then incremental enhancements, gives time to perfecting the product. Late cutting results in hard-to-eliminate inconsistencies, panic and, in the end, shipping a less-than-par experience.

Footnotes:

* It could happen that a project that has a high quality standard and its customers are very happy with it is not performing well financially. However, a project that is not meeting quality standards yet is performing well financially cannot be categorized as a success. Cost is only transferred to the next generation, both in terms of perception (brand, company image) and in terms of internal quality (code, documentation, graphical assets, processes, etc...). 


** Management of expectations and of communities is more important than ever in the days of the Internet. After all, unhappy customers are the most vocal online, as they find the web as an accessible, wide-audience platform to express their frustration. Managing the product image actively through open communication channels with the customers is a way to ensure that the product is evaluated to its true level and it is not drowned in discontent.

GO TO NEXT CHAPTER