The Solutions Team

Software Is Not Written in Factories

The concept of Quality Control works well in automated environments.  QC typically tests products at the end of the factory line:  Is the underwear white?  Does it have any holes?  Is there any bacteria in the tomato paste?

Most software companies follow the same model for quality assurance, to varying degrees of success.  However, the software process doesn't always fit the factory-line model.   Many individuals participate in software production.  The process of creating software product is mysterious; it is neither automated nor repeatable as it relies on skill, instinct, creativity, and personality.  Therefore it's not always correct to assume that what works in the factory will work in software.  This is especially true when it comes to the quality of software.

Glossary

The terms used in this document don't apply to any real software company.  Like most immature markets, the software business has very few well-defined terms, and no aspect of the industry better exemplifies this observation than the structure of software organizations.

Management:  Responsible for organizing the company and defining products.

Development:  Responsible for creating products, adding features, and fixing bugs.

QA:  Stands for Quality Assurance.  Responsible for finding bugs and making sure they get fixed and stay fixed.

Product Delivery:  Responsible for delivering product to customers.  Creates "setup" programs, burns CDs, places files on the web.  The product delivery process  also requires QA.

Quality:  The degree to which a product performs as advertised, without flaws and without unnecessary complications.

Quality Missions Vary

Software companies approach quality differently.  Some QA engineers deserve the title as "engineer," developing test suites against low-level interfaces (e.g., in C++) - this is called "white box" testing.  Other QA engineers perform "black box" testing, using the product mainly from its (usually GUI) user interface.  White box and black box testing both have their place.  However, independent QA groups are chronically ineffective at performing either task.

Regardless of how an organization approaches quality, when a "QA" group exists, the same patterns almost always occur.  I will discuss two relevant quality scenarios.

Scenario 1:  QA and Development are Buddies

In this scenario, QA tries its best to stay out the way of development.  QA's mission is to be helpful, assisting programmers with design reviews and promising automated test suites.  QA appears to be doing something but nobody knows exactly what, especially the developers.

Product is produced very quickly, but with generally low quality.  QA is reluctant to enter bugs for fear of hurting feelings and being in the way.  Implemented features are buggy and hard to use.  Promised functionality fails to materialize because QA isn't aware of the requirements, doesn't care about them, or doesn't have the backbone to report missing features for the same reason they don't report bugs.   Glaring feature omissions and shoddy user interface go unnoticed.  Automated test programs fall behind schedule because QA is overloaded with all of the new product features.

There are many proactive ways in which QA can make this scenario work.  But I have rarely seen them implemented.  Proactiveness really comes down to individual, rather than organizational, effort.  If you're lucky, someone within QA will step up and build test suites.  If you're not so lucky, you may have to hire contractors to do it.  It really comes down to incentives:  if QA and development are buddies, what incentive does QA have to report bugs?  What incentive does QA have to participate in the process?   Incentives must be defined by management.  Without real incentives, QA has no purpose, and most QA members, as humans, would rather punch a clock than take on risks that could mean lack of sleep, loss of time with family, etc.

This scenario is a management failure.   It does very little service to either QA or development.  QA becomes a token entity, providing little value above appearing on an organizational chart to reassure prospects and stockholders. 

Scenario 2:  QA vs. Development

In this scenario, QA loves to find bugs.  There may be a bonus paid for each bug found.  There may even be a penalty (applied to QA or development) for each bug found by the customer after shipping.  In this scenario there are usually a very large number of reported bugs.  Many bugs are superficial in order to meet the quota.   QA takes on a management role, specifying how the product is supposed work in the bug reports.

This scenario breeds contempt between QA and development.  Developers are reluctant to fix bugs because there are so many low quality bug reports.  Much time is wasted arguing about reported bugs are really feature requests.  Managers spend a significant amount of time prioritizing bugs.  Many obvious bugs - reported and unreported - go unfixed because the entire system (including QA and development) can't keep up with the ever-growing bug database.

Total Quality Without QA, or The Foxes Run the Hen House

The above scenarios are extreme examples of what can happen within a software organization.  In the real world, these scenarios occur simultaneously and in varying degrees.   Many other scenarios exist, even some positive ones.   But perhaps, as mentioned earlier, the very concept of QA doesn't fit the software business. At least, it doesn't fit today's immature software industry.

Today's software developers resemble the craftsmen¹ who lived before the industrial revolution.  Developers directly affect the quality of their work.  If developers care deeply about quality, they can create almost flawless works.  On the other hand, they can create classic pieces of crap.  Of course, quality has to be balanced with time constraints.  A "perfect" dialog box isn't worth waiting six months for.

If the responsibility for delivering quality falls onto the shoulders on a separate group, then without fail, quality suffers.  Within development, there always exists that one programmer who consistently throws code "over the wall" to QA without even testing it first.  QA files bugs against the code, and the bugs eventually get fixed.  However, the engineer doesn't have any incentive to change his sloppy work habits.

Therefore, developers should be responsible for ensuring quality.  Developers should write their own test suites that will find bugs early and detect regressions. 

If quality is a real responsibility, however, the organization needs to ensure that the responsibility is carried out.  Quality must be measurable, but how?

Managers Hone The User Interface

Managers are responsible the product's external interface.  Indeed, there's never enough time to specify everything, so often engineers must design user interfaces, including dialog boxes, windows, and drag and drop behavior.  Engineers are generally poor user interface designers due to time constraints.  Therefore managers should be responsible for ensuring the quality of the user interface, and making sure that poor interfaces get fixed.

Managers and Developers Share the Responsibility

Testing involves creativity.  It is not always obvious to tell whether a user interface has been tested completely.  Testing takes both brains and a lot of time.   Therefore managers generally won't have the bandwidth to test all aspects of the user interface.  Detailed testing must be performed by developers and possibly another group that reports to either management or development.  If the testers report to development, they are on the "development track," during which they learn to become developers.  On the other hand, testing can be excellent training for future product managers.  It's not unreasonable to have testers reporting to either group depending on individual preference - indeed, some organizations rotate developers to testing so that engineers learn realistic and lasting lessons about quality. 

The development and management tracks are in sharp contrast to most QA groups that ignore career paths and resent "losing" good engineers.  This sort of competition between QA and rest of the company is counter-productive and promotes a stagnant and unchallenging work environment for QA engineers.  Testers can even be contractors, although contracting out QA work is not advisable since product knowledge is a valuable company asset.  Furthermore, contract testers rarely receive adequate incentives to be truly effective.

When development and management share the responsibility for ensuring quality, there's no need for an autonomous QA group.   For example, Andersen Consulting implements  large-scale software systems for large corporations around the globe, all without a separate QA group.  The quality burden is shared among junior programmers and senior managers.  In fact, junior programmers receive more training on testing their code than writing it.

Low-level testing is essential for software quality, but quality must also be evaluated at a higher level.  The Solutions Team carries out this task.

The Solutions Team

The Solutions Team is responsible for applying the company's internal software to relevant, preferably mission critical projects.  These projects must reflect the needs of the company's customer profile.  The Solutions Team reports to management and has concrete deadlines. 

Few software organizations have a Solutions Team.   For that reason, I took the liberty of creating a new name.  I don't know of an actual name for a group like this, although training and sales groups commonly build "samples" using the company's technology.  However, the samples rarely get the funding and development attention they require to be successful.

I have heard much lip-service to the so-called Solutions Team over the years, primarily voiced by management.  However, resources are rarely allocated towards getting a Solutions Team off the ground.  The Solutions Team can be easily confused with QA, but although QA and the Solutions Team may overlap, the Solutions Team is under the gun to deliver something tangible.  Therefore it's ineffective to hide the Solutions Team inside the QA group.  The Solutions Team should be its own autonomous unit that reports to the CEO, CTO, or VP of Engineering.

The Solutions Team pushes development to make their dates.  If development slips, so does the Solutions Team, and both groups suffer in terms of recognition and perhaps bonuses.  The team simultaneously verifies product features.   Key missing features can't go unnoticed.  Obvious issues with design, usability, and quality quickly bubble to the top.  The Solutions Team makes sure that bugs get reported, get fixed, and stay fixed.  It's in the developers'' best interests to make sure they deliver quality, because ultimately the CEO will find out if they don't.  For this reason developers will be encouraged to write whatever tools they need to ensure quality, such as  test drivers and automated regression test suites.

In most software companies, the beta program partners represent the Solutions Team.   However, beta partners are significantly less effective than an internal Solutions Team because they don't have a fixed loyalty to the software provider.  Beta responses are often sparse and shallow rather than broad and deep.  Beta partners are still necessary, however, especially for testing the product delivery system.

Conclusion

The software business isn't yet ready for QA.  Developers, should be responsible for the quality of their work, which includes writing bug-free code, implementing complete features, and designing easy-to-use interfaces.  It's almost impossible to quantify the value of QA to an organization - their value depends more on individual effort than a proven methodology.  Instead of trying to find that methodology or hiring great QA engineers, software businesses should refocus their QA groups on solving real-world problems with the company's products.

¹Of course, craftsmen can be replaced by "craftspeople."