Software Quality Assurance Fundamentals

CONTENT

CHAPTER 1
1. Organizing for Quality Management 1
1.1 The Quality Management Framework 1
1.1.1 Object (Entity) 2
1.1.2 Product 3
1.1.3 Process 3
1.1.4 Requirement 3
1.1.5 User 4
1.1.6 Evaluation 5
1.1.7 Measure and Measurement 5
1.1.8 Quality 6
1.2 Quality Program Concepts 8
1.2.1 Elements of a Quality Program 8
1.2.2 Considerations 15
1.3 Organizational Aspects of the Quality Program 17
1.4 Quality Program Organizational Relationships 17
1.4.1 Establish Requirements and Control Changes 18
1.4.2 Establish and Implement Methods 20
1.4.3 Evaluate Process and Product Quality 21
1.5 Mapping Quality Program Functions to Project Organizational Entities 22
1.5.1 Planning 23
1.5.2 Establish Requirements and Control Changes 24
1.5.3 Establish and Implement Methods 25
1.5.4 Evaluate Process and Product Quality 27
1.6 Example Organizational Implementations of a Quality Program 27
1.6.1 Project Engineering Process Group 28
1.6.2 Quality Program Structures in Large Projects 28
1.6.3 Quality Program Structures for Small Projects in Large
Organizations 31
1.6.4 Quality Program Structures in Small Organizations with
Small Projects 31

CHAPTER 2
2. Software Quality Lessons Learned from the Quality Experts 35
2.1 Introduction 35
2.2 Kaoru Ishikawa 37
2.3 Joseph M. Juran 39
2.4 Yoji Akao 43
2.5 W. Edwards Deming 44
2.6 Genichi Taguchi 49
2.7 Shigeo Shingo 51
2.8 Philip Crosby 52
2.9 Watts S. Humphrey 56
2.10 Conclusion 60
References 60

CHAPTER 3
3. Commercial and Governmental Standards for Use in Software Quality Assurance 63
3.1 SQA in ISO Standards 63
3.1.1 ISO 9000:2005 and ISO 9001:2000 64
3.1.2 ISO/IEC 90003 64
3.1.3 ISO/IEC 2500n—ISO/IEC 2504n (SQuaRE) 65
3.1.4 ISO/IEC 14598 and ISO/IEC 15504 66
3.1.5 ISO/IEC 9126 67
3.1.6 The Special Role of ISO/IEC 12207 68
3.2 SQA in IEEE Standards 69
3.2.1 IEEE Std 730-2002 69
3.2.2 IEEE Std 829-1998 70
3.2.3 IEEE Std 1028-1997 70
3.2.4 The Special Role of IEEE/EIA 12207 71
3.3 SQA in COBIT® 72
3.4 SQA in ITIL® 74
3.4.1 ISO/IEC 20000 76
3.5 SQA and Other Standards 77
3.5.1 ANSI/EIA-748-A-1998 77
3.5.2 RTCA/DO-178B 79
3.6 Whatever Happened to U.S. Department of Defense Standards? 80
3.6.1 Influential Past Standards 80
3.6.2 SQA in Active DoD Standards 82
3.7 Reminders About Conformance and Certification 83
3.7.1 Conformance 83
3.7.2 Conformance to an Inactive Standard 83
3.7.3 Certification 83
3.8 Future Trends 84
3.8.1 Demand for Personal Credentials Will Increase 84
3.8.2 Systems Engineering and Software Engineering Standards
Will Converge 85

CHAPTER 4
4. Personnel Requirements to Make Software Quality Assurance Work 89
4.1 Introduction 89
4.2 Facing the Challenge 90
4.3 Organization Structure 92
4.4 Identifying Software Quality Assurance Personnel Needs 94
4.5 Characteristics of a Good SQA Engineer 97
4.6 Training the Hardware QA Engineer 99
4.7 Training the Software Engineer 99
4.8 Rotating Software Engineers 101
4.9 New College Graduates 102
4.10 SQA Employment Requisitions 103
4.11 What to Expect from Your SQA Engineering Staff 104
4.12 Developing Career Paths 106
4.13 Recommendations 106

CHAPTER 5
5. Training for Quality Management 111
5.1 Introduction 111
5.2 Context for a Quality Evaluation Training Program 111
5.2.1 Quality Evaluation to Quality Assurance 111
5.2.2 Audience for Quality Evaluation Training 112
5.2.3 Organizational Training Program 112
5.2.4 Needed Skills and Knowledge 113
5.3 Two Examples 116
5.3.1 Evaluation of Adherence to Process (PPQA) 116
5.3.2 Evaluation of Product Quality 118
5.4 Summary 119
Reference 119

CHAPTER 6
6. The Pareto Principle Applied to Software Quality Assurance 121
6.1 Introduction 121
6.2 WWMCCS—Classic Example 1 123
Contents ix6.2.1 Manpower 123
6.2.2 Cost of Contracts 123
6.2.3 By Release 125
6.2.4 By Function 125
6.3 Federal Reserve Bank—Classic Example 2 127
6.4 Defect Identification 132
6.4.1 Rubey’s Defect Data 133
6.4.2 TRW Defect Data 135
6.4.3 Xerox Defect Data 138
6.5 Inspection 140
6.6 Pareto Charts Comparison 143
6.7 Conclusions 145
References 146

CHAPTER 7
7. Inspection as an Up-Front Quality Technique 149
7.1 Origin and Evolution 149
7.2 Context of Use 150
7.3 Scope 150
7.3.1 Software Inspections and Walkthroughs Distinguished 151
7.4 Elements 152
7.4.1 Structured Review Process 153
7.4.2 System of Checklists 156
7.4.3 Rules of Construction 161
7.4.4 Multiple Views 162
7.4.5 Defined Roles of Participants 162
7.4.6 Forms and Reports 164
7.5 Preparation for Expert Use 167
7.6 Measurements 168
7.6.1 National Software Quality Experiment 168
7.6.2 Common Problems Revealed 168
7.6.3 Inspection Lab Operations 169
7.6.4 Defect Type Ranking 169
7.6.5 Return on Investment 170
7.7 Transition from Cost to Quality 171
7.8 Software Inspections Roll Out 173
7.9 Future Directions 175
7.10 Conclusion 177
References 177

CHAPTER 8
8. Software Audit Methods 179
8.1 Introduction 179
8.2 Types of Software Audits 181
8.2.1 Software Piracy Audit 181
8.2.2 Security Audit 183
8.2.3 Information Systems Audit 185
x Contents8.2.4 ISO 9001:2000 Software Audit 187
8.2.5 CMMI®-DEV Appraisal 190
8.2.6 Project Audits (Internal CMMI®-DEV/ISO 9001:2000 Audits) 193
8.2.7 Automated Audits 195
8.3 Preparation for a Software Audit 197
8.4 Performing the Audit 201
8.5 Results and Ramifications 204
8.6 Conclusions 207
References 208

CHAPTER 9
9. Software Safety and Its Relation to Software Quality Assurance 211
9.1 Introduction 211
9.2 Software-Caused Accidents 212
9.3 The Confusing World of Software Safety 212
9.4 Standards, Guidelines, and Certifications 213
9.5 What Does It Take to Develop a Software Safety Assurance Program? 215
9.6 Requirements Drive Safety 217
9.7 Design of a System Safety Program 221
9.8 Hazard Avoidance and Mitigation Technique 223
9.9 Recommendations 223
References 225

CHAPTER 10
10. American Society for Quality’s Software Quality Engineer Certification
Program 227
10.1 ASQ Background 227
10.2 ASQ Certification Program 228
10.2.1 What Is Certification? 228
10.2.2 Why Become Certified? 230
10.2.3 What Is a Certified Software Quality Engineer (CSQE)? 230
10.2.4 What Qualifications Are Necessary to Become a CSQE? 231
10.2.5 How Many People Have Earned Their CSQE? And Who
Are They? 231
10.2.6 Is There Value in the CSQE Certification? 232
10.3 How Is a Certification Exam Developed? 232
10.3.1 Proposal for New Certification 232
10.3.2 Job Analysis 234
10.3.3 Certification Approval 235
10.3.4 Creating the Examination 235
10.3.5 Cut Score Study 236
10.3.6 Examination Administration 236
10.3.7 Sustaining the Examination 237
10.4 How Should You Prepare for the Exam? 237
10.4.1 Apply for the Examination Early 238
10.4.2 What Reference Materials Can Be Used During the Exam? 238
10.4.3 In What Languages Is the Exam Offered? 238
Contents xi10.5 What Is in the Body of Knowledge? 238
10.5.1 Six Levels of Cognition Based on Bloom’s Taxonomy (1956) 250
10.5.2 Sample Questions 250
10.6 Recertification 253

CHAPTER 11
11. CMMI® PPQA Relationship to SQA 257
11.1 Software Quality Engineering/Management 257
11.1.1 Software Quality Engineering/Management Functions 257
11.2 Software Engineering Institute’s CMMI® 259
11.3 PPQA in the CMMI® 262
11.3.1 Process and Product Quality Assurance Purpose Statement 263
11.3.2 Quality Control 263
11.3.3 Quality Assurance 264
11.3.4 Project Quality Plan 264
11.3.5 PPQA as Defined by the CMMI® Specific Goals and Specific
Practices 265
11.3.6 Institutionalization 269
11.3.7 Quality Assurance Representatives 270
11.3.8 What Is the Relationship Between PPQA as Defined in the
CMMI® and SQA? 273
11.4 Approach to Meeting PPQA Requirements 274
11.5 Quality Management and Quality Assurance Infrastructure 274
11.6 Using Criticality and Configuration Management Status Accounting
to Govern Quality 275
11.7 Quality Auditing 277
11.8 Quality Reporting 279
11.9 Proactive Support of Projects 281
11.10 SQA Support Levels 282
11.11 Software Configuration Management 284
11.12 Traps in SQA Implementation of PPQA 286

CHAPTER 12
12. SQA for Small Projects 291
12.1 Introduction 291
12.2 Definitions 292
12.2.1 Small Organization 293
12.2.2 Small Project 293
12.3 Staff Considerations 293
12.3.1 Qualifications 294
12.4 Training Considerations 295
12.4.1 Quality Engineers 295
12.4.2 Mentoring the Project Personnel 295
12.5 What Makes Sense for Your Organization/Project(s)? 296
12.5.1 Tactical 296
12.5.2 Strategic 297
12.6 Success Without Stress and Undue Expense 298
12.6.1 Use a Generic SQA Plan and Schedule 298
12.6.2 Efficiently Audit Work Products 299
12.6.3 Efficiently Review Processes 301
12.6.4 Develop a Quality Engineer’s Guide 302
12.6.5 Provide Senior Management Insight into the Project 302
12.6.6 Act as a “Gatekeeper” for Deliverables 303
12.6.7 Add Engineering Experience 303
12.6.8 Keep an Eye on Configuration Management 303
12.6.9 Walk the Halls 305
12.6.10 Colocate Quality Engineers 305
12.6.11 Share Information 305
12.6.12 Facilitate Process Improvement 306
12.6.13 Institutionalize Processes 306
12.7 Objective Evidence for the Auditor/Appraiser 307
12.8 Compliance with ISO and CMMI® 307
12.8.1 ISO/CMMI® Internal Audits 308
12.8.2 ISO/CMMI® External Audits 308
12.8.3 Document Control 309
12.9 Summary 309
References 310

CHAPTER 13
13. Development Quality Assurance 311
13.1 Introduction 311
13.2 Software QA Versus Traditional QA 312
13.3 Development Quality Assurance 313
13.4 Systems and Software Quality Assurance: An Integrated Approach 314
13.4.1 Process Evaluations 314
13.4.2 Work Product Evaluations 319
13.4.3 Formulating the SSQA Implementation Plan 319
13.4.4 Keeping the SSQA Implementation Plan Current 320
13.4.5 SSQA Tools and Techniques 321
13.4.6 IPT Participation 321
13.4.7 Review of Deliverable Products 322
13.4.8 Participative Activities 322
13.4.9 Results of Evaluations 323
13.5 Systems Quality Assurance 324
13.6 Hardware Design Quality Assurance 324
13.7 Overcoming Cultural Resistance 327
13.8 Conclusion 329
References 330

CHAPTER 14
14. Quality Management in IT 331
14.1 Introduction 331
14.2 Key IT Processes 332
14.2.1 ITSM Processes 332
14.3 IT Best Practices 333
14.3.1 ITIL® 333
14.3.2 SEI CMMI®-SVC 336
14.4 ITSM Standards 337
14.4.1 ISO 20000 337
14.4.2 ISO 20000-1 Content 338
14.4.3 CobiT® 342
14.5 Selecting a Process Improvement Model 347
14.5.1 IT Service Management Self-Assessment 349
14.5.2 Implementing an IT Service Management System 350
14.6 Customer Requirements 352
14.6.1 Service Level Agreements 352
14.6.2 QoS 357
14.7 Monitoring and Measuring ITSM Performance 358
14.7.1 Why Variance Is Difficult to Measure 359
14.8 Procurement Quality—Outstanding 362
14.9 IT Quality Professional 364
14.9.1 Body of Knowledge 365
14.9.2 IT Quality Analyst 365
14.10 Conclusion 368
References 368

CHAPTER 15
15. Costs of Software Quality 371
15.1 Introduction 371
15.2 The Concept of Cost of Software Quality 372
15.2.1 The Concept 372
15.2.2 Objectives of Cost of Software Quality Metrics 373
15.3 Costs of Control 374
15.3.1 Prevention Costs 374
15.3.2 Appraisal Costs 375
15.4 Failure of Control Costs 375
15.4.1 Internal Failure Costs 375
15.4.2 External Failure Costs 376
15.5 Implementation of a Cost of Software Quality System 377
15.5.1 Definition of Cost Items for the CoSQ Model 377
15.5.2 Definition of the Cost Data Collection Method 378
15.5.3 Implementation of a CoSQ System 379
15.6 The Contribution of a CoSQ System to the Organization 379
15.7 Difficulties in the Implementation 380
15.8 Limitations of the Classic CoSQ Model 380
xiv Contents15.9 Extreme Cases of Costs of Software Quality 381
15.10 Conclusion 382

CHAPTER 16
16. Software Quality Assurance Metrics 393
16.1 Introduction 393
16.2 Software Quality Indicators 395
16.3 Practical Software and Systems Measurement (PSM) 396
16.4 CMMI® Measurement and Analysis 403
16.5 CMMI® Higher Maturity Measurements 405
16.6 Practical Implementations 407
16.6.1 Hewlett Packard 407
16.6.2 Quantitative SQA 409
16.6.3 Pragmatic Quality Metrics 409
16.6.4 Effectiveness Measure 410
16.6.5 Team Software Process (TSP®) and Personal Software
Process (PSP®) 411
16.6.6 Software Quality Fault Prediction 412
16.6.7 Measuring Process Improvement Using Stoplight Charts 414
16.6.8 Six Sigma 415
16.6.9 Project Managers Control Panel 415
16.6.10 Predicting Software Quality 419
16.7 Conclusion 421

CHAPTER 17
17. More Reliable Software Faster and Cheaper: An Overview of Software
Reliability Engineering 425
17.1 Introduction 425
17.2 Software Reliability Engineering 425
17.2.1 What it Is and Why it Works 425
17.2.2 A Proven, Standard, Widespread Best Practice 426
17.3 SRE Process and Fone Follower Example 428
17.3.1 Define the Product 430
17.3.2 Implement Operational Profiles 430
17.3.3 Define “Just Right” Reliability 432
17.3.4 Prepare for Test 432
17.3.5 Execute Test 433
17.3.6 Guide Test 433
Contents xv17.3.7 Collect Field Data 435
17.4 Conclusion 435
17.5 To Explore Further 435