Practices for Monitoring SISA Programs
The following practices implement a particular approach to helping the PM and contractor come to a mutual understanding of a program’s progress and the significance of deviations from expectations. The key to that approach is to consolidate a contractor’s progress-related data into a structure, called the “program dashboard,” that covers those areas of contractor performance that enable forecasting future progress. The program dashboard provides the PM with the evidence needed to say, “I can keep the promises I made,” or “here’s the evidence that I can’t keep those promises, so I’m going to have to slip schedule two months.”Program dashboards can take many forms. Not all these examples relate to monitoring SISA programs, but they provide some notion of the types of data that can be collected, organized, and presented in a visual display of the status and progress of a program.
- Address management measures and their use in RFPs and contracts.The PM wants to direct prospective bidders or the awarded contractor to use a specific set of measures. The problems with such an approach were discussed in the challenges above. Rather than directing the contractor to use a specific set of measures, the PM is better served by leveraging the professionalism of the contractor. Certainly, the contractor has questions similar to those of the PM and a need to answer those questions to plan and manage the overall effort and individual tasks. Contractors need to measure and track what they produce. Those are measures that PMs want to see. PMs must make commitments to their stakeholders, and any request to change those commitments must be carefully introduced and socialized. Given the difficulty of specifying the measures in advance, the PM must instead request the contractor to provide measures that address the PM’s questions.The dashboard poses a set of questions—but what gets measured (and thus what measurements are reported) is determined by what the contractor has learned works and has incorporated into a formal measurement program.
Thus, the request for proposals (RFP), statement of work, and contract should
- identify the progress-related questions on which the contractor–PM discussions and program reviews will focus. These questions are general and commonly used by the management of any diligent contractor to better understand what might affect the success of the program.
- request bidders or contractors to indicate how they intend to respond to those questions—what measures will apply?
- identify predefined times relative to milestones or at the beginning of each major development phase when the contractor will report what measures the contractor will use to report progress as a contract deliverable.
Regarding (b) and (c), unforeseeable changes in requirements, process, or technology may require the contractor to update the set of measures used for that phase and communicate this to the PM. The contractor should allow sufficient time for the program office to become better acquainted with the new measures that the contractor will use during that phase.
The PM may also provide an example dashboard in the RFP and contract to help clarify his or her intent.
- Set up the dashboard.In an acquisition context, the PM does not direct how the contractors will perform the work; instead, the PM participates in making decisions about deliverables and prioritizing product content in keeping with commitments to external stakeholders. Some of these decisions will result in a change request that may affect either cost or schedule. Such change requests will need to be reviewed by external stakeholders who will ask for supporting justification. That justification is needed to support the decisions made and the need for changes to existing commitments, which can be shown easily on a dashboard.The contractor’s data is organized into quadrants according to the type of control that the PM, contractor, or stakeholders can exercise.
The program dashboard helps the PM justify what action (e.g., change the direction of the contractor or change the commitments to external stakeholders) is necessary by
- accurately forecasting milestones and delivery of products
- providing clear warnings if the plan is not working or an unplanned event has affected some desired outcome
- supporting re-estimation and re-planning by showing the magnitude of the problem
- providing data that can be used as evidence for stakeholders to reset expectations, take certain actions, and renegotiate commitments
The first three quadrants represent opportunities to control and direct the work with the involvement of appropriate parties (e.g., customer, contractor, major subcontractors):
- Change the scope
- Change schedule and resource allocation
- Change a process
The fourth quadrant represents the target objectives for product quality:
- Frequent quality checks during development help to avoid rework caused by testing failures found late in the project.
- This quadrant records the frequency and outcome of all product quality checks, including the results of testing.
Certainly these quadrants relate to one another. Taking action on one can affect another one. But collectively they help ensure that all the information critical to making decisions is well founded. Thus, you shouldn’t ignore any of these quadrants.
An example dashboard, including measures that it can portray, is described in the Condensed Guide to Software Acquisition Best Practices (slide 4). It doesn’t align into these exact four quadrants, but you should recognize the quadrants in the different arrangement.
- Assign and train staff to interpret the dashboardsThe program dashboard is a tool that helps the PM understand a program’s progress or status; whether the PM or contractor should take action to bring the progress of the program back into alignment with the program’s (and stakeholders’) existing goals and expectations; and when the PM must alert stakeholders to a problem, reset expectations, or renegotiate commitments.A dashboard includes measurements that are organized into one or more charts that show, for example, how the measure changes over time.
Since each of the four quadrants of the dashboard may require several measures or charts, the individual PM is not likely to understand every nuance of their interpretation. An example is a contractor’s reported defect-detection rate during verification (e.g., peer reviews, integration testing). If it is high, does that mean that the resulting product will have few faults and vulnerabilities? Or does it mean that many defects were introduced early and verification is finding only the easily discovered ones? And what do the answers to these questions mean to a PM’s commitments with other stakeholders, especially those who provide services essential to the program (e.g., test-lab access, training, piloting) and who have made plans around a particular delivery date?
The PM can assign measures used for the dashboard charts to different individuals in the program office. Each person should become an expert on interpreting that particular type of measure and changes in that measure. The staff should also be able to identify what additional information is needed to understand the causes and significance of a change in a measure and to probe when a measurement is different than expected. Multiple causes are possible for any deviation in the expected data; in general, it is not possible to diagnose the cause of a deviation (and thus its mitigation or resolution) without discussion with the contractor. This interpretation capability is important not just for the program reviews but also for helping the PM reset expectations with stakeholders for schedule, cost, and quality. To provide a factual and compelling explanation for the redirection or other action taken, the PM needs the background behind the measures and the ability to explain what they mean or don’t mean.
Because different contractors may use different measures, and because the measures may change from phase to phase, staff training on measurements may be particular to the contractor and phase. Small differences in terminology can lead to important nuances of interpretation, and this leads to another reason for assigning different measures to different staff: it allows deep understanding and experience with the particular measures to develop.
- Populate and update the dashboard regularly and when neededInformation contained in the dashboard can require review or change at different times. As the contract proceeds, the PM updates the dashboard with the measurements regularly provided by the contractor. When milestones are reached, the PM needs to determine with contractor input the specific questions that will be asked in the next phase for each quadrant and what measurements the contractor will provide to help answer those questions. It is also necessary to account for changes that may occur asynchronously:
- a contractor may initiate a change (e.g., due to a resolution of a problem; or a change in technology)
- a government agency may initiate a change (e.g., in program scope or funding)
In what follows, we provide motivation for and describe what types of measures might be used in each quadrant of the dashboard.
Cost, Schedule, Resource Availability, and Predictability (Quadrant 1)
Motivation: the PM needs to use forecasts to request commitments from stakeholders. If the forecast is very different from the plan, the plan is probably in error and must be updated.
Earned Value Management (EVM) is a de facto industry standard for cost and schedule reporting. One of the principal reasons is the capability it provides to make bounded forecasts of schedule completion and estimated cost at completion(EAC). Contrast this capability with what kind of forecast one obtains from the frequently used Gantt chart. The Gantt chart shows the plan and the work completed. The PM may be able to interpret the potential for a schedule slip but lacks the basis for predicting how much it will slip. Thus, EVM is more likely to provide realistic forecasts. A realistic forecast of a major cost overrun or schedule slip will cause an immediate reaction by the PM. He or she knows that something must be done and that decisions will have to be made.
Similarly, a resource-availability problem has to be escalated quickly. If the PM is aware that a critical test needs to be performed by the end of the month, he or she must immediately confirm resource availability. Similarly if critical resources are not available when needed, some work will have to be reprioritized and rescheduled. Significant changes may result in a need for program replanning.
Decisions of what to do when cost, schedule, resource, or staffing issues arise must be made carefully but quickly to limit the impacts because so many schedule factors are interdependent. Terrible program-schedule problems occur when different tasks shift onto and off of the project critical path. These problems can make the schedule completely unpredictable and may cause the PM to no longer be trustworthy to external stakeholders.
Scope, Progress, and Change Management (Quadrant 2)
Progress is usually recorded when some portion of the work passes from a single engineer to become a team responsibility and then as the work passes from team to team. Progress must be measured against some planned (collection of) deliverable(s). Often, the deliverable can be structured as a collection of smaller artifacts that can be counted to provide a measure of size. For example, testing progress can be measured as a count or percentage of successful completion of a set of critical test cases. A count or percentage of engineering drawings completed is another example of measurement by collection.
Sometimes deliverable completion is better represented by a combination of measures. Sometimes, measures of physical size or complexity are used (thus allowing “normalization” of the measures, which provides a sounder basis for estimating productivity). Other times, the measures are derived from a decomposition of the effort into smaller tasks to produce the artifact and their attributes; for example, a design may be represented by a set of prototypes, fan-in/fan-out measures, and number of software integration steps. More generally, the completion of some activities may be represented by proxies.
Software progress is usually measured in lines of code or some measure of functional size as the code is stored in the configuration-management repository and passes unit-level testing—as this is the point where responsibility of the code passes from the single developer to a team responsibility. If a measure such as story points is used, the developers already have a notion of a combined measure for size and complexity that is less formal than function points but is probably satisfactory within the team. As a practical matter, it is difficult to rely on story points if one wants a productivity measure for the organization.
Scope, on the other hand, is not about intermediate deliverables. Instead, what is included in the scope is every deliverable that must leave the control of the project team. If the product is to be delivered on two platforms instead of one, that is an element of scope. If the product prototype is to go to a national engineering show, that is a change in scope. If a visiting general officer is planning a visit for a demonstration, that is also a change in scope that will consume resources and has a milestone date. All such deliverables must pass some external quality control. For each element of scope, there is a cost that can be assigned, progress can be measured, and delivery forecasted.
Effectiveness of Process and Risk Management (Quadrant 3)
Acquisition programs hire contractors that promise an effective and productive process. The contractor may even promise to meet ISO 9000 standards or CMMI Level 5. These external measures of process quality are very important to the purchaser and acquirer (see Davenport 2005). Attestations of compliance are evidence that contractors are capable of keeping commitments if they are allowed to follow their defined processes. A PM who asks the contractor to skip steps in a process invites problems that will likely arise sooner or later in the development process. Instead, the acquirer should work to achieve a basic understanding of the processes the contractor intends to use on the program. This pursuit of understanding can begin by having the acquisition team ascertain the boundaries of uncertainty in the product definition and product quality as the program’s stakeholders see it as compared to the contractor’s understanding and what the contractor has and is capable of delivering.
Not many processes need to be monitored, but the contractor’s change-management, quality-management, and risk-management processes are three that do need to be monitored. The contractor should have a change-management process whereby changes to the project plan and to the requirements baseline and other specifications can be monitored. Frequent changes suggest a potential risk inherent in the predictability of forecasts. The contractor should also have a quality-management plan. The quality-management plan should specify the number of resources assigned to verification and quality-assurance activities and describe the frequency of such quality interventions. The quality-management plan should also describe how the results of these quality activities will be aggregated and conveyed to the program office.
A typical mechanism for demonstrating process effectiveness is a measure of defect escapes. The defect-escape report is evidence that the contractor is using frequent quality checks of the work and that the resources assigned to quality activities are properly engaged.
Indicators for risk management also belong to the process-effectiveness quadrant. Risk management by its nature is of equal concern to both the contractor and the program office. Unlike most indicators for effective process, both contractor and program office have equal concerns to handle risk efficiently and effectively. The PM who assumes that the contractor is 100% responsible for managing risk deceives himself. If risk management is not tracked and risks are not mitigated, then the potential problems will become real issues and significant change management will need to occur. Both issues and change-management requests will consume significant resources and delay the program schedule.
Product Quality (Quadrant 4)
Typically, program offices have only the modest goal that the developed product should pass operational test. This is a good goal in and of itself, but operational test is a lagging rather than leading indicator of emerging product quality. This means that, if product quality is inadequate, relying only on operational test to discover problems with product quality leaves few options. A better approach is to identify and address quality problems early, when they can be resolved more quickly and easily, and with less impact to cost, schedule, and stakeholder expectations and commitments. Therefore, PMs should consider how to recognize and track product quality earlier.
For example, there could be a product-quality goal for defect density (perhaps a goal established for each product-development phase), which can work pretty well if the contractor has process and product historical data by which intermediate product quality can be related to final product quality. If the historical data is insufficient or inadequate to establish such relationships, this approach may not be so effective, as it is hard to gauge whether a particular defect density achieved at an early phase of the program will lead to serious problems in the long term.
When historical data is lacking, an alternative approach is to use various process heuristics. An example process heuristic is “the average defect removal rate during operational test is one and a half (1.5) defects per week.” It is hard to do better than that because: (1) operational test failures result in long periods of analysis, rebuilding, and retesting; and (2) many operational test failures become “test blockers” that prevent the execution of other test cases. Such heuristics allow the acquisition team to work backwards from some target product density at release to the beginning of operational test, etc.
Another approach to tracking product quality is to develop validation or assurance cases. Assurance cases represent a potentially superior approach to checking product quality as the development team progresses through product development. It is perhaps the best way to measure (track, evaluate, assess) product quality throughout the life cycle. The earlier webpage on Technical Best Practices for Safety-Critical Systems includes a practice on building safety assurance cases. In some situations, the same approach can be used for other systemic quality attributes, such as system availability and security.
- Include discussion of dashboard changes in program reviews and as neededIt is easy for program reviews to fail to investigate the right questions. For example, one program review went something like this:PM: “Wow, you had this sudden increase in program defects this month, what changed?”
Contractor: “Actually, it’s good news. We got many defects out of the way.”
The contractor completely bypassed the PM’s question, telling the PM something he thought the PM wanted to hear. The PM probably did not learn anything regarding
- how so many defects got introduced into the product
- what the contractor will do differently in the future
- whether stakeholder expectations need to be reset
- the viability of the PM’s commitments to the program’s stakeholders
- what the PM can do to help
Instead, program reviews should include a discussion of recent contractor activities, what the contractor learned, what changed, and how these affect progress toward achieving the program’s objectives. The program dashboard provides a vehicle for covering the right questions and guiding the discussion.
Before the program review, the program office produces the latest dashboard and compares to two (or more) earlier dashboards covering the time since the last program review to help identify where to probe. For example, if this review of the dashboard and other historical data for previous estimates of cost, schedule, etc. suggests a problematic trend, the PM may want to make this a focus of upcoming discussions with the contractor. Coming thus better prepared for the program review, the PM and the acquisition team, referring to the dashboard charts, can ask the question “Why did this change and what is the impact?” They are seeking to understand the reason for a change—not looking for excuses. The questions must be non-threatening but instead have the aim of discovering the causes or drivers of learning and change to help better gauge implications for program objectives, schedule, and commitments with stakeholders.
In this way, the program dashboard helps ask the right questions and helps keep the discussions on target during program reviews. The PM’s thoughts will resemble something like this: “Why did this change, is it something that I need to do something about, is the contractor handling this issue, am I going to be able to keep my commitments to stakeholders, am I going to have to apologize to someone?” In a nutshell, the PM is trying to answer, “What am I going to have to do to ensure the best possible product within cost and schedule constraints?”
A few reminders/cautions: The dashboard doesn’t answer the questions; rather, it tells where to probe. If something curious or potentially troubling is seen on the program dashboard, the dashboard doesn’t provide the answer but helps initiate the discussion needed to get at the answer, which helps determine what, if anything, needs to be done about it. The acquisition team members assigned the particular measures (third practice) perform much of the actual probing. The team member assigned to a particular measure needs to know how and where to probe, to follow up, to be able to understand what a noticeable change in that measure signifies.
- Refresh measures for each new phase/milestoneThe dashboard is a focusing tool to use during program reviews through the life of the program, but the measurements on the dashboard change as the program progresses from phase of work to phase of work. Through the life of the program, the dashboard helps the PM revisit the same basic questions, but they are not the same specific questions. As mentioned in the first practice, at the beginning of a new development phase, contractors should provide the set of measures they intend to use to report progress. The acquisition team needs to become familiar with these measures and their definitions as described in the third practice.An example of how measures may change reflects the difference between development activity and operational testing. The measurement of progress changes from deliverables completed to test cases passed. Test cases passed are easier to observe, and so progress is more apparent. Measuring effectiveness of the process is also performed differently. During operational testing, we should observe how long it takes to resolve discovered defects and how much effort is involved in defect tracking. The cycle time for defect management is often directly related to some set of decision processes, and thus its measurement is an indicator of process effectiveness. Such measures of process effectiveness can be supplemented with a count or percentage of “bad test cases.” A bad test case is a test that fails whose cause is traceable to an error in the definition of the test—thus representing a process problem rather than a product problem.
- Set new expectations and negotiate changes to commitments with stakeholdersAs a result of the program review or other discussion (fourth and fifth practices), the PM may conclude that stakeholder expectations need to be reset or commitments need to be changed. The content of a program dashboard, its analysis, and subsequent discussions with the contractor serve as the justification for the reset/change and also serve as the basis for determining what changes to stakeholder commitments are necessary. Giving due diligence to the changes implied by the dashboard helps to sustain stakeholder confidence in the PM’s ability to recognize when changes are needed and helps maintain stakeholder interest and support for the program.Sometimes, the nature of the change is fairly significant. For example: Prototypes or simulations are typically performed for any high-risk change to the technology used. The challenge the contractor needs to address is not simply to prove that the technology itself works; rather, the challenge is to prove that the contractor can develop a product that uses the new technology in an integrated product and further that the contractor is prepared to test a product that contains the new technology. It is not unusual to discover that the new technology is much more difficult to apply than expected and that more schedule and perhaps additional resources are required. If the prototype is on the critical path for development and testing, it is virtually certain to delay product delivery. Such significant changes are likely to be reflected in all four quadrants of the dashboard.
Remember also that many of a program’s stakeholders have negotiated commitments with their own stakeholders—commitments they are not likely to share with the PM in order to retain flexibility in what they can claim they can or can’t do when renegotiating commitments with the PM. The dashboard evidence the PM provides about his or her own schedule and resource forecasts, product quality, and the other topics touched on by the four quadrants—but that also are of potentially keen interest to stakeholders—gives the PM the leverage he or she needs to enter into some potentially complex discussions with stakeholders.