Monitoring Software-Intensive System Acquisition (SISA) Programs

Our discussion of monitoring SISA programs has four parts. First, we set the context by providing an answer to the question “Why is monitoring an SISA program challenging?” Seven practices for monitoring SISA programs follow. We then briefly address how a program manager (PM) and acquisition team can prepare for and achieve effective results by following these practices. We conclude with a list of selected resources to help you learn more about monitoring SISA programs. Also, we’ve added links to various sources to help amplify a point—these sources may occasionally include material that differs from some of the recommendations below.

Every program is different; judgment is required to implement these practices in a way that benefits you. In particular, be mindful of your mission, goals, processes, and culture. All practices have limitations. We expect that some of these practices will be more relevant to your situation than others, and their applicability will depend on the context to which you apply them. To gain the most benefit, you need to evaluate each practice for its appropriateness and decide how to adapt it, striving for an implementation in which the practices reinforce each other. In particular, the practices are not intended to be in strict sequence: practices may iterate or overlap with others. Also, consider additional practice collections (such as Pitman’s SCRAM approach, which is referenced at the end of this web page). Monitor your adoption and use of these practices and adjust as appropriate.

These practices are certainly not complete—they are a work in progress. We welcome your feedback (use the comments section at the end).

Why is monitoring SISA programs challenging?

Essential to effective program management is the capability to maintain an accurate and current understanding of a program’s status so that issues that threaten program objectives can be identified quickly and dealt with efficiently. Also, the program manager (PM) depends to a large degree on the goodwill and commitments of a program’s many stakeholders. When a program’s status and forecasts frequently change, causing the PM’s promises and assurances to fail, stakeholders may lose confidence and withdraw their time commitments and attention from the PM’s program and instead invest them in less risky undertakings. Thus, the PM cannot afford to break many commitments; the PM wants to identify and resolve issues early before they grow into significant problems that require the attention of external stakeholders. Continual cost overruns or schedule slippage may lead to greater oversight and even program termination.

Monitoring a program’s progress is challenging for several reasons:

1. Contractors don’t understand the PM’s commitments to other stakeholders. Contractors may misunderstand the PM’s role, misinterpreting the PM’s questions as efforts to infringe on the contractor’s responsibilities, and not understanding the important coordination role the PM has with a program’s stakeholders. To be successful, the program must have the support (i.e., resources, services, and attention) of many stakeholders (e.g., for reviews, testing, training, and sponsorship meetings). To sustain this support when other worthy endeavors compete for the same limited resources, the PM negotiates commitments with the program’s stakeholders. The PM becomes dependent on a stakeholder for certain forms of support, and the stakeholder becomes dependent on the accuracy of a PM’s forecasts of what resources will be needed and when. For example, contractors often have the data the PM needs to do these things but don’t recognize this or put it in a form that helps the PM; conversely, without the right data, the PM cannot create accurate forecasts.
2. PMs don’t understand contractor data. The PMs may not have sufficient information to interpret the activity and schedule-related data that they get from contractors. The PM needs to understand the implications such data have for a program’s objectives and the viability of a PM’s existing commitments with stakeholders. Absent such understanding, the PM may waste time addressing the wrong issues, allowing serious problems to go unaddressed and later blindside the PM and stakeholders, resulting in lost time, resources, and goodwill.
3. Changing to a new set of measures is difficult. Measures are important to guiding system and software engineering and program-management decisions, but it takes effort and time to establish an effective measurement program. PMs may believe that they can ask the contractor to provide a particular set of measures and the contractor can easily and quickly make the necessary changes, but that’s rarely the case for several reasons:
   • It takes effort and time for a contractor to introduce a new measure. Doing so may also require changes to operational definitions, analyses, reporting templates, tools, automated data collection, training, data management, access rights, privacy, and so forth.
   • Useful measures are generally obtained by observing and instrumenting a real process. Since each organization will have a unique  process, some measures will also be unique to the organization.
   • There is always something new about a large project, so the organization doing the work will discover new things about the project as it proceeds. The organization may not be able to determine what to measure completely in advance.
   • An acquisition program may have several phases of work, and different measures will have more or less importance in different phases. Rather than asking for a permanent change to the measurement program, sometimes the PM issues a “data call.” With data calls, the contractor does not expect that the measure will be repeated, so he or she gathers whatever is at hand. Such data lacks precision and supporting process. Issued too often, data calls become disruptive. Collecting good data involves instrumenting the process so the data is collected as a natural result (or side effect) of doing the work.

To address these challenges, the PM and contractor should work together to understand how they will use the contractor’s measures  to identify and evaluate potential threats to program objectives and commitments, which can also have an effect on the PM’s commitments to stakeholders. This observation is the basis for these practices.