The next challenge for many project leaders is to collect a variety of data along a chain of impact that shows the project’s value. Figure 13.3 displays a detailed view of the sequential steps that lead to data categorized by the five levels of results. This figure shows the ROI methodology, a step-by-step process beginning with the objectives and concluding with reporting of data. The model assumes that proper analysis is conducted to define need before the steps are taken.

Evaluation Planning

An important first step in planning is to establish or refine the objectives. Ideally, objectives are developed from the initial analysis, described earlier.

Figure 13.3 ROI Process Model.

©2009 ROI Institute, Inc.

On occasion, the evaluation may stop with Level 3 objectives, excluding the impact objectives that are needed to direct higher levels of evaluation. If application and impact objectives are not available, they must be developed using input from job incumbents, analysts, project developers, subject-matter experts, facilitators, and on-the-job team leaders.

Three simple planning documents are developed next: the data collection plan, the ROI analysis plan, and the project plan. These documents should be completed during evaluation planning and before the evaluation project is implemented—ideally, before the program is designed or developed. Figure 13.4 shows a completed data collection plan for a project undertaken to reduce bus drivers’ absenteeism in a major city. This document provides a place for the major elements and issues regarding data collection, described in the next section.

This planning document also captures information on key items that are necessary to develop the actual ROI calculation—all described in the data analysis section. The final plan developed for the evaluation planning phase is a project plan. A project plan consists of a description of the project and brief details, such as duration, target audience, and number of participants. It also shows the timeline of the project, from the planning of the study through the final communication of the results. This plan becomes an operational tool to keep the project on track.

Most of the decisions regarding the process are made as these planning tools are developed. The remainder of the project becomes a methodical, systematic process of implementing the plan.

Collecting Data

Data collection is central to the ROI methodology. Both hard data (representing output, quality, cost, and time) and soft data (including job satisfaction and customer satisfaction) are collected. Data are collected using a variety of methods, listed in Figure 13.5. These are commonly used methods. This figure also shows the methods which are appropriate for the different levels of data.

The important challenge in data collection is to use a method or methods appropriate for the setting and the specific program, within the time and budget constraints of the organization. Figure 13.4 shows the methods, timing, and sources for data collection in the absenteeism reduction project. Additional details on data collection can be found in other references (Phillips & Stawarski, (2008).

Figure 13.4 Data Collection Plan.

*Three questions address how the policy works (employee must answer two correctly); two address employees’ role (employee must answer one correctly). 2009 ROI Institute, Inc.

Figure 13.5 Data Collection Methods.

©2009 ROI Institute, Inc.

Data Analysis

Data analysis involves all steps from data collection to reporting. Five distinct steps are always addressed: isolating the effects of the project, converting data to monetary values, tabulating project costs, calculating ROI, and identifying intangible benefits. Each is briefly discussed next.

Isolating the Effects of the Project. An often overlooked issue in evaluation is the process of isolating the effects of the project. In this step, specific strategies are explored that determine the amount of output performance directly related to the project. This step is essential because many factors will influence performance data. The specific strategies of this step pinpoint the amount of improvement directly related to the project, resulting in increased accuracy and credibility of ROI calculations. The following techniques have been used by organizations to tackle this important issue:

· Control groups

· Trend line analysis

· Forecasting models

· Participant estimates

· Managers’ estimates

· Senior management estimates

· Experts’ input

· Customer input

Collectively, these techniques provide a comprehensive set of tools to handle the important and critical issue of isolating the effects of projects. Additional information on methods of isolation are available in other references (Phillips & Aaron, 2008).

Converting Data to Monetary Values. To calculate the return on investment, Level 4 impact data are converted to monetary values and compared with project costs. This requires that a value be placed on each unit of data connected with the project. Many techniques are available to convert data to monetary values. The specific technique selected depends on the type of data and the situation. The techniques include:

· Use of output data, as standard values

· Cost of quality, usually as a standard value

· Time savings converted to participants’ wage and employee benefits

· An analysis of historical costs

· Use of internal and external experts

· Search of external databases

· Use of participant estimates

· Use of manager estimates

· Soft measures mathematically linked to other measures

This step in the ROI model is important and absolutely necessary in determining the monetary benefits of a project. The process is challenging, particularly with soft data, but can be methodically accomplished using one or more of these strategies. Additional information on data conversion techniques are available in other works (Phillips & Burkett, 2008).

Tabulating Project Costs. An important part of the ROI equation is the calculation of project costs. Tabulating the costs involves monitoring or developing all the related costs of the project targeted for the ROI calculation. Among the cost components to be included are

· Initial analysis costs

· Cost to design and develop the project

· Cost of all project materials

· Costs for the project team

· Cost of the facilities for the project

· Travel, lodging, and meal costs for the participants and team members

· Participants’ salaries (including employee benefits)

· Administrative and overhead costs, allocated in some convenient way

· Evaluation costs

The conservative approach is to include all these costs so that the total is fully loaded.

Calculating the Return on Investment. The return on investment is calculated using the program benefits and costs. The benefits/costs ratio (BCR) is calculated as the project benefits divided by the project costs. In formula the BCR is;

The return on investment is based on the net benefits divided by project costs. The net benefits are calculated as the project benefits minus the project costs. In formula form, the ROI is

This is the same basic formula used in evaluating other investments, in which the ROI is traditionally reported as earnings divided by investment. Additional information on costs and ROI are contained in other works (Phillips & Zuniga, 2008).

Identifying Intangible Benefits

In addition to tangible, monetary benefits, intangible benefits (those not converted to money) are identified for most projects. Intangible benefits include items such as:

· Increased employee engagement

· Increased organizational commitment

· Improved teamwork

· Improved customer service

· Fewer complaints

· Reduced conflict

During data analysis, every attempt is made to convert all data to monetary values. All hard data—such as output, quality, and time—are converted to monetary values. The conversion of soft data is attempted for each data item. However, if the process used for conversion is too subjective or inaccurate and the resulting values lose credibility in the process, then the data are listed as an intangible benefit with the appropriate explanation. For some projects, intangible, nonmonetary benefits are extremely valuable and often carry as much influence as the hard data items.


The final step in the ROI process model is reporting—a critical step that is often deficient in the degree of attention and planning required to ensure its success. The reporting step involves developing appropriate information in impact studies and other brief reports. Careful planning to match the communication method with the audience is essential to ensure that the message is understood and that appropriate actions follow.