The first critical step in assessment is to conduct a process evaluation. It answers the question,"Was the intervention put into place as planned and how was it altered for implementation?" As shown in the figure, a process evaluation focuses on the resources that were employed by the response (inputs) and the activities accomplished with these resources (results), but it does not examine whether the response was effective at reducing the problem (outcomes). For that you need an impact evaluation. An impact evaluation tells you whether the problem changed (Steps 47 to 53).
ROLES OF PROCESS AND IMPACT EVALUATIONS
Both types of evaluations are needed in a POP project. The table summarizes possible conclusions based on the findings of both types of evaluation. A) The response was implemented in accordance with the plans, and there are no other reasonable explanations for the decline. So there is credible evidence that the response caused the reduction. B) The response was implemented as planned, but there was no reduction in the problem. So there is credible evidence that the response was ineffective.
But what if the response was not implemented as planned? In this case, it is hard to come to a useful conclusion. C) If the problem declined, it might mean that the response was accidentally effective or some other factor was responsible. D) If the problem did not decline, then no useful conclusion is possible. Perhaps the implemented response is faulty and the original response would have been effective, or neither is effective. Unless the planned response was implemented, it is hard to learn from an impact evaluation.
Focus of Process and Impact Evaluations
Interpreting Results of Process and Impact Evaluations
| Process Evaluation Results | |||
|---|---|---|---|
| Response implemented as planned | Response not implemented as planned | ||
| Impact Evaluation Results | Problem declined and no other likely cause | A. Evidence that the response caused the decline | C. Suggests that the response was accidentally effective or that other factors may have caused the decline |
| Problem did not decline | B. Evidence that the response was ineffective | D. Little is learned | |
A response is a complex piece of machinery with a variety of components, any of which can go wrong (Step 45). A process evaluation examines which components were carried out successfully. The process evaluation checklist highlights the questions that you should ask.
Scheduling of activities in a problem response is often critical. For this reason, it is useful to create a project timeline showing when key components were implemented. It is also useful to show when other unexpected events occurred and noting publicity so you can check for Anticipatory Benefits (Step 52).
Though unexpected developments can force you to modify a response, some of these developments can be anticipated by understanding what can go wrong with responses. Some of the possible answers are as follows:
- You may have an inadequate understanding of the problem. You may have focused too little on repeat victims, for example. This can be caused by invalid assumptions about the problem or insufficient analysis (you did not look for repeat victimization, for example). If, while developing the response, you can identify weak spots in your analysis, then you can create contingency plans (a plan to address repeat victimization should this prove to be needed).
- Components of the project have failed. The eprocess evaluation checklist shows that there are many potential points of failure. However, not all components are equally important for success. Further, it is sometimes possible to anticipate components with high failure rates. Citizen groups in general are quite variable in their ability to carry out tasks, for example. Building in redundancy or formulating backup plans can mitigate component failure.
- Offenders may react negatively to your response (Step 11). Some forms of negative adaptation can be anticipated and planned for. Sometimes geographical displacement locations can be identified before the response, for example, and advanced protective actions can be taken to immunize them.
- There are unexpected external changes that have an impact on the response. A partner agency's budget may be unexpectedly cut, for example, forcing it to curtail its efforts on the problem. As the problem will not dissipate on its own, the only recourse is to alter the plans.
Process evaluations require information. This information will come largely from members of the problem-solving team, so it is important that they document their activities. Which activities they document and who records it in what detail are questions that should be resolved while planning the response.