Public Policy-Making: Implementation, Evaluation, Change and Termination

After completing the reading assignments and this Study Guide lesson you should be able to achieve the following objectives:

• Define public policy implementation, Stage 5 of the stages heuristic model.
• Explain why consideration should be given to the constraints and challenges of policy implementation during the policy formulation stage.
• Explain who the major policy actor is during the policy implementation stage and explain why.
• Identify and describe three major challenges and pitfalls of the policy implementation stage of the policy process.
• Define public policy evaluation, Stage 6 of the stages heuristic model.
• Identify and explain four key types of policy evaluation as discussed by Theodoulou and Kofinis, 2004.
• Identify and explain seven important barriers to effective policy evaluation as outlined by Peters, 2007.
• Explain the significant differences between internal and external policy evaluators.
• Define public policy change and termination, Stage 7 of the stages heuristic model.
• Identify and describe at least five of the eleven factors frequently cited as necessitating policy change as enumerated by Theodoulou and Kofinis, 2004.
• Explain why policy termination is so rare.

Definition of Public Policy Implementation

What is Policy Implementation?

• Represents the stage where government executes an adopted policy as specified by the legislation or policy action.
• At this stage, various government agnecies and departments, repsonsible for the respective area of policy, are formally made responsible for implementation.
• Policy implementation is what happens after a bill becomes law.

(Theodoulou and Kofinis, 2004)

Once the government has legitimized some form of public policy such as a law, statute, edict, rule, or regulation, the stipulations of that policy must be put into action, administered, and enforced to bring about the desired change sought by the policy-makers. This task defaults to the government executive and necessitates the designation of a government agency as having the responsibility for the new policy. Theoretically the responsible agency is given the requisite resources and authority to ensure that the new policy is carried out as intended, but in reality, this does not always occur.

As discussed in the first four lessons of this course, public policy is implemented to effect some change in the behavior of a target population and it can normally be assumed that this change will ameliorate some public problem. Therefore, it stands to reason that unless the stipulations of a given policy are actually carried out, the problem will persist. As soon as the tenets of the new policy are implemented, a detailed policy evaluation can be conducted to determine if the desired results are being obtained and if not, why not, and what needs to be changed. More will be discussed concerning policy evaluation later in this lesson.

Administrative agencies accomplish most of the day-to-day work of government therefore they have the most immediate and direct impact on the daily lives of citizens than do any other government entities (Anderson, 1990). Dye indicates that implementation �involves all of the activities designed to carry out the policies enacted by the legislative branch. These activities include the creation of new organizations � departments, agencies, bureaus, and so on � or the assignment of new responsibilities to existing organizations. These organizations must translate laws into operational rules and regulations. They must hire personnel, draw up contracts, spend money, and perform tasks. All of these activities involve decisions by bureaucrats � decisions that determine policy (2005, p. 52).� One critical aspect of policy implementation is the high degree of discretion afforded to the bureaucrats and agency procedures to transform laws into action as outlined by Dye above. Administrative decision-making has a significant impact the determination of who receives benefits and who is restricted as a result of the implementation of any policy. Administrative decision-making also has a far reaching impact on society as a result of the promulgation of agency regulations, contracting, licensing, inspections, enforcement, adjudication, and the actual discretion for agencies to interpret their own agency rules.

Definition of Public Policy Implementation, continued

Definition of Policy Implementation

• Policy implementation is the stage in the policy process where policy action occurs to address a recognized policy problem.
• At this stage, the design of a policy proposal is put into effect.
• Policies are executed by respective administrative agencies.
• Selected instruments are applied reflective of the legislative mandate, bureaucratic interpretation, and capacity.
• Specified target populations, and the society, experience the first tangible effects of the policy once implemented.

(Theodoulou and Kofinis, 2004)

Peters notes that �bureaucrats play and important role in interpreting legislation and making regulations to put it into effect; they also make important decisions while applying laws and regulations to individual cases (2007, p. 95).� For now, it is important to understand that government agencies at all levels of government are responsible for the implementation of policy, even if the actual providers of government services are private contractors. The institutional entities that are accountable to the people for the administration and outcome of public policy initiatives are administrative agencies and ultimately those elected legislative officials who enacted the policy in the first place and delegated its execution to the government bureaucracy. In the case of administrative agency rules and regulations, elected officials are still ultimately responsible to the people for the professionalism, efficiency, effectiveness, equality, and equity of the services provided by these agencies. Unfortunately, Congress and the president frequently pass vague, ambiguous, and �lofty� legislation purposely leaving out the details which will need to be filled in by bureaucracy. This legislative technique allows both the Congress and the president to have some degree of plausible deniability if the policy is poorly implemented or if the implementation is a failure altogether. Essentially, this tendency by elected officials to refuse to take responsibility for both the enactment and implementation of laws is an abdication of legislative and executive authority, and many argue that this behavior bestows far too much power on administrative agencies by fiat.

What these few paragraphs indicate is that policy implementation is not a clear cut automatic process that occurs as soon as legislation is passed. Policy is not self-executing. In fact a �large number of factors may limit the ability of a political system to put policies into effect. Rarely will all the factors affect any single policy, but all must be considered when designing a policy and attempting to translate it into real services for citizens. Any one of the factors may be sufficient to cause failure or suboptimal performance by a policy, and all may have to be in good order for the policy to work. In short, it is much easier to prevent a policy from working than it is to make it effective (Peters, 2007, p. 104).� The next few sections along with the assigned course readings will highlight the essential role of policy implementation in the overall policy-making and evaluation process. Additionally, the significance of several key actors involved with policy implementation, and some important challenges that routinely interfere with effective and efficient policy implementation will be discussed.

The Relationship between Policy Design and Policy Implementation

Why Policy Implementation Matters

• Effective and efficient policy actions require consideration to be given to the constraints and challenges of implementation—preferrably even as the policy is formulated.
• How well a policy is implemented has much to do with determining if it is successful or not!

(Theodoulou and Kofinis, 2004)

Regardless of how well intentioned, or how well formulated, or how universally supported in the adoption phase of the policy process, a public policy cannot begin to change the behavior of a target population or solve a specific public problem until someone or some organization implements the policy. Ideally, �each policy includes a design of how a public problem will be resolved. This design will define, in varying detail, the goal(s) of the policy, the set of policy instruments to be used, the agency responsible for implementation, possible timetables, and the target population. The point of policy design is to match the correct set of instruments with the identified problem (Theodoulou and Kofinis, 2004, pp. 168-169),� and to ultimately solve that problem. However, as we have seen throughout this course, it is highly unlikely that any adopted public policy or agency rule or regulation will ever achieve such clarity primarily because of the necessity for policy feasibility and acceptability. The competition inherent in the political stream of the policy process dictates the accommodation of many interests which inevitably dilutes policy details resulting in vague and ambiguous and often confusing legislation.

Frequently, the political give-and-take of the various policy actors in the policy process prevents a thorough understanding, and all too often a misidentification, of the true public problem at hand. Unfortunately, as solutions are developed in the policy formulation stage, improper or inadequate policy instruments are proposed that will have little if any impact. Consequently, policy-makers must be extremely cognizant of the fact that the ills of poorly designed policy cannot be miraculously healed by administrative agencies as they attempt to interpret and implement imperfect legislation. The following text block provides some key considerations that policy-makers should take into account as they work through the policy formulation process. The important point to remember is that the limited understanding of the causality of most social problems makes an in-depth evaluation of potential solutions nearly impossible. Limited understanding and knowledge coupled with the impact of diverse political considerations means that the potential for policy success is limited. Therefore, policy-makers must seriously consider the difficulties and vagaries of policy implementation if they intend to enable any policy solution to have the optimum opportunity to be as successful as bounded rationality allows:

The Relationship: Design and Policy Implementation

• Better policy can result if policy designers and implementors address some or al lof the following questions.
  • Was the design of the policy appropriate given the nature of the problem?
  • Does the design assist or complicat policy implementation?
  • Is the agency organizationally capable of administering the program or policy?
  • Can the instruments be effectively implemented?
  • Are there political obstacles to effective and efficient implementation?
  • What impact will the policy solution, once implemented, have on the target population?
  • How essential are the administrative actors to success of the policy?
  • Have potential factors prevented or made difficlut policy implementation?

  (Theodoulou and Kofinis, 2004)

  page 5 of 17

  The Bureaucracy � The Major Policy Actor of the Implementation Process

  Bureaucracy: The Major PI Actor

  • Rule-making is significant because seldom, if ever, will an adopted policy action, such as a law, encompass all of the necessary details of how, when, and where the policy is to be implemented.
  • Due to this power, the bureaucracy enjoys considerable discretion in interpreting the actual intent, method, and scope of a policy decision.

  (Theodoulou and Kofinis, 2004, p. 171)

  As is the case with all of the stages of the policy process there are a number of institutions and actors who are involved with policy implementation. Some of these participants are directly engaged in the actual administration and implementation of policy while others attempt to influence the direction that public policy will take as a result of how it is administered and implemented. Administrative agencies represent the former while the president, the Congress, the courts, and whole host of interest groups represent the later. The bureaucracy is delegated a significant degree of power during the implementation stage of the policy process because of its �discretion in interpreting the actual intent, method, and scope of a policy decision. This means that the bureaucracy can, to some extent, redefine the intent of a policy action � thereby becoming de facto policy-makers in the process (Theodoulou and Kofinis, 2004, p. 171).� Since many of the specific details of public policy are defined through bureaucratic discretion, decision-making, rule making, regulation, and adjudication it is no wonder that many interest groups persevere in their efforts to mold the ultimate impact that public policy can achieve as a result of how policy is actually implemented.

  Bureaucracy: The Major PI Actor

  • The real power of bureaucracy lies in its ability to define the details of the policy and the details that will affect the method of implementation .
  • Administrative agencies often are given broad and ambiguous statutory mandates that leave them with much discretion to decide what should or should not be done, i.e., rule making:
    • proposed rules
    • interim rules
    • ifinal rules

    (Theodoulou and Kofinis, 2004, pp. 170-171)

    page 6 of 17

    Challenges and Pitfalls of Policy Implementation

    The implementation stage of the policy process is by definition an operational phase where policy is actually translated into action with the hope of solving some public problem. Theodoulou and Kofinis identify three key challenges which they believe can routinely impede the effective implementation of public policy. Those three challenges are:

    Challenges to Policy Implementation

    • Clarity of policy goals
    • Information intelligence
    • Strategic planning

    (Theodoulou and Kofinis, 2004)

    Clarity of Policy Goals:

    �Clear policy or program goals help specify the ends or objectives desired from the policy action. Ideally, policies should be formulated with consideration of what the actual specific goals of the policy are�A goal stated with clarity and specificity not only provides direction but also improves the basis by which policies can be evaluated, for accountability, efficiency, and effectiveness (2004, p. 183).�

    Information Intelligence:

    �Information intelligence refers to the strategic necessity for constant feedback as to how implementation is progressing, as well as�preliminary assessments of impacts�Given the increasing reliance on multiple bureaucracies, information and feedback is essential to gauge the degree of interagency and intergovernmental cooperation and conflict, which may be affecting implementation�Information intelligence requires a high level of communication and feedback not only from the agencies involved�but also�from the population affected by the policy action. Implementation can be improved only if the implementers have insight as to how a policy or program is affecting a target population (2004, p. 184).�

    Strategic Planning:

    �The purpose of strategic planning, within the context of policy implementation, is to highlight the importance of assessing the capacity of an agency to meet specific implementation tasks and goals mandated by the policy decision. Essentially, strategic planning is a tool with which the agency can evaluate its ability to achieve the goals of the policy, as well as plan for how the policy will be executed (2004, pp. 184-185).�

    Steps to Strategic Planning

    • Statement of agency goals, mission, or vision
    • Adoption of a time frame
    • Assessment of present capabilities
    • Assessment of organizational environment
    • Development of a strategic plan
    • Organizational integration

    (Theodoulou and Kofinis, 2004)

    It is important to note that these three key implementation challenges as briefly outlined above do not represent the universe of potential problems that can be encountered during the implementation stage. Peters identifies many other pathologies and variables which can equally impede successful policy implementation. Some of the policy distracters discussed by Peters are similar to those identified by Theodoulou and Kofinis and they include: the vagaries of the legislation, the number and diversity of competing interest groups involved in the policy process, organizational disunity of the institutional actors, standard operating procedures employed by administrative agencies, poor organizational communication both internal and external, the perceived inability of administrative agencies to learn from prior experiences in a reasonably quick period of time, inter-organizational politics and conflict, and the difficulties experienced with vertical and horizontal implementation structures (Peters, 2007). The importance of the knowledge of these challenges is that it helps us to understand that policy implementation is not easy to begin with, and it can be made much more difficult if no attention or consideration is given to these potential pitfalls and the subsequent obstacles that they can engender if they are ignored.

    page 7 of 17

    Definition of Public Policy Evaluation

    What is Policy Evaluation?

    Policy evaluation can be better defined as a process by which general judgments about quality, goal attainment, program effectiveness, impact, and costs can be determined

    (Theodoulou and Kofinis, 2004, p 192)

    Once public policy has been operationalized through the formal adoption of laws, rules, or regulations, and the bureaucracy has taken action to implement the policy, some form of evaluation needs to be accomplished to determine if the policy has achieved the desired outcome or impact. Public policy represents the expenditure of limited public resources and or restrictions on certain types of individual or organizational behavior. Consequently, the public has a right to expect that their government officials are accountable for the validity, efficiency, and effectiveness of those policies. Policy evaluation is therefore an absolutely critical stage in the policy process whereby �we can determine whether a policy�s effects are intended or unintended and whether the results are positive or negative for the target population and society as a whole (Theodoulou and Kofinis, 2004, p. 191).� In essence, policy evaluation is the process used to determine what the consequences of public policy are and what has and has not been achieved.

    What is Policy Evaluation?

    Elected officials, policy makers, community leaders, bureaucrats, and the public want to know what policies work and what policies don't.

    The purpose of evaluation is to determine whether an implemented program is doing what it is supposed to.

    Through evaluation, we can determine whether a policy's effects are intended or unintended and whether the results are positive or negative for the target population and society as a whole.

    (Theodoulou and Kofinis, 2004, p. 191)

    At first glance, policy evaluation appears to be a straightforward concept however a closer inspection of the process reveals that policy evaluation can be equally as political and divisive as any other stage of the policy process. Policy evaluation provides additional opportunities for the myriad political interest groups and policy actors to attempt to influence the life of a specific policy. Favorable evaluations of the impact of a given policy will tend to perpetuate the implementation and life-cycle while unfavorable evaluations may give rise to change or possibly policy termination. Depending on the proclivity of any interest group or policy actor, the perception of how well a policy or program is performing or being implemented can have far-reaching impact.

    The retrospective analysis of any public policy or government action is bounded by a number of �real-world constraints, such as time, budget, ethical considerations, and policy restrictions (Theodoulou and Kofinis, 2004, p. 193),� as well as political ideologies, values, experiences, measurement instruments, goal clarity, and institutional biases. The key to understanding and interpreting the results of any policy evaluation is that some degree of bias is inherent in the process. However, this shortcoming should not prevent efforts to produce fair and unbiased policy evaluation products, at least as much as possible. The objective of policy evaluation is to discover policy flaws and to attempt to correct them given all of the limitations incumbent in the overarching policy process. �In its simplest form, evaluating a public program involves cataloging the goals of the program, measuring the degree to which the goals have been achieved, and, perhaps, suggesting changes that might bring the performance of the organization more in line with the stated purposes of the program (Peters, 2007, p. 163).�

    What is Policy Evaluation?

    • Evaluation is the assessment of whether a set of activities implemented under a specific policy has achieved a given set of objectives.
    • Evaluation is the effort that renders a judgment about program quality.
    • Evaluation is information gathering for the purposes of making decisions about the future of the program.
    • Evaluation is the use of scientific methods to determine how successful implementation and its outcomes have been.

    (Theodoulou and Kofinis, 2004, p. 192)

    page 8 of 17

    Four Types of Policy Evaluation

    Most models of policy evaluation ground their analytical perspective in the logical process used to determine the disparity between what was conceptualized by the initial policy goals and what has actually been accomplished by the policy or program as implemented. However, many other models focus their analysis on different objectives such as what is the true purpose of the evaluation, what is the role of the evaluator in the process, how broad or narrow should the scope of the evaluation be, and finally how should the evaluation be organized and conducted, for example what measurement instruments should be employed to determine success or failure. Theodoulou and Kofinis (2004) identify four generic types of the most commonly used policy evaluation typologies and they are: process evaluation, outcome evaluation, impact evaluation, and cost-benefit analysis.

    Types of Policy Evaluation

    • Process Evaluation
    • Outcome Evaluation
    • Impact Evaluation
    • Cost-Benefit Evaluation

    (Theodoulou and Kofinis, 2004, pp. 193-194)

    Process Evaluation:

    As its name implies this type of evaluation analyzes how well a policy or program is being administered. This type of evaluation is employed more often by program managers to determine what can be done to improve the implementation, the aspects of service delivery, of the program. It does not directly address whether or not the policy or program is achieving the desired outcome or impact on the target population.

    Types of Policy Evaluation: Process Evaluation

    • Determine why a program or policy is performing at current levels.
    • Identify any problems.
    • Develop solutions to the problems.
    • Improve program performance by recommending how solutions should be implemented and evaluated once carried out.

    (Theodoulou and Kofinis, 2004, pp. 193-194)

    Outcome Evaluation:

    Theodoulou and Kofinis somewhat confuse the concepts of outputs and outcomes in their discussion of outcome evaluation. Outputs are measures of government activity such as the number of tax returns processed or the number of social security checks sent out each month. Whereas outcomes are normally considered to be the impact that a policy has on a target population, for example, did the policy produce the desired behavioral change initially sought. In this case, the authors state that outcome evaluation is concerned with outputs. For example, if the stated goal of a welfare policy is to reduce the number of people receiving welfare benefits then a determination is made to see if less people are receiving welfare benefits after program implementation then before. However, what this type of evaluation does not indicate is what happened to the people who used to receive the welfare benefits and who have been forced off the system because they are no longer eligible. Did they find employment? Did they find other means of charity? Have they migrated to a life of crime? Outcome evaluation as described by Theodoulou and Kofinis focuses more on the readily available and tangible results of policy. The actual impact of the policy is the subject of the next type of evaluation.

    Types of Policy Evaluation: Outcome Evaluation

    • Legislative intent
    • Program goals
    • Program elements and indicators
    • Measures of indicators
    • Program outcomes (positive or negative)

    (Theodoulou and Kofinis, 2004, p. 194)

    page 9 of 17

    Impact Evaluation:

    This type of evaluation is what is more commonly perceived as a policy evaluation. The objective of this type of evaluation is to determine whether or not a given public policy or program is in fact achieving the intended impact as visualized by the various policy actors who either supported or opposed the given policy. Using the welfare example once again as discussed above this type of evaluation would answer the question what happened to the people who used to receive the welfare benefits and who have been forced off the system because they are no longer eligible. Did they find employment? Did they find other means of charity? Have they migrated to a life of crime? �In comparison [with outcome evaluation], impact evaluation is concerned with assessing whether the target population is being affected in any way by the introduction and implementation of the policy. There is also concern with the impact of the program on the original problem being addressed�for it is important for both [policy�level managers and policy designers] to ascertain whether target populations are appropriately receiving delivery of a program (Theodoulou and Kofinis, 2004, 194).�

    Types of Policy Evaluation: Impact Evaluation

    • Theoretical goals of the program or policy.
    • The actual goals.
    • Program or policy objectives.
    • Program or policy results and whether they are intended or unintended, positive or negative, in effect.

    (Theodoulou and Kofinis, 2004, p. 194)

    Cost-Benefit Analysis:

    Simply stated, a cost-benefit analysis is the comparison of the costs associated with a policy or program to the benefits generated by the policy. Continuing with the welfare example cited above in the two previous definitions, the tangible cost of a new welfare policy with a goal to reduce the number of welfare recipients could be accurately evaluated to include agency operating costs and the actual monetary cost of the benefits that are provided to welfare recipients. However, the cost-benefit analysis technique is controversial because it is extremely difficult to calculate the intangible costs such as those borne by those individuals who are no longer eligible to receive welfare benefits as a result of the new policy to restrict the number of people on the welfare roles. Additionally, the intangible costs borne by society such as the potential for increased crime or increased public health care costs or the long-term loss of employment opportunity and participation in the market place by those individuals who may have been able to pull themselves out of poverty if they were still eligible for welfare assistance are extremely difficult to calculate. All too often the cost-benefit analysis technique is used because actual real-world costs are easy to obtain, quantify, evaluate, and contrast against a variety of metrics or other policies or programs. Unfortunately, many intangible benefits, such as the advantages gleaned by a well-educated society, may not be readily visible for many years to come, and some intangible benefits are impossible to quantify such as the quality of life. Policy evaluators must constantly be aware that the costs and benefits used in any evaluation may not accurately, if at all, represent the real impact of a given policy or program. Instead, a cost-benefit analysis should be employed as one of several methods used to determine the efficacy or efficiency of government action.

    Types of Policy Evaluation: Cost-Benefit Analysis

    • A method with which to evaluate and assess the effectiveness of a policy's costs, benefits, and outcomes
    • For certain types of programs, such as education or the environment, one could argue that the real benefits do not materialize for years or decades.

    (Theodoulou and Kofinis, 2004, p. 194)

    page 10 of 17

    Seven Important Barriers to Effective Policy Evaluation

    �As increasing pressures are brought to bear on the public sector to perform its role more effectively and efficiently, evaluation will probably become an even greater source of conflict. Negative evaluations of a program�s effectiveness and efficiency now will be more likely to lead to the program�s termination than in more affluent times. The content of an evaluation, the values that are contained in it, and event he organization performing the evaluation will all affect the final assessment. Evaluation research is now a major industry involving numerous consulting firms (�beltway bandits�), universities, and organizations within government itself. These evaluative organizations will have their own perspectives on what is right and wrong in policy and will bring those values with them when they perform an analysis (Peters, 2007, p. 175).�

    As such, Peters identifies seven important barriers to effective policy evaluation that can impede the seemingly simple but operationally difficult process of determining what actually occurred as a result of government action and more specifically determining the level of performance of a public policy of program. It is equally important for both policy evaluators and the consumers of policy evaluation to understand the basic nature of these difficulties. Specifically, anyone engaged in policy evaluation must appreciate the degree of bias and unintentional, albeit sometimes intentional, confusion that can occur as a result of the difference between what is being measured during a policy evaluation and what policy-makers thought the policy should achieve. All too often evaluators, administrators, and the public focus on measurement statistics that are easy to obtain but have no real relationship between what has been accomplished compared to what the original intent of the public policy was. The following text block identifies the seven barriers as enumerated by Peters and it is followed by a brief description of each.

    Barriers to Effective Policy Evaluation

    1. Goal Specification and Goal Change
    2. Measurement
    3. Targets
    4. Efficiency and Effectiveness
    5. Values and Evaluation
    6. Politics
    7. Increasing Requirements for Evaluation

    (Theodoulou and Kofinis, 2004, pp. 163-174)

    1. Goal Specification and Goal Change:

    As outlined in Lessons 3 and 4, problem identification and policy formulation are extremely difficult for a variety of reasons. Not the least of which include political bias, incomplete information (bounded rationality), misunderstanding of the nature of the problem, individual value bias (social construct theory), plurality of competing interests, and inadequate or inappropriate understanding of potential policy alternatives. As a result of these significant limitations, adopted policy is frequently comprised of vague language, ambiguous, contradictory, impractical, or impossible policy goals. Consequently, the administrative agency tasked with policy implementation may need to fill in the gaps, which may or may not coincide with the intentions of policy-makers. Finally, lawmakers may not provide sufficient if any resources to adequately support a policy initiative. This requires bureaucrats to make programmatic decisions, which will unequivocally impact the efficiency and effectiveness of any policy or program.

    2. Measurement:

    Once policy goals have theoretically been identified, communicated, and programs implemented then some type of measurement instrument must be developed to ascertain the extent to which the goals have been achieved. However, most public problems such as national defense, education, poverty, health care, crime, urban and highway planning, and environmental policy are comprised of policy goals that are extremely difficult to measure directly. Consequently, many surrogate metrics are used to circumscribe the level of effectiveness of programs. Frequently these metrics evaluate outputs such as arrest or conviction rates for example because it is nearly impossible to determine if the �right� criminal was arrested and convicted, which of course is the goal. If we use the welfare reform example once again we can see that the mere reduction of the number of people receiving welfare benefits because of new program limitations does not mean that poverty has been eliminated. It simply means that less people are eligible to receive benefits and less people are receiving them. Those individuals who are no longer eligible to receive benefits may very well still be living in poverty and may even been worse off then before because they no longer have government assistance.

    Many other factors can also significantly impair the adequate and appropriate measurement of program and policy goals. For example, the time span that many policies require for their full impact to be felt can be a major problem when elected officials or other policy actors are looking for quick answers to support their own policy positions. The awareness of the time span problem often results in the formulation of public policy that will generate quick and measurable results in the short-term but do not necessarily address the real problem over the long haul. Other measurement problems include the inability to adequately isolate contributing variables of a major public issue such as health care. For example, in poverty stricken areas poor health may be the result of many factors such as poor nutrition, inadequate housing, minimal education, or poor sanitation in addition to a lack of access to quality health care. Therefore, public programs that simply provide some minimal level of access to medical care may have little impact on the overall health of a community because of these other present and contributing factors. However, measurement statistics will frequently focus on the increased availability to health care providers or the number of patient contacts, etc. This data may be valid statistics but it alone does not evaluate the level of success of eliminating the real problem of poor health in a particular community. The important point to remember is that voluminous measurement and statistical data does not in and of itself prove anything if the measurement instruments are evaluating indicators that are not directly related to the problem at hand.

    3. Targets:

    The target population whose behavior is the object of policy action is in many cases as difficult to identify and evaluate as is identifying the problem and formulating policy in the first place. �Programs that have significant effects on the population as a whole may not have the desired effects on the more specific target population. For example, the Medicare program was intended, in part, to benefit less-affluent older people, although all the elderly are eligible for it. However, although the health of the elderly population in general has improved�the health of the neediest elderly has not improved commensurately. And as the program has been implemented, substantial coinsurance has been required, along with substantial deductibles if the insured enters a hospital, so that it is difficult for the neediest elderly citizens to participate (Peters, 2007, p. 170).� In many cases, the need to achieve political feasibility during the policy formulation and legitimation stages of the policy process results in the approval of expanded eligibility for benefits beyond the target population as indicated in the Medicare example just cited. Therefore the remedial intent of the initial policy is diffused amongst a larger group of recipients diluting the ultimate impact of the policy, and making its evaluation more complicated.

    page 11 of 17

    4. Efficiency and Effectiveness:

    The discussion of these two concepts was outlined is Lesson 1 and is provided again here as a review.

    The concept of effectiveness can be seen as an emphasis on whether or not a given policy was able to get the job done. The primary question asked when attempting to determine an evaluation of effectiveness is: Did a given policy achieve its stated objectives? Theodoulou and Kofinis believe that the concept of �effectiveness helps define whether a specific program or service is achieving the intended effect and degree of change in behavior sought by the public policy (2004, p. 17).� Of course a fundamental assumption underlying the concept of effectiveness is that the stated objectives of a given public policy actually address the problem for which the policy was created. On the surface this assumption seems obvious, however many problems especially social problems �have no definitive formulation and hence no agreed-upon criteria to tell when a solution has been found; the choice of a definition of a problem, in fact typically determines its �solution� (Harmon and Mayer, 1986, p. 9).� Many problems faced by institutional policy actors are social in nature and clear cause and effect relationships are difficult to determine and more often than not vehemently disputed. For example, poverty, crime, education, civil rights, immigration, and the environment are just a few of the multitude of policy areas where vigorous and often rancorous debate over the nature of problem identification and policy alternatives is routine. While the academic definition of effectiveness appears to be fairly straightforward and unambiguous, its application can be anything but unequivocal.

    Efficiency, on the other hand, �can be understood in terms of whether a government program or service is operating at the most optimal level in terms of resources � such as time, dollars, or human resources. Put differently, efficiency highlights the importance of whether programs and services are wasting resources (Theodoulou and Kofinis, 2004, p. 17).� Whether in one�s personal life, or in the business world, or in government all resources are finite, consequently, it is a normative desire to minimize waste and to get the most benefit from the expenditure of those limited resources.

    While efficiency and effectiveness represent two separate and distinct evaluation criterion they are not mutually exclusive. For example, a U.S. GAO report discussing Foreign Food Aid indicates that inadequately planned food and transportation procurement procedures resulted in increased logistics costs thereby reducing the quantity, quality, and timeliness of delivery of food aid to stricken areas throughout the world (U.S. GAO, Oct. 2007). Unfortunately, this is an excellent example of how inefficient procedures reduce the amount of intended aid by increasing the cost of service delivery. Additionally, the GAO report indicates that aid workers have experienced difficulty in effectively delivering the food aid that they do receive via the inefficient delivery system. For example, in many cases throughout the world the target population can be found in war torn areas such as Darfur, Sudan, where 460,000 refugees were unable to receive food aid in July 2006 due to excessive violence. This is a case where U.S. humanitarian organizations were ineffective in the attainment of their goals to deliver even the limited food aid that was available to those in need.

    5. Values and Evaluation:

    What an individual experiences throughout a lifetime of perceptions and experiences creates a strong set of values and beliefs about the world around them. However, this awareness is tempered by limited and incomplete knowledge, but this limitation is frequently unacknowledged by most policy actors throughout the policy-making process. Additionally, value systems are not consistent between individuals, even between individuals who have similar experiences and perceptions. What this means is that each evaluator and reviewer filters evaluation data through their own personal set of values. Organizational members and members of the various professions share similar values all of which can come into conflict with each other and with the values of the client and the public. �Assessing a policy is thus not a simple matter of relating a set of known facts about outcomes to a given set of values. As in all aspects of the policy process, the values themselves may be the major source of conflict, and rational argumentation and policy analysis merely the ammunition (Peters, 2007, p. 173).�

    6. Politics:

    A significant obstacle to the fair and unbiased evaluation of any policy or program is the political context within which the evaluation is conducted. There will always be political interests who support the findings of an evaluation and those who oppose them regardless of how objectively accurate the data may be. Those who support a given policy or program tend to be supportive of positive evaluations and contest negative findings while those who oppose a specific policy or program tend to endorse negative evaluation results and dispute positive findings. Additionally, many evaluations are performed on short notice, or are undertaken for more ulterior motives such as to provide �empirical� evidence lending support for decisions that have already been made. The underlying consideration with this evaluation impediment is the realization that frequently evaluations are conducted for many reasons other than the basic altruistic purpose of ensuring the effective and efficient execution of policy goals.

    7. Increasing Requirements for Evaluation:

    The increased emphasis for greater government performance and accountability, which has escalated significantly over the past 20 years, has concurrently intensified the demand for program and policy evaluation. Unfortunately, this increased demand for �objective� evaluation data reinforces many of the evaluation impediments highlighted above. For example, the testing requirements of the �No Child Left Behind Act (NCLB)� have generated considerable debate as to whether or not the test scores required by the program actually represent any indication of improved teaching standards and performance or whether they are even correlated to any increased level of learning on the part of the students. Many opponents of the testing requirements of NCLB state that the educational concentration of the school year is now narrowly focused on the three tested subjects at the expense of other important educational programs such as science, philosophy, history, and the arts. These opponents also emphasize the old adage that states: �what gets measured, gets done.� The point of course is this, just because some output variable is being measured does not mean that the variable has any relationship to the root cause of the public problem or that it is even an accurate representation of program success. In the case of the NCLB the real problem is not poor test scores, it is less than desired intellectual prowess. In fact, some opponents of the NCLB believe that through the limited emphasis on reading, writing, and arithmetic skills, that the NCLB program may in fact be degrading the cognitive capability of future generations of American citizens.

    page 12 of 17

    Essential Activities in the Evaluation Process

    Essential Activities in the Evaluation Process

    • Identification of goals and objectives of the program or policy to make measurement possible.
    • Comprehension of the mission statement or noting the absence of one.
    • Construction of an analytic model of what hte program or policy is expected to achieve; this includes a set of theoretical propositions about means-ends relatinoships.
    • Development of a research design to distinguish program or policy goals from what is actually achieved.
    • Collection of data or actual measurement.
    • Analysis and interpretation of data.

    (Theodoulou and Kofinis, 2004, p. 198)