Wherever "averages" are considered for use in criteria, consideration should be given to the possibly important information that such criteria hide.

(c) In many instances it will be desirable to compare the magnitudes for the criteria with the magnitudes existing in other, similar jurisidictions, both the current magnitudes and those estimated for the future.

For example, local crime rates may be compared with those of other parts of the country (perhaps by using the FBI's uniform crime reports).

The relative conditions, such as displayed by the ratios of the jurisdiction's own crime rates to those of the Nation, or some segment of it, could be used as criteria. It may also be of interest to compare health, education, recreation, unemployment, and housing conditions to conditions elsewhere.

Care should be taken to ascertain that the figures are really comparable since definitions and reporting systems can differ substantially. For example, the crime reports referred to above have been criticized for lack of uniformity.

Too much concentration on "what the other fellow is doing" is not desirable; the absolute forms of the criteria (for example, the total amount of crime in the government's own jurisdiction) should not be neglected."

The list of criteria given in section III does not specifically include comparisons with other jurisdictions. As appropriate, the criteria could readily be modified to reflect such comparisons.

(d) Certain criteria can be displayed either as the "total number" of something or as a "reduction (or increase) in the number" of this thing. For example, "total number of accidents from cause X" could also be shown as "reduction of the total number of accidents from cause X." The use of the term "reduction" implies that there is a base from which the alternatives are measured. When alternative courses of action are being compared, the "reduction" is simply the difference between the base and the number resulting from the alternative.

The "reduction" form is the more direct way of showing effects but does not indicate the level still existing. In the list of criteria in section III both forms are sometimes shown.

(e) Certain of the individual criteria might be combined in various ways to form a new, single criterion. For example, for health programs the "number of sick days" might be multiplied by the severity index (if there is one) to give a "severity-sick-days index." This procedure is sometimes followed in order to reduce the number of criteria for analytical simplification. The list of criteria in section III does not include examples of these combined criteria.

It is also to be noted that the mere fact that the projected magnitudes for a criterion indicate a retrogressing situation (either relative to other jursidictions or even relative to earlier years within the jurisdiction) does not in itself necessarily indicate that the jurisdiction's programs are poor. External conditions outside the control of the jurisdiction (such as a significant shift in the characteristics of the population due to immigration or the entry of a new disease virus from outside) can cause the retrogression. Selection of program alternatives should be made as to which alternative is best relative to the others; i.e., which minimizes the adverse situation, even though none of the alternatives is estimated to cause an absolute improvement in the conditions.

80-976-67- -4

9. Estimates of the criteria magnitudes are needed for each year of the plan

Another aspect of the criteria problem arises from the necessity in program analysis to consider program impacts on each year for several years in the future. Though various pressures usually act to emphasize current and near future needs, good governmental planning obviously requires consideration of the longer range needs. In preparing its plan of action, a government needs to assure that the plan would provide desired goods and services in each year of the plan. Different alternative courses of action will affect different years in different ways. One mix of programs may, for example, result in greater benefits for the near future, while another mix of programs might emphasize current investments that are expected to produce superior benefits in later years. Therefore, in deciding among courses of actions the magnitude of each criterion for each year is an important consideration.

The weighting of the importance of each particular year of the plan will probably be the province of the decisionmakers rather than the analysts. The main job of the analysts will be to provide as complete and accurate information as possible as to the nature and phasing of the program impacts, leaving it to the decisionmakers' judgments for the final weightings of one year versus another.

10. A monetary criterion is always needed

The one common criterion in all problems of choosing among alternative programs is the monetary (i.e., dollar) effects of each alternative. This criterion is not repeated for each major program area in the list of criteria given below, but should be assumed to be pertinent in each case. Primarily nonmonetary criteria are included in the list. In a few cases a monetary criterion seemed to be necessary as a proxy to reflect important social factors, and these are included in the list.

The term "monetary criterion" as used here refers to the actual dollar changes that would occur (for each alternative program mix as compared to some base)-but not including dollar values imputed to nonmonetary things. These dollar changes, whether affecting the Government's own financial picture or that of other sectors of the economy, should be considered in the evaluation. Effects on the various sectors, as well as on the various clientele groups, should be identified separately so that the decisionmakers have a clear perspective of the impacts.

Theoretically, all of the nonmonetary criteria listed below could be translated into dollar values by estimating, in some manner, the dollar "worth" to the government (or to some other specified group) of changes in the magnitudes of each of the criteria. For example, it might be estimated that the population of the jurisdiction would be willing to pay a dollars to reduce the number of criminal homicides per year from Y to Z. It is, however, a premise of this paper that such translations present some almost insurmountable obstacles (at least with the current state of the art of program analysis) and at best will represent the judgment of one limited group of persons at one point in time. Therefore, it is always desirable to display the values for the specific nondollar criteria, such as those listed in section III so that the actual decisionmakers have full information with which

to make their own judgments. Information that attempts to estimate the dollar "value" to specified targets groups for changes in the nondollar criteria can also be presented to the decisionmakers if the analysts believe such information to be useful. For those who insist upon translating all units into dollar terms, the list of criteria might at least be a guide to the major factors to which dollar values have to be attached.

11. The monetary criteria can be very complex

Before we leave the subject of the monetary criteria as applied to program evaluation, the complexity of these criteria should be noted. The monetary effects of a program alternative can be of many types. These effects include the following elements (note that the term "cost" refers only to dollar costs):

(a) Program costs. These are the governmental costs that are incurred in undertaking the activities called for by the program. These costs include the various administrative and other support-type costs as well as those directly incurred.

(b) Program monetary effects within the Government.-As a result of the activities called for by the program, certain Government costs may be increased or decreased. For example, a slum-clearance program might in future years result in reductions in fire and crime protection services for the cleared area; on the other hand it might lead to increased demand for park and recreation services. A slum-clearance program would also have some effect upon the tax base of the jurisdiction.

(c) Program monetary effects outside the government.-Many of the monetary effects of governmental programs will occur outside the jurisdiction, perhaps affecting the private sector of the economy or other jurisdictions. Changes in transportation systems or in housing, for example, will have considerable effect on many types of businesses in the area. Such effects may be important in many kinds of studies. Governments are generally interested, for example, in monetary measures of gross business and income in relation to persons and businesses within their jurisdictions. Specific examples of such economic measures include: manufacturing value added, retail and wholesale sales, amount of bank deposits, and industrial capital expenditures. (However, it should also be recognized that too much emphasis can be placed upon such measures. For example, attracting businesses into the area though increasing total sales and total earnings, could also adversely affect the overall physical attractiveness of the community.) Another example of effects upon the private sector is the effect upon insurance rates of illness, fire, and crime prevention programs.

• Occasionally, the criterion "added future earnings" is used to estimate the value of increasing life expectancy and the value of reducing illness. If this criterion is used alone or predominantly, the evaluation can be misleading. For example, elderly persons or others out of the labor market are at a significant disadvantage, as are housewives (depending upon the amount imputed as their "earnings"). The use of future earnings seems to imply that such factors as growth in "GNP" are the fundamental objectives whereas the value of merely increasing GNP (without, for example, considering per capita GNP and the standard of living) would not seem to be the critical issue in our current society. A very pertinent question is: How does the loss of the individual affect the individual and the remaining population. We know how it affects the individual-and the significant effect is not a monetary one. How it affects the remaining population is a very complex question. Individual population groups, such as those close to the decedents, the taxpayers, the insurance-paying public, the businessman who lose the decedents' spending power, etc., are each affected in different ways. For a more extensive discussion of these points see the Schnelling reference listed in the selected bibliography of this paper,

It is not desirable to add all of these dollar effects (i.e., (a) plus (b) plus (c)) to yield one overall monetary impact. The impact on each sector should be presented to avoid obscuring pertinent considerations.

It will not always be clear whether an item is a "program cost" or a "program monetary effect." (Other terms that have been used to distinguish these are "direct versus indirect" and "primary versus secondary.") However, the important thing is not the classification but the identification and consideration of these monetary effects if significant to the program at hand."

Another major problem in handling monetary changes is the time pattern associated with the cash flow. As discussed in "9", the time pattern is of importance to a government. The use of a "discount" (i.e., interest) rate to translate actual net dollar flow (after consideration of both in-flows or out-flows) into a single "present value" is frequently recommended. This discounting procedure has the advantage of

(a) Reducing the complexity of evaluation by replacing the several dollar figures (i.e., one for each year of the time period) by one number, the present value.

(b) Reflecting the time value of money in the sense that, in general, money this year is worth more than the same amount of money next year since potentially it can be put to work now and grow into a larger sum by next year.

Unfortunately, however, despite the well-grounded economic basis for discounting (as the procedure is commonly called), there are some difficulties and drawbacks such as the following:

(a) First, there is considerable disagreement over the appropriate discount rate to be used. The range usually debated appears to be 4 to 10 percent. The rate chosen can have a significant effect upon the results if, for example, the competing programs have major differences in their expenditure patterns. Nevertheless, uncertainty as to the appropriate rate is not sufficient reason to avoid discounting.

(b) A more important concern to governments is found in the practical constraints in their annual funding capabilities. Major fluctuations in revenue needs from one year to the next may present insurmountable difficulties. It seems, therefore, that whether or not discounting is deemed appropriate, the actual (i.e., unadjusted by the discount rate) time-phased dollar flows should be shown to the decisionmakers. In addition, the discounted present values of the alternatives can be provided (perhaps for more than one discount rate). This suggested procedure applies to monetary flows both inside and outside the Govern

ment.

In most cases, the use of discounting will probably be of secondary concern relative to the many other problems of program analysis.

Nevertheless, as a practical matter, governmental decisionmakers are likely to have strong time preferences as to funding requirements

? Arguments as to whether such cost reductions should be considered as an offset to total costs or as an addition to "benefits" is important if cost-benefit ratios are being used as the primary evaluation criterion; however, reliance on such ratios is not good practice. The question as to whether to consider such cost reductions as belonging on the cost or benefit side should not affect a decision.

Discounting is partly a substitute for the explicit consideration of all effects; that is, if the before-andafter sides of all facets of the economy were explicitly Included, this procedure would directly show the time value of the money, and advantage (b) would not apply.

of their jurisdictions and need to be shown the time-phased monetary implications of the alternative courses of actions.

As has already been noted, the purpose of this discussion has been to indicate the scope and complexity potentially involved with the monetary criterion. Most of the remainder of this paper is directed at nonmonetary criteria.

12. The criteria are not intended for use in organizational evaluations The criteria discussed in this paper are not intended for the purpose of measuring the efficiency of the administrative organizations of a government (such as the police or fire department). The measurement of day-to-day operational performance, though important, is not the subject of this paper. The criteria in this paper are intended for the purpose of evaluating proposed program alternatives, not of evaluating staffs' or departments' current operating efficiency. 13. Measurements of program size are also needed but not as evaluation criteria

Governments that install a formal planning-programing-budgeting system, in addition to undertaking individual program analyses, will probably also prepare a multiyear program and financial plan. One of the main parts of this plan is an "output plan," a presentation that indicates the estimated outputs obtainable from the program plan for each year covered by the plan. At first glance, it might appear that the outputs contained in these output plans should be the evaluation criteria, the measures of effectiveness, utilized in the program analyses such as those discussed and presented in this paper. In practice, however, the "outputs" contained in the formal multiyear plan will probably have to be somewhat different. They are more likely to be measurements that indicate the magnitude or size of the program rather than its effectiveness. "Effectiveness" is too intricate a subject to present simply as a string of numbers not accompanied by evaluative comments. However, certain common and fairly clearly understood effectiveness measures such as crime and accident rates would probably be appropriate and desirable for inclusion in the formal output plan. In any case, measurements indicating the magnitude of each program (for example, the number of persons treated in public hospitals, the number of miles of highway, the number of acres of playgrounds, etc.) will be information useful to readers of the government's formal multiyear program and financial plan.

This paper does not attempt to list the program-size measures that might be appropriate for use in a multiyear program and financial plan. 14. Criteria for government-citizen relations may be desirable

For many of the services which a government provides to its citizens, the pleasantness, courtesy, quietness (e.g., in the case of waste collection), etc., involved in the provision of the service are factors in the overall quality of the service. To some extent these factors are more a problem of operational performance than of pro

Though some of the criteria discussed in this paper probably could be used in the measurement of organizational performance, they would seldom, by themselves be adequate for that purpose. The criteria in this paper are probably too aggregative; seldom will one government department or agency have full control over these criteria. More specific and more directly related criteria (for example, the average time that it takes to get the firetrucks away from the station after an alarm is received, the number of public assistance cases handled per caseworker per month, etc.) are needed to measure organizational efficiency. Such measures are useful in program analysis as planning factors from which estimates of overall program costs and effectiveness are built up, but are not the fundamental criteria sought for program evaluation purposes.