seem that for State and local governments such issues, though of considerable interest, are peripheral to these governments' functions. State and local governments, however, must, of course, be concerned with the notions of individual liberty, privacy, freedom of choice, and democratic processes. The degree to which each program option may impinge upon these individual rights and processes should, of course, be considered in a complete evaluation.

2. There are different "levels" of criteria

One of the major difficulties in specifying criteria is that there are many different levels of criteria. The specific criteria that are appropriate will depend upon the specific problem at hand.

At the highest level we might say that all government programs aim at contributing "to the maintenance and improvement of the well-being of humanity." This overall objective is too general; it is very difficult to measure, is vague, and is not very useful for analysis. The objectives and related criteria presented in the list in section III are at a lower level. However, they are intended to provide the major criteria that should preferably be used in governmental program analyses. These criteria may still be at too high a level for many problems.

Thus, for example, if we are concerned with examining the desirable size and nature of public health nursing services, it is likely to be very difficult to relate some of these services directly to mortality rates, morbidity rates, or days of restricted activity. Preferably, estimates would be made of the effect of alternative levels and mixes of public health nursing services and other types of health service alternatives on each of these criteria. However, because of the difficulty in linking the nursing service programs to these criteria, it may be necessary to use some "indirect," "proxy," or "substitute" criteria. One expedient might be simply to estimate the caseload that can be handled by each public health nursing service program proposed.

Another example: For the objective, "to prevent (deter) crime," judging accused persons can be considered one of the pertinent types of activity. "Judging" itself can be said to have the following subobjectives:

(a) To be fair.

(b) To be swift.

(c) For the guilty, to provide appropriate sentence (neither excessive nor overly lenient).

Alternative programs for "judging" could each be compared through criteria that reflected these subobjectives. Nevertheless, the crucial question would remain as to what extent meeting these criteria to various degrees would deter crime.

Such subcriteria as are indicated in these two examples are not included in the list in section III, but may often be necessary for individual analyses.

Unfortunately, "program size" indicators such as discussed in the public health nursing example (i.e., caseload) tell little about the important effects, e.g., the effects upon community health that the program achieves. Presentation of only this information to the decisionmakers leaves it completely to the decisionmakers to make subjective judgments as to the effects of the service. Presentation of the costs

and the program size indicator for each alternative is better than nothing, but leaves much to be desired.

It should be recognized that, in most cases, at least some information. can be obtained relating programs to the major criteria.

For example, it may well be possible to examine current and past records of the jurisdiction and other jurisdictions and to relate to some extent the more fundamental health criteria to program size; inferences would then be made as to the probable future effects of the newly proposed programs. A second approach is to conduct experiments (controlled as much as is practical) in which characteristics other than those investigated are similar from one group to another. Pertinent information would be kept about these groups, and inferences subsequently would be drawn as to the effects of the program characteristics.

Such information gathering does, of course, cost money. Also, the experimental approach may take a long time before useful results become available-possibly too long for the immediate problem but still useful if similar problems are expected to be of concern when the results do become available. In the absence of analytical techniques that identify the best approach to given objectives, the jurisdiction probably can afford to (and indeed may have to) experiment

to some extent.

The point is that the program analysis should not be quick to accept lower level criteria such as program size indicators as the only criteria on which he can obtain information.

3. Criteria are grouped under seven major program areas

The criteria presented in section III are grouped under each of seven "major program areas":

(a) Personal safety.

(b) Health.

(c) Intellectual development and personal enrichment.

(d) Satisfactory home and community environment.

(e) Economic satisfaction and satisfactory work opportunities. (f) Satisfactory leisure-time opportunities.

(g) Together, these major program areas are intended to encompass the great majority of the activities of a governmental jurisdiction. Though many such classifications could be made, these appear to be a reasonable set for discussion of criteria for evaluation of governmental programs. The assumed components of these major program areas are shown in the appendix.

Transportation-communication-location.

Many, if not most, analyses will at least initially concentrate upon but one part of one of these major program areas. In some of these analyses it may be necessary to utilize lower level criteria.

For example, for an issue raised on manpower training programs the criterion "percent of enrollees satisfactorily completing the training program" might be appropriate. However, as is already noted in "2", unqualified use of such a lower level criterion for program selection could be misleading. The more fundamental problem of government relevant to manpower training is to get unemployed (or underemployed) persons satisfactorily employed and self-sufficient. The mere fact of graduation from a training program does not mean reduced unemployment. Employment

and earning criteria, even if not feasible to use directly as criteria, should be recognized as being more truly the objectives of manpower training.

Most program-oriented categorizations of governmental programs (called program structures in PPB systems) will also contain a major category for general government activities. This will include such activities as the government's financial, legal, and legislative activities. No criteria are included in the list for these activities. The viewpoint of this paper is that these general government activities are primarily supporting services to the other, primary, government functions. That is, these activities are not themselves aimed at achieving fundamental governmental purposes.2

4. More than one criterion will frequently be needed for individual problems

For each of the seven major program areas, several criteria are listed. In some cases there is some overlap and redundancy. However, for the most part, each of the criteria contains some potentially important aspect that is not contained in the other criteria. As has been already noted, the evaluation of program alternatives would be eased considerably if all criteria were commensurable, i.e., expressed in some common unit such as "dollars." However, practically speaking, few major program issues can be meaningfully evaluated solely in terms of a single criterion. The analysts should concentrate upon providing as full and accurate information as possible as to the effects of each program alternative on each of the criteria, leaving it to the decisionmakers to weight the criteria.1

5. Interactions occur among program areas and

among criteria

Though the list of criteria is divided into major program areas, this is not meant to imply that all program analysis problems will necessarily fall into one major program area, and only one. On the contrary, major governmental problems will frequently spill over into more than one program area.

For example, mass transit system proposals could have significant impact on many if not all of the listed major program areas: Traffic safety is directly affected by the substitution of a mass-transit system for individual automobiles; an inexpensive mass-transit system might permit low-income workers to consider job opportunities further away than he can currently afford; families who wished to live further out in the country might be able to do so with a convenient, rapid, inexpensive transit system; recreational opportunities previously too far away and too expensive to reach might be opened to certain segments of the public; individual health and intellectual development might be

2 The point, however, can be made that these activities do contribute to the fundamental function of providing "democracy." 3 However, frequently it may be reasonable to concentrate the analysis on one key, nonmonetary criterion, and treat the other criteria as study constraints or as relatively minor considerations. Even in these instances, however, two criteria, one monetary and the other nonmonetary, will need to be explicitly evalu ated. As is indicated in the introduction to this paper, forcing a dollar value on a nonmonetary criterion does not in general seem a good practice.

A technique occasionally used with multiple criteria is to have experts in the specific field estimate the relative weights of each criterion. By applying the prechosen weights, the multiple criteria can be com bined into one index thus permitting a ranking on the same scale of all the alternatives. As with attempts to translate all nonmonetary criteria into monetary units, such a practice can too easily be misleading. It is the author's belief that if the analysts believe that the resulting information is meaningful, it may be provided to the decisionmaker, but the basic information as to each program alternative's effect upon each of the individual criteria before any weights are applied should always be provided so that meaningful information is not obscured.

furthered (indirectly) by the combination of the preceding effects; certain penalties could also occur, such as the transit system having an adverse effect on the physical attractiveness and living conditions of the areas where it is constructed; air pollution and noise effects would also occur.

Another example is that of education programs that in addition to contributing to individual intellectual development also lead to improved employability and reduction in unemployment. Thus, specific programs may simultaneously have many complex and interacting effects on many program areas and many criteria.

It is important in program analyses to attempt to consider and evaluate all such effects to the extent that they might be important to the decisionmaking process.

6. It will be necessary to distinguish "target groups"

An important aspect of program evaluation is the identification of the specific population groups that receive benefits (or penalties) from each program proposed. Though not specifically included in the list, it will often be appropriate to break down further certain of the criteria into subcriteria in order to distinguish specific clientele, or "target groups." For many issues a government will be interested in distinguishing the effects of alternative programs on specific population groups identified by such characteristics as age, sex, race, income, family size, education, occupation, geographical location, special handicaps, etc.

For example: For many health issues, distinctions by age, income level, family size, etc., may be required to evaluate the effects of various health programs on each category within such groups.

Another example: It will probably be necessary for many law-enforcement issues to distinguish crimes committed by adults from those by juveniles.

Though neither the objectives nor the illustrative criteria listed in section III explicitly single out "equal opportunity" objectives the use of target groups in the criteria will provide information on such objectives.

7. Criteria need to be thoroughly defined

No attempt is made in this paper to define the listed criteria. However, when utilizing criteria it is important to have clear, thorough definitions. In almost all cases, misinterpretations (often subtle ones) can occur if complete definitions are not provided.

For example, for major types of crimes it is necessary to define each type of crime, e.g., does "larceny" include thefts of automobiles and bicycles; does it include thefts of any magnitude or only those beyond a specific dollar value?

Another example: What is meant by "restricted activity" when the number of days of restricted activity for health reasons per person per year is to be estimated?

Again, how is "poverty" defined when the number of persons and families in the jurisdiction's "poverty population" is esti

mated?

Or, what is meant by "substandard" when dwelling units are evaluated?

Definitions should generally specify such things as who is involved, how, what time period is to be covered by the criteria, what geographi

cal location is included, etc. For example, for measuring restricted activity due to health reasons, it is necessary to know

(a) What specifically is meant by "restricted activity"?

(b) Whether the whole population of the jurisdiction is involved or some specific segment such as "all males between the ages of 16 and 21 living in the North Smithtown" section of the city.

(c) Whether the magnitudes are to be on a "per person per year" basis or on some other.

Some attention should also be given to the influence of time which may affect the definitions. For example, wherever a dollar figure is involved in defining a criterion, price-level changes over time may alter the meaning. For example, if "larceny" is defined to include only thefts over $50 at current price levels, price-level rises will automatically bring more thefts into the category even though there is no change in the total number of thefts of each type. Explicit provision will be needed for adjustments of the criterion, based upon price-level changes. Another type of change over time that may occur is change in the jurisdiction's boundaries, possibly requiring adjustments to make compatible the magnitudes assigned to the criteria for different

years.

8. Criteria can be expressed in different forms

Given that a certain factor is considered sufficiently important to be included as an evaluation criterion, there frequently will be a variety of forms in which the criteria can be expressed. Five such choices are noted below:

(a) Both "absolute" numbers and rates are called for by the criteria included in the list. Absolute numbers by themselves can present a misrepresentation of the situation. For example, the total number of various crimes or of traffic accidents, though, of course, of interest in themselves, do not reflect the associated levels of activity. Crime rates and traffic accident rates (the latter related to the volume of traffic) will give improved perspectives as to what is happening in those areas. Both forms are probably needed by the decisionmakers. (b) Some of the criteria listed below call for "averages"; for example, "average waiting time for the use of certain recreational facilities." In such cases, the analysts will frequently also need to consider the distribution of waiting times as well as the average. There is danger that if only the average is considered, important information may be ignored.

For example, the average waiting time throughout the week on a city's golf courses may be 15 minutes, which, if it were applicable at all times, would probably be quite acceptable to the city's golfers. However, the distribution of waiting times for specific times of the week might show prolonged, perhaps severalhour, waits during certain hours of the weekends, probably causing considerable annoyance among golfers and suggesting the need for corrective action. Use of only the overall average would hide the pertinent information.

In the list in section III the dangers of the use of averages for waiting times of recreational facilities have been reduced considerably by requiring the calculation of the averages for specific key periods.