Search

The components of assessment centers are described and evaluative research is reviewed. Predictive validity of assessment centers appears promising, especially for management positions, although the research comes from too few sources and has some confounding methodological problems. Various alleged outcomes of using centers are found to be largely unsupported. [ABSTRACT FROM AUTHOR]

A study of professional employees in two large treatment-oriented organizations reveals a contradiction between the avowed objectives of program research and the acceptance of its findings by those engaged in the program. The ideology of science, which favors the interpretation and communication of research results, often conflicts with organizational mores. These mores favor symbolic research—the ritualistic avowal of the worthwhileness of research about agency problems, coupled with objections to its substance—the advancement of knowledge. Even those expressing a high priority for the use of agency funds or staff time to do research were reluctant to interpret or publish research findings. Interpretation readiness was greater for "encouraging findings" than for "discouraging findings." Only a minority expressed an attitude favorable to taking the risks which are believed to exist when information with evaluative implications is widely disseminated. [ABSTRACT FROM AUTHOR]

Evaluation research on changing social service systems has been plagued with problems attributed to a lack of the stability, required for rigorous evaluation designs. The authors present an alternative conceptualization scheme and model which permits evaluation to be done in such changing environments. The authors distinguish among sets, strategies, and programs and identify a time perspective in the use of evaluative criteria that focus on client relationships both within and outside programs. The model described should permit research teams to address system administration concerns while at the same time taking advantage of the natural changing setting for testing theoretical propositions. [ABSTRACT FROM AUTHOR]

More useful criminal justice evaluation research would differentiate offenses and offenders on the basis of causal theory and would interrelate several levels on abstraction. This is illustrated in correctional practice evaluation by a linkage of behavior modification, symbolic interactionist, and sociocultural diffusion theory, from which three propositions on the effectiveness of specific treatment methods for particular types of offenders are derived. Research thus far supports the validity of these propositions. Boards of autonomous criminologists and public representatives supervising criminal justice statistics and research agencies would foster more grounding of inquiries in policy-relevant theory. [ABSTRACT FROM AUTHOR]

In sum, a study planned to yield equivalent treatment groups and highly interpretable results, based on promises of cooperation from enlightened judges and police officers, was greatly compromised by a variety of failures of the decision-makers involved to follow the experimental prescriptions. The reasons for these failures appear to be that the weight of promises by top-level representatives of bureaucracies is considerably attenuated at the bottom of the organizational structure, absent strong control procedures, and that the day-to-day pressures of bureaucratic "clients"--even when these are law violators--are more weighty than those which researchers can bring to bear.
The main lesson to be gleaned from this experience is that to assure the realization of random or arbitrary procedures in research designs, choice must be eliminated from non-scientific personnel, no matter how well motivated they may appear to be. We have learned that gross departures eau take place from the simple stipulation that a given kind of sentence will be meted out in a given month. Even the clear and simple instruction that a clerk is to administer one procedure to forms bearing a certain last digit and a different procedure to those bearing a different digit could not be accomplished without bias. Our experience has led us to the following suggestions for other researchers planning experimental studies in the legal system and in other similar bureaucratic settings:. [ABSTRACT FROM AUTHOR]

The article describes a program in milieu therapy that the authors designed and operated between September 1970 and January 1972. It was carried out in the Kalamazoo State Hospital, Kalamazoo, Michigan, which had approximately 2,000 patients in 1970. The program aimed to reduce constraints and barriers within the hospital that discouraged discharge, and overcome external or community barriers to discharge. The overarching goal was to create a social structure involving patients and staff that would strengthen patients' motivation and increase their opportunities for leaving the hospital. No program of social intervention can be self-justifying. It was not possible to obtain adequate baseline data prior to intervention so that there might be a simple before-and-after evaluation. As is often the case in such situations, those with authority-the hospital administrators in this instance-were not willing to devote the time, energy, and money that rigorous evaluative research requires. Nevertheless, the soft data available indicate that the program has been successful in attaining the goals set forth for the patient population.

This article discusses methodological factors that affect the quality of evaluation research. At the most elemental level, evaluation research is aimed at determining whether a program was actually carried out. At a more complex level, research is concerned with the effectiveness or the economic attributes of a program. Ideally, such information should contribute to decisions whether to expand, curtail or modify a program. Evaluation research requires that at a minimum, comparison be made between one group that has been exposed to a program and other that has not. Failure to make such comparison logically preempts the researcher from attributing any change observed in the target population to exposure to the program. The use of control groups, randomized placement of individuals in control and experimental groups and various stages of pre-and post-testing of subjects are the basic components of experimental and quasi-experimental designs. Given the problem inherent in experimental designs, a number of alternatives to the experimental design have been suggested. The most viable of these alternatives involved the exploitation of natural conditions that enable the evaluators to approximate the controlled situation.

This paper addresses the question of internal validity in quasi-experimental and nonexperimental social policy research. We focus primary attention on problems of selection in research designs where assignment to experimental conditions occurs on a nonrandom basis. Five different solutions to the problems of selection are discussed-randomization, covariance adjustment, gain scores, matching, and explicit selection-and we examine the conditions under which these solutions are useful. The problem of random measurement error is discussed within the context of our elaboration of these techniques. [ABSTRACT FROM AUTHOR]

This paper delimits and explicates threats to external validity particularly problematic in evaluation research. Five categories of factors are discussed: selection effects, measurement effects, confounded treatment effects, situational effects, and effects due to differential mortality. The paper focuses on pointing up specific ways in which each of the factors threaten generalizability and possible solutions to the methodological problems presented. [ABSTRACT FROM AUTHOR]

Policy experiments represent a powerful tool for testing the consequences of contemplated social programs. However, experimentation in the policy field raises perplexing problems and issues, some of which are not commonly encountered in the literature of experimental social science. This paper examines several issues that seem particularly important and intriguing, drawing heavily on experiences in income maintenance experimentation, one of the largest efforts in policy experimentation to date. [ABSTRACT FROM AUTHOR]

This article discusses coupling randomized experiments and approximations to experiments in social program evaluation. Evaluating experiments are always a debatable issue in social sciences. It is often presumed that conducting experiments and using statistical approximations to experiments are independent rather than interdependent activities. The same presumption is implicit in most texts on quantitative methods of assessment. Similarly, much of the work focusing on approximations to experiments or on quasi-experiments usually provides no information about how to couple randomized experiments with the approximations. The result seems to lead field evaluators to believe that they must choose between an experiment or an approximation to an experiment in any particular evaluation study. The aim of this article is to examine the mutual exclusiveness of these two approaches and their principles. This can, suggestively, be done by means of designing a better program evaluations. The approximation method refers to any statistical technique which does not include the randomization feature, but which is purported to yield fair comparisons and quantification of comparisons. A variety of quasi-experimental or completely non-quantitative evaluation experiments are discussed and evaluated here.

This article deals with validity issues in evaluative research in social sciences. Evaluative research, as a form of social policy research, is expected to provide reliable and valid information useful for policy makers who operate in the world of action. This belief necessitates a variety of research strategies and considerations not ordinarily pertinent to more basic discipline research. Moreover, the political setting in which evaluation research is conducted demands special attention to the fit between methodological techniques and the research problem. Generally, one begins with a dependent variable and hypothesizes about factors that would predict that variable. To deduce hypothesized relationships, one examines theoretical statements about the phenomenon of interest. In evaluation research, the procedure is often reversed. Here one has to begin with an independent variable, the program, and is asked to assess how it affects a set of only vaguely identified goals. One needs to treat as problematic what the treatment is, or what the program inputs are. It is necessary, thus, to identify the program inputs, to assess the degree to which persons administering the treatments agree with that common definition of inputs, and to assess the degree to which the treatment is implemented for the target population in accordance with the definitions.

The national evaluation of an early elementary compensatory education program, Follow Through, is used as the context for discussion of design and analysis problems common to many evaluations. Although experiments have well-known advantages over quasi-experiments, the latter are likely to persist for the foreseeable future. It is argued that more careful planning could avoid many unnecessary weaknesses of past quasi-experiments, and that quasi-experiments can provide valuable information for evaluations. The strengths and weaknesses of several competing analysis strategies for quasi-experiments are considered. The conclusion is that no one strategy is satisfactory. A multiple analysis approach is recommended instead which attempts to capitalize on the unique strengths and control the unique weaknesses of several analysis strategies. [ABSTRACT FROM AUTHOR]

The article presents insight into a five volume journal "Technology and Culture." If history of technology is to be judged as an academic discipline, rather than a high-class sort of engine spotting, it is necessary for the study to proceed to a stage of interpretative and speculative evaluation which will relate these parts to each other and to general principles of organization that are relevant to its specific requirements. The article suggests some ways in which progress could be made towards developing the history of technology into a coherent and analytical discipline. Professor Edwin Layton contributes a third paper to the journal's symposium under the title "Technology as Knowledge," in which he criticizes the usual scholarly definitions of technology because they render it "subordinate to other types of social and intellectual activity" and reasserts the "thought component" in the technological process by stressing the importance of design, which leads him also to emphasize "the importance of social history for the history of technology."

Examines the integration of evaluative research and social planning roles. Concerns on the basic in-home services; Anticipation on the planning and community activities; Modification of evaluation strategy.

The article discusses implications of soaring state expenditures on social program accountability. Like the U.S. Federal Congress, the state legislatures need a mechanism independent of the executive branch to review and analyze existing programs. In addition to describing the current types of state legislative staffing committees employed in research and evaluation, the article points out constraints of executive review. Finally using Ohio as an example, the article offers a proposal to establish a State General Accounting Office based in the legislature.

The primary reason for the non-use of evaluative research is the basing of evaluations on a faulty model of the decision-making process to be affected. The article suggests that the manager requires a causal model based on analysis of the structure of his program, while evaluators tend to rely on models which ignore the causal structures of programs and monitor only inputs to and outputs from the program. The major gains to be had from evaluative research are in determining the effects of operational changes within existing programs.

This article focuses on guidelines for evaluation research. It is possible to separate out two different factors to be considered in designing an effective evaluation strategy. On the one hand, behavioral, political, structural, and methodological problems must be addressed. Monitoring becomes appropriate when there is little agreement upon what the objectives of the program are, where there is little understanding of the effect a program is likely to have or where the anticipated effects are so wide-ranging that only an arbitrary focus would be possible.

The article explores the principles of evaluation research. Evaluation is assessment of value. Value defines an individual or for a group of individuals what ends or means to an end are desirable. Applied to action programs, evaluation research refers to the procedures of fact finding concerning the degree to which a specific action program achieves specific results, both intended and unintended, and which elements in the situation, or in the methods used hamper or foster the process of change. In more practical terms, evaluation aims at providing a systematic and comprehensive measure of success and failure. Evaluation gives evidence how we can learn from the failures observed and how improved action programs in operation can be guided and controlled. In this paper, three dimensions of evaluation research are described. They are: Assessment for whom, recipients field workers and other agents of change, policy makers and the scientific forum. Each of these categories expects a different type of evaluation result; Assessment by whom, self-evaluation, both by the individual and by the local community or other social unit, evaluation by the responsible field workers and other community or social unit, evaluation by the responsible field workers and other professionals, evaluation by local and national superiors and again scientific evaluation. These categories have each their own standards of judgments; The last practical dimension deals with the time and the stage of evaluation. This time-dimension makes the conceptual frame of evaluation operational.

The article discusses the role of social research in action programs in the Philippines. Any program that gets beyond the committee stage is an action program. Action programs are designed to induce change in the attitudes, behavior, practices or organization of people in some target population. Precisely because of the uncertainty element, and because both time and funds are limited, adequate research should be built into action programs for purposes of both evaluation and more fundamental study. The research operation although is logically distinguishable from and contingent upon the action planning, research actually participates in important ways in most phases of the action. Since action is the focus of research, the planning of an action program is logical prior to that of research. Also full-scale experimental design should be emphasized wherever feasible. Action programs provide the greatest opportunity for a social scientist to contribute to current major problems, while continuing his research on such problems. The researcher should serve the policy-maker in a consultative capacity and should neither accept nor have thrust upon him the responsibility for action decisions.

The article evaluates available research on the effectiveness of specific correctional or preventive practices for criminals and delinquents. The author finds that none of the studies of correctional or preventive effectiveness can be described as adequate. The factor most commonly cited as interfering with the proper design and execution of research in corrections is the politics of correctional administrators and the vested interests that they and the personnel involved in the treatment process have in the evaluation of their programs. Moreover, most of the studies emphasize on the practical, as opposed to theoretical, usefulness of correctional research. The author argues that correctional research should no longer be portrayed as action-research, designed to test the effectiveness of various currently existing programs, but as basic theoretical research, designed to develop scientifically reliable knowledge that will be of chiefly theoretical value and only indirectly or eventually of any practical use.

The article cites a study of American Friends Service Committee, the first group to organize volunteer work camps in the U.S., in relation to social interaction. To collect data for this study, the author sent out questionnaires to individual associated with different camp groups to learn of their personal experiences and judgments regarding their camp activities. The author had also access to the files of American Friends Service Committee in Philadelphia from the summer of 1940 and collected magazines and other publications for added information. He enumerated the educational contribution of the work camps to the campers. The author determined that the vital result of the work camp technique decreases social distance and its cooperative nature is essential in promoting democratic living.

The article presents a study which aims to explore the potentialities of the indirect Kirkpatrick and compared it with a more direct scale, the Burgess Marriage Adjustment Form in assessing marital adjustment. The study reveals that Kirkpatrick scale is more reliable than Burgess Marital Adjustment Form since it provides a more uniform and normal distribution of scores. In addition, Kirkpatrick scale is less subject to the common intruding variables of simple, economical, easier to score and administer. Furthermore, it was concluded that the Kirkpatrick Scale of Family Interests warrants increased attention in the statistical and logical analysis of the data and found to be valuable in the study of human relationships.

The article analyzes and discusses policy evaluation and recidivism. It explains the general problems that are concerned with the evaluation of policy impact through non-experimental analysis and controlled experimentation. It is stated that the latter is an alternative response to some of the problems of evaluating policy impact. Similarly, the evaluation of alternative criminal court sentencing policies through non-experimental analysis and controlled random experimentation is described. It is stated that it is difficult to ascertain the precise degree to which criminal court sentencing decisions would affect recidivism rates. Experimental evaluation of Community Treatment Project is cited as an example with its flaws. It is stated that analysis of both the non-experimental and experimental data seem to indicate that those convicted individuals who are granted probation have lower recidivism rates than those who have been incarcerated. Through this study, an attempt has been made to offer more empirical guidance to decision makers and policy evaluation in terms of the goal of reduced recidivism. The study also indicates that factors other than court decisions also have a major impact on recidivism.

Research in the evaluation of field projects is a new subdivision of the social sciences employing techniques derived from statistics, history, anthropology, and econometrics. Its purpose is the objective measurement of change or growth, both the quantitative ‘aggregate’ type and the process of maturation or ‘structural’ growth. Concepts, techniques, principles, administrative problems, and machine aids are discussed. The paper refers frequently to the United Nations and its Specialized Agencies, which are at present the bodies most actively calling for evaluation specialists to guide their field staffs and Technical Assistance programs. [ABSTRACT FROM AUTHOR]

The article evaluates the exchange of persons programs. During the last five years, International Research Associates has carried out a number of evaluation studies of exchange programs in different countries, among students, specialists, and groups of civic and opinion leaders from abroad. In this paper, the authors discuss the research approaches utilized in those studies, point out the usefulness and shortcomings of each, as these are reflected in major findings, and suggest some of the areas in which future research seems most urgently needed. Today there are literally hundreds of organizations in the United States and other countries which sponsor or participate in exchange programs of different kinds. Communities, clubs, foundations, corporations, schools and individuals, as well as national governments, support the foreign exchange-of-persons movement. The influence of personality factors, the expectations and motivations of selectees, different approaches in pre-trip and post-trip orientation, variations in programming and length of stay all remain largely unexplored.

This article addresses the political incursions in the process of evaluation research. Researchers who undertake the evaluation of social action programs are engaged in enterprise fraught with hazards. They are beset by conceptual and methodological problems, problems of relationships, status and function, practical problems and problems of career and reward. There is also an increasing requirement to boost visibility and scope of evaluation research in recent years. The big change that is expected is that both programming and evaluation are now national in scope. Programs may actually become no more standardized in form, content and structure than they ever were, but they are funded from a common source and bear a common name, which is "community action program,".

The article highlights on the research centers and current studies on the commune system in Yugoslavia. Yugoslavia has a considerable number of institutions and organizations which carry on studies and research on the commune system and communal self-management. The Institute of Social Science, and the Federal Institute for Public Administration in Serbia, both in Belgrade, the Institute of Political Science of the Law Faculty of Belgrade University and the Institute of Social Management, in Zagreb are just few of them. Apart from the scientific research institutions, the Federal Executive Council and the Councils of the Federated Republics and their administrative bodies conduct studies on the commune system. The Legal and Political Sciences Department of the Belgrade Institute of Social Sciences has drawn up a research project dealing with the commune as a political and territorial unit for self-management and with the district considered as a political and territorial organization for self-management of associated communes. The Zagreb Institute for Social Management is also doing major research work in the field of communal organization. Proceeding from the principle that Yugoslavia's social system is based on the communes, the institute has made a study of the mechanism of communal management and of organized social activities within the commune in general. In addition to the research work relating to the Varazdin commune, a study group set up by the institute made an analysis during 1958 and 1959 of the organization and activities of producers' assemblies within people's assemblies in the communes of the People's Republic of Croatia.

Most counselors will agree that the profession is in trouble. Many will recognize that the reason we are in trouble is that we have not taken program accountability seriously. This is partially because graduate training for research extinguishes the desire to "find out" rather than nurturing our curiosity. Oetting and Hawkes focus on evaluation as opposed to research, showing that the skills and processes are not the same and that therefore the training should not be the same. Evaluation skills are necessary in all cells of the Cube; in fact, unless the, are operative, counseling may quickly become nondimensional. [ABSTRACT FROM AUTHOR]

Professor George Geiger has provided a very thoughtful framework for describing and clarifying the role of the social scientist in action-research projects. With some minor variations in usage, the term action-reached has continued to denote the activity of applying research methodology to the diagnosis and solution of practical problems of taking action. The author of this article attempts to examine some of the problems of the social scientist who is faced with the decision of whether or not to participate, and how to participate, in action-research with some action group in the society. The invitation probably comes from some group that wants to become more scientific in evaluating its program and making better policy decisions.

This article focuses on the evaluative research relating to curriculum components and organization in the U.S. After the standard compilations of educational research were surveyed, a tentative bibliography was compiled. This bibliography was supplemented by telephone conversations with persons actively engaged in major curriculum revision projects as a check on the literature survey. Standards were established by which each study in the tentative bibliography was judged. These standards included such criteria as the relevance of the problem to current curriculum development, the adequacy of the experimental design, and the appropriateness of the statistical treatment of the data.

This article focuses on the evaluation of the educational program in early and middle childhood and the increased emphasis on the importance of comprehensive evaluation of the general school program. Furthermore, attention was given to the aspects of evaluation related to procedures of reporting to parents, testing, grading, and promotion. Action-research procedures have maintained their place in school instruction and evaluation programs. Investigations of the relationships between various hypothesized predictors and teaching competency were extensive in spite of the many studies purporting to determine concomitants of teacher competency. Finally, many of the major contributions of evaluation were those of textbook writers, and recognition of the importance of adequate evaluation procedures was widespread.

This article has indicated that intelligence testing continued to hold first place in number of articles published within the scope of this summary despite enormous gains in the measurement of aptitude and personality. The bibliography of this chapter represents a high degree of both quantitative and qualitative selection in order to place emphasis on new developments in measurement. Evaluation has been criticized on the same grounds as those on which it has criticized measurement. Failure to supply clearcut data regarding the practices evaluated seriously weakens the scientific validity of the study.

The article discusses the introduction of an action research program in teacher education in the U.S. This article presents a survey over graduates of the Teacher Recruitment and Training Program directed at the San Francisco Unified School District, in cooperation with San Francisco State College. Reports from several schools participating in the program indicates varying degrees of success in developing a program in which practice is more closely articulated to theory. One contribution of the program is the continuous experience in public schools enabling the teacher-trainee to get a more realistic perception of education.

In response to an increasing concern about the low level of utilization of evaluative research findings in public programs, a mathematical model is suggested to predict the probable utilization of a report. The model is named Linked Indices for Assessing Relevance (LIAR). Because of its high predictive accuracy, based on well-established theories of organizational behavior, LIAR has multiple uses for management decisions, including the evaluation of an evaluation division. People will be aware that there is a growing interest in evaluative research on public programs; and that this research must contend with growing doubts about its relevance. A review of the recent literature suggests two schools of thought on the matter: pessimists who say that evaluations are likely to be ignored; and the deeper pessimists who insist that they are likely to be misused. The former school is still in the majority and professional evaluators have given considerable attention to attracting the attention of decision makers to their work. The head of a planning and evaluation division, like those he studies, must at some point account to superiors for his funds.

Focuses on the major themes in social science literature with direct implications for evaluation research. Methods used in evaluative research; Role of action programs as contrasted to external variables; Details on the categories of evaluation.

The article focuses on the more useful categorization of research in social sciences and suggests that each type of research has its associated diffusion system. It shows that the limitations of these diffusion systems condemn so called applied research to frustration. The classification owes much to that proposed by the Zuckerman Committee which offered the useful distinction between pure basic and objective basic research. In social sciences these two categories, together with a third, operational research, and a fourth, action research, will serve as pure types. Each of these four types has associated with it a typical diffusion channel. Diffusion channels, after all, consist of people. If the research is treated as a case study and reported in such a form that it can be discussed and evaluated along with other case studies, some generality is possible. As in the case of operational research, the methods used may have some generality and secondary diffusion channels may be used. In action research the creation and use of a diffusion channel is an essential part of the research. Utilization is built into the research design.

In the past several years it has not been uncommon for the caseworker to take on the responsibilities of research interviewer in addition to his regular social work duties. This has come about for a number of reasons and is mainly to be seen in the action-research type program. It is assumed, often without question, that the caseworker qua caseworker, by virtue of his training and experience in human relations, is admirably suited for the position of research interviewer. The argument of this paper is that he is not so qualified. This erroneous impression has created unnecessary problems in social work-research endeavors. The successful completion of the action-research program is heavily contingent upon the caseworker's ability and willingness to accept the dual role researcher and caseworker. The focus is on the caseworkers in the organization that found themselves with the prospect of accomplishing two diverse tasks casework and research. They were faced with the predicament of reconciling a basic job orientation to a system whose demands they saw as deviating from some rather fundamental tenets of their profession. For a successful action-research program the individual in the focal position must be equipped to act in the capacity of caseworker and researcher simultaneously and with equal effectiveness, but the two roles must remain independent to prevent contamination of one by the other.

The article questions the issue of selection of criteria of value in evaluation research. Evaluation research is explicitly concerned with assessing the quality of public programs. The findings of an evaluation study depend upon the criteria chosen, and the standards employed operationally, as much as they depend upon the workings of the program being evaluated. Public programs should not only work, they should work for the advantage of the community. Standard setters for the criteria of responsiveness and equity are not so easily identified. According to some theories of democracy the legislature should determine what is responsive and equitable public service. In order to include criteria of responsiveness and equity, it is necessary to allow the people being served to provide standards for evaluation. By refining the understanding of citizen satisfaction or dissatisfaction with various aspects of public life, one can more closely judge the responsiveness of particular public programs.

The article presents the proposal of the Committee on Education of the American Association on Mental Deficiency for a two-year action research project on the educable mentally retarded child in the U.S. The improvement of the education programs for the trainable mentally retarded child is necessary because: (1) the nature of change in population trends; and (2) the great interest by parents and other citizens in the development of community facilities. The development of educational program for the trainable mentally retarded child is in urgent need of sound development. The design and framework of the Action Research is presented.

Presents several letters to the editor referencing articles and topics discussed in previous issues. Views on joint interviews in casework; Focuses on differences in the practice and literature in social work; Comments on problems in the area of confidentiality in social work.

The article discusses the importance of local school system research to special education personnel in the U. S. Research helps special education personnel develops approaches necessary for the improvement of special education. The basic concepts of research techniques will enable school personnel to engage in action research in classrooms. Special educators needs to exert adequate efforts to find answers to the problems of educating exceptional children.

Presents a letter to the editor in response to the article "Control Groups and the Placebo Effect in Evaluative Research," published in the January 1962 issue of the journal "Social Work."