Evaluating Research Productivity
no. 105 - June 2001

First Panel of Research Administrators

EVALUATING RESEARCH PRODUCTIVITY OF INDIVIDUAL INVESTIGATORS:
CHAIRMEN'S PERSPECTIVES IN A MEDICAL CENTER

Thomas H. Rosenquist
Vice Chancellor for Research
University of Nebraska Medical Center

The overall success of the research enterprise of any institution depends critically upon the quality and success of its individual investigators. Thus, each institution must develop and implement a well-defined system for recruitment, retention and reward of excellent, well-funded scientists ("the 3 R's"). Both recruitment and retention are dependent upon the adequacy of the reward system.

In the typical model for a reward system in an academic medical center, "reward" means tangible resources that include salary, space, and discretionary dollars. The assignment of these resources is invariably based upon an evaluation of research productivity, according to department chairs in academic medical centers. However, the nature of the evaluation process is highly variable among different medical centers, and among various departments within a given medical center. At the University of Nebraska Medical Center, this variation is manifested by a range of evaluation methods that may be characterized as "Darwinian" on the one extreme, to "Egalitarian" on the other.

The ultimate "Egalitarian" method holds that all investigators are equal in both need and merit, and therefore all receive the same consideration for distribution of resources. Egalitarian methods are by definition more subjective and qualitative; e.g., a person may be judged on "collegiality" or "leadership." Darwinian methods tend to be more objective and quantitative. 

Purely "Darwinian" evaluation is one that compares each investigator with all other relevant investigators, and then bases the assignment of resources (the "Reward") upon this comparison. Investigators who do poorly over time will be weeded out, whereas the strongest, most adaptable, will thrive; hence, "Darwinian."

Most methods for evaluation of individual investigators involve some combination of Egalitarian and Darwinian. In general, basic sciences departments are more on the Darwinian side, whereas clinical departments tend to be more Egalitarian; colleges of medicine typically are more Darwinian than other colleges in a given medical center.

Faculty members, of course, are not completely satisfied in any case. However, it seems that faculty in departments with a major Darwinian component are generally more satisfied with evaluative criteria than they are in circumstances where the chair or another evaluator gives a highly subjective, or no, evaluation. Differences in evaluative criteria among similar departments is a source of discontent; for example, high-performing researchers in a center where some, but not their, department use Darwinian evaluation methods, typically are less satisfied and are more likely to be susceptible to recruitment by other centers.

A model that may be used for the Darwinian method quantifies annual research productivity for all investigators; derives an average; compares each investigator with the average investigator; and then distributes rewards accordingly.

The following formula has been applied to one or more successful applications of the Darwinian model:

Annual Productivity (A) = Publications (P) + Funding (F), where
P = journal "power" X author position;
F = total grant dollars as PI + FTE % paid on grants

"A" is calculated for each investigator (A_inv). Then, all A_inv are used to derive an average of A (A_avg) for all relevant investigators.

           From the above data, an "Annual Productivity Quotient" (A_q) is derived for each investigator, where

A_{q =} A_inv / A_avg.

By this quotient, it is possible to determine how any investigator compares with the average investigator, and to apply this datum to the merit-based distribution of resources. For example, Investigator Smith experiences an outstanding year, with an A_q that is 2.7 times the average for the comparison group (typically, members of a single academic department); Investigator Smith's annual salary increase, for example, could be calculated as 2.7 times the average raise for members of this comparison group. All others in the group are judged and rewarded by the same criteria, yielding 100% distribution based upon merit.

One of the principal criteria in the typical evaluation plan is the level of NIH funding; but the most Darwinian plan of all ("Darwinius Maximus") holds that the only criterion that needs to be evaluated is the level of NIH funding of an individual. In order to achieve NIH funding, it may be reasoned, an investigator must be well-published and have a solid national reputation. To achieve more than one major grant, or to be principal investigator on larger program-type grants, the investigator must have good knowledge and experience with the NIH process; a solid national reputation in research; understanding of the politics of his or her funding institute; and demonstrate a high level of both leadership and collegiality. In a word, success in NIH funding is both necessary and sufficient for judging research merit in an academic health sciences center, according to this plan of evaluation.

In general, even departments that use a highly Darwinian model for annual evaluations (objective, quantitative, limited), also permit a more subjective and qualitative and comprehensive evaluation as part of the kind of episodic evaluation (two or three career episodes) that may lead to promotion or tenure. In this case, objective criteria of research success may be combined with other criteria, such as leadership, professionalism, character, or collegiality; peer review and editorial activity; research awards; or election to national office. Obviously, virtually all of these criteria either reflect a history of strong annual evaluations, or are essential characteristics for obtaining strong annual evaluations. Therefore the annual and the episodic evaluations are part of the same fabric.

There are two recommendations that arise from the above thesis:

First, the most effective tool for evaluating the annual research productivity of an individual investigator is objective, quantitative, and limited to a few specific criteria that are research-sensitive - especially success in publication and in obtaining grant funding, with emphasis upon NIH funding.

Second, within a given health sciences center, it would be best for similar units to use the same evaluation criteria (e.g., all basic sciences departments; all clinical departments). Although it is necessary for such standardized criteria to be imposed by a college- or university-level authority, it is unusual for this to occur. Research administrators should do an evaluation of criteria for evaluation of individuals within their respective institutions, and make recommendations for standardization.

Acknowledgments: The above thesis is based in part upon data obtained from department chairs at the University of Nebraska Medical Center, and from discussions of these topics at meetings of the Association of Anatomy, Cell Biology and Neurobiology Chairs (AACBNC). The research development experience shared by Dr. Roger Markwald was especially productive and useful.