Evaluating Research Productivity
no. 105 - June 2001

RESPONSE TO THE KEYNOTE ADDRESS

Robert E. Barnhill,
Vice Chancellor for Research and Public Service
University of Kansas

Prologue: Research Competitiveness

Research productivity is important to the nation.  Since "we become what we measure," we need good methods of evaluating this productivity.  Joan Lorden has introduced some important questions for our topic today, Evaluating Research Productivity.

It is my experience that leadership at every level is essential for institutional research competitiveness.  This was the principal conclusion from a 1995 conference on Research Competitiveness. The American Association for the Advancement of Science (AAAS) convened some forty people at Kiawah Island, South Carolina.  Included were experts in research policy such as Roger Geiger, Irwin Feller, Susan Cozzens, and Harry Lambright.  The purpose of the meeting was to help EPSCoR states become more competitive in research.  The AAAS invited two outliers, that is, two people who had been successful in non-EPSCoR states, to pass around their secrets of success.  Those two people were George Walker from Indiana University and me, representing Arizona State University.  This was my first meeting with George Walker and also with the national research policy experts.  Subsequently, my institutions, Arizona State University (ASU) and the University of Kansas (KU), and I personally have profited from meeting George Walker and the other research policy gurus.

We prepared manuscripts prior to the 1995 AAAS meeting, which then became a published book (see references).  Roger Geiger's pre-meeting manuscript described the overall research scene, focusing on federal expenditures.  He mentioned that only five universities had made a considerable improvement in research competitiveness in the 1980s and early 1990s; ASU was one of the five.  Geiger went on to say, "Presidential backing for strengthening research is a virtual prerequisite.  In some cases, presidents have identified themselves with ambitious research goals; in others, presidents have more quietly backed the efforts of provosts or vice presidents for research (ASU)."  Geiger also said, "An institutional commitment to research almost presupposes the organization of research administration under a single office.  The office of the vice president for research does far more than standardize research accounting and offer administration support.  It should become the initiator of and advocate for proactive policies." The Kiawah Island conferees agreed that leadership is essential for institutional research competitiveness. This includes not only the president/chancellor, but also the faculty and the rest of the university research community.

Prior Merrill Center Research Policy Meetings

Let me review for you the last three Merrill Center conferences on research policy. The keynote speakers were:

Michael Crow, Columbia University: "Mobilizing for Research Opportunities in the Next Century" - 1998;
Luis Proenza, University of Akron, "Building Cross-University Alliances that Enhance Research" - 1999;
George Walker, Indiana University, "Making Research Part of the Public Agenda" - 2000.

Michael Crow, now the Executive Vice President for Research at Columbia, emphasized the "niche" strategy of highlighting a few areas of institutional expertise.  Luis Proenza, formerly Vice President for Research at Purdue University and now President of the University of Akron, discussed "strategic intent" and its ramifications in collaborative efforts.  George Walker, Vice President for Research and Graduate School Dean at Indiana University, has discussed the Indiana story of mobilizing "the public" to support research.  I will take up each of these three themes in addition to our topic for today.

Current Merrill Center Research Policy Meeting

Our current topic, evaluating research productivity, involves devising ways to measure what we think should be measured.  Joan Lorden has initiated this discussion for us, drawing on her work with Lawrence Martin, SUNY at Stony Brook, and others in the Council on Research Policy and Graduate Education.  This council of chief administrative officers in charge of research policy, administration and graduate education is drawn from the membership of the National Association of State Universities and Land Grant Colleges.  The NASULGC position paper co-authored by Dr. Lorden focuses on the U.S. National Research Council ratings of graduate programs.  "Towards a better way to rate research doctoral programs" is posted on the NASULGC web site: www.nasulgc.org/publications/Towards_A_Better_Way.pdf

Setting the Stage for Success

The new millennium is an exciting time for research in general and science in particular.  It is an excellent time for organizations to take stock of their goals, resources and impacts. 

 Most universities have great stability and long and honorable histories. But these days it is also important that institutions develop a certain level of flexibility so that they can move as quickly as possible when necessary.  Each university must answer these questions:

• Can we remain relevant to today's fast moving world?  Or will we be relegated to a genteel backwater role in American society?
• If we wish to remain, or become, relevant, how can we do it?
• What are reasonable goals and how can we achieve them?
• How can we measure our progress toward these goals?

Michael Crow estimates that in the near future there will be about 75 significant research universities in the US  These select universities will obtain almost all the competitive federal funding. 

Andrew Grove, CEO of Intel Corporation, has written the book, Only the Paranoid Survive.  Let me paraphrase the book's title to: Only the Flexible Will Thrive.  Only those universities which are flexible in their approach and which have clear goals and expectations will do well or even have the chance of being among Crow's 75 universities.

Two years ago, Luis Proenza introduced us to the key concept of "strategic intent," as examined in the book Competing for the Future.  Strategic intent has the attributes of direction, discovery and destiny. 

Direction: "Most companies are over-managed and under-led." That is, "more effort goes into the exercise of control than into the provision of direction."

"Strategic intent should offer employees the enticing spectacle of a new destination or at least new routes to well-known destinations."

Destiny: "Only extraordinary goals provoke extraordinary efforts." Thus, numerical goals are less energizing to employees (or researchers) than goals such as being "the best" in defined competitive areas.

Strategic intent goes beyond strategic planning.  Strategic planning is a "feasibility sieve."  Strategic intent goes beyond the feasible to what is barely possible, e.g., President Kennedy's vision of a space landing on the moon, or our efforts today to find a cure for cancer.

Arrival at Destination

There are several ways to tell that an institution has arrived at a suitable research destination:

• High institutional rankings
• World class research areas
• Cash
• Fullest utilization of university community
• Value added to society

Successful Examples

Let me take you through four examples of strategic intent applied to public universities. 

1. Arizona became a state, the forty-eighth, in 1912.  It still feels like a frontier.  A few of you know that I spent eleven exciting years at Arizona State.  I want to discuss the example of ASU's friendly rival down the road, the University of Arizona.  From Roger Geiger's book, Research and Relevant Knowledge, "the same factors that have been identified in the advancement of other research universities - establishing centers of research excellence, academic leadership, and the availability of resources - were vital to Arizona as well."  In 1959, President Richard A. Harvill stated that "Arizona's role in the expanding research economy would be to concentrate on fields in which it possessed some natural advantage."  (Clark Kerr has also used this phrase "natural advantages.")  At the time, just after Sputnik in 1957, the University of Arizona had only $1 million in federal funding and no nationally recognized departments.  In the years that followed, two centers emerged, one in astronomy and one in anthropology. Each relied on natural advantages: astronomy on Arizona's clear skies and nearby mountains for observatories; and anthropology on the presence of a large number of Native American tribal nations. (There are twenty-one tribal nations in the state.)  In 1966, the two corresponding departments became the first University of Arizona departments to receive national recognition in reputational rankings.

Geiger discerns a pattern to establishing these university centers of research excellence:

• a natural advantage
• topics a little off the beaten academic path
• areas of excellence that have far-reaching effects on the rest of the university.

Note especially Geiger's expansion on his third point:  "achieving these pockets of research excellence overcame a kind of defeatist attitude that was prevalent on the campus."

1. Before going to Arizona State University, I spent twenty-two pleasant years at the University of Utah, in Salt Lake City.  Technology sectors in Salt Lake City account for some $10 billion in annual revenues.  Five of the six key factors in the city's development as a technology center hinge on the University of Utah.  One spin-off company - the Evans & Sutherland Corporation - has helped create more than 150 computer and software companies. In 1965, David Evans came to the University of Utah to chair the Computer Science Department.  In the 1970s, he brought Ivan Sutherland to the university with the strategic intent of forming the premiere computer graphics group in the country.  Evans and Sutherland formed their company in the university's new research park.  (Many of my own students in the mathematics department worked for the new company.)  The University of Utah Research Park was itself a product of strategic intent.  Wayne Brown, Dean of Engineering, worked with President David Gardner to inaugurate the research park.  Their strategic intent was to develop a place where local entrepreneurship and expertise could flower.  Evans & Sutherland became the anchor tenant of the new park.  The three elements of direction, discovery and destiny prevailed for all of these people relative to their respective goals.

1. I now turn to a more recent example, Arizona State University, where I served from 1986-1997. ASU is a large university in the Phoenix metropolitan area, a location with considerable high tech industry.  However, ASU only formally adopted a research mission in 1980.  At about the same time, C. Roland Haden, the new Dean of Engineering, met with local business people who wanted ASU to become a significant research university with the goal of stimulating economic development.  "Engineering Excellence" was born from these meetings and sold to Governor Bruce Babbitt and other political and business leaders.  Unlike many universities in the early 1980s, ASU was growing and thus received new science faculty positions to which excellent people were hired.  This combination of Engineering Excellence and the emphasis on science hiring lifted the entire university (cf. Geiger's remarks above).  At ASU, I served for five years as Chair of Computer Science and Engineering and thus worked within Engineering Excellence on the front lines.  I then served for six years as the University's second Vice President for Research.  During that time, ASU's external funding doubled and, in 1994, ASU became a Research I university for the first time in its history.

2. Finally, Joan Lorden's school, the University of Alabama at Birmingham (UAB), has recently become a major success story. Although it is located in an EPSCOR state, UAB has made great strides in research productivity.  Dr. Lorden shared a little about this leap forward in her presentation.  Here is a longitudinal delineation of the university's federal expenditures over the past twenty years.

Lessons Learned

Strategic intent by top leadership, coupled with natural advantages and local expertise, can lead to research enhancement that lifts the entire institution. Lifting the entire institution is a phenomenon that occurred at all four of the public universities I've mentioned when all the critical elements were in place.                                   

In his book, Only the Paranoid Survive, Andy Grove of Intel includes a variety of useful advice.  As noted earlier, I've modified the title of his book for application to ur universities:  Only the Flexible Will Thrive.   Grove discusses "strategic inflection points," which demarcate times of strategic changes in the performance of a company.  These changes can be either positive or negative.

Positive strategic inflection points are reached more often if we apply strategic intent to our universities. Having goals that reflect our institutional missions can affect major changes in the output of our university research communities. 

Performance Measures

Performance metrics are important because we will become what we measure. Thus we should select and promote measures that reflect values we think are important.

Joan Lorden has played a leading role in bringing these issues to the forefront.  I believe her work with the Council on Research Policy and Graduate Education of NASULGC will have national influence.

Performance measures are used to rank and rate universities nationally, as well as to provide accountability locally.  Well-known rankings are published by US News and World Report, the National Research Council on Graduate Education, the Carnegie Foundation, and in the book by Graham and Diamond, The Rise of American Research Universities.  The numbers collected by the National Science Foundation ("NSF numbers") provide rankings based on both federal research expenditures and all research expenditures.  There are recent interesting studies by The Center at the University of Florida and by the Association of American Universities that use multiple dimensions of quantitative measurements.

A "road map" can be a useful guide.  By "road map" I mean a well thought out formal "action agenda" document.  This concept is adopted from the Japanese semi-conductor industry where it has been used since the early 1980s when Japan became a threat to American dominance in that field.

Universities often do not set research goals or, if they do, the goals don't have quantitative measures.  My counsel is to encourage setting goals that are both ambitious and multidimensional.  I will return to this topic in connection with my present institution, the University of Kansas, in a moment.

If we would like to enlist the public in support of research, it is essential to have quantitative goals that are easily understood by the public. This is another important reason to collect accurate performance measures.

Tactics: Intra- and Inter-institutional

The University of Kansas (KU) provides an interesting case study for us today. When I returned to my undergraduate alma mater in 1997, KU had reached a research equilibrium, wherein its national research ranking was fairly static, and, at the institutional level, little change had occurred within memory. State support of the university had apparently been mediocre for some time and, consequently, support for research was sparse.  However, the faculty and the university appeared to be better than was indicated by the institutional ranking.  In particular, KU had a group of entrepreneurial research centers with faculty eager to step up the pace.

We decided to inventory our intellectual capital on the four KU campuses.  We did this by means of a call to the Deans and Center Directors to elicit faculty proposals for research attention. This was not a formal call for financial proposals, but rather a call for feasibility of "world class" research.  Forty-seven proposals were submitted and a steering committee of Deans, Directors, and others looked for "mega themes," that is, for topics that met three major criteria: at least 50 faculty working in areas that have demonstrated, peer-reviewed strength, that are also of significance to our public.  The steering committee was unanimous in selecting four mega themes: information technology, human biosciences, the human condition, and environmental science and engineering. 

What is "world class" research?  In my opinion, a group is doing world-class research if every international meeting in their area must invite a member of that group to participate.

Next, we inventoried the three research universities of Kansas: the University of Kansas, Kansas State University and Wichita State University.  Partners in this process included the AAAS, KTEC (Kansas Technology Enterprise Corporation), EPSCoR, the Senator Pat Roberts Advisory Committee on Science, Technology and the Future, and KU's Merrill Advanced Studies Center.  In due course, we determined four strategic initiatives in science and technology for the state: information technology, human biosciences, agricultural biotechnology, and aviation.  We are working at the state, regional and national levels to promote these initiatives.

Example of Performance Measures: KU Performance measures that follow national norms are best for national comparisons. The federal research and development expenditures in science and engineering (i.e., those areas with significant external funding) are collected annually by the National Science Foundation (NSF) and are freely available. So this has become our "gold standard" for national comparisons.  Such rankings also provide a surrogate for market share: the percentage of the federal R&D funds obtained by a given university.  Market share corrects for variation in federal Research and Development funds available; since such variation has been considerable over the years, this is an important consideration.

Although federal expenditures in Research and Development are the best available measure of national research competitiveness, this statistic underestimates the local impact of research.  Thus, in Kansas, we use the same methodology as the NSF, but have extended it to include: (1) fields outside of science and engineering, and (2) research training grant expenditures. We call this statistic "research, development and training expenditures" (RD&T).  These figures are particularly helpful in discussing local economic development impacts of university research.

KU's FY99 Rankings

KU advanced in the rankings according to federal R&D in science and engineering, among all universities, by ten positions between fiscal years 1998 and 1999, rising from ninety-third to eighty-third. A corresponding KU rise among public universities was seven positions, from sixtieth to fifty-third.

Rankings Changes: FY98-99

This ranking change was the second largest among the top 100 universities, ASU being the university with the largest change. The average change in ranking, positive or negative, was about three positions.  KU's ranking change within public universities was also the second largest nationally. 

KU's RD&T Expenditures - FY00

KU's federal R&D expenditures in science and engineering rose 20% between fiscal years 1999 and 2000. National rankings for 2000 will be available later from the NSF.  KU's total research, development and training (RD&T) expenditures rose 15% from fiscal year 1999 to 2000.  KU uses NSF methodology to determine its total RD&T number.

KU Longitudinal Studies

It is always wise to study an institution over several years, as exemplified on the longitudinal graphs that follow. In order to provide historical context, here are KU's research productivity rankings among all universities and among all public universities for the last 20 years.  The criterion is federal expenditures in science and engineering.

In the next two graphs, you will see the dollar amounts producing the last few years' rankings, as well as the RD&T numbers.

KU's research strategy - exemplified by its research administration arm, the KU Center for Research, Inc. (KUCR) - involves many aspects, including:

• Promote research as a topic on the campus;
• Stimulate bigger projects: collaborations;
• Provide much better assistance to faculty;
• Make strategic investments of time and money.

Since research administration is a "disruptive technology" (in the sense of Clayton Christensen's book, The Innovator's Dilemma), it must operate relatively autonomously. A necessary condition to maximize research productivity is to minimize internal competition between academic departments and research centers.  We have devised a multiple credit algorithm to accomplish this task. Expenditures are recorded in two lists, one according to departments and one according to centers. This simple expedient has helped reduce competition between departments and centers.

National R&D Environment

Some background is necessary and helpful in understanding what is nationally possible. Lester Thurow, MIT professor of management and economics, wrote the lead article in the June 1999, Atlantic Monthly, "Building Wealth: The New Rules for Individuals, Companies and Nations."

Thurow writes, "A successful knowledge-based economy requires large public investments in education, infrastructure, and research and development."  He quotes rates of return on R&D as: 24% for private investment; 66% for public investment.  ("Public" rates of return mean that the benefits accrue to the whole society.)  "Put simply," Thurow continues, "the payoff from social investment in basic research is as clear as anything is ever going to be in economics."

Some sound bites:

• 50% of economic progress since World War II is due to technology. This includes the fact that almost 3/4 of patents issued depend at least in part on publicly funded research.

• Alan Greenspan has stated that: "the unexpected leap in technology is primarily responsible for the nation's phenomenal economic performance."

• Internet economy: $300 billion with 1.2 million jobs

• Information Technology bits from the PITAC report:

• 1/3 of economic growth;
• 1/3 of all corporate R&D;
• 55% of all venture capital;
• New startup every hour;
• 7.4 million jobs
• 80% higher salaries

My own scientific career in Numerical Analysis and then in Computer Aided Geometric Design causes me to think that information technology advances during the next few years will dwarf what has come before, in terms of ubiquitous computing and visualization possibilities.  These advances will include such visionary topics as molecular level, fault tolerant computer architectures that resemble biological systems, as well as advances in brain imaging and gene therapy due to virtual reality and computational power. For an institution to become a research leader, it must possess leadership that is aware of and can utilize national trends on the local level.

State Rationale for Research

Research universities provide unique cultural and economic advantages to society in general and to local communities in particular. Cultural opportunities include the advantages of a liberal education and all its corollaries. Economic impacts include the value added to graduates' incomes, as well as the economic ripple effect due to R&D dollars.

Economic Impact of Research I University

Graduates are the largest form of technology transfer from research universities.  We have quantified the economic impact of this important asset for our state of Kansas: the annual income of the alumni of our three research universities, who currently reside in Kansas, is $9 billion.  About 1/3 of this total, or $3 billion, is due to the increased salaries they earn because of their degrees from our three universities.  The state tax paid by these graduates is $700 million annually, a figure that exceeds the annual state appropriation to the three universities of $400 million.

What is the ripple effect of R&D funding in Kansas?

The US Department of Commerce estimates that, in Kansas, each $1 million in R&D funding creates 40.6 jobs.  The three Kansas research universities had $335.2 million in RD&T expenditures in fiscal year 2000, which implies that more than 13,600 jobs are due to this source of funding.  Moreover, the average salary in these jobs exceeds the average salary in our state. This type of economic information is what truly catches the attention of state legislators.

A Poll of the Public

Thus, I called together my counterparts from the KU Medical Center, Kansas State University, Wichita State University, and the Kansas Technology Enterprise Corporation (KTEC) to meet Mary Woolley.  The result of this meeting was a poll of the Kansas citizenry. My favorite among the statistics gathered by Ms. Woolley's firm is: 93% of Kansans favor state support of university research, whereas the comparable national average is only 82%.  Senator Pat Roberts announced the results of the survey at a press conference with the CEOs of the three Kansas research universities by his side.

A Tipping Point

Malcolm Gladwell's book, The Tipping Point, indicates that changes by relatively few people can have large impacts.  There are three rules for a tipping point: the Law of the Few, the Stickiness Factor, and the Power of Context. 

My considerable oversimplification of the book is the following:

The Law of the Few: The example of Paul Revere illustrates that some people have exactly the right connections for making a significant impact, while others in the same situation cannot because they do not have these resources. 

The Stickiness Factor: Successful projects frequently have some feature, say, a snappy title or phrase, which makes people remember them favorably. The image "sticks" in their mind.  My own advocacy example is "Selling the Endless Frontier."  This echoes "Science, the Endless Frontier" from Vannevar Bush's letter to Franklin D. Roosevelt encouraging federal support of research after World War II.

The Power of Context: "Environmental tipping points are things that we can change."  Gladwell gives the example of fixing up a small portion of a run-down neighborhood.  By this example, the neighborhood as a whole improves itself.  My hopeful example would be the historical indifference of a legislature to university research.

Black Elk, Oglala Sioux

In the research arena, change is a necessity.  Sometimes it is tempting to think that we have invented everything. I am always brought back to Earth when I turn to this late nineteenth-century saying by Black Elk, an Oglala Sioux elder:  "Little else but weather ever happened in that country - other than the sun and moon and stars going over - and there was little for the old men to do but wait for yesterday." Let us not be like Black Elk's "old men."  Rather, let us embrace change and use it to advance science and society in the twenty-first century.

References

Bush, Vannevar (1945).  Science: The Endless Frontier. National Science Foundation.

Geiger, Roger (1993). Research and Relevant Knowledge:  American Research Universities since World War II.  Oxford University Press.

Gladwell, Malcolm (2000). The Tipping Point: How Little Things Can Make a Big Difference.  Little, Brown and Company.

Graham, Hugh Davis and Diamond, Nancy (1997).  The Rise of American Research Universities: Elites and Challengers in the Postwar Era. Johns Hopkins University Press.

Grove, Andrew S. (1996). Only the Paranoid Survive: How to Exploit the Crisis Points that Challenge Every Company and Career.  Doubleday Publishing Company.

Hamel, Gary and Prahalad, C.K. (1994). Competing for the Future. Harvard Business School Press.

Stokes, Donald E. (1997).  Pasteur's Quadrant: Basic Science and Technological Innovation. Brookings Institution Press.   

Teich, Albert H. (Ed.) (1996).  Competitiveness in Academic Research. American Association for the Advancement of Science.

Thurow, Lester (June 1999).  Building Wealth: the New Rules for Individuals, Companies, and Nations. Atlantic Monthly.