University Governance
Report to the University/Faculty Senates from the Task Force on Learner Outcomes
Fall 2009Summary
General education goals and the curricular options that fulfill them are put in place by the faculty with the intent to facilitate the development of broadly educated graduates of the University of Kansas. While the faculty has some interest in knowing how well that education succeeds in its goals, a recent strategic initiative (2015) highlighted documentation of learning throughout the university. Faculty discussion of the evidence will provide an opportunity to identify areas for continued development, and those conversations can be made available to students and their families, regents and the public to represent institutional commitment to student learning.
The six general education goals in place at KU are more specific than the large-scale skills called for by government panels and measured by standardized tests. On the other hand, they are less refined and articulated than are the goals of some higher education associations that promote measures of liberal arts achievement. Hundreds of campuses are engaged in various forms of assessment, ranging from standardized tests to evaluation of samples of student work taken from courses. Some of these institutions explicitly aim to compare their students’ performance with that of other institutions, while others use benchmarks of local performance only as a guide to identify continuous improvement in student learning.
KU typically uses interviews of undergraduate students from its many schools and the college (http://www2.ku.edu/~oirp/Assess/index.shtml) to assess the level of understanding in the six areas of general education. The results of evaluations done by volunteer faculty members are made available to the community, though no extended public discussions are held about lessons learned or plans for changes in teaching. Given that many units are gathering samples of student work for their own purposes, it would be possible to use shared general education frameworks to categorize student understanding. That description could be compared with indications obtained from standardized tests to see how each represents typical understanding among KU undergraduates. The last review of the general education interview process was in 2001 (reported at the same web site above), and faculty members generally enjoyed their participation and found it valuable to them. This is a good time to explore and discuss KU’s method of measuring our students’ general education, and it is essential to articulate how evidence is to be used to enhance learning at KU and/or to compare KU students with others.
The Task Force has a few recommendations that would move this conversation forward:
Task Force Charge #1: Identify, if possible, those “basic competencies” that all graduates of the University of Kansas should attain.
The current KU goals for general education have been in place for some time, having been revised once, and there appears to be no sentiment in the faculty for wholesale revision of those guidelines. These goals represent a conceptual statement of the values of the KU faculty community on the optimal qualities of an educated graduate of the university. KU’s six general education goals can be found in an appendix at the end of this document and at this web site (http://www2.ku.edu/~oirp/Assess/goals_button.shtml). There are goals that identify the value of knowledge and skills within individual fields of study (especially Goals 2 and 3), and there are goals that focus on general intellectual qualities that allow for integration, analysis, application, articulation, and evaluation of knowledge (found in Goals 1, 2, 3, and 5). Students also should have an awareness and appreciation of the diverse cultures of the U.S. and the world, while also being aware of the complexity and interaction of issues around society, technology, and the natural world (identified in Goals 4 and 5). Finally, Goal 6 urges an ethic of self-discipline, responsibility, and citizenship in local and global contexts.
The current six goals of general education, approved in May 2001, were and have been well received by the faculty. In 2001, the Provost’s Office reported an evaluation of the process of assessing general education at KU (http://www2.ku.edu/~oirp/Assess/index.shtml). Some of its key recommendations include stressing that the university community needs to be better informed on the one hand about KU’s general education goals and on the other about the results of general education assessment. Another key finding was the necessity of university-wide and unit-level consideration of how general education goals fit into the university experience and how each unit contributes to the achievement of those goals. The Task Force agrees with and reaffirms these recommendations and proposes that they be followed through upon the conclusion of any assessment process, whether it be via standardized testing and/or evaluation of samples of student work. Communicating the results of assessment is the first vital step toward using those results to make meaningful changes in teaching and learning.
The Task Force also examined the content of the existing KU goals for general education in the context of external audiences and communities. Though the wording and organization of KU’s goals are different from those of other institutions and associations, the six goals fit well within the national conversation on goals for undergraduate general education. There is wide diversity in the number and range of goals proposed or measured through different approaches. On the lean end, most of the standardized tests available for this purpose measure problem solving, critical thinking, analytical reasoning, and written communication. These represent the basic intellectual skills that people typically assume are acquired during college. On the complex end, many assessment plans involve portfolios or samples of students' work embedded in classes which are subject to categorization and evaluation on as many as a dozen different dimensions, including integrated learning, creativity, teamwork, lifelong learning, ethical reasoning, and quantitative literacy.
An example of a lean approach appeared in 2006, when the Association of Public and Land-grant Universities (APLU) proposed a Voluntary System of Accountability (VSA) program to provide participating institutions the means to supply information to relevant constituents about what the university is doing to ensure students are meeting general education goals. VSA’s simple and basic goals for accountability were designed to encompass the goals the 520 participating institutions across the country. VSA collects data on the following general education goals: critical thinking, analytical reasoning, and written communication skills. VSA selected a smaller number of easily measurable goals so colleges and universities could report quantitative findings on students’ performance in comparison with students at other institutions. VSA emphasizes easily attainable data for an institution’s web site that can be readily available to prospective or current students and their parents, to faculty and staff, and to public policy makers and funders of higher education. KU is a VSA participant (http://www.collegeportraits.org/KS/KU), and it will report data on critical thinking, analytic reasoning, and written communication. KU selected the Collegiate Learning Assessment (CLA) as its tool for standardized measurement of those outcomes.
At the other end of the organizational dimension, the Association of American Colleges and Universities (AACU) has a more complex system of articulating general education goals. Though AACU lists only four primary goals in its general education initiative (knowledge of human cultures and the natural world, intellectual and practical skills, personal and social responsibility, and integrated and applied learning), it has broken these down further into fourteen areas of performance that assess a wide and nuanced variety of skills and competencies. For example, the area called integrated and applied learning covers a wide variety of competencies that are vitally connected to outcomes desired by employers, a group interviewed by the AACU as an interested party in university assessment. Outcomes such as teamwork, integrative learning, foundations and skills for lifelong learning, and creative thinking indicate the wide variety of expectations employers have for degree-holding students in the twenty-first century (along with more basic goals like quantitative literacy and written communication). All fourteen of LEAP’s goals can be found at this link: (http://www.aacu.org/leap/vision.cfm).
KU’s six general education goals fall somewhere between AACU’s nuanced, specific and numerous goals and VSA’s brief but broad-reaching goals. KU’s goals of “Integrate Knowledge” and “Core Skills” tend to cover the same areas addressed in the VSA’s goals and AACU’s more basic goals, while “Critical Inquiry,” “Appreciate Diversity,” “Contemporary Issues” and “Social Responsibility” reach toward the applied learning that is a key focus in the AACU analysis and expressed in its evaluation rubrics. KU general education goals reflect both the general aspirations of the VSA goals while reaching for some of the more specific content related goals articulated by the AACU project.
We also examined the particular goals of specific institutions that made their programs visible. In particular six large public universities have well articulated general education goals along with a system for evaluating how well they are met in student work. The University of Minnesota, the University of Colorado, Illinois State University, Washington State University, and Oklahoma State University have frameworks for general education that are structurally similar to those at KU. Those universities have identified intellectual goals and designated courses as satisfying the general education component of graduation requirements. Details of those programs can be found at each university’s web site:
University of Colorado (http://www.colorado.edu/pba/outcomes/)
Oklahoma State University (http://www.uat.okstate.edu/assessment/index.html)
University of Minnesota (http://www.academic.umn.edu/accountability/reports/2008.html)
Illinois State University (http://www.assessment.ilstu.edu/)
Washington State University (http://wsuctproject.wsu.edu/index.htm)
Based on a reading of the organizational and institutional goals in our review, we note that the KU goals could use further review and updating in light of current understanding of the complex worlds of work and citizenship that face our graduates. Given that there is no indication that our campus is ready to undertake a wholesale reconsideration of the intellectual structure, we are not prepared to recommend such a reframing of goals. Such a review may become more urgent if solid evidence from KU students identifies weaknesses in the preparation that we currently provide. A member of the Task Force (Dan Spencer, School of Business) presented a cogent analysis and critique of how the current goals fare in the context of national conversations on general education. That report is posted in the Task Force Blackboard site (DocumentsàFraming Recommendations), and it would be a good starting point for any group that sets out to reconsider and revise the current KU general education goals. He notes in particular that the local goals have not focused on “Integrated and Applied Learning” as articulated by the AACU reports: “KU’s goals have not kept up with the faster pace of change in the world today that requires an increasing adaptive capacity on the part of our graduates….we certainly need to explore the extent to which KU’s goals resonate with, or fail to resonate with, the emerging consensus regarding contemporary liberal and general education.”
Task Force Charge #2: To the extent possible, work with the Association of Public and Land-grant Universities to contribute to and learn from its study of the measurement of learner outcomes.
There is a strong connection between KU and the VSA project coordinated by APLU; our former Provost is the leader of that project, and a former staff member from the KU Office of Institutional Research and Planning works with the project. KU currently participates in the VSA, providing a wealth of basic student data tracking completion of studies and time to degree. Examples of the data made available to the public include the following: student characteristics, undergraduate success and progress rate, retention rates, student housing, degrees and areas of study, and a number of specific items under student experiences and perceptions, such as active learning and group learning experiences. All the participating schools and corresponding data are found at this web site (http://www.collegeportraits.org/state_list). In addition KU will provide standardized test data on student intellectual skills. At the same time, there is a funded project based in the Center for Teaching Excellence that is also collecting standardized test data on student intellectual skills, along with samples of student work that can be categorized using a framework based on general education programs at universities and colleges around the country. In this way KU is both participating in the early stages of the VSA and offering some feedback to APLU on the quality of the information gained from standardized testing.
In order to obtain readily comparable data, the VSA recommends universities choose one of three different standardized tests. The Collegiate Assessment of Academic Proficiency (CAAP) is a set of standardized tests developed by ACT that are designed to assess general education skills. CAAP allows universities to choose from six different test modules; other than the writing module, each of these areas is tested using a multiple-choice test. CAAP is most often used to test students who have just finished their core classes. The CAAP test can be used to test learning in the institution as a whole both by measuring skill levels in the fourth year of study and by measuring improvement from first to fourth year (called value-added in the testing and production industries). The value added to students’ “intellectual capital” can be determined by performing longitudinal studies, cross-sectional studies or even by linking the CAAP scores to ACT or COMPASS scores.
The Measure of Academic Proficiency and Progress (MAPP), administered by ETS, is a standardized test that combines multiple choice test questions with the option for a writing assessment test. It also provides the option of fifty locally authored questions and nine demographic questions. The multiple-choice section tests mathematics, writing, and reading and critical thinking. Each of these areas is tested on three different levels with specific rubrics for proficiency at each level. Tests can be scored either using norm-referenced scores (scaled scores), which compare the scores of one student or group of students to another, or the same student or group at different points in time; or criterion-referenced scores (proficiency classifications), which measure the level of proficiency obtained on a certain skill set.
The Collegiate Learning Assessment (CLA), developed by the Council for Aid to Education (CAE) and the RAND Corporation, uses a holistic method that tests students in four key areas: critical thinking, analytical reasoning, realistic problem solving and writing. Unlike options-based multiple choice tests, in the CLA students type answers to open-ended questions involving tasks that combine the four skills in order to simulate integrated writing and analysis situations (as opposed to assessments that test each skill individually). The application of these skills includes identifying relevant information in a document, organizing information, making an argument, and proposing solutions to problems. CLA is designed to be used to test a sample of incoming first-year students in the fall semester and a sample of out-going fourth-year students in the spring semester. By doing this, it tests not proficiency levels, but rather increase in those skills that takes place in the years between arrival and completion. CLA can either be used cross-sectionally (testing in one year a set of first-year and a set of fourth-year students), which gives an immediate snapshot of the enhanced skills; or longitudinally, which would track a core group of students throughout their time at the university.
Members of the Task Force strongly favored the CLA among the three choices offered by VSA because of its holistic application of knowledge and skills and its focus on writing instead of multiple choice tests. Since the Provost’s office and the funded grant had independently reached the same preference among the standardized test options, it has been possible to collaborate on a year of administering the CLA to samples of KU students. The first two wave of students taking the CLA were conducted in the fall of 2008 and spring of 2009, and the results are being analyzed. These data will be used to see how well the test matches the assessment of intellectual skill observed in course-embedded assignments by a sample of students who also took the CLA. The CLA scores of fourth-year students in the spring sample can be compared with the scores of first-year students in the fall sample to get a first cross-sectional estimate of the intellectual value added by four years at KU. Since this was the first time KU had used the CLA there were some limits to the representativeness and size of the samples drawn, so the Provost may choose not to post this first data set in the VSA system.
Another feature of this first year of the CLA trial will examine the utility of the data we receive. KU (both the Provost and the grant project) have asked for the data in two forms. The raw scores on each of the two components (the analysis performance task and the writing task) will provide a benchmark of the skill level of our own students, and we can compare the KU distribution of scores with the distribution of scores nationwide. The raw data will also be used for comparison with the hand scored student work as part of the grant project. CLA scores are typically adjusted before they are given to a campus, using the pre-college achievement scores of incoming first year students to predict an expected score given the quality of the entering students. The system calculates the expected score and then reports individual data in terms of their relation to the expected scores (below, at, or above expected). KU will also receive its expected scores for the students who took the test.
The CLA providers intend that this adjusted format will discourage institutions from using the data to make unfair comparisons of student achievement; a school with selective admissions would be expected to show more skill at the fourth year than a school that has relatively open admissions. Instead, CLA urges that institutions compare their own students to the expected scores for their institution. If students are below expected (either at one point in time or in the size of the increase over four years), then the institution would have reason to improve its teaching and learning environment. If on the other hand students are above expected, then the school would want to expand and promote its culture of teaching and learning. Despite the intent of the CLA designers, the VSA format provides prospective students and the public an easy way to make comparisons across institutions, yielding claims that one's tuition dollar returns more value added at institution A than at institution B. Such a consumer guide mentality dominated the work of the federal Spellings Commission challenge to higher education, and the VSA was created to provide a voluntary alternative to a mandated federal system.
In contrast to standardized testing, the approach favored by AACU and its rubric-based analysis of course-embedded student work is explicitly designed to provide a benchmark that is internal to each institution. The evaluations could be tied to a national convention on standards and goals, but they would not provide a number that would be easily compared in the context of the VSA reporting documentation. An institution could use the data to track improvement in its own learning and teaching over time, but there is no attempt to offer a standardized metric that would allow a quick or easy comparison between two institutions.
It is imperative that KU clarify up front its intentions for use of whatever data are collected. If we want to focus on the comparison of general education at KU to general education at other institutions, then we should select a method that will facilitate that comparison. If on the other hand we prefer to focus on improving learning at KU, then we need to have a clear strategy for collecting and responding to our data in a way that enhances learning and teaching based on the evidence we gather (and take seriously). It would be more useful to compare how institutions are using assessment data than it would be to compare the levels of students at those institutions.
In 2008 the results from a national measure of student engagement provided an interesting insight into the uses of such data. The National Survey of Student Engagement has been widely used as a surrogate for direct measures of learning, and institutions whose students report more engagement than average typically report those data in their recruiting materials. When NSSE analyzed its own data, it offered a different message. While some institutions had students who were more engaged than others, the differences between institutions were relatively small. There was a very wide variation in engagement, however, among students within any single institution. The title of the NSSE report is “look within,” and that is their advice to institutions. NSSE suggests institutions should focus on the large number of students in their own institutions whose engagement could be dramatically improved rather than comparing themselves to other campuses. Click on download results at (http://nsse.iub.edu/NSSE_2008_Results/) to examine this report in full.
Based on these results, the Task Force asked the CLA office to provide comparable data for CLA scores. CLA responded with analysis of the raw scores from the 2007-2008 wave of data. These are scores on their tests that have not been adjusted to take into account the entry-level skills of students; these data would maximize the differences among institutions. For first-year students, across all the measures CLA uses, there was much more variability within each institution than between institutions (the differences among students at a single institution contributed three times as much variability as the differences between institutions). For fourth-year students the same pattern emerged but the effect was larger; the differences within each institution contributed four times as much variability as did the comparisons between institutions. Given that CLA typically reports publicly only the adjusted data, the between institution differences will play an even smaller role in the numbers that the public will see on the VSA report. The CLA data give the same message as the NSSE data; the focus of campus inquiry should be on how to improve the education of our own lower reaches rather than focusing on getting our average number higher than a group of comparison institutions.
Given this important reframing from NSSE analysis and from CLA analysis of the reason for gathering institutional data, it is essential that whatever information is obtained should be useful in identifying with some specificity what features of our students’ performance could use improvement. If our primary use of the data will be to enhance teaching in support of the weaker skills, we need to receive data that give appropriately differentiated evidence. Some early adopters of the CLA (such as the University of Nebraska - Lincoln) have reported that the data they received were not helpful in identifying the areas of strength and weakness in student performance; there were only two numbers, the analytic performance task and the writing task. It is possible to see a version of the scoring system that identifies the component skills that go into the number, but the last version of the reports we saw did not provide a breakdown of performance by those components. CLA is developing a program (called “CLA in the Classroom”) designed to improve teaching, but it is very similar to what teaching and learning centers all over the country already do. It is not uniquely tied to the particular data received from an institution.
It is possible that KU would need to use both the CLA and some form of local reading of student work to participate fully in the VSA. Only the CLA will give data appropriate to the VSA goal of providing comparative information for public uses, while only a locally administered analysis of component skills will provide the more specific information needed to develop student performance. Both systems have advocates, and institutions often choose a method based on the purpose for their collection of data. The CLA is expensive to run, and many users report major challenges in recruiting and motivating students for the test; some institutions have carefully studied motivation effects, showing that student performance varies greatly with how motivated the students are. Small institutions that have Dean-level administrators who heavily promote assessment as a campus mission have reported high average levels of student effort on the CLA, but motivation issues surfaced at KU last fall, even with exhortation from faculty members to groups of students.
Using a standardized test requires little or no work from faculty members, and that will be seen as a large advantage when faculty members consider themselves already burdened with administrative tasks. Standardized testing does require a significant amount of effort from academic affairs staff for recruiting and motivating students to participate seriously. Reading student work and evaluating it, however, requires more faculty time, but KU has in the past recruited faculty members for interviewing by providing an evening meal and the company of colleagues for a few hours. Many scholars in the assessment field argue that having active and committed faculty participation is essential to a successful program aimed at improving teaching and learning, and that would favor the evaluation of samples of student writing from courses. The real value of any assessment for the institution itself (as opposed to those examining institutional performance from outside) is in the use of evidence to make changes in teaching and learning; if the comparison function of the evidence dominates the selection of the indicators to be used, there could be a diminution of the value of the evidence for narrowing the relatively large variation of learning within each institution.
Task Force Charge #3: Review current practices and procedures concerning assessment of learning by undergraduates and recommend additions or changes to those practices and procedures.
For more than 15 years the Provost’s Office has provided an estimate of the success of general education through interviews of select fourth-year students. Faculty volunteers are provided with interview prompts and a scoring system keyed to the six KU general education goals. Each year a stratified sample of students representing the College and a few of the schools spend 45 minutes in conversation with three faculty members. The prompting questions provide the student an opportunity to display the skills expected and the professors an opportunity to judge them on a numerical scale. The summary data are provided to deans, shared in a meeting of the faculty raters, and posted on a KU web site. This was an innovative and progressive plan when it was initiated, but there has been no systematic campus conversation on how to use this information to enhance the learning of general educational goals. The procedures for the interview process, along with the data from several years, are found on the KU web site (http://www2.ku.edu/~oirp/Assess/index.shtml).
The current six goals of general education, approved in May 2001, were and have been well received by the faculty. In 2001, the General Education Assessment Process Committee reported the findings of an evaluation of the student interview method of assessing general education at KU; their report is available at the same web site as the data (above). Some of its key recommendations include substantial revision to the process to make it more valid and to engage the campus in using the evidence gathered. The University community needs to be better informed on the one hand about KU’s general education goals and on the other about the results of general education assessment. Another key finding was the necessity of University-wide and unit-level consideration of how general education goals fit into the University experience and how each unit contributes to the achievement of those goals. To this point, there has been little systematic campus conversation on how to use this information to enhance learning within our goals for general education, and the interview method remains largely unchanged. The Task Force agrees with and reaffirms the Committee’s recommendations and proposes that they be followed through. This includes first rethinking the optimal assessment process, whether it be via standardized testing, evaluation of samples of student work, and/or a revised student interview, and second communicating the results of assessment and using those results to make meaningful changes in teaching and learning.
The Task Force explored both standardized testing and assessment using samples of student work as potential alternatives to the interview system. In the last 15 years many other universities have developed sophisticated systems of sampling and evaluating artifacts of student work that are worth considering as complements or alternatives to the current method. The Task Force recognizes the need to establish the efficacy and reliability of both the CLA and collections of artifacts for KU’s purposes. As noted earlier, the Provost and CTE are collaborating in a project to collect data using the CLA in order to report to the VSA. The CLA test is being administered to a sample of KU students throughout this academic year. About two-thirds of these students’ course-based writing will also be assessed using the AACU general education rubrics; these evaluations will be compared with the CLA scores those same students receive in order to test the consistency of these two methods. Preliminary data from this year's dual administration should be available by the end of 2009.
In addition to general education assessment, a number of departments and programs at KU have already implemented assessments at the local level to ensure that their students are graduating with the skills and knowledge necessary to succeed in that field. These departments should serve as a model for what is possible at the institutional level in any form of university-wide assessment. Speech-Language-Hearing, for example, at the request of its national organization, has developed a portfolio-based assessment plan and corresponding rubrics for its Masters students in speech language pathology. Developing this assessment has helped the department move toward a common language used to evaluate students and a means for summarizing performance across students in order to identify areas in which the program may need revision. The School of Pharmacy has developed a portfolio assessment combined with a standardized client evaluation as a means to ensure its graduates are prepared for the rigors of their chosen profession. Students of pharmacy are expected to maintain their own portfolio over the course of their degree; in it they reflect on their strengths and weaknesses as a pharmacist and on the particular kinds of work they’ve been doing. In the standardized client evaluation students are put through a series of scenarios intended to assess their ability to unite theory with practice. More information on each of these departments is available in the unit portfolios on the CTE web site (http://cte.ku.edu/gallery/units/splh/) and (http://cte.ku.edu/gallery/units/pharmacy/index.shtml).
What information did our research uncover?
In order to explore the questions and possibilities raised above, the Task Force explored the assessment procedures of a number of other institutions around the nation, the current literature on both standardized testing and portfolios, and the national conversation on assessment and accountability. All of this information is compiled on a Blackboard page which can be publicly viewed as “Other User” with a username of [_tlo] and the password of [guest1]. The research is presented according to category under the Documents tab. The main categories are: Guest Summary (an overview of the site for visitors), Standardized Test Options, General Literature on Accountability and Assessment, Portfolios, KU Assets, Framing Recommendations, and Closing the Loop. There are also several folders containing information about what other universities are currently doing in order to assess students, some of the details of which are below.
One of the more important results of the Task Force’s research on the question of assessment and accountability was the placement of KU within a national context of assessment, a conversation that is becoming more and more prominent as parents, students, regents, legislators and other constituencies begin demanding a solid “return” on their increasing financial investment in higher education. Much of this conversation stems out of AACU’s Valid Assessment of Learning in Undergraduate Education (VALUE) project, a part of its general education initiative. The advisory board of the VALUE project cautions against a “quick fix” through use of standardized testing, instead recommending e-portfolios as a preferred way to promote and measure student learning. Part of the VALUE project also includes establishing a set of common Essential Learning Outcomes to be articulated in fourteen “metarubrics” compiled out of the general education outcomes of a number of participating universities.
Several universities of comparable size and mission to KU have already begun institution-wide assessment plans, with different institutions utilizing different assessment methods based on needs and the make-up of the school. The University of Texas system administered the CLA test to seniors and freshmen from a number of its institutions during the 2004-2005 school year. Its goals in doing so were to determine how UT students performed in comparison to national scores, to determine the value-added at individual schools, and, most significantly, to set a baseline from which to measure future progress. In order to make a national comparison, the CLA results were converted to the 36-point scale of the ACT. Its report on the test indicates that all of the UT schools performed at or above expectations, even if those expectations placed the school’s scores below the national average. Pedro Reyes, UT’s associate vice chancellor for academic planning and assessment, writes in the UT report on the CLA results that performance-based assessments like the CLA are important for universities to utilize as they are a better measure of the ways in which students will be expected to apply their knowledge, skills and abilities. He further writes that, since “competencies are complementary phenomena that combine skills, abilities, and knowledge,” schools would serve themselves best by employing assessment measures that reflect this integration (Reyes, 2006). As such, he recommends that universities measure four key general intellectual skills assessed by the CLA: critical thinking, problem solving, analytic reasoning and written communication. (Reyes’ report on UT’s administration of the CLA can be found on the Blackboard page, in DocumentsàStandardized TestsàCLA).
In 2001, Oklahoma State University initiated an institutional assessment that sought to measure OSU’s general education outcomes. Its assessment task force decided to implement a dual assessment, in the form of institutional collections of artifacts and a course-content database. As described in OSU’s General Education Assessment Committee’s Annual Report from 2007, evaluations of institutional collections of artifacts “directly assess student achievement of the expected learning outcomes for the general education program, and the course database evaluates how each general education course contributes to student achievement of those articulated outcomes” (Wilbur, et. al., 2007). OSU uses the institutional portfolios to collect evidence of student work from general education courses. These artifacts are grouped according to general education goals: written communication skills, math problem solving skills, science problem solving skills, and critical thinking. Faculty members work in groups to assess student work in each of these areas, using individual rubrics that were developed for each goal. The purpose of the course-content database is to collect information on how each general education course contributes to the university’s stated general education goals. Faculty members upload information on their courses onto a web-based form, where information is requested regarding what general education goals the course meets, what opportunities students are provided with to meet those goals and how the faculty member assesses these goals. According to OSU’s 2007 report, “[t]he database provides a useful tool for holistically evaluating general education course offerings and the extent to which the overall general education goals are targeted across the curriculum.” (This report, along with forms and departmental information, can be found on the Blackboard page in DocumentsàOklahoma State University.) OSU explicitly modeled some of its procedures on the work developed at Washington State University, which has a similar structure in place. In both cases, the identification of clear goals and procedures for sampling and evaluating student work has led to changes in the qualification of courses to count for general education credit.
The University of Nebraska provides a valuable model for institutional assessment as it is focused heavily on ensuring that program and general education assessment leads to meaningful changes in teaching and learning. Its streamlined process, which involves a guidebook for departmental assessment and an online assessment reporting database, demonstrates the ways in which assessment at a research university can be utilized at basic levels that directly affect undergraduate education, such as curriculum reform. (Information on Nebraska’s program, including the guidebook, can be found on the Blackboard page in DocumentsàGeneral Literature).
Illinois State University and Alverno College (both much smaller than KU) assess using portfolios: ISU uses an institutional artifact method similar to the one used by OSU, and Alverno uses an electronic portfolio program maintained by each student over four years (both under Documents, along with an analysis by Dan Spencer).
Neither ISU nor Alverno is close to being a peer university of KU. Nevertheless, the schools’ portfolio-based assessment methods were highly regarded by Task Force members as being able to provide the most meaningful information that could be applied to the task of improving student learning at the general education and discipline levels. At ISU, faculty members voluntarily provide ungraded samples of student work from their classes to the University Assessment Office. These “artifacts” are copied and returned to the faculty member within 24 hours. Each artifact is then evaluated by a team of 2-3 faculty members or administrators with teaching responsibilities to see if the work demonstrates mastery of a set of learner outcomes established by the university. Dan Spencer writes that one of the key advantages to this system is that faculty are able to “self-select” themselves for the process and therefore “would be more likely to comply with newly established procedure and be willing to promote the process to faculty colleagues once its success has been established.” The Alverno model involves students in a continual self-assessment that spans all four years of a student’s tenure at the university, where an electronic portfolio is maintained by the student and reviewed by faculty members to ensure that all students have met eight outcomes established by the university. Spencer writes further that the advantages to this system are the “day to day impact on the quality of student learning” and its allowance of “truly continuous improvement in the learning environment of the student over time.”
Initiating portfolio-based assessment at the institutional level is a daunting task requiring the cooperation and support of a number of different parties. Associate Professor Gregory J. Madden, in the Department of Applied Behavioral Sciences (not a member of the Task Force), cautions against the possible results of an ill-planned portfolio assessment project based on his experience at the University of Wisconsin at Eau Claire. He asserts that unless portfolio assessment becomes part of the university culture, faculty members are paid for their time contributing to assessment, and effective and lasting rubrics are created, portfolios will likely result in few, if any, changes in curriculum. Ann Dueweke, the director of faculty grants and research at Kalamazoo College, cautions similarly about the “institutionalization” of student portfolios. Kalamazoo used portfolios for years before the national move toward assessment began, though as a way for students to reflect on their own work—not, significantly, as a way to perform assessment. When the college began to consider assessment at the institutional level, it initially hoped to shift the purpose of the portfolios over to assessment. Dueweke said that this proved impossible as the portfolios were not given the same attention by students across the board since they were never embedded within a course or a specific academic framework. Jeana Abromeith, chair of the assessment council at Alverno college, also cautions against using student portfolios as a means of assessment if the portfolios are not fully embedded in the institution and connected to clear, transparent educational goals.
It was clear to the Task Force that both standardized testing and assessment of student work can be done well and be done badly. For testing to work well, institutions need to avoid focusing on comparisons across units or campuses; the benefit of well-constructed measures is the precision of the data provided to track improvement over time. The relative ease of data collection and the reliability of the scores would give an easy benchmark to use over time. For rubric based assessment to work, there must be a well structured and supported sampling procedure that is well aligned with the goals to be assessed and sustainable given the professional time available. In both cases, the effort is only useful if the institution actively uses the information gathered to provide focused and selective enhancement of teaching and learning in the areas identified as weaker. Closing the loop, as it is known in this arena, is the single most important part of any program, regardless of the source of the data.
Task Force Recommendations (also found in the Executive Summary)
In light of our review of national practices, examples, and discussions of policy, we offer the following recommendations to KU Faculty Governance:
Clearly identify the purposes for which the evidence of learning is gathered, in consultation with various academic leaders
It is very important to know if the evidence will be used to improve practices and outcomes within the institution or if the evidence will be used to address comparisons between KU and other institutions. The most pressing issue is improving learning within each institution, as that problem is much larger than the differences between institutions. The measures selected and the analysis strategies will be different for data used for comparisons and data used to develop and enhance student understanding. Since the findings will be important to constituents outside the university, this discussion should include those leaders of the university who represent KU to our public.
Develop a prototype system for sustainable sampling and evaluating of student work
Drawing on models from existing large university practices, we recommend creating and using a system for making judgments about student achievement and understanding based on reading of student work. Oklahoma State University, the University of Nebraska, Illinois State University, Washington State University and others serve as good models for efficient and effective sampling and evaluation of student work. These institutions as well as the Association of American Colleges and Universities also provide strong examples of frameworks for evaluation of general education. They provide evidence well suited to improving student learning.
While sampling and evaluating student work, examine the alignment among stated general education goals, the assignments used to generate the work samples, and the courses intended to promote the goals
The process of reading student work and comparing it with criteria matching the goals of general education may give rise to questions about how well the written goals match current students and our educational mission. A close reading of student work provides an ideal occasion for examining the alignment of goals, criteria, and courses, to make sure they work together. Examination of the evidence could lead to adjustments and refinements in any of the three domains.
Continue empirical evaluation of standardized testing in comparison with rating course embedded samples of student work
As KU continues participation in the VSA, there is an opportunity to learn how different methods of measuring learning provide useful information to KU teachers and programs. Faculty governance should encourage ongoing comparison of how the standardized CLA represents learning with how using rubrics to categorize existing course work represents the same kind of learning. In particular, there should be collaboration between the Teagle/Spencer grant evaluating teaching of writing and research skills and the Provost’s Office. Combining those resources can be used to provide a comparison among the traditional KU interviews, the CLA, and organized reading of KU student work. That comparison should help identify what process or combination of processes will be used as evidence of successful general education at KU, and whether there is an advantage to KU to continued participation in the VSA program.
Whatever process is identified for evidence of learning, promote the discussion and use of results beyond simply making them available; the results should be used to identify benchmarks and set internal goals
The primary weakness of any system of measuring general education is the failure to use the information gathered to inform the teaching community and to guide changes in preparation of students’ general intellectual skills. KU’s recent strategic plan, Initiative 2015, should be used as a guide for examining the documentation of learning; changes in teaching and course goals and design should be informed by campus wide conversations about the evidence of understanding. In particular, the campus should focus primarily on how to improve the performance of our own students. Most of the improvement to be gained can be accomplished by identifying local benchmarks and setting internal goals.
Continue to seek and report experience of other campuses
KU should continue to seek out models for successful assessment programs at other institutions, particularly peer institutions, as it moves forward with the assessment process. If one of the goals of assessment is to be comparison among other universities, it will be vital to establish a list of specific peer universities to which to compare the university. The University of Minnesota has utilized this sort of listing in its own assessment process, combining internal assessment and external comparisons into a single, meaningful report of the school’s ability to meet its own established goals. (This report can be found on the Blackboard page: DocumentàFraming Recommendations.) Those institutions with assessment programs already in place will provide both examples and inspiration for the possibilities of university-wide assessment. Institutions such as Oklahoma State University, Illinois State University, Washington State University, the University of Minnesota and the University of Nebraska provide such helpful models. Information on all of these can be found on the Task Force Blackboard page.
Develop public conversations that highlight the uses, benefits and costs of the approaches, sampling the reactions of faculty and other constituencies
Faculty governance should facilitate campus conversations about general education both with faculty and other constituents; these would focus on the costs and benefits of alternative methods of measurement as well as the identification of what is to be accomplished through the measurement. Discussion should include identifying the contributions that faculty would make to the process, thus sampling how much faculty support exists for the effort. There should also be discussion about how the evidence gathered is to be used, both in modifying the curriculum and in reconsidering the academic goals themselves in light of the findings.
Appendix
- KU’s Six General Education Goals
Goal 1: Critical Inquiry
Enhance the skills and knowledge needed to research, organize, evaluate, and apply new information, and develop a spirit of critical inquiry and intellectual integrity.
Goal 2: Integrate Knowledge
Acquire knowledge in the fine arts, the humanities, and the social, natural, and mathematical sciences and be able to integrate that knowledge across disciplines.
Goal 3: Core Skills
Improve the core skills of reading, writing, and numeracy, and enhance communication by clear, effective use of language.
Goal 4: Appreciate Diversity
Understand and appreciate the development, culture, and diversity of the United States and of other societies and nations.
Goal 5: Contemporary Issues
Become aware of contemporary issues in society, technology, and the natural world and appreciate their complexity of cause and consequences.
Goal 6: Social Responsibility
Practice an ethic of self-discipline, social responsibility, and citizenship on a local, national, and international level.
References:
Abromeit, Jeana. Phone Conversation.
Dueweke, Ann. Phone Conversation.
Reyes, Pedro. “Student Learning Assessment in Higher Education.” 2006.
Spencer, Dan. “Liberal Education—Traditional Versus Contemporary Views.” 2008.
McPherson, Peter, and David Shulenburger. “Toward a Voluntary System of Accountability.” (2006)
Wilber, Greg, John Gelder, Frances Griffin, Ed Walkiewicz, Rick Rohrs, Jon Comer, Pam Bowers. “General Education Assessment Committee Annual Report.” 2007
All documents referenced are found on the Task Force Blackboard page.