Viewpoints discussion board

What if the VSA Morphed into the VST?

callen7@illinois.edu — Wed, 15 May 2013 08:15:00 CDT

Good on Doug Lederman at Inside Higher Ed for bringing us up to speed on recent developments related to the six-year old Voluntary System Accountability (VSA). Even though much of what we find today in the way of assessment tools and approaches was either being used on college campuses or on the drawing board prior to 2007, the VSA undoubtedly pushed some aspects of the work further along than would have happened if matters were left to individual institutions. This is surely the case with regard to transparency, a feature of public accountability to which I will return to later.

As Lederman reminded us, the VSA was a timely political response by the postsecondary enterprise that was under what felt like unprecedented scrutiny as to its value. But it was also a much-needed substantive stab at what universities could do to be more forthcoming about their performance to respond to the interests of various parties on and off the campus. Of course, some institutions were well out in front in such efforts. And some national projects, such as the National Survey of Student Engagement, were designed with similar purposes in mind. But in launching the VSA, the Association of Public and Land-Grant Universities (APLU) and the American Association of State Colleges and Universities (AASCU) became the first national institutional membership organizations to lend their imprimaturs to a vehicle designed to encourage universities to measure student learning and to report the findings. Recall that at the time the long-delayed Higher Education Reauthorization was looming. The vast majority of colleges and universities had little to show related to their performance other than what many considered unacceptably low graduation rates (which were artificially dampened by the unfair IPEDS calculation algorithm).

David Shulenburger, the long-serving provost at the University of Kansas, had by then joined the APLU staff, and was one of the architects of the first VSA draft along with Peter McPherson, president of APLU (called the National Association of State Universities and Land-Grant Colleges at the time). McPherson introduced the VSA at the Spellings Commission public hearing in Indianapolis during his testimony as part of a panel on which yours truly was a contributor. I dare say no one else on the panel or the members of the Spellings Commission knew what was coming. Indeed, most of the media attention during and flowing subsequently from that April 2007 event focused on the VSA.

Assessment work has come a long way since the VSA was introduced. And, to its credit, the VSA has both ushered in and attempted to reflect the advances. As Lederman’s article makes plain, the VSA was and continues to be an imperfect solution to a pressing but complicated problem. At the least, its continued presence on the accountability and improvement landscape has prompted others to launch their own transparency efforts, such as the American Association of Community College’s Voluntary Framework for Accountability (VFA), the private sector’s U-CAN, and the short-lived Transparency by Design effort.

To their credit, those responsible for the VSA are contemplating additional changes to make it more useful and, therefore, more attractive to the institutions that must use it if the VSA itself is to be of value. One of the more noteworthy changes to the VSA is to allow universities to populate the College Portrait website of the VSA with multiple forms of evidence of student accomplishment, something that many APLU and AASCU member schools wanted.

Here are four additional challenges the VSA and other similar transparency efforts must address.

Determine which audiences want what kinds of information. External groups such as parents and prospective students have trouble making meaning of test score numbers that supposedly represent the so-called “typical student” enrolled at different universities. They are far more interested in knowing about the experiences of people who are like a particular type of student (oh yes, and they are very interested in cost data!). On the other hand, internal audiences such as faculty and staff may want (or at least expect to see) detailed information— perhaps even dense data displays— accompanied by a careful analysis of which conclusions can be drawn from the data.
Present the information of interest to respective audiences in language that is clear and meaningful. Prospective students, for example, would find helpful seeing and hearing someone explain what someone like them can expect to do if they enrolled at this institution, including the odds that during their studies they would engage in one or more high-impact practices such as study abroad or an internship. Such information, coupled with contextualized interpretations of outcomes measures, will be far more instructive than rows and columns of numbers in the abstract. This suggests that one welcome approach would be a template with links to portals customized for various groups (governing boards, parents, prospective, current and former students, etc.) featuring video snippets from faculty and students. Some institutions do this now, but they are few and far between.
Make accessible information representing both student performance for various program or major fields and student performance data from program/major-fields “rolled up” to represent institutional level performance. One of the criticisms of the VSA (fair, but it also applies to other templates) was that a single number produced by any given test is woefully inadequate to represent the range and depth of learning that occurs on a college campus. In addition, such a number provides little guidance for what faculty and staff could do to improve teaching and learning on the ground—in program or major field classes, labs, and studios—where much of the learning is induced via well-designed assignments and other educationally purposeful tasks. Experiments to aggregate rubric scores at the institutional level such as the initiative at University of Kansas described in Lederman’s article merit our attention. It is a promising, but still challenging frontier for assessment work on campus.
Adopt a qualifications framework to present evidence of student accomplishment across a range of desired outcomes. Inspired by and building on degree qualifications frameworks from other countries, the Degree Qualifications Profile informed by AAC&U’s Essential Learning Outcomes and advanced by Lumina Foundation for Education is one such example. This would nudge the assessment agenda forward by providing some coherence and continuity across individual institutional reports while at the same time allowing a college or university to emphasize distinctive patterns of student proficiencies. Such data would also be much more useful for individual programs and majors committed to identifying places where instruction and student performance need attention. And such an approach could serve as a foundation for documenting the “competencies” (however defined) presented by students whose learning is the product of the combination of, for example, self-selected MOOCs and other delivery systems or life experiences.

To be sure, we are a long way from doing well—at least at scale— the four modifications I’ve briefly described. Taken together, however, they can help de-emphasize the accountability function of learning outcomes assessment and put more weight on using what we are learning to improve the outcomes we seek. This would require, among other things, that institutions be more forthcoming about what they do know about student and institutional performance, and experiment with different ways of reporting this and how they have used the information to what effect.

A year ago, the Kettering Foundation released a report suggesting that the public is less interested in the blunt edges of accountability and more interested in having trustworthy information about such societal institutions as schools and hospitals. After all, the report concluded, most people who pay attention to such matters understand that numbers can be manipulated to tell different stories. What the public wants is assurance that the institution under scrutiny is doing the right things the right way with an eye toward earning and sustaining the public trust. This does not imply ignoring the accountability function of assessment. It does, though, mean that we must be much more transparent about what we are doing. Granted, much remains to be done to develop the kinds of tools and approaches required by both the accountability and improvement purposes of assessment. In the meantime, perhaps if the VSA were to be renamed the Voluntary System of Transparency, it would help focus us more clearly on what we need to do in the near term to enhance student learning and institutional effectiveness.

For additional information about learning outcomes and assessing student learning, see the AAC&U publications, Assessing College Student Learning and Assessing Outcomes and Improving Achievement, and AAC&U’s initiative, Quality Collaboratives.

Where is Culture in Higher Education Assessment and Evaluation?

callen7@illinois.edu — Tue, 14 May 2013 11:15:00 CDT

The Center for Culturally Responsive Evaluation and Assessment (CREA), a new center located in the College of Education at the University of Illinois at Urbana-Champaign, is an interdisciplinary endeavor that brings domestic and international researchers together to address the growing need for practice-relevant and policy-relevant studies that take seriously the influences of cultural norms, practices, and expectations in the design, implementation, and evaluation of social and educational interventions.

Last month, CREA hosted its inaugural conference, “Repositioning Culture in Evaluation and Assessment” in Chicago, Illinois. The conference provided critical insight into the landscape of culturally responsive evaluation and culturally relevant assessment, a space that remains largely uncharted. The conference included internationally recognized keynote speakers and invited panelists, 120 papers, roundtables and symposia and was attended by nearly 300 participants and interested visitors from the U.S.(including participants from Hawaii and Alaska), indigenous nations, and seven non-US countries (including Germany, Canada, Ireland, New Zealand, Greece, Australia and Denmark).

Notably absent from the conference was significant discussion of issues surrounding the role that notions of cultural responsiveness, cultural awareness, and cultural relevance play in evaluating higher education programs and services as well as in assessing higher education outcomes. What little discussion there was of these matters was largely limited to consideration of pedagogical practices and assessment of teacher performance in the context of higher education teacher preparation programs.

Assessment plays a critical role in higher education planning and accreditation. CREA recognizes the importance of assessing learning outcomes at post-secondary institutions and is mindful of the challenges to developing and implementing assessment practices; challenges that are frequently discussed through NILOA’s viewpoints and occasional papers. However, we must be concerned about the limited evidence and lack of sustained discussion on how program and institutional assessment frameworks are responsive to the cultures and cultural contexts of the diverse populations served by institutions of higher education.As the demographic composition of many postsecondary education institutions continues to change, it is imperative that culture is included as a central consideration in designing and implementing systems of assessment. Failure to do so will threaten the validity of inferences about institutional performance based on assessment data.

It is our hope to stimulate discussion at both policy and practice levels of the ways in which higher education institutions document how culture is identified and engaged in assessment planning. There is a growing body of literature that recognizes the critical role of culture in assessment at all levels in P-12 education and provides at least a starting point from which higher education scholars can build. CREA looks forward to supporting these conversations and continuing to advance the central role that culture plays in assessment theory and practice at all levels of education.

About CREA:

CREA’s mission is carried out by a culturally diverse pool of highly trained evaluators, assessment specialists, researchers, and policy analysts from organizations and institutions around the world. Please visit the CREA website for more information about resources and services: www.education.illinois.edu/crea . To join the CREA mailing list, please send your request to crea@education.illinois.edu.

Embedded Assessment and Evidence-Based Curriculum Mapping: The Promise of Learning Analytics

callen7@illinois.edu — Mon, 15 Apr 2013 08:15:00 CDT

Assessing desired learning outcomes effectively is a complex endeavor. Risking over-simplification, I propose we accept the definition offered by Randy Swing (2010) “Assessment is change management.” If we admit that premise and partner it with “Data are resources; people are the agents of decisions and change” (Kramer, Hanson, Olsen, 2010, p.44), then the locus of control for assessment and therefore change, can reside with the people generating the data. In my school, The Wegmans School of Pharmacy at St. John Fisher College in Rochester, New York, that means the faculty.

At Wegmans School of Pharmacy, we have adopted an embedded assessment approach to curriculum mapping and data collection on student learning outcomes achievement. In essence, we capture the data from the course level exams that our faculty members craft to measure student learning. We have accepted the approach proposed by Linda Suskie (2009, p.5), “Assessments that are embedded into individual courses can often provide information on student achievement or program goals, general education goals, and institutional goals.” Furthermore, “…a more deliberate use of existing measures of student success can provide incremental evidence of student learning and move us toward meeting the call of accountability” (McCarthy, Niederjohn & Bosak, 2011, p.81). With this is mind, we tag every question on all course-level exams with multiple codes corresponding to program outcomes, course learning outcomes, and level of Bloom’s Taxonomy. The resulting data are entirely generated from information embedded in our coursework. No additional testing is needed, which addresses a concern of Janet Fontenot’s (2012) that faculty are leery of the additional time required on their part for assessment activities. Furthermore, with embedded assessment the faculty members are the principal source of data, and the people who have the most control over the management of change suggested by data analysis.

At the end of the semester, longitudinal reports are generated that map the frequency of questions related to each course and program level outcome. This density table appears alongside the student achievement for each of those outcomes. For example, data reveal that students were questioned on immunology 45 times with any average percentage correct of 86.5. From a curricular perspective we create an evidence-based curriculum map. Instead of declaring that we address X program outcome in Y course, we now demonstrate that X outcome is tested 45 times with a result of 86.5% student achievement. We analyze the percentage of questions asked at the knowledge, application, and synthesis levels. Based on the data, we recommend change.

At the course level, faculty track coverage of their individual course outcomes. If student achievement is low in one area, then the instructor knows where to spend time in class. The validity of the questions is checked through a peer review process. The faculty member also has descriptive statistics for individual test items, which can help identify poor performing questions which can then be modified or retired. Faculty members also capture reports of student performance on their course learning outcomes to include in their dossiers as evidence of effective teaching.

At the student level, the bi-semester longitudinal reports reveal areas of strength and weakness. For example, a student presented with a 90+ average that appeared to have no areas of concern. After reviewing her longitudinal report, we saw that on questions spanning all coursework involving calculations she earned a 70% average. She now knows where to direct her efforts prior to taking the North American Pharmacist Licensure Examination. After students are presented with their longitudinal reports, they are required to write a reflection on their performance and briefly describe steps they will take to address areas of concern. In this way, we are encouraging students to take responsibility for their learning.

In the Horizon Report: 2013 Higher Education Edition, learning analytics is suggested to be on the two-to-three year horizon. The report cites, “The promise of of learning analytics is actionable data relevant to every tier of the educational system…A key outcome of learning analytics pertains to the student on an individual level…” (p.24) I would suggest that the seeds of learning analytics have already been planted. While we still have far to go, we have begun mining large banks of data to do precisely what is suggested in this report: use the data sets to target areas for improvement. We use our data to inform change at the program, course and student levels.

Overall, our embedded assessment and evidence-based curriculum mapping approach has been well received in the school. We are realizing advantages at the programmatic, course, faculty, and student levels. We believe acceptance of this new approach is in large part to the keeping it simple, consistent with Swing’s advice that assessment is change management and insuring that subsequent changes in curriculum and assessment emanate close to the data source.

References

Fontenot, J. (2012, July 11). Faculty concerns about student learning outcomes assessment. [National Institute for Learning Outcomes Assessment Web log comment]. Retrieved from http://illinois.edu/blog/view/915/76774?displayOrder=desc&displayType=none&displayColumn=created
&displayCount=1

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., and Ludgate, H. (2013). NIMC Horizon Report: 2013 Higher Education Edition. Austin, Texas: The News Media Consortium.

Kramer, G.L., Hanson, C. & Olsen, D. (2010). Assessment frameworks that can make a difference in achieving institutional outcomes. In Kramer and Swing (Ed.). Higher education assessments: Leadership matters (pp.27-56). Landham, MD: Rowman & Littlefield.

McCarthy, M.A., Niederjohn, D.M., & Bosak, T.N. (2011). Embedded assessment: A measure of student learning and teaching effectiveness. Teaching of Psychology, 38(2) 78-82.

Suskie, L. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.

Swing, R.L. (2010). Supporting assessment: cost/benefit considerations. The New England Association of Schools and Colleges annual meeting and conference. Boston, MA.

The DQP and the Creation of the Transformative Education Program at Saint Augustine's University

dumas1@illinois.edu — Wed, 13 Mar 2013 09:30:00 CDT

Saint Augustine’s University in Raleigh, North Carolina, one of the 25 institutions in the CIC/DQP Consortium supported by a grant from the Lumina Foundation, has used the DQP to leverage a transformation of its academic programs in record time. A Transformative Education Program (TEP) for General Education and revised courses in every major were implemented in the fall semester of 2012, just 7 months after the project began. While the Saint Augustine’s experience may be difficult to replicate in other institutional contexts, it does suggest how the DQP can focus and direct plans to modify the curriculum.

The project began with a challenge from Saint Augustine’s president, Dr. Dianne Boardley-Suber, to transform the institution. One aspect of the DQP model that was highly valued by the University was a curriculum that would foster development of cumulative skills, leading students to higher levels of attainment over the course of their education. The General Education Committee was charged with researching other higher education institutions in order to find best practices in pedagogy and assessment that would be compatible with what Saint Augustine’s was trying to do. The TEP core committee conducted surveys of workforce needs in order to map those skills with their courses. Focus groups including faculty, staff, alumni, board members, student leaders and students met and defined the Signature Saint Augustine Student (SSAS) using a set of skills and competencies. Then, faculty met in their departments and schools and as a whole to discuss how and where in the curriculum students would acquire these skills and competencies.

These activities culminated with the stakeholders identifying 10 core competencies that replaced the former general education program:

communications
critical thinking
identity
wellness
STEM and quantitative literacy
civic engagement
global perspective
servant leadership and teamwork
innovation and creativity
artistic literacy
a capstone experience.

These 10 competencies are the Saint Augustine’s University version of the five domains of the DQP spider web (broad integrative learning, intellectual skills, specialized knowledge, applied learning, and civic learning) with “identity” (which includes requirements in African American history) fitting into the 6th domain on the DQP spider web, institution-specific outcomes.

During the spring and summer of 2012, including two week-long all-faculty institutes, the faculty aligned existing and new courses to the core competencies of the TEP, designating courses as T1 (directly supporting a core competency) or T2 (enhancing a core competency). They also identified assessment tools to evaluate the T1 and T2 courses and underwent training on course design, use of rubrics, and other pedagogical strategies. One of the key tactics was the creation of an Implementation Committee (IC), the members of which represented each of the areas of the project plan. The IC communicated their findings to the campus community regularly and solicited ideas from the campus for realizing the ambitious goals of the project.

Two syllabi illustrate that the influence of the DQP is not limited to traditional general education courses.“Foundations of Computer Science” is a TEP T1 course and its syllabus clearly connects the course to the new curriculum. One of the most important DQP concepts, that academic programs should foster cumulative learning, is embodied in the fact that the 3 rubrics on the syllabus describe levels of learning (4 for technological learning and 5 for written and oral communication). Another application of an important DPQ concept, that course expectations be clearly communicated to students, is illustrated in the extensive detail in course assignments, in the clearly described assessment tools used for the assignments, and in the relationship between the course goals and assignments and the institutional mission goals and computer science program goals. For example, in order to achieve the learning objectives for technological literacy, the student must 1)” demonstrate proficient use of various foundational software applications including applications such as word processing, spreadsheets, database, and presentation software,” 2) “demonstrate an understanding and use of the internet and email platforms for information acquisition and communication,” 3) “trouble shoot and solve common technological problems using various programming languages including C++, Java, and Visual Basic,” 4) “demonstrate the ability to design a Webpage or similar project.” Two assignments that measure these 4 outcomes are the development of a web page including graphics, motions, and hyperlinks and an app called “Change Machine” using any of the programming language to divide change into different coin denominations. The Assessment Rubric for technological literacy shows four different levels for the four learning outcomes. For the “benchmark” level for information acquisition, the student does not acquire or evaluate electronic information. For “milestone 1” the student must employ one strategy to acquire and evaluate information using electronic sources to meet the standard at a minimal level. For “milestone 2” the student must use at least three strategies for acquiring information electronically and effectively evaluates the information. For the “capstone” level the student uses multiple strategies for acquiring information electronically and evaluates it effectively. The syllabus also includes a rubric describing in detail the learning outcomes that are required for an A, B, C, or D grade for the web page development assignment. The focus of the course is on developing a basic understanding of computer science and a basic level of technological literacy, but the course is also identified as “writing rich,” but not “speaking rich” (even though some oral skills are developed in the course assignments). Students taking this course are reminded of the broad goals of Saint Augustine’s University Transformative Education Program.

The syllabus for “Role of Sport in Society” states that the course is designated as “writing rich,” “speaking rich,” and “research intensive,” but explicitly states that it is not a service learning course. The course can be used to satisfy part of the Global Perspectives requirement in the Transformative Education Program (TEP). The syllabus shows that the course assessment tools cover two of the six learning objectives for Global Perspectives: “The student can describe cultural differences using verbal and nonverbal communication and begins to negotiate a shared understanding based on those differences” and “The student can develop questions about other cultures and seek out answers to the questions.” These 2 learning objectives are assessed in the assignment to write an international sport comparison research paper and then in the assignment to do a class presentation on the paper. The rubric for Global Perspectives lists four levels of achievement for the six learning outcomes. For the learning outcome “Knowledge of cultural worldview frameworks,” the benchmark level for students is "Demonstrates surface understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.” At Milestone 1, the student demonstrates “partial understanding” of the complexity of these elements, at Milestone 2 “adequate understanding” and at Capstone level “sophisticated understanding” of the same elements. The syllabus for this course includes the same extensive coverage of institutional goals, program goals, and course goals as the syllabus for “Foundations of Computer Science” and also exemplifies the curricular commitment to fostering increasing levels of learning and to clearly communicating expectations.

Every TEP syllabus at Saint Augustine's University includes an assessment tool. The TEP, as the DQP model suggests, involves the entire educational program, not just a collection of general education and required courses. A major curricular revision such as that done at Saint Augustine's University requires strong leadership, campus-wide agreement on mission and focus, and faculty, staff, and student leadership willing to do the very intense work in order to meet the challenge of improving student learning.; however, without the coherent structure of the DQP to provide a model for a broad and clearly articulated curricular structure grounded in extensive, interconnected, and ongoing assessment of student learning, it seems likely that the process of transformation would have taken much longer to realize.

Why Student Learning Outcomes Assessment Is Key to the Future of MOOCs

baker44@illinois.edu — Mon, 03 Dec 2012 10:00:00 CST

Measuring Success in Internationalization: What are Students Learning?

dumas1@illinois.edu — Wed, 14 Nov 2012 09:15:00 CST

Over the past 30 years, various blue-ribbon commissions, association reports and studies have highlighted U.S. students’ woeful lack of foreign language competency and literacy about world geography, politics, and history. The events of September 11, 2001, gave new urgency to the message, providing a wake-up call about the importance of educating Americans about the rest of the world and our inextricably entwined fates. This is not to say that U.S. higher education has been totally inactive with respect to internationalization. Many institutions have had long-standing international partnerships and study abroad programs, and have hosted impressive numbers of international students over time. Yet, rarely are institutional internationalization efforts strategic or coherent, or considered to be central to their academic mission or definition of quality. As is always the case, there is tremendous variation in the quantity, quality, and coherence of internationalization across campuses. But as the race to internationalize intensifies, it becomes all the more important to proceed with intentionality.

For most institutions, “success” in internationalization is judged by a series of widely-used indicators of institutional performance, such as numbers of students going abroad, numbers of international students, or courses offered with an international or global focus. And as institutions around the world take up the challenge of internationalization, a robust literature has emerged outlining institutional indicators that help institutions judge their progress as well as to benchmark (see for example www.impi-project.eu, www.impi-toolbox.eu, www.nuffic.nl/international-organizations/services/quality-assurance-and-internationalization; www.nuffic.nl/mint, http://www.iau-aiu.net/content/global-surveys.)

What these institutional activities mean for student learning is a different matter. Although many institutions cite producing “global citizens” as a goal, few have a clear set of learning outcomes associated with this label, a map of the learning experiences that produce such learning, or an assessment plan in place to determine whether they have achieved their goals. Clearly, institutional performance and the student learning perspectives can be related to each other, but one cannot assume causality in either direction. As anyone who has been engaged in assessing student learning knows all too well, the presence and quality of a given set of institutional activities or the participation rates in various courses or programs do not tell you anything about what students are learning.

The field of education abroad has begun to seriously engage in the question of outcomes. It is no longer deemed acceptable in the field to cite the “it changed my life” argument as the self-evident truth of the positive impact of education abroad. The rapid growth of short-term education abroad programs has put into sharp relief the relationship of the learning achieved in these experiences to different program durations and pedagogies. As students go abroad for shorter periods of time, and are more likely to do so in a faculty-led program in the company of fellow U.S. students, it becomes even more important to determine the impact of these experiences on subject-matter learning, increased global awareness, and development of intercultural skills. The same questions must be asked of longer, more conventional programs, for there is no guarantee that “being there” produces learning, let alone “transformation.” The good news is that institutions are taking up this challenge, increasingly using pre-and post-tests, journals, and portfolios to capture student learning in education abroad (for a list of useful research and resources, see http://www.nafsa.org/resourcelibrary/Default.aspx?id=31791.)

Although education abroad receives a great deal of attention nationally, it is not synonymous with international or global learning. Although it is difficult to estimate the proportion of students who study abroad for credit during their undergraduate year, we do know that only 270,000 students out of more than 20 million enrolled in postsecondary education studied abroad in 2010(IIE, 2011; NCES, 2012). Thus, the key question for higher education institutions is how the overwhelming majority of students who do not go abroad will learn about the world and develop the intercultural skills they will need as citizens and workers. To address this question, institutions will need to be very clear about what knowledge and capacities students must learn, where and how they will learn them, and what constitutes evidence of such learning.

Many institutions begin this work by including global learning as one or more of their stated goals of liberal education. And they need not reinvent the wheel in crafting a specific set of goals. The Association of American Colleges and Universities includes intercultural learning as one of the 15 essential learning outcomes of its VALUE initiative, (http://www.aacu.org/leap/vision.cfm) and also provides specific goals for liberal education and global citizenship (Musil, 2006). The American Council on Education also has also developed a list of global learning goals with institutional examples, drawn from the literature, and categorizing them under knowledge, skills, and attitudes (see Olson, Green, and Hill, 2006). It is important to note, however, that these first steps of stating global learning as a goal and crafting more specific goals are only the beginning of an ongoing process.

Identifying which courses and programs actually enable students to acquire these skills and competencies is more difficult work. Having a global or international requirement as part of the general education sequence is one common way to ensure that every student gets at least a small dose, but certainly not the only one. Institutions also need to look at majors, programs, and individual courses, to map which ones address specific global learning goals.

The next step involves assessment. It is through assessment that institutions can find out whether they are really producing “globally competent” graduates, “global citizens,” or graduates who can navigate multicultural situations. And finally, institutions must take the crucial step of “closing the loop” (Banta & Blaich, 2011) by applying what they learned from assessment to improving curriculum and teaching. I recently produced (2012) a detailed guide on steps in assessing global learning and examples of good practice.

As long as success in internationalization is measured largely or solely by institutional performance, colleges and universities will be missing the mark. Although internationalization, alas, is increasingly a matter of numbers, profile, and branding, the real measure of success should be how well students are equipped to live and work in a rapidly changing global environment.

References:

Banta, T. W., & Blaich, C. (2011). Closing the assessment loop. Change: The Magazine for Higher Learning, 43(1), 22–27.

Green, M. (2012). Measuring and assessing internationalization. Washington, DC: NAFSA: Association of International Educators. Retrieved from www.nafsa.org/epubs

Institute for International Education (2011). Open Doors 2011: Fast Facts. Retrieved from http://www.iie.org/en/Research-and-Publications/Open-Doors

Musil, C. (2006). Assessing Global Learning: Matching Good Intentions with Good Practice. Washington, DC: American Association of Colleges and Universities.

National Center for Education Statistics (2012). Fast Facts: Enrollment. Retrieved from http://nces.ed.gov/fastfacts/display.asp?id=98

Olson, C., Green, M. & Hill, B. (2006). A Handbook for Advancing Comprehensive Internationalization: What Institutions Can Do and What Students Should Learn. Washington, DC: American Council on Education.

Demonstrating How Career Services Contribute to Student Learning

dumas1@illinois.edu — Wed, 14 Nov 2012 09:00:00 CST

Career services are too often thought of separate from the core learning activities in which students engage in classrooms, laboratories and studios. But dividing what students gain from college into academic learning (what happens in the classroom) and personal development (what happens outside of the classroom) is a byproduct of historical, physical structures in higher education institutions. Thinking about student accomplishment as bifurcated in this way does not serve either students or institutions well, as students grow, develop, and learn in a holistic fashion.1

Career services professionals are well-positioned to bridge the gap between the academic learning and personal development outcomes. They are educators who help students learn how to explore career options, make career decisions, and develop career management skills that students will use throughout their lifetime. Career interventions are the medium through which career professionals provide these learning opportunities to students. We intentionally use the term “career interventions,” as opposed to “programs, services, and resources,” as the latter are static entities that focus attention on the activities that are carried out by career professionals. Interventions, on the other hand, focus on the process of helping students change, develop, or move from point A to point B – essentially, to help students learn.

Viewing career interventions as learning opportunities sets a high bar in terms of evaluating the effectiveness of career services on higher education campuses. If education and learning are to be at the core of career services, then learning outcomes must be measured to evaluate success. Fortunately, many career professionals in higher education are no strangers to the process of collecting evidence to show the influence of their work. They have been involved in gathering and interpreting data since the earliest days of the profession. Historically, the most common data-driven strategies for demonstrating the influence of career services have been counting participants, measuring satisfaction, and tracking placement rates. These data tell important parts of the story of career interventions – which students take part, how comfortable students are with their experiences, and where students go after their college experiences.

However, these types of data have limitations that fall short of what contemporary times demand. When participation rates, satisfaction, or placement numbers are low, few insights can be gained regarding how or what to improve. Additionally, these strategies can encourage a “more is better” focus on increasing quantity with little attention to quality. A different approach is needed for career professionals to demonstrate the quality of their career interventions and the difference that career interventions make in students’ lives. Career professionals must build upon past data collection and analysis experiences to rise to the challenge of conducting rigorous, meaningful learning outcomes assessments.

Conducting learning outcomes assessments can be daunting for many career professionals, as it is with many other higher education faculty and staff. One approach is to focus on well-defined programs and services that have clear boundaries – a resume review, a career exploration workshop, career counseling appointments, and so on. Doing so can help keep assessment efforts manageable. Additionally, career professionals can often clearly identify interested parties, such as prospective and current students, families, institution administration, faculty, and student affairs staff who want to know whether career interventions make a difference in students’ lives. Understanding what these audiences want to know offers career professionals clues regarding the types of learning outcomes to focus their assessment efforts on, as well as with whom to share the results.

Furthermore, career services professionals often draw upon established guidelines (e.g., CAS Standards, National Career Development Guidelines)2 and theories (e.g., Holland’s Typology, Planned Happenstance, Social Cognitive Career Theory, Super’s Career-Life Roles) 3 to inform their career interventions. For example, the National Career Development Guidelines offers specific outcomes statements regarding the mastery of career management skills. One such skill, related to career decision-making, is that career development clients should be able to take into consideration how personal priorities, culture, beliefs, and work values affect their decision making by: (a) recognizing the role that these influences play in decision making, (b) showing examples of how these influences have affected them in the past, and (c) evaluating the impact of these influences in current career decision-making processes (p. 10). Attending to the content of guidelines and theory documents such as this helps career professionals clearly express desired milestones or outcomes of career interventions that can be measured in assessment efforts.

Many career professionals are well-positioned to demonstrate the value of their services and career interventions through learning outcomes assessment. A continuing challenge is for career professionals to find a way to embrace the assessment of student learning in their day-to-day practice. Our advice is to think of assessing learning in terms of building a house. That is, have an overall plan, and start with activities that are of reasonable scale, and gradually build up, brick by brick. Focus on laying one brick at a time, no matter how small, and building a strong foundation—a rich body of evidence. Small, positive experiences with learning outcomes assessment can teach useful skills, build confidence and capabilities, and motivate future learning outcomes assessment efforts.4

More information on how career professionals can get started with learning outcomes assessment are described in a recent monograph from the National Career Development Association, entitled: Learning Outcomes Assessment Step-by-Step: Enhancing evidence-based practice in career services (http://tinyurl.com/7sy6krl).

1 Ideas presented in this paragraph are influenced by scholarship such as:

American College Personnel Association. (1996). The Student Learning Imperative. Washington, DC: Author. Retrieved from http://www.acpa.nche.edu/sli/sli.htm

American College Personal Association, & National Association of Student Personnel Administrators. (2004). Learning reconsidered: A campus-wide focus on the student experience. Washington, DC: Author.

2 References for sample professional guidelines:

Dean, L. A. (Ed.). (2009). CAS Professional Standards for Higher Education (7th ed.). Washington, DC: Council for the Advancement of Standards in Higher Education.

America’s Career Resource Network. (2004). National Career Development Guidelines. Retrieved from http://associationdatabase.com/aws/NCDA/asset_manager/get_file/3384?ver=16587

3 References for sample career development theories:

Lent, R. W., Brown, S. D., & Hackett, G. (1994). Toward a unifying social cognitive theory of career and academic interest, choice, and performance [Monograph]. Journal of Vocational Behavior, 45, 79–122.

Mitchell, A., Levin, A., & Krumboltz, J. D. (1999). Planned happenstance: Constructing unexpected career opportunities. Journal of Counseling and Development, 77, 115-124.

Reardon, R. C., & Lenz, J. G. (1998). The Self-Directed Search and Related Holland Career Materials: A practitioner’s guide. Odessa, FL: Psychological Assessment Resources, Inc.

Sampson, J. P., Jr., Reardon, R. C., Peterson, G. W., & Lenz, J. G. (2004). Career counseling and services: A cognitive information processing approach. Belmont, CA: Brooks/Cole.

4 Ideas presented in this paragraph are influenced by scholars such as:

Keeling, R. P., & Associates. (2007, June). Putting Learning Reconsidered into practice: Developing and assessing student learning outcomes. Workshop presented at the National Association of Student Personnel Administrators’ Learning Reconsidered Institute in St. Louis, MO.

Schuh, J. H. (2009). Assessment methods for student affairs. San Francisco, CA: Jossey-Bass.

Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco, CA: Jossey-Bass

The Culture Change Imperative for Learning Assessment

dumas1@illinois.edu — Tue, 16 Oct 2012 08:45:00 CDT

The September 2012 issue of the NILOA Newsletter included NILOA’s 15^th Occasional Paper, “The Seven Red Herrings About Standardized Assessments in Higher Education,” written by Roger Benjamin, with a foreword by Peter Ewell, and including commentaries by Margaret Miller, Terrel Rhodes, Trudy Banta, Gary Pike, and Gordon Davies. The points made in that paper for and against standardized tests of student learning are provocative and clarifying but, as Ewell noted, they are arguments with which we are already quite familiar. Ultimately, how best to assess learning for the purposes of furthering learning and accountability is a question for empirical inquiry that can draw on powerful resources in higher education expertise.

Much assessment of learning already exists on campuses. The vexing problem, however, is that little of it consistently and coherently signals to students the institution’s expectations and the standards shared by faculty and staff. While there are faculty members on every campus who practice exceptionally inventive and effective assessment, such practice rarely is pervasive such that it both purposefully contributes to all students’ learning and informs an institution-level portrait of student learning and development.

Why is systemic assessment still so rare? The answer is embedded in the papers mentioned above: Resistance to learning assessment is in the DNA of the academy’s current culture. Benjamin speaks of this charitably as “institutional inertia.” Davies, more bluntly, notes that despite rhetoric to the contrary, neither higher education’s values nor its rewards for the individuals within it have changed. The academy’s incentive and reward system is not about student learning but about institutional prestige measured by selective admissions, endowment, and research prowess. As Davies put it, “Colleges and universities have a huge investment in the status quo, and they are not likely to support changes that may be needed in what and how they do it.”

But as considerable research and many critics point out, the status quo is no longer tenable. A culture change in higher education is imperative. Far too many college graduates have not achieved widely accepted and significant higher learning outcomes such as the ability to think critically and creatively, speak and write cogently and clearly, solve problems, comprehend complex issues, accept responsibility and accountability, or understand the perspective of others. The central contributor to this learning crisis is culture—both the larger culture surrounding the academy and that within colleges and universities themselves. With regard to the latter, the shared norms, values, standards, expectations, and priorities of teaching and learning on most campuses are not powerful enough to support true higher learning. We do not demand enough from students; our standards are not high enough; we accept half-hearted work from students who have not asked enough of themselves; and we do not support students in asking for more from their teachers. Degrees have become deliverables (purchased, not earned); credit hours are accumulated and courses passed with little concern for coherence or quality because we are not willing to make students work hard to attain shared high standards to earn them. As a result, students do not experience the kind of integrated, holistic, developmental, rigorous undergraduate education they absolutely must have for truly transformative higher learning to occur.

To put student learning at the center of each institution’s work demands that we know the extent to which learning is occurring and that we provide timely and appropriate feedback to students and teachers. To change institutional culture requires that we recognize and embrace the cumulative and collective nature of higher learning and the powerful role that learning assessment plays in outcomes of that nature. Thinking critically and writing creatively, for example, are skills learned cumulatively over the span of the entire undergraduate program. Objectives and standards for excellence in these skills must be shared—intentionally articulated, planned around, and assessed by faculty and staff across all courses and programs. Higher learning requires far more instruction, practice, assessment, and feedback than is currently provided or expected within single courses or other isolated learning experiences.

The assessment challenge of cumulative learning is that it requires faculty to come together—collectively—and to agree on which outcomes, expectations, and standards they share and endorse, and then, throughout their various courses and programs, to reinforce these outcomes, expectations, and standards. The assessment of cumulative learning demands change in the institutional culture of learning, change that requires faculty to significantly raise their expectations and standards for learning outcomes and that ensures the adequate formative and summative assessment of those outcomes. Outcomes, expectations, and standards, moreover, must be transparent. When students engage with faculty and staff in pursuing transparent, institution-wide outcomes, expectations, and standards, and when they receive frequent and appropriate feedback, higher learning improves. In this sense, learning assessment is best understood not as an external imposition by the state or administration but rather as a powerful dimension of teaching and learning derived, practiced, and promoted by faculty and staff to improve the quality and quantity of undergraduate learning.

Given the cumulative and collective nature of higher learning, establishing and sustaining a conscientious, diligent, rigorous, campus-wide regime of learning assessment requires changes not just in attitudes but also in campus policies and commonly agreed practices to advance and sustain a more intentional learning culture. Learning assessment, for example, should not be the burden of a small knot of dedicated faculty and staff who understand its benefits and are willing to suffer its additional costs; when that happens, exhaustion, disenchantment, and frustration are inevitable.

To say that academic culture change—however imperative—is hard is an understatement. The work culture of academia rightfully offers each individual faculty member a great deal of freedom for independent judgments about the aims and content of learning. Yet relationships, not just between faculty and administration but also among faculty members themselves, create cultural and power barriers that are difficult to overcome. Constructing shared outcomes, expectations, standards, and assessment tools, and conducting effective learning assessment requires precious time and effort. Incentive and reward systems are currently skewed against such change. Reappointment, promotion, and tenure criteria need to be adjusted to align with these greater expectations for teaching and for the more time-consuming engagement with students that effective learning assessment requires. Given the limits of most doctoral programs, faculty and staff need better opportunities to learn more about appropriate assessment and how to implement it. And, of course, myriad pros and cons arise with the issue of comparing similar institutions to develop learning benchmarks.

The task list above is hardly exhaustive. This kind of change, ultimately, may be less about expertise and more about will. Changing the academic culture requires sustained, shared, courageous leadership by faculty, staff, administration, and governing boards. Anything less invites those outside the academy to act as referees, which is never good for either the academy or the NFL.

Richard H. Hersh, formerly president of Hobart and William Smith Colleges and Trinity College (Hartford), currently serves as senior consultant for Keeling & Associates, LLC, a higher education consulting practice. Richard P. Keeling, formerly a faculty member and senior student affairs administrator at the University of Virginia and the University of Wisconsin–Madison, leads Keeling & Associates. Hersh and Keeling are the authors of a new book, We’re Losing Our Minds: Rethinking American Higher Education (Palgrave Macmillan, 2012).

Comments on the Commentaries about 'Seven Red Herrings'

dumas1@illinois.edu — Tue, 16 Oct 2012 08:30:00 CDT

I am pleased to accept the invitation to briefly respond to some of the points made by those who commented on my “Seven Red Herrings” paper which appeared in the September 2012 issue of the NILOA monthly newsletter. In his Foreword, Peter Ewell predicted that the merits and role of standardized testing will almost certainly continue to be debated. With this in mind, I also offer a few thoughts about what to expect in the future.

Trudy Banta, Gary Pike, and Terrel Rhodes view the promise and potential of standardized testing differently than Margaret Miller and Gordon Davies. Miller sees standardized measures as essential, because the field demands highly reliable and valid assessment tools. At the same time, she believes formative assessment is important as well, albeit for different purposes. Davies goes a step further by saying that colleges and universities must use standardized student learning outcomes measures to assure the public of that these institutions continue to make meaningful, valued contributions both to individuals and the larger society.

Banta and Pike represent the formative end of the assessment continuum. Most of the arguments they presented in their commentary about standardized assessment measures, particularly the Collegiate Learning Assessment (CLA), have appeared previously. Many of their points have been addressed by CLA staff, the Educational Testing Service (ETS), and other researchers, including a summary of approximately 90 studies (Benjamin, et al. 2012). Although my paper was not about the CLA per se, it is worth summarizing several cogent responses available elsewhere to the Banta and Pike’s main arguments.

For example, average CLA value-added scores are highly reliable especially at the institution level (freshmen=.94; seniors=.86). Aggregate student motivation is not a significant predictor of aggregate CLA performance, and does not invalidate the comparison of colleges based upon CLA scores. Moreover, the types of incentives that students seem to prefer are not related to motivation and performance.

Although we continue to believe that a no-stakes approach is appropriate for the value-added model in higher education, motivation is a problem for individual student results. CAE (Council for Aid to Education) now offers a version of the CLA protocol, CLA+, which is reliable and valid for individual student performance, as does the Education Testing Service with its Proficiency Profile, and the American College Testing Program with its Collegiate Assessment of Academic Progress. It may well be appropriate in the future to attach stakes to the CLA, which, in turn, likely will increase student motivation to do well.

There is no interaction between CLA task content and field of study. Our researchers find that the CLA protocol measures 30% of the knowledge and skills faculty desire. Results are improved significantly if a representative sample is drawn. Finally, that the CLA is highly correlated with the SAT does not mean the two tests measure the same thing. High school grades combined with the CLA predict freshmen and senior GPA at about the same level as the SAT alone. High school grades plus the SAT and CLA generate a higher prediction than either test alone. This would not be true if the SAT and CLA measured the same thing.

Banta and Pike are correct in advocating a focus on disciplines, but stray off track by rejecting that standardized test can accurately measure generic cognitive skills (Benjamin et al. 2012). The mean size effect of the growth in student learning outcomes for all colleges testing annually for the past eight years is approximately .73 standard deviations, demonstrating that college attendance is associated with improving these skills.

Banta and Pike suggest there is qualitative evidence to buttress their claims. It would be helpful to know the evidence to which they refer. Measurement scientists privilege statistical-based evidence. This makes conversation between the two groups difficult. Elsewhere I (Benjamin, 2012) explained what I call the assumption of the equality of fields of inquiry. Faculty members are reluctant to question the legitimacy of fields of inquiry that they may not be familiar with. There are solid reasons for this assumption. For example, an obscure field of molecular biology in veterinary medicine focusing on retroviruses in monkeys was critical in helping researchers develop treatments for AIDS. Breakthroughs in one scientific field may lead to startling breakthroughs in others. Measurement science is a field of inquiry that is too well established to be dismissed by colleagues arguing for formative assessments only. For example, Banta and Pike and Rhodes make good arguments for using e-portfolios to assess student learning. However, e-portfolios do not yet pass muster as tools that are sufficiently reliable and valid to obviate the need for appropriate standardized tests for decisions with stakes attached.

Both Davies and Miller want testing organizations to make public student outcome test results. What I should have said was that external demands will require institutions to make their student learning outcomes transparent and that peer review principles aligned with core values of the academy will provide foundational support for higher education leaders creating assessment reporting systems

Peter Ewell noted that faculty prefer to keep assessment results confidential, for internal use only. It is worth noting that testing organizations can achieve greater economies of scale in test development which lowers the price of individual assessments. Aided by recent developments in education technology, there appears to be a burst of innovation in creating assessments for direct use by faculty as instructional tools. Finally, samples of students tested at individual institutions are seldom large enough for the results to be considered sufficiently reliable. More widely used standardized assessments can boost confidence in the results found at individual institutions.

What We Can Expect

The competency-based model now gaining considerable traction will require assessments that corroborate the efficacy of the student learning claimed. Many of those assessments will be standardized tests. There is and will continue to be ample room for formative and standardized tests in postsecondary education. The issue is how to better leverage the virtues of both, for the benefit of improved teaching and learning for the larger societal goals Davies posited.

This, then, is not the time to defend the status quo. Many colleagues may be comfortable in defending positions that marginalize assessment in postsecondary education. Because increasing numbers of private and public leaders believe human capital is the nation’s principal resource, debates about how to improve education will continue to grow. The rise of Internet-based education and concerns for the quality of higher education provided by more traditional means are fueling external demands for increased transparency, restructuring, and accountability.

External demands for benchmarking student learning outcomes are destined to increase. However, higher education institutions possess a high level of legitimacy and relative autonomy anchored by department-based governance. The initial challenges for increased transparency of student learning outcomes will come from external forces. Responses to these demands will be developed by innovators within the higher education community. We need all hands on deck to experiment with ways to improve teaching and learning.

Finally, higher education institutions must respond to persistent external demands for more systematic evidence about student learning outcomes. In doing so, the enterprise must also maintain faculty autonomy in determining appropriate assessment approaches; reject college and university ranking systems; privilege efforts to improve student learning; develop assessment protocols that combine standardized and formative assessments; and adhere to peer review principles when constructing accountability systems. About this last observation, there seems little to debate.

References

Benjamin, R. (2012). The new limits of education policy: Avoiding a tragedy of the commons. London: Edward Elgar.

Benjamin, R. Elliot S., Klein S., Steedle, J., Zahner, D., & Patterson, J. (2012). The case for generic skills and performance assessment in the United States and international settings. New York: Council for Advancement of Education.

Ethics and Assessment: When the Test is Life Itself

tbrewer2@illinois.edu — Mon, 17 Sep 2012 09:45:00 CDT

By: Edward L. Queen, Ph.D., J.D.
Director of the D. Abbott Turner Program in Ethics and Servant Leadership and Coordinator of Undergraduate Studies at Emory University's Center for Ethics

As the national tragedy that was Watergate progressively unfolded, it became known that John Dean, one of the men at the center of the scandal, had taken, while at Yale, an ethics course from William Sloane Coffin. As one might imagine the fact that one of President Richard Nixon’s scandal-ridden inner circle had been an ethics student of one the president’s most outspoken critics provided a field day for the press. When, as the story goes, a reporter asked the Reverend Coffin how John Dean had done in his class, Coffin responded, “Evidently, he failed.”

This story highlights the fundamental challenge of assessing students’ ethical reasoning. Ultimately the fundamental test or assessment of one’s ethical reasoning or practice is life, as it may well be for all of one’s education.

I write this to make clear what everyone knows but perhaps we fail to say often or convincingly enough. Our assessments or measurements are contingent upon what we define as success. Someone convinced that the goal of education is the formation of an individual into a particular way of being in the world undoubtedly would focus on notably different “measures” than would one who saw that goal as a mastery of a particular body of knowledge.

Assessing ethical formation is possible, though I think it always will be imperfect. My purpose with this essay is to identify the limitations of such assessment and to stimulate a conversation about how we identify what we can assess and, perhaps more importantly, what we cannot assess.

As a rule we cannot assess how students might actually respond in real-life situations. With the rare exceptions of certain experiments we cannot “stress test” our students and, even if we could, most formulations of such stressors would (and should) fail to meet IRB approval or give rise to much more complex analyses than perhaps they have received.

Even having students analyze the most detailed and textually rich ethics case cannot give them the feeling of being a junior employee, with two young children and a recently laid-off spouse who is placed in a situation where she or he feels as though she or he must comply with unethical behavior or risk being fired. Limitations such as this are serious and must be acknowledged.

That said any meaningful attempt to assess students’ ethical reasoning must be able to determine the extent to which students can accomplish the following:

Identify and articulate the ethical issues or challenges within a particular situation or set of facts;
Identify the values at risk in various courses of action;
Identify the value or values that most ought to be served in a particular situation and why those values take precedence;
Choose a course of action designed to serve those values;and
Describe how one might bring about that course of action.

While assessments focusing only on some subset of the above may provide us with important information, we must be clear on what they tell us. A student may be completely competent or even superlative in doing 1 and 2 above and still act totally unethically in that situation. She or he may be willing to say in practice, “Yes, receiving the answers on this test via text messages is wrong and it constitutes cheating but I am going to do it anyway because I want a perfect score.” It is not impossible to have ethically informed sociopaths and psychopaths.

That said, students’ abilities to identify ethical issues are much weaker than one might imagine. Not only are there students who, even when they are told that there are ethical issues in a case, cannot identify them. More fascinating is the large group who identify non-existent ethical issues. The latter results from a tendency to manufacture facts, the inability to think conditionally, as well as a tendency to fixate on a case’s most disturbing elements (its “yuck” factor), even when it is ethically irrelevant. What makes this tendency most disturbing is that it reflects notable weaknesses in students’ overall reasoning and analytical capacities. Here is an instance where simply assessing students’ ethical reasoning may teach us much about their reasoning capacity in general.

Even among those students who are capable of identifying ethical challenges and ordering their values, research suggests that their biggest challenge is determining what to do. Recent graduates entering the work force often are paralyzed when confronted with such challenges because they immediately assume that their only choices are to do nothing or “go nuclear.” This reflects significant weaknesses in our teaching. Hence my insistence on finding some way to assess what students would or could do in a particular situation to ensure that the appropriate values or goods are served.

Ethics inherently are embedded in actions; one is not honest if one lies all the time. If we fail to teach students how to craft ethical actions then, as time will show, it is we who have failed.