Meta-assessment, or the evaluation of assessment practice, can help institutions explain the quality of assessment at every level of a university from program to department to college to the university. Over the past 10 years we have provided feedback to hundreds of university faculty and staff on their assessment reports. We also have trained hundreds of others to do the same. It takes considerable resources and is challenging. Why then should you or anyone else spend time on this seemingly esoteric endeavor? Meta-assessment helps a university get a grip on the quality of its assessment practices and whether student learning is improving. Explicit or implicit, a primary role of an assessment office is to improve assessment practice. However, many assessment offices cannot provide data about the aggregate quality of assessment on campus.
Meta-assessment consists of two major parts. One is easy and the other is challenging. The easy part is capturing the range and saturation of assessment activities underway, which is typically calculated by the number of assessment reports submitted divided by the number of eligible program/units. For example, at James Madison University (JMU) we have 120 academic degree and certificate programs. All 120 submitted a report in the summer of 2012, so the participation for such programs was 120/120 or 100%. This part of meta-assessment is relatively simple to conduct and interpret. Nevertheless, by itself, participation is severely lacking because it does not speak to the quality of assessment.
The other part is evaluating the quality of assessment efforts, which is more important but less straightforward. It requires an institution to develop a rubric or a checklist that specifies characteristics of good assessment practice. These tools not only communicate expectations of good assessment practice but also are used to evaluate assessment reports. These instruments differ by institution, but the most common criteria for quality practice include articulation of objectives, methodology, results, and use of results for improvement.
By evaluating different parts of assessment separately, it allows an institution to identify which aspects of assessment programs are well conceptualized and which aspects could use improvement. At JMU programs were strong on most aspects of assessment except for using results for improvement. In response, we are developing strategies to help programs accomplish this end. One tactic is for the assessment office and the faculty development office to collaborate, helping faculty align assessment, curriculum, and pedagogy at the program level.
Not only does the practice of meta-assessment help the institution manage its assessment practice, but it is very useful when speaking to external stakeholders about accountability. As opposed to many universities that scramble to assemble documentation for accreditation purposes, institutions engaging in meta-assessment will already have a handle on how many assessment reports have been submitted and the quality of assessment. Case in point, JMU received no recommendations for assessment from our recent accreditation visit, in large part because of our convincing meta-assessment data.
Perhaps less intuitive than the other reasons, meta-assessment can improve assessment competence on campus. Among the hundreds of faculty we have trained to evaluate assessment reports, the vast majority comment on their increased assessment acumen. Many then apply these news skills in their programs. For example, many faculty become more adept at integrating their program’s student learning outcomes to every part of the assessment cycle. These reports are in contrast to others where misalignment among student learning outcomes, measures, and data collection design is far too common.
For those interested in conducting meta-assessment, we close with a few tips.
- At the outset decide why your university should conduct meta-assessment. If it is to help with accreditation then make sure that your evaluation criteria, which are usually articulated through a rubric, are synced with the accreditor’s standards.
- If you would like the process to help faculty and staff learn about assessment then make sure that this group participates in evaluating assessment reports. This participation should include training from an assessment expert.
- Last, help administration make decisions based on meta-assessment results. For example, if the results indicate that a few programs are struggling with assessment then the administrators could allocate money to the respective program coordinators for training.
For those who would like to read further about meta-assessment we suggest the following resources:
http://www.jmu.edu/assessment/JMUAssess/APT_HELP_Package.pdf : A compilation of documents used to help faculty write better assessment reports including example reports and a meta-assessment rubric.
Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2012). Demonstrating student success: A practical guide to outcomes-based assessment of learning and development in student affairs. Stylus Publishing, LLC.
Fulcher, K.H. & Bozhidar, B. M. (2012, November-December). Do we practice what we preach? The accountability of an assessment office. Assessment Update. 24(6). 5-7, 14.
Fulcher, K. H., Swain, M. S., & Orem C.D. (2012, January-February).Expectations for assessment reports: A descriptive analysis. Assessment Update, 24(1), 1-2, 14-16.
Ory, J. C. (1992). Meta-assessment: Evaluating assessment activities. Research in Higher Education, 33(4), 467-481.
Keston H. Fulcher, Center for Assessment and Research Studies, James Madison University; Megan Rodgers Good, Department of Graduate Psychology, James Madison University.
Correspondence concerning this blog should be addressed to Keston Fulcher, Center for Assessment and Research Studies, MSC 6806/ 821 S. Main St, Harrisonburg, VA 22807.