At the University of California, Merced, the Students Assessing Teaching and Learning (SATAL) program trains undergraduates to conduct observations that describe (rather than evaluate) what goes on in class. The purpose of SATAL is to provide confidential, formative results of assessment to faculty and other university constituents with results that address the pedagogical impact of instruction and of related co-curricular initiatives. As an assessment support program, SATAL is available to instructional faculty, academic programs and administrative units interested in collecting indirect evidence in support of student learning outcomes. For instance, instructors and staff members might request SATAL students to collect data on classroom activities, courses, and services to gain a better sense of the student experience in their courses. Results of SATAL assessment are entirely confidential and the service is offered at no cost to the user.
Based on a similar program at Brigham Young University, SATAL was launched in 2009 with five students initially. One or more of these students was assigned to work on 28 assessment projects in support of 15 faculty and 13 programs from across the university. For instance, faculty would ask students to conduct observations of teaching and provide summary results of what they had observed. Since then our relatively new program at a new university that was launched in 2005 has nearly tripled in size, with 14 students now, most of whom are juniors and seniors. Excluding classes that SATAL students are currently taking, they can be assigned to any undergraduate class, including those they have previously completed. For fall semester of AY2013-14 we have assisted 20 faculty and 5 academic programs, as well as some administrative units such as housing and student affairs.
The SATAL Program is based on the assumption that undergraduates are well positioned to provide supplemental classroom and co-curricular support. The peer-to-peer nature of the interaction between SATAL students and their peers enrolled in classes being assessed distinctively offers a non-threatening context for open, authentic reflection on learning. SATAL students are carefully selected through interview and faculty recommendation, and most of them will pursue a career in education. Students in the program participate in weekly training workshops. Functioning as a noncredit pedagogy course, the training schedule includes modules featuring the assessment tools for focus groups, ethnographic classroom observations, scripted interviews of individuals or groups, questionnaire surveys, and classroom videotaping. In delivering the training modules, the SATAL program adopts different formats, such as student presentations, peer review activities, reflective writing, debates, and hands-on activities.
SATAL assessment of teaching and learning concentrates foremost on class observations. Rather than using a prescribed protocol to record observations, SATAL students complete the equivalent of an ethnographic description of whatever occurs in a class. We considered using or adapting assessment rubrics from other universities (e.g., Teaching Skills Checklist from UT at Austin, the Teaching Dimensions Observation Protocol or, perhaps the Reformed Teaching Observation Protocol), but ultimately decided that the specific context of a relatively new university enrolling a majority-minority undergraduate population, and that routinely appoints a high percentage of new faculty each year would require a highly contextualized approach for observations of teaching. Those observations can be narrowed to whatever an instructor wants one or more SATAL students to note such as call patterns in response to a teacher's questions or opportunity for students to ask questions and the teacher's response versus redirection to peers for a response. With flipped classrooms, instructors could be interested in receiving feedback on student engagement that confirms their prior attention to posted lecture materials.
Through a series of workshops, students are trained to obtain information as objectively as possible, for instance, by documenting what goes on in the classroom in chronological order without judging if enough time had been allowed for specific tasks. SATAL students work in teams, usually with three students assigned to a large class (~60 to 150 students). Observers will sit in different places and adopt different roles. Two SATAL observers will focus on students' interaction (among themselves and with the instructor) and the third observer will focus on the instructor. SATAL undergraduates are well positioned to provide supplemental classroom support, and they are eager to meet with an instructor afterwards to go over results. During these meetings the instructor has the option to ask SATAL students about what they thought went well and what could use some change. For example, instructors might solicit students' perspective on effective use of class questioning, group interaction, board work, slides design, etc. The findings are completely confidential and for formative purposes only.
Class observations are usually paired with class interviews and/or videotaping and/ or focus groups. With regard to how much data we have collected, the SATAL program has performed 34 observations over the past three years, representing 10% of our assessment requests. Class interviews (94 or 27%), video tapings (110 or 32%), focus groups (72 or 20%), and surveys (38 or 11%) comprise the other 90% of the requests. Faculty members and other constituents have been very satisfied with the helpfulness of SATAL findings in helping them to adjust their courses at UC Merced by increasing student engagement and improving academic performance. These are the top three most frequent changes implemented as a result of the SATAL class assessment: support for the incorporation of mid-lecture (clicker) questions to check students’ attendance and understanding of lecture points, strategies for structuring group work to encourage the participation of quieter students during a flipped classroom teaching approach, and use of minute papers to assess what students take away from the classes. Pre- and post-observation reports provide evidence that these changes have positively impacted the learning environment with regard to student participation during lecture time. Specifically, students are more attentive to class activities, and they interact more often with each other on class activities rather than browsing social media sites. “By incorporating these tools into my lectures, students’ grades increased because they were more engaged with the material” -Natural Sciences faculty member and recipient of a Distinguished Undergraduate Teaching award.