blog postsDiscussing the Data, Making Meaning of the ResultsAug 9, 2012 9:30 am1277 views By: Anne Goodsell LovePh.D., Associate Provost for AssessmentWagner College In examining the outcomes of teaching and learning, when is enough data analysis enough? How are the results of these inquiries most effectively disseminated to faculty and staff? What forms of dissemination and communication are most likely to stimulate productive discussions about assessment results so as to influence decision making and invite additional analysis? Where and how do faculty and staff reflect on the whole cycle of assessment, remembering their goals and making meaning of the results in that context? Conversations about assessment results can happen in faculty meetings and around meal tables, of course, but those are not likely venues for thoughtful reflection. Reflection occurs best when other activity is stilled. The calm surface of a lake or stream reflects the shoreline, but wind, rain, and other kinds of turbulence distort and interrupt the reflection. The same can be said for reflection as a cognitive process in the cycle of assessment. We need time to think by ourselves and to think out loud with others, to sort through information and ideas, and to come to conclusions that again get tested. Talking about the results of assessment as a way of making meaning of the results is part of the data analysis process and, therefore, it needs to be encouraged. In the assessment cycle, this part is the “So what?” that encourages people to see how the data can make a difference in their work. Disseminating and discussing the data, reflecting on data and making meaning of the results—these are processes that my colleagues and I at Wagner College have addressed in our campuswide Bringing Theory to Practice (BTtoP) research initiative exploring the relationships between experiential learning and student well-being. Together, these processes are the means by which data can be digested and transformed into evidence for use in decision making. Disseminating and Discussing the Data Results of the BTtoP research were displayed at an open reception in the employee dining room and at various faculty meetings at Wagner College. Wagner faculty, administrators, and students who viewed those displays were invited to speculate about the meaning of the results, to propose explanations for the relationships reported, and to ask further questions. Members of the BTtoP research team attended these sessions to interact with the attendees and to hear the ideas and questions posed. In addition to these face-to-face sessions (some of which were sparsely attended), with the goal of generating discussion about the data, the results were posted on the college’s assessment website and an email was sent to faculty with a summary of the results and a link to that site. Reflecting on the Data and Making Meaning of the Results For those who participated, the process of discussing and reflecting on educational practices—in this case, the practices involved in establishing experiential learning activities and in connecting in-class and out-of-class learning—was an opportunity to expose assumptions, beliefs, and biases about teaching and learning in these ways. Thinking out loud about what the research results might mean or what might be missing from the results was an opportunity to step back from practice and to reflect on the underlying theories shaping our decisions and actions. For example, as the research project progressed, we went beyond the labels we typically associated with experiential learning (service learning, civic engagement, field trips, etc.) and asked how and why faculty members had changed their experiential learning practices. We also noted that although students were asked to do a variety of informal and formal reflective writing assignments on their experiential learning, Wagner’s faculty and administrators had few structures for this kind of collaborative meaning-making reflection. Conversations about pedagogical practice and the research results that could inform that practice happened in faculty meetings and at annual retreats, but the agendas at those events included many other topics, limiting time for reflection. Conversations with colleagues at SUNY Cortland and Georgetown University who were also conducting BTtoP research revealed similar themes. Research results were disseminated in a variety of ways at these institutions, but actual engagement with the results was limited. Proposals to put the results online—even with access to the results restricted to the campus community—were met with hesitancy, expressed in reservations such as “the findings are only preliminary,” “how publicly should they be made available?” “will anyone read them?” and “will the results be accepted in the spirit in which they are offered—as an invitation to think, reflect, hypothesize, and wonder?” Some of these reservations were reasonable and responsible, but others would have inhibited the dissemination and, therefore, the potential use of the results. When the presentation of research results is added to the agenda of regular meetings there may not be enough time for the discussion to get beneath the surface. Discussions of results and about changes that results might indicate can easily get diverted by questions related to existing programs, staff, resources, or policies. Often, there is a reluctance in such venues even to entertain a conversation about change. What have we learned about facilitating thoughtful discussion and reflection on research results? Presenting research results at regular meetings and displaying the results in high-traffic locations are two ways to work within the existing patterns of events and people at an institution. Retreats offer an outstanding opportunity for focused reflection as long as the retreat’s agenda maintains a similar focus. People are more willing to engage in a discussion of research results when they have been involved in gathering the data for the research. Program-level assessments tend to work better in this regard than institution-level assessments, especially at larger institutions. Ongoing, high-level commitment from faculty and administrators remains key to the success of the assessment process, as sustained momentum (including a budget) over a substantial number of years is necessary to embed within an institution a culture of research and reflection on teaching and learning. Making sure that research about learning outcomes is done well and is also used well are two critical purposes of any assessment cycle. The prospects for achieving these purposes can be enhanced by bringing the process of data analysis into discussions and by faculty, staff, and students making meaning of the data by reflecting together on the results of the analysis.