Viewpoints discussion board | University of Illinois

National Coalition for Learning Outcomes Assesment

Entering content area for Viewpoints discussion board

blog posts

  • What to Consider When Selecting an Assessment Management System

    A few years ago, the primary reason for using a data management system arose from the need to manage large amounts of dynamic data more efficiently.  But in the past few years, there’s been a tectonic shift in public policy that catapulted organizing assessment and institutional effectiveness data to mission-critical status.  Policy shifts include the following: 

    • New laws on the Federal Register requiring State Authorization, reporting on certificate programs leading to Gainful Employment, a new Definition of Credit Hour policy for non-traditional pedagogical delivery methods, new laws on Misrepresentation, and the closing of “safe harbors” in student recruiting;
    • Changes in regulations by the U.S Department of Education to align with modifications to the Federal Definition of a Credit Hour and cross-validation of learning outcomes to assure comparability regardless of mode of pedagogical delivery;
    • Concomitant changes in regional and specialized accrediting requirements placing heavier burdens on institutional effectiveness and outcomes assessment professionals related to reporting Gainful Employment data, student learning outcomes data for all certificate programs, comparability measures for alternative methods of delivery, and post-baccalaureate student data such as employment tracking, debt-earnings ratios, and employer satisfaction data. 

    These changes have many implications.  Among the more important is that a viable data management system must now be capable of reporting data in a variety of new forms to meet the requirements of multiple internal and external agencies.  In addition to keeping abreast of changes in public policy, there are several other reasons for moving expeditiously to acquire or develop an effective assessment management system.  These include the following:

    • Organizing, archiving, and reporting of assessment information in large quantities from disparate academic enterprises.
    • Developing active management tools to substitute for static repositories.
    • Increasing demand for rapid and reliable data flow through vertical and horizontal entities within the university or college.
    • Breaking down silos that emerge when distinct technological infrastructures have evolved too rapidly or are too centralized to accommodate the needs of assessment practitioners.

    To make an informed judgment about whether the institution should consider purchasing an assessment management system, or build one locally, some critical questions and issues must be addressed.  Examples include the following (RiCharde, 2009):

    Cost

    The first cost issue is to determine what the system should accomplish.  System uses range from passive repositories for learning outcomes plans and data summaries to complex curriculum mapping systems such as Weave Online, or a combination accreditation management systems such as Xitracs, or a comprehensive planning and analysis system such as TracDat.

    Also consider the price elasticity of the available systems.  When purchasing a package, multi-year contracts and solid negotiating result in much lower prices. Internally hosted products like TracDat and Digital Measures can run into the tens of thousands of dollars for all available modules while other externally hosted products like Weave Online or Xitracs (which now has an assessment module) can be leased annually for a fraction of that cost or for a few thousand more you can purchase and internally host the software product, e.g. Xitracs.

    Locally Hosted vs. Remotely Hosted

    Do you want to outsource the system so it resides in an external company, such as Weave Online?  Or does it make more sense to purchase a complete system to be run by the local IT department, such as can be done with TracDat?  Your answers will be influenced in part by institutional IT policies.  This means the IT organization needs to be involved early and often before you select or develop a system.  Some institutions favor outsourcing, while others prefer local control.  The major benefit to building your own system is cost.  At the University of South Florida, we have been working on a system for two years. Version 2.0 will roll out during the summer 2012, and 3.0 will be ready by early spring 2013, which requires that a programmer be solely designated to this project.  The salary and benefits are not trivial.  At the same time, the system will be precisely what we require and we will soon begin reaping the benefits of not paying annual fees.

    Breadth and Integration

    Another key set of issues is the comprehensiveness of the system and whether it must be integrated with the institution’s Student Information System (SIS) (e.g., Banner, Datatel, PeopleSoft) or Learning Management System (LMS) such as Blackboard.  Silos spring up when data are “owned” by a specific internal agency. Most assessment practitioners are amazed at how much assessment data are floating around their institutions unclaimed and unused with multi-integrated access to those systems. Some systems such as those produced by Digital Measures and Explorance truly integrate with the SIS while others require Web Services or cumbersome SQL extractions.

    Lessons Learned

    1. Before purchasing a system “spec out” and/or “wire frame” your requirements. This is best accomplished in conjunction with an IT representative. If you plan to purchase from a vendor, formulate a thorough specifications profile. If you plan to develop your own, have IT assist you in developing “wire frames” of what pages will look like. This can be as simple as paper-and-pencil sketches.
    2. Forget integrating with an SIS or LMS unless you wish to use the system to directly gather information from a “hot” (transactional) server or “warm” (operational data) server. These data would include such sources as up-to-the-minute load study information, LMS delivered surveys, and student evaluation of teaching.
    3. Because of shifting public policy, it is essential to purchase or develop the most flexible system available.  By flexibility, I mean, will it be easy to add new categories and entities to the hierarchical entity tree?  Does the system have a built-in meta-assessment or audit tool, and if so, how easy is it to alter the requirements?  For example, the reporting requirements for distance learning are changing, so how easy will it be to set and reset limits on the number and value of learning outcomes?

    Certainly other key issues will arise.  Even so, by addressing those discussed above, assessment practitioners will be much better informed when deciding whether a commercial solution is appropriate or whether constructing one in-house is the best approach.

    References

    RiCharde, R. S. (2009).  Data management and data management tools.  In B. Peat and L.J. Moriarty (eds.). Assessing Criminal Justice/Criminology Education: A resource handbook for educators and administrators (pp. 51-68). Chapel Hill: University of North Carolina Press.