"Heuristic Cognitive Diagnosis When the Q-Matrix is Unknown"
Abstract:
Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning test examinees to proficiency classes are based on maximum-likelihood estimation (MLE) procedures that frequently encounter
difficulties in practical applications: software for these procedures tends to be proprietary and hence unavailable or expensive, convergence rate and computer time depend on having good starting values, and optimal solutions are not guaranteed even when considerable amounts of computer time have been consumed. In response to these difficulties, nonparametric, model-free classification techniques (cluster analysis) have been proposed as heuristic alternatives to MLE procedures. These classification techniques first aggregate each examinee's test-item scores into a profile of attribute sum-scores, which then serve as the basis for assigning examinees to proficiency classes. Like the MLE procedures, the classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known, so the (fallible) judgment of educational experts is used to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This study demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score
profiles (a) does not require knowledge of the Q-matrix and (b) results in a more accurate classification of examinees.
