Methodological issues in diagnosing competencies
Collecting and analysing information on cognitive competencies as part of a large-scale assessment means using methods that must be continually developed and adapted to fit the requirements of the assessment in question. This applies to every stage of collecting and analysing data on student performance. The IQB is working on a series of research questions to help optimise these methods.
The process of compiling test instruments for use in large-scale assessments has to begin with developing matrix sampling designs in which each student only completes a selection of the total test items available. This approach makes it possible to include a large number of items and thus achieve a broad coverage of the test domains without putting individual students under excessive pressure. However, a number of requirements must be taken into account when developing matrix sampling designs. For instance, the design has to ensure that student performance in each competency can be mapped and remain stable on the same scale both within a study and across several studies. At the IQB, we are researching how best to construct these test designs so that they fulfil the requirements as economically as possible.
With regard to the stage of scaling student responses, we address a variety of research questions that concern the dimensionality of the constructs measured and the inclusion of background variables in the calibration process. The IQB is currently doing a series of analyses on the accuracy of performance estimates at a variety of aggregate levels, such as at the level of the states in the Germany. One of the things we are looking into is whether the accuracy of the estimates is affected by differences in the shares of missing values in the background variables used for calibration.
If reports on the results of competency-oriented performance measurements are to be accessible to a variety of actors involved in the education sector, they must present the information clearly and in an easy-to-follow format. Proficiency level models generated using standard-setting processes are an important way of achieving this. They make it possible to transform continuously distributed test results into separate sections of a scale that can be isolated and described according to their content. A variety of procedural methods are used to allocate students to ordinal proficiency levels on the basis of their raw test data. These processes all share a common aim: to assess individual performance not only in relation to the relevant student population but also in terms of solving specific tasks. With its research on the validity of different standard-setting processes, the IQB hopes to help improve the theoretical and empirical foundations on which proficiency level models are developed.