DigiDIn-Kfz - Validation study 1
Documentation and R code
Stefan Hartmann
2024-01-02
1 About this document
1.1 General information
- Project name: DigiDIn-Kfz
- Funding agent: Federal Ministry of Education and Research (Germany)
- Grant No.: 21AP004A
- Funding period: 2020-2023
- Subproject: B - Video-based assessment
- Study: Validation study 1
- Date of data collection: July 2021
- Author: Stefan Hartmann, Pädagogische Hochschule Ludwigsburg
1.2 License
This documentation and the related CSV data file are available under the terms of the Creative Commons Attribution-ShareAlike 4.0 license. They may be copied, shared, transformed, and build upon under the conditions described here: https://creativecommons.org/licenses/by-sa/4.0/
2 About this study
2.1 Synopsis
We investigated the validity of the test score interpretations from a computer-based assessment aiming at the repair-related competences of automotive technician apprentices. Using a known-groups validation design (Cronbach & Meehl, 1955), we compared the test scores of automotive technician apprentices with the scores of electronics technician apprentices. As the test is aimed at measuring car-specific competences, we hypothesise automotive technicians substantially outperform electronics technicians.
The study is published in German (Hartmann, Güzel, & Gschwendtner, 2023a). The main findings are also summed up in the discussion chapter of an English article (Hartmann, Güzel, & Gschwendtner, 2023b).
2.2 Instrument design
The instrument is a computerised fixed test (CFT) which uses scripted video vignettes. The test consists of five independent test booklets, each of which refers to a different car system: brakes, exhaust, lighting, steering, and timing belt. In each test booklet, a series of short video clips of a technician performing repairs on a car are presented on the computer screen. Each video clip is followed by (at least) one selected-response item which requires test takers to judge two short statements about the repair actions shown in the video (fig. 1). The test is in German.
We use a highly standardised “all possible options” (APO) item design in which two statements marked “Option A” and “Option B” about each video clip are presented. Test takers must then mark one out of four possible answers:
Only option A is correct.
Only option B is correct.
Both options, A and B are correct.
Neither option A nor option B is correct.
Correct responses are credited with a score of 1 point (full credit) whereas incorrect responses are scored 0 points (no credit). The sum of correct responses reflects a test taker’s ability to correctly judge the actions shown in the videos, thus representing their procedural knowledge about repairs on the car system the test booklet refers to.
2.3 Sample
The sample of the known-groups study consists of 188 apprentices from Southwest Germany at the third year of formal training (128 automotive technician apprentices, 60 electrical/electronics engineer apprentices). We used an unbalanced incomplete block design, meaning that each participant was given a selection of one or more test booklets, but not all five.
3 Analyses in R
Note. All code was tested using R version 4.3.2.
3.1 Preparation
3.1.1 Increase the maximum length of R console output
(this is optional)
options(max.print = 5000)3.1.2 Set working directory
Linux:
setwd("/home/username/path/to/directory")macOS:
setwd("/Users/username/path/to/directory")`Windows:
setwd("C:/path/to/directory")` Note: Make sure to use forward slashes (/), not backward slashes (\) which you will get if you copy file paths in Windows.
3.1.3 Install required R packages
This is only necessary once, and only if these packages are not yet installed on your computer.
Package ‘car’ (Fox at al., 2023)
install.packages("car")
Package ‘psych’ (Revelle, 2023)
install.packages("psych")
Package ‘CTT’ (Willse, 2018)
install.packages("CTT")
Package ‘yarrr’ (Phillips, 2017)
install.packages("yarrr")
Package ‘lsr’ (Navarro, 2021)
install.packages("lsr")3.1.4 Import CSV data
df <- read.csv(file = "232012_DigiDIn-Kfz_B_Val-1_data.csv" ,
header = TRUE ,
sep = "," ,
dec = "." ,
fileEncoding = "UTF8")3.1.5 Display a list of variables
The first variable, ‘case_number’, is the running number of participants in order of participation. The variables ‘job_designation’, ‘specialisation’, ‘type_of_company’, and ‘school_degree’ contain demographic information. The remaining variables are test items. Their variable names indicate the test booklet:
lights = booklet 1 (electric lighting)
brakes = booklet 2 (hydraulic brakes)
exhaust = booklet 3 (exhaust and catalytic converter)
timing belt = booklet 4 (engine timing belt)
steering = booklet 5 (front steering and suspension)
Item numbers start at 05 in each booklet because the first four items contain an introduction to the instrument and an item example which is not used for scoring.
Note. The following items are intentionally missing because they were added to the instrument at a later time:
lights_09, lights_14, lights_24
brakes_06, brakes_17
timing_belt_05, timing_belt_15, timing_belt_24
steering_11, steering_13, steering_14, steering_19, steering_22, steering_23, steering_24
Two items with the suffix ’_old’ (‘brakes_05_old’ and ‘timing_belt_14_old’) indicate that the versions used in this study are now outdated. In later studies, revised versions of these items were used (‘brakes_05_new’ and ‘timing_belt_14_new’).
colnames(df)## [1] "case_number" "job_designation" "specialisation"
## [4] "type_of_company" "school_degree" "lights_05"
## [7] "lights_06" "lights_07" "lights_08"
## [10] "lights_10" "lights_11" "lights_12"
## [13] "lights_13" "lights_15" "lights_16"
## [16] "lights_17" "lights_18" "lights_19"
## [19] "lights_20" "lights_21" "lights_22"
## [22] "lights_23" "brakes_05_old" "brakes_07"
## [25] "brakes_08" "brakes_09" "brakes_10"
## [28] "brakes_11" "brakes_12" "brakes_13"
## [31] "brakes_14" "brakes_15" "brakes_16"
## [34] "brakes_18" "brakes_19" "brakes_20"
## [37] "brakes_21" "brakes_22" "brakes_23"
## [40] "brakes_24" "exhaust_05" "exhaust_06"
## [43] "exhaust_07" "exhaust_08" "exhaust_09"
## [46] "exhaust_10" "exhaust_11" "exhaust_12"
## [49] "exhaust_13" "exhaust_14" "exhaust_15"
## [52] "exhaust_16" "exhaust_17" "exhaust_18"
## [55] "exhaust_19" "timing_belt_06" "timing_belt_07"
## [58] "timing_belt_08" "timing_belt_09" "timing_belt_10"
## [61] "timing_belt_11" "timing_belt_12" "timing_belt_13"
## [64] "timing_belt_14_old" "timing_belt_16" "timing_belt_17"
## [67] "timing_belt_18" "timing_belt_19" "timing_belt_20"
## [70] "timing_belt_21" "timing_belt_22" "timing_belt_23"
## [73] "steering_05" "steering_06" "steering_07"
## [76] "steering_08" "steering_09" "steering_10"
## [79] "steering_12" "steering_15" "steering_16"
## [82] "steering_17" "steering_18" "steering_20"
## [85] "steering_21"3.2 Frequency analysis
3.2.1 Demographic data
Job designation
There are two job designations, referring to the job program in which the apprentice is trained: automotive technician and electrical/electronics engineer. This variable serves as a grouping variable for the known-groups validation.
table(df$job_designation , exclude = FALSE) ##
## automotive technician electrical/electronics engineer
## 128 60Specialisation
There are three specialisations:
building services (specialisation of the electrical/electronics engineer apprenticeship)
equipment and systems services (specialisation of the electrical/electronics engineer apprenticeship)
light vehicle maintenance and repair (specialisation of the automotive technician apprenticeship)
table(df$specialisation , exclude = FALSE)##
## building services equipment and systems services
## 29 24
## light vehicle maintenance and repair missing by intention
## 110 25Type of company
In the German dual VET system, apprenticeship programs combine classroom-based learning in a vocational school with workplace learning in a company. This variable indicates the type of training company, differentiating dealerships (workshops affiliated with a manufacturer such as Volkswagen or Bosch), independent workshops (workshops that service and maintain products of many different brands), industrial enterprises (companies where products are built at an industrial scale, such as factory sites), and other (companies that don’t fit the other categories, such as municipal enterprises or workshops specialised in classic cars).
table(df$type_of_company , exclude = FALSE)##
## dealership independent workshop industrial enterprise
## 67 39 23
## missing by intention other
## 33 26School degree
Highest school degree of the apprentice, according to the German school system:
lower secondary (Hauptschule) - degree of a school offering ISCED Level 2 education, usually obtained at the end of the 9th or 10th grade
higher secondary (Mittlere Reife) - degree of a school offering ISCED Level 2 education but ranked higher than Hauptschule, usually obtained at the end of the 10th grade
A-levels/high school (Abitur) - high school degree according to ISCED Level 3, usually obtained at the end of the 12th or 13th grade
table(df$school_degree , exclude = FALSE)##
## A-levels/high school (Abitur) higher secondary (Mittlere Reife)
## 23 105
## lower secondary (Hauptschule) missing by intention
## 28 323.2.2 Response data
Response values for each item are 1, 2, 3, and 4, referring to the four response options of the single-best answer items (see chapter ‘Overview’ and fig. 1).
3.2.3 Missing data
A value of ‘9’ indicates an item that was deliberately skipped (missing by intention). Items not presented to a participant by design appear as ‘NA’ (missing by design).
apply(df[c(6:85)] , 2 , table , exclude = FALSE) ## $lights_05
##
## 1 2 3 4 <NA>
## 12 40 63 45 28
##
## $lights_06
##
## 1 2 3 4 9 <NA>
## 14 117 16 12 1 28
##
## $lights_07
##
## 1 2 3 4 <NA>
## 26 76 49 9 28
##
## $lights_08
##
## 1 2 3 4 9 <NA>
## 100 7 49 3 1 28
##
## $lights_10
##
## 1 2 3 4 9 <NA>
## 32 96 9 22 1 28
##
## $lights_11
##
## 1 2 3 4 9 <NA>
## 3 27 113 16 1 28
##
## $lights_12
##
## 1 2 3 4 9 <NA>
## 39 22 88 10 1 28
##
## $lights_13
##
## 1 2 3 4 9 <NA>
## 50 11 47 48 4 28
##
## $lights_15
##
## 1 2 3 4 9 <NA>
## 90 24 35 10 1 28
##
## $lights_16
##
## 1 2 3 4 9 <NA>
## 14 83 52 10 1 28
##
## $lights_17
##
## 1 2 3 4 <NA>
## 52 26 75 7 28
##
## $lights_18
##
## 1 2 3 4 <NA>
## 43 49 36 32 28
##
## $lights_19
##
## 1 2 3 4 <NA>
## 57 19 67 17 28
##
## $lights_20
##
## 1 2 3 4 <NA>
## 62 27 62 9 28
##
## $lights_21
##
## 1 2 3 4 <NA>
## 25 27 102 6 28
##
## $lights_22
##
## 1 2 3 4 <NA>
## 22 11 126 1 28
##
## $lights_23
##
## 1 2 3 4 <NA>
## 7 12 140 1 28
##
## $brakes_05_old
##
## 1 2 3 4 9 <NA>
## 31 29 23 24 1 80
##
## $brakes_07
##
## 1 2 3 4 <NA>
## 9 45 50 4 80
##
## $brakes_08
##
## 1 2 3 4 <NA>
## 35 18 37 18 80
##
## $brakes_09
##
## 1 2 3 4 <NA>
## 66 5 35 2 80
##
## $brakes_10
##
## 1 2 3 4 9 <NA>
## 32 28 38 7 3 80
##
## $brakes_11
##
## 1 2 3 4 9 <NA>
## 29 8 23 47 1 80
##
## $brakes_12
##
## 1 2 3 4 <NA>
## 30 21 49 8 80
##
## $brakes_13
##
## 1 2 3 4 9 <NA>
## 13 21 66 5 3 80
##
## $brakes_14
##
## 1 2 3 4 9 <NA>
## 18 29 55 5 1 80
##
## $brakes_15
##
## 1 2 3 4 9 <NA>
## 43 12 48 4 1 80
##
## $brakes_16
##
## 1 2 3 4 9 <NA>
## 47 11 47 2 1 80
##
## $brakes_18
##
## 1 2 3 4 9 <NA>
## 45 7 52 3 1 80
##
## $brakes_19
##
## 1 2 3 4 9 <NA>
## 33 29 18 26 2 80
##
## $brakes_20
##
## 1 2 3 4 9 <NA>
## 11 31 12 51 3 80
##
## $brakes_21
##
## 1 2 3 4 9 <NA>
## 26 39 39 2 2 80
##
## $brakes_22
##
## 1 2 3 4 9 <NA>
## 10 27 65 3 3 80
##
## $brakes_23
##
## 1 2 3 9 <NA>
## 17 20 70 1 80
##
## $brakes_24
##
## 1 2 3 4 9 <NA>
## 64 17 25 1 1 80
##
## $exhaust_05
##
## 1 2 3 4 <NA>
## 5 25 42 1 115
##
## $exhaust_06
##
## 1 2 3 4 9 <NA>
## 17 37 16 2 1 115
##
## $exhaust_07
##
## 1 2 3 4 <NA>
## 11 26 33 3 115
##
## $exhaust_08
##
## 1 2 3 4 <NA>
## 24 3 44 2 115
##
## $exhaust_09
##
## 1 2 3 4 9 <NA>
## 20 29 19 4 1 115
##
## $exhaust_10
##
## 1 2 3 4 9 <NA>
## 29 17 6 20 1 115
##
## $exhaust_11
##
## 1 2 3 4 <NA>
## 18 22 23 10 115
##
## $exhaust_12
##
## 1 2 3 4 <NA>
## 5 21 39 8 115
##
## $exhaust_13
##
## 1 2 3 4 <NA>
## 4 55 10 4 115
##
## $exhaust_14
##
## 1 2 3 4 9 <NA>
## 12 31 3 24 3 115
##
## $exhaust_15
##
## 1 2 3 4 9 <NA>
## 9 40 8 14 2 115
##
## $exhaust_16
##
## 1 2 3 4 <NA>
## 8 35 26 4 115
##
## $exhaust_17
##
## 1 2 3 4 9 <NA>
## 19 10 36 7 1 115
##
## $exhaust_18
##
## 1 2 3 4 <NA>
## 40 5 26 2 115
##
## $exhaust_19
##
## 1 2 3 <NA>
## 20 8 45 115
##
## $timing_belt_06
##
## 1 2 3 4 <NA>
## 12 29 46 5 96
##
## $timing_belt_07
##
## 1 2 3 4 <NA>
## 19 27 44 2 96
##
## $timing_belt_08
##
## 1 2 3 4 <NA>
## 16 38 29 9 96
##
## $timing_belt_09
##
## 1 2 3 4 <NA>
## 57 9 19 7 96
##
## $timing_belt_10
##
## 1 2 3 4 <NA>
## 56 9 25 2 96
##
## $timing_belt_11
##
## 1 2 3 4 <NA>
## 23 14 52 3 96
##
## $timing_belt_12
##
## 1 2 3 4 <NA>
## 25 23 31 13 96
##
## $timing_belt_13
##
## 1 2 3 4 <NA>
## 7 36 46 3 96
##
## $timing_belt_14_old
##
## 1 2 3 4 9 <NA>
## 15 60 6 10 1 96
##
## $timing_belt_16
##
## 1 2 3 4 <NA>
## 26 14 6 46 96
##
## $timing_belt_17
##
## 1 2 3 <NA>
## 6 22 64 96
##
## $timing_belt_18
##
## 1 2 3 4 9 <NA>
## 15 37 11 28 1 96
##
## $timing_belt_19
##
## 1 2 3 4 9 <NA>
## 28 18 13 31 2 96
##
## $timing_belt_20
##
## 1 2 3 4 9 <NA>
## 37 8 42 4 1 96
##
## $timing_belt_21
##
## 1 2 3 4 <NA>
## 7 26 57 2 96
##
## $timing_belt_22
##
## 1 2 3 4 <NA>
## 15 39 37 1 96
##
## $timing_belt_23
##
## 1 2 3 9 <NA>
## 23 7 60 2 96
##
## $steering_05
##
## 1 2 3 4 <NA>
## 13 41 25 12 97
##
## $steering_06
##
## 1 2 3 4 9 <NA>
## 54 9 22 5 1 97
##
## $steering_07
##
## 1 2 3 4 <NA>
## 32 42 5 12 97
##
## $steering_08
##
## 1 2 3 4 <NA>
## 45 7 30 9 97
##
## $steering_09
##
## 1 2 3 4 <NA>
## 57 13 16 5 97
##
## $steering_10
##
## 1 2 3 4 9 <NA>
## 17 33 37 3 1 97
##
## $steering_12
##
## 1 2 3 4 9 <NA>
## 32 16 18 23 2 97
##
## $steering_15
##
## 1 2 3 9 <NA>
## 32 5 52 2 97
##
## $steering_16
##
## 1 2 3 4 9 <NA>
## 53 14 14 8 2 97
##
## $steering_17
##
## 1 2 3 4 9 <NA>
## 15 39 13 22 2 97
##
## $steering_18
##
## 1 2 3 4 9 <NA>
## 54 2 33 1 1 97
##
## $steering_20
##
## 1 2 3 9 <NA>
## 4 65 21 1 97
##
## $steering_21
##
## 1 2 3 4 9 <NA>
## 24 23 28 15 1 973.3 Items and scales
To judge how well the test items separate between participants with different skill levels, we calculate classical item separation parameters (item-rest correlations) for the items of the five test booklets.
It is expected that for each item, the correct response option (1, 2, 3, or 4) correlates higher with the rest of the booklet than the three incorrect response options (i.e., correct responses to an item are statistically related to the sum of correct responses to all other items of the same test booklet).
As each test booklet addresses a wide range of skills and competences, item separation is expected to be lower than for most psychological tests such as intelligence tests or personality tests.
3.3.1 Preparation: Extract items for each test booklet
We can use grep to extract the item designations for
each test booklet.
Booklet 1 - Lighting system
items_lights <- colnames(df)[grep("lights_" , colnames(df))]
items_lights## [1] "lights_05" "lights_06" "lights_07" "lights_08" "lights_10" "lights_11"
## [7] "lights_12" "lights_13" "lights_15" "lights_16" "lights_17" "lights_18"
## [13] "lights_19" "lights_20" "lights_21" "lights_22" "lights_23"Booklet 2 - Hydraulic brakes
items_brakes <- colnames(df)[grep("brakes_" , colnames(df))]
items_brakes## [1] "brakes_05_old" "brakes_07" "brakes_08" "brakes_09"
## [5] "brakes_10" "brakes_11" "brakes_12" "brakes_13"
## [9] "brakes_14" "brakes_15" "brakes_16" "brakes_18"
## [13] "brakes_19" "brakes_20" "brakes_21" "brakes_22"
## [17] "brakes_23" "brakes_24"Booklet 3 - Exhaust system
items_exhaust <- colnames(df)[grep("exhaust_" , colnames(df))]
items_exhaust## [1] "exhaust_05" "exhaust_06" "exhaust_07" "exhaust_08" "exhaust_09"
## [6] "exhaust_10" "exhaust_11" "exhaust_12" "exhaust_13" "exhaust_14"
## [11] "exhaust_15" "exhaust_16" "exhaust_17" "exhaust_18" "exhaust_19"Booklet 4 - Timing belt
items_timing_belt <- colnames(df)[grep("timing_belt_" , colnames(df))]
items_timing_belt## [1] "timing_belt_06" "timing_belt_07" "timing_belt_08"
## [4] "timing_belt_09" "timing_belt_10" "timing_belt_11"
## [7] "timing_belt_12" "timing_belt_13" "timing_belt_14_old"
## [10] "timing_belt_16" "timing_belt_17" "timing_belt_18"
## [13] "timing_belt_19" "timing_belt_20" "timing_belt_21"
## [16] "timing_belt_22" "timing_belt_23"Booklet 5 - Steering and front axle
items_steering <- colnames(df)[grep("steering_" , colnames(df))]
items_steering## [1] "steering_05" "steering_06" "steering_07" "steering_08" "steering_09"
## [6] "steering_10" "steering_12" "steering_15" "steering_16" "steering_17"
## [11] "steering_18" "steering_20" "steering_21"3.3.2 Add keys
Add information which of the four response options (1, 2, 3, or 4) is correct for each item.
Booklet 1 - Lighting system
keys_lights <- c("4" , "2" , "1" , "1" , "4" , "3" , "3" , "1" , "1" , "4" , "1" , "1" , "4" , "1" , "3" , "3" , "3")Booklet 2 - Hydraulic brakes
keys_brakes <- c("2" , "2" , "4" , "1" , "3" , "4" , "3" , "1" , "3" , "1" , "1" , "1" , "4" , "2" , "1" , "3" , "2" , "1" )Booklet 3 - Exhaust system
keys_exhaust <- c("2" , "2" , "3" , "1" , "1" , "1" , "2" , "3" , "3" , "4" , "4" , "3" , "3" , "1" , "1")Booklet 4 - Timing belt
keys_timing_belt <- c("3" , "2" , "2" , "1" , "1" , "1" , "2" , "3" , "2" , "4" , "3" , "4" , "1" , "1" , "2" , "2" , "3")Booklet 5 - Steering and front axle
keys_steering <- c("2" , "4" , "4" , "1" , "1" , "2" , "4" , "1" , "1" , "2" , "1" , "2" , "4")3.3.3 Distractor analysis
Note. An asterisk indicates the correct response option (1, 2, 3, or
4). A key of “9” indicates that no option was marked (item was
intentionally skipped). Item discrimination parameters (item-rest
correlations) are displayed in the column labelled
discrim.
Booklet 1 - Lighting system
CTT::distractorAnalysis(na.omit(df[, items_lights]) ,
keys_lights ,
nGroups = 3 ,
digits = 2)## $lights_05
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.07 -0.12 -0.01 0.07 0.12 0.06
## 2 2 40 0.25 -0.49 -0.21 0.36 0.16 0.15
## 3 3 63 0.39 -0.25 -0.12 0.40 0.56 0.28
## 4 * 4 45 0.28 0.15 0.34 0.17 0.16 0.51
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 14 0.09 -0.34 -0.13 0.15 0.06 0.02
## 2 * 2 117 0.73 0.23 0.37 0.53 0.91 0.91
## 3 3 16 0.10 -0.35 -0.14 0.17 0.03 0.04
## 4 4 12 0.07 -0.31 -0.10 0.13 0.00 0.04
## 9 9 1 0.01 -0.09 -0.01 0.01 0.00 0.00
##
## $lights_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 26 0.16 -0.08 0.09 0.12 0.19 0.21
## 2 2 76 0.48 -0.29 -0.10 0.52 0.47 0.42
## 3 3 49 0.31 -0.15 0.03 0.29 0.31 0.32
## 4 4 9 0.06 -0.23 -0.01 0.07 0.03 0.06
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 100 0.62 -0.13 0.17 0.59 0.50 0.75
## 2 2 7 0.04 -0.19 -0.05 0.05 0.09 0.00
## 3 3 49 0.31 -0.23 -0.09 0.32 0.41 0.23
## 4 4 3 0.02 -0.18 -0.01 0.03 0.00 0.02
## 9 9 1 0.01 -0.09 -0.01 0.01 0.00 0.00
##
## $lights_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 32 0.20 -0.23 -0.04 0.21 0.22 0.17
## 2 2 96 0.60 -0.29 -0.10 0.63 0.66 0.53
## 3 3 9 0.06 -0.41 -0.09 0.09 0.06 0.00
## 4 * 4 22 0.14 0.23 0.25 0.05 0.06 0.30
## 9 9 1 0.01 -0.09 -0.01 0.01 0.00 0.00
##
## $lights_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 3 0.02 -0.16 -0.01 0.03 0.00 0.02
## 2 2 27 0.17 -0.48 -0.21 0.27 0.12 0.06
## 3 * 3 113 0.71 0.19 0.27 0.56 0.84 0.83
## 4 4 16 0.10 -0.24 -0.04 0.13 0.03 0.09
## 9 9 1 0.01 -0.09 -0.01 0.01 0.00 0.00
##
## $lights_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 39 0.24 -0.28 -0.11 0.28 0.28 0.17
## 2 2 22 0.14 -0.37 -0.14 0.20 0.12 0.06
## 3 * 3 88 0.55 0.14 0.37 0.39 0.59 0.75
## 4 4 10 0.06 -0.35 -0.10 0.12 0.00 0.02
## 9 9 1 0.01 -0.09 -0.01 0.01 0.00 0.00
##
## $lights_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 50 0.31 -0.12 0.09 0.31 0.19 0.40
## 2 2 11 0.07 -0.30 -0.07 0.11 0.03 0.04
## 3 3 47 0.29 -0.24 -0.07 0.29 0.41 0.23
## 4 4 48 0.30 -0.15 0.07 0.25 0.38 0.32
## 9 9 4 0.03 -0.17 -0.02 0.04 0.00 0.02
##
## $lights_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 90 0.56 0.12 0.33 0.43 0.56 0.75
## 2 2 24 0.15 -0.28 -0.12 0.21 0.09 0.09
## 3 3 35 0.22 -0.42 -0.20 0.29 0.25 0.09
## 4 4 10 0.06 -0.24 -0.03 0.07 0.09 0.04
## 9 9 1 0.01 0.06 0.02 0.00 0.00 0.02
##
## $lights_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 14 0.09 -0.18 -0.05 0.11 0.09 0.06
## 2 2 83 0.52 -0.24 0.01 0.53 0.44 0.55
## 3 3 52 0.32 -0.21 0.01 0.31 0.38 0.32
## 4 * 4 10 0.06 -0.03 0.02 0.05 0.06 0.08
## 9 9 1 0.01 -0.02 0.00 0.00 0.03 0.00
##
## $lights_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 52 0.32 0.17 0.38 0.15 0.41 0.53
## 2 2 26 0.16 -0.41 -0.17 0.23 0.19 0.06
## 3 3 75 0.47 -0.35 -0.15 0.55 0.41 0.40
## 4 4 7 0.04 -0.17 -0.06 0.08 0.00 0.02
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 43 0.27 -0.09 0.09 0.23 0.28 0.32
## 2 2 49 0.31 -0.23 0.01 0.31 0.28 0.32
## 3 3 36 0.22 -0.22 0.03 0.21 0.22 0.25
## 4 4 32 0.20 -0.26 -0.14 0.25 0.22 0.11
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 57 0.36 -0.14 0.13 0.31 0.34 0.43
## 2 2 19 0.12 -0.34 -0.11 0.19 0.03 0.08
## 3 3 67 0.42 -0.24 -0.03 0.43 0.44 0.40
## 4 * 4 17 0.11 -0.04 0.01 0.08 0.19 0.09
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 62 0.39 0.01 0.30 0.28 0.31 0.58
## 2 2 27 0.17 -0.38 -0.22 0.25 0.19 0.04
## 3 3 62 0.39 -0.22 -0.05 0.39 0.47 0.34
## 4 4 9 0.06 -0.24 -0.04 0.08 0.03 0.04
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 25 0.16 -0.43 -0.18 0.25 0.06 0.08
## 2 2 27 0.17 -0.50 -0.28 0.32 0.03 0.04
## 3 * 3 102 0.64 0.38 0.53 0.36 0.88 0.89
## 4 4 6 0.04 -0.24 -0.07 0.07 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 22 0.14 -0.40 -0.19 0.23 0.09 0.04
## 2 2 11 0.07 -0.36 -0.11 0.13 0.00 0.02
## 3 * 3 126 0.79 0.21 0.32 0.63 0.91 0.94
## 4 4 1 0.01 -0.13 -0.01 0.01 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $lights_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 7 0.04 -0.35 -0.08 0.08 0.03 0
## 2 2 12 0.07 -0.50 -0.16 0.16 0.00 0
## 3 * 3 140 0.88 0.38 0.25 0.75 0.97 1
## 4 4 1 0.01 -0.13 -0.01 0.01 0.00 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0Booklet 2 - Hydraulic brakes
CTT::distractorAnalysis(na.omit(df[, items_brakes]) ,
keys_brakes ,
nGroups = 3 ,
digits = 2)## $brakes_05_old
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 31 0.29 -0.22 -0.04 0.24 0.41 0.20
## 2 * 2 29 0.27 -0.14 0.01 0.27 0.26 0.28
## 3 3 23 0.21 -0.22 -0.07 0.27 0.15 0.20
## 4 4 24 0.22 -0.05 0.12 0.20 0.18 0.32
## 9 9 1 0.01 -0.18 -0.02 0.02 0.00 0.00
##
## $brakes_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 9 0.08 -0.45 -0.16 0.16 0.03 0.00
## 2 * 2 45 0.42 0.01 0.15 0.33 0.50 0.48
## 3 3 50 0.46 -0.13 0.09 0.43 0.47 0.52
## 4 4 4 0.04 -0.24 -0.08 0.08 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $brakes_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 35 0.32 -0.15 -0.03 0.35 0.29 0.32
## 2 2 18 0.17 -0.33 -0.16 0.16 0.29 0.00
## 3 3 37 0.34 -0.24 -0.01 0.41 0.21 0.40
## 4 * 4 18 0.17 0.09 0.20 0.08 0.21 0.28
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $brakes_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 66 0.61 0.30 0.41 0.43 0.71 0.84
## 2 2 5 0.05 -0.31 -0.08 0.08 0.03 0.00
## 3 3 35 0.32 -0.51 -0.31 0.47 0.24 0.16
## 4 4 2 0.02 -0.09 -0.02 0.02 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $brakes_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 32 0.30 -0.23 -0.17 0.37 0.26 0.20
## 2 2 28 0.26 -0.36 -0.17 0.33 0.24 0.16
## 3 * 3 38 0.35 0.16 0.42 0.22 0.32 0.64
## 4 4 7 0.06 -0.14 -0.04 0.04 0.15 0.00
## 9 9 3 0.03 -0.26 -0.04 0.04 0.03 0.00
##
## $brakes_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 29 0.27 -0.27 -0.13 0.33 0.24 0.20
## 2 2 8 0.07 -0.31 -0.08 0.12 0.03 0.04
## 3 3 23 0.21 -0.34 -0.24 0.24 0.32 0.00
## 4 * 4 47 0.44 0.23 0.47 0.29 0.41 0.76
## 9 9 1 0.01 -0.21 -0.02 0.02 0.00 0.00
##
## $brakes_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 30 0.28 -0.35 -0.19 0.35 0.26 0.16
## 2 2 21 0.19 -0.35 -0.17 0.29 0.12 0.12
## 3 * 3 49 0.45 0.16 0.37 0.31 0.50 0.68
## 4 4 8 0.07 -0.10 -0.02 0.06 0.12 0.04
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $brakes_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 13 0.12 0.13 0.18 0.06 0.12 0.24
## 2 2 21 0.19 -0.19 -0.04 0.20 0.21 0.16
## 3 3 66 0.61 -0.20 -0.01 0.61 0.62 0.60
## 4 4 5 0.05 -0.23 -0.06 0.06 0.06 0.00
## 9 9 3 0.03 -0.24 -0.06 0.06 0.00 0.00
##
## $brakes_14
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 18 0.17 -0.22 -0.02 0.22 0.06 0.20
## 2 2 29 0.27 -0.39 -0.25 0.37 0.24 0.12
## 3 * 3 55 0.51 0.13 0.31 0.33 0.68 0.64
## 4 4 5 0.05 -0.16 -0.02 0.06 0.03 0.04
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00
##
## $brakes_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 43 0.40 0.20 0.36 0.24 0.47 0.60
## 2 2 12 0.11 -0.25 -0.10 0.14 0.12 0.04
## 3 3 48 0.44 -0.40 -0.21 0.53 0.41 0.32
## 4 4 4 0.04 -0.19 -0.02 0.06 0.00 0.04
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00
##
## $brakes_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 47 0.44 0.08 0.27 0.33 0.47 0.60
## 2 2 11 0.10 -0.47 -0.14 0.18 0.03 0.04
## 3 3 47 0.44 -0.22 -0.09 0.45 0.47 0.36
## 4 4 2 0.02 -0.07 -0.02 0.02 0.03 0.00
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00
##
## $brakes_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 45 0.42 0.14 0.35 0.29 0.44 0.64
## 2 2 7 0.06 -0.34 -0.12 0.12 0.03 0.00
## 3 3 52 0.48 -0.29 -0.15 0.51 0.53 0.36
## 4 4 3 0.03 -0.31 -0.06 0.06 0.00 0.00
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00
##
## $brakes_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 33 0.31 -0.40 -0.21 0.41 0.24 0.20
## 2 2 29 0.27 -0.22 -0.05 0.29 0.26 0.24
## 3 3 18 0.17 -0.12 -0.02 0.14 0.24 0.12
## 4 * 4 26 0.24 0.18 0.32 0.12 0.26 0.44
## 9 9 2 0.02 -0.20 -0.04 0.04 0.00 0.00
##
## $brakes_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 11 0.10 -0.36 -0.16 0.16 0.09 0.00
## 2 * 2 31 0.29 0.24 0.54 0.10 0.29 0.64
## 3 3 12 0.11 -0.38 -0.22 0.22 0.03 0.00
## 4 4 51 0.47 -0.19 -0.09 0.45 0.59 0.36
## 9 9 3 0.03 -0.20 -0.06 0.06 0.00 0.00
##
## $brakes_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 26 0.24 0.05 0.26 0.18 0.18 0.44
## 2 2 39 0.36 -0.34 -0.21 0.41 0.41 0.20
## 3 3 39 0.36 -0.20 -0.03 0.35 0.41 0.32
## 4 4 2 0.02 0.02 0.02 0.02 0.00 0.04
## 9 9 2 0.02 -0.12 -0.04 0.04 0.00 0.00
##
## $brakes_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 10 0.09 -0.28 -0.10 0.14 0.06 0.04
## 2 2 27 0.25 -0.35 -0.19 0.35 0.18 0.16
## 3 * 3 65 0.60 0.13 0.33 0.43 0.74 0.76
## 4 4 3 0.03 -0.16 -0.04 0.04 0.03 0.00
## 9 9 3 0.03 -0.07 0.00 0.04 0.00 0.04
##
## $brakes_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 17 0.16 -0.20 0.04 0.16 0.12 0.20
## 2 * 2 20 0.19 0.04 0.20 0.12 0.18 0.32
## 3 3 70 0.65 -0.27 -0.21 0.69 0.71 0.48
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00
##
## $brakes_24
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 64 0.59 0.30 0.53 0.39 0.65 0.92
## 2 2 17 0.16 -0.40 -0.20 0.24 0.12 0.04
## 3 3 25 0.23 -0.43 -0.29 0.33 0.24 0.04
## 4 4 1 0.01 -0.07 -0.02 0.02 0.00 0.00
## 9 9 1 0.01 -0.10 -0.02 0.02 0.00 0.00Booklet 3 - Exhaust system
CTT::distractorAnalysis(na.omit(df[, items_exhaust]) ,
keys_exhaust ,
nGroups = 3 ,
digits = 2)## $exhaust_05
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 5 0.07 -0.31 -0.12 0.12 0.06 0.00
## 2 * 2 25 0.34 0.00 0.23 0.31 0.28 0.53
## 3 3 42 0.58 -0.34 -0.18 0.58 0.66 0.40
## 4 4 1 0.01 0.08 0.07 0.00 0.00 0.07
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 17 0.23 -0.43 -0.24 0.31 0.25 0.07
## 2 * 2 37 0.51 0.22 0.49 0.31 0.53 0.80
## 3 3 16 0.22 -0.37 -0.21 0.35 0.16 0.13
## 4 4 2 0.03 -0.11 0.00 0.00 0.06 0.00
## 9 9 1 0.01 -0.18 -0.04 0.04 0.00 0.00
##
## $exhaust_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 11 0.15 -0.36 -0.20 0.27 0.09 0.07
## 2 2 26 0.36 -0.50 -0.41 0.54 0.31 0.13
## 3 * 3 33 0.45 0.37 0.68 0.12 0.56 0.80
## 4 4 3 0.04 -0.31 -0.08 0.08 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 24 0.33 0.05 0.38 0.15 0.38 0.53
## 2 2 3 0.04 -0.31 -0.12 0.12 0.00 0.00
## 3 3 44 0.60 -0.27 -0.19 0.65 0.62 0.47
## 4 4 2 0.03 -0.37 -0.08 0.08 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 20 0.27 0.07 0.31 0.15 0.28 0.47
## 2 2 29 0.40 -0.39 -0.21 0.54 0.31 0.33
## 3 3 19 0.26 -0.15 0.01 0.19 0.34 0.20
## 4 4 4 0.05 -0.21 -0.08 0.08 0.06 0.00
## 9 9 1 0.01 -0.24 -0.04 0.04 0.00 0.00
##
## $exhaust_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 29 0.40 0.21 0.55 0.12 0.50 0.67
## 2 2 17 0.23 -0.58 -0.37 0.50 0.06 0.13
## 3 3 6 0.08 -0.21 -0.12 0.12 0.09 0.00
## 4 4 20 0.27 -0.15 -0.03 0.23 0.34 0.20
## 9 9 1 0.01 -0.18 -0.04 0.04 0.00 0.00
##
## $exhaust_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 18 0.25 -0.39 -0.32 0.38 0.22 0.07
## 2 * 2 22 0.30 -0.13 0.14 0.19 0.38 0.33
## 3 3 23 0.32 -0.16 0.13 0.27 0.31 0.40
## 4 4 10 0.14 -0.04 0.05 0.15 0.09 0.20
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 5 0.07 -0.38 -0.12 0.12 0.06 0.00
## 2 2 21 0.29 -0.35 -0.24 0.31 0.38 0.07
## 3 * 3 39 0.53 0.31 0.63 0.31 0.53 0.93
## 4 4 8 0.11 -0.47 -0.27 0.27 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 4 0.05 -0.15 -0.08 0.08 0.06 0.0
## 2 2 55 0.75 -0.20 0.03 0.77 0.72 0.8
## 3 * 3 10 0.14 0.07 0.16 0.04 0.19 0.2
## 4 4 4 0.05 -0.36 -0.12 0.12 0.03 0.0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.0
##
## $exhaust_14
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.16 -0.25 -0.13 0.19 0.19 0.07
## 2 2 31 0.42 -0.37 -0.30 0.50 0.47 0.20
## 3 3 3 0.04 -0.31 -0.12 0.12 0.00 0.00
## 4 * 4 24 0.33 0.21 0.58 0.15 0.28 0.73
## 9 9 3 0.04 -0.22 -0.04 0.04 0.06 0.00
##
## $exhaust_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 9 0.12 0.12 0.23 0.04 0.12 0.27
## 2 2 40 0.55 -0.51 -0.33 0.73 0.47 0.40
## 3 3 8 0.11 -0.15 -0.02 0.15 0.06 0.13
## 4 * 4 14 0.19 -0.01 0.12 0.08 0.28 0.20
## 9 9 2 0.03 0.04 0.00 0.00 0.06 0.00
##
## $exhaust_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 8 0.11 -0.21 -0.05 0.12 0.12 0.07
## 2 2 35 0.48 -0.48 -0.42 0.62 0.50 0.20
## 3 * 3 26 0.36 0.06 0.33 0.27 0.31 0.60
## 4 4 4 0.05 0.07 0.13 0.00 0.06 0.13
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 19 0.26 -0.22 -0.10 0.23 0.34 0.13
## 2 2 10 0.14 -0.51 -0.35 0.35 0.03 0.00
## 3 * 3 36 0.49 0.26 0.60 0.27 0.50 0.87
## 4 4 7 0.10 -0.40 -0.15 0.15 0.09 0.00
## 9 9 1 0.01 -0.08 0.00 0.00 0.03 0.00
##
## $exhaust_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 40 0.55 -0.13 -0.07 0.54 0.59 0.47
## 2 2 5 0.07 -0.33 -0.15 0.15 0.03 0.00
## 3 3 26 0.36 -0.11 0.30 0.23 0.38 0.53
## 4 4 2 0.03 -0.29 -0.08 0.08 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $exhaust_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 20 0.27 0.09 0.32 0.08 0.38 0.4
## 2 2 8 0.11 -0.40 -0.19 0.19 0.09 0.0
## 3 3 45 0.62 -0.30 -0.13 0.73 0.53 0.6
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.0Booklet 4 - Timing belt
CTT::distractorAnalysis(na.omit(df[, items_timing_belt]) ,
keys_timing_belt ,
nGroups = 3 ,
digits = 2)## $timing_belt_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.13 -0.27 -0.11 0.15 0.18 0.04
## 2 2 29 0.32 -0.32 -0.19 0.42 0.27 0.23
## 3 * 3 46 0.50 0.16 0.33 0.36 0.48 0.69
## 4 4 5 0.05 -0.18 -0.02 0.06 0.06 0.04
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 19 0.21 -0.28 -0.17 0.24 0.27 0.08
## 2 * 2 27 0.29 0.26 0.38 0.12 0.30 0.50
## 3 3 44 0.48 -0.43 -0.25 0.64 0.39 0.38
## 4 4 2 0.02 0.07 0.04 0.00 0.03 0.04
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 16 0.17 -0.58 -0.36 0.39 0.06 0.04
## 2 * 2 38 0.41 0.48 0.63 0.06 0.55 0.69
## 3 3 29 0.32 -0.36 -0.27 0.42 0.33 0.15
## 4 4 9 0.10 -0.14 -0.01 0.12 0.06 0.12
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 57 0.62 0.09 0.25 0.52 0.61 0.77
## 2 2 9 0.10 -0.34 -0.15 0.15 0.12 0.00
## 3 3 19 0.21 -0.04 0.08 0.15 0.24 0.23
## 4 4 7 0.08 -0.41 -0.18 0.18 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 56 0.61 0.35 0.59 0.33 0.64 0.92
## 2 2 9 0.10 -0.21 -0.08 0.12 0.12 0.04
## 3 3 25 0.27 -0.51 -0.45 0.48 0.24 0.04
## 4 4 2 0.02 -0.26 -0.06 0.06 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 23 0.25 -0.08 0.10 0.21 0.24 0.31
## 2 2 14 0.15 -0.43 -0.27 0.27 0.15 0.00
## 3 3 52 0.57 0.07 0.24 0.45 0.58 0.69
## 4 4 3 0.03 -0.25 -0.06 0.06 0.03 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 25 0.27 -0.43 -0.27 0.42 0.21 0.15
## 2 * 2 23 0.25 0.23 0.44 0.06 0.24 0.50
## 3 3 31 0.34 -0.30 -0.23 0.42 0.36 0.19
## 4 4 13 0.14 0.00 0.06 0.09 0.18 0.15
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 7 0.08 -0.14 -0.09 0.09 0.12 0.00
## 2 2 36 0.39 -0.58 -0.47 0.67 0.27 0.19
## 3 * 3 46 0.50 0.43 0.66 0.15 0.61 0.81
## 4 4 3 0.03 -0.29 -0.09 0.09 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_14_old
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 15 0.16 -0.27 -0.16 0.27 0.09 0.12
## 2 * 2 60 0.65 0.21 0.45 0.39 0.76 0.85
## 3 3 6 0.07 -0.36 -0.15 0.15 0.03 0.00
## 4 4 10 0.11 -0.23 -0.11 0.15 0.12 0.04
## 9 9 1 0.01 -0.10 -0.03 0.03 0.00 0.00
##
## $timing_belt_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 26 0.28 -0.40 -0.35 0.42 0.30 0.08
## 2 2 14 0.15 -0.38 -0.17 0.24 0.12 0.08
## 3 3 6 0.07 -0.04 0.05 0.06 0.03 0.12
## 4 * 4 46 0.50 0.25 0.46 0.27 0.55 0.73
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 6 0.07 -0.40 -0.15 0.15 0.03 0.00
## 2 2 22 0.24 -0.39 -0.24 0.39 0.15 0.15
## 3 * 3 64 0.70 0.28 0.39 0.45 0.82 0.85
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 15 0.16 -0.18 -0.07 0.18 0.18 0.12
## 2 2 37 0.40 -0.30 -0.19 0.42 0.52 0.23
## 3 3 11 0.12 -0.27 -0.10 0.18 0.09 0.08
## 4 * 4 28 0.30 0.22 0.40 0.18 0.21 0.58
## 9 9 1 0.01 -0.27 -0.03 0.03 0.00 0.00
##
## $timing_belt_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 28 0.30 -0.05 0.07 0.24 0.36 0.31
## 2 2 18 0.20 -0.27 -0.13 0.24 0.21 0.12
## 3 3 13 0.14 -0.28 -0.17 0.21 0.15 0.04
## 4 4 31 0.34 0.04 0.30 0.24 0.27 0.54
## 9 9 2 0.02 -0.26 -0.06 0.06 0.00 0.00
##
## $timing_belt_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 37 0.40 0.13 0.38 0.27 0.33 0.65
## 2 2 8 0.09 -0.23 -0.08 0.12 0.09 0.04
## 3 3 42 0.46 -0.34 -0.25 0.52 0.55 0.27
## 4 4 4 0.04 -0.06 -0.02 0.06 0.03 0.04
## 9 9 1 0.01 -0.20 -0.03 0.03 0.00 0.00
##
## $timing_belt_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 7 0.08 -0.28 -0.08 0.12 0.06 0.04
## 2 * 2 26 0.28 -0.04 0.14 0.24 0.24 0.38
## 3 3 57 0.62 -0.21 -0.10 0.64 0.67 0.54
## 4 4 2 0.02 0.10 0.04 0.00 0.03 0.04
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 15 0.16 -0.40 -0.27 0.27 0.18 0.00
## 2 * 2 39 0.42 0.23 0.47 0.18 0.48 0.65
## 3 3 37 0.40 -0.30 -0.17 0.52 0.33 0.35
## 4 4 1 0.01 -0.20 -0.03 0.03 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $timing_belt_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 23 0.25 -0.53 -0.38 0.45 0.18 0.08
## 2 2 7 0.08 -0.22 -0.12 0.12 0.09 0.00
## 3 * 3 60 0.65 0.38 0.56 0.36 0.73 0.92
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
## 9 9 2 0.02 -0.26 -0.06 0.06 0.00 0.00Booklet 5 - Steering and front axle
CTT::distractorAnalysis(na.omit(df[, items_steering]) ,
keys_steering ,
nGroups = 3 ,
digits = 2)## $steering_05
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 13 0.14 -0.25 -0.13 0.18 0.16 0.05
## 2 * 2 41 0.45 -0.02 0.23 0.36 0.53 0.59
## 3 3 25 0.27 -0.50 -0.22 0.36 0.21 0.14
## 4 4 12 0.13 0.03 0.13 0.10 0.11 0.23
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $steering_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 54 0.59 -0.29 -0.17 0.62 0.68 0.45
## 2 2 9 0.10 -0.16 -0.03 0.12 0.05 0.09
## 3 3 22 0.24 -0.17 0.16 0.20 0.21 0.36
## 4 * 4 5 0.05 0.01 0.05 0.04 0.05 0.09
## 9 9 1 0.01 -0.17 -0.02 0.02 0.00 0.00
##
## $steering_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 32 0.35 -0.27 -0.11 0.38 0.37 0.27
## 2 2 42 0.46 -0.22 0.01 0.44 0.53 0.45
## 3 3 5 0.05 -0.30 -0.08 0.08 0.05 0.00
## 4 * 4 12 0.13 -0.01 0.17 0.10 0.05 0.27
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $steering_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 45 0.49 0.16 0.32 0.36 0.63 0.68
## 2 2 7 0.08 -0.28 -0.14 0.14 0.00 0.00
## 3 3 30 0.33 -0.43 -0.09 0.36 0.32 0.27
## 4 4 9 0.10 -0.30 -0.09 0.14 0.05 0.05
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $steering_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 57 0.63 0.14 0.27 0.50 0.79 0.77
## 2 2 13 0.14 -0.39 -0.15 0.20 0.11 0.05
## 3 3 16 0.18 -0.38 -0.10 0.24 0.05 0.14
## 4 4 5 0.05 -0.16 -0.01 0.06 0.05 0.05
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $steering_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 17 0.19 -0.36 -0.08 0.22 0.16 0.14
## 2 * 2 33 0.36 0.17 0.42 0.22 0.42 0.64
## 3 3 37 0.41 -0.39 -0.32 0.50 0.42 0.18
## 4 4 3 0.03 -0.17 0.01 0.04 0.00 0.05
## 9 9 1 0.01 -0.22 -0.02 0.02 0.00 0.00
##
## $steering_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 32 0.35 -0.36 -0.13 0.40 0.32 0.27
## 2 2 16 0.18 -0.24 -0.06 0.20 0.16 0.14
## 3 3 18 0.20 -0.33 -0.17 0.26 0.16 0.09
## 4 * 4 23 0.25 0.19 0.40 0.10 0.37 0.50
## 9 9 2 0.02 -0.20 -0.04 0.04 0.00 0.00
##
## $steering_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 32 0.35 0.27 0.59 0.18 0.32 0.77
## 2 2 5 0.05 -0.30 -0.10 0.10 0.00 0.00
## 3 3 52 0.57 -0.51 -0.45 0.68 0.68 0.23
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
## 9 9 2 0.02 -0.27 -0.04 0.04 0.00 0.00
##
## $steering_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 53 0.58 0.21 0.46 0.40 0.74 0.86
## 2 2 14 0.15 -0.30 -0.15 0.20 0.16 0.05
## 3 3 14 0.15 -0.42 -0.17 0.22 0.11 0.05
## 4 4 8 0.09 -0.28 -0.09 0.14 0.00 0.05
## 9 9 2 0.02 -0.27 -0.04 0.04 0.00 0.00
##
## $steering_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 15 0.16 -0.42 -0.19 0.24 0.11 0.05
## 2 * 2 39 0.43 0.04 0.18 0.32 0.63 0.50
## 3 3 13 0.14 -0.41 -0.11 0.20 0.05 0.09
## 4 4 22 0.24 -0.03 0.16 0.20 0.21 0.36
## 9 9 2 0.02 -0.27 -0.04 0.04 0.00 0.00
##
## $steering_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 54 0.59 0.18 0.32 0.50 0.58 0.82
## 2 2 2 0.02 -0.27 -0.04 0.04 0.00 0.00
## 3 3 33 0.36 -0.50 -0.24 0.42 0.42 0.18
## 4 4 1 0.01 -0.06 -0.02 0.02 0.00 0.00
## 9 9 1 0.01 -0.22 -0.02 0.02 0.00 0.00
##
## $steering_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 4 0.04 -0.23 -0.06 0.06 0.05 0.00
## 2 * 2 65 0.71 0.14 0.31 0.60 0.79 0.91
## 3 3 21 0.23 -0.45 -0.23 0.32 0.16 0.09
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
## 9 9 1 0.01 -0.22 -0.02 0.02 0.00 0.00
##
## $steering_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 24 0.26 -0.12 0.07 0.20 0.42 0.27
## 2 2 23 0.25 -0.23 -0.10 0.28 0.26 0.18
## 3 3 28 0.31 -0.54 -0.35 0.44 0.21 0.09
## 4 * 4 15 0.16 0.24 0.39 0.06 0.11 0.45
## 9 9 1 0.01 -0.22 -0.02 0.02 0.00 0.003.3.4 Recode items
Create a new data frame
Copy case numbers and demographic data to a new data frame named ‘df.dichotom’.
df.dichotom <- df[,c(1:5)]Recode
Recode the marked options (1, 2, 3, or 4) into credits (0 or 1) for each item.
Recoding starts at item 05 for each test block. Items 01 to 04 are not actual items but contain an introduction to the test and an example.
The code below also contains a short summary for each item (as
comments, starting with a hash #).
Booklet 1 - Lighting system
# Item 05 - Checking headlights
df.dichotom$lights_05 <- car::recode(df$lights_05 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 06 - Checking rear lights
df.dichotom$lights_06 <- car::recode(df$lights_06 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 07 - Changing a light bulb
df.dichotom$lights_07 <- car::recode(df$lights_07 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 08 - Diagnosing the cause of a loose connection
df.dichotom$lights_08 <- car::recode(df$lights_08 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_09' is missing here because it was added to the instrument after this study.
# Item 10 - Identifying a wire in a circuit diagram
df.dichotom$lights_10 <- car::recode(df$lights_10 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 11 - Choosing a repair method to fix a ruptured wire
df.dichotom$lights_11 <- car::recode(df$lights_11 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 12 - Judging different repair methods of fixing a ruptured wire
df.dichotom$lights_12 <- car::recode(df$lights_12 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 13 - Identifying wires in a circuit diagram
df.dichotom$lights_13 <- car::recode(df$lights_13 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_14' is missing here because it was added to the instrument after this study.
# Item 15 - Judging the result of an electric measurement
df.dichotom$lights_15 <- car::recode(df$lights_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Judging a method of checking a light bulb
df.dichotom$lights_16 <- car::recode(df$lights_16 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 17 - Judging potential subsequent actions
df.dichotom$lights_17 <- car::recode(df$lights_17 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 18 - Judging a repair method of fixing a ruptured wire
df.dichotom$lights_18 <- car::recode(df$lights_18 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 19 - Explaining why halogen lamps should never be touched with bare fingers
df.dichotom$lights_19 <- car::recode(df$lights_19 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 20 - Explaining what happens if halogen lamps are touched with bare fingers
df.dichotom$lights_20 <- car::recode(df$lights_20 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 21 - Adjusting headlights
df.dichotom$lights_21 <- car::recode(df$lights_21 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 22 - Judging potential subsequent actions
df.dichotom$lights_22 <- car::recode(df$lights_22 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 23 - Explaining the potential risks of wrongly adjusted headlights
df.dichotom$lights_23 <- car::recode(df$lights_23 , "c(3) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_24' is missing here because it was added to the instrument after this study.Booklet 2 - Hydraulic brakes
# Item 05_old - Judging potential diagnostic actions
# Note: In later studies, a revised version of item 'brakes_05_old' was used --> brakes_05_new
df.dichotom$brakes_05_old <- car::recode(df$brakes_05_old , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'brakes_06' is missing here because it was added to the instrument after this study.
# Item 07 - Removing brake pads
df.dichotom$brakes_07 <- car::recode(df$brakes_07 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 08 - Judging (alleged) replacement intervals for brake pads
df.dichotom$brakes_08 <- car::recode(df$brakes_08 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 09 - Deciding which parts of the brake need to be replaced
df.dichotom$brakes_09 <- car::recode(df$brakes_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Fitting new brake pads
df.dichotom$brakes_10 <- car::recode(df$brakes_10 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 11 - Mounting brake calliper
df.dichotom$brakes_11 <- car::recode(df$brakes_11 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 12 - Replacing brake hose
df.dichotom$brakes_12 <- car::recode(df$brakes_12 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 13 - Judging potential diagnostic actions
df.dichotom$brakes_13 <- car::recode(df$brakes_13 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 14 - Explaining signs of wear
df.dichotom$brakes_14 <- car::recode(df$brakes_14 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 15 - Judging the potential consequences of an incomplete repair
df.dichotom$brakes_15 <- car::recode(df$brakes_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Judging potential subsequent actions
df.dichotom$brakes_16 <- car::recode(df$brakes_16 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'brakes_17' is missing here because it was added to the instrument after this study.
# Item 18 - Replacing brake disk
df.dichotom$brakes_18 <- car::recode(df$brakes_18 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 19 - Judging potential subsequent actions
df.dichotom$brakes_19 <- car::recode(df$brakes_19 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 20 - Judging two methods of resetting the position of brake calliper pistons
df.dichotom$brakes_20 <- car::recode(df$brakes_20 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 21 - Explaining why calliper pistons of some brakes must be turned to reset
df.dichotom$brakes_21 <- car::recode(df$brakes_21 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 22 - Assembling rear brakes
df.dichotom$brakes_22 <- car::recode(df$brakes_22 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 23 - Judging potential subsequent actions
df.dichotom$brakes_23 <- car::recode(df$brakes_23 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 24 - Explaining why the brake pedal must be pressed several times after replacing brake pads
df.dichotom$brakes_24 <- car::recode(df$brakes_24 , "c(1) = '1' ; NA = NA ; else = '0'") Booklet 3 - Exhaust system
Note. The entire booklet was removed from the instrument after the first validation study; its items (exhaust_05 to exhaust_19) are therefore missing in subsequent data sets.
# Item 05 - Judging potential diagnostic actions
df.dichotom$exhaust_05 <- car::recode(df$exhaust_05 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 06 - Judging why modern exhaust systems last longer than older models
df.dichotom$exhaust_06 <- car::recode(df$exhaust_06 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 07 - Naming potential locations of Lambda sensors
df.dichotom$exhaust_07 <- car::recode(df$exhaust_07 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 08 - Judging diagnostic actions from the video
df.dichotom$exhaust_08 <- car::recode(df$exhaust_08 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 09 - Deciding which exhaust parts need to be replaced
df.dichotom$exhaust_09 <- car::recode(df$exhaust_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Naming potential consequences of a leaking gasket
df.dichotom$exhaust_10 <- car::recode(df$exhaust_10 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 11 - Naming potential consequences of an air leak near the Lambda sensor
df.dichotom$exhaust_11 <- car::recode(df$exhaust_11 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 12 - Replacing a gasket
df.dichotom$exhaust_12 <- car::recode(df$exhaust_12 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 13 - Replacing Lambda sensor
df.dichotom$exhaust_13 <- car::recode(df$exhaust_13 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 14 - Interpreting a waveform displayed on an oscilloscope
df.dichotom$exhaust_14 <- car::recode(df$exhaust_14 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 15 - Judging potential consequences of a broken Lambda sensor
df.dichotom$exhaust_15 <- car::recode(df$exhaust_15 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 16 - Removing an exhaust pipe
df.dichotom$exhaust_16 <- car::recode(df$exhaust_16 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 17 - Fitting a new exhaust pipe
df.dichotom$exhaust_17 <- car::recode(df$exhaust_17 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 18 - Judging the result of the automotive technician's work in the video
df.dichotom$exhaust_18 <- car::recode(df$exhaust_18 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 19 - Judging potential subsequent actions
df.dichotom$exhaust_19 <- car::recode(df$exhaust_19 , "c(1) = '1' ; NA = NA ; else = '0'") Booklet 4 - Timing belt
# Note: Item 'timing_belt_05' is missing here because it was added to the instrument after this study.
# Item 06 - Removing serpentine belt
df.dichotom$timing_belt_06 <- car::recode(df$timing_belt_06 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 07 - Judging potential consequences of a broken idler pulley
df.dichotom$timing_belt_07 <- car::recode(df$timing_belt_07 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 08 - Naming potential causes of tooth skipping
df.dichotom$timing_belt_08 <- car::recode(df$timing_belt_08 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 09 - Identifying the purpose of a special tool
df.dichotom$timing_belt_09 <- car::recode(df$timing_belt_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Identifying the purpose of another special tool
df.dichotom$timing_belt_10 <- car::recode(df$timing_belt_10 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 11 - Judging potential subsequent actions
df.dichotom$timing_belt_11 <- car::recode(df$timing_belt_11 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 12 - Replacing the timing belt
df.dichotom$timing_belt_12 <- car::recode(df$timing_belt_12 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 13 - Identifying repair steps skipped by the technician in the video
df.dichotom$timing_belt_13 <- car::recode(df$timing_belt_13 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 14 - Naming engine designs with two timing belts
# Note: In subsequent studies, a revised version of item 'timing_belt_14_old' was used --> timing_belt_14_new
df.dichotom$timing_belt_14_old <- car::recode(df$timing_belt_14_old , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'timing_belt_15' is missing here because it was added to the instrument after this study.
# Item 16 - Tensioning the timing belt
df.dichotom$timing_belt_16 <- car::recode(df$timing_belt_16 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 17 - Judging potential subsequent actions
df.dichotom$timing_belt_17 <- car::recode(df$timing_belt_17 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 18 - Attaching crankshaft pulley
df.dichotom$timing_belt_18 <- car::recode(df$timing_belt_18 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 19 - Attaching serpentine belt
df.dichotom$timing_belt_19 <- car::recode(df$timing_belt_19 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 20 - Judging potential subsequent actions
df.dichotom$timing_belt_20 <- car::recode(df$timing_belt_20 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 21 - Judging potential consequences of too high belt tension
df.dichotom$timing_belt_21 <- car::recode(df$timing_belt_21 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 22 - Explaining consequences of a torn timing belt to the engine
df.dichotom$timing_belt_22 <- car::recode(df$timing_belt_22 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 23 - Naming engine parts potentially affected by a torn timing belt
df.dichotom$timing_belt_23 <- car::recode(df$timing_belt_23 , "c(3) = '1' ; NA = NA ; else = '0'")
# Note: Item 'timing_belt_24' is missing here because it was added to the instrument after this study.Booklet 5 - Steering and front axle
# Item 05 - Naming the consequences of a broken anti-roll bar link on the car's operating permit
df.dichotom$steering_05 <- car::recode(df$steering_05 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 06 - Removing a tie rod (part 1)
df.dichotom$steering_06 <- car::recode(df$steering_06 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 07 - Removing a tie rod (part 2)
df.dichotom$steering_07 <- car::recode(df$steering_07 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 08 - Removing a steering boot
df.dichotom$steering_08 <- car::recode(df$steering_08 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 09 - Fitting a new steering boot
df.dichotom$steering_09 <- car::recode(df$steering_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Judging the consequences of insufficient amount of grease
df.dichotom$steering_10 <- car::recode(df$steering_10 , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_11' is missing here because it was added to the instrument after this study.
# Item 12 - Fitting a new tie rod
df.dichotom$steering_12 <- car::recode(df$steering_12 , "c(4) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_13' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_14' is missing here because it was added to the instrument after this study.
# Item 15 - Judging the consequences of a broken anti-roll bar link
df.dichotom$steering_15 <- car::recode(df$steering_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Naming the function of the anti-roll bar link
df.dichotom$steering_16 <- car::recode(df$steering_16 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 17 - Fitting a new anti-roll bar link
df.dichotom$steering_17 <- car::recode(df$steering_17 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 18 - Judging potential subsequent actions
df.dichotom$steering_18 <- car::recode(df$steering_18 , "c(1) = '1' ; NA = NA ; else = '0'") #
# Note: Item 'steering_19' is missing here because it was added to the instrument after this study.
# Item 20 - Judging the consequences of insufficient wheel alignment
df.dichotom$steering_20 <- car::recode(df$steering_20 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 21 - Naming potential reasons for wheel alignment
df.dichotom$steering_21 <- car::recode(df$steering_21 , "c(4) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_22' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_23' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_24' is missing here because it was added to the instrument after this study.3.3.5 Descriptive item coefficients
Display descriptive item coefficients.
psych::describe(df.dichotom[,c(6:85)])## vars n mean sd median trimmed mad min max range skew
## lights_05 1 160 0.28 0.45 0.0 0.23 0.00 0 1 1 0.96
## lights_06 2 160 0.73 0.44 1.0 0.79 0.00 0 1 1 -1.03
## lights_07 3 160 0.16 0.37 0.0 0.08 0.00 0 1 1 1.81
## lights_08 4 160 0.62 0.49 1.0 0.66 0.00 0 1 1 -0.51
## lights_10 5 160 0.14 0.35 0.0 0.05 0.00 0 1 1 2.09
## lights_11 6 160 0.71 0.46 1.0 0.76 0.00 0 1 1 -0.90
## lights_12 7 160 0.55 0.50 1.0 0.56 0.00 0 1 1 -0.20
## lights_13 8 160 0.31 0.46 0.0 0.27 0.00 0 1 1 0.80
## lights_15 9 160 0.56 0.50 1.0 0.58 0.00 0 1 1 -0.25
## lights_16 10 160 0.06 0.24 0.0 0.00 0.00 0 1 1 3.58
## lights_17 11 160 0.32 0.47 0.0 0.28 0.00 0 1 1 0.74
## lights_18 12 160 0.27 0.44 0.0 0.21 0.00 0 1 1 1.03
## lights_19 13 160 0.11 0.31 0.0 0.01 0.00 0 1 1 2.53
## lights_20 14 160 0.39 0.49 0.0 0.36 0.00 0 1 1 0.46
## lights_21 15 160 0.64 0.48 1.0 0.67 0.00 0 1 1 -0.57
## lights_22 16 160 0.79 0.41 1.0 0.86 0.00 0 1 1 -1.39
## lights_23 17 160 0.88 0.33 1.0 0.97 0.00 0 1 1 -2.25
## brakes_05_old 18 108 0.27 0.45 0.0 0.22 0.00 0 1 1 1.03
## brakes_07 19 108 0.42 0.50 0.0 0.40 0.00 0 1 1 0.33
## brakes_08 20 108 0.17 0.37 0.0 0.09 0.00 0 1 1 1.76
## brakes_09 21 108 0.61 0.49 1.0 0.64 0.00 0 1 1 -0.45
## brakes_10 22 108 0.35 0.48 0.0 0.32 0.00 0 1 1 0.61
## brakes_11 23 108 0.44 0.50 0.0 0.42 0.00 0 1 1 0.26
## brakes_12 24 108 0.45 0.50 0.0 0.44 0.00 0 1 1 0.18
## brakes_13 25 108 0.12 0.33 0.0 0.03 0.00 0 1 1 2.30
## brakes_14 26 108 0.51 0.50 1.0 0.51 0.00 0 1 1 -0.04
## brakes_15 27 108 0.40 0.49 0.0 0.38 0.00 0 1 1 0.41
## brakes_16 28 108 0.44 0.50 0.0 0.42 0.00 0 1 1 0.26
## brakes_18 29 108 0.42 0.50 0.0 0.40 0.00 0 1 1 0.33
## brakes_19 30 108 0.24 0.43 0.0 0.18 0.00 0 1 1 1.20
## brakes_20 31 108 0.29 0.45 0.0 0.24 0.00 0 1 1 0.93
## brakes_21 32 108 0.24 0.43 0.0 0.18 0.00 0 1 1 1.20
## brakes_22 33 108 0.60 0.49 1.0 0.62 0.00 0 1 1 -0.41
## brakes_23 34 108 0.19 0.39 0.0 0.11 0.00 0 1 1 1.60
## brakes_24 35 108 0.59 0.49 1.0 0.61 0.00 0 1 1 -0.37
## exhaust_05 36 73 0.34 0.48 0.0 0.31 0.00 0 1 1 0.65
## exhaust_06 37 73 0.51 0.50 1.0 0.51 0.00 0 1 1 -0.03
## exhaust_07 38 73 0.45 0.50 0.0 0.44 0.00 0 1 1 0.19
## exhaust_08 39 73 0.33 0.47 0.0 0.29 0.00 0 1 1 0.71
## exhaust_09 40 73 0.27 0.45 0.0 0.22 0.00 0 1 1 0.99
## exhaust_10 41 73 0.40 0.49 0.0 0.37 0.00 0 1 1 0.41
## exhaust_11 42 73 0.30 0.46 0.0 0.25 0.00 0 1 1 0.85
## exhaust_12 43 73 0.53 0.50 1.0 0.54 0.00 0 1 1 -0.13
## exhaust_13 44 73 0.14 0.35 0.0 0.05 0.00 0 1 1 2.07
## exhaust_14 45 73 0.33 0.47 0.0 0.29 0.00 0 1 1 0.71
## exhaust_15 46 73 0.19 0.40 0.0 0.12 0.00 0 1 1 1.53
## exhaust_16 47 73 0.36 0.48 0.0 0.32 0.00 0 1 1 0.59
## exhaust_17 48 73 0.49 0.50 0.0 0.49 0.00 0 1 1 0.03
## exhaust_18 49 73 0.55 0.50 1.0 0.56 0.00 0 1 1 -0.19
## exhaust_19 50 73 0.27 0.45 0.0 0.22 0.00 0 1 1 0.99
## timing_belt_06 51 92 0.50 0.50 0.5 0.50 0.74 0 1 1 0.00
## timing_belt_07 52 92 0.29 0.46 0.0 0.24 0.00 0 1 1 0.89
## timing_belt_08 53 92 0.41 0.50 0.0 0.39 0.00 0 1 1 0.35
## timing_belt_09 54 92 0.62 0.49 1.0 0.65 0.00 0 1 1 -0.48
## timing_belt_10 55 92 0.61 0.49 1.0 0.64 0.00 0 1 1 -0.44
## timing_belt_11 56 92 0.25 0.44 0.0 0.19 0.00 0 1 1 1.14
## timing_belt_12 57 92 0.25 0.44 0.0 0.19 0.00 0 1 1 1.14
## timing_belt_13 58 92 0.50 0.50 0.5 0.50 0.74 0 1 1 0.00
## timing_belt_14_old 59 92 0.65 0.48 1.0 0.69 0.00 0 1 1 -0.63
## timing_belt_16 60 92 0.50 0.50 0.5 0.50 0.74 0 1 1 0.00
## timing_belt_17 61 92 0.70 0.46 1.0 0.74 0.00 0 1 1 -0.84
## timing_belt_18 62 92 0.30 0.46 0.0 0.26 0.00 0 1 1 0.84
## timing_belt_19 63 92 0.30 0.46 0.0 0.26 0.00 0 1 1 0.84
## timing_belt_20 64 92 0.40 0.49 0.0 0.38 0.00 0 1 1 0.39
## timing_belt_21 65 92 0.28 0.45 0.0 0.23 0.00 0 1 1 0.95
## timing_belt_22 66 92 0.42 0.50 0.0 0.41 0.00 0 1 1 0.30
## timing_belt_23 67 92 0.65 0.48 1.0 0.69 0.00 0 1 1 -0.63
## steering_05 68 91 0.45 0.50 0.0 0.44 0.00 0 1 1 0.20
## steering_06 69 91 0.05 0.23 0.0 0.00 0.00 0 1 1 3.84
## steering_07 70 91 0.13 0.34 0.0 0.04 0.00 0 1 1 2.14
## steering_08 71 91 0.49 0.50 0.0 0.49 0.00 0 1 1 0.02
## steering_09 72 91 0.63 0.49 1.0 0.66 0.00 0 1 1 -0.51
## steering_10 73 91 0.36 0.48 0.0 0.33 0.00 0 1 1 0.56
## steering_12 74 91 0.25 0.44 0.0 0.19 0.00 0 1 1 1.12
## steering_15 75 91 0.35 0.48 0.0 0.32 0.00 0 1 1 0.61
## steering_16 76 91 0.58 0.50 1.0 0.60 0.00 0 1 1 -0.33
## steering_17 77 91 0.43 0.50 0.0 0.41 0.00 0 1 1 0.28
## steering_18 78 91 0.59 0.49 1.0 0.62 0.00 0 1 1 -0.37
## steering_20 79 91 0.71 0.45 1.0 0.77 0.00 0 1 1 -0.93
## steering_21 80 91 0.16 0.37 0.0 0.08 0.00 0 1 1 1.78
## kurtosis se
## lights_05 -1.08 0.04
## lights_06 -0.94 0.04
## lights_07 1.29 0.03
## lights_08 -1.75 0.04
## lights_10 2.36 0.03
## lights_11 -1.20 0.04
## lights_12 -1.97 0.04
## lights_13 -1.37 0.04
## lights_15 -1.95 0.04
## lights_16 10.89 0.02
## lights_17 -1.46 0.04
## lights_18 -0.94 0.04
## lights_19 4.44 0.02
## lights_20 -1.80 0.04
## lights_21 -1.69 0.04
## lights_22 -0.06 0.03
## lights_23 3.07 0.03
## brakes_05_old -0.95 0.04
## brakes_07 -1.91 0.05
## brakes_08 1.12 0.04
## brakes_09 -1.81 0.05
## brakes_10 -1.64 0.05
## brakes_11 -1.95 0.05
## brakes_12 -1.98 0.05
## brakes_13 3.33 0.03
## brakes_14 -2.02 0.05
## brakes_15 -1.85 0.05
## brakes_16 -1.95 0.05
## brakes_18 -1.91 0.05
## brakes_19 -0.57 0.04
## brakes_20 -1.15 0.04
## brakes_21 -0.57 0.04
## brakes_22 -1.85 0.05
## brakes_23 0.56 0.04
## brakes_24 -1.88 0.05
## exhaust_05 -1.60 0.06
## exhaust_06 -2.03 0.06
## exhaust_07 -1.99 0.06
## exhaust_08 -1.51 0.06
## exhaust_09 -1.03 0.05
## exhaust_10 -1.86 0.06
## exhaust_11 -1.30 0.05
## exhaust_12 -2.01 0.06
## exhaust_13 2.31 0.04
## exhaust_14 -1.51 0.06
## exhaust_15 0.36 0.05
## exhaust_16 -1.68 0.06
## exhaust_17 -2.03 0.06
## exhaust_18 -1.99 0.06
## exhaust_19 -1.03 0.05
## timing_belt_06 -2.02 0.05
## timing_belt_07 -1.22 0.05
## timing_belt_08 -1.90 0.05
## timing_belt_09 -1.78 0.05
## timing_belt_10 -1.83 0.05
## timing_belt_11 -0.72 0.05
## timing_belt_12 -0.72 0.05
## timing_belt_13 -2.02 0.05
## timing_belt_14_old -1.62 0.05
## timing_belt_16 -2.02 0.05
## timing_belt_17 -1.31 0.05
## timing_belt_18 -1.31 0.05
## timing_belt_19 -1.31 0.05
## timing_belt_20 -1.87 0.05
## timing_belt_21 -1.11 0.05
## timing_belt_22 -1.93 0.05
## timing_belt_23 -1.62 0.05
## steering_05 -1.98 0.05
## steering_06 12.90 0.02
## steering_07 2.61 0.04
## steering_08 -2.02 0.05
## steering_09 -1.75 0.05
## steering_10 -1.70 0.05
## steering_12 -0.76 0.05
## steering_15 -1.64 0.05
## steering_16 -1.91 0.05
## steering_17 -1.94 0.05
## steering_18 -1.88 0.05
## steering_20 -1.14 0.05
## steering_21 1.17 0.043.3.6 Sum scales
For each test booklet, we calculate raw scores (i.e., the sum of correct responses) and display a descriptive statistic.
Note: As the test booklets differ in length (i.e., number of items), the raw scores cannot be used to compare the difficulty of the booklets. In order to standardise the scores, the raw scores must be divided by the number of items for each booklet.
Booklet 1 - Lighting system
df.dichotom$sum_lights <- rowSums(df.dichotom[,items_lights] , na.rm = FALSE)
psych::describe(df.dichotom$sum_lights)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 160 7.52 2.14 8 7.59 2.97 2 12 10 -0.25 -0.26 0.17Booklet 2 - Hydraulic brakes
df.dichotom$sum_brakes <- rowSums(df.dichotom[,items_brakes] , na.rm = FALSE)
psych::describe(df.dichotom$sum_brakes)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 108 6.73 2.57 7 6.69 2.97 1 13 12 0.09 -0.3 0.25Booklet 3 - Exhaust system
df.dichotom$sum_exhaust <- rowSums(df.dichotom[,items_exhaust] , na.rm = FALSE)
psych::describe(df.dichotom$sum_exhaust)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 73 5.47 2.22 5 5.47 2.97 0 10 10 -0.05 -0.65 0.26Booklet 4 - Timing belt
df.dichotom$sum_timing_belt <- rowSums(df.dichotom[,items_timing_belt] , na.rm = FALSE)
psych::describe(df.dichotom$sum_timing_belt)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 92 7.65 2.92 8 7.68 2.97 1 14 13 -0.08 -0.77 0.3Booklet 5 - Steering and front axle
df.dichotom$sum_steering <- rowSums(df.dichotom[,items_steering] , na.rm = FALSE)
psych::describe(df.dichotom$sum_steering)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 91 5.21 2.04 5 5.23 1.48 1 10 9 -0.06 -0.54 0.213.3.7 Scale reliability (internal consistency)
We calculate two coefficients of internal consistency: Cronbach’s alpha and Guttman’s lambda-6. Note that internal consistency is of limited use for tests measuring a wide range of various abilities (i.e., instruments that are not internally consistent by design). In such cases, the criteria recommended for psychological tests will not be met, and values below .7 can be expected due to the diversity of the construct (Field, Miles, & Field, 2012, p. 799).
However, low internal consistency affects the outcome of analyses such as group comparisons and thereby the inferences based on such tests. Therefore it should be reported. If the association to the underlying variable varies across tests items, Guttman’s lambda-6 is more appropriate than Cronbach’s alpha.
Note. The code below will produce warnings about some items which are negatively correlated with the scales and probably should be reversed. These messages must be ignored as the test does not contain reversed items. In order to increase internal consistency, the items in question might be removed from the scales; however, this will affect ecological validity (see Hartmann, Güzel, & Gschwendtner, 2023b, for details).
Booklet 1 - Lighting system (no items removed)
reli_lights <- psych::alpha(df.dichotom[,items_lights] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_lights$total$raw_alpha ## [1] 0.3342828reli_lights$total$`G6(smc)` ## [1] 0.4203614Booklet 2 - Hydraulic brakes (no items removed)
reli_brakes <- psych::alpha(df.dichotom[,items_brakes] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_brakes$total$raw_alpha ## [1] 0.4387216reli_brakes$total$`G6(smc)` ## [1] 0.5205712Booklet 3 - Exhaust system (no items removed)
reli_exhaust <- psych::alpha(df.dichotom[,items_exhaust] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_exhaust$total$raw_alpha ## [1] 0.3512582reli_exhaust$total$`G6(smc)` ## [1] 0.4843456Booklet 4 - Timing belt (no items removed)
reli_timing_belt <- psych::alpha(df.dichotom[,items_timing_belt] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_timing_belt$total$raw_alpha ## [1] 0.5812294reli_timing_belt$total$`G6(smc)` ## [1] 0.6546225Booklet 5 - Steering and front axle (no items removed)
reli_steering <- psych::alpha(df.dichotom[,items_steering] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_steering$total$raw_alpha ## [1] 0.3917055reli_steering$total$`G6(smc)` ## [1] 0.43665293.4 Known-groups validation
We compare the scores of automotive technician apprentices with the scores of electronics technician apprentices. As the test is aimed at measuring car-specific competences, we hypothesise automotive technicians substantially outperform electronics technicians. If no difference is found or electronics technicians achieve higher scores than automotive technicians, this would indicate a lack of validity.
3.4.1 Visual test
We use RDI plots to assess the distribution of Raw data, Density/normality, and Inferential statistics of group differences. These plots combine raw data points (black dots), density functions (coloured beans), group means (horizontal lines), and 95% confidence intervals of the means (boxes).
Booklet 1 - Lighting system
yarrr::pirateplot(formula = sum_lights ~ job_designation ,
data = df.dichotom ,
inf.method = "ci" ,
theme = 1 ,
cap.beans = FALSE)
Fig. 2. Test scores of automotive technician apprentices and electronics technician apprentices for test booklet 1 (lighting system). Plot shows raw data points (dots), density functions (beans), means (thick horizontal lines), and 95% confidence intervals (boxes).
Booklet 2 - Hydraulic brakes
yarrr::pirateplot(formula = sum_brakes ~ job_designation ,
data = df.dichotom ,
inf.method = "ci" ,
theme = 1 ,
cap.beans = FALSE)
Fig. 3. Test scores of automotive technician apprentices and electronics technician apprentices for test booklet 2 (hydraulic brakes). Plot shows raw data points (dots), density functions (beans), means (thick horizontal lines), and 95% confidence intervals (boxes).
Booklet 3 - Exhaust system
yarrr::pirateplot(formula = sum_exhaust ~ job_designation ,
data = df.dichotom ,
inf.method = "ci" ,
theme = 1 ,
cap.beans = FALSE)
Fig. 4. Test scores of automotive technician apprentices and electronics technician apprentices for test booklet 3 (exhaust system). Plot shows raw data points (dots), density functions (beans), means (thick horizontal lines), and 95% confidence intervals (boxes).
Booklet 4 - Timing belt
yarrr::pirateplot(formula = sum_timing_belt ~ job_designation ,
data = df.dichotom ,
inf.method = "ci" ,
theme = 1 ,
cap.beans = FALSE)
Fig. 5. Test scores of automotive technician apprentices and electronics technician apprentices for test booklet 4 (timing belt). Plot shows raw data points (dots), density functions (beans), means (thick horizontal lines), and 95% confidence intervals (boxes).
Booklet 5 - Steering and front axle
yarrr::pirateplot(formula = sum_steering ~ job_designation ,
data = df.dichotom ,
inf.method = "ci" ,
theme = 1 ,
cap.beans = FALSE)
Fig. 6. Test scores of automotive technician apprentices and electronics technician apprentices for test booklet 5 (steering). Plot shows raw data points (dots), density functions (beans), means (thick horizontal lines), and 95% confidence intervals (boxes).
3.4.2 Inferential statistic
We use Welch’s t-test to compare the scores of the two groups for each of the five test booklets. Welch’s t-test corrects for unequal variances and should be used when two groups have unequal sample sizes (Derrick, Toher, & White, 2016), which is the case in this study. It is robust against departures from normality as long as sample sizes are sufficiently large and distributions are reasonably symmetric (West, 2021). Symmetry of the distributions can be visually assessed in the plots in figures 2 to 6. We do not perform statistical analyses for normality as the (non-)significance of the results is commonly misleading (cf. Field, Miles, & Field, 2012, p. 182).
We use Cohen’s d to estimate effect sizes.
Booklet 1 - Lighting system
t.test(df.dichotom$sum_lights ~ df.dichotom$job_designation ,
paired = FALSE ,
alternative = "greater")##
## Welch Two Sample t-test
##
## data: df.dichotom$sum_lights by df.dichotom$job_designation
## t = 6.6578, df = 104.63, p-value = 6.59e-10
## alternative hypothesis: true difference in means between group automotive technician and group electrical/electronics engineer is greater than 0
## 95 percent confidence interval:
## 1.582215 Inf
## sample estimates:
## mean in group automotive technician
## 8.203704
## mean in group electrical/electronics engineer
## 6.096154Effect size:
lsr::cohensD(df.dichotom$sum_lights ~ df.dichotom$job_designation)## [1] 1.107628
Booklet 2 - Hydraulic brakes
t.test(df.dichotom$sum_brakes ~ df.dichotom$job_designation ,
paired = FALSE ,
alternative = "greater")##
## Welch Two Sample t-test
##
## data: df.dichotom$sum_brakes by df.dichotom$job_designation
## t = 2.8692, df = 48.404, p-value = 0.003042
## alternative hypothesis: true difference in means between group automotive technician and group electrical/electronics engineer is greater than 0
## 95 percent confidence interval:
## 0.630991 Inf
## sample estimates:
## mean in group automotive technician
## 7.111111
## mean in group electrical/electronics engineer
## 5.592593Effect size:
lsr::cohensD(df.dichotom$sum_brakes ~ df.dichotom$job_designation)## [1] 0.6092409
Booklet 3 - Exhaust system
t.test(df.dichotom$sum_exhaust ~ df.dichotom$job_designation ,
paired = FALSE ,
alternative = "greater")##
## Welch Two Sample t-test
##
## data: df.dichotom$sum_exhaust by df.dichotom$job_designation
## t = 3.89, df = 39.831, p-value = 0.0001857
## alternative hypothesis: true difference in means between group automotive technician and group electrical/electronics engineer is greater than 0
## 95 percent confidence interval:
## 1.105821 Inf
## sample estimates:
## mean in group automotive technician
## 6.00
## mean in group electrical/electronics engineer
## 4.05Effect size:
lsr::cohensD(df.dichotom$sum_exhaust ~ df.dichotom$job_designation)## [1] 0.9503083
Booklet 4 - Timing belt
t.test(df.dichotom$sum_timing_belt ~ df.dichotom$job_designation ,
paired = FALSE ,
alternative = "greater")##
## Welch Two Sample t-test
##
## data: df.dichotom$sum_timing_belt by df.dichotom$job_designation
## t = 4.2214, df = 30.158, p-value = 0.0001027
## alternative hypothesis: true difference in means between group automotive technician and group electrical/electronics engineer is greater than 0
## 95 percent confidence interval:
## 1.720936 Inf
## sample estimates:
## mean in group automotive technician
## 8.277778
## mean in group electrical/electronics engineer
## 5.400000Effect size:
lsr::cohensD(df.dichotom$sum_timing_belt ~ df.dichotom$job_designation)## [1] 1.07311
Booklet 5 - Steering and front axle
t.test(df.dichotom$sum_steering ~ df.dichotom$job_designation ,
paired = FALSE ,
alternative = "greater")##
## Welch Two Sample t-test
##
## data: df.dichotom$sum_steering by df.dichotom$job_designation
## t = 3.7511, df = 21.874, p-value = 0.0005565
## alternative hypothesis: true difference in means between group automotive technician and group electrical/electronics engineer is greater than 0
## 95 percent confidence interval:
## 0.9575095 Inf
## sample estimates:
## mean in group automotive technician
## 5.480519
## mean in group electrical/electronics engineer
## 3.714286Effect size:
lsr::cohensD(df.dichotom$sum_steering ~ df.dichotom$job_designation)## [1] 0.90891483.4.3 Discussion
We investigated the validity of the test score interpretations from a computer-based assessment aiming at the repair-related competences of automotive technician apprentices. Using a known-groups validation design (Cronbach & Meehl, 1955), we compared the test scores of automotive technician apprentices with the scores of electronics technician apprentices. As the test is aimed at measuring car-specific competences, we hypothesise automotive technicians substantially outperform electronics technicians.
Results confirm this hypothesis for all five test booklets. The test scores of the automotive technician group are significantly higher than the scores of the electronics technician group. The effect sizes are medium (> .50) for the ‘hydraulic brakes’ booklet and large (> .80) for the other booklets. This finding suggests that the chances of solving the test items are substantially higher for people who are trained in diagnosing, maintaining, and repairing cars than for people who are not. As the two groups are known to be rather similar except for the focus of their professional training, it is unlikely the test score differences are based on generic skills such as intelligence, problem solving, or reading ability (cf. Nickolaus, Knöll, & Gschwendtner, 2006; Gschwendtner, Abele, & Nickolaus, 2009).
To judge validity, it is recommended to evaluate evidence from various sources (AERA, APA, & NCME, 2014). In our project, the validity evidence from this known-groups study is complemented by evidence from a hands-on validation study (Hartmann, Güzel, & Gschwendtner, 2023b). Setting the findings of the two studies into relation, we found solid evidence supporting the validity assumption. But we were also able to identify limitations of our video-based test (Hartmann, Güzel, & Gschwendtner, 2023a,b). We advise potential test users to inform themselves about these limitations before using the instrument.
4 References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME] (2014). Standards for Educational and Psychological Testing. Washington, D.C.: American Educational Research Association.
Cronbach, L. J.; Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.
Derrick, B., Toher, D., & White, P. (2016). Why Welch’s test is Type I error robust. Quantitative Methods for Psychology, 12(1), 30-38. https://uwe-repository.worktribe.com/output/915692
Field, A., Miles, J., & Field, Z. (2012). Discovering statistics using R. London: Sage.
Gschwendtner, T., Abele, S., & Nickolaus, R. (2009). Computersimulierte Arbeitsproben: Eine Validierungsstudie am Beispiel der Fehlerdiagnoseleistungen von Kfz-Mechatronikern. Zeitschrift für Berufs- und Wirtschaftspädagogik, 105(4), 557–578. https://doi.org/10.25162/zbw-2009-0038
Hartmann, S., Güzel, E., & Gschwendtner, T. (2023a). Berufsbezogene Fähigkeiten digital messen: Herausforderungen, Möglichkeiten und Grenzen am Beispiel videovignettenbasierter Prüfungsaufgaben für das Kfz-Handwerk. Berufsbildung in Wissenschaft und Praxis, 52, 26-30. https://nbn-resolving.org/urn:nbn:de:0035-bwp-23326-2
Hartmann, S., Güzel, E., & Gschwendtner, T. (2023b). Digital measurement of hands-on performance? Ecological validation of a computer-based assessment of automotive repair skills. Empirical Research in Vocational Education and Training, 15(15). https://doi.org/10.1186/s40461-023-00153-x
Nickolaus, R., Knöll, B., & Gschwendtner, T. (2006). Methodische Präferenzen und ihre Effekte auf die Kompetenz- und Motivationsentwicklung: Ergebnisse aus Studien in anforderungsdifferenten elektrotechnischen Ausbildungsberufen in der Grundbildung. Zeitschrift für Berufs- und Wirtschaftspädagogik, 102(4), 552–577.
West, R. M. (2021). Best practice in statistics: Use the Welch t-test when testing the difference between two groups. Annals of Clinical Biochemistry: International Journal of Laboratory Medicine, 5(4), 267-269. https://doi.org/10.1177/000456322199208