DigiDIn-Kfz - Validation study 2
Documentation and R code
Stefan Hartmann
2024-09-20
1 About this document
1.1 General information
- Project name: DigiDIn-Kfz
- Funding agent: Federal Ministry of Education and Research (Germany)
- Grant No.: 21AP004A
- Funding period: 2020-2023
- Subproject: B - Video-based assessment
- Study: Validation study 2
- Date of data collection: November 2021
- Author: Stefan Hartmann, Pädagogische Hochschule Ludwigsburg
1.2 License
This documentation and the related CSV data file are available under the terms of the Creative Commons Attribution-ShareAlike 4.0 license. They may be copied, shared, transformed, and build upon under the conditions described here: https://creativecommons.org/licenses/by-sa/4.0/
2 About this study
2.1 Synopsis
We investigated the validity of the test score interpretations from a computer-based assessment (CBA). The CBA utilises scripted video vignettes to measure repair-related competences of automotive technician apprentices. Focusing on ecological validity (Gouvier, Barker, & Musso, 2010), we hypothesised the ability to correctly judge repair actions shown in the videos of the CBA is a valid predictor of the ability to perform such repair actions on real cars in a hands-on scenario. Our theoretical approach, method, and preliminary findings are published in German (Hartmann, Güzel, & Gschwendtner, 2023a) as well as in English (Hartmann, Güzel, & Gschwendtner, 2023b).
Note. After completing data input and carefully revising the analysis procedures, the correlations between CBA and hands-on test scores presented here somewhat differ from the numbers reported in the publications; however, the overall findings remain unchanged, indicating that evidence for ecological validity is generally weak (see chapter ‘Discussion’ for details).
2.2 Research design
Our instrument is a computerised fixed test (CFT) which uses scripted video vignettes. The intended purpose of the test is to be used in the final exams (“Gesellenprüfungen”) for automotive technicians in the German VET system. The test originally consisted of five independent test booklets, each of which refers to a different car system: brakes, exhaust, lighting, steering, and timing belt. One of the booklets (exhaust) was removed from the instrument prior to this validation study after it was judged by experts to be of limited relevance for exams.
In each computer-based test booklet, a series of short video clips showing a technician performing repairs on a car is presented on the computer screen. Each video clip is followed by (at least) one selected-response item which requires test takers to judge two short statements about the repair actions shown in the video (fig. 1). The test is in German.
Figure 1. English translation
of an example item from the computerised fixed test. First, the video
clip is played back automatically. After playback, the item is displayed
on the computer screen.
We use a highly standardised “all possible options” (APO) item design in which two statements marked “Option A” and “Option B” about each video clip are presented. Test takers must then mark one out of four possible answers:
Only option A is correct.
Only option B is correct.
Both options, A and B are correct.
Neither option A nor option B is correct.
Correct responses are credited with a score of 1 point (full credit) whereas incorrect responses are scored 0 points (no credit). The sum of correct responses of a booklet reflects a test taker’s ability to correctly judge the actions shown in the videos, thus representing their procedural knowledge about the repairs on the car system the test booklet refers to. Hypothetically, this procedural knowledge should be closely related to the ability of practically performing such repairs.
To investigate this hypothesis, we designed a performance assessment requiring automotive technicians to carry out real repairs on the car systems used in the video-based CBA in a realistic workshop scenario (fig. 2).
Figure 2. The four hands-on
stations used in the performance assessment: (1) Electric lighting, (2)
brakes, (3) steering, (4) timing belt.
(Image credit: Emre Güzel | https://creativecommons.org/licenses/by-sa/4.0/)
At each station, a participants’ hands-on performance was rated by an expert on a highly standardised protocol (fig. 3). The protocols contained comprehensive lists of anticipated actions for each repair step, and the experts marked if an action was performed (“yes”) or not performed (“no”). Each adequate and necessary action was scored one credit (1 point), whereas actions inadeqate or unnecessary for the repair in question were scored zero credits (0 points). For each station individually, the sum of adequate and necessary actions reflects the test takers’ ability to perform the repair. The protocols were for the experts’ eyes only and were not revealed to the participants.
Figure 3. Protocol page from
the hands-on performance assessment “brakes” (in German). At
the end of each line, an expert marked if the described action was
performed (Ja/yes) or not performed (Nein/no) by the participant. In
addition to these pre-defined actions, the experts were given room to
note additional actions not covered by the list. Green checkboxes (not
highlighted on the original protocol) represent correct actions for this
task, which were scored 1 credit each. In this particular case, the
person scored three out of eight credits on this task.
2.3 Sample and Design
The sample of the hands-on validation study consisted of 88 automotive technician apprentices from Southwest Germany at the third year of formal training. We used a balanced incomplete block design (fig. 4), meaning that each participant had to work a selection of two hands-on stations and two computer-based test booklets instead of all four.
To increase the number of apprentices tested at a time, we used identical parallel versions of the stations (labelled “1” and “2” in fig. 4). The design was layed out to test 24 participants per day on five consecutive days, allowing for a total of 120 participants. However, not all slots were filled with participants. A total of 91 apprentices were tested, with 88 of them completing their hands-on stations and the following computer-based tests.
Figure 4. Design of the
hands-on validation study. Each unique code (and respective colour)
represents one participant. The design was layed out to test 24
apprentices. The same design was used on five consecutive days, allowing
for a total of 120 participants to be tested.
3 Analyses in R
Note. All code was tested using R version 4.3.2.
3.1 Preparation
3.1.1 Increase the maximum length of R console output
(this is optional)
options(max.print = 5000)3.1.2 Set working directory
Linux:
setwd("/home/username/path/to/directory")macOS:
setwd("/Users/username/path/to/directory")Windows:
setwd("C:/path/to/directory")Note. Make sure to use forward slashes (/), not backward slashes (\) which you will get if you copy file paths in Windows.
3.1.3 Install required R packages
This is only necessary once, and only if these packages are not yet installed on your computer.
Package ‘car’ (Fox at al., 2023)
install.packages("car")
Package ‘psych’ (Revelle, 2023)
install.packages("psych")
Package ‘CTT’ (Willse, 2018)
install.packages("CTT")
Package ‘ggplot2’ (Wickham et al., 2024)
install.packages("ggplot2")3.1.4 Import CSV data
df <- read.csv(file = "df.csv" ,
header = TRUE ,
sep = "," ,
dec = "." ,
fileEncoding = "UTF8")3.1.5 Display a list of variables
The first variable, ‘ID’, is an anonymous unique identifier for each participant. It consists of six characters: The first two indicate the day of participation (in German); the last four indicate the hands-on stations and CBA test booklets the persons have worked on according to the design plan in Figure 4.
The variables ‘type_of_company’ and ‘school_degree’ contain demographic information.
The remaining variables are test items. Their variable names indicate the CBA test booklet / hands-on station:
cba_lights = CBA booklet 1 (electric lighting)
cba_brakes = CBA booklet 2 (hydraulic brakes)
cba_timing belt = CBA booklet 3 (engine timing belt)
cba_steering = CBA booklet 4 (steering and front axle)
handson_lights = hands-on protocol 1 (electric lighting)
handson_brakes = hands-on protocol 2 (hydraulic brakes)
handson_timing belt = hands-on protocol 3 (engine timing belt)
handson_steering = hands-on protocol 4 (steering and front axle)
Note. A fifth CBA booklet (‘exhaust’) from an earlier version of the instrument was not used in this study. Item numbers start at 05 in each CBA booklet because items 01 to 04 contain an introduction to the instrument and an example which is not used for scoring. The following items are intentionally missing because they were added to the instrument at a later time:
cba_lights_09, cba_lights_14, cba_lights_24
cba_brakes_06, cba_brakes_17
cba_timing_belt_05, cba_timing_belt_15, cba_timing_belt_24
cba_steering_11, cba_steering_13, cba_steering_14, cba_steering_19, cba_steering_22, cba_steering_23, cba_steering_24
Note. Two items with the suffix ’_old’ (‘cba_brakes_05_old’ and ‘cba_timing_belt_14_old’) indicate that the versions used in this study are now outdated. In later studies, revised versions of these items were used (‘cba_brakes_05_new’ and ‘cba_timing_belt_14_new’).
colnames(df)## [1] "ID" "type_of_company"
## [3] "school_degree" "cba_lights_05"
## [5] "cba_lights_06" "cba_lights_07"
## [7] "cba_lights_08" "cba_lights_10"
## [9] "cba_lights_11" "cba_lights_12"
## [11] "cba_lights_13" "cba_lights_15"
## [13] "cba_lights_16" "cba_lights_17"
## [15] "cba_lights_18" "cba_lights_19"
## [17] "cba_lights_20" "cba_lights_21"
## [19] "cba_lights_22" "cba_lights_23"
## [21] "cba_brakes_05_old" "cba_brakes_07"
## [23] "cba_brakes_08" "cba_brakes_09"
## [25] "cba_brakes_10" "cba_brakes_11"
## [27] "cba_brakes_12" "cba_brakes_13"
## [29] "cba_brakes_14" "cba_brakes_15"
## [31] "cba_brakes_16" "cba_brakes_18"
## [33] "cba_brakes_19" "cba_brakes_20"
## [35] "cba_brakes_21" "cba_brakes_22"
## [37] "cba_brakes_23" "cba_brakes_24"
## [39] "cba_steering_05" "cba_steering_06"
## [41] "cba_steering_07" "cba_steering_08"
## [43] "cba_steering_09" "cba_steering_10"
## [45] "cba_steering_12" "cba_steering_15"
## [47] "cba_steering_16" "cba_steering_17"
## [49] "cba_steering_18" "cba_steering_20"
## [51] "cba_steering_21" "cba_timing_belt_06"
## [53] "cba_timing_belt_07" "cba_timing_belt_08"
## [55] "cba_timing_belt_09" "cba_timing_belt_10"
## [57] "cba_timing_belt_11" "cba_timing_belt_12"
## [59] "cba_timing_belt_13" "cba_timing_belt_14_old"
## [61] "cba_timing_belt_16" "cba_timing_belt_17"
## [63] "cba_timing_belt_18" "cba_timing_belt_19"
## [65] "cba_timing_belt_20" "cba_timing_belt_21"
## [67] "cba_timing_belt_22" "cba_timing_belt_23"
## [69] "handson_lights_09a" "handson_lights_09b"
## [71] "handson_lights_09c" "handson_lights_10_11_16a"
## [73] "handson_lights_10_11_16b" "handson_lights_10_11_16c"
## [75] "handson_lights_10_11_16d" "handson_lights_10_11_16e"
## [77] "handson_lights_10_11_16f" "handson_lights_10_11_16g"
## [79] "handson_lights_10_11_16h" "handson_lights_12a"
## [81] "handson_lights_12b" "handson_lights_12c"
## [83] "handson_lights_12d" "handson_lights_12e"
## [85] "handson_lights_12f" "handson_lights_13_14_15a"
## [87] "handson_lights_13_14_15b" "handson_lights_13_14_15c"
## [89] "handson_lights_13_14_15d" "handson_lights_13_14_15e"
## [91] "handson_lights_13_14_15f" "handson_lights_13_14_15g"
## [93] "handson_lights_13_14_15h" "handson_lights_13_14_15i"
## [95] "handson_lights_13_14_15k" "handson_brakes_06a"
## [97] "handson_brakes_06b" "handson_brakes_06c"
## [99] "handson_brakes_06d" "handson_brakes_06e"
## [101] "handson_brakes_06f" "handson_brakes_06g"
## [103] "handson_brakes_06h" "handson_brakes_06i"
## [105] "handson_brakes_06k" "handson_brakes_06l"
## [107] "handson_brakes_08a" "handson_brakes_08b"
## [109] "handson_brakes_08c" "handson_brakes_08d"
## [111] "handson_brakes_08e" "handson_brakes_08f"
## [113] "handson_brakes_08g" "handson_brakes_08h"
## [115] "handson_brakes_08i" "handson_brakes_09_10a"
## [117] "handson_brakes_09_10b" "handson_brakes_09_10c"
## [119] "handson_brakes_09_10d" "handson_brakes_09_10e"
## [121] "handson_brakes_09_10f" "handson_brakes_09_10g"
## [123] "handson_brakes_09_10h" "handson_brakes_09_10i"
## [125] "handson_brakes_16a" "handson_brakes_16b"
## [127] "handson_brakes_16c" "handson_brakes_16d"
## [129] "handson_brakes_16e" "handson_brakes_16f"
## [131] "handson_brakes_16g" "handson_brakes_16h"
## [133] "handson_brakes_16i" "handson_brakes_17_21a"
## [135] "handson_brakes_17_21b" "handson_brakes_17_21c"
## [137] "handson_brakes_17_21d" "handson_brakes_17_21e"
## [139] "handson_brakes_17_21f" "handson_brakes_17_21g"
## [141] "handson_brakes_17_21h" "handson_brakes_17_21i"
## [143] "handson_steering_07a" "handson_steering_07b"
## [145] "handson_steering_07c" "handson_steering_07d"
## [147] "handson_steering_07e" "handson_steering_07f"
## [149] "handson_steering_07g" "handson_steering_07h"
## [151] "handson_steering_07i" "handson_steering_07k"
## [153] "handson_steering_07l" "handson_steering_08a"
## [155] "handson_steering_08b" "handson_steering_08c"
## [157] "handson_steering_08d" "handson_steering_08e"
## [159] "handson_steering_08f" "handson_steering_09a"
## [161] "handson_steering_09b" "handson_steering_09c"
## [163] "handson_steering_09d" "handson_steering_09e"
## [165] "handson_steering_09f" "handson_steering_11a"
## [167] "handson_steering_11b" "handson_steering_11c"
## [169] "handson_steering_11d" "handson_steering_11e"
## [171] "handson_steering_11f" "handson_steering_11g"
## [173] "handson_steering_11h" "handson_steering_11i"
## [175] "handson_steering_11k" "handson_steering_11l"
## [177] "handson_steering_11m" "handson_steering_11n"
## [179] "handson_steering_11o" "handson_steering_15a"
## [181] "handson_steering_15b" "handson_steering_15c"
## [183] "handson_steering_15d" "handson_steering_15e"
## [185] "handson_timing_belt_10a" "handson_timing_belt_10b"
## [187] "handson_timing_belt_10c" "handson_timing_belt_10d"
## [189] "handson_timing_belt_10e" "handson_timing_belt_10f"
## [191] "handson_timing_belt_10g" "handson_timing_belt_10h"
## [193] "handson_timing_belt_10i" "handson_timing_belt_10k"
## [195] "handson_timing_belt_10l" "handson_timing_belt_11_12a"
## [197] "handson_timing_belt_11_12b" "handson_timing_belt_11_12c"
## [199] "handson_timing_belt_11_12d" "handson_timing_belt_11_12e"
## [201] "handson_timing_belt_11_12f" "handson_timing_belt_11_12g"
## [203] "handson_timing_belt_11_12h" "handson_timing_belt_11_12i"
## [205] "handson_timing_belt_11_12k" "handson_timing_belt_11_12l"
## [207] "handson_timing_belt_14_15a" "handson_timing_belt_14_15b"
## [209] "handson_timing_belt_14_15c" "handson_timing_belt_14_15d"
## [211] "handson_timing_belt_14_15e" "handson_timing_belt_14_15f"
## [213] "handson_timing_belt_14_15g" "handson_timing_belt_14_15h"
## [215] "handson_timing_belt_14_15i" "handson_timing_belt_14_15k"
## [217] "handson_timing_belt_14_15l" "handson_timing_belt_14_15m"
## [219] "handson_timing_belt_14_15n" "handson_timing_belt_14_15o"
## [221] "handson_timing_belt_16a" "handson_timing_belt_16b"
## [223] "handson_timing_belt_16c" "handson_timing_belt_16d"
## [225] "handson_timing_belt_16e" "handson_timing_belt_16f"
## [227] "handson_timing_belt_16g" "handson_timing_belt_16h"
## [229] "handson_timing_belt_16i" "handson_timing_belt_16k"3.1.6 Frequency analysis: Demographic data
3.1.6.1 Type of company
In the German dual VET system, apprenticeship programs combine classroom-based learning in a vocational school with workplace learning in a company. This variable indicates the type of training company, differentiating dealerships (workshops affiliated with a manufacturer such as Volkswagen or Bosch) and independent workshops (workshops that service and maintain products of many different brands).
table(df$type_of_company , exclude = FALSE)##
## dealership independent workshop missing by intention
## 64 22 23.1.6.2 School degree
Highest school degree of the apprentice, according to the German school system:
lower secondary (Hauptschule) - degree of a school offering ISCED Level 2 education, usually obtained at the end of the 9th or 10th grade
higher secondary (Mittlere Reife) - degree of a school offering ISCED Level 2 education but ranked higher than Hauptschule, usually obtained at the end of the 10th grade
A-levels/high school (Abitur) - high school degree according to ISCED Level 3, usually obtained at the end of the 12th or 13th grade
table(df$school_degree , exclude = FALSE)##
## A-levels/high school (Abitur) higher secondary (Mittlere Reife)
## 22 45
## lower secondary (Hauptschule) other
## 19 23.2 Computer-based test
3.2.1 Frequency analysis: Response data and missing data
Response values for each item are ‘1’, ‘2’, ‘3’, and ‘4’, referring to the four response options of the single-best answer items (see chapter 2.2 and Figure 1). A value of ‘9’ indicates an item that was deliberately skipped (missing by intention). Items not presented to a participant by design appear as ‘NA’ (missing by design).
apply(df[c(4:68)] , 2 , table , exclude = FALSE) ## $cba_lights_05
##
## 1 2 3 4 <NA>
## 2 8 16 21 41
##
## $cba_lights_06
##
## 1 2 3 <NA>
## 2 41 4 41
##
## $cba_lights_07
##
## 1 2 3 4 <NA>
## 7 16 20 4 41
##
## $cba_lights_08
##
## 1 2 3 4 <NA>
## 35 2 9 1 41
##
## $cba_lights_10
##
## 1 2 3 4 <NA>
## 10 28 1 8 41
##
## $cba_lights_11
##
## 1 2 3 4 <NA>
## 1 13 31 2 41
##
## $cba_lights_12
##
## 1 2 3 4 <NA>
## 12 6 22 7 41
##
## $cba_lights_13
##
## 1 2 3 4 <NA>
## 16 3 11 17 41
##
## $cba_lights_15
##
## 1 2 3 4 <NA>
## 31 2 10 4 41
##
## $cba_lights_16
##
## 1 2 3 4 <NA>
## 5 22 16 4 41
##
## $cba_lights_17
##
## 1 2 3 4 <NA>
## 12 10 24 1 41
##
## $cba_lights_18
##
## 1 2 3 4 <NA>
## 8 17 14 8 41
##
## $cba_lights_19
##
## 1 2 3 4 <NA>
## 17 7 20 3 41
##
## $cba_lights_20
##
## 1 2 3 4 <NA>
## 17 9 20 1 41
##
## $cba_lights_21
##
## 1 2 3 <NA>
## 3 2 42 41
##
## $cba_lights_22
##
## 1 2 3 <NA>
## 2 4 41 41
##
## $cba_lights_23
##
## 2 3 9 <NA>
## 4 42 1 41
##
## $cba_brakes_05_old
##
## 1 2 3 4 <NA>
## 10 6 9 15 48
##
## $cba_brakes_07
##
## 1 2 3 4 <NA>
## 1 15 22 2 48
##
## $cba_brakes_08
##
## 1 2 3 4 <NA>
## 14 9 8 9 48
##
## $cba_brakes_09
##
## 1 3 4 <NA>
## 34 5 1 48
##
## $cba_brakes_10
##
## 1 2 3 4 <NA>
## 12 12 13 3 48
##
## $cba_brakes_11
##
## 1 2 3 4 <NA>
## 11 2 3 24 48
##
## $cba_brakes_12
##
## 1 2 3 4 <NA>
## 12 10 17 1 48
##
## $cba_brakes_13
##
## 1 2 3 4 <NA>
## 6 10 22 2 48
##
## $cba_brakes_14
##
## 1 2 3 4 <NA>
## 8 18 12 2 48
##
## $cba_brakes_15
##
## 1 2 3 4 <NA>
## 20 4 15 1 48
##
## $cba_brakes_16
##
## 1 2 3 4 <NA>
## 20 1 18 1 48
##
## $cba_brakes_18
##
## 1 2 3 <NA>
## 11 1 28 48
##
## $cba_brakes_19
##
## 1 2 3 4 <NA>
## 5 17 5 13 48
##
## $cba_brakes_20
##
## 2 3 4 <NA>
## 18 1 21 48
##
## $cba_brakes_21
##
## 1 2 3 <NA>
## 14 17 9 48
##
## $cba_brakes_22
##
## 1 2 3 <NA>
## 2 12 26 48
##
## $cba_brakes_23
##
## 1 2 3 <NA>
## 5 10 25 48
##
## $cba_brakes_24
##
## 1 2 3 <NA>
## 37 1 2 48
##
## $cba_steering_05
##
## 1 2 3 4 <NA>
## 3 33 6 6 40
##
## $cba_steering_06
##
## 1 2 3 4 <NA>
## 13 14 19 2 40
##
## $cba_steering_07
##
## 1 2 4 <NA>
## 9 24 15 40
##
## $cba_steering_08
##
## 1 2 3 4 <NA>
## 38 1 8 1 40
##
## $cba_steering_09
##
## 1 2 3 <NA>
## 33 5 10 40
##
## $cba_steering_10
##
## 1 2 3 <NA>
## 4 24 20 40
##
## $cba_steering_12
##
## 1 2 3 4 <NA>
## 10 10 11 17 40
##
## $cba_steering_15
##
## 1 2 3 <NA>
## 17 1 30 40
##
## $cba_steering_16
##
## 1 2 3 4 <NA>
## 34 3 10 1 40
##
## $cba_steering_17
##
## 1 2 3 4 <NA>
## 4 23 4 17 40
##
## $cba_steering_18
##
## 1 3 <NA>
## 42 6 40
##
## $cba_steering_20
##
## 2 3 <NA>
## 28 20 40
##
## $cba_steering_21
##
## 1 2 3 4 <NA>
## 12 11 13 12 40
##
## $cba_timing_belt_06
##
## 1 2 3 4 <NA>
## 11 7 21 1 48
##
## $cba_timing_belt_07
##
## 1 2 3 4 <NA>
## 7 12 20 1 48
##
## $cba_timing_belt_08
##
## 1 2 3 4 <NA>
## 3 18 8 11 48
##
## $cba_timing_belt_09
##
## 1 2 3 4 <NA>
## 28 5 6 1 48
##
## $cba_timing_belt_10
##
## 1 2 3 <NA>
## 30 1 9 48
##
## $cba_timing_belt_11
##
## 1 2 3 4 <NA>
## 15 4 20 1 48
##
## $cba_timing_belt_12
##
## 1 2 3 4 <NA>
## 6 20 7 7 48
##
## $cba_timing_belt_13
##
## 1 2 3 4 <NA>
## 8 11 17 4 48
##
## $cba_timing_belt_14_old
##
## 1 2 3 4 9 <NA>
## 11 23 1 3 2 48
##
## $cba_timing_belt_16
##
## 1 2 3 4 <NA>
## 6 2 3 29 48
##
## $cba_timing_belt_17
##
## 1 2 3 4 <NA>
## 2 12 25 1 48
##
## $cba_timing_belt_18
##
## 1 2 3 4 9 <NA>
## 6 11 2 20 1 48
##
## $cba_timing_belt_19
##
## 1 2 3 4 <NA>
## 7 8 8 17 48
##
## $cba_timing_belt_20
##
## 1 2 3 4 <NA>
## 18 4 15 3 48
##
## $cba_timing_belt_21
##
## 2 3 <NA>
## 15 25 48
##
## $cba_timing_belt_22
##
## 1 2 3 <NA>
## 2 27 11 48
##
## $cba_timing_belt_23
##
## 1 3 <NA>
## 8 32 483.2.2 Item separation and distractor analysis
To judge how well the test items separate between participants with different skill levels, we calculate classical item separation parameters (item-rest correlations) for the items of the four test booklets.
It is expected that for each item, the correct response option (1, 2, 3, or 4) correlates higher with the rest of the booklet than the three incorrect response options (i.e., correct responses to an item are statistically related to the sum of correct responses to all other items of the same test booklet).
As each test booklet addresses a wide range of skills and competences, item separation is expected to be lower than for most psychological tests such as intelligence tests or personality tests.
3.2.2.1 Preparation: Extract items for each test booklet
We can use grep to extract the item designations for
each test booklet.
Booklet 1 - Lighting system
items_lights <- colnames(df)[grep("cba_lights_" , colnames(df))]
items_lights## [1] "cba_lights_05" "cba_lights_06" "cba_lights_07" "cba_lights_08"
## [5] "cba_lights_10" "cba_lights_11" "cba_lights_12" "cba_lights_13"
## [9] "cba_lights_15" "cba_lights_16" "cba_lights_17" "cba_lights_18"
## [13] "cba_lights_19" "cba_lights_20" "cba_lights_21" "cba_lights_22"
## [17] "cba_lights_23"Booklet 2 - Hydraulic brakes
items_brakes <- colnames(df)[grep("cba_brakes_" , colnames(df))]
items_brakes## [1] "cba_brakes_05_old" "cba_brakes_07" "cba_brakes_08"
## [4] "cba_brakes_09" "cba_brakes_10" "cba_brakes_11"
## [7] "cba_brakes_12" "cba_brakes_13" "cba_brakes_14"
## [10] "cba_brakes_15" "cba_brakes_16" "cba_brakes_18"
## [13] "cba_brakes_19" "cba_brakes_20" "cba_brakes_21"
## [16] "cba_brakes_22" "cba_brakes_23" "cba_brakes_24"Booklet 3 - Timing belt
items_timing_belt <- colnames(df)[grep("cba_timing_belt_" , colnames(df))]
items_timing_belt## [1] "cba_timing_belt_06" "cba_timing_belt_07" "cba_timing_belt_08"
## [4] "cba_timing_belt_09" "cba_timing_belt_10" "cba_timing_belt_11"
## [7] "cba_timing_belt_12" "cba_timing_belt_13" "cba_timing_belt_14_old"
## [10] "cba_timing_belt_16" "cba_timing_belt_17" "cba_timing_belt_18"
## [13] "cba_timing_belt_19" "cba_timing_belt_20" "cba_timing_belt_21"
## [16] "cba_timing_belt_22" "cba_timing_belt_23"Booklet 4 - Steering and front axle
items_steering <- colnames(df)[grep("cba_steering_" , colnames(df))]
items_steering## [1] "cba_steering_05" "cba_steering_06" "cba_steering_07" "cba_steering_08"
## [5] "cba_steering_09" "cba_steering_10" "cba_steering_12" "cba_steering_15"
## [9] "cba_steering_16" "cba_steering_17" "cba_steering_18" "cba_steering_20"
## [13] "cba_steering_21"
3.2.2.2 Add keys
Add information which of the four response options (1, 2, 3, or 4) is correct for each item.
Booklet 1 - Lighting system
keys_lights <- c("4" , "2" , "1" , "1" , "4" , "3" , "3" , "1" , "1" , "4" , "1" , "1" , "4" , "1" , "3" , "3" , "3")Booklet 2 - Hydraulic brakes
keys_brakes <- c("2" , "2" , "4" , "1" , "3" , "4" , "3" , "1" , "3" , "1" , "1" , "1" , "4" , "2" , "1" , "3" , "2" , "1" )Booklet 3 - Timing belt
keys_timing_belt <- c("3" , "2" , "2" , "1" , "1" , "1" , "2" , "3" , "2" , "4" , "3" , "4" , "1" , "1" , "2" , "2" , "3")Booklet 4 - Steering and front axle
keys_steering <- c("2" , "4" , "4" , "1" , "1" , "2" , "4" , "1" , "1" , "2" , "1" , "2" , "4")
3.2.2.3 Perform analysis
An asterisk * indicates the correct response option (1,
2, 3, or 4). A key of “9” indicates that no option was marked (item was
intentionally skipped). Item discrimination parameters (item-rest
correlations) are displayed in the column labelled
discrim.
Booklet 1 - Lighting system
CTT::distractorAnalysis(na.omit(df[, items_lights]) ,
keys_lights ,
nGroups = 3 ,
digits = 2)## $cba_lights_05
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.04 -0.07 0.00 0.00 0.10 0.00
## 2 2 8 0.17 -0.45 -0.22 0.22 0.19 0.00
## 3 3 16 0.34 -0.06 0.15 0.22 0.43 0.38
## 4 * 4 21 0.45 -0.31 0.07 0.56 0.29 0.62
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.04 -0.19 -0.06 0.06 0.05 0
## 2 * 2 41 0.87 0.22 0.28 0.72 0.95 1
## 3 3 4 0.09 -0.55 -0.22 0.22 0.00 0
## 4 4 0 0.00 NA 0.00 0.00 0.00 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0
##
## $cba_lights_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 7 0.15 -0.27 0.03 0.22 0.05 0.25
## 2 2 16 0.34 -0.31 0.03 0.22 0.48 0.25
## 3 3 20 0.43 -0.26 -0.12 0.50 0.38 0.38
## 4 4 4 0.09 -0.05 0.07 0.06 0.10 0.12
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 35 0.74 0.14 0.39 0.61 0.76 1
## 2 2 2 0.04 -0.13 -0.06 0.06 0.05 0
## 3 3 9 0.19 -0.59 -0.33 0.33 0.14 0
## 4 4 1 0.02 -0.10 0.00 0.00 0.05 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0
##
## $cba_lights_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 10 0.21 -0.15 0.28 0.22 0.10 0.50
## 2 2 28 0.60 -0.48 -0.43 0.56 0.81 0.12
## 3 3 1 0.02 -0.18 -0.06 0.06 0.00 0.00
## 4 * 4 8 0.17 0.02 0.21 0.17 0.10 0.38
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 1 0.02 -0.18 -0.06 0.06 0.00 0
## 2 2 13 0.28 -0.49 -0.44 0.44 0.24 0
## 3 * 3 31 0.66 0.12 0.61 0.39 0.76 1
## 4 4 2 0.04 -0.37 -0.11 0.11 0.00 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0
##
## $cba_lights_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.26 -0.18 0.10 0.28 0.19 0.38
## 2 2 6 0.13 -0.43 -0.15 0.28 0.00 0.12
## 3 * 3 22 0.47 -0.05 0.28 0.22 0.67 0.50
## 4 4 7 0.15 -0.39 -0.22 0.22 0.14 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 16 0.34 0.08 0.58 0.17 0.33 0.75
## 2 2 3 0.06 -0.26 -0.11 0.11 0.05 0.00
## 3 3 11 0.23 -0.33 -0.15 0.28 0.24 0.12
## 4 4 17 0.36 -0.45 -0.32 0.44 0.38 0.12
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 31 0.66 0.12 0.38 0.50 0.71 0.88
## 2 2 2 0.04 -0.49 -0.11 0.11 0.00 0.00
## 3 3 10 0.21 -0.32 -0.10 0.22 0.24 0.12
## 4 4 4 0.09 -0.39 -0.17 0.17 0.05 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 5 0.11 -0.32 0.01 0.11 0.10 0.12
## 2 2 22 0.47 -0.08 0.18 0.44 0.43 0.62
## 3 3 16 0.34 -0.49 -0.32 0.44 0.33 0.12
## 4 * 4 4 0.09 0.04 0.12 0.00 0.14 0.12
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 12 0.26 -0.01 0.33 0.17 0.24 0.50
## 2 2 10 0.21 -0.21 0.08 0.17 0.24 0.25
## 3 3 24 0.51 -0.41 -0.36 0.61 0.52 0.25
## 4 4 1 0.02 -0.43 -0.06 0.06 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 8 0.17 0.05 0.19 0.06 0.24 0.25
## 2 2 17 0.36 -0.45 -0.32 0.44 0.38 0.12
## 3 3 14 0.30 -0.28 0.04 0.33 0.24 0.38
## 4 4 8 0.17 -0.22 0.08 0.17 0.14 0.25
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 17 0.36 -0.10 0.28 0.22 0.43 0.5
## 2 2 7 0.15 -0.27 -0.17 0.17 0.19 0.0
## 3 3 20 0.43 -0.29 0.00 0.50 0.33 0.5
## 4 * 4 3 0.06 -0.40 -0.11 0.11 0.05 0.0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.0
##
## $cba_lights_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 17 0.36 -0.07 0.28 0.22 0.43 0.50
## 2 2 9 0.19 -0.40 -0.15 0.28 0.14 0.12
## 3 3 20 0.43 -0.24 -0.07 0.44 0.43 0.38
## 4 4 1 0.02 -0.43 -0.06 0.06 0.00 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_lights_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 3 0.06 -0.40 -0.11 0.11 0.05 0
## 2 2 2 0.04 -0.26 -0.11 0.11 0.00 0
## 3 * 3 42 0.89 0.13 0.22 0.78 0.95 1
## 4 4 0 0.00 NA 0.00 0.00 0.00 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0
##
## $cba_lights_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.04 -0.13 -0.06 0.06 0.05 0
## 2 2 4 0.09 -0.47 -0.11 0.11 0.10 0
## 3 * 3 41 0.87 0.10 0.17 0.83 0.86 1
## 4 4 0 0.00 NA 0.00 0.00 0.00 0
## 9 9 0 0.00 NA 0.00 0.00 0.00 0
##
## $cba_lights_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 0 0.00 NA 0.00 0.00 0 0
## 2 2 4 0.09 -0.55 -0.22 0.22 0 0
## 3 * 3 42 0.89 0.27 0.28 0.72 1 1
## 4 4 0 0.00 NA 0.00 0.00 0 0
## 9 9 1 0.02 -0.18 -0.06 0.06 0 0Booklet 2 - Hydraulic brakes
CTT::distractorAnalysis(na.omit(df[, items_brakes]) ,
keys_brakes ,
nGroups = 3 ,
digits = 2)## $cba_brakes_05_old
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 10 0.25 -0.20 -0.03 0.26 0.25 0.23
## 2 * 2 6 0.15 0.00 0.02 0.13 0.25 0.15
## 3 3 9 0.22 -0.43 -0.15 0.30 0.00 0.15
## 4 4 15 0.38 -0.11 0.16 0.30 0.50 0.46
##
## $cba_brakes_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 1 0.03 -0.26 -0.04 0.04 0.0 0.00
## 2 * 2 15 0.38 0.18 0.40 0.22 0.5 0.62
## 3 3 22 0.55 -0.50 -0.39 0.70 0.5 0.31
## 4 4 2 0.05 -0.11 0.03 0.04 0.0 0.08
##
## $cba_brakes_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 14 0.35 -0.18 -0.08 0.39 0.25 0.31
## 2 2 9 0.22 -0.35 -0.11 0.26 0.25 0.15
## 3 3 8 0.20 -0.21 0.13 0.17 0.00 0.31
## 4 * 4 9 0.22 -0.02 0.06 0.17 0.50 0.23
##
## $cba_brakes_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 34 0.85 -0.14 0.05 0.87 0.50 0.92
## 2 2 0 0.00 NA 0.00 0.00 0.00 0.00
## 3 3 5 0.12 -0.19 -0.05 0.13 0.25 0.08
## 4 4 1 0.03 -0.04 0.00 0.00 0.25 0.00
##
## $cba_brakes_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.30 -0.44 -0.31 0.39 0.50 0.08
## 2 2 12 0.30 -0.37 -0.12 0.35 0.25 0.23
## 3 * 3 13 0.32 0.30 0.52 0.17 0.00 0.69
## 4 4 3 0.07 -0.20 -0.09 0.09 0.25 0.00
##
## $cba_brakes_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 11 0.28 -0.47 -0.36 0.43 0.00 0.08
## 2 2 2 0.05 -0.16 -0.04 0.04 0.25 0.00
## 3 3 3 0.07 -0.24 -0.01 0.09 0.00 0.08
## 4 * 4 24 0.60 0.16 0.41 0.43 0.75 0.85
##
## $cba_brakes_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.30 -0.33 -0.15 0.30 0.75 0.15
## 2 2 10 0.25 -0.39 -0.19 0.35 0.00 0.15
## 3 * 3 17 0.42 0.16 0.39 0.30 0.25 0.69
## 4 4 1 0.03 -0.26 -0.04 0.04 0.00 0.00
##
## $cba_brakes_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 6 0.15 -0.12 0.02 0.13 0.25 0.15
## 2 2 10 0.25 -0.32 -0.23 0.30 0.50 0.08
## 3 3 22 0.55 -0.15 0.17 0.52 0.25 0.69
## 4 4 2 0.05 -0.06 0.03 0.04 0.00 0.08
##
## $cba_brakes_14
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 8 0.20 -0.29 0.01 0.22 0 0.23
## 2 2 18 0.45 -0.33 -0.25 0.48 1 0.23
## 3 * 3 12 0.30 0.05 0.20 0.26 0 0.46
## 4 4 2 0.05 -0.16 0.03 0.04 0 0.08
##
## $cba_brakes_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 20 0.50 0.06 0.10 0.43 0.75 0.54
## 2 2 4 0.10 -0.19 0.07 0.09 0.00 0.15
## 3 3 15 0.38 -0.41 -0.13 0.43 0.25 0.31
## 4 4 1 0.03 -0.19 -0.04 0.04 0.00 0.00
##
## $cba_brakes_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 20 0.50 -0.23 0.03 0.43 1 0.46
## 2 2 1 0.03 -0.26 -0.04 0.04 0 0.00
## 3 3 18 0.45 -0.18 -0.06 0.52 0 0.46
## 4 4 1 0.03 0.03 0.08 0.00 0 0.08
##
## $cba_brakes_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 11 0.28 0.25 0.29 0.17 0.25 0.46
## 2 2 1 0.03 -0.26 -0.04 0.04 0.00 0.00
## 3 3 28 0.70 -0.52 -0.24 0.78 0.75 0.54
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_brakes_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 5 0.12 -0.19 0.07 0.09 0.25 0.15
## 2 2 17 0.42 -0.36 -0.21 0.52 0.25 0.31
## 3 3 5 0.12 -0.32 -0.10 0.17 0.00 0.08
## 4 * 4 13 0.32 0.10 0.24 0.22 0.50 0.46
##
## $cba_brakes_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 0 0.00 NA 0.00 0.00 0 0.00
## 2 * 2 18 0.45 0.19 0.54 0.30 0 0.85
## 3 3 1 0.03 -0.19 -0.04 0.04 0 0.00
## 4 4 21 0.52 -0.54 -0.50 0.65 1 0.15
##
## $cba_brakes_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 14 0.35 0.13 0.28 0.26 0.25 0.54
## 2 2 17 0.42 -0.36 -0.13 0.43 0.75 0.31
## 3 3 9 0.22 -0.38 -0.15 0.30 0.00 0.15
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_brakes_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.05 -0.16 -0.09 0.09 0.00 0.00
## 2 2 12 0.30 -0.42 -0.24 0.39 0.25 0.15
## 3 * 3 26 0.65 0.04 0.32 0.52 0.75 0.85
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_brakes_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 5 0.12 -0.47 -0.22 0.22 0.00 0.00
## 2 * 2 10 0.25 0.09 0.18 0.13 0.75 0.31
## 3 3 25 0.62 -0.23 0.04 0.65 0.25 0.69
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_brakes_24
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 37 0.92 0.00 0.09 0.91 0.75 1
## 2 2 1 0.03 -0.26 -0.04 0.04 0.00 0
## 3 3 2 0.05 -0.11 -0.04 0.04 0.25 0
## 4 4 0 0.00 NA 0.00 0.00 0.00 0Booklet 3 - Timing belt
CTT::distractorAnalysis(na.omit(df[, items_timing_belt]) ,
keys_timing_belt ,
nGroups = 3 ,
digits = 2)## $cba_timing_belt_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 11 0.28 -0.34 -0.20 0.35 0.3 0.15
## 2 2 7 0.17 -0.28 -0.16 0.24 0.2 0.08
## 3 * 3 21 0.52 0.16 0.42 0.35 0.5 0.77
## 4 4 1 0.03 -0.31 -0.06 0.06 0.0 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 7 0.17 -0.19 -0.08 0.24 0.1 0.15
## 2 * 2 12 0.30 -0.05 0.15 0.24 0.3 0.38
## 3 3 20 0.50 -0.28 -0.07 0.53 0.5 0.46
## 4 4 1 0.03 -0.06 0.00 0.00 0.1 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 3 0.07 -0.21 -0.04 0.12 0.0 0.08
## 2 * 2 18 0.45 0.16 0.17 0.29 0.7 0.46
## 3 3 8 0.20 -0.42 -0.22 0.29 0.2 0.08
## 4 4 11 0.28 -0.24 0.09 0.29 0.1 0.38
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 28 0.70 0.27 0.47 0.53 0.6 1
## 2 2 5 0.12 -0.48 -0.24 0.24 0.1 0
## 3 3 6 0.15 -0.31 -0.18 0.18 0.3 0
## 4 4 1 0.03 -0.12 -0.06 0.06 0.0 0
## 9 9 0 0.00 NA 0.00 0.00 0.0 0
##
## $cba_timing_belt_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 30 0.75 0.24 0.26 0.59 0.9 0.85
## 2 2 1 0.03 -0.24 -0.06 0.06 0.0 0.00
## 3 3 9 0.22 -0.47 -0.20 0.35 0.1 0.15
## 4 4 0 0.00 NA 0.00 0.00 0.0 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_11
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 15 0.38 0.06 0.24 0.29 0.3 0.54
## 2 2 4 0.10 0.06 0.15 0.00 0.2 0.15
## 3 3 20 0.50 -0.54 -0.48 0.71 0.5 0.23
## 4 4 1 0.03 0.07 0.08 0.00 0.0 0.08
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 6 0.15 -0.41 -0.29 0.29 0.1 0.00
## 2 * 2 20 0.50 0.16 0.53 0.24 0.6 0.77
## 3 3 7 0.17 -0.40 -0.29 0.29 0.2 0.00
## 4 4 7 0.17 -0.06 0.05 0.18 0.1 0.23
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_13
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 8 0.20 -0.35 -0.08 0.24 0.2 0.15
## 2 2 11 0.28 -0.22 0.01 0.29 0.2 0.31
## 3 * 3 17 0.42 0.14 0.24 0.29 0.5 0.54
## 4 4 4 0.10 -0.30 -0.18 0.18 0.1 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_14_old
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 11 0.28 -0.26 -0.05 0.35 0.1 0.31
## 2 * 2 23 0.58 0.10 0.28 0.41 0.7 0.69
## 3 3 1 0.03 -0.18 -0.06 0.06 0.0 0.00
## 4 4 3 0.07 -0.31 -0.12 0.12 0.1 0.00
## 9 9 2 0.05 -0.21 -0.06 0.06 0.1 0.00
##
## $cba_timing_belt_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 6 0.15 -0.23 -0.16 0.24 0.1 0.08
## 2 2 2 0.05 -0.39 -0.12 0.12 0.0 0.00
## 3 3 3 0.07 -0.10 -0.06 0.06 0.2 0.00
## 4 * 4 29 0.72 0.05 0.33 0.59 0.7 0.92
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.05 0.06 0.08 0.00 0.1 0.08
## 2 2 12 0.30 -0.56 -0.47 0.47 0.4 0.00
## 3 * 3 25 0.62 0.12 0.32 0.53 0.5 0.85
## 4 4 1 0.03 0.07 0.08 0.00 0.0 0.08
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 6 0.15 -0.34 -0.16 0.24 0.1 0.08
## 2 2 11 0.28 -0.44 -0.26 0.41 0.2 0.15
## 3 3 2 0.05 -0.12 -0.06 0.06 0.1 0.00
## 4 * 4 20 0.50 0.27 0.53 0.24 0.6 0.77
## 9 9 1 0.03 -0.12 -0.06 0.06 0.0 0.00
##
## $cba_timing_belt_19
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 7 0.17 -0.03 0.17 0.06 0.3 0.23
## 2 2 8 0.20 -0.07 0.05 0.18 0.2 0.23
## 3 3 8 0.20 -0.28 -0.20 0.35 0.0 0.15
## 4 4 17 0.42 -0.24 -0.03 0.41 0.5 0.38
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 18 0.45 0.29 0.40 0.29 0.4 0.69
## 2 2 4 0.10 -0.20 -0.04 0.12 0.1 0.08
## 3 3 15 0.38 -0.51 -0.30 0.53 0.3 0.23
## 4 4 3 0.07 -0.21 -0.06 0.06 0.2 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 0 0.00 NA 0.00 0.00 0.0 0.00
## 2 * 2 15 0.38 -0.05 0.09 0.29 0.5 0.38
## 3 3 25 0.62 -0.32 -0.09 0.71 0.5 0.62
## 4 4 0 0.00 NA 0.00 0.00 0.0 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_22
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 2 0.05 -0.17 -0.06 0.06 0.1 0.00
## 2 * 2 27 0.68 0.28 0.39 0.53 0.6 0.92
## 3 3 11 0.28 -0.55 -0.33 0.41 0.3 0.08
## 4 4 0 0.00 NA 0.00 0.00 0.0 0.00
## 9 9 0 0.00 NA 0.00 0.00 0.0 0.00
##
## $cba_timing_belt_23
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 8 0.2 -0.44 -0.29 0.29 0.3 0
## 2 2 0 0.0 NA 0.00 0.00 0.0 0
## 3 * 3 32 0.8 0.15 0.29 0.71 0.7 1
## 4 4 0 0.0 NA 0.00 0.00 0.0 0
## 9 9 0 0.0 NA 0.00 0.00 0.0 0Booklet 4 - Steering and front axle
CTT::distractorAnalysis(na.omit(df[, items_steering]) ,
keys_steering ,
nGroups = 3 ,
digits = 2)## $cba_steering_05
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 3 0.06 -0.44 -0.15 0.15 0.00 0
## 2 * 2 33 0.69 0.22 0.55 0.45 0.71 1
## 3 3 6 0.12 -0.34 -0.15 0.15 0.21 0
## 4 4 6 0.12 -0.47 -0.25 0.25 0.07 0
##
## $cba_steering_06
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 13 0.27 -0.22 0.04 0.25 0.29 0.29
## 2 2 14 0.29 -0.21 0.04 0.25 0.36 0.29
## 3 3 19 0.40 -0.42 -0.14 0.50 0.29 0.36
## 4 * 4 2 0.04 -0.01 0.07 0.00 0.07 0.07
##
## $cba_steering_07
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 9 0.19 -0.46 -0.23 0.3 0.14 0.07
## 2 2 24 0.50 -0.41 -0.17 0.6 0.43 0.43
## 3 3 0 0.00 NA 0.00 0.0 0.00 0.00
## 4 * 4 15 0.31 0.07 0.40 0.1 0.43 0.50
##
## $cba_steering_08
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 38 0.79 0.04 0.23 0.70 0.79 0.93
## 2 2 1 0.02 -0.23 -0.05 0.05 0.00 0.00
## 3 3 8 0.17 -0.41 -0.13 0.20 0.21 0.07
## 4 4 1 0.02 -0.23 -0.05 0.05 0.00 0.00
##
## $cba_steering_09
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 33 0.69 0.16 0.48 0.45 0.79 0.93
## 2 2 5 0.10 -0.23 -0.03 0.10 0.14 0.07
## 3 3 10 0.21 -0.62 -0.45 0.45 0.07 0.00
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_steering_10
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 4 0.08 -0.24 -0.10 0.1 0.14 0.00
## 2 * 2 24 0.50 0.00 0.31 0.4 0.43 0.71
## 3 3 20 0.42 -0.52 -0.21 0.5 0.43 0.29
## 4 4 0 0.00 NA 0.00 0.0 0.00 0.00
##
## $cba_steering_12
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 10 0.21 -0.07 0.14 0.15 0.21 0.29
## 2 2 10 0.21 -0.65 -0.40 0.40 0.14 0.00
## 3 3 11 0.23 -0.28 -0.06 0.20 0.36 0.14
## 4 * 4 17 0.35 -0.02 0.32 0.25 0.29 0.57
##
## $cba_steering_15
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 17 0.35 -0.10 0.27 0.30 0.21 0.57
## 2 2 1 0.02 -0.14 -0.05 0.05 0.00 0.00
## 3 3 30 0.62 -0.44 -0.22 0.65 0.79 0.43
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_steering_16
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 34 0.71 -0.17 0.14 0.65 0.71 0.79
## 2 2 3 0.06 -0.04 0.07 0.00 0.14 0.07
## 3 3 10 0.21 -0.37 -0.16 0.30 0.14 0.14
## 4 4 1 0.02 -0.23 -0.05 0.05 0.00 0.00
##
## $cba_steering_17
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 4 0.08 -0.37 -0.15 0.15 0.07 0.00
## 2 * 2 23 0.48 0.13 0.39 0.25 0.64 0.64
## 3 3 4 0.08 -0.32 -0.08 0.15 0.00 0.07
## 4 4 17 0.35 -0.48 -0.16 0.45 0.29 0.29
##
## $cba_steering_18
## correct key n rspP pbis discrim lower mid66 upper
## 1 * 1 42 0.88 -0.06 0.13 0.8 0.93 0.93
## 2 2 0 0.00 NA 0.00 0.0 0.00 0.00
## 3 3 6 0.12 -0.34 -0.13 0.2 0.07 0.07
## 4 4 0 0.00 NA 0.00 0.0 0.00 0.00
##
## $cba_steering_20
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 0 0.00 NA 0.00 0.00 0.00 0.00
## 2 * 2 28 0.58 -0.06 0.26 0.45 0.64 0.71
## 3 3 20 0.42 -0.50 -0.26 0.55 0.36 0.29
## 4 4 0 0.00 NA 0.00 0.00 0.00 0.00
##
## $cba_steering_21
## correct key n rspP pbis discrim lower mid66 upper
## 1 1 12 0.25 -0.27 0.04 0.25 0.21 0.29
## 2 2 11 0.23 -0.19 0.16 0.20 0.14 0.36
## 3 3 13 0.27 -0.40 -0.09 0.30 0.29 0.21
## 4 * 4 12 0.25 -0.16 -0.11 0.25 0.36 0.143.2.3 Recode items
3.2.3.1 Create a new data frame
Copy case numbers and demographic data to a new data frame named ‘df.dichotom’.
df.dichotom <- df[,c(1:3)]
3.2.3.2 Recode
Recode the marked options (1, 2, 3, or 4) into credits (0 or 1) for each item.
The below code contains a short summary for each item (as comments,
starting with a hash #).
Booklet 1 - Lighting system
# Item 05 - Checking headlights
df.dichotom$cba_lights_05 <- car::recode(df$cba_lights_05 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 06 - Checking rear lights
df.dichotom$cba_lights_06 <- car::recode(df$cba_lights_06 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 07 - Changing a light bulb
df.dichotom$cba_lights_07 <- car::recode(df$cba_lights_07 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 08 - Diagnosing the cause of a loose connection
df.dichotom$cba_lights_08 <- car::recode(df$cba_lights_08 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_09' is missing here because it was added to the instrument after this study.
# Item 10 - Identifying a wire in a circuit diagram
df.dichotom$cba_lights_10 <- car::recode(df$cba_lights_10 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 11 - Choosing a repair method to fix a ruptured wire
df.dichotom$cba_lights_11 <- car::recode(df$cba_lights_11 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 12 - Judging different repair methods of fixing a ruptured wire
df.dichotom$cba_lights_12 <- car::recode(df$cba_lights_12 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 13 - Identifying wires in a circuit diagram
df.dichotom$cba_lights_13 <- car::recode(df$cba_lights_13 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_14' is missing here because it was added to the instrument after this study.
# Item 15 - Judging the result of an electric measurement
df.dichotom$cba_lights_15 <- car::recode(df$cba_lights_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Judging a method of checking a light bulb
df.dichotom$cba_lights_16 <- car::recode(df$cba_lights_16 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 17 - Judging potential subsequent actions
df.dichotom$cba_lights_17 <- car::recode(df$cba_lights_17 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 18 - Judging a repair method of fixing a ruptured wire
df.dichotom$cba_lights_18 <- car::recode(df$cba_lights_18 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 19 - Explaining why halogen lamps should never be touched with bare fingers
df.dichotom$cba_lights_19 <- car::recode(df$cba_lights_19 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 20 - Explaining what happens if halogen lamps are touched with bare fingers
df.dichotom$cba_lights_20 <- car::recode(df$cba_lights_20 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 21 - Adjusting headlights
df.dichotom$cba_lights_21 <- car::recode(df$cba_lights_21 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 22 - Judging potential subsequent actions
df.dichotom$cba_lights_22 <- car::recode(df$cba_lights_22 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 23 - Explaining the potential risks of wrongly adjusted headlights
df.dichotom$cba_lights_23 <- car::recode(df$cba_lights_23 , "c(3) = '1' ; NA = NA ; else = '0'")
# Note: Item 'lights_24' is missing here because it was added to the instrument after this study.Booklet 2 - Hydraulic brakes
# Item 05_old - Judging potential diagnostic actions
# Note: In later studies, a revised version of item 'brakes_05_old' was used --> brakes_05_new
df.dichotom$cba_brakes_05_old <- car::recode(df$cba_brakes_05_old , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'brakes_06' is missing here because it was added to the instrument after this study.
# Item 07 - Removing brake pads
df.dichotom$cba_brakes_07 <- car::recode(df$cba_brakes_07 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 08 - Judging (alleged) replacement intervals for brake pads
df.dichotom$cba_brakes_08 <- car::recode(df$cba_brakes_08 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 09 - Deciding which parts of the brake need to be replaced
df.dichotom$cba_brakes_09 <- car::recode(df$cba_brakes_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Fitting new brake pads
df.dichotom$cba_brakes_10 <- car::recode(df$cba_brakes_10 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 11 - Mounting brake calliper
df.dichotom$cba_brakes_11 <- car::recode(df$cba_brakes_11 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 12 - Replacing brake hose
df.dichotom$cba_brakes_12 <- car::recode(df$cba_brakes_12 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 13 - Judging potential diagnostic actions
df.dichotom$cba_brakes_13 <- car::recode(df$cba_brakes_13 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 14 - Explaining signs of wear
df.dichotom$cba_brakes_14 <- car::recode(df$cba_brakes_14 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 15 - Judging the potential consequences of an incomplete repair
df.dichotom$cba_brakes_15 <- car::recode(df$cba_brakes_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Judging potential subsequent actions
df.dichotom$cba_brakes_16 <- car::recode(df$cba_brakes_16 , "c(1) = '1' ; NA = NA ; else = '0'")
# Note: Item 'brakes_17' is missing here because it was added to the instrument after this study.
# Item 18 - Replacing brake disk
df.dichotom$cba_brakes_18 <- car::recode(df$cba_brakes_18 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 19 - Judging potential subsequent actions
df.dichotom$cba_brakes_19 <- car::recode(df$cba_brakes_19 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 20 - Judging two methods of resetting the position of brake calliper pistons
df.dichotom$cba_brakes_20 <- car::recode(df$cba_brakes_20 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 21 - Explaining why calliper pistons of some brakes must be turned to reset
df.dichotom$cba_brakes_21 <- car::recode(df$cba_brakes_21 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 22 - Assembling rear brakes
df.dichotom$cba_brakes_22 <- car::recode(df$cba_brakes_22 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 23 - Judging potential subsequent actions
df.dichotom$cba_brakes_23 <- car::recode(df$cba_brakes_23 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 24 - Explaining why the brake pedal must be pressed several times after replacing brake pads
df.dichotom$cba_brakes_24 <- car::recode(df$cba_brakes_24 , "c(1) = '1' ; NA = NA ; else = '0'") Booklet 3 - Timing belt
# Note: Item 'timing_belt_05' is missing here because it was added to the instrument after this study.
# Item 06 - Removing serpentine belt
df.dichotom$cba_timing_belt_06 <- car::recode(df$cba_timing_belt_06 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 07 - Judging potential consequences of a broken idler pulley
df.dichotom$cba_timing_belt_07 <- car::recode(df$cba_timing_belt_07 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 08 - Naming potential causes of tooth skipping
df.dichotom$cba_timing_belt_08 <- car::recode(df$cba_timing_belt_08 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 09 - Identifying the purpose of a special tool
df.dichotom$cba_timing_belt_09 <- car::recode(df$cba_timing_belt_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Identifying the purpose of another special tool
df.dichotom$cba_timing_belt_10 <- car::recode(df$cba_timing_belt_10 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 11 - Judging potential subsequent actions
df.dichotom$cba_timing_belt_11 <- car::recode(df$cba_timing_belt_11 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 12 - Replacing the timing belt
df.dichotom$cba_timing_belt_12 <- car::recode(df$cba_timing_belt_12 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 13 - Identifying repair steps skipped by the technician in the video
df.dichotom$cba_timing_belt_13 <- car::recode(df$cba_timing_belt_13 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 14 - Naming engine designs with two timing belts
# Note: In subsequent studies, a revised version of item 'timing_belt_14_old' was used --> timing_belt_14_new
df.dichotom$cba_timing_belt_14_old <- car::recode(df$cba_timing_belt_14_old , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'timing_belt_15' is missing here because it was added to the instrument after this study.
# Item 16 - Tensioning the timing belt
df.dichotom$cba_timing_belt_16 <- car::recode(df$cba_timing_belt_16 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 17 - Judging potential subsequent actions
df.dichotom$cba_timing_belt_17 <- car::recode(df$cba_timing_belt_17 , "c(3) = '1' ; NA = NA ; else = '0'")
# Item 18 - Attaching crankshaft pulley
df.dichotom$cba_timing_belt_18 <- car::recode(df$cba_timing_belt_18 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 19 - Attaching serpentine belt
df.dichotom$cba_timing_belt_19 <- car::recode(df$cba_timing_belt_19 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 20 - Judging potential subsequent actions
df.dichotom$cba_timing_belt_20 <- car::recode(df$cba_timing_belt_20 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 21 - Judging potential consequences of too high belt tension
df.dichotom$cba_timing_belt_21 <- car::recode(df$cba_timing_belt_21 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 22 - Explaining consequences of a torn timing belt to the engine
df.dichotom$cba_timing_belt_22 <- car::recode(df$cba_timing_belt_22 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 23 - Naming engine parts potentially affected by a torn timing belt
df.dichotom$cba_timing_belt_23 <- car::recode(df$cba_timing_belt_23 , "c(3) = '1' ; NA = NA ; else = '0'")
# Note: Item 'timing_belt_24' is missing here because it was added to the instrument after this study.Booklet 4 - Steering and front axle
# Item 05 - Naming the consequences of a broken anti-roll bar link on the car's operating permit
df.dichotom$cba_steering_05 <- car::recode(df$cba_steering_05 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 06 - Removing a tie rod (part 1)
df.dichotom$cba_steering_06 <- car::recode(df$cba_steering_06 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 07 - Removing a tie rod (part 2)
df.dichotom$cba_steering_07 <- car::recode(df$cba_steering_07 , "c(4) = '1' ; NA = NA ; else = '0'")
# Item 08 - Removing a steering boot
df.dichotom$cba_steering_08 <- car::recode(df$cba_steering_08 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 09 - Fitting a new steering boot
df.dichotom$cba_steering_09 <- car::recode(df$cba_steering_09 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 10 - Judging the consequences of insufficient amount of grease
df.dichotom$cba_steering_10 <- car::recode(df$cba_steering_10 , "c(2) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_11' is missing here because it was added to the instrument after this study.
# Item 12 - Fitting a new tie rod
df.dichotom$cba_steering_12 <- car::recode(df$cba_steering_12 , "c(4) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_13' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_14' is missing here because it was added to the instrument after this study.
# Item 15 - Judging the consequences of a broken anti-roll bar link
df.dichotom$cba_steering_15 <- car::recode(df$cba_steering_15 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 16 - Naming the function of the anti-roll bar link
df.dichotom$cba_steering_16 <- car::recode(df$cba_steering_16 , "c(1) = '1' ; NA = NA ; else = '0'")
# Item 17 - Fitting a new anti-roll bar link
df.dichotom$cba_steering_17 <- car::recode(df$cba_steering_17 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 18 - Judging potential subsequent actions
df.dichotom$cba_steering_18 <- car::recode(df$cba_steering_18 , "c(1) = '1' ; NA = NA ; else = '0'") #
# Note: Item 'steering_19' is missing here because it was added to the instrument after this study.
# Item 20 - Judging the consequences of insufficient wheel alignment
df.dichotom$cba_steering_20 <- car::recode(df$cba_steering_20 , "c(2) = '1' ; NA = NA ; else = '0'")
# Item 21 - Naming potential reasons for wheel alignment
df.dichotom$cba_steering_21 <- car::recode(df$cba_steering_21 , "c(4) = '1' ; NA = NA ; else = '0'")
# Note: Item 'steering_22' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_23' is missing here because it was added to the instrument after this study.
# Note: Item 'steering_24' is missing here because it was added to the instrument after this study.3.2.4 Descriptive item coefficients
Display descriptive item coefficients.
psych::describe(df.dichotom[,c(4:68)])## vars n mean sd median trimmed mad min max range
## cba_lights_05 1 47 0.45 0.50 0.0 0.44 0.00 0 1 1
## cba_lights_06 2 47 0.87 0.34 1.0 0.95 0.00 0 1 1
## cba_lights_07 3 47 0.15 0.36 0.0 0.08 0.00 0 1 1
## cba_lights_08 4 47 0.74 0.44 1.0 0.79 0.00 0 1 1
## cba_lights_10 5 47 0.17 0.38 0.0 0.10 0.00 0 1 1
## cba_lights_11 6 47 0.66 0.48 1.0 0.69 0.00 0 1 1
## cba_lights_12 7 47 0.47 0.50 0.0 0.46 0.00 0 1 1
## cba_lights_13 8 47 0.34 0.48 0.0 0.31 0.00 0 1 1
## cba_lights_15 9 47 0.66 0.48 1.0 0.69 0.00 0 1 1
## cba_lights_16 10 47 0.09 0.28 0.0 0.00 0.00 0 1 1
## cba_lights_17 11 47 0.26 0.44 0.0 0.21 0.00 0 1 1
## cba_lights_18 12 47 0.17 0.38 0.0 0.10 0.00 0 1 1
## cba_lights_19 13 47 0.06 0.25 0.0 0.00 0.00 0 1 1
## cba_lights_20 14 47 0.36 0.49 0.0 0.33 0.00 0 1 1
## cba_lights_21 15 47 0.89 0.31 1.0 0.97 0.00 0 1 1
## cba_lights_22 16 47 0.87 0.34 1.0 0.95 0.00 0 1 1
## cba_lights_23 17 47 0.89 0.31 1.0 0.97 0.00 0 1 1
## cba_brakes_05_old 18 40 0.15 0.36 0.0 0.06 0.00 0 1 1
## cba_brakes_07 19 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## cba_brakes_08 20 40 0.22 0.42 0.0 0.16 0.00 0 1 1
## cba_brakes_09 21 40 0.85 0.36 1.0 0.94 0.00 0 1 1
## cba_brakes_10 22 40 0.32 0.47 0.0 0.28 0.00 0 1 1
## cba_brakes_11 23 40 0.60 0.50 1.0 0.62 0.00 0 1 1
## cba_brakes_12 24 40 0.42 0.50 0.0 0.41 0.00 0 1 1
## cba_brakes_13 25 40 0.15 0.36 0.0 0.06 0.00 0 1 1
## cba_brakes_14 26 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## cba_brakes_15 27 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## cba_brakes_16 28 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## cba_brakes_18 29 40 0.28 0.45 0.0 0.22 0.00 0 1 1
## cba_brakes_19 30 40 0.32 0.47 0.0 0.28 0.00 0 1 1
## cba_brakes_20 31 40 0.45 0.50 0.0 0.44 0.00 0 1 1
## cba_brakes_21 32 40 0.35 0.48 0.0 0.31 0.00 0 1 1
## cba_brakes_22 33 40 0.65 0.48 1.0 0.69 0.00 0 1 1
## cba_brakes_23 34 40 0.25 0.44 0.0 0.19 0.00 0 1 1
## cba_brakes_24 35 40 0.92 0.27 1.0 1.00 0.00 0 1 1
## cba_timing_belt_06 36 40 0.52 0.51 1.0 0.53 0.00 0 1 1
## cba_timing_belt_07 37 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## cba_timing_belt_08 38 40 0.45 0.50 0.0 0.44 0.00 0 1 1
## cba_timing_belt_09 39 40 0.70 0.46 1.0 0.75 0.00 0 1 1
## cba_timing_belt_10 40 40 0.75 0.44 1.0 0.81 0.00 0 1 1
## cba_timing_belt_11 41 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## cba_timing_belt_12 42 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## cba_timing_belt_13 43 40 0.42 0.50 0.0 0.41 0.00 0 1 1
## cba_timing_belt_14_old 44 40 0.58 0.50 1.0 0.59 0.00 0 1 1
## cba_timing_belt_16 45 40 0.72 0.45 1.0 0.78 0.00 0 1 1
## cba_timing_belt_17 46 40 0.62 0.49 1.0 0.66 0.00 0 1 1
## cba_timing_belt_18 47 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## cba_timing_belt_19 48 40 0.17 0.38 0.0 0.09 0.00 0 1 1
## cba_timing_belt_20 49 40 0.45 0.50 0.0 0.44 0.00 0 1 1
## cba_timing_belt_21 50 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## cba_timing_belt_22 51 40 0.68 0.47 1.0 0.72 0.00 0 1 1
## cba_timing_belt_23 52 40 0.80 0.41 1.0 0.88 0.00 0 1 1
## cba_steering_05 53 48 0.69 0.47 1.0 0.73 0.00 0 1 1
## cba_steering_06 54 48 0.04 0.20 0.0 0.00 0.00 0 1 1
## cba_steering_07 55 48 0.31 0.47 0.0 0.28 0.00 0 1 1
## cba_steering_08 56 48 0.79 0.41 1.0 0.85 0.00 0 1 1
## cba_steering_09 57 48 0.69 0.47 1.0 0.72 0.00 0 1 1
## cba_steering_10 58 48 0.50 0.51 0.5 0.50 0.74 0 1 1
## cba_steering_12 59 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## cba_steering_15 60 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## cba_steering_16 61 48 0.71 0.46 1.0 0.75 0.00 0 1 1
## cba_steering_17 62 48 0.48 0.50 0.0 0.48 0.00 0 1 1
## cba_steering_18 63 48 0.88 0.33 1.0 0.95 0.00 0 1 1
## cba_steering_20 64 48 0.58 0.50 1.0 0.60 0.00 0 1 1
## cba_steering_21 65 48 0.25 0.44 0.0 0.20 0.00 0 1 1
## skew kurtosis se
## cba_lights_05 0.21 -2.00 0.07
## cba_lights_06 -2.16 2.73 0.05
## cba_lights_07 1.91 1.68 0.05
## cba_lights_08 -1.09 -0.84 0.06
## cba_lights_10 1.70 0.91 0.06
## cba_lights_11 -0.65 -1.61 0.07
## cba_lights_12 0.12 -2.03 0.07
## cba_lights_13 0.65 -1.61 0.07
## cba_lights_15 -0.65 -1.61 0.07
## cba_lights_16 2.88 6.43 0.04
## cba_lights_17 1.09 -0.84 0.06
## cba_lights_18 1.70 0.91 0.06
## cba_lights_19 3.46 10.16 0.04
## cba_lights_20 0.56 -1.72 0.07
## cba_lights_21 -2.47 4.20 0.05
## cba_lights_22 -2.16 2.73 0.05
## cba_lights_23 -2.47 4.20 0.05
## cba_brakes_05_old 1.89 1.60 0.06
## cba_brakes_07 0.50 -1.80 0.08
## cba_brakes_08 1.27 -0.40 0.07
## cba_brakes_09 -1.89 1.60 0.06
## cba_brakes_10 0.72 -1.52 0.07
## cba_brakes_11 -0.39 -1.89 0.08
## cba_brakes_12 0.29 -1.96 0.08
## cba_brakes_13 1.89 1.60 0.06
## cba_brakes_14 0.84 -1.33 0.07
## cba_brakes_15 0.00 -2.05 0.08
## cba_brakes_16 0.00 -2.05 0.08
## cba_brakes_18 0.97 -1.08 0.07
## cba_brakes_19 0.72 -1.52 0.07
## cba_brakes_20 0.19 -2.01 0.08
## cba_brakes_21 0.61 -1.67 0.08
## cba_brakes_22 -0.61 -1.67 0.08
## cba_brakes_23 1.11 -0.78 0.07
## cba_brakes_24 -3.11 7.85 0.04
## cba_timing_belt_06 -0.10 -2.04 0.08
## cba_timing_belt_07 0.84 -1.33 0.07
## cba_timing_belt_08 0.19 -2.01 0.08
## cba_timing_belt_09 -0.84 -1.33 0.07
## cba_timing_belt_10 -1.11 -0.78 0.07
## cba_timing_belt_11 0.50 -1.80 0.08
## cba_timing_belt_12 0.00 -2.05 0.08
## cba_timing_belt_13 0.29 -1.96 0.08
## cba_timing_belt_14_old -0.29 -1.96 0.08
## cba_timing_belt_16 -0.97 -1.08 0.07
## cba_timing_belt_17 -0.50 -1.80 0.08
## cba_timing_belt_18 0.00 -2.05 0.08
## cba_timing_belt_19 1.65 0.73 0.06
## cba_timing_belt_20 0.19 -2.01 0.08
## cba_timing_belt_21 0.50 -1.80 0.08
## cba_timing_belt_22 -0.72 -1.52 0.07
## cba_timing_belt_23 -1.44 0.09 0.06
## cba_steering_05 -0.78 -1.41 0.07
## cba_steering_06 4.44 18.13 0.03
## cba_steering_07 0.78 -1.41 0.07
## cba_steering_08 -1.39 -0.06 0.06
## cba_steering_09 -0.78 -1.41 0.07
## cba_steering_10 0.00 -2.04 0.07
## cba_steering_12 0.59 -1.68 0.07
## cba_steering_15 0.59 -1.68 0.07
## cba_steering_16 -0.89 -1.24 0.07
## cba_steering_17 0.08 -2.03 0.07
## cba_steering_18 -2.20 2.89 0.05
## cba_steering_20 -0.33 -1.93 0.07
## cba_steering_21 1.12 -0.76 0.063.2.5 Sum scales
For each test booklet, we calculate raw scores (i.e., the sum of correct responses) and display a descriptive statistic.
Note. As the test booklets differ in length (i.e., number of items), the raw scores cannot be used to compare the difficulty of the booklets.
Booklet 1 - Lighting system
df.dichotom$sum_lights <- rowSums(df.dichotom[,items_lights] , na.rm = FALSE)
psych::describe(df.dichotom$sum_lights)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 47 8.11 1.68 8 8.13 1.48 4 11 7 -0.03 -0.35 0.25Booklet 2 - Hydraulic brakes
df.dichotom$sum_brakes <- rowSums(df.dichotom[,items_brakes] , na.rm = FALSE)
psych::describe(df.dichotom$sum_brakes)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 40 7.62 2.22 7 7.44 2.22 4 14 10 0.73 0.11 0.35Booklet 3 - Timing belt
df.dichotom$sum_timing_belt <- rowSums(df.dichotom[,items_timing_belt] , na.rm = FALSE)
psych::describe(df.dichotom$sum_timing_belt)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 40 8.93 2.56 9 8.94 2.97 4 14 10 -0.09 -0.96 0.4Booklet 4 - Steering and front axle
df.dichotom$sum_steering <- rowSums(df.dichotom[,items_steering] , na.rm = FALSE)
psych::describe(df.dichotom$sum_steering)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 48 6.62 1.62 7 6.67 1.48 3 10 7 -0.22 -0.51 0.233.2.6 Scale reliability (internal consistency)
We calculate two coefficients of internal consistency: Cronbach’s Alpha and Guttman’s Lambda-6. Note that internal consistency is of limited use for tests measuring a wide range of various abilities (i.e., instruments that are not internally consistent by design). In such cases, the criteria recommended for psychological tests will not be met, and values below .7 can be expected due to the diversity of the construct (Field, Miles, & Field, 2012, p. 799).
However, low internal consistency affects the outcome of analyses such as group comparisons and thereby the inferences based on such tests. Therefore it should be reported. If the association to the underlying variable varies across tests items, Guttman’s Lambda-6 is more appropriate than Cronbach’s Alpha.
As discussed in our research article, Digital measurement of hands‑on performance? Ecological validation of a computer‑based assessment of automotive repair skills (Hartmann, Güzel, & Geschwendtner, 2023b), internal consistency of the scales could be improved by removing items with unsatisfactory discrimination parameters (i.e., low or negative item-rest correlations) from the scales. However, this would in return significantly affect the test’s ecological validity: Each of the video-based tests (lighting, brakes, timing belt, steering) represents a complete repair task, addressing each of its steps without skipping any important actions. If items were removed from the tests, the tests would no longer be valid in terms of a complete and realistic representation of the underlying tasks.
Note. The code below will produce warnings about some items which are negatively correlated with the scales and probably should be reversed. These messages must be ignored as the test does not contain reversed items.
Booklet 1 - Lighting system (no items removed)
reli_lights <- psych::alpha(df.dichotom[,items_lights] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_lights$total$raw_alpha ## [1] 0.01351517reli_lights$total$`G6(smc)` ## [1] 0.3174202Booklet 2 - Hydraulic brakes (no items removed)
reli_brakes <- psych::alpha(df.dichotom[,items_brakes] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_brakes$total$raw_alpha ## [1] 0.265847reli_brakes$total$`G6(smc)` ## [1] 0.5462895Booklet 3 - Timing belt (no items removed)
reli_timing_belt <- psych::alpha(df.dichotom[,items_timing_belt] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_timing_belt$total$raw_alpha ## [1] 0.4341331reli_timing_belt$total$`G6(smc)` ## [1] 0.6611437Booklet 4 - Steering and front axle (no items removed)
reli_steering <- psych::alpha(df.dichotom[,items_steering] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_steering$total$raw_alpha ## [1] 0.006226054reli_steering$total$`G6(smc)` ## [1] 0.20802433.3 Repair protocols
3.3.1 Frequency analysis: Response data and missing data
Response values for each item are ‘0’ (Action was not performed), and ‘1’ (Action was performed; see chapter 2.2 and Figure 3). A value of ‘7’ indicates the participant ran out of time before reaching this action (missing not reached). A value of ‘9’ indicates the expert observing the participant marked neither option on the protocol (missing by intention). Items not presented to a participant by design appear as ‘NA’ (missing by design).
apply(df[c(69:230)] , 2 , table , exclude = FALSE) ## $handson_lights_09a
##
## 0 1 <NA>
## 8 40 40
##
## $handson_lights_09b
##
## 0 1 9 <NA>
## 22 25 1 40
##
## $handson_lights_09c
##
## 0 1 9 <NA>
## 14 21 13 40
##
## $handson_lights_10_11_16a
##
## 0 1 <NA>
## 43 5 40
##
## $handson_lights_10_11_16b
##
## 0 1 <NA>
## 45 3 40
##
## $handson_lights_10_11_16c
##
## 0 1 <NA>
## 33 15 40
##
## $handson_lights_10_11_16d
##
## 0 1 9 <NA>
## 30 4 14 40
##
## $handson_lights_10_11_16e
##
## 0 1 9 <NA>
## 23 24 1 40
##
## $handson_lights_10_11_16f
##
## 0 1 9 <NA>
## 44 3 1 40
##
## $handson_lights_10_11_16g
##
## 0 1 9 <NA>
## 24 22 2 40
##
## $handson_lights_10_11_16h
##
## 0 1 9 <NA>
## 3 36 9 40
##
## $handson_lights_12a
##
## 0 1 <NA>
## 1 47 40
##
## $handson_lights_12b
##
## 0 1 9 <NA>
## 29 14 5 40
##
## $handson_lights_12c
##
## 0 1 9 <NA>
## 26 17 5 40
##
## $handson_lights_12d
##
## 0 1 9 <NA>
## 25 18 5 40
##
## $handson_lights_12e
##
## 0 1 9 <NA>
## 21 19 8 40
##
## $handson_lights_12f
##
## 0 1 9 <NA>
## 10 21 17 40
##
## $handson_lights_13_14_15a
##
## 0 1 9 <NA>
## 39 8 1 40
##
## $handson_lights_13_14_15b
##
## 0 1 <NA>
## 18 30 40
##
## $handson_lights_13_14_15c
##
## 0 1 9 <NA>
## 35 12 1 40
##
## $handson_lights_13_14_15d
##
## 0 1 9 <NA>
## 34 11 3 40
##
## $handson_lights_13_14_15e
##
## 0 1 9 <NA>
## 32 13 3 40
##
## $handson_lights_13_14_15f
##
## 0 1 9 <NA>
## 24 23 1 40
##
## $handson_lights_13_14_15g
##
## 0 1 9 <NA>
## 40 6 2 40
##
## $handson_lights_13_14_15h
##
## 0 1 9 <NA>
## 30 17 1 40
##
## $handson_lights_13_14_15i
##
## 0 1 <NA>
## 35 13 40
##
## $handson_lights_13_14_15k
##
## 0 1 9 <NA>
## 1 45 2 40
##
## $handson_brakes_06a
##
## 0 1 9 <NA>
## 30 8 1 49
##
## $handson_brakes_06b
##
## 0 1 9 <NA>
## 31 7 1 49
##
## $handson_brakes_06c
##
## 0 1 <NA>
## 2 37 49
##
## $handson_brakes_06d
##
## 0 1 <NA>
## 38 1 49
##
## $handson_brakes_06e
##
## 0 1 <NA>
## 38 1 49
##
## $handson_brakes_06f
##
## 0 9 <NA>
## 37 2 49
##
## $handson_brakes_06g
##
## 0 1 <NA>
## 8 31 49
##
## $handson_brakes_06h
##
## 0 1 9 <NA>
## 1 37 1 49
##
## $handson_brakes_06i
##
## 0 1 9 <NA>
## 1 37 1 49
##
## $handson_brakes_06k
##
## 0 1 9 <NA>
## 7 31 1 49
##
## $handson_brakes_06l
##
## 0 1 9 <NA>
## 25 3 11 49
##
## $handson_brakes_08a
##
## 0 1 9 <NA>
## 27 10 2 49
##
## $handson_brakes_08b
##
## 0 1 <NA>
## 19 20 49
##
## $handson_brakes_08c
##
## 0 1 9 <NA>
## 24 14 1 49
##
## $handson_brakes_08d
##
## 0 1 9 <NA>
## 23 14 2 49
##
## $handson_brakes_08e
##
## 0 1 <NA>
## 6 33 49
##
## $handson_brakes_08f
##
## 0 1 <NA>
## 22 17 49
##
## $handson_brakes_08g
##
## 0 1 <NA>
## 37 2 49
##
## $handson_brakes_08h
##
## 0 1 <NA>
## 37 2 49
##
## $handson_brakes_08i
##
## 0 1 9 <NA>
## 25 4 10 49
##
## $handson_brakes_09_10a
##
## 0 1 <NA>
## 2 37 49
##
## $handson_brakes_09_10b
##
## 0 1 <NA>
## 37 2 49
##
## $handson_brakes_09_10c
##
## 0 1 <NA>
## 19 20 49
##
## $handson_brakes_09_10d
##
## 0 1 <NA>
## 31 8 49
##
## $handson_brakes_09_10e
##
## 0 1 <NA>
## 23 16 49
##
## $handson_brakes_09_10f
##
## 1 9 <NA>
## 38 1 49
##
## $handson_brakes_09_10g
##
## 0 1 9 <NA>
## 1 36 2 49
##
## $handson_brakes_09_10h
##
## 0 1 9 <NA>
## 13 25 1 49
##
## $handson_brakes_09_10i
##
## 0 1 9 <NA>
## 22 6 11 49
##
## $handson_brakes_16a
##
## 1 <NA>
## 39 49
##
## $handson_brakes_16b
##
## 0 1 <NA>
## 8 31 49
##
## $handson_brakes_16c
##
## 0 1 <NA>
## 35 4 49
##
## $handson_brakes_16d
##
## 0 1 <NA>
## 34 5 49
##
## $handson_brakes_16e
##
## 0 1 <NA>
## 33 6 49
##
## $handson_brakes_16f
##
## 0 1 <NA>
## 1 38 49
##
## $handson_brakes_16g
##
## 0 1 <NA>
## 1 38 49
##
## $handson_brakes_16h
##
## 0 1 <NA>
## 13 26 49
##
## $handson_brakes_16i
##
## 0 9 <NA>
## 25 14 49
##
## $handson_brakes_17_21a
##
## 0 1 <NA>
## 8 31 49
##
## $handson_brakes_17_21b
##
## 0 1 <NA>
## 6 33 49
##
## $handson_brakes_17_21c
##
## 0 1 <NA>
## 4 35 49
##
## $handson_brakes_17_21d
##
## 0 1 9 <NA>
## 10 28 1 49
##
## $handson_brakes_17_21e
##
## 0 1 9 <NA>
## 31 7 1 49
##
## $handson_brakes_17_21f
##
## 0 1 9 <NA>
## 13 25 1 49
##
## $handson_brakes_17_21g
##
## 0 1 9 <NA>
## 22 16 1 49
##
## $handson_brakes_17_21h
##
## 0 1 <NA>
## 12 27 49
##
## $handson_brakes_17_21i
##
## 0 1 9 <NA>
## 16 11 12 49
##
## $handson_steering_07a
##
## 0 1 9 <NA>
## 17 30 1 40
##
## $handson_steering_07b
##
## 0 1 9 <NA>
## 38 9 1 40
##
## $handson_steering_07c
##
## 0 1 9 <NA>
## 35 12 1 40
##
## $handson_steering_07d
##
## 0 1 9 <NA>
## 38 9 1 40
##
## $handson_steering_07e
##
## 0 1 9 <NA>
## 36 2 10 40
##
## $handson_steering_07f
##
## 0 1 9 <NA>
## 5 28 15 40
##
## $handson_steering_07g
##
## 0 1 9 <NA>
## 44 3 1 40
##
## $handson_steering_07h
##
## 0 1 9 <NA>
## 2 42 4 40
##
## $handson_steering_07i
##
## 0 1 9 <NA>
## 26 18 4 40
##
## $handson_steering_07k
##
## 0 1 9 <NA>
## 2 45 1 40
##
## $handson_steering_07l
##
## 0 1 9 <NA>
## 17 9 22 40
##
## $handson_steering_08a
##
## 0 1 9 <NA>
## 41 5 2 40
##
## $handson_steering_08b
##
## 0 1 9 <NA>
## 6 40 2 40
##
## $handson_steering_08c
##
## 0 1 9 <NA>
## 14 31 3 40
##
## $handson_steering_08d
##
## 0 1 9 <NA>
## 36 10 2 40
##
## $handson_steering_08e
##
## 0 1 9 <NA>
## 2 44 2 40
##
## $handson_steering_08f
##
## 0 1 9 <NA>
## 18 3 27 40
##
## $handson_steering_09a
##
## 0 1 9 <NA>
## 37 6 5 40
##
## $handson_steering_09b
##
## 0 1 9 <NA>
## 32 10 6 40
##
## $handson_steering_09c
##
## 0 1 9 <NA>
## 3 41 4 40
##
## $handson_steering_09d
##
## 0 1 9 <NA>
## 9 33 6 40
##
## $handson_steering_09e
##
## 0 1 9 <NA>
## 19 23 6 40
##
## $handson_steering_09f
##
## 0 1 9 <NA>
## 16 12 20 40
##
## $handson_steering_11a
##
## 0 1 9 <NA>
## 2 42 4 40
##
## $handson_steering_11b
##
## 0 1 9 <NA>
## 2 42 4 40
##
## $handson_steering_11c
##
## 0 1 9 <NA>
## 6 35 7 40
##
## $handson_steering_11d
##
## 0 9 <NA>
## 24 24 40
##
## $handson_steering_11e
##
## 0 1 9 <NA>
## 24 16 8 40
##
## $handson_steering_11f
##
## 0 1 9 <NA>
## 11 27 10 40
##
## $handson_steering_11g
##
## 0 1 9 <NA>
## 12 26 10 40
##
## $handson_steering_11h
##
## 0 1 9 <NA>
## 33 4 11 40
##
## $handson_steering_11i
##
## 0 1 9 <NA>
## 28 10 10 40
##
## $handson_steering_11k
##
## 0 1 9 <NA>
## 29 9 10 40
##
## $handson_steering_11l
##
## 0 1 9 <NA>
## 11 22 15 40
##
## $handson_steering_11m
##
## 0 1 9 <NA>
## 26 7 15 40
##
## $handson_steering_11n
##
## 0 1 9 <NA>
## 20 13 15 40
##
## $handson_steering_11o
##
## 0 1 9 <NA>
## 16 3 29 40
##
## $handson_steering_15a
##
## 0 1 9 <NA>
## 1 32 15 40
##
## $handson_steering_15b
##
## 0 1 9 <NA>
## 14 18 16 40
##
## $handson_steering_15c
##
## 0 1 9 <NA>
## 3 31 14 40
##
## $handson_steering_15d
##
## 0 1 9 <NA>
## 16 17 15 40
##
## $handson_steering_15e
##
## 0 1 9 <NA>
## 17 9 22 40
##
## $handson_timing_belt_10a
##
## 1 9 <NA>
## 39 1 48
##
## $handson_timing_belt_10b
##
## 0 1 9 <NA>
## 2 37 1 48
##
## $handson_timing_belt_10c
##
## 0 1 7 9 <NA>
## 34 1 1 4 48
##
## $handson_timing_belt_10d
##
## 0 1 7 9 <NA>
## 16 20 1 3 48
##
## $handson_timing_belt_10e
##
## 0 1 7 9 <NA>
## 8 29 1 2 48
##
## $handson_timing_belt_10f
##
## 0 1 7 9 <NA>
## 17 20 1 2 48
##
## $handson_timing_belt_10g
##
## 0 1 7 9 <NA>
## 17 20 1 2 48
##
## $handson_timing_belt_10h
##
## 0 1 7 9 <NA>
## 14 23 1 2 48
##
## $handson_timing_belt_10i
##
## 0 1 7 9 <NA>
## 29 7 1 3 48
##
## $handson_timing_belt_10k
##
## 0 1 7 9 <NA>
## 17 19 1 3 48
##
## $handson_timing_belt_10l
##
## 0 1 7 9 <NA>
## 28 7 1 4 48
##
## $handson_timing_belt_11_12a
##
## 0 1 7 9 <NA>
## 1 17 21 1 48
##
## $handson_timing_belt_11_12b
##
## 0 1 7 9 <NA>
## 2 16 21 1 48
##
## $handson_timing_belt_11_12c
##
## 0 1 7 9 <NA>
## 1 17 21 1 48
##
## $handson_timing_belt_11_12d
##
## 0 1 7 9 <NA>
## 6 12 21 1 48
##
## $handson_timing_belt_11_12e
##
## 0 1 7 9 <NA>
## 3 15 21 1 48
##
## $handson_timing_belt_11_12f
##
## 0 1 7 9 <NA>
## 2 16 21 1 48
##
## $handson_timing_belt_11_12g
##
## 0 1 7 9 <NA>
## 2 16 21 1 48
##
## $handson_timing_belt_11_12h
##
## 0 1 7 9 <NA>
## 4 13 21 2 48
##
## $handson_timing_belt_11_12i
##
## 0 1 7 9 <NA>
## 3 15 21 1 48
##
## $handson_timing_belt_11_12k
##
## 0 1 7 9 <NA>
## 8 10 21 1 48
##
## $handson_timing_belt_11_12l
##
## 0 1 7 9 <NA>
## 18 3 18 1 48
##
## $handson_timing_belt_14_15a
##
## 0 1 7 9 <NA>
## 5 12 22 1 48
##
## $handson_timing_belt_14_15b
##
## 0 1 7 9 <NA>
## 4 13 22 1 48
##
## $handson_timing_belt_14_15c
##
## 0 1 7 9 <NA>
## 3 8 22 7 48
##
## $handson_timing_belt_14_15d
##
## 0 1 7 9 <NA>
## 2 15 22 1 48
##
## $handson_timing_belt_14_15e
##
## 0 1 7 9 <NA>
## 2 15 22 1 48
##
## $handson_timing_belt_14_15f
##
## 0 1 7 9 <NA>
## 2 15 22 1 48
##
## $handson_timing_belt_14_15g
##
## 0 1 7 9 <NA>
## 4 12 23 1 48
##
## $handson_timing_belt_14_15h
##
## 0 1 7 9 <NA>
## 4 12 23 1 48
##
## $handson_timing_belt_14_15i
##
## 0 1 7 9 <NA>
## 4 12 23 1 48
##
## $handson_timing_belt_14_15k
##
## 0 1 7 9 <NA>
## 8 8 23 1 48
##
## $handson_timing_belt_14_15l
##
## 0 1 7 9 <NA>
## 7 9 23 1 48
##
## $handson_timing_belt_14_15m
##
## 0 1 7 9 <NA>
## 8 8 23 1 48
##
## $handson_timing_belt_14_15n
##
## 0 1 7 9 <NA>
## 11 5 22 2 48
##
## $handson_timing_belt_14_15o
##
## 0 1 7 9 <NA>
## 14 5 19 2 48
##
## $handson_timing_belt_16a
##
## 0 1 7 9 <NA>
## 1 11 27 1 48
##
## $handson_timing_belt_16b
##
## 0 1 7 9 <NA>
## 2 9 28 1 48
##
## $handson_timing_belt_16c
##
## 0 1 7 9 <NA>
## 7 3 29 1 48
##
## $handson_timing_belt_16d
##
## 0 7 9 <NA>
## 10 29 1 48
##
## $handson_timing_belt_16e
##
## 0 1 7 9 <NA>
## 9 1 29 1 48
##
## $handson_timing_belt_16f
##
## 0 1 7 9 <NA>
## 6 4 29 1 48
##
## $handson_timing_belt_16g
##
## 0 1 7 9 <NA>
## 6 4 29 1 48
##
## $handson_timing_belt_16h
##
## 0 1 7 9 <NA>
## 9 1 29 1 48
##
## $handson_timing_belt_16i
##
## 0 7 9 <NA>
## 10 29 1 48
##
## $handson_timing_belt_16k
##
## 0 1 7 9 <NA>
## 10 3 26 1 483.3.2 Recode correct actions
The current item values indicate whether an action was performed (1) or not performed (0). In order to measure the participants’ repair skills, the items must be recoded into these values:
correct action was performed (1 credit) or not performed (0 credits),
incorrect action was performed (0 credits) or not performed (1 credit).
The new values are written to the ‘df.dichotom’ data frame.
Note. We recode missing values into zero (0 credits). If two items on a protocol reflect the same action (e.g., item 1 reflects a correct action, and item 2 reflects the opposite incorrect action), only the first item will be used for scoring. Irrelevant actions are not used for scoring and will be skipped.
The below code contains a short summary for each item (as comments,
starting with a hash #).
Station 1 - Lighting system
# Item handson_lights_09a:
# 'inspects the circuit diagram' (correct action)
df.dichotom$handson_lights_09ax <- car::recode(df$handson_lights_09a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_09b:
# 'identifies the correct component (blinker)' (correct action)
df.dichotom$handson_lights_09bx <- car::recode(df$handson_lights_09b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_10_11_16a:
# 'cuts the frayed wire ends cleanly with a side cutter' (correct action)
df.dichotom$handson_lights_10_11_16ax <- car::recode(df$handson_lights_10_11_16a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_10_11_16b:
# 'removes the wire insulation with wire strippers to a length of approximately 5 to 7 mm' (correct action)
df.dichotom$handson_lights_10_11_16bx <- car::recode(df$handson_lights_10_11_16b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_10_11_16c:
# 'repairs the wire professionally with a solder connector' (correct action)
df.dichotom$handson_lights_10_11_16cx <- car::recode(df$handson_lights_10_11_16c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_10_11_16d:
# 'shields other cables against heat during the procedure' (correct action)
df.dichotom$handson_lights_10_11_16dx <- car::recode(df$handson_lights_10_11_16d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_12a:
# 'inspects the circuit diagram' (correct action)
df.dichotom$handson_lights_12ax <- car::recode(df$handson_lights_12a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_12b:
# 'attaches yellow wire to pin 1' (correct action)
df.dichotom$handson_lights_12bx <- car::recode(df$handson_lights_12b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_12c:
# 'attaches brown wire to pin 3' (correct action)
df.dichotom$handson_lights_12cx <- car::recode(df$handson_lights_12c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_12d:
# 'attaches brown-white wire to pin 5' (correct action)
df.dichotom$handson_lights_12dx <- car::recode(df$handson_lights_12d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15a:
# 'checks the bulb for continuity/resistance with a multimeter' (correct action)
df.dichotom$handson_lights_13_14_15ax <- car::recode(df$handson_lights_13_14_15a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15c:
# 'checks power supply in the headlight' (correct action)
df.dichotom$handson_lights_13_14_15cx <- car::recode(df$handson_lights_13_14_15c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15d:
# 'checks power supply at the connector (pin 8 against pin 9)' (correct action)
df.dichotom$handson_lights_13_14_15dx <- car::recode(df$handson_lights_13_14_15d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15e:
# 'checks the fuse for continuity/resistance with a multimeter' (correct action)
df.dichotom$handson_lights_13_14_15ex <- car::recode(df$handson_lights_13_14_15e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15g:
# 'checks continuity/resistance of the power supply cable with a multimeter' (correct action)
df.dichotom$handson_lights_13_14_15gx <- car::recode(df$handson_lights_13_14_15g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15h:
# 'identifies cause of the fault (1/2): defective bulb' (correct action)
df.dichotom$handson_lights_13_14_15hx <- car::recode(df$handson_lights_13_14_15h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_lights_13_14_15i:
# 'identifies cause of the fault (2/2): broken wire inside the headlight' (correct action)
df.dichotom$handson_lights_13_14_15ix <- car::recode(df$handson_lights_13_14_15i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")Station 2 - Hydraulic brakes
# Item handson_brakes_06a:
# 'checks brake fluid level' (correct action)
df.dichotom$handson_brakes_06ax <- car::recode(df$handson_brakes_06a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_06b:
# 'draws off brake fluid with a syringe' (incorrect action)
df.dichotom$handson_brakes_06bx <- car::recode(df$handson_brakes_06b ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_06c:
# 'loosens the brake calliper bolts using a ratchet and socket' (correct action)
df.dichotom$handson_brakes_06cx <- car::recode(df$handson_brakes_06c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_06d:
# 'loosens the brake calliper bolts using an Allen key' (correct action)
df.dichotom$handson_brakes_06dx <- car::recode(df$handson_brakes_06d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_06f:
# 'attempts to loosen tight brake pads with a hammer' (incorrect action)
df.dichotom$handson_brakes_06fx <- car::recode(df$handson_brakes_06f ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_06g:
# 'attempts to reset piston position with a screwdriver' (incorrect action)
df.dichotom$handson_brakes_06gx <- car::recode(df$handson_brakes_06g ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_06h:
# 'removes brake calliper from the brake calliper carrier' (correct action)
df.dichotom$handson_brakes_06hx <- car::recode(df$handson_brakes_06h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_06i:
# 'removes brake pads' (correct action)
df.dichotom$handson_brakes_06ix <- car::recode(df$handson_brakes_06i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_06k:
# 'secures brake calliper with a hook' (correct action)
df.dichotom$handson_brakes_06kx <- car::recode(df$handson_brakes_06k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_08a:
# 'measures thickness of both brake pads' (correct action)
df.dichotom$handson_brakes_08ax <- car::recode(df$handson_brakes_08a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_08b:
# 'measures thickness of brake disk' (correct action)
df.dichotom$handson_brakes_08bx <- car::recode(df$handson_brakes_08b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_08c:
# 'visually inspects brake hose' (correct action)
df.dichotom$handson_brakes_08cx <- car::recode(df$handson_brakes_08c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_08d:
# 'visually inspects rubber seal on brake calliper' (correct action)
df.dichotom$handson_brakes_08dx <- car::recode(df$handson_brakes_08d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_08e:
# 'proposes to replace brake pads' (incorrect action)
df.dichotom$handson_brakes_08ex <- car::recode(df$handson_brakes_08e ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_08f:
# 'proposes to replace brake disk' (incorrect action)
df.dichotom$handson_brakes_08fx <- car::recode(df$handson_brakes_08f ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_08g:
# 'proposes to replace brake hose' (incorrect action)
df.dichotom$handson_brakes_08gx <- car::recode(df$handson_brakes_08g ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_08h:
# 'proposes to replace rubber seal or entire brake calliper' (incorrect action)
df.dichotom$handson_brakes_08hx <- car::recode(df$handson_brakes_08h ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_09_10a:
# 'resets position of brake piston with adequate tool' (correct action)
df.dichotom$handson_brakes_09_10ax <- car::recode(df$handson_brakes_09_10a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_09_10b:
# 'attempts to reset position of brake piston with waterpump pliers' (incorrect action)
df.dichotom$handson_brakes_09_10bx <- car::recode(df$handson_brakes_09_10b ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_09_10c:
# 'attempts to remove brake dust with a wire brush' (correct action)
df.dichotom$handson_brakes_09_10cx <- car::recode(df$handson_brakes_09_10c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_09_10d:
# 'attempts to use copper spray' (incorrect action)
df.dichotom$handson_brakes_09_10dx <- car::recode(df$handson_brakes_09_10d ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_09_10e:
# 'attempts to use brake degreaser' (correct action)
df.dichotom$handson_brakes_09_10ex <- car::recode(df$handson_brakes_09_10e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_09_10f:
# 'installs brake pads' (correct action)
df.dichotom$handson_brakes_09_10fx <- car::recode(df$handson_brakes_09_10f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_09_10g:
# 'connects cable of brake wear indicator' (correct action)
df.dichotom$handson_brakes_09_10gx <- car::recode(df$handson_brakes_09_10g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_09_10h:
# 'tightens brake calliper bolts with correct torque' (correct action)
df.dichotom$handson_brakes_09_10hx <- car::recode(df$handson_brakes_09_10h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16a:
# 'removes rear brake disk' (correct action)
df.dichotom$handson_brakes_16ax <- car::recode(df$handson_brakes_16a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16b:
# 'attempts to remove brake dust with a wire brush' (correct action)
df.dichotom$handson_brakes_16bx <- car::recode(df$handson_brakes_16b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16c:
# 'attempts to apply grease to rear wheel hub' (incorrect action)
df.dichotom$handson_brakes_16cx <- car::recode(df$handson_brakes_16c ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_16d:
# 'attempts to apply copper spray to rear wheel hub' (incorrect action)
df.dichotom$handson_brakes_16dx <- car::recode(df$handson_brakes_16d ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_16e:
# 'attempts to apply ceramic paste to rear wheel hub' (correct action)
df.dichotom$handson_brakes_16ex <- car::recode(df$handson_brakes_16e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16f:
# 'installs rear brake disk' (correct action)
df.dichotom$handson_brakes_16fx <- car::recode(df$handson_brakes_16f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16g:
# 'attaches lock screw' (correct action)
df.dichotom$handson_brakes_16gx <- car::recode(df$handson_brakes_16g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_16h:
# 'fastens lock screw with correct torque' (correct action)
df.dichotom$handson_brakes_16hx <- car::recode(df$handson_brakes_16h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21a:
# 'attaches rear calliper carrier' (correct action)
df.dichotom$handson_brakes_17_21ax <- car::recode(df$handson_brakes_17_21a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21b:
# 'attaches rear brake calliper' (correct action)
df.dichotom$handson_brakes_17_21bx <- car::recode(df$handson_brakes_17_21b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21c:
# 'suggests to mount wheels' (correct action)
df.dichotom$handson_brakes_17_21cx <- car::recode(df$handson_brakes_17_21c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21d:
# 'suggests to push brake pedal several times' (correct action)
df.dichotom$handson_brakes_17_21dx <- car::recode(df$handson_brakes_17_21d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21e:
# 'suggests to bleed brake system' (incorrect action)
df.dichotom$handson_brakes_17_21ex <- car::recode(df$handson_brakes_17_21e ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_brakes_17_21f:
# 'suggests to check brake fluid level' (correct action)
df.dichotom$handson_brakes_17_21fx <- car::recode(df$handson_brakes_17_21f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21g:
# 'suggests to bed in brake' (correct action)
df.dichotom$handson_brakes_17_21gx <- car::recode(df$handson_brakes_17_21g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_brakes_17_21h:
# 'suggests taking a test drive' (incorrect action)
df.dichotom$handson_brakes_17_21hx <- car::recode(df$handson_brakes_17_21h ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")Station 3 - Timing belt
# Item handson_timing_belt_10a:
# 'turns crankshaft clockwise to top dead centre (TDC)' (correct action)
df.dichotom$handson_timing_belt_10ax <- car::recode(df$handson_timing_belt_10a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10b:
# 'attaches locking tool to camshaft' (correct action)
df.dichotom$handson_timing_belt_10bx <- car::recode(df$handson_timing_belt_10b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10c:
# 'loosens pulley bolt with an impact wrench' (incorrect action)
df.dichotom$handson_timing_belt_10cx <- car::recode(df$handson_timing_belt_10c ,
"0 = '1' ; c(9 , 7 , 1) = 0 ; NA = NA")
# Item handson_timing_belt_10d:
# 'loosens pulley polt with a torque wrench' (correct action)
df.dichotom$handson_timing_belt_10dx <- car::recode(df$handson_timing_belt_10d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10e:
# 'secures pulley with specialised tool 3415[/1]' (correct action)
df.dichotom$handson_timing_belt_10ex <- car::recode(df$handson_timing_belt_10e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10f:
# 'screws the pulley bolt back in …' (correct action)
df.dichotom$handson_timing_belt_10fx <- car::recode(df$handson_timing_belt_10f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10g:
# '… and uses two washers under the bolt' (correct action)
df.dichotom$handson_timing_belt_10gx <- car::recode(df$handson_timing_belt_10g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10h:
# 'releases tension of the timing belt tensioning pulley' (correct action)
df.dichotom$handson_timing_belt_10hx <- car::recode(df$handson_timing_belt_10h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10i:
# 'completely removes the timing belt tensioner pulley' (correct action)
df.dichotom$handson_timing_belt_10ix <- car::recode(df$handson_timing_belt_10i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_10k:
# 'removes main timing belt' (correct action)
df.dichotom$handson_timing_belt_10kx <- car::recode(df$handson_timing_belt_10k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12a:
# 'releases tension of the secondary belt tensioning pulley' (correct action)
df.dichotom$handson_timing_belt_11_12ax <- car::recode(df$handson_timing_belt_11_12a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12b:
# 'completely removes the secondary belt tensioner pulley' (correct action)
df.dichotom$handson_timing_belt_11_12bx <- car::recode(df$handson_timing_belt_11_12b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12c:
# 'removes secondary belt' (correct action)
df.dichotom$handson_timing_belt_11_12cx <- car::recode(df$handson_timing_belt_11_12c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12d:
# 'checks timing mark on the crankshaft' (correct action)
df.dichotom$handson_timing_belt_11_12dx <- car::recode(df$handson_timing_belt_11_12d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12e:
# 'checks position of both camshafts' (correct action)
df.dichotom$handson_timing_belt_11_12ex <- car::recode(df$handson_timing_belt_11_12e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12f:
# 'installs new secondary belt' (correct action)
df.dichotom$handson_timing_belt_11_12fx <- car::recode(df$handson_timing_belt_11_12f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12g:
# 'installs secondary belt tensioner pulley' (correct action)
df.dichotom$handson_timing_belt_11_12gx <- car::recode(df$handson_timing_belt_11_12g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12h:
# 'turns secondary pulley counter-clockwise' (correct action)
df.dichotom$handson_timing_belt_11_12hx <- car::recode(df$handson_timing_belt_11_12h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12i:
# 'tightens secondary tensioner pulley' (correct action)
df.dichotom$handson_timing_belt_11_12ix <- car::recode(df$handson_timing_belt_11_12i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_11_12k:
# 'tightens bolt with correct torque (20 Nm)' (correct action)
df.dichotom$handson_timing_belt_11_12kx <- car::recode(df$handson_timing_belt_11_12k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15a:
# 'checks timing mark on the crankshaft again' (correct action)
df.dichotom$handson_timing_belt_14_15ax <- car::recode(df$handson_timing_belt_14_15a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15b:
# 'checks position of both camshafts again' (correct action)
df.dichotom$handson_timing_belt_14_15bx <- car::recode(df$handson_timing_belt_14_15b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15c:
# 'installs main belt tensioner pulley' (correct action)
df.dichotom$handson_timing_belt_14_15cx <- car::recode(df$handson_timing_belt_14_15c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15d:
# 'attaches main timing belt …' (correct action)
df.dichotom$handson_timing_belt_14_15dx <- car::recode(df$handson_timing_belt_14_15d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15e:
# '… starting at the water pump …' (correct action)
df.dichotom$handson_timing_belt_14_15ex <- car::recode(df$handson_timing_belt_14_15e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15f:
# '… continuing clockwise' (correct action)
df.dichotom$handson_timing_belt_14_15fx <- car::recode(df$handson_timing_belt_14_15f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15g:
# 'tensions the tensioning pulley up to the mark' (correct action)
df.dichotom$handson_timing_belt_14_15gx <- car::recode(df$handson_timing_belt_14_15g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15h:
# 'fastens tensioning pulley …' (correct action)
df.dichotom$handson_timing_belt_14_15hx <- car::recode(df$handson_timing_belt_14_15h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15i:
# '… while holding it in position with an Allen wrench' (correct action)
df.dichotom$handson_timing_belt_14_15ix <- car::recode(df$handson_timing_belt_14_15i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15k:
# 'tightens tensioning pulley bolt with correct torque (20 Nm)' (correct action)
df.dichotom$handson_timing_belt_14_15kx <- car::recode(df$handson_timing_belt_14_15k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15l:
# 're-checks crankshaft timing mark' (correct action)
df.dichotom$handson_timing_belt_14_15lx <- car::recode(df$handson_timing_belt_14_15l ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15m:
# 're-checks camshaft positions' (correct action)
df.dichotom$handson_timing_belt_14_15mx <- car::recode(df$handson_timing_belt_14_15m ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_14_15n:
# 'turns crankshaft 720 degrees clockwise' (correct action)
df.dichotom$handson_timing_belt_14_15nx <- car::recode(df$handson_timing_belt_14_15n ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16a:
# 'removes bolt (and washers, if applied) from crankshaft' (correct action)
df.dichotom$handson_timing_belt_16ax <- car::recode(df$handson_timing_belt_16a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16b:
# 'attaches crankshaft pulley' (correct action)
df.dichotom$handson_timing_belt_16bx <- car::recode(df$handson_timing_belt_16b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16c:
# 'fastens pulley with a new bolt' (correct action)
df.dichotom$handson_timing_belt_16cx <- car::recode(df$handson_timing_belt_16c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16d:
# 'fastens bolt with an impact wrench' (incorrect action)
df.dichotom$handson_timing_belt_16dx <- car::recode(df$handson_timing_belt_16d ,
"0 = '1' ; c(9 , 7 , 1) = 0 ; NA = NA")
# Item handson_timing_belt_16e:
# 'fastens bolt with a ratcheting socket wrench' (correct action)
df.dichotom$handson_timing_belt_16ex <- car::recode(df$handson_timing_belt_16e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16f:
# 'fastens bolt with a toque wrench' (correct action)
df.dichotom$handson_timing_belt_16fx <- car::recode(df$handson_timing_belt_16f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16g:
# 'fastens bolt with correct torque (depending on version)' (correct action)
df.dichotom$handson_timing_belt_16gx <- car::recode(df$handson_timing_belt_16g ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_timing_belt_16h:
# 'turns the screw by 90 degrees (using a specialised tool)' (correct action)
df.dichotom$handson_timing_belt_16hx <- car::recode(df$handson_timing_belt_16h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")Station 4 - Steering and front axle
# Item handson_steering_07a:
# 'loosens the nut of the tie rod end with a ratchet' (incorrect action)
df.dichotom$handson_steering_07ax <- car::recode(df$handson_steering_07a ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_07b:
# 'loosens the nut of the tie rod end with an impact wrench' (incorrect action)
df.dichotom$handson_steering_07bx <- car::recode(df$handson_steering_07b ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_07c:
# 'loosens the nut of the tie rod end with a ring or open-end spanner' (correct action)
df.dichotom$handson_steering_07cx <- car::recode(df$handson_steering_07c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_07d:
# 'holds the bolt of the tie rod end in place with an Allen key' (correct action)
df.dichotom$handson_steering_07dx <- car::recode(df$handson_steering_07d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_07e:
# 'attempts to knock the tie rod end out of the steering knuckle with a hammer' (incorrect action)
df.dichotom$handson_steering_07ex <- car::recode(df$handson_steering_07e ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_07f:
# 'presses the tie rod end out of the steering knuckle with a special tool' (correct action)
df.dichotom$handson_steering_07fx <- car::recode(df$handson_steering_07f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_07h:
# 'loosens the counter nut …' (correct action)
df.dichotom$handson_steering_07hx <- car::recode(df$handson_steering_07h ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_07i:
# '… while holding the tie rod with an open-end spanner' (correct action)
df.dichotom$handson_steering_07ix <- car::recode(df$handson_steering_07i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_07k:
# 'removes tie rod end' (correct action)
df.dichotom$handson_steering_07kx <- car::recode(df$handson_steering_07k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_08a:
# 'opens small clamping ring with standard pliers' (incorrect action)
df.dichotom$handson_steering_08ax <- car::recode(df$handson_steering_08a ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_08b:
# 'opens small clamping ring with special pliers' (correct action)
df.dichotom$handson_steering_08bx <- car::recode(df$handson_steering_08b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_08c:
# 'opens large clamping ring with a screwdriver' (correct action)
df.dichotom$handson_steering_08cx <- car::recode(df$handson_steering_08c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_08d:
# 'opens large clamping ring with side cutters' (correct action)
df.dichotom$handson_steering_08dx <- car::recode(df$handson_steering_08d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_08e:
# 'pulls steering boot off the tie rod' (correct action)
df.dichotom$handson_steering_08ex <- car::recode(df$handson_steering_08e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_09a:
# 'removes old grease from the inner tie rod joint' (correct action)
df.dichotom$handson_steering_09ax <- car::recode(df$handson_steering_09a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_09b:
# 'applies fresh grease to inner tie rod joint' (correct action)
df.dichotom$handson_steering_09bx <- car::recode(df$handson_steering_09b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_09c:
# 'attaches new steering boot' (correct action)
df.dichotom$handson_steering_09cx <- car::recode(df$handson_steering_09c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_09d:
# 'fastens small clamping ring with special pliers' (correct action)
df.dichotom$handson_steering_09dx <- car::recode(df$handson_steering_09d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_09e:
# 'fastens large clamping ring with special pliers' (correct action)
df.dichotom$handson_steering_09ex <- car::recode(df$handson_steering_09e ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11a:
# 'screws on counter nut' (correct action)
df.dichotom$handson_steering_11ax <- car::recode(df$handson_steering_11a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11b:
# 'attaches steering rod end to steering rod' (correct action)
df.dichotom$handson_steering_11bx <- car::recode(df$handson_steering_11b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11c:
# 'fastens counter nut' (correct action)
df.dichotom$handson_steering_11cx <- car::recode(df$handson_steering_11c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11f:
# 'secures the tapered bolt of the tie rod end with a new self-locking nut' (correct action)
df.dichotom$handson_steering_11fx <- car::recode(df$handson_steering_11f ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11g:
# 'fastens self-locking nut with a ratchet' (incorrect action)
df.dichotom$handson_steering_11gx <- car::recode(df$handson_steering_11g ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_11h:
# 'fastens self-locking nut with an impact wrench' (incorrect action)
df.dichotom$handson_steering_11hx <- car::recode(df$handson_steering_11h ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_11i:
# 'fastens self-locking nut with a ring or open-end spanner' (correct action)
df.dichotom$handson_steering_11ix <- car::recode(df$handson_steering_11i ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11k:
# 'holds the bolt of the tie rod end in place with an Allen key' (correct action)
df.dichotom$handson_steering_11kx <- car::recode(df$handson_steering_11k ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11l:
# 'fastens self-locking nut with correct torque (20 Nm)' (correct action)
df.dichotom$handson_steering_11lx <- car::recode(df$handson_steering_11l ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11m:
# 'turns self-locking nut by 90 degrees using a special tool' (correct action)
df.dichotom$handson_steering_11mx <- car::recode(df$handson_steering_11m ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_11n:
# 'turns self-locking nut by 90 degrees by eye' (incorrect action)
df.dichotom$handson_steering_11nx <- car::recode(df$handson_steering_11n ,
"0 = '1' ; c(9 , 7 , 1) = '0' ; NA = NA")
# Item handson_steering_15a:
# 'suggests to mount wheels' (correct action)
df.dichotom$handson_steering_15ax <- car::recode(df$handson_steering_15a ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_15b:
# 'suggests to tighten wheel bolts with correct torque' (correct action)
df.dichotom$handson_steering_15bx <- car::recode(df$handson_steering_15b ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_15c:
# 'suggests to perform wheel alignment procedure' (correct action)
df.dichotom$handson_steering_15cx <- car::recode(df$handson_steering_15c ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")
# Item handson_steering_15d:
# 'suggests to go for a test drive' (correct action)
df.dichotom$handson_steering_15dx <- car::recode(df$handson_steering_15d ,
"c(9 , 7 , 0) = '0' ; 1 = '1' ; NA = NA")3.3.3 Descriptive item coefficients
Display descriptive item coefficients.
psych::describe(df.dichotom[,c(73:205)])## vars n mean sd median trimmed mad min max range
## handson_lights_09ax 1 48 0.83 0.38 1.0 0.90 0.00 0 1 1
## handson_lights_09bx 2 48 0.52 0.50 1.0 0.52 0.00 0 1 1
## handson_lights_10_11_16ax 3 48 0.10 0.31 0.0 0.02 0.00 0 1 1
## handson_lights_10_11_16bx 4 48 0.06 0.24 0.0 0.00 0.00 0 1 1
## handson_lights_10_11_16cx 5 48 0.31 0.47 0.0 0.27 0.00 0 1 1
## handson_lights_10_11_16dx 6 48 0.08 0.28 0.0 0.00 0.00 0 1 1
## handson_lights_12ax 7 48 0.98 0.14 1.0 1.00 0.00 0 1 1
## handson_lights_12bx 8 48 0.29 0.46 0.0 0.25 0.00 0 1 1
## handson_lights_12cx 9 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## handson_lights_12dx 10 48 0.38 0.49 0.0 0.35 0.00 0 1 1
## handson_lights_13_14_15ax 11 48 0.17 0.38 0.0 0.10 0.00 0 1 1
## handson_lights_13_14_15cx 12 48 0.25 0.44 0.0 0.20 0.00 0 1 1
## handson_lights_13_14_15dx 13 48 0.23 0.42 0.0 0.18 0.00 0 1 1
## handson_lights_13_14_15ex 14 48 0.27 0.45 0.0 0.23 0.00 0 1 1
## handson_lights_13_14_15gx 15 48 0.12 0.33 0.0 0.05 0.00 0 1 1
## handson_lights_13_14_15hx 16 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## handson_lights_13_14_15ix 17 48 0.27 0.45 0.0 0.22 0.00 0 1 1
## handson_brakes_06ax 18 39 0.21 0.41 0.0 0.15 0.00 0 1 1
## handson_brakes_06bx 19 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_06cx 20 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_06dx 21 39 0.03 0.16 0.0 0.00 0.00 0 1 1
## handson_brakes_06fx 22 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_06gx 23 39 0.21 0.41 0.0 0.15 0.00 0 1 1
## handson_brakes_06hx 24 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_06ix 25 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_06kx 26 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_08ax 27 39 0.26 0.44 0.0 0.21 0.00 0 1 1
## handson_brakes_08bx 28 39 0.51 0.51 1.0 0.52 0.00 0 1 1
## handson_brakes_08cx 29 39 0.36 0.49 0.0 0.33 0.00 0 1 1
## handson_brakes_08dx 30 39 0.36 0.49 0.0 0.33 0.00 0 1 1
## handson_brakes_08ex 31 39 0.15 0.37 0.0 0.09 0.00 0 1 1
## handson_brakes_08fx 32 39 0.56 0.50 1.0 0.58 0.00 0 1 1
## handson_brakes_08gx 33 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_08hx 34 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_09_10ax 35 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_09_10bx 36 39 0.95 0.22 1.0 1.00 0.00 0 1 1
## handson_brakes_09_10cx 37 39 0.51 0.51 1.0 0.52 0.00 0 1 1
## handson_brakes_09_10dx 38 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_09_10ex 39 39 0.41 0.50 0.0 0.39 0.00 0 1 1
## handson_brakes_09_10fx 40 39 0.97 0.16 1.0 1.00 0.00 0 1 1
## handson_brakes_09_10gx 41 39 0.92 0.27 1.0 1.00 0.00 0 1 1
## handson_brakes_09_10hx 42 39 0.64 0.49 1.0 0.67 0.00 0 1 1
## handson_brakes_16ax 43 39 1.00 0.00 1.0 1.00 0.00 1 1 0
## handson_brakes_16bx 44 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_16cx 45 39 0.90 0.31 1.0 0.97 0.00 0 1 1
## handson_brakes_16dx 46 39 0.87 0.34 1.0 0.94 0.00 0 1 1
## handson_brakes_16ex 47 39 0.15 0.37 0.0 0.09 0.00 0 1 1
## handson_brakes_16fx 48 39 0.97 0.16 1.0 1.00 0.00 0 1 1
## handson_brakes_16gx 49 39 0.97 0.16 1.0 1.00 0.00 0 1 1
## handson_brakes_16hx 50 39 0.67 0.48 1.0 0.70 0.00 0 1 1
## handson_brakes_17_21ax 51 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_17_21bx 52 39 0.85 0.37 1.0 0.91 0.00 0 1 1
## handson_brakes_17_21cx 53 39 0.90 0.31 1.0 0.97 0.00 0 1 1
## handson_brakes_17_21dx 54 39 0.72 0.46 1.0 0.76 0.00 0 1 1
## handson_brakes_17_21ex 55 39 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_brakes_17_21fx 56 39 0.64 0.49 1.0 0.67 0.00 0 1 1
## handson_brakes_17_21gx 57 39 0.41 0.50 0.0 0.39 0.00 0 1 1
## handson_brakes_17_21hx 58 39 0.31 0.47 0.0 0.27 0.00 0 1 1
## handson_timing_belt_10ax 59 40 0.98 0.16 1.0 1.00 0.00 0 1 1
## handson_timing_belt_10bx 60 40 0.92 0.27 1.0 1.00 0.00 0 1 1
## handson_timing_belt_10cx 61 40 0.85 0.36 1.0 0.94 0.00 0 1 1
## handson_timing_belt_10dx 62 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## handson_timing_belt_10ex 63 40 0.72 0.45 1.0 0.78 0.00 0 1 1
## handson_timing_belt_10fx 64 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## handson_timing_belt_10gx 65 40 0.50 0.51 0.5 0.50 0.74 0 1 1
## handson_timing_belt_10hx 66 40 0.58 0.50 1.0 0.59 0.00 0 1 1
## handson_timing_belt_10ix 67 40 0.17 0.38 0.0 0.09 0.00 0 1 1
## handson_timing_belt_10kx 68 40 0.48 0.51 0.0 0.47 0.00 0 1 1
## handson_timing_belt_11_12ax 69 40 0.42 0.50 0.0 0.41 0.00 0 1 1
## handson_timing_belt_11_12bx 70 40 0.40 0.50 0.0 0.38 0.00 0 1 1
## handson_timing_belt_11_12cx 71 40 0.42 0.50 0.0 0.41 0.00 0 1 1
## handson_timing_belt_11_12dx 72 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## handson_timing_belt_11_12ex 73 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## handson_timing_belt_11_12fx 74 40 0.40 0.50 0.0 0.38 0.00 0 1 1
## handson_timing_belt_11_12gx 75 40 0.40 0.50 0.0 0.38 0.00 0 1 1
## handson_timing_belt_11_12hx 76 40 0.32 0.47 0.0 0.28 0.00 0 1 1
## handson_timing_belt_11_12ix 77 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## handson_timing_belt_11_12kx 78 40 0.25 0.44 0.0 0.19 0.00 0 1 1
## handson_timing_belt_14_15ax 79 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## handson_timing_belt_14_15bx 80 40 0.32 0.47 0.0 0.28 0.00 0 1 1
## handson_timing_belt_14_15cx 81 40 0.20 0.41 0.0 0.12 0.00 0 1 1
## handson_timing_belt_14_15dx 82 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## handson_timing_belt_14_15ex 83 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## handson_timing_belt_14_15fx 84 40 0.38 0.49 0.0 0.34 0.00 0 1 1
## handson_timing_belt_14_15gx 85 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## handson_timing_belt_14_15hx 86 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## handson_timing_belt_14_15ix 87 40 0.30 0.46 0.0 0.25 0.00 0 1 1
## handson_timing_belt_14_15kx 88 40 0.20 0.41 0.0 0.12 0.00 0 1 1
## handson_timing_belt_14_15lx 89 40 0.22 0.42 0.0 0.16 0.00 0 1 1
## handson_timing_belt_14_15mx 90 40 0.20 0.41 0.0 0.12 0.00 0 1 1
## handson_timing_belt_14_15nx 91 40 0.12 0.33 0.0 0.03 0.00 0 1 1
## handson_timing_belt_16ax 92 40 0.28 0.45 0.0 0.22 0.00 0 1 1
## handson_timing_belt_16bx 93 40 0.22 0.42 0.0 0.16 0.00 0 1 1
## handson_timing_belt_16cx 94 40 0.08 0.27 0.0 0.00 0.00 0 1 1
## handson_timing_belt_16dx 95 40 0.25 0.44 0.0 0.19 0.00 0 1 1
## handson_timing_belt_16ex 96 40 0.02 0.16 0.0 0.00 0.00 0 1 1
## handson_timing_belt_16fx 97 40 0.10 0.30 0.0 0.00 0.00 0 1 1
## handson_timing_belt_16gx 98 40 0.10 0.30 0.0 0.00 0.00 0 1 1
## handson_timing_belt_16hx 99 40 0.02 0.16 0.0 0.00 0.00 0 1 1
## handson_steering_07ax 100 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## handson_steering_07bx 101 48 0.79 0.41 1.0 0.85 0.00 0 1 1
## handson_steering_07cx 102 48 0.25 0.44 0.0 0.20 0.00 0 1 1
## handson_steering_07dx 103 48 0.19 0.39 0.0 0.12 0.00 0 1 1
## handson_steering_07ex 104 48 0.75 0.44 1.0 0.80 0.00 0 1 1
## handson_steering_07fx 105 48 0.58 0.50 1.0 0.60 0.00 0 1 1
## handson_steering_07hx 106 48 0.88 0.33 1.0 0.95 0.00 0 1 1
## handson_steering_07ix 107 48 0.38 0.49 0.0 0.35 0.00 0 1 1
## handson_steering_07kx 108 48 0.94 0.24 1.0 1.00 0.00 0 1 1
## handson_steering_08ax 109 48 0.85 0.36 1.0 0.92 0.00 0 1 1
## handson_steering_08bx 110 48 0.83 0.38 1.0 0.90 0.00 0 1 1
## handson_steering_08cx 111 48 0.65 0.48 1.0 0.68 0.00 0 1 1
## handson_steering_08dx 112 48 0.21 0.41 0.0 0.15 0.00 0 1 1
## handson_steering_08ex 113 48 0.92 0.28 1.0 1.00 0.00 0 1 1
## handson_steering_09ax 114 48 0.12 0.33 0.0 0.05 0.00 0 1 1
## handson_steering_09bx 115 48 0.21 0.41 0.0 0.15 0.00 0 1 1
## handson_steering_09cx 116 48 0.85 0.36 1.0 0.92 0.00 0 1 1
## handson_steering_09dx 117 48 0.69 0.47 1.0 0.73 0.00 0 1 1
## handson_steering_09ex 118 48 0.48 0.50 0.0 0.48 0.00 0 1 1
## handson_steering_11ax 119 48 0.88 0.33 1.0 0.95 0.00 0 1 1
## handson_steering_11bx 120 48 0.88 0.33 1.0 0.95 0.00 0 1 1
## handson_steering_11cx 121 48 0.73 0.45 1.0 0.77 0.00 0 1 1
## handson_steering_11fx 122 48 0.56 0.50 1.0 0.57 0.00 0 1 1
## handson_steering_11gx 123 48 0.25 0.44 0.0 0.20 0.00 0 1 1
## handson_steering_11hx 124 48 0.69 0.47 1.0 0.73 0.00 0 1 1
## handson_steering_11ix 125 48 0.21 0.41 0.0 0.15 0.00 0 1 1
## handson_steering_11kx 126 48 0.19 0.39 0.0 0.12 0.00 0 1 1
## handson_steering_11lx 127 48 0.46 0.50 0.0 0.45 0.00 0 1 1
## handson_steering_11mx 128 48 0.15 0.36 0.0 0.07 0.00 0 1 1
## handson_steering_11nx 129 48 0.42 0.50 0.0 0.40 0.00 0 1 1
## handson_steering_15ax 130 48 0.67 0.48 1.0 0.70 0.00 0 1 1
## handson_steering_15bx 131 48 0.38 0.49 0.0 0.35 0.00 0 1 1
## handson_steering_15cx 132 48 0.65 0.48 1.0 0.68 0.00 0 1 1
## handson_steering_15dx 133 48 0.35 0.48 0.0 0.32 0.00 0 1 1
## skew kurtosis se
## handson_lights_09ax -1.73 1.03 0.05
## handson_lights_09bx -0.08 -2.03 0.07
## handson_lights_10_11_16ax 2.51 4.40 0.04
## handson_lights_10_11_16bx 3.50 10.49 0.04
## handson_lights_10_11_16cx 0.78 -1.41 0.07
## handson_lights_10_11_16dx 2.92 6.67 0.04
## handson_lights_12ax -6.50 41.12 0.02
## handson_lights_12bx 0.89 -1.24 0.07
## handson_lights_12cx 0.59 -1.68 0.07
## handson_lights_12dx 0.50 -1.79 0.07
## handson_lights_13_14_15ax 1.73 1.03 0.05
## handson_lights_13_14_15cx 1.12 -0.76 0.06
## handson_lights_13_14_15dx 1.25 -0.45 0.06
## handson_lights_13_14_15ex 1.00 -1.02 0.06
## handson_lights_13_14_15gx 2.20 2.89 0.05
## handson_lights_13_14_15hx 0.59 -1.68 0.07
## handson_lights_13_14_15ix 1.00 -1.02 0.06
## handson_brakes_06ax 1.40 -0.03 0.07
## handson_brakes_06bx -1.40 -0.03 0.07
## handson_brakes_06cx -3.91 13.67 0.04
## handson_brakes_06dx 5.77 32.15 0.03
## handson_brakes_06fx -3.91 13.67 0.04
## handson_brakes_06gx 1.40 -0.03 0.07
## handson_brakes_06hx -3.91 13.67 0.04
## handson_brakes_06ix -3.91 13.67 0.04
## handson_brakes_06kx -1.40 -0.03 0.07
## handson_brakes_08ax 1.07 -0.87 0.07
## handson_brakes_08bx -0.05 -2.05 0.08
## handson_brakes_08cx 0.57 -1.72 0.08
## handson_brakes_08dx 0.57 -1.72 0.08
## handson_brakes_08ex 1.85 1.44 0.06
## handson_brakes_08fx -0.25 -1.99 0.08
## handson_brakes_08gx -3.91 13.67 0.04
## handson_brakes_08hx -3.91 13.67 0.04
## handson_brakes_09_10ax -3.91 13.67 0.04
## handson_brakes_09_10bx -3.91 13.67 0.04
## handson_brakes_09_10cx -0.05 -2.05 0.08
## handson_brakes_09_10dx -1.40 -0.03 0.07
## handson_brakes_09_10ex 0.35 -1.92 0.08
## handson_brakes_09_10fx -5.77 32.15 0.03
## handson_brakes_09_10gx -3.05 7.52 0.04
## handson_brakes_09_10hx -0.57 -1.72 0.08
## handson_brakes_16ax NaN NaN 0.00
## handson_brakes_16bx -1.40 -0.03 0.07
## handson_brakes_16cx -2.52 4.47 0.05
## handson_brakes_16dx -2.14 2.65 0.05
## handson_brakes_16ex 1.85 1.44 0.06
## handson_brakes_16fx -5.77 32.15 0.03
## handson_brakes_16gx -5.77 32.15 0.03
## handson_brakes_16hx -0.68 -1.58 0.08
## handson_brakes_17_21ax -1.40 -0.03 0.07
## handson_brakes_17_21bx -1.85 1.44 0.06
## handson_brakes_17_21cx -2.52 4.47 0.05
## handson_brakes_17_21dx -0.93 -1.16 0.07
## handson_brakes_17_21ex -1.40 -0.03 0.07
## handson_brakes_17_21fx -0.57 -1.72 0.08
## handson_brakes_17_21gx 0.35 -1.92 0.08
## handson_brakes_17_21hx 0.80 -1.39 0.07
## handson_timing_belt_10ax -5.86 33.15 0.02
## handson_timing_belt_10bx -3.11 7.85 0.04
## handson_timing_belt_10cx -1.89 1.60 0.06
## handson_timing_belt_10dx 0.00 -2.05 0.08
## handson_timing_belt_10ex -0.97 -1.08 0.07
## handson_timing_belt_10fx 0.00 -2.05 0.08
## handson_timing_belt_10gx 0.00 -2.05 0.08
## handson_timing_belt_10hx -0.29 -1.96 0.08
## handson_timing_belt_10ix 1.65 0.73 0.06
## handson_timing_belt_10kx 0.10 -2.04 0.08
## handson_timing_belt_11_12ax 0.29 -1.96 0.08
## handson_timing_belt_11_12bx 0.39 -1.89 0.08
## handson_timing_belt_11_12cx 0.29 -1.96 0.08
## handson_timing_belt_11_12dx 0.84 -1.33 0.07
## handson_timing_belt_11_12ex 0.50 -1.80 0.08
## handson_timing_belt_11_12fx 0.39 -1.89 0.08
## handson_timing_belt_11_12gx 0.39 -1.89 0.08
## handson_timing_belt_11_12hx 0.72 -1.52 0.07
## handson_timing_belt_11_12ix 0.50 -1.80 0.08
## handson_timing_belt_11_12kx 1.11 -0.78 0.07
## handson_timing_belt_14_15ax 0.84 -1.33 0.07
## handson_timing_belt_14_15bx 0.72 -1.52 0.07
## handson_timing_belt_14_15cx 1.44 0.09 0.06
## handson_timing_belt_14_15dx 0.50 -1.80 0.08
## handson_timing_belt_14_15ex 0.50 -1.80 0.08
## handson_timing_belt_14_15fx 0.50 -1.80 0.08
## handson_timing_belt_14_15gx 0.84 -1.33 0.07
## handson_timing_belt_14_15hx 0.84 -1.33 0.07
## handson_timing_belt_14_15ix 0.84 -1.33 0.07
## handson_timing_belt_14_15kx 1.44 0.09 0.06
## handson_timing_belt_14_15lx 1.27 -0.40 0.07
## handson_timing_belt_14_15mx 1.44 0.09 0.06
## handson_timing_belt_14_15nx 2.18 2.84 0.05
## handson_timing_belt_16ax 0.97 -1.08 0.07
## handson_timing_belt_16bx 1.27 -0.40 0.07
## handson_timing_belt_16cx 3.11 7.85 0.04
## handson_timing_belt_16dx 1.11 -0.78 0.07
## handson_timing_belt_16ex 5.86 33.15 0.02
## handson_timing_belt_16fx 2.57 4.71 0.05
## handson_timing_belt_16gx 2.57 4.71 0.05
## handson_timing_belt_16hx 5.86 33.15 0.02
## handson_steering_07ax 0.59 -1.68 0.07
## handson_steering_07bx -1.39 -0.06 0.06
## handson_steering_07cx 1.12 -0.76 0.06
## handson_steering_07dx 1.55 0.42 0.06
## handson_steering_07ex -1.12 -0.76 0.06
## handson_steering_07fx -0.33 -1.93 0.07
## handson_steering_07hx -2.20 2.89 0.05
## handson_steering_07ix 0.50 -1.79 0.07
## handson_steering_07kx -3.50 10.49 0.04
## handson_steering_08ax -1.94 1.82 0.05
## handson_steering_08bx -1.73 1.03 0.05
## handson_steering_08cx -0.59 -1.68 0.07
## handson_steering_08dx 1.39 -0.06 0.06
## handson_steering_08ex -2.92 6.67 0.04
## handson_steering_09ax 2.20 2.89 0.05
## handson_steering_09bx 1.39 -0.06 0.06
## handson_steering_09cx -1.94 1.82 0.05
## handson_steering_09dx -0.78 -1.41 0.07
## handson_steering_09ex 0.08 -2.03 0.07
## handson_steering_11ax -2.20 2.89 0.05
## handson_steering_11bx -2.20 2.89 0.05
## handson_steering_11cx -1.00 -1.02 0.06
## handson_steering_11fx -0.24 -1.98 0.07
## handson_steering_11gx 1.12 -0.76 0.06
## handson_steering_11hx -0.78 -1.41 0.07
## handson_steering_11ix 1.39 -0.06 0.06
## handson_steering_11kx 1.55 0.42 0.06
## handson_steering_11lx 0.16 -2.01 0.07
## handson_steering_11mx 1.94 1.82 0.05
## handson_steering_11nx 0.33 -1.93 0.07
## handson_steering_15ax -0.69 -1.56 0.07
## handson_steering_15bx 0.50 -1.79 0.07
## handson_steering_15cx -0.59 -1.68 0.07
## handson_steering_15dx 0.59 -1.68 0.073.3.4 Sum scales
Display a list of variables, extract protocol items for each of the four stations, and calculate sum scores of correctly performed actions for each station.
3.3.4.1 Display a list of variables
colnames(df.dichotom)## [1] "ID" "type_of_company"
## [3] "school_degree" "cba_lights_05"
## [5] "cba_lights_06" "cba_lights_07"
## [7] "cba_lights_08" "cba_lights_10"
## [9] "cba_lights_11" "cba_lights_12"
## [11] "cba_lights_13" "cba_lights_15"
## [13] "cba_lights_16" "cba_lights_17"
## [15] "cba_lights_18" "cba_lights_19"
## [17] "cba_lights_20" "cba_lights_21"
## [19] "cba_lights_22" "cba_lights_23"
## [21] "cba_brakes_05_old" "cba_brakes_07"
## [23] "cba_brakes_08" "cba_brakes_09"
## [25] "cba_brakes_10" "cba_brakes_11"
## [27] "cba_brakes_12" "cba_brakes_13"
## [29] "cba_brakes_14" "cba_brakes_15"
## [31] "cba_brakes_16" "cba_brakes_18"
## [33] "cba_brakes_19" "cba_brakes_20"
## [35] "cba_brakes_21" "cba_brakes_22"
## [37] "cba_brakes_23" "cba_brakes_24"
## [39] "cba_timing_belt_06" "cba_timing_belt_07"
## [41] "cba_timing_belt_08" "cba_timing_belt_09"
## [43] "cba_timing_belt_10" "cba_timing_belt_11"
## [45] "cba_timing_belt_12" "cba_timing_belt_13"
## [47] "cba_timing_belt_14_old" "cba_timing_belt_16"
## [49] "cba_timing_belt_17" "cba_timing_belt_18"
## [51] "cba_timing_belt_19" "cba_timing_belt_20"
## [53] "cba_timing_belt_21" "cba_timing_belt_22"
## [55] "cba_timing_belt_23" "cba_steering_05"
## [57] "cba_steering_06" "cba_steering_07"
## [59] "cba_steering_08" "cba_steering_09"
## [61] "cba_steering_10" "cba_steering_12"
## [63] "cba_steering_15" "cba_steering_16"
## [65] "cba_steering_17" "cba_steering_18"
## [67] "cba_steering_20" "cba_steering_21"
## [69] "sum_lights" "sum_brakes"
## [71] "sum_timing_belt" "sum_steering"
## [73] "handson_lights_09ax" "handson_lights_09bx"
## [75] "handson_lights_10_11_16ax" "handson_lights_10_11_16bx"
## [77] "handson_lights_10_11_16cx" "handson_lights_10_11_16dx"
## [79] "handson_lights_12ax" "handson_lights_12bx"
## [81] "handson_lights_12cx" "handson_lights_12dx"
## [83] "handson_lights_13_14_15ax" "handson_lights_13_14_15cx"
## [85] "handson_lights_13_14_15dx" "handson_lights_13_14_15ex"
## [87] "handson_lights_13_14_15gx" "handson_lights_13_14_15hx"
## [89] "handson_lights_13_14_15ix" "handson_brakes_06ax"
## [91] "handson_brakes_06bx" "handson_brakes_06cx"
## [93] "handson_brakes_06dx" "handson_brakes_06fx"
## [95] "handson_brakes_06gx" "handson_brakes_06hx"
## [97] "handson_brakes_06ix" "handson_brakes_06kx"
## [99] "handson_brakes_08ax" "handson_brakes_08bx"
## [101] "handson_brakes_08cx" "handson_brakes_08dx"
## [103] "handson_brakes_08ex" "handson_brakes_08fx"
## [105] "handson_brakes_08gx" "handson_brakes_08hx"
## [107] "handson_brakes_09_10ax" "handson_brakes_09_10bx"
## [109] "handson_brakes_09_10cx" "handson_brakes_09_10dx"
## [111] "handson_brakes_09_10ex" "handson_brakes_09_10fx"
## [113] "handson_brakes_09_10gx" "handson_brakes_09_10hx"
## [115] "handson_brakes_16ax" "handson_brakes_16bx"
## [117] "handson_brakes_16cx" "handson_brakes_16dx"
## [119] "handson_brakes_16ex" "handson_brakes_16fx"
## [121] "handson_brakes_16gx" "handson_brakes_16hx"
## [123] "handson_brakes_17_21ax" "handson_brakes_17_21bx"
## [125] "handson_brakes_17_21cx" "handson_brakes_17_21dx"
## [127] "handson_brakes_17_21ex" "handson_brakes_17_21fx"
## [129] "handson_brakes_17_21gx" "handson_brakes_17_21hx"
## [131] "handson_timing_belt_10ax" "handson_timing_belt_10bx"
## [133] "handson_timing_belt_10cx" "handson_timing_belt_10dx"
## [135] "handson_timing_belt_10ex" "handson_timing_belt_10fx"
## [137] "handson_timing_belt_10gx" "handson_timing_belt_10hx"
## [139] "handson_timing_belt_10ix" "handson_timing_belt_10kx"
## [141] "handson_timing_belt_11_12ax" "handson_timing_belt_11_12bx"
## [143] "handson_timing_belt_11_12cx" "handson_timing_belt_11_12dx"
## [145] "handson_timing_belt_11_12ex" "handson_timing_belt_11_12fx"
## [147] "handson_timing_belt_11_12gx" "handson_timing_belt_11_12hx"
## [149] "handson_timing_belt_11_12ix" "handson_timing_belt_11_12kx"
## [151] "handson_timing_belt_14_15ax" "handson_timing_belt_14_15bx"
## [153] "handson_timing_belt_14_15cx" "handson_timing_belt_14_15dx"
## [155] "handson_timing_belt_14_15ex" "handson_timing_belt_14_15fx"
## [157] "handson_timing_belt_14_15gx" "handson_timing_belt_14_15hx"
## [159] "handson_timing_belt_14_15ix" "handson_timing_belt_14_15kx"
## [161] "handson_timing_belt_14_15lx" "handson_timing_belt_14_15mx"
## [163] "handson_timing_belt_14_15nx" "handson_timing_belt_16ax"
## [165] "handson_timing_belt_16bx" "handson_timing_belt_16cx"
## [167] "handson_timing_belt_16dx" "handson_timing_belt_16ex"
## [169] "handson_timing_belt_16fx" "handson_timing_belt_16gx"
## [171] "handson_timing_belt_16hx" "handson_steering_07ax"
## [173] "handson_steering_07bx" "handson_steering_07cx"
## [175] "handson_steering_07dx" "handson_steering_07ex"
## [177] "handson_steering_07fx" "handson_steering_07hx"
## [179] "handson_steering_07ix" "handson_steering_07kx"
## [181] "handson_steering_08ax" "handson_steering_08bx"
## [183] "handson_steering_08cx" "handson_steering_08dx"
## [185] "handson_steering_08ex" "handson_steering_09ax"
## [187] "handson_steering_09bx" "handson_steering_09cx"
## [189] "handson_steering_09dx" "handson_steering_09ex"
## [191] "handson_steering_11ax" "handson_steering_11bx"
## [193] "handson_steering_11cx" "handson_steering_11fx"
## [195] "handson_steering_11gx" "handson_steering_11hx"
## [197] "handson_steering_11ix" "handson_steering_11kx"
## [199] "handson_steering_11lx" "handson_steering_11mx"
## [201] "handson_steering_11nx" "handson_steering_15ax"
## [203] "handson_steering_15bx" "handson_steering_15cx"
## [205] "handson_steering_15dx"
3.3.4.2 Extract protocol items for each of the four stations
# Lights
handson_lights <- colnames(df.dichotom)[grep("handson_lights_" , colnames(df.dichotom))]
handson_lights## [1] "handson_lights_09ax" "handson_lights_09bx"
## [3] "handson_lights_10_11_16ax" "handson_lights_10_11_16bx"
## [5] "handson_lights_10_11_16cx" "handson_lights_10_11_16dx"
## [7] "handson_lights_12ax" "handson_lights_12bx"
## [9] "handson_lights_12cx" "handson_lights_12dx"
## [11] "handson_lights_13_14_15ax" "handson_lights_13_14_15cx"
## [13] "handson_lights_13_14_15dx" "handson_lights_13_14_15ex"
## [15] "handson_lights_13_14_15gx" "handson_lights_13_14_15hx"
## [17] "handson_lights_13_14_15ix"# Brakes
handson_brakes <- colnames(df.dichotom)[grep("handson_brakes_" , colnames(df.dichotom))]
handson_brakes## [1] "handson_brakes_06ax" "handson_brakes_06bx" "handson_brakes_06cx"
## [4] "handson_brakes_06dx" "handson_brakes_06fx" "handson_brakes_06gx"
## [7] "handson_brakes_06hx" "handson_brakes_06ix" "handson_brakes_06kx"
## [10] "handson_brakes_08ax" "handson_brakes_08bx" "handson_brakes_08cx"
## [13] "handson_brakes_08dx" "handson_brakes_08ex" "handson_brakes_08fx"
## [16] "handson_brakes_08gx" "handson_brakes_08hx" "handson_brakes_09_10ax"
## [19] "handson_brakes_09_10bx" "handson_brakes_09_10cx" "handson_brakes_09_10dx"
## [22] "handson_brakes_09_10ex" "handson_brakes_09_10fx" "handson_brakes_09_10gx"
## [25] "handson_brakes_09_10hx" "handson_brakes_16ax" "handson_brakes_16bx"
## [28] "handson_brakes_16cx" "handson_brakes_16dx" "handson_brakes_16ex"
## [31] "handson_brakes_16fx" "handson_brakes_16gx" "handson_brakes_16hx"
## [34] "handson_brakes_17_21ax" "handson_brakes_17_21bx" "handson_brakes_17_21cx"
## [37] "handson_brakes_17_21dx" "handson_brakes_17_21ex" "handson_brakes_17_21fx"
## [40] "handson_brakes_17_21gx" "handson_brakes_17_21hx"# Timing belt
handson_timing_belt <- colnames(df.dichotom)[grep("handson_timing_belt_" , colnames(df.dichotom))]
handson_timing_belt## [1] "handson_timing_belt_10ax" "handson_timing_belt_10bx"
## [3] "handson_timing_belt_10cx" "handson_timing_belt_10dx"
## [5] "handson_timing_belt_10ex" "handson_timing_belt_10fx"
## [7] "handson_timing_belt_10gx" "handson_timing_belt_10hx"
## [9] "handson_timing_belt_10ix" "handson_timing_belt_10kx"
## [11] "handson_timing_belt_11_12ax" "handson_timing_belt_11_12bx"
## [13] "handson_timing_belt_11_12cx" "handson_timing_belt_11_12dx"
## [15] "handson_timing_belt_11_12ex" "handson_timing_belt_11_12fx"
## [17] "handson_timing_belt_11_12gx" "handson_timing_belt_11_12hx"
## [19] "handson_timing_belt_11_12ix" "handson_timing_belt_11_12kx"
## [21] "handson_timing_belt_14_15ax" "handson_timing_belt_14_15bx"
## [23] "handson_timing_belt_14_15cx" "handson_timing_belt_14_15dx"
## [25] "handson_timing_belt_14_15ex" "handson_timing_belt_14_15fx"
## [27] "handson_timing_belt_14_15gx" "handson_timing_belt_14_15hx"
## [29] "handson_timing_belt_14_15ix" "handson_timing_belt_14_15kx"
## [31] "handson_timing_belt_14_15lx" "handson_timing_belt_14_15mx"
## [33] "handson_timing_belt_14_15nx" "handson_timing_belt_16ax"
## [35] "handson_timing_belt_16bx" "handson_timing_belt_16cx"
## [37] "handson_timing_belt_16dx" "handson_timing_belt_16ex"
## [39] "handson_timing_belt_16fx" "handson_timing_belt_16gx"
## [41] "handson_timing_belt_16hx"# Steering and front axle
handson_steering <- colnames(df.dichotom)[grep("handson_steering_" , colnames(df.dichotom))]
handson_steering## [1] "handson_steering_07ax" "handson_steering_07bx" "handson_steering_07cx"
## [4] "handson_steering_07dx" "handson_steering_07ex" "handson_steering_07fx"
## [7] "handson_steering_07hx" "handson_steering_07ix" "handson_steering_07kx"
## [10] "handson_steering_08ax" "handson_steering_08bx" "handson_steering_08cx"
## [13] "handson_steering_08dx" "handson_steering_08ex" "handson_steering_09ax"
## [16] "handson_steering_09bx" "handson_steering_09cx" "handson_steering_09dx"
## [19] "handson_steering_09ex" "handson_steering_11ax" "handson_steering_11bx"
## [22] "handson_steering_11cx" "handson_steering_11fx" "handson_steering_11gx"
## [25] "handson_steering_11hx" "handson_steering_11ix" "handson_steering_11kx"
## [28] "handson_steering_11lx" "handson_steering_11mx" "handson_steering_11nx"
## [31] "handson_steering_15ax" "handson_steering_15bx" "handson_steering_15cx"
## [34] "handson_steering_15dx"
3.3.4.3 Calculate sum scores of correctly performed actions for each station
For each hands-on station, we calculate raw scores (i.e., the sum of correctly performed actions) and display a descriptive statistic.
Note. As the hands-on protocols differ in length (i.e., number of actions for the tasks), the raw scores cannot be used to compare the difficulty of the stations.
# Lights
df.dichotom$sum_handson_lights <- rowSums(df.dichotom[,handson_lights] , na.rm = FALSE)
psych::describe(df.dichotom$sum_handson_lights)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 48 5.58 2.28 5 5.53 2.97 1 10 9 0.23 -0.97 0.33# Brakes
df.dichotom$sum_handson_brakes <- rowSums(df.dichotom[,handson_brakes] , na.rm = FALSE)
psych::describe(df.dichotom$sum_handson_brakes)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 39 27.82 4.11 28 27.85 5.93 19 36 17 0 -0.91 0.66# Timing belt
df.dichotom$sum_handson_timing_belt <- rowSums(df.dichotom[,handson_timing_belt] , na.rm = FALSE)
psych::describe(df.dichotom$sum_handson_timing_belt)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 40 14.55 12.98 7 13.38 7.41 0 39 39 0.58 -1.35 2.05# Steering and front axle
df.dichotom$sum_handson_steering <- rowSums(df.dichotom[,handson_steering] , na.rm = FALSE)
psych::describe(df.dichotom$sum_handson_steering)## vars n mean sd median trimmed mad min max range skew kurtosis se
## X1 1 48 18.35 5.81 19 18.85 5.19 3 27 24 -0.74 0.09 0.843.3.5 Scale reliability
We calculate two coefficients of internal consistency: Cronbach’s Alpha and Guttman’s Lambda-6 (see section 3.2.9 for details).
Note. The code below will produce warnings about some items which are negatively correlated with the scales and probably should be reversed. These messages must be ignored as reversed items (i.e., incorrect actions) are already recoded.
Station 1 - Lighting system
reli_handson_lights <- psych::alpha(df.dichotom[,handson_lights] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_handson_lights$total$raw_alpha ## [1] 0.4852834reli_handson_lights$total$`G6(smc)` ## [1] 0.7063037Station 2 - Hydraulic brakes
reli_handson_brakes <- psych::alpha(df.dichotom[,handson_brakes] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_handson_brakes$total$raw_alpha ## [1] 0.6810014reli_handson_brakes$total$`G6(smc)` ## [1] 0.9888014Station 3 - Timing belt
reli_handson_timing_belt <- psych::alpha(df.dichotom[,handson_timing_belt] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_handson_timing_belt$total$raw_alpha ## [1] 0.9782516reli_handson_timing_belt$total$`G6(smc)` ## [1] 0.9844092Station 4 - Steering and front axle
reli_handson_steering <- psych::alpha(df.dichotom[,handson_steering] ,
check.keys = FALSE)Display Cronbach’s alpha and Guttman’s lambda-6.
reli_handson_steering$total$raw_alpha ## [1] 0.840987reli_handson_steering$total$`G6(smc)` ## [1] 0.96284043.4 Correlations
To judge how well the CBA test scores predict hands-on performance (and vice versa), we calculate correlations between the CBA booklets and the corresponding hands-on stations.
3.4.1 Visual tests
Note. This code will produce messages and warnings related to missing data, which can be ignored.
Lighting system
library("ggplot2")
ggplot(df.dichotom, aes(sum_lights , sum_handson_lights)) +
geom_point(position = position_jitter(w = 0.03, h = 0.03) ,
alpha = 0.5) +
scale_x_continuous(name = "Video-based assessment “lighting system”\n(total score)") +
scale_y_continuous(name = "Hands-on assessment “lighting system”\n(total score)") +
stat_smooth(method = 'loess' , se = TRUE ,
linewidth = 0.3) +
stat_smooth(method = "lm" , se = FALSE ,
linewidth = 0.3 ,
linetype = "dashed") 
Hydraulic brakes
ggplot(df.dichotom, aes(sum_brakes , sum_handson_brakes)) +
geom_point(position = position_jitter(w = 0.03, h = 0.03) ,
alpha = 0.5) +
scale_x_continuous(name = "Video-based assessment “hydraulic brakes”\n(total score)") +
scale_y_continuous(name = "Hands-on assessment “hydraulic brakes”\n(total score)") +
stat_smooth(method = 'loess' , se = TRUE ,
linewidth = 0.3) +
stat_smooth(method = "lm" , se = FALSE ,
linewidth = 0.3 ,
linetype = "dashed") 
Timing belt
ggplot(df.dichotom, aes(sum_timing_belt , sum_handson_timing_belt)) +
geom_point(position = position_jitter(w = 0.03, h = 0.03) ,
alpha = 0.5) +
scale_x_continuous(name = "Video-based assessment “timing belt”\n(total score)") +
scale_y_continuous(name = "Hands-on assessment “timing belt”\n(total score)") +
stat_smooth(method = 'loess' , se = TRUE ,
linewidth = 0.3) +
stat_smooth(method = "lm" , se = FALSE ,
linewidth = 0.3 ,
linetype = "dashed") 
Steering and front axle
ggplot(df.dichotom, aes(sum_steering , sum_handson_steering)) +
geom_point(position = position_jitter(w = 0.03, h = 0.03) ,
alpha = 0.5) +
scale_x_continuous(name = "Video-based assessment “steering and front axle”\n(total score)") +
scale_y_continuous(name = "Hands-on assessment “steering and front axle”\n(total score)") +
stat_smooth(method = 'loess' , se = TRUE ,
linewidth = 0.3) +
stat_smooth(method = "lm" , se = FALSE ,
linewidth = 0.3 ,
linetype = "dashed") 
3.4.2 Inferential statistic
We use Pearson’s correlation (and the t-test to investigate significance) as it has good power to detect any effects and is rather insensitive to violations of the normality and measurement scale assumptions (see Havlicek & Peterson, 1976). However, Pearson’s r is a test for linear relationships (which we hypothesised and therefore test for). The scatterplots above indicate a non-linear relationship for one of our tests. Pearson’s r will only test for the linear component of that relationship (i.e. the dotted line in the plot) which in this case is near zero.
Note. To calculate non-parametric
correlations, change method = "pearson" to
method = "spearman" or method = "kendall" in
the below code.
cor.test(df.dichotom$sum_lights ,
df.dichotom$sum_handson_lights ,
method = "pearson" ,
alternative = "greater")##
## Pearson's product-moment correlation
##
## data: df.dichotom$sum_lights and df.dichotom$sum_handson_lights
## t = 1.3087, df = 45, p-value = 0.09864
## alternative hypothesis: true correlation is greater than 0
## 95 percent confidence interval:
## -0.05404939 1.00000000
## sample estimates:
## cor
## 0.191476cor.test(df.dichotom$sum_brakes ,
df.dichotom$sum_handson_brakes ,
method = "pearson" ,
alternative = "greater")##
## Pearson's product-moment correlation
##
## data: df.dichotom$sum_brakes and df.dichotom$sum_handson_brakes
## t = 1.0307, df = 37, p-value = 0.1547
## alternative hypothesis: true correlation is greater than 0
## 95 percent confidence interval:
## -0.1051041 1.0000000
## sample estimates:
## cor
## 0.1670677cor.test(df.dichotom$sum_timing_belt ,
df.dichotom$sum_handson_timing_belt ,
method = "pearson" ,
alternative = "greater")##
## Pearson's product-moment correlation
##
## data: df.dichotom$sum_timing_belt and df.dichotom$sum_handson_timing_belt
## t = 2.3609, df = 38, p-value = 0.01173
## alternative hypothesis: true correlation is greater than 0
## 95 percent confidence interval:
## 0.1034079 1.0000000
## sample estimates:
## cor
## 0.3576522cor.test(df.dichotom$sum_steering ,
df.dichotom$sum_handson_steering ,
method = "pearson" ,
alternative = "greater")##
## Pearson's product-moment correlation
##
## data: df.dichotom$sum_steering and df.dichotom$sum_handson_steering
## t = 0.32818, df = 46, p-value = 0.3721
## alternative hypothesis: true correlation is greater than 0
## 95 percent confidence interval:
## -0.1943283 1.0000000
## sample estimates:
## cor
## 0.048331143.4.3 Discussion
We investigated the ecological validity of performance measures from a computer-based assessment (CBA) that utilises scripted video vignettes. We hypothesised the ability to correctly judge video-taped repair actions on a car is a good predictor of the ability to carry out such repairs in real-life scenarios. The correlations between video judgements and hands-on performance were substantially lower than expected, indicating low to insufficient ecological validity of the test score interpretations. Furthermore, the internal consistency of the hands-on stations was much higher than the internal consistency of the CBA, suggesting standardised observations of hands-on performance are a more reliable measure than our CBA test scores.
To judge validity, it is recommended to evaluate evidence from various sources (AERA, APA, & NCME, 2014). In our project, the validity evidence from this hands-on validation study is complemented by evidence from a known-groups study (Hartmann, Güzel, & Gschwendtner, 2023a). Setting the findings of the two studies into relation, we were able to identify both strengths and limitations of our video-based test (Hartmann, Güzel, & Gschwendtner, 2023a,b). We advise potential test users to inform themselves about these limitations before using the instrument.
4 References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME] (2014). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.
Field, A., Miles, J., & Field, Z. (2012). Discovering statistics using R. London: Sage.
Gouvier W., Barker A., & Musso, M. (2010). Ecological validity. In N.J. Salkind (ed.), Encyclopedia of research design. SAGE, London, pp. 399–404. https://doi.org/10.4135/9781412961288.n126
Hartmann, S., Güzel, E., & Gschwendtner, T. (2023a). Berufsbezogene Fähigkeiten digital messen: Herausforderungen, Möglichkeiten und Grenzen am Beispiel videovignettenbasierter Prüfungsaufgaben für das Kfz-Handwerk. Berufsbildung in Wissenschaft und Praxis, 52, 26-30. https://nbn-resolving.org/urn:nbn:de:0035-bwp-23326-2
Hartmann, S., Güzel, E., & Gschwendtner, T. (2023b). Digital measurement of hands-on performance? Ecological validation of a computer-based assessment of automotive repair skills. Empirical Research in Vocational Education and Training, 15(15). https://doi.org/10.1186/s40461-023-00153-x
Havlicek, L. L., & Peterson, N. L. (1976). Robustness of the Pearson correlation against violations of assumptions. Perceptual and Motor Skills, 43(3), 1319-1334. https://doi.org/10.2466/pms.1976.43.3f.1