2) Estimating Consequences for Subsequent Analyses
Florian Berding and Julia Pargmann
2022-12-22
Source:vignettes/cons_and_values.Rmd
cons_and_values.Rmd1 Introduction
In the first vignette Get started, rules
of thumb are used to evaluate the degree of reliability. In this
vignette, we would like to demonstrate that a more situation-specific
evaluation of reliability is possible. In doing so, we continue the
example of the first vignette, which you can find here. The focus of this kind of analysis is the
function get_consequences(), which can be used to evaluate
pretests of coding schemes, as well as for planning or evaluating
existing studies.
The cut-off values for judging reliability are derived in a way that ensures a high data quality in many practical situations. In the study of Berding and Pargmann (2022), the most demanding situations are chosen. Thus, for some applications, these rules of thumb may be too strict. Additionally, it may be helpful to understand the consequences induced by specific reliability values.
To illustrate this kind of analysis, we need to extend our example by introducing a new variable. Let us assume that all participants of the exam were interviewed in order to assess how confident the students are about their abilities to successfully pass the exam. Let us further assume that the degree of confidence varies between “low”, “medium” and “high”.
In analogy to studies investigating the relationship between achievement and self-concept (Huang 2011; Möller et al. 2020), we assume that participants with better exams have higher confidence in their abilities. Figure 1 illustrates this relationship.
Figure 1: Assumed Relationship between Performance and Ability Self Concept
To prove this relationship, we use the data from the exams and the
rated interviews. Whether or not we can draw the right conclusions from
the data depends on the reliability of the generated data, the sample
size and the sample method. This can be explored in more detail with the
function get_consequences()
2 Using get_consequences()
The function get_consequences() provides information
about the impact of reliability on significance testing and drawing
conclusions. It requires at least four arguments. With
measure_typ you can decide which measure of reliability you
would like to use on the scale level. Currently, we recommend to use
measure_typ = "dynamic_iota_index" because in our analysis,
this measure showed the highest value for R² in predicting
different kinds of data quality.
With measure_1_val you can set the reliability of the
independent variable. In our example, this is the performance a
participant shows in their exam. Here, the value for Dynamic Iota
Index is about .267 (see Get started).
With measure_2_val you can set the reliability of the
dependent variable. If this value is not set explicitly, the analysis
assumes the same reliability as for the independent variable. In our
example, the dependent variable refers to the data of the interviews
showing participants’ confidence in their abilities. Let us assume that
the corresponding value of Dynamic Iota Index is about
.879.
The argument data_type sets the scale level. Currently,
“nominal” and “ordinal” are possible. In the case of nominal data, all
results of the function refer to significance tests with Cramer’s
V. In the case of ordinal data, all statistics refer to
significance tests with Kendall’s Tau. In our example, both
scales form an ordinal scale.
The argument strength is closely connected to the data
type and refers to the true strength of a relationship
between the independent and dependent variable. The argument can be set
to “no”, “weak”, “medium” and “strong”. These categories are based on
the work of Cohen (1988), who classified statistical measures according
to their relevance for real-world applications. Thus,“no” does not imply
a value of 0 for Kendall’s Tau, but a small value around 0.
“Strong” does not imply a perfect relationship, but rather refers to
values above .5. Cohen’s (1988) work does not explicitly deal with
Kendall’s Tau but instead employs Pearson correlation. For
Cramer’s V, the situation is more complicated, as the class of
the effect size depends on the number of categories. However, this is
considered in the function. In our example, we assume a medium
relationship between performance and confidence.
The argument sample_size refers to the sample size of a
planned or already realized study. In our example, three raters judged
the written exams of 318 participants and analyzed their corresponding
interviews.
Finally, level refers to the certainty level of the
calculated prediction intervals. A prediction interval characterizes the
probability that the true value is within a specific range around the
prediction (Afifi et al. 2020, p. 119). In the current example we choose
95%.
library(iotarelr)
#> Loading required package: ggplot2
#> Warning: package 'ggplot2' was built under R version 4.1.3
#> Loading required package: ggalluvial
#> Warning: package 'ggalluvial' was built under R version 4.1.3
get_consequences(measure_typ = "dynamic_iota_index",
measure_1_val = .267,
measure_2_val = .879,
data_type = "ordinal",
strength = "medium",
sample_size=318,
level = 0.95)
#> lower 0.95 % mean upper 0.95 % practically no effect
#> deviation 0.107 0.260 0.413 0.020
#> classification rate 0.002 0.009 0.028 0.000
#> risk of Type I errors 0.064 0.215 0.479 0.011
#> practically weak effect
#> deviation 0.695
#> classification rate 0.000
#> risk of Type I errors 0.095The function calculates three important aspects when investigating relationships.
Deviation: The first row called “deviation” characterizes the expected deviation between the estimated sample effect size and the true sample effect size. Since we are using ordinal data, the effect size is Kendall’s Tau. The mean value implies that we expect that Kendall’s Tau differs from an error-free assessment by about .260 units. This is quite high if we use Cohen’s (1988) classification for Pearson correlation. Here, the meaning of a correlation changes every .20 units. The upper and lower values mean that with a certainty of 95%, the estimated value for Kendall’s Tau differs from an error-free estimation by .107 to .413 units.
The column “practically no effect” reports the probability that the effect size does not deviate more than .1 units. With a reliability of .267 for the exams and of .879 for the interviews, this chance is about 2%, which is very low. The last column, “practically weak effect”, reports the probability that Kendall’s Tau deviates less than .3 units from an error-free measurement. This probability is about 70%. At first glance, this seems to be very high. However, this probability implies that in nearly one third of all studies with the same research design, the values deviate by more than .3 units.
Classification rate: The second row is closely connected to the deviation. It describes the chance to correctly classify an effect size as practically not relevant, weak, medium or strong, based on Cohen’s (1988) classification. The mean value implies that the chance to correctly classify the effect size is about .9%, which is very low. The upper and lower values state that with a certainty of 95%, the chance to correctly classify the effect size is between .2 and 2.8%. It becomes clear that with the current reliability of .267 for the exams and .879 for the interviews, the risk of drawing the wrong conclusion about the strength of the relationship is immense.
The column “practically no effect” reports the probability that the chance to correctly classify the effect size is at least 95%, while the last column reports that the probability to correctly classify the effect size is at least 90%. In both cases the probability is zero.
Type I errors: The last row refers to the risk of Type I errors. Type I errors mean in this context that the significance test implies the acceptance of the null hypothesis, while an error-free measurement would imply the rejection of the null hypothesis. In other words: The results of the significance test imply that there is no relationship, although an error-free measurement would imply the acceptance of a relationship.
The mean value implies that we have to expect a chance of 21.5% of a Type I error. That is, in 21.5% of cases, the results of the significance tests imply that there is no relationship between participants’ performance in the exam and the confidence in their abilities, although an error-free measurement would imply the existence of such a relationship. The lower and upper values imply that with a certainty of 95%, the risk for Type I errors is between 6.4% and 47.9%. Thus, there is a high risk for drawing the wrong conclusions.
The column “practically no effect” reports the probability that the risk of Type I errors does not exceed 5% while the last column reports the probability that the risk of Type I errors does not exceed 10%. In the current example, the probability for no relevant effect is about 1.1 % and for a weak effect about 9.5%.
Summing up, the information provided by
get_consequences() helps to make the suggested cut-off
values more specific. The information can be used to judge the degree of
reliability in a more situation-specifical manner and can support both
the planning and the evaluation of studies.
3 Limitations
It is important to note that this analysis has some limitations. First, only two types of significance tests are supported (Cramer’s V and Kendall’s Tau). Second, the analysis assumes that the independent and dependent variables both have the same number of categories. Thus, please use the results as an orientation.
References
- Afifi, A., May, S., Donatello, R. A. & Clark, V. A. (2020). Practical Multivariate Analysis (6th ed.). Chapman and Hall/ CRC Texts in Statistical Science Ser. CRC Press LLC.
- Huang, C. (2011). Self-concept and academic achievement: a meta-analysis of longitudinal relations. Journal of school psychology, 49(5), 505–528. https://doi.org/10.1016/j.jsp.2011.07.001
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd Ed.). Taylor & Francis.
- Möller, J., Zitzmann, S., Helm, F., Machts, N. & Wolff, F. (2020). A Meta-Analysis of Relations Between Achievement and Self-Concept. Review of Educational Research, 90(3), 376–419. https://doi.org/10.3102/0034654320919354