How do use a statistical test, the Krippendorff alpha, to check the inter-rater reliability of a variable with ordinal data. In this example, six observers have rated 30 students. The question was: “How would you rate this individual student?” Excellent was coded with the digit 1. Above average 2. Average 3. Below average was coded with a 4. Here, in this example, I have six judges, and there are no missing data. However, this statistical test can be used with any number of judges, with or without missing data. In SPSS, click “File”, “Open”, and “Syntax”. Open the macro “kalpha.sps”. If you don’t have this special file, please see my previous video entitled “Nominal Dichotomous Yes/No Data”, where I show you how to find and download it. Execute this macro, and run the statistical tests by clicking “File”, “New”, and “Syntax”. Then type kalpha judges equal teachers one two three four five and six slash level equal 2… Here, 2 indicates that we’re dealing with ordinal data. … slash detail equals 0 slash boot equals ten thousand. Then “Run”, “All”. Here, the Krippendorff alpha reliability estimate is 0.6159. An alpha below 0.67 indicates a really low inter-rater reliability. It’s crappy, in other words. Ideally it should be over 0.8. Below 0.8 but above 0.67 indicates low reliability. The table shows that there is an estimated 70.01 percent chance that the alpha value would be below 0.67 if the whole population would be tested. Now, a Krippendorff alpha of just 0.6159 is perhaps too low to be used in a report but I still want to include an example here just to show you how these types of results are written out. Let’s do it again with another set of example data. Here we have four observers rating 12 individuals. There are missing data but that’s no problem for the Krippendorff alpha test. In fact, this is one great advantage compared to other statistical tests. “File”, “New”, and “Syntax”. Kalpha judges equals obsa b, c, and d slash level equals 2 slash detail equals zero slash boot equals ten. thousand. The Krippendorff alpha reliability estimate here is 0.8095, which is better. There’s a 6.46 percent chance that the alpha would be below 0.67 if the whole population would be tested. Here’s an example of how it can be reported in text. Thank you very much for watching my videos! Bye.