Ordinal data: Krippendorff alpha inter-rater reliability test


How do use a statistical test, the Krippendorff alpha, to check the inter-rater reliability of a variable with ordinal data. In this example, six observers have rated 30 students. The question was: “How would you rate this individual student?” Excellent was coded with the digit 1. Above average 2. Average 3. Below average was coded with a 4. Here, in this example, I have six judges, and there are no missing data. However, this statistical test can be used with any number of judges, with or without missing data. In SPSS, click “File”, “Open”, and “Syntax”. Open the macro “kalpha.sps”. If you don’t have this special file, please see my previous video entitled “Nominal Dichotomous Yes/No Data”, where I show you how to find and download it. Execute this macro, and run the statistical tests by clicking “File”, “New”, and “Syntax”. Then type kalpha judges equal teachers one two three four five and six slash level equal 2… Here, 2 indicates that we’re dealing with ordinal data. … slash detail equals 0 slash boot equals ten thousand. Then “Run”, “All”. Here, the Krippendorff alpha reliability estimate is 0.6159. An alpha below 0.67 indicates a really low inter-rater reliability. It’s crappy, in other words. Ideally it should be over 0.8. Below 0.8 but above 0.67 indicates low reliability. The table shows that there is an estimated 70.01 percent chance that the alpha value would be below 0.67 if the whole population would be tested. Now, a Krippendorff alpha of just 0.6159 is perhaps too low to be used in a report but I still want to include an example here just to show you how these types of results are written out. Let’s do it again with another set of example data. Here we have four observers rating 12 individuals. There are missing data but that’s no problem for the Krippendorff alpha test. In fact, this is one great advantage compared to other statistical tests. “File”, “New”, and “Syntax”. Kalpha judges equals obsa b, c, and d slash level equals 2 slash detail equals zero slash boot equals ten. thousand. The Krippendorff alpha reliability estimate here is 0.8095, which is better. There’s a 6.46 percent chance that the alpha would be below 0.67 if the whole population would be tested. Here’s an example of how it can be reported in text. Thank you very much for watching my videos! Bye.

4 Comments

  1. Hi Kent. Thank you for the video. I was wondering if you could help me. I have 4 judges and 23 items. I was wondering if you could assist me in analysing the inter-rater reliability for the judges on each item. Is this possible? Have you done a video on this?

    Also, the macro that I download does not seem to be working. The output file says this:
    print/title = "ERROR: Input Reliability Data Matrix Exhibits No Variation.".

    >Error # 4285 in column 7. Text: title
    >Incorrect variable name: either the name is more than 64 characters, or it is
    >not defined by a previous command.
    >Execution of this command stops.

    I am working on SPSS 20

Leave a Reply

Your email address will not be published. Required fields are marked *