IOA calculator for all nine interobserver agreement methods

Interobserver agreement (IOA) is the percentage of agreement between two observers who watched the same session and recorded the same behavior independently. In ABA, IOA is the standard check that data are believable: if two trained observers using the same operational definition get very different numbers, neither set of data is solid enough for a treatment decision.

IOA doesn’t prove the data are accurate. Two observers can agree and both be wrong. What it shows is that the definition is clear enough and the observers are consistent enough that the numbers are believable. Low agreement is usually a sign that the definition is vague, that one observer drifted, or that the measurement system doesn’t fit the behavior.

A common rule is that IOA should be at least 80%, and many published studies aim for 90% or higher. IOA is usually collected in at least 20% of sessions, spread across every phase and condition, not bunched at the start.

Studying for the RBT? Measurement is its own domain on the exam, Data Collection and Graphing, and RBTs are often the second observer in an IOA check. The free RBT practice test has a full set of data collection questions with every answer explained.

IOA for event recording compares two observers’ counts of the same behavior. There are three formulas: total count IOA, mean count-per-interval IOA and exact count-per-interval IOA. Total count is the most lenient, exact count-per-interval the strictest. All three divide agreement by the possible agreement and multiply by 100 to give a percentage.

Use these when observers counted how many times the behavior happened, for example on a frequency sheet from the free ABA data sheet generator. Splitting the session into short intervals and comparing counts per interval is stricter than comparing the session totals, because totals can match while the observers counted different responses.

• Total count IOA = smaller total count ÷ larger total count × 100. Observer 1 counts 9 hits, observer 2 counts 10: 9 ÷ 10 × 100 = 90%.
• Mean count-per-interval IOA = sum of (smaller ÷ larger count in each interval) ÷ number of intervals × 100. An interval where both observers counted 0 counts as full agreement (1).
• Exact count-per-interval IOA = intervals with the exact same count ÷ total intervals × 100. The strictest of the three: a count of 2 against 3 earns nothing.

Trial-by-trial IOA and the three interval IOA methods all compare two observers item by item: each trial or interval is scored as an agreement or a disagreement. Trial-by-trial IOA fits discrete trial teaching. Interval-by-interval, scored-interval and unscored-interval IOA fit partial interval, whole interval and momentary time sampling data.

Use trial-by-trial IOA for discrete trial teaching, where each trial is scored correct or incorrect. Use one of the interval methods for partial interval, whole interval or momentary time sampling data. Which interval method you report depends on how often the behavior happens.

• Trial-by-trial IOA = trials scored the same ÷ total trials × 100.
• Interval-by-interval IOA = intervals agreed ÷ (intervals agreed + intervals disagreed) × 100. Agreement on an empty interval counts, which inflates IOA when the behavior is rare.
• Scored-interval IOA = intervals both observers scored ÷ intervals either observer scored × 100. Use it for low-rate behavior, so a long run of empty intervals can’t hide disagreement.
• Unscored-interval IOA = intervals both observers left unscored ÷ intervals either observer left unscored × 100. Use it for high-rate behavior, the mirror image of scored-interval IOA.

Duration IOA compares how long two observers timed the same behavior. Total duration IOA divides the shorter total by the longer total. Mean duration-per-occurrence IOA compares each episode on its own and averages the ratios, so it is the more demanding of the two and the better choice when episodes vary in length.

• Total duration IOA = shorter total duration ÷ longer total duration × 100. 2 minutes against 2 minutes 30 seconds: 120 ÷ 150 × 100 = 80%.
• Mean duration-per-occurrence IOA = sum of (shorter ÷ longer duration of each occurrence) ÷ number of occurrences × 100.

Low IOA, below about 80%, usually means a vague operational definition, observer drift or a measurement system that doesn’t fit the behavior. The fix is to find where the observers disagreed, tighten the definition, retrain both observers on the same examples and collect IOA again before the data drive any decision.

Start with the table under the result and look at where the disagreements fall. If they cluster around one part of the session, the observers may have lost sight of the learner or started their timers at different moments. If they are spread evenly, the definition is the more likely culprit.

Tighten the operational definition with examples and non-examples (the data sheets page walks through how to write one), practice with a video until both observers agree, and run IOA again. Report the IOA for each phase separately so a reader can see that agreement held up throughout, and show the phases on the graph with the free ABA graph maker.