Academic Journal
View in EDS

Achieving Inter-Rater Agreement and Inter-Rater Reliability to Assess Fidelity of an Occupation-Based Coaching (OBC) Clinical Trial Intervention.

Bibliographic Details
Title: Achieving Inter-Rater Agreement and Inter-Rater Reliability to Assess Fidelity of an Occupation-Based Coaching (OBC) Clinical Trial Intervention.
Authors: Abbott, Amy Ann, Shin, Julia, Carlson, Kathryn, Russell, Marion, Qi, Yongyue, Storm, Hannah, Jewell, Vanessa Dawn
Source: British Journal of Occupational Therapy; Mar2025, Vol. 88 Issue 3, p133-141, 9p
Subject Terms: FAMILIES & psychology, MEDICAL protocols, TYPE 1 diabetes, SELF-efficacy, RESEARCH funding, CRONBACH'S alpha, PILOT projects, EVALUATION of medical care, TEACHING methods, DESCRIPTIVE statistics, OCCUPATIONAL therapy, TELEMEDICINE, CAREGIVERS, CHRONIC diseases, QUALITY of life, DATA analysis software, INTER-observer reliability, VIDEO recording
Abstract: Introduction: Establishing inter-rater agreement and reliability ascertains that multiple raters consistently evaluate observed interventions to ensure that clinical research protocols are delivered as intended by the trial protocol. Purpose: Using the Guidelines for Reporting Reliability and Agreement Studies, we (a) exemplified the steps to establish inter-rater reliability and inter-rater agreement on the occupation-based coaching Video Evaluation Tool and (b) evaluated best practices that promoted high inter-rater reliability and inter-rater agreement between blinded raters prior to starting a pilot randomized controlled trial. The randomized controlled trial examined the preliminary effectiveness of occupation-based coaching via telehealth for rural families with children living with type 1 diabetes to improve family quality of life, participation, self-efficacy, and child health outcomes. Method: We created a library of 13 occupation-based coaching videos portraying a range of evaluations, scores, and ratings. The inter-rater agreement and reliability on the occupation-based coaching Video Evaluation Tool were established through the iterations of (a) blinded rater training, (b) data collection using the tool, and (c) statistical analysis using Cohen's kappa and Cronbach's alpha. Findings: Occurrence and Non-Occurrence Checklist (κ = 0.881, p < 0.001); "Caregiver Talk" and "Interventionist Talk Analysis" (ICC = 0.991–0.999, p < 0.001); Evidence of Independent Capacity Rating (ICC = 0.867 p = 0.006). Conclusion: Strong inter-rater reliability and inter-rater agreement was established by engaging two blinded raters through multifaceted training, integrating real-life clients and contexts into the instrumentation and training, and precisely defined rubric criteria. By employing such practices, high inter-rater reliability and agreement can be achieved in clinical research involving interventions and instruments that are highly subjective and individualized. To ascertain greater scientific confidence in the intervention effect, developing a multidomain fidelity framework and establishing high inter-rater agreement and reliability in the instruments a priori to implementation of clinical trials are necessary. [ABSTRACT FROM AUTHOR]
Copyright of British Journal of Occupational Therapy is the property of Sage Publications Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Complementary Index
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edb&genre=article&issn=03080226&ISBN=&volume=88&issue=3&date=20250301&spage=133&pages=133-141&title=British Journal of Occupational Therapy&atitle=Achieving%20Inter-Rater%20Agreement%20and%20Inter-Rater%20Reliability%20to%20Assess%20Fidelity%20of%20an%20Occupation-Based%20Coaching%20%28OBC%29%20Clinical%20Trial%20Intervention.&aulast=Abbott%2C%20Amy%20Ann&id=DOI:10.1177/03080226241283292
    Name: Full Text Finder (for New FTF UI) (s8985755)
    Category: fullText
    Text: Find It @ SCU Libraries
    MouseOverText: Find It @ SCU Libraries
Header DbId: edb
DbLabel: Complementary Index
An: 183370650
RelevancyScore: 1082
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 1082.12561035156
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Achieving Inter-Rater Agreement and Inter-Rater Reliability to Assess Fidelity of an Occupation-Based Coaching (OBC) Clinical Trial Intervention.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Abbott%2C+Amy+Ann%22&quot;&gt;Abbott, Amy Ann&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Shin%2C+Julia%22&quot;&gt;Shin, Julia&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Carlson%2C+Kathryn%22&quot;&gt;Carlson, Kathryn&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Russell%2C+Marion%22&quot;&gt;Russell, Marion&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Qi%2C+Yongyue%22&quot;&gt;Qi, Yongyue&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Storm%2C+Hannah%22&quot;&gt;Storm, Hannah&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Jewell%2C+Vanessa+Dawn%22&quot;&gt;Jewell, Vanessa Dawn&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: British Journal of Occupational Therapy; Mar2025, Vol. 88 Issue 3, p133-141, 9p
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22FAMILIES+%26+psychology%22&quot;&gt;FAMILIES &amp; psychology&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22MEDICAL+protocols%22&quot;&gt;MEDICAL protocols&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22TYPE+1+diabetes%22&quot;&gt;TYPE 1 diabetes&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22SELF-efficacy%22&quot;&gt;SELF-efficacy&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22RESEARCH+funding%22&quot;&gt;RESEARCH funding&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22CRONBACH&#39;S+alpha%22&quot;&gt;CRONBACH&#39;S alpha&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22PILOT+projects%22&quot;&gt;PILOT projects&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22EVALUATION+of+medical+care%22&quot;&gt;EVALUATION of medical care&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22TEACHING+methods%22&quot;&gt;TEACHING methods&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22DESCRIPTIVE+statistics%22&quot;&gt;DESCRIPTIVE statistics&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22OCCUPATIONAL+therapy%22&quot;&gt;OCCUPATIONAL therapy&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22TELEMEDICINE%22&quot;&gt;TELEMEDICINE&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22CAREGIVERS%22&quot;&gt;CAREGIVERS&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22CHRONIC+diseases%22&quot;&gt;CHRONIC diseases&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22QUALITY+of+life%22&quot;&gt;QUALITY of life&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22DATA+analysis+software%22&quot;&gt;DATA analysis software&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22INTER-observer+reliability%22&quot;&gt;INTER-observer reliability&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22VIDEO+recording%22&quot;&gt;VIDEO recording&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Introduction: Establishing inter-rater agreement and reliability ascertains that multiple raters consistently evaluate observed interventions to ensure that clinical research protocols are delivered as intended by the trial protocol. Purpose: Using the Guidelines for Reporting Reliability and Agreement Studies, we (a) exemplified the steps to establish inter-rater reliability and inter-rater agreement on the occupation-based coaching Video Evaluation Tool and (b) evaluated best practices that promoted high inter-rater reliability and inter-rater agreement between blinded raters prior to starting a pilot randomized controlled trial. The randomized controlled trial examined the preliminary effectiveness of occupation-based coaching via telehealth for rural families with children living with type 1 diabetes to improve family quality of life, participation, self-efficacy, and child health outcomes. Method: We created a library of 13 occupation-based coaching videos portraying a range of evaluations, scores, and ratings. The inter-rater agreement and reliability on the occupation-based coaching Video Evaluation Tool were established through the iterations of (a) blinded rater training, (b) data collection using the tool, and (c) statistical analysis using Cohen&#39;s kappa and Cronbach&#39;s alpha. Findings: Occurrence and Non-Occurrence Checklist (κ = 0.881, p &lt; 0.001); &quot;Caregiver Talk&quot; and &quot;Interventionist Talk Analysis&quot; (ICC = 0.991–0.999, p &lt; 0.001); Evidence of Independent Capacity Rating (ICC = 0.867 p = 0.006). Conclusion: Strong inter-rater reliability and inter-rater agreement was established by engaging two blinded raters through multifaceted training, integrating real-life clients and contexts into the instrumentation and training, and precisely defined rubric criteria. By employing such practices, high inter-rater reliability and agreement can be achieved in clinical research involving interventions and instruments that are highly subjective and individualized. To ascertain greater scientific confidence in the intervention effect, developing a multidomain fidelity framework and establishing high inter-rater agreement and reliability in the instruments a priori to implementation of clinical trials are necessary. [ABSTRACT FROM AUTHOR]
– Name: Abstract
  Label:
  Group: Ab
  Data: &lt;i&gt;Copyright of British Journal of Occupational Therapy is the property of Sage Publications Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder&#39;s express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.&lt;/i&gt; (Copyright applies to all Abstracts.)
PLink https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edb&AN=183370650
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/03080226241283292
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 9
        StartPage: 133
    Subjects:
      – SubjectFull: FAMILIES & psychology
        Type: general
      – SubjectFull: MEDICAL protocols
        Type: general
      – SubjectFull: TYPE 1 diabetes
        Type: general
      – SubjectFull: SELF-efficacy
        Type: general
      – SubjectFull: RESEARCH funding
        Type: general
      – SubjectFull: CRONBACH'S alpha
        Type: general
      – SubjectFull: PILOT projects
        Type: general
      – SubjectFull: EVALUATION of medical care
        Type: general
      – SubjectFull: TEACHING methods
        Type: general
      – SubjectFull: DESCRIPTIVE statistics
        Type: general
      – SubjectFull: OCCUPATIONAL therapy
        Type: general
      – SubjectFull: TELEMEDICINE
        Type: general
      – SubjectFull: CAREGIVERS
        Type: general
      – SubjectFull: CHRONIC diseases
        Type: general
      – SubjectFull: QUALITY of life
        Type: general
      – SubjectFull: DATA analysis software
        Type: general
      – SubjectFull: INTER-observer reliability
        Type: general
      – SubjectFull: VIDEO recording
        Type: general
    Titles:
      – TitleFull: Achieving Inter-Rater Agreement and Inter-Rater Reliability to Assess Fidelity of an Occupation-Based Coaching (OBC) Clinical Trial Intervention.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Abbott, Amy Ann
      – PersonEntity:
          Name:
            NameFull: Shin, Julia
      – PersonEntity:
          Name:
            NameFull: Carlson, Kathryn
      – PersonEntity:
          Name:
            NameFull: Russell, Marion
      – PersonEntity:
          Name:
            NameFull: Qi, Yongyue
      – PersonEntity:
          Name:
            NameFull: Storm, Hannah
      – PersonEntity:
          Name:
            NameFull: Jewell, Vanessa Dawn
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Text: Mar2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 03080226
          Numbering:
            – Type: volume
              Value: 88
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: British Journal of Occupational Therapy
              Type: main
ResultId 1