Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist
Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114)....
| Authors: | ; ; ; ; ; ; ; ; |
|---|---|
| Format: | Article |
| Status: | Published version |
| Publication date: | 2010 |
| Country: | España |
| Institution: | Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| Repository: | Recercat. Dipósit de la Recerca de Catalunya |
| OAI Identifier: | oai:recercat.cat:10230/16458 |
| Online Access: | http://hdl.handle.net/10230/16458 http://dx.doi.org/10.1186/1471-2288-10-82 |
| Access Level: | Open access |
| Keyword: | Indicadors de salut Salut pública -- Avaluació Salut pública -- Enquestes |
| _version_ | 1878738872539545600 |
|---|---|
| author | Mokkink, Lidwine B. |
| author2 | Terwee, Caroline B. Gibbons, Elisabeth Stratford, Paul W. Alonso Caballero, Jordi Patrick, Donald L. Knol, Dirk L. Bouter, Lex M. De Vet, Henrica C. W. |
| author2_role | author author author author author author author author |
| author_browse | Alonso Caballero, Jordi Bouter, Lex M. De Vet, Henrica C. W. Gibbons, Elisabeth Knol, Dirk L. Mokkink, Lidwine B. Patrick, Donald L. Stratford, Paul W. Terwee, Caroline B. |
| author_facet | Mokkink, Lidwine B. Terwee, Caroline B. Gibbons, Elisabeth Stratford, Paul W. Alonso Caballero, Jordi Patrick, Donald L. Knol, Dirk L. Bouter, Lex M. De Vet, Henrica C. W. |
| author_role | author |
| collection | Recercat. Dipósit de la Recerca de Catalunya |
| description | Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions./nConclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved. |
| eu_rights_str_mv | openAccess |
| format | article |
| id | ES_b6b79c775a560df0ee4e77e1cd2c2dec |
| institution | Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| instname_str | Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| network_acronym_str | ES |
| network_name_str | España |
| oai_identifier_str | oai:recercat.cat:10230/16458 |
| publishDate | 2010 |
| publishDateSort | 2010 |
| publisher | BioMed Central |
| publisherStr | BioMed Central |
| reponame_str | Recercat. Dipósit de la Recerca de Catalunya |
| spelling | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist Mokkink, Lidwine B. Terwee, Caroline B. Gibbons, Elisabeth Stratford, Paul W. Alonso Caballero, Jordi Patrick, Donald L. Knol, Dirk L. Bouter, Lex M. De Vet, Henrica C. W. Indicadors de salut Salut pública -- Avaluació Salut pública -- Enquestes Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions./nConclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved. BioMed Central http://hdl.handle.net/10230/16458 http://dx.doi.org/10.1186/1471-2288-10-82 |
| spellingShingle | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist Mokkink, Lidwine B. Indicadors de salut Salut pública -- Avaluació Salut pública -- Enquestes |
| status_str | publishedVersion |
| title | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| title_full | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| title_fullStr | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| title_full_unstemmed | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| title_short | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| title_sort | Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist |
| topic | Indicadors de salut Salut pública -- Avaluació Salut pública -- Enquestes |
| topic_facet | Indicadors de salut Salut pública -- Avaluació Salut pública -- Enquestes |
| url | http://hdl.handle.net/10230/16458 http://dx.doi.org/10.1186/1471-2288-10-82 |