/*++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ INTER_RATER macro v1.0 8/1/2002 - 8/27/2002 PROGRAM: Inter_Rater(InputData=,DataType=, VarianceType=,CategoryFile=,OutFile=_OuputFile); AUTHOR: Kilem Gwet, Ph.D. PURPOSE: 1) To compute the First-Order Agreement Coefficient AC1 among two or multiple raters and the associated precision measures as discussed by Kilem Gwet (2001). The ratings of subjects are assumed to be on a nominal scale. 2) To compute the Kappa statistic and its standard error as discussed by Fleiss (1971). 3) AC1 and Kappa statistics are also calculated for each category as discussed by Fleiss (1971). When calculated for a specific category, the extent of agreement between raters is said to be evaluated conditionally on that category. 4) The standard errors of the Kappa statistic and conditional Kappa statistics are disccused by Fleiss(1971). This SAS macro implements Fleiss' equations to evaluate these standard errors. For the AC1 statistic and conditional AC1 statistics (on response category), standard errors can be calculated conditionally on the current sample of raters or unconditionally. The user of this macro must request unconditional standard errors to obtain both conditional and unconditional standard errors. The AC1 statistic is extensively discused by Gwet (2001). USAGE: The %INTER_RATER must first be defined during a SAS session to make it available. To define this macro, the user should submit this file to the SAS system. There are two methods for using this macro: a) The first method is to cut this macro and paste it at the beginning of a SAS program so that the macro becomes part of the program. Following the SAS macro should be the preparation of the input dataset (InputFile.sas7bdat for example) that will be supplied as parameter to the macro. After defining the input dataset, the user should call the macro as follows: %Inter_Rater(InputData=InputFile, DataType=, VarianceType=, CategoryFile=); These parameters are defined later in this section. b) The second method, which I personally prefer consists of keeping the %INTER_RATER macro as a separate program file called INTER_RATER.MAC for instance. Suppose that this program file is fully specified as "C:\My SAS Programs\Macros\INTER_RATER.MAC". Then the user's SAS program should have the following 3 components: - Input data definition (in data step for instance) - Inclusion of the macro in the program as follows: %include 'C:\My SAS Programs\Macros\INTER_RATER.MAC'; - Use of the macro by specifying %Inter_Rater(InputData=InputFile, DataType=, VarianceType=, CategoryFile=); The parameters to specify are described as follows: InputData= Identifies the input data set to be analyzed. This parameter is required. If it is not specified, the macro will stop. This SAS data set can take two forms, which are best described by an example. Suppose that 3 raters have classified each of 10 subjects into one of 4 possible response categories. The ratings can be described by providing the classi- fication data as follows: TABLE 1. Classification Data Set Subject Peter Paul Michael 1 1 2 3 2 1 3 4 3 4 2 2 4 3 3 3 5 2 2 1 6 4 4 4 7 1 4 2 8 2 2 2 9 3 4 4 10 1 1 1 The subject column contains subject identifiers and can be alphabetic or numeric. The remaining variables contain the raters' ratings. In the above table, we have Peter = 2 for Subject 8. This indicates that Peter classified that subject into category 2. The user may assign any name to these variables, which could be alphabetic or numeric. The second form for the input data set provides the distribution of raters by subject and response cate- gory. This file provides agreement data as opposed to basic classification data. The agreement data set corresponding the ratings shown above is defined as follows: TABLE 2. Agreement Table Subject Category1 Category2 Category3 Category4 1 1 1 1 0 2 1 0 1 1 3 0 2 0 1 4 0 0 3 0 5 1 2 0 0 6 0 0 0 3 7 1 1 0 1 8 0 3 0 0 9 0 0 1 2 10 3 0 0 0 The user of this macro may use any variable name to designate the subject and the categories. Instead of using Subject Category1, Category2, Category3, and Category4, one may use the following variables: Patient, Depression, Personality_disorder, Schizophrenia, Neurosis, and Other. It follows from the above table that for Subject = 10, we have Category1 = 3. This indicates that all 3 raters classified subject 10 into category 1. DataType= Identifies which of the 2 forms described for the input file is used. When the input file provides the basic classification data (i.e. as in Table 1), then one should specify DataType=C (upper or lower case would be acceptable). If on the other hand the input data provides agreement data (i.e. as in Table 2), then one should specifiy DataType=A. VarianceType= Indicates whether the standard error of the AC1 statistic should only be calculated conditionally on the sample of raters (VarianceType=C) or whether the unconditional standard error should also be calculated (VarianceType=U). CategoryFile= This is a one-variable dataset containing a description of all response categories used in the reliability study. This variable can be alphabetic or numeric. It is recom- memnded to have short descriptions as these should match the content of Table 1. An example of such a file is the following: Category D P S N O where D, P, S, N, and O may respectively stand for Depression Personality disorder, Shizophrenia, Neurosis, and Other. Any classification data set (as Table 1) based on these categories should use these category descriptions. OutFile= This parameter specifies the output file that contains all the results and intermediate results such as the chance-agreement or overall agreement probabilities as well as the PI classifi- cation probabilities. The default name of the output file is _OutputFile, but can be changed by the user. REFERENCES 1. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters., Psychological Bulletin, 76, 378-382. 2. Gwet, K. (2002). Handbook of Inter-Rater Reliability. STATAXIS Publishing Company. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*/ %MACRO CreateAgreeFile(ClassifData,CategoryFile,qCategories); /* The objective of this SAS macro is to create the agreement file AgreeFile with "Subjects" rows and "qCategories+1" columns. AgreeFile is used by other SAS macros to produce statistics and associated standard errors. The input file "ClassifData" is a subject-level file that contains one categorical variable for each rater, showing the category into which the subject is classified. ---*/ %IF &ErrorFound %THEN %GOTO EXIT; %ELSE %DO; %LET DSID = %SYSFUNC(OPEN(&ClassifData)); %IF &DSID %THEN %DO; /*-- compute the number of subjects and number of raters --- */ %LET Subjects = %SYSFUNC(ATTRN(&DSID,NOBS)); %LET Raters = %EVAL(%SYSFUNC(ATTRN(&DSID,NVARS))-1); %LET NumberOfVars = %SYSFUNC(ATTRN(&DSID,NVARS)); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &ClassifData ****; %LET ErrorFound = 1; %GOTO EXIT; %END; /*--- Read all variable names from the ClassifData file ----*/ %LET DSID = %SYSFUNC(OPEN(&ClassifData)); %DO I=1 %TO &NumberOfVars; %LET ClassVar&I = %SYSFUNC(VARNAME(&DSID,&I)); %END; %LET RC = %SYSFUNC(CLOSE(&DSID)); /*--- To construct a modified classification data file where ----*/ /*--- categories are numbered as 1, 2, ... qCategories ----*/ DATA _ClassFile(KEEP=&ClassVar1 _RaterName _RowNum _ColNum _CatLabel ); IF _N_=1 THEN _RowNum = 1; _ColNum = 1; SET &ClassifData; %DO I=2 %TO &NumberOfVars; %LET _R=%EVAL(&I-1); _RaterName = "Rater&_R"; _CatLabel = &&ClassVar&I; OUTPUT; _ColNum+1; %END; _RowNum+1; RUN; %LET DSID = %SYSFUNC(OPEN(&CategoryFile)); %LET _CatVar = %SYSFUNC(VARNAME(&DSID,1)); %LET RC = %SYSFUNC(CLOSE(&DSID)); DATA _CatFile(KEEP=_CatLabel _CatID); SET &CategoryFile; _CatLabel = &_CatVar; _CatID = _N_; RUN; PROC SORT DATA=_ClassFile;BY _CatLabel;RUN; PROC SORT DATA=_CatFile;BY _CatLabel;RUN; DATA _ClassFile; MERGE _ClassFile(IN=IN1) _CatFile(IN=IN2); BY _CatLabel; IF IN1; RUN; PROC SORT DATA=_ClassFile;BY _RowNum _ColNum;RUN; DATA _NewClassifData(DROP=_RowNum _ColNum _RaterName &ClassVar1 _CatLabel _CatID _q); SET _ClassFile; ARRAY _RaterVars{&Raters} _Rater1-_Rater&Raters; RETAIN _Rater1-_Rater&Raters; BY _RowNum; IF FIRST._RowNum THEN _q = 1; _RaterVars(_q) = _CatID; _q+1; IF LAST._RowNum THEN DO; _Subject = &ClassVar1; OUTPUT; END; RUN; DATA AgreeFile; SET _NewClassifData; ARRAY Category{&qCategories} Category1-Category&qCategories; ARRAY Rater{&Raters} _Rater1-_Rater&Raters; DO q=1 TO &qCategories; Category{q}=0; DO r=1 TO &Raters; IF Rater{r}=q THEN Category{q}+1; END; END; DROP q r _Rater1-_Rater&Raters; RUN; %END; %EXIT: TITLE1;TITLE2;TITLE3;RUN; %MEND CreateAgreeFile; %MACRO P2aValues(ClassifData,AgreeFile,qCategories); /*-- This macro computes the unconditional P2a probability as well as the conditional P2a.q (q=1,...,Q) probabilities needed to obtain the unconditional and conditional variances of the AC1 statistics ---*/ %LET DSID = %SYSFUNC(OPEN(&ClassifData)); %IF &DSID %THEN %DO; %LET Subjects = %SYSFUNC(ATTRN(&DSID,NOBS)); %LET Raters = %EVAL(%SYSFUNC(ATTRN(&DSID,NVARS))-1); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &ClassifData ****; %LET ErrorFound = 1; %GOTO EXIT; %END; PROC DATASETS LIB=WORK NOLIST; DELETE BigDeltaFile; RUN; DATA DummyFile; DO Mkey=1 TO &Subjects; DO DUMMY=1 TO &Subjects;OUTPUT;END;END; RUN; /*-- DeltaFile&q is an &Subjects x &Raters indicator matrix of 0's and 1's obtained from ClassifData. The (i,r) element of DeltaFile&q is 1 if rater r has classified subject i into category q, and will be 0 otherwise. ----*/ %DO q=1 %TO &qCategories; DATA DeltaFile&q(RENAME=(_Subject=Subject&q) DROP=r _Rater1-_Rater&Raters); SET &ClassifData; ARRAY Rater{&Raters} _Rater1-_Rater&Raters; ARRAY Delta{&Raters} DeltaVar1-DeltaVar&Raters; DO r=1 TO &Raters; IF Rater(r)=&q THEN Delta(r)=1; ELSE Delta(r)=0; END; Mkey=_N_; DATA WorkDeltaFile(RENAME=(Subject&q=Subject) DROP=Mkey rr DeltaVar1-DeltaVar&Raters); MERGE DeltaFile&q DummyFile; BY Mkey; ARRAY Delta{&Raters} DeltaVar1-DeltaVar&Raters; ARRAY NewDelta{&Raters} DeltaV1-DeltaV&Raters; DO rr=1 TO &Raters; NewDelta(rr)=Delta(rr); END; lCategory = &q; RUN; PROC APPEND BASE=BigDeltaFile NEW=WorkDeltaFile FORCE;RUN; %END; PROC SORT DATA=BigDeltaFile;BY DUMMY;RUN; %DO q=1 %TO &qCategories; DATA DeltaFile&q; SET DeltaFile&q; DUMMY=_N_; RUN; PROC SORT DATA=DeltaFile&q;BY DUMMY;RUN; DATA WorkMfile(DROP=DUMMY); MERGE DeltaFile&q BigDeltaFile; BY DUMMY; RUN; PROC SORT DATA=WorkMfile;BY Subject&q;RUN; DATA MFile&q(RENAME=(Subject&q=SubjectID) KEEP=Subject&q SumOfM&q) MFileCond&q(RENAME=(Subject&q=SubjectID) KEEP=Subject&q SumOfMC&q); ARRAY Delta{&Raters} DeltaVar1-DeltaVar&Raters; ARRAY NewDelta{&Raters} DeltaV1-DeltaV&Raters; SET WorkMfile; BY Subject&q; Mfactor=0; DO r=1 TO &Raters; Mfactor+Delta(r)*NewDelta(r); END; SumOfM&q+Mfactor*(Mfactor-1); IF lCategory=&q THEN SumOfMC&q+Mfactor*(Mfactor-1); IF LAST.Subject&q THEN DO; OUTPUT MFile&q; OUTPUT MFileCond&q; SumOfM&q=0; SumOfMC&q=0; END; RUN; PROC SORT DATA=MFile&q;BY SubjectID;RUN; PROC SORT DATA=MFileCond&q;BY SubjectID;RUN; %IF &q=1 %THEN %DO; DATA FinalMfile; SET MFile&q; RUN; DATA FinalMfileCond; SET MFileCond&q; RUN; %END; %ELSE %DO; DATA FinalMfile; MERGE FinalMfile MFile&q; BY SubjectID; RUN; DATA FinalMfileCond; MERGE FinalMfileCond MFileCond&q; BY SubjectID; RUN; %END; %END; /*-- To calculate the P2a probability ---*/ DATA _NULL_; SET FinalMfile END=EOF; ARRAY Pvars(&qCategories) SumOfM1-SumOfM&qCategories; DO I=1 TO &qCategories; p2a+Pvars(I); END; IF EOF THEN DO; p2a = p2a/(&Raters*(&Raters-1)*(&Subjects**2)); CALL SYMPUT('P2a',p2a); END; RUN; DATA P2a_qFile(KEEP=qVar P2aq); SET FinalMfileCond END=EOF; ARRAY Pvars(&qCategories) SumOfMC1-SumOfMC&qCategories; ARRAY P2aqVars(&qCategories) P2aqV1-P2aqV&qCategories; DO q=1 TO &qCategories; P2aqVars(q)+Pvars(q); END; IF EOF THEN DO; DO q=1 TO &qCategories; P2aq = P2aqVars(q); qVar = q; OUTPUT; END; END; RUN; /*--- To calculate the averages \ov{r}_{+q} ---*/ DATA RqFile(KEEP=qVar r_qVar); SET &AgreeFile END=EOF; ARRAY CatVars(&qCategories) Category1-Category&qCategories; ARRAY rqVars(&qCategories) MeanR1-MeanR&qCategories; DO q=1 TO &qCategories; rqVars(q)+CatVars(q); END; IF EOF THEN DO; DO q=1 TO &qCategories; r_qVar = rqVars(q)/&Subjects; qVar = q; OUTPUT; END; END; RUN; PROC SORT DATA=RqFile;BY qVar;RUN; PROC SORT DATA=P2a_qFile;BY qVar;RUN; DATA P2aqData(KEEP=qVar P2aq); MERGE RqFile P2a_qFile; BY qVar; P2aq = P2aq/(r_qVar*(&Raters-1)*(&Subjects**2)); RUN; DATA P2aqData(KEEP=q P2aq); DO q=1 TO &qCategories+1; IF q<=&qCategories THEN DO; SET P2aqData; OUTPUT; END; ELSE DO; q=%EVAL(&qCategories+1);P2aq=&P2a; OUTPUT; END; END; RUN; %EXIT: TITLE1;TITLE2;TITLE3;RUN; %MEND P2aValues; %MACRO RaterAgreement(AgreeFile); /*-- This macro computes the Kappa and AC1 statistics as well as the corresponding standard errors, Z and P values. For the AC1 statistic, the standard error is calculated conditionally on the rater sample. That is the result the results cannot be extrapolated to a set of raters who did not participate to the the reliability experiment. Nevertheless, the results can be extrapolated to a bigger universe of subjects. ---*/ %LET DSID = %SYSFUNC(OPEN(&AgreeFile)); %IF &DSID %THEN %DO; %LET nSubjects = %SYSFUNC(ATTRN(&DSID,NOBS)); %LET qCategories = %EVAL(%SYSFUNC(ATTRN(&DSID,NVARS))-1); %LET NumberOfVars= %SYSFUNC(ATTRN(&DSID,NVARS)); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &AgreeFile ****; %LET ErrorFound = 1; %GOTO EXIT; %END; %IF %UPCASE(&DataType) EQ A %THEN %DO;/*-- Rename variables as SUBJECT, Category1,....---*/ %LET DSID = %SYSFUNC(OPEN(&AgreeFile)); %DO I=1 %TO &NumberOfVars; %LET ClassVar&I = %SYSFUNC(VARNAME(&DSID,&I)); %END; %LET RC = %SYSFUNC(CLOSE(&DSID)); DATA &AgreeFile; SET &AgreeFile; RENAME &ClassVar1=_Subject; %DO I=2 %TO &NumberOfVars; %LET II=%EVAL(&I-1); RENAME &&ClassVar&I=Category&II; %END; RUN; %END; /*-- To compute the number of raters rRaters --*/ DATA _NULL_; SET &AgreeFile; ARRAY CatV(&qCategories) Category1-Category&qCategories; DO q=1 TO &qCategories; rRaters+CatV(q); END; CALL SYMPUT('rRaters',LEFT(rRaters)); STOP; RUN; /*-- To check for missing values ----*/ DATA _NULL_; SET &AgreeFile; ARRAY CatVars{&qCategories} Category1-Category&qCategories; NumRaters=0; DO q=1 TO &qCategories; NumRaters+CatVars(q); END; IF NumRaters NOT= &rRaters THEN DO; CALL SYMPUT('MissError',1); STOP; END; RUN; DATA StatFile(DROP=q) CategFile(keep=q Subjects Raters rBarQ Pe_gamma Pe_kappa PaQ PiHatQ FleissVar AC1_CondVar KappaStat AC1Statistic StdErrKappa StdErrAC1cond Z_Kappa Z_AC1 PvalKappa PvalAC1); SET &AgreeFile END=EOF; Subjects = &nSubjects; Raters = &rRaters; ARRAY CatVars(&qCategories) Category1-Category&qCategories; ARRAY rBar_q(&qCategories) rBar_1-rBar_&qCategories; ARRAY Pa_q(&qCategories) Pa_1-Pa_&qCategories; ARRAY PiHat_q(&qCategories) PiHat_1-PiHat_&qCategories; ARRAY FleissVar_q(&qCategories) FleissVar_1-FleissVar_&qCategories; ARRAY AC1Var_q(&qCategories) AC1Var_1-AC1Var_&qCategories; ARRAY SumOfPaiq2_q(&qCategories) SumOfPaiq2_1-SumOfPaiq2_&qCategories; SumRi = 0; DO q=1 TO &qCategories; SumRi+(CatVars(q)*(CatVars(q)-1)); rBar_q(q)+CatVars(q); Pa_q(q)+(CatVars(q)*(CatVars(q)-1)); SumOfPaiq2_q(q)+(CatVars(q)*(CatVars(q)-1))**2; END; Pa_i=SumRi/(&rRaters*(&rRaters-1)); SumR+SumRi; SumOfPa_i2+Pa_i**2;/*-- used to compute the uvariance of AC1 estimator --*/ IF EOF THEN DO; OverallPa=SumR/(&nSubjects*&rRaters*(&rRaters-1)); DO q=1 TO &qCategories; rBar_q(q)=rBar_q(q)/&nSubjects; Pa_q(q)=Pa_q(q)/(&nSubjects*rBar_q(q)*(&rRaters-1)); SumOfPaiq2_q(q)=SumOfPaiq2_q(q)/((rBar_q(q)*(&rRaters-1))**2); PiHat_q(q)=rBar_q(q)/&rRaters; Pe_gamma+PiHat_q(q)*(1-PiHat_q(q)); Pe_kappa+PiHat_q(q)**2; PiHatQ3+PiHat_q(q)**3; AC1Var_q(q)= SumOfPaiq2_q(q)-&nSubjects*(Pa_q(q)**2); FleissVar_q(q) = ((1+2*(&rRaters-1)*PiHat_q(q))**2 +2*(&rRaters-1)*PiHat_q(q)*(1-PiHat_q(q))) /((&nSubjects*&rRaters*(&rRaters-1)**2) *PiHat_q(q)*(1-PiHat_q(q))); END; Pe_gamma = Pe_gamma/(&qCategories-1); FleissVarUncond = (2*(Pe_kappa-(2*&rRaters-3)*(Pe_kappa**2) +2*(&rRaters-2)*PiHatQ3))/(&nSubjects*&rRaters *(&rRaters-1)*((1-Pe_kappa)**2)); AC1VarCond = (SumOfPa_i2-&nSubjects*(OverallPa**2)) /(&nSubjects*(&nSubjects-1)*((1-Pe_gamma)**2)); /*-- create output variables for the category-level file ---*/ DO q=1 TO &qCategories; rBarQ = rBar_q(q); PaQ = Pa_q(q); PiHatQ= PiHat_q(q); FleissVar = FleissVar_q(q); StdErrKappa = SQRT(FleissVar); AC1_CondVar = AC1Var_q(q)/(&nSubjects*(&nSubjects-1)*((1-Pe_gamma)**2)); StdErrAC1cond = SQRT(AC1_CondVar); KappaStat = (Pa_q(q)-PiHat_q(q))/(1-PiHat_q(q)); AC1Statistic = (Pa_q(q)-Pe_gamma)/(1-Pe_gamma); Z_Kappa=KappaStat/StdErrKappa; IF StdErrAC1cond>0 THEN Z_AC1=AC1Statistic/StdErrAC1cond;ELSE Z_AC1=.; PvalKappa=1-PROBNORM(Z_Kappa); PvalAC1=1-PROBNORM(Z_AC1); OUTPUT CategFile; END; /*-- to print overall statistics in the category-level file ---*/ q=%EVAL(&qCategories+1); rBarQ = 1.0; PaQ = OverallPa; PiHatQ= 1.0; FleissVar = FleissVarUncond; StdErrKappa = SQRT(FleissVar); AC1_CondVar = AC1VarCond; StdErrAC1cond = SQRT(AC1_CondVar); KappaStat = (OverallPa-Pe_kappa)/(1-Pe_kappa); AC1Statistic = (OverallPa-Pe_gamma)/(1-Pe_gamma); Z_Kappa=KappaStat/StdErrKappa; IF StdErrAC1cond>0 THEN Z_AC1=AC1Statistic/StdErrAC1cond;ELSE Z_AC1=.; PvalKappa=1-PROBNORM(Z_Kappa); PvalAC1=1-PROBNORM(Z_AC1); OUTPUT CategFile; END; OUTPUT StatFile; RUN; %EXIT: TITLE1;TITLE2;TITLE3;RUN; %MEND RaterAgreement; %MACRO Inter_Rater(InputData=,DataType=A,VarianceType=C,CategoryFile=,OutFile=_OuputFile); OPTIONS NONOTES; TITLE2 "INTER_RATER macro"; %LET MissError=0;/* Macro %RaterAgreement set this to 1 if there are missing ratings*/ %LET ErrorFound=0; %IF %UPCASE(&InputData) EQ %THEN %DO; %PUT "ERROR: *** An input data must be specified ***"; %GOTO EXIT; %END; /*-- Computing the number of categories ----*/ %LET DSID = %SYSFUNC(OPEN(&CategoryFile)); %IF &DSID %THEN %DO; %LET Ncategories = %SYSFUNC(ATTRN(&DSID,NOBS)); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &CategoryFile ****; %GOTO EXIT; %END; /*-- Verifying that the input file provides raw data on subject classification (DataType=C) or provides the distribution of raters by subject and response category (DataType=A). DataType=C refers to a subject-level dataset with the following variables: Subject _Rater1 _Rater2 .... and DataType=A refers to a subject-level dataset with the following variable: Subject Category1 Category2 .... ---*/ %IF %UPCASE(&DataType) NE A AND %UPCASE(&DataType) NE C %THEN %DO; DATA _NULL_; PUT "!-------------------------------------------------------------------------!"; PUT "! ERROR: *** DataType= must be set either to A (if an nxQ agreement file !"; PUT "! containing the distribution of raters by subject and category !"; PUT "! is supplied or to C (if an nxr classification file containing !"; PUT "! for each subject and rater, the corresponding categorization !"; PUT "!-------------------------------------------------------------------------!"; RUN; %GOTO EXIT; %END; %ELSE %IF %UPCASE(&DataType) EQ C %THEN/*-- To compute the number of raters ---*/ %DO; %LET DSID = %SYSFUNC(OPEN(&InputData)); %IF &DSID %THEN %DO; %LET rRaters = %EVAL(%SYSFUNC(ATTRN(&DSID,NVARS))-1); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &AgreeFile ****; %GOTO EXIT; %END; %END; %ELSE %DO; DATA _NULL_; SET &InputData; ARRAY CatV(&Ncategories) Category1-Category&Ncategories; rRaters = 0; DO q=1 TO &Ncategories; rRaters+CatV(q); END; CALL SYMPUT('rRaters',LEFT(rRaters)); STOP; RUN; %END; /*-- To compute the number of subjects ---*/ %LET DSID = %SYSFUNC(OPEN(&InputData)); %IF &DSID %THEN %DO; %LET nSubjects= %SYSFUNC(ATTRN(&DSID,NOBS)); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &AgreeFile ****; %GOTO EXIT; %END; /*-- Variance calculation ----*/ %IF %UPCASE(&VarianceType) NE U %THEN %DO; %IF %UPCASE(&DataType) EQ A %THEN %RaterAgreement(&InputData); %ELSE %DO; %CreateAgreeFile(&InputData,&CategoryFile,&Ncategories); %RaterAgreement(AgreeFile); %END; %END; %ELSE %DO; %IF %UPCASE(&DataType)=C %THEN %DO; %CreateAgreeFile(&InputData,&CategoryFile,&Ncategories); %P2aValues(_NewClassifData,AgreeFile,&Ncategories); %RaterAgreement(AgreeFile); %IF &ErrorFound %THEN %GOTO EXIT; %ELSE %DO; PROC SORT DATA=CategFile;BY q;RUN; PROC SORT DATA=P2aqData;BY q;RUN; DATA CategFile; MERGE CategFile P2aqData; BY q; Cr = 1/(&rRaters*(&rRaters-1)*((1-Pe_gamma)**2)); AC1_UcondVar = AC1_CondVar+Cr*(P2aq+(Paq-P2aq)/&nSubjects); StdErrAC1_U = SQRT(AC1_UcondVar); IF StdErrAC1_U>0 THEN Z_AC1_U = AC1Statistic/StdErrAC1_U; ELSE Z_AC1_U=.; PvalAC1_U = 1-PROBNORM(Z_AC1_U); RUN; %END; %END; %ELSE %DO; DATA _NULL_; PUT "!----------------------------------------------------------------!"; PUT "!ERROR: *** In order to compute the unconditional variance of the!"; PUT "! AC1 statistic(VarianceType=U), it is necessary to provide!"; PUT "! an input file of type DataType=C showing the specific ca-!"; PUT "! tegories into which the subjects are classified. ********!"; PUT "!----------------------------------------------------------------!"; RUN; %GOTO EXIT; %END; %END; %IF &MissError=1 %THEN %DO; %PUT "!-----------------------------------------------------!"; %PUT "!*** ERROR: There are missing ratings in the dataset. !"; %PUT "! Each rater must classify every subject !"; %PUT "! into one of &Ncategories categories. !"; %PUT "!-----------------------------------------------------!"; %GOTO EXIT; %END; /**** Read all variable names from the ClassifData file ****/ %LET DSID = %SYSFUNC(OPEN(&CategoryFile)); %IF &DSID %THEN %DO; %LET _CategoryVar = %SYSFUNC(VARNAME(&DSID,1)); %LET _CategType = %SYSFUNC(VARTYPE(&DSID,1)); %LET RC = %SYSFUNC(CLOSE(&DSID)); %END; %ELSE %DO; %PUT **** The system could not open data set &CatgoryFile ****; %LET ErrorFound = 1; %GOTO EXIT; %END; PROC TRANSPOSE DATA=&CategoryFile OUT=_TransFile; VAR &_CategoryVar; RUN; DATA _NULL_; SET _TransFile; IF _N_=1 THEN DO; %DO _q=1 %TO &Ncategories; CALL SYMPUT("_CatLabel&_q",COL&_q); %END; END; RUN; PROC FORMAT; VALUE _fCateg %EVAL(&Ncategories+1)="Overall" %DO _q=1 %TO &Ncategories; %IF &_CategType=C %THEN &_q = "&&_CatLabel&_q"; %ELSE &_q = &&_CatLabel&_q; %END;; RUN; PROC PRINT DATA=CategFile NOOBS LABEL; VAR q KappaStat StdErrKappa Z_Kappa PvalKappa; LABEL KappaStat="Kappa" StdErrKappa="Standard Error" q="Category" Z_Kappa="Z" PvalKappa="Prob>Z"; FORMAT q _fCateg.; TITLE2 "INTER_RATER macro (v 1.0)"; TITLE3 "Kappa statistics: conditional and unconditional analyses"; TITLE4; RUN; PROC PRINT DATA=CategFile NOOBS LABEL; VAR q AC1Statistic StdErrAC1cond Z_AC1 PvalAC1; LABEL AC1Statistic="AC1 statistic" StdErrAC1cond="Standard Error" q="Category" Z_AC1="Z" PvalAC1="Prob>Z"; FORMAT q _fCateg.; TITLE2 "INTER_RATER macro (v 1.0)"; TITLE3 "AC1 statistics: conditional and unconditional analyses"; TITLE4 " Inference based on conditional variances of AC1"; RUN; %IF %UPCASE(&VarianceType)=U %THEN %DO; PROC PRINT DATA=CategFile NOOBS LABEL; VAR q AC1Statistic StdErrAC1_U Z_AC1_U PvalAC1_U; LABEL AC1Statistic="AC1 statistic" StdErrAC1_U="Standard Error" q="Category" Z_AC1_U="Z" PvalAC1_U="Prob>Z"; FORMAT q _fCateg.; TITLE2 "INTER_RATER macro (v 1.0)"; TITLE3 "AC1 statistics: conditional and unconditional analyses"; TITLE4 " Inference based on unconditional variances of AC1"; RUN; %END; /*-- Creating the output file -----*/ %IF %UPCASE(&OutFile) EQ %THEN %DO; DATA _OutputFile; SET CategFile; FORMAT q _fCateg.; RUN; %END; %ELSE %DO; DATA &OutFile; SET CategFile; FORMAT q _fCateg.; RUN; %END; %EXIT: TITLE1;TITLE2;TITLE3;RUN; %MEND Inter_Rater;