Study of surgeons' agreement on the presence of bony anomalies on coronal CT of paranasal sinuses. (Original Article).

Abstract

We conducted a study to determine the level of agreement among five surgeons who assessed the surgical risk to key neighboring structures prior to endoscopic sinus surgery as depicted on coronal computed tomography (CT). The five observers studied preoperative CT scans that had been performed on 29 patients. Two scoring systems were used to rate each scan: the Lund-Mackay system and our own risk-assessment system. Kappa analysis was used to measure interobserver agreement. We found that the overall level of agreement among the five surgeons according to the Lund-Mackay criteria was moderate, while agreement according to our risk-assessment system was only slight. We conclude that surgeons' agreement on the bony detail of risk structures is not as strong as our agreement on other factors, such as staging disease.

Introduction

Computed tomography (CT) is the preoperative imaging modality preferred by endoscopic sinus surgeons. It provides information on the extent of disease, areas of sinus drainage, and details of regional anatomy. (1) Several investigators have based disease-staging systems on CT because it is a readily available source of hard, objective evidence on the extent of pathology. (2,3) Oluwole et al (2) compared some of these systems with regard to inter- and intraobserver agreement, and they established that the Lund-Mackay scoring system3 is simple and reproducible.

Of greater immediate importance to the operating surgeon is CT's value as a road map. The surgeon should use the scan to form a three-dimensional image of the relationships of the nasal cavity and sinuses to the anterior skull base and the orbit. Investigators have used cadaver dissections to identify and measure anatomic landmarks that can aid the surgeon and help prevent complications. (4,5) Similar efforts have been undertaken to use CT to identify those anatomic variations that can predispose surgeons to making an inadvertent penetration of the orbit or cranial cavity. (6)

Although surgeons agree on staging and diagnosing mucosal disease, agreement on the interpretation of anatomic variations has not been shown. Melhem et al demonstrated disagreement on anatomic anomalies in the course of a study on optimizing CT evaluation for endoscopic surgery. (7) Melhem' s group, which was made up of four neuroradiologists and one otolaryngologist, found a significant degree of interobserver disagreement in the identification of 10 anatomic structures and regions. This clinical disagreement occurred despite the use of strict and previously agreed-upon anatomic definitions.

It is widely believed that CT is a safeguard against complications and that its interpretation can be an important medicolegal document in a court of law. (8) We set out to determine the extent of agreement among five endoscopic sinus surgeons with respect to anomalies that constitute a surgical risk to the patient as shown on the preoperative CT scan.

Materials and methods

We retrospectively reviewed the cases of 29 consecutive patients--17 women and 12 men, aged 18 to 74 years (mean: 42.8)--who had undergone CT in the radiology department of Raigmore Hospital in Inverness, Scotland, between March 1 and May 31, 1996. CT had been performed in preparation for endoscopic sinus surgery following the failure of medical treatment for chronic rhinosinusitis. We did not include in our study group any patient who had previously undergone sinus surgery, who had been previously subjected to facial or anterior skull-base trauma, or who had been suspected clinically of having neoplastic disease, nor did we include any patients who were younger than 16 years of age. Imaging had been performed with the IGE 9800 Quick scanner and was restricted to coronal cuts only.

Five observers (three consultant otolaryngologists and two specialist registrars) in two centers (the Raigmore Hospital and the Ninewells Hospitals in Dundee, Scotland) took part in the study. One of the consultants (P.S.W.) was a dedicated rhinologist, while the other two regularly performed endoscopic surgery as part of their practices (W.S.M. and L.G.M.). The two registrars were year-2 (H.E.) and year-4 (M.O.) trainees who had a special interest in rhinology; both had attended endoscopic surgery courses and were regularly operating under supervision in their units. Each observer rated the scans individually and was blind to the findings of other observers as well as to each patient's history, management, operative findings, and outcome.

The scans were rated according to the Lund-Mackay system (3) (table 1) and a customized risk-assessment system of our own design (table 2). The former provided a good baseline for interobserver agreement; we based the latter on information derived from the literature. (9,10) Prior to making our interpretations of CT images, our five observers agreed on all definitions, bearing in mind those factors that lead to clinical disagreement. Our interpretations of the CT images of the first 10 patients constituted a pilot study, and we found that intraobserver agreement was high (kappa [kappa] value: 0.9). Our risk-assessment system rated 11 anatomic locations according to the presence or absence of a variation. We used nonweighted [kappa] analysis for measuring our degree of agreement (table 3). The level of interobserver agreement was measured individually for each location and for each side, and the strength of agreement was classified into one of five categories as suggested by Landis and Koch. (11)

Results

Our overall degree of agreement according to the five Lund-Mackay parameters was in the moderate category ([kappa] value: 0.51) (table 4). Scores ranged from 0.57 (frontal sinus) to 0.39 (ostiomeatal complex), and five of the six were in the moderate range.

Our overall level of agreement according to our customized risk-assessment system was in the slight category (kappa value: 0.16) (table 5). Agreement ranged from substantial (0.65 for an absent frontal sinus) to poor (--0.05 for Hailer's cells). These ranges were much wider than those seen with the Lund-Mackay system.

Discussion

Our study shows that surgeons do not agree on anatomic bony details on coronal CT scans of the sinuses as much as we do with regard to staging mucosal disease.

Preoperative CT in endoscopic sinus surgery has two main aims: It is an important staging tool, and it has an even more crucial role as a road map for the safe execution of surgery. (10) The incidence of complications during endoscopic sinus surgery has been shown to be associated with the experience of the surgeon, the extent of disease, and previous surgery. (12) Although there is no robust scientific evidence that associates complication rates with imaging factors, CT is essential, and it can be used as a medicolegal document if a complication ensues. (8) Molony and Ah-See expressed concern that one in seven surgeons who took part in their mail questionnaire survey did not always obtain CT prior to endoscopic sinus surgery. (13) They also reported that despite the importance attached to axial cuts in delineating posterior relations, (1) 48% of their respondents depended solely on coronal views. Our findings add to these concerns. Although the five surgeons in our study generally agreed on diagnostic gradin g, the level of agreement on risk factors in certain areas was poor--the worst scores being those for Hailer's cells, the absence of a sphenoid sinus, and the bony cover of the internal carotid artery.

Our study involved surgeons whose experience in endoscopic sinus surgery was representative of mainstream U.K. peers. We chose as a benchmark the Lund-Mackay scoring system because it is known for its reproducibility and simplicity. (2) Our pilot study confirmed these points. Our choice of the bony landmarks and variations was based on those cited in major references. To avoid additional complexity, we scored these variables as simply as possible (0 for the absence of an anomaly and 1 for its presence). Our patients were typical of those generally seen in a district general hospital.

We acknowledge certain limitations of our study, some of which were inherent in its design and others that were related to general issues of clinical disagreement. The main design limitations were the relatively small number of observers and the variation in our experience. In addition, greater levels of agreement in our assessments of the relationships of structures such as the internal carotid artery and the optic nerve might have been achieved had we also studied axial-plane CTs, but we chose to use only coronal slices because this practice is most representative of current U.K. practice. (13)

The causes of clinical disagreement are well established and are relevant to our findings. A report was published in 1980 that identified five principal factors behind clinical disagreements: subtle degrees in a variation, the presence of several aspects to each variation, entrapment by prior expectation, varying definitions for some entities in the literature, and incorrect use of the tool being studied. (14)

Bolger et al (15) in 1991 and Lund et a1 (8) in 1997 reported that there were wide variations in the reported prevalence of anomalies in the agger nasi, concha bullosa, and Haller's cells. Indeed, in their report of 202 patients, Bolger et al found that 98.5% had anomalies of the agger nasi, 53.0% had anomalies of the concha bullosa, and 45.0% had anomalies of Haller's cells. Also in 1991, Lloyd et al reported on 100 patients and found these same anomalies in only 14.0, 24.0, and 15.0% of patients, respectively. (16) Finally, in 1997, Jones et al found corresponding anomaly rates of 95.5, 20.0, and 9.0% in a group of 200 patients. (17) Bolger et al attributed the wide variations to a lack of consensus on clear definitions of some of these anomalies. (15) An effort to circumvent these problems was exemplified by the work of Stammberger and Kennedy in 1995. (18) They suggested a unified system of terminology that was designed to promote consistent international language for surgeons and to serve as a basis for discussions among anatomists.

Finally, there is little doubt that axial-plane CT studies are better than coronal cuts in demonstrating the relationship of the sphenoid and posterior ethmoid sinuses to the orbit. It is possible that the practice of limiting examination to the coronal planes fails to provide surgeons with adequate information on the vital structures that are related to the posterior ethmoid cells and the sphenoid.

Our study shows that surgeons' agreement on the bony detail of risk structures is poor compared with our agreement on staging disease. We believe that a greater emphasis should be placed on formal radiologic teaching as part of training for endoscopic sinus surgery. There is also a need for uniform and accurate definitions in order to reduce confusion in the risk assessment of patients who undergo such surgery.

Table 1

The Lund-Mackay scoring system (3)

Structure                Left        Right

Maxillary sinus          0, 1, or 2  0, 1, or 2
Anterior ethmoid sinus   0, 1, or 2  0, 1, or 2
Posterior ethmoid sinus  0, 1, or 2  0, 1, or 2
Frontal sinus            0, 1, or 2  0, 1, or 2
Sphenoid sinus           0, 1, or 2  0, 1, or 2
Ostiomeatal complex      0 or 2      0 or 2

Key: 0: no abnormality; 1: partial opacification; 2: total
opacification.

Table 2

Our risk-assessment scoring system

Structure                    Left   Right

Absent frontal sinus        0 or 1  0 or 1
Absent sphenoid sinus       0 or 1  0 or 1
Agger nasi cells            0 or 1  0 or 1
Asymmetric sphenoid septum  0 or 1  0 or 1
Cribriform plate            0 or 1  0 or 1
Haller's cells              0 or 1  0 or 1
Internal carotid cover      0 or 1  0 or 1
Lamina papyracea            0 or 1  0 or 1
Optic nerve cover           0 or 1  0 or 1
Pneumatized ethmoid roof    0 or 1  0 or 1
Uncinate process            0 or 1  0 or 1

Key: 0: normal; 1: definite anomaly.

Table 3

Categories of agreement according to kappa analysis (11)

Kappa value   Strength of agreement

<0.00         Poor
0.00 to 0.20  Slight
0.21 to 0.40  Fair
0.41 to 0.60  Moderate
0.61 to 0.80  Substantial
0.81 to 1.00  Almost perfect

Table 4

Our degree of agreement according to the Lund-Mackay system

                         Kappa  Degree of
Structure                value  agreement

Frontal sinus            0.57   Moderate
Maxillary sinus          0.54
Sphenoid sinus           0.53
Posterior ethmoid sinus  0.48
Anterior ethmoid sinus   0.44

Ostiomeatal complex      0.39   Fair

Overall                  0.51   Moderate

Table 5

Our degree of agreement according to our risk-assessment system

                            Kappa  Degree of
Structure                   value  agreement

Absent frontal sinus         0.65  Substantial

Pneumatized ethmoid roof     0.45  Moderate
Uncinate process             0.45

Cribriform plate             0.21  Fair

Asymmetric sphenoid septum   0.19  Slight
Lamina papyracea             0.11
Agger nasi cells             0.09
Optic nerve cover            0.09
Absent sphenoid sinus        0.00
Internal carotid cover       0.00

Hailer's cells              -0.05  Poor

Overall                      0.16  Slight

References

(1.) Zinreich SJ, Kennedy DW, Rosenbaum AE, et al. Paranasal sinuses: CT imaging requirements for endoscopic surgery. Radiology 1987;163:769-75.

(2.) Oluwole M, Russell N, Tan L, et al. A comparison of computerized tomographic staging systems in chronic sinusitis. Clin Otolaryngol 1996;21:91-5.

(3.) Lund VJ, Mackay IS. Staging in rhinosinusitis. Rhinology 1993;31:183-4.

(4.) Rontal M, Rontal E. Studying whole-mounted sections of the paranasal sinuses to understand the complications of endoscopic sinus surgery. Laryngoscope 1991;l01:361-6.

(5.) Streitmann MJ, Otto RA, Sakai CS, Anatomic considerations in complications of endoscopic and intranasal sinus surgery. Ann Otol Rhinol Laryngol 1994;103:105-9.

(6.) Meyers RM, Valvassori G. Interpretation of anatomic variations of computed tomography scans of the sinuses: A surgeon's perspective. Laryngoscope 1998;l08:422-5.

(7.) Melhem ER, Oliverio PJ, Benson ML, et al. Optimal CT evaluation for functional endoscopic sinus surgery. AJNR Am J Neuroradiol 1996;17:181-8.

(8.) Lund VJ, Wright A, Yiotakis J. Complications and medicolegal aspects of endoscopic sinus surgery. J R Soc Med 1997;90:422-8.

(9.) Mafee MF, Chow JM, Meyers R. Functional endoscopic sinus surgery: Anatomy, CT screening, indications, and complications. MR Am J Roentgenol 1993;160:735-44.

(10.) Stammberger H. Functional Endoscopic Sinus Surgery. Philadelphia: B.C. Decker, 1991:142-3.

(11.) Landis JR. Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.

(12.) Stankiewicz JA. Complications in endoscopic intranasal ethmoidectomy: An update. Laryngoscope 1989;99:686-90.

(13.) Molony NC, Ah-See KW. The contemporary practice of functional endoscopic sinus surgery: A nationwide survey. Clin Otolaryngol 1998:23:331-8.

(14.) No authors listed. Clinical disagreement: I. How often it occurs and why. Can Med Assoc J 1950;123:499-504.

(15.) Bolger WE, Butzin EA, Parsons DS. Paranasal sinus bony anatomic variations and mucosal abnormalities: CT analysis for endoscopic sinus surgery. Laryngoscope 1991;10l:56-64.

(16.) Lloyd GA, Lund VJ, Scadding GK. CT of the paranasal sinuses and functional endoscopic sinus surgery: A critical analysis of 100 symptomatic patients. J Laryngol Otol 1991;105:181-5.

(17.) Jones NS, Strobl A, Holland I. A study of the CT findings in 100 patients with rhinosinusitis and 100 controls. Clin Otolaryngol 1997;22:47-51.

(18.) Stammberger HR, Kennedy DW. Paranasal sinuses: Anatomic terminology and nomenclature. The Anatomic Terminology Group. Ann Otol Rhinol Laryngol Suppl 1995;167:7-16.

From the Department of Otolaryngology, Ninewells Teaching Hospitals, Dundee, Scotland (Dr. El-Hakim, Dr. Oluwole, and Dr. White), and the Department of Otolaryngology, Raigmore Hospital, Inverness, Scotland (Dr. MacKerrow and Dr. McClymont).

Reprint requests: Dr. Hamdy El-Hakim, Paediatric Otolaryngology, 2C3.59 Walter MacKenzie Centre, Edmonton T6G 2B7 Alberta, Canada. Phone: [phone]; fax: [phone]; e-mail: [e-mail]

Originally presented during a meeting of the Otorhinolaryngological Research Society; October 1997; London.