«MIST (Melbourne Initial Screening Test) Evaluation Studies February 1998 Part 1: Reliability and validity testing of the MIST Part 2: A negative ...»
MIST (Melbourne Initial Screening Test) Evaluation Studies
Part 1: Reliability and validity testing of the MIST
Part 2: A negative predictive study of the MIST
Part 3: A positive predictive study of the MIST
Linda Santamaria, MAppSc, DipAppSc, DOBA
Shayne Brown, MAppSc, DipAppSc, DOBA
Ian Story, PhD, BBSc
Department of Clinical Vision Sciences
La Trobe University
Address for Correspondence
Department of Surgery, Monash University Monash Medical Centre Level 5, Block E 246 Clayton Rd Clayton, VIC, 3168 Project Funding The MIST implementation and evaluation projects were funded by The Department of Human Services, Victoria, Australia.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) MIST (Melbourne Initial Screening Test) evaluation studies ABSTRACT The aim of these three studies was to assess the test-retest reliability of the MIST (Melbourne Initial Screening Test), a new vision screening tool designed for use by vision screeners of preschool children and to determine the positive and negative predictive value.
In the first study, 471 children participated in two sessions, either with the Sheridan Gardiner Singles (SGS) or the MIST. The children were allocated to four groups - Group A, test-retest with SGS, one orthoptist; Group B, test-retest with MIST, one orthoptist; Group C, test-retest with MIST, two different orthoptists; or Group D, test-retest with MIST, an orthoptist and a nurse. The test-retest reliability of the MIST demonstrated correlations of 0.79 and 0.78 for the right and left eyes respectively, in comparison to the SGS score of 0.71 and 0.79. The inter-rater reliability of the MIST demonstrated correlations of 0.60 and 0.74 for orthoptists, and 0.63 and
0.60 with orthoptists in comparison to nurses. These results confirm that the MIST is a reliable screening tool for pre-school vision screeners.
In the second study, 201 children participated in a gold standard orthoptic and ophthalmic examination after passing their MIST screening. The negative predictive value of the MIST vision screening program was found to be 97.5%. Five children were found to have failed the gold standard examination. Four of the children had an astigmatic error of 1.50 dioptres in one eye, one of whom also failed the visual acuity criterion. One other child had an intermittent esotropia. This result means that 2.5% of the children who pass the MIST vision screening test may actually have strabismus, amblyopia or a refractive error, something to be noted in any screening program. Both nurses and parents must be aware of this information when vision screening is performed.
In the third study, forms were returned for 3,854 children referred from the MIST vision screening program over a three-year period. Visual assessment results of 2,623 of these children were obtained from examining clinicians. The positive predictive value of the MIST was found to be 44.4%. Given the less than 5% prevalence of amblyopia in children of this age group, this over-referral rate would be considered acceptable. A balance must be reached between the positive and the negative predictive values. The follow-up examination of each child is a cost to the health care system to be considered in balance with the early detection of a visual problem and any long-term costs associated with visual disorders.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) The MIST (Melbourne Initial Screening Test) has been designed as a simplified vision screening test for 3.5 to 4.5-year old children, to be performed by maternal and child health nurses in the state of Victoria.1 It was introduced in 1998 as part of the visual surveillance program, combined with observation of ocular position and corneal reflections and questions regarding family history and ocular problems. As previously described by Brown and Story1 the MIST, a letter matching test, has a pass/fail method of assessment rather than a threshold test of visual acuity, with five test letters of 3/5 size. Initial results showed that it was easy to administer by the nurses and had a high compliance rate.
The maternal and child health nurses perform vision screening as part of a general developmental assessment, allowing earlier screening at minimal cost to the health system. As part of a wider assessment this allows vision screening in the context of the child’s whole development. This surveillance program provides not only assessment, but also gives the chance to raise parental awareness and provide incidental education on many health and development issues.
Since the implementation of the MIST in 1998 there has been much public debate concerning the value of preschool vision screening in the absence of randomised control studies investigating the age effect of amblyopia treatment and outcomes, with the recommendation that programs should be discontinued.2 Various authors then emphasised the need for further research and it may be suggested that in the absence of appropriate evidence on such aspects as the natural history of amblyopia, the age effect of amblyopia treatment outcomes, or whether amblyopia is disabling, that preschool vision screening should continue in the most cost-effective way.3-5 The MIST forms part of the routine general developmental assessment in a similar manner to that reported by Thorburn and Roland6. Previous reports have suggested that orthoptists are more effective at vision screening than other health professionals,7, 8 however studies have shown that screening can be performed effectively by others providing that training is given to ensure high quality assessments.6 The accepted criteria for a screening test are that it must be simple, reliable and valid.9 The purpose of the evaluation project, funded by The Department of Human Services, was to evaluate the effectiveness of the MIST as a valid vision screening tool, with three separate studies, the reliability and validity testing of the MIST, a negative predictive study and a positive predictive study.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) Part 1: Reliability and validity testing of the MIST
INTRODUCTIONThe reliability of any screening test must be evaluated not in the clinic, but in the hands and venue of the screening personnel. In this context, test-retest results could be considered in two ways, either as the repeatability of an individual test score or of a pass/fail classification only.10, 11 The former may provide more detail of individual test results, but in the context of vision screening, pass or fail is the information required to make a referral decision.
There have been few studies on the test-retest reliability of available visual acuity tests. One issue to be considered is the number of children who are able to complete a test at the first presentation. Hered and colleagues12 found with the HOTV chart at 3 metres, that 93% of 3 to 5year olds were testable at the first session compared to 97% at a second session, with 85% of 3year olds and 98% of 4-year olds testable at the first session. Similar results were reported by other studies.11, 13 A study, with single optotypes at 6 metres reported a lower testability rate of 64% in children less than 3 years old, but 93% in 4-year olds.14 Another, using isolated surrounded optotypes at 4 metres, found 67% of 3-year olds and 87% of 4-year olds testable.15 These and other studies show an increase in testability with increased age, decreased testing distance and repeat testing.16 Test-retest reliability studies have reported widely varying results for correlational values. Hered and colleagues,12 grading each child as pass, fail or untestable, reported an F value of 0.54 for the HOTV chart with 3 to 5-year old children, or a value of 0.71 after excluding all children untestable at the first session. In comparison, Friendly11 found a rho value of only 0.16 for a slightly wider age group, and Sprague et al13 a correlation coefficient of 0.398, using continuous variable measurements. A more recent study, using isolated surrounded HOTV letters, reported a correlation of 0.82 in a group of 2 to 7-year old children.15 McGraw et al17 studied the test-retest reliability of a single letter test at 3 metres and found a coefficient of repeatability of two lines difference in a group of children with mean age 5.3 years. In adult populations, the correlation coefficients have been reported to be much higher at 0.9818 or 0.8419 for the Bailey-Lovie LogMAR chart. Comparisons between studies are difficult as methodology and statistical analyses vary widely.
For a screening test to be efficient it must have a high compliance rate, so that a result can be obtained from the majority of participants. As a screening program is designed only to identify those at risk, an actual measurement of visual acuity is not required. This may in effect mean a balance between ease of administration and accuracy of the result. The referral practice would tend to be towards referring any child who shows any risk of failing the test. The aim of this study was to assess the test-retest reliability of the MIST. The test-retest reliability of the Sheridan Gardner Singles (SGS) was assessed as it is a standard clinical test, commonly used in clinics and vision screening programs. The MIST was then evaluated for its reliability in comparison to this clinical test, both with experienced eye health personnel and within its screening context with maternal and child health nurses.
Participants Children were recruited from kindergartens and child care centres in the Melbourne metropolitan region. After an initial approach to centres, written consent was obtained from those wishing to participate. Parental Information and Consent forms were then distributed, to be signed and returned prior to testing. The study was approved by the Human Ethics Committee, La Trobe University, and the Department of Human Services Ethics Committee.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) Procedures Vision was tested with either the SGS at 6 metres or with the MIST at 3 metres, both letter matching tasks. The right eye was tested first as a routine procedure. The time taken for the vision tests was recorded.
The aim was to assess each child twice, by either the same or a different tester. Each centre was allocated to one of four groups. The SGS test-retest (Group A) and the MIST test-retest (Group
B) were retested by the same orthoptist. Two other orthoptists were involved in the MIST interrater test by different orthoptists (Group C). Seven maternal and child health nurses were involved in the MIST inter-rater test of orthoptists in comparison to nurses (Group D).
The results of each test were graded as pass, fail or untestable. As the data was ordinal measurement, correlational analysis was performed with Spearman’s rho (ρ). Due to the extreme skew in the distribution of the responses, where the agreement is predominantly in one cell, measures of agreement such as Cohen’s kappa will tend to underestimate the true agreement. In this study, where the children were from a normal population, the vast majority of results would be in the pass/pass cell, with only a few in the fail/fail cell, so it is not possible to statistically test the amount of disagreement.
An orthoptist or a nurse, with either the SGS or the MIST, tested a total of 583 children. As some were absent at the time of one of the testing sessions, 471 children participated in the full testretest study. The children ranged in age from 35 to 67 months. The number of children, their mean age and the period of time between the two testing sessions are presented in Table 1.
Group A: Test-retest reliability of the SGS, conducted by one orthoptist In this group of children, Orthoptist 1 tested the children using the SGS at both sessions. A referral is recommended in cases where the vision for either one or both eyes is classed as a fail, therefore the results are classed as a pass if the child passes with each eye or a fail if the child fails in either eye. The contingency table presents the result for each child, for the test and retest (Table 2).
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) Table 2 Contingency table of the pass/fail results for SGS retest, conducted by one orthoptist
The SGS vision score was categorised as the line of visual acuity achieved, from 6/60 to 6/6.
Using Spearman’s correlation coefficient of this score, a significant relationship was found between the SGS test and retest results for both the right and left eyes (ρ = 0.71, ρ = 0.79, respectively, p = 0.0001).
Group B: Test-retest reliability of the MIST, conducted by one orthoptist In this group of children, both testing sessions were performed using the MIST by the same orthoptist (Orthoptist 1). The contingency table presents the pass/fail/untestable results for the MIST test-retest (Table 3).
The MIST is scored as the number of correct letters. Using Spearman’s correlation coefficient of the MIST score, a significant relationship was found between the MIST test and retest results for both the right and left eyes (ρ = 0.79, ρ = 0.78, respectively, p = 0.0001).
Group C: Inter-rater reliability of the MIST, conducted by two different orthoptists In this group of children, both testing sessions were performed using the MIST, but with a different orthoptist at each session. Three orthoptists were involved with the MIST testing procedure in different combinations (Orthoptists 1, 2 & 3).
The contingency table shows the pass/fail/untestable results for the MIST for each child (Table 4).
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) Using Spearman’s correlation coefficient of the MIST score, a significant relationship was found between the MIST test and retest results for both the right and left eyes (ρ = 0.60, ρ = 0.74, respectively, p = 0.0001).