«MIST (Melbourne Initial Screening Test) Evaluation Studies February 1998 Part 1: Reliability and validity testing of the MIST Part 2: A negative ...»
Group D: Inter-rater reliability of the MIST, orthoptist and nurse In this group of children, both testing sessions were performed using the MIST, an orthoptist at one session and a nurse at the other. One orthoptist (Orthoptist 1) and seven nurses were involved with the MIST testing procedure. Seventy-nine of the children (59%) were seen by the orthoptist initially, followed by a nurse. Regardless of tester, 86% of children passed this vision test at the first session, increasing to 93% by the second test. The number of children who could not be tested decreased from 6.8% to 1.5% on retest.
As the aim in this group of children was a comparison of the orthoptist’s and the nurses’ results with the MIST, analysis was done comparing these two results rather than those of Test 1 against Test 2. The contingency table shows the pass/fail/untestable results for the MIST test-retest sessions, orthoptist compared to nurse (Table 5).
Table 5 Contingency table of the pass/fail results for MIST retest, orthoptist versus nurse
Using Spearman’s correlation coefficient of the MIST score, a significant relationship was found between the orthoptist and nurse MIST results for both the right and left eyes (ρ = 0.63, ρ = 0.60, respectively, p = 0.0001).
Testing time The time taken to perform the SGS and the MIST is presented in Table 6.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) DISCUSSION Test-retest and inter-rater reliability of SGS and MIST The first stage of this study was to determine the test-retest reliability of the SGS. If visual acuity was recorded as the line achieved, then it could be seen that SGS test-retest reliability shows a correlation of 0.71 in the right eye and 0.79 in the left eye. This high correlation showed good but not perfect reliability of this test, with some variation in the test-retest results expected for children in this age group. One reason for the test-retest reliability being a little higher in the left than the right eye may be due to a learning effect, as the right eye was always tested first, therefore learning and confidence may increase within the testing session, tending towards a higher correlation for the left eye. This study has shown that the SGS test is a reliable clinical test for the test-retest of visual acuity with children in the 3.5 to 4.5-year age group. These results are difficult to compare with any previous studies as the methods of analysis differ, with the previously reported correlation results varying from 0.16 to 0.54 for vision charts,11-13 versus 0.82 for a surrounded single test.15 The MIST test-retest correlation, when tested by the same orthoptist, was 0.79 for the right eye and 0.78 for the left eye. This result demonstrates a level of reliability similar to that of the SGS, with a slightly higher result in the right eye for the MIST than the SGS. It may be considered that the MIST, being a simpler test than the SGS is less learning dependent within the session than the SGS, and so shows a more equal correlation with each eye. However, this is not particularly shown in the other two MIST test-retest groups. In the context of repeat testing by one eye health care professional, the MIST produces as good a test-retest correlation as the SGS.
With two different orthoptists performing the MIST, the inter-rater reliability values were 0.60 with the right eye and 0.74 with the left eye. This moderate to good correlation, indicates a substantial relationship, though a little less than with one tester. This was achieved with a combination of three orthoptists, in different combinations of first and second tester. The use of three rather than two orthoptists for this assessment may have decreased the inter-rater reliability, but allowed an evaluation within the context of different clinical testers.
The inter-rater relationship between the MIST results gained by an orthoptist and a nurse also produced a moderate to good correlation, with 0.63 in the right eye and 0.60 in the left eye. This was achieved with the use of seven nurses as testers, which may reduce the reliability result, but would provide a more realistic evaluation of the test in the screening context of many different nurses.
Pass/fail results for test-retest of SGS and MIST The aim of a screening test is to determine those children who may have a vision problem and require further assessment. One factor required for the success of a screening program is that the test has a high compliance rate so that the majority of the children can be successfully graded as pass or fail. The number of children achieving a pass level at the first test was identical for both the MIST and the SGS, at a level of 86% with an orthoptist, increasing to around 92% on retest.
This increase in testability of a clinical test from the first to a second session confirms previous reports.12, 13, 16 When comparing the test completion rates of the nurses to that of the orthoptists, it was found that more children were successfully tested by an orthoptist (93%) than by a nurse (87%).
The number of children who were untestable on the first test with the SGS was 10%, decreasing to 2% on the retest. For the two MIST retest groups, the number of children untestable was around 5% to 7%, with no real change from the first to the second test. This higher rate of compliance at the first test might be explained again by the simpler design of the MIST, with learning having a lesser effect from the first to the second test. However, this is comprised of a higher fail rate at the first session with the MIST than with the SGS. One hypothesis to explain this may be that the simpler design of the MIST means that the test is more in the control of the This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) child being tested than in the skill and experience of the tester. In this case the less cooperative children are more likely to fail in the first session. A similar number of children in both the SGS and the MIST retest groups, 9% and 8.4% respectively, failed or were untestable at the first test and then passed at the second test, showing a learning effect. In a screening context these children would represent false positive results.
One group of interest in a screening program are the children who were untestable at the first test and then determined as a fail at the retest session. As the numbers of these children in each of the groups was small, it was not possible to determine any increased risk factor, but the question would remain after a single testing, whether these children are unable to complete the task due to their cognitive ability, requiring sustained concentration, or to their visual ability. It could also be seen that a small number of children actually passed the first test and then failed or were untestable at the retest. This occurred in both the SGS and MIST retest groups. All of the five children who changed from a pass to a fail on retest were aged 45 months or younger. It must be noted, that in the normal population at this age a small number of children may be expected to be unable to complete the task and some variation in cooperation from one session to another may be expected.
In the orthoptist/nurse MIST retest group there was a small group of children who were passed by one clinician, but were failed or untestable by the other. Three children were passed by a nurse but failed by an orthoptist. Two of these children were passed on their second test, one passed their first test. Six children were passed by the orthoptist, but failed by the nurse, three were failed on their first test and three on the retest. As some were failed at each session, the learning effect cannot be considered the major reason for these discrepant results. These children with conflicting results from an orthoptist and a nurse pose several questions. There were a larger number of children failed by a nurse but passed by an orthoptist, which may be due to the relative lack of experience of the nurses in vision testing of small children.
It can be seen that for testing by an orthoptist, the mean time required to perform the MIST is less than that required to complete the SGS. The nurses required more time to complete the MIST than did the orthoptist, but again less time for the repeat test. This is understandable given the difference between an experienced eye health care professional and the nurses for whom the test had been relatively recently introduced. It can be inferred that the nurses would take much longer to perform the SGS than the MIST, so the MIST is a quicker screening test to perform, an advantage given that the vision screening is performed within the context of a total developmental screening session. This compares favourably with Thorburn and Roland’s report6 that vision testing with single letters by health visitors took 8.1 minutes within the routine developmental check.
The results from this study would confirm those of other studies that both the testability rate and number of children passing a second test is increased.12, 13, 16 The MIST may appear to be biased towards a higher fail rate, particularly in the first session, but this cost of the false positive results must be balanced by the risk associated with false negative results from a vision screening program. In a screening program a decision must be made whether to refer after one screening test or to retest each child. This study would confirm the recommendation to retest each child who fails or is untestable at the initial vision screening,12, 13, 16 however, this must be considered in the light of the nurses’ workplace, and the fact that the 3.5-year assessment is the last for children within this system. If there is a high level of false referrals, the issue of costs to the public health system, and the financial and emotional costs to parents whose child is referred on for further assessment must be considered. However, this must be balanced by the consideration of the nurses’ time and organisation required for a second screening session and the number of children who may not return for a retest, with the risk that they actually have a fail result and so do not gain further follow-up.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) CONCLUSION In conclusion this study has demonstrated that the MIST has a pass rate as high as that of the SGS, with 86% of children passing each at the first test, with the nurses showing a similar rate to the orthoptists. The number of children who were untestable with the MIST was in the order of 5% to 7%, lower than that of the SGS, but this was complemented by an increased fail rate at the first test. The MIST has been demonstrated to be quicker to perform than the standard clinical test of SGS, taking approximately 2.25 to 5 minutes for the nurses to perform on children for the first time, with a mean time of 4.3 minutes.
The test-retest reliability of the MIST demonstrated as strong a relationship as that of the SGS, confirming the reliability of this new vision screening test in comparison to one of the standard clinical tests. In the context of a vision screening program, in the hands of maternal and child health nurses, rather than the clinical context of eye health professionals, the inter-tester correlation is not quite as strong, but a substantial relationship was still found. It may be expected that the orthoptist/nurse inter-rater reliability would improve as the nurses become more experienced with the test.
This document is managed by the Department of Education and Early Childhood Development, Victoria (as of 27 August 2007) Part 2: A negative predictive study of the MIST