«IHC Membrane Image Analysis User’s Guide ©Copyright 2007 Aperio Technologies, Inc. Part Number/Revision: MAN-0026, Revision B Date: January 2, ...»
6. Export the algorithm with its parameter setting as a macro to a file. It might be a good idea to use a filename that identifies the stain, the manufacturer and the tissue type as well as the Training algorithm mode. Store the algorithm macro with the training data sets so you know where it belongs and can find it easily.
7. Import the algorithm macro for automatic training into Spectrum Information Manager. It might be a good idea to use a Macro Name that identifies the stain, the manufacturer and the tissue type as well as the Training algorithm mode to identify this macro by in Spectrum Information Manager.
8. Run the automatic training algorithm macro on all digital slides in the training data sets except those that you want to exclude from the training data set. Note that there is no need to create a markup image for algorithm training as the scoring scheme parameters that determine the classification of the cells have not been determined yet and the markup images for the cell feature detection parameters have already been generated. So you may not want to select the option to create a markup image when running the algorithm macro as this accelerates the processing.
9. Delete the automatic training algorithm macro in Spectrum Information Manager when the training is done to ensure that no further digital slides are added to the training by accident and your training results are preserved.
36 IHC Membrane Image Analysis User’s GuideChapter 6 – Algorithm Setup
10. Retrieve the trained scoring scheme parameters provided by the automatic training. The trained scoring scheme parameters are provided in a file with the extension.trr in the folder defined by the Classifier Definition File in the ScanScope program folder under AAF and Algorithms (\Program Files\ScanScope\AAF\Algorithms\Classifier Definition File). Open the file with Notepad. The last four numbers in descending order correspond to the cell classification thresholds. The first three correspond to the intensity thresholds (e.g., Weak (1+) Threshold = 200, Moderate (2+) Threshold = 175 and Strong (3+) Threshold = 140) and the fourth one to the completeness threshold (e.g., Completeness Threshold = 60).
IHC Membrane Algorithm8.001 2.000000e+002 1.750000e+002 1.400000e+002 6.000000e+001
Create Final Algorithm Macro Now that you have determined the best parameter settings based on the training data sets, create the final algorithm macro to be used for the analysis.
1. Start with the automatic training algorithm macro and set the cell nuclear classification thresholds to the values provided by the automatic training.
2. To use the algorithm for analysis, set Use Mode to 0 – Analysis/Tuning. and Markup Image Type to 1 – Analysis.
3. Export the algorithm with its parameter setting as a macro to a file. It might be a good idea to use a filename that identifies the stain, the manufacturer and the tissue type as well as the Analysis algorithm mode. Store the algorithm macro with the training data sets so you know what it belongs to and can find it easily.
4. Import the final algorithm macro for analysis into Spectrum Information Manager. It might be a good idea to use a Macro Name that identifies the stain, the manufacturer and the tissue type to identify this macro by in Spectrum Information Manager.
Validate Algorithm Setup To validate the setup of the algorithm, run the IHC algorithm on the training data sets, then review the markup images and analyze the scores provided by the algorithm against the scores provided by the pathologists.
The scores provided by the pathologists and the algorithm can easily be exported from Spectrum Information Manager into an Excel Spreadsheet. Spectrum Information Manager exports a csv file (comma separated text file). To open the file directly in Excel, make sure you have Excel installed on your computer.
1. To export the annotations layers from Spectrum Information Manager select all digital slides under Specimen Digital Slides by selecting the check box in the title bar, and then click Export.
You can choose to either Open or Save the file. All the information from all layers and from all selected digital slides is exported.
In Excel you can delete or hide the columns you are not interested in, resulting in a summary table like the following. If you use multiple copies of the training data set, you would export a summary table for each copy of the training data set.
HER2 Score - HER2 Score Pathologist - Algorithm Use the summary tables for the manual read of the digital slide and image analysis as well as the manual read of the glass slide scores to do your statistical analysis to evaluate how good your agreement is between image analysis and the manual scoring for the training data set.
If you find problems with the markup images or do not achieve good agreement on the training data set, you need to investigate what the origin of the problem is and re-do the setup of the algorithm.
Once the algorithm setup has been completed, access rights to the specific application data groups can be restricted to Read Only or No Access to make sure that no data can be modified. If you want other pathologists or customers to be able to view the data used for the setup you can provide them Read Only access rights.
38 IHC Membrane Image Analysis User’s Guide Validation How well the IHC Membrane Image Analysis algorithm will perform for your application will depend on your setup of the algorithm and your slide preparation. Before using the IHC Membrane Image Analysis algorithm for analysis, we recommend validating the performance of the algorithm.
How to validate the IHC Membrane Image Analysis algorithm for your application is entirely up to you.
We show an example of what we think could be a possible study design and how to use Spectrum Information Manager to execute the study.
Study Design Example As there is no gold standard for IHC scoring, we base the validation on the scores from a panel of three pathologists. The goal for the performance validation study should be to demonstrate substantial equivalence of IHC Membrane Image Analysis algorithm to conventional manual microscopy. This can be demonstrated by calculating measures of reliability separately for pathologists using manual microscopy and the image analysis system, and across both methods. Substantive equivalence will be established by finding no substantial differences for the separate and combined estimates of reliability.
The lab should use FDA approved kits/reagents for HER2. The lab should employ appropriate morphological studies and controls as specified in the instructions for the IHC kits/reagents.
The evaluation slides should be taken from the clinical operation as the cases come in. A sufficient number of slides per HER2 score: 0, 1+, 2+ and 3+ are required to provide sufficient power for the statistical analysis.
The slides in the data set should pass a quality inspection by a pathologist in terms of having a good representation of tumor regions present in the specimen and cleanness and good condition of the slides in order to be included in the validation study.
The slides should be made anonymous and therefore not contain any human subject information.
The study should have three blind evaluation arms: manual microscopy, digital slide scoring and image analysis.
Manual Microscopy Three pathologists should score independently the glass slides using manual microscopy.
Between the manual microscopy and the digital slide scoring there should be a wash-out period of at least one week and the slides should be randomized and scanned.
Digital Slide Scoring The same three pathologists should then score independently the digital slides and outline the tumor regions.
Image Analysis The image analysis should then be performed on all digital slides analyzing them by the pathologists’ outlined tumor regions. We separate the image analysis from the digital slide scoring so that we could easily re-run the validation study when new algorithm versions are released by just re-running the algorithm and performing the statistical analysis.
Validate Algorithm Performance The performance validation data set can be organized in Spectrum Information Manager exactly the same way as the training data set for the algorithm setup using the same cases (projects) for the specific applications.
The only difference is the annotation file that only needs to include the HER2 score.
The rest of the validation is the same as the validation of the algorithm setup.
If you find problems with the markup images or do not achieve good agreement between scores, you need to investigate what the origin of the problem is and go back to the setup of the algorithm.
Once the algorithm validation has been completed, access rights to the specific application data groups can be restricted to Read Only or No Access to make sure that no data can be modified. If you want other pathologists or customers to be able to view the data used for the validation you can provide them Read Only access rights.