Tweet
WHO external quality assessment project for the detection of subtype influenza A viruses by polymerase chain reaction ? summary analysis, 2007 and 2008 [WHO WER]
Weekly epidemiological record - 7 NOVEMBER 2008, 83rd YEAR - No. 45, 2008, 83, 401?412 - http://www.who.int/wer - 2008, 83, 401?412 No. 45 - WORLD HEALTH ORGANIZATION - Geneva
WHO external quality assessment project for the detection of subtype influenza A viruses by polymerase chain reaction ? summary analysis, 2007 and 2008
[Edited version. Full text with accompanying tables at: http://www.who.int/wer/2008/wer8345.pdf ]
Introduction
For more than 50 years, National Influenza Centres (NICs)1 have been the backbone of WHO?s Global Influenza Surveillance Network and WHO?s Global Influenza Programme.
The centres collect specimens, conduct preliminary analyses and send representative virus isolates in a timely manner to WHO collaborating centres to support the annual recommendation of
influenza vaccine composition for the next season.
The tests used are mainly virus isolation and haemagglutination inhibition.
Since early 2004 the ongoing presence of the A(H5N1) strain of avian influenza has increased the need to include polymerase chain reaction (PCR) as part of the laboratory process in order to improve the accuracy and efficiency of diagnosis.
PCR has become the principal laboratory test for detecting H5 infections, and it has facilitated patient management, outbreak response and pandemic preparedness.
Although many NICs had started using PCR prior to 2006, until that time no globally coordinated external quality assessment programme existed to monitor the quality and comparability of laboratories? performance. WHO?s external quality assessment project for the detection of influenza virus type A by PCR was established to improve the global laboratory capacity for diagnosing both seasonal influenza and avian influenza.
It specifically aims to monitor the quality and standards of performance of NICs and other influenza laboratories in detecting influenza A virus subtypes using PCR as well as to promote good laboratory practice.
Coordinated by WHO?s Global Influenza Programme, based at WHO?s headquarters, the external quality assessment project is being conducted by the WHO reference laboratory for diagnosis of influenza A/H5 infection, National Influenza Centre, Centre for Health Protection, Hong Kong, Special Administrative Region of China (Hong Kong SAR), with support from WHO?s collaborating
centres on influenza, other influenza A/H5 reference laboratories and WHO regional offices.
This report summarizes the results of laboratories that participated in the assessment exercise by analysing 4 influenza test panels dispatched over a 2-year period from 2007 to 2008.
Preparation of panels
Panels sent to participating laboratories consisted of vacuum-dried RNA specimens extracted from influenza A(H1N1), A(H3N2) and A(H5N1) viruses. Prior to dispatch, the panels were tested by 7 laboratories in Hong Kong SAR to assess sample quality and homogeneity. To assess the effects of storage and transportation at ambient temperature, samples were tested after 7 days of storage at 37 ?C using both conventional and realtime PCR assays2 to confirm sample stability. The samples were quantified by real-time PCR assay using known synthetic RNA standards.
Composition of panels
The 4 panels consisted of coded samples containing RNA at different concentrations from 5 strains representing different genetic clades (1 and 2) of influenza A(H5N1) viruses, A(H1N1) viruses and A(H3N2) viruses.
Samples that contained no virus were also included.
Details of the composition of the panels are shown in Table 1. Participants were instructed to reconstitute each sample with the provided reconstitution buffer prior to testing. Additionally, a questionnaire was included to obtain information on detection methods used, target genes tested and primer and probe sequences employed.
Distribution of panels and response of participants
NICs and other national influenza laboratories were invited to participate before the panels were dispatched. Panel 1 was dispatched between February 2007 and March 2007; panel 2 was dispatched between August 2007 and October 2007; panel 3 was dispatched between January 2008 and February 2008; and panel 4 was dispatched between June 2008 and July 2008. All panels were dispatched from the Centre for Health Protection in Hong Kong SAR at ambient temperature by courier service to participating laboratories in all 6 WHO regions.
Participating laboratories were requested to notify the Centre for Health Protection immediately on receipt of the panel, either by fax or e-mail, and to return their results within 1 month. Each laboratory was assigned a participant identity code known only to the Centre for Health Protection and the Global Infl uenza Programme.
Laboratories that did not respond within the specified time frame were reminded by e-mail. A preliminary report, including correct results, was sent to participants shortly after the closing date for each panel.
Results for panel 1 were reported by 64 laboratories from 54 countries, areas or territories; results for panel 2 were reported by 83 laboratories from 66 countries, areas or territories; results for panel 3 were reported by 95 laboratories from 77 countries, areas or territories; and results for panel 4 were reported by 109 laboratories from 83 countries, areas or territories.
Details are shown in Table 2 and Table 3.
Most participants received the panels within 1 week (panel 1, 81% received within 1 week; panel 2, 87%; panel 3, 86%; panel 4, 85%). Across all 4 panels, nearly half (47%) of participating laboratories were situated in WHO?s European Region; 9% were in the African Region; 15% were in the Region of the Americas; 5% were in the Eastern Mediterranean Region; 5% in the South-East Asia Region; and 19% in the Western Pacific Region.
Results
All results returned before the closing date for each panel were included in the analysis, and a full report on laboratories? performance on the panel was sent to all participating laboratories within 1 month after release of the preliminary report.
This assessment required qualitative results (that is, positive or negative) of the target genes tested; quantitative results (that is, copies/μL and threshold cycle value) were used only for reference. The performance of individual laboratories was assessed by adding up the number of correct results. The following standards were used when assessing laboratories? results:
(i) failing to detect H5 samples and/or reporting the results as non-H5 subtype were recorded as an incorrect response;
(ii) failing to detect H1 samples and/or reporting the results as non-H1 subtype were recorded as an incorrect response;
(iii) failing to detect H3 samples and/or reporting the results as non-H3 subtype were recorded as an incorrect response;
(iv) failing to report correct influenza A test results for H1/H3 samples if H1/H3 subtyping was not performed was recorded as an incorrect response;
(v) reporting positive results for a sample that did not contain any viral RNA was recorded as an incorrect response.
Performance of the laboratories
Panel 1
A total of 43 participants returned correct results for all 10 samples in panel 1 (Table 3). An additional 6 participants returned correct results for 9 samples and 9 participants for 5?8 samples; 6 participants returned correct results for less than half of the samples. For the 2 negative samples (denoted as 2007-5 and 2007-10), 9 participants reported positive results. The corresponding
false-positive rates were 8% and 6%, respectively (Table 1).
The 3 strongly positive H5 samples that contained higher concentrations of viral RNA (denoted 2007-1, 2007-6, 2007-3) were correctly reported by 56 (88%) participants, 58 (91%) and 57 (89%), respectively. The 3 less strongly positive H5 samples (denoted 2007-7, 2007-2, 2007-9) were correctly reported by 53 (83%), 55 (86%) and 53 (83%) participants, respectively.
The 1 sample of H1 and 1 sample of H3 were correctly reported by 57 (89%) participants.
Panel 2
A total of 54 participants returned correct results for all 14 samples (Table 3). An additional 14 participants returned correct results for 13 samples and 12 participants for 8?12 samples; 3 participants returned correct results for less than half of the samples.
For the 4 negative samples (2007-15, 2007-18, 2007-21, 2007-24), 5 participants reported positive results. The corresponding false-positive rates were 2% for 2007-15, 4% for 2007-18, 4% for 2007-21 and 1% for 2007-24 (Table 1).
The 4 samples of H5 with higher concentrations (2007- 11, 2007-22, 2007-23, 2007-16) were correctly reported by 79 (95%), 77 (93%), 80 (96%) and 79 (95%) participants, respectively. The 4 samples of H5 with lower concentrations (2007-19, 2007-12, 2007-14, 2007-20) were correctly reported by 76 (92%), 76 (92%), 77 (93%) and 75 (90%) participants, respectively.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 74 (89%) and 73 (88%) participants, respectively.
Panel 3
A total of 70 participants returned correct results for all 10 samples (Table 3). An additional 10 participants returned correct results for 9 samples and 11 participants for 6?8 samples; 4 participants returned correct results for less than half of the samples.
For the 2 negative samples (2008-04, 2008-08), 2 participants reported positive results. The corresponding false-positive rates were 1% and 2%, respectively (Table 1).
The 4 samples of H5 (2008-05, 2008-01, 2008-02, 2008-03) were correctly reported by 85 (89%), 88 (93%), 90 (95%) and 89 (94%) participants, respectively. For the 2 duplicate H5 samples (2008-07, 2008-09), the performance of the participants was consistent: both were correctly reported by 89 (94%) participants.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 83 (87%) and 84 (88%) participants, respectively.
Panel 4
A total of 84 participants returned correct results for all 10 samples (Table 3). An additional 11 participants returned correct results for 9 samples and 10 participants for 6?8 samples; 4 participants returned correct results for less than half of the samples.
For the 2 negative samples (2008-19, 2008-20), 6 participants reported positive results. The corresponding false-positive rates were 5% and 2%, respectively (Table 1).
The 4 samples of H5 with a single concentration (2008- 11, 2008-13, 2008-15, 2008-17) were correctly reported by 104 (95%), 105 (96%), 104 (95%) and 105 (96%) participants, respectively. For the clade 2.3.2 strain with higher and lower concentrations (2008-16 and 2008-12, respectively), correct results were reported by 104 (95%) and 103 (94%) participants, respectively.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 93 (85%) and 96 (88%) participants, respectively.
Methods of detection
To allow further comparison of performance and its relationship to the PCR method used, information on assay designs that may affect test results was obtained from participants for comprehensive review and analysis.
There were considerable variations in the PCR protocols applied by participants. For primers and probes for the H5 gene alone, the responding laboratories cited as many as 35 different references for their detection methods. The details on target genes, detection methods and sources of the primers and probes were included in the summary report distributed to all participants together with the results of the analysis.
The target most often used to detect influenza type A viruses was the M gene. Other targets, such as the NS and NP genes, were also used. A minority of participants also performed PCR to detect influenza type B viruses.
For panel 2, panel 3 and panel 4, almost all participants performed H5 subtyping, and >80% of participants reported using H1 and H3 gene targets to subtype influenza type A viruses. Subtyping of the H7 gene was performed by 2?13% participants, of the H9 gene by 1?6% participants, of the N1 gene by 24?37% of participants and of the N2 gene by 14?28% of participants.
The nucleic acid amplification tests used mainly had been developed in-house, and primers and probes were most commonly adapted from other researchers. Commercial kits were used by 11?21% of participants to detect A/M, H5 and N1 genes; commercial kits were rarely used to detect H1, H3 and H7 genes. Detection methods were based on either conventional PCR or real-time PCR. For panel 2 and panel 3, real-time detection was most often used for A/M and H5 genes; conventional PCR was most often used for H1 and H3 genes. For panel 4, real-time detection was most often used for A/M, H5, H1 and H3 genes.
Comparison of laboratories? performance on all panels
During a 2-year period from 2007 to 2008, the proportion of participants with entirely correct results increased from 65% to 77%, whereas the proportion of participants with incorrect H5 subtyping fell from 23% to 12% and false-positive results fell from 14% to 6% (Table 4). False-positive results for negative samples remained at <10% from panel 2 onward. Of the 868 tests
performed on negative samples (panel 1, 128 tests; panel 2, 332 tests; panel 3, 190 tests; panel 4, 218 tests), 32 (3.7%) tests were false-positives; the most frequent false-positive results reported were for influenza A virus subtype H5 (Table 5).
To evaluate the overall performance of participants across the panels, results were categorized as ?good? when >90% were correct and ?less good? where <90% were correct. When all panels were compared, the proportion of participants returning less good results fell from 23% to 13% (Table 3). The improvement in results could be attributed to participation in earlier panels. Of
15 laboratories returning less good results for panel 1, 12 participated in all panels; the number classifi ed as having less good results decreased from 12 to 4 for panel 2 and from 4 to 2 for panel 3; it remained 2 for panel 4.
A total of 44 laboratories returned less good results for >1 of the panels; 8 laboratories participated in only 1 panel; the remaining 36 laboratories participated in >1 panel. These 36 participants were categorized as belonging to 1 of 3 groups according to their performance record: as an ?improved performer? (participant returned less good results in earlier panel or panels but good results in a later panel or panels), a ?fluctuating performer? (participant returned good results in an earlier panel or panels but less good results in a later panel or panels) or a ?persistently poor performer?
(participant returned less good results in all panels). In total, 22 laboratories were classifi ed as improved performers, 12 as fluctuating and 2 as persistently poor.
The improvement in H5 subtyping was evident when results for samples with the same concentration were compared. For H1 and H3 subtyping, it appears there is still room for improvement (Table 1).
For all 4 panels, the percentage of laboratories returning all correct results in the European Region was >70%, but in the African Region and the Eastern Mediterranean Region the percentages remained <70% (Fig. 1).
The sharp decline (>15%) in performance in the Region of the Americas and the South-East Asia Region on panel 2 was probably due to the large number of firsttime participants; a similar situation occurred in the African Region for panel 3 and panel 4 (Table 6). For panel 4, the performance of the South-East Asia and Western Pacific regions was comparable to that of the European Region.
Factors affecting performance
There was considerable variation in the PCR protocols applied by participants. Because of the multitude of test procedures used by a small number of participants, a meaningful statistical analysis to determine whether 1 individual protocol is significantly better than others was not possible. However, we have compared and analysed the use of in-house methods versus commercial kits, and the use of conventional PCR versus real-time PCR.
Analysis showed that in 2 of 3 panels, laboratories using real-time PCR for >1 test to detect the H5 gene demonstrated significantly better overall performance than laboratories using only conventional PCR (panel 2, P=0.002; panel 3, P=0.006) (Table 7). The use of commercial kits, however, did not have a significant effect on participants? overall performance (panel 2, P=0.678;
panel 3, P=0.637; panel 4, P=0.188).
Discussion
The number of laboratories participating in external quality assessment increased steadily during the 2 years from 2007 to 2008. Efforts were made to overcome communication and logistic challenges by communicating with the appropriate responsible officer in the laboratories, obtaining import permits and resolving shipping problems (Table 8). This increase in the number of participants also reflects improvement in the diagnostic capacity of global influenza laboratories: an increasing number of laboratories are equipped with PCR technology to detect influenza A.
The accurate detection of different subtypes of influenza A virus can help to target resources for disease prevention and control. The panels were designed to focus on different aspects of the testing process. Theoretically, a larger number of samples with different subtypes and concentrations would be more likely to detect errors but would probably create an undue burden on
many participants. Except for panel 2, which consisted of 14 samples, each panel consisted of 10 samples; this is considered an acceptable size panel by participants.
The aim of the panel 1 was to assess laboratories? performance in detecting different concentrations and strains of H5 viruses: 3 different H5 viruses with high and low concentrations were included. For panel 2, an additional clade of H5 virus with high and low concentrations was included. Because we learnt from panel 1 that false-positive amplifi cation was a problem for 9 laboratories, 4 negative samples were included to assess potential contamination problems in the participating laboratories.
Based on feedback from participants and the decrease in the contamination rate from 14% to 6%, it was decided to reduce the number of negative samples in panel 3. In order to evaluate inter-panel reproducibility, panel 3 and panel 4 were similar in composition and concentration: each contained 5 strains of clade 1 and clade 2 H5 viruses, making it possible to evaluate the
effect of testing parameters on the detection of different H5 clades.
The primary purpose of this project was to determine whether participants were capable of obtaining correct results. It was not intended to assess the accuracy of particular methods. However, the results of the analyses of the 4 panels yielded useful information on the performance of the particular methods used. PCR primers and probes from multiple sources were used by participants.
Although it is statistically impossible to determine whether 1 set of PCR primers and probes produced significantly better results than others, overall analysis shows that no difference in performance was observed between the use of in-house methods and commercial kits.
Detecting influenza A virus in a clinical sample is a complex, multistep procedure. Each laboratory should have its own testing strategy using optimized protocols for different target genes. The design of the primers and probes alone may not explain why participants failed to detect H5 samples, since base mismatch in some primer and probe sets would still be able to detect all H5 samples (data not shown). Optimization of the PCR reaction seems to be a more important factor: this could explain why performance among laboratories varied even though they were using the same sources of PCR primers and probes.
The scope of this project is limited to the amplification part of the testing process. Steps such as specimen collection and RNA extraction may contribute to the overall accuracy and reliability of test results. It must also be understood that samples for the external quality assessment project were handled differently from real clinical samples. However, the results of this project did show there were improvements in performance, indicating that some problems in laboratory testing had been overcome. The use of real-time PCR appeared to lead to better performance in panel 2 and panel 3 but not in panel 4. This difference is most likely due to the improvement in performing H5 subtyping by the conventional method. The performance of the laboratories in the South-East Asia Region and the Western Pacific Region in panel 4 are comparable to that of the European Region: results were 100% correct in >80% of laboratories.
The significant improvement in performance in the South-East Asia and Western Pacific regions is closely linked to the knowledge and skills acquired through various channels. These channels include WHO?s training workshop for the detection of influenza A viruses by PCR, conducted for laboratories with incorrect H5 results in these 2 regions at the Centre for Health Protection, Hong Kong SAR, in February 2008 just before panel 3 was sent. Of the 9 laboratories that participated in the workshop, 8 achieved 100% correct results for panel 4, while 1 had 100% correct results
for H5 samples but not for H1 and H3 samples.
NICs and national influenza laboratories in countries without NICs are encouraged to participate in the external quality assessment project to benefi t from continual monitoring of laboratory performance. It is expected that through this project high-quality influenza diagnostic capacity can be achieved, maintained and improved globally.
Editorial note.
The full report for each panel contains more information than that presented in this summary. All data from the project will be used as a foundation to assess needs and plans for action by the Global Influenza Surveillance Network.
WHO would like to thank all NICs and other influenza laboratories for participating in the project, for the time they spent completing the questionnaires and for their willingness to share information for this analysis. For more information, please contact the WHO Global Influenza Programme, Geneva, Switzerland (e-mail: GISN@who.int).
-
1) A list of NICs can be found at http://www.who.int/csr/disease/influ.../en/index.html.
2) Recommendations and laboratory procedures for detection of influenza A(H5N1) virus in specimens from suspected human cases. Geneva, World Health Organization, 2007 (http://www.who.int/csr/disease/avian...testsAug07.pdf, accessed 22 October 2008). See recommendations on conventional RT?PCR and real-time RT-PCR.
-
-----
Weekly epidemiological record - 7 NOVEMBER 2008, 83rd YEAR - No. 45, 2008, 83, 401?412 - http://www.who.int/wer - 2008, 83, 401?412 No. 45 - WORLD HEALTH ORGANIZATION - Geneva
WHO external quality assessment project for the detection of subtype influenza A viruses by polymerase chain reaction ? summary analysis, 2007 and 2008
[Edited version. Full text with accompanying tables at: http://www.who.int/wer/2008/wer8345.pdf ]
Introduction
For more than 50 years, National Influenza Centres (NICs)1 have been the backbone of WHO?s Global Influenza Surveillance Network and WHO?s Global Influenza Programme.
The centres collect specimens, conduct preliminary analyses and send representative virus isolates in a timely manner to WHO collaborating centres to support the annual recommendation of
influenza vaccine composition for the next season.
The tests used are mainly virus isolation and haemagglutination inhibition.
Since early 2004 the ongoing presence of the A(H5N1) strain of avian influenza has increased the need to include polymerase chain reaction (PCR) as part of the laboratory process in order to improve the accuracy and efficiency of diagnosis.
PCR has become the principal laboratory test for detecting H5 infections, and it has facilitated patient management, outbreak response and pandemic preparedness.
Although many NICs had started using PCR prior to 2006, until that time no globally coordinated external quality assessment programme existed to monitor the quality and comparability of laboratories? performance. WHO?s external quality assessment project for the detection of influenza virus type A by PCR was established to improve the global laboratory capacity for diagnosing both seasonal influenza and avian influenza.
It specifically aims to monitor the quality and standards of performance of NICs and other influenza laboratories in detecting influenza A virus subtypes using PCR as well as to promote good laboratory practice.
Coordinated by WHO?s Global Influenza Programme, based at WHO?s headquarters, the external quality assessment project is being conducted by the WHO reference laboratory for diagnosis of influenza A/H5 infection, National Influenza Centre, Centre for Health Protection, Hong Kong, Special Administrative Region of China (Hong Kong SAR), with support from WHO?s collaborating
centres on influenza, other influenza A/H5 reference laboratories and WHO regional offices.
This report summarizes the results of laboratories that participated in the assessment exercise by analysing 4 influenza test panels dispatched over a 2-year period from 2007 to 2008.
Preparation of panels
Panels sent to participating laboratories consisted of vacuum-dried RNA specimens extracted from influenza A(H1N1), A(H3N2) and A(H5N1) viruses. Prior to dispatch, the panels were tested by 7 laboratories in Hong Kong SAR to assess sample quality and homogeneity. To assess the effects of storage and transportation at ambient temperature, samples were tested after 7 days of storage at 37 ?C using both conventional and realtime PCR assays2 to confirm sample stability. The samples were quantified by real-time PCR assay using known synthetic RNA standards.
Composition of panels
The 4 panels consisted of coded samples containing RNA at different concentrations from 5 strains representing different genetic clades (1 and 2) of influenza A(H5N1) viruses, A(H1N1) viruses and A(H3N2) viruses.
Samples that contained no virus were also included.
Details of the composition of the panels are shown in Table 1. Participants were instructed to reconstitute each sample with the provided reconstitution buffer prior to testing. Additionally, a questionnaire was included to obtain information on detection methods used, target genes tested and primer and probe sequences employed.
Distribution of panels and response of participants
NICs and other national influenza laboratories were invited to participate before the panels were dispatched. Panel 1 was dispatched between February 2007 and March 2007; panel 2 was dispatched between August 2007 and October 2007; panel 3 was dispatched between January 2008 and February 2008; and panel 4 was dispatched between June 2008 and July 2008. All panels were dispatched from the Centre for Health Protection in Hong Kong SAR at ambient temperature by courier service to participating laboratories in all 6 WHO regions.
Participating laboratories were requested to notify the Centre for Health Protection immediately on receipt of the panel, either by fax or e-mail, and to return their results within 1 month. Each laboratory was assigned a participant identity code known only to the Centre for Health Protection and the Global Infl uenza Programme.
Laboratories that did not respond within the specified time frame were reminded by e-mail. A preliminary report, including correct results, was sent to participants shortly after the closing date for each panel.
Results for panel 1 were reported by 64 laboratories from 54 countries, areas or territories; results for panel 2 were reported by 83 laboratories from 66 countries, areas or territories; results for panel 3 were reported by 95 laboratories from 77 countries, areas or territories; and results for panel 4 were reported by 109 laboratories from 83 countries, areas or territories.
Details are shown in Table 2 and Table 3.
Most participants received the panels within 1 week (panel 1, 81% received within 1 week; panel 2, 87%; panel 3, 86%; panel 4, 85%). Across all 4 panels, nearly half (47%) of participating laboratories were situated in WHO?s European Region; 9% were in the African Region; 15% were in the Region of the Americas; 5% were in the Eastern Mediterranean Region; 5% in the South-East Asia Region; and 19% in the Western Pacific Region.
Results
All results returned before the closing date for each panel were included in the analysis, and a full report on laboratories? performance on the panel was sent to all participating laboratories within 1 month after release of the preliminary report.
This assessment required qualitative results (that is, positive or negative) of the target genes tested; quantitative results (that is, copies/μL and threshold cycle value) were used only for reference. The performance of individual laboratories was assessed by adding up the number of correct results. The following standards were used when assessing laboratories? results:
(i) failing to detect H5 samples and/or reporting the results as non-H5 subtype were recorded as an incorrect response;
(ii) failing to detect H1 samples and/or reporting the results as non-H1 subtype were recorded as an incorrect response;
(iii) failing to detect H3 samples and/or reporting the results as non-H3 subtype were recorded as an incorrect response;
(iv) failing to report correct influenza A test results for H1/H3 samples if H1/H3 subtyping was not performed was recorded as an incorrect response;
(v) reporting positive results for a sample that did not contain any viral RNA was recorded as an incorrect response.
Performance of the laboratories
Panel 1
A total of 43 participants returned correct results for all 10 samples in panel 1 (Table 3). An additional 6 participants returned correct results for 9 samples and 9 participants for 5?8 samples; 6 participants returned correct results for less than half of the samples. For the 2 negative samples (denoted as 2007-5 and 2007-10), 9 participants reported positive results. The corresponding
false-positive rates were 8% and 6%, respectively (Table 1).
The 3 strongly positive H5 samples that contained higher concentrations of viral RNA (denoted 2007-1, 2007-6, 2007-3) were correctly reported by 56 (88%) participants, 58 (91%) and 57 (89%), respectively. The 3 less strongly positive H5 samples (denoted 2007-7, 2007-2, 2007-9) were correctly reported by 53 (83%), 55 (86%) and 53 (83%) participants, respectively.
The 1 sample of H1 and 1 sample of H3 were correctly reported by 57 (89%) participants.
Panel 2
A total of 54 participants returned correct results for all 14 samples (Table 3). An additional 14 participants returned correct results for 13 samples and 12 participants for 8?12 samples; 3 participants returned correct results for less than half of the samples.
For the 4 negative samples (2007-15, 2007-18, 2007-21, 2007-24), 5 participants reported positive results. The corresponding false-positive rates were 2% for 2007-15, 4% for 2007-18, 4% for 2007-21 and 1% for 2007-24 (Table 1).
The 4 samples of H5 with higher concentrations (2007- 11, 2007-22, 2007-23, 2007-16) were correctly reported by 79 (95%), 77 (93%), 80 (96%) and 79 (95%) participants, respectively. The 4 samples of H5 with lower concentrations (2007-19, 2007-12, 2007-14, 2007-20) were correctly reported by 76 (92%), 76 (92%), 77 (93%) and 75 (90%) participants, respectively.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 74 (89%) and 73 (88%) participants, respectively.
Panel 3
A total of 70 participants returned correct results for all 10 samples (Table 3). An additional 10 participants returned correct results for 9 samples and 11 participants for 6?8 samples; 4 participants returned correct results for less than half of the samples.
For the 2 negative samples (2008-04, 2008-08), 2 participants reported positive results. The corresponding false-positive rates were 1% and 2%, respectively (Table 1).
The 4 samples of H5 (2008-05, 2008-01, 2008-02, 2008-03) were correctly reported by 85 (89%), 88 (93%), 90 (95%) and 89 (94%) participants, respectively. For the 2 duplicate H5 samples (2008-07, 2008-09), the performance of the participants was consistent: both were correctly reported by 89 (94%) participants.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 83 (87%) and 84 (88%) participants, respectively.
Panel 4
A total of 84 participants returned correct results for all 10 samples (Table 3). An additional 11 participants returned correct results for 9 samples and 10 participants for 6?8 samples; 4 participants returned correct results for less than half of the samples.
For the 2 negative samples (2008-19, 2008-20), 6 participants reported positive results. The corresponding false-positive rates were 5% and 2%, respectively (Table 1).
The 4 samples of H5 with a single concentration (2008- 11, 2008-13, 2008-15, 2008-17) were correctly reported by 104 (95%), 105 (96%), 104 (95%) and 105 (96%) participants, respectively. For the clade 2.3.2 strain with higher and lower concentrations (2008-16 and 2008-12, respectively), correct results were reported by 104 (95%) and 103 (94%) participants, respectively.
The 1 sample of H1 and the 1 sample of H3 were correctly reported by 93 (85%) and 96 (88%) participants, respectively.
Methods of detection
To allow further comparison of performance and its relationship to the PCR method used, information on assay designs that may affect test results was obtained from participants for comprehensive review and analysis.
There were considerable variations in the PCR protocols applied by participants. For primers and probes for the H5 gene alone, the responding laboratories cited as many as 35 different references for their detection methods. The details on target genes, detection methods and sources of the primers and probes were included in the summary report distributed to all participants together with the results of the analysis.
The target most often used to detect influenza type A viruses was the M gene. Other targets, such as the NS and NP genes, were also used. A minority of participants also performed PCR to detect influenza type B viruses.
For panel 2, panel 3 and panel 4, almost all participants performed H5 subtyping, and >80% of participants reported using H1 and H3 gene targets to subtype influenza type A viruses. Subtyping of the H7 gene was performed by 2?13% participants, of the H9 gene by 1?6% participants, of the N1 gene by 24?37% of participants and of the N2 gene by 14?28% of participants.
The nucleic acid amplification tests used mainly had been developed in-house, and primers and probes were most commonly adapted from other researchers. Commercial kits were used by 11?21% of participants to detect A/M, H5 and N1 genes; commercial kits were rarely used to detect H1, H3 and H7 genes. Detection methods were based on either conventional PCR or real-time PCR. For panel 2 and panel 3, real-time detection was most often used for A/M and H5 genes; conventional PCR was most often used for H1 and H3 genes. For panel 4, real-time detection was most often used for A/M, H5, H1 and H3 genes.
Comparison of laboratories? performance on all panels
During a 2-year period from 2007 to 2008, the proportion of participants with entirely correct results increased from 65% to 77%, whereas the proportion of participants with incorrect H5 subtyping fell from 23% to 12% and false-positive results fell from 14% to 6% (Table 4). False-positive results for negative samples remained at <10% from panel 2 onward. Of the 868 tests
performed on negative samples (panel 1, 128 tests; panel 2, 332 tests; panel 3, 190 tests; panel 4, 218 tests), 32 (3.7%) tests were false-positives; the most frequent false-positive results reported were for influenza A virus subtype H5 (Table 5).
To evaluate the overall performance of participants across the panels, results were categorized as ?good? when >90% were correct and ?less good? where <90% were correct. When all panels were compared, the proportion of participants returning less good results fell from 23% to 13% (Table 3). The improvement in results could be attributed to participation in earlier panels. Of
15 laboratories returning less good results for panel 1, 12 participated in all panels; the number classifi ed as having less good results decreased from 12 to 4 for panel 2 and from 4 to 2 for panel 3; it remained 2 for panel 4.
A total of 44 laboratories returned less good results for >1 of the panels; 8 laboratories participated in only 1 panel; the remaining 36 laboratories participated in >1 panel. These 36 participants were categorized as belonging to 1 of 3 groups according to their performance record: as an ?improved performer? (participant returned less good results in earlier panel or panels but good results in a later panel or panels), a ?fluctuating performer? (participant returned good results in an earlier panel or panels but less good results in a later panel or panels) or a ?persistently poor performer?
(participant returned less good results in all panels). In total, 22 laboratories were classifi ed as improved performers, 12 as fluctuating and 2 as persistently poor.
The improvement in H5 subtyping was evident when results for samples with the same concentration were compared. For H1 and H3 subtyping, it appears there is still room for improvement (Table 1).
For all 4 panels, the percentage of laboratories returning all correct results in the European Region was >70%, but in the African Region and the Eastern Mediterranean Region the percentages remained <70% (Fig. 1).
The sharp decline (>15%) in performance in the Region of the Americas and the South-East Asia Region on panel 2 was probably due to the large number of firsttime participants; a similar situation occurred in the African Region for panel 3 and panel 4 (Table 6). For panel 4, the performance of the South-East Asia and Western Pacific regions was comparable to that of the European Region.
Factors affecting performance
There was considerable variation in the PCR protocols applied by participants. Because of the multitude of test procedures used by a small number of participants, a meaningful statistical analysis to determine whether 1 individual protocol is significantly better than others was not possible. However, we have compared and analysed the use of in-house methods versus commercial kits, and the use of conventional PCR versus real-time PCR.
Analysis showed that in 2 of 3 panels, laboratories using real-time PCR for >1 test to detect the H5 gene demonstrated significantly better overall performance than laboratories using only conventional PCR (panel 2, P=0.002; panel 3, P=0.006) (Table 7). The use of commercial kits, however, did not have a significant effect on participants? overall performance (panel 2, P=0.678;
panel 3, P=0.637; panel 4, P=0.188).
Discussion
The number of laboratories participating in external quality assessment increased steadily during the 2 years from 2007 to 2008. Efforts were made to overcome communication and logistic challenges by communicating with the appropriate responsible officer in the laboratories, obtaining import permits and resolving shipping problems (Table 8). This increase in the number of participants also reflects improvement in the diagnostic capacity of global influenza laboratories: an increasing number of laboratories are equipped with PCR technology to detect influenza A.
The accurate detection of different subtypes of influenza A virus can help to target resources for disease prevention and control. The panels were designed to focus on different aspects of the testing process. Theoretically, a larger number of samples with different subtypes and concentrations would be more likely to detect errors but would probably create an undue burden on
many participants. Except for panel 2, which consisted of 14 samples, each panel consisted of 10 samples; this is considered an acceptable size panel by participants.
The aim of the panel 1 was to assess laboratories? performance in detecting different concentrations and strains of H5 viruses: 3 different H5 viruses with high and low concentrations were included. For panel 2, an additional clade of H5 virus with high and low concentrations was included. Because we learnt from panel 1 that false-positive amplifi cation was a problem for 9 laboratories, 4 negative samples were included to assess potential contamination problems in the participating laboratories.
Based on feedback from participants and the decrease in the contamination rate from 14% to 6%, it was decided to reduce the number of negative samples in panel 3. In order to evaluate inter-panel reproducibility, panel 3 and panel 4 were similar in composition and concentration: each contained 5 strains of clade 1 and clade 2 H5 viruses, making it possible to evaluate the
effect of testing parameters on the detection of different H5 clades.
The primary purpose of this project was to determine whether participants were capable of obtaining correct results. It was not intended to assess the accuracy of particular methods. However, the results of the analyses of the 4 panels yielded useful information on the performance of the particular methods used. PCR primers and probes from multiple sources were used by participants.
Although it is statistically impossible to determine whether 1 set of PCR primers and probes produced significantly better results than others, overall analysis shows that no difference in performance was observed between the use of in-house methods and commercial kits.
Detecting influenza A virus in a clinical sample is a complex, multistep procedure. Each laboratory should have its own testing strategy using optimized protocols for different target genes. The design of the primers and probes alone may not explain why participants failed to detect H5 samples, since base mismatch in some primer and probe sets would still be able to detect all H5 samples (data not shown). Optimization of the PCR reaction seems to be a more important factor: this could explain why performance among laboratories varied even though they were using the same sources of PCR primers and probes.
The scope of this project is limited to the amplification part of the testing process. Steps such as specimen collection and RNA extraction may contribute to the overall accuracy and reliability of test results. It must also be understood that samples for the external quality assessment project were handled differently from real clinical samples. However, the results of this project did show there were improvements in performance, indicating that some problems in laboratory testing had been overcome. The use of real-time PCR appeared to lead to better performance in panel 2 and panel 3 but not in panel 4. This difference is most likely due to the improvement in performing H5 subtyping by the conventional method. The performance of the laboratories in the South-East Asia Region and the Western Pacific Region in panel 4 are comparable to that of the European Region: results were 100% correct in >80% of laboratories.
The significant improvement in performance in the South-East Asia and Western Pacific regions is closely linked to the knowledge and skills acquired through various channels. These channels include WHO?s training workshop for the detection of influenza A viruses by PCR, conducted for laboratories with incorrect H5 results in these 2 regions at the Centre for Health Protection, Hong Kong SAR, in February 2008 just before panel 3 was sent. Of the 9 laboratories that participated in the workshop, 8 achieved 100% correct results for panel 4, while 1 had 100% correct results
for H5 samples but not for H1 and H3 samples.
NICs and national influenza laboratories in countries without NICs are encouraged to participate in the external quality assessment project to benefi t from continual monitoring of laboratory performance. It is expected that through this project high-quality influenza diagnostic capacity can be achieved, maintained and improved globally.
Editorial note.
The full report for each panel contains more information than that presented in this summary. All data from the project will be used as a foundation to assess needs and plans for action by the Global Influenza Surveillance Network.
WHO would like to thank all NICs and other influenza laboratories for participating in the project, for the time they spent completing the questionnaires and for their willingness to share information for this analysis. For more information, please contact the WHO Global Influenza Programme, Geneva, Switzerland (e-mail: GISN@who.int).
-
1) A list of NICs can be found at http://www.who.int/csr/disease/influ.../en/index.html.
2) Recommendations and laboratory procedures for detection of influenza A(H5N1) virus in specimens from suspected human cases. Geneva, World Health Organization, 2007 (http://www.who.int/csr/disease/avian...testsAug07.pdf, accessed 22 October 2008). See recommendations on conventional RT?PCR and real-time RT-PCR.
-
-----