Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices (Euro Surveill., edited)
Eurosurveillance, Volume 14, Issue 44, 05 November 2009
Surveillance and outbreak reports
Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices
M Coory 1, K Grant 2, H Kelly 2,3
1. School of Population Health, University of Queensland, Herston, Australia
2. Victorian Infectious Diseases Reference Laboratory, Melbourne, Australia
3. School of Population Health, University of Melbourne, Melbourne, Australia
Citation style for this article: Coory M, Grant K, Kelly H. Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices. Euro Surveill. 2009;14(44):pii=19387. Available online: LINK
Date of submission: 13 October 2009
Standard sources of data for influenza surveillance include notifications of laboratory-confirmed cases and notifications from sentinel general practices. These data are not always available in a timely fashion, leading to proposals to use more immediate data sources such as over-the-counter drug sales, ambulance call-outs and web searches to monitor influenza-like illness (ILI). We aimed to assess data from a deputising medical service as another source of data for timely syndromic influenza surveillance. We measured the extent of agreement between the weekly percentage of patients with ILI reported from sentinel general practices and the corresponding weekly percentage reported from a deputising medical service in Victoria, Australia over ten years, from 1999 to 2008. There was good agreement between the two data sources, with suitably narrow limits of agreement. The deputising medical service did not use a standardised definition of ILI and is not supplemented by laboratory confirmation of suspected cases. Nevertheless, the results of this study show that such data can provide low cost and timely ILI surveillance.
Introduction
In temperate southern Australia, the influenza season occurs between May (late autumn) and October (early spring). Sentinel general-practitioner (GP) surveillance, operational in Victoria during the influenza season, reports weekly on the number of patients fulfilling the Australian nationally agreed case definition of influenza-like illness (ILI): cough, fever and fatigue. Respiratory specimens taken from a proportion of cases permit diagnosis of laboratory-confirmed influenza [1]. Not all ILI cases are confirmed as influenza. In Victoria, Australia, the proportion of confirmed cases between 2003-2007 varied from 18-47%, annually [2].
Besides notifications from sentinel GPs, another standard method of influenza surveillance is to count the number of laboratory-confirmed cases notified to a public health authority [1]. Both these standard data sources, which involve laboratory testing, are associated with a reporting lag due to the time taken for specimen testing and reporting. For instance, the median interval between symptom onset and registration for a laboratory test was three days for a patient recruited through sentinel GPs in Victoria in 2007 and 2008.
To overcome the problem of delay, surveillance using more immediate data sources without laboratory confirmation, referred to as syndromic surveillance, have been implemented. These include over-the-counter drug sales [3], telephone calls to health information lines such as nurse on call [4], ambulance call-outs [5], school or workforce absenteeism [6,7], and web searches [8-10].
One surveillance source, previously described by Turner and Kelly [11] but not formally assessed, is a deputising medical service, that is, an out-of-hours service for GP consultations. Many deputising services record the reason for the call-out and the final diagnosis in an electronic database, such as the GP house call surveillance system in Bordeaux, France [12]. The aim of this study was to measure the extent of agreement between ILI surveillance data from the deputising service and data from the sentinel GP system in Melbourne, Australia, in order to assess whether the former could be used for routine influenza surveillance.
Methods
The Melbourne Medical Deputising Service (MMDS) is a deputising, out-of-hours general practice service. Deputising doctors attend patients in their homes within a 45 km radius of the Melbourne Central Business District. Demographic (e.g. age, sex) and clinical data (e.g. diagnosis) are entered by the deputising doctor into a customised database, usually within 24 hours of the consultation. Access to the data is available on a password-protected page of the MMDS website. The data are available for use in a surveillance system as soon as they are entered, i.e. within 24 hours of the consultation.
We routinely obtain the proportion of ILI call-outs from the MMDS once a week, although they could be obtained daily with a 24-hour lag. The weekly data extraction uses a validated search algorithm that identifies the number of call-outs for ILI. This is divided by the total number of call-outs for that week and expressed as a percentage per 100 call-outs. MMDS data are available throughout the year. The search algorithm has been validated by manual confirmation of the diagnosis of all patients identified by the search algorithm for week 34 in the years 2002 to 2007, a week of high activity for all years in that period,. The search algorithm successfully identified ILI call-outs searching for the terms ?flu? and ?influ? and excluding terms such as ?reflux? and ?fluid? that included the letters ?flu?. New exclusion terms, ?fluvax?, ?at risk? and ?immunisation?, were added to the algorithm in 2009 to exclude pandemic H1N1 influenza contacts who received prophylactic antiviral treatment.
For the sentinel GP system, we used the number of consultations that met the nationally agreed definition of ILI expressed as a percentage of total visits as the comparator. We then assessed the degree of consensus between this measure and that from the MMDS, using a standard statistical method developed by Bland and Altman [13,14]. This method is based on reporting the difference between the two measures, and the 95% limits of agreement, which provide an interval in which 95% of the differences between the two measurements are expected to lie. If the limits of agreement describe differences that are not of material importance, the data sources can be used interchangeably.
As described by Bland and Altman [14], it is not unusual for the difference between two measures and the standard deviation to increase with increasing values of the two measures being assessed, and this should be accounted for in the statistical analysis, otherwise the limits of agreement will be too wide for low values of weekly ILI proportions and too narrow for high values. Accordingly, we regressed the difference of the weekly ILI percentages on their average, using absolute residuals to estimate the standard deviation.
To further assess the comparability of the two surveillance systems, we calculated the area under the receiver operator characteristic (ROC) curve for the 10 years of data from the deputising service against the weeks with higher than expected seasonal activity as currently defined by a sentinel GP weekly ILI percentage of 1.5%, described by Watts et al. [15]. In the context of this study, as described by Bland and Altman [16], an area under the ROC curve of 0.5 would mean that the deputising service was no better than chance in detecting the influenza season, while a value of 1.0 would mean that it was a perfect measure. Confidence intervals for the area under the ROC curve were obtained using the algorithm of DeLong et al. [17].
Results
From 1999 to 2008, the weekly percentages of ILI reported through the deputising service were similar to the percentages seen in the sentinel GP system during periods of low seasonal activity, but were larger in periods of higher activity, although this was less evident in later years (Figure 1). The difference between the two ILI data sources was small, but increased during the peak of the season, with data from the deputising service recording higher values than data from the sentinel GPs (Table). (...)
The 95% limits of agreement increased with increasing ILI activity, the importance of which, as noted by Bland and Altman [13], is a matter of judgement, rather than a statistical issue. Our judgement is that the limits of agreement are appropriately small during periods of normal seasonal ILI activity as well as at the start and end of the season, and that the wider limits at the peak of the season, or in seasons of higher activity, are of no material importance.
The area under the ROC curve was 0.91 (95% confidence interval (CI) 0.83, 0.98), confirming very close agreement between the systems when dichotomised around ILI activity describing normal and higher than expected seasonal activity.
Having both surveillance systems in place has been very useful in the H1N1 influenza pandemic of 2009 as the two surveillance systems provided complementary and confirmatory surveillance data when influenza A(H1N1)v was the dominant circulating strain [20]. As with previous years, however, ILI proportions from the two surveillance systems were more similar for lower values (Figure 2). (...)
Discussion
There was good agreement between the weekly percentages of ILI in the deputising service and sentinel GP system, although the agreement for high ILI values was not as close as for lower values. This is probably because the deputising service is an out-of-hours service, which is likely to have a higher percentage of call-outs for acute illnesses, such as influenza. The deputising service is also less likely to see non-acute illnesses, effectively increasing the ILI percentage relative to sentinel GPs who would continue to see patients for chronic diseases during the peak of the influenza season. Moreover all ILI consultations are captured by the deputing service database, whereas GP data are recorded on paper forms which makes complete capture of all ILI patients is unlikely. This would reduce the reported ILI percentage from sentinel GPs compared with the deputising service.
We did not use the correlation coefficient to assess whether the deputising service data were equivalent to the sentinel GP data as some authors have done [8], because this approach has been questioned in a series of much cited papers by Bland and Altman [13,14,18,19]. There are two reasons for not using the correlation coefficient to assess equivalence of two data sources: First, if the values of the data vary across a wide range, as is the case for ILI data from both deputising service and sentinel GPs, the correlation coefficient will be close to 1.0 even if one measure is not a good substitute for the other. Second, correlation ignores any systematic bias between the two measures. To overcome these problems, Bland and Altman recommended reporting the difference, or bias, between the two measures and the 95% limits of agreement and we have followed their advice in this study.
We did not examine agreement for different age groups. However, for the most recent five-year period included in the analysis (2004-2008), the percentage of ILI cases under the age of 15 years was similar in the two systems (19.5% in the deputising service versus 18.8% in the sentinel GP system), while the ILI cases from the deputising service were slightly older than those from sentinel general practice (mean 40.7 years versus 39.9 years) and showed more variation (standard deviation 25.6 versus 20.3). This was because of the growing number of out-of-hours consultations by the deputising medical service at care facilities for the elderly in the latter years of surveillance; 8.6% of ILI cases identified by the deputising service were 80 years or older while the corresponding percentage for sentinel general practice was only 2.2%.
Deputising medical service surveillance does not use a standardised definition of ILI and is not supplemented by laboratory confirmation of suspected influenza cases. Nevertheless we have shown that data from a deputising medical service can provide low cost and timely ILI surveillance throughout the year, equivalent to ILI surveillance provided by sentinel GPs. Further confirming its utility, surveillance data from the deputising service confirmed the onset and peak of ILI activity during the 2009 pandemic in Victoria.
Acknowledgements
We gratefully acknowledge the ongoing support of the Executive Director of MMDS, Ms Josie Adams and database support from Steven Long of SL Digital.
References
1. Grant K, Carville K, Fielding J, Barr I, Tran T, Riddell M, et al. High proportion of influenza B in the 2008 influenza season in Victoria. Communicable Diseases Intelligence. 2009; Forthcoming.
2. Kelly HA, Carville K, Grant K, Jacoby P, Thomas T, Barr I. Estimation of Influenza Vaccine Effectiveness from Routine Surveillance Data. PlosOne. 2009; Forthcoming.
3. Vergu E, Grais RF, Sarter H, Fagot JP, Lambert B, Valleron AJ, et al. Medication sales and syndromic surveillance, France. Emerg Infect Dis. 2006;12(3):416-21.
4. Espino JU, Hogan WR, Wagner MM. Telephone triage: a timely data source for surveillance of influenza-like diseases. AMIA Annu Symp Proc. 2003:215-9.
5. Coory MD, Kelly H, Tippett V. Assessment of ambulance dispatch data for surveillance of influenza-like illness in Melbourne, Australia. Public Health. 2009;123(2):163-8.
6. Besculides M, Heffernan R, Mostashari F, Weiss D. Evaluation of school absenteeism data for early outbreak detection, New York City. BMC Public Health. 2005;5:105.
7. Zhao H, Joseph C, Phin N. Outbreaks of influenza and influenza-like illness in schools in England and Wales, 2005/06. Euro Surveill. 2007;12(5):E3-4.
8. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using search engine query data. Nature. 2009;457(7232):1012-4.
9. Polgreen PM, Chen Y, Pennock DM, Nelson FD. Using internet searches for influenza surveillance. Clin Infect Dis. 2008;47(11):1443-8.
10. Butler D. Web data predict flu. Nature. 2008;456(7220):287-8.
11. Turner J, Kelly H. A medical locum service as a site for sentinel influenza surveillance. Euro Surveill. 2005;10(4):pii: 530. Available from: LINK
12. Flamand C, Larrieu S, Couvy F, Jouves B, Josseran L, Filleul L. Validation of a syndromic surveillance system using a general practitioner house calls network, Bordeaux, France. Euro Surveill. 2008;13(25):pii: 18905. Available from: LINK
13. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-10.
14. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135-60.
15. Watts CG, Andrews RM, Druce JD, Kelly HA. Establishing thresholds for influenza surveillance in Victoria. Aust N Z J Public Health. 2003;27(4):409-12.
16. Altman DG, Bland JM. Diagnostic tests 3: receiver operating characteristic plots. BMJ. 1994;309(6948):188.
17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-45.
18. Altman DG, Bland JM. Measurement in medicine: The analysis of method comparison studies. The Statistician. 1983;32:307-17.
19. Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet. 1995;346(8982):1085-7.
20. Kelly H, Grant K. Interim analysis of pandemic influenza (H1N1) 2009 in Australia: surveillance trends, age of infection and effectiveness of seasonal vaccination. Euro Surveill. 2009;14(31):pii: 19288. Available from: LINK
-
<cite cite="http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19387">Eurosurveillance - View Article</cite>
Surveillance and outbreak reports
Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices
M Coory 1, K Grant 2, H Kelly 2,3
1. School of Population Health, University of Queensland, Herston, Australia
2. Victorian Infectious Diseases Reference Laboratory, Melbourne, Australia
3. School of Population Health, University of Melbourne, Melbourne, Australia
Citation style for this article: Coory M, Grant K, Kelly H. Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices. Euro Surveill. 2009;14(44):pii=19387. Available online: LINK
Date of submission: 13 October 2009
Standard sources of data for influenza surveillance include notifications of laboratory-confirmed cases and notifications from sentinel general practices. These data are not always available in a timely fashion, leading to proposals to use more immediate data sources such as over-the-counter drug sales, ambulance call-outs and web searches to monitor influenza-like illness (ILI). We aimed to assess data from a deputising medical service as another source of data for timely syndromic influenza surveillance. We measured the extent of agreement between the weekly percentage of patients with ILI reported from sentinel general practices and the corresponding weekly percentage reported from a deputising medical service in Victoria, Australia over ten years, from 1999 to 2008. There was good agreement between the two data sources, with suitably narrow limits of agreement. The deputising medical service did not use a standardised definition of ILI and is not supplemented by laboratory confirmation of suspected cases. Nevertheless, the results of this study show that such data can provide low cost and timely ILI surveillance.
Introduction
In temperate southern Australia, the influenza season occurs between May (late autumn) and October (early spring). Sentinel general-practitioner (GP) surveillance, operational in Victoria during the influenza season, reports weekly on the number of patients fulfilling the Australian nationally agreed case definition of influenza-like illness (ILI): cough, fever and fatigue. Respiratory specimens taken from a proportion of cases permit diagnosis of laboratory-confirmed influenza [1]. Not all ILI cases are confirmed as influenza. In Victoria, Australia, the proportion of confirmed cases between 2003-2007 varied from 18-47%, annually [2].
Besides notifications from sentinel GPs, another standard method of influenza surveillance is to count the number of laboratory-confirmed cases notified to a public health authority [1]. Both these standard data sources, which involve laboratory testing, are associated with a reporting lag due to the time taken for specimen testing and reporting. For instance, the median interval between symptom onset and registration for a laboratory test was three days for a patient recruited through sentinel GPs in Victoria in 2007 and 2008.
To overcome the problem of delay, surveillance using more immediate data sources without laboratory confirmation, referred to as syndromic surveillance, have been implemented. These include over-the-counter drug sales [3], telephone calls to health information lines such as nurse on call [4], ambulance call-outs [5], school or workforce absenteeism [6,7], and web searches [8-10].
One surveillance source, previously described by Turner and Kelly [11] but not formally assessed, is a deputising medical service, that is, an out-of-hours service for GP consultations. Many deputising services record the reason for the call-out and the final diagnosis in an electronic database, such as the GP house call surveillance system in Bordeaux, France [12]. The aim of this study was to measure the extent of agreement between ILI surveillance data from the deputising service and data from the sentinel GP system in Melbourne, Australia, in order to assess whether the former could be used for routine influenza surveillance.
Methods
The Melbourne Medical Deputising Service (MMDS) is a deputising, out-of-hours general practice service. Deputising doctors attend patients in their homes within a 45 km radius of the Melbourne Central Business District. Demographic (e.g. age, sex) and clinical data (e.g. diagnosis) are entered by the deputising doctor into a customised database, usually within 24 hours of the consultation. Access to the data is available on a password-protected page of the MMDS website. The data are available for use in a surveillance system as soon as they are entered, i.e. within 24 hours of the consultation.
We routinely obtain the proportion of ILI call-outs from the MMDS once a week, although they could be obtained daily with a 24-hour lag. The weekly data extraction uses a validated search algorithm that identifies the number of call-outs for ILI. This is divided by the total number of call-outs for that week and expressed as a percentage per 100 call-outs. MMDS data are available throughout the year. The search algorithm has been validated by manual confirmation of the diagnosis of all patients identified by the search algorithm for week 34 in the years 2002 to 2007, a week of high activity for all years in that period,. The search algorithm successfully identified ILI call-outs searching for the terms ?flu? and ?influ? and excluding terms such as ?reflux? and ?fluid? that included the letters ?flu?. New exclusion terms, ?fluvax?, ?at risk? and ?immunisation?, were added to the algorithm in 2009 to exclude pandemic H1N1 influenza contacts who received prophylactic antiviral treatment.
For the sentinel GP system, we used the number of consultations that met the nationally agreed definition of ILI expressed as a percentage of total visits as the comparator. We then assessed the degree of consensus between this measure and that from the MMDS, using a standard statistical method developed by Bland and Altman [13,14]. This method is based on reporting the difference between the two measures, and the 95% limits of agreement, which provide an interval in which 95% of the differences between the two measurements are expected to lie. If the limits of agreement describe differences that are not of material importance, the data sources can be used interchangeably.
As described by Bland and Altman [14], it is not unusual for the difference between two measures and the standard deviation to increase with increasing values of the two measures being assessed, and this should be accounted for in the statistical analysis, otherwise the limits of agreement will be too wide for low values of weekly ILI proportions and too narrow for high values. Accordingly, we regressed the difference of the weekly ILI percentages on their average, using absolute residuals to estimate the standard deviation.
To further assess the comparability of the two surveillance systems, we calculated the area under the receiver operator characteristic (ROC) curve for the 10 years of data from the deputising service against the weeks with higher than expected seasonal activity as currently defined by a sentinel GP weekly ILI percentage of 1.5%, described by Watts et al. [15]. In the context of this study, as described by Bland and Altman [16], an area under the ROC curve of 0.5 would mean that the deputising service was no better than chance in detecting the influenza season, while a value of 1.0 would mean that it was a perfect measure. Confidence intervals for the area under the ROC curve were obtained using the algorithm of DeLong et al. [17].
Results
From 1999 to 2008, the weekly percentages of ILI reported through the deputising service were similar to the percentages seen in the sentinel GP system during periods of low seasonal activity, but were larger in periods of higher activity, although this was less evident in later years (Figure 1). The difference between the two ILI data sources was small, but increased during the peak of the season, with data from the deputising service recording higher values than data from the sentinel GPs (Table). (...)
The 95% limits of agreement increased with increasing ILI activity, the importance of which, as noted by Bland and Altman [13], is a matter of judgement, rather than a statistical issue. Our judgement is that the limits of agreement are appropriately small during periods of normal seasonal ILI activity as well as at the start and end of the season, and that the wider limits at the peak of the season, or in seasons of higher activity, are of no material importance.
The area under the ROC curve was 0.91 (95% confidence interval (CI) 0.83, 0.98), confirming very close agreement between the systems when dichotomised around ILI activity describing normal and higher than expected seasonal activity.
Having both surveillance systems in place has been very useful in the H1N1 influenza pandemic of 2009 as the two surveillance systems provided complementary and confirmatory surveillance data when influenza A(H1N1)v was the dominant circulating strain [20]. As with previous years, however, ILI proportions from the two surveillance systems were more similar for lower values (Figure 2). (...)
Discussion
There was good agreement between the weekly percentages of ILI in the deputising service and sentinel GP system, although the agreement for high ILI values was not as close as for lower values. This is probably because the deputising service is an out-of-hours service, which is likely to have a higher percentage of call-outs for acute illnesses, such as influenza. The deputising service is also less likely to see non-acute illnesses, effectively increasing the ILI percentage relative to sentinel GPs who would continue to see patients for chronic diseases during the peak of the influenza season. Moreover all ILI consultations are captured by the deputing service database, whereas GP data are recorded on paper forms which makes complete capture of all ILI patients is unlikely. This would reduce the reported ILI percentage from sentinel GPs compared with the deputising service.
We did not use the correlation coefficient to assess whether the deputising service data were equivalent to the sentinel GP data as some authors have done [8], because this approach has been questioned in a series of much cited papers by Bland and Altman [13,14,18,19]. There are two reasons for not using the correlation coefficient to assess equivalence of two data sources: First, if the values of the data vary across a wide range, as is the case for ILI data from both deputising service and sentinel GPs, the correlation coefficient will be close to 1.0 even if one measure is not a good substitute for the other. Second, correlation ignores any systematic bias between the two measures. To overcome these problems, Bland and Altman recommended reporting the difference, or bias, between the two measures and the 95% limits of agreement and we have followed their advice in this study.
We did not examine agreement for different age groups. However, for the most recent five-year period included in the analysis (2004-2008), the percentage of ILI cases under the age of 15 years was similar in the two systems (19.5% in the deputising service versus 18.8% in the sentinel GP system), while the ILI cases from the deputising service were slightly older than those from sentinel general practice (mean 40.7 years versus 39.9 years) and showed more variation (standard deviation 25.6 versus 20.3). This was because of the growing number of out-of-hours consultations by the deputising medical service at care facilities for the elderly in the latter years of surveillance; 8.6% of ILI cases identified by the deputising service were 80 years or older while the corresponding percentage for sentinel general practice was only 2.2%.
Deputising medical service surveillance does not use a standardised definition of ILI and is not supplemented by laboratory confirmation of suspected influenza cases. Nevertheless we have shown that data from a deputising medical service can provide low cost and timely ILI surveillance throughout the year, equivalent to ILI surveillance provided by sentinel GPs. Further confirming its utility, surveillance data from the deputising service confirmed the onset and peak of ILI activity during the 2009 pandemic in Victoria.
Acknowledgements
We gratefully acknowledge the ongoing support of the Executive Director of MMDS, Ms Josie Adams and database support from Steven Long of SL Digital.
References
1. Grant K, Carville K, Fielding J, Barr I, Tran T, Riddell M, et al. High proportion of influenza B in the 2008 influenza season in Victoria. Communicable Diseases Intelligence. 2009; Forthcoming.
2. Kelly HA, Carville K, Grant K, Jacoby P, Thomas T, Barr I. Estimation of Influenza Vaccine Effectiveness from Routine Surveillance Data. PlosOne. 2009; Forthcoming.
3. Vergu E, Grais RF, Sarter H, Fagot JP, Lambert B, Valleron AJ, et al. Medication sales and syndromic surveillance, France. Emerg Infect Dis. 2006;12(3):416-21.
4. Espino JU, Hogan WR, Wagner MM. Telephone triage: a timely data source for surveillance of influenza-like diseases. AMIA Annu Symp Proc. 2003:215-9.
5. Coory MD, Kelly H, Tippett V. Assessment of ambulance dispatch data for surveillance of influenza-like illness in Melbourne, Australia. Public Health. 2009;123(2):163-8.
6. Besculides M, Heffernan R, Mostashari F, Weiss D. Evaluation of school absenteeism data for early outbreak detection, New York City. BMC Public Health. 2005;5:105.
7. Zhao H, Joseph C, Phin N. Outbreaks of influenza and influenza-like illness in schools in England and Wales, 2005/06. Euro Surveill. 2007;12(5):E3-4.
8. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using search engine query data. Nature. 2009;457(7232):1012-4.
9. Polgreen PM, Chen Y, Pennock DM, Nelson FD. Using internet searches for influenza surveillance. Clin Infect Dis. 2008;47(11):1443-8.
10. Butler D. Web data predict flu. Nature. 2008;456(7220):287-8.
11. Turner J, Kelly H. A medical locum service as a site for sentinel influenza surveillance. Euro Surveill. 2005;10(4):pii: 530. Available from: LINK
12. Flamand C, Larrieu S, Couvy F, Jouves B, Josseran L, Filleul L. Validation of a syndromic surveillance system using a general practitioner house calls network, Bordeaux, France. Euro Surveill. 2008;13(25):pii: 18905. Available from: LINK
13. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-10.
14. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135-60.
15. Watts CG, Andrews RM, Druce JD, Kelly HA. Establishing thresholds for influenza surveillance in Victoria. Aust N Z J Public Health. 2003;27(4):409-12.
16. Altman DG, Bland JM. Diagnostic tests 3: receiver operating characteristic plots. BMJ. 1994;309(6948):188.
17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-45.
18. Altman DG, Bland JM. Measurement in medicine: The analysis of method comparison studies. The Statistician. 1983;32:307-17.
19. Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet. 1995;346(8982):1085-7.
20. Kelly H, Grant K. Interim analysis of pandemic influenza (H1N1) 2009 in Australia: surveillance trends, age of infection and effectiveness of seasonal vaccination. Euro Surveill. 2009;14(31):pii: 19288. Available from: LINK
-