EXECUTIVE SUMMARY

RCPA QAP was engaged by the Department of Health & Ageing (DoHA), in a project to review the Role of External Quality Assurance in Identifying Poor Laboratory Performance. The project objectives were to determine if the regular EQA that laboratories perform can be used to identify unacceptable performance earlier than the usual three year NATA accreditation cycle, by establishing a framework and criteria to identify suboptimal laboratory performance, to allow early intervention in order to minimise risk to patients.

An earlier project in Chemical Pathology QAP developed “Key Performance Indicators” for “EQA participation”, “performance of all analytes” and “performance of indicator analytes”. Analysis of data retrospectively from 2006 to 2010 indicated a level of non‐submission of results, and a higher level (approximately 25%) of late submission of results by Australian laboratories. A review of the “All Analytes” KPI and “Indicator Analytes” KPI demonstrated that there were probably no diagnostic Australian laboratories considered to be technically poorly performing over this period. These KPIs will need to be reviewed in future. Due to confidentiality concerns, however, this data was unable to be correlated with NATA on‐site assessment records.

Key Performance Indicators were developed for Anatomical Pathology in this project. Several scoring methods have been developed, using a novel non‐parametric scoring system for measuring participants’ performance for each survey case, and allowing a peer review comparison. Analysis of data retrospectively in the General and Breast diagnostic modules showed significant levels of survey non‐participation by laboratories. Performance across surveys indicated a small population of participants that consistently ranked in the lowest 10% of cumulative performance. Benchmarks
were developed, although a further period of time will be required to trial and validate these adequately.

Extensive stakeholder consultation with Anatomical Pathologists has identified several key operational issues with the Anatomical Pathology QAP;

• EQA is perceived as educational by many pathologists, so selection of cases should be better suited to assessment of performance rather than educational value
• There are no clear regulatory guidelines (NPAAC) for enrolment of anatomical pathology laboratories; this should be reviewed to allow consideration of a laboratory’s scope of practice
• There are concerns that single pathologist / few pathologist laboratories are at risk of being assessed more rigorously as individuals than large laboratories containing many pathologists,
• Virtual images are not used in routine laboratories for diagnosis

Performance benchmarks have been developed in the Transfusion QAP, including review of performance for a single survey and cumulative performance over a six month period. Trials of the benchmarks to date have been successful, with letters of notification being sent to four laboratories to enable them to complete a timely corrective action.

Existing benchmarks for gynaecological Cytopathology have been revisited, and new benchmarks have been developed which include both EQA results and the National Performance Measures data. Data analysed retrospectively from 2004 – 2010 for the GYN module showed that those laboratories that had major errors in the GYN module had more Performance Measures outside the national standard. Benchmarks have been set at one major error, three unacceptable responses in the GYN module, being outside the national performance measures or non submission of results. Trials have resulted in letters being sent to 10 laboratories. The feedback from participants has been predominately positive, and any negative feedback has been addressed and overcome.

A novel protocol has been developed to monitor EQA results for both laboratory performance and for the performance of in vitro diagnostic medical devices. The framework (Appendix A), has been through extensive consultation, and provides a mechanism of escalation and notification to participants, manufacturers or their agents. The notification of suboptimal laboratory performance makes provision for escalation to NATA. Legal advice received recently prevents RCPA QAP from forwarding any participant results to NATA without the written consent from the participant (laboratory). The capacity to do this is critical for the framework to function effectively, and the regulatory framework needs to be reviewed urgently to overcome this barrier.

Software development for this project has two elements, the development of calculations for the newly developed benchmarks and scoring systems, and the development of a novel electronic platform where both laboratories and NATA can review a summary of results and performance. The development of the first element has been partly completed, whilst the ‘dashboard’ concept is still at the early stages of development. Establishing an early warning system must be balanced with the need to avoid collusion, and the manner in which active notification to participants and NATA is perceived. RCPA QAP prefers an approach to initially assist participants in addressing issues of performance and maintaining accreditation.

Significant effort has gone into this project and establishing an early warning system has been demonstrated to be achievable during this project. However, a pilot phase of six months (Jan‐ June 2012) is insufficient time to draw any conclusions to the effectiveness of the framework or the benchmarks for performance. RCPA QAP recommends a further minimum pilot phase of two years to test the framework and benchmarks, which will give further time to overcome the barriers of validation of data with NATA and allow resolution of other internal RCPA issues within the Anatomical Pathology QAP.