Poor quality clinical data informing NICE decisions on over half of treatments
The quality of evidence submitted to the National Institute for Health and Care Excellence (NICE), informing its decisions to recommend technologies for use in the NHS, was poor in more than half of cases, according to a 20-year analysis.
The data quality submitted for health technology appraisals by manufacturers between 2000 and 2019 was consistently poor, with no improvement during that time, the analysis shows.
NICE advises the NHS on the clinical and cost effectiveness of both new and existing technologies, through an independent body of specialists who make up its appraisal committee.
This committee’s decisions are based on reports from its independent technology assessors, plus advice from consultees, clinical, NHS commissioning and patient experts.
In recent years, NICE has expanded the evidence it considers to include a broader range of factors that influence health: registry data, national statistics, surveys, clinical practice recommendations, expert opinions and additional knowledge from manufacturers.
In light of this, the researchers wanted to systematically review all NICE’s active technology appraisals published between 2000 and 2019 to scrutinise the clinical evidence submitted by the manufacturers and assess its quality for decision-making.
They extracted data from the independent assessment group and evidence review group reports and final appraisal determinations on the quality of submitted randomised controlled clinical trials, and the overall quality of evidence submitted for decision-making.
For single technology appraisals (STAs), which evaluate a single product, device, or technology for a single indication, and usually involve new drugs or indications, they also extracted data on quality of life evidence and comparative clinical evidence.
Each category was scored for quality: two for good one for acceptable, zero for poor, and minus one for unacceptable. The scores were peer reviewed by all the members of the research team to try and minimise bias.
In all, the evidence for 409 technology appraisals was analysed: 104 multiple technology appraisals (MTA), which evaluate technologies that share one or more criteria, and 305 STAs.
The appraisals included 25 non-pharmaceutical products, 14 medical devices, six other therapies, five surgical procedures, and 384 drugs.
In two thirds of all appraisals, the overall quality of evidence was judged to be either poor (55%) or unacceptable (10%). The quality of evidence was judged acceptable in a third (34%) and good in only 1%.
In nearly four out of ten (39%) STAs, the quality of comparative evidence was considered poor, and in 17% unacceptable. In 44% the quality of quality of life data was considered poor, and unacceptable in 15%. In only a third of STA appraisals was the quality deemed acceptable, and good in only 7%.
Based on analysis of the comments from the review group reports, over half of the clinical trials presented in the manufacturers’ submissions for all appraisals were deemed to be either poor (41%) or unacceptable (10%) quality. Just under half were considered to be of acceptable (42%) or good (7%) quality.
Weak or insufficient evidence from poorly conducted clinical trials was often used because it was the only evidence available. But even when the trials had been done well, and the evidence was comprehensive, the comparators were often unsuitable for decision-making in the NHS context.
And over the entire 20-year period the overall quality of evidence submitted to NICE didn’t change and was consistently poor, the analysis indicated.
‘We found that the primary components of clinical evidence … that influence patients and are crucial for NICE’s decision making framework are of poor quality,’ the authors said.
‘Since the evidence bar continues to be lowered, it is essential to have [health technology assessment] bodies and payers’ input to ensure that the generation of evidence submitted to NICE is strengthened.
‘However, it is essential that stakeholders are aware of this and that organisations put more effort into generating high-quality evidence pre-market and post-market entry.’
Image: Adi Goldstein
Vaccine expert claims technology is key to cutting NHS costs