Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that facilitates research into pragmatic trials. It collects and distributes cleaned trial data, ratings and evaluations using PRECIS-2. This permits a variety of meta-epidemiological analyses that examine the effect of treatment across trials of various levels of pragmatism.
Background
Pragmatic trials provide evidence from the real world that can be used to make clinical decisions. The term "pragmatic" however, is used inconsistently and its definition and evaluation require clarification. Pragmatic trials are designed to inform clinical practices and policy decisions rather than confirm a physiological hypothesis or clinical hypothesis. A pragmatic study should strive to be as close as is possible to the real-world clinical practice which include the recruitment of participants, setting up, delivery and execution of interventions, determination and analysis results, as well as primary analyses. This is a significant difference between explanation-based trials, as described by Schwartz and Lellouch1 that are designed to prove the hypothesis in a more thorough manner.
Trials that are truly pragmatic should avoid attempting to blind participants or the clinicians as this could cause distortions in estimates of the effect of treatment. Pragmatic trials should also seek to recruit patients from a variety of health care settings, to ensure that their findings can be applied to the real world.
Finally, pragmatic trials must be focused on outcomes that matter to patients, like quality of life and functional recovery. This is particularly relevant when trials involve surgical procedures that are invasive or may have harmful adverse effects. The CRASH trial29 compared a 2-page report with an electronic monitoring system for patients in hospitals with chronic heart failure. The catheter trial28, however utilized symptomatic catheter-related urinary tract infections as its primary outcome.
In addition to these features pragmatic trials should reduce the procedures for conducting trials and data collection requirements in order to reduce costs. Finaly, pragmatic trials should aim to make their results as relevant to real-world clinical practices as possible. This can be accomplished by ensuring that their analysis is based on the intention-to treat approach (as described within CONSORT extensions).
Many RCTs which do not meet the criteria for pragmatism but contain features in opposition to pragmatism, have been published in journals of varying types and incorrectly labeled pragmatic. This could lead to false claims of pragmatism and the usage of the term should be standardised. The development of the PRECIS-2 tool, which offers an objective standard for assessing pragmatic characteristics is a great first step.
Methods
In a pragmatic study it is the intention to inform policy or clinical decisions by showing how an intervention could be implemented into routine care. This is different from explanatory trials that test hypotheses regarding the causal-effect relationship in idealized situations. In this way, pragmatic trials could have a lower internal validity than explanation studies and be more prone to biases in their design analysis, conduct, and design. Despite their limitations, pragmatic studies can be a valuable source of information for decision-making within the healthcare context.
The PRECIS-2 tool evaluates an RCT on 9 domains, ranging between 1 and 5 (very pragmatist). In this study, the recruitment, organization, flexibility in delivery, flexible adherence and follow-up domains were awarded high scores, but the primary outcome and the method of missing data fell below the pragmatic limit. This suggests that it is possible to design a trial using high-quality pragmatic features, without damaging the quality of its results.
It is hard to determine the amount of pragmatism within a specific trial because pragmatism does not possess a specific characteristic. Certain aspects of a research study can be more pragmatic than other. A trial's pragmatism could be affected by changes to the protocol or the logistics during the trial. In addition 36% of 89 pragmatic trials discovered by Koppenaal et al were placebo-controlled or conducted before licensing and most were single-center. They are not close to the standard practice, and can only be referred to as pragmatic if their sponsors accept that such trials are not blinded.
Furthermore, a common feature of pragmatic trials is that researchers attempt to make their findings more relevant by analyzing subgroups of the trial. This can result in unbalanced analyses with less statistical power. This increases the risk of omitting or misinterpreting differences in the primary outcomes. This was the case in the meta-analysis of pragmatic trials as secondary outcomes were not corrected for covariates' differences at baseline.
Additionally practical trials can have challenges with respect to the collection and interpretation of safety data. This is due to the fact that adverse events are usually self-reported, and are prone to delays, errors or coding errors. It is therefore important to improve the quality of outcomes ascertainment in these trials, and ideally by using national registries rather than relying on participants to report adverse events in the trial's database.
Results
Although the definition of pragmatism may not require that all trials are 100 100% pragmatic, there are advantages to including pragmatic components in clinical trials. These include:
Increasing sensitivity to real-world issues, reducing study size and cost as well as allowing trial results to be more quickly transferred into real-world clinical practice (by including patients who are routinely treated). However, pragmatic trials may be a challenge. For example, the right kind of heterogeneity can allow a trial to generalise its results to many different settings and patients. However the wrong kind of heterogeneity can reduce assay sensitivity and therefore decrease the ability of a trial to detect minor treatment effects.
Numerous studies have attempted to categorize pragmatic trials with a variety of definitions and scoring systems. Schwartz and Lellouch1 developed an approach to distinguish between explanation-based trials that support a clinical or physiological hypothesis, and pragmatic trials that aid in the selection of appropriate therapies in clinical practice. he has a good point was comprised of nine domains, each scored on a scale of 1-5, with 1 indicating more explanatory and 5 indicating more practical. The domains were recruitment setting, setting, intervention delivery with flexibility, follow-up and primary analysis.
The initial PRECIS tool3 had similar domains and an assessment scale ranging from 1 to 5. Koppenaal et. al10 devised an adaptation of this assessment, called the Pragmascope which was more user-friendly to use for systematic reviews. They found that pragmatic reviews scored higher on average in most domains, but scored lower in the primary analysis domain.
This distinction in the main analysis domain could be explained by the fact that most pragmatic trials analyze their data in the intention to treat method, whereas some explanatory trials do not. The overall score was lower for pragmatic systematic reviews when the domains on organisation, flexible delivery and follow-up were merged.
It is important to note that the term "pragmatic trial" does not necessarily mean a low-quality trial, and there is an increasing number of clinical trials (as defined by MEDLINE search, however this is not specific or sensitive) which use the word 'pragmatic' in their abstract or title. These terms may signal a greater understanding of pragmatism in abstracts and titles, however it's unclear whether this is evident in the content.
Conclusions
As appreciation for the value of real-world evidence grows popular and pragmatic trials have gained popularity in research. They are randomized trials that compare real world alternatives to new treatments that are being developed. They include patient populations that are more similar to those who receive treatment in regular medical care. This approach has the potential to overcome limitations of observational studies which include the limitations of relying on volunteers, and the limited availability and the variability of coding in national registry systems.
Pragmatic trials offer other advantages, like the ability to draw on existing data sources and a greater likelihood of detecting meaningful distinctions from traditional trials. However, pragmatic trials may still have limitations that undermine their validity and generalizability. Participation rates in some trials could be lower than expected due to the healthy-volunteering effect, financial incentives or competition from other research studies. The necessity to recruit people in a timely manner also limits the sample size and the impact of many pragmatic trials. Additionally some pragmatic trials lack controls to ensure that the observed differences aren't due to biases in trial conduct.
The authors of the Pragmatic Free Trial Meta identified RCTs published up to 2022 that self-described as pragmatism. The PRECIS-2 tool was used to evaluate the degree of pragmatism. It includes domains such as eligibility criteria as well as recruitment flexibility and adherence to intervention and follow-up. They found 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or above) in at least one of these domains.
Trials with a high pragmatism score tend to have more expansive eligibility criteria than traditional RCTs that have specific criteria that are unlikely to be used in the clinical setting, and contain patients from a broad range of hospitals. These characteristics, according to the authors, can make pragmatic trials more useful and applicable in everyday practice. However they do not ensure that a study is free of bias. Furthermore, the pragmatism of a trial is not a fixed attribute and a pragmatic trial that does not contain all the characteristics of a explanatory trial may yield reliable and relevant results.