CER: BIO’s comments on PCORI’s Draft Methodology Report

Re:  Patient-Centered Outcomes Research Institute (PCORI) Draft Methodology Report

Dear Dr. Selby:

The Biotechnology Industry Organization (BIO) is pleased to submit the following comments on the Patient-Centered Outcomes Research Institute’s (PCORI’s) Draft Methodology Report, published on the group’s website on July 23, 2012.[1] BIO is the largest trade organization to serve and represent the biotechnology industry in the United States and around the globe. BIO represents more than 1,100 biotechnology companies, academic institutions, state biotechnology centers, and related organizations in the United States. BIO members are involved in the research and development of novel interventions to prevent, treat, and cure diseases through the most advanced science.

BIO supports PCORI’s goal of increasing the availability of accurate, scientific evidence to inform clinical decision-making. BIO supported the creation of PCORI to conduct comparative clinical effectiveness research, and we maintain an ongoing desire to see the Institute successfully carry out its statutory mandate, of which this methodology report is a part. This letter provides feedback on the draft methodology report as it relates to the fulfillment of PCORI’s mission, improved and unbiased transparency around potential conflicts of interest, the appropriateness of proposed methodological standards for rare disease research, and the importance of adopting a systematic and transparent public input process.

I.                   Fulfilling the PCORI Mission

A.     BIO supports the patient-centered focus of the draft methodology report, but asks PCORI to engage with patients on additional issues relevant to designing and conducting patient-centered outcomes research (PCOR).

 BIO applauds the draft methodology report’s patient-centered focus. A significant portion of the report is spent establishing standards for patient input and participation at each stage of the research undertaking, aligning the report closely with PCORI’s mission. Specifically, engaging with the patient perspective from the initial study design stages is crucial to accomplish the aim of making research results more relevant to the aspects of treatment that are most relevant to patients.

To improve upon this strong start, PCORI should expand the list of potentially relevant issues on which researchers should consider the patient perspective to improve the relevance of PCOR to patient decision-making, to include interpreting and understanding relevant medical practices and study design, issues of confidentiality, and ethical, legal, and regulatory concerns. Considering a broader list of issues on which the inclusion of patient perspectives might impact how researchers design and conduct PCOR will further strengthen the relevance of the research results to patient decision-making.

B.     PCORI should remove all references to cost as an element of PCOR in the final methodology report.

While BIO recognizes that the cost of treatment has the potential to impact patient and provider decisions about care, the draft methodology report inappropriately identifies cost as a potential endpoint of research studies undertaken by PCORI. The report states that “the Committee’s view is that in the context of PCOR, cost, like other aspects of the healthcare delivery system, can be a factor in the effectiveness of care if it influences choices made by patients and clinicians.”[2] This is in direct conflict with the authorizing statute’s specific prohibition of PCORI from considering cost effectiveness in studies of comparative effectiveness.[3] Therefore, to comply with this statute, PCORI should remove from the draft methodology report all references to cost as a PCOR endpoint. The final report should further clarify that an assessment of cost in PCOR is only appropriate if its aim is to identify cost as a potential confounding element of a research study; this would be important because it could improve interpretation and assessment of the research results.

C.      PCORI should limit the methodology report to fulfilling only the requirements set forth by the statute.

The Patient Protection and Affordable Care Act (ACA) of 2010 requires the methodology report to fulfill two main responsibilities: that it “contain recommendations for the Institute to adopt methodological standards developed and updated by the methodology committee as well as other actions deemed necessary to comply with such methodological standards” and that it develop a translation table.[4] Yet the diversity of other subjects the report discusses—specifically the dissemination of research results and research priorities—distracts from the mandated focus on scientifically-derived methodological standards for PCOR and the framework underpinning the development of a translation table. To strictly comply with the statute’s requirement, which benefits the overall focus and clarity of the proposed standards and translation table, PCORI should omit all non-mandated subjects from the final methodology report. Instead, the PCORI Board should comment on these topics, if it feels this is necessary, in a separate, more appropriate forum.

D.      PCORI should clarify the process of interaction between users of the translation table.

 One of the two statutory requirements of the methodology committee is to develop a translation table to pair specific PCOR research questions with appropriate methodologies.[5] This table is meant to be integral to standardize PCOR, improve the replicability of its results and their applicability to real-world decision-making, and ensure that innovation and the realities of conducting non-traditional scientific research on patient-centered outcomes remains at the forefront of the PCORI-funded work. Rather than propose a specific format for the translation table, the draft methodology report outlines a framework on which to build a multi-faceted decision tree to serve as the translation ‘tool’. While BIO recognizes PCORI’s efforts to root the tool itself in scientific methodological standards, the process that will govern the interaction between the tool’s users—investigators in designing study methodology and PCORI in making funding decisions—is unclear. A lack of standardization and predictability of this interaction may result in a convoluted and inefficient funding process, ultimately delaying the timely conduct of research. The PCORI Board, in the forum it deems most appropriate, should provide further clarity about the interaction, including if it intends to engage in an iterative process with other users of the tool in devising an appropriate study methodology, and at what point in the funding process (prior to, during, or after funding has been granted) this process would occur. Resolutions to these issues are crucial prior to the start of research to resolve any issues that might otherwise interfere with conducting robust PCOR.

E.     PCORI should clarify its statement implying the creation of multiple versions of the proposed methodological standards that vary based on the role of the researcher and/or funder.

In the introduction to the proposed methodological standards, PCORI states that “in this report we focus on patients rather than on other health decision stakeholders, who will be the focus of future standards”, but does not further elaborate.[6] PCORI needs to clarify if it intends methodological standards for conducting PCOR to vary depending on the perspective of the researcher or funder (e.g., as a payer, provider, clinician, health system administrator, regulator, policy maker). BIO discourages an approach to conducting PCOR that sets different methodological standards for different types of researchers and funders, and reminds PCORI that such an approach would countermand the statute’s intent that “any methodological standards developed and updated under this subclause shall be scientifically based”.[7] If PCORI is proposing such an approach, it should provide a comprehensive rationale for its position and allow the public an opportunity to provide comments.

F.      PCORI should appropriately contextualize comparisons of the value of different PCOR studies and clarify that such comparisons should extend beyond a Value of Information Analysis (VOI).

BIO acknowledges that statue allows PCORI to take into account “the relative value (determined based on the cost of conducting research compared to the potential usefulness of the information produced by the research) associated with different types of research” when determining its agenda.[8] BIO reiterates its position that the draft methodology report was not intended to be an exercise in priority-setting (as stated above in section C), and that PCORI should address these issues in an alternate forum that it deems most appropriate. Nonetheless, in evaluating trade-offs between types of research, PCORI should look beyond just employing VOI, which is the primary focus of the draft methodology report’s discussion on the issue of assigning relative value to research. While other research agencies, like the Agency for Healthcare Research and Quality (AHRQ), have employed VOI in their own priority-setting endeavors, PCORI should address how the differences between traditional research and PCOR may impact the utility of VOI as a value measurement.[9] PCORI should also address concerns that VOI may not be appropriately able to account for significant differences across PCOR studies related to differences in diseases, inclusion criteria, primary endpoints, and study assumptions. PCORI should clarify how VOI is relevant to decision-making on the individual patient level and within and across different patient subpopulations. Finally, PCORI should detail how VOI fits into the broader context of defining methodological standards for conducting PCOR and/or pairing research methodologies with specific study questions (the subject of the draft report).

II.                Transparency around Potential Conflicts of Interest: PCORI should promote a fair assessment of the merits of research conducted by experts in the field, no matter their affiliation.

The language in the draft methodology’s section ‘Problems that PCORI Hopes to Address’ is unduly biased against industry-sponsored research, as it specifically cites an example of distrust as “a pharmaceutical company that conducts studies of its own products… so as to support an application for regulatory approval and ultimately success in the marketplace.”[10] This statement presents a skewed view of research conducted by biopharmaceutical companies—who are most often the only entities to sponsor research that results in the development of new treatments for patients with serious diseases. It ignores the fact that such research and approvals are strictly regulated.

PCORI should remove the language that implies that research conducted by pharmaceutical companies is inherently untrustworthy. PCORI should instead clarify that valuable research can come from a variety of sources, and that the authenticity and scientific validity of research must be determined independent of a study’s source. PCORI should also reiterate that transparency around financial relationships that might potentially constitute a conflict of interest is universally important no matter the research source. Finally, PCORI should construct a framework for formal oversight and audit of transparency, with opportunities for public comment, which leverages and builds from existing oversight models already in use by research institutions and agencies (e.g., the Institutional Review Board).

III. Methodological Appropriateness for Rare Disease Studies: PCORI should clarify that some of the methodological standards may not always be appropriately applied to studies of treatments for rare diseases.

BIO appreciates the draft methodology report’s acknowledgement that the focus of PCORI’s priorities must strike a balance between “widespread health threats”[11] and those impacting very small patient populations (i.e., rare diseases). However, the draft report does not adequately detail which of its standards should not be applied to rare disease PCOR—because of factors like inherently small patient populations—and what, if any, alternative standards may be appropriately substituted in such circumstances (e.g., Bayesian methods for observational studies). For instance, meeting some of the standards outlined in this document will require research conducted in samples of sufficient size, which is not always possible in rare disease research.

Without further clarification, BIO is concerned that because some of the current methodological standards are inappropriate measures by which to judge rare disease studies, those studies will suffer a competitive disadvantage in the funding process. To avoid this, PCORI should qualify which of the report’s proposed methodological standards can, and cannot, be appropriately applied to rare disease PCOR, as well as indicate potentially appropriate substitute methods where available. This strategy represents a broader-based, systematic effort to incorporate the rare disease community into PCOR, in addition to utilizing the rare-disease-specific expert advisory panels, provided for in statute, whose purpose is to evaluate the merits of a specific rare disease research study proposal.[12] Similarly, the Methodology Committee may consider proposing methods for rare disease researchers to leverage existing resources (e.g., datasets maintained by patient advocacy organizations) to assist in meeting the draft report’s proposed methodological standards when possible.

IV.              Length of Comment Period and Use of Public Comments: PCORI should standardize its comment period, and develop a formal process for detailing how public comments are incorporated in the final methodology report.

BIO appreciates PCORI’s stated encouragement of public comments and intention to use stakeholder feedback to refine the draft methodology report before its final publication. Nonetheless, to improve consistency and predictability going forward, BIO asks PCORI to standardize its comment period to 60 days, a precedent set by other governmental agencies (e.g., AHRQ, Centers for Medicare and Medicaid Services [CMS]).

BIO also continues to be concerned that PCORI has no formal mechanism for incorporating or declining to incorporate public feedback. As we have previously recommended, PCORI should develop and describe in detail a systematic process regarding how it will synthesize and incorporate comments into the final draft of the methodology report. BIO suggests that PCORI prepare a document, such as those released by CMS during comment periods, that compiles stakeholders’ comments and discusses the rationale behind PCORI’s final decisions regarding inclusion of those comments. Understanding how public comments are used will improve their value to both the institute and stakeholders.

                    V.                 Conclusion

BIO appreciates the opportunity to comment on the draft methodology report. We look forward to continuing to work with the institute to provide patients with information relevant to their concerns and perspective that can assist them in making better health care decisions. Thank you for your attention to this very important matter.