Article Text

Utility of routinely collected electronic health records data to support effectiveness evaluations in inflammatory bowel disease: a pilot study of tofacitinib
  1. Vivek Ashok Rudrapatna1,2,
  2. Benjamin Scott Glicksberg1,3 and
  3. Atul Janardhan Butte1,4
  1. 1Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, California, USA
  2. 2Division of Gastroenterology, Department of Medicine, University of California, San Francisco, CA, USA
  3. 3Mount Sinai Medical Center, New York, New York, USA
  4. 4Center for Data-Driven Insights and Innovation, University of California Health System, Oakland, California, USA
  1. Correspondence to Dr Atul Janardhan Butte; atul.butte{at}ucsf.edu

Abstract

Objectives Electronic health records (EHR) are receiving growing attention from regulators, biopharmaceuticals and payors as a potential source of real-world evidence. However, their suitability for the study of diseases with complex activity measures is unclear. We sought to evaluate the use of EHR data for estimating treatment effectiveness in inflammatory bowel disease (IBD), using tofacitinib as a use case.

Methods Records from the University of California, San Francisco (6/2012 to 4/2019) were queried to identify tofacitinib-treated IBD patients. Disease activity variables at baseline and follow-up were manually abstracted according to a preregistered protocol. The proportion of patients meeting the endpoints of recent randomised trials in ulcerative colitis (UC) and Crohn’s disease (CD) was assessed.

Results 86 patients initiated tofacitinib. Baseline characteristics of the real-world and trial cohorts were similar, except for universal failure of tumour necrosis factor inhibitors in the former. 54% (UC) and 62% (CD) of patients had complete capture of disease activity at baseline (month −6 to 0), while only 32% (UC) and 69% (CD) of patients had complete follow-up data (month 2 to 8). Using data imputation, we estimated the proportion achieving the trial primary endpoints as being similar to the published estimates for both UC (16%, p value=0.5) and CD (38%, p-value=0.8).

Discussion/Conclusion This pilot study reproduced trial-based estimates of tofacitinib efficacy despite its use in a different cohort but revealed substantial missingness in routinely collected data. Future work is needed to strengthen EHR data and enable real-world evidence in complex diseases like IBD.

  • computing methodologies
  • medical informatics

Data availability statement

Data are available in a public, open access repository. The analytic code in the form of a R markdown file as well as the accompanying data set needed to reproduce the analysis in this work are available in a Docker container to all investigators without restriction (https://doi.org/10.7272/Q6PZ5715). These individual participant data were de-identified to comply with the US Department of Health and Human Services ‘Safe Harbor’ guidance and applicable laws and regulations concerning privacy and/or security of personal information. The data dictionary is documented within the study protocol section of Supplemental Content.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Summary

What is already known?

  • Real-world data (RWD) are receiving increasing attention from regulators, payors and biopharmaceuticals as an emerging source of evidence on treatment effects.

  • Although electronic health records (EHR) data are an important and granular source of RWD, their suitability for real-world evidence remains unknown in part due to their complexity.

  • Tofacitinib was recently approved for the inflammatory bowel disease (IBD-subtype ulcerative colitis, but its effectiveness and safety in real-world cohorts who may not meet trial eligibility criteria is unclear.

What does this paper add?

  • Although EHR data contain much of the data needed to assess treatment effectiveness in IBD, we found these data to be less accessible (primarily found within free text) and associated with significant missing values at baseline and follow-up.

  • We propose an approach for estimating real-world treatment effects from these data using data abstraction protocols and methods for stochastic imputation of missing data.

  • Although a majority of the patients treated at our centre did not meet the eligibility criteria corresponding to randomised trials of tofacitinib in IBD, its effectiveness appeared to be the same as that measured in the trials.

Introduction

Real-world evidence (RWE) refers to the use of observational data to support inference on diseases and treatments. This area has been growing for a variety of reasons,1–4 including (1) rising costs and other challenges to the feasibility of randomised trials,5 (2) concerns that trial cohorts may be unrepresentative of real-world patients6 7 and (3) the emergence of new datasets and methods for assessing treatment in routine clinical contexts.

Of the sources of real-world data (RWD) that are being explored for this purpose, electronic health records (EHR) are receiving particular attention. They have served as the primary ledger for clinical encounters over two decades and capture rich data on exposures and outcomes. However, this optimism has been tempered by several challenges.1 Beyond limitations common to observational settings (eg, confounding, mismeasurement), EHR data is commonly captured in free text rather than a tabular format. This creates a challenge for the study of diseases whose assessments may be captured in narratives (eg, patient-reported outcomes). Such data typically require the use of text processing, methods that can achieve high accuracy but remain laborious. However, the utility of pursuing these approaches remains unclear because the availability of the underlying data (ie, disease activity scores) in free text is understudied.

An example of a disease currently assessed by complex measures is inflammatory bowel disease (IBD). IBD is comprised of two subtypes, ulcerative colitis (UC) and Crohn’s disease (CD). Treatment involves immunosuppression that is usually continued until treatment failure (eg, inadequate efficacy, adverse events). In trials, effectiveness is measured according to the Mayo Score and Crohn’s Disease Activity Index (CDAI) for UC and CD respectively.

The first small molecule approved for IBD is tofacitinib. Tofacitinib induced clinical remission in 18.5% of the 476 participants with UC who were treated for 8 weeks (OCTAVE 1) and maintained remission in 34.3% of the 197 induction responders assigned to 52 weeks of continued treatment.8 Tofacitinib was also evaluated in phase 2b randomised controlled trials (RCTs) of CD.9 In these trials, 43% of the 86 patients randomised to the 10 mg arm achieved clinical remission following induction (week 8) and 39.5% of the 60 induction responders assigned to the 5 mg arm achieved response or remission at week 26. However, unlike for UC, tofacitinib did not show statistical superiority to placebo for CD and thus was never approved for that indication. Nonetheless, it has sometimes been used off-label for CD.

In this pilot study, we assessed the utility of EHR data for treatment effectiveness evaluations in a cohort of patients with IBD treated with tofacitinib. Our primary objective was to assess disease activity data at timepoints roughly corresponding to the trial endpoints. An exploratory objective was to estimate tofacitinib’s effectiveness using EHR data and compare it with the trials. Other exploratory objectives included characterising differences in patient cohorts, time-to-treatment-failure and the reasons for treatment failure.

Methods

This retrospective cohort study of patients with IBD treated with tofacitinib was performed according to the STROBE and RECORD guidelines (online supplemental file 1).

Supplemental material

Patient identification

We directly queried an existing database derived from all EHR records at the University of California, San Francisco (UCSF). This previously described database10 contains records from 6/2012 (instantiation of the Epic EHR) through 4/2019 (query date) and includes diagnoses, procedures, demographics and medications. Eligible records met these criteria: (1) age over 18 years, (2) a tofacitinib order and (3) a gastroenterologist-assigned IBD diagnosis code (ICD-10-CM K50*/K51*) (table 1). Records meeting the above informatics criteria were all manually reviewed to identify a cohort of all adult patients at UCSF who had (1) ever been prescribed tofacitinib for the treatment of IBD and (2) initiated treatment.

Table 1

Baseline demographics

Study endpoints

The primary endpoint was the proportion of patients with complete measurements of the Mayo Score and CDAI at baseline and follow-up. For this pilot study, baseline was defined as month −6 to 0 relative to the start date of tofacitinib, and follow-up was defined as month 2 to 8. These time-windows were chosen to reflect typical patterns of data collection in clinical practice while also allowing for rough comparisons to the timepoints assessed in trials.

An exploratory endpoint was the proportion of patients meeting the endpoints as defined by the OCTAVE trials8 in UC and the CD trials by Panés et al9 (see ‘Comparison to trial endpoints’ below). Other exploratory endpoints included characterising differences in patient cohorts, time-to-treatment-failure, and the reasons for treatment failure.

Disease activity scores

The Mayo score is scored on a 0–12 scale corresponding to the sum of four equally weighted subscores. The CDAI ranges from 0 to over 600; it incorporates three patient-reported outcomes, comorbidities, weight, haematocrit and medication use. In the gastroenterology clinic at UCSF, elements of these scores are individually captured in clinical narratives as relevant to the provision of routine care; these are not captured as structured data (eg, ‘smartforms’).

Data quality, completeness, and handling of missing data

We assessed the quality of the data in detail prior to proceeding with downstream analysis. We annotated missing data and characterised its distribution (figures 1 and 2). The proportion of patients with complete capture of the Mayo score and CDAI at baseline and follow-up were computed (primary endpoint). We also computed the proportion of non-missing data elements taken as a whole.

Figure 1

Distribution of missing data in the ulcerative colitis dataset. Variables are listed on the x-axis in order of decreasing missingness. Each row in the y-axis corresponds to a different patient. Variable abbreviations correspond to the following: MRN=medical record number; dateOnset=date of disease onset; diseaseDuration=length of disease; diseaseLocation=location of disease by Montreal classification; startDate=date of treatment initiation; lastDate=date of last known use of treatment; status=0 if still using tofacitinib at last date, 1 if no longer using tofacitinib at last date; cotherapy=use of other concomitant therapies (eg, mesalamine, curcumin, simple carbohydrate diet); bslnCrp=baseline C reactive protein; fuCrp=follow up C reactive protein; bslnCal=baseline faecal calprotectin; fuCal=follow-up faecal calprotectin; bslnSteroid=baseline corticosteroid use; fuSteroid=follow-up corticosteroid use; bslnStoolFreq=baseline Mayo stool frequency subscore; fuStoolFreq=follow-up Mayo Stool Frequency subscore; bslnBleed=baseline Mayo Rectal Bleeding subscore; fuBleed=follow-up Mayo Rectal Bleeding subscore; bslnPGA=baseline Mayo Physician Global Assessment subscore; fuPGA=follow-up Mayo Physician Global Assessment subscore; bslnEndo=baseline Mayo Endoscopic subscore; fuEndo=follow-up Mayo Endoscopic subscore.

Figure 2

Distribution of missing data in the Crohn’s disease dataset. Variables are listed on the x-axis in order of decreasing missingness. Each row in the y-axis corresponds to a different patient. Covariate abbreviations are as follows: ‘bsln’ and ‘Fu’ prefixes refer to variable at baseline or at follow-up; MRN=medical record number; dxDate=diagnosis date; startDate=date of treatment initiation; lastDate=date of last known use of treatment; status=0 if still using tofacitinib at last date, 1 if no longer using tofacitinib at last date; ageAtDx=age at diagnosis; ageAtStart=age at treatment start; Wt=weight; idealWt=ideal wt; ht=height; priorSurg=history of prior gastrointestinal surgery; locationIC=disease location in the lower gastrointestinal tract; locationPerianal=presence of disease in the perianal region; locationUGI=disease location in the upper gastrointestinal tract; Ostomy=presence of an ostomy; Imm=use of immunomodulators; ASA=use of aminosalicylates; steroid=use of corticosteroids; complications=complications CDAI subscore; wellbeing=wellbeing CDAI subscore; AbdPain=abdominal pain CDAI subscore; penetrating=penetrating disease behaviour; structuring=structuring disease behaviour; Hct=haematocrit; hxTNFiFail=history of TNF inhibitor failure; oneTNFiFail=history of only one prior TNF inhibitor failure; multipleTNFiFailure=history of multiple TNF inhibitor failures; TNFiFailMech=classification of TNF inhibitor failure. CDAI, Crohn’s Disease Activity Index; TNF, tumour necrosis factor.

We handled missing data using a model-based approach, which relies on the data meeting the missing at random assumption. This was deemed plausible because (1) the clinical decision to pursue additional testing is typically dictated by the results of other correlated data and the risks/benefits of additional studies, and (2) we collected a wide range of auxiliary variables that inform clinical decision making (see ‘Covariate abstraction’).

We performed multiple imputation by chained equations using random forest models (online supplemental file 1). These methods have a lower false discovery rate than last-observation-carried-forward,11 a method commonly used in IBD trials.

Covariate abstraction

Patient records were reviewed via the clinician-facing interface, which contains all clinical data, including notes, patient-provider messaging, procedure reports and laboratory results (online supplemental eTable 1). The EHR contains all clinical data generated within UCSF as well as that shared from other health systems during clinical care.

All patients were assessed by the time-to-treatment-failure, defined as either a lack of efficacy or a significant adverse event recognised by both the clinician and the patient (figure 2). This variable was distinguished from treatment non-compliance defined as a patient-initiated discontinuation against medical advice. This was separately measured during abstraction and was found to be available for all patients (online supplemental file 1). Patients who had not failed treatment at the time of data abstraction were treated as having had non-informatively censored events. Treatment discontinuation due to loss of insurance coverage as well as relocation or other lost-to-follow-up events were rare and were treated as non-informatively censored.

A random sample of the patient records in this study was selected for abstraction of the remaining variables. This was done to strike a balance between estimating parameters with reasonable precision and the effort required for this manual review process (32 and 47 variables per record for UC and CD). The full list and definition of these variables is available in the protocol (online supplemental file 1).

CDAI elements incorporating an average daily rating over 7 days were calculated by extrapolating from a single day’s mention within the time windows mentioned above. This decision was made based on accepted practices of the handling missing CDAI diary data in registrational trials (eg, UNITI-1 Statistical Analysis Plan section 5.2.112) and the methods used to derive the CDAI.13

Comparison to trial outcomes

An exploratory endpoint of this study involved estimating the proportion of patients meeting the endpoint of the trials. As mentioned, a follow-up window of months 2–8 after treatment was used to assess disease activity after initiating treatment. This window was chosen to account for the typical follow-up time in practice, but does not precisely match either the induction or maintenance endpoint times for either OCTAVE (weeks 8 and 52) or the corresponding CD trials9 (weeks 8 and 26).

Because our timepoint more closely matched that of maintenance than of induction, and because each trial only assessed remission among those achieving treatment response following induction (ie, others were assumed to be maintenance-phase non-responders), we recomputed the maintenance endpoint probability as the probability of induction patients being eligible for the maintenance trial by the probability of maintenance response among those enrolled. This probability was statistically compared with the endpoint probabilities in the UCSF-cohort.

These binary endpoints were computed using the same definitions as those in the corresponding trials. For UC, this was the proportion with a total Mayo score ≤2, no individual subscore greater than 1 and a rectal bleeding subscore of 0. For CD, this corresponded to the probability that a patient had either achieved a 100-point reduction in the CDAI from baseline or an absolute CDAI less than 150 at follow-up.

Statistics/computing

We computed point estimated and performed hypothesis testing using Wald test statistics with pooled standard errors.14 For analyses comparing the probability of remission in the real-world cohort with that of the RCTs, the prespecified null hypothesis was these two probabilities were equal. We estimated the time-to-treatment-failure survival distributions using the product-limit estimator. No competing events were observed. Code written in R was independently reviewed by a co-author. Data and analysis files were version-controlled using Docker.

Results

Cohort identification

We identified 115 patient records following a query of our EHR database. Manual review confirmed that 86 patients—68 with UC and 18 with CD—had initiated tofacitinib specifically to treat IBD (figure 3). The other 29 patients were excluded during this process for multiple reasons, including failure to start treatment due to payor denial, the decision to forgo the ordered medical treatment in favour of surgery and treatment initiated by a non-gastroenterologist for another autoimmune condition. Non-compliance, defined as patient-initiated discontinuation of tofacitinib against medical advice, was rare (4%) in this cohort.

Figure 3

Cohort selection schematic. EHR,electronic health records; GI, gastrointestinal; IBD, inflammatory boweldisease; UCSF, University of California, San Francisco.

Data completeness

Out of 28 patients with UC randomly sampled for full assessment of the Mayo score and all other auxiliary variables at baseline and follow-up, 15 (54%) had a complete capture of the Mayo score at baseline and 9 (32%) had a complete capture at follow-up. The least available subscore was endoscopy (figure 1). With respect to the partial Mayo score, 21 (75%) and 17 (61%) were complete at these timepoints. Out of 13 patients with CD sampled, 8 (62%) had complete capture of the CDAI at baseline and 9 (69%) had this available at follow-up.

Taken as a proportion of the total number of collected variables, 13% of the UC-related data and 9% of the CD-related were missing (figures 1 and 2). These missing data were handled by multiple imputation.

Cohort characterization

The baseline demographics of the subjects under study in the UCSF and RCT cohorts were similar (table 1). Notable differences include the universal failure of TNF inhibitors in the UCSF cohort, as well as a longer duration of disease in the patients with UC. Patient groups had similar baseline Mayo scores, C-reactive protein levels and prevalent corticosteroid use. Sixty-one per cent of the cohort had been using corticosteroids at baseline. Thirty-nine per cent of the cohort used at least one form of additional IBD treatment: these included mesalamine, curcumin and dietary changes.

Zero per cent of the patients with UC initiated on tofacitinib met the eligibility criteria of the corresponding phase 3 RCT.8 The reasons for this were multifactorial (online supplemental eTable 2) but include use of vedolizumab within the previous year, high-dose steroids at the time of treatment initiation and the possibility of requiring surgery during the treatment period.

We separately explored what proportion of patients met the specific RCT entry criteria defined by the Mayo score and CDAI for UC and CD, respectively. Ninety-three per cent (73–98) of the patients with UC had an eligible baseline Mayo score (6–12), whereas 50% (19–82) of the patients with CD had a baseline CDAI within the eligibility range of the corresponding RCT (220–450).

Effectiveness and safety

Time-to-treatment-failure analysis on the full cohort revealed similar survival distributions irrespective of IBD disease subtype (online supplemental eFigure 1). The overall probability of incident users continuing tofacitinib long-term was 68% (58%–80%). All failure events occurred within the first 7 months; among continued responders by month 6, the probability of sustained absence of treatment failure was 94%. Of note, the first use of the tofacitinib occurred in 2013, and the longest duration of effectiveness data relevant to treatment maintenance was 3.7 years.

We assessed the reasons for treatment failure (online supplemental eFigure 2). In the UC cohort, there were 17 treatment failure events: 12 with insufficient treatment efficacy, 4 with adverse events/intolerances and 1 due to patient preference. Of the 12 efficacy failures, 8 patients (67%) contained evidence of ongoing inflammation on the basis of biomarkers, imaging or lower endoscopy performed within the 2-month period prior to treatment failure. All patients who did not undergo objective confirmation of inflammation during this timeframe did have objective evidence of inflammation prior to treatment initiation. All but one patient with inadequate response completed a minimum of 7 weeks of treatment induction (11 weeks on average) prior to the adjudication of treatment failure.

In the CD cohort, there were five treatment failure events: one due to an adverse event (zoster) and one due to insufficient efficacy (all with concomitant objective evidence of ongoing inflammation). These patients completed 13.2 weeks of treatment on average.

Twenty-two per cent of all subjects participating in the induction phase of the UC RCT8 met the primary maintenance endpoint of week 52 clinical remission. We observed a similar response (16%) in the corresponding UCSF cohort (6%–37%, p value=0.5). Similarly, the proportion achieving the primary endpoint in the CD RCT9 (34%) was similar to the point estimate of the real-world cohort (38%, p-value=0.8).

We explored the extent to which steroid use may account for some of these results. In the UC cohort, 33% of patients had been using steroids at the time of follow-up. Among the patients who had been using steroids at baseline, 56% were steroid-free at the time of follow-up.

Discussion

We assessed the completeness of routinely collected EHR data to support RWE studies of diseases with complex activity measures. Taking a use case of tofacitinib as used to treat IBD (both on-label and off-label), we found that the capture of the total Mayo score and the CDAI s currently modest at best, even at a tertiary-care medical centre.

On exploratory analyses, the real-world effectiveness of this drug appeared to be consistent with its published effectiveness from randomised trials despite its use in a substantially different cohort. We found that patients with IBD using tofacitinib appear to generally tolerate it well and that unlike biologics commonly used for IBD, secondary loss of response events for this small molecule was uncommon.

RWD has been receiving growing interest from a variety of parties including the FDA3 and EMA,15 biopharmaceuticals and payors. Despite this interest, it must be recognised that not all RWD are created equal. Unlike prospectively planned disease and treatment registries, the EHR data capture mechanism has historically been designed with other objectives in mind: healthcare coordination and delivery, revenue generation and medico-legal documentation among others.

Our pilot study highlights the substantial work that will be needed to close the quality gap between retrospective EHR data and prospective data and realise the promise of RWE. We outline the root causes of this quality gap as well as outline potential solutions in table 2. Many of these solutions will ultimately require a close partnership between the many stakeholders in real-world clinical care: clinicians, patients, health IT, operations and especially payors. Undoubtedly, this may require a significant investment in both time and money by these participants. However, we are of the opinion that the eventual rewards are worth the investment. These include the ability to better measure the quality of care, discover practice-changing evidence and enable continuous-improving learning health systems.

Table 2

Potential approaches to strengthen routinely collected electronic health records data and better support real-world evidence studies

Strengths of this study include the use of a preregistered protocol and analysis plan, the use of rigorous methods for handling missing data,11 as well as openly available code accompanied by deidentified raw EHR data in order to maximise the reproducibility and reusability of this work. The primary limitation of this work lies in its inability to draw inferences related to the real-world effectiveness of tofacitinib.

Conclusion

Routinely collected EHR data currently has uneven capture of the data needed to optimally assess IBD treatment effectiveness at baseline and follow-up. This work provides several insights into real-world practice, including typical patterns of data collection and the real-world effectiveness and safety of tofacitinib for IBD. It also offers an analytical approach to the analysis of missing real-world data. Future efforts are needed to improve inference from these data, such better data capture mechanisms and novel measures more suitable to routine care.

Data availability statement

Data are available in a public, open access repository. The analytic code in the form of a R markdown file as well as the accompanying data set needed to reproduce the analysis in this work are available in a Docker container to all investigators without restriction (https://doi.org/10.7272/Q6PZ5715). These individual participant data were de-identified to comply with the US Department of Health and Human Services ‘Safe Harbor’ guidance and applicable laws and regulations concerning privacy and/or security of personal information. The data dictionary is documented within the study protocol section of Supplemental Content.

Ethics statements

Ethics approval

We obtained Institutional Review Board approval to obtain patient data and abstract all covariates.

Acknowledgments

The authors thank the UCSF Academic Research Services and Clinical Data Research Consultation services for clinical informatics support. The authors would like to acknowledge Dana Ludwig for his help in deidentifying and interpreting the UCSF EHR.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @vivicality, @BenGlicksberg, @atulbutte

  • Contributors VAR and AB conceived the project. VAR designed the chart review protocol, performed chart extraction, conducted statistical analyses and drafted this manuscript. BG performed code review and critically edited this manuscript. AB supervised the project and critically edited this manuscript.

  • Funding Research reported in this publication was supported by funding from the UCSF Bakar Computational Health Sciences Institute and the National Center for Advancing Translational Sciences of the National Institutes of Health under award UL1 TR001872. VAR was supported by the National Institute of Diabetes and Digestive and Kidney Disease of the National Institutes of Health grant under award T32 DK007007.

  • Disclaimer The contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

  • Competing interests Atul Butte is a co-founder and consultant to Personalis and NuMedii; consultant to Samsung, Mango Tree Corporation, and in the recent past, 10x Genomics, Helix, Pathway Genomics, and Verinata (Illumina); has served on paid advisory panels or boards for Geisinger Health, Regenstrief Institute, Gerson Lehman Group, AlphaSights, Covance, Novartis, Genentech, Merck, and Roche; is a shareholder in Personalis and NuMedii; is a minor shareholder in Apple, Facebook, Alphabet (Google), Microsoft, Amazon, Snap, Snowflake, 10x Genomics, Illumina, Nuna Health, Assay Depot (Scientist.com), Vet24seven, Regeneron, Sanofi, Royalty Pharma, Pfizer, BioNTech, AstraZeneca, Moderna, Biogen, Twist Bioscience, Pacific Biosciences, Editas Medicine, Invitae, and Sutro, and several other non-health related companies and mutual funds; and has received honoraria and travel reimbursement for invited talks from Johnson and Johnson, Roche, Genentech, Pfizer, Merck, Lilly, Takeda, Varian, Mars, Siemens, Optum, Abbott, Celgene, AstraZeneca, AbbVie, Westat, several investment and venture capital firms, and many academic institutions, medical or disease specific foundations and associations, and health systems. Atul Butte receives royalty payments through Stanford University, for several patents and other disclosures licensed to NuMedii and Personalis. Atul Butte’s research has been funded by NIH, Northrup Grumman (as the prime on an NIH contract), Genentech, Johnson and Johnson, FDA, Robert Wood Johnson Foundation, Leon Lowenstein Foundation, Intervalien Foundation, Priscilla Chan and Mark Zuckerberg, the Barbara and Gerson Bakar Foundation, and in the recent past, the March of Dimes, Juvenile Diabetes Research Foundation, California Governor’s Office of Planning and Research, California Institute for Regenerative Medicine, L’Oreal, and Progenity.

  • Provenance and peer review Commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.