key: cord-0130148-et1qmhrg
authors: Adibi, Amin; Harvard, Stephanie; Sadatsafavi, Mohsen
title: Programmable Interface for Statistical&Simulation Models (PRISM): Towards Greater Accessibility of Clinical and Healthcare Decision Models
date: 2022-02-16
journal: nan
DOI: nan
sha: 2269deb438aef341541a12f9e6bc33e0a283dad3
doc_id: 130148
cord_uid: et1qmhrg

Background: Increasingly, decision-making in healthcare relies on computer models, be it clinical prediction models at point of care or decision-analytic models at the policymaking level. Given the important role models play in both contexts, their structure and implementation be rigorously scrutinized. The ability to interrogate input/output associations without facing barriers can improve quality assurance mechanisms while satisfying privacy/confidentiality concerns and facilitating the integration of models into decision-making. This paper reports on the development of Programmable Interface for Statistical&Simulation Models (PRISM), a cloud-based platform for model accessibility. Methods: PRISM emphasizes two main principles: 1) minimal specifications on the side of model developer to make the model fit for cloud hosting, and 2) making client access completely independent of the resource requirement and software dependencies of the model. The server architecture integrates a RESTful Application Programming Interface (API) infrastructure, JSON for data transfer, a routing layer for access management, container technology for management of computer resources and package dependencies, and the capacity for synchronous or asynchronous model calls. Results: We discuss the architecture, the minimal API standards that enable a universal language for access to such models, the underlying server infrastructure, and the standards used for data transfer. An instance of PRISM is available as a service via the Peer Models Network http://peermodelsnetwork.com. Through a series of case studies, we demonstrate how interrogating models becomes possible in standardized fashion, in a way that is irrespective of the specifics of any model. Conclusions: We have developed a publicly accessible platform and minimalist standards that facilitate model accessibility for both clinical and policy models.

Medical decision making increasingly relies on computational models. In clinical practice, clinical prediction models assist physicians in diagnosing patients and planning the course of treatment based on the projected prognosis of individual patients. Governments and regulators rely on decision-analytic or health economics models to decide which competing heath technologies provide the best outcome for the money spent, while public health units use infectious disease and environmental exposure models to plan and guide exposure control strategies.

Models typically represent substantial intellectual and monetary investments. They are often prototyped by academic, government, or industry researchers with the intention of producing the results that are published in academic papers or reports, integrated into routine care, submitted to approval agencies, or used to inform specific technology adoption or policy decisions within a jurisdiction. Developing well-documented, optimized, and easy-to-use software is rarely the goal of modeling units in such organizations, and hardly within their means.

In this manuscript, we focus on a particular challenge facing all these modelling efforts: difficulties in in accessing model outputs. As we describe, this lack of model accessibility prevents thorough model interrogation, resulting in lack of trust and potentially inaccurate predictions, causes unnecessary repetition leading to wasteful research, and slows the uptake and application of such models. We briefly review efforts to overcome these challenges as they apply to modelling in healthcare, and propose PRISM, a cloud-based model accessibility platform for models developed in R.

Clinical prediction and policy models should be scrutinized before their output informs decisions that can potentially affect millions of lives. To support this process, reporting guidelines for clinical prediction models [1] and health economic evaluations [2] recommend providing details on the rationale and basis for model assumptions and the types of validation undertaken, and sufficiently transparent reporting to enable others to replicate the model, evaluate its validity, and assess the model's overall usefulness to inform decision-making. Others have called for making models open-source [3] and following standard coding practices [4] to improve transparency and facilitate model-sharing. A particularly laudable effort is the Global Health Cost-Effectiveness Analysis Registry that provides a growing database of open-source models (http://ghcearegistry.com/orchard/about-the-clearinghouse) [5] .

One possibility is to move beyond reporting and mere code-sharing and facilitate direct access to models, providing both technical and non-technical stakeholders the opportunity to interrogate models and explore different sets of inputs and outputs and the impact of altering model assumptions. In the context of modeling, 'accessibility', the ability of an end-user to interrogate the input/outputs of a model, is a particularly robust and often overlooked form of 'transparency'.

There are various reasons for pursuing accessibility as a form of transparency, for both clinical predication and decision-analytic models. For one, increasingly, attention is being paid to social and ethical value judgments in modelling [6] [7] [8] , leading to calls for greater reflection and information on models' adequacy-for-purpose from the perspective of different stakeholders. Even when the models are open source, many stakeholders lack the technical knowledge and means to access and examine the model for themselves. A lack of accessibility also hampers further developments, re-use, and clinical implementation of models, and leads to repetition and wasteful research. We have previously called for a survival-of-the-fittest approach towards validation of healthcare models, where multiple clinical prediction models can be set up in a pipeline to receive real-time streaming data (e.g., from an electronic medical records (EMR) system) and compete for better predictive performance over time [9] . An analogous setup for policy models is prospective testing of model projections with the arrival of new studies, as it is practiced by the Mt Hood Diabetes Challenge Network [10] . Direct model access would greatly facilitate not only these initiatives but also integration of healthcare models into other pieces of software and hardware, including but not limited to diagnostic devices, EMR systems, web and smart phone applications, and software packages used for research.

Lacking direct model access, peer-reviewers, policymakers, and other users of models often rely on selected results from a model published in a manuscript or report to judge the quality and usefulness of the model. While some clinical prediction models are simple and can easily be programmed by the user, others require complex computations. Many decision models are so complex that it would be nearly impossible to fully document their inner workings (to the point of enabling reproducibility) within the confines of a typical scientific report. While an increasing push for transparency has compelled many modellers to make their models open-source, it is often the case that very few people, other than the developers of the model, are practically able to quality-certify a model based on its code. This is particularly the case for the larger and more complex 'whole disease' models that are being advocated for to reduce duplicate efforts in decision modeling and represent the broader context of decision problems [11] . It is not feasible for a peerreviewer to vet, in limited time, the internal and external validity of a large decision model that was developed over the course of several years by several developers. Even when the code is made available, a common challenge is in getting a complex model to run on the local computing environment of the user, due to the potential need for specialized software environment or hardware specification.

For clinical prediction models, the need for accessibility stems from the direct applicability of such models at point of care, such as integration within Clinical Decision Support Systems and EMR. Currently, clinicians might find themselves limited to models provided by a certain commercial entity. A troubling trend in recent years has been an increasing number of undervalidated models or proprietary models embedded in EMR systems that have not gone through extensive peer review and independent validation [12] . If all clinical prediction models in the same clinical area could be accessed in standardized ways, then health providers could make an informed decision to pick the model, if any, that would have the best performance in their local setting, while being able to monitor troublesome trends such as dataset shift and calibration drift in real-time [13, 14] .

Several challenges need to be addressed before enabling public access to models and some of these challenges are common to both clinical prediction and policy models. Some models might require on-demand access to individual patient data that cannot be made public. For example, a clinical prediction model for lung function decline handles missing predictors by fitting a new model on the original development data 'on demand' [15] . It is also common for cost-effectiveness models, notably those submitted to health technology assessment agencies by pharmaceutical companies, to contain intellectual property and proprietary information that the sponsor is unwilling to share with the public. A practical model accessibility platform should be able to protect confidential information such as patient data and confidential pricing.

Health technology assessment models are traditionally implemented in Microsoft Excel, which is not an ideal platform for complex model development [16] . While models built in Excel can be shared, they are incredibly difficult to test and debug, offer little in the way of version control and documentation of model development, and at times require the user to have high-end processing capabilities or face very long running times. Occasionally, web applications or dashboard prototypes that accompany the submission help end-users interactively explore some of the results. Well-resourced modeling efforts might have their customized implementation

solutions. An example is the OncoSim models in Canada [17] .

Clinical prediction models that are picked up by guidelines such as the Atherosclerotic Cardiovascular Disease Risk Estimator [18] usually have publicly accessible web portals. While web apps do make healthcare models more accessible, professional development and deployment of web apps is often too expensive for model development teams and in-house-developed web apps often come with poorly designed user interfaces and unreliable back-ends that affect user experience. Further, even professionally developed web applications still do not provide programmatic access to the model required for high-throughput validation and testing.

Recently, the Shiny web app framework for R programmers has attained substantial popularity among decision modelers [19] . Shiny provides customized web interface to R functions and can enable rich, event-driven interactive web application and is more accessible to researchers compared with common web programming languages. However, this platform does not offer programmatic access through standard Application Programming Interfaces (APIs) access to models (unless the user signs up for the costly premium RStudio Connect service) and as such, does not enable large-scale model validation and implementation. There are some other commercial entities that offer programmable APIs, for example, the evidencio (https://www.evidencio.com/home) platform that provides standard web portals and API for some clinical prediction models, but any professional use of this service requires payment.

We believe model developers and users should be able to focus on the aspects of modeling that concern them, not on the specifics of model deployment or access. On the side of model developers, who are often experts in the science of modeling but are not software engineers or programming experts, this means that no additional requirement for making a model available is required. On the side of the end-users, they can now focus on interrogating or using a model, rather than nuances of how each model can be accessed. Such separation of concerns is achieved by providing an accessibility platform that imposes minimal requirements for the developers and a unified interface for clients that hides the technical complexity and dependencies of particular models.

We have taken an early attempt on this front by creating the Programmable Interface for

Statistical & Simulation Models (PRISM). PRISM is a readily available cloud-based web API service that puts R models on the cloud and provide access to them through standardized http calls. As a motivating example, consider the following curl command and its response: curl \ -X POST \ -H "x-prism-auth-user: REPLACE_WITH_API_KEY" \ -H "Content-Type: application/json" \ -d '{"func":["prism_model_run"],"model_input":[{"male":1,"age":70,"smoker":1,"FEV1":2.5,"height":1.68, "weight":65}]}' \ https://prism.peermodelsnetwork.com/route/fev1/run ["{\"Year\":[0,1,2,3,4,5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 

Smoking\",\"Smoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSm oking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"Quit sSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\",\"QuitsSmoking\"]}"]

The above command calls a clinical prediction model of lung function in COPD patients [20] . The server returns the results including projected lung function over the next 15 years under the two scenarios of continuing to smoke or quitting smoking. The user gets such results without having to install R, the R package that implements the model (and its long list of dependencies) and does not have to be worried about computational resources on their computer. Given such http calls are now implemented in standard libraries in many programming environments (e.g., the httpr package for R, or request library for python), access to a model like this can be done in any programming environment and the results be processed (e.g., turned into a graph) as the user demands.

The major design principles underlying PRISM include:

1. Web-API access. Models that sit on the cloud are easily updated by developers and accessed by end-users. Testing, debugging, and package dependencies are all managed on the cloud and behind the scene.

2. Minimal specification. This principle recognizes that model developers should worry about the content of their model and not about how easily it can be accessed on the cloud. Accordingly, PRISM in its core will only require the model to be turned into a standard R package, with a minimal set of functions exposed in a standardized way. Creating comprehensive standards for model accessibility requires coordinated efforts by the broader community.

3. Scalability: the proposed solution should be ready to host models of any level of complexity.

As well, conflicts due to the dependency of the models on additional components (e.g., specific packages) should be avoided by design. The above design principles distinguish PRISM from other means of facilitating model accessibility, reviewed above, and such differences are summarized in Table 1 . remote procedure calls to R [21] . Open CPU all exposed functions of a given R package (i.e., the model) through HTTP calls. Function calls are typically enabled via synchronous HTTP POST requests. The request will return the result of the R function call as well as information about the call in a temporary web address. OpenCPU also automatically captures the graphical output of the R function call and makes it available in the temporary address. OpenCPU provides parallel processing as well as sandboxing (preventing unauthorized access to key system resources by unauthorized code). Open-CPU is by default a stateless synchronous platform, indicating that upon receiving a call request, it will start an R process, process the request, return the results, and terminate the R process. This contrasts with state-based platforms like Shiny that keep the R session open for as long as the user is connected.

Data are transferred between the client and server in JSON (Javascript Object Notation) format. JSON is an industry-standard data coding format that is implemented in many programming languages and data environments, including in R (e.g., jsonlite package) [22] .

OpenCPU by default exposes all functions in a package and makes their source code visible. While this is advantageous for many publicly available models, it cannot per se accommodate privacy and confidentiality requirements. Further, OpenCPU does not provide, by default, any quota on the use of system's resources by a given user (so submitting several demanding requests can effectively shut down the server). The access layer, implemented in the freely available CapRover software [23] , enables user management and logging of information.

User management is performed through API keys. Each API key is unique to each user and enables different levels of access to the model. The API key also enables monitoring and logging of model calls (for example, specifying a quota on CPU usage such that a particular user does not overwhelm the server). API keys are passed to the server via standard http headers. An error code of 0 indicates that the request was submitted successfully. Note that the call address indicates the asynchronous nature of this request.

We deployed an instance of the PRISM server as a part of the 

One of the challenges of working with many models is that each model requires the user to follow a different syntax and workflow in terms of the functions and the specific order in which they need to be called, and the input variables and parameters that need to be provided. The PRISM solution to this problem is a standardization wrapper layer for each model, which enables the user accessing the cloud to follow a standard workflow with a unified syntax when calling different models on the cloud server.

Our guiding principle in designing minimal standardization requirement was such that no more standards are imposed on the modeler.

Before a model developed in R can be hosted on PRISM, the model and the wrapper file should be turned into a standard R package if it is not already in that format [24] . For an example model package, see the source code of the R package for the Acute COPD Exacerbation Prediction Tool at https://github.com/resplab/accept [25] . There are several tools for turning a set of R functions to an R package without the end-user requiring a deep understanding of the internal structure of R packages [24] . For models that have previously been published as an R package, the same package can be used for PRISM, as a generic secondary R package (bridgePrism, https://github.com/resplab/bridgePrism) can interact with the R model.

For the model to be accessilbe, at the minimum, the package should expose one function, prism_model_run(model_input). This function must take model_input as a list, call the model with the appropriate model-specific command and return results as a list.

The function prism_get_default_input() should return the 'default' input set of the model. In general, the standard workflow for programmatic access to models on the peermodels cloud includes the following steps:

1. Retrieving default input structure and values. Depending on how computationally intensive a model might be, the model will return results instantly (i.e. synchronously) or with a delay (asynchronously). The architecture of these two modes of running models is described below.

The R package peermodels provides wrapper functions that simplify access to models on the Peer Models Network cloud. The peermodels package is particularly useful for data wrangling, processing customized web apps, and statistical analysis of the outputs of a model. The case studies in the later sections of this manuscript provide examples of such scenarios. The peermodels package is available on the Comprehensive R Archive Network (CRAN) [26] .

The Evaluation Platform in COPD is a Canadian decision-analytic model that uses discreteevent simulation modelling to model different pathways of care in COPD in order to evaluate policy choices by projecting health and cost outcomes at the societal level [27] . EPIC is currently deployed on the PRISM server and is available for both synchronous and asynchronous calls. EPIC is accessible through the peermodels package in R, as well as an Excel spreadsheet and standard RESTful API calls from other programming languages.

The following code snippet in R retrieves default inputs of this model:

> library(peermodels) > input <-get_default_input(model_name = "epic", api_key = "YOUR_API_KEY") Calling server at https://prism. An asynchronously will generate a token which can be used later to retrieve the results:

> model_run(input = input, model_name = "epic", api_key = "YOUR_API_KEY", async = TRUE) Calling server at https://prism.peermodelsnetwork.com/route/epic/async/run 

projects the rate of all and severe exacerbations of COPD within the next year [25] .

The following code snippet in R can get the default input structure for this model. 

The fields of predictive analytics and decision analysis in healthcare are currently at a pivot point. On one hand, clinical prediction is gaining significant momentum under the purview of the so-called "precision medicine" initiatives [28] . Integrating prediction models with increasingly available patient-level data, be it genetic, -omics, administrative, or through EMR systems and smart devices, is bringing about countless possibilities. On the other hand, the recent increase in the use of models in healthcare decision-making has not been without its challenges [29] . Notably, there has been concerns about scientific rigor and validity of models [30] , potential conflicts of interests on the part of model developers [31] , wasteful repeated attempts to build similar models [9] , ethics of modelling and management of value judgments [6, 7] , and lack of transparency [32] .

Efforts to bolster trust in healthcare models have thus far mainly focused on transparency of the modelling process and the final model, improved reporting, and patient and stakeholder involvement. The peermodels cloud system presented here focuses on making models more accessible, making it easier to examine the model's assumptions and evaluate its adequacy for purpose. Moreover, programmatic access to models can standardize model validation and integration efforts and make them more efficient. To showcase how this is done in practice, we provide two case studies here, one with a decision-analytic health economics model and another with a clinical prediction model.

We kept the specifications for PRISM at a minimum (with the only mandatory function being prism_model_run). This is a manifestation of our 'separation of concerns' principle and recognizes the general desire among modelers to focus their efforts on proper modeling rather than following sophisticated publishing standards. However, we recognize that standardizing model calls can have substantial benefits. For example, a cost-effectiveness model can follow standards on reporting payoffs, net benefits, as well as common standards for setting the time horizon, running a probabilistic analysis at given times, and so on. If these standards are implemented, common libraries can be developed to generate standard outputs form such models. For example, Value of Information (VoI) analyses that work on standard outputs of a probabilistic analysis can then be called, regardless of the specifics of any model. For clinical prediction models, naming conventions such as those specified by Fast Healthcare

Interoperability Resources (FHIR) [33] will facilitate automatic integration of such models into electronic health records, or facilitate automatic external validation, or even recalibration, of such models with the arrival of new data. However, we think specifying such common standards is not a mandate of a model accessibility platform like PRISM and will require broader conversation among all stakeholders in the field. A platform like PRISM can provide a basis for implementation of such standards.

The stateless server technology implemented in PRISM is in contrast with naturally statebased systems such as Shiny. A Shiny app keeps an R process on the server up and running for as long as the user is connected to the application. The benefit of such a solution is the interactive session. For example, the user can run probabilistic analysis and then use the result in multiple instances such as drawing the [cost-effectiveness?] acceptability curve or performing

VoI. The downside of this approach is the scalability of the server if multiple users are connected. In PRISM, the emphasis is on secure stateless server calls after which the R process is terminated. A main advantage of this is the as-required consumption of server resources, as well as the R process starting afresh with a new call thus not being at the risk of becoming corrupted with previous function calls.

Concerns about scientific validity, reproducibility, and management of value judgments in healthcare modelling has led to initiatives that encourage open-source modelling and adherence to reporting guidelines. PRISM cloud infrastructure complements these efforts by providing a standard framework for direct access to healthcare models on the cloud. The PRISM platform enables a wide variety of audiences with different levels of technical expertise to examine models, while at the same time facilitating reproducibility, validation studies, and implementation studies.

YOUR_API_KEY") Calling server at

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Explanation and Elaboration: A Report of the ISPOR CHEERS II Good Practices Task Force

A Call for Open-Source Cost-Effectiveness Analysis

A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling

Center for Evaluation of Value and Risk in Health (CEVR)

Social, ethical, and other value judgments in health economics modelling

Value judgments in a COVID-19 vaccination model: A case study in the need for public involvement in health-oriented modelling

Purposes and duties in scientific modelling

Validation and Utility Testing of Clinical Prediction Models: Time to Change the Approach

Exploring Structural Uncertainty and Impact of Health State Utility Values on Lifetime Outcomes in Diabetes Economic Simulation Models: Findings from the Ninth Mount Hood Diabetes Quality-of-Life Challenge. Med Decis Mak Int

Whole Disease Modeling to Inform Resource Allocation Decisions in Cancer: A Methodological Framework

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

The Clinician and Dataset Shift in Artificial Intelligence

Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic

An Individualized Prediction Model for Long-term Lung Function Trajectory and Risk of COPD in the General Population

You Still Using Excel? The Advantages of Modern Software Tools for Health Technology Assessment

The OncoSim model: development and use for better decision-making in Canadian cancer control

Use of Risk Assessment Tools to Guide Decision-Making in the Primary Prevention of Atherosclerotic Cardiovascular Disease: A Special Report From the American Heart Association and American College of Cardiology

Making health economic models Shiny: A tutorial

Individualized prediction of lung-function decline in chronic obstructive pulmonary disease

The OpenCPU System: Towards a Universal Interface for Scientific Computing through Separation of Concerns. ArXiv14064806 Cs Stat Published Online First: 3

Foundations of JSON Schema

The Acute COPD Exacerbation Prediction Tool (ACCEPT): a modelling study

peermodels: Client-Side R API Wrapper for Peer Models Network Model Repository

Development and Validation of the Evaluation Platform in COPD (EPIC): A Population-Based Outcomes Model of COPD for Canada

A new initiative on precision medicine

Purposes and duties in scientific modelling

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal

Modeling in pharmacoeconomic studies: funding sources and outcomes

Transparency in Decision Modelling: What, Why, Who and How?

SMART on FHIR: a standards-based, interoperable apps platform for electronic health records