key: cord-0710827-ogvvtz9o
authors: Sperl-Hillen, Joann M.; Crain, A. Lauren; Chumba, Lilian; Ekstrom, Heidi L.; Appana, Deepika; Kopski, Kristen M.; Wetmore, James B.; Wheeler, James; Ishani, Areef; O'Connor, Patrick J.
title: Pragmatic clinic randomized trial to improve chronic Kidney disease care: Design and adaptation due to COVID disruptions
date: 2021-07-13
journal: Contemp Clin Trials
DOI: 10.1016/j.cct.2021.106501
sha: a4398860d6dd2dad7862466c58c3305eb2a91ead
doc_id: 710827
cord_uid: ogvvtz9o

Background We describe a clinic-randomized trial to improve chronic kidney disease (CKD) care through a CKD-clinical decision support (CKD-CDS) intervention in primary care clinics and the challenges we encountered due to COVID-19 care disruption. Methods/design Primary care clinics (N = 32) were randomized to usual care (UC) or to CKD-CDS. Between April 17, 2019 and March 14, 2020, more than 7000 patients had accrued for analysis by meeting study-eligibility criteria at an index office visit: age 18–75, laboratory criteria for stage 3 or 4 CKD (eGFR 15–59 mL/min/1.73 m2), and one or more opportunities algorithmically identified to improve CKD care such as blood pressure (BP) or glucose control, angiotensin converting enzyme inhibitor (ACEI) or angiotensin receptor blocker (ARB) use, discontinuance of a nonsteroidal anti-inflammatory drug (NSAID), or nephrology referral. At CKD-CDS clinics, CDS provided individualized treatment suggestions that were printed for patients and clinicians at the start of office encounters and were viewable within the electronic health record. By initial design, the impact of the CKD-CDS intervention on care gaps was to be assessed 12 months after the index date, but COVID-19 caused major disruptions to care delivery during the intervention period. In response to disruptions, the intervention was temporarily suspended while we expanded CDS use for telehealth encounters and programmed new criteria for displaying the CKD-CDS to intervention patients due to clinic closures and scheduling changes. Discussion We describe a NIH-funded pragmatic trial of web-based EHR-integrated CKD-CDS and modifications necessary mid-study to complete the study as intended in the face of COVID-19 pandemic challenges.

by eGFR values do not have an diagnostic code identified for it in the electronic record. [3] Furthermore, national CKD guidelines recommend blood pressure control, use of angiotensin converting enzyme inhibitors (ACEIs) or angiotensin receptor blockers (ARBs) in adults with albuminuria, optimal glucose control in patients with diabetes, and avoidance of nonsteroidal ant-inflammatory agents (NSAIDs) to slow progression of renal disease and reduce the occurrence of major cardiovascular events. [4] [5] [6] Even when laboratory data is present to substantiate a CKD diagnosis, primary care clinicians often do not recognize or diagnose it, and, as a result, these care goals are often not achieved. Patients with stage 3b (eGFR 30-44 mL/min/1.73 m 2) or stage 4 CKD (eGFR 15-29 mL/min/1.73 m 2 ) often are not referred to nephrology early enough to have meaningful discussions about advanced CKD care and renal replacement options such as dialysis or renal transplantation. [7] Preliminary data collected for this study as preparation for research (unpublished) from 17 primary care clinics in Minnesota in 2015-2016 showed that of 5766 patients aged 18-75 years with confirmed eGFR 15-59 mL/min/1.73 m 2 , 51% did not have a CKD diagnosis, 63% had a blood pressure (BP) above 130/80 mm Hg, 50% of those with diabetes had glycated hemoglobin (A1C) ≥7%, and merely one in four had a test for albuminuria within the last year.

Of those with hypertension or albuminuria (and without hyperkalemia), barely more than half (57%) had been prescribed an ACEI or ARB. Of patients with more advanced CKD (eGFR 15-29 mL/min/1.73 m 2 , or eGFR 30-44 mL/min/1.73 m 2 with albuminuria), less than half (45%) had seen a nephrologist within the last 2 years.

The COVID-19 pandemic was a major unanticipated challenge requiring us to make substantial overhauls to the intervention delivery mechanism that required rapid regulatory and funder approval to implement. This manuscript describes the original study, the challenges that emerged, and some of the advantages and limitations of the adaptations that were implemented to respond to COVID related care delivery disruptions.

The overall consort flow of the study showing enrollment, randomization, allocation, follow up and analysis is shown in Figure 1 . Further details about the study design are described below. Clinic Randomization: Clinics were randomized 1:1 to intervention or UC using covariate constrained randomization (CCR). [24] The SAS-based CCR macro [25] identified a pool of randomization schemes that would balance five selected clinic covariates across the intervention and UC arms of the study. CCR then randomly selects one randomization scheme J o u r n a l P r e -p r o o f for the study randomization. The clinic balancing covariates were calculated in February, 2019: 1) care delivery system (PN or HP), 2) clinic eligibility for another cluster-randomized trial that started in the month prior to CKD-CDS randomization 3) number of CKD patients, 4) proportion of CKD patients with Medicaid health insurance coverage, and 5) proportion of hypertension patients meeting the care group's performance measure for achieving the blood pressure goal of < 140/90 mm Hg. CKD-CDS software programming, software training and intervention delivery all required that the investigative team, clinic leaders and clinicians not be blinded to clinic allocation. Patients were assigned to the intervention or UC control group based on the assignment of the clinic at which their index visit took place.

Patient Accrual: Patients were accrued into the study if they met the inclusion and exclusion criteria outlined below at an index visit. Automated calculations to assess these criteria were performed by the CDS tool at the beginning of each primary care encounter in randomized clinics over the course of the accrual period. Risk is highly mitigated in several ways: (a) all the data servers reside within the HealthPartners firewall (b) Instead of exchanging the medical record number, an artificially generated Study ID was assigned at the initial visit and used as a token to link the patient in the CDS application with the one in the EMR (c) the patient identifiable information that was initially exchanged included name, date of visit, and dates of labs. Once the operational need to display the patient name on the CDS was fulfilled, the name was scrubbed from the system. Also, a scrubbed limited dataset needed for analysis was removed from the transactional system in order to mitigate risk of data exposure and limit data access to only programmers on the research team.

The final system architecture passed security checks and penetration testing before deployment. Furthermore, the layers of the infrastructure are regularly tested for vulnerabilities by our internal security team and also scanned annually by external security firms.

CKD-CDS Pilot: The CKD-CDS was first turned on for a nephrology department and specialty feedback was obtained on the CKD-CDS content for accuracy and clarity. Nephrologists were trained to use the CKD-CDS and how to provide feedback to the research team either electronically through email or a "suggestions tab" on the CDS display, or to write feedback directly on the patient's printed CDS forms and send it back to the research team via internal mail routing.

Prior and during pilot implementation of the CDS, we conducted clinician interviews to gain their perception of the usefulness of Priority Wizard and CKD content. The information learned from these processes were used to further modify and improve the tool.

CDS Data Flow: Figure 2 describes how data related to the CKD-CDS intervention flowed.

Hosting the CDS on a secure web service securely linked with the EHR allowed for maximum efficiency and versatility. Maintaining the CKD-CDS on a single web service allowed the study team to make necessary updates to risk equations and algorithms relatively easily. It also avoided potentially disruptive changes to the CDS with Epic upgrades and allowed the CDS to J o u r n a l P r e -p r o o f run without risk of slowing down the EHR production environment. The average web service run time was less than one second. Most importantly, the web service algorithms not only drove the intervention, but also assured complete collection of necessary data for analysis at both intervention and control clinics. This greatly reduced the data collection burden and increased the accuracy of collecting data needed for Data and Safety Monitoring of the study, and for assessment of intervention effectiveness at the end of the study.

The CKD-CDS content was incorporated into a pre-existing CDS tool used for diabetes and CV risk. When the CKD content was added, the nonproprietary CDS tool was redesigned and rebranded from the name CV Wizard © to Priority Wizard © . The clinic workflow for CDS use was developed previously by our research team in conjunction with clinic leaders in previous successful studies [16, 17, 19, 26] and subsequently adopted and maintained by the care system for diabetes and CV risk factor management. For patients who presented for an appointment at any randomized primary care clinic (intervention or control), when a blood pressure was entered into the electronic health record (EHR) and the vitals section closed, selected clinical data were sent from the EHR to the CDS website to assess eligibility for the Priority Wizard © intervention.

CKD-CDS description: The CKD-CDS components included four major features as shown in For the subset of study-eligible CKD patients with sufficient clinical data at the index visit the CDS also estimated and displayed the 5 year probability of kidney failure (dialysis or transplantation) using data on age, sex, eGFR, and albuminuria levels and a published equation (Tangri) validated in more than 700,000 individuals worldwide [27, 28] and used in many online medical calculators.

For any eligible patient encounter, the CDS was viewable by clinicians within the EHR and also printable in two companion versions, one designed for patients that displayed "Kidney Health" as a priority ( Figure 4 ) and one designed for clinicians that displayed "Chronic Kidney Disease" as a priority ( Figure 5 ). for the clinician to review. While there was some resistance to printing, we justified printing the tools by rooming staff for the following reasons: 1) In a clinician survey, 90% of respondents wanted the CDS printed, 70% said they reviewed the printed materials with patients frequently or always, and 90% said that their patients like the information on the handouts; 2) In a pilot that did not prompt rooming staff to print materials for targeted patients with high risk situations, the CDS was triggered for less than 20% of targeted patients, 3) in an online patient questionnaire called MY VOICE used to obtain patient feedback on the prototype CDS patient interface, the most highly preferred time to receive the CDS information was during the rooming process while waiting for the clinician (41%) compared to electronically through the EHR patient portal before the appointment (20%) or after the appointment (20%). With respect to major CV risk factors, J o u r n a l P r e -p r o o f 63% of patients said they were likely to take action on a risk factor because of the handout, 58%

said it is very convincing to get them to stop smoking, and 55% to improve their cholesterol.

The CDS was also viewable by clinicians online within the EHR, and a smart dot phrase (.ckdrisk) was developed to facilitate clinician documentation of the CKD status within EHR notes and to summarize CKD-related information in the patient's printed after visit summary.

An advantage to this CDS system is that the CDS could be easily updated/refreshed by rooming staff or clinicians if new results became available (such as a repeat BP or a point of care A1C).

In control clinics, the same data flow and workflow was used to generate the Priority Wizard © , but it contained no CKD-CDS information and the BPA to rooming staff was not displayed for patients meeting the CKD intervention eligibility criteria alone.

Data Sources: The CDS web service collected data elements documented in the electronic health record during the five-year periods prior to each web service call in both intervention and control clinics. It also retained analytic data elements that originated in the EHR and outputs from web service algorithms on a secure server, including a randomly-generated patient J o u r n a l P r e -p r o o f

Fidelity to the Intervention: Use of the CKD-CDS intervention was estimated by tracking CDS print rates at the system, clinic, and clinician levels. The clinics were given the goal to print the CDS at 75% of encounters targeted with a BPA. Monthly print rate reports were provided to clinic leadership comparing print rates across clinics listed in the order of highest to lowest print rate for the current month, and by displaying the print rates of each clinic for the past 12 months in a grid that highlighted print rate levels in green (meeting goal), yellow (close to meeting goal) and red (not close to meeting goal). Each clinic leader was also given a clinician-level use rates report so that they could troubleshoot low use rates at the clinician level. In addition to distribution of monthly print rate information, study staff periodically contacted the lead nurse at clinics with low print rates to inquire if assistance in improving rates was desired. Use of Priority Wizard © was not monetarily incentivized. Figure 6 Table 1 . *care gaps for the analysis denominators are the same as those defined in the inclusion criteria Analysis Plan: Study hypotheses posit that the CKD-CDS intervention will reduce deficits in key elements of CKD care relative to UC. The planned analysis for each outcome accommodates the covariate constrained cluster randomization and its clinic level intraclass correlation coefficient (ICC) estimated from pilot data. Data elements required for calculating the outcomes are extracted from EHR production tables. The absence of documentation of a care process, vital sign, or medication will not be interpreted as a missing value but rather as indicative of a care process or test not having been performed or medication not prescribed within the health system. Truly missing observations (e.g., systolic BP measured, value not available) will be extremely rare, undetectable and assumed to be missing at random (MAR).

The primary study hypotheses will be tested using random coefficients models in which each outcome will be predicted using distribution-appropriate link functions from clinic fixed effects for treatment group and the balancing factors; patient fixed effects for pre-randomization characteristics (e.g., outcome status); and random effects to accommodate clustering (e.g., clinics, patients within clinics), as needed. A priori power analyses estimated the minimum detectable difference in each of the 5 outcomes listed in Table 1 based on the patient sample size, event rate and clinic and patient level ICCs calculated from pilot data. Absolute betweengroups differences of 10% (BP control, ACEI or ARB use, glucose control) and 20% (recognition of CKD, Nephrology referral) were determined to be clinically meaningful. The study is adequately powered to detect a clinically meaningful difference for all major study hypotheses at 80% with two-tailed alpha of 0.05.

The COVID-19 pandemic emerged as a major disruptive force in primary care clinics starting in

March 2020 in the middle of this 3-year study, about one month prior to the end of the planned 12-month accrual period. The most immediate impact of the pandemic was a dramatic decrease in office-based clinical encounters, which, by design, were the point of delivery for the CKD-CDS intervention. At the clinics involved in this study, virtual clinical encounters by video or telephone suddenly became the recommended mode for all appointments other than those that required in-person evaluation for diagnosis or treatment. These changes required some primary care clinicians and many patients to relocate primary care services to new locations or seek care through virtual visits that were not necessarily scheduled through their home clinic. An additional challenge was that the CKD-CDS intervention was not originally programmed for use at video visits. This "perfect storm" of COVID-19-related disruptions had the potential to undermine the intervention and the integrity of this clinic randomized study design.

Based on preliminary data and power analysis, we aimed to accrue about 6100 study-eligible adults with stage 3 or 4 CKD (eGFR 15-59 mL/min/1.73 m 2 ) over a 12-month accrual period.

Near the start of the pandemic disruptions, we had already accrued more than 7000 patients.

Due to the dramatic reduction in the number of possible index visits, potential selection effects of patients attending in-person office visits and inability at the time to deliver the intervention outside of in-person office visits, the research team decided to end accrual of new patients on March 13, 2021 . From the start of the pandemic, video visits could be scheduled either at a specific clinic, or within a pool of clinicians drawn from all clinics. Visits with a clinician pool that was drawn from both intervention and UC clinics created potential contamination of care delivered to patients whose index visit had been either in an intervention or UC clinic. This problem was aggravated a few weeks later, when several randomized primary care clinics were closed on short notice, and several more randomized clinics were temporarily re-allocated to J o u r n a l P r e -p r o o f provide care only to patients with respiratory symptoms. Due to disruption of the nested patientprovider-clinic structure, the study team temporarily suspended the CKD-CDS intervention on March 23, 2021. During the suspension, which lasted until August 19, 2020, no CKD-CDS content was presented, automatically or on demand, at any type of clinical encounter.

During the suspension, several intervention modifications were made. The determination of whether to display the CDS at encounters was initially determined by the clinic location of the office visit (clinic intervention assignment). Due to concern about feasibility of maintaining the intended clinic level randomization process going forward due to video visit locations and clinic closures, the study team modified the criteria that determines whether CKD-CDS can be displayed at a CKD-eligible visit from using the clinic location of the current visit (i.e., intervention or control clinic location) to the clinic location of the index visit. The goal of this modification was to preserve the integrity of the intervention status assigned to each patient at their index visit regardless of the clinic location, clinician, or encounter modality (telehealth vs. in person clinical encounter) at their subsequent clinical encounters. Thus a patient randomized to usual care at index would never receive the CDS if they were to be seen by a clinician at an intervention randomized clinic and a patient randomized to intervention at index would continue to receive the CKD-CDS if seen at a clinic randomized to usual care. This modification was designed to adhere to intent to treat principles as much as possible and preserved the initial randomization structure to a greater extent than other options that were considered. During the suspension period, the web-service was also reprogrammed to add an Epic trigger for a web service call at the opening of a video encounter so that the CDS could be viewed in video visits.

Eligible patients had a BPA display in the clinician's BPA section recommending use of the CDS. Because rooming staff were not involved in a rooming process for telehealth encounters and printing would not be helpful, there was no pop-up BPA programmed. Because the software our care system used for video visits did not include a feature for screen sharing, there was not J o u r n a l P r e -p r o o f a practical way to share the CKD-CDS with patients other than verbally. The rates of telehealth encounters and clinician rates of opening the CDS during telehealth encounters will be monitored closely as low use rates in telehealth encounters could impact study outcomes.

These study modifications required approval of the NIH, DSMB, IRB, and care delivery leadership. After a 5-month suspension, the intervention was restarted at eligible clinical encounters and available for use at office and telehealth encounters for intervention assigned patients. Because of the intervention suspension, the planned follow up time for analysis was extended from 12 to 18 months. Because the grant permitted a no cost extension, the study timeline and budget were adapted accordingly.

By early October 2020, all but one of the closed primary care clinics had reopened and all clinics had returned to blended operation seeing both routine office visits and patients with respiratory symptoms. One randomized intervention clinic was permanently closed. Telehealth Informed Consent: A waiver of patient and provider consent for this study was requested and granted by the HealthPartners Internal Review Board (IRB).

This cluster-randomized pragmatic trial conducted at 32 primary care clinics is designed to improve CKD care management and outcomes for patients with stage 3 or 4 CKD (eGFR 15-59 mL/min/1.73 m 2 ). The CKD-CDS identifies patients with stage 3 or 4 CKD and one or more selected care gaps immediately before the start of a primary care clinical encounter and creates an opportunity for timely intervention to recognize and improve CKD care. The high frequency of care gaps we identified through preliminary data underscores the urgent need to improve CKD care.

Effectiveness of the CDS system will be assessed using an intent-to-treat analysis, and outcomes will be assessed through a combination of EHR data and data harvested by the CDS tool itself. Thus, the CDS system acts as both a means of intervention delivery and a means of data collection. Data needed for analysis, including both typical EHR data and CKD-CDS algorithm output for all patients are securely stored as a limited data set on the firewall protected CDS web server and linked for each study patient using a unique random study ID number assigned electronically by the CDS at the index visit. This repository of data can also be used to 

The waiver of patient consent granted by the IRB was critical, as the office-based intervention would not have been feasible if formal consent was required for large numbers of study eligible patients in the primary care delivery setting, particularly at UC clinics.

A co-variate constrained randomization procedure proved to be a practical method to randomize clinics and created acceptable balance on patient characteristics, insurance status, and a co-existing research initiative related to medication adherence implemented during the CKD study intervention period.

Having dedicated web and Epic programmers supported by research funding was key to making timely programming changes required when intervention adaptations were necessary. In this study our research programmers were able to quickly program the intervention to work in telehealth encounters and to change criteria for displaying CDS from the clinic level to the patient level, even when most other organizational programming resources were assigned to a variety of critical pandemic-related tasks. Reliance on stressed information J o u r n a l P r e -p r o o f technology resources of the care delivery system could have resulted in delays that would likely have made study completion impossible within the study timeline and budget.

Lesson 5: Pragmatic trials have practical limitations that often need to be recognized and accepted. One limitation of this study is that approximately 25% of CKD-CDS eligible patients in usual care clinics also received CDS for cardiovascular or diabetes risks due to system-wide CDS installations that pre-dated the CKD-CDS study. Identification of these patients within the data repository will allow us to perform secondary sensitivity analysis to evaluate the magnitude of this effect.

Study design and analysis adaptations that solve one problem may introduce other problems. While study eligible patients will not cross over from intervention to control or vice versa, the modifications to the intervention and analysis made to address pandemic-related disruptions could potentially lead to a potential learning contamination effect of UC clinicians ( learning contamination could occur if a patient originally assigned to CKD-CDS intervention has a post-index encounter with a UC clinician). This limitation was discussed at length by the study team, project officer, and DSMB with acknowledgement that a beneficial intervention effect might be harder to detect as statistically significant, but a positive result of the study would still be very meaningful. We will be able to quantify the amount of patient crossover to UC clinician encounters and could evaluate the extent of a learning effect on UC clinicians by comparing changes in CKD outcomes in non-study clinics. If a learning effect could be identified, it could actually enhance the value of the CKD-CDS intervention.

The change in deployment of the intervention to accommodate telehealth visits and clinic closings could make it more challenging to assess CKD-CDS intervention fidelity.

Extending the use of the CKD-CDS intervention to new types of clinical encounters and other adaptations to the study design will increase variation in how the CDS tools are actually used, especially for purposes of shared decision making. There was a steep learning curve when J o u r n a l P r e -p r o o f clinicians were required to rapidly pivot to new video technology. However, gathering qualitative and quantitative data on how clinicians use the tool and how much they value it will provide important new information relevant to future CDS design, and will have a bearing on how to interpret the main study findings. Although the study team was able to devise technical solutions to challenges posed by COVID-19, the impact of COVID-19 on patient behavior and receipt of health services through telehealth encounters could impact the potential for the intervention to improve outcomes in a time of greatly increased stress for both clinicians and patients.

Almost 30 years ago heath care leaders recognized the potential of EHR-linked CDS to improve chronic disease and preventive care. [29] Despite profound advances in health information technology and the wide use of sophisticated EHRs through most of the U.S. health care system, that potential to improve chronic disease care has yet to be fully realized. [30, 31] In particular, there have been few successful efforts to systematically improve CKD care in primary care settings, despite persistent observed deficits in care. [1, 2, [9] [10] [11] Here we report our ongoing efforts to use EHR-linked web-based CDS interventions to improve CKD care, and provide examples of some of the challenges that need to be overcome to achieve this goal. In particular, we describe how disruptions to the delivery of primary care during the COVID-19

pandemic threatened the intervention delivery, thus giving rise to innovations and adaptations,

including new ability to deliver CKD-CDS at virtual clinical encounters that occur outside primary care clinic walls. However, the development of some of these innovations, and the very positive response of primary care clinicians and patients to these innovations will inform future efforts to improve the design, delivery, and effectiveness of CDS interventions that aspire to improve chronic disease care in primary care settings. Importantly, the study modifications made in the face of COVID-19 pandemic adversities adhered to intent to treat principles by preserving the initial randomization structure to a greater extent than other solutions to the challenges that J o u r n a l P r e -p r o o f Journal Pre-proof were considered. The most important lesson we learned from the challenges we experienced was the importance of having good relationships, respect, and collaboration between the research project staff, care delivery system leaders, and IT departments to implement timely solutions.

Trial Status: The trial was funded through the National Institute of Diabetes and Digestive and Kidney Diseases (R18DK118463). The CKD-CDS intervention will become available throughout the system at the request of the care delivery systems at the conclusion of the study. Analyses is 5-6 months delayed due to the COVID-related temporary intervention suspension, and trial results are expected in late 2021.

J o u r n a l P r e -p r o o f

Centers for Disease Control and Prevention (CDC)

Chronic Kidney Disease (CKD) Surveillance System

Chronic Kidney Disease (CKD)

Surveillance System

Kidney Disease: Improving Global Outcomes (KDIGO) Diabetes Work Group

Clinical Practice Guideline for Diabetes Management in Chronic Kidney Disease

KDOQI Clinical Practice Guideline for Diabetes and CKD: 2012 Update

Executive summary of the 2020 KDIGO Diabetes Management in CKD Guideline: evidence-based advances in monitoring and treatment

Adherence to chronic kidney disease guidelines in primary care patients is associated with comorbidity

Use of clinical decision support systems for kidney-related drug prescribing: a systematic review

Electronic Decision Support for Management of CKD in Primary Care: A Pragmatic Randomized Trial

Impact of a Primary Care CKD Registry in a US Public Safety-Net Health Care Delivery System: A Pragmatic Randomized Trial

Effect of 2 Clinical Decision Support Strategies on Chronic Kidney Disease Outcomes in Primary Care: A Cluster Randomized Trial

Improving outcomes with clinical decision support an implementer's guide

Enabling health care decisionmaking through clinical decision support and knowledge management

Effect of clinical decision-support systems: a systematic review

Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success

Impact of Electronic Health Record Clinical Decision Support on Diabetes Care: A Randomized Trial

Clinical decision support directed to primary care patients and providers reduces cardiovascular risk: a randomized trial

Outpatient EHR-based diabetes clinical decision support that works: lessons learned from implementing Diabetes Wizard

Clinical Decision Support for Recognition and Management of Hypertension: A Randomized Trial

Cost-Effectiveness of an Electronic Medical Record Based Clinical Decision Support System

Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy

Harms, benefits, and the nature of interventions in pragmatic clinical trials

What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation

Covariate-based constrained randomization of group-randomized trials

A SAS Macro for Covariate-Constrained Randomization of General Cluster-Randomized and Unstratified Designs

Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review

A predictive model for progression of chronic kidney disease to kidney failure

Multinational Assessment of Accuracy of Equations for Predicting Risk of Kidney Failure: A Meta-analysis

The computer-based patient record. An essential technology for health care

Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures

Clinical decision support in electronic prescribing: recommendations and an action plan: report of the joint clinical decision support workgroup