2023 Real World Testing Results
This page contains a list of the steps taken to conduct the annual Real World Testing requirements for ONC certification for the 2023 reporting year. The Results within this were reviewed as Screenshots and spreadsheets for their compliance with the criteria defined in the test plan. These artifacts will be maintained by the health IT developer for audit purposes or further requests.
General Information #
Developer Name: Canvas Medical, Inc
Product Name(s): Canvas
Version Number(s): 1
Certified Health IT: 2015 Cures Update
CHPL Product Number: 15.04.04.3112.Canv.01.00.1.220523
Developer Real World Testing URL: https://www.canvasmedical.com/compliance/onc/mandatory-disclosures
Changes to Original Plan #
| Summary of Change | Reason | Impact |
|---|---|---|
| 70.315 (g)(7): Application Access - Patient Selection: We intended to report on the number of patient-restricted tokens with a reduced set of scope. We instead tested the functionality through quarterly interactive live testing using the Inferno Tests. | Our quarterly reports showed no to little use of this functionality. When reviewing the results, the numbers were questioned due to specific knowledge of a customer’s recent adoption. It was discovered that the reports would only reflect 10 hours of tokens, as they were being deleted after 10 hours due to our security protocols. | The functionality was tested live and was also confirmed to be working based on customer feedback. Our team is implementing logging to have better reporting of this feature in the future. |
| 170.315 (e)(1): View, Download, and Transmit to 3rd Party (Cures Update): We combined the metric to group viewed and downloaded. We also conducted live testing to confirm both options were available to patients. | Our reporting could not differentiate between the two actions taken by patients. | Our real world metrics do not necessarily confirm that both interactions are working as intended so we added live interactive testing to confirm. |
Withdrawn Products #
We have not withdrawn any products within the past year that we previously included in the Real World Testing plan.
Summary and Key Findings #
Canvas Medical v1 is a certified electronic health record (ehr) that is sold to ambulatory primary care clinics and tech-enabled multispecialty provider organizations operating in hybrid settings (telehealth and in-person).
Functionality within Canvas is shared across all supported care settings and so our Real World Testing plan was designed to incorporate data from our entire customer base.
Our test plan focused on capturing and documenting the number of instances that each certified capability (listed below) is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability using interactive testing methods in a semi-controlled setting as close to a “real world” implementation as possible.
Adoption of certified functionality has been low within our customer base. Many of our customers are either direct-to-consumer or focus on employer or commercial contracts.
We did run across some issues during live testing with the functionality that is not widely used by customers. We make fast and frequent changes to our platform, and in doing so, unintentionally altered some of the lesser used and therefore understood features. Post testing - all non-conformities were reported within 30 days and quickly corrected.
We also discovered some deficits in our reporting capabilities. We set out to capture each measure at the most granular level. Additional logging is needed to do that in some areas. That work has been started in order to better support the 2024 reporting year.
Standards Updates (SVAP and USCDI) #
For CY 2023, Canvas did not make any version updates on approved standards through the SVAP process. This applies to all test scenarios described within.
Care Settings #
Canvas markets to provider organizations that operate in hybrid settings. Most customers deliver care to patients using a combination of of the following:
- In-person, office-based visits
- Synchronous telehealth visits
- Asynchronous digital consultations
- In-Facility or in-home visits
- Field-based visits
Metrics and Outcomes #
170.315 (b)(1): Transitions of Care (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Create a Valid CCDA INTERACTIVE: 1) A CCDA of each type (Referral Note and CCD) will be created in Canvas and uploaded to another EHR. A user in the receiving EHR will demonstrate a triumphant display of all required elements. | Expected Outcome: Success is when a different EHR receives and recognizes each type of CCDA as conformant. | Q1: Pass Q2: Fail Q3: Pass Q4: Pass | The CCD created in Q2 failed to load into athenanet. The ETT identified several non-conformities that had been introduced and were reported within 30 days. We entered a CAP to make the necessary corrections |
| Create and send a CCDA 1) Number of CCDAs sent via edge protocols | Expected Outcome: Success will be measured through volume. As our customers implement this tool, usage will demonstrate that the capability is available and effective. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | none |
| Receive and display a CCDA 1) Number of CCDAs received via edge protocols | Expected Outcome: Success will be measured through volume. As our customers implement this tool, usage will demonstrate that the capability is available and effective. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 11 | none |
170.315 (b)(2): Clinical Information Reconciliation and Incorporation (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Receive and reconcile a CCDA 1) Percentage of CCDAs imported successfully 2) Number of CCDA Import Notes Created 3) Number of CCDA Import Notes Reconciled and Locked | Expected Outcome: Success will be measured through volume. Our expectation is there will be low utilization by providers with a high success rate (with expected errors). | 1) Q1: 75% (3/4) Q2: 0% (0/2) Q3: 0% 0/9 Q4: 40% 2/5 2) Q1: 3 Q2: 0 Q3: 0 Q4 2 3) Q1: 0 Q2: 0 Q3: 0 Q4: 0 | The small sample size is mainly reflective of some testing by customers. The high error rates were evaluated and the errors were expected and due to file non-conformance or the inability to match patients. |
170.315 (b)(3): Electronic Prescribing (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Enable a user to perform the following (...) prescription-related electronic transactions 1) Number of and success rates for: NewRx RxChange CancelRx RxRenewal RxFill Medication History | Justification: Evaluating a statistically significant sample size of electronic prescriptions spanning multiple organizations using Canvas will demonstrate the real-world utility of the feature. Expected Outcome: Transactions are successfully delivered with standard errors (e.g., pharmacy does not support electronic transactions). Data validation errors are prevented, or the end-user is notified of the errors when applicable. | Pass NewRx Q1: 32423 99.12% Q2: 26949 99.25% Q3: 33140 98.77% Q4: 34211 98.34% RxChange n/a CancelRx n/a RxRenewal Q1: 24635 98.72% Q2: 26991 98.23% Q3: 28213 98.38% Q4: 30484 98.61% RxFill Q1: 14535 100% Q2: 23472 100% Q3: 227696 100% Q4: 510027 100% Medication History Q1: n/a Q2: n/a Q3: n/a Q4: 191676 100% | Our customers did not opt in or adopt some of the workflows to facilitate certain transaction types Users received common validation warnings/errors from the application as expected for missing data requirements/etc. |
170.315 (c)(1): Clinical Quality Measures - Record and Export #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Record & Export 1) Number of measures recorded during the period 2) Number of QRDA Category 1 files exported | Expected Outcome: Success will be measured through volume. We expect moderate utilization of our eCQMs with high success rates but limited utilization of the export functionality. | Pass 1) Q1: 0 Q2: 0 Q3: 0 Q4: 0 2) Q1: 0 Q2: 0 Q3: 0 Q4: 0 | No adoption |
170.315 (c)(2): Clinical Quality Measures - Import and Calculate #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Import & Calculate every CQM (clinical quality measure) 1) Number of QRDA Category 1 files imported (if applicable) | Expected Outcome: Success will be measured through volume. We expect limited utilization of the import functionality. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | No adoption |
170.315 (c)(3): Clinical Quality Measures - Report (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Enable a user to create a data file for transmission electronically 1) Number of QRDA Category 3 aggregate report(s) created over the period | Expected Outcome: Success will be measured through volume. We expect limited utilization of the reporting functionality. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | No adoption |
170.315 (e)(1): View, Download, and Transmit to 3rd Party (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Preview CCD or Download CCD 1) System logs will be evaluated to identify patients with a successful CCD document preview or download in the patient app. | Justification: Many of our customers have developed their own patient-facing applications. For those using the Canvas patient web app, usage will demonstrate the ability of a patient to download a CCD document template in the live production environment of their patient app. Expected Outcome: Success is defined by the number of patients with successful CCD document previews or downloads. We expect usage to be limited. | Pass Q1: 14 Q2: 14 Q3: 5 Q4: 14 | Our current reporting does not differentiate between a patient viewing and downloading their data. We are adding additional logging to track. |
| Transmit CCD 1) System logs will be evaluated to identify patients with a successful CCD document transmission in the patient app. | Justification: This measure will demonstrate the ability of a patient to transmit a CCD document template in the live production environment of their patient app. The ability to send via Direct is tied to the implementation of DataMotion as a relied upon software. Expected Outcome: CCD documents were successfully sent via email and Direct with the expected errors (e.g., invalid direct address, lack of response, etc.). We expect usage to be limited as many of our customers have developed their own patient-facing applications. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | Adoption of DataMotion was limited due to customers not leveraging our patient experience. |
170.315 (f)(1): Transmission to Immunization Registries #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Record immunizations and generate the HL7 v2.5.1 Z22 VXU immunization information messages 1) Percentage of immunization records submitted to immunization registries | Justification: We intend to record the frequency that immunization data is submitted to registries to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Expected Outcome: We expect limited volume with a high rate of success. | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | No adoption |
170.315 (f)(5): Transmission to Public Health Agencies - Electronic Case Reporting (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Consume and Maintain Table of Reportable Condition Trigger CodesCreate a case report for the patient encounter(s) based on a matched trigger INTERACTIVE 1) Provider will document a condition from the trigger code table in a test encounter | Justification: While Canvas provides the capability for electronic case reporting, there has been zero adoption to date. Therefore, we plan to demonstrate real-world performance through interactive testing. Expected Outcome: Success is when a case report is generated for the patient encounter(s) based on a matched trigger. | Q1: Fail Q2: Pass Q3: Pass Q4: Pass | A nonconformity was discovered during Q1 testing and reported within 30 days. It was resolved under a CAP. |
170.315 (g)(7): Application Access - Patient Selection #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Receive a request with sufficient information to uniquely identify a patient and return an ID or other token that can be used by an application to subsequently execute requests for that patient's data. INTERACTIVE 1) Run the inferno tests on the production API service with test patient data. | Justification: The evaluation of a statistically significant sample size of API requests spanning a broad spectrum of API Information Sources will demonstrate the real-world utility of the API Expected Outcome: Success is defined by passing the Inferno test using the production API and verifying that the documentation is complete | Q1: Pass Q2: Pass Q3: Pass Q4: Pass | We ran into reporting limitations and therefore leveraged our inferno test results to demonstrate the capability |
170.315 (g)(9): Application Access - All Data Request (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Respond to requests for patient data for all of the data categories specified in the Common Clinical Data Set. 1) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token | Justification: The evaluation of a statistically significant sample size of API requests spanning a broad spectrum of API Information Sources will demonstrate the real-world utility of the API Expected Outcome: Our expectation is there will be low utilization by external applications with a high success rate (including expected errors that could include failure in authorization/authentication, incorrectly formatted request, etc.) | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 961 | None |
| Respond to requests for patient data associated with a specific date as well as requests for patient data within a specified date range. 1) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token for a specific date range | Justification: The evaluation of a statistically significant sample size of API requests spanning a broad spectrum of API Information Sources will demonstrate the real-world utility of the API Expected Outcome: Our expectation is there will be low utilization by external applications with a high success rate (including expected errors that could include failure in authorization/authentication, incorrectly formatted request, etc.) | Pass Q1: 0 Q2: 0 Q3: 0 Q4: 0 | None |
170.315 (g)(10): Standardized API for Patient and Population Services (Cures Update) #
| Requirement(s) | Justification & Expected Outcome | Results | Challenges Encountered |
|---|---|---|---|
| Standardized API for Patient and Population Services INTERACTIVE 1) Run the inferno tests on the production API service with test patient data.2) Review of the documentation for required elements | Justification: The standardized testing tool reflects a wider variety of features that our customers employ and is therefore the most thorough method of showing real-world capabilities. Expected Outcome: Success is defined by passing all sections of the Inferno test using the production API and verifying that the documentation is complete | Q1: Fail Q2: Pass Q3: Pass Q4: Pass | An issue was discovered during Q1 testing and reported within 30 days. No non-conformity was found. |
Key Milestones #
| Requirement(s) | Justification & Expected Outcome |
| Finalization of the Real World Testing plan | December 2022 |
| Development of candidate list of providers to assist with interactive Real World Testing | December 2022 |
| Data collection and interactive testing | 2023 - Quarterly |
| Validation of expected outcomes | 2023 - Quarterly |
| Analysis and report creation | January 2023 |
| Submit Real World Testing Report to ACB | February 2024 |