ONC Self-Declaration Testing: The Right, The Wrong, and The Overlooked
A few weeks ago ONC announced a major change in their testing and certification process. They introduced “self-declaration” as a method for ONC-ATLs to accept test results for over half of the criteria in 2015 Edition, and ONC announced a scaling back on their focus on randomized surveillance. It has generated significant conversation, including notable criticism by some. My thoughts on what they got right, what they got wrong, and what people are overlooking.
What They Got Right Overall, I like this approach, and I applaud ONC in the general direction of both efforts. Self-declaration means developers do not demonstrate their functionality before on ONC-ATL through inspection testing in a live event, but instead, they declare its compliance and provide a paperwork attestation to this fact. Thus, the ONC-ATL does not "see" the functionality demonstrated that they are reporting is compliant. With an exception of a few criteria, discussed below, nearly all of the self-declaration criteria are those which are relatively simple and have a very low testing failure rate within ONC-ATL test events. As ONC noted, many of the self-declaration are gap eligible so they likely would not be tested for returning products.
More importantly, many of these criteria are still tested indirectly through other criteria. From an official testing perspective, the best way to verify criteria like smoking status or medication list were properly implemented is not through visual inspection of the UX but by their electronic summary record. Using the summary record and other files created by the system and then exchanging and incorporating them into another system is a better way of testing compliance to a standard. Since transition of care (315.b.1) and other criteria involving a test tool are still tested, much of the functionality in the self-declaration criteria will still be covered.
There have been some complaints from industry stakeholders that this will diminish certification and allow issues like the DoJ discovered with eClinicalWorks to become more prevalent. However, the example of DoJ/eClinicalWorks is actually more proof that the method of examining the “surface” of the UX display is not really sufficient. As noted above, a better method of examination is examining the files created through a variety of scenarios.
What They Got Wrong My complaints on self-declaration is two-fold. One, I felt that the selection of the criteria to be eligible for self-declaration should have been chosen differently. I don’t believe they proactively engaged ONC-ATLs in advance to ask them their input on criteria which would justify self-declaration, and I think the ONC-ATLs could have given some good feedback to direct this effort. The selection of the criteria seemed more of a blanket decision primarily on not involving a test tool. However, there are some criteria which are not addressed indirectly via a test tool and have a higher likelihood of errors.
The major criteria marked as self-declaration which strike me as needing inspection testing is auditable events and audit log reporting (d.2 and d.3). There is so much which can go wrong with auditable events, and with security breaches and other privacy challenges in the marketplace, this criteria needs thorough testing. Also, the new criteria like Implantable Devices (a.14) and Patient Generated Health Data (e.3) should be tested as these are not widely implemented capabilities yet. Criteria which have some current vagueness or ambiguities in terms of their implementation design warrant inspection testing.
My second complaint is the timing. I truly respect ONC and what they have done with advancing health IT in our country, and I give them high marks on their work over these nearly 10 years of 201X Edition criteria. However, the chief critique I offer over this time is that their timing of enforcement of new regulations or sub-regulatory requirements very often puts ONC-ACBs, ONC-ATLs, and developers in a very difficult spot by having to pivot so quickly. In this case, a better approach would have been to make self-declaration required by January 1, 2018, but allow ONC-ATLs to implement it sooner if they desire. That provides a 3-month window for ONC-ACBs and ONC-ATLs to properly update their quality system (which is also an ONC requirement that does not lend itself to fast pivots) and better implement this requirement. Also, I think they should allow developers the option of electing to test through demonstration rather than solely through self-declaration. There is value for some developers in choosing this path if they want the additional confidence that comes from a 3rd party evaluation. As much as developers may find demonstration testing nerve wracking, there is also a comfort in completing this before 3rd party experts. ONC has made clear they will still hold developer to the same level of compliance if they were tested normally so self-declaration actually increase the risk for both ONC-ACBs and developers in terms of certification compliance.
What People Are Overlooking Still, I consider the introduction of self-declaration testing a net positive, and it is my personal opinion that this change is not going to suddenly introduce new undiscovered errors into the health IT space in a noticeable way. To me, the bigger news is moving away from randomized surveillance, and this change is being largely ignored.
In the 2015 Edition Rule, the introduction of randomized surveillance was a major point of emphasis by ONC, and to see them effectively remove it - their statement is that they will not be expecting ONC-ACBs to enforce it as part of their certification accreditation - is a major reversal of policy.
In my time with Drummond Group as an ONC-ACB, randomized surveillance was a difficult requirement to enforce for many reasons, most notably the challenge in getting providers to donate their time even though they were not experiencing any current issues, and it was a significant drain on our internal resources. To have the requirement relaxed is a welcome addition.
As part of its reasoning, ONC stated it wanted ONC-ACBs “to prioritize complaint driven, or reactive, surveillance”. I recognize this change is driven partly because of the direction of this current administration to reduce the number of regulations, and this was a way to remove one that would have little industry objection. However, I believe this also a recognition of the new world of health IT we are now in regarding findings of non-compliance, and that is what people are missing.
With the news of the DoJ investigation into eClinicalWorks, everything has changed. Developers have legitimate fear of running afoul of government investigations. Coupled with the recent ONC Enhanced Oversight and Accountability rule, you have a much more active engagement from the government looking for EHRs which are non-compliant. This attention has also increased awareness of clinicians and hospitals to report errors and complaints to their ONC-ACBs or to ONC directly. As a result, ONC-ACBs do not need to go “looking” for errors in randomized surveillance, but instead, these complaints will come to them. Developers may have found randomized surveillance aggravating for the hassle, but it was generally low risk.
There is sense of true concern from EHR developers of not wanting to run aground of non-compliance of ONC requirements or MIPS calculations or general patient safety concerns which can draw attention of government eyes. To provide developers with greater confidence, Chart Lux offers quality assurance testing to assist in ONC-ATL self-declaration preparation, MIPS readiness, and general patient safety design. Contact us for more information on how we can help.
