Ehr Testing Challenge

THE EHR TESTING CHALLENGEPerfecting the Process to Yield Better Results

www.qasymphony.com • 1-844-798-4386

Contents

Overview .............................................................................................................. 1

The Unique Challenges Healthcare Organizations Face .......................... 2

Defining Testing Coverage, Scope & Priority Using Risk Analysis .......... 3

The Must-have Testing Strategy ..................................................................... 5

Selecting a Testing Method ............................................................................. 6

Workflow Distinction & Definition ................................................................. 9

Test Results & Defect Reporting .................................................................... 10

Conclusion ........................................................................................................... 12


Electronic Health Records (EHR) have become an essential

part of the way healthcare organizations operate. But, the

quality of the EHR software has become a sore spot for

users. Because these systems are often highly complex,

interoperable and not-so-intuitive, testing by the end user

is absolutely critical. Why? Think of all the workflows in a

healthcare operation: from admissions, patient visits and

testing to labs, medications and prescriptions — not to

mention documentation requirements, interoperability

verification and analytics / reporting. Workflows also

vary amongst facilities, clinicians, physicians, accounting

and regulatory administrators. The possibilities are not

endless, but they are daunting.

Ultimately, healthcare systems themselves — be it a

physician practice, practice group, acute hospital system

or a post-acute system — are responsible for patient

safety, dosage accuracy, data integrity and patient

information security. And yet, healthcare systems

generally don’t have teams of professional software testers

or quality analysts that test their software and ensure that

everything is running smoothly — for all users. That testing

burden often falls on the clinician, which means that

it’s often the last step and one that’s performed under

extreme time pressure. For that reason, it’s rare that

these healthcare systems are able to implement testing

processes that scale and allow for re-usability thanks to

the great pressure surrounding testing.

A common complaint amongst EHR users is that they

don’t know what exactly they need to test. They make

statements like: “As an end user, I don’t access the back-

end systems to view code or process details so I’m reliant

on being able to see, touch and test all the UI workflows

for each user type.”

What’s more, interoperability testing is a difficult task since

those systems are designed to run in production, not in

test. Most can be configured to test on a staging server,

but the environment is never exactly the same as production.

With the Right Tools and Processes in Place, EHR Testing CanActually be Incredibly Effective and Perfectly Painless.

Overview


Additionally, there are multiple types of interoperability systems live at

any given time — and typically from multiple vendors. In most systems,

there’s pharmacy (prescriptions), patient-facing documents and records,

lab test results, and MRI or other imaging-related records — not to

mention medical devices and software-controlled surgery devices.

With each additional system comes more test formats, test cases and

repositories, and less traceability back to what change in which system

caused a test to fail.

All this proves that the depth and breadth of tests that must be

performed each and every time software is updated can be a monster

to manage. So how can healthcare systems wrangle it in the midst of

all their time and resource constraints? Even with the perfect testing

strategy and tools, it isn’t easy. End user testing is critical to both patient

care and clinician morale, and a poorly tested EHR negatively impacts the

health of the business itself.

If we agree user testing is essential, then what can be done to create

an effective and efficient method of testing? Ultimately, just like patient

intake or discharge, software testing is a critical workflow for hospitals

that requires attention all its own. By investing time in the development

and optimization of a sound testing strategy, healthcare organizations

can ensure the quality of their applications, and use the software to their

best advantage.

In this white paper, we offer healthcare organizations insight into techniques

that will help them continuously improve their testing practices.

The Unique Challenges Healthcare Organizations Face

Testing EHR software for a healthcare organization presents unique

challenges. How do you know what to test? What are all the “things”

you should test? The objective, or point, of testing is to ensure the

application meets your regulatory needs, your workflow needs, and your

users trust it to provide valid and accurate information. The first step

in ensuring testing success is planning a test strategy. A test strategy

defines testing scope, priority based on analyzing the risk factors

for each application workflow or operation, based both on past

history as well as recent configurations and changes.

DOCUMENTS & RECORDS

The many systems involved in patient care can make it

incredibly difficult to determine what causes test failures

DOCUMENTS & RECORDS

MRI OR OTHER IMAGE RECORDS

LAB TEST RESULTS

PHAMARCY

http://www.qasymphony.com


End users don’t need to be professional software testers if

they have a clear plan and set of testing assets to execute.

When you have a plan that defines what’s tested, by whom

and how deep into each workflow the tester needs to go,

end users are empowered to test applications the right way.

Developing a plan is as easy as:

• Defining end user group workflows

• Mapping out clinician workflows for each non-

physician role

• Mapping out physician workflows — and don’t forget

lab results, radiology, pharmacy and reports, both

patient-and system-based.

• Using risk analysis to define your test coverage

Defining Testing Coverage, Scope & Priority Using Risk Analysis

Risk analysis is a method of determining what needs to be

tested for each workflow and ranking it by priority. As end

users, you know which workflows are critical and which

are lower priority. Keep in mind that testing occurs on all

priority levels so don’t think that only the critical workflows

are tested. Risk analysis provides a way to narrow down

which workflows and functions should be tested first and

most frequently. Some workflows need to be tested with

multiple data points and across different environments,

while others might be less mission-critical and require one

pass — or may not need to be tested at all.

Risk analysis can be done using a spreadsheet, a Word

document or — better yet — a purpose-built test case

management platform that can allow for dynamic planning

and prioritization. For simplicity’s sake, the example we

describe here uses a spreadsheet for risk analysis.

When creating a risk analysis spreadsheet, the left side

should list the workflows and their functions — such as

all the tasks a clinician performs in a software application.

It would also include an admission workflow, including

the tasks for admitting patients, capturing accurate

demographic data, allergy information, medication and

health history. Additionally, users may be responsible

for documenting and verifying insurance coverage by

payer. We’ve barely scratched the surface in the patient

admission process, and already there’s a large amount of

testing to cover.

In addition to helping simplify and document the workflow

processes, the risk analysis grid offers additional distinct

advantages. Once you’ve done the initial work of creating

it, the grid is easy to update and use the next time you

need to plan testing.

Risk analysis is a method of determining what needs to be tested for each workflow and ranking it by priority.


Given typical unforeseen hiccups, we recognize we’re never able to test every workflow we want to in the depth we’d like,

so establishing priorities up front allows us to focus on core testing responsibilities as timelines shorten.

Too often, healthcare organizations only think about testing when their testing process is already in flight, and they

don’t spend the necessary time between test cycles to retrospectively optimize the process. With the approach we’ve

described here, you’ll know — and be able to prove — that your software works accurately for every functional use

within the organization.

The risk assessment score is determined by multiplying the assigned weight (impact) by the business risk (likelihood percentage). So, the most risky areas are the ones where the potential impact is high (i.e. patient death) and the likelihood is also high (i.e. part of the new patient enrollment path).

If you look at the example “grid” above, you’ll see the functions listed on the left as well as the various “weight” factors

that create a calculation and determine the risk value on the right. For a healthcare organization, you’ll want to change the

headings to match your needs, and as a team determine the rankings of each defined function. Once each is ranked, then

the final calculation is used to determine what value constitutes a high, medium or low risk.

Once the risk level is determined, create your test strategy by ranking the functions in priority order based on risk. When

pressed for time, start with the highs, mediums and then lows. In subsequent testing rounds, consider mixing the risks

and testing the mix to ensure full coverage without testing every function every time.

Please note that risk scores will vary in each particular organization.

HIGH

Risk score > 467

MEDIUM

Risk score >234 and <466

LOW

Risk score zero and < 233

DEFECT/FUNCTION/ACCEPTANCE CRITERIA ASSIGNED WEIGHT BUSINESS RISK FUNCTIONAL

IMPORTANCE ... RISK SCORE RISK LEVEL

WEIGHT SCORE WEIGHT SCORE ...

Medication Orders FDB 10 10 100 10 100 ... 600 High

Medication Orders Direct 5 10 50 5 50 ... 190 Low

Medication Orders Search 7 10 70 7 49 ... 329 Medium

Medication Orders Copy 5 5 25 5 25 ... 175 Medium

Order Sets FDB 10 10 100 10 100 ... 640 Low


TEST STRATEGY TEMPLATE<Project or Release>

<Author name - Version 1.0>

Project definition & objective<A brief introduction of the overall project.>

Test DefinitionFeatures Tested<List all the application functions that are planned for testing.>

Test DeliverablesTesting TasksDevelop test cases.Conduct test cases reviews with team members.Perform tests.Report bugs.Update the Team’s Risk Analysis grid (if needed).

The Must-have Testing Strategy

In software development teams, the testing strategy is generally called a “test plan.” Traditionally it’s a long document

that’s meant to be reviewed and approved by upper management prior to any test execution. In reality, the test plan is an

extremely long and tedious read, containing the detail of all test cases and requirements that nearly no one, except the

authoring tester, reads.

The testing strategy is meant to be short, concise and to the point so users at any level can read through it in under 10

minutes. It’s an outline that includes a description of who’s testing what, where and when, and it provides the organization

with a history of the testing effort as well as an overall plan. Again, it’s short, quick and concise — just like the generalized

template below.

TEST CASE TEMPLATE<List the test case ID and title or brief description OR include the path to locate >

ID# –Meds Summary Window_Physician

Testing Resources & Responsibility<List the testing resources assigned and their QA responsibilities.>

Testing Environment, Date/Time<List the testing resources assigned and their QA responsibilities.>

Risk & Mitigation

<List any known risks and their mitigation strategy.>

RISK MITIGATION & COMMENTS

<Example>

Full regression testing that includes different client types is not possible Selected 2 standard customer scenarios (Medicare A, Medicaid).

1

2

3

1

2

3


If you search the Internet, you’ll find a number of different free

templates and samples for testing strategies. The important idea is

to find the one that meets your needs and will be adopted by your

testers. The testing strategy should be a living, breathing document

that’s read and understood by all parties involved in the testing effort.

Minimally speaking, the testing strategy defines who’s testing what,

where and when. Once execution is complete, it can also be used to

track results if you don’t want to use a separate document. Sometimes

minimizing the number of documents that have to be created, tracked

and used results in greater team efficiency.

Once a testing strategy is defined, it’s subject to change until testing

is complete since you want the testing strategy to be a record of the

testing event so it serves as a historical record. By keeping a record

of each testing effort, the team can learn from the past and make

the process better by implementing changes to it. Continuously

improving testing is only possible if past testing events are

documented or known.

Additionally, the testing strategy is a useful tool for communicating

with upper management or even vendors who need to know what

tests are being planned and executed. This keeps the organization’s

resources from having to repeat what’s being tested, where, when

and by whom to multiple sources. They can simply share the testing

strategy and focus on actual work to be done.

Selecting a Testing Method

Once you’ve gotten the risks documented and measured and the

organization’s testing strategy is defined, you’re ready to select a

testing method. But how do you know what testing method will work

best for your organization, or which one will provide the best results

Let’s break it down.

It’s hard to sift through all the buzzwords in the software development

industry — one of the most popular of which is Agile. Agile is a great

methodology for teams with dedicated, experienced developers

http://www.qasymphony.com

7 www.qasymphony.com • 1-844-798-43867

and testers who work closely together and collaborate

frequently. But for healthcare organizations, Agile has

significant shortcomings in terms of adoption. Agile

is meant to be fast and flexible, which doesn’t always

translate well in the highly regulated healthcare industry.

In addition, Agile doesn’t always work well when many

systems need to be coordinated on a single release cycle

— which is often the case in healthcare.

Another problem with Agile for healthcare application

testing is that at the end user level, there really isn’t room

for flexibility. Either the system works or it doesn’t — you

can’t be flexible on regulations, dosage accuracy or vitals

monitoring. Close is not good enough when it comes

to patient care or proving that an organization meets

government mandatory regulations.

One popular technique for overcoming this is building a

more traditional — or waterfall — suite of manual test

cases managed in a dedicated test case management

system. Implementing test case management to house

your suite takes time and effort, but it is a long-term win.

The tests are re-usable for as long as the software is in

place, and results can be tracked release over release.

While the tests may need to be updated if workflows or

major software changes occur, they can generally be used

for many years without major effort. For most test case

management tools, manual test case suites are defined as

scripted tests that are written as step-by-step instructions

on how to proceed through a function or workflow and

provide documentation on the expected results.

If you want to give testers greater freedom to challenge

your system in new and exciting ways, you should try

exploratory testing, which gives your testers a more

flexible approach. Once testers are satisfied with the

expected workflow processes, exploratory testing

empowers them to use your system like real patients and

clinicians would — and document new errors while in the

actual experience. None of the barriers that traditionally

come with a scripted test case are there to get in the

way. Instead, a tester can take many different paths in

an attempt to “break” the application. For example, they

can try to get the system to generate an incorrect dosage

calculation, place a medication order on a patient with a

severe allergy to it, pull a document onto the wrong user

record or attach a prescription for patient A on patient B.


With exploratory testing, anything a user can think of that

could make an application fail is fair game.

One important thing to remember when using exploratory

testing is to take clear notes on what’s been tested to

ensure adequate coverage in case of an audit. Many

vendors will provide automated documentation tools to

assist in this effort since clinicians may not have the time

or knowledge to capture this documentation during the

actual testing cycle. It’s also essential that testers focus on

finding errors and not following “rules.” Exploratory testing

is meant to find errors outside the lines, and it adds value

to testing because it finds errors that may go unnoticed

until a user makes an entry mistake or skips a step or two

in the expected workflow.

If an organization is pressed for time and the testing

resources don’t need detailed or explicit test steps,

consider using functional checklists. Checklists are simply

lists of the main functions each user role performs and

the expected result. Using your risk analysis grid data,

just create simple checklists using the application of your

choice with check boxes to indicate they’re complete.

Checklists work well when the resources doing the testing

are intimately familiar with the software application and

the role workflows. Also, checklists are relatively easy to

save, update and manage online or offline. Just remember:

their success at testing depends on the knowledge of the

tester as well as the quality of the testing process and the

assets provided.

Lastly, another process popular with many vendors

is automated testing. Automated testing works best

for organizations with more extensive IT or software

development resources. Automated testing is great for

performing repetitive tasks and validating scenarios with

multiple rows of data, but it doesn’t replace the need for

testers to define the automated tests needed and validate

the end user workflow experience. Automation also tends

to require regular maintenance, so increasing the number

of automated tests will require an increase in the IT

personnel necessary to support them.

What tools are available to give control and visibility into

all of these different types of testing present in today’s

leading hospitals? Using spreadsheets is common in the

healthcare arena — but spreadsheets are limited

in functionality.

If you’re using exploratory testing, it’s important to remember to take clear notes on what’s been tested so you have adequate coverage in case of an audit.

9www.qasymphony.com • 1-844-798-43869

Spreadsheets don’t provide useful test metric information and

they’re difficult to manage efficiently, especially in large teams. Just

as hospitals invested in EHR systems as their patient data and

workflows grew in complexity, they’re now investing in test case

management solutions to replace their growing repository of test

cases managed in spreadsheets.

Workflow Distinction & Definition

When defining role-based tests, remember to include all the roles

affected by the software. Plan who’s going to execute each workflow

— from clinicians and physicians to administrators — and make sure

any changes made up front don’t impact downstream accounting or

finance functions. It’s especially important when there are questions

about functionality or the user role isn’t actually the person doing

the testing. For example, it’s rare for physicians, administration

heads or the CFO to actually test the software, and if they aren’t

going to test their workflows, it’s critical that whoever does has a

well-defined test script or functional checklist that incorporates

those users’ needs and expectations.

When creating workflows, it’s important to interview or collect input

for each role. You’ll want to find out:

• What frustrates them in the software?

• Where exactly does the software fail them?

• Do they have to duplicate tasks?

• Can they get to the information they need quickly?

• What functions in the software do they not trust?

Often, discussions about how to best test software become

insightful conversations about how to best build and configure the

application. One common EHR complaint amongst physicians, for

example, is that software adds unnecessary tasks that interrupt

their workflow and causes them rework. Similarly, clinicians often

complain about inaccurate lab results with incorrect or missing value

indicators. Knowing what parts of the software are problematic to

end users helps you plan the testing effort and understand where to

focus the most attention.

A common EHRcomplaint amongstphysicians is thatsoftware addsunnecessary tasksthat interrupt theirworkflow and causesrework.


Once a testing method is chosen and the test strategy and role workflows are defined, it’s time verify. When using role

defined workflows, the testing results are highly dependent on the validity of the workflow, so make sure at least one expert

from each area reviews the workflow tests fully. A solid, reviewed test case provides improved and valid testing results.

Test Results & Defect Reporting

As the testing effort progresses, defects will be found and documented, and will likely need to be reported. When

reporting these issues to vendors or support personnel, it’s critical to explain the steps that led to the failure in detail so it

can be replicated.

First, capture the title and description in a format that includes the functional area, followed by the role or workflow and

then a short description. For example, as a clinician discovers that when he sends a verbal medication order to a physician

for approval, the application fails to update the medication order status during the active session. He has to save, log

out and log back in to see the correct status. So how can the clinician explain these errors to the vendor in a way that

allows them to understand the nature of the problem and the need for an immediate fix? The statement below is a good

example of how to communicate the issue:

Medication processing physician approval request status fails to update during active session.

In this statement, the functional area is first, so it’s easy to see where the problem occurs; next comes the role it affects,

accompanied by a succinct description. Last is the status, worded in a way that that’s clear and concise, and tells software

development team exactly what’s wrong. Next, follow up with accurate steps to reproduce.

When reportingfailures to vendorsor supportpersonnel, it’scritical to explainthe steps you tookin detail.


The steps to reproduce the issue are key for error

reproduction and getting the issue corrected as soon

as possible. Be as detailed and explicit as you can, and

remember that you’re explaining your steps to a

complete stranger with the end goal of getting the

problem corrected. Below is a good example.

Example:

“Log in” as a clinician with access to

place medication orders.

Navigate to the physician order entry

page by clicking the “Medication

Orders” link.

Select a patient with multiple existing

and approved medication orders.

“Add” an order for Levothyroxine

125 mcg with a frequency of Daily

for 90 days.

Click the option to “Send to Physician”

for approval button.

Click “Refresh” or allow the window to

refresh and updated the view.

View the order details and the

order status.

Clear, concise and detailed. It’s important to indicate the

user role that’s logged in and performing the action.

It’s also significant that the problem occurs on patients

with existing orders. These steps are configuration or

situational indicators that are helpful for development

teams when trying to reproduce a customer user defect.

In step 5, you’re indicating the action you’re trying to take,

but you also want to include what you expect to occur and

what’s actually happening.

After the steps to reproduce, add two additional sections:

“Expected Results” and “Actual Results”. For the example

above, the expected and actual results look like:

Expected Results: The “Send to Physician for Approval”

button should update the medication order status to “Sent

for Approval” immediately.

Actual Result: The “Send to Physician for Approval”

button remains as “Draft Order”. The status fails to

update unless the user saves, logs out of the application

and then logs back in. Once the user logs in again, the

status is updated.

Since EHR applications tend to be highly configurable,

fully describing the problem and what the user expects

to see is essential for communicating the nature of a

problem and its impact on the end user. Clinicians often

struggle with getting the detailed reproduction data they

need to close out defects on the first attempt, which is

where a defect documentation tool like qTest eXplorer

becomes invaluable. These tools record every aspect

of the testing session and automatically create detailed

documentation. Defect reports written with distinct

steps, expected results and a problem description are

more likely to get fixed — and the clearer the problem is,

the less likely it is to get pushed off to later or ignored due to

a lack of understanding. Using a tool like qTest eXplorer also

helps with documentation for an audit — making the entire

process easier and more effective.

1

2

3

4

5

6

7

12

Conclusion

EHR software updates occur on a regular basis — and each time they

do, it creates risk. The challenges of testing an EHR application for each

update across all user roles and functions can only be overcome with

a good testing strategy, a full understanding of the risk areas, clearly

defined user workflows and testing methods that are maintainable and

re-usable.

When testing, it’s critical to maintain a history of the effort so future

events can be improved as needed. Knowing that testing will result

in finding defects, reporting them clearly and concisely yields a plan

for working around errors and getting issues fixed faster. Too many

healthcare organizations don’t have the data they need to make

accurate testing decisions for one simple reason: they’re struggling to

manage their testing activities in spreadsheets.

EHR software is as complex as healthcare organizations themselves,

and the variations in workflow and configuration create systems prone

to failure. The burden of ensuring that systems work, falls on the end

user at the point of patient care — making end user testing essential to

the overall success of the organization, its employees and its patients.

For all these reasons, organized end user testing is not optional for

healthcare organizations; it can be the difference between success and

failure. When it’s planned, organized and executed with methods suited

to the organization, testing can be effectively and efficiently executed —

without all the aches and pains.

Protect your business, employees and patients — test often and test

organized for far better results.

References• Becker’s Health IT & CIO Review, The problem with EHRs: 5 complaints from CIOs,

Akanksha Jayanthi, January 20, 2015.

• InPracSys, TOP 7 PHYSICIAN EHR COMPLAINTS, Vlad Hurduc.

• Testing Computer Software, Kaner, Nguyen, Falk. 1999 Wiley Publishing.

• How to Break Software, James A Whittaker, 2003 Addison-Wesley

Protect your business,employees andpatients — testoften and testorganized for farbetter results.

QASymphony was named a Cool Vendor in Application Development by Gartner in 2015 and is headquartered in Atlanta, GA.

Learn more at www.QASymphony.com or call 844-798-4386

To create solutions for Agile development teams that significantly improve speed, efficiency and collaboration throughout the software testing process.

THE QASYMPHONY MISSION —

Sign-up for a FREE TRIAL | Request a FREE PERSONAL DEMO