You’ll have to hear us out, but we think the Martian has something to teach us in healthcare analytics. The Martian is a captivating story of teamwork, creative problem-solving, and the awesome powers of math and science. It’s also a great reminder of the critical step of validation… the step the team at NASA skipped in an attempt to fast-track emergency supplies for the astronaut stranded on Mars. The result was a rocket that exploded within seconds of takeoff.

In our analytics projects, rockets don’t explode if we don’t validate before publishing an application, but there are still consequences: data telling a misleading story or calculating inaccurately erodes trust faster than my daughter devours the frosting off a cupcake.

Imagine our wonder recently, then, when one of our customers, Lakeland Health of southwest Michigan, hit a home run on their validation phase, completing all the validation for a complex application in a 5 days – how did they do it? What are the secrets?! Before we reveal the secret sauce, first a quick look at what wasn’t the secret.

• Contrary to what you might think, the team that completed validation was not built from workflow and EMR experts. In fact, only about half of the group working on validation considered themselves even proficient in the EMR application from which many of the metrics were sourced.
• Also a surprise was that the 4.5 people who worked for 5 business days to complete validation did not sit, nose to the grindstone for 8+ hours each day for a week. The average time spent per person was about 15 hours per week.

So, what did it take to be the gold standard of validation?

Secret #1: An amazing metric workbook

Ok, this one wasn’t much of a surprise. We’ve always related the time spent on metric definitions to the efficiency of validation. What this team did better than others was have a clear understanding of how the metric workbook would be used during validation, so they could thoroughly populate it to support that use. Because the team working on validation didn’t have equal expertise in the workflows driving the metrics, the metric workbook had to be populated with information that leveled the playing field, with a lot of specifics about the source values, the configuration variables, and SQL queries for each metric. Which leads to our next secret…

Secret #2: Expertise in SQL

We knew that SQL expertise was important, but we were surprised that it was more important than workflow expertise for the validation team. Each member of the team utilized their expertise in SQL to both better understand the metrics they were validating and to validate the metric calculations themselves. They also used queries to creatively check that the raw data tied, so if a number still wasn’t tying in the UI, the problem could be more localized for faster troubleshooting.

Secret #3: Order of Operations

We have always started validation working from the visualization layer back to the source.  If an aggregate number didn’t look right on your overview dashboard, we would dig into what calculations were performed in the UI, then what transforms created the application fields, and then what extractions created the field. But if you flip the order of operations and start from your extract and validate the code, then move to the transforms and validate the code matches the Metric Workbook logic, you are more likely to catch the error earlier in the process. But for all of this to work you need…

Secret #4: Knowledge Transfer before Validation

Most people want to get through the project as quickly as possible to stop the ever-increasing demand and then learn how to maintain the project once it is live.  Our successful customer had a completely different mindset coming into the project which demanded knowledge transfer occur throughout the project and, specifically to focus on knowledge transfer as a means of preparing for validation. We trained the team on the data structure, each transform and how it related to the definition requirements in the metric workbook before validations started.  By front loading training, validation was more informed.

Secret #5: Autonomy in work with focus on outcomes

Validation can be tough. Part of the success of this project was the fact that all 5 people on the validation team understood the purpose and process of validation, and then could approach validation in whatever way they chose, based on their expertise. As a process worked well for one person, the others could learn and adapt their process. Sometimes, the first step was to run ad-hoc queries to really understand the metric definition before starting to validate it; sometimes it made sense to dig in to the EMR data dictionary to better understand the data elements themselves.

All of these culminate to our final secret from this outstanding validation effort…

Secret #6: Teamwork

This team did an outstanding job of, well, being a team, which made the rest of the validation process smooth, efficient, and collaborative. It started with a manager who embodied the best of all the team – he was seasoned in the EMR application and workflows, and fluent in SQL. He was able to assign metrics to the rest of the team based on their expertise, and help facilitate resolving issues and sharing understanding across the team. The collaborative dynamic he established throughout the validation phase was a big factor in staying on track.

We love when our clients are successful, especially when they share their secrets and successes with us so we can share them with our other projects. Everyone wins! Are you interested in learning more about how we can help you visualize your data to take meaningful action? Let’s talk.