Stop the presses! A readiness-measure review paper came out this morning in Implementation Research and Practice, a brand new journal from the SIRC folks focusing on implementation issues in behavioral health. Bryan Weiner had a readiness measure review paper about 12 years ago that was extremely influential in my own thinking on how readiness can be an important organization construct in helping to measure change.
full disclosure, I’m an author on one of the papers/measures that didn’t fair so well (Scott et al. 2017)
However, where this new paper really falls apart is in how it thinks about the pragmatic application of measures. So, here is how they looked at the papers:
PAPERS includes nine criteria for assessing psychometric properties: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. In addition, PAPERS includes five criteria for assessing pragmatic properties of measures: cost, language accessibility, assessor burden (training), assessor burden (interpretation), and length .
So that’s an okay set of criteria, but the pragmatism properties falls really really short. PAPERS talks about things that contribute to whether a measure may be used. However, what can also really help a measure is whether or not it can be adapted (so can it be made more meaningful to respondents) and whether it not leads to tangible interpretations.
First, it’s extremely rare that an intervention is generic. People aren’t just implementing a change; they are implementing a specific change with it’s own specific requirements. A more useful assessment is one that can be customized for the specific use case.
Second, when something is being put into practice, it’s critical that the data needs to have actionable implications for people on the ground. Otherwise, it becomes just research data. That’s OKAY!, but I’m more interested in how assessment can stimulate changes. Sometimes, people talk about assessment as a being an intervention in and of itself. That’s certainly one of the foundations of Empowerment Evaluation and other evaluation-capacity building strategies.
But, back to the article. It’s okay. It’s research focused. And it certainly doesn’t provide any guidance about what a change agent or implementer should do when putting something new into practice. There is likely a tradeoff between psychometric validity and adaption. However, these constructs aren’t precise enough to worry too much about that. After all, what’s the difference between a 5 and 5.5 on leadership engagement?
My two cents; please push back and let me know that I’m a MORAN. Read for yourself at: Measuring readiness for implementation: A systematic review of measures’ psychometric and pragmatic properties. But, do remember the words of Thom Yorke.
An empowered and informed member of society (pragmatism not idealism).
Fitter Happier