We love ranking and rating things and have come to rely on them. What’s the highest ranked college that your kid can get into? Which restaurant has the best Yelp review? Which beach did TripAdvisor rank as the best in the world? Rankings and ratings are increasingly permeating healthcare, too. US News and World Report already releases annual and ‘scientific’ hospital rankings over which administrators obsess. Soon, if not already, physicians will also be officially ranked and rated and pay for performance could become the norm. If you raise this topic with surgeons, you quickly realize that most are not too excited about this.
Of course everyone wants performance to continually improve. If they didn’t then there would be no need for research projects, CME, conferences or even reviewing tough cases with your colleagues. As a group, we tend to be intrinsically motivated and work hard to continually improve the outcomes of our patients. So, if that is the case, why wouldn’t all of us be completely supportive of “pay for performance” and the ubiquitous patient satisfaction surveys (HCAHPS, OAS CAHPS, CG CAHPS…etc)?
The first challenge in pay for performance is defining “performance”. How do you objectively measure the “performance” of a physician? Administrators assume that this is pretty straightforward. Can’t you just get patient satisfaction scores and outcome measures and maybe some administrative data like number of patients seen per day, and be done with it? Well, let’s begin with patient outcome: How do you value the objective outcome of the patient as compared to the subjective outcome of the patient? And how does this relate to the patient’s satisfaction? The relationship between these three things are often independent of each other. As an example, imagine the patient post-op from a PIP fracture dislocation with no pain and near full strength and ROM but the patient is unhappy with the residual swelling…in addition, they are very upset because of rude nursing staff and difficulty getting a hold of the physician’s staff when the pharmacy wouldn’t fill their prescription.
The next obstacle is consideration of the factors that affect patient outcomes. Of course the surgeon is a component of this. However, the nursing staff, office staff, therapists, scrub technician, central processing staff and every other staff member that the patient either directly or indirectly interacts with, also has an impact on their outcome. Even the facility itself can have an effect on outcome. What about the patient-specific factors? It is logical that patient-specific factors such as socioeconomic status, family support, employment status, compliance, health literacy and a multitude of other factors, all contribute in some way or another to outcome. To express it mathematically: Performance = physician + hospital + health care staff + patient physical health + patient psyche + medical device company + weather conditions affecting patient + …….+, + all the hundreds and thousands of interactions between the components of the health care system.
In Systems Thinking we learn that it is a combination and relationship of all of these factors (The System) that produces the outcome. The surgeon is an important part, but just a part, of The System. So, when you measure physician performance what you are actually measuring is The System in which the physician works. Therefore, how can you accurately compare one physician to another physician when they are taking care of a different set of patients and work in different clinics with different staff on different days, etc? Well…the simple answer is that you can’t. Your measurements are of the two different Systems. Therefore, you are not comparing the performance of one physician with another physician but rather the one System with the other System.
So far, we have seen that: 1) measuring performance is much harder than administrators like to believe and 2) physician-specific performance cannot feasibly be measured. This is a concept that will take time to fully grasp and, for some administrators, will never be understood. Does this mean that we should not bother measuring the performance of physicians? No, data is very important. However, it is the analyses and the conclusions that we make, where we must be very careful. If we can’t practically measure the performance of the physician independent of the System then we cannot reliably rank or rate physicians. That means, ranking a group of physicians in the top 5% or another group of physicians in the 30th percentile is fallacious.
If you can’t feasibly measure individual performance, then you can’t award an individual for their specific performance. This will undoubtedly unsettle many managers who rely on carrots and sticks to “motivate” employees. Performance data is important, but we need to know how to collect, quantify and analyze the data. Each data value is not an independent measure of performance (remember that performance comes from the System!). To collect and analyze performance data scientifically, and determine whether there is a positive or negative outcome related to a physician(s) requires application of the Shewhart theory and use of control charts for data analysis. This sounds complicated, right? Isn’t it just easier to carry on with the way that we’re doing it now? When the laboratory measures a patient’s HIV viral load, you better believe that they employ great precision and scientific analysis to make sure that the result is accurate and reflective of the true health state of the patient. Why wouldn’t we employ the same amount of science and knowledge when we rank and rate people?
Even if you could reliably measure the physician’s specific performance (independent of the multitude of other factors that affect performance) and could then pay the physician according to his/her “performance”, what is the point? How will this change behavior – short term and long term? Also, what is the Shewhart theory and control charts? Will this put me to sleep minutes after I start reading about it? How do we use this knowledge to make meaningful improvements and improve performance throughout all spectrums of the health care system? All of this will be covered in my next Focus post(s).
This is an exercise in futility, yet the ivory tower academic thinkers believe this is a valid goal. An analogy can be drawn to the compensation of teachers. There is no reasonable way to accurately determine a teacher’s performance unless you take into consideration factors on the student side… ie: IQ, culture, family structure, behavior, IQ (Yes I said IQ twice) . The same goes for patients.
No my academic friends, the best way to determine how doctors should be paid is by adherence to a free market system and letting the chips fall where they may. It is competition that keeps costs down and quality up. Anything else is an artificial accounting and subject to bias, corruption, and abuse.
Dr Gordon, you are 100% correct. Measuring a teacher’s performance and then ranking and rating that teacher compared to other teachers is just as ludicrous. That teacher’s “performance” is dependent on a multitude of factors outside of that teacher’s control. When we do this to individuals (i.e. inappropriately attribute the performance of a system to an individual), people start to cheat and come up with all sorts of tactics to falsely improve their “performance”. Can anyone think of any current examples? Maybe Wells Fargo employees that made up false accounts/opened excessive accounts, Teachers in the Atlanta school district that cheated to artificially improve the test scores of their students, Hospitals administrators that try to get very sick patients out of the ICU and into hospice to prevent their mortality rates from suffering…etc.
One doesn’t have to read the business section of the newspaper long before you read about some group in some company that was caught fudging the figures so that they met an unreasonable quota placed on them by management.
Jacques, this is an important topic that unfortunately does not get discussed enough by the physicians on the front lines – those who are both intimately concerned with performance to provide the best care for their patients and forced to justify to administrators “poor outcomes” that may have just as much to do with the price of parking at their hospital.
Rather than appreciating the view of everything around and how it all fits together to make a beautiful panorama, the administrators up in their ivory tower seem to focus on singular “perceived blights” below. The view is never good enough – it can always be more perfect. Everything has to be “measured”, everything has to “measure up”.
I agree with Scott’s comment on the free market system and competition being a “natural” regulation of the system. However, I think that some degree of measuring performance is appropriate. When it comes to keeping costs down, I don’t think many people would disagree that a surgeon who unethically up-codes a carpal tunnel release to get better reimbursement from insurance is failing both patients and the system. The cost of a surgeon’s procedure is a perfectly reasonable “performance measure” to judge them by; but the consumer of the system (the patient) is often not the best person to judge that performance, so the “free market” fails them. Where does the market play a role and where does it not?
When it comes to true surgical “outcomes”, however, we WILL be measured on our “performance” by administrators and bureaucrats who have no direct relationship to a satisfied patient and have no understanding of what they are “measuring”. There will be no future health care system where that isn’t a part of our practices. As hand surgeons, we will be best served to DEFINE what we value as appropriate outcomes and measure them OURSELVES, BEFORE those at the top of the tower do it for us.