A further consideration regarding pay-for-performance and the measurement of healthcare disparities involves the fact that pay-for-performance programs that consider healthcare disparities issues are likely to be focused more on disparities in control among subpopulations deemed to need special attention, such as persons diagnosed as hypertensive, than disparities in processes among the population at large. As discussed in references 6,8-10, 14 of my earlier comment,[1] the former focus involves truncated populations, where the distributions of factors associated with an outcome will tend not to be normal even when the overall distributions of which they are a part are perfectly normal. These references maintain that such fact would seem not to materially alter the typical patterns of changes of measures of differences between rates as the prevalence of an outcome changes (except for odds ratio), as illustrated, say, in Figures 6 and 7 of reference 8 to the earlier comment (though such issue certainly deserves further attention).[2]

But that disparities are examined within truncated populations would have material implications with regard to the approach described in the earlier comment that estimated differences between means of hypothesized distributions. As I have stressed, even as to an overall population, such approach is rather speculative given that we do not know the extent to which the underlying distributions of the groups being compared are in fact normal. But, for reasons explained with respect to Tables 6-8 of reference 14 of the earlier comment, such approach seems fundamentally problematic with regard to distributions in special needs populations that, being truncated portions of larger distributions, are almost certain not to be normal.

While the above comments are intended mainly to concern situations where control disparities are examined, I note that data discussed in the earlier comment involved treatment decisions concerning a population that had experienced acute myocardial infarction. While this a special needs population, I am nevertheless inclined to think that the white and black distributions of factors associate with receiving CABG tend to be more like those in an overall population (that is, tending toward normal) than those in a truncated population, such as, say, the white and black distributions of factors associated with control of hypertension within a population diagnosed as hypertensive. But to the extent that the distributions of factors are more like the latter than the former, it would call further into question the reliability of results of the approach described in the earlier comment.

In any case, that the examination of healthcare disparities in the context of pay-for-performance programs is likely to more often involve disparities in control than disparities in processes may well further complicate the task of devising rational methods for appraising performance.

Notes:

1. Scanlan JP. Pay-for-performance implications of the failure to recognize the way changes in prevalence of an outcome affect measures of racial disparities in experiencing the outcome. Journal Review Feb. 8, 2008 (responding to Werner, RM, Asch DA, Polsky D. Racial profiling: The unintended consequences of coronary artery bypass graft report cards. Circulation 2005;111:1257–63): http://www.journalreview.org/view_pubmed_arti...

2. References 6 and 8-10 explain why a study found improvements in care to reduce absolute differences in process outcomes but not reduce (or increase) absolute differences in clinical control outcomes. Such explanation involves the fact that improvements in the process outcome rates examined tended to involve relatively high favorable outcome rates (that is, in what is termed Zone B in the figures in references 6 of the earlier comment and where further increases tend to reduce absolute differences) while improvements in the control rates examined tended to involve relatively low favorable outcome rates (that is, in what is termed Zone B in the figures in references 6 of the earlier comment and where further increases tend to increase absolute differences). But such explanation involves a different issue from that of whether, within a population needing control, correlations between various measures and the prevalence of an outcome are similar to those observed in the overall population.

Werner et al.[1] found that after New York implemented a CABG report card, racial disparities in CABG rates increased. Such finding has been increasingly cited in discussions of the way pay-for-performance may affect healthcare disparities. A recent article described the Werner study as the only study of such effects so far.[2]

Like virtually all health and healthcare disparities research to date, however, the Werner study suffers from the failure to recognize the way measures of differences between rates tend to change solely because of changes in the prevalence of an outcome. CABG use increased substantially during the period examined by Werner et al. The white rate increased from 3.6% of AMI patients in 1988-1991 to 8.0% in 1992-1995, while the black rate increased from 0.9% to 3.0% during this period. It is the increase in the absolute difference between these rates – from 2.7 percentage points in 1988-1991 to 5.0 percentage points in 1992-1995 – that underlies the study’s finding that has received such attention.

But changes in the difference between the black and white rates need to be examined with a recognition of what would typically occur solely because of the general increase in CABG use. To begin with, for reasons explained in references 3 through 16, and in several score references at http://www.jpscanlan.com/homepage/measuringhl... during a period of increase in CABG one would typically observe a decrease in the relative difference between rates of receiving CABG and an increase in the relative difference between rates of failing to receive CABG. And that is what did occur. The ratio of white rate of receiving CABG to that of the black rate declined from 4.0 in 1988-1991 to 2.7 in 1992-1995; and the ratio of the black rate of failing to receive CABG to that of the white rate increased from 1.03 (99.1%/96.4%) in 1988-1991 to 1.05 (97.0%/92.0%) in 1992-1995.

In the case of the absolute difference between rates on which Werner et al. rely, for reasons explained in references 4 through 15, one would expect such difference to increase (as did occur and which was interpreted by Werner et al. to reflect a meaningful increase in disparity). On the other hand, for reasons explained in most of those references, one would expect the difference measured in odds ratios to decline, which also occurred. The ratio of the white to black odds of receiving the procedure declined from 4.1 in 1988-1991 to 2.8 in 1992-1995.

The references explain these issues, and why none of these changes ought alone to be regarded as indicating a meaningful change in disparity, at sufficient length that there is no reason for extensive discussion here. I do note, however, that references 6-7 and 9-13 discuss the Sehgal article[17] that Werner et al. cite as finding that quality improvements reduced racial disparities. Those references, particularly number 12, explain why one would expect absolute differences typically to decline in the context addressed by Sehgal at the same time that one would expect absolute differences typically to increase in the context addressed by Werner et al.

There remains the issue of whether disparities changed in some meaningful sense after implementation of the report card program (perhaps increasing as a result of the avoidance of high risk patients, as suggested by Werner et al.). As discussed in many of the references, identifying meaningful changes in disparities is fraught with difficulties, particularly in the common situation where, as here, all the measures change in the standard direction. Nevertheless, employing the approach addressed briefly in the latter part of reference 6, and discussed in the contexts of actual data in references 14 and 15 (with much attention devoted to the speculation involved in such approach), one finds that the estimated difference between black and white means of hypothesized distributions of factors associated with the outcome declined from .57 standard deviations in 1989-1992 to .47 standard deviations in 1992-1995. That is, such approach shows a decline in the disparity. But too much uncertainty is involved with the approach to place much weight on such result.

Other data are presented in the Werner study and a good part of its analysis involves the comparison of patterns of change in New York with those in other states. But I think the discussion above is sufficient to illustrate the problems with appraising changes in disparities by means of changes in absolute differences (or relative differences, etc.) without regard to the ways certain measures of differences between rates tend usually to change as overall prevalence changes. I will note, however, that the approach just referenced would show that in the other states the decline in estimated difference between means would have been from .40 to approximately .28 standard deviations. While perhaps suggesting a slightly larger meaningful decline in those states than in New York (which would seem consistent with the reasoning of the authors), I doubt that such difference would approach significance

Assuming the validity of my reasoning in the various references, virtually all health disparities research that has relied on dichotomous measures, or measures that are functions of dichotomies, is suspect for failing to consider the way such measures change solely because of changes in prevalence. But, while that would mean that substantial resources may have been wasted on such research, it is unclear that its errors have otherwise caused concrete harms. That is, whatever such research tends to show, policy makers have continued to implement what appear to them to be sensible measures for improving the health of the community. I doubt, for example, that anyone would consider eliminating something like the Back to Sleep Program because it led to increased socioeconomic (relative) differences in SIDS rates (as discussed in reference 16). Pay-for-performance, and particularly where performance will be in some part measured by perceived effects on healthcare disparities (as, say, is discussed in reference 18), adds an additional wrinkle to the matter. But whether or not paying for performance as evaluated by perceived effects on disparities without regard to the way various measures of disparity change solely as a result of changes in the prevalence of an outcome has the potential to cause concrete harms, it has the potential to undermine what might otherwise be a useful means of promoting healthcare.

References:

1. Werner, RM, Asch DA, Polsky D. Racial profiling: The unintended consequences of coronary artery bypass graft report cards. Circulation 2005;111:1257–63.

2. Chien AT, Chin MH, Davis AM, Casalino LP. Pay for performance, public reporting, and racial disparities in health car: how are programs being designed. Med Car Res Rev 2007;64:283S-304S.

3. Scanlan JP. Race and mortality. Society 2000;37(2):19-35 (reprinted in Current 2000 (Feb)): http://www.jpscanlan.com/images/Race_and_Mort...

4. Scanlan JP. Can we actually measure health disparities? Chance 2006:19(2):47-51: http://www.jpscanlan.com/images/Can_We_Actual...

5. Scanlan JP. Measuring health disparities. J Public Health Manag Pract 2006;12(3):293-296, responding to Keppel KG, Pearcy JN. Measuring relative disparities in terms of adverse events. J Public Health Manag Pract 2005;11(6):479–483: http://www.nursingcenter.com/library/JournalA...

6. Scanlan JP. Can We Actually Measure Health Disparities, presented at the 7th International Conference on Health Policy Statistics, Philadelphia, PA, Jan 17-18, 2008 (invited session): PowerPoint Presentation: http://www.jpscanlan.com/images/2008_ICHPS.pp... Oral Presentation: http://www.jpscanlan.com/images/2008_ICHPS_Or...

7. Scanlan JP. Measurement Problems in the National Healthcare Disparities Report, presented at American Public Health Association 135th Annual Meeting & Exposition, Washington, DC, Nov. 3-7, 2007: PowerPoint Presentation: http://www.jpscanlan.com/images/APHA_2007_Pre... Presentation: http://www.jpscanlan.com/images/ORAL_ANNOTATE...

8. Scanlan JP. Methodological Issues in Comparing the Size of Differences between Rates of Experiencing or Avoiding an Outcome in Different Settings, presented at the British Society for Populations Studies Conference 2007, St. Andrews, Scotland, Sept. 11-13, 2007: PowerPoint Presentation: http://www.jpscanlan.com/images/2007_BSPS_Pre... Presentation: http://www.jpscanlan.com/images/2007_BSPS_Ora...

9. Scanlan JP. Understanding the ways improvements in quality affect different measures of disparities in healthcare outcomes regardless of meaningful changes in the relationships between two groups’ distributions of factors associated with the outcome. Journal Review Aug. 30, 2007, responding to Sequist TD, Adams AS, Zhang F, Ross-Degnan D, Ayanian JZ. The effect of quality improvement on racial disparities in diabetes care. Arch Intern Med. 2006;166:675-681: http://www.journalreview.org/view_pubmed_arti...

10. Scanlan JP. Understanding patterns of correlations between plan quality and different measures of healthcare disparities. Journal Review Aug. 30, 2007, responding to Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ. Relationship between quality of care and racial disparities in Medicare health plans. JAMA 2006;296:1998-2004: http://www.journalreview.org/view_pubmed_arti...

11. Scanlan JP. Recognizing the way correlations between improvements in healthcare and reductions in healthcare disparities tend to turn on the choice of disparities measure. Journal Review Nov. 9, 2007, responding to Kaytur FA, Clancy CM. Improving quality and reducing disparities. JAMA 2003;289:1033-34: http://www.journalreview.org/view_pubmed_arti...

12. Scanlan JP. Effects of choice measure on determination of whether health care disparities are increasing or decreasing. Journal Review May 1, 2007, responding to Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ. Trends in the quality of care and racial disparities in Medicare managed care. N Engl J Med 2005;353:692-700 (and several other articles in the same issue):http://www.journalreview.org/view_pubmed_arti...

13. Scanlan JP. Correction to statements concerning the measurement of healthcare disparities in the National Healthcare Disparities Reports in earlier comment on Vaccarino et al. Journal Review Nov. 6, 2007: http://www.journalreview.org/view_pubmed_arti...

14. Scanlan JP. Comparing the size of inequalities in dichotomous measures in light of the standard correlations between such measures and the prevalence of an outcome. Journal Review Jan. 14, 2008, responding to Boström G, Rosén M. Measuring social inequalities in health – politics or science? Scan J Public Health 2003;31:211-215: http://www.journalreview.org/view_pubmed_arti... (version with properly formatted tables: http://www.jpscanlan.com/images/Bostrom_and_R...

15. Scanlan JP. Comparing health inequalities across time and place with an understanding of the usual correlations between various measures of difference and overall prevalences. Journal Review Jan. 30, 2008, responding to Moser K, Frost C, Leon D. Comparing health inequalities across time and place—rate ratios and rate differences lead to different conclusions: analysis of cross-sectional data from 22 countries 1991–200. Int J Epidemiol 2007;36:1285-1291: http://www.journalreview.org/view_pubmed_arti...

16. Scanlan JP. Changing social inequalities in SIDS. Am J Pub Health Dec. 11, 2005 (responding to Pickett et al. Widening social inequalities in risk for sudden infant death syndrome. Am J Pub Health 2005;95:97-81): http://www.ajph.org/cgi/eletters/95/11/1976

17. Sehgal AR. Impact of quality improvement efforts on race and sex disparities in hemodialysis. JAMA 2003;289:996-1000.

18. Massachusetts Medicaid Policy Institute. Pay-for Performance to Reduce Racial and Ethnic Disparities in Health Care in the Massachusetts Medicaid Program: Recommendations of the Massachusetts Medicaid Disparities Roundtable, July 2007: http://www.massmedicaid.org/pdfs/2007-7_dispa...