CBC hospital rankings draw condemnation and praise

A special report from CBC's the fifth estate ranking Canadian hospitals is receiving a mixed reaction from hospital officials and medical associations, but many experts are saying measuring success matters for hospital improvement.

In an interview with CBC’s Mark Kelley, the Ontario Hospital Association's CEO and president, Pat Campbell, criticized the scoring system, saying the group had some "significant concerns with the methodology."

"Patients providing commentary on hospitals, CBC providing commentary on hospitals, that's all fair game. What we’re concerned about is patients relying on this information to make decisions on when and how to access care when it’s not necessarily grounded in good evidence," she said.

• Vist CBC's Rate My Hospital website
• Inside Rate My Hospital: How we measured performance

Rate My Hospital assessed 239 facilities based on data reported by hospitals to the Canadian Institute for Health Information. CBC was advised on the selection and use of CIHI's data by a five-member expert panel. 

Rate My Hospital rated facilities only against others of the same size, as defined by CIHI. CBC rated hospitals on performance indicators that measure patient results after common medical and surgical procedures.

Methodology misunderstood

Most hospitals received a B grade, indicating their outcomes in measured areas are similar to those of a typical hospital of the same size. Ten hospitals showed substantially better outcomes, 20 demonstrated better outcomes, 34 scored lower outcomes and eight displayed substantially lower outcomes.

CIHI uses 21 clinical and six financial indicators to measure hospital performance. CBC used five of those indicators for its study, looking at rates of:

• Death after major surgery.
• Surgical and medical patients who experienced unexpected complications tied to nursing care during a hospital stay.
• Patients who were readmitted following surgery or medical treatment.

Vancouver Coastal Health, which manages Vancouver Hospital and Health Sciences Centre, a facility that received a D ranking, echoed OHA's methodology concerns in a statement. The health authority disagreed with CBC's decision to use five of CIHI's clinical indicators and said "basing an overall hospital rating on a very limited number of indicators is misleading, and the results can be highly skewed."

• Find and rate your hospital

But  B.C. Health Minister Margaret MacDiarmid said she hoped health authorities would use the findings to improve.

'The important information is what we do with it. So, drilling down into some of those results and saying why are the results different, what's it telling us and what do we need to do about it?' — Maura Davies, CEO and president of Saskatoon Health Region

"The Ministry of Health and the health authorities should take this very seriously and look at it carefully and see if there is a lesson there for us, or are there areas we're not already focused on in terms of quality improvement that we should be focusing on," she said.

In Saskatchewan's Sunrise Health Region, where Yorkton Regional Health Centre received a D, spokeswoman Sharon Tropin said the CBC's rating tool could have used more than five indicators. "Why these five indicators were used to the exclusion of others is not understood," she said.

OHA's Campbell agreed: "Hospitals are very large very complex organizations, and to be rating anything on the basis of five measures is incomplete at best and potentially misleading at worst."

• Read about how the fifth estate came up with its ratings

CBC's expert panel addressed these issues during the development of the evaluation system, noting many of CIHI's indicators focus on specialties not offered by all hospitals, such as obstetrics and cardiac care. CBC limited its rankings to the five measures that assessed the quality of general surgical care and medical treatment only.

Including specialty services would have significantly reduced the number of hospitals CBC could have ranked.

Focus on improvement

Maura Davies, the CEO and president of Saskatoon Health Region, where three hospitals scored A+, B and C, disagreed with CBC's methodology. However, she said, the rankings are helpful because they increase awareness that measurement matters.

"The important information is what we do with it," said Davies. "So, drilling down into some of those results and saying why are the results different, what's it telling us and what do we need to do about it?"

Public hospital rankings are intended to drive change.

They can "focus hospitals' and clinicians' attention on important quality of care issues that need improvement," said Jack Tu, a member of CBC's expert panel advising the Rate My Hospital project. Tu is a cardiologist at the Schulich Heart Centre at Sunnybrook Hospital in Toronto.

In the rankings, Windsor Regional Hospital scored a B, demonstrating an average performance for a hospital its size.

"We're not pleased with just being average. We feel we're better than average and we can do a lot better," said hospital CEO David Musyj.

The hospital has already proved low rankings can spur it into change.

Windsor Regional and another local hospital underwent a structural overhaul a few years ago after ranking well below average on patient mortality, said Musyj.

"If you would have [done] this report five or six years ago, I bet we would have been a C or a D," he said. "Maybe next time, we'll be at an A or A+. I'm confident we'll get there."

To contact the Rate My Hospital team with tips or information related to the series, please email [email protected].