Making sense of congenital heart surgery data

Conquering CHD is committed to empowering patients and families. Patients and families deserve essential information to help make important health care decisions. As this data becomes available it can be very hard to understand.  It is our hope that this resource can help.  For a .pdf version of this document, click here.

When your child has congenital heart disease, at times it can feel like there are so many decisions to make, especially, when faced with surgery or another medical procedure.  During these times of stress, it is important to partner with your child’s medical team.

One key decision you may be thinking about is where your child receives care. More and more information about different hospitals is now available to the public for you to review.  However, understanding this information can be a challenge.

Much of the information to look at includes several different numbers.  Our goal is to describe what some of these numbers mean, offer things to think about when reviewing this information, and encourage you to use publicly available data as “one piece of the puzzle” when making health care decisions.

Above all, please use this information in a conversation with your child’s medical team when making important decisions.

 

A little bit of background

Understanding congenital heart surgery information that is available to the public can be very complicated, even for people who work with the numbers all the time.  We realize you may find that some of this information is difficult to understand.  It is important information and we encourage you to talk with your child’s medical team to help you.

Here, we are going to focus on explaining one specific set of information reported by the Society of Thoracic Surgeons (STS). Of all programs involved in reporting congenital heart surgery information to the public, the STS program is currently the largest.  The information from the STS is also used by other groups who report congenital heart surgery data to the public, including Consumer Reports and US News & World Report.

The STS routinely collects multiple pieces of information about congenital heart surgery to form a database.  This information is often called “data” and includes a variety of different numbers along with other information.

Approximately 120 hospitals around the country who perform congenital heart surgery provide information to the STS database. In 2015, the STS began sharing some of this information on their website for those hospitals that agreed to share their information with the public.

The reporting website can be found at:  http://publicreporting.sts.org/chsd.  This information is updated once a year.  Details from the STS about the information contained on the website can be found here.

The STS data

The STS reporting website lists several pieces of information for each hospital participating in public reporting. As you read on, we’ve tried to help you understand what some of the words and numbers mean that you will find on the website. Some of this information may be easy to recognize and understand.  Some of it is more complicated.

In addition to the information below, we have also developed an appendix that can be used with this guide that may provide additional helpful information.

Surgeons

The name of each surgeon at the hospital who performs congenital heart surgery was listed on previous versions of the STS website, but is not listed in the current version.  There is a link to the hospital’s website where you may find some of this information.  You may also want to inquire about the surgeon’s board certification and level of experience.  In addition, it is important to remember that the care of patients with congenital heart disease involves multiple team members working together including the surgeon. You can use our guided questions tool as a reference for other questions you may want to ask about the care team.

Outcomes

Outcome means the result of the surgery.  For each hospital, information is displayed on deaths after surgery.  These include all deaths that occur during the hospitalization in which the surgery is performed, or death that occurs in any location within 30 days of surgery. All deaths are included – the cause doesn’t matter. Each hospital uses the same criteria, which means they cannot pick and choose which deaths to report.  The STS periodically checks the information to make sure they are not missing any deaths; about 10% of the hospitals participating in the STS Congenital Heart Surgery Database are randomly selected each year to have their data checked or audited.

The STS is also working on developing methods to report other types of important outcomes such as complications after surgery, and the number of days a child stays in the hospital after surgery.

Timeframe

The information is gathered about all surgeries that took place at a hospital during a span of the last four years.  Although you may want to see only the most recent numbers, like over the past month or past year, this can have some downsides. Because only a small number of surgeries may be performed over a short period of time by any one hospital, comparing these small numbers may make it seem like there are no real differences. Research studies have shown that putting the numbers together into 4-year timeframes gives the best chance that any differences will become easier to see.

You may want to ask whether the hospital you are considering has had any major changes recently (like changes in surgeons, cardiologists, the unit patients are treated in, etc.), and if so, work with your child’s healthcare provider to understand some of their most recent data.

Data Presentation

The information is presented in two ways on the website:

1. Overall: Overall numbers for all of the congenital heart surgeries.
2. Separate STAT Categories:  Separated into categories of surgeries as described below.

STAT categories

It would be great if the outcome information we are talking about were reported for each specific heart defect or type of surgery, but this can be challenging.  Because there are so many kinds of heart defects, a hospital may not see many of the exact same type of defects in a given year. These small numbers can be hard to understand or compare as described in the section above.  Combining operations together into larger groups or categories (the STAT categories) can help.

To help organize surgeries into categories based on how risky or complex they are, the STS places each surgery type into one of five “STAT” categories. Category 1 includes surgeries with the lowest risk of death, and category 5 includes the surgeries with the highest risk of death. STAT is short for:  Society of Thoracic Surgeons – European Association for Cardio-Thoracic Surgery. These categories were created after looking at 77,294 heart surgeries across North America and Europe.

A brief listing of common types of surgeries included in each STAT category is included in the table below.  More detail can be found in this article:  O’Brien SM, Clarke DR, Jacobs JP, Jacobs ML, et al. An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg. 2009;138:1139-53.

 

Common types of surgery within each STAT Category
STAT Category 1	atrial septal defect repair, ventricular septal defect repair, coarctation repair, subaortic stenosis resection, pulmonary valve replacement, conduit replacement
STAT Category 2	Tetralogy of Fallot repair, Fontan operation, Ross operation
STAT Category 3	Hemi-fontan operation, arterial switch operation, complete atrioventricular septal defect repair
STAT Category 4	aortic arch repair, arterial switch operation with ventricular septal defect closure, heart transplant, aorto-pulmonary shunt, total anomalous pulmonary venous return repair, truncus arteriosus repair
STAT Category 5	Norwood (stage I) operation, hybrid stage 1 procedure, double switch operation, truncus arteriosus with interrupted aortic arch repair

Mortality Data

Let’s take a look at how the information about death (mortality) is displayed on the STS website:

#/Eligible and Observed:

The first column of numbers on the website shows the following numbers for each hospital:

# – total number of deaths during the 4-year timeframe.

Eligible – total number of surgeries performed during the same timeframe.

This number is particularly important because it tells us the volume or number of surgeries the hospital performs.  See below and the appendix for more information about how to understand this information and why the volume of surgeries a hospital performs is an important piece of information to consider in addition to the number of deaths or mortalities.

Observed – the rate of death calculated by dividing the two numbers above:  the number of deaths divided by the total number of eligible surgeries.

Even though the numbers in this calculation are pretty easy to understand, they should not be used to label the quality of a congenital heart hospital. This is because the types of congenital heart disease and surgeries performed, along with important characteristics of the patients (such as whether they were born prematurely or have some type of genetic syndrome or other medical problem besides their heart disease) can be very different across hospitals. Some hospitals treat a lot of patients with very complex or severe heart disease and other conditions, while other hospitals treat fewer of these patients.  This is known as having a different “case mix”. If you compared only the simple calculations above, some hospitals could have a higher rate of death just because they take care of more complicated and complex patients, not because they are providing lower quality or worse care.

So, in order to better understand how a hospital is doing, more advanced methods are necessary.

Expected: 

This column lists the expected rate of death.

This number is calculated from a math formula that is used to predict how many deaths we could expect to see at a hospital when taking into account the severity or complexity of patients they treat. This formula uses information like the type and complexity of the congenital heart surgery performed, and the characteristics of the patient such as how old they are, how much they weigh at the time of surgery, whether the patient has a genetic syndrome or other medical problems, etc.

Importantly, hospitals don’t get to pick their own expected rate, it is assigned for all hospitals based on the information in the math formula created by STS.

The formula is not perfect. It does not take into account every important factor that relates to how complex or sick a child may be at the time of surgery, but it is the best one that has been developed to date and it has been improved and updated over time.

O/E (95% CI):

This column lists what is known as an observed/expected (O/E) ratio and a 95% confidence interval.

Observed/Expected (O/E) ratio – This ratio is calculated by using the data described above, and dividing the number of observed or actual deaths at the hospital by the number of expected deaths from the math formula (based on how complex or sick the patients at that hospital are).

If the O/E is 1 – A ratio of 1 means the hospital has a similar number of deaths to what would be expected for the type and complexity of patients that they treat (their case mix)

If the O/E is less than 1 – A ratio less than 1 means the hospital had fewer deaths than what would be expected for the type and complexity of the patients that they treat. Thus, a lower O/E ratio is better.

If the O/E is greater than 1 – A ratio greater than 1 means a hospital has more deaths than what would be expected for the type and complexity of the patients that they treat.

95% Confidence Interval (95%) – The O/E ratio is displayed with what is known as a 95% confidence interval, which is very important. The calculations from the math formulas described above are complicated, and it is difficult to be completely certain about what the exact O/E ratio is. The confidence interval shows the range of numbers where the O/E ratio could fall for each hospital. This range is just as important as the O/E value itself.  Taken together, the O/E ratio and 95% confidence interval is believed to be one of the most important pieces of information displayed on the website.

Adj. Rate (95% CI):

This piece of information can be challenging to interpret.  It represents the adjusted mortality rate with 95% confidence interval.  It is calculated from a math equation using some of the information described in more detail above.  It represents what the hospital’s rate of death would be if their results were in theory applied across the entire population of patients within the STS Congenital Heart Surgery Database.

Star rating

In an effort to help make it easier to understand the information we’ve been talking about, STS has given each hospital a star rating ranging from one to three stars. This rating is based on the O/E ratio and 95% confidence interval.

One star = More deaths than expected.  These hospitals have more deaths than would be expected based on the type and complexity of patients they treat (their case mix).  Their O/E ratio and entire 95% confidence interval are greater than 1.

Two star = Deaths similar to expected. These hospitals have a rate of death similar to what would be expected based for the type and complexity of patients they treat. Their O/E and 95% confidence interval overlaps with 1.

Three star = Fewer deaths than expected.  These hospitals have fewer deaths than would be expected for the type and complexity of patients they treat. Their O/E ratio and entire 95% confidence interval are less than 1.

Example –

Star rating                   Example of O/E ratio (95% CI)

1 star                                       1.4 (1.1 – 1.6)

2 star                                       0.8 (0.7 – 1.1)

3 star                                       0.6 (0.5 – 0.9)

It is important to note that based on the definitions of the star categories described above and the way the math formulas work, the vast majority of hospitals fall into the two star category. This is talked about in more detail below.

What does all of this mean?

This information can be hard for families and even providers to understand. There are a few key points to be aware of:

• Not all hospitals share their data. Of the hospitals currently participating in the STS Congenital Heart Surgery Database, about half currently agree to share their information on the STS public reporting site (see the appendix for more details). If you are considering a hospital that does not report their information you should ask them why. They likely still send their data to the STS Database, but have decided to not share it publicly. You can work with your child’s medical team to ask for the same information that would be posted to the STS website.

• Remember the timeframe of reporting. As described above, there are several reasons why the 4-year timeframe for reporting the information was chosen. You may want to ask whether the hospital you are considering has had any major changes recently, and if so, work with your child’s healthcare provider to understand some of their most recent data.

• Not all outcomes are reported. Current public reporting efforts focus on mortality (death) that occurs in the short-term after surgery. Efforts are ongoing to develop the methods necessary to share information on complications that occur after surgery, number of days in the hospital, and other longer-term outcomes. In the meantime, you should talk about other outcomes that are important to you with your child’s healthcare team and work with them to understand any available information.

• The raw numbers may be easier to understand but they SHOULD NOT be used to judge the quality of a congenital heart hospital. The raw data (#/Eligible and Observed) do not consider the severity or complexity of patients the hospital treats. Some hospitals may have higher mortality rates simply because they take care of sicker or more complex children.

• Use the information that takes into account the type and complexity of patients the hospital treats. The additional information calculated from the math formulas described above is much better to use in understanding how a hospital is performing compared to what would be expected for the type and complexity of patients that they treat. The math formulas allow the calculation of the O/E ratios and star ratings.

• Look beyond the star ratings. There are a few things to know about the star ratings calculated from the math formulas:

Small changes can influence a hospital’s star rating. Any time you group hospitals into categories, a few more or less deaths can cause the hospital to change from one category or group to another. The same is true for the star rating system.  For example, in one reporting period a hospital had an observed mortality rate of 3.7% with an O/E ratio of 0.84 (95% confidence interval 0.66-1.05) resulting in classification as a 2-star center. The year prior, the observed mortality rate was 3.8% with an O/E ratio of 0.8 (95% confidence interval 0.6-0.9) resulting in classification as a 3-star center.

The majority of hospitals are classified in the 2-star category. Because of the way the math formulas used to calculate the star rating work, the majority of hospitals are classified within the 2-star category. There may be important differences between some of these hospitals that you don’t see when you look only at the star ratings.

Use the more detailed information on the STS website. Due to the reasons above it is important to look beyond the star rating to more of the detailed information presented on the STS website. Remember that lower O/E ratios are better, but also that small differences in the O/E ratios themselves may not be important. The confidence intervals around the O/E ratio are just as important to consider because they describe the range of values where the O/E ratio is likely to fall.

• The information on the website can’t necessarily be used to directly compare two hospitals without knowing more of the story. This point is critical to understand.  The math formulas used to calculate the information such as the O/E ratio and star ratings tell us how a hospital is performing compared to what would be expected specifically for the type and complexity of patients they treat (case mix).

For example, a hospital can receive a 3-star rating by doing well in caring for a high-complexity patient population (very sick/complex patients), doing well in caring for a low-complexity patient population, or somewhere in between.

A hospital who does well in treating lower complexity patients can’t really be directly compared to a hospital who treats higher complexity patients (sicker or more complex children). It can’t be assumed that this first hospital would have the same success if they were treating the more complex or sicker patients that the second hospital treats.

For this reason, it is important to understand not just the information about the outcomes (the O/E ratios, star ratings, etc.) but also the type and complexity of the patients that the hospital treats in order to make the best judgement.

This is really important especially when deciding where to seek care for a child with more complex heart disease. In this situation, you may want to understand not only which hospitals have good results but whether they achieved these results through taking care of a high complexity patient population. 

 

• Understanding the type and complexity of patients a hospital treats. So how do you understand this important piece of the information – the type of patients that a hospital treats? There is no “gold standard” or best way to do this that has been looked at in any research studies to date, and it can be challenging to understand based on the data currently presented on the website.  Here is one way to begin to understand this information:

You can look at the number or volume of patients a hospital treats in the high complexity or high risk STAT categories (STAT categories 4 and 5). Remember this information is found under the “Eligible” column described above. A higher number means the hospital treats more of these types of cases.  You can also understand a hospital’s results or mortality information (O/E ratio and 95% confidence interval) specifically for these higher complexity operations by looking at the information presented in these rows of the table for each hospital on the STS website.

In addition to helping to put the information above, such as star ratings, in better context, it is also important to note that prior research studies have shown that in general that hospitals that perform a higher number of operations generally have better results, particularly for children who have higher complexity types of heart disease. This is known as the “volume-outcome” relationship.

• More information, including the range of values from hospitals currently reporting on the STS website, can be found in the appendix.

How can I use this data?

Publicly reported information can be a useful tool when making health care decisions. However, it is important to remember that the information shared, as well as this guide, cannot by itself tell families where they should seek care for their child.

There are often many things that impact a family’s healthcare decisions and this information is best used as just one tool to help to make those decisions.

We recommend reviewing the publicly reported STS data and the key points included in this guide with your child’s healthcare team as a first step.  Don’t be afraid to ask questions until you understand how this information impacts your child.

For more information, visit www.conqueringchd.org.

Authors:

Sara K. Pasquali MD MHS¹, Jeffrey P. Jacobs MD², Michael Kim PhD³, Amy Basken MS⁴

¹ University of Michigan, C.S. Mott Children’s Hospital, CCHD Medical Advisory Board, ² Johns Hopkins University, Johns Hopkins All Children’s Hospital, CCHD Medical Advisory Board, ³ Parent, CCHD Steering Committee, ⁴ Parent, CCHD Director of Programs

Acknowledgements:

CCHD would also like to thank editors David Kasnic¹ and Nicolas Madsen MD MPH²

¹Parent, CCHD Executive Director; ²Cincinnati Children’s Hospital and Medical Center, CCHD Medical Advisory Board Vice-Chair