Study Points to Biases in Online Physician Ratings

Physicians have long argued that doctor rating websites provide biased information that does not accurately reflect the quality of care patients receive.

In a statement, the American Medical Association said, "Patient satisfaction is important to physicians, and patients should have access to credible information so they can be confident in their choice of physician, but public web forums have many shortcomings." It added, "Some allow postings to be published anonymously, and there is no guarantee that the opinions about a physician even come from that physician's patient. People may express dissatisfaction on these forums because they wanted a medication that wasn't medically necessary, or because they didn't receive a prescription or service that was delayed or denied by their insurance company."

Physicians might be partially right about biases in online ratings, but not in the way many of those doctors assumed.

A new study by researchers at the University of Maryland's Robert H. Smith School of Business and the University of Minnesota's School of Public Health examined the informativeness of online ratings about primary care physicians. Researchers found that low-quality physicians actually are less likely to be reviewed online. In fact, the majority of online physician ratings are very positive.

Guodong "Gordon" Gao -- an assistant professor at the University of Maryland and one of the study's authors -- noted that out of about 300,000 physicians ratings studied, half were given the highest score possible -- a five out of five.

So are physicians' concerns about patients relying on online ratings to make health care decisions -- much the same way they turn to such sites to make decisions about where to eat or book a hotel -- off base? Maybe, but maybe not.

Study Details

For the study -- titled, "The Information Value of Online Physician Ratings" -- researchers relied on several data sets, including:

• An offline patient survey conducted by consumer advocacy group Consumers' Checkbook;
• Online physician ratings collected from RateMDs.com;
• Information from the 2007 Economic Census on population and median income in the areas the physicians practice in; and
• State medical board websites for physician information, such as board certification, physician experience and specialty.

The study focused on general practitioners, or family care physicians. The final data set included 1,513 physicians who had been evaluated through offline surveys and 696 doctors who had been rated online. Researchers then compared physician scores on the offline survey with the online physician ratings.

Study Findings

The study found a correlation between a high offline survey rating and the probability of a physician being rated online. If a physician received a low score in the online survey, there was a low probability of him or her being rated online.

Gao said, "I think it comes from human nature. We are pretty reluctant to talk about negative things."

However, the study did find that the probability of being reviewed online increases dramatically for physicians rated in the extreme low end of the quality spectrum in the offline survey. In addition, patients who rate doctors online are more likely to exaggerate their opinions, according to the study.

Researchers say these findings show that the "sound of silence" effect is seen across the quality spectrum of physicians, while the "bad mouthing" effect is visible among very low-end quality doctors.

The findings also showed a strong correlation between the online ratings and patients' offline opinions, but the association was strongest for physicians with the lowest quality ratings. Therefore, online ratings identifying low-quality physicians tend to be the most informative.

Gao explained that if "a physician is exceptionally good and another physician is relatively good, both physicians will [have] a decent chance to get a 5 out of 5." He added, "The online ratings cannot really separate the very best from good."

On the other hand, physicians with negative ratings are not as clustered together. "There's a lot of variation in those ratings. So it's actually easy for us to tell which physician is worse than the other if they are both in the low end," Gao said.

Online Doctor Ratings Useful, But Proceed With Caution

AMA said, "Online opinions of physicians should be taken with a grain of salt and certainly not be the sole resource when looking for physician information. More reliable options are available like checking with your state medical board or asking trusted family and friends about their physicians. Patients can also ask their current physician for a referral." It added, "Choosing a physician is more complicated than choosing a good restaurant, and patients owe it to themselves to use the best available resources when researching a physician."

Although the study found potential biases in online physician ratings, Gao maintains that such information is still useful. He said that "patients really need this information," adding, "This is in high demand, so these websites provide a way for us to help each other sharing our wisdom of the crowd. I think it is a very good thing."

He noted that the study did find "a pretty strong correlation between the online ratings and quality. At the low end, it's much higher than at the high end, but overall the quality correlation is still there. So it has value."

Gao said the onus is on patients to use online ratings with caution. He said that patients need to "realize that these ratings are posted by people who are motivated to rate, which means that the ratings tend to be extremes -- either very negative or very positive."

He added that it is important to keep in mind that physician rating websites still are at an early stage. He noted that the average number of online ratings for each physician is three. Because the sample sizes are so small, the variation in the information can be high. Gao said, "Don't rely on one or two ratings to judge a physician."

Lastly, patients should consult information beyond online ratings when making a decision about a health care provider, Gao said. "Go and talk to the people surrounding you and look at other information, such as physician awards or whether a physician has a malpractice history."

He said that patients should "make judgments on comprehensive information, not just one or two ratings online."

How To Improve Online Rating Sites

There also are ways to improve the physician rating websites, themselves, according to Gao.

He said that "if the number of ratings [for a physician] is low, like one or two, you probably shouldn't put it on the website because then that physician's reputation is determined by one or two patients, [whose ratings] can be highly volatile."

Another physician rating website best practice is to give doctors the opportunity to respond to patients' reviews. Patients "want to hear both parties' arguments" he said, adding that allowing physicians to respond provides "a more complete picture."

In addition to patient ratings, websites should offer other data, such as objective quality measures from the federal government, according to Gao. He said that while such information is not as easy to understand, "there needs to be a way to integrate those different sources of information and make it easily digestible to people like us."

Future Research

Gao and his colleagues are conducting ongoing research to see if their findings about general practitioner ratings can be extended to other specialties.

He said he also plans to study whether online physician ratings "reflect true clinical quality."

Gao noted that the study used an offline patient survey to measure quality but that it also is important to examine whether clinical quality can be reflected in physician ratings.

He said, "There's a lot of debate here. A lot of physicians say that patients are not in a position to judge a physician's quality." Gao said he hopes future research can determine "to what extent patients can judge a physician's quality."

Read more: http://www.ihealthbeat.org/features/2011/study-points-to-biases-in-online-physician-ratings.aspx#ixzz1WtgdDcUE