Thursday, May 30, 2013

Are Fox News Polls Biased?

Especially after the outcome of the mid-term election, I think there is a common contention among some groups that there is something wonky about Fox News when it comes to reporting polls relative to President Obama and the Democratic Party in general.

Graph 1
Looking at the 795 polls (RealClearPolitics.com) thus far produced shows that there are 55 different pollsters, with the top 4 are Fox, CNN, Gallup, and Rasmussen (See Graph 1).  Taking a more detailed look at the top 4 we see that there seems to be a much larger proportional block of those who disapprove of the president in the Rasmussen Reports and Fox News relative to Gallup and CNN (Graph 2).  A similar pattern appears in the approval polls numbers.

Over time Obama's ratings have moved quite a bit.  It is hard to identify a clear pattern as far as one pollster being higher than another in the long run (Graph 3) though in the short run (Graph 4) it appears that CNN consistently reports high approval numbers and Rasmussen consistently reports low approval values.

Graph 2
When comparing polls among the top 4 there is no sense of objectiveness since there is no clarity as to which poll is closer to the objective truth if there is systematic bias in one or more of the pollsters.  However, what we can instead do is compare the polls to that of the average within each poll collection phase (say every 50 polls).  Removing the time average for every 50 polls we can see a bit more clearly how each pollster is behaving (Graph 5)

A pollster will be above the average 50% of the time and below the average 50% of the time if the pollster is unbiased and the average is unbiased.   We can see that for the polls looking at the average approval gap (Table 1, Column 3 and 4). In a similar manner by looking at how individual polls diverge from the average poll, we can take the average of this divergence.  If the poll is fair then the average gap will be close to zero.  If this gap is positive then it means on average the pollster reports polls levels of either approval or disapproval above the average poll value for the period.  If the average is negative then it means the pollster reported average is less than the polling average (Table 1, Column 1 and 2).
Graph 3

Looking at Table 1 we can see that CNN tends to over-report both approval and disapproval numbers relative to the typical poll.  CNN is more likely to over-report disapproval numbers.  Fox on the other hand
is prove to under-reporting both approval and disapproval numbers though Fox is much more likely to under-report approval numbers.  Gallup on the other hand is almost the mirror of Fox except that it tends to much more heavily under-report disapproval numbers rather than approval numbers. 

Graph 4
Finally, looking at the Rasmussen Reports it is clear that there is something fishy going on.  The average disapproval rating report is nearly 7 points above those of other polling locations while the approval rating is under-reported by nearly 2 points.  In addition, every poll that is put out by Rasmussen has reported 100% of the times disapproval rates above the average.  This is an astounding number since it is clearly highly unlikely.

However, since I am not an expert in polling at all, I am not sure if Rasmussen is doing something legitimate and well known.  It is very clear to me from the data however that they are not to be considered equivalent to the other three major polling agencies.  Looking at wikipedia, it is well understood that Rasmussen has a different type of polling by selecting individual with a plan to only capture "likely voters".
Graph 5

Overall it looks like Fox is off the hook.  Table 2 shows them slightly under-reporting approval rates while CNN is over-reporting.  The difference in disapproval rates though is not statistically significant for Fox though it is for CNN as well (Table 3)

Find the code to produce all of these results at:
https://github.com/EconometricsBySimulation/2013-05-30-FoxNews/blob/master/analysis.r


Table 1: Averages and counts
Average Gap
Count
Approve
Disapprove
Approve
Disapprove
CNN/Opinion Research
1.23
1.90
0.63
0.69
FOX News
-1.40
-0.31
0.31
0.44
Gallup
-0.41
-1.53
0.38
0.34
Rasmussen Reports
-1.86
6.73
0.32
1.00


Table 2: T-tests of Approval Gap

Estimate
Std. Error
t value
Pr(>|t|)
PollsterCNN/Opinion Research
1.232823
0.471684
2.613664
0.009437
PollsterFOX News
-1.40239
0.440775
-3.18165
0.001627
PollsterGallup
-0.41156
0.415242
-0.99114
0.322465
PollsterRasmussen Reports
-1.85953
0.431748
-4.30697
2.28E-05


Table 3: T-tests of Disapproval Gap

Estimate
Std. Error
t value
Pr(>|t|)
PollsterCNN/Opinion Research
1.896855
0.572502
3.313273
0.001042
PollsterFOX News
-0.31254
0.534987
-0.58419
0.559557
PollsterGallup
-1.52794
0.503997
-3.03164
0.002658
PollsterRasmussen Reports
6.7325
0.524031
12.84753
4.51E-30

12 comments:

  1. Thanks for sharing. Nice use of graphics.

    ReplyDelete
  2. Shouldn't your headline say "Fox news polls are not biased"? That seems to be your conclusion. Or you could have made a headline that says "Are CNN news polls biased?", since your last paragraph says they over-report. To make a headline that suggests one thing, then say the opposite in your last paragraph, seems misleading to me.

    ReplyDelete
    Replies
    1. Good point! Though as far as CNN goes, I think they have the most centric of the polls since they are both over-reporting approval ratings and over-reporting disapproval ratings in equal regards. It is hard to argue that they are systematically biased. As for the headline, I am not really arguing that Fox is not biased.

      I have been waiting for someone to say well, what would happen if you removed the big 4 from the data then estimated the difference and I am pretty sure without the Rausmussen Reports moving the mean substantially then Fox would look pretty biased. But I stated my method and I tested it and so I wrote it up as I found the data.

      Delete
    2. I'd say the title is fine. It peaks interest and often people I encounter say Fox is biased so it's in a way addressing a quasi research question. Your title probably causes more to click and read than to say "CNN is biased" and asking it in the form of a question, rather than stating "Fox is not biased", makes me curious and I dove in. I'd say nice job with the title. Again thanks for sharing.

      Delete
  3. Rasmussen is well known as right-wing biased (and the owner is an evangelical Christian). They shape the question or lump results to get what they want. When George Bush was president, their results showed much higher approval ratings than any other poll. Now that a Democrat is President, they shape the question to get the lowest approval. During elections, they typically show the Republican in the lead until right before the election when they show results consistent with other surveys. They are not a survey outfit, they are propagandists.

    ReplyDelete
    Replies
    1. I was honestly completely unaware. I was just interested in seeing what I could find with the RealClearPolitics data.

      Delete
    2. Is that why Rassmussen was rated the most accurate polling firm during the 2008 elections?

      Delete
    3. Actually, after all the presidential votes were counted, it appears that Rasmussen was rated 10th out of 20 in 2008 (http://www.fordham.edu/images/academics/graduate_schools/gsas/elections_and_campaign_/2008%20poll%20accuracy%20panagopoulos.pdf) and 25th out of 28 in 2012 (http://www.fordham.edu/images/academics/graduate_schools/gsas/elections_and_campaign_/poll_accuracy_2012_presidential_election_110712.pdf).

      Delete
  4. Great post and plots. Minor note: you can upload the data directly to the GitHub repo, and open it with something like downloader::download().

    ReplyDelete
    Replies
    1. Really good point. I had assumed it was so but I could not figure out how to do it. If you had some sample code that would be great!

      Delete
  5. No problem:

    1) Move the CSV file into the same folder as your R script

    2) Synchronize the repo to upload the file.

    3) Install Winston Chang's downloader package to deal with HTTPS links:

    install.packages("downloader", quiet = T)

    4) Change the top of your code to:

    require(downloader)
    link = "https://raw.github.com/EconometricsBySimulation/2013-05-30-FoxNews/master/2013-05-29-approval.csv"
    file = "approval.csv"
    download(link, file)
    approval.rating <- read.csv(file, stringsAsFactors = FALSE)

    Hope this helps! If it does not work, I'll clone your repo and provide you with a full-fledged example. I have more on the topic in my R course.

    ReplyDelete
  6. Guys from the SmarterPoland foundation prepared a nice R/Shiny webapp that does a similar thing with political polls in Poland. The app is here: http://glimmer.rstudio.com/sondaze/demo1/ (in Polish, but there's not a lot of text—I guess it will be bearable with some Google Translate magic ;-); tabs are political parties, colors denote different public opinion agency). They also found that some polls seem to misreport data.

    ReplyDelete