Are Fox News Polls Biased?

Especially after the outcome of the mid-term election, I think there is a common contention among some groups that there is something wonky about Fox News when it comes to reporting polls relative to President Obama and the Democratic Party in general.

Graph 1

Looking at the 795 polls (RealClearPolitics.com) thus far produced shows that there are 55 different pollsters, with the top 4 are Fox, CNN, Gallup, and Rasmussen (See Graph 1).  Taking a more detailed look at the top 4 we see that there seems to be a much larger proportional block of those who disapprove of the president in the Rasmussen Reports and Fox News relative to Gallup and CNN (Graph 2).  A similar pattern appears in the approval polls numbers.

Over time Obama's ratings have moved quite a bit.  It is hard to identify a clear pattern as far as one pollster being higher than another in the long run (Graph 3) though in the short run (Graph 4) it appears that CNN consistently reports high approval numbers and Rasmussen consistently reports low approval values.

Graph 2

When comparing polls among the top 4 there is no sense of objectiveness since there is no clarity as to which poll is closer to the objective truth if there is systematic bias in one or more of the pollsters.  However, what we can instead do is compare the polls to that of the average within each poll collection phase (say every 50 polls).  Removing the time average for every 50 polls we can see a bit more clearly how each pollster is behaving (Graph 5)

A pollster will be above the average 50% of the time and below the average 50% of the time if the pollster is unbiased and the average is unbiased.   We can see that for the polls looking at the average approval gap (Table 1, Column 3 and 4). In a similar manner by looking at how individual polls diverge from the average poll, we can take the average of this divergence.  If the poll is fair then the average gap will be close to zero.  If this gap is positive then it means on average the pollster reports polls levels of either approval or disapproval above the average poll value for the period.  If the average is negative then it means the pollster reported average is less than the polling average (Table 1, Column 1 and 2).

Graph 3

Looking at Table 1 we can see that CNN tends to over-report both approval and disapproval numbers relative to the typical poll.  CNN is more likely to over-report disapproval numbers.  Fox on the other hand
is prove to under-reporting both approval and disapproval numbers though Fox is much more likely to under-report approval numbers.  Gallup on the other hand is almost the mirror of Fox except that it tends to much more heavily under-report disapproval numbers rather than approval numbers. 

Graph 4

Finally, looking at the Rasmussen Reports it is clear that there is something fishy going on.  The average disapproval rating report is nearly 7 points above those of other polling locations while the approval rating is under-reported by nearly 2 points.  In addition, every poll that is put out by Rasmussen has reported 100% of the times disapproval rates above the average.  This is an astounding number since it is clearly highly unlikely.

However, since I am not an expert in polling at all, I am not sure if Rasmussen is doing something legitimate and well known.  It is very clear to me from the data however that they are not to be considered equivalent to the other three major polling agencies.  Looking at wikipedia, it is well understood that Rasmussen has a different type of polling by selecting individual with a plan to only capture "likely voters".

Graph 5

Overall it looks like Fox is off the hook.  Table 2 shows them slightly under-reporting approval rates while CNN is over-reporting.  The difference in disapproval rates though is not statistically significant for Fox though it is for CNN as well (Table 3)

Find the code to produce all of these results at:
https://github.com/EconometricsBySimulation/2013-05-30-FoxNews/blob/master/analysis.r

Table 1: Averages and counts

	Average Gap		Count
	Approve	Disapprove	Approve	Disapprove
CNN/Opinion Research	1.23	1.90	0.63	0.69
FOX News	-1.40	-0.31	0.31	0.44
Gallup	-0.41	-1.53	0.38	0.34
Rasmussen Reports	-1.86	6.73	0.32	1.00

Table 2: T-tests of Approval Gap

	Estimate	Std. Error	t value	Pr(>|t|)
PollsterCNN/Opinion Research	1.232823	0.471684	2.613664	0.009437
PollsterFOX News	-1.40239	0.440775	-3.18165	0.001627
PollsterGallup	-0.41156	0.415242	-0.99114	0.322465
PollsterRasmussen Reports	-1.85953	0.431748	-4.30697	2.28E-05

Table 3: T-tests of Disapproval Gap

	Estimate	Std. Error	t value	Pr(>|t|)
PollsterCNN/Opinion Research	1.896855	0.572502	3.313273	0.001042
PollsterFOX News	-0.31254	0.534987	-0.58419	0.559557
PollsterGallup	-1.52794	0.503997	-3.03164	0.002658
PollsterRasmussen Reports	6.7325	0.524031	12.84753	4.51E-30