Sunday, August 26, 2012

The Ground Continues to Shrink Under the Council of Canadians

It just keeps looking worse and worse for the Council of Canadians.

Frank Graves is a name that is extremely familiar to followers of Canadian politics. In 2010, Graves gave advice to then-Liberal leader Michael Ignatieff that he invoke a "culture war" between the Liberal Party and the governing Conservative Party. It did not end well for the Liberal Party. It was serious enough that Graves lamely tried to explain the entire thing away -- although he never really has.

So it should come as no surprise that Graves -- President of the EKOS polling firm -- is also instrumental in the Council of Canadians' bid to overturn the 2011 election results in seven ridings all won by Conservative candidates.

Apparently, Graves conducted a poll that concluded that, despite the absence of tens of thousands of such complaints, tens of thousands of Canadians -- opposition supporters in all -- were targeted by a highly-organized voter suppression effort.

As it turns out, however, Graves' poll has some very serious flaws in it. Reported, by all people, by Stephen Maher and Glen McGregor.

Ruth Corbin, the CEO of Toronto-based CorbinPartners, has found some very serious problems with Graves' poll. Central to Corbin's analysis is the use of an Interactive Voice Response (IVR) system which actually does nothing to confirm who is taking the survey. The survey could have been answered by absolutely anyone, including minors. IVR's tend to suffer from receiving very low response rates, posing serious problems for the scientific sampling of respondents, and making the survey very susceptible to a self-selection bias among respondents.

Some of Corbin's criticisms can be accounted for and explained away pretty easily. But this critique of IVR technology isn't. This becomes clear once you realize that Graves, EKOS, and Council of Canadians, for their own part, actually have no good answer this critique. Apparently the best they could do was enlist University of Toronto Political Science Neil Nevitte tp lamely complain that Corbin's critique was "not generous."

Wow. Is that it?

Look deeper than Nevitte's lame complaint. Maher and McGregor clearly didn't. Nevitte himself overlooks the presence of the self-selection bias in the Graves survey. He notes that “If the calls regarding the change in location of polling stations were random, then there should be no differences in the frequency with which people with different partisan inclinations would report that they were contacted.”

That would make sense. But having used an IVR system, what evidence does Graves have to show that his sampling was truly random? And what evidence does Graves have that months of media bluster about "misleading robocalls" leading up to his survey didn't taint the results? What guarantee does Nevitte have? After carefully considering both Corbin's argument, and Nevitte's counter-argument, the answer becomes immediately apparent: given their use of IVR, the answer is "absolutely none." While a poli sci prof may be able to overlook something like that, an industry professional like Corbin would not.

The more you dig into this Council of Canadians case, the less and less solid the evidence becomes. Remember that they have to show that not only were there misdeeds in the 2011 election, but that it likely affected the result.

The evidence they have is far from rock-solid. No wonder they've been trying to bend the rules while looking for more.


  1. in the spirit of fair play perhaps you could review our point by point response in the back of this affadavit. The design wasn't perfect as we acknowledged but the critique was full of errors. We would have done this study in the very same manner if the CPC had hired us to do it.

    Frank Graves

  2. I'm quite content to deal with the demonstrably salient points. As I explained, any number of Corbin's critiques very much can be explained away. But clearly even you can't deny that the use of IVR technology is a key weakness in this particular poll, as you haven't even attempted to do so.

    In fact, the complaint that Corbin's analysis is "not generous" is itself, IMO, an admission that there's far more to that particular critique than you're ostensibly attempting to let on.

  3. Patrick,
    We did actually respond extensively to this critique and we had included an annex dealing with IVR which was unfortunately dropped in the final version of the original report . This was not Ms. Corbin's fault but her depiction of IVR is inconsistent with the expert literature and our own experience and testing (as we noted in our response). The most prestigious survey expert organizations in the world (AAPOR and PEW) have concluded that IVR methods (properly applied) yield accuracy similar or better to live interviewer). Our own proof of concept testing , including a large experiment were appended to our response and shows similar favourable results. Ms. Corbin claimed that we had done no call backs, we had done up to seven and a minimum of four calls before retiring a number. She also claimed that we had not collected demographic data to test comparabilty to Census parameters; we did.
    We used a high quality random digit dialing sample provided by a very reputable sampling house. The sample was definitely a random probability sample with calculable margins of error (there are of course other sources of surver error). Ms. Corbin cites personal experience with a response rate of one tenth of one percent . This overstates non response by a factor of several hundered times . She also cites a non expert who refers to IVR as massively flawed while ignoring the comments of the top third party experts on this topic. I do agree that IVR suffers repuational issues in Canada (much less so in the USA) where it is a very popular methodology . No survey method is perfect but the IVR approach we applied definitely produced sound estimates of population parameters in the areas we studied. Thanks for the opportunity to respond