The battle between Daily Kos and pollster Research 2000 went from ugly to surreal last week, as the website and its founder, Markos ("Kos") Moulitsas, filed suit and pollster Del Ali fired back with a lengthy, frequently rambling reply to TPM's Justin Elliot. The Washington Post's Greg Sargent points out that the coming lawsuit could provide an "unprecedented look at the inside of a professional polling operation." I would argue that it already has, although "professional" is not necessarily the adjective I would choose.
Consider what we have learned in just the past few days. From Elliot's reporting, for example, we know that the Olney, Maryland address listed on the Research 2000 website is a post office box and the company "does not appear as incorporated on the state business records database." Ali told Elliott "he incorporated with 'self-proprietorship' in 2000."
That profile fits the description of his business that Ali gave the Baltimore Daily Record in 2006 (h/t Harry Enten). The article described Research 2000 as having just "three part time employees" and being "quite a bit smaller" than Mason-Dixon Polling & Research Inc., where Ali had worked until starting his own business in late 1998. "The actual legwork" of his business, the article said, "is farmed out to professional call banks." Ali also claimed that his firm did considerable non-media work:
Ali said diversification - working with interest groups as well as media - is important for business, and that there is a misconception surrounding polling contracts with large news agencies. We don't make a great deal of money, Ali said. If I were to depend on making ends meet with media polls, then I'd be broke.
Running a mom-and-pop polling business is not incriminating, in and of itself. Many research companies, including my former business, are small shops that depend entirely on third-party call centers to conduct live telephone interviews. But it's important to consider the Research 2000 profile in terms of the sheer volume of work it claims it did for Daily Kos, especially given that Ali told me, as recently as four weeks ago, that his work for Kos and progressive PACs was "less than 15% of our overall business." If that is true, the volume of surveys that Research 2000 farmed out to call centers over the last two years was extraordinary. Several very large call centers would have been involved. Why have we heard nothing from them?
From Yahoo Politics' John Cook we learn that court records show Ali "has been sued numerous times in his home state of Maryland for nonpayment of debt and has been hit with several tax liens," including a $2,360 lien just two months ago. Cook also notes that Ali and his company were sued eight years ago for $5,692 for non-payment to "polling and research company" RT Nielson Company (now known as NSON). Their website confirms that NSON "specializes in telephone data collection" and provides these services "to many market and opinion research consulting companies."
Maryland Court records also show a judgment against Del Ali for $5,714.09 from a suit filed by Zogby International in 2001 (document obtained via search here). So we do have documentation to show that Research 2000 was doing business with research companies and survey call centers, albeit eight to ten years ago.
The Daily Kos complaint, published by Greg Sargent, provides new information on the financial side of the polling partnership from the Daily Kos perspective. Shortly after the 2008 elections, for example, Daily Kos entered into an agreement "reached orally" to conduct 150 polls for the website over the following year, including a weekly national survey and various statewide polls to be conducted "as requested." Kos agreed to make an initial payment in late 2008 and two "lump sum" payments in 2009. The complaint implies (though does not state explicitly) that the parties agreed to either a total amount or a set cost per poll (or both).
The complaint goes on to explain that Kos agreed to advance the second lump sum payment to May 2009 -- right around tax time -- in exchange for an additional 59 polls "to be performed free of charge." Ali requested the advance and offered the free polls in exchange, according to the filing, "claiming it would provide 'immense' help for cash flow reasons."
What the Daily Kos complaint omits is any discussion of the dollar amounts involved. Just how much did they pay for hundreds of thousands of interviews conducted over the last two years? What was the typical cost per interview (especially when we include those 59 free surveys)? The answers to those questions alone will tell us whether Research 2000 could have plausibly conducted live telephone interviewing on such a large scale. As both Patrick Ruffini and Nate Silver have speculated, Ali appears to have been charging absurdly low prices given the likely budget of a site like Daily Kos and the realities of the costs of farming out live interviewing.
Moreover, the financial arrangement described in the complaint -- pre-negotiated lump sums for hundreds of surveys with no written contract -- is also extraordinary. Telephone interviewing costs vary considerably depending on the number of interviews, the length of the questionnaire, the incidence of the target population (how many non-registrants or non-voters need to be screened out) and several other factors. I know of no call center that would agree to field a survey without an advance bid based on precise specification of all of these variables. Given this potential variability and the relatively low profit margins typically involved, pollsters, call centers and their clients are usually careful about nailing down the specifications in advance. The idea that a pollster would propose conducting 59 free polls as a means of obtaining, as Nate Silver puts it, a short-term loan with an "alarmingly high interest rate," is simply unheard of.
While the story told by the Kos filing was strange, the controversy grew even more surreal after Ali "lashed out" at Moulitsas and others in a rambling 1,100-word statement sent via email to TPM's Justin Elliot on Thursday. Ali claims in his statement that the Daily Kos complaint contains "many lies and fabrications," that "every charge against my company and myself are pure lies, plain and simple," and that Kos still owes him a "six figure payment."
Ali promises to "expose" the alleged mistruths "in litigation, not in the media" and says calls by the National Council of Public Polls (NCPP) and others to "just release the data and explain your methodology" indicate a bias toward Kos "and a disregard for the legal process."
Hardly. I am not a lawyer, but I find it difficult to believe that the release of exculpatory evidence now would in any way prejudice Ali's ultimate defense in court. If the surveys were genuine, then raw data files exist somewhere, at least for the most recent surveys. If the cross-tabulations published on Daily Kos are genuine, then statistical software exists somewhere that can replicate the tabulations published on Daily Kos -- including the strange matching odd or even pattern observed by Grebner, Weissman and Weissman. This is not be the stuff of advanced statistical analysis: Either the data and processes exist and can be replicated, or they do not and cannot.
Moreover, if Research 2000 actually conducted the literally hundreds of thousands of live interviews behind the results published on Daily Kos since January 2009 (I count well over 200,000 reported for their national surveys and U.S. Senate surveys alone), a wealth of documentation and eyewitness should be readily available that would be easily understood by mere statistical mortals: Call center invoices, testimony from interviewers, supervisors and the employees that prepared cross-tabulations. That sort of evidence helped send a call center owner to jail in an unrelated Connecticut case in 2006. That sort of evidence could also help vindicate Ali and Research right now -- but only if it exists.
By far the most troubling part of Ali's response comes in these two sentences (left in their original form including typographical errors):
Regardless though. to you so-called polling experts, each sub grouping, gender, race, party ID, etc must equal the top line number or come pretty darn close. Yes we weight heavily and I will, using te margin of error adjust the top line and when adjusted under my discretion as both a pollster and social scientist, therefore all sub groups must be adjusted as well.
"Top line" in this context means the results for the full sample rather than a subgroup, but it still unclear exactly which "top line numbers" Ali is referring to. If he means the results of attitude questions -- vote preference horse-race numbers, favorable ratings, issue questions or possibly even the party identification question -- he comes close to admitting a practice that every pollster I know would consider deceptive and unethical. "Scientific" political surveys are supposed to provide objective measurements of attitudes and preferences. As such pollsters and social scientists never have the "discretion" to simply "adjust" the substantive results of their surveys, within the margin of error or otherwise. As a pollster friend put it in an email he sent me a few minutes after reading Ali's statement: "That's not polling. It's Jeanne Dixon polling."
Pollsters and social scientists do often adjust their top line demographic results, and some will weight on attitude measurements like party identification, to correct for non-response bias (though party weighting continues to be subject of considerable debate in the industry). In either case, however, the adjustment needs to be grounded in prior empirical evidence -- U.S. census demographic estimates or, perhaps, previous surveys of the same population -- and not merely the whim of the researcher.
Because of the apparent lack of a written contract,** the Daily Kos complaint relies in part on the concept of an "implied warranty," the idea grounded in common law that transactions involve certain inherent understandings between a buyer and seller. Most reasonable people would agree that a political poll should be an objective measurement based on survey data that has been "adjusted" only as necessary to correct statistical bias. If Del Ali believes a pollster has the discretion to "adjust" results arbitrarily within the margin of error, he has been selling something very different than the rest of us have been (figuratively) buying.
Greg Sargent was right. The legal process of discovery, if this case gets that far, will provide truly full disclosure. But what we have learned so far is already very troubling.
[Typos corrected]
**Update (7/6): Markos Moulitsas emails to say that while was no formal "boilerplate" contract, "we hashed out our agreement via email." To be clear, a legally binding contract between two parties does not require a written document.