The Failures Discovered in Diebold's Tabulation System Are Even Worse Than Previously Reported...
...And Never Would Have Been Found Were it Not for the 'Humboldt Transparency Project'...
By Brad Friedman on 12/12/2008, 1:17pm PT  

Following up on the latest monumental failure by Diebold/Premier --- whose voting system was recently found by a citizens' transparency project to have lost hundreds of ballots in a single precinct in Humboldt County, CA --- Wired's Kim Zetter covered the issue earlier this week.

Her article touched on a number of technical issues involved in the failure of the Diebold GEMS central tabulator which, though we hadn't noted them in our own coverage, are nonetheless important, even if somewhat down in the weeds, and perhaps most of note to technical election integrity geeks only.

As quoted below in some detail from Zetter's report, she advances the story by noting:

  • The problem, which Diebold admits has been in its system for years, likely wouldn't have been noticed at all were it not for the unique, experimental, citizen-led "Humboldt Transparency Project." Even CA's required 1% random post-election "audit" wouldn't have uncovered the problem, since absentee ballots are not included in that audit.
  • The chances of the problem occurring, which resulted in ballots being deleted without notice by the Diebold GEMS tabulator, are rather high, even if chances of discovery of the error by election officials are very low.
  • Diebold's explanation for the failure may or may not be legitimate.
  • The GEMS internal "audit logs" (the GEMS system was used in 34 states around the nation last November) aren't actually audit logs at all, and may not actually note the complete set of events which occurred on the voting system, even if anyone is allowed to review those logs in the first place.

Let's take a look at the details...

Thanks to the unique oversight program allowed by Humboldt County's election chief Carolyn Crnich, the deck of some 200 absentee ballots scanned, but then deleted from the totals without notice by the Diebold/Premier GEMS tabulator, was discovered and the previously-certified results had to be corrected and re-certified.

The special project used open-source software, written locally, and off-the-shelf scanners to scan the same paper ballots that were scanned and tabulated by the Diebold op-scan system. However, had it not been for that parallel counting project, using alternate hardware and software, the previously-certified results --- which did not include the ballots deleted by Diebold --- would have stood, and the lost ballots would likely never have been found, as Zetter notes:

It's important to note that Crnich would never have discovered the problem through her standard canvassing procedures --- since the votes were still in the system after the canvas was completed --- nor would she have discovered it while conducting a mandatory manual audit that California counties are required to do.

The audit requires every county to hand-count ballots in 1 percent of randomly chosen precincts in order to compare the totals against digital tallies. But the audit involves only ballots cast physically at a precinct, not mail-in ballots, which are the ballots that the Premier/Diebold system dropped in Humboldt. Even if the Humboldt ballots had been precinct-cast ballots, Crnich would not have known they'd been dropped from the system if they weren't cast in a precinct that was included in the 1 percent of precincts that were hand counted.

But while the chances of the problem that occurred in Humboldt are actually very high, the chances of discovering the failure are low, unless the ballots are rescanned or recounted by an alternate system that actually works, like the one created by citizens, for free, on open-source software in Humboldt...

Premier explained that due to a programming problem, the first "deck" or batch of ballots that is counted by the GEMS software sometimes gets randomly deleted if any subsequent deck is intentionally deleted. The GEMS system names the first deck of ballots "deck 0," with subsequent batches called "deck 1," "deck 2," etc. For some reason "deck 0" is sometimes erased from the system if any other deck is erased. Since it's common for officials to intentionally erase a deck in the normal counting process if they've made an error and want to rescan a deck, the chance that a GEMS system containing this flaw will delete a batch of ballots is pretty high.

The system never provides any indication to election officials when it's deleting a batch of ballots in this manner.

That last graf, noting that no indication of the error is given to officials, is both troubling and reminiscent of Diebold's other recent admission that its GEMS central tabulator --- on both the paper ballot systems and touch-screen systems --- routinely drops thousands of ballots when they are uploaded to the central GEMS server, but without giving any indication of that failure to the system administrator, who is left believing that all votes were properly recorded.

Ohio's Secretary of State Jennifer Brunner is currently suing Diebold/Premier over that particular failure, as she and I discussed in an exclusive interview late in the Summer, though no other state --- including California --- is currently suing Diebold/Premier to our knowledge. That, even though the same GEMS system is currently used in some 34 states around the nation.

With all of this, however, it's not even certain that the reason Diebold offers for the deletion of those ballots was actually the reason for the deletion, as noted by Zetter:

After examining Humboldt's database, Premier determined that the "deck 0" in Humboldt was deleted at some point in between processing decks 131 and 135, but so far Crnich has been unable to determine what caused the deletion. She said she did at one point abort deck 132, instead of deleting it, when she made a mistake with it, but that occurred before election day, and the "deck 0" batch of ballots was still in the system on November 23rd, after she'd aborted deck 132. She couldn't recall deleting any other deck after election night or after the 23rd that might have caused "deck 0" to disappear in the manner Premier described.

And, as if all of that is not bad enough, as they say on Late-Night TV: But wait! There's more! And, ultimately, it could be the worse flaw of all!

As it turns out, the GEMS system's internal audit logs, used (when election officials and/or courts will allow it) to look back and determine what operations were actually carried out by the system and when, doesn't appear to be a real "audit log" at all!

Rather, as Parke Bostrom, one of the Humboldt citizens who worked on the project described it, the audit log is actually a "'re-imagining' of what GEMS would like the audit log to be, based on whatever information GEMS happens to remember at the end of the vote counting process."

The deletion of "deck 0" wasn't the only problem with the GEMS system. As I mentioned previously, the audit log not only didn't show that "deck 0" had been deleted, it never showed that the deck existed in the first place.

The system creates a "deck 0" for each ballot type that is scanned. This means, the system should have three "deck 0" entries in the log --- one for vote-by-mail ballots, one for provisional ballots, and one for regular ballots cast at the precinct. Crnich found that the log did show a "deck 0" for provisional ballots and precinct-cast ballots but none for vote-by-mail ballots, even though the machine had printed a receipt at the time that an election worker had scanned the ballots into the machine. In fact, the regular audit log provides no record of any files that were deleted, including deck 132, which was deleted when Crnich intentionally aborted it. She said she had to go back to a backup of the log, created before the election, to find any indication that "deck 0" had ever been created.

Parke Bostrom, one of the Transparency Project volunteers, wrote in a blog post about the issue, "This means the audit log is not truly a 'log' in the classical computer program sense, but is rather a 're-imagining' of what GEMS would like the audit log to be, based on whatever information GEMS happens to remember at the end of the vote counting process."

Zetter didn't offer the link to Bostrom's blog post, but it's right here. He notes, in addition to the points mentioned by Zetter above, what appears to be a contradiction to her reported information on the other two "deck 0" batches --- for provisional ballots and for mail-ballot-precincts (those precincts that are too small and remote to have actual physical precincts, so they are all done as vote-by-mail).

While Zetter notes that the other two "deck 0" batches did show up as having been scanned in the "audit log," in fact Bostrom notes that they "do not appear in the audit log, but apparently their votes are included in the final report."

Not sure whether it's Bostrom or Zetter who has that point correct, and I've been on the road and out-of-pocket for weeks, so don't have the time for the moment to find out. But, suffice it to say, if we're talking about "audit logs" that are made at the end of the process, instead of concurrently, as events occur, then they are worthless for purposes of auditing, no matter what.

Bostrom adds one other point of note in his blog item which illustrates just how difficult it is to audit the job that Diebold's GEMS system --- again, it's used in 34 states! --- has done in scanning ballots, even if (and that's a big "if"!) officials and/or courts allow regular citizens to examine the system at all:

To the best of my knowledge, GEMS does not have the capability to export machine readable per precinct/per deck results in an easily analyzable format. If GEMS had the capability to export machine readable information, it would be a lot easier to audit GEMS (even without the [Election Transparency Project software and hardware]). AS it stands, it is very difficult to notice that decks or precincts have been dropped from the final report due to this serious deficiency in the design of GEMS. (If GEMS does have such a data export capability, no one has been able to tell me how to use it - not even Premier's election support specialists.)

As we noted in our Monday coverageof this story, this entire sorry affair reveals yet again the enormous scam behind the entire federal e-voting legislation contained in the Help America Vote Act (HAVA) of 2002, allocating some $3.9 billion for states to hand over to private voting machine corporations like Diebold for their proprietary trade secret-protected voting systems which don't work.

In the meantime, on the other hand, a handful of citizens in Humboldt County, using fully open source code, developed at no cost to the county, with off-the-shelf commercial scanners, have created a system in a number of months that did a better job of counting ballots than Diebold, even with all of their millions/billions of tax-payer dollars, hundreds of programmers, secret hardware and software, and enormously expensive long-term maintenance contracts.

HAVA also created the U.S. Election Assistance Commission (EAC) which was supposed to have served as a "clearinghouse" for voting system problems such as the one that hit Humboldt. The flaw in Diebold's system has been known since 2004, but the EAC failed to notify anybody about it, and has failed to set up any useful clearinghouse system at all, in the 6 years since HAVA was signed into law.

Given all of those failures, every jurisdiction in the country which insists on using optical-scanners should immediately look at moving over to the software and hardware developed by the citizens in Humboldt. Diebold/Premier should be dumped, sued for fraud, malfeasance, and gross negligence and be put out of business --- at least the elections business --- once and for all.

As to what should happen to the EAC, we'll leave that for now, for another day...

As long promised, The BRAD BLOG has covered your electoral system 2008, fiercely and independently, like no other media outlet in the nation. Please support our work with a donation to help us keep going. If you like, we'll send you some great, award-winning election integrity documentary films in return! Details on that right here...