Blogged by John Gideon and Brad Friedman
The results of California Secretary of State Debra Bowen’s “top-to-bottom review” of electronic voting systems previously approved for use by her predecessor is still underway. But before any of the findings from her teams of security specialists, software analysts and voting systems experts have been made public, the unprecedented analysis has already revealed a disturbing anomaly which may have far-reaching implications for both state and federal voting systems laws across the country.
As The BRAD BLOG reported exclusively almost three weeks ago — precisely zero media outlets bothered to file their own reports on this matter until last weekend — all voting machine vendors certified in California had submitted their source code to Bowen for the review, except for ES&S, America’s largest voting machine company.
After their refusal to submit the code as required for the test, Bowen demanded the source code used for the InkaVote Plus voting systems marketed by ES&S, and used exclusively in Los Angeles, be released to the state by the escrow firm which had been holding it as per state law.
Following Bowen’s demand to the escrow company, Iron Mountain Intellectual Property Management, ES&S reluctantly agreed to give their own version of the source code to the state.
Oddly enough at the time, the voting machine company, in an arrogant letter to Bowen (posted here in full by The BRAD BLOG), demanded that she withdraw her request to receive the version of the source code already stored in escrow at Iron Mountain. The letter succeeded in keeping our already-raised eyebrows at full perk, as the demand that Bowen not review the code in escrow, but rather look only at the one ES&S was sending, raised several troubling questions. Among them, we wondered at the time if perhaps the version stored in escrow was not the version actually used on the county’s voting systems during last year’s election. If so, there could be enormous ramifications for the company, and for the idea of escrowed source code for voting systems in general.
Over the weekend, an article in the Los Angeles Daily News, the first organization to jump into this matter following our series of reports, filed a story on the matter which began to validate our suspicions. The paper reported that due to the late submission, the InkaVote Plus system would not be included in Bowen’s “top-to-bottom review”, presenting questions about which voting system would be allowed for use in 2008, in the country’s most populous county. LA County is larger than many states in America.
It’s as yet unclear whether Bowen will completely decertify the InkaVote Plus system for use, or whether she will take other steps.
Perhaps more disturbingly, however, the Daily News report includes comments from CA’s Deputy SoS for Voting Systems, Lowell Finley, indicating that our concerns about differences in the submitted and escrowed source code may have been precisely on target.
We contacted Bowen’s office for more details, and they shared with us the letter sent from Finley back to ES&S in response to the company’s curious demands. The letter is posted in full at the end of this article. And if the issue Finley raises is indeed true, there may be a whole lotta trouble ahead…
“With regard to the InkaVote Plus source code,” Finley writes in the letter, “it has come to our attention that there are version number discrepancies between the description provided by ES&S to Iron Mountain of the source code deposited in escrow and the description of the system as certified by the Secretary of State on April 21, 2006.”
“As you know, Section 19213 of the Elections Code prohibits any change to a voting system after it has been certified without written notice to and approval by the Secretary, and Section 19103(a) also prohibits use of a voting system if this requirement is not met.”
Finley suggests that perhaps the version number discrepancies “may represent no more than typographical errors,” before confirming that his office will, in fact, “continue to insist on access to the escrowed source code.”
This issue is no small matter.
If ES&S failed to escrow the version of software that was actually certified by the state, the “serious problem” Finley refers to in the letter raises a number of serious questions, all of which could have great potential national consequences. Among the questions:
- Which set of source code, with what version number was actually used in the last election in Los Angeles County? And was it a certified version at all?
- Was the source code recently provided to the state by ES&S the same version as is in escrow at Iron Mountain or that which was certified for use by the state?
- Was the source code provided by ES&S, as per state law, ever federally inspected by an Independent Test Authority (ITA) and qualified by the National Association of State Election Directors (NASED)?
- Why is there even any discussion about whether to decertify the InkaVote Plus voting system at this point? State law appears to have been violated and it is the Secretary of State’s duty to take action.
If Finley’s comments are accurate, ES&S may have violated state election law. California Election Law, Section 19103 states:
In other words, when a vendor changes versions they are mandated, by law, to put those changes into escrow. They must also have those changes certified at both the federal and state level before the updated software may be used in any California election.
If the system used in Los Angeles Co. has been modified in any way without any notification to the Secretary of State’s office — and that now appears to be very possible — state law allows administrative relief to include a fine of up to $10,000 per offense, a ban from use of the vendor’s product in the state for up to three years, and/or a refund of all money paid by a locality for the voting system in question ($25 million in the case of Los Angeles County).
Prior to the 2004 Presidential Election, when it was revealed that Diebold, Inc. had installed uncertified hardware and software for their touch-screen (DRE) voting systems in several California counties, then-Secretary of State Kevin Shelley decertified the systems, and banned Diebold from further selling that system in the state. Shelley’s successor, Bruce McPherson, who was later appointed by Gov. Arnold Schwarzenegger, surprised many by re-certifying the Diebold TSx system despite the discovery of myriad security flaws and previously undisclosed source code which violated the Federal Voting Systems Standards that all systems in the state must comply with prior to being state-certified.
So how far will the state now go in regards to ES&S under a new Secretary of State? Bowen defeated McPherson last November, largely on the promise of taking voting system security serious in the Golden State for the first time since Shelley was removed from office.
The LA County Matter May Have National Consequences…
The questions begged by this matter may have far reaching national consequences. The practice of requiring the escrowing of voting system source code, for later review as needed (for example, in the event that problems or questions are revealed during an election) has been gaining traction around the country. A number of states, as well as pending legislation in both the U.S. House (Rep. Rush Holt’s HR 811) and Senate (Sen. Diane Feinstein’s S. 1487) require voting machine companies to submit their source code into escrow for use in a later review as may be required.
However, as the Los Angeles situation reveals, there may be few, if any, safeguards keeping a vendor from storing one version in escrow and then using a complete different version in actual elections.
Such a circumstance would not likely be revealed until, and unless, a problem is later discovered. The result could be a false sense of security by voters and elections officials that the escrowing of voting system source code might actually offer any transparency or safety whatsoever.
Rush Holt’s controversial federal Election Reform bill, HR 811, when originally introduced, had been written to require complete disclosure of all voting system source code to any member of the public who might be interested in reviewing it. Once the bill underwent changes in the U.S. House Administrative Committee, however, and after intense lobbying by voting machine companies, the language had been changed to require only the escrowing of source code for possible review by “experts”, under non-disclosure agreements, in the event of a problem discovered during an election.
But if vendors are responsible for policing themselves in determining which version of source code will be submitted to escrow, the effectiveness of the entire matter of escrowing source code comes into question.
As Bowen’s attempt to access the InkaVote Plus source code from escrow is the first known instance of attempting to require such a release, it’s troubling that there are already questions about the validity of the practice.
Finley’s letter also goes on to describe another possible violation of state law discovered in the 2006 escrow contract between ES&S and Iron Mountain.
According to the missive, the contract (which we have not yet seen) is said to violate the law “which specifically gives the Secretary of State the right of access to escrowed source code for any purpose that is in furtherance of her responsibility for certifying and conducting periodic reviews of voting systems.”
Given ES&S’ earlier instructions ordering Iron Mountain not to release the source code to Bowen, it sounds as if they may have written something into the contract disallowing release to the SoS without explicit prior approval.
Old Questions Still Unanswered…
Some questions we’d asked previously of Bowen’s office still remain unanswered, though they have told The BRAD BLOG they are working towards sending us the answers.
Among the still-unanswered questions concerning ES&S and the InkaVote Plus system specifically:
- Why did ES&S directly provide the source code to the state and not just allow Iron Mountain to deliver the code that is supposed to be on escrow at that facility for purposes such as the state’s lawful inspection? (The answer may be in the contractual violation referred to by Finley in his letter.)
- What steps will be taken to ensure that ES&S has complied with state law and the certification agreement regarding that escrow?
- Did ES&S include the environment and compiler used to build the software from the source code?
- If so, will the SoS be doing their own trusted build to compare against the software used in the last election?
- If not, how will they confirm that the source code was actually used to build the software used in the last election?
We will, of course, keep you up to date as we’re able to gain answers to so many of the questions this entire matter now raises.
The 3-page July 5, 2007 letter from Deputy SoS Lowell Finley to ES&S’ Steven M. Pearson follows in full below. A PDF version is here…
Can we just skip to Friday so Bush and Cheney can resign.
These folks working on behalf of California elections need to be extended our full support and encouragement. Just think: this is but one case.
If the contract does not conform to State law, then can ES&S claim to have legally escrowed the source code?
Iron Mountain is in a tough spot here. If the escrow contract is between ES&S and LA County, then Iron Mountain has no choice but to prevent the release of the source code to the State of California. To do otherwise is to tell other customers of Iron Mountain that your escrowed source code is not safe with Iron Mountain. Iron Mountain will release your source code to parties outside of the escrow contract.
But, the more ES&S fights and succeeds to block the release of source code from Iron Mountain, the more ES&S loses with regard to the California escrow statute as a whole. This is because the more ES&S and LA County “win” on the Iron Mountain release matter, the more they “lose” on complying with the statutory requirement of 19103. (a) to have:
An exact copy of the source code for all ballot tally software programs certified by the Secretary of State, including all changes or modifications and new or amended versions, shall be placed in an approved escrow facility prior to its use.
Which one has the vote flipping bug in it, the one already in use in the machines in LA County, or the software in escrow at Iron Mountain ?
Floridiot #4 –
This is the InkaVote Plus system. It’s not a DRE but a ballot marking device w/ optical scan.
If the contract is contrary to state law, it could be considered void or voidable. Perhaps L.A. County should send the machines back, demand a refund, and then sue.
This is almost EXACTLY the same scam that Diebold was running in California (and no doubt in all the other states in which they have contracts), and the documents I exposed proved it. The certified version of the software Diebold CLAIMED they were using in their California voting machines was NOT THE SAME version they actually WERE using. After receiving state certification of their software, they took the software away and made changes to it, then used the changed software in the voting machines, not the certified software. And they willingly and knowingly LIED to the state of California by signing documents attesting to the false fact that the software they were using was the same as the certified software.
Liars, all of them. And we continue to let these criminals run our elections for us? Wake up, America! Your democracy is nearly dead; between the junta currently in power stripping away your constitutional rights and declaring itself to be above the law, to the perversion an subversion of the justice department and the federal courts, to the voting machine companies like Diebold and ES&S knowingly and willingly, with malice aforethought, subverting elections and thus our entire democracy, this country is going down fast. I really don’t know if we’ll ever be able to reverse or even stop the destruction of our democratic republic.
America, we hardly knew ye. Canada is looking so good to me right now…
Who is in charge of elections in California?
ES&S states in its June letter that “ES&S expects that the SOS and each of the examiners will strictly comply with the NDAs….”
Apparently, ES&S does NOT expect that IT will have to “strictly comply” with California law or the requests of the highest elected officials in the state of California regarding elections.
Source code is only one of many things that have to be checked for a top to bottom review. In and of itself source code is a small item, but if unreviewed it is a safe haven for irregularity and fraud, and therefore looms larger in a state of un-review than it would if reviewed.
But it’s clear that the people of the state of California are not fully in charge of their own elections, according to ES&S.
{Ed Note: Comment spam removed.}
… John Gideon said…
“Floridiot #4 – This is the InkaVote Plus system. It’s not a DRE but a ballot marking device w/ optical scan”
Er… John… this invalidates Floridiot’s question… how?
If I recall correctly opscans were the first machines to be publicly hacked with vote-flipping software, right?
Jerry Brown is the Attorney General of California now. Time for him to open a can of whupass.
The other arrogant election machine companies need to see ES&S’s knickers pulled down and their little arse reddened with the people’s paddle.
Yup, I was thinkin the same thing.
It doesn’t matter what kind of system it is, it still has flipping software in it, with a verifiable paper ballot that they won’t let us look at.
Zap and Floridiot
I really can’t recall any reports of vote switching on optical-scan machines. I’m not saying it hasn’t happened; it just hasn’t been reported.
Yes, they can be hacked. No doubt about it and I’m not supporting optical-scan over DREs.
John…Hacking Democracy?
Ahhh… got it. You’re referring to the “Push one candidate and another candidate gets the vote” routine often witnessed on DRE’s.
But the term “flipping” has often been used in speaking of the internals of voting software design which is how I took Floridiot’s meaning.
Not quite as urgent as clearing up the “election fraud” vs. “voter fraud” errors… 🙂
If Los Angeles thinks they have problems with ES&S, we in San Diego have Deborah Seiler as our County Registrar. After the county bought 10,000 Deibold machines in 2003, we hire their sale representative as our county registrar..who knew?!
http://www.nctimes.com/articles..._105_11_07.txt
John in 13 said:
************************************
Precinct 66 (FL-24) in Seminole County Florida AFFIDAVITS collected by people volunteering for Clint Curtis has reported our results to be 6%-24 different from the official results. Precinct 66 all voting types, but our descrepancy (I believe) was Omniscan machines, where the paper ballot is not countable once it is fed thru the Omniscan machines (which Charlie Crist is trying to ram thru in Florida for 2008.)
Just because the House Admin Committee dismissed the case doesn’t mean that vote flipping for Omniscan machines has not been reported.
Just to clear things up, I believe Gideon was referencing the “vote flipping” that we’ve seen reported many times on touch-screen DRE systems.
A semantic confusion. Obviously, results are frequently flipped (either on purpose or via programming errors) on op-scan systems as well as DRE systems.
I’m not an optimist.
I’ve been saying since before Bowen became SOS, that the most we can expect of Bowen is to run out the clock through hearings, reviews, and investigations, and then, just prior to elections, issue waivers to all the voting machines, even those which Bowen might decertify, in the interests of holding elections.
Look people, HAVA, Congress, State and local elections officials, and the voting machine vendors stole your tax money fair and square–billions of dollars of it, and they used it to install machines to steal your vote. They’re not giving either the money or the vote back–that’s not how they operate. And even if they go to prison, like Bob Ney, the author of HAVA, the courts can’t order them to make restitution to democracy.
The Holt bill, the Feinstein bill, and any other election legislation other than Kucinich’s hand-counted paper ballots bill, are designed to steal more taxpayer money and more votes. Was it Bev Harris who first described voting machines as $7,000 pencils? Which wouldn’t be so bad except that they write with invisible lead and you need more machines to read them–and you can never prove whether they were tallied correctly or not.
Brad and a lot of others have touted the benefits of having a paper ballot. I’m in San Diego and we learned the hard way that having a paper ballot is like having a ticket to a ball game that was over two days ago. The ticket is valid, but the game’s over, so it is useless. Unless that paper ballot is hand counted on election night in full public view, with no machines that can cast doubt on the results and throw the election to Congress or the courts, it isn’t worth the paper it’s printed on.
What would paper ballots have proven in the Christine Jennings case? That 18,000 votes were lost? But we already know that, even without a paper trail. And Democrat Pelosi chose to swear in Republican Buchanan anyway, because Pelosi needed more Republican votes to help keep impeachment off the table. And if you’d had paper ballots in Sarasota, and had gone to Pelosi with those paper ballots and shown that Jennings had really won with a huge majority, Pelosi still wouldn’t have sworn in anybody who might not have helped Pelosi keep impeachment off the table. Clint Curtis had sworn affidavits from voters, and Pelosi wouldn’t even look at them. It isn’t as if Pelosi cares about voters, any more that Bush and Cheney do.
I hope I’m wrong about Bowen. It would be nice to be wrong for a change. 😉
If they don’t let us see that code pretty soon, I think we can throw out the “programming error” thingie 😉
Check out the last five paragraphs of this story, if you disagree with me about Bowen:
http://www.dailynews.com/news/ci_6318162
John
Nobody can see vote flipping on the opscams but that sure does not mean it is not there. I believe TRUTHIS ALL was among those who posted here that the opscams were likely switching more votes in Florida than the Dre’s based on the exit poll data. The fact that a crime can be hidden doesn’t make it any less of a crime.
Mark S. said:
Please don’t conflate paper ballots with paper trails. Had we had paper ballots, marked by the voter, not only wouldn’t there have been 18,000 “lost” votes, but if there had been — as reported, for example, by op-scan machines — we’d have been able to go back and determine the true intent of the voters.
Adding paper trails to the touch-screen DRE systems used in Sarasota, would, of course, have been useless. Despite so many in the Election Integrity community (including Rep. Rush Holt) arguing, without evidence, that Holt’s mandated DRE “paper trails” would have made a difference in that election. It wouldn’t have.
Paper ballots however, would have!
Everyone. Take a deep breath. I know that optical scan machines can be hacked and I know that they are just as apt to report the incorrect results as any other system in use.
However, I am also a researcher/compiler/reporter and I have to go on reported instances. I have never read an instance where a voter has claimed that an optical scan machine changed their vote. There may be anecdotal evidence but that’s it.
Let’s not let this distract us from the information that is in this article.
They will not easily give up anything that might incriminate them which means the source code held in Escrow will probably incriminate them. By this point in time I don’t know why they just don’t seize the code before some “accident” befalls it. Amazing how incriminating evidence has a way of getting “lost” before it can be proved to be incriminating.
Good work Brad at keeping this public issue public. The more eyes on it the better. Thanks…Joe in Spfd.MO
BJOBOTTS:
Well, in this case, if the code went “missing” it might do to Iron Mountain what the Enron affair did to a “Big Five” accounting firm called Arthur Anderson. (Lesson to corporate monsters: You’re never too big to go down.)
As soon as your escrow service “loses” something this important, other government entities are going to stop thinking of (or allowing) Iron Mountain to be used as an escrow.
SEE WHAT A FIASCO THIS SIMPLE REQUEST HAS TURNED INTO.
We have a problem here so show us the source code you have in escrow.
No, you can’t have the one in escrow, but here take this one it should be the same. How are we going to know if we don’t get the original source code you are holding in Escrow?
This is my vote. I don’t want it privatized or the way of checking how my vote was registered and counted, held in some secret coded machine software by a private company which I must petition to check if their voting machines are operating properly. Am I the only one who sees this as a recipe for disaster?
Keep up the good work Brad Blog. Our whole system of government rests on this, so thanks.
Common Cause had an article in the Bangor News in Maine advocating H.R.811.
John
You never will hear from anybody who seen his vote switched by a opscam. The flippy can not be seen. That is no indication whatsoever that it is not happening. The only clue we can get is from the exit polls since the won’t let us count the ballots. The exit polls have been screaming loud and clear that they have indeed been switching votes in a set direction in recent elections. They may well have been doing so long ago but nobody was watching like they have been recently. As an expert in probability theory I sure can’t come up with any reason to put any more faith in the opscams than the dre’s! An optimist might call then opscans but I sure don’t!
Why would you expect the escrowed version to match the version being used? After the Diebold scandal in 2003, Conny McCormack told the Los Angeles Times:
Also, Brad, you said state law allows up to $10,000 per offense. But the California Elections Code section 19214.5 states:
So, if the 5,000 InkaVote+ machines in Los Angeles all have a version not in escrow, that is a fine up to $50,000,000! And ES&S has a lot of other business in the state, with the AutoMARK system, so they won’t want to just pull out.
– Jerry
* The McCormack quote is cited on several web pages, including:
By Kim Alexander, California Voter Foundation,
By Doug Jones,
In the ACM Risks forum,
etc.
Point of clarification for you, Brad. The R&G audits ordered by Shelley revealed that uncertified software was installed by Diebold in 17 CA counties, but this included optical scan as well as DRE systems. I’ve been particularly interested in this point for a quite a while, attempting to determine who approved or allowed such illegal tampering with our equipment here in Humboldt County. That this happened seems to me to be proof of either complicity or negligence on the part of one or more people in our elections department. My investigation has not resolved this to my satisfaction. I’ve always encouraged people in the other impacted counties to research this too. The questions are the same.
Julie, #17
So Florida isn’t going to recountable paper ballots???
So Christ isn’t the hero he’s been made out to be in our election reform movement? If what you say is true, he’s just another sleazy Pub…
What’s the story?
shw
Black box voting on unverifiable systems – an abysmally stupid idea.
As Bad as the Florida 2000 vote tally was – at least there were real ballots to argue over – and that punch card system could be corrected for less than a dollar’s worth of parts per machine with a flash light bulb and D size battery – punch through completely and the light goes on when the stylus completes the circuit – gees rocket science.
These computerized machines:
Source code should be public domain, a smart high school kid could write these systems. Easily verifiable check sums on the source code and executable code. The hardware should be easily examinable – not only are wireless connections a potential problem – the power grid can be used as a network as well.
Multiple logs recording every keystroke are easily faked on the fly. I write database triggers every day – it would be cake to display one thing on the screen (what the voter thinks they are doing) record what you want in as many ‘audit’ logs as desired.
Best solution:
Paper ballots counted at each and every precinct. True more labor intensive – but if your going to ‘fix’ a major election – a lot more risky collusion will be required to pull it off then with these magic black boxes. But what do I know, I’m only a database administrator and programmer with over 20 years in the biz.
A bigger problem – controlling who you may vote for no matter how the ballots are counted. Who is Carl Romanelli?
Folks,
There is one secure system, and one secure system only: PAPER BALLOTS, HAND COUNTED. The sooner we prove to the majority of American that every other option is an invitation to election theft, the sooner we can get to the one and only solution and the reinstatement of our representative republic (which we lost in Dec. 2000). Fortunately, the “Holt” Bill, HR-811, is seeing more an more resistance — as it should because it is a clear prescription for dictatorship — and, best of all, Dennis Kucinich is introducing a new bill (as yet unnumbered) that will REQUIRE PAPER BALLOTS, HAND COUNTED FOR ALL PRESIDENTIAL ELECTIONS STARTING WITH THE 2008 ELECTION. Folks, get on top of this now; starting calling and writing your “representatives” and DEMANDING that Kucinich’s bill pass and not the Holt Dictatorship bill. Act as if your life depends on it; it does.
In L.A. the InkaVote Plus machines are only used to notify voters of an undervote or an overvote. Many millions of HAVA money went for this purpose only. The machines can tabluate votes, but I doubt if they ever will because I’ve heard they fail too often.
L.A.’s current vote tabulation system has not been looked at by state election officials for many years. I believe the last time they looked was 1998. We’ve had changes to our software since then. Our registrar said a few years ago that L.A. and other counties make changes to software without notifying the secretary of state. Once GEMS2 passes federal and state certification we will have a Diebold system counting all the votes here.
There is something wrong with the escrow process and state law if the secretary of state cannot easily get at the escrowed source code.
Patriot #34 said:
As I have sedd ad nauseum, paraphrasing Stalin:
Stalin was pre-computer but not pre-dishonesty. That human essence is everywhere isn’t it?
Bob Young #29 said:
I am not so sure about that and therefore suggest that you read this 1988 article and this 1988 NIST Report. They describe electronic voting issues going back to 1964 … over 40 years.
Bob also said:
And that really is the bottom line Bob, faith.
There are varying degrees of faith in the american election system. I have never heard anyone express absolute faith in that system, however, often I hear people say they feel “better” about one system over another.
Subconsciously we know the truth. The truth is that there is an element of religiousity to our election system, founded upon faith.
That is why the election integrity movement has so many of the characteristics of religious faith. Dogma, sectarianism, and sectarian wars, which Brad has described as “just short of a shooting war”.