Vote of no confidence

When Mikhail Motorin, one of the people behind neural network-based election-monitoring software Revisor, heard that psephologists and activists from voter rights NGO Golos were having their homes searched by police in early October 2022, he immediately understood the way the wind was blowing.

The day before the searches, Golos co-chairman Roman Udot had presented the results of its AI analysis of video footage from the September 2021 parliamentary elections to a European conference. The focus of his presentation was the 17 million extra votes he said had been cast for Russia’s ruling party, United Russia.

Motorin was in no doubt that the searches related to the presentation. He quickly packed his bags, took his hard drives and left Moscow.

In 2020, Revisor analysed footage recorded at 1,238 polling stations in 29 Russian regions on polling day in 2018 in an attempt to retroactively uncover the electoral fraud widely believed to have taken place. Indeed, after counting votes and comparing them with the official turnout, the software found discrepancies in 11 regions so huge that it made it impossible for the election result to have been accurate to the extent that even who had won was questionable, according to another Revisor creator, Vlad Matveyev. 

says Ivan Shukshin, another one of Revisor’s creators. After the outbreak of the war in Ukraine, he and many of the neural network’s other developers left Russia. Matveyev, Shukshin and several dozen other Russians programmers, IT specialists and analysts went on to create and train a neural network that so frightened the authorities that they ultimately blocked access to live streams broadcast from polling stations.

The germ of an idea

Motorin and Matveyev first came up with the idea of creating a neural network to monitor election turnout in late 2019. IT specialist Matveyev was already well versed in elections, having first worked as an election observer at the age of 18, before subsequently working in polling stations and as a campaign advisor for political candidates.

When presenting Putin with certification of his win in April 2018, the head of the Central Electoral Commission (CEC), Ella Pamfilova, congratulated him on a “clear and convincing victory”. Federation Council speaker Valentina Matviyenko gushed that the elections would go down in history “as the most honest, open and transparent”, noting the “national approval” for the president’s course. 

Needless to say, that wasn’t entirely the case. The findings of an interim Golos report based on volunteer analysis of video footage from 233 polling stations in 17 regions revealed a discrepancy in turnout equivalent to 81,731 voters. Based on election statistics, analyst Sergey Shpilkin estimated that 10 million additional votes had found their way into ballot boxes nationwide.

Matveyev was well aware of the link between voter turnout and voting results when he read the study. Inflating turnout meant adding votes in favour of a certain candidate or party. There was clearly a need for an automated system to monitor turnout discrepancies.

The idea both men had involved teaching AI to recognise people voting, specifically how to distinguish them from non-voters in a polling station. This involved teaching AI to recognise ballot boxes of all shapes and sizes using footage recorded during the disputed 2018 presidential election.

Losing faith

Shukshin, a web developer and graduate of the Moscow Institute of Physics and Technology, had the idea of automating the work of a video observer at about the same time. In 2018, he was involved in reviewing and analysing video footage recorded during that year’s presidential elections for independent election observers, including Golos.

He decided to check polling stations with a turnout of 85% and found three — 708, 709 and 710 — in the village of Bryukhovetskaya, in the Krasnodar region of southern Russia. By watching the footage of voting at all three, he calculated that staff at each had inflated voter turnout and awarded exactly 90% of the total vote to the United Russia candidate. Shukshin later shared video footage with a running count of the number of votes cast at each polling station. Over the course of voting, all three polling stations had transmitted incorrect turnout figures to the regional election commission.

Shukshin discovered first hand how prone to abuse Russia’s electoral system was during the 2011 election. He and his wife had both voted for the liberal Yabloko party at a polling station in a village in the Lipetsk region of central Russia, only to find that the party had received zero votes in the polling station’s final tally. 

“At that time, I had this internal dialogue going on, wondering what had made Navalny so radical, why he called United Russia a party of crooks and thieves. But after that zero, I came home and wrote on LiveJournal, ‘United Russia is a party of crooks and thieves,’” he recalled.

After seeing what happened in Bryukhovetskaya, he tracked down video footage of voting in the 2011 election at the Lipetsk polling station. Watching the 12-hour recording, Shukshin came to understand that tracking the number of voters could be automated and he began working on bots able to keep an accurate tally of turnout.

Within a year, Shukshin and Matveyev had met up and began working on Revisor.

How to recognise a ballot box

Matveyev and programmers from his team broke down 200 terabytes of images of ballot boxes from video footage recorded during various elections between 2014 and 2018. There were different types of ballot box, shot from different angles, in different lighting, and their various dimensions and sizes were fed into AI so that it would always be able to recognise a ballot box from afar. Programmers from India responded to their online search for help. Matveyev and Motorin checked their results.

The team soon had a data set — a set of pictures with tens of thousands of ballot boxes marked out. The system began to learn and detect them quite accurately. The images were updated regularly. Often, AI found ballot boxes where the human eye had failed to.

The next step was to automate the count. That meant teaching the neural network to distinguish the voter from the rest of the people in the polling station and count them as soon as they put their ballot in the ballot box.

A data set of images was prepared for each region and the system was gradually taught to recognise a voter by characteristic shoulder, forearm and hand movements, from various angles, in a crowd, in bright light, and in the shade. In the interests of voter privacy, the neural network was not taught to identify individual voters or determine for whom they had cast their vote.

They now had a sample for processing: decent footage from 1,925 polling stations.

First, they tested the program on a relatively small sample. Then they hired a lot of computers and started processing the video. The results didn’t come as much of a surprise to Matveyev, but his colleagues with less experience of elections were stunned.

From faith to fact

The system successfully processed 1,238 polling stations from the sample, but another 687 were found to be inaccurate. This was due to the ballot boxes being placed in unexpected positions at 165 polling stations, as well as the initial version of Revisor being unable to deal with poor lighting in many cases. Nevertheless, it still revealed large-scale inflation of turnout: in some areas, the discrepancy was as high as 70%.

“Many of the guys, especially those with no election experience, were very surprised, said they couldn’t believe it,” says Matveyev. In his words, Revisor showed that fraud “went from being a matter of faith to mathematical fact”.

As the data sample was not big enough to be extrapolated to represent voter behaviour and turnout nationwide, Motorin says it would be incorrect to claim these results were representative of Russia as a whole, but insists they were perfectly valid for the 11 regions in question.

Revisor’s creators never tire of stressing that their program merely counts and records discrepancies, and neither makes accusations nor claims to verify voting results. Nevertheless, they have no doubt that the discrepancies it discovered were sufficiently large to cast doubt on the election results.

Volunteers manually reviewed 198 polling stations to check the quality of the neural network’s work. Under human scrutiny, the number of violations found by Revisor in the 2018 presidential election appear to have been on the lower side, and in reality there were far more.

Smear campaign fail

An internal commission at the Federation Council, Russia’s upper house of parliament, discussed the threat posed by “foreign software” such as Revisor at a meeting in June 2021, noting that “foreign centres” intended to use such programs to discredit Russia’s parliamentary elections. 

Why the program was deemed to be foreign was not explained, though “credible sources” were referred to. A report outlined the insidious plan for AI to monitor election footage in order to create an illusion of mass electoral fraud, boosted by “so-called volunteers selected and trained by the Americans”.

Matveyev and his colleagues only discovered that Revisor had been created abroad and that they had been recruited and trained by the Americans about a month later. All that time, they had been busy working to improve the neural network, attempting to understand the circumstances in which it made mistakes. They tested it using polling stations where both AI and Motorin had checked turnout.

The system was upgraded continuously until it was ready for use in September 2021. Just days before the election, disaster struck when public access to polling station video streams was blocked, officially, due to a very sudden lack of funds. The CEC appeared genuinely afraid that irrefutable evidence of mass voter fraud would come to light. That evidence emerged soon afterwards nonetheless.

The rule, not the exception

The developers finally had a stroke of luck that December when independent news outlet Meduza received a leaked archive of parliamentary election footage from over 9,000 polling stations in 19 regions. Meduza handed the data dump to Matveyev, Shukshin and their colleagues.

According to the automated Vybory system, 3.2 million voters voted at these 9,000 polling stations, whereas Revisor counted just over 2 million, meaning that roughly one third of votes were fake. Meduza estimated United Russia had added about 17 million fraudulent votes in all.

Shukshin says, though he concedes that he can’t say how many current United Russia deputies would have lost their seats were it not for ballot box stuffing.

Two days before the war began, a majority of deputies in the State Duma ratified treaties of friendship and mutual assistance with the self-proclaimed “Donetsk people’s republic” and “Luhansk people’s republic” in occupied eastern Ukraine, in a case of illegitimately elected representatives voting to create a legal framework for an illegitimately elected president to start a war. 

At this point, Shukshin and his family left the country. Udot left too, while Motorin went into hiding for several months. 

“If access to those broadcasts hadn’t been blocked, we would have announced how many honest and how many criminal polling stations we have on the Wednesday after the election,” Shukshin says wistfully. Others had clearly foreseen the likelihood of that happening too, only they regarded it as a threat. When Putin decreed that there should be live broadcasts from polling stations in 2012 to prevent accusations of fraud and quash doubts about the election’s legitimacy, he could never have suspected that same footage would become evidence of just how illegitimate he was. 

Some names in this story have been changed to protect individuals who remain in Russia.