Big Data and the Soviet Ghosts

In a 2009 CNBC, Maria Bartiromo asked Google’s Eric Schmidt if people should treat Google like their best friend. His reply reflects the ethos of the modern software world. He said that you shouldn’t do anything that you wouldn’t want to get on the internet. In other words, everything people do should be recorded. The modern character of the new software world is Big Data. Big Data and the machine learning tools bring a ghostly vision to the software world that Soviet technocrats, who ran the Soviet Union, experimented with during the 60s and 70s. That we can record everything in society, and develop a rational way to understand society, for the purpose of guiding it to new prosperity.

Here is a clip from the documentary Pandora’s Box:

In the clip, it shows how information from all of Soviet society is fed into computer systems in Gosplan. The belief being that with all of this data, you can use models to determine if society is following the plan. It was a tool the technocrats used to control society in an efficient way. Without the computer systems, all of Soviet society would have to be employed to crunch the economic data mechanically.

Big Data  promises the same dream, but comes at it from a different side. Instead of developing rational models and applying them to large amounts of data, machine learning is employed to bubble up predictive models and to find connections between disparate features automatically. But the goal is still the same, to make sense of vasts amount of data for the purpose of control. The control the Soviet Union used was straight-forward top down. The control modern western industry employs is indirect and bottom up. This is what advertising companies do, and the dream that Google and friends sell to them.

But just like the Soviet Gosplan, the Big Data software world requires enormous amounts of centralization. Big Data requires enormous star networks that funnel every piece of data through them. The dream of the internet being a distributed system has been crushed by the client-server architecture of the Web. And because this centralization is so valuable, huge investments were made in the mid to late 90s that led to the Dot Com Bubble, and from it the internet giants we have to day such as Google, Amazon, Ebay, and Yahoo.

Just as the centralized Soviet economy and computer systems produced gross absurdities in Soviet society, the centralization of information has produced vast obscurities in the lives of this generation. As demonstrated by luke warm reaction to the recent revelations about the NSA, to the Great Recession and the computerization of trade, to the perverse incentives Facebook and Twitter provide for exhibitionist behaviours, to the death march of consumerism despite the hard evidence that global warming will lead to a miserable existence for hundreds of millions of people within decades.

The private life is dead, so that google and facebook can help others sell you things you don’t need to keep the economy humming. And it all works too! But in a much more hidden, mysterious way than Gosplan could have ever hoped.

Bonus Clip describing the obscurities that developed in Soviet society