"Do you have self-repairing serverless realtime microservices?" - "I don't know, that stuff is handled by AI on our Big Data."

Blog posts are binary:

Category 1: What you'd like to hear about

Category 2: What you should know about

This is Category 2 ;)

Yes, some of this might seem to be taking position against my own blog posts. That's ok. To minimize issues, one of the projects I'm linking to has decided to drop hundreds of lines of Code to have investors keep track of their investments themselves: Thus, responsibility for getting things right is back where it belongs, not within The Code...

Are you sure you need this fancy AI-controlled self-repairing serverless realtime microservices that run in the fabric of the internet? With all the exciting new tech around, it's good to take a step back once in a while to see if you actually need what that guy in the nice suit tells you. This post shines a spotlight on some nice and shiny technology from a perspective you don't usually get to see: Does your new technology make things easier? Or does it introduce risks that are hard to control?

Why do we like buzzwords that much? Don't we see these projects fail one after another? We see these projects fail. And these projects fail because Life does not do a good job adjusting the incentives; a set of incentives we optimize our lives by to have a good time: Who cares what's coming next? Catch is: Optimizing our lives by "It does not matter what's coming next, here's me and my life" is extremely dangerous. Bad consequences rarely are based on evil genius or stupidity, just the achievement of designing processes under the assumption that this design process should boost a career or two.

That's important! Optimizing gone wrong or bad weighting of variables is the most common reason for neural networks to not do what they're intended to do.

A real-life development that has happened in the last decades: People don't stay for long in one company anymore. Thirty years ago, you had to explain why you were applying to a new company after twenty years of service for another. Now, you have to explain why you have stuck with one company for twenty years in a row; people do get not-hired for having too little stations in their résumé - leading to actual people that tend to boost their résumé rather than getting solid workings into their current company. If you happen to recognize yourself: I don't mean to attack you or anyone else. I want you to understand the psychology behind it: If you know about peoples' motivations, you can set up a track that's more likely to end with a good result.

Optimizing a Résumé

Get in the hot new shit.

• 80s: Data Center. Unix. Multi-User
• 90s: Network. Distributed. X.400/500. CORBA
• 00s: WWW. Web 2.0. Scaled Web. CDN. LDAP. Virtualization.
• 10s: Cloud. Container. Apps. Containered Apps. Big Data. AI. NoSQL. Blockchain.

Please do not assume that I disrespect this new technology. Being new does not mean it's crap, sometimes people misunderstand hot new shit for having died and don't see it living on in something new, LDAP does not sound nice so people call it Active Directory these days (that's ok but does it run in a container?).

Do take a minute (or 20) of your time to understand what you might get to decide on, if you dare, get some quick knowledge to your boss: Sometimes life is better by gaining a reputation in one company rather than switching every other year. You'll be smarter after bad stuff happens, if you take a moment you might be smarter earlier and avoid bad things happening :)

Case Studies

1: Blockchain

Don't worry if you don't know about inner workings of math or blockchains. It's not about these few lines, it's that it fits on these few lines. There are three basic types of cryptography:

(1) Symmetric Encryption

Plain Text + Key = Bogus

Bogus - Key = Plain Text

(2) Hash-Function

Hash(no matter how much data) = 256 bit value [depends on Hash function]. It is not trivial to find a data value to a hash value, neither is it likely to have two different input values resulting in the same hash (mind Rainbow Tables and some very rare occasions of equal hash outputs)

(3) Asymmetric Cryptography

One keypair: Public and Private Key.

Data * Private Key = Data has been signed by me, if data turns out useful when you apply my public key, you can rest assured that it is my signature.

Data * Your Public Key = Data is encrypted to you. Noone but the owner of the corresponding private key is able to extract the data; not even I am able to decrypt the data I encrypted using your public key.

Crypto-Logfiles

A combination of Hash-Functions and Asymmetric Cryptography can be used to log data. You begin using a random value and encrypting this with someone else's public key. Now you add some data to the log, apply Hash(Random Value, Data), add that to the log, remove the previous random value and use the hash value as new random input for the next log input. A malicious actor is thus not able to manipulate the log at any other place than adding a bogus value to the log, changing data within the log file invalidates the entire log beginning from the line where the bogus data's hash does not correspond with the following lines original input. Only the person owning the private key can reconstruct at which point the log got corrupted; malicious actors can truncate or delete the log. This is an obvious manipulation and results in corresponding action -- if you add a timestamp to the log file, the person owning the private key can find out from which moment in time the log is no longer reliable.

Surprise: This is not Blockchain. This is a logfile resilient to manipulation. This is what the banks want.

So what's Blockchain?

It's a mechanism to determine consensus on distributed databases (a combination of Hot New Shit from the 80s and 90s): A set of actors hold data on the state of the world but do not trust each other and are incentivized to scam others. How do we agree on what's real or not?

Proof-of-Work

The Blockchain does not work using log lines but on binding blocks containing a set of log lines (e.g. Bitcoin transactions) -- everyone races to find a Hash on the past and new blocks' data plus a given value, a grain of Salt. Salt does not really matter but the Hash is supposed to have a certain number of 0s (Bitcoin) or match some other pattern. The Blockchain World relies on a competition amongst each other - to prevent someone from appearing with loads of rented cloud computing power to overpower your logs. Remember: You can't find the data from a Hash so all you can do is try; the Bitcoin Protocol adjusts difficulty levels to maintain a steady state in the Bitcoin world. Problem is: this results in massive amounts of energy consumed (solutions to this maybe Ethereum's proposed switch to Proof-of-Stake or other protocols' Proof-of-Storage Capacity).

The transactions themselves are signed by the sender, "Miners" can not change values, sender or receiver without invalidating the block - but they can refrain from hashing a transaction which might be a bad thing happening.

We simply assume that noone controls more than 50% of the computing power and as such we can trust each other not to trust each other. The current Bitcoin Cash Wars show that a blockchain needs a reasonable number of Miners to prevent a 51% attack, a glance at Bitcoin-Blockchain.com's Hash Rate Distribution shows that it is not unreasonable for a cartel of few people to form, thus dominating the Bitcoin-Blockchain.

Following this reasoning, you should defínitely ask questions if someone tries to sell you a private blockchain: Worst Case is that a malicious actor could rent some cloud computing to destroy your accounting logs.

2: Smart Contracts

The idea behind Smart Contracts is to put machine-readable contracts online to save on lawyers' costs: "IF EVENT=X MOVE 500$ FROM A TO B". These Contracts are put on the blockchain so noone can claim they do not exist. The downside: We're talking code. Code without bugs is hard to achieve, not even "Hello World!" is definitely without bugs. I won't link to patch databases of Adobe, Apple, Google, Linux, Microsoft, Oracle. World-class companies that have loads of bugs, some fixed, some not. Buggy contracts are something we already have by the dozen while they're analogue: we can talk to our partner about possible flaws we did not recognize when signing the contract, eventually a court will decide what's going to happen. Smart Contracts have already been exploited.

Following this, you should definitely ask questions if someone tries to sell you Smart Contracts.

3: Big Data

If Data happens to appear, just push it to storage. Storage is cheap, maybe we can get some useful information later. Maybe we even get to find some way of putting that data to use. First issue: That's not ethical, it undermindes democracy. That's un-american. The Soviet Union tried that in Germany, China is doing it now - why is this not reason enough?!?. Second issue: It's illegal in the European Union, Data Protection Laws require a reason for storing data. Third issue: When there's food, rats come. Fourth issue: Everyone's collecting data, noone has done something cool yet. Fifth issue: One Company's Data Loss is someone elses Data Gain.

Following this, you should definitely ask for a reference if someone tries to sell you Big Data.

4: Machine Learning

Human brains have neurons, right? Let's put that into code: One side gives input, the other receives output. We do some weighting for inputs, the coded neuron has a treshold - once the number of inputs passes the treshold, the neuron outputs 1, else 0. This stuff is not programmed, it's trained. You send in some input which is expected to return a certain output, then you go back and re-do the weighting until the neural network's output matches your expectations. Then you do Feed Forward: One neurons output is another neurons input. Add some layers fo that to have a recurrent neural network, later neurons output is fed back to early neurons input. A Deep Recurrent Feed Forward Machine Learning Neural Network puts likeliness into consideration. We hope that the neural network learns its own training data but we don't have a clue what it is doing.

It's important to know that once trained, the neural network is deterministic. You can not understand or debug it, you can only add to the training part. There are no bugs in the code, it's been trained this way. In this current world, noone is responsible for issueing bad software; with Machine Learning, people don't even have to try anymore!

Consider this situation: A Bank uses Machine Learning to decide things. The algorithm decides to freeze a bank account. The account owner goes bankrupt. The account owner commits suicide. "Can't do anything boss, the machine has trained itself to freeze that account". That's bad...

...other things Machine Learning has achieved? A robot that was rewarded for staying on track learned that it just as to move from one end of a straight lane to the other and back and thus managed to stay on track indefinitely. A sorting program made its output an empty list which, by evaluation metrics, was the most efficient way to output a sorted list. A Tetris Machine paused the game indefinitely to keep itself from losing.

Summary

Don't let others confuse you with fancy words. Blockchain, at this moment in time, usually creates more problems than it solves. Big Data creates more problems than it solves. Machine Learning creates more problems than it solves. Hot New Shit someone just told me: By applying machine-learned Big Data Analysis to encrypted data we can crack the iPhone's internal encryption faster than anyone else. Would you believe me if I told you that this works? Think about it again and maybe it's the better move to gain a good reputation in your company by doing doing the right decision rather than applying for a new job with your résumé saying "I am the guy who let the previous company spend millions of dollars on the technology to decrypt strongly encrypted data by running a statistical analysis on it. Didn't work though, but the new tech guy seems do to a good job there."

Machine Learning does solve some things, yes. But it's not a universal magic trick.

Big Data did not solve anything yet. It doesn't even attempt to be a magic trick.

Blockchain does solve some things for some people. But it's not a universal magic trick.

Smart Contracts do solve some things for some people. But they're not a universal magic trick :)