That strikes me as a very lucid and concise formulation of The Problem. Zang. Yes.
I'm fortunate to have some nerd friends in the HFT programming community and they are big supporters of the
"Opaque Intelligence" concept...summarized in two:
...the problem isn’t artificial intelligence but opaque intelligence. Algorithms have now become so sophisticated that we human’s can’t really understand why they are telling us what they are telling us.
Dual angles of probably the same issue:
1. They can't explain a lot of the best successes. "Reverse tracing," my favorite stock mentor would call it -- working backwards to determine why decisions were made and then timing out
when that trade could have been submitted even more profitably. It's also a great way to learn, backwards, how financial news coverage affects actual stock prices. Apparently this is usually impossible on the best picks a sophisticated, long-term algorithm / machine learning project makes. They are totally
opaque to human observers.
2. Getting algorithms to lie is apparently hard. Humans, natural born liars that we be, are perfectly suited to the labyrinth of winks, nods and social signaling they need to negotiate even when interacting with
other humans on a screen. HFT algos are more like a driven MMA fighter, immediately exploiting any weakness their penetration testing reveals -- that metaphor obviously falls apart because a human being can only be struck in the skull so many times, whereas you can extract a few billion from market liquidity in the space of ~60 seconds.
Really, it's a compelling argument for functional AI, isn't it?
"We turned the robot broker on and he immediately concluded the entire market was over-valued, and it would be easy to scale enough paper/derivative leverage to force other players to agree." I don't think that robot is broken...All that said, the infosec working concept of APT - "Advanced Persistent Threats," folks who have you permanently, totally compromised and you don't know it - pales in comparison to HFT algorithms with a time preference on the scale of decades and a daily MO of total invisibility. Hopefully there are a couple dozen in play any given second NYSE is open.