Attacking AIXI via its memory

Can we really define something as abstract as “intelligence” in order to state what an idealized Artificial Intelligence would do?

Yes, we can, according to Marcus Hutter. On the linked page, he describes the AIXI formalism which has many nice properties and features. Among those is a short and colorful mathematical statement which defines what an implementation of AIXI would do:

See here for explanation of the terms. I just wanted it here for the color.

There are also many objections to it, starting with the fact that it’s uncomputable. A common behavioral objection is that an implementation of AIXI would immediately succumb to “wireheading”, as it is a very pure reinforcement learner. A performance-based objection is that its reliance on deductive certainty, rather than probabilities, would make it less accurate than an untrained human child at guessing the next digit in a redundantized random string — one where an initial string of digits is generated by a truly random process (e.g. 7,2,4,0,3,…) and then each digit is repeated at least once (e.g. 7,7,7,2,2,4,4,0,0,0,0,0,3,3,…).

I have a new objection closely related to the above. Since an AIXI never forgets, and isn’t permitted to ignore input data, AIXI is vulnerable to a form of denial-of-service attack. If there were an AIXI-based agent, an attacker would have only to send it a stream of truly random data, such as provided by Australian National University’s quantum random number generator. Since every bit sent from ANU QRNG to the AIXI implementation is fundamentally unpredictable, the length [l(q)] of the shortest program [q] that generates the input stream exactly would have to grow by one bit, on average. Send it ten random bits and it will have to grow its program by about ten bits. Send it 10GB and it will have to grow by about 10GB.

The lesson to be learned from this attack is that any AI that undergoes constant learning will need one or both of the following:

• the training data needs to be kept safe from attackers, or
• the AI needs to be able to forget training data that was not useful

That latter statement is deliberately foggy and philosophical. “Forgetting” might imply information loss via summarizing data into a statistic or wholesale deletion of untrusted data, and “useful” might imply a utility function separate from the reinforcement learner or built into it — but I want to emphasize the conclusion is an informal heuristic, lacking any mathematical rigor.