Virge's Law
It was perfectly logical but nobody expected it.
The output was wrong. It disagreed with the taxation department's assessment. She ran the accountancy software again. It was right now. It agreed with the assessment, but she had changed nothing. There was no reason for the output to be any different. Memory may be deceptive at times, but the data submitted to the tax department was a permanent record of the error.
Does this scene sound familiar - software producing different results at different times when it is supposed to reliably calculate the same results? Many artificial intelligence researchers have toyed with the idea that self-awareness and the illusion of free will are generated purely through extreme complexity. The increments of complexity from simple logic to logic with memory, rules, pattern matching, learning algorithms and predictive capabilities can each be understood by themselves, but there comes point where a total system becomes too complex to have its macroscopic behaviour understood in terms of its component algorithms.
An accountancy program should just add up the numbers and spit out an answer. It should be simple, but its creators had to evolve its design and capabilities to keep up with one of the fastest growing bodies of chaotic complexity in the known universe - Australian tax law. This is a system of laws embracing the arc of history, the whole gamut of modern human life and a fair smattering of wild abstract fantasy. The program creators thought that they were dealing with something the IT industry describes as "a bugger of a thing to debug", but they had inadvertently come up against a fundamental barrier. They had met Virge's Law:
Any program of sufficient complexity to cope with Australian tax law is also sufficiently complex to exhibit emergent self-determination.