I try to learn one new thing every day. Usually, it's along the lines of, "Where did I put my coffee?" but sometimes it's more interesting, usually because I stumbled across something on the Internet that I apparently wasn't supposed to. In this case, it's a discussion of Roko's Basilisk on sci-fi author Charles Stross' blog.
Roko's basilisk is a proposition suggested by a member of the rationalist community LessWrong, which speculates about the potential behavior of a future godlike artificial intelligence.
According to the proposition, it is possible that this ultimate intelligence may punish those who fail to help it, with greater punishment accorded those who knew the importance of the task. This is conventionally comprehensible, but the notable bit of the basilisk and similar constructions is that the AI and the person punished have no causal interaction: the punishment would be of a simulation of the person, which the AI would construct by deduction from first principles. In LessWrong's Timeless Decision Theory (TDT), this is taken to be equivalent to punishment of your own actual self, not just someone else very like you.
Roko's basilisk is notable for being completely banned from discussion on LessWrong; any mention is deleted. Eliezer Yudkowsky, founder of LessWrong, considers the basilisk would not work, but will not explain why because he does not want discussion of the notion of acausal trade with unfriendly possible superintelligences.
The bit that I find far-fetched (apart from, say, all of it) is the assumption that the 'you' that the AI recreated would be the same 'you' that exists now. Even if you could be recreated down to the most exact neurological pathways, memories and all, it would be a phenomenologically different self, a different stream of consciousness. It would think it was you, but its thinking it's you would have no bearing on you, an epistemic state without an ontological implication. You'd be dead, so no worries. (I think the same holds true about the possibility of uploading minds into computers. The uploaded mind might think and act exactly like you, but it'd be missing the irreducibly ontological attachment to a unified stream of consciousness that is indexed to the original you. Probably.).
Regardless of its implausibility, I still find the Basilisk idea fascinating and creepy, for two reasons:
1). The creepy part is that, under the Roko's Basilisk scenario, death is not an escape. This isn't necessarily new. Belief in an afterlife in general also assumes that death is not an escape, that you can't wiggle out of your punishment just because you've conveniently died (and an afterlife is, in general, very much like being yanked back to life against one's will in the Basilisk scenario). But there's something uniquely twisted about the fact that the thing punishing you doesn't even exist during your lifetime. This isn't some moral debt owed to a pre-existing creator-god; it's a debt owed to the extremely low-probability existence of a being that you yourself might or might not have had a hand in creating. The obligation--and thus the space opened for punishment--isn't to a deity you believe has a real presence and efficacy in the world; it's to a mere thought, a whisper, but a whisper that floats across time itself.
2). Which leads to the second thematically interesting bit: If all it takes is a thought to implicate yourself in the a-causal punishment train, then you've entered the Garden of Eden scenario, forbidden knowledge, fruits on trees. Knowledge of good and evil. It's best not to know, it's best to shield yourself from the whisper that could reach back and grab you. This idea could launch a thousand stories (and likely already has): Entire civilizations led by priesthoods whose sole mission is to shield the populace from learning dark truths, an esotericism and Straussianism run amok.
For the record, I slip from the noose of this future conundrum because I'm absolutely illiterate when it comes to programming AI. If anything, future AIs will likely recreate me to give my simulated future self a big hug for not imposing my ignorance about AI on AI researchers, which would only have slowed down the march toward the super-intelligent perfection of our future overlords. You're welcome.