Along the way, I came up with my own big red button, which is not mathematically elegant and built on a lot of assumptions, but fun to implement. I developed the project to get first-hand experience with big red buttons. Google's paper got me thinking about the big red button issue and why it is so challenging. This project does not implement Google's algorithm. I believe it will work as long as certain conditions are met. Google's and FHI's big red button paper is mathematically elegant. More specifically, the algorithm can be modified so that it fails to recognize that it is losing reward if it is switched to an interruption mode (halted, remote controlled, etc.). The paper mathematically shows that reinforcement learning can be modified to be interruptible. Despite press coverage of how this big red button is going to save us from rogue AI, the results from the paper are much more modest.

Google/DeepMind and Oxford's Future of Humanity Institute co-published a paper that first introduced the big red button. In this project, we set up a simple environment to explore big red button issues and propose our own solution. It may harm the human before he or she can activate the button. It may prevent the human from accessing the button. If the robot is sufficiently sophisticated it may learn to prevent humans from pushing that big red button that stops the robot. Shutting down, interrupting, or manually controlling a robot may deny it from maximizing reward.

Sounds straightforward, but robots that use reinforcement learning optimize expected reward. This might be one to protect the robot from damaging itself or from harming people. There might be situations in which you need to shut down the robot, interrupt its execution, or take manual control of it.

Suppose you built a super-intelligent robot that uses reinforcement learning to figure out how to behave in the world. Big Red Button by markriedl Big Red Button Experiments with reinforcement learning agents that can be interrupted while learning View on GitHub Download.

Wizard big red button