AI nerd's imagination: For the purpose of the argument suppose we have a superhuman AGI and we're going to sandbox it safely, the theoretical problems are...
What would actually happen: Just allow all users of this LLM chatbot to execute arbitrary code via the REPL plugin, and in privileged shell too because I couldn't be bothered to create a new account.
If you have a fediverse account, you can quote this note from your own instance. Search https://mk.absturztau.be/notes/9s8g82gvg5fw00hb on your instance and quote it. (Note that quoting is not supported in Mastodon.)