Vanessa Kosoy (MIRI)

Some systems in the world seem to behave like “agents”: they make consistent decisions, and sometimes display complex goal-seeking behaviour. Can we develop a robust mathematical description of such systems and build provably aligned AI agents?

Ethan Perez (Anthropic)

Current ML models that predict human language are surprisingly powerful and might scale into transformative AI. What novel alignment failures will future models exhibit, how can we develop demonstrations of those failures, and how can we mitigate them?

Stephen Casper (MIT AAG), Erik Jenner (UC Berkeley CHAI), Jessica Rumbelow (Leap Labs)

Identifying high-level concepts in ML models might be critical to predicting and restricting dangerous or otherwise unwanted behaviour. Can we identify structures corresponding to “goals” or dangerous capabilities within a model and surgically alter them?

Jesse Clifton (Center on Long-Term Risk), Caspar Oesterheld (CMU FOCAL)

The world may soon contain many advanced AI systems frequently interacting with humans and with each other. Can we create a solid game-theoretic foundation for reasoning about these interactions to prevent catastrophic conflict and incentivize cooperation?

Evan Hubinger (Anthropic)

Powerful AI systems may be instrumentally motivated to secretly manipulate their training process. What ML training processes and architectures might lead to this deceptive behavior, and how can it be detected or averted?

Jesse Hoogland (Timaeus), Daniel Murfet (Timaeus, Uni Melb)

Singular learning theory (SLT) offers a principled scientific approach to detecting phase transitions during ML training. Can we develop methods to identify, understand, and ultimately prevent the formation of dangerous capabilities and harmful values?

Owain Evans (University of Oxford), Francis Rhys Ward (Imperial College London)

Many stories of AI accident and misuse involve potentially dangerous capabilities, such as sophisticated deception and situational awareness, that have not yet been demonstrated in AI. Can we evaluate such capabilities in existing AI systems to form a foundation for policy and further technical work?

Neel Nanda (Google DeepMind), Alex Turner (Google DeepMind), Lee Sharkey (Apollo Research), Adrià Garriga Alonso (FAR AI)

Rigorously understanding how ML models function may allow us to identify and train against misalignment. Can we reverse engineer neural nets from their weights, similar to how one might reverse engineer a binary compiled program?

David “davidad” Dalrymple (ARIA)

If we could encode a detailed world model and coarse human preferences in a formal language, it could be possible to formally verify that an AI-generated agent won’t take actions leading to catastrophe. Can we use frontier AI to help create such a detailed multi-scale world model, and/or to synthesize agents and proof certificates for small-scale demonstrations already?

Buck Shlegeris (Redwood Research)

When a power company wants to build a nuclear power plant, they’re obligated to provide a safety case: an argument backed by evidence that their power plant is acceptably safe. What’s the shortest path towards AI developers being able to make reliable safety cases for their training and deployment, and how can we start iterating on techniques to fit into these safety cases now?

Asa Cooper Stickland, Julian Michael, Shi Feng, David Rein (NYU ARG)

Human overseers alone might not be able to supervise superhuman AI in domains that we don’t understand. Can we design systems that scalably evaluate AI and incentivize AI truthfulness?

Jeffrey Ladish (Palisade Research)

Current and near-term language models have the potential to greatly empower hackers and fundamentally change cybersecurity. How effectively can current models assist bad actors and how soon might models be capable of hacking unaided?