Cristian Trout

In the face of disaster, I predict the government will be forced to play insurer of last resort, whether for a particular lab, or society at large. (See this, for example). Designed well, I believe a federal insurance backstop could internalize catastrophic negative externalities; designed poorly, it will simply be a subsidy for AI companies. I want to design the good version, so we have it ready.

I encourage people with inverse game theory (mechanism design) expertise to apply, but don't be deterred if you don't have this expertise.

Apply

View all streams

Stream overview

I'm trying to design the good version of a US federal insurance backstop for catastrophic risks from AI (read "disasters costing >~$10B"). This draws on precedents like the Price-Anderson Act for nuclear, and the Terrorism Risk Insurance Act.

One version of this research project is quite technical: use inverse game theory to design a mechanism to forecast catastrophic risk from AI. This probably looks like modifying existing forecasting methods like prediction markets and peer prediction mechanisms. The goal: help the government better predict losses, so as to charge risk-priced insurance premiums and incentivize investment in risk mitigations. I believe this is the critical challenge to making a good backstop.

Another version of this project is more like a traditional policy paper. It would step back and compare the different ways the government could play insurer of last resort (e.g. insuring labs vs. reinsuring insurers; pricing the risk in sophisticated ways without requiring additional security measures vs. not attempting risk-pricing but mandating security measures).

Another version of this project is even more high-level, comparing federal insurance backstops with other PPPs that would let the government manage catastrophic risk. I'm much less interested in this version of the project.

Mentors

Cristian Trout

Artificial Intelligence Underwriting Company

Research Fellow

SF Bay Area

—

Strategy & Forecasting, Policy & Governance

Cristian is a Research Fellow at Artificial Intelligence Underwriting Company (AIUC). Insurers have been known to play the role of private regulators (such as in commercial nuclear power); his work broadly focuses on how we might steer the insurance market for AI toward an effective private governance regime.

He was previously a Winter Fellow at the Centre for the Governance of AI, and independent researcher at the AI Safety Student Team at Harvard. He has an M.A. in Philosophy from the University of British Columbia.

Mentorship style

1 hour weekly meetings by default for high-level guidance. I'm active on Slack and typically respond within a day for quick questions or conceptual (not code) debugging. Expect async back-and-forth on experiment design and results between meetings. Scholars can also schedule ad-hoc calls if they're stuck or want to brainstorm—just ping me on Slack.

Depending on the project, I may help with writing.

Representative papers

https://arxiv.org/abs/2409.06672

https://arxiv.org/abs/2409.07277

https://papers.ssrn.com/abstract=5588732

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5339887

Scholars we are looking for

If interested in the technical paper, applicants must:

Have mechanism design or forecasting design experience (minimum: took graduate level courses on the topic)
Have experience with experimental design (e.g. running forecasting surveys)

For all applicants:

Preferred:

Published papers in relevant field
Graduate degree research experience

Nice to haves:

A sense of what makes an actionable, useful policy paper (green flag: you also roll your eyes when you see a paper title like "Towards a taxonomy of frameworks for a principles-based approach to...")
Some knowledge of how insurance works
Good writing skills

Not a good fit:

Scholars who want a lot of freedom over what their research direction will be

Scholars may find their own collaborators if they wish.

Project selection

For technical versions of this project, I suspect the project will automatically be fairly tightly scoped based on the scholar's expertise. I will pose the core challenge and over the first week, the scholar and I will hammer out exactly what theoretical questions need answering + empirical surveys need running.

For non-technical versions of this project, I will pitch a few different projects and scholars will try ones they find interesting for a week. In week 2 we'll settle one together.

Community at MATS

MATS Research phase provides scholars with a community of peers.

During the Research phase, scholars work out of a shared office, have shared housing, and are supported by a full-time Community Manager.

Working in a community of independent researchers gives scholars easy access to future collaborators, a deeper understanding of other alignment agendas, and a social network in the alignment community.

Previous MATS cohorts included regular lightning talks, scholar-led study groups on mechanistic interpretability and linear algebra, and hackathons. Other impromptu office events included group-jailbreaking Bing chat and exchanging hundreds of anonymous compliment notes. Scholars organized social activities outside of work, including road trips to Yosemite, visits to San Francisco, and joining ACX meetups.