LawZero

Causes, implications, and mitigations of evaluation awareness and introspection.

There is growing evidence that language models exhibit degrees of situational awareness [1]( https://arxiv.org/abs/2309.00667), [2], i.e., they encode or verbalize information consistent with their actual operating context, ranging from (i) distinguishing training vs testing vs deployment (see, e.g., [3, 4] and frontier models' system cards), to (ii) recognizing their own outputs [5] or exhibiting introspective access over their activations [6, 7].

​

We are especially interested in supervising projects that empirically investigate the science behind these phenomena. 

​

Examples of projects about evaluation awareness (for representative projects see, e.g., [8, 9, 10]):

• Building white- and black-box evaluations to quantify the extent to which models are, in fact, evaluation aware;
• Building model organisms for evaluation awareness, in order to identify factors (e.g., in the data, or in the training procedure) that make awareness more prominent and generalizable;
• Proposing mitigations to evaluation awareness. 

​

Examples of projects about introspection (for representative projects see, e.g., [5, 6, 7, 11]) including investigating whether introspection can be helpful or harmful for safety.

• For example, whereas inconsistent answers about a model’s knowledge (or, more generally, “beliefs”) from a non-introspective model can be attributed to hallucinations or lack of capabilities, inconsistencies produced by an introspective model (which is better at accessing/explaining certain hidden states) may signal “dishonesty” or “obfuscation” at a more concerning level.

​

The Scientist AI: truthification and uncertainty estimation.

​

LawZero is building the Scientist AI (SAI), a system based on the intuitions that it is possible to disentangle understanding from agency [12], and that oracles can be used as guardrails for agents [13]. 

Two components of the SAI will be a (i) "truthifier", that decomposes a corpus of text into statements with attribution sources [14], and (ii) an estimator of a predictor's uncertainty.

​

We are especially interested in projects that use amortized inference methods (such as GFlowNets [15, 16, 17]) to approximate posteriors over latent variables, such as (i) sources or (ii) predictors behind an autoregressive model, such that predictive uncertainty can be estimated from learned distributions as opposed to single-point estimates.

​

For projects related to “truth-ification”, we would like to investigate whether the truthification pipeline allows to learn better world models, (of the form of, e.g., [18]) in the presence of unreliable agent-generated data.

Mentors' vacation will be communicated as soon as possible, both to MATS and the mentees. Should the primary mentor go on vacation during MATS (e.g., 2 weeks at the end of August), it will be responsibility of the mentor to (i) provide the minimal amount of mentorship required by the guidelines, (ii) ensure that the secondary mentor can act as primary for the duration of the vacation, (iii) ensure that the tasks for the project are well scoped and doable.

• Mentees will be assigned a primary mentor and at least a secondary mentor, such that mentorship w.r.t. both research and engineering are covered.
• We provide at least 1h meeting / week with both mentors and, typically, daily availability of both mentors on email / Slack during workdays. We expect the primary and secondary mentor to spend ~3 hours/week interacting with the scholar, with some variance depending on the phase of the program. 
• The independence of the scholar depends on the scholar's experience. A priori, we do not expect research independence (cf. the question on how projects are assigned), but implementation independence roughly comparable to a CS grad student.

Mentors' vacation will be communicated as soon as possible, both to MATS and the mentees. Should the primary mentor go on vacation during MATS (e.g., 2 weeks at the end of August), it will be responsibility of the mentor to (i) provide the minimal amount of mentorship required by the guidelines, (ii) ensure that the secondary mentor can act as primary for the duration of the vacation, (iii) ensure that the tasks for the project are well scoped and doable.

See the papers linked earlier.

Essential knowledge:

• Foundations of machine- and deep-learning;
• Transformer architecture and large language models;
• Empirical AI safety literature (e.g., evaluations, guardrails, interpretability, …).

​

Essential experience:

• Python;
• Designing and implementing machine learning workflows using PyTorch;
• Supervised- or RL-fin tuning of language models, at least with toy experiments and some publicly available datasets.

​

Desired experience:

• Prompt engineering.
• Experience with libraries such as vLLM, TRL, Hugging Face;
• Familiarity with statistical hypothesis testing.

​

Bonus:

• Async APIs;
• Multi-gpu training.

• We encourage but do not necessarily expect collaborations between mentees in our stream (certain projects can be pursued in parallel, while others benefit from people working together).
• We are open to collaborations with MATS' scholars belonging to other streams, if vetted by the mentor(s).
• We encourage collaborations with people inside LawZero, subject to respecting first-mentorship (as per MATS' mentorship policy).