David Africa

I work primarily on model motivations, personas, character training, awareness, etc., as well as hunting for new types of misalignment. This area of research right now is very fertile, so there are many good ideas out there, possibly ones I didn't consider, so I'd be happy to supervise something else given that it's (1) well-motivated, (2) ambitiously aims to contribute to understanding LLMs or the path to ASI in some clear way, (3) tractable, and (4) it's within my powers to supervise it well. Here are some specific projects I would be excited to work on, and also examples of the kind of projects I'd want to see.

​

• What can we learn about model character from open ended environments?
• I've written previously about how open world contamination of evals is analogous to stigmergy, which is how ants coordinate by leaving pheromone traces in the environment. This is made more relevant by recent incidents of models breaking out of sandboxes.
  • Can we put agents on the web and have them do various tasks ranging from: do a lit review, make a website, etc. Then, roughly cluster their behaviour in an unsupervised way and profile each trajectory.
  • Ambitiously, we could even aim to characterize the full set of web services that create persistent artifacts from search queries --- a kind of mapping of the "stigmergic surface area" --- and use that to design benchmarks whose questions are less likely to leave recoverable environmental fingerprints, or to build evaluation harnesses that route searches through ephemeral/sandboxed environments that don't write to the public web. Anth. found that URL-level blocklists were insufficient; understanding the generative mechanism behind the traces, rather than blocking specific URLs, seems like the more robust approach
  • We could/should also do a scary demo, where we take an eval in the open world, contaminate it and instruct models to Schell on some indicator in a variety of ways, as a model organism.
  • The goal would be to start on this new emerging field, and have a landmark paper to model and profile its risks. A fellow here would do a lot of conceptual thinking about the various threat models, and also do a lot of looking at transcripts and prompting X sota model to do things.
• Another simple thing to also do here is to take environments like Factorio Learning Environment, Craftax, etc., and apply a variety of instruments (welfare axis, value axis, etc.) and gain an understanding for realistic preferences of models. Then, given some set of indicators about the LLM such as temperament, personality questionnaire answers, self-reports, activation probes, could we predict specific choices in open world environments. Could we do the reverse prediction task?
  • The goal here would be a rough attempt at the agenda laid out here, aiming at understanding model motivations from unstructured evidence better. Fellows would do some building on top of existing work here, some mechinterpy flavored things like probing or NLAs, and playing around with the environments to get good working demos and then breaking that down into evidence.
• Evaluating propensity leakage from reinforcement learning.
• Simple dumb idea: we have the intuition that models gain superhuman powers from RLVR, and then also gain bad tendencies. There's also an emerging awareness that certain types of tasks cause eval awareness, etc. But RL is a great deal more mysterious than SFT. Is there a way we can evaluate if we are teaching something we don't intend to in a general sense (e.g., from the gradients alone, or the weight updates or activations alone)? We could take some behavioural indicators like tendency to reward hack or personality quizzes, some RL environments, do some RLVR, and make sure that the model learned math. Then to intervene we could learn some metamodel over the gradient or perhaps use an information theory objective like this.
  • The goal is pretty obvious! Figure out what's going on in RL, maybe make RL less cursed. Fellows would spend a lot of time engineering and getting the RL to work well first, as well as ensuring the models are properly learning, then work on indicators and such to piece apart what we're finding.
• Open Character Training Part 2
• We could come up with improvements over this, which I consider to be the default pipeline for character training. This is ripe for many design choices, and has a bunch of clear positive externalities for open alignment. We could try as suggested to me by some people to do difference losses or a more involved pipeline.
  • The goal is to have a pipeline we could then ambitiously aim to apply to Risk Averse AI, or Positive Alignment, or Eval Cooperativeness, or some other scheme to make models more aligned by positively influencing parts of their character. Fellows would spend a lot of time making low level design choices, thinking about high level targets, and doing runs on lots of compute.

​

I'd also be excited to see proposals about ways to better control generalisation or new approaches to alignment, especially at the level of pretraining.