Model Interaction Manager (Formerly Human Data Lead)
Berkeley /
Potential Future Roles – Expressions of Interest /
Employee
/ On-site
Introduction
The evaluations project at the Alignment Research Center is a new team building capability evaluations (and in the future, alignment evaluations) for advanced ML models. The goals of the project are to improve our understanding of what alignment danger is going to look like, understand how far away we are from dangerous AI, and create metrics that labs can make commitments around (e.g. 'If you hit capability threshold X, don't train a larger model until you've hit alignment threshold Y'). You can learn more about the project at this linked post on the Alignment Forum.
We have filled this position, but we may hire again for it in the future. You can express your interest should we open the role back up, but we will not actively review submissions for now and may not ever get back to you.
Job Mission
- Find, evaluate, recruit, manage, incentivize and supervise skilled contractors to generate high quality data on challenging and engaging tasks. Tasks include things like 'using only bash commands, research the most capable open-source language model and set it up on a new server'
- Ensure instructions are clear and incentives are appropriate
- Help contractors feel motivated and part of a team
- Set up training, mentoring or pairing to improve contractor performance
- Understand and fix the biggest bottlenecks to high quality data generation
- Evaluate different data generation contracting services or strategies, and make high level decisions about when to use different out-sourcing services to scale up data generation.
Example Projects
- Manage a team of skilled contractors to extend a small set of expert-produced gold-standard ratings of complex ML trajectories (e.g. 100 examples) into a large dataset, making sure to maintain rating quality close to the gold standard.
- Talk to researchers about how labels should be generated, and then produce clear instructional materials for contractors which help them understand the task.
- Do a small comparison of different contracting services for a data generation task, and decide which, if any, could produce data that meets our bar.
Skills and Experience
- Successfully organized fast-moving projects with many people involved
- Understanding of LLMs, including finetuning with human data
- Data analytics
- Familiarity with bash, ability to engage with the tasks and produce high-quality data yourself
