Member of Technical Staff, Evals/Environments

Member of Technical Staff, Evals/Environments

NY, SF, Remote

Why RoVR

At RoVR, we’re on a mission to turn practices into AI-Native, Self-Optimizing Learning Health Clinics. At RoVR, we are building a multimodal, agentic operating system to launch, manage, and scale A-LHCs nationwide. We collaborate with large health organizations as well as local physician groups, modernizing legacy operations to launch these next-generation clinics.

Central to this vision are at RoVR, autonomous agents that orchestrate and optimize clinical, administrative, and patient engagement processes. By alleviating repetitive tasks, RoVRs free providers and admins to focus on proactive decision-making guided by actionable insights, laying the groundwork for a more responsive, outcome-driven healthcare ecosystem.

By uniting clinical and operational intelligence in one cohesive platform, we're building the infrastructure for a Learning Health System that can collaborate with and empower clinicians to redefine the patient experience and transform health outcomes.

Based in the heart of NYC and backed by leading venture capital firms/visionary healthcare leaders, RoVR is on a mission to transform the future of healthcare. We are seeking mission-driven engineers, designers, and operators to join our team and help us build the foundational operating system that will drive medicine for the next century.

The Role

We are looking for an engineer to build the environments models are trained and judged in: simulated clinics with simulated people, the tasks that populate them, and the graders that score what an agent did. As the owner of evaluation, you would make it sensitive to real improvement, resistant to gaming, and usable as a training signal, and decide what the evidence has to look like before a model ships.

Responsibilities

  • Design RL environments that replicate clinical workflows end to end: state, tools, counterparties and consequences.

  • Build task generators and dynamic datasets with adjustable difficulty, so environments do not go stale as models improve.

  • Develop verifiers and reward signals: deterministic where possible, model-graded where not, expert-adjudicated where it matters.

  • Build the evaluation harness: agents at scale, reproducibility, telemetry, and the reports that decide what ships.

  • Keep the internal and public benchmarks honest as the system learns to game them.

Minimum qualifications

  • Bachelor's degree or equivalent experience in computer science or a related technical field.

  • 3+ years in software, simulation or ML infrastructure, with an environment or evaluation system others have trained against.

  • Strong Python and systems skills; browser or API simulation (Playwright or similar) and distributed execution.

  • Working RL understanding: reward design, environment dynamics, evaluation leakage.

  • Ability to turn "handle this correctly" into a scored task.

  • Suspicion of your own metrics.

Preferred qualifications

  • Public benchmarks or leaderboards.

  • Enough healthcare operations knowledge to know which failures matter.

Logistics

  • Location: San Francisco or New York, on-site.

  • Employment type: full-time.

  • Compensation: competitive salary and meaningful equity.

  • Visa sponsorship is available.

RoVR is an equal opportunity employer and does not discriminate on the basis of race, gender, sexual orientation, gender identity/expression, national origin, disability, age, genetic information, veteran status, marital status, pregnancy or related condition, or any other basis protected by law.

Apply for this Job