Senior Data Scientist - Generative AI

Tech Stack

Responsibilities

• Develop Evaluation Frameworks: Design and operationalize rigorous evaluation systems for GenAI features or internal AI Agents, including experiment design, dataset design, label reliability analysis, and implementing and finetuning LLM-as-judge methods.
• Run Rigorous Experiments: Conduct online experiments (A/B tests) and causal inference to quantify the impact of GenAI features or AI-assisted coding tools, identifying opportunities, measuring lift, and ensuring statistical rigor.
• Define Success Metrics: Partner with cross-functional teams to define leading/lagging indicators for GenAI safety and user satisfaction, or for engineering productivity and code health.
• Build Automated Systems: Research and apply state-of-the-art methodologies to build reproducible evaluation tooling and agentic workflows that lift rigor and efficiency across the company.
• Drive Strategy & Visibility: Develop dashboards and reporting frameworks that reveal trends and translate complex data into clear, prioritized recommendations for leadership.

Soft Skills

Benefits

• Equity
• Health Insurance

Culture

Requirements