Applied Machine Learning Engineer – Evaluation

Overview

• Drive the development of advanced machine learning evaluation frameworks to ensure robust model reliability and performance across diverse applications.
• Collaborate with multidisciplinary teams to align evaluation strategies with real-world product objectives and cutting-edge technologies.
• Design scalable pipelines and tools to automate evaluation processes, enhancing efficiency and accuracy in model assessments.
• Develop innovative metrics and ground truth definitions to measure and improve machine learning model quality effectively.
• Create visualization dashboards to monitor trends, regressions, and evaluation results for actionable insights.
• Contribute to continuous workflow improvements, accelerating iteration cycles and enhancing signal quality.
• Work in a fully remote environment, fostering collaboration and innovation across diverse and talented teams.

Key Responsibilities & Duties

• Define and implement evaluation metrics and ground truth for machine learning systems, ensuring clarity and reproducibility.
• Develop automated pipelines for model evaluation and integrate results into centralized systems for seamless analysis.
• Create human-in-the-loop tools for error analysis and failure categorization to refine model performance.
• Build internal dashboards to visualize evaluation outcomes and monitor trends over time effectively.
• Collaborate with machine learning, product, and research teams to ensure evaluations align with practical use cases.
• Optimize workflows to enhance efficiency, reliability, and scalability of evaluation processes across projects.
• Influence and establish evaluation standards across teams to maintain consistency and uphold quality benchmarks.

Job Requirements

• Bachelor of Science degree in a relevant field is required.
• Minimum of 4 years of professional experience in applied machine learning or machine learning engineering.
• Proven expertise in designing and implementing machine learning evaluation frameworks in production environments.
• Proficiency in metrics design, ground truth construction, and automated evaluation pipelines.
• Experience with tools such as Weights & Biases for experiment tracking and analysis.
• Ability to develop internal tools and dashboards to support machine learning workflows.
• Preferred experience in natural language processing, computer vision, or multimodal model evaluation.