Applied Machine Learning Engineer – Evaluation

Overview

• Lead the development of innovative evaluation frameworks for machine learning models, ensuring reliability and robust performance metrics.
• Collaborate with cross-functional teams to align evaluation strategies with real-world applications and product goals.
• Design scalable pipelines and tools to automate model evaluation processes, enhancing efficiency and accuracy.
• Implement advanced metrics and ground truth definitions to measure and improve model quality.
• Develop visualization tools and dashboards to monitor trends, regressions, and evaluation results effectively.
• Contribute to the continuous improvement of workflows, accelerating iteration cycles and enhancing signal quality.
• Work in a fully remote environment, fostering collaboration and innovation across diverse teams.

Key Responsibilities & Duties

• Define and implement evaluation metrics and ground truth for machine learning systems, ensuring clarity and reproducibility.
• Develop automated pipelines for model evaluation and integrate results into centralized systems.
• Create human-in-the-loop tools for error analysis and failure categorization.
• Build internal dashboards to visualize evaluation outcomes and monitor trends over time.
• Collaborate with machine learning, product, and research teams to ensure evaluations align with practical use cases.
• Optimize workflows to enhance efficiency, reliability, and scalability of evaluation processes.
• Influence and establish evaluation standards across teams to maintain consistency and quality.

Job Requirements

• Bachelor of Science degree in a relevant field is required.
• Minimum of 4 years of professional experience in applied machine learning or machine learning engineering.
• Proven expertise in designing and implementing machine learning evaluation frameworks in production environments.
• Proficiency in metrics design, ground truth construction, and automated evaluation pipelines.
• Experience with tools such as Weights & Biases for experiment tracking and analysis.
• Ability to develop internal tools and dashboards to support machine learning workflows.
• Preferred experience in natural language processing, computer vision, or multimodal model evaluation.