Contribute to cutting-edge AI evaluation in a flexible remote environment. Shape the future of AI coding agents with impactful engineering tasks. Collaborate independently while leveraging your professional expertise.
Senior Software Developer
in Information Technology ContractJob Detail
Job Description
Senior Software Developer Overview
- The Senior Software Developer role involves evaluating AI coding agents in realistic codebases and designing tasks to test their capabilities.
- Candidates will leverage their expertise in shipping production systems to assess and grade AI-generated outputs.
- Work includes self-directed, long-form tasks spanning multiple languages and problem types, fully remote with flexible scheduling.
- Tasks include bug fixes, feature implementation, refactoring, pull request reviews, debugging write-ups, and API contract creation.
- The role emphasizes professional judgment, attention to detail, and strong written communication skills.
- Candidates must be comfortable with terminal tools, git, debugging, and reading unfamiliar codebases.
- Experience with Docker, CI, and general familiarity with AI/LLM tools is beneficial but not mandatory.
- Ideal candidates are senior contributors with a track record of shipped systems and independent work.
Senior Software Developer Key Responsibilities & Duties
- Evaluate AI coding agents by designing and grading tasks based on professional engineering standards.
- Create realistic engineering tasks drawn from personal practice and assess AI agent outputs.
- Develop tests and rubrics to define correct deliverables and identify failures in agent outputs.
- Perform comparative analysis of AI outputs, providing detailed feedback and grading results.
- Collaborate remotely, managing tasks independently while adhering to high professional standards.
- Contribute to debugging, refactoring, and feature implementations with a focus on quality and precision.
- Utilize expertise in software engineering to enhance AI testing methodologies and outcomes.
- Engage in continuous learning and application of AI/LLM tools to improve task design and evaluation.
Senior Software Developer Job Requirements
- Minimum of 2 years professional software engineering experience, with a preference for senior-level contributors.
- Proficiency in at least one common stack, such as Python, Java, Go, Rust, or TypeScript.
- Strong skills in terminal tools, git, debugging, and rapid codebase orientation.
- Experience with Docker, CI systems, and familiarity with AI/LLM tools is advantageous.
- Ability to work independently, manage tasks remotely, and communicate effectively in writing.
- Demonstrated ability to ship and maintain production systems with high standards of quality.
- Titles such as Senior Software Engineer, Staff Engineer, or Tech Lead are indicative of ideal candidates.
- Open to specialists in frontend, backend, mobile, or infrastructure engineering with verifiable work.
- ShareAustin:
