Contribute to cutting-edge AI coding evaluations in a fully remote role. Collaborate with innovative technologies to enhance coding agent capabilities. Flexible work arrangements and impactful engineering opportunities.
Software Engineer
in Information Technology ContractJob Detail
Job Description
Software Engineer Overview
- As a Software Engineer, contribute to evaluating AI coding agents within realistic codebases.
- Design and test tasks that assess AI agent performance against professional standards.
- Engage in long-form, self-directed work across multiple programming languages and problem types.
- Work remotely without production on-call or shift work requirements.
- Leverage your engineering judgment to grade AI agent outputs effectively.
- Collaborate with cutting-edge AI technologies to enhance coding agent capabilities.
- Contribute to creating rubrics and tests defining correct deliverables for AI agents.
- Analyze and improve AI coding agent outputs by identifying and flagging failures.
Software Engineer Key Responsibilities & Duties
- Evaluate AI coding agents' performance on tasks derived from real-world engineering practices.
- Design and implement tasks such as bug fixes, feature development, and refactoring.
- Conduct pull request reviews and debugging write-ups based on logs and traces.
- Compare outputs from AI agents, determining quality and areas of improvement.
- Develop tests and rubrics to evaluate deliverables against professional standards.
- Identify and document issues such as brittle changes, missed edge cases, and fabricated files.
- Collaborate with AI/LLM tools to enhance coding agent performance.
- Work independently to deliver high-quality assessments and improvements.
Software Engineer Job Requirements
- Professional experience in software engineering, with 2+ years preferred.
- Proficiency in at least one common programming stack such as Python, JavaScript, or Java.
- Comfortable with terminal operations, debugging, and unfamiliar codebases.
- Experience with Docker and CI tools is advantageous.
- Strong written communication skills for explaining technical issues clearly.
- Familiarity with AI/LLM tools, with user-level expertise being sufficient.
- Ability to work independently and remotely, managing self-directed tasks effectively.
- Depth in specialized areas such as frontend, backend, or mobile engineering is a plus.
- ShareAustin:
