Senior Software Engineer - AI Code Evaluation

Gramian Consultancy is a boutique firm specializing in IT professional services and engineering talent solutions. Leveraging deep expertise in software engineering and leadership, Gramian partners with companies to build high-performing teams by connecting them with professionals who align closely with their technical and cultural needs. Currently, we are seeking senior software engineers with over seven years of industry experience to contribute to the training and evaluation of large language models (LLMs). This role focuses on reviewing, analyzing, and enhancing AI-generated code by applying sound engineering judgment to evaluate correctness, security, scalability, maintainability, and production readiness. Unlike traditional software testing or manual QA roles, this position requires deep technical insight to analyze unfamiliar codebases, identify root causes of issues, compare alternative implementations, and develop robust solutions and evaluation criteria.

Key Responsibilities
- Review and evaluate AI-generated code across multiple programming languages and diverse software engineering scenarios.
- Assess code for correctness, reliability, security, scalability, readability, and maintainability.
- Identify bugs, logical errors, incomplete implementations, edge-case failures, and architectural weaknesses.
- Analyze unfamiliar codebases and understand the implications of proposed changes.
- Review bug fixes, feature implementations, refactoring efforts, API integrations, configuration changes, and database operations.
- Compare alternative implementations to determine if they meet technical requirements.
- Rewrite or improve code to produce high-quality reference solutions.
- Provide clear, technical explanations of identified issues and recommend improvements.
- Develop evaluation criteria, technical annotations, and rubrics for assessing code quality.
- Collaborate closely with researchers and engineering teams to design coding benchmarks and enhance LLM evaluation methodologies.

Required Qualifications
- Minimum of 7 years of professional software engineering experience.
- Strong proficiency in at least one programming language such as Python, JavaScript/TypeScript, Java, C++, Go, C#, Ruby, PHP, or Rust.
- Extensive experience building, maintaining, debugging, and reviewing production-grade software.
- Solid understanding of software design principles, including clean code practices, modular architecture, abstraction, error handling, and maintainability.
- Proven ability to identify functional, performance, security, and design issues within complex codebases.
- Excellent debugging, root-cause analysis, and problem-solving skills.
- Experience conducting code reviews and working within collaborative software development workflows.
- Familiarity with Git and modern software engineering practices.
- Strong written English skills with the ability to communicate technical feedback clearly and effectively.
- Good understanding of data structures, algorithms, APIs, databases, and application architecture.

This role offers an opportunity to work at the intersection of software engineering and AI, contributing to cutting-edge developments in large language model evaluation. Candidates passionate about code quality and AI innovation will find this position both challenging and rewarding.