We are seeking a motivated professional to contribute to the development of large language model (LLM) evaluation and training datasets aimed at enhancing LLM capabilities in solving realistic software engineering challenges. This role involves creating verifiable software engineering tasks derived from public repository histories using a synthetic approach combined with human-in-the-loop validation. The project focuses on expanding dataset coverage across various programming languages, difficulty levels, and task types to ensure comprehensive training and evaluation.
Key Responsibilities
- Design and develop verifiable software engineering tasks based on public code repositories.
- Implement synthetic data generation methods integrated with human-in-the-loop processes to ensure task quality and accuracy.
- Expand dataset diversity by incorporating multiple programming languages and varying levels of task complexity.
- Collaborate with cross-functional teams to refine dataset requirements and improve evaluation metrics.
- Analyze and validate dataset outputs to maintain high standards of reliability and relevance for LLM training.
- Document methodologies and contribute to continuous improvement of dataset creation workflows.
Required Qualifications
- Strong background in software engineering with hands-on experience in multiple programming languages.
- Familiarity with version control systems and public code repositories such as GitHub.
- Experience in dataset creation, data annotation, or synthetic data generation techniques.
- Understanding of machine learning concepts, particularly related to natural language processing or LLMs.
- Ability to work collaboratively in a team environment and communicate technical information effectively.
- Detail-oriented mindset with a focus on data quality and verifiability.
Preferred Qualifications and Benefits
- Experience working with human-in-the-loop systems or crowdsourcing platforms for data validation.
- Knowledge of software engineering education or assessment methodologies.
- Prior involvement in research or projects related to LLM training or evaluation.
- Opportunities for professional growth within an innovative AI research environment.
- Exposure to cutting-edge developments in AI and software engineering intersection.
This position offers the chance to contribute to pioneering work at the intersection of artificial intelligence and software engineering, helping to build datasets that enable LLMs to better understand and solve real-world coding problems.
Job Details
What is your Competitive Advantage?
Mona Sharif
Senior Team Lead
Konext PVT. LTD
Babar Ahmed Khan
Territory Sales Manager
Engro Foods
Zohaib Ali
Marketing Manager
Digital Online Devices
Zubair Anjum
Business Excellence Specialist
Nestle Pakistan
Ali Aziz
CEO
Shopistan
Sehrish Azhar
Motivational Speaker & Blogger
Living Upbeat
Aqib Irshad
Head of HR, Training & OD
FINCA
Ayesha Shaukat
Social Media Marketing Manager
Ivoke
Muzammil Baig
Digital Marketing Manager
IT Services Group
Usman Habib
Recruitment & Training Executive
ACS Synergy (Pvt) Ltd.
