Posted Jul 19, 2026 18 views
We are seeking a highly skilled Microsoft 365 / Dynamics 365 Copilot Agent Developer with extensive experience in AI agent testing, evaluation, quality engineering, and automation. The ideal candidate will be responsible for designing, developing, testing, and optimizing enterprise-grade Copilot agents using Microsoft Copilot Studio, Microsoft 365 Copilot, Dynamics 365 Copilot, Power Platform, Azure AI, and Copilot Studio Kit. This role focuses on building scalable test frameworks, automated evaluation pipelines, benchmark datasets, and quality validation processes specifically tailored for Generative AI solutions within enterprise environments.
Key Responsibilities
Design and develop scalable test automation frameworks and evaluation pipelines for Microsoft 365 and Dynamics 365 Copilot agents. Utilize Copilot Studio Kit, Azure Application Insights, AI Builder, Dataverse, Direct Line APIs, and Azure AI services to validate agent accuracy, groundedness, response quality, relevance, safety, and overall business effectiveness. Implement multi-turn conversational agent testing, including response match, topic match, generative answer validation, and attachment testing. Create and maintain automated regression testing frameworks to ensure continuous quality assurance of Copilot solutions.
Analyze conversation transcripts, telemetry data, knowledge retrieval effectiveness, and agent KPIs to identify areas for improvement. Apply prompt engineering techniques, Retrieval-Augmented Generation (RAG), and large language model (LLM) evaluation methods to enhance agent performance. Conduct hallucination detection, groundedness testing, benchmark management, and continuous quality monitoring to uphold AI governance and Responsible AI principles.
Collaborate closely with solution architects, business stakeholders, developers, and QA teams to ensure Copilot agents meet enterprise standards for reliability, security, compliance, and user satisfaction. Integrate with Azure DevOps and GitHub CI/CD pipelines while implementing Microsoft Entra ID security best practices. Utilize Power BI for analytics and reporting on agent performance and quality metrics.
Required Qualifications
Proven experience with Microsoft Copilot Studio, Microsoft 365 Copilot extensibility, and Dynamics 365 Copilot development. Strong hands-on skills in Power Platform components including Power Apps, Power Automate, and Dataverse. Expertise in Azure AI Services and Azure OpenAI, including Copilot Studio Kit (Agent Kit) and AI Builder. Demonstrated ability in prompt engineering and Retrieval-Augmented Generation (RAG) techniques.
Experience in multi-turn conversation testing and generative AI evaluation. Proficiency in developing agent benchmarks and conducting groundedness, relevance, completeness, and accuracy testing. Skilled in hallucination detection and response, topic, and attachment match testing methodologies. Familiarity with Azure Application Insights, Direct Line API integration, and Power BI analytics.
Knowledge of Microsoft Entra ID security protocols and CI/CD processes using Azure DevOps or GitHub. Experience building test automation frameworks and applying AI governance and Responsible AI principles. Ability to analyze conversation transcripts and monitor agent performance with KPI reporting. Understanding of Dynamics 365 Sales, Customer Service, and Customer Insights modules.
Preferred Qualifications and Certifications
AI-102 Azure AI Engineer Associate
PL-400 Power Platform Developer
PL-600 Power Platform Solution Architect
MB-210 Dynamics 365 Sales
MB-230 Dynamics 365 Customer Service
Microsoft Copilot Studio Applied Skills certification
This position offers a unique opportunity to work at the forefront of AI-driven enterprise solutions, contributing to the development and quality assurance of innovative Copilot agents that drive business productivity and enhance user engagement.
Skills
Job Details
ITC WORLDWIDE
What is your Competitive Advantage?
Muzammil Baig
Digital Marketing Manager
IT Services Group
Aqib Irshad
Head of HR, Training & OD
FINCA
Ali Aziz
CEO
Shopistan
Zohaib Ali
Marketing Manager
Digital Online Devices
Usman Habib
Recruitment & Training Executive
ACS Synergy (Pvt) Ltd.
Babar Ahmed Khan
Territory Sales Manager
Engro Foods
Mona Sharif
Senior Team Lead
Konext PVT. LTD
Zubair Anjum
Business Excellence Specialist
Nestle Pakistan
Sehrish Azhar
Motivational Speaker & Blogger
Living Upbeat
Ayesha Shaukat
Social Media Marketing Manager
Ivoke
