AI Engineer

Plutus21 is looking for an AI Engineer to join its core platform team with a focus on agent evaluation and governance. In this role, you will develop and maintain systems that rigorously test, approve, and monitor AI agents both before and after deployment. Ensuring governance rules are embedded and enforced in code at runtime is a key responsibility. You will collaborate closely with the Principal Platform Engineer, Cloud Engineer, as well as product and client engagement teams. This position plays a vital role in preparing AI capabilities for safe and compliant use by external clients.

Key Responsibilities

- Develop and maintain a comprehensive evaluation framework that includes datasets, metrics, defined thresholds, and a go-live gate for each AI agent capability.
- Conduct re-evaluations whenever there are changes to an agent’s model or instructions, identifying any regressions prior to release.
- Build a governed agent runtime environment that assigns accountability for each action, requires human approval when necessary, and enforces deterministic checks before irreversible operations.
- Implement rule-level and campaign-level approval processes for high-volume actions, including statistically sized review samples before release and automatic suspension if quality thresholds are not met.
- Design workflows for reviewing and evaluating AI-generated content that is shared outside the client organization.
- Ensure every model call is logged with details such as the model version, instruction set, data provenance, and applied quality checks.
- Maintain platform independence from any single model provider to ensure flexibility and resilience.
- Collaborate with the Principal Platform Engineer to translate written governance rules into executable validation code.
- Work closely with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
- Document design decisions clearly and thoroughly before implementation begins.

Required Qualifications

- Proven experience deploying multi-step, tool-using large language model (LLM) agents into production environments beyond prototype stages.
- Demonstrated ability to build evaluation datasets and metrics, set quality thresholds, and detect regressions caused by model or prompt changes.
- Solid understanding of applied statistics, including sample sizing, confidence intervals, and error rate estimation from samples.
- Experience designing systems where governance rules—such as approvals, checks, and kill switches—are enforced programmatically rather than relying on the model alone.
- Strong software engineering skills in Python or TypeScript, including writing tests, conducting code reviews, and developing maintainable services.
- Excellent written communication skills, capable of producing clear design documentation and working from established decisions.

Preferred Qualifications

- Experience with knowledge graphs or ontologies (e.g., LinkML, SHACL, OWL/RDF) and translating domain-specific rules into validation logic.
- Background in building human-in-the-loop systems, including review queues, escalation paths, and approval workflows.
- Familiarity with retrieval systems over governed data, including access control and data classification.
- Experience working in regulated industries such as financial services or privacy-sensitive environments.
- Exposure to multiple AI model providers, supporting platform independence and flexibility.

This role offers the opportunity to work at the forefront of AI governance and evaluation, ensuring that advanced AI agents operate safely and reliably in client-facing environments. If you have a strong engineering background combined with a passion for AI safety and governance, Plutus21 invites you to apply and contribute to shaping the future of AI deployment.