AI Engineer

Plutus21 is looking for an AI Engineer to join its core platform team, focusing on agent evaluation and governance. In this role, you will be responsible for building and maintaining systems that test, approve, and monitor AI agents both before and after deployment. You will ensure governance rules are enforced programmatically at runtime, working closely with the Principal Platform Engineer, Cloud Engineer, as well as product and client engagement teams. This position plays a crucial role in preparing AI capabilities for safe and compliant use by external clients.

Key Responsibilities

- Develop and maintain a comprehensive evaluation framework that includes datasets, metrics, defined thresholds, and go-live gates for each AI agent capability.
- Conduct re-evaluations whenever there are changes to an agent’s model or instructions, identifying any regressions prior to release.
- Design and implement a governed agent runtime environment that assigns accountability for each action, requires human approval when necessary, and applies deterministic checks before irreversible actions are taken.
- Create rule-level and campaign-level approval processes for high-volume actions, including statistically sized review samples before release and automatic suspension if quality thresholds are not met.
- Establish review and evaluation workflows for AI-generated content that is shared outside the client organization.
- Ensure that every model invocation is logged with details such as the model version, instruction set, data provenance, and quality checks applied.
- Maintain platform independence from any single AI model provider to ensure flexibility and resilience.
- Collaborate with the Principal Platform Engineer to translate written governance rules into executable code validations.
- Work closely with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
- Document all design decisions clearly and thoroughly before implementation begins.

Required Qualifications

- Proven experience deploying multi-step, tool-using large language model (LLM) agents into production environments beyond prototype stages.
- Demonstrated ability to build evaluation datasets and metrics, set quality thresholds, and detect regressions across model or prompt changes.
- Practical knowledge of applied statistics, including sample sizing, confidence intervals, and error rate estimation from samples.
- Experience designing systems where governance rules—such as approvals, checks, and kill switches—are enforced programmatically rather than relying solely on the model.
- Strong software engineering skills in Python or TypeScript, including writing tests, performing code reviews, and developing maintainable services.
- Excellent written communication skills with the ability to produce clear design documentation and work effectively from documented decisions.

Preferred Qualifications

- Experience with knowledge graphs or ontologies (e.g., LinkML, SHACL, OWL/RDF) and translating domain rules into automated validation.
- Background in building human-in-the-loop systems, including review queues, escalation paths, and approval workflows.
- Familiarity with retrieval systems over governed data, including access control and data classification.
- Experience working in regulated industries such as financial services or privacy-sensitive environments.
- Exposure to multiple AI model providers, supporting platform independence and flexibility.

This role offers an opportunity to work at the forefront of AI governance, ensuring that advanced AI agents operate safely, transparently, and in compliance with client and regulatory requirements.