Senior AI Evaluation Engineer (CICD & Platform)

Your work days are brighter here. We’re obsessed with making hard work pay off, for our people, our customers, and the world around us. As a Fortune 500 company and a leading AI platform for managing people, money, and agents, we’re shaping the future of work so teams can reach their potential and focus on what matters most. The minute you join, you’ll feel it. Not just in the products we build, but in how we show up for each other. Our culture is rooted in integrity, empathy, and shared enthusiasm. We’re in this together, tackling big challenges with bold ideas and genuine care. We look for curious minds and courageous collaborators who bring sun-drenched optimism and drive. Whether you're building smarter solutions, supporting customers, or creating a space where everyone belongs, you’ll do meaningful work with Workmates who’ve got your back. In return, we’ll give you the trust to take risks, the tools to grow, the skills to develop and the support of a company invested in you for the long haul. So, if you want to inspire a brighter work day for everyone, including yourself, you’ve found a match in Workday, and we hope to be a match for you too. About the Team The Forward Intelligence team owns the skills that power Workday's deployment agent—turning the deep configuration expertise and hard-earned wisdom of our Domain Experts and Deployment Consultants into production-grade, fully tested skills that the agent can execute lawfully in live customer tenants. What we own: The end-to-end skill engine and governance ecosystem, spanning skill knowledge extraction, promotion pipelines, evaluation frameworks, self-correcting drift detection, and developer authoring environments. How we build: Automated testing harnesses and release-gating pipelines running at enterprise scale, ensuring deterministic execution, strict schema compliance, and zero regressions for skills executing in live customer tenants. Agentic engineering: We build evaluation tooling, synthetic datasets, and automated red-teaming pipelines that help engineers test how AI agents and Model Context Protocol (MCP) skills perform across complex tenant environments. How we work: Close end-to-end collaboration across domain experts, AI platform engineers, and site reliability teams to transform domain knowledge into trustworthy, production-ready AI skills. About the Role We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform. In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments. You will bridge the gap between AI skill development and production platform engineering—leveraging modern evaluation frameworks (e.g., LangSmith, Braintrust, DeepEval) alongside CI/CD pipelines to evaluate agent trajectories, test schema contracts, and certify every skill before deployment. Key Responsibilities • Continuous Evaluation: . Build tiered evaluation pipelines—separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates. • CI/CD Quality Gates & MLOps Infrastructure: Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated "pass/fail" release gates on every pull request. • Deterministic Code & Schema Validation: . Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge-case evaluati