Principal Software Engineer, Orchestration

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. Roblox’s data platform powers analytics, ML, and product decisions for a platform serving 200M+ daily active users. As a Principal Software Engineer in the Data Platform Orchestration team, you will be the technical leader for the system that schedules and runs the company’s data workloads. Our team owns Apache Airflow and the Kubernetes (EKS) infrastructure underneath it, running 8,000+ DAGs every day, most of them Spark jobs. You will set the long-term architecture for how Roblox orchestrates data at scale, and you will be hands-on in making it faster, more reliable, and easier for hundreds of engineers to use. You Will: • Define the Orchestration Strategy: Own the multi-year architecture for Airflow and the surrounding platform, including scheduler and executor design, multi-tenancy, upgrades and migrations, and where the platform should evolve next. • Scale and Harden Airflow: Find and remove scaling limits across the scheduler, metadata database, executors, and DAG parsing. Make sure thousands of daily DAGs run with predictable latency and strong SLAs. • Own Orchestration on Kubernetes: Lead the design and operation of our EKS clusters for orchestration and Spark workloads, including autoscaling, resource isolation, cost efficiency, and deployment automation. • Build Platform Tooling for Internal Engineers: Design the APIs, custom operators and providers, libraries, and self-service tooling that let data teams build, deploy, and monitor pipelines safely. • Drive Observability and Reliability: Establish SLOs, alerting, and diagnostics for pipelines and platform components. Lead incident reviews and fix the root causes in code, not just in runbooks. • Lead Across Teams: Work with data platform, governance, data science, and product engineering teams to align roadmaps, guide technical decisions, and build consensus on complex initiatives. • Raise the Engineering Bar: Set standards for code quality, design, and operational readiness. Mentor staff, senior, and mid-level engineers. You Have: • B.S. equivalent in CS or sufficient experience. • [10+] years of experience designing, building, and operating production-grade, large-scale distributed systems. • Workflow Orchestration Depth: Deep, hands-on experience designing, running, and scaling a production data-related workflow orchestrator or job scheduler (thousands of workflows or DAGs a day). Apache Airflow is ideal, but experience with an equivalent system (Dagster, Prefect, Temporal, or a large in-house scheduler) is also welcome. You understand scheduler and executor design, metadata store performance, task isolation, retries and idempotency, and safe upgrades and migrations. • Kubernetes Expertise: Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling (Karpenter or Cluster Autoscaler), and GitOps-style deployment. • Software Engineering Foundations: Strong proficiency in Python, plus Go or Java, with a record of writing clean, well-tested