Senior Data Engineer

OAG is the essential intelligence partner transforming global travel. For nearly a century, we've been the foundation the travel industry builds on, trusted, proven, essential. Today, we're going beyond. We combine unmatched data quality, quantity and access with breakthrough technology and deep industry expertise to deliver what the industry truly needs: not just more data, but better answers. From real-time operational intelligence to strategic decision-making, we power the entire travel ecosystem, from airlines and airports to technology companies and financial institutions. We are actively embedding AI throughout our intelligence layer, combining nearly a century of proprietary data with the latest technology to deliver answers that were simply not possible before. We exist because travel moves the world, and we move travel. By transforming complexity into clarity and questions into answers, we give our partners the intelligence they need to thrive in an evolving world. For more information visit www.oag.com ABOUT THE ROLE: You will join a team of software and data engineers modernizing OAG’s core Flight Status data platform. You will play a key role in building high-throughput, near real-time ingestion and processing pipelines that handle global flight status data using Azure Databricks and Delta Lake . As OAG actively embeds AI across its intelligence layer and adopts modern agentic development practices, you will leverage AI-assisted tools to accelerate delivery, improve system observability, and support AI-driven data products. WHAT YOU’LL BE DOING: • Deliver Real-Time Data Pipelines: Build, scale, and maintain high-volume batch and Structured Streaming pipelines to ingest multi-source flight data into the Lakehouse platform. • Drive Engineering Standards: Implement reusable data patterns, automated data quality controls, and observability to ensure data consistency, accuracy, and reliability. • Optimize Platform Performance: Monitor and tune compute resources, streaming triggers, and storage for cost efficiency, low latency, and high scalability. • Automate Workflows & DevOps: Design automated job orchestration using Databricks Workflows and maintain CI/CD pipelines for reliable deployments. • Leverage Modern AI Workflows: Integrate AI-assisted tools into daily engineering practices to boost productivity, edge-case testing, and pipeline automation. • Collaborate & Lead: Work closely with product, analytics, and platform teams to define data contracts, review code, and mentor engineers across the domain. WHAT WE’RE LOOKING FOR: • Senior Data Engineering Experience: Strong track record building and scaling production-grade streaming and batch ETL/ELT data pipelines. • Databricks & Governance Mastery: Hands-on proficiency with Azure Databricks, Apache Spark (PySpark), Structured Streaming, Delta Lake, and Unity Catalog (for fine-grained access control, data governance, and lineage). • Advanced Pipeline Design: Deep understanding of modern data patterns, including Medallion architecture, Change Data Capture (CDC), SCD Type 1/2, MERGE/upsert strategies, and schema evolution. • Orchestration & Streaming: Practical experience building job dependencies with Databricks Workflows and integrating event streaming platforms (e.g., Azure Event Hubs, Kafka). • DevOps & Observability Focus: Strong experience with automated testing, CI/CD tools (Azure DevOps or GitHub Actions), and data observability practices. • AI Tooling Adoption: Active experience using AI coding assistants (e.g., GitHub Copilot, Cursor) to accelerate development while maintaining high code quality and ownership. • Ownership & Collaboration: Strong analytical skills, a results-oriented mindset, and excellent technical communication skills. IF YOU HA