Databricks Data Engineer

Job Description:  We are looking for a highly experienced Data Engineer to join our product organization in a full-time role. This position focuses on Big Data systems design and data architecture, with a special emphasis on Spark and Delta/Data Lake technology. In this role, you will help design, develop, enhance, and maintain complex data pipeline and API products that manage business-critical operations and large-scale analytics. We are looking for applicants who demonstrate a capability to learn new concepts quickly and bring a robust background in data engineering or software engineering expertise. Responsibilities • Collaborate: Work collaboratively with other engineers to architect and implement complex systems with minimal oversight, while partnering with team leadership to identify how best to improve and expand platform capabilities. • Build & Design: Design, develop, and maintain complex data pipeline products that support business-critical operations and large-scale analytics applications. • Support the Team: Partner with analytics, data science, and engineering teams to understand and solve their unique data needs and challenges. • Continuous Learning: Dedicate time to staying current with the latest developments in the space and embrace new concepts to keep up with fast-moving data engineering technology. • Autonomy: Enjoy a role that offers strong independence and autonomy while contributing to the technical maturity of the organization. Qualifications & Skills • Experience: 5+ years of Data Engineering experience, including 4+ years designing and building Databricks data pipelines is essential. While Azure cloud experience is preferred, we are happy to consider experience with AWS, GCP, or other cloud platforms. • Technical Stack: • 4+ years of hands-on experience with Python, Pyspark, or SparkSQL is key. Experience with Scala is a plus. • 4+ years of experience with Big Data pipelines or DAG Tools (such as Airflow, Dbt, Data Factory, or similar). • 4+ years of Spark experience, especially with Databricks Spark and Delta Lake. • 4+ years of hands-on experience implementing Big Data solutions in a cloud ecosystem, including Data/Delta Lakes. • 5+ years of relevant software development experience, including Python 3.x, Django/Flask Frameworks, FastAPI (or other standard industry API), and relational databases (SQL/ORM). • Additional Skills (Great to have): • Experience with Microsoft Fabric specifically with Pyspark on Fabric and Fabric Pipelines • Experience with conceptual, logical, and/or physical database designs. • Strong SQL experience, specifically writing complex, highly optimized queries across large volumes of data. • Strong data modeling/profiling capabilities using Kimball/star schema methodology as well as medallion architecture. • Professional experience with Kafka, EventHub or other live streaming technologies. • Familiarity with database deployment pipelines (e.g., dacpac or similar). • Experience with unit testing or data quality frameworks. Beware of scams Our recruiting team may communicate with candidates via our @hitachisolutions.com domain email address and/or via our SmartRecruiters (Applicant Tracking System) notification@smartrecruiters.com domain email address regarding your application and interview requests.   All offers will originate from our @hitachisolutions.com domain email address. If you receive an offer or information from someone purporting to be an employee of Hitachi Solutions from any other domain, it may not be legitimate.