Databricks Engineer

Databricks Data EngineerLocation: New York, NY — HybridAbout the RoleWe are seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions. You will work closely with data architects, analysts and business stakeholders to deliver reliable, high-quality data for analytics, reporting and AI initiatives.This is a hands-on engineering role requiring strong experience with Databricks, PySpark and SQL.Key ResponsibilitiesDesign, develop and maintain data pipelines using Databricks, PySpark and SQL.Build batch and streaming workflows to ingest data from databases, APIs and enterprise applications.Develop and maintain Delta Lake tables across bronze, silver and gold layers.Implement data transformation, validation and quality controls to ensure accurate, reliable datasets.Optimise Spark workloads, cluster configurations and processing costs.Implement governance, access controls and data discovery through Unity Catalog.Troubleshoot pipeline failures and resolve performance and reliability issues.Build reusable engineering components and follow coding, testing and documentation standards.Support version control and automated deployments through Git and CI/CD pipelines.Collaborate with stakeholders to translate business requirements into technical solutions.Required ExperienceStrong commercial experience in data engineering, including hands-on Databricks development.Proficiency in Python, PySpark and SQL.Experience building ETL/ELT pipelines and working with large datasets.Practical knowledge of Apache Spark, Delta Lake and lakehouse architecture.Experience with at least one major cloud platform: Azure, AWS or GCP.Understanding of data modelling, warehousing and data quality practices.Experience troubleshooting distributed processing workloads and improving performance.Familiarity with Git, CI/CD and software engineering best practices.Strong communication skills and the ability to collaborate with technical and business teams.Ability to work on a hybrid basis in New York.Preferred ExperienceDatabricks certification, such as Data Engineer Associate or Professional.Experience with Unity Catalog, Lakeflow Jobs and Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables).Experience with streaming, change data capture and incremental processing.Familiarity with Terraform or other infrastructure-as-code tools.Experience supporting data platforms used for analytics, machine learning or AI.