Senior Data Engineer Databricks

Job Title: Senior Data Engineer (Databricks)
Primary Location: Chicago, IL (Hybrid 3 days onsite / 2 days remote)
Position Type: Full-Time, Direct Hire

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field
• 6+ years of hands-on data engineering experience, with a track record of owning pipelines end to end: ingestion, validation, transformation, and governance
• Real, hands-on Databricks experience (Azure preferred; AWS or GCP acceptable), including Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, and Workflows/Jobs
• Strong coding skills in Python, SQL, and Apache Spark that go well beyond basic scripting
• Proven experience designing and building scalable ETL/ELT pipelines, including reusable, metadata-driven pipelines
• Experience with object storage, lakehouse engines, orchestration tools, and streaming or CDC pipelines
• Solid data modeling, schema design, and performance tuning experience
• Working knowledge of Git-based workflows, CI/CD, test automation, and monitoring and alerting
• Strong troubleshooting skills and a solid grasp of engineering fundamentals: scalability, reliability, and maintainability
• The ability to clearly explain what you built, why it mattered, and how it solved a business or platform problem

• Experience building or migrating a data platform from the ground up
• Infrastructure as Code (Terraform, Bicep, ARM) and Governance as Code experience
• Multi-cloud background with depth in Azure
• Legacy data warehousing and/or Hadoop experience
• Deep Databricks exposure beyond basic pipelines, such as enterprise-scale work and optimization
• A track record of mentoring and upskilling peers
• Agile delivery experience

• Lead the design and build of scalable data pipelines and models in Azure Databricks on a brand-new platform with no production code yet
• Own pipelines end to end, from ingesting source systems to transforming and modeling data to producing outputs for downstream analytics, BI, and AI/ML use cases
• Refactor and migrate multiple legacy data warehouses into the centralized Databricks lakehouse
• Work at the code level every day in Python, SQL, and Spark, building pipelines with Databricks Asset Bundles, Spark workflows, and API-driven jobs
• Establish engineering standards for CI/CD (GitHub/Azure DevOps), automated data testing, monitoring, and alerting
• Help shape and enforce standards for data modeling and governance
• Troubleshoot and optimize performance across the platform
• Mentor and coach junior engineers, conduct code reviews, and lead by example in design, coding, and testing practices
• Partner with Data Engineering, BI, AI, Finance, and product owners, and help translate technical work into business impact

The expected salary range for this position is $150,000-$164,000 per year, depending on experience and qualifications. This role also qualifies for comprehensive benefits such as health insurance, 401(k), and paid time off.