Lead Data Engineer Azure Databricks

Job Title: Lead Data Engineer Azure Databricks

Location: Remote/Hybrid
Duration :6+ Months

Job Summary

We are looking for an experienced Lead Data Engineer to support enterprise data engineering, analytics, and cloud initiatives. The ideal candidate will have strong expertise in designing, developing, and optimizing large-scale data pipelines, cloud data platforms, and ETL/ELT processes.

The candidate will work closely with business stakeholders, data analysts, data scientists, and engineering teams to deliver reliable, scalable, and secure data solutions that support reporting, analytics, and business decision-making.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing solutions.
• Build and optimize ETL/ELT workflows for large-scale data integration.
• Work with Azure Databricks and other cloud-based data engineering technologies.
• Develop data solutions using SQL, Python, Spark, and related technologies.
• Work with structured and unstructured datasets to support analytics and reporting.
• Design and maintain data warehouses and data models.
• Ensure data quality, security, governance, and compliance.
• Collaborate with business and technical teams to understand data requirements.
• Troubleshoot performance issues and optimize data processing workflows.
• Follow cloud engineering best practices to deliver reliable and scalable solutions.

Required Qualifications

• 10+ years of overall IT experience.
• 5+ years of experience in cloud leadership roles.
• 5+ years of experience in Data Engineering or related data development roles.
• Strong experience with SQL and relational databases.
• Hands-on experience with ETL/ELT development.
• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Strong programming skills in Python, Spark, or similar technologies.
• Experience developing and optimizing large-scale data pipelines.
• Knowledge of data warehousing, dimensional modeling, and data integration.
• Understanding of data governance, security, and compliance practices.
• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

• Experience with Azure Databricks, Azure Data Factory, Azure Synapse, or Snowflake.
• Experience with Hadoop or other big data technologies.
• Healthcare or healthcare payer industry experience.
• Experience supporting Medicare, Medicaid, or healthcare analytics projects.
• Familiarity with Power BI, Tableau, or other reporting tools.
• Exposure to machine learning and advanced analytics environments.
• Experience working in regulated healthcare environments.