Skip to main content
Loading
loadingbar
Loading, Please wait..!!

Data Engineer & Databricks Tech Lead

  • Data Engineer & Databricks Tech Lead job type Posted on: Nov 19, 2026
  • Data Engineer & Databricks Tech Lead experience level VSG Business Solutions LLC
  • Data Engineer & Databricks Tech Lead employment type New York City, New York
  • Data Engineer & Databricks Tech Lead employment type Onsite
  • Data Engineer & Databricks Tech Lead salary Full-time

Point Apply Here APPLY LATER

Curious about compensation?

Explore the historical salary trends, average pay, and estimated compensation for Data Engineer & Databricks Tech Lead roles in New York.

View Salary Guide →

Job Title :

Data Engineer & Databricks Tech Lead

Job Type :

Full-Time

Job Location :

New York City New York United States

Remote :

No

Jobcon Logo Job Description :

Locations -
  • Arlington / Dallas, TX
  • Fort Lauderdale, FL
  • Louisville, KY
  • Boston, MA
  • Nashville, TN
  • Chicago, IL
  • New York City, NY
  • Tampa, FL
Role Summary

Design, build, and lead delivery of scalable data pipelines and lakehouse solutions on Databricks. The Tech Lead sets technical direction, enforces engineering standards, and mentors a team of data engineers while remaining hands-on in architecture and development.

Key Technical Responsibilities
  • Architect and implement end-to-end data solutions on the Databricks Lakehouse Platform using the medallion (bronze/silver/gold) pattern
  • Build and optimize batch and streaming pipelines with PySpark, Spark SQL, and Structured Streaming
  • Design and manage Delta Lake tables, including schema evolution, time travel, Z-ordering, partitioning, and compaction
  • Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and tools such as Airflow or Azure Data Factory
  • Implement data governance and security with Unity Catalog (access controls, lineage, auditing)
  • Tune Spark jobs and cluster configurations for performance and cost efficiency
  • Establish CI/CD, testing, and DevOps practices for data pipelines (Git, Databricks Asset Bundles, Terraform, Azure DevOps/GitHub Actions)
  • Implement data quality, monitoring, and observability frameworks
  • Lead code reviews, define best practices, and provide technical guidance to engineers
  • Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into technical designs
  • Support integration of ML/AI workloads (MLflow, feature engineering) where applicable
Required Qualifications
  • 10+ years of experience in data engineering; 3+ years in a technical lead role
  • 4+ years of hands-on Databricks experience in production environments
  • Expert-level Python (PySpark) and SQL
  • Strong working knowledge of Apache Spark internals, including the execution model, shuffles, and join strategies
  • Deep experience with Delta Lake and lakehouse architecture
  • Experience with at least one major cloud platform (Azure, AWS, or GCP) and its data services (e.g., ADLS/S3, Event Hubs/Kinesis, Synapse/Redshift)
  • Solid data modeling skills (dimensional modeling, Data Vault, star/snowflake schemas)
  • Experience with ETL/ELT design, and with ingesting from APIs, databases, files, and streaming sources (Kafka)
  • Proficiency with version control and CI/CD pipelines
  • Strong understanding of data governance, security, and compliance practices
Preferred Qualifications
  • Databricks certifications (Data Engineer Associate/Professional, Spark Developer)
  • Experience with Unity Catalog, DLT, Databricks SQL, and Auto Loader
  • Infrastructure-as-code experience (Terraform)
  • Familiarity with dbt, Great Expectations, or similar data quality tooling
  • Experience migrating legacy data warehouses or Hadoop workloads to Databricks
  • Exposure to MLOps and MLflow
  • Experience in large enterprise environments

Jobcon Logo Position Details

Posted:

Nov 19, 2026

Reference Number:

8297-8310

Employment:

Full-time

City:

New York City

Job Origin:

CIEPAL_ORGANIC_FEED

Share this job:

  • linkedin

Jobcon Logo
A job sourcing event
In Dallas Fort Worth
Aug 19, 2017 9am-6pm
All job seekers welcome!

Data Engineer & Databricks Tech Lead    Apply

Click on the below icons to share this job to Linkedin, Twitter!

Locations -

  • Arlington / Dallas, TX
  • Fort Lauderdale, FL
  • Louisville, KY
  • Boston, MA
  • Nashville, TN
  • Chicago, IL
  • New York City, NY
  • Tampa, FL
Role Summary

Design, build, and lead delivery of scalable data pipelines and lakehouse solutions on Databricks. The Tech Lead sets technical direction, enforces engineering standards, and mentors a team of data engineers while remaining hands-on in architecture and development.

Key Technical Responsibilities
  • Architect and implement end-to-end data solutions on the Databricks Lakehouse Platform using the medallion (bronze/silver/gold) pattern
  • Build and optimize batch and streaming pipelines with PySpark, Spark SQL, and Structured Streaming
  • Design and manage Delta Lake tables, including schema evolution, time travel, Z-ordering, partitioning, and compaction
  • Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and tools such as Airflow or Azure Data Factory
  • Implement data governance and security with Unity Catalog (access controls, lineage, auditing)
  • Tune Spark jobs and cluster configurations for performance and cost efficiency
  • Establish CI/CD, testing, and DevOps practices for data pipelines (Git, Databricks Asset Bundles, Terraform, Azure DevOps/GitHub Actions)
  • Implement data quality, monitoring, and observability frameworks
  • Lead code reviews, define best practices, and provide technical guidance to engineers
  • Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into technical designs
  • Support integration of ML/AI workloads (MLflow, feature engineering) where applicable
Required Qualifications
  • 10+ years of experience in data engineering; 3+ years in a technical lead role
  • 4+ years of hands-on Databricks experience in production environments
  • Expert-level Python (PySpark) and SQL
  • Strong working knowledge of Apache Spark internals, including the execution model, shuffles, and join strategies
  • Deep experience with Delta Lake and lakehouse architecture
  • Experience with at least one major cloud platform (Azure, AWS, or GCP) and its data services (e.g., ADLS/S3, Event Hubs/Kinesis, Synapse/Redshift)
  • Solid data modeling skills (dimensional modeling, Data Vault, star/snowflake schemas)
  • Experience with ETL/ELT design, and with ingesting from APIs, databases, files, and streaming sources (Kafka)
  • Proficiency with version control and CI/CD pipelines
  • Strong understanding of data governance, security, and compliance practices
Preferred Qualifications
  • Databricks certifications (Data Engineer Associate/Professional, Spark Developer)
  • Experience with Unity Catalog, DLT, Databricks SQL, and Auto Loader
  • Infrastructure-as-code experience (Terraform)
  • Familiarity with dbt, Great Expectations, or similar data quality tooling
  • Experience migrating legacy data warehouses or Hadoop workloads to Databricks
  • Exposure to MLOps and MLflow
  • Experience in large enterprise environments

Loading
Please wait..!!