Loading, Please wait..!!
Data Engineer & Databricks Tech Lead
-
Posted on: Nov 19, 2026
-
New York City, New York
-
Onsite
-
Full-time
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for Data Engineer & Databricks Tech Lead roles in New York.
View Salary Guide →Find your match !
Job Title :
Data Engineer & Databricks Tech Lead
Job Type :
Full-Time
Job Location :
New York City New York United States
Remote :
No
Job Description :
- Arlington / Dallas, TX
- Fort Lauderdale, FL
- Louisville, KY
- Boston, MA
- Nashville, TN
- Chicago, IL
- New York City, NY
- Tampa, FL
Design, build, and lead delivery of scalable data pipelines and lakehouse solutions on Databricks. The Tech Lead sets technical direction, enforces engineering standards, and mentors a team of data engineers while remaining hands-on in architecture and development.
Key Technical Responsibilities- Architect and implement end-to-end data solutions on the Databricks Lakehouse Platform using the medallion (bronze/silver/gold) pattern
- Build and optimize batch and streaming pipelines with PySpark, Spark SQL, and Structured Streaming
- Design and manage Delta Lake tables, including schema evolution, time travel, Z-ordering, partitioning, and compaction
- Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and tools such as Airflow or Azure Data Factory
- Implement data governance and security with Unity Catalog (access controls, lineage, auditing)
- Tune Spark jobs and cluster configurations for performance and cost efficiency
- Establish CI/CD, testing, and DevOps practices for data pipelines (Git, Databricks Asset Bundles, Terraform, Azure DevOps/GitHub Actions)
- Implement data quality, monitoring, and observability frameworks
- Lead code reviews, define best practices, and provide technical guidance to engineers
- Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into technical designs
- Support integration of ML/AI workloads (MLflow, feature engineering) where applicable
- 10+ years of experience in data engineering; 3+ years in a technical lead role
- 4+ years of hands-on Databricks experience in production environments
- Expert-level Python (PySpark) and SQL
- Strong working knowledge of Apache Spark internals, including the execution model, shuffles, and join strategies
- Deep experience with Delta Lake and lakehouse architecture
- Experience with at least one major cloud platform (Azure, AWS, or GCP) and its data services (e.g., ADLS/S3, Event Hubs/Kinesis, Synapse/Redshift)
- Solid data modeling skills (dimensional modeling, Data Vault, star/snowflake schemas)
- Experience with ETL/ELT design, and with ingesting from APIs, databases, files, and streaming sources (Kafka)
- Proficiency with version control and CI/CD pipelines
- Strong understanding of data governance, security, and compliance practices
- Databricks certifications (Data Engineer Associate/Professional, Spark Developer)
- Experience with Unity Catalog, DLT, Databricks SQL, and Auto Loader
- Infrastructure-as-code experience (Terraform)
- Familiarity with dbt, Great Expectations, or similar data quality tooling
- Experience migrating legacy data warehouses or Hadoop workloads to Databricks
- Exposure to MLOps and MLflow
- Experience in large enterprise environments
Position Details
Posted:
Nov 19, 2026
Reference Number:
8297-8310
Employment:
Full-time
City:
New York City
Job Origin:
CIEPAL_ORGANIC_FEED
Share this job:
Please wait..!!

