Lead Data Engineer

Job Title: Senior Databricks Engineer
Location: 100% Remote
Interview Process: Video (2 Rounds)

Job Description: Lead/Senior Databricks Data Engineer Position: Lead Databricks Data Engineer / Databricks Architect Experience: 10+ Years Location: Remote

Job Summary We are looking for an experienced Lead Databricks Data Engineer / Databricks Architect with 10+ years of overall experience in Data Engineering and strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud-based data platforms. The candidate will be responsible for designing and implementing scalable Lakehouse architectures, enterprise data pipelines, data integration solutions, governance frameworks, and high-performance analytics platforms using Databricks.

• Design and develop scalable data engineering solutions using Databricks and Lakehouse architecture.
• Build robust ETL/ELT pipelines using PySpark, Spark SQL, Python, and SQL.
• Design and implement Bronze, Silver, and Gold/Medallion architecture.
• Develop and optimize Delta Lake tables, including MERGE, schema evolution, Change Data Feed, and incremental processing.
• Build batch and real-time/streaming pipelines using Structured Streaming, Auto Loader, and Lakeflow.
• Develop and manage Databricks Jobs/Workflows for pipeline orchestration, scheduling, dependencies, retries, and monitoring.
• Implement enterprise data governance using Unity Catalog, including access control, data lineage, auditing, catalogs, schemas, and external locations.
• Perform Spark and Databricks performance tuning, including cluster configuration, partitioning, caching, query optimization, Photon, and workload optimization.
• Design data models supporting Data Warehousing, BI, Analytics, and AI/ML workloads.
• Integrate Databricks with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Work with cloud services such as AWS S3, Azure ADLS Gen2, Azure Data Factory, AWS Glue, Synapse, Event Hubs/Kafka/Kinesis, as applicable.
• Implement CI/CD pipelines using Git, Azure DevOps/GitHub/Jenkins and Databricks deployment capabilities.
• Work with Terraform/IaC for infrastructure provisioning and automation.
• Troubleshoot production pipeline failures, performance issues, data-quality problems, and Spark/cluster issues.
• Establish data quality, monitoring, logging, and observability practices.
• Provide technical leadership, code reviews, architecture guidance, and mentorship to junior/mid-level engineers.
• Collaborate with Data Architects, Data Scientists, Business Analysts, DevOps teams, and application teams.

• Databricks
• Databricks Lakehouse Platform
• Delta Lake
• Unity Catalog
• Databricks Workflows/Jobs
• Lakeflow / Delta Live Tables
• Auto Loader
• Databricks SQL
• Databricks notebooks
• Databricks Asset Bundles
• Photon
• Cluster/workload optimization
• Big Data
• Apache Spark
• PySpark
• Spark SQL
• Structured Streaming
• Kafka
• Batch and real-time data processing
• Python
• SQL
• PySpark
• Scala
• AWS: S3, Glue, EMR, Lambda, Redshift, IAM, Kinesis
• Azure: ADLS Gen2, ADF, Synapse, Azure DevOps, Event Hubs, Key Vault
• Google Cloud Platform: GCS, BigQuery, Dataflow, Pub/Sub
• ETL/ELT
• Data Warehousing
• Dimensional Modeling
• Data Lake/Lakehouse
• Medallion Architecture
• CDC
• Data Quality
• Data Governance
• Metadata and Data Lineage
• Git
• Azure DevOps / GitHub
• Jenkins
• Terraform
• CI/CD automation
• Infrastructure as Code

• MLflow
• Databricks Machine Learning
• Feature Store
• Mosaic AI / GenAI
• dbt
• Apache Airflow
• Power BI / Tableau
• Delta Sharing
• Lakehouse Federation
• Liquid Clustering
• Data security, PII masking

Qualifications

Bachelor's degree in Computer Science, Engineering, Information Technology, or related field. 10+ years of experience in Data Engineering / Big Data / Analytics. 4+ years of hands-on Databricks experience preferred. Strong experience designing enterprise-scale data platforms. Demonstrated experience leading technical projects and mentoring engineers. Strong communication and stakeholder-management skills.

For applications and inquiries,