InfoBeans is hiring an experienced Senior / Lead Data Engineer to design, build, and optimize enterprise‑scale data platforms. The ideal candidate has strong hands‑on expertise with Azure Databricks, PySpark, ADF, Delta Lake, and modern data warehousing. In this role, you will work closely with architects, analysts, and business stakeholders to deliver high‑performance, reliable, and scalable data solutions.If you thrive in fast‑paced environments and enjoy solving complex data challenges, we’d love to connect.Title: Senior / Lead Data Engineer (Databricks)Experience: 10+ yearsContract: C2H / W2 / 1099Duration: 12 monthsJob Description:Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategiesDelta Lake Internals & Performance Tuning - Optimizing Delta performance, Understanding Delta logs & transaction protocol, Time travel & schema evolution best practices, CDC (Change Data Capture) patterns, Multi-hop architecture (Bronze/Silver/Gold)Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies), Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using reposUnity Catalog & Enterprise Data Governance - Fine-grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions modelsDatabricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queriesStreaming & Real‑Time Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late-arriving data, Stateful stream processingAdvanced Cluster & Compute Management - Spot vs on-demand clusters, Photon runtime vs standard, Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboardsBest Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints)DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub ActionsNice to have:Azure Data Factory - Understanding of Pipelines, Activities, and Datasets, Linked Services configuration, Integration Runtime types (Auto‑IR, Self‑Hosted IR, Azure IR), Source and Sink concepts in ADFETL/ELT Data Integration - Building data ingestion pipelines (batch + incremental loads), Designing end‑to‑end data transformation
Click on the below icons to share this job to Linkedin, Twitter!
InfoBeans is hiring an experienced Senior / Lead Data Engineer to design, build, and optimize enterprise‑scale data platforms. The ideal candidate has strong hands‑on expertise with Azure Databricks, PySpark, ADF, Delta Lake, and modern data warehousing. In this role, you will work closely with architects, analysts, and business stakeholders to deliver high‑performance, reliable, and scalable data solutions.If you thrive in fast‑paced environments and enjoy solving complex data challenges, we’d love to connect.Title: Senior / Lead Data Engineer (Databricks)Experience: 10+ yearsContract: C2H / W2 / 1099Duration: 12 monthsJob Description:Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategiesDelta Lake Internals & Performance Tuning - Optimizing Delta performance, Understanding Delta logs & transaction protocol, Time travel & schema evolution best practices, CDC (Change Data Capture) patterns, Multi-hop architecture (Bronze/Silver/Gold)Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies), Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using reposUnity Catalog & Enterprise Data Governance - Fine-grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions modelsDatabricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queriesStreaming & Real‑Time Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late-arriving data, Stateful stream processingAdvanced Cluster & Compute Management - Spot vs on-demand clusters, Photon runtime vs standard, Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboardsBest Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints)DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub ActionsNice to have:Azure Data Factory - Understanding of Pipelines, Activities, and Datasets, Linked Services configuration, Integration Runtime types (Auto‑IR, Self‑Hosted IR, Azure IR), Source and Sink concepts in ADFETL/ELT Data Integration - Building data ingestion pipelines (batch + incremental loads), Designing end‑to‑end data transformation
yes
Send Us Your Feedback
Sign
In
Please check your email. We have sent you a password reset link.
This link will expire in one hour.
Please input your account's email
Apply to Lead Full Stack Engineer
InfoBeans • Charlotte, North Carolina
✓
Application Started!
Redirecting you to complete your application...
Get $5 added to your wallet for registering an account. Refer others and get $1 for each successful referral. Earn while you search for jobs and redeem in gift cards!Register to search from millions of jobs across the United States. Upload your resume and get matched with top employers today!
Welcome to SnapRecruit! Your account has been created via Google.
Get $5.00 Added to Your Wallet!
Upload your resume below and apply to this job to have $5.00 added directly to your wallet, activate your account, and qualify for high-priority employer matches.
Perks & Rewards You Receive with SnapRecruit:
$5.00 Resume Bonus
Added automatically to your wallet when you apply with your resume.
$1.00 Per Referral
Earn $1.00 cash rewards for every friend who joins SnapRecruit in the US.
Earn While You Search
Get credited $0.02 - $0.05 every time you explore sponsored job listings.
Amazon eGift Cards
Redeem your earned wallet funds for Amazon eGift Cards delivered to your email.
Get $5 added to your wallet for registering anRegister to search from millions of jobs across the United States. Upload your resume and get matched with top employers today!
account.
Refers others and get $1 for each successful referral.
Earn while you search for jobs and redeem in
gift cards!
Upload your resume for a free professional assessment
Loading, Please wait..!!
Send this job to my email
Please wait..!!
Confirmation
Are you sure?
List of resumes
Confirmation
It is working for Chrome Mozilla only for now
We use essential cookies to keep SnapRecruit secure and reliable. With your permission, we also use optional analytics, AI support chat, and push-alert cookies to improve your job search experience. You can change or withdraw your choice anytime.
Privacy Policy •
Cookie Policy •
Your Privacy Rights