Databricks Engineer

Position: Databricks EngineerLocation: Englewood Cliffs, New Jersey / New York, NY (final approved worksite to be confirmed)Work Arrangement: 100% Onsite — No Remote or HybridStart Date: Immediate Availability RequiredBackground Check: MandatoryExperience: 5+ Years of Relevant Data Engineering ExperienceOverviewThe Databricks Engineer will design, develop, optimize, and support enterprise-grade data engineering solutions using Databricks, Apache Spark, Python, SQL, and PySpark. Working in a fully onsite environment, this individual will build scalable ETL/ELT pipelines and lakehouse architectures that transform structured and unstructured data into reliable, accessible, and business-ready datasets.This position requires strong hands-on engineering capabilities, including Delta Lake implementation, distributed data processing, data modeling, performance tuning, streaming data processing, and integration with cloud platforms such as AWS, Azure, or GCP.The engineer will collaborate closely with data scientists, analysts, technical teams, and business stakeholders to ensure data pipelines support analytical, operational, and enterprise requirements.Success will be measured by the reliability and accuracy of delivered pipelines, performance and scalability of data workflows, efficiency of cloud resource utilization, and successful delivery of production-ready data solutions.Purpose: Play a central role in building and improving the data engineering infrastructure that enables enterprise analytics, reporting, and data-driven decision-making. Your workObjectives1. Design and Deliver Reliable Databricks ETL/ELT Pipelines. Within the first 30–90 days, assess assigned data sources, transformation requirements, and existing processing workflows to design and implement production-ready ETL/ELT pipelines using Databricks, Apache Spark, Python, SQL, and PySpark. Develop reusable and maintainable processing components that support structured and unstructured data while meeting agreed business requirements. Incorporate appropriate validation, error handling, monitoring, and operational controls to support reliable data delivery. Success will be measured by acceptance of completed pipelines, data accuracy, successful scheduled execution, and achievement of agreed processing and delivery targets.2. Build and Maintain a Scalable Delta Lakehouse Architecture. Within the first 60–120 days, contribute to the design and enhancement of a scalable data lakehouse environment using Databricks, Delta Lake, and appropriate cloud storage and processing services. Implement data ingestion, transformation, storage, and consumption patterns that support analytical and operational workloads, including streaming data where required. Apply appropriate data modeling, partitioning, schema management, and data organization practices to improve maintainability and accessibility. Success will be measured by successful integration of assigned datasets, adherence to agreed architectural standards, reliable data availability, and demonstrated support for expected workload volumes.3. Optimize Data Processing Performance, Scalability, and Cost. Within the first 90–150 days, evaluate existing Spark workloads, Databricks jobs, SQL transformations, and cloud resource usage to identify performance bottlenecks and inefficient processing patterns. Implement improvements involving query optimization, data layout, partitioning, cluster configuration, and appropriate Delta Lake capabilities while preserving data correctness. Establish baseline measurements and compare results after approved changes to demonstrate improvements in processing efficiency and operational stability. Success will be measured by verified improvements against agreed benchmarks for job execution time, resource consumption, processing throughput, or cost per workload.4. Deliver Production-Ready Data Solutions with Effective Operational Support. Within the first 90–180 days, take ownership of assigned data engineering deliverables from technical design through development, testing, deployment, monitoring, and production support. Collaborate with data scientists, analysts, business stakeholders, and platform engineers to ensure that delivered datasets and pipelines meet functional, technical, and operational requirements. Use Git, approved CI/CD practices, monitoring capabilities, and structured troubleshooting methods to reduce deployment risk and address production issues. Success will be measured by timely delivery of accepted solutions, successful deployments, reliable pipeline operations, effective incident resolution, and positive feedback from downstream data consumers.1. Assess the Existing Databricks and Cloud Data Environment. During the first 15–30 days, review the assigned Databricks workspace, Apache Spark workloads, cloud integrations, source systems, and existing ETL/ELT processes. Identify key datasets, transformation requirements, architectural dependencies, current monitoring practices, and known technical issues.2. Develop and Validate Data Ingestion and Transformation Pipelines. During the first 30–60 days, build or enhance assigned batch and, where required, streaming pipelines using Databricks, Python, PySpark, SQL, and Apache Spark. Implement transformation logic, data quality checks, exception handling, and appropriate processing controls for structured and unstructured data.3. Implement Delta Lake Storage and Data Modeling Patterns. During the first 45–90 days, develop or enhance Delta Lake tables, data structures, and processing patterns to support reliable storage and consumption of enterprise datasets. Apply appropriate data modeling, schema management, incremental processing, and data organization techniques based on the requirements of assigned workloads.4. Tune Spark Jobs and Improve Processing Efficiency. During the first 60–120 days, analyze execution plans, job performance, data distribution, and compute utilization to identify bottlenecks in assigned Databricks workloads. Implement appropriate optimizations involving Spark transformations, SQL queries, partitioning, data layout, and cluster or job configurations.