Databricks Engineer

This position requires a mandatory in person interview and in person onboarding.

Senior Data Engineer Contract Length: Until 12/31/26 with possible extension

Location: New York, NY

Work Setup: Hybrid

We are seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data platforms that power enterprise analytics, machine learning, and AI-driven solutions. This role will build reliable, high-performance data ecosystems using Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS while supporting modern Generative AI initiatives.

• Design, build, and maintain scalable batch and real-time data pipelines for structured, semi-structured, and unstructured data.
• Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion (Bronze, Silver, Gold) design patterns.
• Develop distributed processing and streaming solutions using PySpark, Spark SQL, Kafka, Spark Structured Streaming, and AWS Kinesis.
• Architect AWS-based solutions using S3, EMR, EC2, Athena, Redshift, RDS, Lambda, IAM, SNS, and SQS.
• Optimize performance through partitioning, clustering, caching, query tuning, and cost-efficient storage and processing strategies.
• Ensure data quality, governance, lineage, security, and compliance across data ecosystems.
• Develop orchestration and scheduling frameworks using Airflow and Databricks Workflows and build CI/CD automation.
• Support migrations from traditional Hadoop and EMR environments to modern cloud-native platforms.
• Implement Generative AI solutions leveraging LLMs, RAG, Vector Databases, AI Agents, and Model Context Protocol (MCP).
• Collaborate with architects, analysts, data scientists, and business stakeholders to deliver enterprise-grade solutions.

• Bachelor's or Master's degree in Computer Engineering, Computer Science, Information Systems, or a related field.
• 8+ years of experience in software engineering, data engineering, or big data platform development.
• Strong experience designing and implementing enterprise-scale data pipelines.
• Hands-on expertise with: Python, PySpark, Spark SQL, SQL, Kafka, Databricks, Delta Lake, Snowflake, Hive
• Experience building data solutions on AWS cloud platforms.
• Strong understanding of distributed computing, data modeling, and large-scale data processing.
• Experience with Git-based development workflows and CI/CD practices.
• Excellent analytical, troubleshooting, and problem-solving skills.

• Experience with real-time streaming architectures and event-driven systems.
• Knowledge of data governance, metadata management, and data quality frameworks.
• Experience with generative AI technologies including: LLMs, RAG, Vector Databases, AI Agents, MCP integrations
• Experience developing developer productivity tools and AI-assisted engineering workflows.
• Exposure to enterprise supply chain, retail, healthcare, or manufacturing data domains.
• AWS certifications are highly preferred.

Python, Java, SQL, Shell Scripting, C/C++, PySpark, Spark SQL, Hive, Databricks, Delta Lake, Snowflake, Kafka, HBase, Sqoop, Apache Airflow, Databricks Workflows, Oozie, AWS, Tableau, Git, Docker, Splunk, IntelliJ IDEA, PyCharm, Cursor, Generative AI, LLMs, RAG, Agentic AI Systems, MCP, and Vector Databases.

AWS Certified Solutions Architect Associate, AWS Certified Cloud Practitioner, and Databricks Certifications (preferred).

What Success Looks Like

The Senior Data Engineer will deliver highly scalable and reliable data pipelines, improve platform performance and cost efficiency, enable enterprise analytics and AI initiatives through trusted data products, modernize data platforms, and leverage AI technologies to enhance engineering productivity, automation, and innovation.

We look forward to reviewing your application!

For applications and inquiries,