Databricks Engineer

Job Title Databricks Engineer

Location Remote / Hybrid (as applicable)

Job Summary

STAFFXPERT LLC is seeking a Databricks Engineer on behalf of our client in Remote/Hybrid location. This role requires a highly skilled data engineering professional with proven experience designing, developing, and optimizing data solutions on the Databricks platform. The ideal candidate will bring strong expertise in PySpark, SQL, Delta Lake, data quality frameworks, and workflow modernization initiatives, with a track record of delivering enterprise-scale Databricks solutions in production environments.

Key Responsibilities

• Design, develop, and optimize scalable Delta Lake data pipelines using PySpark and SQL.

• Implement data transformation processes including data cleansing, deduplication, validation, and MERGE/upsert operations.

• Build and maintain data quality frameworks, monitoring solutions, and reporting mechanisms.

• Develop Databricks SQL dashboards and AI/BI reporting solutions to track data quality metrics and remediation activities.

• Migrate existing workflows from legacy ETL tools to Databricks using PySpark, SQL, and Lakeflow Jobs.

• Collaborate with cross-functional teams, stakeholders, and developers to deliver high-quality data solutions.

• Perform performance tuning and troubleshooting to improve reliability, scalability, and efficiency of data pipelines.

• Provide technical guidance, best practices, and knowledge transfer to internal and client teams.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent experience.

• Strong hands-on experience with Databricks, PySpark, SQL, and Delta Lake.

• Proven experience building, optimizing, and maintaining large-scale data processing pipelines.

• Expertise in implementing data quality, validation, monitoring, and governance frameworks.

• Experience creating data visualization, reporting, and dashboard solutions using Databricks SQL.

• Strong understanding of modern data lake and data warehouse architectures.

• Excellent analytical, problem-solving, and communication skills.

• Demonstrated experience delivering Databricks-based solutions in production environments.

Preferred Qualifications

• Experience migrating workflows from Alteryx or similar ETL platforms to Databricks.

• Familiarity with Lakehouse architecture and cloud platforms such as AWS, Azure, or GCP.

• Experience mentoring technical teams and leading knowledge-transfer sessions.

• Databricks certifications are a plus.