Databricks Engineer

Responsibilities

• Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks
• Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads
• Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments
• Create agents for metadata management, lineage tracking, and governance automation within Databricks
• Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic
• Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines
• Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases
• Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap
• Establish best practices for agent development, testing, deployment, and observability within Azure Databricks
• Document agent workflows, decision logic, and operational runbooks for team knowledge sharing
• Evaluate emerging AI/agent frameworks and technologies; recommend adoption where applicable

Education Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred)

Experience 10+ years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure) Proven experience developing automation solutions, agents, or AI-powered tools (e.g., using LLMs, prompt engineering, or orchestration frameworks like Databricks Agent Framework (or LangChain, AutoGen) Strong proficiency in Python and SQL with focus on production-grade code quality Demonstrated experience with Azure Databricks (notebooks, jobs, workflows, DLT) or equivalent modern data platforms Hands-on experience with CI/CD/CT tools (GitHub Actions, Azure DevOps Pipelines) and automation frameworks Experience building data quality frameworks, testing patterns, or QA automation Prior work in regulated or risk-sensitive environments preferred

• Expert-level Python programming with strong software engineering practices (testing, documentation, version control)
• Strong SQL and PySpark optimization skills
• Hands-on experience with LLM APIs, prompt engineering, and AI agent frameworks (e.g., Databricks Agent Bricks, LangChain, OpenAI, Azure OpenAI)
• Proficiency in Azure Databricks, including Delta Lake, workflows, and Databricks Asset Bundles
• Experience designing and implementing automation workflows using orchestration tools or serverless platforms
• Strong problem-solving and systems thinking skills; ability to decompose complex workflows into automatable components
• Excellent communication and documentation skills; ability to articulate technical concepts and agent design rationale
• Self-directed learner with curiosity about emerging AI/automation technologies
• Ability to collaborate effectively with cross-functional teams (data engineers, architects, QA, operations)

For applications and inquiries,