Databricks Engineer

Data Engineer/Architect with AI Agent Automation Engineer (Databricks) Azure Databricks Environment******* Austin, TX Need only Locals or very nearby places within 50Miles Hybrid Role (3d onsite in a week) Duration: 12+ Months Must have skills: Databricks( 5-10 Years), Azure cloud, LLM, AI Responsibilities Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments Create agents for metadata management, lineage tracking, and governance automation within Databricks Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap Establish best practices for agent development, testing, deployment, and observability within Azure Databricks Document agent workflows, decision logic, and operational runbooks for team knowledge sharing Evaluate emerging AI/agent frameworks and technologies; recommend adoption where applicable Job Requirements Education Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred) Experience 10+ years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure) Proven experience developing automation solutions, agents, or AI-powered tools (e.g., using LLMs, prompt engineering, or orchestration frameworks like Databricks Agent Framework (or LangChain, AutoGen) Strong proficiency in Python and SQL with focus on production-grade code quality Demonstrated experience with Azure Databricks (notebooks, jobs, workflows, DLT) or equivalent modern data platforms Hands-on experience with CI/CD/CT tools (GitHub Actions, Azure DevOps Pipelines) and automation frameworks Experience building data quality frameworks, testing patterns, or QA automation Prior work in regulated or risk-sensitive environments preferred Skills & Competencies Expert-level Python programming with strong software engineering practices (testing, documentation, version control) Strong SQL and PySpark optimization skills Hands-on experience with LLM APIs, prompt engineering, and AI agent frameworks (e.g., Databricks Agent Bricks, LangChain, OpenAI, Azure OpenAI) Proficiency in Azure Databricks, including Delta Lake, workflows, and Databricks Asset Bundles Experience designing and implementing automation workflows using orchestration tools or serverless platforms Strong problem-solving and systems thinking skills; ability to decompose complex workflows into automatable components Excellent communication and documentation skills; ability to articulate technical concepts and agent design rationale Self-directed learner with curiosity about emerging AI/automation technologies Ability to collaborate effectively with cross-functional teams (data engineers, architects, QA, operations)

For applications and inquiries,