Databricks Engineer

Mandatory skills:

• Data Engineering Design and develop scalable ETL/ELT pipelines using Databricks, Spark, and SQL.
• Build data ingestion frameworks for structured and unstructured documents.
• Develop data quality, reconciliation, and validation frameworks.
• Design medallion architecture (Bronze, Silver, Gold) for document processing workflows.
• Implement metadata management and lineage tracking.

• Build Intelligent Document Processing (IDP) solutions using LLMs.
• Develop AI agents to perform: Contract clause extraction, Invoice validation, Payment term identification, Business rule evaluation, Exception handling workflows.
• Implement RAG (Retrieval Augmented Generation) solutions.
• Design multi-agent workflows using Microsoft Agentic Framework, LangChain, LangGraph, CrewAI, Semantic Kernel or similar.
• Develop prompt engineering and evaluation frameworks.
• Build human-in-the-loop review workflows.

• Integrate OCR platforms such as Azure AI Document Intelligence, Azure OCR, AWS Textract, Google Document AI, ABBYY Extract for vendor information, invoice details, PO details, payment schedules, contract clauses, pricing schedules.
• Improve extraction accuracy through AI-assisted validation.

• Work closely with business SMEs and invoicing teams.
• Conduct requirement workshops.
• Present solution architecture and implementation progress to stakeholders.

• Databricks
• Apache Spark (PySpark)
• SQL
• Python
• Azure Data Engineering
• Data Modeling
• Gen AI / LLM Integration
• Agentic AI Frameworks
• RAG Architecture
• OCR / Document Intelligence
• REST API Development
• AI Technologies
• Azure OpenAI GPT Models
• Semantic Kernel
• LangChain
• LangGraph
• CrewAI
• Vector Databases
• Cloud Platforms Azure, AWS, Google Cloud Platform

• Contract lifecycle management domain knowledge
• Accounts Payable / Invoicing experience
• Knowledge Graph integration experience
• Healthcare or Financial Services domain experience

For applications and inquiries,