Loading, Please wait..!!
Senior Databricks Engineer
-
Posted on: Oct 23, 2026
-
Atlanta, Georgia
-
Onsite
-
CTC
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for Senior Databricks Engineer roles in Georgia.
View Salary Guide →Find your match !
Job Title :
Senior Databricks Engineer
Job Type :
Contract
Job Location :
Atlanta Georgia United States
Remote :
No
Job Description :
Title: Senior Databricks Engineer
Location: Atlanta, GA/Raleigh, NC (Hybrid)
Type: Contract
Day-to-Day Job Duties:
- Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
- Build document ingestion, classification, extraction, validation, and transformation pipelines for PDFs, emails, attachments, Excel/XLSX files, insurance applications, Statements of Values (SOVs), and other business documents.
- Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured documents.
- Troubleshoot false positives, incorrect metadata, and inaccurate field values generated by LLM-based extraction pipelines.
- Implement document classification, source-specific extraction rules, data validation, reconciliation, and quality-control mechanisms.
- Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
- Design and implement ontology-based data models representing insurance entities, business concepts, and relationships.
- Map extracted document attributes to standardized ontology concepts and metadata structures.
- Optimize the processing of large and complex Excel/XLSX and SOV files, addressing performance, scalability, and concurrency challenges.
- Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
- Develop and maintain APIs and data products to support downstream application integration.
- Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
- Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.
Basic Qualifications:
- Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
- Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines.
- Minimum 5+ years of experience with Python, PySpark, and SQL.
- Minimum 3+ years of experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
- Minimum 2+ years of experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
- Minimum 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
- Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
- Minimum 2+ years of experience designing JSON outputs, APIs, and data products for downstream application consumption.
- Minimum 2+ years of experience with PostgreSQL, Lakebase, or similar relational database technologies.
- Minimum 2+ years of experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
- Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
- Strong analytical, problem-solving, communication, and cross-functional collaboration skills.
Nice to Have:
- Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
- Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
- Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
- Experience building custom ontologies, knowledge graphs, or semantic data layers.
- Experience with document AI, OCR, information extraction, and document intelligence platforms.
- Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Experience optimizing large-scale document ingestion and distributed processing workloads
Position Details
Posted:
Oct 23, 2026
Reference Number:
101958-5195
Employment:
CTC
City:
Atlanta
Job Origin:
CIEPAL_ORGANIC_FEED
Share this job:
Please wait..!!

