Databricks Platform Lead / Technical Delivery Lead

Job Title: Databricks Platform Engineering Lead / Technical Delivery LeadWork Location: Warren, NJ (Hybrid)Employment Type: Fulltime/PermanentNeed Independent Candidates Only | Local or NearbyMinimum Years of Experience: 10+ YearsNeed Someone who has experience in Python/PySpark and AWS Glue, EMR, S3Develop and optimize analytical datasets and workloads involving Redshift and AthenaProject OverviewWe are seeking an experienced Databricks Platform Engineering Led to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.Technical and Delivery Expertise10+ years of overall technology experience.5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.Strong hands-on experience with:Azure DatabricksDatabricks account and workspace administrationUnity CatalogApache Spark and PySparkPython and SQLDelta Lake and Lakehouse architectureAzure Data Lake StorageAzure Data Factory or comparable integration toolsDatabricks Workflows and job orchestrationInfrastructure as code using TerraformCI/CD and automated deployment practicesStrong understanding of:Databricks workspace provisioning and administrationDevelopment, test, and production environment managementCompute provisioning and cluster configurationCluster policies, pools, runtimes, and serverless computePlatform security, identity, and access managementService principals, managed identities, and secrets managementAzure networking, private endpoints, and secure connectivityUnity Catalog architecture, permissions, lineage, and governancePlatform monitoring, logging, auditability, and Observability.Databricks performance, capacity, and cost optimizationProduction support and platform-incident managementGood understanding of data-engineering practices, including:Batch and streaming data ingestionETL/ELT pipeline developmentMedallion architectureData transformations and enrichmentData quality, validation, and reconciliationSchema management and data lineageSpark workload and query optimizationExperience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.Preferred QualificationsPrior experience implementing a Databricks-enabled pricing, underwriting, actuarial, insurance analytics, or financial-services application.Experience working with global insurance or reinsurance carriers.Understanding of reinsurance pricing, underwriting, treaty, and facultative business processes.Experience integrating Databricks with:Pricing enginesSQL Server applicationsPolicy, claims, and accounting systemsData warehouses and reporting platformsInternal and third-party data sourcesExperience establishing reusable Databricks platform standards, engineering frameworks, and reference architectures.Experience implementing a hybrid solution of Databricks and SQLExperience defining the division of responsibility among platform engineering, data engineering, application development, DevOps, data science, and production-support teams.Familiarity with MLflow, model lifecycle management, model serving, or actuarial-model integration.Experience implementing data governance, metadata management, lineage, role-based access, audit controls, and regulatory controls.Familiarity with Azure DevOps, Terraform, Power BI, Microsoft Purview, Azure Monitor, or comparable tools.Databricks and Microsoft Azure certifications are preferred.Experience supporting business-critical production platforms with defined availability, recovery, security, and performance requirements.The Resource Will Be Expected To:Own the engineering and operational management of the Databricks platform supporting the enterprise pricing application.Lead end-to-end technical delivery of platform enhancements, environment changes, integrations, releases, and production improvements.Provision and administer Databricks accounts, workspaces, compute resources, clusters, policies, and supporting Azure services.Define and enforce platform standards covering architecture, security, governance, compute, deployment, monitoring, and cost management.Design and maintain appropriate separation across development, test, UAT, and production environments.Implement infrastructure as code and automated deployment practices for Databricks and supporting cloud resources.Establish secure connectivity between Databricks, SQL Server, cloud storage, the pricing engine, the underwriting workbench, and other enterprise systems.Configure and maintain identity, access, security, secrets, networking, and governance controls.Define Unity Catalog structures, access models, data ownership, lineage, and auditability standards.Establish cluster policies, runtime standards, compute controls, and workload-management practices.Monitor and optimize platform availability, reliability, performance, capacity, and cost.Troubleshoot workspace, authentication, networking, compute, storage, access, and platform-performance issues.Lead root-cause analysis and corrective actions for platform incidents and recurring operational problems.Provide technical oversight to data engineers building ingestion pipelines, transformations, curated datasets, and pricing data products.Review pipeline designs, Spark workloads, data-processing patterns, deployment approaches, and production-readiness plans.Ensure that data-engineering solutions comply with platform, security, governance, quality, and performance standards.Coordinate platform engineering, data engineering, application development, QA, DevOps, infrastructure, security, and production-support teams.Independently identify technical and delivery impacts across Databricks, the SQL Server pricing engine, the underwriting workbench, and integrated systems.Support successful SIT, UAT, regression testing, release validation, deployment, cutover, and post-production stabilization.Communicate technical risks, delivery dependencies, architectural decisions, and recovery plans to business and technology stakeholders.Serve as the primary technical point of contact for the Databricks platform and its integration with the enterprise pricing application.Serve as a subject matter expert on Databricks platform engineering, administration, governance, security, reliability, and operational best practices.