Lead Databricks Architect

Lead Databricks ArchitectLocation: Indianapolis, IN — Hybrid, 3 days onsite/weekIndustry: PharmaceuticalContract: Long Term ContractInterview/Selection: Deep technical background review — only highly qualified candidatesRole OverviewWe are seeking a Lead Databricks Architect to lead the architecture, modernization, governance, and platform strategy for a large-scale Databricks environment running on AWS.This is a true enterprise architecture and platform leadership role, not a general Databricks Developer or Data Engineer position.The ideal candidate must have multiple enterprise-scale Databricks implementations/integrations and hands-on experience designing and migrating complex AWS data platforms into a Databricks Lakehouse architecture.The candidate will lead architecture across multiple Databricks workspaces, establish platform standards, define governance and security models, and provide hands-on technical leadership throughout migration and implementation.Key ResponsibilitiesLead architecture and modernization of an enterprise Databricks platform on AWS.Design and implement multi-workspace Databricks architectures.Establish standards for Unity Catalog, security, governance, RBAC/ABAC, data sharing, and access controls.Lead migrations from AWS Glue, Aurora PostgreSQL/PostgreSQL, and existing AWS data lakes to Databricks Lakehouse.Design scalable raw-to-refine/medallion data architectures using Databricks and Unity Catalog.Build metadata-driven and configuration-based ingestion pipelines.Develop and maintain Databricks Lakeflow Declarative Pipelines (formerly DLT).Implement CDC, materialized views, schema evolution, and incremental data processing.Integrate Databricks with Veeva Vault/Veeva Direct Data API (DDAPI) and related connectors.Develop advanced Databricks SQL, Delta Lake, Spark/PySpark solutions.Build reusable architectural patterns, accelerators, templates, and engineering standards.Implement Terraform-based Infrastructure as Code for Databricks workspaces, Unity Catalog, service principals, IAM, and permissions.Build/manage AWS infrastructure using CloudFormation, including S3, Glue, Lambda, Step Functions, EventBridge, and DynamoDB.Establish enterprise CI/CD and automated deployment processes, including GitHub Actions.Drive platform governance, compliance, security, privacy, auditability, and data-quality standards.Troubleshoot complex pipeline, schema evolution, integration, and platform issues.Partner with data engineering, security, infrastructure, architecture, regulatory, and business teams.Provide technical leadership and mentorship to engineering teams.Document architecture decisions, standards, deployment patterns, and operational procedures.Must-Have SkillsCandidates must have strong experience in the following:Multiple enterprise-scale Databricks implementations/integrationsDatabricks Architecture on AWSMulti-workspace Databricks environmentsAWS Glue → Databricks migrationsAWS Data Lake → Databricks Lakehouse modernizationAurora PostgreSQL/PostgreSQL → Databricks migrationUnity CatalogDelta LakeDatabricks WorkflowsDatabricks SQL / SQL WarehousesApache Spark / PySparkLakeflow Declarative Pipelines / DLTAWS S3, IAM, networking, and securityTerraform / Infrastructure as CodeCI/CD for enterprise data platformsPlatform governance, security, and architectural standardsEnterprise migration strategy and reference architectureIdeal Candidate ProfileThe strongest candidate will be someone who has already architected and delivered multiple enterprise Databricks implementations on AWS, rather than someone who has primarily worked as a Databricks developer.Top priority: Proven enterprise Databricks architecture + multiple implementations + AWS migration experience.Important: Only submit candidates who can clearly demonstrate these experiences on their resume. The hiring manager will conduct a deep technical review of their background.Submission DetailsPlease provide the following with each candidate submission:Full NameCurrent LocationPhone NumberEmail AddressAvailability / Notice PeriodExpected RateWork AuthorizationLinkedIn ProfileTotal Years of ExperienceYears of Databricks ExperienceYears of AWS ExperienceNumber of Enterprise Databricks Implementations LedDatabricks Architecture / Platform Leadership ExperienceAWS Glue → Databricks Migration ExperienceAurora PostgreSQL/PostgreSQL → Databricks Migration ExperienceAWS Data Lake → Databricks Lakehouse Migration ExperienceUnity Catalog / Multi-Workspace ExperienceTerraform / IaC / CI-CD ExperiencePharma / Healthcare / Regulated Industry ExperienceVeeva Vault / Veeva DDAPI Experience, if applicableIndianapolis, IN Hybrid Availability — 3 Days Onsite/Week Note: Please submit only candidates who have multiple enterprise-level Databricks implementations/integrations and can demonstrate hands-on Databricks architecture and AWS migration experience.Thanks & RegardsNajeeb Khan (Account Manager - US Staffing )Three, 400 E Royal, Ln #290, Irving, TX 75039.Email: najeeb@kaltechsoft.com