Loading, Please wait..!!
Senior Infrastructure Engineer – Terraform & Databricks
-
Posted on: Nov 17, 2026
-
San Francisco, California
-
Onsite
-
Full-time
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for Senior Infrastructure Engineer – Terraform & Databricks roles in California.
View Salary Guide →Find your match !
Job Title :
Senior Infrastructure Engineer – Terraform & Databricks
Job Type :
Full-Time
Job Location :
San Francisco California United States
Remote :
No
Job Description :
Senior Infrastructure Engineer Terraform & Databricks
San Francisco, CA
Need candidates who can work onsite from Day 1 (3 days a week : Tuesday, Wednesday and Thursday 3 days work from office) NO REMOTE
Highly preferred local candidates
Experience: 09+ years
About the role
Building an enterprise Databricks Lakehouse on AWS, and all of it is managed as code. In this role you'll design, build and run the Terraform/Terragrunt foundation behind our Databricks workspaces, Unity Catalog, compute and security controls.
You'll work closely with Cloud Infrastructure/SRE, Security Engineering and Data Engineering. Your goal is to take our platform from sandbox to production-ready multi-environment setup that is secure, well governed and cost-efficient.
What you will do
- Design, build and maintain reusable Terraform modules for Databricks account-level (MWS) and workspace-level resources. This covers workspaces, Unity Catalog (metastores, catalogs, schemas, volumes, grants), cluster policies, instance pools, SQL warehouses, jobs, DLT pipelines and service principals.
- Use Terragrunt to manage environments consistently across sandbox, nonprod, preprod and prod, with minimal configuration drift.
- Provision and secure the supporting AWS infrastructure: customer-managed VPCs, PrivateLink/VPC endpoints, security groups, NACLs, IAM roles and policies, S3, KMS customer-managed keys (CMK) and Secrets Manager.
- Lead security hardening. That means fixing findings from security architecture reviews and cloud security posture management (CSPM) tools such as Wiz, enforcing PrivateLink-only access, IP access lists, CMK encryption and secret/OAuth token rotation.
- Set up identify and access controls, including Okta SSO and SCIM sync, role-based (RBAC) and tag-based (ABAC) access in Unity Catalog, and least-privilege service principals.
- Build and own CI/CD for infrastructure. This includes GitHub Actions plan/apply workflows with approvals, pre-commit hooks, Terratest, and security scanning with Checkov or tfsec.
- Set up monitoring and alerting with CloudWatch, Grafana and Splunk log forwarding. That covers cluster and job health, token expiry and cost anomalies.
- Drive FinOps practices: cluster policy guardrails, spot usage, moving work-loads from all-purpose to job clusters, enforcing tagging, and analysing DBU usage.
- Enable integrations with Dagster, dbt Cloud and BI tools such as Tableau, Power BI and Looker via SQL warehouses.
- Write and maintain architecture docs, ADRs, runbooks and module references, and enforce naming and tagging standards.
- Databricks Hands-on build data pipelines using Pyspark
- Used Agentic IDE to build data pipelines
- Understands Medallion architecture
- Understands ML flow Model inferencing
- Have domain expertise or understanding of Credit decisioning or Risk or Servicing (collections)
Skills Required
- 7+ years in infrastructure, platform or DevOps engineering with atleast 3 years of hands-on Terraform in production. Terragrunt experience is strongly preferred.
- 2+ years running Databricks on AWS, including workspace provisioning, Unity Catalog, cluster policies and the Databricks Terraform provider.
- Deep AWS knowledge; VPC networking, PrivateLink, IAM, KMS, S3, Secrets Manager and CloudWatch.
- Experience with CI/CD for Infrastructure as code (GitHub Actions, Jenkins or similar) and with GitOps workflows.
- Strong grasp of cloud security and compliance: encryption, network isolation, least privilege and audit logging. Experience in regulated environments such as financial services is a plus.
- Scripting skills in Python and/or Bash. Go is a plus for Terratest.
- Comfortable with Git, Code reviews and writing clear technical documentation.
Nice to have
- Databricks certifications (Data Engineer Associate/Professional or Platform Administrator), or HashiCorp Terraform Associate.
- Experience with Dela Live Tables, Vector Search, Model Serving or MLops Infrastructure.
- Familiarilty with Dagster, Dbt, Spark and Delta Lake.
- Experience with Wiz, Checkov, tfsec or similar policy-as-code and CSPM tools.
- Background in FinOps or cloud cost optimization.
- Experience with SOC 2 or similar compliance frameworks
Position Details
Posted:
Nov 17, 2026
Reference Number:
34424-5039
Employment:
Full-time
City:
San Francisco
Job Origin:
CIEPAL_ORGANIC_FEED
Share this job:
Please wait..!!

