Data Engineer

Responsibilities and duties:

Data Engineering

• Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
• Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
• Optimize Databricks jobs, compute, and data layouts for performance and cost.
• Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
• Implement data quality checks, monitoring, error handling, and recovery mechanisms.

Cloud & Platform Engineering

• Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry and Azure Monitor.
• Experience with cloud AI services including vector search and embedding pipelines.
• Improve platform performance, scalability, reliability, security, and cost efficiency.
• Implement secure identity, and access management using Azure managed identities, Key Vault, secrets managed via scopes.
• Support development, staging, and production environments.

Automation & Operations

• Improve platform observability through logging, monitoring, dashboards, and alerting.
• Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
• Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
• Support production incidents and root-cause analysis.
• Collaborate with application, AI/ML, and data engineering teams.

Requirements:

• 3 5 years of experience in data engineering, cloud engineering, or a related role.
• Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
• Strong skills in Python/PySpark, SQL.
• Experience building and operating production data ETL/ELT pipelines.
• Experience with relational and semi-structured/unstructured data, database technologies.
• Experience with Spark, Delta Lake, and lakehouse architectures.
• Working knowledge of Databricks Unity Catalog and data-access controls.
• Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
• Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.

Nice to Have:

• Experience with Terraform or ARM templates.
• Experience with Azure DevOps or GitHub Actions.
• Familiarity with Azure networking, including VNets and private endpoints.
• Experience with Azure Data Factory, Databricks Workflows, or Airflow.
• Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.

For a further discussion or to find out more, contact Indu Sri Lakavath on (909) 271-0953 or apply now.