Lead Data Engineer

Position: Senior Data Engineer

Experience: 10+ Years

Employment Type: Contract W2

We are looking for an experienced Senior Data Engineer with 10+ years of experience in designing, developing, and maintaining scalable data pipelines and enterprise data platforms. The ideal candidate should have strong expertise in Python, SQL, PySpark, ETL/ELT, cloud platforms, data warehousing, and modern data engineering technologies.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines for large and complex datasets.
• Develop data processing solutions using Python, PySpark, and SQL.
• Build and optimize data pipelines using tools such as AWS Glue, Informatica, Azure Data Factory, Databricks, or equivalent technologies.
• Design and implement data warehouses, data lakes, and lakehouse solutions.
• Work with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Develop data ingestion and transformation processes from structured and unstructured data sources.
• Perform data profiling, validation, cleansing, and quality checks.
• Optimize SQL queries, Spark jobs, and data pipelines for performance and scalability.
• Work with databases including Oracle, PostgreSQL, SQL Server, Teradata, Snowflake, BigQuery, or Redshift.
• Implement CI/CD processes and version control using Git, Jenkins, GitHub Actions, or Azure DevOps.
• Collaborate with Data Architects, Business Analysts, DevOps Engineers, and application teams.
• Monitor production pipelines, troubleshoot failures, and resolve data-related issues.
• Ensure data security, governance, lineage, and compliance standards are followed.
• Participate in architecture discussions, code reviews, technical documentation, and mentoring of junior engineers.

Required Skills

• 10+ years of experience in Data Engineering or related fields.
• Strong hands-on experience with Python, PySpark, and SQL.
• Strong understanding of ETL/ELT concepts and data pipeline development.
• Experience with AWS, Azure, or Google Cloud Platform cloud environments.
• Strong experience with Databricks / Spark and distributed data processing.
• Experience with Snowflake, BigQuery, Redshift, Teradata, Oracle, or PostgreSQL.
• Experience with data lakes, data warehouses, and lakehouse architecture.
• Knowledge of Kafka or other streaming technologies is preferred.
• Experience with Git and CI/CD tools.
• Strong troubleshooting, analytical, and problem-solving skills.
• Excellent communication and collaboration skills.

Preferred Skills

• Experience with Informatica PowerCenter/IICS/IDMC, Talend, AWS Glue, ADF, or similar ETL tools.
• Experience with Airflow, Control-M, or other workflow orchestration tools.
• Knowledge of Terraform and cloud infrastructure.
• Experience with Delta Lake / Delta Live Tables.
• Knowledge of data governance, metadata management, and data quality frameworks.
• Experience working in Agile/Scrum environments.

For applications and inquiries,