Key Responsibilities

• Design, build, and maintain ETL/ELT data pipelines
• Develop Python-based data processing applications
• Work with structured and unstructured data at scale
• Integrate data from multiple sources (APIs, databases, files, streams)
• Optimize data workflows for performance and reliability
• Ensure data quality, validation, and monitoring
• Collaborate with data scientists, analysts, and backend teams
• Manage and maintain data warehouses/lakes
• Implement logging, error handling, and automation
• Follow best practices for security and compliance

Required Skills

Programming

• Strong Python (Pandas, NumPy, PySpark)
• Writing clean, modular, and testable code

Databases & Storage

• SQL (PostgreSQL, MySQL, SQL Server)
• NoSQL (MongoDB, Cassandra optional)
• Data Warehouses (Snowflake, Redshift, BigQuery)

Big Data & Processing

• Apache Spark, Hadoop (preferred)
• Batch and streaming data processing

Cloud Platforms

• AWS / Azure / GCP

S3, Lambda, Glue, Dataflow, BigQuery, etc.

Data Engineering Tools

• Airflow, Prefect, Luigi (orchestration)
• Kafka / PubSub (streaming optional)
• DBT (data transformation)

DevOps & Other

• Git, CI/CD
• Docker, Kubernetes (nice to have)
• Linux basics