Job Title: Data Engineer
Interview: Virtual
Job Duration: 6+ Months
Job Location: Hopkins, MN

Job Description:
We are seeking a Data Engineer with expertise in building scalable data pipelines and managing big data solutions in Azure. This role involves collaborating with cross-functional teams to design, develop, and optimize data workflows while ensuring alignment with industry standards and best practices.

Key Responsibilities:

• Design and develop data pipelines in Azure using Spark technologies to process and manage large datasets.
• Build and maintain Zafin and AC360 data pipelines to ensure seamless integration and high-performance data processing.
• Develop Airflow DAGs for automated scheduling and orchestration of workflows.
• Collaborate with product owners, business stakeholders, and engineering teams to gather and define technical requirements.
• Apply a DevOps mindset to production systems, incorporating automation, active alerting, and self-healing techniques.
• Optimize big data solutions to ensure scalability, reliability, and efficiency.
• Perform peer code reviews and define best engineering practices to maintain high-quality deliverables.
• Document technical designs, workflows, and engineering artifacts to support team knowledge sharing and maintenance.
• Partner with key stakeholders to ensure end-to-end testing and smooth deployment of software products.

Qualifications:

• Big Data Expertise: 10+ years of experience managing large-scale datasets and distributed systems.
• Spark (Scala, SQL, Py): 5+ years of hands-on experience.
• Cloud Technologies: 4+ years of experience with Azure, Kubernetes, and PostGres.
• DevOps and CI/CD: Proficiency in Maven, Git, Jenkins, Docker, and Kubernetes.
• Programming Skills: Strong knowledge of Python, Java, and GoLang (3+ years).
• Airflow and Elastic Stack: 3+ years of experience with tools like Kibana and ELK.
• Soft Skills: Excellent verbal and written communication skills (10+ years).
• Analytical and Problem-Solving Skills: Strong technical and logical capabilities (10+ years).

Preferred Skills:

• Experience with Spark, Scala, Python, and the Azure technology stack.
• Familiarity with microservices-based applications and best practices for data engineering.