Role: IT Infrastructure Management - Data Warehouse Developer
Locations: Hopkins, MN; Hybrid

Duration: Contract

Interview Process - 1 Rounds; Mode of Interview: Video; Realtime Coding Session

Job Description:
Can you please provide a summary of the project/initiatives which describes what's being done?

• Build AC360 data pipelines in Azure using Spark technologies. As part of this initiative, develop Zafin data pipelines in Azure.

What are the top 5-10 responsibilities for this position? (Please be detailed as to what the candidate is expected to do or complete on a daily basis)

• Identify requirements for RMS replacement and develop solutions in Azure.
• Re-engineer solution to retrofit the rules with AC360 in Azure.
• Register data pipelines in Azure, onboard the dataset and enable data processing in Azure.
• Build Airflow dags and ensure scheduling is inbuilt with integration.
• Apply Dev/Ops mindset, take ownership of production success, optimize operation success via automation/active alerting/self-healing, and lead the resolution of production issues to ensure high availability and performance.
• Develop high-quality code, define best engineering practice, perform peer code reviews to ensure successful deliverable with engineering excellence
• Document engineering artifacts such as technical design document, flowcharts, system context diagrams, code comments etc.
• Collaborate cross-functionally with product owner, data scientists, business users, project managers and other engineers to achieve elegant solutions.
• Collaborate with other engineers on the team to elevate technology and consistently apply best
• Data Engineer will execute true end-to-end development and testing of software products in an agile lifecycle. Partnering with key technical and business stakeholders, ensure the quality of the final product through manual and automated tests, and support the new software as it is moved to production.

What skills/technologies are required (please include the number of years of experience required)?

• Good knowledge of Mainframe COBOL to understand the rules in Strata rules engine.
• Knowledge of Spark-Scala or Spark-SQL, Spark-Py - 5 years
• Experience in airflow, elastic, Azure 3+ years
• Experience in managing big data sets 10+ years
• Experience building micro-services based applications 5+ years
• Experience in Azure, Kubernetes, Postgres.
• Experience in Utilizing tools such as Maven, Docker, Kubernetes, Kibana, ELK, Jenkins and frameworks such as Spring Boot to develop and deploy microservices 4+ years
• Experience using CICD processes for application software integration and deployment using Maven, Git, Jenkins 4+ years
• Experience building scalable and resilient applications in private or public cloud environments and cloud technologies 4+ years
• Experience with server side languages such as Python, Java, Golang 3+ years
• Excellent verbal and written communication skills 10 years or higher
• Considerable technical, logical, analytical and problem-solving skills 10 years or higher

What skills/attributes are preferred (these are a desired, not required)?

• Spark, Scala, Python, Azure stack.

Responsibilities:

• Responsible for all phases of the software development lifecycle (analysis, design, and implementation).
• Provide application development and maintenance for all phases of the software development lifecycle, including analysis, design, and implementation.
• Design and maintain complex ETL (Extract, Transform and Load) processes that load from both data files and other databases.
• Develop and maintain complex PL/SQL functions and procedures.
• Create and optimize complex SQL queries for reports and applications.
• Provide input to project plans that includes estimates of tasks, duties and technologies for future releases and write technical documentation.
• Provide end user support and problem diagnosis and resolution.
• Perform all assignments in accordance with ESH&Q, Security, and business practice requirements and policies.
• Provide expert level development support of PL/SQL functions, procedures, and database architecture
• Provide advanced SQL tuning skills to complex queries, ETL's, and data processing.
• Provide advanced support towards the IT needs for financial and/or budget and planning applications.

Qualifications

• Broad experience developing and maintaining custom ETL scripts.
• Comprehensive knowledge and experience with relational database design, techniques, and development methodologies
• Broad experience using SQL, PL/SQL, and Oracle 11g or higher.
• Broad experience developing ETL processes within Oracle database environments and non-Oracle environments using SQL Server, MySQL, File Transfer, or similar technologies.
• Proficient interpersonal skills necessary to effectively collaborate as part of a cohesive development team working with multiple customers and organizations simultaneously with time constraints.
• Proficient written and verbal communication skills necessary to effectively prepare, present and explain technical information to others.
• Significant experience supporting financial and business applications.
• Advanced experience with SQL Tuning and ETL Optimization.
• Significant experience with data virtualization tools such as Denodo.