• Snapboard
  • Activity
  • Reports
  • Campaign
Welcome ,

Chat with the recruiter

...Minimize

Hey I'm Online! Leave me a message.
Let me know if you have any questions.

Data Engineer with Spark exp.

In California / United States

Save this job

Data Engineer with Spark exp.   

Click on the below icons to share this job to Linkedin, Twitter!

JOB TITLE:

Data Engineer with Spark exp.

JOB TYPE:

JOB SKILLS:

JOB LOCATION:

Santa Clara California / United States

JOB DESCRIPTION:

Job title: Data Engineer with Spark exp

Location: Remote

Duration: Long term

W2 Candidates

This position is not open for C2C

Our client is looking for someone to be based in the US, but are flexible on specific location. Candidate will be expected to work US East Coast hours with occasional flexibility needed to work with client team based in India.

Job Overview:

  • One of our large clients is expanding their current data footprint on the cloud to provide analytics, BI and data APIs. Majority of data will be batch processed with data validation, data quality and transformation into a multitude of data platforms such as Redshift, Postgres and Hive.
  • A Senior Technical Consultant is expected to be knowledgeable in two or more technologies within (a given Solutions/Practice area). The Senior Technical Consultant is expected to have strong development and programming skills in Spark with a focus on Scala/Java and other ETL development experience in the big data space. You are expected to be experienced and fluent in agile development and agile tools as well as code repositories and agile SLDC/DevOps frameworks.
  • You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high quality solutions focused on our clients' business needs.

Responsibilities:

  • Work with data engineering team to define and develop data ingestion, validation, transformation and data engineering code.
  • Develop open source platform components using Spark, Scala, Java, Oozie, Hive and other components
  • Document code artifacts and participate in developing user documentation and run books
  • Troubleshoot deployment to various environments and provide test support.
  • Participate in design sessions, demos and prototype sessions, testing and training workshops with business users and other IT associates

Qualifications:

  • Data Engineers are required to design and build data products and data pipelines. They will ensure the robust flow of data from acquisition through curation, and governance. Data Engineers will enable data as a service and drive all critical data driven initiatives.
  • Experts of the data and its application by users
  • Understands data landscape and environments: sources, elements, update freq, completeness, stewards/contacts, platforms
  • Manages ETL: use programming and tools for data ingestion, configure pipelines, apply transformations and decoding, integrate and fuse data, move and securely deliver
  • Translates business requirements to build repeatable, sustainable, efficient, coded processes that can be productionized by Software Engineers and readily modified by other Data Engineers
  • Creates POC processes in Dev/QA and works with Software Engineers to productionize and align best coding and process practices that maximize efficiency, speed, stability, system resources and capabilities
  • Leverages frameworks in place with big data tools: Hadoop, Spark, Python, Kafka, etc.
  • Experience with relational SQL and NoSQL databases
  • Awareness of and compliance with: data privacy, security, legal and contractual guidelines
  • Incorporates data quality and privacy checks/alerts to minimize bad data being consumed by end users, models and dashboards, and to protect customer data; revises checks for new data issues
  • Maintains feedback loop with Data Stewards on data issues, standards, fit for use (Data Stewardship is a subset of data engineering which would include responsibilities like data curation)
  • Validates data products and pipelines are functioning as expected following system or application upgrades, source changes, etc.
  • Responsible for data architecture including sources, table structures, physical models
  • Works closely with Architects to align systems, tools and applications being utilized with business use case and performance requirements
  • Communicates with end users to set expectations and ensure alignment around data accuracy, completeness, timeliness and consistency
  • Provides data product support and maintenance
  • Establishes, tracks and monitors KPIs related to specific data products and deliverables

Preferred Skills and Education:

  • Bachelors degree in Computer Science or related field
  • Certification in Spark, Azure or other cloud platform

Position Details

POSTED:

Sep 10, 2021

EMPLOYMENT:

INDUSTRY:

SNAPRECRUIT ID:

S1617465812705555

LOCATION:

California / United States

CITY:

Santa Clara

Job Origin:

CEIPAL_ORGANIC_FEED

A job sourcing event
In Dallas Fort Worth
Aug 19, 2017 9am-6pm
All job seekers welcome!

Data Engineer with Spark exp.    Apply

Click on the below icons to share this job to Linkedin, Twitter!

Job title: Data Engineer with Spark exp

Location: Remote

Duration: Long term

W2 Candidates

This position is not open for C2C

Our client is looking for someone to be based in the US, but are flexible on specific location. Candidate will be expected to work US East Coast hours with occasional flexibility needed to work with client team based in India.

Job Overview:

  • One of our large clients is expanding their current data footprint on the cloud to provide analytics, BI and data APIs. Majority of data will be batch processed with data validation, data quality and transformation into a multitude of data platforms such as Redshift, Postgres and Hive.
  • A Senior Technical Consultant is expected to be knowledgeable in two or more technologies within (a given Solutions/Practice area). The Senior Technical Consultant is expected to have strong development and programming skills in Spark with a focus on Scala/Java and other ETL development experience in the big data space. You are expected to be experienced and fluent in agile development and agile tools as well as code repositories and agile SLDC/DevOps frameworks.
  • You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high quality solutions focused on our clients' business needs.

Responsibilities:

  • Work with data engineering team to define and develop data ingestion, validation, transformation and data engineering code.
  • Develop open source platform components using Spark, Scala, Java, Oozie, Hive and other components
  • Document code artifacts and participate in developing user documentation and run books
  • Troubleshoot deployment to various environments and provide test support.
  • Participate in design sessions, demos and prototype sessions, testing and training workshops with business users and other IT associates

Qualifications:

  • Data Engineers are required to design and build data products and data pipelines. They will ensure the robust flow of data from acquisition through curation, and governance. Data Engineers will enable data as a service and drive all critical data driven initiatives.
  • Experts of the data and its application by users
  • Understands data landscape and environments: sources, elements, update freq, completeness, stewards/contacts, platforms
  • Manages ETL: use programming and tools for data ingestion, configure pipelines, apply transformations and decoding, integrate and fuse data, move and securely deliver
  • Translates business requirements to build repeatable, sustainable, efficient, coded processes that can be productionized by Software Engineers and readily modified by other Data Engineers
  • Creates POC processes in Dev/QA and works with Software Engineers to productionize and align best coding and process practices that maximize efficiency, speed, stability, system resources and capabilities
  • Leverages frameworks in place with big data tools: Hadoop, Spark, Python, Kafka, etc.
  • Experience with relational SQL and NoSQL databases
  • Awareness of and compliance with: data privacy, security, legal and contractual guidelines
  • Incorporates data quality and privacy checks/alerts to minimize bad data being consumed by end users, models and dashboards, and to protect customer data; revises checks for new data issues
  • Maintains feedback loop with Data Stewards on data issues, standards, fit for use (Data Stewardship is a subset of data engineering which would include responsibilities like data curation)
  • Validates data products and pipelines are functioning as expected following system or application upgrades, source changes, etc.
  • Responsible for data architecture including sources, table structures, physical models
  • Works closely with Architects to align systems, tools and applications being utilized with business use case and performance requirements
  • Communicates with end users to set expectations and ensure alignment around data accuracy, completeness, timeliness and consistency
  • Provides data product support and maintenance
  • Establishes, tracks and monitors KPIs related to specific data products and deliverables

Preferred Skills and Education:

  • Bachelors degree in Computer Science or related field
  • Certification in Spark, Azure or other cloud platform


Please wait..!!