Senior Big Data Engineer

Design and develop scalable data pipelines and solutions using Python and PySpark. Utilize big data technologies such as Hadoop, Spark, Kafka, or similar tools for processing and analyzing large datasets. Develop and maintain ETL processes to extract, transform, and load data into data lakes or warehouses. Collaborate with data engineers and scientists to implement machine learning models and algorithms. Optimize and tune data processing workflows for performance and efficiency. Implement data governance and security measures to ensure data integrity and privacy. Create and maintain documentation for data pipelines, workflows, and processes. Provide technical leadership and mentorship to junior team members. Skills Required Python: Proficiency in Python programming for data manipulation and analysis. PySpark: Experience with PySpark for processing large-scale data. Big Data Technologies: Strong understanding and practical experience with big data technologies such as Hadoop, Spark, Kafka, etc. ETL Processes: Knowledge of designing and implementing ETL processes for data integration. Data Processing: Ability to work with large datasets, perform data cleansing, transformations, and aggregations. Machine Learning: Familiarity with machine learning concepts and experience implementing ML models. Data Governance: Understanding of data governance principles and experience implementing data security measures. Documentation: Ability to create clear and concise documentation for data pipelines and processes. Collaboration: Strong teamwork and collaboration skills to work with cross-functional teams. Problem-Solving: Analytical and problem-solving skills to optimize data workflows and processes. Leadership: For senior roles, the ability to provide technical leadership, mentorship, and guidance to junior team members. SQL: Knowledge of SQL for querying and manipulating data in databases. Cloud Platforms: Experience with cloud platforms like AWS, Azure, or Google Cloud is a plus. #J-18808-Ljbffr