Data Engineering Lead

Location: Irving, TX (Onsite – Customer Office Premises)
 

We are seeking an experienced Data Engineering Lead with strong expertise in Python, AWS data services, scalable data pipelines, and microservices architecture. The ideal candidate will lead the design and development of cloud-native data ingestion and streaming solutions, ensuring high performance, data integrity, and reliability across enterprise data platforms.

• Collaborate with Principal Software Engineers and Data Architects to define solution architecture.

• Lead the design and optimization of scalable data pipelines and Python-based microservices for real-time and batch processing.

• Develop cloud-native data ingestion and streaming solutions using AWS services such as Kinesis, S3, DynamoDB, and EventBridge.

• Own data integration frameworks and source data pipelines supporting CI Autonomy initiatives.

• Translate business and technical requirements into scalable data architectures, workflows, mappings, and system designs.

• Establish automated testing, data quality controls, and validation frameworks.

• Monitor production data platforms, perform performance tuning, and conduct root-cause analysis using tools such as AWS CloudWatch.

• Ensure high availability, service reliability, data integrity, and compliance across distributed data ecosystems.

• Lead teams in delivering high-throughput and scalable data engineering solutions.

• Bachelor's degree in Computer Science, Computer Engineering, or a related field.

• 8+ years of experience in data engineering or related disciplines, with increasing responsibility.

• Extensive experience with large-scale enterprise data platforms, preferably Caterpillar data platforms such as Helios Data Platform.

• Strong hands-on experience developing and deploying Python solutions in production environments.

• Experience leading teams in building high-throughput, scalable data pipelines.

• Strong hands-on expertise with AWS services, including:

  • Amazon Kinesis

  • Amazon S3

  • Amazon DynamoDB

  • Amazon EventBridge

• Strong SQL skills, including data quality and validation practices.

• Experience developing microservices for real-time data ingestion.

• Experience with relational and NoSQL databases.

• Experience with CI/CD and deployment tools such as Azure DevOps, Jira, and Jenkins.

• Knowledge of monitoring, testing, automation, and data integrity across distributed and streaming systems.