Please Accept our Privacy Policy
Location: Irving, TX (Onsite – Customer Office Premises)
\n
We are seeking an experienced Data Engineering Lead with strong expertise in Python, AWS data services, scalable data pipelines, and microservices architecture. The ideal candidate will lead the design and development of cloud-native data ingestion and streaming solutions, ensuring high performance, data integrity, and reliability across enterprise data platforms.
\n\nKey Responsibilities\n\nCollaborate with Principal Software Engineers and Data Architects to define solution architecture.
\nLead the design and optimization of scalable data pipelines and Python-based microservices for real-time and batch processing.
\nDevelop cloud-native data ingestion and streaming solutions using AWS services such as Kinesis, S3, DynamoDB, and EventBridge.
\nOwn data integration frameworks and source data pipelines supporting CI Autonomy initiatives.
\nTranslate business and technical requirements into scalable data architectures, workflows, mappings, and system designs.
\nEstablish automated testing, data quality controls, and validation frameworks.
\nMonitor production data platforms, perform performance tuning, and conduct root-cause analysis using tools such as AWS CloudWatch.
\nEnsure high availability, service reliability, data integrity, and compliance across distributed data ecosystems.
\nLead teams in delivering high-throughput and scalable data engineering solutions.
\nBachelor's degree in Computer Science, Computer Engineering, or a related field.
\n8+ years of experience in data engineering or related disciplines, with increasing responsibility.
\nExtensive experience with large-scale enterprise data platforms, preferably Caterpillar data platforms such as Helios Data Platform.
\nStrong hands-on experience developing and deploying Python solutions in production environments.
\nExperience leading teams in building high-throughput, scalable data pipelines.
\nStrong hands-on expertise with AWS services, including:
\n\nAmazon Kinesis
\nAmazon S3
\nAmazon DynamoDB
\nAmazon EventBridge
\nStrong SQL skills, including data quality and validation practices.
\nExperience developing microservices for real-time data ingestion.
\nExperience with relational and NoSQL databases.
\nExperience with CI/CD and deployment tools such as Azure DevOps, Jira, and Jenkins.
\nKnowledge of monitoring, testing, automation, and data integrity across distributed and streaming systems.
\n