Lead/Senior Data Engineer
Remote (for candidates physically present in US), United States of America
Job TitleLead/Senior Data Engineer
- Build and manage the data processing pipeline
- Ensure reliable processing of data, keeping error counts in check, add data validation checks, and ensure performance of data pipeline is good.
- Responsible for building the infrastructure for running the data pipeline
- Work with data ingestion and other feature teams to ensure the data pipeline runs smoothly.
- Telecommuting is OK
- No Agencies Please
- You are currently a technical lead managing a very large scale data processing pipeline.
- You are a hands-on technical leader
- You have 12+ years of professional software development experience
- You have 6+ years of experience with developing data applications in Python.
- You have a deep knowledge of building, scaling and managing a large scale real-time data pipeline (processing 100s of millions of events per day).
- You have a deep knowledge and experience working with two or more of the following.
- Large scale message queues like Kafka / Kinesis
- Real-time stream processing systems like Spark, Apache Flink, Kinesis Data Streams
- Have experience in Airflow
- You have experience with understanding the trade-offs between scalability, reliability, -developer/ops overhead and infrastructure cost of building scalable data pipelines.
- You have expert knowledge of AWS. Should have built or managed a large scale data pipeline hosted within AWS.
- You have significant experience working with either SQL based databases (PostGres, etc) or MongoDB.
- You have the ability to define the roadmap and drive execution of the team.
About the Company
This is a Full Time Opportunity with one of our client, who are into the AI and ML space. They have their own patented products and have got fundings in Millions recently.
- Contact: Suhail
- E-mail contact: email@example.com
- Web: https://www.careers-page.com/technocred/job/65V95W