Notice: While JavaScript is not essential for this website, your interaction with the content will be limited. Please turn JavaScript on for the full experience.

Lead/Senior Data Engineer
TechnoCred
Remote (for candidates physically present in US), United States of America

Job Title

Lead/Senior Data Engineer

Job Description

  • Build and manage the data processing pipeline
  • Ensure reliable processing of data, keeping error counts in check, add data validation checks, and ensure performance of data pipeline is good.
  • Responsible for building the infrastructure for running the data pipeline
  • Work with data ingestion and other feature teams to ensure the data pipeline runs smoothly.

Restrictions

  • Telecommuting is OK
  • No Agencies Please

Requirements

  • You are currently a technical lead managing a very large scale data processing pipeline.
  • You are a hands-on technical leader
  • You have 12+ years of professional software development experience
  • You have 6+ years of experience with developing data applications in Python.
  • You have a deep knowledge of building, scaling and managing a large scale real-time data pipeline (processing 100s of millions of events per day).
  • You have a deep knowledge and experience working with two or more of the following.
  • Large scale message queues like Kafka / Kinesis
  • Real-time stream processing systems like Spark, Apache Flink, Kinesis Data Streams
  • Have experience in Airflow
  • You have experience with understanding the trade-offs between scalability, reliability, -developer/ops overhead and infrastructure cost of building scalable data pipelines.
  • You have expert knowledge of AWS. Should have built or managed a large scale data pipeline hosted within AWS.
  • You have significant experience working with either SQL based databases (PostGres, etc) or MongoDB.
  • You have the ability to define the roadmap and drive execution of the team.

About the Company

This is a Full Time Opportunity with one of our client, who are into the AI and ML space. They have their own patented products and have got fundings in Millions recently.

Contact Info

Previous Senior Software Engineer, NxCodes Ltd in Remote, UK Next Data Analyst - Heavy Industry, Transition Zero in London, United Kingdom