Data Engineer - Lead

Mumbai, Maharashtra
1. Engineering – Data Engineering
Full time
As a lead engineer in the Data Infrastructure team, you will build platforms and tools that churn through, process & analyze petabytes of data and lead a robust team
You will work on technologies (such as: Apache Kafka, Apache Spark, Aerospike, Redshift and such ) to build a scalable infrastructure that delivers recommendations to our users in real-time.
The pace of our growth is incredible. If you enjoy working on problems at scale and if you want to build our platform for the next 100 million, join us!

Responsibilities

    • You will work closely with Software Engineers & ML engineers to build data infrastructure that fuels the needs of multiple teams, systems and products
    • You will automate manual processes, optimize data-delivery and build the infrastructure required for optimal extraction, transformation and loading of data required for a wide variety of use-cases using SQL/Spark.
    • You will build stream processing pipelines and tools to support a vast variety of analytics and audit use-cases.
    • You will continuously evaluate relevant technologies, influence and drive architecture and design discussions
    • You will work in cross-functional team and collaborate with peers during entire SDLC

Must haves

    • BE/B.Tech/BS/MS/PhD in Computer Science or a related field (ideal)
    • Lead a team in the past
    • Minimum 4 years of work experience building data warehouse and BI systems
    • Strong Java skills
    • Experience in either Go or Python (plus to have)
    • Experience in Apache Spark, Hadoop, Redshift, Athena
    • Strong understanding of database and storage fundamentals
    • Experience with the AWS stack
    • Ability to create data-flow design and write complex SQL / Spark based transformations
    • Experience working on real time streaming data pipelines using Spark Streaming or Storm
At Hotstar, we have over 100 million users on Play Store & capture close to a billion clickstream messages daily. The engineering team at Hotstar is at the centre of action and is responsible for creating unmatchable user experience. Our engineers solve real life complex problems and create compelling experiences for our customers.