Senior Distributed Systems Engineer - Data Platform Insights Services
Los Gatos, California
Streaming – Data Platform Engineering
At Netflix, the big data platform is the core foundation in driving all of our product decisions that directly impact our customer experience. It is leveraged across the company to build data models for recommendations, analyze customer streaming experience, content take-rate, plus more. It is critical to the success of our business as we heavily rely on analyzing data to revolutionize internet TV!
We are the data infrastructure team that builds an ecosystem of microservices and near real-time data ingestion pipelines that expose the big data warehouse and big data platform as a robust and highly available service to the rest of Netflix. We are constantly evolving our infrastructure. We are designing new microservices to take our architecture to the next level to keep pace with our increasing data and metadata needs.
Specifically, we are working on:
- an intelligent error classification system that can detect and accurately attribute the cause of an error when a process fails using data points from thousands of machines and distributed services
- a predictive SLA service that can intelligently detect which processing pipelines are keeping up or lagging behind by combining information like SLAs, dependency graphs, past performance, with system load and performance
- a ‘data-driven’ data platform that collects all the data it needs not only to interpret failures and quantify systems performance; but also to predict job trends
- If this aligns well with your passion, we would love to hear from you!
This would be your dream job if you enjoy:
- Working with a massive amount of data (100+ PB) that is growing rapidly.
- Understanding and solving real business needs at large scale by applying your software engineering and analytical problem solving skills.
- Architecting and building a robust, scalability, and highly available distributed infrastructure.
- Leading cross-functional initiatives and collaborating with engineers across teams.
- Sharing our experiences with the open source communities and contributing to Netflix OSS.
- You have 6+ years of experience in building large scale distributed applications.
- You are proficient in design and development of RESTful web services.
- You are an expert in Java. Python expertise is a plus.
- You have a BS/MS/PhD in Computer Science or related field.
- Most importantly, you are thrilled to help solve our big data challenges while revolutionizing internet TV!
To learn more about the team, here is our recent talk at re:Invent conference (in slides or video) and Strata that describes what we do in the big data space!