Big Data Engineer

São Paulo /
engine /
Full-time
At WhyLabs, we’re on a mission to build an interface between AI applications and human operators. WhyLabs started at the Allen Institute for AI (AI2), a fundamental research institute as an Observability Platform for making AI robust and reliable. From deploying in some of the world’s largest and most sophisticated tech organizations, to empowering individual data scientists and AI practitioners — we’ve seen firsthand how our tools for model and data health democratize and accelerate AI in the real world.

Success at WhyLabs relies on combining outstanding engineers and product designers, with best-in-class technologies to build a product that’s loved by both customers and open source users. Our team is responsible for scaling data pipelines, developing elegant and intuitive interfaces, employing state-of-the-art ML modeling techniques, and leveraging best practices for seamless deployment.

Why work with us?

This is a rare opportunity to be part of something unique, where your contributions will make a huge impact to both WhyLabs customers and the entire company. At WhyLabs, we are defining the AI Observability category with an exceptional product. Every person who joins the team is jumping onto a rocket ship, and will play a huge part in helping us get to the destination. What’s more, you’ll be working with a talented team who have a track record of building successful products at companies like Cloudflare, Lyft, Stripe, Amazon, and Microsoft. We’re solving technical and complex problems, and we’re doing it at scale.

As a Data Tier engineer at WhyLabs, your main goal will be to understand our customers' needs and translate them into secure and reliable systems within our platform. You will unlock engaging user experiences and build on top of the data firehose produced by our processing and monitoring layers.

This is an exciting position at the very heart of WhyLabs. You will work with many different technologies to design and build services that do everything from serving data insights and sending notifications to managing users and enforcing security boundaries. You will also create APIs for both internal and external consumption, defining the interop standards for the WhyLabs Platform. And you will have full support of our data scientists, data engineers, and the front end team as you work on features that span our entire product offering.

The Daily

    • Build, maintain, and debug data pipelines empowering a new breed of ML monitoring
    • Extend Apache Druid to enable new streaming mergeable probabilistic algorithms
    • Work anywhere with a fully remote team

The Essentials

    •  5+ years of JVM based coding experience
    • Experience with performance optimization in highly scalable architectures
    • Experience writing distributed batch ETL jobs using frameworks such as Spark
    • Experience working with distributed NoSQL databases
    • Experience working with cloud services (preferably AWS) and CI/CD pipelines

The Nice to Haves

    • Experience with data engineering, machine learning, or working in a startup is a huge plus
    • Exposure to database performance optimization
    • Some experience with infrastructure-as-code frameworks (Pulumi, AWS CDK, or similar)
    • Druid or other time series database experience
    • Exposure to Datalake design (EG deltalake)
Be Your Best At WhyLabs

At WhyLabs, we have our eyes set on an ambitious goal: to build the interface between humans and AI applications. As teams across industries adopt AI, WhyLabs enables them to operate with certainty by streamlining model monitoring, preventing costly model failures, and facilitating cross-functional collaboration. We realize people do not fit into neat boxes. At WhyLabs, we actively work to create an environment that values end-to-end ownership, diverse forms of impact, and opportunities for personal growth. We cannot complete our mission without building a diverse and inclusive team. Learn more about us at www.whylabs.ai.

At WhyLabs, you’ll be supported by an amazing team and an amazing set of benefits. We offer comprehensive medical, dental, and vision plans for employees and their families. You’ll enjoy four weeks of vacation each year and our matching 401k program lets employees plan for their future.

WhyLabs is proud to be an Equal Employment Opportunity employer and is committed to building a team that represents a variety of backgrounds, perspectives, and skills. WhyLabs embraces diversity and provides equal employment opportunities to all employees and applicants for employment. WhyLabs prohibits discrimination and harassment of any type on the basis of race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local law. All employment is decided on the basis of qualifications, performance, merit, and business need.

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

This role is US-based in Seattle, with the option to work remotely #LI-Remote.