Software Engineer - Data Infrastructure
Data Science & Engineering
Shopify’s platform is growing at an incredible rate, generating vast amounts of data. To keep up, the data platform team just completed a migration out of our on premise data center (DC) and into Google Cloud (GCP). The work was worth it; we can react faster to the needs of the Data Science and Engineering team, the rest of Shopify, and most importantly, our merchants.
We’ve still got a lot of work to do and that’s where you can help! The challenges include building out our deployment tools to provide blue/green and zero downtime deploys for our services; work with Google’s Beyondcorp security model to continue to serve our data in a secure and auditable way without adding too much overhead; build up our next generation streaming low-latency pipeline to make the data we provide even more useful and timely.
We’re looking for software engineers that are interested in collaborating on these challenges with an SRE mindset and deploying platform services on a very large scale. You’ll need a curiosity of how our systems work under the hood and how we can leverage them for building a data platform to truly help and grow the hundreds of thousands of entrepreneurs on Shopify’s platform. We’d love to grow our team with someone that is passionate about helping us solve these challenges and continue to build out our data platform to make commerce better for everyone.
Requirements for the role:
- A systems-level approach. You’ve worked across in the stack, from the OS all the way up.
- Comfort with multiple languages. You’re a low-level generalist who is comfortable with multiple languages such as Go, Python and languages which target the JVM like Scala or Java.
- A mind for scale. You’re curious about building large distributed systems, and often ask the question, “but does it scale?”.
- Well-founded opinions about writing code and approaching problems. You’re comfortable with automated testing, code refactoring, and software engineering best practices.
- Passion for working with a team. You value collaborating on problems, asking questions, delivering feedback, and supporting others in their goals.
- Working with data at petabyte scale.
- Running or using Kafka, Spark, Presto, Hive, YARN or Hadoop.
- Any Cloud Platform (GCP, AWS or Azure): Compute instances, deployment tooling, storage.
- Developing and orchestrating large Docker deployments both with Kubernetes and without.
- Using log aggregation technologies and monitoring tools (e.g. Splunk, Datadog, Stackdriver).
- Ensuring that our data platform stays online, secure, and performant.
- Developing configuration management and automation tools.
- Building out our monitoring and analytics tooling to get insights about our platform usage.
- Providing automation for on-demand test cluster deployments.
- Deploying machine learning infrastructure to our clusters.
- Building a world-class data analytics platform to help both internal and external customers, focusing on making the lives of our hundreds of thousands of merchants better.
We know that looking for a new role can be both exciting and time-consuming, and we truly appreciate your effort. Yandu is looking forward to learning more about you.
And remember, we want to know what you're really interested in building and why you want to build it at Shopify, so please give us as much detail on this as you'd like in your cover letter - we do love a good story. 👍 📖