Backend Workflow Engineer
We’re looking for an experienced Software Engineer who is excited to work on our Workflow engine, leveraging and improving our open-source Digdag project and its integration as a key component of our product. As part of the Backend team, you’re likely to work on a number of different challenges that cross our entire stack, designing and implementing new features, and building upon key aspects of our technology. This team provides a foundational component in our public cloud data platform that leverages adapted open-source query engines (Hadoop/Hive, Presto) serving over half a million customer queries per day, over data ingestion rates of more than a million rows per second.
Because our Workflow engine is customer-facing, you can expect to have direct contact with many of our most active users, hearing their feedback firsthand and working with our Product team to incorporate their input. As part of our larger Backend team, you will also be regularly involved in production operation of our systems. Being able to combine customer experience and operational experience will provide many opportunities to directly impact the direction of our product development and key platform improvements.
Success in this role requires a commitment to collaboration and maintaining an open work environment, and comfort doing so in a globally distributed team. Engineers at Arm Treasure Data appreciate and value working alongside their team members, both inside their immediate team and across the entire organization. They recognize that a key aspect of a responsible and open engineering culture is being comfortable with asking for help, and always being willing to go the extra mile to help each other and the customer.
If this sounds like the kind of opportunity you’ve been looking for, then we’re going to need your resume of course, but more importantly include a short note giving us a sense of why you think you are absolutely the right person for this job and how you are going to meet and exceed the objectives outlined below.
Things you will do
Design and implementation of new features for our Workflow engine, solving problems including distributed execution of complex execution graphs and hosting customer-defined applications within workflows.
・Release of some Workflow features to our open-source workflow project,Digdag.
Write testable, maintainable, observable code with an eye to enabling iterative improvement and experimentation.
・Work with UX and Frontend teams to improve the customer's day-to-day experience.
・Work with Customer Success and Sales Engineering teams to directly solve customer problems and pain points.
・Gain experience operating our Workflow engine, and using that experience for further improvement of the Arm Treasure Data product and Digdag as an open source project.
・Work with Backend and SRE team members to improve the delivery of tested systems to production.
・Participate in on-call rotation with other members of the Backend team.
・With team leadership and the Product Management team, help define and drive the product roadmap.
Your background and skills should include
・A BS in Computer Science or a related field, or equivalent work experience.
・A minimum of 4 years of professional experience as a Software Engineer.
・Demonstrated ability working collaboratively in cross-functional teams and a strong track record for delivery as part of a team.
・Industry experience running services in public cloud IaaS providers, specifically around computing, storage, relational databases, and load balancers to achieve service redundancy and robustness.
・Experience with one or more JVM languages (e.g. Java, Kotlin, Scala.)
・A solid understanding of computer science (data structures, algorithms, etc.)
Experience designing and developing APIs, middleware, and/or backends to support data-heavy analysis systems.
・Articulate and personable with strong spoken and written language abilities.
・Able to work with a distributed team.
・Ability to handle stressful situations with rigor and composure.
・Experience mentoring and collaborating directly with other team members.
・Strong self-motivation, and sensitivity about on-time delivery.
We would be thrilled if you
・Have experience with workflow engines like Digdag or Apache Airflow.
・Have had experience building and managing data-centric services that support a large user base.
・Have strong UNIX systems programming background which should include experience with operating systems, multithreading, distributed computing, and related challenges.
・Are knowledgeable of PostgreSQL, Hadoop, Presto, MySQL, or other open-source distributed database/engines.
・Are familiar with security best practices.
・Have hands-on experience with infrastructure as code (Terraform, Chef).
・Take equal pride in optimizing as well as building systems and are able to share a success story around the former.
・Own or are actively contributing to any open-source project.