Senior Hadoop Infrastructure Engineer - Data Platform
At Paytm Labs, we build technologies that powers Paytm India, the world's’ fastest growing mobile payments and commerce ecosystem. In addition to, the Paytm Canada app. We use our skills and our biggest asset – data, to make our dent in this universe.
We are committed to offering the most transparent, secure, and personalized consumer experience to over 230 million users. We believe that this kind of scale, and the unique problems that it presents attracts curious candidates like yourself.
DaaS (Data as a Service) team at Paytm Labs operates a charter of capturing, storing and processing data reliably at scale. The DaaS Team makes this data available for a large set of products that are used for internal and external services. The core infrastructure that powers this platform operates at a scale of speed, performance, and complexity that few others can claim. The issues we face with large-scale data storage, low-latency retrievals, high volume requests and high availability are common yet complex. To help us solve these challenges, we are looking for the best of the best engineering talent to come and join our cool & rewarding environment! Paytm Labs is in a hyper growth phase, and this is a stellar opportunity to make an impact. As Senior Hadoop Engineer, you will be a core contributor in the Data Platform team and help us deliver the world-class Data Platform Paytm is gearing towards for data products growth.
• Get stuff done: A problem partially solved today is better than a perfect solution next year. Have an idea during the night? Discuss it with your team in the morning, code it, push it at noon, test it in the afternoon and deploy it the next morning.
• You will participate in and build tools to diagnose and fix complex distributed systems handling petabytes of data & drive opportunities to automate infrastructure, deployments, and observability of data services.
• You will test, monitor, administer, optimize and operate multiple Hadoop / Spark clusters across cloud providers - AWS, GCP, Aliyun and on premise data centers, primarily in Python , Java and Scala.
• Investigate emerging technologies in Hadoop ecosystem that relate to our needs and implement those technologies.
• Partner with Hadoop developers in building best practices for Data Warehouse and analytics environment. Share an on-call rotation and handle service incidents.
• Working with Big Data tools and building high performance, high throughput, and distributed data pipeline and big data platform with Hadoop, Spark, Kakfa, Hive, and Presto.
• Minimum 3+ years of handling services in a large scale distributed systems environment, preferably Hadoop.
• Hands on experience with Hadoop (or similar) ecosystem - Yarn, Hive, HDFS, Spark, Presto, Parquet, HBase
• Experience with workflow management (Airflow, Oozie, Azkaban)
Familiarity with systems management tools (Puppet, Chef, Capistrano, etc)
• Knowledge of Linux operating system internals, file systems, disk/storage technologies and storage protocols and networking stack.
• Proven knowledge of systems programming (bash and shell tools) and/or at least one scripting language (Python, Ruby, Perl).
• Be adaptable and able to focus on the simplest, most efficient & reliable solutions.
What We Offer:
• We are proud to announce that we have been certified as a Great Place to Work !
• A collaborative, open work environment that fosters ownership, creativity, and urgency.
• Group Health Benefits plan and enrollment right from the day you start working, no waiting period.
• We believe in hard work, fun, innovation, and team building. Every other month, we plan an activity to celebrate that. So far, we've been biking, axe-throwing, bowling, rocking climbing, and go-karting.
• Chocolates & Snacks: Our "Chief Chocolate Officer" is in charge of stocking high-quality chocolates for all.
• Birthdays are celebrated with cakes, and special office events have beer and whiskey.
• We also get a weekly delivery of groceries to our office.
• Ping Pong: A chance to be Paytm Labs table tennis champion!
*Selected candidates will be given a technical challenge*
We thank all applicants, however, only those selected for an interview will be contacted.
Paytm Labs is committed to meeting the accessibility needs of all individuals in accordance with the Accessibility for Ontarians with Disabilities Act (AODA) and the Ontario Human Rights Code (OHRC). Should you require accommodations during the recruitment and selection process, please let us know. Paytm Labs is an equal opportunity employer.