Data Pipeline Engineer
Engineering – Data Engineering
Who we are
TFG is the largest mobile game company in Latin America, and one of the largest in the world. In 8 years, we have released over 70 games, including hits such as Sniper 3D, the leading FPS game on App Store and Google Play Store, and Colorfy, the world's most popular coloring app. Our games have been downloaded 1 billion times in 125 countries. The team started with two brothers, and now there are around 320 of us – and counting. To build the very best mobile games, we gather exceptional talent in software engineering, art and animation, product design and management, marketing, and data science.
About the Team
Data Engineering has a significant part in all of our strategic efforts and decisions at TFG. Our mission is to provide our company with complete, secure, reliable, high quality, and highly available data. To accomplish this mission, we are looking for engineers to help us to develop cutting-edge data science infrastructure. We love working with large datasets, low latency data systems, and complex business logic.
About the Role
At Data Pipelines, we are focused on gathering and implement requirements from stakeholders to build robust pipelines based on what they need to improve our decision making. Our current tech stack for data processing is Hadoop (MR), Hive, Presto and Spark, the challenge here is dealing with +1PB of historical data and Spark streaming for near real-time processing with a volume around 200k RPM. For this role, what we want is to increase the quality and production of data pipelines to make the most of the new opportunities of our industry.
More about you
- Enjoy working with complex business logic and deal with large datasets;
- Smart and creative, both, you have the ability and persistence to solve problems, big and small. Curious by nature, you're constantly looking for ways to improve upon things;
- Demonstrate critical thinking and problem-solving capabilities both independently and collaboratively;
- You're flexible, fearless, and excited to help build something;
- You're hands-on, in the right ways; willing and able to do whatever is needed, no matter the task.
What you’ll do
- Working proactively and closely with data scientists and marketing analysts on the various group projects, collecting data and developing pipelines;
- Analyzing, troubleshooting, and resolving data quality issues. Being proactive in identifying issues will be a key for your success;
- Automate and improve data quality checks to avoid wasting hours of human resources in debugging and thousand of dollars in operational misleadings;
- Implement business requirements and structure our data lake to support easy access to the data ensuring high quality and versatility.
What you'll need
- BS in Computer Science or a related fieldAt least 3 years of experience as a Data engineer;
- At least 2 years of experience with SparkSQL;
- Strong experience with Python;
- Highly skilled with SQL and building workflows through ETL/ELT;
- Experience with orchestration frameworks like airflow or Luigi.
- Relevant experience with Hadoop ecosystem (Hive, HDFS, Yarn);
- Experience with Databricks;
- Experience with Looker;
- Already designed/built an ETL to create a robust Data lake that you are proud of;
We welcome people from all backgrounds who seek the opportunity to help build the best gaming company, where everyone thrives.