Senior Data Engineer (m/f/d) - Berlin
Perpetua provides growth optimization and reporting technology for the world’s smartest eCommerce businesses. Through the platform, brands and sellers create goals based on strategy and rely on Perpetua’s best-in-class, always-on optimization to execute tactically. Perpetua’s intelligent reporting tools will also surface the most important metrics you need in order to easily quantify and examine the investment of your advertising dollars into your business. Integrations with Amazon, Instacart, Walmart and more ensure brands achieve full-funnel engagement and maximum visibility.
We are looking for a Senior Data Engineer with a strong understanding of data engineering and data ops topics to work with us and assist our data team.
What You’ll Do:
- Build and improve the infrastructure required for high-performance and scalable ingestion, transformation, and retrieval of terabytes of data from a variety of sources
- Work with a mix of structured and unstructured data across multi cloud-based batch- and streaming architectures
- Create and maintain data tools for business intelligence and data scientist teams which assist them in building and optimizing our product
- Identify, design, and implement internal process improvements: optimizing data delivery, automating manual processes, upscaling of the infrastructure, etc.
- Work in a multi cloud environment and establish common data practices across the organization regarding access control, security and data documentation
Who You Are:
- 3+ years of experience with object-oriented programming languages e.g Python, Scala, and/or Java
- 3+ years of experience with creating, deploying, and maintaining Apache Spark (Pyspark and Scala) applications with demonstrable deployment experience with AWS-based platforms e.g Databricks, EMR, Glue.
- Experience with workflow orchestration using Airflow
- Significant experience with using AWS Cloud services, including S3, EC2, VPCs, RDS, Lambda
- Experience with Databricks platform and Delta lakes will be highly regarded
- Deployment of infrastructure using Terraform
Bonus Points if you:
- Have experience using hive metastore implementations like glue, databricks, hive, metastore, etc
- Have experience with querying, designing, and tuning relational databases. Deep knowledge of SQL (MySql/Postgresql)
- Have working experience with massively parallel processing databases such as Redshift, or distributed NoSQL databases such as Cassandara, DynamoDB
- Can use SQL querying tools such as AWS Athena, PrestoDB
- Have experience with metadata catalog management and data lineage
- Have experience with streaming solutions Kafka, Kinesis, and equivalent processing frameworks (e.g Flink, Spark Streams)
- Acquired GCP knowledge, especially Big Query
- Worked with other data engineering tools/frameworks such as Apache Beam or Dask.
Working at Perpetua:
- Perpetua is operating in an incredible market with nearly unlimited growth potential. Our team has grown from 40 to over 200 in the last year. Our head office is in Toronto and we have offices in London, Berlin, Sao Paulo, Tokyo, and Shanghai.
- Flexible work arrangements - remote, hybrid or in office
- Here at Perpetua one of our values is “have presence”, we encourage team members to work in person 1-2 times per week
- Flexible time off policy
- A very well-located office in the heart of Berlin Mitte including a rooftop terrace with a pool
- Professional coffee machine, and the best coffee beans Berlin has to offer ;)
- Free food and drinks, regular team events
- Visa sponsorship and assistance for non-EU citizens possible
- A friendly, open, multicultural work environment with colleagues from over 35 countries
- Learn more about us here: Life at Perpetua
At Perpetua, we are dedicated to pursuing and hiring a diverse workforce with varied experiences, perspectives and opinions. We believe diversity helps our team perform better and enables us to build an outstanding product for our customers. We are an equal opportunity employer and are committed to working with applicants requesting accommodation at any stage of the hiring process.