Data Engineer - (Spark, Scala, Python, Cassandra, Elasticsearch, AWS, Airflow, SQL)
Bangalore, India
Technology – Other /
Full Time /
Hybrid
At Nielsen, we believe that career growth is a partnership. You ultimately own, fuel and set the journey. By joining our team of nearly 14,000 associates, you will become part of a community that will help you to succeed. We champion you because when you succeed, we do too. Embark on a new initiative, explore a fresh approach, and take license to think big, so we can all continuously improve. We enable your best to power our future.
Responsibilities
- Work closely with team leads and backend developers to design and develop functional, robust pipelines to support internal and customer needs
- Write both unit and integration tests, and develop automation tools for daily tasks
- Develop high quality, well documented, and efficient code
- Manage and optimize scalable pipelines in the cloud
- Optimize internal and external applications for performance and scalability
- Develop automated tests to ensure business needs are met, and write unit, integration, or data quality tests
- Communicate regularly with stakeholders, project managers, quality assurance teams, and other developers regarding progress on long-term technology roadmap
- Recommend systems solutions by comparing advantages and disadvantages of custom development and purchased alternatives
Key Skills
- Domain Expertise
- 2+ years of experience as a software/data engineer
- Bachelor’s degree in Computer Science, MIS, or Engineering
- Technical Skills
- Experience in software development using programming languages & tools/services: Java or Scala, Big Data, Hadoop, Spark, Spark SQL, Presto \ Hive, Cloud (preferably AWS), Docker, RDBMS (such as Postgres and/or Oracle), Linux, Shell scripting, GitLab, Airflow, Cassandra & Elasticsearch.
- Experience in big data processing tools/languages using Apache Spark Scala.
- Experience with orchestration tools: Apache Airflow or similar tools.
- Strong knowledge on Unix/Linux OS, commands, shell scripting, python, JSON, YAML.
- Agile scrum experience in application development is required.
- Strong knowledge in AWS S3, PostgreSQL or MySQL.
- Strong knowledge in AWS Compute: EC2, EMR, AWS Lambda.
- Strong knowledge in Gitlab /Bitbucket .
- AWS Certification is a plus
- "Big data" systems and analysis
- Experience with data warehouses or data lakes
- Mindset and attributes
- Strong communication skills with ability to communicate complex technical concepts and align organization on decisions
- Sound problem-solving skills with the ability to quickly process complex information and present it clearly and simply
- Utilizes team collaboration to create innovative solutions efficiently
