Data Engineering -Senior Data Engineer
Noida, Uttar Pradesh /
Technology – DWH /
For this role, we require someone with strong product design sense. The position requires one to work on complex technical projects and closely work with peers in an innovative and fast-paced environment.
· Grow our analytics capabilities with faster, more reliable data pipelines, and better tools, handling petabytes of data every day.
· Brainstorm and create new platforms that can help in our quest to make data available to cluster users in all shapes and forms, with low latency and horizontal scalability.
· Make changes to our data platform, refactoring/redesigning as needed and diagnosing any problems across the entire technical stack.
· Design and develop a real-time events pipeline for Data ingestion for real-time dash-boarding.
· Develop complex and efficient functions to transform raw data sources into powerful, reliable components of our data lake.
· Design & implement new components and various emerging technologies in Hadoop Eco System, and successful execution of various projects.
· Optimize and improve existing features or data processes for performance and stability.
· Conduct peer design and code reviews.
· Write unit tests and support continuous integration.
· Be obsessed about quality and ensure minimal production downtimes.
· Mentor peers, share information and knowledge and help build a great team.
· Monitor job performances, file system/disk-space management, cluster & database connectivity, log files, management of backup/security and troubleshooting various user issues.
· Collaborate with various cross-functional teams: infrastructure, network, database.
· Fluent with data structures, algorithms and design patterns.
· Strong hands-on experience with Hadoop, MapReduce, Hive, Spark.
· Excellent programming/debugging skills in Java/Scala.
· Experience with any scripting language such as Python, Bash etc.
· Good to have experience of working with noSQL databases like Hbase, Cassandra.
· Hands on programming experience with multithreaded applications.
· Good to have experience in Database, SQL, messaging queues like Kafka.
· Good to have experience in developing streaming applications eg Spark Streaming, Flink, Storm, etc.
· Good to have experience with AWS and cloud technologies such as S3
· Experience with caching architectures like Redis, Memcached etc.
· Memory optimization and GC tuning.
· Experience with profiling and performance optimizations.
· Experience with agile development methodologies and DevOps action.