Staff Platform Engineer - Big Data Platform Management
Singapore, Singapore
Cloud Infrastructure and Services – Cloud Infrastructure & Services /
Full-time /
On-site
CSIT develops products to advance the national security interests of Singapore. We use our products in a wide range of operations, including but not limited to Counter-terrorism and Computer Network Defence. We are looking for talented Senior Big Data Platform Staff Engineer. Join the awesome CSIT family and use cutting-edge technologies to protect the nation.
Role
- We are seeking an experienced Big Data Platform Staff Engineer to join our team, responsible for managing and optimizing our Big Data infrastructure platform. The successful candidate will have a deep understanding of Big Data technologies, architectures, and systems, as well as experience with managing large-scale data processing and analytics environments. The role will involve designing, implementing, and maintaining the Big Data infrastructure platform, ensuring high availability, scalability, and performance.
Responsibilities
- Design, implement, and manage the Big Data infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
- Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
- Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
- Ensure the security and governance of the Big Data infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
- Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data infrastructure platform.
- Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.
Requirements (Minimum Qualifications)
- Bachelor's degree in Computer Science, Engineering, or a related field.
- At least 8 years of experience in managing Big Data Platforms. Proven experience with technologies like Spark, Kafka, Hadoop, Elastic Stack, Kudu, NiFi and Kafka.
- Strong knowledge of Big Data architectures, including data lakes, data warehouses, and data pipelines.
- Experience in troubleshooting and resolving issues of the Big Data stack, ranging from hardware, networking, operating system to software
- Strong understanding of data storage systems, including HDFS, S3, and other object storage systems.
- Experience with scripting languages, such as Python, Scala, or Java.
- Experience in managing physical and virtual infrastructures.
- Experience in supporting search workloads in multi-terabytes and billions of records.
- Experience in distributed systems.
- Proven experience in using DevOps tools and CI/CD pipelines effectively.
- Experience in leading engineering teams.
Nice to Have
- Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer.
- Experience with cloud-based Big Data platforms, such as AWS EMR, Google Cloud Dataproc, or Azure HDInsight.
- Experience with machine learning and deep learning frameworks, such as TensorFlow, PyTorch, or Scikit-learn.
- Familiarity with containerization, such as Docker or Kubernetes.
- Experience with agile development methodologies and version control systems, such as Git.
- Experience with cloud computing platforms, such as AWS, Azure, or Google Cloud.
- Experience with containerization, such as Docker or Kubernetes.
Why join us?
- The work is purposeful and meaningful
- You will work with the best engineers
- We work with modern technologies and tech stacks
- We have excellent engineering culture and work-life balance
- We aspire to engineering and operational excellence
- We empower to innovate
- We grow together as a family
As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.
