Senior Network Engineer - Supercomputing
Sunnyvale, CA
Engineering /
Full-time /
On-site
About the Institute of Foundation Models
We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.
The Role
As part of IFM’s Supercomputing team, you will design, optimize, and maintain low-latency, high-bandwidth networking solutions that power some of the world’s largest GPU supercomputing clusters. You’ll work on both network software and systems that enable AI training and inference workloads, using cutting-edge technologies such as NVIDIA’s RDMA-capable solutions, InfiniBand, RoCE, and GPUDirect RDMA. Our integrated product stack covers the full lifecycle of network management—from metric collection and configuration deployment to zero-touch provisioning, real-time monitoring, alerting, and auto-remediation. You will also be responsible for investigating, debugging, and quickly resolving any network-side errors in collaboration with cross-functional teams, ensuring the highest degree of reliability and performance.
Job Responsibilities
- Design & Optimization: Develop and tune RDMA-based communication systems leveraging NVIDIA GPUs, Mellanox NICs (InfiniBand, RoCE), and low-level networking technologies to support ultra-fast data transfers between nodes.
- Performance Engineering: Implement and optimize GPUDirect RDMA to enable direct memory access between GPUs and network interfaces, minimizing CPU overhead.
- Automation & Monitoring: Build network-aware software and observability tools with extensive metrics coverage, automate configuration management, and ensure robust, secure deployment pipelines through Infrastructure-as-Code (IaC) best practices.
- Integration & Collaboration: Integrate RDMA solutions within Kubernetes-based workloads and containerized environments. Collaborate closely with AI researchers, network engineers, and infrastructure teams to accelerate data pipelines and optimize collective communications using NCCL, MPI, and SHARP.
- Troubleshooting: Quickly investigate, debug, and resolve network-side issues across the full stack—from physical InfiniBand fabrics to high-level orchestration services—ensuring continuous operational excellence.
Tech Stack
- Languages & Tools: Python, Go, Rust, C/C++
- Networking Protocols & Technologies: TCP/IP, BGP, RDMA, InfiniBand, RoCE, SHARP, GPUDirect RDMA
- AI & HPC Communication Frameworks: NCCL, MPI
- Container & Orchestration: Kubernetes
- Cluster Management: Slurm
- Monitoring & Automation: Prometheus, Grafana, Ansible, Terraform
- Hardware: NVIDIA GPUs, Mellanox networking solutions
Professional Experience
- High-Performance Networks: Hands-on experience with NVIDIA RDMA technologies (e.g., GPUDirect RDMA, RoCE, InfiniBand) in HPC or AI supercomputing environments.
- Job Scheduling & Cluster Management: Familiarity with Slurm workload manager and experience troubleshooting and optimizing network performance within Slurm-managed environments.
- Advanced Communication Frameworks: Proven expertise in optimizing distributed systems using NCCL, SHARP, MPI, or similar frameworks tailored for GPU-accelerated workloads.
- Programming & System Optimization: Proficiency in Python, Go, and low-level programming languages such as Rust, C, or C++ to design and optimize networking software.
- Networking Fundamentals: In-depth knowledge of network protocols (TCP/IP, BGP, RDMA) and network architectures, both physical and logical.
- Kubernetes & Containerization: Familiarity with Kubernetes networking and experience integrating RDMA into containerized environments.
- Troubleshooting & Debugging: Strong analytical and debugging skills with a track record of rapidly resolving network-side errors and performance bottlenecks.
- Collaboration & Metrics-Driven Approach: Experience working closely with network engineers and systems architects, using extensive metrics to drive prioritization and improvements.
$200,000 - $400,000 a year
Salary depends on level.
Visa Sponsorship
This position is eligible for visa sponsorship.
Benefits Include
*Comprehensive medical, dental, and vision benefits
*Bonus
*401K Plan
*Generous paid time off, sick leave and holidays
*Paid Parental Leave
*Employee Assistance Program
*Life insurance and disability
