Performance Engineer

About CuspAI CuspAI is the frontier AI company on a mission to solve the breakthrough materials needed to power human progress. While nature took billions of years to perfect molecules, we are harnessing AI to unlock trillion-dollar materials breakthroughs in months, not millennia. Our founding team is the most cited in the world, comprised of world-class researchers in AI, chemistry and engineering. We are working on some of the hardest and most important challenges including energy, clean water, the future of compute, and carbon capture, and this is just the start of what our 'search engine' for next-generation materials will unlock. We invite you to be part of a diverse, innovative team at the intersection of AI and materials science, working to create impactful partnerships that drive innovation, scalability, and industry collaboration. This work matters. Your work matters. We’re on the cusp of the on-demand materials era. Join us. The Role Due to the rapid scaling of our compute requirements, we are seeking a Performance Engineer to serve as the vital bridge between our frontier AI models and our high-performance computing infrastructure. Your Impact In this role you will be building the optimization layers and profiling infrastructure that maximizes the efficiency of our large-scale training and inference workloads, which is critical for accelerating our materials discovery pipeline and ensuring our researchers can iterate at the speed of thought.   At CuspAI, compute is our most precious resource. Your main focus initially will be profiling our current training runs to identify bottlenecks in Model FLOP Utilization (MFU). Over time you are likely to get involved in designing custom kernels and low-level optimizations that allow us to train larger, more complex models for chemistry and physics that were previously computationally prohibitive. What You Will Do Training & Inference Optimization • Analyze model architectures to optimize how mathematical operations map onto hardware, specifically focusing on NVIDIA GPU kernels and collective communication layers (NCCL). • Implement state-of-the-art efficiency techniques such as quantization, model pruning, and distillation to lower latency for our materials “search engine” APIs. • Streamline data pipelines to ensure high-performance accelerators are never “starved” for data during massive training runs. Profiling, Observability & Cost Management • Use advanced observability tools to trace end-to-end execution of training jobs, pinpointing exact locations of processing clogs or interconnect contention. • Build and maintain real-time dashboards tracking vital system metrics, including MFU, memory bandwidth utilization, and cluster uptime. • Monitor and justify the utilization of our AI accelerators (e.g. H100 clusters), preventing idle time and memory leaks to ensure every dollar of compute spend translates into scientific progress. • Develop scaling laws and internal tools to predict system behavior across compute, memory, and networking as our models grow in parameter count. Interdisciplinary Collaboration • Work closely with AI Researchers to adapt model architectures for better hardware compatibility without sacrificing scientific accuracy. • Partner with the Infrastructure team to influence the design of our future compute clusters based on performance data and profiling results. Must Have Skills and Qualifications: • You are someone who gets excited about the opportunity to enable scientists to work on world changing challenges in this domain, with a personal interest in the potential applications of the technology that Cusp is building. • You have 3+ years of experience in performance engineering, site reliability engineering, or low-level software devel