Managing Supercomputing Resources

During my time at a major Australian public research university, I managed and allocated compute time on Gadi, a high-performance computing cluster at the NCI, for scientific researchers. Originally, this process relied on multiple Excel spreadsheets and manual calculations, making it cumbersome and error-prone.

To improve efficiency and accuracy, I transitioned our compute request process from manual emails to a standardised form using REDCap, a secure, web-based platform designed for collecting structured data and managing research and administrative workflows. Using this structured intake, I calculated fair HPC allocations based on researcher requests and historic utilisation against the university's limited compute time allocated from Gadi. I then monitored quarterly allocations and actual usage to present clear findings to the steering committee, tracking whether we were effectively using our grant, giving us the evidence needed to confidently justify investing in additional compute time from Gadi or strategically scale back to save costs. The new system cut management time by 50% and significantly reduced errors, making HPC resource management far more efficient.

Images by Taylor Vick and Alexandre Debiève, sourced from Unsplash.