hello, i'm

Neil McGlohon

Research Scientist · Software Engineer · Daydreamer

I design and build the runtime software behind large-scale AI and HPC hardware — currently the inference stack for IBM's AIU NorthPole accelerator. Before that, interconnect networks, parallel simulation, and the occasional dissertation.

View CV Get in touch
scroll

01 · selected work

Projects

A handful of systems I've helped design, build, or keep running — from rack-scale LLM inference appliances to the HPC network simulators that trained me to think about them.

02 · about

A bit about me

I am a staff research scientist at IBM Research in Almaden. While my team is based in the Bay Area, I work remotely from Troy, NY, on the brain-inspired computing team behind the IBM AIU NorthPole AI hardware accelerator.

My post-doctoral research has focused primarily on parallel and high performance computing and artificial intelligence, with earlier work in distributed/cloud computing and machine learning. Outside of work I like mountain biking, playing ukulele, and adventures with my dog.

Education

Doctor of Philosophy
Computer Science
Rensselaer Polytechnic Institute
August 2021
Master of Science
Computer Science
Rensselaer Polytechnic Institute
May 2016
Bachelor of Science
Physics
University of Oklahoma
May 2014

03 · contact

Get in touch

LinkedIn
@nmcglo GitHub
@nmcglo
Download résumé Full CV

IBM AIU NorthPole Rack-Scale LLM Inference

IBM AIU NorthPole LLM rack, annotated IBM AIU NorthPole LLM system diagram

I was a primary contributor to the vertical integration of the IBM AIU NorthPole chip into an end-to-end LLM inference system comprising 288 NorthPole accelerator cards, a high-performance runtime stack, and a containerized inference pipeline.

The research prototype system delivers 115 peta-ops at 4-bit precision and 3.7 PB/s of memory bandwidth across eighteen 2U servers. It only consumes 30 kW of power, enabling deployment in existing data centers (cloud or on-prem) without requiring exotic communication fabrics, custom hardware integration, liquid cooling, or facility power upgrades.

The modular, scalable, and reconfigurable system can run 3 simultaneous instances of the IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users at a per-user inter-token latency of 2.8 ms — 30,000 tokens/sec in total. The same system can run 18 instances of a 3-billion-parameter model at the same context length and user count, achieving 1 ms inter-token latency for 504,000 tokens/sec total.

Summarized in a pre-print on ArXiV: arxiv.org/abs/2511.15950

IBM AIU NorthPole Server

IBM AIU NorthPole node breakout

I wrote a bespoke LLM inference engine for the IBM AIU NorthPole chip. An initial demonstration featured the IBM Granite 3B model on a compact 2U research prototype using the NorthPole inference chip. Early benchmarking showed 28,356 tokens/sec of system throughput and sub-1ms/token (per-user) latency.

NorthPole is optimized for the two conflicting objectives of energy efficiency and low latency. In the low-latency regime, NorthPole (12nm) provides 72.7× better energy efficiency (tokens/second/W) than a state-of-the-art 4nm GPU. In the high-energy-efficiency regime, NorthPole (12nm) provides 46.9× better latency (ms/token) than a 5nm GPU.

Summarized in a paper at IEEE HPEC 2024: ieeexplore.ieee.org/document/10938418

IBM AIU NorthPole Edge Inference

IBM AIU NorthPole ResNet comparison IBM AIU NorthPole ResNet results IBM AIU NorthPole YOLOv4 comparison IBM AIU NorthPole YOLOv4 results

Commonly, high-performance AI runs primarily in the data center. While training belongs there, there's significant value in inference operating at the edge, minimizing transmission cost and latency for in-situ, mission-critical scenarios where microseconds matter.

To enable inference outside the data center, users need AI accelerators with both high performance and high energy efficiency, in a form factor built for edge deployment.

The IBM AIU NorthPole chip fulfills that need — capable of inference with orders-of-magnitude better energy efficiency than comparable silicon-process GPU systems, while simultaneously improving throughput and latency by orders of magnitude.

I joined the NorthPole team after the Science journal paper's author cutoff, but contributed to the work and the unpublished evolutions that followed: science.org/doi/full/10.1126/science.adh1174

CODES HPC Network Simulator

CODES simulator diagram

The Co-Design of Exascale Storage Architectures (CODES) project is a parallel discrete event simulation framework that enables massive-scale simulation of high performance computing interconnects, workloads, and technologies. I am the primary maintainer of this project's repository.

github.com/codes-org/codes

ROSS Simulator

ROSS logical processes

The Rensselaer Optimistic Simulation System (ROSS) is a parallel discrete event simulation engine designed with many-processor scalability in mind. It executes models optimistically, letting each physical processor process simulation events without forced periodic inter-core synchronization — yielding significant performance gains over other parallel execution modes. I am a contributor to this project and repository.

github.com/ross-org/ross

Interconnect Topology Research

Fit Fly interconnect topology

In the process of writing my Ph.D. dissertation, I spent years studying the history and state of the art of high performance computing communication interconnects. These networks of switches and network interface cards are the backbone of every supercomputer. The power, speed, efficiency, and cost of a supercomputer depends heavily on a suitable choice of network topology.

Congestion Control

Traffic class controller

My Ph.D. dissertation focused on the analysis of congestion in high performance computing communication interconnects and methods for both treating it once it occurs and preventing it in the first place. Passive prevention can take the form of Quality of Service techniques. Active abatement measures are called Congestion Management: strategies employed to treat congestion as quickly and efficiently as possible without negatively affecting other unrelated operators on the network.