High-Performance AI Inference
Researching how lower precision, model optimisation and execution strategies can reduce latency and compute requirements.
Active Experimentation
Why This Matters
A model can be accurate in the laboratory and still be unsuitable for a real operational system if inference is too slow or too expensive.
This is especially important for multimodal models and scenes containing several people or objects, where computational cost can rise quickly.
Our inference research focuses on the trade-off between accuracy, latency, memory use and hardware requirements.
Areas Under Investigation
| Area | Research goal |
|---|---|
| BF16 | Reduce compute cost while maintaining model quality |
| FP8 | Explore lower-precision execution on supported accelerated hardware |
| Quantisation | Reduce memory footprint and execution requirements |
| Model sizing | Compare smaller and larger model families for operational tasks |
| Batching | Increase throughput when several inputs can be processed together |
| Runtime optimisation | Reduce framework and execution overhead |
| Input resolution | Balance visual detail against processing cost |
| Model architecture | Evaluate whether different architectures provide better latency/quality trade-offs |
Latency vs Throughput
These two objectives are related but not identical.
Latency asks:
How quickly can we process one event?
Throughput asks:
How many events can we process over a period of time?
A real-time system may need both: low latency for urgent events and enough throughput to support several cameras or several people in the same frame.
Multimodal Inference
Vision-language models are a particular focus because their execution cost can be much higher than conventional object detectors.
We are exploring approaches such as:
- Smaller model variants.
- Reduced numerical precision.
- More efficient image tokenisation.
- Selective region processing.
- Detector-assisted multimodal inference.
- Parallel or staged processing.
Research Goal
The objective is not simply to make models faster.
It is to identify the smallest and most efficient computational path that still produces operationally useful intelligence.
That principle is essential for deploying advanced visual AI beyond large datacentres and into practical edge environments.