We've just released new improvements!
New Hardware Support
Intel Ice Lake instances from both AWS and GCP are now available. We also have plans to onboard Ice Lake instances from Azure in coming weeks.
TVM, ONNX, TensorFlow, and TFLite engines are supported on these devices with comparable coverage as on other x86 cloud CPUs in our platform.
Jetson Nanos are also available!
Improved ONNX-RT Support
The OctoML Platform now has a higher success rate for dynamic model inputs! OctoML now supports the TensorRT EP for ONNX Runtime, which improves coverage for ONNX-RT acceleration on GPUs and simplifies the steps users could take to reproduce our results.
Previously, we offered ONNX acceleration via the onnx-tensorrt tool.
OctoML has updated our hardware selection UX, to make it easier to search for and add hardware among the many hardware options we offer.