We've just released new improvements!

New Hardware Support

  • Intel Ice Lake instances from both AWS and GCP are now available. We also have plans to onboard Ice Lake instances from Azure in coming weeks.

  • TVM, ONNX, TensorFlow, and TFLite engines are supported on these devices with comparable coverage as on other x86 cloud CPUs in our platform.

  • Jetson Nanos are also available!

Improved ONNX-RT Support

  • The OctoML Platform now has a higher success rate for dynamic model inputs! OctoML now supports the TensorRT EP for ONNX Runtime, which improves coverage for ONNX-RT acceleration on GPUs and simplifies the steps users could take to reproduce our results.

  • Previously, we offered ONNX acceleration via the onnx-tensorrt tool.

UX Update

  • OctoML has updated our hardware selection UX, to make it easier to search for and add hardware among the many hardware options we offer.

Did this answer your question?