The OctoML Platform infers the input name, type, and dimensions for models upon upload. If your model features dynamic batch sizes or input string length, you must designate a static, placeholder batch size/string length for the optimization.
We recommend selecting a batch size or string length that you anticipate will work for most inputs in your data set, and then padding or clipping as necessary. You may want to experiment with optimizing for different sizes or lengths and testing each model variant against your data sets to determine real world performance.
For some models you may need to enter inputs manually, particularly if your model has a dynamic input shape. Using a third-party viewer for machine learning models like Netron may be useful to confirming correct input names, types, and dimensions.
For ONNX models, the input layer name is the layer name in the provided ONNX file, commonly input or data. The input layer type corresponds to the corresponding NumPy type name used by the model’s input layer.
For example, specify
float32fornp.float32, oruint8fornp.uint8. The input layer shape is a comma-separated list of layer dimension sizes. For image-based models, these are provided in NCHW format, for example,1,3,512,512for a model configured with a batch size of 1, three channels, and of height/width of 512x512 pixels.
Torchscript (PyTorch) models embed the least information about their input shapes. As such, you’ll need to provide not only input dimensions but also the total number of input dimensions. See the example below from our WebUI, where an ImageNet model’s inputs are properly provided.
The Input Name, by convention, is usually simply numbered as “input0” and “input1”, but can be set arbitrarily without disrupting the function of the model
