Data Structures

This section describes the set of available data structures present in the MLI interface:

mli_tensor Structure

All functions use the mli_tensor structure as the main container for input and output data. It represents a multi-dimensional array of elements. The mli_tensor structure describes the shape of this array, its data format, and the way it is organized in memory.

typedef struct mli_tensor {
   mli_data_container data;
   uint32_t shape[MLI_MAX_RANK];
   int32_t mem_stride[MLI_MAX_RANK];
   uint32_t rank;
   mli_element_type el_type;
   mli_element_params el_params;
} mli_tensor;

The table mli_tensor Structure Field Descriptions describes the fields in the mli_tensor structure.

mli_tensor Structure Field Descriptions

Field name

type

Comment

data

mli_data_container

The meaning of this field varies based on the setting of the rank field:

  • rank  > 0: General Tensor. The data contains a pointer to the data.

  • rank == 0: Scalar tensor. The data holds only a single value and this value is directly stored into this field.

Type of the data preserved in this container must be in sync with el_type field.

See mli_data_container Structure for more info.

shape

uint32_t[]

Array with tensor dimensions. Dimensions are stored in order starting from the one with the largest stride between the data portions. For example, for tensor T of shape (channels, height width) stored in HWC layout, shape[0] = height, shape[1] = width, shape[2] = channels. Shape[3] is unused. The size of the array is defined by MLI_MAX_RANK*.

mem_stride

int32_t[]

Array with the distance (in elements) to the next element in the same dimension. Positive values are supported only. To compute the size in bytes, the number of elements needs to be multiplied by the bytes per element. For example, for a matrix A(rows,columns), mem_stride[1] contains the distance to the next element (=1 in this example), and mem_stride[0] contains the distance from one row to the next (=columns in this example). The size of the array is defined by MLI_MAX_RANK*.

Values of mem_stride array must decrease gradually and must not be less than if they would be computed from the shape. For example, for a tensor of shape \((Height, Width, Channels)\):

  • mem_stride[0] >= 1 x Channels x Width AND mem_stride[0] >= mem_stride[1]

  • mem_stride[1] >= 1*Channels AND mem_stride[1] >= mem_stride[2]

  • mem_stride[2] >= 1

mli_move is the only function which can write the mem_stride field of the dst tensor. Other kernels don’t update this field

rank

uint32_t

Number of dimensions of this tensor (Must be less or equal to MLI_MAX_RANK*)

el_type

mli_element_type

Enum depicting the type of the element stored in the tensor. See mli_element_type Enumeration for more info.

el_params

mli_element_params

Union of structs containing the quantization parameters of the elements stored in the tensor. Details on supported quantization schemes are discussed in Data Formats

See mli_element_params Union for more info.

Note

MLI_MAX_RANK is set to 4.

Important

mli_tensor is valid if all its fields are populated in a non-contradictory way which implies:

  • rank and shape fields are aligned with mem_strides field

  • data container points to a memory region of el_type elements or contains a single element itself (see mli_data_container Structure). Its capacity is enough to preserve data described by rank, shape and mem_stride fields.

  • el_params structure is filled properly according to el_type field and related quantization scheme (see Data Formats)

mli_data_container Structure

mli_data_container is a container to represent polymorphic data. It stores pointer to data or a single value that intend to be directly used in arithmetical operations.

mli_data_container is defined as follows:

typedef struct _mli_data_container {
  uint32_t  capacity;
  union {
    int32_t*  pi32;
    int16_t*  pi16;
    int8_t*   pi8;
    float*    pf32;
    int32_t   i32;
    int16_t   i16;
    int8_t    i8;
    float     f32;
  } mem;
} mli_data_container;

mli_data_container Structure Field Description describes the fields in the mli_data_container struture.

mli_data_container Structure Field Description

Field Name

Type

Comment

capacity

uint32_t

Size in bytes of the memory that the mem field points to. In case there is no buffer attached, the capacity must be set to 0.

mem

Union

This field is the union of different possible data container types. In case capacity is set to 0, this field is not a pointer, but it contains the data itself.

mem.pi32

int32_t *

Pointer to array of 32 bit signed integer values.

mem.pi16

int16_t *

Pointer to array of 16 bit signed integer values

mem.pi8

int8_t *

Pointer to array of 8 bit signed integer values

mem.pf32

float *

Pointer to array of 32bit single precision floating point value

mem.i32

int32_t

32 bit signed integer value.

mem.i16

int16_t

16 bit signed integer value.

mem.i8

int8_t

8 bit signed integer value.

mem.f32

float

32bit single precision floating point value.

mli_element_type Enumeration

mli_element_type enumeration defines the basic element type stored in tensor structure. Based on this information, library functions may define sizes, algorithms for processing, and other implementation specific things. mli_element_type is defined as follows:

typedef enum {
   MLI_EL_FX_4  = 0x004,
   MLI_EL_FX_8  = 0x008,
   MLI_EL_FX_16 = 0x010,
   MLI_EL_SA_8  = 0x108,
   MLI_EL_SA_32 = 0x120,
   MLI_EL_FP_16 = 0x210,
   MLI_EL_FP_32 = 0x220
} mli_element_type;

mli_element_type Enumeration Values Description describes the entities in the mli_element_type union.

mli_element_type Enumeration Values Description

Enumeration Value

Description

MLI_EL_FX_4

4 bit depth fixed point data. For future use.

MLI_EL_FX_8

8 bit depth fixed point data. See fx8 in MLI Data Formats

MLI_EL_FX_16

16 bit depth fixed point data. See fx16 in MLI Data Formats

MLI_EL_SA_8

8 bit asymetrical signed data. See sa8 in MLI Data Formats

MLI_EL_SA_32

32 bit asymetrical signed data. See sa32 in MLI Data Formats

MLI_EL_FP_16

Half precision floating point data. For future use.

MLI_EL_FP_32

Single precision floating point data. See fp32 in MLI Data Formats

mli_element_params Union

mli_element_params stores data quantization parameters required for arithmetical operations with tensor elements. Details on supported quantization schemes are discussed in Data Formats.

mli_element_params is defined as follows:

typedef union _mli_element_params {
   struct{
      uint32_t frac_bits;
   } fx; /* FiXed point \*/

   struct {
      mli_el_param_type type;
      mli_data_container zero_point;
      mli_data_container scale;
      mli_data_container scale_frac_bits;
      int32_t dim;
   } sa; /* Signed Asymmetric \*/
} mli_element_params;

mli_el_param_type is defined as follows:

typedef enum {
   MLI_EL_PARAM_SC16_ZP16 = 0
} mli_el_param_type;

mli_element_params Union Field Description describes the fields in the mli_element_params union. Several members of this union are used to support per-axis quantization. sa.dim indicates over which axis (dimension) of the tensor the quantization parameters can vary. For instance in a CHW layout, dim = 0 means that for each channel there is a different zero point and a different scale factor. The size of these arrays is the same as the number of channels in the tensor (array_size = shape[dim]).

mli_element_params Union Field Description

Field Name

Type

Comment

fx.frac_bits

uint8_t

Number of fractional bits.

sa.type

mli_el_param_type

Enum depicting the types of the quantization parameters in the tensor. Only MLI_EL_PARAM_SC16_ZP16 is currently supported which reflects the following parameters according the description below.

sa.dim

int32_t

Tensor dimension to which the arrays of quantization parameters apply

sa.zeropoint

mli_data_container

16-bit signed integer zero-point offset.

  • sa.dim < 0: Single value for all data in tensor.

  • sa.dim >= 0: Pointer to an array of zero points relating to configured dimension (sa.dim).

See mli_data_container Structure for more info.

sa.scale

mli_data_container

16-bit signed integer scale factors. Only positive scale factors are supported.

  • If sa.dim < 0: sa.scale is a single value for all data in tensor

  • If sa.dim >= 0: sa.scale is a pointer to an array of scale factors related to configured dimension (sa.dim).

See mli_data_container Structure for more info.

sa.scale_frac_bits

mli_data_container

8-bit signed integer exponent of values in sa.scale field. The field stores the exponent as the number of fractional bits.

  • If sa.dim < 0: sa.scale_frac_bits is a single value

  • If sa.dim >= 0: sa.scale_frac_bits is a pointer to an array of frac bits per each value in sa.scale array.

See mli_data_container Structure for more info.

Example

FX16 tensor might be populated in the following way:

mli_tensor tsr_fx16 = {0};

// Filling quantization params
tsr_fx16.el_type = MLI_EL_FX_16;
tsr_fx16.el_params.fx.frac_bits = 12;

// Filling other fields of tsr_fx16

Example

SA8 tensor quantized on per-tensor level might be populated in the following way:

mli_tensor tsr_sa8 = {0};

// Filling quantization params
tsr_sa8.el_type = MLI_EL_SA8;
tsr_sa8.el_params.sa.type = MLI_EL_PARAM_SC16_ZP16;
tsr_sa8.el_params.sa.dim = -1; // e.g Per-Tensor (all values shares the same quant params)

// Set all capacities to 0 as values are directly stored inside the container
tsr_sa8.el_params.sa.zero_point.capacity = 0;
tsr_sa8.el_params.sa.scale_frac_bits.capacity = 0;
tsr_sa8.el_params.sa.scale.capacity = 0;

tsr_sa8.el_params.sa.zero_point.mem.i16 = -128;
tsr_sa8.el_params.sa.scale_frac_bits.mem.i8 = 3;
tsr_sa8.el_params.sa.scale.mem.i16 = 5; // (5 \ 2^3) = 0.625

// Filling other fields of tsr_sa8

Example

SA8 tensor quantized on per-axis level might be populated in the following way:

mli_tensor tsr_sa8_per_axis = {0};
int16_t scales[] = {...};
int8_t scales_frac[] = {...};
int16_t zero_points[] = {...};

// Filling quantization params
tsr_sa8_per_axis.el_type = MLI_EL_SA8;
tsr_sa8_per_axis.el_params.sa.type = MLI_EL_PARAM_SC16_ZP16;
tsr_sa8_per_axis.el_params.sa.dim = 0; // e.g Per 0th dimension

tsr_sa8_per_axis.el_params.sa.zero_point.mem.pi16 = zero_points;
tsr_sa8_per_axis.el_params.sa.zero_point.capacity = sizeof(zero_points);

tsr_sa8_per_axis.el_params.sa.scale_frac_bits.mem.pi8 = scales_frac;
tsr_sa8_per_axis.el_params.sa.scale_frac_bits.capacity = sizeof(scales_frac);

tsr_sa8_per_axis.el_params.sa.scale.mem.pi16 = scales;
tsr_sa8_per_axis.el_params.sa.scale.capacity = sizeof(scales);

// Filling other fields of tsr_sa8_per_axis

mli_lut Structure

Several functions use a look-up table (LUT) to perform data transformation. The LUT represents a function in a table form that can be used to transform input values (function argument) to output values (function result). The mli_lut structure is a representation of such a table.

The mli_lut struct describes the data in the LUT, including the format of its input and output.

typedef struct _mli_lut{
   mli_data_container data;
   mli_element_type type;
   int32_t length;
   int32_t in_frac_bits;
   int32_t out_frac_bits;
   int32_t input_offset;
   int32_t output_offset;
} mli_lut;

The following table describes the fields in the mli_lut structure.

mli_lut Structure Field Descriptions

Field name

type

Comment

data

mli_data_container

This field has a union of different possible data container types. Pointer of specified type (see the type field in this table) should point to an array with the LUT table data.

See mli_data_container Structure for more info.

data.capacity

uint32_t

Size in bytes of the allocated memory that the data field points to.

type

mli_element_type

Enum depicting the type of the element stored in the data field. Values in this enum are listed in section mli_element_type Enumeration. Only MLI_EL_FX_16 entity is currently supported.

length

int32_t

Number of values stored in the LUT table

in_frac_bits

int32_t

Number of fractional bits for the LUT input (argument)

out_frac_bits

int32_t

Number of fractional bits for the LUT output (result)

input_offset

int32_t

Offset of input argument which is added before applying the LUT function.

output_offset

int32_t

Offset of output which is subtracted from LUT function result.

Kernel Specific Configuration Structures

A significant number of MLI kernels must be configured by specific parameters, which influence calculations and results, but are not directly related to input data. For example, padding and stride values are parameters of the convolution layer and the type of ReLU is a parameter for ReLU transform layer. All specific parameters for particular primitive type are grouped into structures. This document describes these structures along with the kernel description they relate to. The following tables describe fields of existing MLI configuration structures: