Encoding Format Specification¶

AIMET Quantization Simulation determines scale/offset values for activation and parameter tensors in the model. This scale/offset information is also referred to as ‘quantization encoding’. When a model is exported using QuantizationSimModel.export() API, an encoding file is also exported that contains quantization encodings for the model. This encoding file can then be used by a target runtime like Qualcomm® AI Engine Direct when running the model on-target.

The encoding file uses a JSON syntax. The file format is usable with model exports from aimet_torch, aimet_tensorflow, and aimet_onnx.

1. Versioning¶

Encoding format will follow XX.YY.ZZ versioning format as describe below,

XX = Major Revision
YY = Minor Revision
ZZ = Patching version

A change in major revision indicates substantial change to the format, updates to minor version indicates additional information element being added to encoding format and might require update to fully consume the encodings. The patching version shall be updated to indicate minor updates to quantization simulation e.g. bug fix etc.

2. Version 0.6.1¶

2.1. Encoding specification¶

“version”: “string”
“activation_encodings”:
{
    <tensor_name>: [Encoding, …]
}
“param_encodings”
{
    <tensor_name>: [Encoding, …]
}
"quantizer_args":
{
     "activation_bitwidth": integer,
     "dtype": string,
     "is_symmetric": string,
     "param_bitwidth": integer,
     "per_channel_quantization": string,
     "quant_scheme": "string"
}

Where,

"version” is set to “0.6.1”
<tensor_name> is a string representing the tensor in onnx or tensorflow graph.

‘Encoding’ structure shall include an encoding field “dtype” to specify the datatype used for simulating the tensor.

Encoding:{
    dtype: string
    bitwidth: integer
    is_symmetric: string
    max: float
    min: float
    offset: integer
    scale: float
}

Where,

dtype: allowed choices int, float
bitwidth: constraints >=4 and <=32
is_symmetric: allowed choices True, False

when dtype is set to ‘float’, Encoding shall have the following fields

Encoding:{
    dtype: string
    bitwidth: integer
}

bitwidth defines the precision of the tensor being generated by the producer and consumed by the downstream consumer(s).

The quantizer_args structure describes the settings used to configure the quantization simulation model, and contains usable information about how encodings were computed. The field is auto-populated and should not require a manual edit from users. It can be broken down as follows:

activation_bitwidth: Indicates the bit-width set for all activation encodings.
dtype: Indicates if computation occurred in floating point or integer precision.
is_symmetric: If set to true, it indicates that parameter encodings were computed symmetrically.
param_bitwidth: Indicates the bit-width set for all parameter encodings.
per_channel_quantization: If set to True, then quantization encodings were computed for each channel axis of the tensor.
quant_scheme: Indicates the quantization algorithm used, which may be one of post_training_tf or post_training_tf_enhanced.

The intended usage of quantizer_args is to provide debugging information for customers who may need to perform post-quantization tasks, which could benefit from knowledge of how the encoding information was obtained.

3. Version 1.0.0¶

Note

Encoding version 1.0.0 is only supported for aimet_torch and aimet_onnx.

Note

The default encoding version is 0.6.1. To export with 1.0.0, add from aimet_common import quantsim and quantsim.encoding_version = '1.0.0' prior to sim.export().

Changes from 0.6.1:

Activation and parameter encodings are now no longer dictionaries mapping tensor names to encoding dictionaries, but instead are lists of encoding dictionaries where the tensor names are another entry in the encoding dictionary.
Fields present in the encoding dictionary have been reworked or removed for conciseness. Refer to the table below for details on which fields are present for each encoding type.
Notably, per channel encodings are now contained in a single encoding dictionary instead of a list of encodings with length num_channels. Instead, scale and offset fields are now 1-D arrays of length num_channels.
Encodings for per-block quantization and Low Power Blockwise Quantization are now supported.

3.1. Encoding specification¶

Top level structure¶
Key	Value type	Description
version	string	Encoding file version
activation_encodings	list of Encoding dictionaries	Encodings for each activation tensor
param_encodings	list of Encoding dictionaries	Encodings for each param tensor
quantizer_args	dict	Arguments used to instantiate QuantizationSimModel (refer to Quantizer Args structure for details)
excluded_layers	list	List of excluded layers

The below table describes how the Encoding dictionary looks for different quantization types: Per Tensor, Per Channel, Per Block, and Low Power Blockwise Quantization (LPBQ). Certain keys will only be present for certain quantization types, as indicated in the table.

Encoding dictionary structure¶
Key	Value type	Description	Per Tensor	Per Channel	Per Block	LPBQ
name	string	Tensor name	X	X	X	X
enc_type	string	Encoding type (refer to EncodingType for valid strings)	X	X	X	X
dtype	string	Data type (refer to DataType for valid strings)	X	X	X	X
block_size	uint32	Block size			X (INT only)	X
bw	uint8	Encoding bw (>=4 and <= 32)	X	X	X	X
is_sym	bool	True if encoding is symmetric, False otherwise	X	X	X	X
scale	fp32[]	Flattened array of scales	X (INT only)	X (INT only)	X (INT only)	X
offset	int32[]	Flattened array of offsets	X (INT only)	X (INT only)	X (INT only)	X
compressed_bw	uint8	Compressed bw				X
per_block_int_scale	uint16[]	Flattened array of scales per block				X

Encoding type¶
Enum	Description
PER_TENSOR	Denotes Per Tensor quantization
PER_CHANNEL	Denotes Per Channel quantization
PER_BLOCK	Denotes Per Block quantization
LPBQ	Denotes LPBQ quantization

Data type¶
Enum	Description
INT	Denotes integer quantization
FLOAT	Denotes floating point quantization

Quantizer Args structure¶
Key	Value type	Description
activation_bitwidth	uint8	Indicates the bit-width set for all activation encodings
dtype	string	Indicates if computation occurred in floating point or integer precision
is_symmetric	bool	If set to true, it indicates that parameter encodings were computed symmetrically
param_bitwidth	uint8	Indicates the bit-width set for all parameter encodings
per_channel_quantization	bool	If set to True, then quantization encodings were computed for each channel axis of the tensor
quant_scheme	string	Indicates the quantization algorithm used, which may be one of post_training_tf or post_training_tf_enhanced

For Per Channel quantization, the channel axis is defined to be the output channel dimension. For Per Block quantization, the channel axis is the output channel dimension while the block axis is the input channel dimension.
For Per Tensor quantization, scales and offsets will be a 1-D array of length 1. For Per Channel quantization, the length will be the the number of output channels. For Per Block quantization, the length will be number of output channels x number of input channels / block size