--aic-batch-json-input JSON Format

The --aic-batch-json-input option accepts a JSON file that specifies one or more inference requests, each with its own set of input and output buffers.

JSON Structure

The top-level object must contain an "IO-files" key whose value is an array of IO sets. Each IO set is an array of IO entry objects, and each IO set corresponds to one inference. Two IO sets may have different numbers of entries and different field values.

{
  "IO-files": [
    [ <io-entry>, <io-entry>, ... ],
    [ <io-entry>, <io-entry>, ... ]
  ]
}

IO Entry Fields

Required Fields

Field

Description

path

Path to the buffer file. Accepts an absolute path or a path relative to the location of the JSON input file.

Input files are used by the runtime for inference. Output files are used only for validation with -c; the output file name generated by --write-output-dir is determined by the runtime API and is independent of this field.

data-type or elem-size

Specifies the element data type. At least one of these fields must be present for every IO entry.

data-type accepts a string (see Supported Data Types).

elem-size accepts the element size in bytes (e.g. 4 for float).

The two fields are interchangeable.

io-direction

Direction of the buffer. Accepted values: "in" (input) or "out" (output).

map-to

Name of the network buffer (bufferName) this IO entry maps to. Accepts a string matching the buffer name in the network descriptor.

Optional Fields

Field

Description

dims

Shape of the IO buffer as an array of integers. Accepts between 1 and 8 dimensions (inclusive). Use [0] for empty buffers (not []).

Although optional, supplying ``dims`` is highly recommended for specialized buffers and partial output buffers.

When dims is supplied it is validated against the network descriptor according to the buffer type (see Buffer Types). When omitted, the dimension defaults to the value in the network descriptor.

skip-validation

Boolean. Valid only for output entries. When true, this output buffer is excluded from the -c output validation check.

Supported Data Types

The data-type field accepts the following string values:

Value

Size (bytes)

"int8_t"

1

"uint8_t"

1

"int16_t"

2

"uint16_t"

2

"float16"

2

"int"

4

"uint"

4

"float"

4

"int64_t"

8

"uint64_t"

8

Buffer Types

The network descriptor controls how dims is validated for each buffer:

Regular buffer (is_partial_allowed = false)

The supplied dims must exactly match the dimensions in the network descriptor.

Partial buffer without specialization (is_partial_allowed = true, allowed_shapes is empty)

The cumulative product of the supplied dims must be less than or equal to the cumulative product of the dimensions in the network descriptor.

Partial buffer with specialization (is_partial_allowed = true, allowed_shapes is non-empty)

The supplied dims must exactly match one of the dimension combinations listed in allowed_shapes in the network descriptor.

Examples

Regular Network — One Inference

Network IO layout:

Input[0]:  bufferName=in0, data-type=int
Input[1]:  bufferName=in1, data-type=float
Output[0]: bufferName=out, data-type=int

{"IO-files": [
  [
    { "path":"output.raw",  "elem-size": 4,          "io-direction":"out", "map-to":"out" },
    { "path":"input0.raw",  "data-type": "float",    "io-direction":"in",  "map-to":"in0" },
    { "path":"input1.raw",  "data-type": "int",      "io-direction":"in",  "map-to":"in1" }
  ]
]}

Regular Network — Two Inferences

IO entry order within each IO set does not need to be consistent across sets.

{"IO-files": [
  [
    { "path":"output.raw",  "elem-size": 4,       "io-direction":"out", "map-to":"out" },
    { "path":"input0.raw",  "data-type": "float", "io-direction":"in",  "map-to":"in0" },
    { "path":"input1.raw",  "data-type": "int",   "io-direction":"in",  "map-to":"in1" }
  ],
  [
    { "path":"input1.raw",  "data-type": "int",   "io-direction":"in",  "map-to":"in1" },
    { "path":"output.raw",  "elem-size": 4,       "io-direction":"out", "map-to":"out" },
    { "path":"input0.raw",  "data-type": "float", "io-direction":"in",  "map-to":"in0" }
  ]
]}

Non-Specialized Network with Partial Buffers

For partial output buffers, dims is highly recommended. The accepted value must be within [0, max_dims] as specified in the network descriptor.

Network IO layout:

Input[0]:  bufferName=in0, is_partial_allowed=true,  data-type=float, dims=[32]
Input[1]:  bufferName=in1, is_partial_allowed=false, data-type=float, dims=[128]
Output[0]: bufferName=out0, is_partial_allowed=true, data-type=float, dims=[64]
Output[1]: bufferName=out1, is_partial_allowed=false, data-type=float, dims=[32]

{"IO-files": [
  [
    { "path":"output1.raw", "data-type": "float",          "io-direction":"out", "map-to":"out1" },
    { "path":"input0.raw",  "data-type": "float",          "io-direction":"in",  "map-to":"in0"  },
    { "path":"input1.raw",  "data-type": "float",          "io-direction":"in",  "map-to":"in1"  },
    { "path":"output0.raw", "dims":[60], "data-type": "float", "io-direction":"out", "map-to":"out0" }
  ]
]}

Note

dims is recommended for out0 because it is a partial output buffer. Any value from [0] to [64] is accepted by the runtime.

Specialized Network

For specialized networks, dims is highly recommended for both input and output buffers. Each IO set’s dimensions must match one of the combinations defined by allowed_shapes in the network descriptor. Different IO sets may use different allowed_shapes combinations.

Network IO layout:

Input[0]:  bufferName=in0, is_partial_allowed=true,  data-type=float
Input[1]:  bufferName=in1, is_partial_allowed=false, data-type=float
Output[0]: bufferName=out0, is_partial_allowed=true, data-type=float
Output[1]: bufferName=out1, is_partial_allowed=false, data-type=float

Allowed_shapes[0]: in0=[1,32],  in1=[1,128], out0=[1,64], out1=[1,32]
Allowed_shapes[1]: in0=[1,1],   in1=[1,2],   out0=[1,3],  out1=[1,4]

{"IO-files": [
  [
    { "path":"input1.raw",  "dims":[1,128], "data-type": "float", "io-direction":"in",  "map-to":"in1"  },
    { "path":"input0.raw",  "dims":[1,32],  "data-type": "float", "io-direction":"in",  "map-to":"in0"  },
    { "path":"output1.raw", "dims":[1,32],  "data-type": "float", "io-direction":"out", "map-to":"out1" },
    { "path":"output0.raw", "dims":[1,64],  "data-type": "float", "io-direction":"out", "map-to":"out0" }
  ],
  [
    { "path":"input1.raw",  "dims":[1,2],   "data-type": "float", "io-direction":"in",  "map-to":"in1"  },
    { "path":"input0.raw",  "dims":[1,1],   "data-type": "float", "io-direction":"in",  "map-to":"in0"  },
    { "path":"output1.raw", "dims":[1,4],   "data-type": "float", "io-direction":"out", "map-to":"out1" },
    { "path":"output0.raw", "dims":[1,3],   "data-type": "float", "io-direction":"out", "map-to":"out0" }
  ]
]}

Selective IO

A buffer can be omitted from an IO set (dropped) if all of the following conditions are met in the network descriptor:

  • is_partial_allowed = true

  • skip_partial_header = true

  • The DMA buffer has allow_skip = true

Dropped buffers are not submitted for inference and are not written by --write-output-dir. Different IO sets may drop different buffers, so IO sets can have different numbers of entries.

Example — specialized network, ``in0`` and ``out0`` dropped:

{"IO-files": [
  [
    { "path":"input1.raw",  "dims":[1,128], "data-type": "float", "io-direction":"in",  "map-to":"in1"  },
    { "path":"output1.raw", "dims":[1,32],  "data-type": "float", "io-direction":"out", "map-to":"out1" }
  ]
]}

Example — non-specialized network with partial buffers, mixed dropping across IO sets:

{"IO-files": [
  [
    { "path":"input0.raw",  "data-type": "float",        "io-direction":"in",  "map-to":"in0"  },
    { "path":"input1.raw",  "data-type": "float",        "io-direction":"in",  "map-to":"in1"  },
    { "path":"output1.raw", "dims":[30], "data-type": "float", "io-direction":"out", "map-to":"out1" }
  ],
  [
    { "path":"input1.raw",  "data-type": "float",        "io-direction":"in",  "map-to":"in1"  },
    { "path":"output1.raw", "dims":[0],  "data-type": "float", "io-direction":"out", "map-to":"out1" }
  ]
]}

Option

Interaction with --aic-batch-json-input

-c, --check-output

Validates inference output against the output files specified in the JSON. Output entries with "skip-validation": true are excluded.

--time <seconds>

IO sets are submitted in round-robin order until the specified duration elapses.

-n, --num-iter <i>

IO sets are submitted in round-robin order until n inferences have been submitted.

-S, --set-size <i>

Controls the number of execution objects (execObjs) created by the runtime (default: 10). Setting -S 1 enforces the execution order of IO sets in the JSON file.

-T, --threads-per-queue <i>

Controls the number of processing queues (default: 4). Setting -T 1 enforces the execution order of IO sets in the JSON file.

Limitations

  • User buffer allocation on demand is only supported with --aic-batch-json-input. Other input options may result in high memory usage because user buffers are not allocated on demand.