class trtutils.parallel.image.ParallelDetector(engines: Sequence[EngineInfo], warmup_iterations: int = 10, input_range: tuple[float, float] = (0.0, 1.0), preprocessor: str = 'trt', resize_method: str = 'letterbox', conf_thres: float = 0.1, nms_iou_thres: float = 0.5, mean: tuple[float, float, float] | None = None, std: tuple[float, float, float] | None = None, backend: str = 'auto', *, warmup: bool | None = None, pagelocked_mem: bool | None = None, unified_mem: bool | None = None, cuda_graph: bool | None = None, extra_nms: bool | None = None, agnostic_nms: bool | None = None, sequential_load: bool | None = None, no_warn: bool | None = None, verbose: bool | None = None)[source]¶

Bases: object

A parallel implementation of Detector.

Allows multiple version of Detector to be allocated and executed at the same time. Primarily useful for multi-gpu/multi-accelerator systems such as the NVIDIA Jetson series. Since the TensorRT engines are compiled for a specific device, no device specification is needed inside of this class.

property models: list[Detector]¶

Get the underlying Detector models.

Returns:: A list of the underlying models.
Return type:: list[Detector]
Raises:: RuntimeError – If any values are None, not initialized yet.

get_model(modelid: int) → Detector[source]¶

Get a Detector model with id.

Parameters:: modelid (int) – The model ID to get. Based on original list passed during init.
Returns:: The Detector model
Return type:: Detector
Raises:: RuntimeError – If access is attempted before init is complete

get_model_profiling(modelid: int) → tuple[float, float, float][source]¶

Get the latency of a specific model as profiled in thread.

Parameters:: modelid (int) – The model ID of the profiling to get.
Returns:: The start time, end time, and delta
Return type:: tuple[float, float, float]

get_profiling() → list[tuple[float, float, float]][source]¶

Get all the profiling results for all models.

Returns:: The profiling data
Return type:: list[tuple[float, float, float]]

stop() → None[source]¶: Stop the threads.

preprocess(inputs: list[list[np.ndarray]], resize: str = 'letterbox', method: str | None = None, *, no_copy: bool | None = None, verbose: bool | None = None) → tuple[list[np.ndarray], list[list[tuple[float, float]]], list[list[tuple[float, float]]]][source]¶

Preprocess inputs for inference.

Parameters:

inputs (list[list[np.ndarray]]) – The inputs to preprocess, one batch per model.
resize (str) – The method to resize the image with. By default letterbox, options are [letterbox, linear]
method (str, optional) – The underlying preprocessor to use. Options are ‘cpu’ and ‘cuda’. By default None, which will use the preprocessor stated in the constructor.
no_copy (bool, optional) – If True and using CUDA, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The preprocessed tensors, ratios per image per model, and padding per image per model.

Return type:

tuple[list[np.ndarray], list[list[tuple[float, float]]], list[list[tuple[float, float]]]]

Raises:

ValueError – If inputs do not match the number of models

preprocess_model(images: list[np.ndarray], modelid: int, resize: str = 'letterbox', method: str | None = None, *, no_copy: bool | None = None, verbose: bool | None = None) → tuple[np.ndarray, list[tuple[float, float]], list[tuple[float, float]]][source]¶

Preprocess a batch of images for a specific model.

Parameters:

images (list[np.ndarray]) – The batch of images to preprocess.
modelid (int) – The model to preprocess the data for.
resize (str) – The method to resize the image with. By default letterbox, options are [letterbox, linear]
method (str, optional) – The underlying preprocessor to use. Options are ‘cpu’ and ‘cuda’. By default None, which will use the preprocessor stated in the constructor.
no_copy (bool, optional) – If True and using CUDA, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The preprocessed tensor, ratios per image, and padding per image.

Return type:

tuple[np.ndarray, list[tuple[float, float]], list[tuple[float, float]]]

postprocess(outputs: list[list[np.ndarray]], ratios: list[list[tuple[float, float]]], paddings: list[list[tuple[float, float]]], *, no_copy: bool | None = None, verbose: bool | None = None) → list[list[list[np.ndarray]]][source]¶

Postprocess outputs for inference.

Parameters:

outputs (list[list[np.ndarray]]) – The raw outputs per model.
ratios (list[list[tuple[float, float]]]) – The ratios per image per model.
paddings (list[list[tuple[float, float]]]) – The paddings per image per model.
no_copy (bool, optional) – If True, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The postprocessed outputs per image per model.

Return type:

list[list[list[np.ndarray]]]

Raises:

ValueError – If outputs do not match the number of models

postprocess_model(outputs: list[np.ndarray], modelid: int, ratios: list[tuple[float, float]], padding: list[tuple[float, float]], *, no_copy: bool | None = None, verbose: bool | None = None) → list[list[np.ndarray]][source]¶

Postprocess outputs for a specific model.

Parameters:

outputs (list[np.ndarray]) – The raw outputs to postprocess.
modelid (int) – The model to postprocess the data for.
ratios (list[tuple[float, float]]) – The ratios per image from preprocessing.
padding (list[tuple[float, float]]) – The padding per image from preprocessing.
no_copy (bool, optional) – If True, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The postprocessed outputs per image.

Return type:

list[list[np.ndarray]]

get_detections(outputs: list[list[list[np.ndarray]]], *, verbose: bool | None = None) → list[list[list[tuple[tuple[int, int, int, int], float, int]]]][source]¶

Get the detections of the YOLO models.

Parameters:

outputs (list[list[list[np.ndarray]]]) – The postprocessed outputs per image per model.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The detections per image per model.

Return type:

list[list[list[tuple[tuple[int, int, int, int], float, int]]]]

get_detections_model(outputs: list[list[np.ndarray]], modelid: int, *, verbose: bool | None = None) → list[list[tuple[tuple[int, int, int, int], float, int]]][source]¶

Get the detections for a batch from a single model.

Parameters:

outputs (list[list[np.ndarray]]) – The postprocessed outputs per image.
modelid (int) – The model ID of which model is forming detections.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The detections per image.

Return type:

list[list[tuple[tuple[int, int, int, int], float, int]]]

submit(inputs: list[list[np.ndarray]], ratios: list[list[tuple[float, float]]] | None = None, paddings: list[list[tuple[float, float]]] | None = None, preprocess_method: str | None = None, *, preprocessed: bool | None = None, postprocess: bool | None = None, no_copy: bool | None = None, verbose: bool | None = None) → None[source]¶

Submit batches to all models.

Parameters:

inputs (list[list[np.ndarray]]) – The batches to pass to each model.
ratios (list[list[tuple[float, float]]], optional) – The ratios per image per model.
paddings (list[list[tuple[float, float]]], optional) – The padding per image per model.
preprocess_method (str, optional) – The method to use for preprocessing. Options are ‘cpu’, ‘cuda’, ‘trt’. By default None, which will use the preprocessor stated in the constructor.
preprocessed (bool, optional) – Whether or not the inputs are preprocessed
postprocess (bool, optional) – Whether or not to postprocess the outputs right away
no_copy (bool, optional) – If True, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Raises:

ValueError – If the input length does not match the models If preprocessed is True, but ratios/paddings not provided

submit_model(images: list[np.ndarray], modelid: int, ratios: list[tuple[float, float]] | None = None, padding: list[tuple[float, float]] | None = None, preprocess_method: str | None = None, *, preprocessed: bool | None = None, postprocess: bool | None = None, no_copy: bool | None = None, verbose: bool | None = None) → None[source]¶

Submit a batch to a specific model.

Parameters:

images (list[np.ndarray]) – The batch of images to send to the model.
modelid (int) – The specific model index to send the data to.
ratios (list[tuple[float, float]], optional) – The ratios per image from preprocessing.
padding (list[tuple[float, float]], optional) – The padding per image from preprocessing.
preprocess_method (str, optional) – The method to use for preprocessing. Options are ‘cpu’, ‘cuda’, ‘trt’. By default None, which will use the preprocessor stated in the constructor.
preprocessed (bool, optional) – Whether or not the inputs are preprocessed.
postprocess (bool, optional) – Whether or not to perform postprocessing.
no_copy (bool, optional) – If True, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

get_random_input() → list[list[np.ndarray]][source]¶

Get random inputs (one per model).

Returns:: The random inputs per model.
Return type:: list[list[np.ndarray]]

mock_submit(data: list[list[ndarray]] | list[ndarray] | None = None, modelid: int | None = None) → None[source]¶

Perform a mock submit for all models or a specific model.

Parameters:

data (list[list[np.ndarray]], list[np.ndarray], optional) – The inputs to use for the inference. If modelid is specified, should be list[np.ndarray] (single batch). Otherwise should be list[list[np.ndarray]] (batch per model).
modelid (int, optional) – The specific engine to perform a mock submit for.

Raises:

ValueError – If specified modelid, but gave list of batches If gave single batch, but did not specify modelid

retrieve(*, verbose: bool | None = None) → tuple[list[list[list[np.ndarray]] | list[np.ndarray]], list[list[tuple[float, float]] | None], list[list[tuple[float, float]] | None]][source]¶

Get outputs back from all the models.

Parameters:

verbose (bool, optional) – Whether or not to log additional information.

Returns:

tuple[ – list[list[list[np.ndarray]] | list[np.ndarray]], list[list[tuple[float, float]] | None], list[list[tuple[float, float]] | None],
] – The outputs per image per model, ratios per image per model, padding per image per model.

retrieve_model(modelid: int, *, verbose: bool | None = None) → tuple[list[list[np.ndarray]] | list[np.ndarray], list[tuple[float, float]] | None, list[tuple[float, float]] | None][source]¶

Get the outputs from a specific model.

Parameters:

modelid (int) – The model to retrieve data from.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The outputs per image, ratios per image, and padding per image.

Return type:

tuple[list[list[np.ndarray]] | list[np.ndarray], list[tuple[float, float]] | None, list[tuple[float, float]] | None]

end2end(inputs: list[list[np.ndarray]], ratios: list[list[tuple[float, float]]] | None = None, paddings: list[list[tuple[float, float]]] | None = None, *, preprocessed: bool | None = None, postprocess: bool | None = None, no_copy: bool | None = None, verbose: bool | None = None) → list[list[list[tuple[tuple[int, int, int, int], float, int]]]][source]¶

Perform end-to-end inference for all models.

Parameters:

inputs (list[list[np.ndarray]]) – The batches to pass to each model.
ratios (list[list[tuple[float, float]]], optional) – The ratios per image per model.
paddings (list[list[tuple[float, float]]], optional) – The padding per image per model.
preprocessed (bool, optional) – Whether or not the inputs are preprocessed
postprocess (bool, optional) – Whether or not to postprocess the outputs right away
no_copy (bool, optional) – If True, do not copy the data from the allocated memory. If the data is not copied, it WILL BE OVERWRITTEN INPLACE once new data is generated.
verbose (bool, optional) – Whether or not to log additional information.

Returns:

The detections per image per model.

Return type:

list[list[list[tuple[tuple[int, int, int, int], float, int]]]]

Raises:

ValueError – If postprocess is False when calling end2end.
RuntimeError – If postprocessed outputs are not available for end2end.