trtutils.parallel package

Subpackages

• trtutils.parallel.image package
  • Module contents
    • Classes
    • EngineInfo
      • EngineInfo.engine_path
      • EngineInfo.detector_class
      • EngineInfo.dla_core
      • EngineInfo.input_range
      • EngineInfo.preprocessor
      • EngineInfo.resize_method
      • EngineInfo.conf_thres
      • EngineInfo.nms_iou_thres
      • EngineInfo.mean
      • EngineInfo.std
      • EngineInfo.input_schema
      • EngineInfo.output_schema
      • EngineInfo.backend
      • EngineInfo.warmup
      • EngineInfo.pagelocked_mem
      • EngineInfo.unified_mem
      • EngineInfo.cuda_graph
      • EngineInfo.extra_nms
      • EngineInfo.agnostic_nms
    • ParallelDetector
      • ParallelDetector.models
      • ParallelDetector.get_model()
      • ParallelDetector.get_model_profiling()
      • ParallelDetector.get_profiling()
      • ParallelDetector.stop()
      • ParallelDetector.preprocess()
      • ParallelDetector.preprocess_model()
      • ParallelDetector.postprocess()
      • ParallelDetector.postprocess_model()
      • ParallelDetector.get_detections()
      • ParallelDetector.get_detections_model()
      • ParallelDetector.submit()
      • ParallelDetector.submit_model()
      • ParallelDetector.get_random_input()
      • ParallelDetector.mock_submit()
      • ParallelDetector.retrieve()
      • ParallelDetector.retrieve_model()
      • ParallelDetector.end2end()

Module contents

Parallel implementations of TensorRT engines and models.

Submodules

image

Parallel implementations of image models.

Classes

QueuedTRTEngine

A class for running a TRTEngine in a separate thread asynchronously.

ParallelTRTEngines

A class for running many TRTEngines in parallel.

class trtutils.parallel.ParallelTRTEngines(engines: Sequence[TRTEngine | Path | str | tuple[TRTEngine | Path | str, int] | tuple[TRTEngine | Path | str, int | None, int | None]], warmup_iterations: int = 5, *, warmup: bool | None = None, cuda_graph: bool | None = None)

Bases: object

Handle many TRTEngines in parallel.

get_random_input(*, new: bool | None = None) → list[list[np.ndarray]]

Get a random input to the underlying TRTEngines.

Parameters:

new (bool, optional) – Whether or not to get a new input or the cached already generated one. By default, None/False

Returns:

The random inputs.

Return type:

list[list[np.ndarray]]

stop() → None

Stop the underlying engine threads.

submit(inputs: list[list[np.ndarray]]) → None

Submit data to be processed by the engines.

Parameters:

inputs (list[list[np.ndarray]]) – The inputs to pass to the engines. Should be a list of the same lenght of engines created.

Raises:

ValueError – If the inputs are not the same size as the engines.

mock_submit() → None

Send random data to the engines.

retrieve(timeout: float | None = None) → list[list[np.ndarray] | None]

Get the outputs from the engines.

Parameters:

timeout (float, optional) – Timeout for waiting for data.

Returns:

The output from the engines.

Return type:

list[np.ndarray]

class trtutils.parallel.QueuedTRTEngine(engine: TRTEngine | Path | str, warmup_iterations: int = 5, dla_core: int | None = None, device: int | None = None, *, warmup: bool | None = None, cuda_graph: bool | None = None)

Bases: object

Interact with TRTEngine over Thread and Queue.

property input_spec: list[tuple[list[int], np.dtype]]

Get the specs for the input tensor of the network. Useful to prepare memory allocations.

Returns:

A list with two items per element, the shape and (numpy) datatype of each input tensor.

Return type:

list[tuple[list[int], np.dtype]]

property input_shapes: list[tuple[int, ...]]

Get the shapes for the input tensors of the network.

Returns:

A list with the shape of each input tensor.

Return type:

list[tuple[int, …]]

property input_dtypes: list[np.dtype]

Get the datatypes for the input tensors of the network.

Returns:

A list with the datatype of each input tensor.

Return type:

list[np.dtype]

property output_spec: list[tuple[list[int], np.dtype]]

Get the specs for the output tensor of the network. Useful to prepare memory allocations.

Returns:

A list with two items per element, the shape and (numpy) datatype of each output tensor.

Return type:

list[tuple[list[int], np.dtype]]

property output_shapes: list[tuple[int, ...]]

Get the shapes for the output tensors of the network.

Returns:

A list with the shape of each output tensor.

Return type:

list[tuple[int, …]]

property output_dtypes: list[np.dtype]

Get the datatypes for the output tensors of the network.

Returns:

A list with the datatype of each output tensor.

Return type:

list[np.dtype]

get_random_input(*, new: bool | None = None) → list[np.ndarray]

Get a random input to the underlying TRTEngine.

Parameters:

new (bool, optional) – Whether or not to get a new input or the cached already generated one. By default, None/False

Returns:

The random input.

Return type:

list[np.ndarray]

stop() → None

Stop the thread containing the TRTEngine.

submit(data: list[np.ndarray]) → None

Put data in the input queue.

Parameters:

data (list[np.ndarray]) – The data to have the engine run.

mock_submit() → None

Send a random input to the engine.

retrieve(timeout: float | None = None) → list[np.ndarray] | None

Get an output from the engine thread.

Parameters:

timeout (float, optional) – Timeout for waiting for data.

Returns:

The output from the engine.

Return type:

list[np.ndarray]