sensor

Submodules

sapien.sensor.activelight module

sapien.sensor.depth_processor module

sapien.sensor.depth_processor.calc_depth_and_pointcloud(disparity: ndarray, mask: ndarray, q: ndarray, no_pointcloud: bool = False) → Tuple[ndarray, object]

Calculate depth and pointcloud.

Parameters:

• disparity – Disparity

• mask – Valid mask

• q – Perspective transformation matrix

• no_pointcloud

Return depth:

Depth

Return pointcloud:

Pointcloud

sapien.sensor.depth_processor.calc_disparity(imgl: ndarray, imgr: ndarray, method: str, *, ndisp: int = 128, min_disp: int = 0, use_census: bool = True, census_wsize: int = 7) → ndarray

Calculate disparity given a rectified image pair.

Parameters:

• imgl – Left image

• imgr – Right image

• method – SGBM or BM

• ndisp – max disparity

• min_disp – min disparity

Returns:

disparity

sapien.sensor.depth_processor.calc_main_depth_from_left_right_ir(ir_l: ndarray, ir_r: ndarray, l2r: ndarray, l2rgb: ndarray, k_l: ndarray, k_r: ndarray, k_main: ndarray, map1: ndarray, map2: ndarray, q: ndarray, method: str = 'SGBM', ndisp: int = 128, use_noise: bool = False, use_census: bool = True, register_depth: bool = True, register_blur_ksize: int = 5, main_cam_size=(1920, 1080), census_wsize=7, **kwargs) → ndarray

Calculate depth for rgb camera from left right ir images.

Parameters:

• ir_l – left ir image

• ir_r – right ir image

• l2r – Change-of-coordinate matrix from left camera’s frame to right camera’s frame (in OpenCV coordinate system)

• l2rgb – Change-of-coordinate matrix from left camera’s frame to RGB camera’s frame (in OpenCV coordinate system)

• k_l – left intrinsic matrix

• k_r – right intrinsic matrix

• k_main – rgb intrinsic matrix

• map1 – Left map for rectification

• map2 – Right map for rectification

• q – Perspective transformation matrix (for cv2.reprojectImageTo3D)

• method – method for depth calculation (SGBM or BM)

• use_noise – whether to simulate ir noise before processing

Return depth:

calculated depth

sapien.sensor.depth_processor.calc_rectified_stereo_pair(imgl: ndarray, imgr: ndarray, map1: ndarray, map2: ndarray) → Tuple[ndarray, ndarray]

Rectify an image pair with given camera parameters.

Parameters:

• imgl – Left image

• imgr – Right image

• map1 – Map for left camera

• map2 – Map for right camera

Return imgl_rect:

Rectified left image

Return imgr_rect:

Rectified right image

sapien.sensor.depth_processor.depth_post_processing(depth: ndarray, ksize: int = 5) → ndarray

sapien.sensor.depth_processor.get_census(img: ndarray, wsize: int = 7) → ndarray

sapien.sensor.depth_processor.init_rectify_stereo(w: int, h: int, kl: ndarray, kr: ndarray, rt: ndarray, distortl: ndarray | None = None, distortr: ndarray | None = None) → Tuple[ndarray, ndarray, ndarray]

Initiate parameters needed for rectification with given camera parameters.

Parameters:

• w – width of the image

• h – height of the image

• kl – Left intrinsic matrix

• kr – Right intrinsic matrix

• rt – Extrinsic matrix (left to right)

• distortl – Left distortion coefficients

• distortr – Right distortion coefficients

Return map1:

Map for left camera

Return map2:

Map for right camera

Return q:

Perspective transformation matrix (for cv2.reprojectImageTo3D)

sapien.sensor.depth_processor.pad_lr(img: ndarray, ndisp: int) → ndarray

sapien.sensor.depth_processor.sim_ir_noise(img: ndarray, scale: float = 0.0, blur_ksize: int = 0, blur_ksigma: float = 0.03, speckle_shape: float = 398.12, speckle_scale: float = 0.00254, gaussian_mu: float = -0.231, gaussian_sigma: float = 0.83, seed: int = 0) → ndarray

Simulate IR camera noise.

Noise model from Landau et al. Simulating Kinect Infrared and Depth Images

Parameters:

• img – Input IR image

• scale – Scale for downsampling & applying gaussian blur

• blur_ksize – Kernel size for gaussian blur

• blur_ksigma – Kernel sigma for gaussian blur

• speckle_shape – Shape parameter for speckle noise (Gamma distribution)

• speckle_scale – Scale parameter for speckle noise (Gamma distribution)

• gaussian_mu – mu for additive gaussian noise

• gaussian_sigma – sigma for additive gaussian noise

• seed – random seed used for numpy

Returns:

Processed IR image

sapien.sensor.depth_processor.unpad_lr(img: ndarray, ndisp: int) → ndarray

sapien.sensor.sensor_base module

class sapien.sensor.sensor_base.SensorEntity

Bases: object

sapien.sensor.simsense_component module

class sapien.sensor.simsense_component.SimSenseComponent(rgb_resolution: tuple, ir_resolution: tuple, rgb_intrinsic: ndarray, ir_intrinsic: ndarray, trans_pose_l: Pose, trans_pose_r: Pose, min_depth: float, max_depth: float, ir_noise_seed: int, ir_speckle_noise: float, ir_thermal_noise: float, rectified: bool, census_width: int, census_height: int, max_disp: int, block_width: int, block_height: int, p1_penalty: int, p2_penalty: int, uniqueness_ratio: int, lr_max_diff: int, median_filter_size: int, depth_dilation: bool)

Bases: Component

compute(left: ndarray | CudaArray, right: ndarray | CudaArray, bbox_start: tuple | None = None, bbox_size: tuple | None = None) → None

get_cuda() → CudaArray

get_ndarray() → ndarray

get_point_cloud_cuda() → CudaArray

get_point_cloud_ndarray() → ndarray

get_rgb_point_cloud_cuda(rgba_cuda: CudaArray) → CudaArray

get_rgb_point_cloud_ndarray(rgba_cuda: CudaArray) → ndarray

on_add_to_scene(scene)

on_remove_from_scene(scene)

sapien.sensor.stereodepth module

class sapien.sensor.stereodepth.StereoDepthSensor(config: StereoDepthSensorConfig, mount_entity: Entity, pose: Pose | None = None)

Bases: object

This class simulates an active stereo depth sensor. It has one RGB camera and two infrared camera. Depth is computed via semi-global block matching. Refer to StereoDepthSensorConfig for configurable parameters. The computed depth map will be presented in RGB camera frame.

compute_depth(bbox_start: tuple | None = None, bbox_size: tuple | None = None)

get_config()

get_depth()

Note: Returned depth map will be of the same resolution and frame of RGB camera.

get_depth_cuda()

Note: Returned depth map will be of the same resolution and frame of RGB camera.

get_ir()

Note: Noise simulation won’t be reflected here.

get_pointcloud(with_rgb: bool = False)

Note: Returned point cloud is from RGB camera’s with x rightward, y downward, z forward.

get_pointcloud_cuda(with_rgb: bool = False)

Note: Returned point cloud is from RGB camera’s with x rightward, y downward, z forward.

get_pose()

get_rgb()

get_rgba_cuda()

set_census_window_size(census_width: int, census_height: int)

Parameters:

• census_width – Width of the center-symmetric census transform window. This must be an odd number.

• census_height – Height of the center-symmetric census transform window. This must be an odd number.

set_ir_noise(ir_speckle_noise: float, ir_thermal_noise: float)

Parameters:

• ir_speckle_noise – Scale for simulating infrared speckle noise. Set to 0 will disable noise simulation.

• ir_thermal_noise – Scale for simulating infrared thermal noise. Only effective when speckle noise is on.

set_local_pose(pose: Pose)

Set local pose of the sensor relative to mounted actor.

set_lr_max_diff(lr_max_diff: int)

Parameters:

lr_max_diff – Maximum allowed difference in the left-right consistency check. Set it to 255 to disable the check.

set_matching_block_size(block_width: int, block_height: int)

Parameters:

• block_width – Width of the matched block. This must be an odd number.

• block_height – Height of the matched block. This must be an odd number.

set_penalties(p1_penalty: int, p2_penalty: int)

Parameters:

• p1_penalty – P1 penalty for semi-global matching algorithm.

• p2_penalty – P2 penalty for semi-global matching algorithm.

set_uniqueness_ratio(uniqueness_ratio: int)

Parameters:

uniqueness_ratio – Margin in percentage by which the minimum computed cost should win the second best (not considering best match’s adjacent pixels) cost to consider the found match valid.

take_picture(infrared_only: bool = False)

Note: We expect one scene.update_render() call before calling take_picture().

Parameters:

infrared_only – If true, only take infrared pictures without taking RGB picture.

class sapien.sensor.stereodepth.StereoDepthSensorConfig

Bases: object

An instance of this class is required to initialize StereoDepthSensor.

block_height

Height of the matched block for stereo matching. This must be an odd number.

block_width

Width of the matched block for stereo matching. This must be an odd number.

census_height

Height of the center-symmetric census transform window for stereo matching. This must be an odd number.

census_width

Width of the center-symmetric census transform window for steoreo matching. This must be an odd number.

depth_dilation

Final depth will be transformed into the same size of rgb_resolution, and depth dilation can dilate the final depth map to avoid holes. Recommended to set as true if rgb_resolution is greater than ir_resolution.

ir_intrinsic

Intrinsic matrix of the infrared cameras.

ir_noise_seed

Random seed for simulating infrared noise.

ir_resolution

Resolution of the infrared cameras (width x height).

ir_speckle_noise

Scale for simulating infrared speckle noise. Set to 0 will disable noise simulation.

ir_thermal_noise

Scale for simulating infrared thermal noise. Only effective when speckle noise is on.

light_pattern

Path to active light pattern file. Use RGB modality if set to None.

lr_max_diff

Maximum allowed difference of the left-right consistency check in stereo matching. Set it to 255 will disable the check.

max_depth

Maximum valid depth (non-inclusive) in meters.

max_disp

Maximum disparity (non-inclusive) for stereo matching.

median_filter_size

Size of the median filter for disparity post-processing. Choices are 1, 3, 5, 7. Set to 1 will disable median filter.

min_depth

Minimum valid depth in meters.

p1_penalty

P1 penalty for semi-global matching algorithm.

p2_penalty

P2 penalty for semi-global matching algorithm.

rectified

Set to true if left and right infrared cameras have a common image plane, otherwise false.

rgb_intrinsic

Intrinsic matrix of the RGB camera.

rgb_resolution

Resolution of the RGB camera (width x height).

trans_pose_l

Relative pose of the left infrared camera to the RGB camera.

trans_pose_r

Relative pose of the right infrared camera to the RGB camera.

uniqueness_ratio

How much (in percentage) the minimum cost computed in stereo matching must exceed the second best cost (ignoring best match’s adjacent pixels) to be considered valid.

Module contents