Ray Tracing Renderer

In this tutorial, you will learn the following:

• Enable ray tracing in SAPIEN renderer

The full script can be downloaded from rt.py, rt_mat.py.

Ray tracing vs. rasterization

In the previous tutorials, we have learned how to set up a basic scene with SAPIEN and acquire rendering results under the default settings using SapienRenderer. By default, SapienRenderer uses a high-efficiency rasterization-based rendering pipeline, making it suitable for data-intensive tasks such as reinforcement learning.

However, though fast, the rasterization-based renderer is not physically-grounded, and cannot faithfully model many real-world effects, e.g., indirect lighting, realistic shadows, reflections and refractions, making the results overly flat and lack realism. On the other end, ray tracing renderer simulates how light rays interact with objects in a physically correct manner, and produces images that can be indistinguishable from those captured by a camera.

From A Shader-Based Ray Tracing Engine, Park et al.

Ray tracing with SAPIEN

In SAPIEN 3, the default renderer supports both rasterization and ray tracing, and different cameras can use different rendering pipelines. Choosing a pipeline is done through specifying a shader pack, which is a directory containing a collection of glsl files.

To use the ray-tracing pipeline, simply add the following lines before creating a camera or a viewer.

    sapien.render.set_camera_shader_dir("rt")
    sapien.render.set_viewer_shader_dir("rt")

That’s it! You can now rerun the script with raytracing renderer. The result would look like:

You may find that the result looks more realistic with the ray tracing shader. However the result contains noise due to under-sampling. To reduce the noise, one way is to increase the sample-per-pixel for the renderer. To achieve this, simply change the rt_samples_per_pixel in render_config.

    sapien.render.set_ray_tracing_samples_per_pixel(4)  # change to 256 for less noise

Increasing the spp will affect the rendering speed directly. A cheaper way to reduce the noise is using a denoiser. SAPIEN supports the OptiX denoiser built into NVIDIA’s GPU driver and Intel’s open source denoiser OpenImageDenoise

    sapien.render.set_ray_tracing_denoiser("none") # change to "optix" or "oidn"

Note

You are required to have an NVIDIA GPU with driver version >= 525 installed to use OptiX or OpenImageDenoise.

Reflection and refraction

Ray tracing allows SAPIEN to render realistic reflection and refractions.

We will create a scene in SAPIEN and render with ray tracing turned on and off. First, let’s setup the environment:

        sapien.render.set_viewer_shader_dir("rt")
        sapien.render.set_ray_tracing_samples_per_pixel(64)
        sapien.render.set_ray_tracing_denoiser("optix")

    scene = sapien.Scene()

    camera_mount = scene.create_actor_builder().build_kinematic()
    camera = scene.add_mounted_camera(
        name="camera",
        mount=camera_mount,
        pose=sapien.Pose(),  # relative to the mounted actor
        width=1280,
        height=720,
        fovy=np.deg2rad(45),
        near=0.1,
        far=100,
    )

    camera_mount.set_pose(
        Pose([-0.28, -0.28, 0.46], [0.8876263, -0.135299, 0.3266407, 0.2951603])
    )

    ground_material = sapien.render.RenderMaterial()
    ground_material.base_color = np.array([202, 164, 114, 256]) / 256
    ground_material.specular = 0.5
    scene.add_ground(0, render_material=ground_material)
    scene.set_timestep(1 / 240)

    scene.set_ambient_light([0.3, 0.3, 0.3])
    scene.add_directional_light(
        [0, 0.5, -1],
        color=[3.0, 3.0, 3.0],
        shadow=True,
        shadow_scale=2.0,
        shadow_map_size=4096,  # these are only needed for rasterization
    )

    builder = scene.create_actor_builder()

We add a flag ray_tracing to allow switching between rasterization and ray tracing. Next, let’s build the scene. First, we create a rough bluish sphere:

    material.base_color = [0.2, 0.2, 0.8, 1.0]
    material.roughness = 0.5
    material.metallic = 0.0
    builder.add_sphere_visual(radius=0.06, material=material)
    builder.add_sphere_collision(radius=0.06)
    sphere1 = builder.build()
    sphere1.set_pose(Pose(p=[-0.05, 0.05, 0.06]))

    builder = scene.create_actor_builder()

Next, we create a rough transparent sphere:

    material.ior = 1.2
    material.transmission = 1.0
    material.base_color = [1.0, 1.0, 1.0, 1.0]
    material.roughness = 0.15
    material.metallic = 0.0
    builder.add_sphere_visual(radius=0.07, material=material)
    builder.add_sphere_collision(radius=0.07)
    sphere2 = builder.build()
    sphere2.set_pose(Pose(p=[0.05, -0.05, 0.07]))

    builder = scene.create_actor_builder()

Generally, setting a large transmission value will lead to a transparent material. Similarly, we can add a capsule and a box with complex materials:

    material.base_color = [0.8, 0.7, 0.1, 1.0]
    material.roughness = 0.01
    material.metallic = 1.0
    builder.add_capsule_visual(radius=0.02, half_length=0.1, material=material)
    builder.add_capsule_collision(radius=0.02, half_length=0.1)
    cap = builder.build()
    cap.set_pose(
        Pose(p=[0.15, -0.01, 0.01], q=transforms3d.euler.euler2quat(0, 0, -0.7))
    )

    builder = scene.create_actor_builder()
    material = sapien.render.RenderMaterial()
    material.base_color = [0.8, 0.2, 0.2, 1.0]
    material.roughness = 0.005
    material.metallic = 1.0
    builder.add_box_visual(half_size=[0.09, 0.09, 0.09], material=material)
    builder.add_box_collision(half_size=[0.09, 0.09, 0.09])
    box = builder.build()
    box.set_pose(Pose(p=[0.05, 0.17, 0.09]))

    builder = scene.create_actor_builder()

Finally, let’s load an external mesh and assign a highly metallic material to that object:

    material.base_color = [0.9, 0.6, 0.5, 1.0]
    material.roughness = 0.0
    material.metallic = 1.0
    builder.add_visual_from_file(
        "../assets/objects/suzanne.dae", scale=[0.1, 0.1, 0.1], material=material
    )
    builder.add_box_collision(half_size=[0.1, 0.1, 0.1])
    box = builder.build()

After building the scene, we can get rendering results from the camera:

    camera.take_picture()

    rgb = camera.get_picture("Color")
    rgb = im.fromarray((rgb * 255).astype(np.uint8))
    rgb.save(f'mat_{"rt" if ray_tracing else "rast"}.png')
    rgb.show()

Result with default rasterizer

Result with ray tracer