Babylon.js World Space

UIBabylon renders uno UI into a texture and displays it on a plane inside a Babylon.js scene. The plane can move, rotate, and scale while its UI remains interactive.

This section continues the Babylon.js guide. As before, keep the shared WebGPU resources, engine, scene, and camera. The following examples inject the UI directly onto a plane in the scene instead of drawing it as a background or foreground overlay.

1. Create the UI plane

Replace the UI import and UI.create() call with UIBabylon. Pass it the scene as well as the shared resources. It returns the uno UI instance and a Babylon.js plane with its texture and material already connected:

import UIBabylon from '@uno.build/ui/UIBabylon'
import { StandardMaterial } from '@babylonjs/core/Materials/standardMaterial.js'
import { Color3 } from '@babylonjs/core/Maths/math.color.js'

const UI_SIZE = 400
const device_pixel_ratio = window.devicePixelRatio

const { ui, plane, material, texture } = await UIBabylon.create({
  scene,
  resources,
  device_pixel_ratio,
  texture_width: Math.round(UI_SIZE * device_pixel_ratio),
  texture_height: Math.round(UI_SIZE * device_pixel_ratio),
  world_width: 1,
  world_height: 1,
  createMaterial() {
    const material = new StandardMaterial('ui-material', scene)
    material.disableLighting = true
    material.emissiveColor = Color3.White()
    return material
  },
})

ui.setViewport(UI_SIZE, UI_SIZE)

• texture_width, texture_height: Sets the resolution of the UI texture in physical pixels.
• world_width, world_height: Sets the size of the plane in Babylon.js world units.
• ui.setViewport(width, height): Sets the space available for the UI layout in logical pixels.
• createMaterial: This callback lets you define the material applied to the plane. The default material is StandardMaterial; the example disables lighting and uses white emission so the UI does not need a scene light.

2. Set up the camera

import { FreeCamera } from '@babylonjs/core/Cameras/freeCamera.js'
import { Vector3 } from '@babylonjs/core/Maths/math.vector.js'

const camera = new FreeCamera('camera', new Vector3(0, 0, -3), scene)
camera.fov = Math.PI / 3
camera.minZ = 0.1
camera.maxZ = 100
camera.setTarget(Vector3.Zero())
ui.setCamera(camera)

With ui.setCamera(camera), it casts a ray onto the plane and converts the hit position into UI layout coordinates. The component's existing pointer, click, and focus handlers work as the plane rotates.

For pointer, click, and wheel events, the payload described in Event data also includes distance_to_camera when the ray hits the plane. It is the distance from the camera to the hit position in Babylon.js world units. The x and y fields remain coordinates in the UI's logical coordinate space.

3. Render the scene

By default, plane is a Babylon.js mesh created with MeshBuilder.CreatePlane() using world_width and world_height. It already belongs to the scene passed to UIBabylon.create(), so there is no separate step to add it.

Draw the UI texture before rendering the scene so the plane displays the updated layout:

scene.autoClear = true

engine.runRenderLoop(() => {
  plane.rotation.y = Math.sin(performance.now() / 2000) * 0.8

  ui.draw()
  scene.render()
})

Here, ui.draw() updates the plane's texture rather than the canvas. Babylon.js can clear the canvas normally before drawing the scene.

Customize the plane and hit mapping

Use the optional createPlane() callback to display the UI on a different Babylon.js mesh, such as a box, sphere, or cylinder. It receives the configured UI material and the requested world_width and world_height. Return the mesh as plane; additional resources such as geometry and body_material are also returned by UIBabylon.create().

For custom interaction, return an optional mapIntersection() callback alongside the mesh. It receives the closest Babylon.js PickingInfo for the mesh and returns UV coordinates as { x, y }, or null to reject the hit. uno converts these UVs into UI layout coordinates. Without this callback, uno uses intersection.getTextureCoordinates() directly.

For example, add this createPlane option to UIBabylon.create() to display the UI on only the front (-Z) face of a box and make only that face interactive:

import { MultiMaterial } from '@babylonjs/core/Materials/multiMaterial.js'
import { MeshBuilder } from '@babylonjs/core/Meshes/meshBuilder.js'
import { SubMesh } from '@babylonjs/core/Meshes/subMesh.js'

const { ui, plane } = await UIBabylon.create({
  //...
  createPlane({ material, world_width, world_height }) {
    const plane = MeshBuilder.CreateBox(
      'ui-box',
      {
        width: world_width,
        height: world_height,
        depth: world_width,
      },
      scene,
    )
    const body_material = new StandardMaterial('body-material', scene)
    body_material.disableLighting = true
    body_material.emissiveColor = Color3.FromHexString('#F5F2EC')

    const box_material = new MultiMaterial('box-material', scene)
    box_material.subMaterials = [body_material, material, body_material, body_material, body_material, body_material]
    plane.material = box_material
    plane.releaseSubMeshes()
    for (let face = 0; face < 6; face++) {
      new SubMesh(face, face * 4, 4, face * 6, 6, plane)
    }

    return {
      plane,
      geometry: plane.geometry,
      body_material,
      box_material,
      mapIntersection(intersection) {
        return intersection.subMeshId === 1 ? intersection.getTextureCoordinates() : null
      },
    }
  },
})

Each face gets its own submesh and material slot. Index 1 selects the -Z face, which faces the camera in this example. Hits on the other faces return null; uno does not continue looking for an interactive face behind them.

mapIntersection() can also transform the UVs before returning them, allowing you to adapt interaction to a custom texture layout. It changes how hits are mapped to the UI; it does not change how the texture is rendered.

Cleanup

World-space UIs have resources owned by uno and resources managed by the rendering engine. ui.destroy() releases the uno nodes, listeners, renderer allocations, and offscreen GPU texture. It does not remove the plane from the scene or dispose the engine objects returned by create().

Before cleanup, stop this UI's draw callbacks and remove any event listeners added by your application. If a framework manages the layout, call its root's unmount() while the UI is still alive.

Keep the material and texture returned by UIBabylon.create(), as shown above. Dispose the mesh, then the material and texture:

plane.dispose()
material.dispose()
texture.dispose()
ui.destroy()

Disposing the default mesh removes it from the scene and releases its geometry when no other mesh uses it. The separate calls release the material and texture.

uno releases the backing GPU texture in ui.destroy(); do not call gpu_texture.destroy() separately.

This example assumes the default plane resources and a material used only by this UI. If createPlane() or createMaterial() creates additional resources, release those too. Keep shared materials and geometries alive until their last user is removed.

After the last overlay or world-space UI using the shared image and font atlases has been destroyed, dispose the resource store:

resources.dispose()

This does not destroy the GPU device or canvas context. Dispose the scene, renderer, application, or engine separately when shutting down the application; keep them alive when removing only one UI panel.

Demo

This demo displays one rotating UI plane using the component from the React demo.