Configure RViz with AI

Describe the view you want in plain language. The AI proposes display changes, AutoDiag checks every one of them against your live system, and nothing changes until you press Apply.

Example

Show the lidar point cloud colored by height in the base_link frame, keep it light-weight. My laser scan shows nothing.

The plan appears as a checklist:

• Add PointCloud2 display "Lidar Cloud" on /lidar/points
• Set Color Transformer of "Lidar Cloud" to AxisColor (needed for Axis)
• Set Axis of "Lidar Cloud" to Z
• Set Style of "Lidar Cloud" to Points
• Set Decay Time of "Lidar Cloud" to 0
• Set Topic/Depth of "Lidar Cloud" to 1
• Set Topic/Reliability Policy of "LaserScan" to Best Effort (AutoDiag QoS fix)
• Set Fixed Frame to base_link (from your request)

Untick anything you don't want, then press Apply checked. Each line gets ✔ applied, ✖ failed (with the reason), or ⏳ waiting for data.

What happens behind the scenes

sequenceDiagram
    autonumber
    actor You
    participant Panel as RViz panel
    participant Server as AutoDiag server
    participant AI as AI model
    participant ROS as ROS 2 graph
    You->>Panel: Describe the view
    Panel->>Server: Request + current displays, Fixed Frame, TF frames + conversation
    par Live context
        Server->>ROS: List topics and types
    and
        Server->>ROS: Inspect QoS of each display's topic
    end
    Server->>AI: Context + request (JSON schema enforced)
    AI-->>Server: Proposed actions
    Server->>Server: Validate against topics, types, frames, display catalog
    Server->>Server: Add missing dependencies and diagnosed QoS fixes
    Server-->>Panel: Checklist, rejected items with reasons, notes
    You->>Panel: Apply checked
    Panel->>Panel: Snapshot for Undo, then apply each action
    Panel-->>You: ✔ / ✖ / ⏳ per action

What the AI can change

Action	Example
Add a display	PointCloud2 on /points, Map on /map, TF
Remove a display	"remove the old laser scan"
Set the topic	"use /front/scan instead"
Set a property	Style, Size (m) / Size (Pixels), Color Transformer, Axis, Decay Time, Alpha, QoS (Topic/Reliability Policy, Topic/Depth…)
Enable or disable	"hide the map"
Rename	"call it Front Lidar"
Set the Fixed Frame	"view everything in base_link"

Supported display types: PointCloud2, PointCloud, LaserScan, Image, Camera, DepthCloud, Map, Path, Odometry, Pose, PoseWithCovariance, PoseArray, PointStamped, Polygon, Marker, MarkerArray, Range, GridCells, TwistStamped, Wrench, TF, Grid, Axes and RobotModel.

The safety net

Small models in particular make mistakes. AutoDiag doesn't trust the plan blindly:

Validated

Unknown display types, missing displays and topics that exist with the wrong message type are rejected with a reason. Display-name typos like laser_scan vs LaserScan are matched.

Completed

Dependent settings the model forgot are added (for example Color Transformer: AxisColor when it only set Axis). If your request mentions "frame" and names exactly one TF frame from the tree (other than the current Fixed Frame), that frame is applied even if the model skipped it.

Diagnosed

AutoDiag inspects the topics of up to six displays itself. If the publisher offers Best Effort but the display requests Reliable, or durability doesn't match (including a Volatile display on a latched Transient Local publisher), the QoS fix is added, regardless of what the model proposed. A Best Effort display on a latched topic gets Reliable: the latched sample only reaches Reliable readers. Displays the plan adds get the same check, so a new point cloud on a best-effort lidar topic is not left empty, and a QoS setting the model proposes that would receive nothing is corrected (a RobotModel's settings sit under its Description Topic). If the Fixed Frame does not exist in the TF tree (RViz then shows nothing at all, e.g. an old odom_combined config), the plan switches it to map, or odom when there is no map. QoS facts come from one complete discovery pass over the ROS graph.

Reversible

Every Apply takes a snapshot first. Undo restores all displays and the Fixed Frame from before the last Apply.

Before any data exists

You can set up RViz before your sensors run. Displays are added with the topic you name, or the conventional one (/scan, /points, /camera/image_raw, /map, /odom, …), and marked not publishing yet. Settings that only appear once data arrives are retried for up to 30 seconds.

Conversations

Follow-ups work: after "Add a point cloud for /points" you can say "make those points bigger and color them by height", and the existing display is changed instead of a new one added. The panel remembers what was proposed, applied, skipped, failed or undone. New conversation forgets it; history is never written to disk.

Choosing a model

Pick Provider and Model at the top of the tab. It's the same selection as the AI assistant tab.

Model	Planning time	Notes
Local qwen3:4b-instruct	about 1–2 min on a CPU	Offline; fine for simple requests
Claude, ChatGPT, Gemini	a few seconds	Better at complex, multi-part requests

What the model receives:

Sent	Never sent
Your request; earlier turns; display names, types, topics, properties and status; Fixed Frame; TF frame names; live topic names and types; diagnosed QoS facts	Snapshots, message contents, node names, hostnames, IP addresses, home paths, API keys

For a cloud model, this machine's hostname, IP addresses and home paths in that text are replaced with tokens first. See AI providers.

Limits

• RViz itself can't downsample point clouds. That needs a filter node such as pcl_ros VoxelGrid; the AI says so in its notes and offers the lighter-weight display settings instead.
• Display types from extra plugins (for example rviz_imu_plugin) aren't in the catalog yet.
• It changes the RViz view only, never the robot, nodes or ROS configuration.