Instrumentation

How the RobotOps Trace SDK turns your robot into traces — the auto-init model, the framework integrations, the optional annotations, and what each span carries.

This page is the depth behind the Quickstart. If you just want it running, start there.

The carrier model

TraceHouse uses a carrier model, like Datadog or OpenTelemetry:

1. The SDK runs inside your nodes. It produces spans deterministically from in-process events — callbacks, action calls, ticks, and your own annotations.
2. The agent (robot-agent) runs next to your nodes and is the carrier: it receives spans over a local socket and forwards them to TraceHouse. It also keeps collecting logs, metrics, TF, and MCAP independently.

No forks, no framework rebuilds. You keep stock upstream MoveIt / Nav2 / BehaviorTree.CPP / ros2_control and add our packages as normal dependencies.

Turn it on — env-default auto-init

Auto-init is the Datadog -javaagent model: set it once in the launch environment and every node auto-instruments with zero per-node code.

Language	Mechanism	Set
C++	LD_PRELOAD a constructor lib (librobotops_trace_autoinit.so) — runs RobotOps::init() on load	LD_PRELOAD=/opt/ros/<distro>/lib/librobotops_trace_autoinit.so
Python	An installed .pth / sitecustomize hook calls robotops.init() on interpreter start	ROBOTOPS_TRACE_AUTOINIT=1

The installer writes both into /etc/robotops/trace.env. Apply that env to your ROS launch environment — your node's systemd unit (EnvironmentFile=/etc/robotops/trace.env), docker run -e, a launch-file SetEnvironmentVariable, or source /etc/robotops/trace.env before ros2 launch — see the Quickstart deployment tabs.

:::caution Scope it to ROS, not the whole machine By default the installer does not add a global /etc/profile.d preload — that would LD_PRELOAD the tracer (and its dependencies) into every process on the box (ssh, ls, cron, system daemons), which is at odds with the lightweight goal. Keep auto-init in the environment of the nodes you want traced. The aggressive preload-everything option is opt-in only: re-run the installer with ROBOTOPS_GLOBAL_AUTOINIT=1. :::

Explicit init (the override path)

A node launched outside the auto-init environment still works — initialize the SDK explicitly at startup:

C++

#include <robotops/trace.hpp>

int main(int argc, char** argv) {
  RobotOps::init();          // explicit init — equivalent to the LD_PRELOAD path
  rclcpp::init(argc, argv);
  // ...
}

Python

import robotops

robotops.init()              # explicit init — equivalent to ROBOTOPS_TRACE_AUTOINIT=1
import rclpy
rclpy.init()

Integrations — pick what you run

The SDK core is framework-agnostic. The integrations are à-la-carte packages — install only the ones matching your stack.

Integration	apt package	PyPI	Produces
rclcpp	ros-<distro>-robotops-trace-rclcpp	—	Executor callback spans + rclcpp_action goal-UUID stitching + DDS content keys
rclpy	ros-<distro>-robotops-trace-rclpy	robotops-trace-rclpy	rclpy callback + action spans (monkey-patched)
BehaviorTree.CPP	ros-<distro>-robotops-trace-bt-cpp	—	BT tick callbacks (covers Nav2 + MoveIt Pro)
ros2_control	ros-<distro>-robotops-trace-ros2-control	—	Controller / FollowJointTrajectory boundary (RT-safe)
MoveIt ★	ros-<distro>-robotops-trace-moveit	—	TrajectoryExecutionManager internal-async capture/restore
semantic conventions	ros-<distro>-robotops-trace-semconv	(bundled)	The shared robot.* / ros.* attribute vocabulary

:::caution MoveIt ★ — the one exception to "no forks" The MoveIt internal-async coverage needs a hook that isn't upstream yet. If you need it before it lands, install our build of that one package — it's minimized and being pushed upstream. Everything else is a stock-upstream + normal-dependency install. :::

:::note Python (rclpy) on PyPI: coming soon robotops-trace and robotops-trace-rclpy on PyPI are not published yet — the PyPI org is being set up. The apt (dev channel) C++ lane is live today. ROS-coupled fleets can also get robotops_trace_rclpy as an ament_python deb via apt once published. :::

Annotate (optional)

Auto-init gives you a span for every framework callback for free. Add one line for first-party depth inside your own code:

C++

void GraspNode::plan() {
  ROBOTOPS_TRACE("plan_grasp");   // RAII span for this scope
  // ... your planning code ...
}                                 // span ends automatically

Python

@robotops.trace                   # span around the whole function
def plan_grasp(self):
    ...

# or, for a scoped block:
with robotops.span("plan_grasp"):
    ...

Annotated spans nest automatically under the surrounding callback span via thread-local (C++) / contextvars (Python) propagation — including across async boundaries.

What you get

With the SDK installed and auto-init on, TraceHouse shows:

• Deterministic intra-process traces — exact parent/child causality within a process, captured from real events, not statistical sampling.
• Auto-spans per callback — every rclcpp / rclpy executor callback (and BehaviorTree.CPP tick) becomes a span, tagged with robot.callback.type (subscription / timer / service / action / client).
• Deterministic action-hop stitching — action client and server spans are joined by robot.action.goal_id (the goal UUID), so e.g. a NavigateToPose is one trace across the process boundary, with robot.action.status and robot.action.result.
• Best-effort topic/service stitching — pub/sub and service hops are correlated via DDS content keys (ros.publisher_gid, ros.source_timestamp, ros.message.content_hash).
• Black-box spans for uninstrumented code — frameworks you haven't added an integration for still appear as opaque spans. Missing coverage is a gap, never a failure — it never crashes or blocks your node.

It all renders in TraceHouse as waterfalls with span attributes and events, and is queryable with ROSQL.

Semantic conventions

Spans are described by a shared, stable robotics vocabulary so traces are queryable the same way across frameworks. Two namespaces:

• robot.* — robotics-concept keys (portable): robot.action.*, robot.callback.type, robot.transform.*, robot.joint.*, robot.trajectory.*, robot.target.*, robot.component.name.
• ros.* — the ROS mapping keys (present only when the transport is ROS): ros.node, ros.topic, ros.service, ros.message.type, and the content-correlation keys above.

The authoritative source is the robotops_trace_semconv package (a C++ header + a Python module); see the Glossary for the key dictionary.

What's next

• Quickstart — the install + turn-on flow
• TraceHouse Agent Overview — what the carrier collects beyond traces
• Related Libraries — the SDK repos and the rest of the ecosystem