At work, I was tasked to set up Datadog metrics from our Temporal workflows. I'll spare you the details of my failed attempts and go straight to what worked for me.
The Temporal TypeScript SDK provides examples for tracing with OpenTelemetry.
1import { DefaultLogger, Worker, Runtime } from "@temporalio/worker";
2import { Resource } from "@opentelemetry/resources";
3import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
4import { ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
5import { NodeSDK } from "@opentelemetry/sdk-node";
6import {
7 OpenTelemetryActivityInboundInterceptor,
8 makeWorkflowExporter,
9} from "@temporalio/interceptors-opentelemetry/lib/worker";
10import * as activities from "./activities";
11
12async function main() {
13 const resource = new Resource({
14 [SemanticResourceAttributes.SERVICE_NAME]: "interceptors-sample-worker",
15 });
16 // Export spans to console for simplicity
17 const exporter = new ConsoleSpanExporter();
18
19 const otel = new NodeSDK({ traceExporter: exporter, resource });
20 await otel.start();
21
22 // Silence the Worker logs to better see the span output in this sample
23 Runtime.install({ logger: new DefaultLogger("WARN") });
24
25 const worker = await Worker.create({
26 workflowsPath: require.resolve("./workflows"),
27 activities,
28 taskQueue: "interceptors-opentelemetry-example",
29 sinks: {
30 exporter: makeWorkflowExporter(exporter, resource),
31 },
32 // Registers opentelemetry interceptors for Workflow and Activity calls
33 interceptors: {
34 // example contains both workflow and interceptors
35 workflowModules: [require.resolve("./workflows")],
36 activityInbound: [(ctx) => new OpenTelemetryActivityInboundInterceptor(ctx)],
37 },
38 });
39 try {
40 await worker.run();
41 } finally {
42 await otel.shutdown();
43 }
44}
45
46main().then(
47 () => void process.exit(0),
48 (err) => {
49 console.error(err);
50 process.exit(1);
51 },
52);
This example prints out spans but I want them to be exported to Datadog. There doesn't seem to be an exporter for Node but I kept seeing recommendations for OLTP ingestion via the Datadog agent in the Datadog docs. I gave it a try and it worked perfectly with the OTLPTraceExporter OpenTelemetry provides.
Our Temporal worker is deployed as a Google Cloud Run service and instrumenting it with Datadog is as simple as changing its entrypoint. Enabling OTLP ingestion is also as easy.
1COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
2COPY --from=datadog/dd-lib-js-init /operator-build/node_modules /dd_tracer/node/
3ENV DD_SERVICE=datadog-demo-run-nodejs
4ENV DD_ENV=datadog-demo
5ENV DD_VERSION=1
6# Enable OTLP ingestion on the Datadog Agent
7ENV DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=localhost:4318
8ENTRYPOINT ["/app/datadog-init"]
9CMD ["/nodejs/bin/node", "/path/to/your/app.js"]
Lastly, a quick switcheroo and we're done!
1- import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-base';
2+ import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
1- // Export spans to console for simplicity
2- const exporter = new ConsoleSpanExporter();
3+ const exporter = new OTLPTraceExporter();