Metrics and Monitoring

Configure and use Tekton Pipelines metrics for observability

Tekton Pipelines exposes metrics for monitoring pipeline execution, controller performance, and resource utilization.

Metrics Endpoint

Metrics are available at the tekton-pipelines-controller service on port 9090:

kubectl port-forward -n tekton-pipelines service/tekton-pipelines-controller 9090

Access metrics at: http://127.0.0.1:9090/metrics

Available Metrics

All metrics are experimental and subject to change.

PipelineRun Metrics

tekton_pipelines_controller_pipelinerun_duration_seconds · Histogram/Gauge

Duration of PipelineRuns in seconds.

Labels:

• pipeline - Pipeline name (optional)
• pipelinerun - PipelineRun name (optional)
• status - Completion status
• namespace - PipelineRun namespace
• reason - Completion reason (optional)

Variants: _bucket, _sum, _count

tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds · Histogram/Gauge

Duration of TaskRuns within PipelineRuns in seconds.

Labels:

• pipeline - Pipeline name (optional)
• pipelinerun - PipelineRun name (optional)
• task - Task name (optional)
• taskrun - TaskRun name (optional)
• status - Completion status
• namespace - Namespace
• reason - Completion reason (optional)

Variants: _bucket, _sum, _count

tekton_pipelines_controller_pipelinerun_total · Counter

Total number of PipelineRuns.

Labels:

• status - Completion status

tekton_pipelines_controller_running_pipelineruns · Gauge

Number of currently running PipelineRuns.

TaskRun Metrics

tekton_pipelines_controller_taskrun_duration_seconds · Histogram/Gauge

Duration of TaskRuns in seconds.

Labels:

• task - Task name (optional)
• taskrun - TaskRun name (optional)
• status - Completion status
• namespace - TaskRun namespace
• reason - Completion reason (optional)

Variants: _bucket, _sum, _count

tekton_pipelines_controller_taskrun_total · Counter

Total number of TaskRuns.

Labels:

• status - Completion status

tekton_pipelines_controller_running_taskruns · Gauge

Number of currently running TaskRuns.

Throttling Metrics

tekton_pipelines_controller_running_taskruns_throttled_by_quota · Gauge

Number of TaskRuns throttled by resource quota.

Labels:

• namespace - TaskRun namespace (optional)

tekton_pipelines_controller_running_taskruns_throttled_by_node · Gauge

Number of TaskRuns throttled by node availability.

Labels:

• namespace - TaskRun namespace (optional)

Client Metrics

tekton_pipelines_controller_client_latency · Histogram

Kubernetes API client latency in milliseconds.

Variants: _bucket, _sum, _count

Metrics Configuration

Configure metrics behavior in the config-observability ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: tekton-pipelines
data:
  metrics-protocol: prometheus
  metrics.taskrun.level: "task"
  metrics.taskrun.duration-type: "histogram"
  metrics.pipelinerun.level: "pipeline"
  metrics.pipelinerun.duration-type: "histogram"
  metrics.count.enable-reason: "false"
  metrics.running-pipelinerun.level: ""

TaskRun Metrics Level

metrics.taskrun.level · string · default: "task"

Granularity level for TaskRun metrics.

• taskrun - Include taskrun label (highest cardinality)
• task - Include task label, exclude taskrun label
• namespace - Include only namespace label (lowest cardinality)

data:
  metrics.taskrun.level: "namespace"

TaskRun Duration Type

metrics.taskrun.duration-type · string · default: "histogram"

Metric type for TaskRun duration.

• histogram - Histogram with buckets for duration distribution
• lastvalue - Gauge with last observed duration

data:
  metrics.taskrun.duration-type: "lastvalue"

Histogram is not available when taskrun or pipelinerun labels are selected (leads to single bar).

PipelineRun Metrics Level

metrics.pipelinerun.level · string · default: "pipeline"

Granularity level for PipelineRun metrics.

• pipelinerun - Include pipelinerun label (highest cardinality)
• pipeline - Include pipeline label, exclude pipelinerun label
• namespace - Include only namespace label (lowest cardinality)

data:
  metrics.pipelinerun.level: "namespace"

Running PipelineRun Level

metrics.running-pipelinerun.level · string · default: ""

Granularity level for running PipelineRun count metrics.

• pipelinerun - Include pipelinerun label
• pipeline - Include pipeline label
• namespace - Include namespace label
• "" (empty) - Cluster level, no labels

data:
  metrics.running-pipelinerun.level: "namespace"

PipelineRun Duration Type

metrics.pipelinerun.duration-type · string · default: "histogram"

Metric type for PipelineRun duration.

• histogram - Histogram with buckets
• lastvalue - Gauge with last value

data:
  metrics.pipelinerun.duration-type: "lastvalue"

Reason Label

metrics.count.enable-reason · boolean · default: "false"

Include reason label on duration metrics.

data:
  metrics.count.enable-reason: "true"

Does not affect total counters (*_total), which always include reason.

Throttle Namespace Label

metrics.taskrun.throttle.enable-namespace · boolean · default: "false"

Include namespace label on throttle metrics.

data:
  metrics.taskrun.throttle.enable-namespace: "true"

OpenTelemetry Configuration

Metrics Protocol

metrics-protocol · string · default: "prometheus"

Protocol for metrics export.

Options: prometheus, grpc, http/protobuf, none

data:
  metrics-protocol: "grpc"

metrics-endpoint · string

Metrics endpoint for gRPC/HTTP protocols.

data:
  metrics-endpoint: "otel-collector:4317"

metrics-export-interval · duration

Metrics export interval.

data:
  metrics-export-interval: "30s"

Tracing Configuration

tracing-protocol · string · default: "none"

Protocol for tracing export.

Options: grpc, http/protobuf, none, stdout

data:
  tracing-protocol: "grpc"

tracing-endpoint · string

Tracing endpoint for gRPC/HTTP protocols.

data:
  tracing-endpoint: "otel-collector:4317"

tracing-sampling-rate · string · default: "1.0"

Tracing sampling rate (0.0 to 1.0).

data:
  tracing-sampling-rate: "0.1"

Runtime Profiling

runtime-profiling · string · default: "disabled"

Enable runtime profiling.

Options: enabled, disabled

data:
  runtime-profiling: "enabled"

runtime-export-interval · duration · default: "15s"

Runtime metrics export interval.

data:
  runtime-export-interval: "30s"

Prometheus Integration

ServiceMonitor

For Prometheus Operator, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-pipelines-controller
  namespace: tekton-pipelines
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: controller
  endpoints:
  - port: http-metrics
    interval: 30s

Scrape Configuration

For standard Prometheus, add scrape configuration:

scrape_configs:
  - job_name: 'tekton-pipelines'
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - tekton-pipelines
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      action: keep
      regex: tekton-pipelines-controller
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: http-metrics

Grafana Dashboards

Example Prometheus queries for Grafana:

PipelineRun Success Rate

sum(rate(tekton_pipelines_controller_pipelinerun_total{status="success"}[5m]))
/
sum(rate(tekton_pipelines_controller_pipelinerun_total[5m]))

Average PipelineRun Duration

rate(tekton_pipelines_controller_pipelinerun_duration_seconds_sum[5m])
/
rate(tekton_pipelines_controller_pipelinerun_duration_seconds_count[5m])

Running PipelineRuns by Namespace

tekton_pipelines_controller_running_pipelineruns{namespace=~".*"}

TaskRun Throttling

sum by (namespace) (tekton_pipelines_controller_running_taskruns_throttled_by_quota)

Best Practices

1. Use namespace-level metrics in production to avoid unbounded cardinality
2. Enable reason labels only when needed for debugging
3. Monitor throttling metrics to identify resource quota issues
4. Set appropriate scrape intervals (30s recommended)
5. Use histogram type for duration metrics when aggregating across multiple resources
6. Configure retention policies in your metrics backend
7. Alert on high failure rates and long-running pipelines

TaskRun and PipelineRun level metrics are not recommended for production as they lead to unbounded cardinality, which can degrade observability database performance.

Verification

Verify metrics configuration is applied:

# Port forward to metrics endpoint
kubectl port-forward -n tekton-pipelines service/tekton-pipelines-controller 9090

# Check metrics are exposed
curl http://127.0.0.1:9090/metrics | grep tekton_pipelines

# Verify specific metric
curl http://127.0.0.1:9090/metrics | grep pipelinerun_duration_seconds