Performance Tuning - Tekton Pipelines

Tekton Pipelines performance can be tuned by adjusting controller parameters including thread count, API throttling, and resource limits.

Overview

Three primary parameters impact Tekton controller performance:

ThreadsPerController - Number of goroutines for processing the work queue
QPS (Queries Per Second) - Maximum queries to the Kubernetes API server
Burst - Maximum burst for API throttling

Default Values

Out-of-the-box configuration:

Parameter	Default Value	Source
ThreadsPerController	2	knative/pkg
QPS	5.0	client-go
Burst	10	client-go

QPS and Burst values are multiplied by 2 internally, so the actual values are double the configured values.

Configuration Methods

Performance parameters can be configured using:

Command-line flags in the controller deployment
Environment variables

Command-line flags take precedence over environment variables.

Configuring via Command-Line Flags

Modify the controller deployment in config/controller.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tekton-pipelines-controller
  namespace: tekton-pipelines
spec:
  template:
    spec:
      serviceAccountName: tekton-pipelines-controller
      containers:
      - name: tekton-pipelines-controller
        image: ko://github.com/tektoncd/pipeline/cmd/controller
        args:
        - "-kube-api-qps=50"
        - "-kube-api-burst=50"
        - "-threads-per-controller=32"
        # Other flags...

Apply the changes:

kubectl apply -f config/controller.yaml

Available Flags

-threads-per-controller

integer

default:"2"

Number of threads (goroutines) to create per controller for processing the work queue.Higher values increase parallelism but consume more memory.

args:
  - "-threads-per-controller=32"

-kube-api-qps

float

default:"5.0"

Maximum queries per second to the Kubernetes API server from this client.Note: Actual QPS is multiplied by 2 internally.

args:
  - "-kube-api-qps=50"

With this configuration, actual QPS = 100.

-kube-api-burst

integer

default:"10"

Maximum burst for throttling API requests.Note: Actual burst is multiplied by 2 internally.

args:
  - "-kube-api-burst=50"

With this configuration, actual burst = 100.

Configuring via Environment Variables

Alternatively, use environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tekton-pipelines-controller
  namespace: tekton-pipelines
spec:
  template:
    spec:
      containers:
      - name: tekton-pipelines-controller
        image: ko://github.com/tektoncd/pipeline/cmd/controller
        env:
        - name: THREADS_PER_CONTROLLER
          value: "32"
        - name: KUBE_API_QPS
          value: "50"
        - name: KUBE_API_BURST
          value: "50"

THREADS_PER_CONTROLLER

string

Environment variable for threads per controller.

env:
  - name: THREADS_PER_CONTROLLER
    value: "32"

KUBE_API_QPS

string

Environment variable for API queries per second.

env:
  - name: KUBE_API_QPS
    value: "50"

KUBE_API_BURST

string

Environment variable for API burst throttling.

env:
  - name: KUBE_API_BURST
    value: "50"

Performance Tuning Guidelines

Small Deployments (< 100 PipelineRuns/day)

Use default values:

args:
  - "-threads-per-controller=2"
  - "-kube-api-qps=5"
  - "-kube-api-burst=10"

Medium Deployments (100-1000 PipelineRuns/day)

Increase concurrency:

args:
  - "-threads-per-controller=16"
  - "-kube-api-qps=25"
  - "-kube-api-burst=50"

Large Deployments (> 1000 PipelineRuns/day)

Maximize throughput:

args:
  - "-threads-per-controller=32"
  - "-kube-api-qps=50"
  - "-kube-api-burst=100"

High-Concurrency Scenarios

For clusters with many simultaneous PipelineRuns:

args:
  - "-threads-per-controller=64"
  - "-kube-api-qps=100"
  - "-kube-api-burst=200"

Resource Requirements

Adjust controller resource limits based on performance configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tekton-pipelines-controller
  namespace: tekton-pipelines
spec:
  template:
    spec:
      containers:
      - name: tekton-pipelines-controller
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        args:
        - "-threads-per-controller=32"
        - "-kube-api-qps=50"
        - "-kube-api-burst=50"

Resource Scaling Guidelines

Threads	CPU Request	Memory Request	CPU Limit	Memory Limit
2 (default)	100m	256Mi	500m	512Mi
16	250m	512Mi	1000m	1Gi
32	500m	512Mi	2000m	2Gi
64	1000m	1Gi	4000m	4Gi

Monitoring Performance

Key Metrics

Monitor these metrics to assess controller performance:

# Work queue depth
workqueue_depth{name="pipelinerun"}

# Reconciliation latency
controller_reconcile_duration_seconds{controller="pipelinerun-controller"}

# API client latency
tekton_pipelines_controller_client_latency_bucket

# Running resources
tekton_pipelines_controller_running_pipelineruns
tekton_pipelines_controller_running_taskruns

Performance Indicators

Good Performance:

Work queue depth remains low (< 10)
Reconciliation latency < 1s (p95)
API client latency < 100ms (p95)
No throttling errors in logs

Poor Performance:

Work queue depth grows unbounded
Reconciliation latency > 5s (p95)
API client latency > 500ms (p95)
Frequent “rate limit exceeded” errors

Troubleshooting

High Work Queue Depth

Symptoms: Work queue depth metric increases continuously Solutions:

Increase threads-per-controller
Verify API server health
Check for slow reconciliation (enable debug logging)

kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller --tail=100

API Rate Limiting

Symptoms: Logs show “rate limit exceeded” or “client rate limiter Wait” Solutions:

Increase kube-api-qps and kube-api-burst
Verify API server capacity
Consider cluster API server scaling

# Check for rate limit errors
kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller | grep -i "rate limit"

High Memory Usage

Symptoms: Controller pods are OOMKilled or approaching memory limits Solutions:

Increase memory limits
Reduce threads-per-controller if excessively high
Check for memory leaks (file issue if found)

# Check memory usage
kubectl top pod -n tekton-pipelines -l app=tekton-pipelines-controller

Slow Reconciliation

Symptoms: PipelineRuns take long to start or complete Solutions:

Enable debug logging to identify bottlenecks
Increase threads-per-controller
Verify webhook performance
Check node and pod resource availability

Complete High-Performance Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tekton-pipelines-controller
  namespace: tekton-pipelines
spec:
  replicas: 3  # HA configuration
  template:
    spec:
      serviceAccountName: tekton-pipelines-controller
      containers:
      - name: tekton-pipelines-controller
        image: ko://github.com/tektoncd/pipeline/cmd/controller
        args:
        # Performance tuning
        - "-threads-per-controller=32"
        - "-kube-api-qps=50"
        - "-kube-api-burst=50"
        # Other required args
        - "-entrypoint-image=ko://github.com/tektoncd/pipeline/cmd/entrypoint"
        - "-nop-image=ko://github.com/tektoncd/pipeline/cmd/nop"
        - "-sidecarlogresults-image=ko://github.com/tektoncd/pipeline/cmd/sidecarlogresults"
        - "-workingdirinit-image=ko://github.com/tektoncd/pipeline/cmd/workingdirinit"
        - "-shell-image=cgr.dev/chainguard/busybox@sha256:19f02276bf8dbdd62f069b922f10c65262cc34b710eea26ff928129a736be791"
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        env:
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: CONFIG_LOGGING_NAME
          value: config-logging
        - name: CONFIG_OBSERVABILITY_NAME
          value: config-observability
        - name: CONFIG_FEATURE_FLAGS_NAME
          value: feature-flags
        - name: METRICS_DOMAIN
          value: tekton.dev/pipeline
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-leader-election-controller
  namespace: tekton-pipelines
data:
  buckets: "10"  # Max buckets for HA
  lease-duration: "60s"
  renew-deadline: "40s"
  retry-period: "10s"

Best Practices

Start with defaults and increase gradually based on metrics
Match thread count to workload - Don’t over-provision
Monitor API server impact when increasing QPS/Burst
Adjust resource limits proportionally with thread count
Enable HA for production deployments (3+ replicas)
Use HPA for webhook to handle variable load
Test changes in non-production environment first
Monitor continuously after configuration changes
Document tuning decisions for future reference
Review quarterly and adjust based on workload changes

​Overview

​Default Values

​Configuration Methods

​Configuring via Command-Line Flags

​Available Flags

​Configuring via Environment Variables

​Performance Tuning Guidelines

​Small Deployments (< 100 PipelineRuns/day)

​Medium Deployments (100-1000 PipelineRuns/day)

​Large Deployments (> 1000 PipelineRuns/day)

​High-Concurrency Scenarios

​Resource Requirements

​Resource Scaling Guidelines

​Monitoring Performance

​Key Metrics

​Performance Indicators

​Troubleshooting

​High Work Queue Depth

​API Rate Limiting

​High Memory Usage

​Slow Reconciliation

​Complete High-Performance Configuration

​Best Practices

Overview

Default Values

Configuration Methods

Configuring via Command-Line Flags

Available Flags

Configuring via Environment Variables

Performance Tuning Guidelines

Small Deployments (< 100 PipelineRuns/day)

Medium Deployments (100-1000 PipelineRuns/day)

Large Deployments (> 1000 PipelineRuns/day)

High-Concurrency Scenarios

Resource Requirements

Resource Scaling Guidelines

Monitoring Performance

Key Metrics

Performance Indicators

Troubleshooting

High Work Queue Depth

API Rate Limiting

High Memory Usage

Slow Reconciliation

Complete High-Performance Configuration

Best Practices