Deploying OpenTelemetry Collector and Jaeger for Distributed Tracing on a VPS

What we are setting up and why

Distributed tracing is needed when a single user request passes through multiple components: a frontend, API, queue, worker, database, third-party payment service, or LLM API. Ordinary logs show individual events and metrics show the overall load picture, but they do not quickly answer the question: “Why did a specific user request take 8 seconds to execute, and which service caused the delay?”

A trace consists of related spans. Each span describes an operation: an HTTP request, SQL query, message publication to Kafka, job execution, or request to Redis. A span has a start time, duration, status, attributes, and the identifiers trace_id and span_id. If services pass trace context through HTTP headers or a queue, Jaeger assembles the request path into a single tree.

In this setup, OpenTelemetry Collector serves as an intermediary layer between applications and the trace storage. Applications send data to the Collector using the standard OTLP protocol. The Collector receives the data, adds useful attributes, limits memory usage, performs batching, and exports traces to Jaeger. This keeps applications independent of a specific backend: later, Jaeger can be replaced with Tempo, OpenSearch, a ClickHouse-compatible storage system, or a cloud service without rewriting the instrumentation.

What you will have as a result

After completing the instructions, you will have a domain such as traces.example.com with HTTPS access to the Jaeger UI. Applications will be able to send OTLP/gRPC traces to https://otel.example.com:4317 or to the Collector’s private IP without TLS if all services are located on an internal network or connected through WireGuard.

In Jaeger, you will be able to select a service, operation, and time range, open an individual trace, view a waterfall diagram of spans, identify a bottleneck, and navigate to related logs using trace_id. In the configuration, we will add the attributes deployment.environment and host.name so that production, staging, and different servers are not mixed in the same search.

Self-hosted or cloud-managed tracing

Cloud observability platforms usually offer a quick start, ready-made alerts, long-term storage, and a scalable backend. The disadvantages are charges based on ingestion volume, vendor dependency, transmission of technical metadata to a third party, and the risk of an unexpected bill increase during a traffic spike.

Self-hosted OpenTelemetry Collector and Jaeger on a VPS are suitable for small teams, internal services, early-stage SaaS products, test environments, and projects with data-control requirements. You choose the retention period, access controls, backups, and hosting location yourself. The price of this flexibility is the need to monitor disk space, updates, security, and growth in trace volume.

Jaeger with Badger in this article is a practical option for small and medium volumes: dozens of services and a moderate number of requests. Long-term storage, high cardinality, multi-terabyte traffic, and multiple Collector instances require a separate backend: OpenSearch, Elasticsearch, ClickHouse, or another supported production storage system. The Collector-based architecture remains the same; only the exporter and Jaeger backend change.

What VPS configuration is needed for this task

Resource consumption depends less on the number of services than on the number of spans per second, the average size of attributes, and the retention period. A single HTTP request can create 3–5 spans in a simple API or 30–100 spans in a microservice system with SQL, Redis, queues, and external APIs. Do not record HTTP request bodies, tokens, passwords, or large payloads: this is dangerous and quickly fills the storage.

For an initial production deployment, choose 4 vCPU, 8 GB RAM, 120 GB NVMe, and a 1 Gbps port. This reserve will allow the Collector to work with a memory limiter and queues, Jaeger to cache data, and the system to avoid swapping during short-term load spikes. As a neutral option, you can choose a VPS with the specified characteristics, but before ordering, check the storage type, traffic limit, IPv4 address, and whether the disk can be expanded without migration.

How to estimate disk space

Start with measurements rather than assumptions. Suppose an application creates 20 spans per request, processes 10 requests per second, and one span occupies an average of 1–3 KB after serialization and indexing. That is already approximately 17–52 GB per day before accounting for compression, metadata, and sampling. With 10% sampling, the volume will decrease approximately tenfold, making sampling the main regulator of storage costs.

For the first month, allocate at least twice as much space as the estimated volume for the retention period. This leaves room for Docker image updates, backup archives, and sudden traffic growth. Configure an alert when disk usage reaches 70%, rather than 95%: Badger and Docker can degrade sharply when free space runs out.

When you need dedicated rather than a VPS

A VPS is suitable while the load is predictable and disk latency is not critical. A dedicated server makes sense with consistently high ingestion, long retention periods, the need for multiple NVMe drives in RAID, CPU and IOPS isolation requirements, or the use of a resource-intensive backend such as OpenSearch. Dedicated hosting is also reasonable if the tracing system serves dozens of teams and becomes an internal observability platform.

Do not move Jaeger to a dedicated server simply because the interface shows tens of thousands of traces. First, enable head sampling in the SDK, remove unnecessary attributes, reduce retention, check disk speed, and separate production and staging into different services. This usually has a more noticeable effect than immediately increasing the server size.

Choosing a location

Location affects telemetry transmission latency, legal requirements, and traffic costs. The Collector should be located close to the services sending traces: in the same data center, region, or at least on the same continent. For synchronous OTLP/gRPC, high RTT increases application overhead, especially on an unreliable network.

If production services are located in different regions, do not make every service write over the internet to a single Collector. Deploy local Collector instances close to the applications and send the data onward asynchronously or to a central backend. One VPS is sufficient to start, but preserve the ability to split the architecture this way.

Server Preparation

Ubuntu Server 24.04 LTS is used below. Run commands as root only during the initial preparation stage, then work under the separate otel user. It is assumed that you have the domains traces.example.com for the Jaeger UI and, if necessary, otel.example.com for the public OTLP endpoint.

Before configuring TLS, create an A record for traces.example.com pointing to the server’s public IPv4 address. If you plan to accept OTLP through a separate domain, also add a record for otel.example.com. You can check DNS from a local computer with the command dig +short traces.example.com.

Operating System Update

First install the accumulated updates and basic utilities. After updating the kernel, reboot the server before continuing with Docker configuration.

# Updates package indexes and installs security updates.
sudo apt update && sudo apt upgrade -y

# Installs diagnostic tools, firewall, editor, and HTTPS repository support.
sudo apt install -y ca-certificates curl gnupg lsb-release \
  ufw fail2ban unattended-upgrades jq vim htop dnsutils

# Reboots the server if the kernel or system libraries were updated.
sudo reboot

User, SSH Keys, and sudo

Create a separate account. Do not use a password for SSH: an Ed25519 key is easier to protect against brute-force attacks and is more convenient for automation. Before disabling root login, be sure to open a second SSH session and verify that the new user can log in and run sudo.

# Creates the otel user with a home directory and adds it to the sudo group.
sudo adduser otel
sudo usermod -aG sudo otel

# Creates the key directory and copies the current authorised_keys for the new user.
sudo install -d -m 700 -o otel -g otel /home/otel/.ssh
sudo cp /root/.ssh/authorized_keys /home/otel/.ssh/authorized_keys
sudo chown otel:otel /home/otel/.ssh/authorized_keys
sudo chmod 600 /home/otel/.ssh/authorized_keys

Edit the SSH configuration. If you are connecting as a user other than root or your provider stores the key elsewhere, do not copy the file blindly: add your public key with ssh-copy-id otel@SERVER_IP from a local machine.

# Opens a separate SSH configuration file to strengthen access settings.
sudo vim /etc/ssh/sshd_config.d/99-hardening.conf

# Checks the configuration syntax and restarts SSH only if successful.
sudo sshd -t && sudo systemctl restart ssh

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
X11Forwarding no
MaxAuthTries 3
AllowUsers otel

Firewall and Fail2ban

Expose only SSH, HTTP, and HTTPS externally. Ports 4317, 4318, 16686, and 14268 should not be published to the entire internet. If applications are located on other servers, later allow OTLP only from their private subnet or through WireGuard.

# Allows SSH before enabling the firewall; otherwise you may lose access to the server.
sudo ufw allow OpenSSH

# Allows Caddy to accept HTTP for ACME verification and HTTPS for the Jaeger UI.
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enables the firewall and displays the active rules.
sudo ufw enable
sudo ufw status verbose

# Enables Fail2ban to protect SSH against repeated failed login attempts.
sudo systemctl enable --now fail2ban

# Verifies that the SSH jail is active.
sudo fail2ban-client status sshd

An important Docker feature: published container ports may bypass ordinary UFW rules because of iptables rules created by Docker. In this guide, OTLP ports are not published by default at all. If you later add them to the ports section, restrict access with separate rules in the DOCKER-USER chain or use private networking/VPN.

Software Installation — Step by Step

We use Docker Engine 27.x or newer, Docker Compose Plugin 2.x, OpenTelemetry Collector Contrib 0.120.x, and Jaeger all-in-one 1.67.x as a compatible stable set. Before a production update, check the available image tags against the official release notes and test the update in staging: the Collector version and Jaeger image may change independently.

Installing Docker Engine from the Official Repository

Do not use the old docker.io package from the standard Ubuntu repository: it may be significantly out of date. The following commands add the official Docker repository and install the engine together with the Compose plugin.

# Removes conflicting old Docker packages, if they were installed.
sudo apt remove -y docker.io docker-doc docker-compose docker-compose-v2 \
  podman-docker containerd runc || true

# Creates a directory for keys of third-party APT repositories.
sudo install -m 0755 -d /etc/apt/keyrings

# Downloads and saves the Docker repository key in keyring format.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Adds the official Docker APT repository for the current Ubuntu release.
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Installs Docker Engine, CLI, containerd, and the Compose plugin.
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

# Starts Docker at boot and checks the installed version.
sudo systemctl enable --now docker
sudo docker version
sudo docker compose version

# Allows the otel user to manage Docker without sudo.
sudo usermod -aG docker otel

After adding the user to the docker group, log out of the SSH session and reconnect as otel. Membership in the Docker group effectively grants root-equivalent access to the server, so add only trusted administrators to it.

Project Directories and Secrets

Store the Compose file, configurations, and data in a predictable directory. The /opt/observability directory is usually more convenient than a home directory: it is easier to include in backups and less likely to be accidentally deleted when changing users.

# Creates the directory structure for configurations, Caddy, and Jaeger persistent data.
sudo mkdir -p /opt/observability/{otel-collector,caddy,data/jaeger}
sudo chown -R otel:otel /opt/observability

# Changes to the project working directory.
cd /opt/observability

# Creates .env with permissions for the owner only.
umask 077
touch .env
chmod 600 .env

The secret in this configuration will be the password hash for the Jaeger UI. Caddy does not store the password in plain text. Generate the hash interactively: the command will request a password and output a string beginning with $2a$ or a similar bcrypt hash.

# Generates a bcrypt password hash for HTTP Basic Authentication in Caddy.
docker run --rm caddy:2 caddy hash-password --plaintext 'ЗАМЕНИТЕ_НА_ДЛИННЫЙ_ПАРОЛЬ'

# Opens the environment variables file to enter the domains and generated hash.
vim /opt/observability/.env

Do not insert the actual password into docker-compose.yml, do not send .env to a public Git repository, and do not use the same password for SSH and the web interface. For team access, it is better to use a VPN or an external identity proxy, while leaving Basic Auth as an additional layer of protection.

Checking System Limits

Jaeger with Badger uses the filesystem more intensively than a simple stateless container. Before starting, check the available disk space and disable swap only if you have sufficient RAM and a clear OOM policy. For a small VPS, swap is useful as an emergency buffer, but continuous swap activity means that the server does not have enough memory.

# Displays free disk space, available RAM, and whether swap is present.
df -h /
free -h
swapon --show

# Creates Docker logging with a size limit so that container logs do not fill the disk.
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}
EOF

# Restarts Docker to apply the logging policy.
sudo systemctl restart docker

OpenTelemetry Collector, Jaeger, and HTTPS Configuration

Below is a configuration in which only the Caddy ports 80 and 443 are published externally. The Collector is accessible within the Docker network. This is a secure option if the applications are running on the same server or in the same Docker network. For remote applications, we will later show how to carefully open the OTLP ports for a trusted subnet.

Environment Variables File

Replace the domain, email, and bcrypt hash. In the value of CADDY_BASIC_AUTH_HASH, Docker Compose may interpret the $ character as variable substitution, so each $ is replaced with a double $$. This is normal Compose behavior.

# /opt/observability/.env
TRACES_DOMAIN=traces.example.com
[email protected]

# Вставьте результат caddy hash-password, заменив каждый $ на $$.
CADDY_BASIC_AUTH_HASH=$$2a$$14$$REPLACE_WITH_YOUR_BCRYPT_HASH

# Среда добавляется Collector ко всем входящим трейсам.
DEPLOYMENT_ENVIRONMENT=production

OpenTelemetry Collector Configuration

The Collector accepts OTLP over gRPC and HTTP, protects the server from running out of memory via memory_limiter, groups spans into batches, and sends them to Jaeger over OTLP/gRPC. Important: the receiver listens on 0.0.0.0 inside the container, but this does not make the port accessible from the internet until it is published in Compose.

# /opt/observability/otel-collector/config.yaml
extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  resource:
    attributes:
      - key: deployment.environment
        value: ${env:DEPLOYMENT_ENVIRONMENT}
        action: upsert
      - key: host.name
        value: tracing-vps
        action: upsert

  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 10000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/jaeger]

  telemetry:
    logs:
      level: info

The 512 MiB limit is intended for a server with 8 GB of RAM. On a VPS with 4 GB, reduce it to 256 MiB; on a server with 16 GB, it can be increased to 1024 MiB if justified by the workload. When the limit is reached, the Collector will begin rejecting some data instead of causing an OOM and bringing down the entire stack.

Docker Compose: Collector, Jaeger, and Caddy

Jaeger all-in-one receives traces from the Collector on the internal port 4317. The Badger directories are mounted using bind mounts, so removing the container will not delete the data. The BADGER_SPAN_STORE_TTL variable sets a retention period of 168 hours, or 7 days. Adjust the retention period according to available disk space and investigation requirements.

# /opt/observability/docker-compose.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.67.0
    container_name: jaeger
    restart: unless-stopped
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
      SPAN_STORAGE_TYPE: badger
      BADGER_EPHEMERAL: "false"
      BADGER_DIRECTORY_VALUE: /badger/data
      BADGER_DIRECTORY_KEY: /badger/key
      BADGER_SPAN_STORE_TTL: 168h
      MEMORY_MAX_TRACES: "0"
    volumes:
      - ./data/jaeger:/badger
    networks:
      - tracing

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.120.0
    container_name: otel-collector
    restart: unless-stopped
    command: ["--config=/etc/otelcol-contrib/config.yaml"]
    environment:
      DEPLOYMENT_ENVIRONMENT: ${DEPLOYMENT_ENVIRONMENT}
    volumes:
      - ./otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro
    depends_on:
      - jaeger
    networks:
      - tracing

  caddy:
    image: caddy:2.8.4
    container_name: caddy
    restart: unless-stopped
    environment:
      TRACES_DOMAIN: ${TRACES_DOMAIN}
      ACME_EMAIL: ${ACME_EMAIL}
      CADDY_BASIC_AUTH_HASH: ${CADDY_BASIC_AUTH_HASH}
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      - jaeger
    networks:
      - tracing

networks:
  tracing:
    name: tracing
    driver: bridge

volumes:
  caddy_data:
  caddy_config:

Caddy and HTTPS Access to the Jaeger UI

Caddy automatically obtains and renews the TLS certificate through ACME if the domain's DNS already points to the server and TCP ports 80 and 443 are accessible externally. Basic Auth protects the interface from accidental trace viewing. Do not consider Basic Auth a full-fledged access control system: for corporate use, add a VPN, SSO proxy, or network ACL.

# /opt/observability/caddy/Caddyfile
{
    email {$ACME_EMAIL}
    auto_https prefer_wildcard
}

{$TRACES_DOMAIN} {
    encode zstd gzip

    basic_auth {
        observability {$CADDY_BASIC_AUTH_HASH}
    }

    reverse_proxy jaeger:16686

    header {
        -Server
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "no-referrer"
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
    }

    log {
        output stdout
        format json
    }
}

Starting and Checking the Containers

Check the Compose syntax before starting. Then download the images and start the services in the background. The first TLS certificate issuance may take up to a minute; do not change DNS or restart Caddy unnecessarily.

# Проверяет подстановку переменных и итоговую конфигурацию Docker Compose.
cd /opt/observability
docker compose config > /tmp/observability-compose-rendered.yml

# Скачивает зафиксированные версии образов и запускает стек в фоне.
docker compose pull
docker compose up -d

# Показывает состояние контейнеров и последние строки журналов.
docker compose ps
docker compose logs --tail=100 otel-collector jaeger caddy

# Проверяет health endpoint Collector из его Docker-сети.
docker exec otel-collector wget -qO- http://127.0.0.1:13133/ || true

# Проверяет, что Jaeger API отвечает внутри контейнера.
docker exec jaeger wget -qO- http://127.0.0.1:16686/api/services

# Проверяет публичный HTTPS endpoint; потребуется пароль Basic Auth.
curl -I -u observability:'ВАШ_ПАРОЛЬ' https://traces.example.com

Open https://traces.example.com in a browser, enter the user observability and the password used to generate the hash. If the Jaeger UI loads but the service list is empty, this is normal: services will appear only after the first trace is sent.

How to Send a Test Trace

The simplest test is to temporarily run the telemetrygen container on the same Docker network. It will create test spans and send them to the Collector. Then, in Jaeger, select the telemetrygen service, the traces operation, and click Find Traces.

# Отправляет 20 тестовых trace в секунду в Collector внутри Docker-сети в течение 10 секунд.
docker run --rm --network tracing \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:0.120.0 \
  traces --otlp-endpoint otel-collector:4317 \
  --otlp-insecure --rate 20 --duration 10s

# Проверяет, что сервис telemetrygen появился в Jaeger API.
docker exec jaeger wget -qO- http://127.0.0.1:16686/api/services | jq .

Connecting Remote Applications

The safest approach is to connect the servers through WireGuard, Tailscale, or the provider's private network and publish the Collector ports only on the private interface. If this is not possible, add ports 4317 and 4318 to Compose, but allow access only from the application's known IP address. Do not open them to 0.0.0.0/0: anyone could send junk spans and fill your disk.

# Добавьте в сервис otel-collector только при необходимости внешнего OTLP.
# Лучше заменить 10.20.0.10 на приватный IP интерфейса сервера.
    ports:
      - "10.20.0.10:4317:4317"
      - "10.20.0.10:4318:4318"

For an application written in Python, Node.js, Go, Java, or .NET, specify the endpoint through environment variables. Instrumentation with OpenTelemetry SDK libraries is performed separately, but the basic variables are nearly the same for all languages.

# Пример общих переменных окружения приложения для OTLP/gRPC.
export OTEL_SERVICE_NAME=orders-api
export OTEL_RESOURCE_ATTRIBUTES="service.version=2026.03,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://10.20.0.10:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"

Sampling 0.1 means recording approximately 10% of new traces. The parent sampler preserves the parent request's decision across all child services, so you will not get fragments of a single trace. For important errors, you can later implement tail sampling in the Collector: it allows you to retain 100% of traces with errors and slow requests, but requires more memory and more careful configuration.

Backups and Maintenance

In this deployment, you need to back up the /opt/observability/data/jaeger directory, the Compose file, the Collector configuration, the Caddyfile, the .env file, and the Caddy volume data. The latter contains TLS certificates and ACME state; without them, Caddy can issue the certificate again, but recovery will be slower and may run into rate limits.

Do not consider a Docker image a backup. The image can be downloaded again, but traces in Badger and the configuration will not restore themselves. Also, do not store the only copy of a backup on the same VPS: disk loss, accidental deletion, or server compromise will destroy both the data and the backup.

Backup with restic to S3-compatible storage

Restic encrypts content before sending it and supports deduplication, so it is well suited for daily backups. Create a bucket with an external S3 provider or on a separate MinIO server. Store access credentials in a root-accessible file, not on the command line or in shell history.

# Installs restic from the Ubuntu repository.
sudo apt install -y restic

# Creates a backup secrets directory accessible only to root.
sudo install -d -m 700 /root/.config/restic

# Creates the S3 variables and repository password file.
sudo vim /root/.config/restic/observability.env
sudo chmod 600 /root/.config/restic/observability.env

# /root/.config/restic/observability.env
export RESTIC_REPOSITORY="s3:https://s3.example.net/observability-restic"
export RESTIC_PASSWORD="GENERATE_A_SEPARATE_LONG_PASSWORD"
export AWS_ACCESS_KEY_ID="REPLACE_ME"
export AWS_SECRET_ACCESS_KEY="REPLACE_ME"
export AWS_DEFAULT_REGION="us-east-1"

# Initializes the encrypted restic repository once.
sudo bash -c 'source /root/.config/restic/observability.env && restic init'

# Creates a directory for administrator scripts.
sudo install -d -m 700 /usr/local/sbin

# Creates the daily backup script.
sudo vim /usr/local/sbin/backup-observability.sh
sudo chmod 700 /usr/local/sbin/backup-observability.sh

#!/usr/bin/env bash
set -euo pipefail

source /root/.config/restic/observability.env

cd /opt/observability

# Stops writing to Badger for the duration of the consistent backup.
docker compose stop jaeger

# Backs up the configurations, Jaeger data, and Caddy Docker volume.
restic backup \
  /opt/observability/otel-collector \
  /opt/observability/caddy \
  /opt/observability/data/jaeger \
  /opt/observability/docker-compose.yml \
  /opt/observability/.env \
  /var/lib/docker/volumes/observability_caddy_data \
  --tag observability

# Starts Jaeger even if a subsequent cleanup error occurs.
docker compose start jaeger

# Keeps 7 daily, 4 weekly, and 6 monthly snapshots.
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# Checks the integrity of part of the repository data.
restic check --read-data-subset=5%

To guarantee that Jaeger starts even if an error occurs, you can add trap 'docker compose start jaeger' EXIT to the script. With a small amount of data, briefly stopping Jaeger at night is acceptable. For large production storages, use the backend’s native snapshot mechanisms instead of copying files from a running database.

# Adds a daily backup at 03:20 and writes the log to a protected file.
sudo crontab -e

# Entry for the root crontab:
20 3   * /usr/local/sbin/backup-observability.sh >> /var/log/backup-observability.log 2>&1

Restoration testing

A backup without a restoration test is merely an assumption. Once a month, restore the data to a separate test VPS or at least to a temporary directory. Verify that the Jaeger UI opens after starting Compose and that old services and traces are available in search.

# Shows available snapshots in the restic repository.
sudo bash -c 'source /root/.config/restic/observability.env && restic snapshots'

# Restores the latest snapshot to a temporary directory for verification.
sudo mkdir -p /tmp/observability-restore
sudo bash -c 'source /root/.config/restic/observability.env && \
  restic restore latest --target /tmp/observability-restore'

Updates and retention

For a single VPS, perform updates during a maintenance window: make a backup, read the image changelogs, update one component, check its health, and only then proceed to the next. The docker compose pull && docker compose up -d command causes a brief interruption in trace reception. SDKs usually buffer some data, but you cannot rely on this without testing.

A rolling update is needed when losing even a few seconds of telemetry is unacceptable. In that case, run at least two Collector instances behind a load balancer, and use queues and an HA backend for Jaeger. A single Jaeger all-in-one with local Badger does not support a true rolling upgrade without interruption and is not a highly available configuration.

# Checks disk, container, and Jaeger data usage.
df -h /
docker stats --no-stream
sudo du -sh /opt/observability/data/jaeger

# Updates images only after creating a verified backup.
cd /opt/observability
docker compose pull
docker compose up -d
docker compose logs --tail=100

Troubleshooting and FAQ

Why does the Jaeger UI open, but the service list is empty?

An empty list means that Jaeger has not yet received a valid trace or that the Collector cannot export data. First, perform the test with telemetrygen from the configuration section. Then check docker compose logs otel-collector: look for exporter, connection refused, or deadline exceeded errors. Make sure the exporter endpoint is set to jaeger:4317 and that COLLECTOR_OTLP_ENABLED=true is enabled in Jaeger. Also verify that the application uses the traces exporter specifically, rather than only metrics or logs.

Why does the application receive connection refused on port 4317?

By default, port 4317 is not published externally in this article, so a remote server cannot connect to the Collector. This is intentional. For applications on the same host, use the Docker network and address otel-collector:4317. For another server, set up WireGuard or a private network, then safely publish the port on the private IP. Also check that the application is not trying to use the HTTP endpoint 4318 with the gRPC protocol, or vice versa.

Caddy is not obtaining a TLS certificate: what should I check?

Check that the domain’s A record points to the VPS public IP: dig +short traces.example.com. Then make sure ports 80 and 443 are allowed in UFW and in the provider’s network firewall panel. Check docker compose logs caddy; ACME errors usually directly indicate DNS issues, a closed port, or a certificate request rate limit. Do not use a CDN proxy mode that interferes with the HTTP challenge until you have confirmed that Caddy has obtained its first certificate.

The Collector consumes a lot of RAM or restarts with OOMKilled. How can I fix it?

First, confirm the cause with docker inspect otel-collector and check the OOMKilled field. Reduce limit_mib and spike_limit_mib in the memory_limiter processor, reduce queue_size, lower sampling in the SDK, and disable excessive attributes. Do not increase memory limits indefinitely: if the trace flow exceeds the VPS capacity, the Collector will only accumulate more data before crashing. For high load, add a second Collector and a load balancer.

The disk fills up quickly even though retention is set to 7 days.

Check the actual Badger size with du -sh /opt/observability/data/jaeger, Docker logs with docker system df, and system logs with journalctl --disk-usage. Retention does not always free space immediately: storage compaction takes time. Reduce sampling, shorten BADGER_SPAN_STORE_TTL, remove unused Docker images with docker image prune, and do not write request/response bodies to span attributes. Adding disk space is safer before usage reaches 90%.

What is the minimum suitable VPS configuration?

For tests, a home lab, or staging, 2 vCPU, 4 GB RAM, 40 GB NVMe, and 100 Mbps are sufficient. Such a server is suitable for several applications with 5–20% sampling and short retention. For production, it is better to choose 4 vCPU, 8 GB RAM, and 100–160 GB NVMe, leaving capacity for Collector queues, caching, updates, and data growth. If the disk is a slow HDD, the tracing backend may lag even with sufficient CPU.

Which should I choose for this task—a VPS or dedicated server?

Start with a VPS if this is one project with several services and retention of up to several weeks. A VPS is easier to scale in terms of CPU, RAM, and disk, and OpenTelemetry Collector works well in this environment. A dedicated server is needed for consistently high span ingestion, strict IOPS requirements, multiple NVMe drives in RAID, long-term storage, and a heavy backend instead of local Badger. The transition should be justified by measurements: CPU load, memory pressure, disk latency, and storage fill rate.

Can I send personal data, JWTs, or SQL parameters to the Collector?

No, not by default. Trace attributes are often accessible to developers, included in backups, and may be stored for weeks. Exclude Authorization headers, cookies, passwords, tokens, full request bodies, email addresses, and payment data. In the SDK, use an attribute allowlist, and add processors to the Collector to remove sensitive keys. For SQL, it is better to record the operation type and table name rather than the complete query with user parameters.

Why does a trace consist of one span even though the request passes through several services?

The usual cause is context loss between services. Check that the HTTP client automatically forwards W3C Trace Context headers: traceparent and tracestate. For queues, make sure the producer writes the context to message headers and the consumer extracts it before creating a child span. All services must use a compatible propagator, usually tracecontext. Also check that different services are not generating a new root trace because auto-instrumentation is disabled.

Conclusions and Next Steps

You now have a self-hosted distributed tracing stack: applications send OTLP traces to the OpenTelemetry Collector, the Collector forwards them to Jaeger, and the Jaeger UI is available over HTTPS with basic protection. The configuration is suitable for initial production use under moderate load and gives you control over data, retention, and infrastructure.

1. Instrument the main HTTP inputs, outgoing HTTP calls, SQL, Redis, and queues; start with one critical user flow.
2. Add rule-based sampling: retain all errors and slow requests, while recording successful typical requests at a lower probability.
3. As load grows, separate the Collector and Jaeger onto different servers, move storage to a production backend, and configure monitoring for CPU, RAM, disk, queue size, and ingestion rate.

Regularly test backup restoration, available disk space, and image currency. Tracing is useful only when the required trace can be found quickly at the moment of an incident.