WhisperX on VPS: Transcription with Timestamps and Speaker Diarization

What We Are Configuring and Why

WhisperX is a local transcription tool built around Whisper and faster-whisper models. Unlike a basic Whisper launch, it adds alignment of the text with the audio signal: as a result, you can obtain not only a phrase with an approximate start time, but also timestamps for individual words. This is useful for subtitles, interview transcripts, podcast editing, searching video archives, and preparing meeting minutes.

The second important feature is diarization, meaning speaker separation. After processing a recording, the service assigns labels such as SPEAKER_00, SPEAKER_01, and so on to segments. WhisperX does not know people’s names automatically: it determines which fragments belong to the same voice. Names can later be changed in a subtitle editor or in the JSON result.

In this guide, an internal HTTP API in Python will be deployed. It accepts the path to an already uploaded media file, launches WhisperX inside a Docker container, saves the results to a task directory, and returns JSON. Caddy will be installed in front of the API; it automatically obtains a TLS certificate from Let’s Encrypt and protects the service with HTTP Basic Auth. This approach is suitable for a single VPS owner, a small team, or an internal SaaS tool.

What You Will Have After Configuration

• An HTTPS address such as https://transcribe.example.com.
• A protected API with /health and /transcribe endpoints.
• Processing of MP3, WAV, M4A, MP4, MKV, and other formats supported by FFmpeg.
• Automatic generation of result.json, result.srt, result.vtt, and result.txt.
• Segment and word timestamps for supported languages.
• Separation of utterances into a specified number of speakers.
• Automatic container restart after a VPS reboot.

What the Processing Flow Looks Like

1. You upload the source file to the /opt/whisperx/data/inbox directory via SFTP, SCP, rsync, or a separate upload form.
2. The client sends the filename, language, and model parameters to the API.
3. FastAPI launches a separate WhisperX process inside the container.
4. WhisperX extracts the audio track through FFmpeg, recognizes the speech, and obtains the segments.
5. The alignment model refines word boundaries if an alignment model is available for the language.
6. pyannote.audio performs diarization and associates time intervals with speakers.
7. The API saves the finished artifacts to the task directory and provides links to them through internal file serving.

Cloud Services vs. Self-Hosted: What to Choose

A self-hosted option is especially justified when recordings contain trade secrets, personal data, or medical or legal information. It is also cost-effective for regularly processing dozens of hours of content per month. However, the server cannot be left unattended: you need to monitor disk usage, update images, and control access to the source recordings.

Important: diarization is not personal identification. The SPEAKER_00 label means “the same voice in this recording,” not a specific person. Named recognition requires separate models and a lawful basis for processing biometric data.

What VPS Configuration Is Needed for This Task

The main resource for WhisperX is not the CPU but the GPU video memory. The service works on a CPU, but processing long files will be slow, and diarization places an additional load on memory. A CPU VPS is suitable for occasional short transcriptions; for podcasts, video lectures, and task queues, choose a GPU VPS with an NVIDIA GPU and CUDA access.

For the basic setup described, a GPU VPS with 8–12 vCPUs, 32 GB of RAM, an NVMe disk of at least 300 GB, and an NVIDIA GPU with 16 GB of VRAM is a sensible choice. When choosing, you can select a VPS with the specified characteristics if the configuration explicitly lists the graphics card model, VRAM capacity, CUDA support, and the ability to use the GPU from Docker.

Model Selection and Its Impact on Resources

The listed values are approximate: actual memory usage is affected by batch size, compute type, recording length, diarization, and library versions. GPUs typically use compute_type=float16. On a CPU, use int8; otherwise, processing will become noticeably slower and may not fit in RAM.

When a VPS Is Not Enough

A dedicated server is needed when the GPU cannot be passed through to the virtual machine, guaranteed performance without interference from neighboring tenants is required, continuous tasks or processing of confidential archives measured in terabytes are planned. It is also justified when using multiple GPUs and processing dozens of files in parallel. For one or two sequential tasks, a GPU VPS is usually simpler, cheaper, and easier to scale by changing the plan.

Server Location

Location primarily affects the upload latency for source files, the jurisdiction governing personal data, and the team’s access speed to results. If videos are recorded in Europe and contain data belonging to EU customers, it is reasonable to choose a European data center and determine the retention period in advance. For files sized 5–20 GB, bandwidth matters more than latency: you need a real inbound and outbound connection of at least 1 Gbit/s, or close to it, without aggressive traffic limits.

Server Preparation

The instructions below assume a fresh Ubuntu Server 24.04 LTS installation. In 2026, this is a stable and convenient option for Docker, NVIDIA Container Toolkit, and Caddy. Perform the initial steps as the user who obtained access after provisioning. Do not expose the API before configuring SSH keys and the firewall.

Update the System and Install Basic Utilities

sudo apt update && sudo apt upgrade -y
sudo apt install -y ca-certificates curl gnupg git jq ufw fail2ban \
  unattended-upgrades rsync python3-venv

The first command installs the latest security patches; the second adds utilities for installing Docker, diagnostics, the firewall, and copying backups.

Create a Separate Administrator

sudo adduser deploy
sudo usermod -aG sudo deploy
sudo mkdir -p /home/deploy/.ssh
sudo chmod 700 /home/deploy/.ssh

The deploy user will perform administrative actions through sudo. Add your public SSH key to the /home/deploy/.ssh/authorized_keys file and set its permissions to 600.

sudo nano /home/deploy/.ssh/authorized_keys
sudo chmod 600 /home/deploy/.ssh/authorized_keys
sudo chown -R deploy:deploy /home/deploy/.ssh

Insert one line containing the public key, for example one beginning with ssh-ed25519. Before disabling password login, be sure to open a second SSH session and verify that logging in as the deploy user with the key works.

Disable Password-Based SSH Access

sudo tee /etc/ssh/sshd_config.d/99-hardening.conf > /dev/null <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
X11Forwarding no
MaxAuthTries 3
EOF
sudo sshd -t && sudo systemctl reload ssh

The command checks the SSH configuration syntax before applying it. If you make a mistake and immediately restart SSH, you may lose access to the server, so do not close the current session until you have successfully verified login in a new one.

Configure UFW and Fail2ban

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable
sudo systemctl enable --now fail2ban

Only SSH, HTTP, and HTTPS are open. The internal API port 8000 is not exposed externally: Caddy will access it through the Docker local network. You can check the rules with the sudo ufw status verbose command.

Prepare DNS

Create an A record, for example transcribe.example.com, pointing to the VPS’s public IPv4 address. If you use IPv6, add an AAAA record only when the IPv6 firewall is configured correctly. Before launching Caddy, make sure that the dig +short transcribe.example.com command returns your server’s IP; otherwise, automatic certificate acquisition will not work.

Check the GPU Before Installing the Application

If you selected a GPU VPS, the NVIDIA driver is usually already installed in the provider’s image. Perform a check:

nvidia-smi

The output should show the driver version, the supported CUDA version, and the graphics card. If the command is not found or shows a driver communication error, fix this first in the VPS control panel or install a suitable NVIDIA driver for Ubuntu. Do not proceed to containers until nvidia-smi works on the host.

Software Installation — Step by Step

Docker Engine 27+ or a newer compatible release, Docker Compose v2, and NVIDIA Container Toolkit will be used to isolate dependencies. WhisperX heavily depends on PyTorch, CUDA, FFmpeg, CTranslate2, and pyannote.audio; the container eliminates conflicts between system packages.

Install Docker Engine from the official repository

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

This block adds the signing key for the official Docker repository. Do not install the legacy docker.io package alongside Docker Engine from the official repository.

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

The command connects the repository and installs Docker Engine, Buildx, and Compose. Verify the installation:

sudo docker run --rm hello-world
sudo usermod -aG docker deploy

The first run will download a test image and confirm that the daemon is working. After adding the user to the docker group, log out of SSH and log back in. Membership in this group grants privileges close to root, so add only administrators to it.

Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

This block adds the official NVIDIA Container Toolkit repository. It allows a Docker container to use the GPU driver installed on the host system.

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

After configuring the runtime, test GPU passthrough into the container. The CUDA tag below is an example of a compatible base image; if issues arise, compare its version with the driver version from nvidia-smi.

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi

If your graphics card is visible in the container output, the GPU is ready for WhisperX. On a CPU-only VPS, skip this step and set DEVICE=cpu and COMPUTE_TYPE=int8 in the configuration below.

Create the project structure

sudo mkdir -p /opt/whisperx/{app,data/inbox,data/jobs,models,caddy}
sudo chown -R deploy:deploy /opt/whisperx
cd /opt/whisperx

The data/inbox directory stores input files, data/jobs stores results, and models stores downloaded models. Models must be kept on a persistent volume; otherwise, they will be downloaded again when the container is recreated.

Create a Dockerfile for the API and WhisperX

FROM nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04

ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
ENV PIP_NO_CACHE_DIR=1

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip python3-venv ffmpeg git ca-certificates \
    && rm -rf /var/lib/apt/lists/

RUN python3 -m pip install --break-system-packages \
    "torch==2.6.0" "torchaudio==2.6.0" \
    --index-url https://download.pytorch.org/whl/cu126

RUN python3 -m pip install --break-system-packages \
    "whisperx==3.3.1" "fastapi==0.115.8" \
    "uvicorn[standard]==0.34.0" "python-multipart==0.0.20"

WORKDIR /app
COPY app/ /app/
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

mkdir -p /opt/whisperx/app
nano /opt/whisperx/app/main.py

import json
import os
import subprocess
import uuid
from pathlib import Path

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

app = FastAPI(title="WhisperX API", version="1.0")

INBOX = Path("/data/inbox").resolve()
JOBS = Path("/data/jobs").resolve()
DEVICE = os.getenv("DEVICE", "cuda")
COMPUTE_TYPE = os.getenv("COMPUTE_TYPE", "float16")
DEFAULT_MODEL = os.getenv("WHISPER_MODEL", "large-v3")
HF_TOKEN = os.getenv("HF_TOKEN", "")

class TranscribeRequest(BaseModel):
    filename: str = Field(pattern=r"^[A-Za-z0-9._ -]+$")
    language: str = Field(default="ru", min_length=2, max_length=5)
    model: str = Field(default=DEFAULT_MODEL)
    min_speakers: int | None = Field(default=None, ge=1, le=20)
    max_speakers: int | None = Field(default=None, ge=1, le=20)

@app.get("/health")
def health():
    return {"status": "ok", "device": DEVICE, "model": DEFAULT_MODEL}

@app.post("/transcribe")
def transcribe(payload: TranscribeRequest):
    source = (INBOX / payload.filename).resolve()
    if INBOX not in source.parents or not source.is_file():
        raise HTTPException(status_code=404, detail="Файл не найден в inbox")

    job_id = str(uuid.uuid4())
    output_dir = JOBS / job_id
    output_dir.mkdir(parents=True, exist_ok=False)

    command = [
        "whisperx", str(source),
        "--model", payload.model,
        "--language", payload.language,
        "--device", DEVICE,
        "--compute_type", COMPUTE_TYPE,
        "--output_dir", str(output_dir),
        "--output_format", "all",
    ]

    if HF_TOKEN:
        command.extend(["--diarize", "--hf_token", HF_TOKEN])

    if payload.min_speakers:
        command.extend(["--min_speakers", str(payload.min_speakers)])
    if payload.max_speakers:
        command.extend(["--max_speakers", str(payload.max_speakers)])

    completed = subprocess.run(
        command, text=True, stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT, timeout=14400
    )

    (output_dir / "whisperx.log").write_text(
        completed.stdout, encoding="utf-8"
    )

    if completed.returncode != 0:
        raise HTTPException(
            status_code=500,
            detail={"job_id": job_id, "log": completed.stdout[-3000:]}
        )

    files = [item.name for item in output_dir.iterdir() if item.is_file()]
    return {"job_id": job_id, "status": "done", "files": files}

sudo apt install -y restic
sudo mkdir -p /root/.config/restic
sudo chmod 700 /root/.config/restic
sudo nano /root/.config/restic/whisperx.env
sudo chmod 600 /root/.config/restic/whisperx.env

RESTIC_REPOSITORY=s3:https://s3.example.net/whisperx-backups
RESTIC_PASSWORD=long_random_repository_password
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
AWS_DEFAULT_REGION=us-east-1

sudo bash -c 'source /root/.config/restic/whisperx.env && restic init'

sudo nano /usr/local/sbin/backup-whisperx.sh
sudo chmod 700 /usr/local/sbin/backup-whisperx.sh

#!/usr/bin/env bash
set -euo pipefail

source /root/.config/restic/whisperx.env

restic backup /opt/whisperx \
  --exclude='/opt/whisperx/models' \
  --exclude='/opt/whisperx/data/inbox/.tmp' \
  --tag whisperx

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check

The script excludes the model cache: it can be downloaded again when needed and often occupies tens of gigabytes. If you have a slow internet connection or fast disaster recovery is critical, remove the models exclusion, but estimate storage and traffic costs in advance.

sudo crontab -e

20 3   * /usr/local/sbin/backup-whisperx.sh >> /var/log/backup-whisperx.log 2>&1

The task runs daily at 03:20. Be sure to test recovery once a month in a separate directory or on a test VPS:

sudo bash -c 'source /root/.config/restic/whisperx.env && \
  restic restore latest --target /tmp/whisperx-restore-test'

Updates without surprises

A rolling update is usually acceptable for Caddy: change the pinned image tag, run docker compose pull caddy, and then docker compose up -d. Update WhisperX, PyTorch, CUDA, or pyannote.audio during a maintenance window. These components may change model requirements, argument formats, and VRAM consumption.

1. Back up the project and record the current versions: docker compose images.
2. Create a copy of the directory on a test server or a separate project branch.
3. Build the new image and process a reference recording in Russian and in the languages you need.
4. Compare quality, speed, the presence of speaker labels, and the JSON format.
5. Only update production after verification, and retain the ability to roll back to the previous image tag.

Monitoring disk, GPU, and logs

df -h /opt/whisperx
docker system df
nvidia-smi
docker compose logs --since=24h whisperx | tail -n 200

Keep at least 20% free space on NVMe: temporary audio files, models, and results can sharply increase disk usage. Once a week, delete unnecessary source files and old results according to the approved retention period. Do not blindly use docker system prune -a in production: the command may remove images needed for a quick rollback.

Troubleshooting + FAQ

Why does the container report “could not select device driver nvidia”?

The error means Docker cannot see the NVIDIA runtime. First check nvidia-smi on the VPS itself: without a working driver, the container cannot fix anything. Then run dpkg -l | grep nvidia-container-toolkit and repeat the configuration with sudo nvidia-ctk runtime configure --runtime=docker, then restart Docker. The check docker run --rm --gpus all ... nvidia-smi must work before launching WhisperX.

Why does CUDA out of memory occur when launching large-v3?

There is not enough video memory for the model, batches, alignment, or diarization. First check usage with nvidia-smi and stop other GPU processes. Then reduce the model to medium, use compute_type=float16, and do not run multiple transcriptions in parallel. If large-v3 quality is required, you need a GPU with more VRAM, usually 16–24 GB depending on the workload.

Diarization does not start or an error appears when accessing the pyannote model

Check that HF_TOKEN is set in .env, does not contain extra quotes, and that the container was recreated after changing the file: docker compose up -d --force-recreate whisperx. The token must have read permission, and the account must have accepted the access terms for pyannote gated models. Check the full task log in whisperx.log: it usually indicates the exact reason for the denial.

Why are there no precise word timestamps in the result?

WhisperX adds word-level timestamps through a separate alignment model, but it is not available for every language and depends on audio quality. Check whether the language parameter is passed correctly: use ru for Russian, not an arbitrary language name. In noisy recordings with music, overlapping voices, or a poor microphone, word boundaries may be omitted. Segment timestamps are usually still retained.

The API returns 502 Bad Gateway through Caddy

Status code 502 means Caddy did not receive a valid response from the API container. Check the status with docker compose ps and the logs with docker compose logs whisperx. A common cause is that the container exited because of a Python error or insufficient RAM or VRAM at startup. Also check that the Caddyfile specifies the whisperx:8000 host matching the Compose service name, rather than an external IP address.

What is the minimum suitable VPS configuration?

The minimum for experimentation is 8 vCPU, 16 GB RAM, and 150 GB NVMe without a GPU. On such a machine, use small, DEVICE=cpu, and COMPUTE_TYPE=int8; processing one long recording may take longer than its actual duration. For comfortable regular work with diarization, the practical minimum is 8 vCPU, 32 GB RAM, 300 GB NVMe, and an NVIDIA GPU with 12–16 GB VRAM.

What should I choose for this task — VPS or dedicated?

A GPU VPS is suitable for a personal service, a small team, and variable load: it is faster to deploy and usually easier to scale. Choose dedicated if you need guaranteed performance, continuous processing, multiple GPUs, large local archives, or strict isolation requirements. More important than the rental format are an NVIDIA GPU, sufficient VRAM, an NVMe disk, and the ability to use GPUs inside Docker containers.

Why is transcription too slow?

First make sure the application is actually using the GPU: the /health response should contain device: cuda, and during a task the nvidia-smi command should show a Python process. If the CPU is used, check the Docker GPU runtime settings. On a GPU, speed also depends on the model, input file quality, and diarization. Use medium for a draft pass, and run large-v3 only for the final text.

Conclusions and next steps

Your own WhisperX service is now running on the VPS: it accepts locally uploaded recordings and creates transcripts, subtitles, timestamps, and speaker labels. Access is protected with HTTPS and Basic Auth, while configuration, results, and secrets can be copied to external storage through restic.

1. Add a Redis task queue and separate worker containers if you need to process multiple files without long HTTP requests.
2. Create a simple upload web form with file size limits, a task log, and automatic source file cleanup.
3. Test the medium and large-v3 models on your recordings, measure speed, and choose the balance between quality, VRAM, and infrastructure cost.