Set Up LocalAI on AlmaLinux 10 with Docker Compose

LocalAI is an open-source AI runtime that exposes an OpenAI-compatible API and built-in WebUI, which makes it useful for self-hosted assistants, private testing, offline development, and internal AI workflows that should not depend on an external inference provider.

In this guide, we deploy LocalAI on a fresh AlmaLinux 10.1 server, install Docker from Docker’s official repository, pin the current LocalAI stable image tag, keep the service bound to 127.0.0.1:8080 instead of exposing it publicly, validate the API locally, and access the WebUI through an SSH tunnel rather than forcing SSL onto a service that does not need it for a safe base installation.

Why Use LocalAI on AlmaLinux 10?

• AlmaLinux 10 provides a modern enterprise-style base with a long support window.
• LocalAI delivers an OpenAI-compatible API plus a built-in WebUI without requiring a cloud AI service.
• Docker Compose keeps the deployment reproducible and easy to update.
• Binding LocalAI to localhost only is a safer default than publishing an unauthenticated AI endpoint directly to the internet.

Before You Begin

Make sure you have the following before you start:

• A fresh AlmaLinux 10 server
• Root or sudo access
• An SSH client on your local machine for tunneling the WebUI
• Enough disk space for the LocalAI image and any models you plan to download later

1. Verify the AlmaLinux 10 Release

Start by confirming that the server is actually running AlmaLinux 10.1.

cat /etc/os-release

2. Install Docker Engine and the Compose Plugin

The LocalAI installation docs recommend a container-based deployment. On AlmaLinux 10, the cleanest path is Docker’s official repository so you get a current Engine and Compose plugin instead of relying on older or incomplete packages.

dnf -y install dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf makecache
dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

systemctl enable --now docker

docker --version
docker compose version
systemctl is-active docker

On the validated server, this installed Docker Engine 29.3.1, Docker Compose 5.1.1, and left the Docker service active after boot.

3. Deploy LocalAI 4.0.0 with Docker Compose

For a safer default setup, this guide binds LocalAI to 127.0.0.1:8080 instead of exposing it publicly. That keeps the API reachable on the server and through an SSH tunnel without adding a reverse proxy or SSL layer that the base installation does not require.

mkdir -p /opt/localai/models /opt/localai/data

cat > /opt/localai/compose.yaml <<'EOF'
services:
  localai:
    image: localai/localai:v4.0.0
    container_name: local-ai
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models
      - ./data:/data
EOF

cd /opt/localai
docker compose up -d
sleep 25

docker compose ps

On the live server, Docker pulled the official localai/localai:v4.0.0 image, created the stack successfully, and exposed the container only on the localhost listener.

4. Validate the Local API and Container Health

Before you worry about browser access, confirm that the container is healthy and the local API responds on the server itself.

cd /opt/localai
docker compose ps
docker compose images
docker inspect --format '{{.State.Health.Status}}' local-ai

for _ in $(seq 1 24); do
  if curl -fsS http://127.0.0.1:8080/readyz >/dev/null 2>&1; then
    break
  fi
  sleep 5
done

curl -fsS http://127.0.0.1:8080/readyz
printf '\n'
curl -fsS http://127.0.0.1:8080/v1/models
printf '\n'

On the validated deployment, the LocalAI container moved from starting to healthy, the readiness endpoint responded successfully, and the models endpoint returned an empty list, which is expected before you import or download any models.

5. Access the LocalAI WebUI Through an SSH Tunnel

Because this guide keeps LocalAI on localhost only, use an SSH tunnel from your own machine to reach the WebUI securely without opening port 8080 to the public internet.

ssh -N -L 18080:127.0.0.1:8080 root@YOUR_SERVER_IP

Then open the following URL in your local browser:

http://127.0.0.1:18080/

On the validated server, the tunnel exposed the LocalAI WebUI cleanly in the browser while the service remained bound to the loopback interface on the server itself.

6. Run a Final Service Check

Finish by confirming the container image tag, health status, loopback listener, and API response one more time.

cd /opt/localai
docker compose ps
docker compose images
docker inspect --format '{{.State.Health.Status}}' local-ai
ss -lntp | grep ':8080'
curl -fsS http://127.0.0.1:8080/readyz
printf '\n'
curl -fsS http://127.0.0.1:8080/v1/models
printf '\n'

On the live server, LocalAI remained healthy on 127.0.0.1:8080, the readiness endpoint responded successfully, and the API was ready for model imports or further configuration.

Hardening Notes

• This guide intentionally avoids public exposure and SSL because LocalAI does not require HTTPS for the base localhost-only install.
• If you later publish LocalAI behind a reverse proxy, add authentication and TLS before exposing it publicly.
• The empty /v1/models result is normal on a fresh install; you still need to add models after deployment.
• Model storage can grow quickly, so keep an eye on disk usage under /opt/localai/models and /opt/localai/data.

Conclusion

You now have LocalAI v4.0.0 running on AlmaLinux 10.1 with Docker Compose, bound to 127.0.0.1:8080, validated locally through the API, and accessible through an SSH tunnel for WebUI use. This gives you a clean and safer base installation without forcing a public SSL layer onto a service that does not need it for its initial deployment.

Christian Wells