byeCloud: GitLab with Docker and Traefik

published on 02. November 2018 in category byeCloud , Tags: byecloud selfhosted docker gitlab traefik

For some months now I’m running a private GitLab server. I really enjoy using it, especially with all the great features like the Docker Container Registry and GitLab Pages to host static pages, even with own domains. Normally I would prefer a more lightweight solution, such as Gitea but GitLab has so many advantages that, at least for me, this is currently the only way to go. However, it felt tedious setting it up, even with Docker. You have to configure GitLab to serve stuff using plain HTTP, provide different ports for different apps to be able to create own vhosts in the reverse proxy and so on. So I decided to quickly write up what I did to get it working. Maybe I’m wrong and there’s a much easier way to do it but I couldn’t find it. Additionally, in the meanwhile I switched from Caddy as a reverse proxy to Traefik since it can attach directly to the Docker daemon, listens to changes in the domain configuration, request new HTTPS certificates on the fly while new containers are spawned and - best of all - I don’t need a separate configuration file. So the guide this time is still using Docker and docker-compose, but Traefik instead of Caddy. But you can basically use any reverse proxy to set it up, like nginx-proxy.

Compose setup

With this configuration I am able to run GitLab, the Container Registry and GitLab Pages on different subdomains with one IP address, each with it’s own certificate off-loaded by Traefik. Additionally I have one GitLab Runner to run CI jobs from GitLab (actually I have multiple on different hosts now).

First off, here’s my docker-compose.yml. Down below I’ll explain a little more about my GITLAB_OMNIBUS_CONFIG:

version: '3'
services:
  traefik:
    image: traefik:latest
    restart: always
    command: --docker
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/etc/traefik/traefik.toml
      - ./acme.json:/acme.json

  gitlab:
    image: gitlab/gitlab-ce:latest
    restart: always
    hostname: gitlab.example.com
    # I had problems with the health check. Sometimes it reported unhealthyness and therefore Traefik removed
    # the container, so I turned it off. Maybe it works by now.
    healthcheck:
      disable: true
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        nginx['http2_enabled'] = false

        nginx['proxy_set_headers'] = {
          "Host" => "$$http_host",
          "X-Real-IP" => "$$remote_addr",
          "X-Forwarded-For" => "$$proxy_add_x_forwarded_for",
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }

        gitlab_rails['gitlab_shell_ssh_port'] = 22

        registry_external_url 'https://registry.gitlab.example.com'
        registry_nginx['listen_port'] = 5100
        registry_nginx['listen_https'] = false

        registry_nginx['proxy_set_headers'] = {
          "Host" => "$$http_host",
          "X-Real-IP" => "$$remote_addr",
          "X-Forwarded-For" => "$$proxy_add_x_forwarded_for",
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }

        pages_external_url 'https://pages.gitlab.example.com'
        pages_nginx['listen_port'] = 5200
        pages_nginx['listen_https'] = false

        pages_nginx['proxy_set_headers'] = {
          "Host" => "$$http_host",
          "X-Real-IP" => "$$remote_addr",
          "X-Forwarded-For" => "$$proxy_add_x_forwarded_for",
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }

        gitlab_pages['inplace_chroot'] = true
        gitlab_pages['external_http'] = ['gitlab:5201']        
    volumes:
      - gitlab-config:/etc/gitlab
      - gitlab-logs:/var/log/gitlab
      - gitlab-data:/var/opt/gitlab
    ports:
      # Feel free to map this to a different port if that one is in use already
      - "22:22"
    labels:
      - traefik.enable=true
      # Host settings for GitLab itself
      - traefik.gitlab.frontend.rule=Host:gitlab.example.com
      - traefik.gitlab.port=80
      # Host settings for the registry
      - traefik.registry.frontend.rule=Host:registry.gitlab.example.com
      - traefik.registry.port=5100
      # Host settings for GitLab pages. Since I don't have a wildcard certificate, I list every domain on it's own here
      - traefik.pages.frontend.rule=Host:pages.gitlab.example.com,username.pages.gitlab.example.com
      - traefik.pages.port=5201

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    restart: always
    volumes:
      # Mount Docker socket for dind
      - /var/run/docker.sock:/var/run/docker.sock
      # We need to slightly modify the runner config, so mount it here and just create an empty file for the beginning
      # Make sure that the permissions are correct so that the container can write to it
      - ./runner.toml:/etc/gitlab-runner/config.toml

volumes:
  gitlab-config:
  gitlab-logs:
  gitlab-data:
  gitlab-runner-data:

Details I want to point out:

external_url 'https://gitlab.example.com': External URL as it will be seen by GitLab users, so with HTTPS, even if GitLab itself only serves HTTP (same for registry_external_url and pages_external_url)

gitlab_pages['inplace_chroot'] = true: Seems like when you use Docker data volumes, you need this, otherwise it shows this in the log: “Failed to bind mount /gitlab-data/shared/pages on /tmp/gitlab-pages-xyz/pages. operation not permitted”

gitlab_pages['external_http'] = ['gitlab:5201']: Tell GitLab to use an external HTTP server, like Traefik in our case, to handle custom domains. The documentation says that you’d need to point an additional IP address here that you want to use but I figured out that just using the GitLab container name works as well, so no need to purchase an additional domain :-)

Here’s my Traefik configuration, also with comments, called traefik.toml in my case:

# Show Docker events for now
logLevel = "INFO"
defaultEntryPoints = ["https", "http"]

# Force HTTPS
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]

[acme]
email = "webmaster@example.com"
# Store certificates in /acme.json
storage = "acme.json"
acmeLogging = true
onHostRule = true
onDemand = false
# Use HTTP challenge as my DNS provider does not support DNS challenge
entryPoint = "https"
  [acme.httpChallenge]
  entryPoint = "http"

[docker]
endpoint = "unix:///var/run/docker.sock"
watch = true
# Only expose services that are enabled explicitly
exposedbydefault = false

Connecting the Runner

After starting everything and setting a password for the GitLab administrator account, you can register your GitLab runner. Run the register command inside the container:

docker-compose run --rm gitlab-runner register

Choose “docker” as a runner type. You will be asked for your GitLab URL, which would be https://gitlab.example.com in our example. The runner token can be obtained from the GitLab admin area at Overview -> Runners. After the register command is done it will not work directly. Since we use Docker in Docker (our runner runs inside a Docker container and is able to use Docker on it’s own), we need to set our runner to privileged mode. Open the runner.toml file now and add the privileged = true line to your runner:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "example runner"
  url = "https://gitlab.example.com/"
  token = "123"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    # Add this line
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Afterwards just restart the container using

docker-compose restart gitlab-runner

and you should be good to go.

Setting up Pages

In the label traefik.pages.frontend.rule of my docker-compose.yml you can see that I defined a list of domains I want to use with GitLab Pages. Normally this is done using a wildcard DNS but since my DNS provider does not support Let’s Encrypt DNS Challenge and I want TLS certificates for all my pages I cannot use a wildcard subdomain. Workaround: define every domain manually. But I only use two or three domains, so it’s fine.

Afterwards, just create a repository and use the CI to build your page to static HTML. Using an artifact it can be automatically deployed to GitLab pages. In the Settings of your Project you will be able to define a custom domain for your page that needs to be pointed to port 5201 of your GitLab container, just like I did it with Traefik in the docker-compose.yml above.

At https://gitlab.com/pages/plain-html you can find an example on how to serve a plain HTML page. Another example is Hugo, which I use for my blog. Here’s my gitlab-ci.yml:

image: dettmering/hugo-build

pages:
  script:
  # Build the page
  - hugo --config config_prod.toml
  artifacts:
    paths:
    # Public the public folder to GitLab pages
    - public
  only:
  - master

As an living example, this page is actually served through GitLab pages :-)

davd.io

byeCloud: GitLab with Docker and Traefik

Table of contents

Compose setup

Connecting the Runner

Setting up Pages