Missing static files on fresh install

[maitredede]

Hello,

I have deployed a fresh install of wger with :

• chart version 0.2.4
• wget latest (at date of writing, may be 2.3)

In my values file, I have activated storage and nginx, and set my storageClass.

When it was all ready, I browsed to my url, but all static files went 404.

To fix it, I have executed in the pod : ./manager.py collect-static -v 2 --no-input (the --clear returned an error due to volume permissions).

Since it is a time-consuming operation, I think it may not be added as an init container, but as a helm hook.

Or have I missed something ?

[maitredede]

Looks like it is due to the --clear argument on application startup, with permissions on lost+found directory.

A workaround fr this would be by using a single PVC, with volumeMounts.subPath. So the lost+found would be outside the mounted directory. But this would be hard to upgrade existing installations without reinstalling.

[bbkz]

Hi,

thank you for the report. Normally the static files are collected on every start of the container, see the entrypoint.sh of the containerfile:

• https://github.com/wger-project/wger/blob/b2d5b2a78413b218c95063156e774c024576f587/extras/docker/production/entrypoint.sh#L24

But this will not be called when DJANGO_DEBUG is set, so maybe you have defined that?

As you discovered as well starting the container can take quite some time, especially for the first time as it also has to setup the whole database.

As for the lost+found, then you must be using a ext4 filesystem? I am using a cephfs storage backend in my cluster, so my volumes don't have this issue... what storage solution do you use?

Yours,
bbk

[bbkz]

Oh.. i just checked what i do in the helm chart is:

  - name: DJANGO_DEBUG
    {{- if .Values.app.nginx.enabled }}
    value: "False"
    {{- else }}
    value: "True"
    {{- end }}

So app.nginx.enable must be set in your values.yaml and don't override the DJANGO_DEBUG.

[maitredede]

I am using https://longhorn.io/ as volume provider. I have also installed https://github.com/topolvm/topolvm and https://rook.io/ (for ceph orchestration, but I have not yet configured a ceph cluster).

The lost+found folder would be present on ext4 volumes, that is the case for longhorn, topolvm, and ceph RBD. And on many clusters I think.

The value for app.nginx.enabled is true in my values file :

app:
  timezone: "UTC"
  persistence:
    enabled: true
    storageClass: longhorn
    accessModes:
      - ReadWriteMany
    sizeMedia: 8Gi
    sizeStatic: 8Gi
    sizeCeleryBeat: 1Gi
  resources:
    requests:
      memory: 512Mi
      cpu: 100m
    limits:
      memory: 1Gi
      cpu: 500m
  nginx:
    enabled: true
    image: nginx:stable
    imagePullPolicy: Always
  mail:
    enabled: false

ingress:
  enabled: true
  ingressClassName: nginx
  url: workout.local
  tls: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
postgres:
  enabled: true
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  storage:
    requestedSize: 10Gi
    className: longhorn

redis:
  storage:
    requestedSize: 10Gi
    className: longhorn

celery:
  enabled: true
  flower:
    enabled: true

[bbkz]

Thank you for the information. I'll try to setup a environment reproducing the error. From your values.yaml the entrypoint.sh should run

python3 manage.py collectstatic --no-input

on the first startup, without the --clear parameter (empty).

[bbkz]

I'm just collecting some more information about this topic... here are some of my findings:

• [Question] is it possible to initialize pvc without lost+found directory? longhorn/longhorn#1580
  • Longhorn supports the fstype xfs to workaround this issue, what i read between the lines is that other applications will suffer the same issue
• https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/values.yaml#L520
  • with ceph-csi-rbd you can pass the fstype parameter to choose xfs, i guess you can do that with rook as well

So meanwhile a workaround can be to choose xfs instead of ext4.

If i get it right the only way to reproduce this is using longhorn or something similar in k8s, that makes is possible to use blockdevices with RWX?

[maitredede]

There are multiple access modes that can be used : https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

But it depends on the driver. A drivers list : https://kubernetes-csi.github.io/docs/drivers.html

I don't think the volume mode parameter is usefull in our case : https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-mode

Since default value of volumeMode is filesystem, the argument fstype make sense here, and is by default (in most cases) ext4.

A workaround using fstype: xfs can be a solution, but having a single pvc/pv with multiple subPath can be an other solution, so you have only one volume to handle. It may be less fine-grained than your helm chart, but it is a proposal 😃.

Also I have an other related question... Why is a volume required to hold static files ? Can't they be generated in container image ? Since like the name said, it should be static...

[maitredede]

Well, I have looked at the https://github.com/wger-project/wger/blob/master/extras/docker/production/Dockerfile. It is not really built to fully use kubernetes components 😃

I will try to make an alternative container image and helm chart. I will be focusing on security (readonly file system), pregenerated static files, other volume management...

[bbkz]

It is alway nice to learn more ;-) normally a blockdevice can only be mounted once at a time from the kernel, so longhorn is some kind of wrapper around that and makes one able to use it as a filesystem (thus RWX)?

The static files is a django thing. Django comes a long way it has a separation of static files (css, js, pics,...) the media files (content, user uploaded stuff) and the application code.

Where as the python code of django is interpreted by gunicorn and accessed by one or more webservers the static and media files are served directly from the webserver(s) but generated by the django code.

[bbkz]

I'm really looking forward to have it lifted to the next level!

[rolandgeider]

Where as the python code of django is interpreted by gunicorn and accessed by one or more webservers the static and media files are served directly from the webserver(s) but generated by the django code.

the static files are not really pregenerated, django just collects them to one place (this could also be a CDN or another server somewhere else), it's then the responsibility of the web server to actually serve them.

But I'm of course with bbkz, if you have suggestions for improving the docker file, we want to hear them :)

[maitredede]

I have started something here : https://github.com/maitredede/wger-wger/tree/feature/docker-rework

Based on production docker image, the objective is to have the image filesystem mounted as readonly, so static files have to be generated at build time. I have used docker multi-staged builds to :

• move yarn/sass compilation to its own stage
• removed rust, cargo, yarn and sassc from build
  I have also built a docker-compose file with :
• bitnami containers for postgresql and redis (accent is put on security, helm charts are ready and powerfull for kubernetes installations)
• I have added a catchall mail server : emails can be sent with smtp, but never goes online, and there is a webmail to get them (or using imap)
• some tasks are moved outside main web application : migration, exercises sync (with photo/video), online fixture and ingredients...

For kubernetes installation, I have started with a Kustomize file (I will craft a helm chart later). External data sync tasks will be run as cronjob.

I still have to figure out some bugs I have (I have http 500 errors, I can't load online fixtures due to duplicate key value violates unique constraint...).

This is a work in progress...