dont cap grace period at 0 for soft evictions

[cjroebuck]

the following flag is currently defaulted on the kubelet: --eviction-max-pod-grace-period="0" which renders any terminationGracePeriodSeconds set on the pod spec useless when said pod is (softly) evicted.

Would be nice to set this to a larger period (300s / 5m?) in order to support pods that need to terminate gracefully when handling long running connections or processes.

Pods that are evicted due to 'soft' thresholds being met are effectively being hard-killed which doesn't seem right to me.

[timoreimann]

@cjroebuck 👋

Being able to use soft evictions in DOKS sounds like a reasonable request. Apart from the grace period itself, I believe the major challenge is to figure out what the right thresholds are for the individual resource types (memory, disk, etc.). Ideally, we want to be able to use a default value that works for most (all?) clusters.

[timoreimann]

@cjroebuck would it be possible for you to set resource requests on your affected workload(s) high enough so that pod evictions do not even become an issue in the first place?