Kyverno values#

Values under hw-kyverno configure the Kyverno cluster policies and the policy exceptions Hopsworks workloads need.

Generated from the Hopsworks Helm chart 5.1.0 (Hopsworks 5.1.0).

Deployed according to the first of these values that is set: global._hopsworks.kyverno.enabled, global._hopsworks.full_platform.

General#

Defaults as YAML

hw-kyverno:
  hopsworkslib: {}

hw-kyverno #: Type object, default {}. override hw-kyverno values
hw-kyverno.hopsworkslib #: Type object, default {}. override hopsworkslib values

policies#

Defaults as YAML

hw-kyverno:
  policies:
    addCertificatesVolume:
      autogenControllers: DaemonSet,Deployment,Job,StatefulSet
      cacertsConfigMap: ca-pemstore
      enabled: false
      envs: []
      extraAnnotations: []
      hopsworksProjectLabelKey: hopsworks.ai/project
      initContainers:
        extraAnnotations: []
        labels: []
      labels:
      - key: job-type
        value: check-image-exist
      - key: job-type
        value: tag
      - key: job-type
        value: list-tags
      - key: job-type
        value: docker-build
      - key: job-type
        value: delete
      - key: job-type
        value: git-command
      mountPath: ''
    prohibitHostPath:
      enabled: false

hw-kyverno.policies #

Type object. Configuration for Hopsworks Kyverno policies and exceptions

Default

addCertificatesVolume:
  autogenControllers: DaemonSet,Deployment,Job,StatefulSet
  cacertsConfigMap: ca-pemstore
  enabled: false
  envs: []
  extraAnnotations: []
  hopsworksProjectLabelKey: hopsworks.ai/project
  initContainers:
    extraAnnotations: []
    labels: []
  labels:
  - key: job-type
    value: check-image-exist
  - key: job-type
    value: tag
  - key: job-type
    value: list-tags
  - key: job-type
    value: docker-build
  - key: job-type
    value: delete
  - key: job-type
    value: git-command
  mountPath: ''
prohibitHostPath:
  enabled: false

hw-kyverno.policies.addCertificatesVolume #

Type object. Configuration to add custom certificates to pods as a mounted volume

Default

autogenControllers: DaemonSet,Deployment,Job,StatefulSet
cacertsConfigMap: ca-pemstore
enabled: false
envs: []
extraAnnotations: []
hopsworksProjectLabelKey: hopsworks.ai/project
initContainers:
  extraAnnotations: []
  labels: []
labels:
- key: job-type
  value: check-image-exist
- key: job-type
  value: tag
- key: job-type
  value: list-tags
- key: job-type
  value: docker-build
- key: job-type
  value: delete
- key: job-type
  value: git-command
mountPath: ''

hw-kyverno.policies.addCertificatesVolume.autogenControllers #

Type string, default "DaemonSet,Deployment,Job,StatefulSet". the list of controllers separated by comma to generate the policy for

hw-kyverno.policies.addCertificatesVolume.cacertsConfigMap #

Type string, default "ca-pemstore". The name of the configmap containing the certificates. The configmap should contains the ca-certificates.crt trusted by the user to replace the whole /etc/ssl/certs. It should also include the root certificate(s) if any defined for OAUTH or LDAP identity providers. Also, if using a hosted object storage with a custom CA, it should includes the java/cacerts to trust that object storage host and that require updating the hopsfs.namenode.jvmOpts = "-Djavax.net.ssl.trustStore=/etc/ssl/certs/my-cacerts -Djavax.net.ssl.trustStorePassword=changeit" and similarly for the hopsfs.datanode.jvmOpts.

hw-kyverno.policies.addCertificatesVolume.enabled #

Type bool, default false. Enable add certificates volume

hw-kyverno.policies.addCertificatesVolume.envs #

Type list, default []. The array of environment variables with their values that you would like to inject to the pods along side the certificates volume.

hw-kyverno.policies.addCertificatesVolume.extraAnnotations #

Type list, default []. The array of extra annotations to use when filtering which pods to inject the certificates volume into.

hw-kyverno.policies.addCertificatesVolume.hopsworksProjectLabelKey #

Type string, default "hopsworks.ai/project". the name of the label key to identify hopsworks user project namespaces

hw-kyverno.policies.addCertificatesVolume.initContainers #

Type object, default {"extraAnnotations":[],"labels":[]}. Opt-in for injecting the certificates volume into init containers. When enabled, a second mutate rule is rendered that targets spec.initContainers and is gated by an OR of the dedicated annotation (default kyverno-inject-certs-init=enabled), extraAnnotations, and labels configured below. Pods must opt in via at least one of these matchers.

hw-kyverno.policies.addCertificatesVolume.initContainers.extraAnnotations #

Type list, default []. Init-specific extra annotations that opt a pod's init containers into certificate volume injection. ORed with the dedicated init-container annotation and labels below. Defaults to empty so init injection is opt-in by design.

hw-kyverno.policies.addCertificatesVolume.initContainers.labels #

Type list, default []. Init-specific labels that opt a pod's init containers into certificate volume injection. ORed with the dedicated init-container annotation and extraAnnotations. Defaults to empty so third-party operator-injected init containers are not silently mutated.

hw-kyverno.policies.addCertificatesVolume.labels #

Type list. The array of labels to use when filtering which pods to inject the certificates volume into. The default list includes job-type=check-image-exist, job-type=tag, job-type=list-tags, job-type=docker-build, job-type=delete for docker operations, that is needed if using a registry with custom CA. The default list also includes job-type=git-command for git operations, that is needed if using a git host with custom CA.

Default

- key: job-type
  value: check-image-exist
- key: job-type
  value: tag
- key: job-type
  value: list-tags
- key: job-type
  value: docker-build
- key: job-type
  value: delete
- key: job-type
  value: git-command

hw-kyverno.policies.addCertificatesVolume.mountPath #

Type string, default "". Path to mount the certificates volume

hw-kyverno.policies.prohibitHostPath #

Type object, default {"enabled":false}. Configuration for disabling hostPath volumes in Pods

hw-kyverno.policies.prohibitHostPath.enabled #

Type bool, default false. Enable hostPath policy and relevant exceptions

policyExceptions#

Defaults as YAML

hw-kyverno:
  policyExceptions:
    airflow:
      enabled: true
    buildkitd:
      appLabel: buildkitd
      enabled: true
      namePrefix: buildkitd
      rootless: false
    dockerRegistryConfigurer:
      enabled: true
    filebeat:
      enabled: true
    jobs:
      enabled: true
    jupyter:
      enabled: true
    knativeDryRun:
      enabled: true
    opensearch:
      enabled: true
    prometheusNodeExporter:
      enabled: true
    pythonDeployment:
      enabled: true
    pythonapp:
      enabled: true
    rondb:
      enabled: true
    spark:
      enabled: true
      rssAppName: rss-hops
    systemJobs:
      enabled: true
    terminal:
      enabled: true
    trino:
      enabled: true
    vllm:
      enabled: true

hw-kyverno.policyExceptions #

Type object. Configuration for PolicyExceptions to exempt specific services from Kyverno PSS Restricted policies

Default

airflow:
  enabled: true
buildkitd:
  appLabel: buildkitd
  enabled: true
  namePrefix: buildkitd
  rootless: false
dockerRegistryConfigurer:
  enabled: true
filebeat:
  enabled: true
jobs:
  enabled: true
jupyter:
  enabled: true
knativeDryRun:
  enabled: true
opensearch:
  enabled: true
prometheusNodeExporter:
  enabled: true
pythonDeployment:
  enabled: true
pythonapp:
  enabled: true
rondb:
  enabled: true
spark:
  enabled: true
  rssAppName: rss-hops
systemJobs:
  enabled: true
terminal:
  enabled: true
trino:
  enabled: true
vllm:
  enabled: true

hw-kyverno.policyExceptions.airflow #

Type object, default {"enabled":true}. PolicyException for airflow-scheduler Deployment. Airflow has a mount-airflow-folders sidecar that requires privileged access for FUSE mounting of HopsFS. This exception is enabled by default and only activates when both the airflow service and hw-kyverno are enabled.

hw-kyverno.policyExceptions.airflow.enabled #

Type bool, default true. Enable PolicyException for airflow-scheduler. Set to false to disable even when airflow service is enabled.

hw-kyverno.policyExceptions.buildkitd #

Type object. PolicyException for the persistent BuildKit daemon (global._hopsworks.buildkitd.enabled). The system-jobs exception matches Jobs by job-type label and so never reaches this StatefulSet, which would then be rejected on a Kyverno cluster.

Default

appLabel: buildkitd
enabled: true
namePrefix: buildkitd
rootless: false

hw-kyverno.policyExceptions.buildkitd.appLabel #

Type string, default "buildkitd". Pod label the exception matches, together with the name prefix below. Tracks hopsworks.buildkitd.name, which is what the StatefulSet sets as its app label.

hw-kyverno.policyExceptions.buildkitd.enabled #

Type bool, default true. Enable PolicyException for the persistent BuildKit daemon. Also gated on global._hopsworks.buildkitd.enabled, which is what turns the daemon itself on, so this defaults true without becoming a standing grant: the exception renders only where the daemon does. Turning it off on a Kyverno cluster that runs the daemon gets it rejected at admission, since the exception grants the rootful union (privileged, host namespaces, uid 0).

hw-kyverno.policyExceptions.buildkitd.namePrefix #

Type string, default "buildkitd". Name prefix the exception matches, so the grant is not reachable by anything that merely wears the app label. StatefulSet pods are -.

hw-kyverno.policyExceptions.buildkitd.rootless #

Type bool, default false. Grant only the policies a rootless daemon needs, dropping the privileged-container and host-namespace exceptions. Defaults false, which grants the union, because a rootful daemon set to true is rejected at admission whereas a rootless daemon set to false merely carries two exceptions it does not use. Set true alongside hopsworks.buildkitd.rootless.enabled.

hw-kyverno.policyExceptions.dockerRegistryConfigurer #

Type object, default {"enabled":true}. PolicyException for docker-registry-configurer DaemonSet. This service requires privileged access, hostPID, hostNetwork, and hostPath to configure container runtimes on nodes.

hw-kyverno.policyExceptions.dockerRegistryConfigurer.enabled #

Type bool, default true. Enable PolicyException for docker-registry-configurer

hw-kyverno.policyExceptions.filebeat #

Type object, default {"enabled":true}. PolicyException for filebeat DaemonSet. Filebeat requires hostNetwork, hostPath volumes, and runs as root to collect logs from all nodes.

hw-kyverno.policyExceptions.filebeat.enabled #

Type bool, default true. Enable PolicyException for filebeat

hw-kyverno.policyExceptions.jobs #

Type object, default {"enabled":true}. PolicyException for user jobs. These pods contain a hopsfsmount sidecar that requires privileged access for FUSE mounting of HopsFS.

hw-kyverno.policyExceptions.jobs.enabled #

Type bool, default true. Enable PolicyException for user jobs

hw-kyverno.policyExceptions.jupyter #

Type object, default {"enabled":true}. PolicyException for Jupyter server deployments. These pods contain a hopsfsmount sidecar that requires privileged access for FUSE mounting of HopsFS.

hw-kyverno.policyExceptions.jupyter.enabled #

Type bool, default true. Enable PolicyException for Jupyter server deployments

hw-kyverno.policyExceptions.knativeDryRun #

Type object, default {"enabled":true}. PolicyException for the throwaway Pod that Knative's webhook dry-run creates to validate a revision's pod spec. It carries no serving labels, so the label-scoped serving exceptions cannot match it, and a hopsfsmount FUSE sidecar makes it fail the restricted policies.

hw-kyverno.policyExceptions.knativeDryRun.enabled #

Type bool, default true. Enable PolicyException for Knative pod-spec dry-run validation

hw-kyverno.policyExceptions.opensearch #

Type object, default {"enabled":true}. PolicyException for opensearch StatefulSet. OpenSearch requires a privileged init container to configure vm.max_map_count kernel parameter. Only needed when olk.opensearch.setVMMaxMapCount is true.

hw-kyverno.policyExceptions.opensearch.enabled #

Type bool, default true. Enable PolicyException for opensearch

hw-kyverno.policyExceptions.prometheusNodeExporter #

Type object, default {"enabled":true}. PolicyException for prometheus-node-exporter DaemonSet. Node exporter requires hostNetwork and hostPath to collect node-level metrics.

hw-kyverno.policyExceptions.prometheusNodeExporter.enabled #

Type bool, default true. Enable PolicyException for prometheus-node-exporter

hw-kyverno.policyExceptions.pythonDeployment #

Type object, default {"enabled":true}. PolicyException for Python (model-server: python) KServe serving deployments. Agent deployments inject a root, privileged hopsfsmount FUSE sidecar (HWORKS-2871) that does not meet the restricted policy requirements.

hw-kyverno.policyExceptions.pythonDeployment.enabled #

Type bool, default true. Enable PolicyException for Python model serving deployments

hw-kyverno.policyExceptions.pythonapp #

Type object, default {"enabled":true}. PolicyException for Python app deployments (custom apps, Streamlit, Gradio). These pods contain a hopsfsmount sidecar that requires privileged access for FUSE mounting of HopsFS.

hw-kyverno.policyExceptions.pythonapp.enabled #

Type bool, default true. Enable PolicyException for Python app deployments

hw-kyverno.policyExceptions.rondb #

Type object, default {"enabled":true}. PolicyException for RonDB mysqlds StatefulSet. Mysqlds uses the SYS_NICE capability for process scheduling priority tuning. Only needed when rondb.rondb.meta.mysqld.addSysNiceCapability is true.

hw-kyverno.policyExceptions.rondb.enabled #

Type bool, default true. Enable PolicyException for RonDB mysqlds

hw-kyverno.policyExceptions.spark #

Type object, default {"enabled":true,"rssAppName":"rss-hops"}. PolicyException for Spark-related resources including: (1) Spark driver and executor pods created by spark-operator - these pods have container-level security contexts but lack pod-level security context support in older spark-operator versions, (2) RSS (Remote Shuffle Service) coordinator and shuffle server Deployments/StatefulSets - these are dynamically created by the Uniffle controller with security contexts configured via CRD spec, and (3) the spark-operator Helm hook Job that applies CRDs on install/upgrade - the upstream chart hardcodes its securityContext without runAsNonRoot/seccompProfile and exposes no values to set them.

hw-kyverno.policyExceptions.spark.enabled #

Type bool, default true. Enable PolicyException for Spark-related resources (spark-operator driver/executor pods, RSS coordinator/shuffle server Deployments/StatefulSets, and the spark-operator CRD upgrade hook Job)

hw-kyverno.policyExceptions.spark.rssAppName #

Type string, default "rss-hops". The app name of the RemoteShuffleService resource

hw-kyverno.policyExceptions.systemJobs #

Type object, default {"enabled":true}. PolicyException for Hopsworks system jobs including docker operations (docker-build, check-image-exist, tag, delete, list-tags), image validation (check-image), and conda library operations (list-libraries, export-libraries, conda-search-libraries). These jobs require privileged access to run buildkit/podman for building container images.

hw-kyverno.policyExceptions.systemJobs.enabled #

Type bool, default true. Enable PolicyException for Hopsworks system jobs

hw-kyverno.policyExceptions.terminal #

Type object, default {"enabled":true}. PolicyException for terminal server deployments. These pods contain a hopsfsmount sidecar that requires privileged access for FUSE mounting of HopsFS.

hw-kyverno.policyExceptions.terminal.enabled #

Type bool, default true. Enable PolicyException for terminal server deployments

hw-kyverno.policyExceptions.trino #

Type object, default {"enabled":true}. PolicyException for the Trino coordinator, workers and test coordinator. Their hopsfs-mount sidecar FUSE-mounts the credential-file store, which needs /dev/fuse and Bidirectional mount propagation, so it runs privileged and as root. Only rendered when the mountable-secret store is enabled, so a cluster without it grants no exception.

hw-kyverno.policyExceptions.trino.enabled #

Type bool, default true. Enable PolicyException for trino

hw-kyverno.policyExceptions.vllm #

Type object, default {"enabled":true}. PolicyException for vLLM model serving deployments created by KServe. These pods are deployed with default KServe/vLLM configurations that may not meet all restricted policy requirements.

hw-kyverno.policyExceptions.vllm.enabled #

Type bool, default true. Enable PolicyException for vLLM model serving deployments