Service Log Labels#

Introduction#

Filebeat attaches Kubernetes metadata to every service log document before Logstash forwards it to OpenSearch. Only an explicit set of pod, namespace and node labels is kept.

The set is bounded because every distinct label key becomes a field in the shared .services-* index mapping. OpenSearch rejects documents once an index exceeds index.mapping.total_fields.limit, which defaults to 1000 fields. A cluster that attaches every pod and node label, including the label sets that cloud providers and node feature discovery add, reaches that limit and then stops indexing service logs.

In this guide you will learn which labels are kept by default, and how to append to or replace that set.

Default pod labels#

Default namespace labels#

Filebeat collects logs from the release namespace, from namespaces carrying either of these two labels, and from any namespace listed in olk.filebeat.extraNamespaces.

Default node labels#

How labels appear in a log document#

{
  "kubernetes": {
    "labels": { "app": "namenode", "app_kubernetes_io/name": "hopsfs" },
    "namespace_labels": { "hopsworks_ai/project": "demo" },
    "node": { "labels": { "kubernetes_io/hostname": "worker-1" } }
  }
}

Dots in a label key are replaced by underscores. In OpenSearch Dashboards, search for kubernetes.labels.app_kubernetes_io/name, not kubernetes.labels.app.kubernetes.io/name.

Labels Hopsworks' log filtering reads#

Four defaults are unioned into the effective set whatever the lists below say, because losing one breaks log collection rather than degrading a field:

name and hopsworks.ai/onlinefs-cluster are read by the chart's own Filebeat configuration rather than by a pipeline. hopsworks.ai/project is read by both: the Filebeat namespace gate and the discriminator.conf pipeline, which derives the project field from it. They cost eight fields of the 1000-field budget, not four: a dynamically mapped string label is indexed as text plus a .keyword sub-field, so every key counts twice. The same doubling applies to any label you append, which halves the effective budget.

Every default is in this category: the audit of the shipped set found no key that is collected without something reading it. They live in the logFiltering* values, so the set is visible and auditable:

olk:
  filebeat:
    kubernetesMetadata:
      logFilteringPodLabels: [ ... ]
      logFilteringNamespaceLabels:
        - "hopsworks.ai/project"
        - "hopsworks.ai/onlinefs-cluster"
      logFilteringNodeLabels:
        - "kubernetes.io/hostname"

The effective set for a dimension is logFiltering* plus extra*, de-duplicated. Removing a key from logFiltering* is possible, since Helm cannot make a value read-only, and breaks whatever reads it: the four above stop log collection, and the rest stop a Logstash routing or enrichment branch from matching. A location that genuinely needs different metadata should set addKubernetesMetadata: false and supply its own processors instead.

A log location that genuinely needs different metadata should instead set addKubernetesMetadata: false and supply its own processors, as described below.

Metadata collection per log location#

Collection is switched on per entry of olk.filebeat.logs_locations:

olk:
  filebeat:
    logs_locations:
      - name: containerd
        path: /var/log/containers
        mountPaths:
          - /var/log/containers
          - /var/log/pods
        logtype: log
        glob: "/*.log"
        addKubernetesMetadata: true
        processors: []

Helm replaces lists instead of merging them, so overriding logs_locations replaces the chart's entry in full, addKubernetesMetadata included.

Set addKubernetesMetadata: false only for a location that supplies its own add_kubernetes_metadata in processors, which keeps two metadata processors from running on the same location.

Append a label#

Use the extra lists to keep the defaults and add to them:

olk:
  filebeat:
    kubernetesMetadata:
      extraPodLabels:
        - "my.corp/team"
      extraNamespaceLabels:
        - "my.corp/cost-center"
      extraNodeLabels:
        - "topology.kubernetes.io/zone"

The extra lists are appended to the defaults and de-duplicated, so repeating a default is harmless. Label keys named by olk.logstash.extendServicesPipeline are added automatically and do not need an entry here.

An appended label is stored on the log document and is searchable and aggregatable in OpenSearch Dashboards. Routing and field extraction are done by the Logstash pipelines, which read a fixed set of keys, so an appended label does not change how a log line is parsed.

Replace the filtering set#

Override logFiltering* to replace the set that Hopsworks' filtering reads. Only do this if you know what stops working:

olk:
  filebeat:
    kubernetesMetadata:
      logFilteringPodLabels:
        - "app"
        - "my.corp/team"

The logFiltering* lists hold the keys Hopsworks' own log filtering reads, so removing one silently breaks whatever reads it.

Collect annotations#

Annotations are not collected by default. Add the keys you need:

olk:
  filebeat:
    kubernetesMetadata:
      podAnnotations:
        - "my.corp/owner"
      namespaceAnnotations: []
      nodeAnnotations: []

Deployment and cron job names#

Two toggles control whether kubernetes.deployment.name and kubernetes.cronjob.name are attached:

olk:
  filebeat:
    kubernetesMetadata:
      deployment: true
      cronjob: true

Set a toggle to true to add the field, false to drop it, or leave it unset to keep Filebeat's own default.

Going Further#

See Services Logs for accessing the collected logs in OpenSearch Dashboards.