Skip to content

Add per-resource label selector support via --label-selectors flag #2885

@superbrothers

Description

@superbrothers

What would you like to be added:

A new CLI flag to support per-resource label selectors, for example:

--label-selectors=nodes=<labelSelector>,pods=<labelSelector>

This would allow specifying Kubernetes label selectors per resource type.
Only the specified resource types would be filtered; others remain cluster-wide.

Example:

--label-selectors=nodes=tenant=team-a,pods=app=frontend

Why is this needed:

KSM currently performs cluster-wide List/Watch operations for most resources.

In some environments, restricting the observed resource set is desirable:

  • Running multiple KSM instances for sharding
  • Node-scoped deployments (e.g. DaemonSet-based KSM)
  • Reducing API server load and KSM memory usage
  • Multi-tenant environments

A concrete example:

In multi-tenant clusters, nodes are often labeled per tenant (e.g. tenant=team-a).
It is desirable to expose only the Node metrics matching a specific label selector to each tenant, so that:

  • Each tenant receives metrics only for its own nodes
  • Node-level isolation is preserved
  • KSM does not need to cache the full cluster resource set

Important technical note:

KSM exposes object labels as separate *_labels metrics (e.g. kube_node_labels{node="..."} ...).
However, those label values are not necessarily attached as metric labels on all related resource metrics. In practice, this means Prometheus-side metric_relabel_configs cannot reliably filter Node metrics by node labels, because the label information may only be present in separate *_labels metrics rather than as labels on the Node metrics themselves.

As a result, downstream filtering in Prometheus is not sufficient to implement node-level multi-tenant isolation. Filtering must occur at the informer/KSM level.

Existing mechanisms such as --namespaces are insufficient because:

  • Nodes are not namespaced
  • Namespace-based filtering does not solve node-level isolation
  • --node only filters Pods via a field selector and does not apply to Nodes

Describe the solution you'd like:

Introduce:

--label-selectors=<resourceType>=<labelSelector>,...

Behavior:

  • Comma-separated resourceType=labelSelector
  • Applied to ListOptions.LabelSelector of the corresponding informer
  • Disabled by default (backward compatible)

If both:

  • --node (field selector for Pods)
  • --label-selectors (label selector)

are specified, they are applied independently:

  • FieldSelector for --node
  • LabelSelector for --label-selectors

Additional context:

When selectors are applied, kube-state-metrics would expose metrics only for the selected subset of resources. It would not attempt to preserve cluster-wide semantics. This behavior would be explicitly documented.

This feature would enable cleaner sharding models and improve scalability for large or multi-tenant clusters.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    Status

    Needs Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions