High CPU usage (2000–8000%) from otelcol on IBM zlinux Systems (s390x / IFL x390)

[nguyenken77-hash]

We are observing extremely high and sustained CPU usage from otelcol running on IBM zlinux Systems (s390x / IFL x390) in a non-production environment. Since late last week, multiple systems have entered CPU RED state due to otelcol consumption. The collector is using significantly more CPU than any of our application workloads. This behavior is consistent across multiple hosts.

otelcol--4118645 on xxxxx14 → 8319%
otelcol--4177419 xxxxxx11 → 2668%
otelcol--1174656 on xxxxxx1 → 2924%
otelcol--1574336 on xxxxW11 → 1801.98%
otelcol--4186187 on xxxxxP14 → 2839%

Collector version: [.133.0]
Deployment method: Tarball
OS: [Red Hat Enterprise Linux release 9.6 s390x GNU/Linux ]
Number of CPU cores per host: [2] Memory [12]

1. Is high CPU consumption expected on s390x architecture?
2. Are there known performance issues specific to IBM z / s390x?
3. Are there recommended tuning parameters for this platform?
4. Is there a way to cap or limit CPU usage?
5. Are certain processors/exporters known to be inefficient on s390x?
6. Are there ways to throttle or control CPU usage?

[dosubot]

Hi @nguyenken77-hash! I'm Dosu and I’m helping the opentelemetry-collector-releases team.

I found some information that may help diagnose this issue. While there are no documented s390x-specific CPU performance issues (s390x is a supported Tier 3 platform), the CPU usage you're describing (2000-8000%) suggests something unusual is happening.

To your questions:

1. Not expected - This level of CPU consumption is abnormal regardless of architecture. High CPU typically indicates the Collector cannot process/export data as fast as it's receiving it.

2. No known s390x issues - No architecture-specific bugs have been reported.

3-6. Resource control options (the Collector doesn't have direct CPU throttling):

processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 4000
    spike_limit_mib: 500
  batch:
    send_batch_size: 10000
    timeout: 10s

Immediate diagnostic steps:

1. Enable pprof to profile the collector:

   extensions:
     pprof:
       endpoint: localhost:1777

   Then access http://localhost:1777/debug/pprof/ during high CPU to identify the bottleneck.

2. Check these metrics via Prometheus or your monitoring:

   • otelcol_processor_refused_spans - memory limiter blocking data
   • otelcol_exporter_queue_size / otelcol_exporter_queue_capacity - queue buildup indicates backend too slow

3. Review your configuration for CPU-intensive components: tail sampling, transform processor, group by trace, or kubernetes attributes processors.

4. Check for high cardinality metrics - cardinality explosion can cause unbounded CPU growth.

Could you share your collector configuration? That would help identify potential bottlenecks. Also, did anything change "late last week" when this started - new receivers, increased data volume, or backend changes?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[nguyenken77-hash]

Here is the collector configuration file. We recently started using OpenTelemetry, so this is a new setup.
extensions:

Basic auth for Grafana Cloud

basicauth/grafana_cloud:
client_auth:
username: "xxxxxx"
password: "xxxxxxxxxxxxx"

File storage for offset tracking

file_storage:
directory: /var/lib/otelcol-contrib/file_storage
timeout: 10s

Health check extension

health_check:
endpoint: 0.0.0.0:13133
path: /health

receivers:
filelog:
include:
- /var/log/messages
- /var/log/secure
- /var/log/.log
- /var/log/atp/.log
- /var/log/atp//.log
- /var/log/atp///.log
- /var/log/atp////.log
- /opt/controller/ctm/proclog/.log

- /var/log/atp/**/*.log.

exclude:
  # Exclude ALL .log.owner files anywhere under /var/log/atp
  - /var/log/atp/**/*.log.owner
  - /var/log/dnf.log
  - /var/log/hawkey.log
  - /var/log/dnf*.log
  - /var/log/hawkey*.log
  - /var/log/wtmp*
  - /var/log/btmp*
  - /var/log/lastlog
  - /var/log/tallylog

# CRITICAL SETTINGS FOR REAL-TIME COLLECTION
start_at: beginning                    # Start at end of file (only NEW logs from now)
storage: file_storage            # Link to storage extension for offset tracking

include_file_path: true
include_file_name: true

# Fingerprint settings
fingerprint_size: 4kb            # Smaller fingerprint (faster)
max_log_size: 10MiB
max_concurrent_files: 1024

# Poll frequently for real-time collection
poll_interval: 200ms             # Check every 200ms for new content

# Force flush frequently
force_flush_period: 500ms

processors:

Memory limiter - CRITICAL for production (should be first processor)

memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128

Batch processor for better performance

batch:
timeout: 10s
send_batch_size: 1024

Resource detection processor

resourcedetection:
detectors: [system, env]
timeout: 5s
system:
hostname_sources: ["os"]

Resource processor to add custom attributes

resource:
attributes:
- key: deployment.environment
action: upsert
value: qtst
- key: service.name
action: upsert
value: qtst-logs
exporters:

Grafana Cloud OTLP exporter

otlphttp/grafana_cloud:
endpoint: "https://otlp-gateway-prod-us-east-3.grafana.net/otlp"
auth:
authenticator: basicauth/grafana_cloud
compression: gzip
timeout: 30s
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s

service:
extensions:
- basicauth/grafana_cloud
- file_storage
- health_check
telemetry:
logs:
level: info
initial_fields:
service: otel-collector

pipelines:
logs:
receivers: [filelog]
processors: [memory_limiter, resourcedetection, resource, batch]
exporters: [otlphttp/grafana_cloud]

[dosubot]

Thanks for sharing the configuration! I can see several settings that are likely contributing to your extreme CPU usage:

Primary suspects:

1. Aggressive poll_interval: 200ms - Polling every 200ms means the collector is checking all matched files 5 times per second. Combined with your glob patterns, this creates enormous overhead.

2. Deep recursive glob patterns - These patterns expand to scan many directories constantly:

   - /var/log/atp/*/*.log
   - /var/log/atp/*/*/*.log  
   - /var/log/atp/*/*/*/*.log

3. max_concurrent_files: 1024 - With aggressive polling, tracking up to 1024 files simultaneously at 200ms intervals is extremely resource-intensive.

4. start_at: beginning - On a new deployment, this causes the collector to read ALL existing log content from all matched files at startup, creating a massive initial load.

Recommended changes:

filelog:
  # Increase poll interval significantly
  poll_interval: 1s              # Was 200ms - 5x reduction in file checks
  
  # Start from end for production (avoid processing historical logs)
  start_at: end                  # Was: beginning
  
  # Reduce concurrent file tracking
  max_concurrent_files: 256      # Was: 1024
  
  # Increase flush period
  force_flush_period: 5s         # Was: 500ms
  
  # Simplify patterns - use ** instead of multiple levels
  include:
    - /var/log/messages
    - /var/log/secure
    - /var/log/*.log
    - /var/log/atp/**/*.log      # Single pattern instead of 4
    - /opt/controller/ctm/proclog/*.log

Also consider filtering unwanted telemetry with the filter processor if you don't need all log content, and monitor otelcol_processor_refused_spans to see if the memory limiter is being triggered.

The combination of 200ms polling + deep glob patterns + 1024 concurrent files + reading from beginning is almost certainly the root cause of your 2000-8000% CPU usage.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[nguyenken77-hash]

when I use - /var/log/atp/**/*.log I see logs skipped

[mx-psi]

@bwalk-at-ibm @rrschulze could you take a look as owners of the s390x platform?

[bwalk-at-ibm]

@nguyenken77-hash Is this an s390-only issue? Have you tried running your collector configuration on a x86-machine? If the bot's analysis is correct, the issue should also occur on a non-s390 machine. I'd like to assure that this is a platform bug before investigating.

[nguyenken77-hash]

@bwalk-at-ibm, We don't have any x86 linux in house to test. It's very random when I use recursive. It pickup some logs but not other. As soon as I changed to single it picked up all logs. Thanks

[bwalk-at-ibm]

Can you please provide a minimal reproducible config? If that config depends on external resources, I need some context on those too (e.g. number of files affected by the globs and so on).

I can give it a try to reproduce it then.

[nguyenken77-hash]

Below is the configuration file.
Initially, we used the following recursive include patterns:
/var/log/atp/**/*.log
/var/log/messages
However, we noticed that SystemOut.log was not being collected, even though it resides under the same /var/log/atp/ path covered by the recursive pattern.
After we explicitly added the full path to SystemOut.log as a separate include entry, the OTel agent began collecting it successfully.

Thanks

receivers:

File logs receiver

filelog:
include:

- /var/log/atp/fmgqcell/fmgapp011/SystemOut.log

  - /var/log/messages
  - /var/log/secure
  - /var/log/*.log
  - /var/log/atp/*.log
  - /var/log/atp/*/*.log
  - /var/log/atp/*/*/*.log
  - /var/log/atp/*/*/*/*.log
  - /opt/controller/ctm/proclog/*.log

- /var/log/atp/**/*.log

exclude:
  # Exclude ALL .log.owner files anywhere under /var/log/atpco
  - /var/log/atp/**/*.log.owner
  - /var/log/dnf*.log
  - /var/log/hawkey*.log
  - /var/log/wtmp*
  - /var/log/btmp*
  - /var/log/lastlog
  - /var/log/tallylog
  - /var/log/atp/**/nodeagent/**
  - /var/log/atp/**/dmgr/**
  - /var/log/atp/**/ffdc/**
  - /var/log/atp/**/aspectj/**


# CRITICAL SETTINGS FOR REAL-TIME COLLECTION
start_at: end                    # Was:beginning
storage: file_storage            # Link to storage extension for offset tracking

include_file_path: true
include_file_name: true

# Fingerprint settings
fingerprint_size: 4kb            # Smaller fingerprint (faster)
max_log_size: 10MiB
max_concurrent_files: 256        # Was: 1024

# Poll frequently for real-time collection
poll_interval: 1s             # Check every 200ms for new content

# Force flush frequently
force_flush_period: 5s        # Was: 500ms

processors:

Memory limiter - CRITICAL for production (should be first processor)

memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128

Batch processor for better performance

batch:
timeout: 10s
send_batch_size: 1024

Resource detection processor

resourcedetection:
detectors: [system, env]
timeout: 5s
system:
hostname_sources: ["os"]

Resource processor to add custom attributes

resource:
attributes:
- key: deployment.environment
action: upsert
value: htst
- key: service.name
action: upsert
value: htst-logs

- key: service.namespace

action: upsert

value: atp

exporters:

Grafana Cloud OTLP exporter

otlphttp/grafana_cloud:
endpoint: "https://otlp-gateway-prod-us-east-3.grafana.net/otlp"
auth:
authenticator: basicauth/grafana_cloud
compression: gzip
timeout: 30s
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s

service:
extensions:
- basicauth/grafana_cloud
- file_storage
- health_check

- pprof

- zpages

telemetry:
logs:
level: info
initial_fields:
service: otel-collector

metrics:

level: detailed

pipelines:
# Logs pipeline
logs:
receivers: [filelog]
processors: [memory_limiter, resourcedetection, resource, batch]
exporters: [otlphttp/grafana_cloud]

[bwalk-at-ibm]

Sorry, I need you to condense it down to the bare minimum configuration which can reproduce the issue. I don't have the context to reproduce your environment myself.

[mowies]

also, using proper codeblocks really help others to easily follow the configuration you post :)