ingest-worker

WarningSample

ingest-worker · UAT · local/worker · us-east-1 · active

host demo-linux-02 · connection 10.0.12.12 · 1.12.0 · e171cc4

All applications

Kind

Sample

worker

Python 3.12

Requests

Sample

per minute

Error rate

Sample

1.8%

sample fleet

p95 latency

Sample

940 ms

sample fleet

Dependencies

Sample /health statuses · website, API, database, Redis, storage

Sample

Website

Healthy

API

Warning

Database

Healthy

Redis

Healthy

Storage

Warning

Linked server

demo-linux-02 · sample host capacity

SampleOpen host

UAT · local/linux · us-east-1 · connection 10.0.12.12

CPU

Sample

71.8%

Demo-Linux-02

RAM

Sample

82.4%

Ubuntu 24.04 LTS

Disk

Sample

64.1%

Root mount

30-day API growth

Sample

+9.0%

Sample series

30-day user growth

Sample

+4.0%

Sample series

30-day storage growth

Sample

+62.0%

Sample series

ingest-worker error rate is elevated

Warning

Sample

What is happening: Error rate 1.8% since 09:55 UTC, p95 940 ms.
Impact: ingest-worker on demo-linux-02 (UAT) · backlog replay in progress.
Possible cause: Host memory 82.4% and rising; coincident /var/log growth on the same server.
Recommended next action: Open Demo-Linux-02, inspect RSS and /var/log, then pause debug logging.

Open ingest-worker

Memory pressure on Demo-Linux-02

Warning

Sample

What is happening: Memory above 80% for 15 minutes (82.4% now).
Impact: UAT worker host demo-linux-02 · ingest-worker sharing the machine.
Possible cause: Worker RSS climbed after a backlog replay. Swap activity is rising.
Recommended next action: Inspect ingest-worker RSS on Demo-Linux-02 and reduce replay concurrency.

Open Demo-Linux-02

Unexpected /var/log growth

Warning

Sample

What is happening: /var/log is 22.4% of disk, 18.6% in 24 hours.
Impact: Storage on demo-linux-02 is the fastest-growing path in the sample fleet.
Possible cause: Application debug logging left enabled after yesterday’s load test.
Recommended next action: Disable debug logging on ingest-worker and re-check /var/log growth.

Open storage

Requests / min

Sample series

Sample

Previous 6h is not in this sample series

ingest-worker

Error rate

Sample series

Sample

Previous 6h is not in this sample series

ingest-worker

p95 latency

Sample series

Sample

Previous 6h is not in this sample series

ingest-worker

Related alerts

Sample

Severity	Alert	Source	State	Age
warning	Memory usage above 80% for 15 minutes	Demo-Linux-02	firing	47m ago
warning	Disk growth rate elevated on /var/log	Demo-Linux-02	firing	2h ago
critical	ingest-worker error rate above 1.5%	ingest-worker	acknowledged	3h ago

Related incidents

Sample

Memory pressure on Demo-Linux-02
Worker RSS climbed after a backlog replay. Swap activity is rising.
Unplanned /var/log growth
Application debug logging left enabled after yesterday’s load test.