https://runbooks.safetywing.dev/runbooks/node/Recent content in Node on SafetyWing RunbooksHugoen-ushttps://runbooks.safetywing.dev/runbooks/node/nodefilesystemalmostfull/Mon, 01 Jan 0001 00:00:00 +0000https://runbooks.safetywing.dev/runbooks/node/nodefilesystemalmostfull/<h2 id="meaning">Meaning<a class="anchor" href="#meaning">#</a></h2> <p>A node filesystem is running low on free space. This is a supplemental platform-tier rule on top of the kube-prometheus-stack node-exporter mixin.</p> <p>Fires when: available space on a non-ephemeral filesystem drops below the configured ratio.</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-promql" data-lang="promql"><span style="display:flex;"><span><span style="color:#f92672">(</span>node_filesystem_avail_bytes{fstype<span style="color:#f92672">!~</span>&#34;<span style="color:#e6db74">tmpfs|overlay|squashfs</span>&#34;} </span></span><span style="display:flex;"><span> <span style="color:#f92672">/</span> node_filesystem_size_bytes{fstype<span style="color:#f92672">!~</span>&#34;<span style="color:#e6db74">tmpfs|overlay|squashfs</span>&#34;}<span style="color:#f92672">)</span> <span style="color:#f92672">&lt;</span> <span style="color:#f92672">&lt;</span>ratio<span style="color:#f92672">&gt;</span></span></span></code></pre></div><p><code>for: 15m</code>, severity <code>ticket</code>, tier <code>platform</code> (cluster-wide, no <code>environment</code> label). The offending filesystem is identified by the <code>instance</code> and <code>mountpoint</code> labels.</p> <h2 id="impact">Impact<a class="anchor" href="#impact">#</a></h2> <p>A full filesystem on a node can wedge the kubelet, fail image pulls, block container log writes, evict pods (ephemeral-storage pressure), and on Talos can disrupt the system partition. If the node hosts stateful workloads (Ceph OSDs, MySQL/MOCO, RabbitMQ), data writes can stall or fail.</p>