• Redjard@reddthat.com
    link
    fedilink
    arrow-up
    2
    ·
    23 hours ago

    There was a dedicated person on call, but it happened to be when they were away.
    The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages.
    The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn’t noticed.
    Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident.