Load (computing)

In UNIX computing, the system load is a measure of the amount of computational work that a computer system performs. The load average represents the average system load over a period of time. It conventionally appears in the form of three numbers which represent the system load during the last one-, five-, and fifteen-minute periods.

htop displaying a significant computing load (top right: Load average:)

Unix-style load calculation

All Unix and Unix-like systems generate a dimensionless metric of three "load average" numbers in the kernel. Users can easily query the current result from a Unix shell by running the uptime command:

$ uptime
 14:34:03 up 10:43,  4 users,  load average: 0.06, 0.11, 0.09

The w and top commands show the same three load average numbers, as do a range of graphical user interface utilities. The underlying interface is getloadavg(), a C function present on most UNIX systems (but is not part of POSIX).^[1]

An idle computer has a load number of 0 (the idle process is not counted). Each process using or waiting for CPU (the ready queue or run queue) increments the load number by 1. Each process that terminates decrements it by 1. Most UNIX systems count only processes in the running (on CPU) or runnable (waiting for CPU) states (state R).

Systems calculate the load average as the exponentially damped/weighted moving average of the load number. The three values of load average refer to the past one, five, and fifteen minutes of system operation.^[2]

Mathematically speaking, all three values always average all the system load since the system started up. They all decay exponentially, but they decay at different speeds: they decay exponentially by e after 1, 5, and 15 minutes respectively. Hence, the 1-minute load average consists of 63% (more precisely: 1 - 1/e) of the load from the last minute and 37% (1/e) of the average load since start up, excluding the last minute. For the 5- and 15-minute load averages, the same 63%/37% ratio is computed over 5 minutes and 15 minutes, respectively. Therefore, it is not technically accurate that the 1-minute load average only includes the last 60 seconds of activity, as it includes 37% of the activity from the past, but it is correct to state that it includes mostly the last minute.

Interpretation

For single-CPU systems that are CPU bound, one can think of load average as a measure of system utilization during the respective time period. For systems with multiple CPUs, one must divide the load by the number of processors in order to get a comparable measure.

For example, one can interpret a load average of "1.73 0.60 7.98" on a single-CPU system as:

• During the last minute, the system was overloaded by 73% on average (1.73 runnable processes, so that 0.73 processes had to wait for a turn for a single CPU system on average).
• During the last 5 minutes, the CPU was idling 40% of the time, on average.
• During the last 15 minutes, the system was overloaded 698% on average (7.98 runnable processes, so that 6.98 processes had to wait for a turn for a single CPU system on average).

This means that this system (CPU, disk, memory, etc.) could have handled all the work scheduled for the last minute if it were 1.73 times as fast.

In a system with four CPUs, a load average of 3.73 would indicate that there were, on average, 3.73 processes ready to run, and each one could be scheduled into a CPU.

On modern UNIX systems, the treatment of threading with respect to load averages varies. Some systems treat threads as processes for the purposes of load average calculation: each thread waiting to run will add 1 to the load. However, other systems, especially systems implementing so-called M:N threading, use different strategies such as counting the process exactly once for the purpose of load (regardless of the number of threads), or counting only threads currently exposed by the user-thread scheduler to the kernel, which may depend on the level of concurrency set on the process. Linux appears to count each thread separately as adding 1 to the load.^[3]

Variation by OS

Linux

In addition to processes in "R" state, Linux also includes processes in uninterruptible sleep states (usually waiting for disk activity; state "D"), which can lead to markedly different results if many processes remain blocked in I/O due to a busy or stalled I/O system.^[4] This, for example, includes processes blocking due to an NFS server failure or too slow media (e.g., USB 1.x storage devices). Such circumstances can result in an elevated load average, which does not reflect an actual increase in CPU use (but still gives an idea of how long users have to wait).

In operating systems based on the Linux kernel, load information can be easily accessed by reading the /proc/loadavg file. This file also provides information on the instataneous load:^[5]:

$ cat /proc/loadavg
0.06, 0.11, 0.09 2/329 6270

Here the first three numbers are the regular load average values. "2/329" indicates that there are 2 kernel scheduling entities (processes, threads) on the system current running or runnable, and that there are 329 such entities in total. "6270" refers to the process ID of the most-recently-created process. In other words, it provides an instant measurement of the running or runnable (R) component of load.^[5]

Linux procfs offers further information on CPU time through the file /proc/stat.^[6] (procfs is part of the Filesystem Hierarchy Standard [FHS], but it is not limited to FHS-enabled systems.)^[7] There is minor variation between linux kernels on different instruction set architectures with regards to what information is provided, but for load calculation these are almost always available:^[8]

• A cpu line showing the breakdown into CPU time spent in user mode, nice (low-priority user mode), system mode, on idle task, iowait task (unreliable on multicore systems), irq task (serving interrupts), etc. This information is counted since system start, so to obtain a recent reading one needs to subtract the current reading from a reading from some time (e.g. 1 second) ago.
• One line per CPU core to provide per-core information, starting with cpu0. Same format as the "cpu" line. Counted since system start.
• procs_running, the current count of running and runnable (state "R") processes. Same as the count from /proc/loadavg.
• procs_blocked, the current count of processes in uninterruptible sleep (state "D").

This kind of instantaneous information can be used to calculate load averages over different timestamps compared to what's offered by the system by default. For example, the following program continuously displays load average values for periods from 10 seconds to one hour:

#!/usr/bin/env python3
import time
from datetime import datetime
from math import exp, log
from typing import Tuple
from os import getloadavg

LOGSYSPERIODS = [log(x) for x in [60, 300, 900]]
REFRESH_RATE = ((5**0.5) + 1) / 2  # Ripke's golden ratio
PERIODS = [10, 30, 60, 120, 300, 900, 1800, 3600]

class LoadEntry:
    def __init__(self, avg: float, exp: float):
        self.avg = avg
        self.exp = exp


def initialize_loads(now: int | float, lavgs: dict[int, LoadEntry]):
    """Initialize load averages based on linear interpolation of system load averages and instant load count."""
    sys = getloadavg()
    slopes = ((sys[0] - now) / 60, (sys[1] - sys[0]) / 240, (sys[2] - sys[1]) / 600)
    for period in [10, 30, 60, 120, 300, 900, 1800, 3600]:
        exp_factor = exp(-REFRESH_RATE / period)
        if period < 60:
            est_avg = now + slopes[0] * (period - 60)
        elif period < 300:
            est_avg = sys[0] + slopes[1] * (period - 300)
        else:
            est_avg = sys[1] + slopes[2] * (period - 900)
        lavgs[period] = LoadEntry(avg=max(est_avg, 0), exp=exp_factor)

def get_current_load() -> int:
    load = 0
    with open("/proc/stat", "r") as f:
        for line in f:
            if line.startswith("procs_running "):
                load += int(line.split()[1])  # Read procs_running
                load -= 1  # Subtract one for ourself
            elif line.startswith("procs_blocked "):
                load += int(line.split()[1])  # Read procs_blocked
    return load

def update_loads(lavgs: dict[int, LoadEntry], current_load: int | float) -> None:
    for _, entry in lavgs.items():
        entry.avg = entry.avg * entry.exp + current_load * (1 - entry.exp)

def main():
    lavgs: dict[int, LoadEntry] = {}
    current_load = get_current_load()
    initialize_loads(current_load, lavgs)
    heading = ["SYS-TIME", "INSTANT"] + [str(x) for x in PERIODS]
    print("\t".join(heading))
    while True:
        # Always print before recalculating
        entries = [datetime.now().strftime("%H:%M:%S"), str(current_load)]
        entries += [f"{entry.avg:.4f}" for entry in lavgs.values()]
        print("\t".join(entries), flush=True)

        # Sleep and wait. This assumes that the time used to update a new line is
        # minuscule compared to the time slept. If this is not the case, a
        # sleepuntil()-type reckoning should be used.
        time.sleep(REFRESH_RATE)

        # Print
        current_load = get_current_load()
        update_loads(lavgs, current_load)

if __name__ == "__main__":
    main()

Unix-like and Unix-inspired systems

The number of threads in runnable (R) and uninterruptable (D) states can be obtained almost every most Unix-like system by parsing the output of the ps command, specifically ps Max -o stat. These include FreeBSD, Linux, macOS, and many more. Although the "state" column is not a part of base POSIX, it is found on every system that implements X/Open System Interfaces (XSI) in the form of ps -l.^[9] The following modification would make the above portable to these systems:

from os import popen

def get_sys_loadavg_uptime() -> Tuple[float, float, float]:
    with popen("uptime", "r") as f:
        parts = f.read().split()
        return float(parts[-3].strip(',')), float(parts[-2].strip(',')), float(parts[-1])

def get_current_load_ps_o_s() -> int:
    with popen("ps Max -o stat", "r") as f:
        states = map(f, line: line.split()[-1])  # Get the last column. Required on macOS!
        # R is running on Linux and macOS. D is uninterruptable on Linux, but on macOS it's called U.
        return sum(1 if state[0] in "RUD" for state in states) - 1

Plan 9 from Bell Labs also has an analogous concept of process states.

Microsoft Windows

Although the Windows OS does not keep track of a "load average", it does have a performance counter System\Processor Queue Length accessible through Performance Monitor as well as various APIs. This value reports the number of threads waiting to run, without including running threads, so to obtain an approximation of Unix-style load one would need to add the current CPU utilization percent multiplied by the number of cores. (There are separate counters for disk queue length, which do include currently-running threads.)^[10]

from pyperfmon import pyperfmon
from os import cpu_count

pm = pyperfmon.pyperfmon()
ncores = os.cpu_count()
get_counter = lambda x: pm.getCounter(x)

def get_current_load_perfmon() -> float:
    return (
        get_counter(r"System\Processor Queue Length")
        + get_counter(r"Processor\_Total\% Processor Time") * ncores
        + get_counter(r"PhysicalDisk\_Total\Current Disk Queue Length")
    )

CPU load vis-à-vis CPU utilization

The comparative study of different load indices carried out by Ferrari et al. reported that CPU load information based upon the CPU queue length does much better in load balancing compared to CPU utilization. The reason CPU queue length did better is probably because when a host is heavily loaded, its CPU utilization is likely to be close to 100%, and it is unable to reflect the exact load level of the utilization. In contrast, CPU queue lengths can directly reflect the amount of load on a CPU. As an example, two systems, one with 3 and the other with 6 processes in the queue, are both very likely to have utilizations close to 100%, although they obviously differ.^[11]

Reckoning CPU load

On Linux systems, the load-average is not calculated on each clock tick, but driven by a variable value that is based on the Hz frequency setting and tested on each clock tick. This setting defines the kernel clock tick rate in hertz (times per second), and it defaults to 100 for 10 ms ticks. Kernel activities use this number of ticks to time themselves. Specifically, the calc_load() function (in timer.c), which calculates the load average, runs every LOAD_FREQ (5*HZ+1) ticks, i.e. a tiny bit over 5 s.

unsigned long avenrun[3];

static inline void calc_load(unsigned long ticks)
{
    unsigned long active_tasks; /* fixed-point */
    static int count = LOAD_FREQ;

    count -= ticks;
    if (count < 0) {
        count += LOAD_FREQ;
        active_tasks = count_active_tasks();
        CALC_LOAD(avenrun[0], EXP_1, active_tasks);
        CALC_LOAD(avenrun[1], EXP_5, active_tasks);
        CALC_LOAD(avenrun[2], EXP_15, active_tasks);
    }
}

The avenrun array contains 1-minute, 5-minute and 15-minute average. The CALC_LOAD macro and its associated values are defined in sched.h:

#define FSHIFT     11           /* number of bits of precision */
#define FIXED_1    (1<<FSHIFT)  /* 1.0 as fixed-point */
#define LOAD_FREQ  (5*HZ+1)     /* 5 sec intervals */
#define EXP_1      1884         /* 1/exp(5sec/1min) as fixed-point */
#define EXP_5      2014         /* 1/exp(5sec/5min) */
#define EXP_15     2037         /* 1/exp(5sec/15min) */

/* This macro calculates the exponential moving average */
#define CALC_LOAD(load,exp,n) \
    load *= exp; \
    load += n*(FIXED_1-exp); \
    load >>= FSHIFT;

The associated Linux kernel source code also mentions that the use of 11 fractional bits in the above fixed point calculation prevents reducing the interval (LOAD_FREQ) much lower. For example, a two-second interval would result in the EXP values becoming 1981, 2034 and 2043, nearly saturating the precision available (0–2047).^[12]

The "sampled" calculation of load averages is a somewhat common behavior; FreeBSD, too, only refreshes the value every five seconds. The interval is usually taken to not be exact so that they do not collect processes that are scheduled to fire at a certain moment. This is the reason for "+1" in the Linux code from above; FreeBSD instead uses a pseudorandom offset added to the interval.^[13]\

Ripke Klaus has shown in 2011 that the "+1" modification alone is not sufficient to avoid Moiré artifacts from regularly-scheduled processes. His experiments suggest 4.61 to be a better value: 0.61 is close to the golden ratio, which helps spread out the sample-point among fractional seconds. At the same time, 4.61 is close to 60⁄13, so the property of 5 s being an integer fraction of 60 s is maintained.^[14][15] Ripke's change is common among Android system kernels, although the exact expression used (4*HZ+61) assumes an HZ of 100.^[16] 60*HZ/13 would be more appropriate for varying values of HZ. The new values would be:^[15]

#define LOAD_FREQ  (60*HZ/13)   /* 60/13 ~ 4.61 sec intervals */
#define EXP_1      1896         /* 1/exp(4.61sec/1min)  = 1/exp(1/13) as fixed-point */
#define EXP_5      2017         /* 1/exp(4.61sec/5min)  = 1/exp(1/13/5)  */
#define EXP_15     2038         /* 1/exp(4.61sec/15min) = 1/exp(1/13/15) */

Other system performance commands

Other commands for assessing system performance include:

• uptime – the system reliability and load average
• top – for an overall system view
  • htop – interactive process viewer
• vmstat – vmstat reports information about runnable or blocked processes, memory, paging, block I/O, traps, and CPU.
• dool (formerly dstat),^[17] atop – helps correlate all existing resource data for processes, memory, paging, block I/O, traps, and CPU activity.
• iftop – interactive network traffic viewer per interface
• nethogs – interactive network traffic viewer per process
• iotop – interactive I/O viewer^[18]
• iostat – for storage I/O statistics
• netstat – for network statistics
• mpstat – for CPU statistics
• tload – load average graph for terminal
• xload – load average graph for X

See also

• CPU usage