Follow

For those who don't know the difference between the different kinds of memory usage on Linux

Virtual memory is a measure of how much memory - real and "virtual" - is mapped into the process space. This can go up without increasing the memory pressure on your system by doing things like mmaping files.

Resident memory is a more accurate measure of your application's use of RAM. The etymology is a little bit dated, but basically this is how much RAM your program takes up when it's not the active task. If you want to know about RAM usage, this is the only thing you need to look at.

Shared memory is that which is currently or theoretically possible to share between processes, such as shared libraries or shm files.

@sir I thought anonymous memory is how much it takes up when it's not an active task. Resident is how much of its memory (anonymous or file-backed) sits currently in RAM, as opposed to being on disk (swap if anonymous, otherwise the file it comes from).

@sir I believe it was somewhere in /lib/modules/$(uname -r)/build/Documentation, lemme see if I can find it

@sir
reading section 2.2.1 of admin-guide/cgroup-v1/memory.rst, and it looks like I was wrong: RSS only counts resident anonymous pages. IOW, it doesn't count file-backed pages, and it doesn't count anonymous pages that are swapped out.

kernel.org/doc/Documentation/a

@sir but a more appropriate resource is probably man 5 proc:

* VmRSS: Resident set size. Note that the value here is the sum of RssAnon, RssFile, and RssShmem.

* RssAnon: Size of resident anonymous memory. (since Linux 4.5).

* RssFile: Size of resident file mappings. (since Linux 4.5).

* RssShmem: Size of resident shared memory (includes System V shared memory, mappings from tmpfs(5), and shared anonymous mappings). (since Linux 4.5).

I wonder which one htop displays as RES

@Wolf480pl @sir htop on Linux read /proc/<pid>/statm. The resident part of that (second number) is VmRSS.

Source: Documentation/filesystems/proc.txt (search for statm)

For OpenBSD and DragonFlyBSD it also clearly uses VM RSS. For other BSDs it's not immediately obvious.

Source: github.com/hishamhm/htop/searc

@sir Quite importantly: "Why are virtual and shared memory different?"

In Linux (and other OSes with memory overcommit, which OpenBSD doesn't allow,) when a program asks the OS for memory, the OS doesn't actually give it that memory. Instead it allocates the memory when you actually try to use it(read or write.) You're basically telling the OS "I need a book with this many blank pages", it gives you an empty binder, and it brings you a new sheet whenever you want to write a new page.

@ari this is true and important but also misleading. Most applications aren't allocating pages worth of memory and not using them. In practice the largest sink in virt is mmap'd files.

@sir I mean, technically the entire virt sink is mmapped files since pretty much every malloc is based on mmap :) I'm kinda interested in the malloc vs mmap breakdown for various programs though.

@ari malloc is backed by mmap, too. But notice I said "mmaped files"

@sir @ari I know you wrote "most applications," and two applications aren't "most", but one very important class of applications that relies heavily on "over-committed" memory is web servers like Apache.

@sir @ari When I was working full-time in a data center, probably the #1 thing which caused cron to die (of all things) was OOM randomly killing it because some extremely poorly written server-side code was sopping up all the system resources that Apache over-committed on its behalf. That frequently lead to logs failing to rotate, which lead to / filling up, which lead to ... well, all-too-frequent remote hands requests.

@sir @ari Moral of the story: if you want your system to have long up-times, you need to understand the nature and behavior of the kernel's over-commit policy and *preferably* run software that does not rely so heavily upon it.

Sign in to participate in the conversation
Mastodon

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!