Created a Perl script to record and report memory usage of processes to discover heavy memory greedy processes.
Here is the source code, below this text block Perl Pod documentation and usage.
In the "real life" troubleshooting I need it for, it did reveal some bloating processes, but only after some time, had to increase "nice" priority for the script, because when the system get under stress, kind of everything gets slow, including the processes inspection.
Another alternative approach that I've tried was to keep htop running on the target system, record it with kazam and then using on monitoring charts could identify actually watch the interesting moments of the system behavior.
process-mem-report - record & print top N memory consumers over time
process-mem-report
--help
--runtime|t number of minutes to run the measurement, default infinite
--maxlen|l maximum output line length, default 80
--maxlines|n number of top memory greedy processes to show, default 30
--refresh|r refresh rate in seconds, default 10
--write|w write to file
--wstrftime|f add strftime formatted suffix to write file
Will go through all procesed and read their memory usage using smaps and show top N. After refresh time will re-read all process together with previous N will show new top N.
If executed unprivileged, will consider processe of current user only.
$ process-mem-report -n 10
--------------------------------------------------
2020-02-22T22:22:14 885.48M /usr/lib/thunderbird/thunderbird
2020-02-22T22:22:14 653.38M firefox-esr
2020-02-22T22:22:15 487.50M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
2020-02-22T22:22:15 451.92M /usr/lib/chromium/chromium --show-component-extensi…
2020-02-22T22:22:14 380.55M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
2020-02-22T22:22:14 377.14M cinnamon --replace
2020-02-22T22:22:15 321.80M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
2020-02-22T22:22:14 308.17M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
2020-02-22T22:22:15 257.77M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
2020-02-22T22:22:15 247.15M /usr/lib/firefox-esr/firefox-esr -contentproc -chil…
max mem used: 7.50G at: 2020-02-22T22:22:15
since: 2020-02-22T22:22:13
# hourly written report of top memory greedy processes
$ sudo process-mem-report -r 5 -n 80 -w mem-top_ -f %Y%m%d_%H