In the comments on my post about Load Average Monitoring [1] an anonymous person recommended that I investigate PSI. As an aside, why do I get so many great comments anonymously? Don’t people want to get credit for having good ideas and learning about new technology before others?
PSI is the Pressure Stall Information subsystem for Linux that is included in kernels 4.20 and above, if you want to use it in Debian then you need a kernel from Testing or Unstable (Bullseye has kernel 4.19). The place to start reading about PSI is the main Facebook page about it, it was originally developed at Facebook [2].
I am a little confused by the actual numbers I get out of PSI, while for the load average I can often see where they come from (EG have 2 processes each taking 100% of a core and the load average will be about 2) it’s difficult to work out where the PSI numbers come from. For my own use I decided to treat them as unscaled numbers that just indicate problems, higher number is worse and not worry too much about what the number really means.
With the cgroup2 interface which is supported by the version of systemd in Testing (and which has been included in Debian backports for Buster) you get PSI files for each cgroup. I’ve just uploaded version 1.3.5-2 of etbemon (package mon) to Debian/Unstable which displays the cgroups with PSI numbers greater than 0.5% when the load average test fails.
System CPU Pressure: avg10=0.87 avg60=0.99 avg300=1.00 total=20556310510 /system.slice avg10=0.86 avg60=0.92 avg300=0.97 total=18238772699 /system.slice/system-tor.slice avg10=0.85 avg60=0.69 avg300=0.60 total=11996599996 /system.slice/system-tor.slice/tor@default.service avg10=0.83 avg60=0.69 avg300=0.59 total=5358485146 System IO Pressure: avg10=18.30 avg60=35.85 avg300=42.85 total=310383148314 full avg10=13.95 avg60=27.72 avg300=33.60 total=216001337513 /system.slice avg10=2.78 avg60=3.86 avg300=5.74 total=51574347007 /system.slice full avg10=1.87 avg60=2.87 avg300=4.36 total=35513103577 /system.slice/mariadb.service avg10=1.33 avg60=3.07 avg300=3.68 total=2559016514 /system.slice/mariadb.service full avg10=1.29 avg60=3.01 avg300=3.61 total=2508485595 /system.slice/matrix-synapse.service avg10=2.74 avg60=3.92 avg300=4.95 total=20466738903 /system.slice/matrix-synapse.service full avg10=2.74 avg60=3.92 avg300=4.95 total=20435187166
Above is an extract from the output of the loadaverage check. It shows that tor is a major user of CPU time (the VM runs a ToR relay node and has close to 100% of one core devoted to that task). It also shows that Mariadb and Matrix are the main users of disk IO. When I installed Matrix the Debian package told me that using SQLite would give lower performance than MySQL, but that didn’t seem like a big deal as the server only has a few users. Maybe I should move Matrix to the Mariadb instance. to improve overall system performance.
So far I have not written any code to display the memory PSI files. I don’t have a lack of RAM on systems I run at the moment and don’t have a good test case for this. I welcome patches from people who have the ability to test this and get some benefit from it.
We are probably about 6 months away from a new release of Debian and this is probably the last thing I need to do to make etbemon ready for that.
>it’s difficult to work out where the PSI numbers come from. For my own use I decided to treat them as unscaled numbers that just indicate problems, higher number is worse and not worry too much about what the number really means.
The number is the time in microseconds the system was unable to schedule a task due to the ressource being busy. For example:
System CPU Pressure: avg10=0.87 avg60=0.99 avg300=1.00 total=20556310510
This means that some tasks were delayed 20556310510 microseconds since the start of the system. The avg field are an average on the last seconds in percent. So, in the last 10 seconds, 0.87% of the tasks were delayed because of the pressure on the CPU.