OS is Debian on both servers. I found some examples showing how to measure TCP latency with nmap: nmap --packet-trace -p 22 192.168.0.10 But I'm getting some negative results for latency. ie, it will print some output like: Host is up (-0.0023s latency) at the end. What is nmap doing? Is it measuring the time difference between SYN and SYN-ACK? If so, how is it getting a negative result sometimes? Does it require clocks to be very precisely synced between the two servers? And in addition, can I use ssh as a good TCP test server or should I use a different service? Is there another tool I can use to measure TCP latency? I don't know if I trust nmap due to the negative results for the latency measurement.

I am building an embedded device that runs Linux (Armbian) and receives a steady stream of UDP data. It has both Ethernet and WiFi connectivity. Ethernet works fine, however when using WiFi, the data stream will occasionally appear to suddenly pause for up to a second, no data is received during this period. The packets are not lost, however, they seem to just be queued in the network driver stack somewhere. And after the pause is up, the entire backlog is dumped on the application. This would be fine if I had a reasonable buffer to work with, but my stream is very time-sensitive, I cannot afford jitter more than 50ms or so. So these pauses lead to big buffer underruns in my application. The delay is at the receiver, earlier than userspace, since tshark running there shows a gap in packets when it happens. But this gap is not present in wireshark on the client PC. It is hard to tell whether the spikes correlate with activity, since activity is very evenly distributed on the device, there is nothing else of note running on there than my application. It is not correlated with time, it seems random. Where can I begin to troubleshoot this?

I develop a git-tracked codebase that has a lot of files. This code must be run on a remote machine. So every time I make a change locally, I must then sync it to the remote and run the new code on the remote. I want to eliminate the manual sync step. Things I've tried: * Simply doing git push (locally) and git pull (remotely) works but leads to many useless commits. Also, sometimes I am only trying small changes and I don't want those to be committed. * Mounting the remote dir locally with sshfs works, but there is a slight latency in file access. I don't mind, but PyCharm relies on it and many features break. Is there a better way to accomplish this?

I am running a system with RT-Preempt kernel v6.4.8-rt8 CPU is 12th Gen Intel(R) Core(TM) i7-12800HE This system is receiving data through the CAN Bus. Every 1 millisecond a burst of 3 messages is received in the hardware CAN interface. Most of the time (99.99998%) of the time the messages are available on the socket to be received within 1ms of the previous group. But once per minute approximately, there is a delay of 4-6 ms in which the messages has been received in the hardware but is not available on the socket when calling recv. Things I have checked: - My application is calling deterministically recv every 1ms, the data is not there. - With candump I get the confirmation that the messages are received every 1 ms by checking the hardware timestamp. - The IRQ associated with the CAN device is bound to a CPU core (2) that is not used by my RT application (CPU core 6) ## The hack If I execute stress --cpu 1 and bind the stress CPU core to the same core as the IRQ the delays disappear. So I assume that a delay in waking up the CPU is causing the culprit. I have tried all of the following to try to keep the CPU awake without using stress. - performance /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor - performance /sys/devices/system/cpu/cpu2/cpufreq/energy_performance_preference - idle=poll intel_idle.max_cstate=0 in /etc/default/grub and update-grub; reboot Still have delays with all these applied. I am out of ideas, I cannot switch to acpi-cpufreq because it does not respect the integrated GPU power consumption (which is not being used during these tests). Any other knobs I can turn to decrease that latency or ways I can understand the root issue?

I’m working with an STM32MP157D-DK1, trying to use the hwlat tracer for the board's arm Cortex-A7 CPU, to check for typical hardware latency. The following attempt was made to use the hwlat tracer on the board:

# mount -t debugfs debugfs /sys/kernel/debug
# mount -t tracefs nodev /sys/kernel/tracing
# cd /sys/kernel/tracing
# cat hwlat_detector/window
1000000
# cat hwlat_detector/width
900000
# cat tracing_thresh
0
# cat tracing_max_latency
0
# cat tracing_cpumask
3
# cat hwlat_detector/mode
none [round-robin] per-cpu
# cat tracing_on
0
# cat current_tracer
nop
# echo hwlat > current_tracer

Assigning # echo hwlat > current_tracer would set tracing_thresh to 10, so:

# echo 0 > tracing_thresh
# cat tracing_thresh
0
# echo 1 > tracing_on
# sleep 1800
# cat trace


# tracer: hwlat
#
# entries-in-buffer/entries-written: 46/46   #P:2
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
           -159      d....   240.855853: #1     inner/outer(us):    0/2     ts:946699046.769930071 count:2
           -159      d....   241.871956: #2     inner/outer(us):    0/1     ts:946699047.786034280 count:2
           -159      d....   242.891945: #3     inner/outer(us):    0/1     ts:946699048.806022489 count:2
           -159      d....   243.911906: #4     inner/outer(us):    0/1     ts:946699049.825984865 count:2
           -159      d....   246.971967: #5     inner/outer(us):    0/1     ts:946699052.886044616 count:2
           -159      d....   252.071924: #6     inner/outer(us):    0/1     ts:946699057.986000660 count:2
           -159      d....   257.171915: #7     inner/outer(us):    0/1     ts:946699063.085992329 count:1
           -159      d....   272.471929: #8     inner/outer(us):    0/1     ts:946699078.386005920 count:2
           -159      d....   273.491923: #9     inner/outer(us):    0/1     ts:946699079.406001504 count:2
           -159      d....   277.571906: #10    inner/outer(us):    0/1     ts:946699083.485984589 count:2
           -159      d....   278.591918: #11    inner/outer(us):    0/1     ts:946699084.505996048 count:2
           -159      d....   318.371905: #12    inner/outer(us):    0/1     ts:946699124.285982817 count:2
           -159      d....   346.931903: #13    inner/outer(us):    0/1     ts:946699152.845980372 count:2
           -159      d....   393.851943: #14    inner/outer(us):    0/1     ts:946699199.766020728 count:2
           -159      d....   420.371906: #15    inner/outer(us):    0/1     ts:946699226.285984865 count:1
           -159      d....   487.691904: #16    inner/outer(us):    0/1     ts:946699293.605983314 count:2
           -159      d....   614.171939: #17    inner/outer(us):    0/1     ts:946699420.086017333 count:2
           -159      d....   646.811933: #18    inner/outer(us):    0/1     ts:946699452.726012015 count:1
           -159      d....   650.891904: #19    inner/outer(us):    0/1     ts:946699456.805984600 count:2
           -159      d....   705.971922: #20    inner/outer(us):    0/1     ts:946699511.886000877 count:2
           -159      d....   708.011907: #21    inner/outer(us):    0/1     ts:946699513.925987669 count:1
           -159      d....   763.091907: #22    inner/outer(us):    0/1     ts:946699569.005986862 count:2
           -159      d....   824.291904: #23    inner/outer(us):    0/1     ts:946699630.205985225 count:2
           -159      d....   893.651899: #24    inner/outer(us):    0/1     ts:946699699.565979716 count:2
           -159      d....   895.691904: #25    inner/outer(us):    0/1     ts:946699701.605985717 count:2
           -159      d....   912.011931: #26    inner/outer(us):    0/1     ts:946699717.926011266 count:2
           -159      d....   913.031903: #27    inner/outer(us):    0/1     ts:946699718.945983725 count:2
           -159      d....   930.371903: #28    inner/outer(us):    0/1     ts:946699736.285984275 count:2
           -159      d....   934.451904: #29    inner/outer(us):    0/1     ts:946699740.365986111 count:2
           -159      d....  1034.411903: #30    inner/outer(us):    0/1     ts:946699840.325984200 count:2
           -159      d....  1049.711894: #31    inner/outer(us):    0/1     ts:946699855.625976207 count:1
           -159      d....  1211.891897: #32    inner/outer(us):    0/1     ts:946700017.805979076 count:2
           -159      d....  1222.091906: #33    inner/outer(us):    0/1     ts:946700028.005987831 count:2
           -159      d....  1336.331903: #34    inner/outer(us):    0/1     ts:946700142.245984969 count:2
           -159      d....  1338.371900: #35    inner/outer(us):    0/1     ts:946700144.285983803 count:2
           -159      d....  1356.731931: #36    inner/outer(us):    0/1     ts:946700162.646014437 count:2
           -159      d....  1477.091900: #37    inner/outer(us):    0/1     ts:946700283.005983161 count:2
           -159      d....  1514.831892: #38    inner/outer(us):    0/1     ts:946700320.745975929 count:2
           -159      d....  1530.131903: #39    inner/outer(us):    0/1     ts:946700336.045985853 count:2
           -159      d....  1533.191895: #40    inner/outer(us):    0/1     ts:946700339.105978979 count:1
           -159      d....  1534.211901: #41    inner/outer(us):    0/1     ts:946700340.125985563 count:2
           -159      d....  1571.951896: #42    inner/outer(us):    0/1     ts:946700377.865979289 count:2
           -159      d....  1639.271892: #43    inner/outer(us):    0/1     ts:946700445.185976197 count:2
           -159      d....  1664.771895: #44    inner/outer(us):    0/1     ts:946700470.685979917 count:2
           -159      d....  1679.051899: #45    inner/outer(us):    0/1     ts:946700484.965983799 count:1
           -159      d....  1768.811900: #46    inner/outer(us):    0/1     ts:946700574.725985300 count:2

After a half an hour sleep, this is what is logged to file trace. I'm trying to understand this result. Trying to take assistance in Steven Rosetedt's walkthrough of the hwlat tracer output . In another 20 minutes test I ran with similar conditions, there also appeared a line where numerous encounters of a 3 millisecond latency occurred, rather than the frequent 0, 1 or 2 microsec. It looks like this (see #16's count):

# tracer: hwlat
#
# entries-in-buffer/entries-written: 20/20   #P:2
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
           -169      d....   328.696323: #1     inner/outer(us):    0/2     ts:946685131.612004155 count:2
           -169      d....   329.711962: #2     inner/outer(us):    0/1     ts:946685132.627644489 count:2
           -169      d....   332.771947: #3     inner/outer(us):    0/1     ts:946685135.687627240 count:2
           -169      d....   333.791924: #4     inner/outer(us):    0/1     ts:946685136.707605282 count:2
           -169      d....   334.811973: #5     inner/outer(us):    0/1     ts:946685137.727653991 count:2
           -169      d....   335.831926: #6     inner/outer(us):    0/1     ts:946685138.747608742 count:2
           -169      d....   336.851921: #7     inner/outer(us):    0/1     ts:946685139.767602242 count:2
           -169      d....   345.011927: #8     inner/outer(us):    0/1     ts:946685147.927608371 count:2
           -169      d....   365.411950: #9     inner/outer(us):    0/1     ts:946685168.327631214 count:2
           -169      d....   455.171927: #10    inner/outer(us):    0/1     ts:946685258.087609424 count:2
           -169      d....   471.491916: #11    inner/outer(us):    0/1     ts:946685274.407599140 count:2
           -169      d....   616.331919: #12    inner/outer(us):    0/1     ts:946685419.247601959 count:2
           -169      d....   805.031908: #13    inner/outer(us):    0/1     ts:946685607.947590632 count:2
           -169      d....  1285.451914: #14    inner/outer(us):    0/1     ts:946686088.367599861 count:2
           -169      d....  1324.211913: #15    inner/outer(us):    0/1     ts:946686127.127599213 count:2
           -169      d....  1540.451925: #16    inner/outer(us):    1/3     ts:946686344.024041733 count:457
           -169      d....  1541.471970: #17    inner/outer(us):    0/2     ts:946686344.387656150 count:2
           -169      d....  1546.571943: #18    inner/outer(us):    0/1     ts:946686349.487628111 count:2
           -169      d....  1547.591934: #19    inner/outer(us):    0/2     ts:946686350.507622153 count:2
           -169      d....  1548.611929: #20    inner/outer(us):    0/1     ts:946686351.527615528 count:2

Since then I have made more half an hour/whole hour tests, their results are similar to those of the first log file shown. Please note that according to the hardware latency detector documentation : > The tracer hwlat_detector is a special purpose tracer that is used to > detect large system latencies induced by the behavior of certain > underlying hardware or firmware, independent of Linux itself. The code > was developed originally to detect SMIs (System Management Interrupts) > on x86 systems, **however there is nothing x86 specific about this > patchset.** I'm assuming that the reasons that x86 processors need SMIs (temperature management, ...) are inevitably present on arm cores too. How should these results be interpreted? \ Hardware latency from the Cortex-A7 doesn't exceed 3 microseconds?\ Is there something wrong with what I’m doing?

Sadly the issue reported and described here: https://unix.stackexchange.com/questions/149029/pernicious-usb-stick-stall-problem-reverting-workaround-fix and https://unix.stackexchange.com/questions/474216/is-writeback-throttling-a-solution-to-the-usb-stick-stall-problem continues to be unresolved in modern Linux distros as of 2024 despite the availability of the BDI interface introduced in Linux 6.2 released in February 2023. This can be addressed by a simple udev rule invoking a script which sets sane writeback cache values for USB mass storage devices.

When using the terminal on code-server (VSCode probably does it as well), input is instantly shown on screen (grayed out, until the server actually responds): When I connect to the remote machine using the OpenSSH client, however, the input is echoed only after a short delay. That's usually not very noticeable when latency is low, but makes it hard to type on slower connections. Using the -T option to disable TTY allocation solves the issue, but severely limits the usability of the terminal. Is there any SSH client with a similar feature to code-server?

I have been reading today about linux-lowlatency and I am more and more confused. First of all, is the install as simple as sudo apt install linux-lowlatency and then reboot? Do I need to select a different boot option in the Grub menu? I gave it a quick shot and uname -r still returned 6.2 which seems to be the generic version? Secondly, I'm using POP_OS 22.04 LTS, basically Ubuntu, and the kernel is in version 6.2. Yet, linux-lowlatency seems to be stuck at 5.15 (https://packages.ubuntu.com/jammy/linux-lowlatency) . What is the reason behind this version lag and can it be an issue?

I can't find any good information on the rt and lowlatency Linux kernels. I am wondering why anybody would *not* want to use a lowlatency kernel. Also, if anyone can tell what the specific differences are, that would be great too.

Running on MBP M1 with 10 cores. Running 1 thread client (wrk) and 1 thread server (java). A youtube is also running on the system. Load average 2 out of 10. Except GC, which could be the reasons of the wild variance in latency below?

"percentile": 0,
			"latency_in_microseconds": 10
		},
		{
			"percentile": 50,
			"latency_in_microseconds": 16
		},
		{
			"percentile": 99,
			"latency_in_microseconds": 51
		},
		{
			"percentile": 99.9,
			"latency_in_microseconds": 1500
		},
		{
			"percentile": 99.99,
			"latency_in_microseconds": 9598
		},
		{
			"percentile": 99.999,
			"latency_in_microseconds": 15021
		}

The same variance happens when using nginx, or a rust web server. The same on the M1, in a Linux docker, or on a Linux EC2 machine. What could cause such a huge variation from 10 microseconds to 15 milliseconds? Thanks

I have a non-stellar SSH connection to some machine; the echo time is... I don't know, 0.2 seconds or so. Anyway, when I start vim, say with no input file, I experience a long delay, of about 3 to 4 seconds, which - when I connect over a lower-latency SSH connection (i.e. within the same network segment), I don't experience. This delay is _before_ vim has written anything to the terminal, and hence before it can expect me to input anything - so I'm wondering what's causing it to try to communicate over the connection, and what exactly it's communicating. I used strace -rr, looking for differences of over 0.1 seconds, and noticed a bunch of these lines:

0.284973 recvfrom(3, "\1\0\v\0\0\0\340\1", 8, 0, NULL, NULL) = 8
     0.133300 recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\1\0\0\0\0\0\1\205\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
     0.138255 recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\2\0\0\0\0\0\377\377?\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
     0.148123 recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\10\4\0M\1\0\0\37\0\0\0\0\0\0\0001\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1364

(and this repeats itself again and again.) What's going on with these messages? And - how can I make vim skip these communications attempts and just come up faster?

I cannot explain this, I will start by listing these speed test results this one is the linux host running mxlinux latest version this one is the windows 10 running inside vmware **on the same computer** I'm trying to figure out why it's THAT bad on linux I have tried the web interface, I'm getting the same kind of result every single time I have tried different distribution (live usb) same thing I have tried different browser, again same thing I don't think it should matter but just in case; I have a gigabyte x570 aorus pro with amd 3900x, 32 gig ram and linux is installed on an nvme I wonder if it's something on my side or speedtest is that bad with linux or auto tunning of sysctl with tcp is the cause or some weird/obscure setting with the kernel. I don't know if it matter too, my router is a mikrotik hap ac2, the same cable is used on the linux host and the windows 10 vm I'm open to any suggestion, tweak, etc. it's really bad for no reason. **edit 1** I forgot to add that from my router with what is called btest, I was able to get the same speed and even faster than on windows 10 with a public server I have a NAS connected to the same router, I ran iperf and I'm getting my full 1gb speed between linux and my NAS **Edit 2** **Edit 3** I think I found the culprit. with mikrotik router, there is a packet sniffer that show a lot of information and one of them is which cpu is used. the mikrotik that I have got 4 cpu under windows, it use multiple cpu while under linux it use only one cpu during the speedtest something under linux is making it use only one cpu **Edit 4** look like I need to upgrade the firmware of my router to the latest version 7.x I am using latest version of 6.x and they added a feature in 7.x where I can lower cpu usage while using vlan. I quickly tested and it look like it solve my issue, getting same speed under linux as windows screenshot and more, I will look at this more this weekend because I tested with a bare bone configuration, not my real one. **Edit 5** so it look like I had to upgrade to latest version 7.x to be able to get the maximum speed under linux... **Edit 6** I had to use the new feature fast track that work with vlan-filtering since 7.2

I'm using a system with multiple storage devices with different write throughput. As explained in question https://unix.stackexchange.com/q/480399/20336 Linux allows a single slow device to use nearly all of disk cache for write buffering by default, which results in poor performance for all processes, even when those processes write to other devices. And the situation gets even worse if any process uses void sync(void) which stalls for writing the whole cache. If a process is e.g. writing an ISO image to slow USB memory stick, the whole image may be in kernel cache from the start of the process so sync() would end up waiting for the whole ISO image being written to the slow memory stick. Even without processes calling void sync(void), all programs will suffer slowdown because instead of using background writing as usual, all the programs are forced to use syncronous writing if current system wide write cached exceeds /proc/sys/vm/dirty_background_bytes bytes. I would want to avoid this by preventing a single (or multiple running in parallel) slow storage devices from slowing down the whole system. As far as I know, this requires limiting the cache usage for write buffering allowed a single slow device. The cache should be big enough that the slow and fast device are the bottleneck for any process using those devices but those processes are not limited by other devices, as happens with default configuration. I've figured out that if I limit /proc/sys/vm/dirty_background_bytes to 50 MB and /proc/sys/vm/dirty_bytes to 200 MB then the latency of the system never gets truly bad but I still get some slowdown when I write to slow devices. I think this happens because all writing is forced syncronous when more than 50 MB dirty cache is in use. If we assume a situation where process writing to the slow memory has 52 MB in the cache and another process wants to write 4 KB file to another fast SSD device, that 4 KB write has to be syncronous, too, which causes slowdown to SSD device speeds instead of running at RAM speeds. On the other hand, the max 200 MB write cache may be too little when writing to truly fast SSD devices because the process generating the data may not be fast enough to fill the cache. As a result, heavily limiting those two kernel settings is a tradeoff for avoiding worst case latency but getting non-optimal average performance. I know that if I set device BDI max_ratio to value less than 100, then that will be used as a percentage of the whole write cache that can be used for that device. However, the default for any device is 100 so any slow device is allowed to force the whole system to slow down. I've tested that setting max_ratio to value way below 100 works nicely and prevents any slowdown from the slow device because that device cannot waste all of the cache. **How to set max_ratio to value less than 100 for all devices, including devices connected in the future?** I could write a script that runs during system boot and configures all the connected devices but then newly connected storage (be it USB, eSATA or any other connection method) is allowed to take all the write cache.

Does anyone see what could be causing this intermittent latency? At various times all commands become slow on login, compute and head nodes and only with logins associated with Lustre, i.e., root and local logins do not have this latency. 45 errors on openat syscall does not sound normal. strace -ttt -T -C -w touch newfilethat 1662064377.602061 execve("/usr/bin/touch", ["touch", "newfilethat"], 0x7fffffffd278 /* 75 vars */) = 0 1662064377.602361 brk(NULL) = 0x55555576b000 1662064377.602385 arch_prctl(0x3001 /* ARCH_??? */, 0x7fffffffd1a0) = -1 EINVAL (Invalid argument) 1662064377.602443 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 1662064377.602516 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/haswell/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602582 stat("/path/to/slurm/current/lib64/slurm/tls/haswell/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602610 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/haswell/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602634 stat("/path/to/slurm/current/lib64/slurm/tls/haswell/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602657 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602680 stat("/path/to/slurm/current/lib64/slurm/tls/haswell/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602703 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602725 stat("/path/to/slurm/current/lib64/slurm/tls/haswell", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602748 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602772 stat("/path/to/slurm/current/lib64/slurm/tls/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602794 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602817 stat("/path/to/slurm/current/lib64/slurm/tls/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602840 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602862 stat("/path/to/slurm/current/lib64/slurm/tls/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602885 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602907 stat("/path/to/slurm/current/lib64/slurm/tls", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602930 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/haswell/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.602955 stat("/path/to/slurm/current/lib64/slurm/haswell/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.602979 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/haswell/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603002 stat("/path/to/slurm/current/lib64/slurm/haswell/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603025 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603048 stat("/path/to/slurm/current/lib64/slurm/haswell/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603071 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603094 stat("/path/to/slurm/current/lib64/slurm/haswell", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603117 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603140 stat("/path/to/slurm/current/lib64/slurm/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603164 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603187 stat("/path/to/slurm/current/lib64/slurm/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603211 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603233 stat("/path/to/slurm/current/lib64/slurm/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603257 openat(AT_FDCWD, "/path/to/slurm/current/lib64/slurm/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603280 stat("/path/to/slurm/current/lib64/slurm", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0 1662064377.603308 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/haswell/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603332 stat("/path/to/slurm/current/lib64/tls/haswell/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603356 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/haswell/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603379 stat("/path/to/slurm/current/lib64/tls/haswell/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603402 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603424 stat("/path/to/slurm/current/lib64/tls/haswell/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603446 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603469 stat("/path/to/slurm/current/lib64/tls/haswell", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603491 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603517 stat("/path/to/slurm/current/lib64/tls/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603540 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603569 stat("/path/to/slurm/current/lib64/tls/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603592 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603615 stat("/path/to/slurm/current/lib64/tls/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603638 openat(AT_FDCWD, "/path/to/slurm/current/lib64/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603661 stat("/path/to/slurm/current/lib64/tls", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603684 openat(AT_FDCWD, "/path/to/slurm/current/lib64/haswell/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603707 stat("/path/to/slurm/current/lib64/haswell/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603730 openat(AT_FDCWD, "/path/to/slurm/current/lib64/haswell/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603753 stat("/path/to/slurm/current/lib64/haswell/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603775 openat(AT_FDCWD, "/path/to/slurm/current/lib64/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603798 stat("/path/to/slurm/current/lib64/haswell/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603820 openat(AT_FDCWD, "/path/to/slurm/current/lib64/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603844 stat("/path/to/slurm/current/lib64/haswell", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603867 openat(AT_FDCWD, "/path/to/slurm/current/lib64/avx512_1/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603890 stat("/path/to/slurm/current/lib64/avx512_1/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603913 openat(AT_FDCWD, "/path/to/slurm/current/lib64/avx512_1/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603935 stat("/path/to/slurm/current/lib64/avx512_1", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.603958 openat(AT_FDCWD, "/path/to/slurm/current/lib64/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.603981 stat("/path/to/slurm/current/lib64/x86_64", 0x7fffffffc3f0) = -1 ENOENT (No such file or directory) 1662064377.604004 openat(AT_FDCWD, "/path/to/slurm/current/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.604028 stat("/path/to/slurm/current/lib64", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 1662064377.604053 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 1662064377.604074 fstat(3, {st_mode=S_IFREG|0644, st_size=93583, ...}) = 0 1662064377.604095 mmap(NULL, 93583, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555537000 1662064377.604117 close(3) = 0 1662064377.604139 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 1662064377.604160 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2607\2\0\0\0\0\0"..., 832) = 832 1662064377.604183 fstat(3, {st_mode=S_IFREG|0755, st_size=3149120, ...}) = 0 1662064377.604203 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x155555535000 1662064377.604225 mmap(NULL, 3938144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x155554f69000 1662064377.604245 mprotect(0x155555122000, 2093056, PROT_NONE) = 0 1662064377.604268 mmap(0x155555321000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x155555321000 1662064377.604294 mmap(0x155555327000, 14176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x155555327000 1662064377.604318 close(3) = 0 1662064377.604345 arch_prctl(ARCH_SET_FS, 0x155555536580) = 0 1662064377.604394 mprotect(0x155555321000, 16384, PROT_READ) = 0 1662064377.604426 mprotect(0x555555769000, 4096, PROT_READ) = 0 1662064377.604448 mprotect(0x155555553000, 4096, PROT_READ) = 0 1662064377.604468 munmap(0x155555537000, 93583) = 0 1662064377.604539 brk(NULL) = 0x55555576b000 1662064377.604559 brk(0x55555578c000) = 0x55555578c000 1662064377.604579 brk(NULL) = 0x55555578c000 1662064377.604601 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.604631 openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3 1662064377.604652 fstat(3, {st_mode=S_IFREG|0644, st_size=2997, ...}) = 0 1662064377.604673 read(3, "# Locale name alias data base.\n#"..., 4096) = 2997 1662064377.604703 read(3, "", 4096) = 0 1662064377.604722 close(3) = 0 1662064377.604750 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.604772 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = 3 1662064377.604793 fstat(3, {st_mode=S_IFREG|0644, st_size=368, ...}) = 0 1662064377.604812 mmap(NULL, 368, PROT_READ, MAP_PRIVATE, 3, 0) = 0x15555554d000 1662064377.604832 close(3) = 0 1662064377.604852 openat(AT_FDCWD, "/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3 1662064377.604872 fstat(3, {st_mode=S_IFREG|0644, st_size=26998, ...}) = 0 1662064377.604892 mmap(NULL, 26998, PROT_READ, MAP_SHARED, 3, 0) = 0x155555546000 1662064377.604911 close(3) = 0 1662064377.604935 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.604956 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_MEASUREMENT", O_RDONLY|O_CLOEXEC) = 3 1662064377.604976 fstat(3, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0 1662064377.604996 mmap(NULL, 23, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555545000 1662064377.605017 close(3) = 0 1662064377.605040 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_TELEPHONE", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605060 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_TELEPHONE", O_RDONLY|O_CLOEXEC) = 3 1662064377.605081 fstat(3, {st_mode=S_IFREG|0644, st_size=59, ...}) = 0 1662064377.605101 mmap(NULL, 59, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555544000 1662064377.605121 close(3) = 0 1662064377.605143 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_ADDRESS", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605164 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_ADDRESS", O_RDONLY|O_CLOEXEC) = 3 1662064377.605184 fstat(3, {st_mode=S_IFREG|0644, st_size=167, ...}) = 0 1662064377.605203 mmap(NULL, 167, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555543000 1662064377.605223 close(3) = 0 1662064377.605245 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_NAME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605266 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_NAME", O_RDONLY|O_CLOEXEC) = 3 1662064377.605286 fstat(3, {st_mode=S_IFREG|0644, st_size=77, ...}) = 0 1662064377.605305 mmap(NULL, 77, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555542000 1662064377.605324 close(3) = 0 1662064377.605347 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_PAPER", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605367 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_PAPER", O_RDONLY|O_CLOEXEC) = 3 1662064377.605388 fstat(3, {st_mode=S_IFREG|0644, st_size=34, ...}) = 0 1662064377.605407 mmap(NULL, 34, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555541000 1662064377.605426 close(3) = 0 1662064377.605450 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605471 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3 1662064377.605492 fstat(3, {st_mode=S_IFDIR|0755, st_size=29, ...}) = 0 1662064377.605514 close(3) = 0 1662064377.605533 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3 1662064377.605554 fstat(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0 1662064377.605573 mmap(NULL, 57, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155555540000 1662064377.605592 close(3) = 0 1662064377.605614 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_MONETARY", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605636 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_MONETARY", O_RDONLY|O_CLOEXEC) = 3 1662064377.605656 fstat(3, {st_mode=S_IFREG|0644, st_size=286, ...}) = 0 1662064377.605675 mmap(NULL, 286, PROT_READ, MAP_PRIVATE, 3, 0) = 0x15555553f000 1662064377.605695 close(3) = 0 1662064377.605717 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_COLLATE", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605738 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_COLLATE", O_RDONLY|O_CLOEXEC) = 3 1662064377.605758 fstat(3, {st_mode=S_IFREG|0644, st_size=2586930, ...}) = 0 1662064377.605777 mmap(NULL, 2586930, PROT_READ, MAP_PRIVATE, 3, 0) = 0x155554cf1000 1662064377.605797 close(3) = 0 1662064377.605825 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605847 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = 3 1662064377.605868 fstat(3, {st_mode=S_IFREG|0644, st_size=3316, ...}) = 0 1662064377.605889 mmap(NULL, 3316, PROT_READ, MAP_PRIVATE, 3, 0) = 0x15555553e000 1662064377.605909 close(3) = 0 1662064377.605931 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.605952 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = 3 1662064377.605972 fstat(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0 1662064377.605991 mmap(NULL, 54, PROT_READ, MAP_PRIVATE, 3, 0) = 0x15555553d000 1662064377.606011 close(3) = 0 1662064377.606035 openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1662064377.606056 openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = 3 1662064377.606076 fstat(3, {st_mode=S_IFREG|0644, st_size=337024, ...}) = 0 1662064377.606095 mmap(NULL, 337024, PROT_READ, MAP_PRIVATE, 3, 0) = 0x1555554e2000 1662064377.606115 close(3) = 0 1662064377.606142 openat(AT_FDCWD, "newfilethat", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = 3 1662064411.577554 dup2(3, 0) = 0 1662064411.577613 close(3) = 0 1662064411.577662 utimensat(0, NULL, NULL, 0) = 0 1662064411.595221 close(0) = 0 1662064411.595453 close(1) = 0 1662064411.595474 close(2) = 0 1662064411.595512 exit_group(0) = ? 1662064411.595645 +++ exited with 0 +++ % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.94 33.971842 539235 63 45 openat 0.05 0.017515 17514 1 utimensat 0.00 0.000348 16 21 close 0.00 0.000280 8 32 30 stat 0.00 0.000204 203 1 execve 0.00 0.000129 7 18 mmap 0.00 0.000106 6 17 fstat 0.00 0.000034 8 4 mprotect 0.00 0.000028 6 4 brk 0.00 0.000020 6 3 read 0.00 0.000018 18 1 dup2 0.00 0.000012 6 2 1 arch_prctl 0.00 0.000011 11 1 munmap 0.00 0.000008 7 1 1 access ------ ----------- ----------- --------- --------- ---------------- 100.00 33.990554 169 77 total

I have a system with very spikey memory usage and lots of disk activity. As such, I'd like to have files in the cache as much as possible but I'd like to have enough free RAM ready to go to quickly respond to user mode processes that require more RAM for a short time. I'm basically trying to reduce latency for sudden user mode memory allocations by "always" having the typical amount of RAM needed ready to go. I'm expecting memory spikes in range 300–500 MB so I'd like to have that amount of RAM free most of the time. I know that increasing /proc/sys/vm/vfs_cache_pressure keeps more of the disk cache in RAM in case of memory pressure so I've set it to 120 (this improves caching of dentries which I've found good for my workload). I also know about /proc/sys/vm/min_free_kbytes which reserves RAM so I've set it to 300000 to make sure there's always 300 MB free ready for use. However, it seems that user mode processes are not allowed to use the RAM reserved by min_free_kbytes. **Is there some kernel setting that would allow me to tell the kernel that it should *try* to keep e.g. 300 MB free RAM ready at all times?** That is, if an user mode program wants RAM, give it immediately from that reserve *and* start a background process to make more free RAM (e.g. write dirty pages, drop some disk caches) to match that reserve again for future user mode processes. If I've understood things correctly, I'm looking for a setting to reserve memory in addition to min_free_kbytes that cannot be used for disk cache but is free to be allocated by any user processes.

I have a raspberry pi 4B running Gentoo, with the 5.15.44 raspi kernel & realtime patches applied to it. The kernel is configured for CONFIG_PREEMPT_RT (big surprise), as well as CONFIG_HZ_PERIODIC, CPU_FREQ_DEFAULT_GOV_PERFORMANCE, (some other options I fail to remember off the top of my head) and the raspi itself is running at 1,800 GHz clock speed. The problem I'm intending to solve is as follows: I have attached an external device to the GPIO pins of the raspi. The device uses 1 pin as both input & output. For both input & output, communication is done in such a way that every 4 us (microseconds), 1 bit is transferred. A low bit (0) is classified by a low signal for 3us, followed by a high signal for 1us. A high bit (1) is classified by a low signal for 1us, followed by a high signal for 3us. The external device will periodically send the sequence of 000000001 (this takes around 36us) to the raspi, after which it will expect a response of 64 bit to be returned. To read the data of the external device, I've written a small program in C++ to read the current pin state, as well as print out the time in microseconds since the start of the program. Here is an example in which the program reads a bit of value 0 being transmitted:

Pin State: 0 | Time Dif: 7365.79
Pin State: 0 | Time Dif: 7365.92
Pin State: 0 | Time Dif: 7366.05
Pin State: 0 | Time Dif: 7366.36
Pin State: 0 | Time Dif: 7366.49
Pin State: 0 | Time Dif: 7366.62
Pin State: 0 | Time Dif: 7366.75
Pin State: 0 | Time Dif: 7366.88
Pin State: 0 | Time Dif: 7367.01
Pin State: 0 | Time Dif: 7367.14
Pin State: 0 | Time Dif: 7367.27
Pin State: 0 | Time Dif: 7367.4
Pin State: 0 | Time Dif: 7367.55
Pin State: 0 | Time Dif: 7367.68
Pin State: 0 | Time Dif: 7367.81
Pin State: 0 | Time Dif: 7367.94
Pin State: 0 | Time Dif: 7368.07
Pin State: 0 | Time Dif: 7368.2
Pin State: 0 | Time Dif: 7368.33
Pin State: 0 | Time Dif: 7368.46
Pin State: 0 | Time Dif: 7368.61
Pin State: 0 | Time Dif: 7368.74
Pin State: 0 | Time Dif: 7368.86
Pin State: 0 | Time Dif: 7368.99
Pin State: 0 | Time Dif: 7369.12
Pin State: 0 | Time Dif: 7369.25
Pin State: 0 | Time Dif: 7369.38
Pin State: 0 | Time Dif: 7369.51
Pin State: 0 | Time Dif: 7369.66
Pin State: 0 | Time Dif: 7369.79
Pin State: 1 | Time Dif: 7369.92
Pin State: 1 | Time Dif: 7370.05
Pin State: 1 | Time Dif: 7370.18
Pin State: 1 | Time Dif: 7370.31
Pin State: 1 | Time Dif: 7370.44

(I should note at this point that this is being printed **after** the program has finished executing, not during the reading of these values) So far so good. However, there is one small problem I haven't been able to figure out. Every now and then, the program will skip a few microseconds. This is quite a problem in this use case, since it means that multiple bits of information will get lost:

Pin State: 0 | Time Dif: 7384.2
Pin State: 0 | Time Dif: 7384.33
Pin State: 1 | Time Dif: 7390.4
Pin State: 1 | Time Dif: 7390.53

As you can see here, it jumped almost 6 entire microseconds, which is 1.5 bits of information lost. This isn't a huge deal if it's just about getting the polling request from the device, but when it comes to sending information to it, I'm afraid everything will arrive jumbled (the device does not perform any sort of error correction). Now my question is, how do I prevent this from happening? Is there some kernel config I can set, or running the process under a certain priority? I've tried chrt -rr 99 but to no avail. Any help would be greatly appreciated.

On a system with an existing multi-stage qdisc setup, we need to introduce extra latency (at least fixed, but fixed with a small variation would be a nice option to have). The canonical way to do this on Linux is to use the netem qdisc. However, this cannot work here because netem does not work with other qdiscs (this is a [well-documented](https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#the-netem-qdisc-does-not-work-in-conjunction-with-other-qdiscs) limitation, and one which a coworker has verified himself). Putting a VM onto the machine that does nothing but netem not only seems like overkill but will also massively complicate routing and WLAN configuration so that’s a step I would prefer not to take. In case it’s relevant, the current setup is a combination of htb (used to limit bandwidth only… we probably should have used tbf instead but this is what we had when I joined) and fq_codel for ECN marking, both with custom patches. I’m not averse to patching this into either… As requested, here is a sample setup, using stock htb/fq_codel for easier testing:

#!/bin/mksh
set -ex
dev=eth0
rate=1000
sudo tc qdisc add dev $dev root handle 1: htb default 1
sudo tc class add dev $dev parent 1: classid 1:1 htb rate ${rate}kbit ceil ${rate}kbit prio 1
sudo tc qdisc add dev $dev parent 1:1 handle 2: fq_codel

I'm trying to disable local timer interrupts for low latency. I have full tickless mode enabled in the kernel config and I have the boot parameter nohz_full set for the cores in question. However when I look at the interrupt count through /proc/interrupts I see the local timer interrupts counting up 1000 times a sec per core meaning full tickless isn't working. The tickless mode documentation says that for tickless to work that only one running process needs to be on that core. When I look at top, I see the following under a given core (core 1 in this example): 19 root RT 0 0 0 0 S 0.0 0.0 0:00.00 1 watchdog/1 20 root -2 0 0 0 0 S 0.0 0.0 0:02.15 1 rcuc/1 21 root RT 0 0 0 0 S 0.0 0.0 0:00.04 1 migration/1 22 root -2 0 0 0 0 S 0.0 0.0 0:00.25 1 ksoftirqd/1 23 root RT 0 0 0 0 S 0.0 0.0 0:00.00 1 posixcputmr/1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 1 kworker/1:0 25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 1 kworker/1:0H I do know that some of these are kernel threads. Are these the reason why my full tickless mode isn't working?

I am trying to build an application that can measure the latency/processing time of graphics frameworks on Linux. My idea is to implement simple programs that react to an input event (e.g. mouse click) by changing the screen's color (e.g. from black to white) with different graphics and UI frameworks (e.g. SDL, OpenGL, Qt, ...). To measure each program's latency, I want to implement a separate program that measures the time from an input event arriving at the machine (I'm using evdev for this) to a pixel being updated in some kind of framebuffer (as close to the application as possible). The second program should also be independent of vblank events, as I'm interested in the time the rendering is done, not the time users might be able to see something. My problem is getting the framebuffer content with the second program. I already managed to get framebuffer content with fbdev or libdrm (based on this tutorial ), but both require the programs to be run in a terminal without an active XServer (which I would prefer due to external validity). I have also already tried to use MIT-XShm to retrieve the X framebuffer's content, but it seems too slow for my problem, even when reading only one pixel (around 4 milliseconds with harsh outliers). This is how I currently use XShm, in case it might help. // get one pixel at coordinates x/y with XShm XShmGetImage(display, rootWindow, image, x, y, 0x00ffffff); // store red value into variable unsigned int color = image->data[2] ; if(color != lastcolor) { log(time_micros(), color); } Is there a quick and reliable way to retrieve framebuffer content (either from X or from DRM) with an XServer running? Or is XShm the best we can do at the moment?

I've noticed when I SSH into a remote machine over a slow link, SSH seems to "stick" after a relatively large amount of data is transferred. For example, typing text over the link is fine, but running ls /etc will freeze the connection for a few minutes. I would understand if the delay was because I was maxing out the connection speed and it returned to normal once the data had been transferred, but the connection freezes for far longer than you'd expect for the amount of data actually involved. The weird thing is that if I open two SSH connections to and from the same machines, when one of them has frozen, the other one still works fine. So I can't be maxing out the connection speed, otherwise they would both freeze at the same time. There is no traffic shaping active on either box or (as far as I can tell) the routers in between, so it shouldn't be anything dropping packets to keep the average transfer speed within a certain range. Can anyone suggest anything that could cause this kind of behaviour, or anything further to check? The same thing happens with scp and sshfs, with scp reporting a huge transfer rate (many MB/sec, then the speed slowly falls back to stalled for a few minutes, then if I'm lucky it'll repeat until the file finishes transferring.) sshfs works but often (not always) after a file is saved the mount point is non-responsive for a few minutes, temporarily blocking any program that tries to access it. EDIT: I tried using iperf and get some interesting stats: Local: 0.0- 0.9 sec 256 KBytes 2.25 Mbits/sec Remote: 0.0- 7.0 sec 256 KBytes 302 Kbits/sec Local: 0.0-15.9 sec 1.00 MBytes 529 Kbits/sec Remote: 0.0-16.0 sec 1.00 MBytes 524 Kbits/sec It looks like below a certain amout of data, the local end can send a lot faster than the remote can receive. I guess this isn't an SSH problem after all. I will investigate some TCP settings that might adjust this, but if anyone knows any for sure please advise!