I'm working on an networking lab tool leveraging QEMU-based VM virtualization and docker technology to run VMs and containers respectively on a Linux host. The underlying lab connectivity is implemented by using linux bridges. I have a linux Ubuntu guest running inside a QEMU VM that features a virtio-net paravirualized interface with TAP backend. Such TAP interface is connected to a linux bridge's port on the host. root@eve-ng62-28:~# brctl show vnet0_3 bridge name bridge id STP enabled interfaces vnet0_3 8000.d63b1f37e4ba no vnet0_9_2 vunl0_3_3 vunl0_7_0 vunl0_9_2 root@eve-ng62-28:~# root@eve-ng62-28:~# ethtool -i vunl0_7_0 driver: tun version: 1.6 firmware-version: expansion-rom-version: bus-info: tap supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no root@eve-ng62-28:~# I'm using Linux ftrace via trace-cmd frontend to dig into some details, see also https://unix.stackexchange.com/questions/797717/tcp-checksum-offloading-on-virtio-net-paravirtualized-interfaces root@eve-ng62-28:~# trace-cmd start -e net:netif_receive_skb_entry -f "name == 'vunl0_7_0'" root@eve-ng62-28:~# root@eve-ng62-28:~# trace-cmd show # tracer: nop # # entries-in-buffer/entries-written: 1/1 #P:48 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | qemu-system-x86-600348 b.... 66505.777999: netif_receive_skb_entry: dev=vunl0_7_0 napi_id=0x0 queue_mapping=1 skbaddr=0000000006a1cc35 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=3 hash=0x00000000 l4_hash=0 len=60 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0x0 As you can see, linux guest sends outgoing TCP packets to the virtio-net network interface setting CHECKSUM_PARTIAL(3) tag in the ip_summed field within sk_buff struct. My question is related to the TASK-PID field shown by trace-cmd show. 600348 is the PID of the qemu-system-x86_64 process's instance associated to the VM. As required I edit this to provide the question: is the TASK-PID shown the PID or TID of the process/thread that is the context the TAP driver runs into ?

I've come across the following DTrace one-liner on https://wiki.freebsd.org/DTrace/One-Liners :

# Summarize TCP life span in seconds:
dtrace -n 'fbt::tcp_close:entry { 
    @["TCP life span (seconds):"] = quantize((uint32_t)(ticks - args->t_starttime) / hz); 
}'

It does not work on latest FreeBSD 15.0-CURRENT anymore. It errors out with:

dtrace: invalid probe specifier fbt::tcp_close:entry {
 @["TCP life span (seconds):"] = 
    quantize((uint32_t)(ticks - args->t_starttime) / hz); 
}:
  in action list: no symbolic type information is available for kernel`ticks:
    No type information available for symbol

error text above manually wrapped/indented Any ideas why DTrace cannot find ` ticks `? It finds hz `` just fine. According to the wiki page, all the one liners were tested in the past so that is not a typo. Also, I tried including sys/kernel.h as it seems to be where ticks is declared, but that did not help (it ends up needing sys/queue.h as well and even then it does not work).

I am analyzing some trace-cmd (event) output filtered for some specific PID. The workflow is as follows (parse.py is just a custom python to parse for specific PIDs, $EVENTS_ARGS is just a list of the events to be traced):

trace-cmd record -b 10000 $EVENT_ARGS -o output_file -F -c executable_to_trace
trace-cmd report outputfile > trace.txt
parse.py trace.txt

And parsing produces a list of events for the chosen PID(s). Here's a relevant excerpt:

-1342    3231.385036: sched_switch:         taskset:1342  D ==> swapper/4:0 
	sh-1342    3231.385357: task_rename:          pid=1342 oldcomm=taskset newcomm=launch_function oom_score_adj=0
	sh-1342    3231.385473: sched_switch:         launch_function:1342  D ==> swapper/4:0 
	sh-1342    3231.385660: sched_process_exec:   filename=/shared/loadgen/payload/launch_function.out pid=1342 old_pid=1342
	sh-1342    3231.385830: sched_switch:         launch_function:1342  D ==> swapper/4:0

However, even after these **task_rename** and **sched_process_exec** events, following events (*like sched_switch*) still log the comm field as sh, even though the actual process name should have changed. To clarify further, trace-cmd seems to use *ftrace*'s *record-cmd* option which matches to the *comm* I am seeing in the trace-cmd output, but shouldn't that handle *task_rename*s better? I know I can rewrite this during post-processing by replacing the comm with the latest task_rename, but I’m curious why the original trace still shows the outdated comm *6.5.9 kernel*

I know that I can redirect the xtrace output to some_file with something like this: exec 2>> some_file set +x ...but this sends to some_file not only the xtrace output, but also any other content originally sent to fd 2, which includes most error messages and warnings, all unrelated to xtrace. Is there a way to capture *only* the xtrace output in some_file? I should add that I'm looking for a way to do this that would distort as little as possible the xtrace output itself, and the timing information gathered through a PS4 setting like, e.g. zmodload zsh/datetime export PS4='${(j::)epochtime} %N:%i> '

tldr: Does mtrace still work or am I just doing it wrong? I was attempting to use mtrace and have been unable to get it to write data to a file. I followed the instructions in man 3 mtrace: t_mtrace.c:

#include 
#include 
#include 

int
main(int argc, char *argv[])
{
    mtrace();

    for (int j = 0; j < 2; j++)
        malloc(100);            /* Never freed--a memory leak */

    calloc(16, 16);             /* Never freed--a memory leak */
    exit(EXIT_SUCCESS);
}

Then running this in bash:

gcc -g t_mtrace.c -O0 -o t_mtrace
export MALLOC_TRACE=/tmp/t
./t_mtrace
mtrace ./t_mtrace $MALLOC_TRACE

but the file /tmp/t (or any other file I attempt to use) is not created. When I create an empty file with that name it remains zero length. I've tried using relative paths in the MALLOC_TRACE. I tried adding setenv("MALLOC_TRACE", "/tmp/t", 1); inside the program before the mtrace() call. I've tried adding muntrace() before the program terminates. I've tried these tactics on Ubuntu 22.04 and Fedora 39, and I get the same result: the trace file is empty. The ctime and mtime on the file (if I create it in advance) are unchanged when I run the program. I've verified the permissions of the file and its parent directory are read/writable. strace isn't showing that the file in question is stated, much less opened. This occurs using Glibc 2.35 on Ubuntu and 2.38 on Fedora. This isn't a question on how to profile or check for memory leaks. I realize I can do this with valgrind or half a dozen other programs, this is mostly a curiosity and me wanting to know if this is a bug that might need to be patched or whether the man page needs updating (or whether I'm just misapprehending something and the only problem is sitting in my chair).

I am currently working on some PCI-e device stuff, that requires me to get the trace for DMA transactions on a specific (say GPU or NIC ) PCI card. After doing some research I found a useful post by xealits, but it seems doesn't tell how many bytes are read/written in a certain DMA transaction. I then found a kernel module named dma_debug may fully meet my expectations I tried to recompile kernel v6.7 with configure options: CONFIG_DMA_API_DEBUG=y CONFIG_DEBUG_FS=y CONFIG_DMA_API_DEBUG_SG=y CONFIG_DMA_API_DEBUG_FENCE=y CONFIG_DMA_API_DEBUG_EVENTS=y and add dma_debug.debug_dma=1 to grub default booting variables. But unfortunately, dmesg seems completely identical to when booting the old kernel. Also, dma_debug didn't show up in /sys/kernel/debug. Am I missing something? How can I get the dma_debug to work？ If this module is deprecated, is there any substitute method?

I am using the bpf_printk() helper to print to the kernel debug to trace some BPF programs. The usage is as follows:

#include 

#include 
#include 

#include 

char LICENSE[] SEC("license") = "GPL";

SEC("sk_msg")
int http_state_machine(struct sk_msg_md *msg) {
  long len = (long)msg->data_end - (long)msg->data;
  if (len > 0) {
    bpf_printk("Message length: %ld\n", len);
  }

  return SK_PASS;
}

Then I try to read the trace_pipe via sudo cat /sys/kernel/debug/tracing/trace_pipe > check.log. My goal is to read only the first few lines of the trace, instead of the entire trace. However, this takes a very long time (in the order of minutes) to move all the output into check.log. I learnt from [this](https://unix.stackexchange.com/a/684146/441335) answer, that the output of the trace_pipe is also the same in the static file /sys/kernel/debug/tracing/trace. And the trace file has nearly 10k lines. My question is: How can I clear the entire output of the trace_pipe and/or trace - either by moving the entire output to a new file or discarding the output (after having read the first few lines)?

Suppose a user runs the following command: zcat file.gz | grep something | gzip > grepped.gz I'm looking for a kernel feature (a BPF filter perhaps?) that would note all of the execves, chain together their stdins/stdouts and reconstruct that in a similar form, putting it into system logs. Is there a way to do that without interfacing with the shells?

How to use a Systemtap probe on a kernel module function while that module is loaded. I am trying to print the functions being called for a particular module while it is being loaded. Suppose I have a kernel module hello.ko which is not loaded. Now I want to trace a function hello_init() of that module. I tried with the following Systemtap script but it did not work. Command: stap test10.stp -c "modprobe hello" --> this printed nothing Systemtap script:

#!/usr/bin/env stap

global traces

probe module("hello").function("hello_init") {
    printf("hello")
    print_stack(backtrace())
}

Kernel module:

#include        
#include        
#include          
MODULE_LICENSE("GPL");

static int __init hello_init(void)
{
printk(KERN_INFO "Hello world\n");
return 0;
}

static void __exit hello_exit(void)
{
printk(KERN_INFO "Done\n");
}

module_init(hello_init);
module_exit(hello_exit);

On Windows, there is a program available for download which is part of what's called the sysinternals suite of tools. This program is called Process Explorer 64 aka ProcExp64. The program is useful for amongst many other things, identifying potential malware on a machine because one common behavior of malware is to launch and kill processes essentially faster than the typical program and/or the human eye could perceive/notice. This posed a challenge using the Windows Task Manager, because a malware may quickly launch, launch a child process, inject into another process and then kill the child all in under 1 second, for example (potentially even under 200ms). ProcExp64 has what's called Difference Highlight Duration which highlights a killed process to a specified color and keeps it in the process list for some duration, even after it's killed. In fact, the user can even specify to "scroll to new process", which I simply use sort by PID for on linux in htop. I do see an update time setting in htop but if I make that larger, it may miss the launch altogether. On Linux, I've found a similar issue and I'm not sure how to solve it. In this case, I am not trying to observe malware, but I am trying to observe other, non-malicious processes that begin and exit quickly. An example is that I noticed that when I enter the command ps aux | grep ld-linux, I get an output, but then when I view htop, there is no such process still running because I believe the ld-linux executable exits and remains present only in shared object (.so) space rather than as a separate process (I could be wrong, but this is how I understand what's happening). One reason I suspect this is the case is that every time I run the aforementioned command, the PID is incremented further and further for ld-linux because I think that is the instance that was loaded in the actual loading process for ps itself (again, just speculation). Is there a way to observe this in action using htop or some other tool to put some kind of delay on the existence of the ld-linux loader process in the process table/list, and otherwise highlight it so that it stands out?

I'd like to be able to concatenate a string and an int using strjoin(), e.g., strjoin("ada", args->unit_number); but in order to do that I have to make sure that the int is first converted to a string. --- I'm running DTrace on FreeBSD 13.1-RELEASE.

I want to trace every IO operation an application does in its lifetime and don't want any caching effect to interfere. I cannot change the application source code, using a shim would be OK. Performance is not an issue, I just want to record what an application does even if it takes hours. The final goal of this is to use the traces to try to improve heuristics related to filesystems, for example, prefetching. Ideally the output would be something like , but just would probably be enough. Here are my ideas so far and why I think they won't work: 1. Trace all block-level accesses, for example, IOs sent to an SSD. This requires disabling readahead and page cache. The former is easy to do, the latter seems to be impossible. Flushing the page cache every X units of time is out of the question since there is no guarantee every read bypasses the page cache. Disabling only write-cache is also not enough. 2. Using eBPF to trace all filesystem operations, like this article does . There are two issues here: apparently only the first request of a file is recorded (from the blog: "The probe we implemented ... identify the first time a block is requested"), and I wrote a replica of what the blog does and the reads are not printed outSource is here][2] (edit: the reason why it wasn't showing is that the file I was reading was on an NFS drive. Whoops). The BCC script shows some IOs happening on the system, but none from the touchblk application are printed out. I have no idea why or how. Edit: here are some other ideas, but I don't know how to implement yet: 3. Always mark pages as unavailable, so that they are always fetched from disk. 3. TLB poisoning. Am I overthinking this? Is there either a solution that I didn't think about that does this or a fix to one of the problems I identified? Thanks!

I am working on software that communicates with a PCI card through direct memory access (DMA) transactions. My programs use a suit of drivers and a library that handles the DMA. Everything runs on a Red Hat Linux. To test and measure the performance of my programs I would like to trace the start and end of the DMA transactions. Now I do this by looking at a couple of functions in the library: * dma_from_host and dma_to_host that initiate the transactions by configuring the values in the registers of the card and writing 1 to a register called DMA_DESC_ENABLE * dma_wait that waits until the transaction has finished by continuously checking the value of the DMA_DESC_ENABLE register. But I would like to have a more robust confirmation that a transaction has started and a more precise signal when the transaction has ended. Something from Linux or hardware itself would be the best. I understand that in principle it is a cumbersome situation. The idea of DMA is that the hardware (the PCI card or the DMA controller on the motherboard) copies things directly into the memory of the process, bypassing the CPU and the OS. But I hope that it does not just copy things into RAM without notifying the CPU somehow. Are there some standard ways to trace these transactions or it is very platform-specific? Are there some special interrupts that notify the CPU about the start and end of the DMA? I could not spot anything like that in the drivers that I use. But I am not experienced with drivers, so I could have easily looked at wrong places. Another idea, are there any PMU-like hardware monitors that could provide this information? Something that just counts transactions on PCI lanes? Also an idea, do I understand right that one could write a custom DMA-tracer as a Linux module or a BPF program that would continuously check the value of that DMA_DESC_ENABLE register? Is this a viable approach? Are there known tracers like that?

Sounds dumb, but can you extract application name from a packet/pcap. For eg: If a packet destination is to chrome process, can you extract that information from packet?

I would like to generate a log of all virtual memory accesses performed in user mode and kernel mode as a result of running some program. Besides collecting memory access locations, I also want to capture other state information (e.g., instruction pointer, thread identifier). I anticipate that I won't be able to collect all of my desired statistics with any tool out of the box. I intend on doing this profiling off-line, so I'm not concerned about the performance impacts. In fact, depending on what is available, it would be helpful to know which tools can record all memory accesses and which can only sample. I was originally going to augment Valgrind's lacky tool until I realized that it only records user mode memory accesses. Looking into what other tools I might use, I'm at a loss at how I can quickly determine which tool is capable of capturing the information I want. Here are some resources I've found that have gotten me started: - [Brendan Gregg's _Choosing a Linux Tracer_](http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html) - [Julia Evans' _Linux tracing systems & how they fit together_](https://jvns.ca/blog/2017/07/05/linux-tracing-systems/)

How can this be done? I don't see any applicable option in the manual. I have positively checked that indentation breaks after ten million lines. You can check it like: $ (for i in seq 0 10000000; do echo "$i"; done) | nl I don't often generate so many lines, but I don't want it to break like it does. How can this be done?

I experience quite a bit of fan activity on my laptop, even the machine should be idle. When I take a look at my processes with top I have a pgrep process appear once in a while, taking some CPU. It is run as root user. Does anyone know what this is for? **Is it something to be aware of, or is it just a normal system routine?** Operating System: Debian GNU/Linux bullseye/sid Kernel: Linux 5.6.0-2-686-pae Architecture: x86 I run a while true; do pgrep -au root | grep pgrep; done loop to see when it is launched. The log looks like this: 22713 pgrep -n startx 22713 pgrep -n startx 22767 pgrep -n startx 22767 pgrep -n startx[...] **is there a way I can see what initiates the process?** ______ **Update:** running the script from @hauke-laging I can see that /etc/acpi/power.sh seems to be the parent process: ------------------------- 10359 10358 root root pgrep -n startx parent process: 10358 5645 root root /bin/sh /etc/acpi/power.sh here it is: #!/bin/sh test -f /usr/share/acpi-support/key-constants || exit 0 . /usr/share/acpi-support/power-funcs . /usr/share/acpi-support/policy-funcs if { CheckPolicy || CheckUPowerPolicy; }; then exit fi cat /usr/share/acpi-support/power-funcs | grep pgrep **there we go:** startx=$(pgrep -n startx || :)

**Question** Does the tracing pipes in Linux, for tracing events, accepts simultaneous readers? **Context** I am trying to get traces from the

subsystem while running some benchmarks. This subsystem reports hardware error events using trace events. The tool I am using to get the events in the **RASdaemon** tool. I know the tool subscribe to the

FIFO on each cpu. Furthermore, the benchmark subscribe to performance events which, **I believe that**, are also advertised through the same pipes. The

works like the

which has the following description: > The output is the same as the “trace” file but this file is meant to be streamed with live tracing. Reads from this file will block until new data is retrieved. Unlike the “trace” file, this file is a consumer. This means reading from this file causes sequential reads to display more current data. Once data is read from this file, it is consumed, and will not be read again with a sequential read. The “trace” file is static, and if the tracer is not adding more data, it will display the same information every time it is read. Unlike the “trace” file, opening this file for reading will not temporarily disable tracing. I understand the concept of FIFO being a consumer file and why **each read will produce different data**. However, do the Kernel make any arrangements for multiple subscribers (reader) to these events?

The gvim program is stuck for 3 seconds before opening any file. This issue is isolated to gvim, vim opens just fine. Therefore, the cause must be somewhere in X or some related service and not vim. I have tried disabling all my vim plugins and the vimrc file. Here is the symptom: $ time strace -o strace.gvim -f -yy -T gvim -c q time: Real 0m3.5s User 0m0.0s System 0m0.1s While vim runs fine: $ time strace -o strace.vim -f -yy -T vim -c q time: Real 0m0.2s User 0m0.0s System 0m0.0s To get to the bottom of this, I am looking for what is taking such a long time: ...It looks like it is related to the dbus service somehow. $ grep ' 0x7ffcaa8f8670) = 0 4848 kill(4907, SIGKILL) = -1 ESRCH (No such process) 4848 write(1, "unix:abstract=/tmp/dbus-8t4OQuvB"..., 73) = 73 @@@@@@@@ 4586 "unix:abstract=/tmp/dbus-8t4OQuvB"..., 1024) = 73 4848 write(1, "+\23\0\0", 4 4586 read(7, -- 4586 "", 1024) = 0 4586 wait4(4848, @@@@@@@@ 4900 "", 4) = 0 4848 +++ exited with 0 +++ 4900 write(2>, "EOF in dbus-launch reading PID f"..., 47) = 47 -- 4586 close(4) = 0 4586 poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, 0 @@@@@@@@ 4579 "ok\0", 9) = 3 4586 ) = 0 (Timeout) 4586 poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, 0 Looking further up, I discover that the ~/.dbus folder is owned by root: 4848 mkdir("/home/jaroslav/.dbus", 0700) = -1 EEXIST (File exists) 4848 mkdir("/home/jaroslav/.dbus/session-bus", 0700) = -1 EACCES (Permission denied) 4848 write(2, "Unable to create /home/jaroslav/"..., 50) = 50 - $ ls -ld ~/.dbus drwx------ 1 root root 1 191206 18:50 /home/jaroslav/.dbus Changing the owner solves the issue and gvim starts instantaneously. # strace is fine, but Is it possible to somewhow trace all kernel calls that take more than 3 seconds and print their stacks (Linux 4.19.72)?

I am trying to install LTTng. I was installing lttng-tools-2.6.0 and got the following error: configure: error: Cannot find libuuid uuid_generate nor libc uuid_create. Use LDFLAGS=-Ldir to specify their location. I don't know how to specify -L/usr/local/lib in LDFLAGS.