Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
2
votes
1
answers
29
views
DTrace cannot reference the external kernel variable `ticks` on FreeBSD
I've come across the following DTrace one-liner on https://wiki.freebsd.org/DTrace/One-Liners: ``` # Summarize TCP life span in seconds: dtrace -n 'fbt::tcp_close:entry { @["TCP life span (seconds):"] = quantize((uint32_t)(`ticks - args[0]->t_starttime) / `hz); }' ``` It does not work on latest Free...
I've come across the following DTrace one-liner on https://wiki.freebsd.org/DTrace/One-Liners :
# Summarize TCP life span in seconds:
dtrace -n 'fbt::tcp_close:entry {
@["TCP life span (seconds):"] = quantize((uint32_t)(ticks - args->t_starttime) /
hz);
}'
It does not work on latest FreeBSD 15.0-CURRENT anymore. It errors out with:
dtrace: invalid probe specifier fbt::tcp_close:entry {
@["TCP life span (seconds):"] =
quantize((uint32_t)(ticks - args->t_starttime) /
hz);
}:
in action list: no symbolic type information is available for kernel`ticks:
No type information available for symbol
error text above manually wrapped/indented
Any ideas why DTrace cannot find `
ticks `? It finds
hz `` just fine.
According to the wiki page, all the one liners were tested in the past so that is not a typo. Also, I tried including sys/kernel.h
as it seems to be where ticks
is declared, but that did not help (it ends up needing sys/queue.h as well and even then it does not work).
Mateusz Piotrowski
(4983 rep)
Jun 23, 2025, 01:09 PM
• Last activity: Jun 24, 2025, 08:53 AM
1
votes
1
answers
38
views
Why does timestamp increase after a call to chill() but vtimestamp and walltimestamp do not?
I'd like to understand why calling `chill()` in a DTrace action block increases the `timestamp` variable, but not `vtimestamp` and `walltimestamp`. Here's an example showing `timestamp` increasing after a call to `chill()`: ```console # dtrace -w -c true -n 'pid$target:::entry {self->t = $1; chill(1...
I'd like to understand why calling
chill()
in a DTrace action block increases the timestamp
variable, but not vtimestamp
and walltimestamp
.
Here's an example showing timestamp
increasing after a call to chill()
:
# dtrace -w -c true -n 'pid$target:::entry {self->t = $1; chill(1); printf("%d\n", $1 - self->t);}' timestamp
dtrace: description 'pid$target:::entry ' matched 3082 probes
dtrace: allowing destructive actions
dtrace: pid 6734 has exited
CPU ID FUNCTION:NAME
5 83475 _r_debug_postinit:entry 11258
5 85771 atexit:entry 2218
5 86468 __libc_atexit:entry 491
5 86428 exit:entry 441
5 85397 __cxa_thread_call_dtors:entry 441
5 86213 __cxa_finalize:entry 447
5 86213 __cxa_finalize:entry 565
5 83470 _rtld_addr_phdr:entry 454
5 86213 __cxa_finalize:entry 431
5 83470 _rtld_addr_phdr:entry 1645
5 84405 _exit:entry 432
If we run the same script but use walltimestamp
(or vtimestamp
), we'll see the counter did not increase:
# dtrace -w -c true -n 'pid$target:::entry {self->t = $1; chill(1); printf("%d\n", $1 - self->t);}' walltimestamp
dtrace: description 'pid$target:::entry ' matched 3082 probes
dtrace: allowing destructive actions
dtrace: pid 6707 has exited
CPU ID FUNCTION:NAME
4 83475 _r_debug_postinit:entry 0
4 85771 atexit:entry 0
4 86468 __libc_atexit:entry 0
4 86428 exit:entry 0
4 85397 __cxa_thread_call_dtors:entry 0
4 86213 __cxa_finalize:entry 0
4 86213 __cxa_finalize:entry 0
4 83470 _rtld_addr_phdr:entry 0
4 86213 __cxa_finalize:entry 0
4 83470 _rtld_addr_phdr:entry 0
4 84405 _exit:entry 0
This is understandable for vtimestamp
, as it does not increase when executing DTrace code, but I don't understand walltimestamp
's behavior here.
I'm running FreeBSD 13.1-RELEASE-p1 here on amd64.
Mateusz Piotrowski
(4983 rep)
Nov 17, 2022, 02:08 PM
• Last activity: Nov 24, 2022, 11:26 AM
2
votes
1
answers
59
views
How to convert an int to a string in DTrace?
I'd like to be able to concatenate a string and an int using `strjoin()`, e.g., strjoin("ada", args[1]->unit_number); but in order to do that I have to make sure that the int is first converted to a string. --- I'm running DTrace on FreeBSD 13.1-RELEASE.
I'd like to be able to concatenate a string and an int using
strjoin()
, e.g.,
strjoin("ada", args->unit_number);
but in order to do that I have to make sure that the int is first converted to a string.
---
I'm running DTrace on FreeBSD 13.1-RELEASE.
Mateusz Piotrowski
(4983 rep)
Aug 30, 2022, 04:03 PM
• Last activity: Aug 30, 2022, 04:27 PM
2
votes
1
answers
224
views
How to trace and print out all files that are accessed / used when bash starts?
I'm having a problem and need to print out to the console all files that are being access each time `bash` is executed on macOS. Is there a relatively easy method for doing this? Perhaps `dtrace` might make this fairly straight forward?
I'm having a problem and need to print out to the console all files that are being access each time
bash
is executed on macOS.
Is there a relatively easy method for doing this? Perhaps dtrace
might make this fairly straight forward?
ylluminate
(686 rep)
Jan 6, 2020, 09:05 PM
• Last activity: Jan 8, 2020, 05:17 AM
6
votes
1
answers
1166
views
Finding the source of CoW page faults on OS X
I'm trying to find the source of CoW page faults in some C code on OS X. I would like to use the [vminfo DTrace provider](http://dtrace.org/guide/chp-vminfo.html), but `vminfo` isn't available on OS X. On Linux, I can use SystemTap to print a stack trace on a CoW fault. Is there a way to do this on...
I'm trying to find the source of CoW page faults in some C code on OS X. I would like to use the [vminfo DTrace provider](http://dtrace.org/guide/chp-vminfo.html) , but
vminfo
isn't available on OS X. On Linux, I can use SystemTap to print a stack trace on a CoW fault. Is there a way to do this on OS X?
Aaron Patterson
(163 rep)
Mar 25, 2016, 06:43 PM
• Last activity: Apr 5, 2019, 10:41 PM
2
votes
1
answers
75
views
How to measure changes to the filesystem done by a certain application?
I am using a command line application which creates a great amount of intermediate artifacts to the file system and then at some point deletes them. On environments where storage space is constrained this can pose a problem. I'd like to know if there's a tool which would allow me to run the applicat...
I am using a command line application which creates a great amount of intermediate artifacts to the file system and then at some point deletes them.
On environments where storage space is constrained this can pose a problem.
I'd like to know if there's a tool which would allow me to run the application under and then summarise the changes that were done to the filesystem when it terminates (mainly amount of created/deleted files and their sizes).
carlossless
(243 rep)
Aug 12, 2016, 12:56 PM
• Last activity: Nov 1, 2018, 01:34 PM
1
votes
1
answers
75
views
Modifying standard DTT scripts to use -c instead of $1
There are lots of great dtrace programs out there and some of them require a PID to be passed to them, in order for them to trace that PID. Inside the dtrace scripts that argument is stored in $1. Is it possible to just simply replace $1 with $target and -c on the cmdline, and rest of the script log...
There are lots of great dtrace programs out there and some of them require a PID to be passed to them, in order for them to trace that PID. Inside the dtrace scripts that argument is stored in $1. Is it possible to just simply replace $1 with $target and -c on the cmdline, and rest of the script logic remains the same?
Is it just as simple as I think it should be?
Junaid Shahid
(79 rep)
Nov 22, 2017, 02:44 AM
• Last activity: Sep 5, 2018, 06:01 PM
2
votes
0
answers
576
views
Tools to debug slow NFS on OSX
I have 2 mac pros connected to a Nexenta NAS. They have a fast network connection, the same ping time, and the routes to the server are the same (both are plugged into the same switch). One computer has very slow reads and writes on the NAS when the files are larger that about 20 KB (found with `iop...
I have 2 mac pros connected to a Nexenta NAS. They have a fast network connection, the same ping time, and the routes to the server are the same (both are plugged into the same switch).
One computer has very slow reads and writes on the NAS when the files are larger that about 20 KB (found with
ioping
). It responds to ping, doesn't drop packets, it can be accessed with SSH and works fine.I've checked the network over that interface and as far as I can tell there is nothing wrong unless I am connecting to the NAS **with NFS specifically**.
**What tools are available on Mac OS to understand NFS performance problems?**
Dan
(9622 rep)
Dec 15, 2017, 10:54 PM
2
votes
0
answers
120
views
osx dtruss doesn't report opening ~/.bash_profile while tracing bash?
Let's assume I have this line at the start of my `~/.bash_profile`: echo "*** THIS IS ~/.bash_profile RUNNING ***" On a Linux machine (Ubuntu 14.04), I can inspect which files `bash` loads at startup with `strace`, so I do this: strace -f bash --login 2>&1 | tee /tmp/log.strace # type [ENTER] here,...
Let's assume I have this line at the start of my
~/.bash_profile
:
echo "*** THIS IS ~/.bash_profile RUNNING ***"
On a Linux machine (Ubuntu 14.04), I can inspect which files bash
loads at startup with strace
, so I do this:
strace -f bash --login 2>&1 | tee /tmp/log.strace
# type [ENTER] here, or "hello" [ENTER], then Ctrl+C to exit
grep 'bash_' /tmp/log.strace
The result is as expected:
> faccessat(AT_FDCWD, "/etc/profile.d/bash_completion.sh", R_OK) = 0
> open("/etc/profile.d/bash_completion.sh", O_RDONLY) = 3
> open("~/.bash_profile", O_RDONLY) = 3
> read(3, "echo \"*** THIS IS ~/.bash_profil"..., 48) = 48
> write(1, "*** THIS IS ~/.bash_profile RUNN"..., 40*** THIS IS ~/.bash_profile RUNNING ***
However, I've logged in on an OSX 10.9 machine via ssh
, and I need to do the same. Since there is no strace
on OSX, I used dtruss
, like this:
dtruss -f bash --login 2>&1 | tee /tmp/log.dtruss
# type [ENTER] here, or "hello" [ENTER], then Ctrl+C to exit
grep 'bash_' /tmp/log.dtruss
Strangely, there seems to be no mention of ~/.bash_profile
ever being loaded:
> $ grep 'bash_' /tmp/log.dtruss
> $
..., - even if, if I just run bash --login
on that OSX machine, I can see the above echo
being printed, which means ~/.bash_profile
must be loaded?!
Of course, dtruss
does report accessing other files:
$ grep 'open\|stat' /tmp/log.dtruss
41819/0xce5a2: stat64("/AppleInternal\0", 0x7FFF5CBC2A88, 0x0) = -1 Err#2
41819/0xce5a2: stat64("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x7FFF5CBC23F8, 0x7FFF5CBC3330) = 0 0
41819/0xce5a2: open("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x0, 0x0) = 3 0
41819/0xce5a2: stat64("/usr/lib/libncurses.5.4.dylib\0", 0x7FFF5CBC2208, 0x7FFF5CBC30A0) = 0 0
...
41819/0xce5a2: open("/dev/tty\0", 0x6, 0x7FFF79D33940) = 3 0
41819/0xce5a2: open_nocancel("/usr/share/locale/en_US.UTF-8/LC_COLLATE\0", 0x0, 0x1B6) = 3 0
...
41819/0xce5a2: stat64("~/.fastlane/bin/bash\0", 0x7FFF5CBC37E0, 0x0) = -1 Err#2
41819/0xce5a2: stat64("/usr/bin/bash\0", 0x7FFF5CBC37E0, 0x0) = -1 Err#2
41819/0xce5a2: stat64("/bin/bash\0", 0x7FFF5CBC37E0, 0x0) = 0 0
41819/0xce5a2: stat64("/bin/bash\0", 0x7FFF5CBC3820, 0x0) = 0 0
...
..., for instance we can see that the $HOME
directory is accessed as part of searching through $PATH
(and in fact, that PATH="~/.fastlane/bin:$PATH"
*is* set in that very same ~/.bash_profile
).
My question is - how come this happens? Is there a special invocation of dtruss
that I need to use, so it reports when files like ~/.bash_profile
are accessed? Or is there another program on OSX that I should use to achieve the same kind of file open tracing, that strace
allows on Linux? Or is the process starting on OSX so different, that it "loads" ~/.bash_profile
for a bash
process somehow in the background, before bash
ever starts running as a standalone process?
sdaau
(7098 rep)
Oct 24, 2017, 03:00 PM
• Last activity: Oct 24, 2017, 03:07 PM
3
votes
0
answers
204
views
Unix OSX invisible process, how do I identify further information?
Using execsnoop -v, I note a dtrace error for two processes. ID3 (ID 630) and ID2 (ID 360). ~ 03:59 am ∆:ps -p 260 PID TTY TIME CMD 260 ?? 0:02.36 /usr/libexec/UserEventAgent (Aqua) ~ 03:59 am ∆:ps -p 630 PID TTY TIME CMD ~ 03:59 am ∆:sudo execsnoop -v Password: STRTIME UID PID PPID ARGS dtrace: err...
Using execsnoop -v, I note a dtrace error for two processes. ID3 (ID 630) and ID2 (ID 360).
~ 03:59 am ∆:ps -p 260
PID TTY TIME CMD
260 ?? 0:02.36 /usr/libexec/UserEventAgent (Aqua)
~ 03:59 am ∆:ps -p 630
PID TTY TIME CMD
~ 03:59 am ∆:sudo execsnoop -v
Password:
STRTIME UID PID PPID ARGS
dtrace: error on enabled probe ID 2 (ID 260: syscall::execve:return): invalid >kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 3 (ID 630: syscall::posix_spawn:return): >invalid kernel access in action #8 at DIF offset 0
dtrace: error on enabled probe ID 2 (ID 260: syscall::execve:return): invalid >kernel access in action #8 at DIF offset 0
^C
~ 04:01 am ∆:ps -p 3
PID TTY TIME CMD
~ 04:01 am ∆:ps -p 2
PID TTY TIME CMD
My understanding is this is generated by a process that has a hold on dtrace being active for that process.
I note that the processes do not show up in the top command list nor in Activity Monitor. The two processes reoccur with a full restart, and are hence consistent and I presume some OSX process. Just puzzling that they can not be fully identified.
Curious to understand what is going on here.
Cam_Aust
(131 rep)
Dec 26, 2016, 04:19 PM
• Last activity: Dec 26, 2016, 10:30 PM
6
votes
1
answers
703
views
DTrace to trap any chmod applied to certain files
Underneath the Mac OS X directory `/audit` I have certain files which users can access and `chmod` to their liking. I need to audit any `chmod` done on any files by recording the time, user and file being `chmod`, especially the latter. I can `dtrace -n 'syscall::chmod:entry'` and detect the events,...
Underneath the Mac OS X directory
/audit
I have certain files which users can access and chmod
to their liking.
I need to audit any chmod
done on any files by recording the time, user and file being chmod
, especially the latter.
I can dtrace -n 'syscall::chmod:entry'
and detect the events, how do I read the first argument to chmod
?
man 2 chmod
tells me the path is in the first argument:
chmod(const char *path, mode_t mode);
but how can I read args
? I think I am doing this the wrong way around.. perhaps entry doesn't correspond to the actual syscall?
If I have a probe I can monitor, how can I check which parameters it offers for access and what types they are? I am assuming some pointers will need to be dereferenced based on their data layout..
Robottinosino
(5431 rep)
Sep 10, 2011, 10:42 AM
• Last activity: Jul 18, 2016, 09:38 PM
14
votes
1
answers
7260
views
Error on enabled probe: syscall::open_nocancel:entry): invalid user access in action #2 at DIF
I've the following one-liner to show files opened by process: sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }' however I've plenty of repeated errors such as: > `dtrace: error on enabled probe ID 4 (ID 946: syscall::open_nocancel:entry): invalid user access in acti...
I've the following one-liner to show files opened by process:
sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
however I've plenty of repeated errors such as:
>
dtrace: error on enabled probe ID 4 (ID 946: syscall::open_nocancel:entry): invalid user access in action #2 at DIF offset 24
>
> dtrace: error on enabled probe ID 7 (ID 160: syscall::open:entry): invalid user access in action #2 at DIF offset 24
I'm aware that I can suppress them by redirecting to 2> /dev/null
.
What these errors means and why they're happening?
Is it dtrace
fault, or some specific process causing that? And how this problem can be addressed?
I'm using OS X 10.11.2
kenorb
(22004 rep)
Apr 8, 2016, 03:45 PM
• Last activity: Apr 27, 2016, 02:05 PM
0
votes
1
answers
82
views
Uprobes Status on the present linux kernel
I know uprobes was introduced on kernel 3.5 has it survived or is it dropped, is it still available on kernel 4.x
I know uprobes was introduced on kernel 3.5 has it survived or is it dropped, is it still available on kernel 4.x
SAS
(31 rep)
May 8, 2015, 10:36 AM
• Last activity: Apr 8, 2016, 03:50 PM
7
votes
2
answers
9133
views
Measure disk IO latencies of a running process
I'm trying to measure the disk IO latencies of a running process to make a histogram. I could do this with DTrace in operating systems that provide it (e.g. as in [this Joyent paper][1]), but my application is running in Linux. My first thought was to try `perf`, and I can get counters but I can't f...
I'm trying to measure the disk IO latencies of a running process to make a histogram.
I could do this with DTrace in operating systems that provide it (e.g. as in this Joyent paper ), but my application is running in Linux. My first thought was to try
perf
, and I can get counters but I can't find any way to get time deltas. I can get time deltas with strace
(e.g. strace -e read -T
), but I'm not sure if I can restrict the trace to disk IO (this system has a busy network interface as well).
Is there any way to do this in Linux?
ajduff574
(233 rep)
Apr 21, 2013, 04:29 PM
• Last activity: Apr 22, 2013, 07:29 PM
4
votes
1
answers
715
views
Are there awk versions that provide syntax for computing aggregations?
From time to time I find myself writing awk scripts that compute some simple statistics. For example computing a histogram, the average of a value, the standard deviation or even the variance ... Doing that again and again with helper arrays/variables and for-loops in the `END` clause etc. feels a l...
From time to time I find myself writing awk scripts that compute some simple statistics. For example computing a histogram, the average of a value, the standard deviation or even the variance ...
Doing that again and again with helper arrays/variables and for-loops in the
END
clause etc. feels a little bit tedious and error-prone.
In Dtrace there is a quite awesome syntax for such tasks which they call aggregations . It is similar to the concept/API of Accumulators in the Boost C++ library .
Thus my question: are there awk variants which provide similar concepts/syntax that allow for convenient and iterative computation of such statistics?
An imaginative example of such syntax:
$ someawk '{ @time[$1] = avg($2) }' measurements.log
prog1 150
prog2 200
....
(where the 1st column contains the program name, the 2nd the runtime of one measurement, measurements.log
contains multiple measurements for each program and the aggregate function avg
computes the average)
maxschlepzig
(59492 rep)
Dec 25, 2012, 07:00 PM
• Last activity: Dec 25, 2012, 11:38 PM
3
votes
2
answers
954
views
Java process, swap (/proc Solaris10) memory highly utilized
How can I analyze high swap memory utilization of Solaris 10 for Java processes?
How can I analyze high swap memory utilization of Solaris 10 for Java processes?
Vishal
(131 rep)
Dec 30, 2011, 04:35 PM
• Last activity: Aug 4, 2012, 08:50 PM
Showing page 1 of 16 total questions