Sample Header Ad - 728x90

Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

0 votes
1 answers
2353 views
systemd & python logging
I wrote a simple python3 skript and want to make a service from it. Here are the two files #!/bin/python3 while True: print('True') Systemd Service: [Unit] Description=True Service [Service] Type=simple ExecStart=python3 /root/print_true.py StandardOutput=journal+console [Install] WantedBy=multi-use...
I wrote a simple python3 skript and want to make a service from it. Here are the two files #!/bin/python3 while True: print('True') Systemd Service: [Unit] Description=True Service [Service] Type=simple ExecStart=python3 /root/print_true.py StandardOutput=journal+console [Install] WantedBy=multi-user.target The service is starting but i cant see the output anywhere. Not in
-u true-service.service
, not in
status true-service
and not in
. I want to have it only in journal but how can i accomplish that? I cant seem to find a working answer.
Nico (123 rep)
Jun 28, 2020, 08:57 AM • Last activity: Aug 6, 2025, 06:05 AM
6 votes
2 answers
568 views
journalctl seemingly can't handle too much logs
I have a systemctl service that generates rather high volume of logs, on the order of 20-50 Gb per day. The service runs on a VPS with Ubuntu 24.04 LTS. The service is written in node.js, and simply `console.log`s log messages. As it is running as a systemd service, these logs are picked by journald...
I have a systemctl service that generates rather high volume of logs, on the order of 20-50 Gb per day. The service runs on a VPS with Ubuntu 24.04 LTS. The service is written in node.js, and simply console.logs log messages. As it is running as a systemd service, these logs are picked by journald, and I can then view and analyze them using journalctl. From time to time I observe the following problems: - The logs in journalctl are severly delayed. Namely, the service logs current time every minute, and in journalctl output I see records such as Jul 07 18:29:43 vc start-manager.sh: Current time: 2025-07-07 18:24:43 The first timestamp comes from journalctl, the second comes from the service; the difference can be up to 20 minutes and more - systemd-journald process eats 20-40% CPU as reported by htop - The node.js service gradually increases its heap size (as reported by node.js's process.memoryUsage()), until it runs out of heap and crashes. I've dumped node.js heap and saw that almost all memory is eaten by log strings (that actually should be immediately written to stdout). So my assumption is that journald can not manage such a large flow of logs, the stdout pipe from the service to journald becomes full, and nodejs apparently starts bufferizing log messages. This explains both high heap usage and log delays. Overall, when this happens, the system doesn't look overloaded: la15 is around 3 (with 8 vcpu), memory usage is around 2-5 Gb out of 16 Gb, and disk write is around 5 MBytes/s (almost all by systemd-journald), which does not sound too much (sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync reports 600-700 MB/s). There are also periods when everything works fine, logs are not delayed, systemd-journald consumes much less CPU, and node.js service does not use too much heap. I do not see much difference between these two regimes; in particular, the log speed by the service is approximately the same. My journald config (in /etc/systemd/journald.conf.d/journald.conf) is
[Journal]
SystemMaxUse=100G
SystemKeepFree=30G
SystemMaxFileSize=1G
SystemMaxFiles=100
RateLimitInterval=0
RateLimitBurst=0
How can I diagnose and fix this behavior? Or am I hitting inherent journald performance?
Petr (163 rep)
Jul 7, 2025, 07:25 PM • Last activity: Jul 10, 2025, 09:42 PM
2 votes
1 answers
5447 views
systemd: my service stopped working with code=killed
My service has suddenly stopped working, `systemctl status` reports: Main PID: 5459 (code=killed, signal=TERM) Also, I checked through `journald` for my service (`journalctl -u myservice.service`) and the last entry is: systemd[1]: Stopped MyService Service. Does this mean it was manually stopped wi...
My service has suddenly stopped working, systemctl status reports: Main PID: 5459 (code=killed, signal=TERM) Also, I checked through journald for my service (journalctl -u myservice.service) and the last entry is: systemd: Stopped MyService Service. Does this mean it was manually stopped with systemctl stop or it might as well mean it has crashed? Is there a way to tell?
Mark (1943 rep)
Jun 7, 2022, 09:23 PM • Last activity: Jul 2, 2025, 07:08 PM
2 votes
2 answers
2480 views
Can't get kernel messages logged to the serial console of a systemd system
I would like to log the kernel messages of system to the serial console on ttyS0. A similar problem was addressed under [How to get kernel messages on serial console on a systemd system?][1] but I still don't get it to work. I've got `systemd.journald.max_level_console=debug` `console=ttyS0,115200`...
I would like to log the kernel messages of system to the serial console on ttyS0. A similar problem was addressed under How to get kernel messages on serial console on a systemd system? but I still don't get it to work. I've got systemd.journald.max_level_console=debug console=ttyS0,115200 loglevel=7 on my kernel command line and ShowStatus=no set in */etc/systemd/system.conf* Nevertheless, while I'll see kernel messages on the console at the beginning of the boot process, after the Journal Service is started I wont get any more kernel messages logged to the console. If I, as an example, connect a USB mouse to the board, there is nothing logged to the console but journalctl -f shows: Feb 11 10:09:45 a20 kernel: usb 3-1: new low-speed USB device number 4 using ohci-platform Feb 11 10:09:45 a20 kernel: usb 3-1: New USB device found, idVendor=046d, idProduct=c069, bcdDevice=56.01 Feb 11 10:09:45 a20 kernel: usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Feb 11 10:09:45 a20 kernel: usb 3-1: Product: USB Laser Mouse Feb 11 10:09:45 a20 kernel: usb 3-1: Manufacturer: Logitech Feb 11 10:09:45 a20 kernel: input: Logitech USB Laser Mouse as /devices/platform/soc/1c14400.usb/usb3/3-1/3-1:1.0/0003:046D:C069.0005/input/input6 Feb 11 10:09:45 a20 kernel: hid-generic 0003:046D:C069.0005: input,hidraw2: USB HID v1.10 Mouse [Logitech USB Laser Mouse] on usb-1c14400.usb-1/input0 Any thoughts?
bey0nd (977 rep)
Feb 11, 2020, 11:09 AM • Last activity: Jun 17, 2025, 02:09 PM
-3 votes
1 answers
65 views
Linux Mint: /var/log shows 165GB disk usage but no large files found
My Linux Mint system reports that /var/log occupies 165 GB of space, but all subdirectories and files within it are small (total <100 MB). This is causing the root partition (/) to fill up, preventing normal system operation Checked disk usage with `sudo du -hxd1 /var/log | sort -hr` – no large file...
My Linux Mint system reports that /var/log occupies 165 GB of space, but all subdirectories and files within it are small (total <100 MB). This is causing the root partition (/) to fill up, preventing normal system operation Checked disk usage with sudo du -hxd1 /var/log | sort -hr – no large files found. Cleared journal logs:
sudo journalctl --vacuum-size=0
sudo rm -rf /var/log/journal/*
sudo systemctl restart systemd-journald
Looked for "deleted but open" files with
sudo lsof | grep deleted | grep '/var/log'
– no results. Verified filesystem integrity with
sudo e2fsck -n /dev/sda3
– no errors found. Checked extended attributes (getfattr, lsattr) – nothing unusual. Restarted the system multiple times – no change.
df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb3       199G  193G  6.3G  97% /


sudo du -hxd1 /var/log:
165G    /var/log
4.0K    /var/log/private
12M     /var/log/apt
... (other small directories)
- What could explain the discrepancy between du and df? - How can I safely reclaim the "phantom" 165 GB in /var/log?
Karol Kr&#243;l (5 rep)
Jun 7, 2025, 10:59 AM • Last activity: Jun 7, 2025, 11:10 AM
1 votes
2 answers
105 views
systemd message litters tuigreet / greetd
A failing mount unit's systemd message somehow ends up on the tuigreet / greetd screen. I suspected the problem stemmed from using VT1 for greetd which is used by systemd to log messages there but the following configurations didn't solve the issue. At least this is how I interpreted this section an...
A failing mount unit's systemd message somehow ends up on the tuigreet / greetd screen. I suspected the problem stemmed from using VT1 for greetd which is used by systemd to log messages there but the following configurations didn't solve the issue. At least this is how I interpreted this section and concluded the attempt below. **/etc/greetd/config.toml** [terminal] vt = 2 [default_session] command = "tuigreet --greeting Authenticate into Jotunheim --time --time-format '%m-%d | %H:%M:%S' --remember" user = "greeter" **/etc/systemd/system/greetd.service.d/override.conf** [Unit] After=getty@tty2.service Conflicts=getty@tty2.service **Grandpa screenshot** k (I know what this lingering mount unit is but I suppose making it not log anything won't solve my issue for good because than any other failing unit will cause the same in future.)
Hermes Hariwald (11 rep)
Jun 2, 2025, 04:44 AM • Last activity: Jun 7, 2025, 07:16 AM
4 votes
1 answers
5769 views
Is it possible to reduce systemd-journald size stored format (and thus size)?
I'm using systemd on raspberrypi machine with yocto based system. Recently I had some problems with redirecting messages to rsyslog.socket so I decided to get rid of `rsyslog` completely in favour of of `journald`. After doing so I have noticed that size of journald files is much bigger than I previ...
I'm using systemd on raspberrypi machine with yocto based system. Recently I had some problems with redirecting messages to rsyslog.socket so I decided to get rid of rsyslog completely in favour of of journald. After doing so I have noticed that size of journald files is much bigger than I previously thought it would be. root@rpiDev: ~ $ journalctl -o cat > /tmp/journals-cat.txt root@rpiDev: ~ $ journalctl -o export > /tmp/journals-exp.txt root@rpiDev: ~ $ journalctl -o verbose > /tmp/journals-verb.txt root@rpiDev: ~ $ journalctl -a -m > /tmp/journals.txt ### This is what I need! root@rpiDev: ~ $ journalctl -a -m -o verbose > /tmp/journals-everything.txt root@rpiDev: ~ $ du -sh /tmp/journals* /var/log/journal/ ; journalctl --disk-usage 468.0K /tmp/journals-cat.txt 15.7M /tmp/journals-everything.txt 4.7M /tmp/journals-exp.txt 4.9M /tmp/journals-verb.txt 2.3M /tmp/journals.txt 41.0M /var/log/journal/ Archived and active journals take up 12.5M on disk. Comparing the sizes it looks like binary files created by journald are much bigger than merged (-m) logs. What I actually need is what is inside /tmp/journals.txt. **Question:** Is it possible to reduce amount of stuff stored by journald in it's binary files to what I noticed when running journalctl -a -m? In other words: can I disable storing all of the information that is not important to me and use journald just as I would syslog? My problem can be solved by disabling permanent storing of journald logs and forwarding them to syslog, but maybe it is possible without bringing back rsyslog? EDIT: Parameters mentioned by some users do not help me here. - Using SystemMaxUse= and RuntimeMaxUse= only sets the maximum size of the files stored- I can have smaller files with the same amount of not needed info and therefore even less actual logs. - Using MaxLevel...= sets the maximum log level stored in the journal. That is also not what I need here. EDIT2: My solution: I have decided to store logs in syslog (I use rsyslog).
In my journald.conf I have set Storage=volatile and used SystemMaxUse=64M and RuntimeMaxUse=64M to limit disk usage by journald.
I also enabled ForwardToSyslog=yes so now I have my old syslog solution working and I'm also able to view runtime journald logs.
lewiatan (1149 rep)
Jan 6, 2017, 01:39 PM • Last activity: Jun 5, 2025, 01:07 AM
0 votes
0 answers
62 views
/dev/log is a socket not a symbolic link
I am trying to understand why one of my system has /dev/log as a socket and other has /dev/log as a symbolic link. [ec2-user@ip-171-31-12-17 log]$ file /dev/log /dev/log: symbolic link to /run/systemd/journal/dev-log [ec2-user@ip-171-31-12-18 log]$ file /dev/log /dev/log: socket I have tried checkin...
I am trying to understand why one of my system has /dev/log as a socket and other has /dev/log as a symbolic link. [ec2-user@ip-171-31-12-17 log]$ file /dev/log /dev/log: symbolic link to /run/systemd/journal/dev-log [ec2-user@ip-171-31-12-18 log]$ file /dev/log /dev/log: socket I have tried checking on other systems and found out most of the RHEL has /dev/log as link not a socket. That means I have to make the other systems /dev/log as a link not socket. Does journalctl logs appear in /var/log/messages ? when i have /dev/log as socket instead of link. How do I make things normal ? By the way OS version is RHEL 7.9 for /dev/log : socket
Boogeyman (1 rep)
May 25, 2025, 06:57 AM
5 votes
1 answers
2843 views
How would I limit users' sudo permissions for journalctl to a specific unit?
I want to provide our users access to `journalctl` of a specific unit. For e.g., `httpd`. My initial thought was to specify the command spec like this: /usr/bin/journalctl -u httpd.service * This will give them access to useful options such as `--all`, `--full`, `--follow`, etc.. However, quickly no...
I want to provide our users access to journalctl of a specific unit. For e.g., httpd. My initial thought was to specify the command spec like this: /usr/bin/journalctl -u httpd.service * This will give them access to useful options such as --all, --full, --follow, etc.. However, quickly noticed that it will also allow them to specify a different unit (e.g., -u tomcat). Is by specifying each of the acceptable options the only way to do this? Or can I leverage something like ACLs on a file for more granular access?
Belmin Fernandez (9877 rep)
Mar 24, 2016, 02:02 PM • Last activity: May 25, 2025, 05:05 AM
0 votes
1 answers
33 views
Why syslogd receives the message and not journald?
I have tried to log into `journald` using `logger`. `logger test`, `journalctl |grep test` does not show any output. The message does not reach journald even though it listening and logger is writing (I checked with strace) to the same socket address `/run/systemd/journal/dev-log` I foud the message...
I have tried to log into journald using logger. logger test, journalctl |grep test does not show any output. The message does not reach journald even though it listening and logger is writing (I checked with strace) to the same socket address /run/systemd/journal/dev-log I foud the message in /var/log/messages, the message appears in syslog messages, and not in journalctl. I checked which process listen to /dev/log, I found both syslog and systemd root@raspberrypi4-64:~# lsof /dev/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd 1 root 32u unix 0x0000000077b4fabe 0t0 220 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) systemd-j 129 root 5u unix 0x0000000077b4fabe 0t0 220 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) syslogd 253 root 0u unix 0x000000009c5fe5d3 0t0 601 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) After disabling syslog.service and rebooting, the logger messages are prompted in journalctl logs. 2 daemons bound to the same socket path (but not to same inode), why syslogd receives the messages and not journald ? is there some priority to be respected when two daemons listen to the same socket path? I've read this answer but still does not full answer my question.
maths soso (3 rep)
May 8, 2025, 03:13 PM • Last activity: May 8, 2025, 06:03 PM
2 votes
2 answers
95 views
How to preserve newline characters in journald logs from PHP script stdout?
I have some service, which executes php scripts and sends its stdout to journald. Journald splits the log record by newline. This can be read under lineMax: - [journald.conf(5) - systemd | Debian Manpage][1] For example: ``` 1 ``` Then I get from journalctl ``` May php[107850]: Thisisline 1 May php[...
I have some service, which executes php scripts and sends its stdout to journald. Journald splits the log record by newline. This can be read under lineMax: - journald.conf(5) - systemd | Debian Manpage For example:
1
Then I get from journalctl
May php: Thisisline 1
May php: This is line 2
May php: This is line 3
May myapp: Thisisline 1
                   This is line 2
                   This is line 3
Is there any way to preserve newline characters or support multiline (e.g., stack traces or formatted output) so that they appear as a single entry in the journal without the logger API? Any advice how to approach it?
Dano Logos (21 rep)
May 2, 2025, 11:32 AM • Last activity: May 3, 2025, 09:26 AM
0 votes
0 answers
6430 views
systemd-journald and rsyslogd: high cpu usage
This is a fresh Debian 10 installation that's two-three weeks old. For the last couple of days, my system has been acting up, it would just slow down and I would not be able to use `sudo` in the terminal as it won't prompt for authentication. When I check CPU usage using `htop`, it shows that `syste...
This is a fresh Debian 10 installation that's two-three weeks old. For the last couple of days, my system has been acting up, it would just slow down and I would not be able to use sudo in the terminal as it won't prompt for authentication. When I check CPU usage using htop, it shows that systemd-journald is using quite a lot of CPU, more than 100%. One time there was two or three processes/thread of rsyslogd using around 50% CPU each. I haven't seen rsyslogd using that much CPU after that. Since I could not use sudo, I wasn't able to check the logs, but today I found out that I could just su and become root while sudo wasn't working and I checked the logs. The following is an excerpt from journalctl: Aug 25 22:27:34 asgard systemd-journald: Missed 51 kernel messages Aug 25 22:27:34 asgard kernel: wlp0s20f3: Failed check-sdata-in-driver check, flags: 0x4 Aug 25 22:27:34 asgard kernel: WARNING: CPU: 0 PID: 2221 at net/mac80211/driver-ops.h:19 drv_sta_state+0x265/0x3e0 [mac80211] Aug 25 22:27:34 asgard systemd-journald: Missed 56 kernel messages Aug 25 22:27:34 asgard kernel: drbg ansi_cprng cfg80211 dell_rbtn processor_thermal_device iTCO_vendor_support irqbypass intel_soc_dts_iosf ecdh_generic sg joydev dell_laptop hid_multitouch idma64 crct10dif_pclmul crc32_pclmul int3403_thermal rfkill int3400_thermal int340x_thermal_zone wmi_bmof dell_smo8800 dell_wmi intel_hid pcc_cpufreq dell_smbios dell_smm_hwmon dcdbas intel_pch_thermal ucsi_acpi typec_ucsi dell_wmi_descriptor typec acpi_thermal_rel evdev sparse_keymap pcspkr serio_raw ghash_clmulni_intel acpi_tad acpi_pad ac intel_cstate intel_uncore xt_conntrack nft_compat intel_rapl_perf efi_pstore efivars battery nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink efivarfs ip_tables x_tables autofs4 ext4 And the following part from journalctl is all over the log: Aug 25 22:27:34 asgard systemd-journald: Missed 55 kernel messages Aug 25 22:27:34 asgard kernel: crc16 mbcache jbd2 fscrypto ecb btrfs xor zstd_decompress zstd_compress xxhash raid6_pq libcrc32c crc32c_generic sd_mod hid_generic crc32c_intel i2c_designware_platform i2c_designware_core i915 psmouse aesni_intel xhci_pci aes_x86_64 crypto_simd xhci_hcd cryptd glue_helper i2c_i801 ahci libahci libata i2c_algo_bit usbcore r8169 drm_kms_helper realtek sdhci_pci libphy cqhci scsi_mod sdhci drm mmc_core intel_lpss_pci intel_lpss mfd_core usb_common i2c_hid hid wmi video button From dmesg: [ 2420.057787] wlp0s20f3: Failed check-sdata-in-driver check, flags: 0x4 [ 2420.057800] Modules linked in: uinput cmac rfcomm bnep ctr ccm fuse binfmt_misc nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 nls_ascii snd_soc_skl nls_cp437 vfat fat xt_hl ip6_tables ip6t_rt snd_hda_codec_hdmi snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi nf_log_ipv4 snd_hda_codec_realtek nf_log_common snd_soc_core snd_hda_codec_generic ipt_REJECT nf_reject_ipv4 snd_compress xt_LOG snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm uvcvideo intel_rapl snd_timer snd x86_pkg_temp_thermal intel_powerclamp nft_limit coretemp videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common kvm_intel videodev iTCO_wdt soundcore xt_limit arc4 media iwlmvm mac80211 btusb btrtl btbcm btintel bluetooth xt_addrtype kvm iwlwifi xt_tcpudp mei_me mei [ 2420.057845] RSP: 0018:ffffbfd40269f9b0 EFLAGS: 00010282 [ 2420.057850] FS: 0000000000000000(0000) GS:ffff9f0f26200000(0000) knlGS:0000000000000000 [ 2420.057879] __sta_info_flush+0x15e/0x1c0 [mac80211] [ 2420.057892] ieee80211_set_disassoc+0xbe/0x550 [mac80211] [ 2420.057903] ieee80211_mgd_deauth.cold.57+0x47/0x1b5 [mac80211] [ 2420.057914] cfg80211_mlme_deauth+0xb3/0x1d0 [cfg80211] [ 2420.057926] cfg80211_mlme_down+0x66/0x90 [cfg80211] [ 2420.057937] cfg80211_disconnect+0x128/0x1e0 [cfg80211] [ 2420.057946] cfg80211_leave+0x26/0x40 [cfg80211] [ 2420.057954] cfg80211_netdev_notifier_call+0xcd/0x600 [cfg80211] [ 2420.057956] ? syscall_return_via_sysret+0x14/0x83 [ 2420.057957] ? ret_from_fork+0x1a/0x40 [ 2420.057970] ? ieee80211_reconfig+0xd5/0x1420 [mac80211] [ 2420.057984] ? report_bug+0xb0/0xd0 [ 2420.057985] ? inetdev_event+0x46/0x580 [ 2420.057988] notifier_call_chain+0x47/0x70 [ 2420.057992] dev_close_many+0x9f/0x160 [ 2420.058002] cfg80211_shutdown_all_interfaces+0x6d/0xc0 [cfg80211] [ 2420.058015] ? rcu_exp_wait_wake+0x250/0x250 [ 2420.058016] ? try_to_del_timer_sync+0x4d/0x80 [ 2420.058028] process_one_work+0x1a7/0x3a0 [ 2420.058030] worker_thread+0x30/0x390 [ 2420.058031] ? create_worker+0x1a0/0x1a0 [ 2420.058035] ? kthread_bind+0x30/0x30 [ 2420.058043] ------------[ cut here ]------------ [ 2420.058056] Modules linked in: uinput cmac rfcomm bnep ctr ccm fuse binfmt_misc nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 nls_ascii snd_soc_skl nls_cp437 vfat fat xt_hl ip6_tables ip6t_rt snd_hda_codec_hdmi snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi nf_log_ipv4 snd_hda_codec_realtek nf_log_common snd_soc_core snd_hda_codec_generic ipt_REJECT nf_reject_ipv4 snd_compress xt_LOG snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm uvcvideo intel_rapl snd_timer snd x86_pkg_temp_thermal intel_powerclamp nft_limit coretemp videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common kvm_intel videodev iTCO_wdt soundcore xt_limit arc4 media iwlmvm mac80211 btusb btrtl btbcm btintel bluetooth xt_addrtype kvm iwlwifi xt_tcpudp mei_me mei [ 2420.058101] RSP: 0018:ffffbfd40269f9b0 EFLAGS: 00010282 [ 2420.058107] FS: 0000000000000000(0000) GS:ffff9f0f26200000(0000) knlGS:0000000000000000 [ 2420.058136] __sta_info_flush+0x15e/0x1c0 [mac80211] [ 2420.058147] ieee80211_set_disassoc+0xbe/0x550 [mac80211] [ 2420.058159] ieee80211_mgd_deauth.cold.57+0x47/0x1b5 [mac80211] [ 2420.058170] cfg80211_mlme_deauth+0xb3/0x1d0 [cfg80211] [ 2420.058181] cfg80211_mlme_down+0x66/0x90 [cfg80211] [ 2420.058193] cfg80211_disconnect+0x128/0x1e0 [cfg80211] [ 2420.058202] cfg80211_leave+0x26/0x40 [cfg80211] [ 2420.058210] cfg80211_netdev_notifier_call+0xcd/0x600 [cfg80211] [ 2420.058212] ? syscall_return_via_sysret+0x14/0x83 [ 2420.058213] ? ret_from_fork+0x1a/0x40 [ 2420.058238] ? ieee80211_reconfig+0xd5/0x1420 [mac80211] [ 2420.058240] ? report_bug+0xb0/0xd0 [ 2420.058241] ? inetdev_event+0x46/0x580 [ 2420.058245] notifier_call_chain+0x47/0x70 [ 2420.058248] dev_close_many+0x9f/0x160 [ 2420.058269] ieee80211_reconfig+0xa3/0x1420 [mac80211] [ 2420.058271] ? rcu_exp_wait_wake+0x250/0x250 [ 2420.058272] ? try_to_del_timer_sync+0x4d/0x80 [ 2420.058285] worker_thread+0x30/0x390 [ 2420.058288] kthread+0x112/0x130 [ 2420.058291] ret_from_fork+0x1f/0x40 [ 2420.058298] ------------[ cut here ]------------ Similar to journalctl messages, the above logs from dmesg were also repeated endlessly in the dmesg. (I think these dmesg logs were displayed as I tried to shutdown the system when the system hangs, but I couldn't make sure as the messages were rapidly moving.) I thing the slow down is caused by excessive logging as /var/log/kern.log, /var/log/messages and /var/log/syslog are about 23GB each. I am not sure what causes this much errors/logs, but I suspect it's the wi-fi of the laptop. The WiFi chip is an Intel® Wireless-AC 9560 and I have been having some connection issues with wifi. (The wifi would disconnect all of a sudden and the network manager would show device is not ready. When I reboot, the wifi option is gone from Network manager. This has occurred twice and toggling the Secure Boot in UEFI to OFF/ON would fix it.) I didn't notice any trouble with wifi when the system became slow. - Here's another related question from Askubuntu: https://askubuntu.com/questions/1251908/syslog-and-kern-log-keeps-filling-up-and-network-stops-working - A bug report from Red Hat: https://bugzilla.redhat.com/show_bug.cgi?id=1851185 - An Ubuntu Bug report: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1882419/comments/12 - Bug report from Linux Kernal: https://bugzilla.kernel.org/show_bug.cgi?id=98321 What exactly is happening here and how do I fix this?
RogUE (207 rep)
Aug 25, 2020, 05:54 PM • Last activity: Apr 25, 2025, 09:02 AM
2 votes
1 answers
1994 views
How should I manage the logs from my own systemd services?
I have deployed a couple of daemons on ubuntu boxes and wrapped them as systemd services. Their standard output is redirected to the journal by default which is what I was hoping for. Now, it turns out that the default configuration on ubuntu (I guess same on debian) is to make journald forward all...
I have deployed a couple of daemons on ubuntu boxes and wrapped them as systemd services. Their standard output is redirected to the journal by default which is what I was hoping for. Now, it turns out that the default configuration on ubuntu (I guess same on debian) is to make journald forward all events to /run/systemd/journal/syslog (see /etc/systemd/journald.conf: #ForwardToSyslog=yes) and have rsyslog pull data from there with the imuxsock module (see /etc/rsyslog.conf). By default, I also see that because in /etc/rsyslog.d/50-default.conf, we have this line: > *.*;auth,authpriv.none -/var/log/syslog i.e., all non-auth data ends up in /var/log/syslog Because I wanted to track the output of my services in journald, I made the journald log file persistent by creating /var/log/journal (in /etc/systemd/journald.conf, Storage's default value is auto). The result is that now all my log data is duplicated: it is stored once in journald and once in rsyslog's /var/log/syslog. So, for various reasons, I really want to keep my own service's data in journald but I really do not want to duplicate that data (there are a lot of logs !). I can see a couple of options: 1. disable rsyslog entirely. I am worried I might miss a lot of data from other services if I do this: who knows what other code in my infrastructure reads /var/log/* 2. try to disable only daemon.info because this is what appears to be the default log facility/level picked for my services. I am worried the following might make me ignore other useful messages that just happen to have the same facility/level :/ >*.*;auth,authpriv.none;daemon.!=info -/var/log/syslog 3. Change the rsyslog/journald integration to use the imjournal input module, ignore imuxsock (just like on fedora), and write rsyslog rules more specific to each of my services Now, the question is: what would be the recommended way to proceed ?
mathieu (213 rep)
Jan 17, 2017, 10:32 AM • Last activity: Apr 23, 2025, 11:03 AM
1 votes
2 answers
108 views
systemd-journald writes an average of 16MB per hour!
OS: Ubuntu MATE 24.04.2 LTS I added the system monitor to my top panel and noticed much disk activity. At first I thought the [main 'culprit' was `jbd2`][1] -- Journaling Block Device. But jbd2 (["a journal to protect the filesystem against metadata inconsistencies in the case of a system crash"][2]...
OS: Ubuntu MATE 24.04.2 LTS I added the system monitor to my top panel and noticed much disk activity. At first I thought the main 'culprit' was jbd2 -- Journaling Block Device. But jbd2 ("a journal to protect the filesystem against metadata inconsistencies in the case of a system crash" ) will/should only write to disk in response to other things writing to disk. Running sudo iotop -a, I saw that (*) the main thing writing to disk is *systemd-journald*: Over 3h40m it wrote 60MB! I understand that this is binary data but, if it was text, even at 4 bytes per character, that's '*War and Peace*' every 7 hours! * Is this 'normal'? * What's it writing that requires so much data? * Should I be worried about my SSD system disk? * Can I (safely) reduce the amount (and frequency) of data written? (*) the monster that is Firefox notwithstanding
AlanQ (97 rep)
Apr 18, 2025, 05:53 PM • Last activity: Apr 19, 2025, 11:30 AM
1 votes
1 answers
452 views
Log all journald messages to AWS Cloudwatch
I'm migrating our existing Amazon Linux 2 servers to Amazon Linux 2023. One of the changes is that the AL2023 now uses journald for it's logging. I have the requirement to have all logging in AWS Cloudwatch. I've already enabled the normal cloudwatch agent for metrics and a few logfiles. This is wor...
I'm migrating our existing Amazon Linux 2 servers to Amazon Linux 2023. One of the changes is that the AL2023 now uses journald for it's logging. I have the requirement to have all logging in AWS Cloudwatch. I've already enabled the normal cloudwatch agent for metrics and a few logfiles. This is working as expected. For journald I've checked various options and decided to implement [journald-cloudwatch-logs](https://github.com/saymedia/journald-cloudwatch-logs) , as it looked the most promising. I've configured it but I have a problem: * The service starts successful, but I simply do not see any logging in cloudwatch. This is the config:
aws_region = "eu-central-1"
ec2_instance_id = "i-xxxxxxxxxx"
log_group = "LogGroup"
log_stream = "server01-systemctl"
state_file = "/var/lib/journald-cloudwatch-logs/state"
log_priority = "INFO"
I've checked journalctl but no errors or any other indication something is wrong. Also, because the normal cloudwatch agent logs successfully to another log stream I know the IAM permissions are fine. My questions: * How can I make journald-cloudwatch-logs to actually log to cloudwatch? Any help is greatly appreciated. Edit: I'm also open for other approached to log journald messages to cloudwatch
GetShifting (153 rep)
Apr 11, 2025, 09:21 AM • Last activity: Apr 18, 2025, 07:56 AM
11 votes
2 answers
6586 views
The simple but correct way to have a bash script send output to systemd journal?
I have simply been sticking something like this at the top of my bash scripts. This still works with systemd. (I have used it for a long time and I don't fully understand it. It just works, so I use it.) exec 5> >(logger -t $0) BASH_XTRACEFD="5" PS4='$LINENO: ' set -x Even with systemd, the above co...
I have simply been sticking something like this at the top of my bash scripts. This still works with systemd. (I have used it for a long time and I don't fully understand it. It just works, so I use it.) exec 5> >(logger -t $0) BASH_XTRACEFD="5" PS4='$LINENO: ' set -x Even with systemd, the above commands give the expected output in journalctl. However, it seems that maybe I should be using something like this instead. systemd-cat -t $0 How should I alter my current logging commands under systemd? (I'm looking for the same kind of simple-minded solution where I can paste a few lines and get some output in the journal.)
MountainX (18888 rep)
Sep 19, 2017, 04:39 AM • Last activity: Apr 11, 2025, 09:27 PM
11 votes
3 answers
31018 views
How can journald be disabled?
I've recently learned that `journalctl` is occupying a big chunk of my 16GB SD card (Raspberry Pi): ```bash $ journalctl --disk-usage Archived and active journals take up 312.1M in the file system. ``` I don't feel that `journalctl` and `journald` are [pulling their weight](https://idioms.thefreedic...
I've recently learned that journalctl is occupying a big chunk of my 16GB SD card (Raspberry Pi):
$ journalctl --disk-usage
Archived and active journals take up 312.1M in the file system.
I don't feel that journalctl and journald are [pulling their weight](https://idioms.thefreedictionary.com/pull+your+weight) in my use case for this machine. It's an old-ish RPi, and rsyslog is also running. I estimate my need for and use of journalctl is maybe [*"once in a blue moon"*](https://idioms.thefreedictionary.com/once+in+a+blue+moon) . Consequently, I decided I would disable journald - which "feeds" journalctl. I assumed this would be straightforward, using systemctl to stop, or perhaps just disableing the systemd-journald.service so that it would not start on the next boot. Before pulling the plug, I decided to do some research. Instead of finding thousands of references that offered "how-to" advice, there were remarkably few results addressing my specific search term: *"how to disable journald"*. Instead, the results mostly offered advice on reducing journald's *resource consumption*. I did find a couple of references that gave me pause: An old thread in the ArchLinux forum suggested it was not possible to disable journald without [*repercussions*](https://bbs.archlinux.org/viewtopic.php?id=149884) ; i.e. "Masking systemd-journald causes all kinds of dependency failures and drops you at an emergency prompt." But this post is now 10 years old... The [systemd-journald.service manual](https://manpages.debian.org/bullseye/systemd/systemd-journald.service.8.en.html) says, "stopping it [systemd-journald.service] is not recommended.". The documentation proceeds from there to discuss *namespaces*? I've learned that the usual command to prevent systemd units from starting has no effect; i.e. it starts normally:
$ sudo systemctl disable systemd-journald.service
$ sudo reboot 

# ... and after boot & login:

$ systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Fri 2022-06-03 07:30:29 UTC; 1min 59s ago
TriggeredBy: ● systemd-journald-audit.socket
             ● systemd-journald.socket
             ● systemd-journald-dev-log.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 134 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 1598)
        CPU: 820ms
     CGroup: /system.slice/systemd-journald.service
             └─134 /lib/systemd/systemd-journald

...
$
#### How can journald be disabled? ... or can it be disabled? If not, why would the systemd developers force this on users? (OK, yeah - asking for an opinion, so forget that part of the question.)
Seamus (3772 rep)
Jun 3, 2022, 06:06 AM • Last activity: Apr 1, 2025, 07:53 PM
9 votes
4 answers
10790 views
How to keep systemd journal for one year?
By default my system had about ~500MB max size which had about 1 month of logs. System Journal (/var/log/journal/dd35c7606a5645c5acc9908470c45159) is 483.2M, max 491.5M, 8.2M free. No idea how the 491.5M limit was decided. I changed the below setting and restarted the system └──> grep SystemMaxUse /...
By default my system had about ~500MB max size which had about 1 month of logs. System Journal (/var/log/journal/dd35c7606a5645c5acc9908470c45159) is 483.2M, max 491.5M, 8.2M free. No idea how the 491.5M limit was decided. I changed the below setting and restarted the system └──> grep SystemMaxUse /etc/systemd/journald.conf SystemMaxUse=25G Now the max journal size is 1.0G. Why is it not 25G? System Journal (/var/log/journal/dd35c7606a5645c5acc9908470c45159) is 568.6M, max 1.0G, 455.3M free. I do have enough free space (94%) in the disk └──> df -h /var/log/journal Filesystem Size Used Avail Use% Mounted on /dev/sda1 885G 50G 791G 6% /var How to increase max journal size even further? If possible, I would like to keep the default limit for 1 month but instead of deleting old journal files, save it some where where in can be queried on demand. **Update:** Just to clarify, my journal files are now deleted after 500MB cap, I want to increase the limit to 25G. Even after setting SystemMaxUse=25G, the limit does not increase past 1GB.
balki (4717 rep)
Jul 12, 2022, 02:38 PM • Last activity: Mar 30, 2025, 06:31 PM
0 votes
0 answers
149 views
tails booting issue
[![enter image description here][1]][1] [1]: https://i.sstatic.net/19ZirPk3.jpg im getting this error while booting tails from a usb stick. i tried the same usb with another laptop it works fine. but trying with my asus g14 its not working, the error is shown in the screenshot. please help. thankyou...
enter image description here im getting this error while booting tails from a usb stick. i tried the same usb with another laptop it works fine. but trying with my asus g14 its not working, the error is shown in the screenshot. please help. thankyou :)
joel (1 rep)
Mar 29, 2025, 09:26 AM • Last activity: Mar 29, 2025, 10:18 AM
1 votes
1 answers
77 views
Logging Cisco logs on remote linux syslog
Good evening, I am running multiple cisco routers/switches and a virtualized debian install. In order to have proper forensic capabilities in cse of attack/breach/malfunction I wish to have remote logging of routers/switches messages in a remote facility to be able to read them even in case of hardw...
Good evening, I am running multiple cisco routers/switches and a virtualized debian install. In order to have proper forensic capabilities in cse of attack/breach/malfunction I wish to have remote logging of routers/switches messages in a remote facility to be able to read them even in case of hardware shutdown/reboot. My idea was to remotely log messages from the cisco routers to a virtualized debian host running syslog I did this in the past with debian stretch and it worked by setting different facility codes and properly log-rotate with cron job. I now see that bookworm is removing syslog and doing all with journalctl. I see that I can also install cuncurrenlty syslog and configure it as I did, but I wonder if there is a way to avoid having local logging done to the syslog and only remote syslog messages being log to /var/log/* while local system is still handled by journal(d/ctl) I guess I could do the socket listening logging to proper facilities files and local syslog logging to /dev/null but I do not like it .... anybody has done something similar? Or a better idea? I am asking in advance so when I prepare the virtual machine dedicated to logging I can properly set it up. Thanks for any pointers. Fabio
fsardone (11 rep)
Dec 25, 2024, 06:10 PM • Last activity: Feb 28, 2025, 09:10 AM
Showing page 1 of 20 total questions