Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
2
votes
3
answers
2809
views
Parallelize recursive deletion with find
I want to recursively delete all files that end with `.in`. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From [this thread][1], it looks like it's possible to use `xargs` or `make` to parallelize `find`. Is this application of find possibl...
I want to recursively delete all files that end with
.in
. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From this thread , it looks like it's possible to use xargs
or make
to parallelize find
. Is this application of find possible to parallelize?
Here is my current serial command:
find . -name "*.in" -type f -delete
kilojoules
(169 rep)
Mar 6, 2017, 04:57 PM
• Last activity: Jul 26, 2025, 03:16 PM
86
votes
6
answers
114690
views
How to determine the maximum number to pass to make -j option?
I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the `-j` option. How can I programmatically choose that value, e.g. in a shell script? Is the output of `nproc` equivalent to the number of threads I have available to compile with? `make...
I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the
-j
option. How can I programmatically choose that value, e.g. in a shell script?
Is the output of nproc
equivalent to the number of threads I have available to compile with?
make -j1
make -j16
tarabyte
(4506 rep)
Jun 9, 2015, 09:36 PM
• Last activity: Mar 8, 2025, 10:07 AM
3
votes
1
answers
883
views
Only one of the two processor cores is recognized by Linux Mint on my laptop
My computer seems to have only one of the two processor cores operating. inxi CPU~Single core Intel Core2 Duo T6400 (-UP-) speed~2000 MHz (max) Kernel~4.4.0-53-generic x86_64 Up~2:40 Mem~1180.8/3951.5MB HDD~80.0GB(34.1% used) Procs~160 Client~Shell inxi~2.2.35 lscpu Arquitectura: x86_64 modo(s) de o...
My computer seems to have only one of the two processor cores operating.
inxi CPU~Single core Intel Core2 Duo T6400 (-UP-) speed~2000 MHz (max) Kernel~4.4.0-53-generic x86_64 Up~2:40 Mem~1180.8/3951.5MB HDD~80.0GB(34.1% used) Procs~160 Client~Shell inxi~2.2.35
lscpu Arquitectura: x86_64 modo(s) de operación de las CPUs:32-bit, 64-bit Orden de bytes: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Hilo(s) de procesamiento por núcleo:1 Núcleo(s) por «socket»:1 Socket(s): 1 Modo(s) NUMA: 1 ID de fabricante: GenuineIntel Familia de CPU: 6 Modelo: 23 Model name: Intel(R) Core(TM)2 Duo CPU T6400 @ 2.00GHz Revisión: 10 CPU MHz: 2000.000 CPU max MHz: 2000,0000 CPU min MHz: 1200,0000 BogoMIPS: 4000.41 Caché L1d: 32K Caché L1i: 32K Caché L2: 2048K NUMA node0 CPU(s): 0 Flags: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dthermMy computer is an ASUS F50SL laptop, which came with Windows Vista factory installed. And the BIOS lacks options to enable / disable advanced ACPI options, such as Acpi apic or Acpi 2.0... Options, that you can find in desktop BIOS versions; for example. This is the information
inxi -F System: Host: x-PC1 Kernel: 4.4.0-53-generic x86_64 (64 bit) Desktop: MATE 1.16.1 Distro: Linux Mint 18.1 Serena Machine: System: ASUSTeK product: F50SL v: 1.0 Mobo: PEGATRON model: F50SL v: 1.0 Bios: American Megatrends v: 209 date: 11/12/2009 CPU: Single core Intel Core2 Duo T6400 (-UP-) cache: 2048 KB speed: 2000 MHz (max) Graphics: Card: Advanced Micro Devices [AMD/ATI] RV710/M92 [Mobility Radeon HD 4530/4570/545v] Display Server: X.Org 1.18.4 drivers: fbdev,ati (unloaded: vesa,radeon) Resolution: 1024x768@76.00hz GLX Renderer: Gallium 0.4 on llvmpipe (LLVM 3.8, 128 bits) GLX Version: 3.0 Mesa 11.2.0 Audio: Card-1 Advanced Micro Devices [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] driver: snd_hda_intel Card-2 Silicon Integrated Systems [SiS] Azalia Audio Controller driver: snd_hda_intel Sound: Advanced Linux Sound Architecture v: k4.4.0-53-generic Network: Card-1: Silicon Integrated Systems [SiS] 191 Gigabit Ethernet Adapter driver: sis190 IF: enp0s4 state: up speed: 100 Mbps duplex: full mac: e0:cb:4e:69:13:c4 Card-2: Qualcomm Atheros AR928X Wireless Network Adapter (PCI-Express) driver: ath9k IF: wlp2s0 state: down mac: 00:22:43:6f:d3:08 Drives: HDD Total Size: 80.0GB (34.1% used) ID-1: /dev/sda model: INTEL_SSDSA2M080 size: 80.0GB Partition: ID-1: / size: 32G used: 8.7G (30%) fs: ext4 dev: /dev/sda1 ID-2: swap-1 size: 8.59GB used: 0.00GB (0%) fs: swap dev: /dev/sda5 RAID: No RAID devices: /proc/mdstat, md_mod kernel module present Sensors: System Temperatures: cpu: 69.0C mobo: N/A Fan Speeds (in rpm): cpu: N/A Info: Processes: 160 Uptime: 2:54 Memory: 1256.8/3951.5MB Client: Shell (bash) inxi: 2.2.35So; to be able to boot the computer correctly I am forced to make certain modifications to the grub file in the path: etc/default/grub I made the following necessary changes:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset nolapic acpi_osi=\"Windows 2006\""And after this I updated the grub with sudo update-grub Since apic is linked to the processor management and to be able to boot I have to disable it by manually entering changes to the GRUB file. I think those changes I have set, have to do with the deactivation of one of the CPU cores. What can I do to make both cores of my processor work instead of just one?
Unjuhkijgh
(31 rep)
May 25, 2017, 02:10 PM
• Last activity: Jun 30, 2023, 01:05 PM
-2
votes
2
answers
6424
views
Difference between cores and logical processors
I have a question about core and logical processors. So, my computer has 1 CPU with 2 cores and 4 logical processors. Does this mean that my computer has 4 threads, 2 threads for every core? Than why it says 2975 Threads if I have only 4? [![enter image description here][1]][1] [1]: https://i.sstati...
I have a question about core and logical processors. So, my computer has 1 CPU with 2 cores and 4 logical processors. Does this mean that my computer has 4 threads, 2 threads for every core? Than why it says 2975 Threads if I have only 4?
Thank you!

elvis
(105 rep)
Apr 18, 2023, 11:46 AM
• Last activity: Apr 18, 2023, 12:30 PM
82
votes
5
answers
188425
views
So what are logical cpu cores (as opposed to physical cpu cores)?
I was googling about how I could find the number of CPUs in a machine and I found some posts but I am confused as some mentioned that you get the logical cores vs physical cores etc. So what is the difference between logical and physical cores and is there a way I could get the physical cores only?...
I was googling about how I could find the number of CPUs in a machine and I found some posts but I am confused as some mentioned that you get the logical cores vs physical cores etc.
So what is the difference between logical and physical cores and is there a way I could get the physical cores only? Or does it make sense to include logical cores in our count?
Jim
(10550 rep)
Aug 26, 2013, 10:03 PM
• Last activity: Mar 10, 2023, 10:07 AM
3
votes
2
answers
4717
views
Simple BASH script to become a multi-process/'threaded' BASH script?
I have the following, working BASH script, which I run on a Mac Pro 2010/Mojarve OS: #!/bin/bash c=0 cnt=0 # count up wav files cnt=$(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav" | wc -l) echo "there are $cnt .wav voice samples." # go through and run rhubarb on them for f in $(fin...
I have the following, working BASH script, which I run on a Mac Pro 2010/Mojarve OS:
#!/bin/bash
c=0
cnt=0
# count up wav files
cnt=$(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav" | wc -l)
echo "there are $cnt .wav voice samples."
# go through and run rhubarb on them
for f in $(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav")
do
c=$((c+1))
echo "$c of $cnt";
f=$(basename "$f" .wav)
/hummdinger/LoCI/LoCI_orig/TSV/rhubarb-lip-sync-1.10.0-osx/rhubarb /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/"$f".wav -o /hummdinger/LoCI/LoCI_orig/LOCI_GAME_FILES/Compiled/Windows/sync/"$f".tsv
done;
It takes a list of WAV files, goes through each, scans the file and then produces output and stores the generated TSV files somewhere else. The point of 'rhubarb' is to produce lip-sync information from a recording (the WAV files). etc. etc. blah blah.
The one problem with this script is it takes ~10-12 HOURS to run over about 3,000 wav files. On my crappier, non-ECC ram, one-time-it-corrupted-the-whole-lot-and-I-made-a-vow-to-never-use-it-again Mac Mini 2018, it took about *3* hours.
But this is a Mac Pro, meaning though it's old (2010), but it's very reliable and has 12x Xeons. This is pretty low intensity work, so I'm missing out on that extra juice by making it single processor. I just want to get this script working with 10-15-30 threads, and hopefully this will speed it up and it'll be done in an hour or less; not most of the day.
My thoughts are: divide the directory of WAVs into groups of (total_files/15), put these listings in file1-15.txt, then read each one back and process it in 15 separate threads. But that's about as far as I've gotten :P
Can anyone help with making this a muti-process script? I'm an amateur and made this script with help from reddit.
Benjamin Rich
(31 rep)
Jun 21, 2021, 11:28 PM
• Last activity: Jun 25, 2021, 11:13 AM
0
votes
2
answers
238
views
Is opening new tab equivalent to opening new window?
Is opening new tab in the terminal window is equivalent to opening a new tab in terms of resources? I need to run 16 scripts in parallel. I do not know whether opening tabs or new terminals is the best practice? I want to get the maximum performance but without exhausting the resources to an extent...
Is opening new tab in the terminal window is equivalent to opening a new tab in terms of resources? I need to run 16 scripts in parallel. I do not know whether opening tabs or new terminals is the best practice? I want to get the maximum performance but without exhausting the resources to an extent that degrades the performance.
user9371654
(881 rep)
Oct 9, 2018, 03:26 PM
• Last activity: Feb 12, 2020, 05:57 PM
2
votes
1
answers
1040
views
GNU Parallel only using one CPU Core
I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without considerable effort), however, I run this training with several different seeds, and all these seeds can...
I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without considerable effort), however, I run this training with several different seeds, and all these seeds can easily run on different cores.
This is how I am using parallel -
#!/bin/bash
parallel ./train.py config/config.yml _results/ \
--seed {1} \
::: {1..5}
When I run this, parallel puts all 5 processes on the same core (core0), and each of them has 20% CPU usage (as checked using htop).
If I run another set of training using the same command, 5 more processes get added to core0, and now they all show 10% CPU usage.
I am on Ubuntu 18.04
Operating System: Ubuntu 18.04.4 LTS
Kernel: Linux 5.3.0-28-generic
and Ryzen 5 3600
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 113
model name : AMD Ryzen 5 3600 6-Core Processor
stepping : 0
microcode : 0x8701013
cpu MHz : 3868.329
cache size : 512 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
...
My current (non-)solution is to use taskset after starting the training to put each seed on a different core.
pushpreet
(21 rep)
Feb 3, 2020, 02:33 AM
• Last activity: Feb 3, 2020, 07:12 PM
0
votes
0
answers
315
views
Allow bash script to run commands on multiple processors
I have a script `compute_all_matrices.sh` which launches deepTools' `computeMatrix` for a whole bunch of different files with various option. The user manual for `computeMatrix` states: >` --numberOfProcessors INT, -p INT` > > Number of processors to use. Type "max/2" to use half the maximum number...
I have a script
compute_all_matrices.sh
which launches deepTools' computeMatrix
for a whole bunch of different files with various option. The user manual for computeMatrix
states:
> --numberOfProcessors INT, -p INT
>
> Number of processors to use. Type "max/2" to use half the maximum number of processors or "max" to use all available processors. (default: 1)
I'd like to use up to 8 processors for each one, one after the other, and there are other people using these processors so I can't take them all. However, when I check on htop
, it looks like the script compute_all_matrices.sh
is only using one processor.
Is there any way to allow the "superscript" to use multiple processors for its "subscripts"?
**EDIT**
I am usure why, but apparently my superscript was using all of the processors, just not all of the time. I think I must have only checked the state of CPU usage when it was using only 1 processor. Is this possible? If so, I believe that this question can be closed.
Whitehot
(245 rep)
Jan 14, 2020, 02:41 PM
• Last activity: Jan 14, 2020, 04:27 PM
0
votes
0
answers
98
views
How RAM is distributed among processes in MPI?
For example for a 4GB RAM and 2 processes, each process will take 2 GB of memory?
For example for a 4GB RAM and 2 processes, each process will take 2 GB of memory?
Arvanitis
(11 rep)
Jun 30, 2019, 05:41 PM
1
votes
0
answers
32
views
Why do we use dynamic memory in ccNUMA systems when we talk about data distribution into locality domains by first touch policy?
In many books, when they talk about first touch policy in ccNUMA systems they are using dynamic memory allocation when they distribute data across locality domains. What if, for example, we have an array in stack?Does first touch policy work in the same way?
In many books, when they talk about first touch policy in ccNUMA systems they are using dynamic memory allocation when they distribute data across locality domains. What if, for example, we have an array in stack?Does first touch policy work in the same way?
Arvanitis
(11 rep)
Jun 30, 2019, 11:26 AM
• Last activity: Jun 30, 2019, 01:02 PM
0
votes
0
answers
557
views
Process freezes when using plotting with multiprocessing in Python
I'm parallelizing a process by using the multiprocessing module. The function that is parallelized does plotting and using `savefig` module saves the plot in pdf format. However, when the code is run, only one of the iterations results in an image while other images although created are of size 0 by...
I'm parallelizing a process by using the multiprocessing module. The function that is parallelized does plotting and using
savefig
module saves the plot in pdf format.
However, when the code is run, only one of the iterations results in an image while other images although created are of size 0 bytes.
The following error is thrown and the computer freezes:
Gdk-Message: 14:44:50.467: python2.7: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
I'm using Ubuntu 18.04.2 LTS on a laptop with 4 cores.
Sameeresque
(101 rep)
Jun 19, 2019, 03:25 PM
3
votes
3
answers
5278
views
The number of processors and cores in my systems
This is the output of `cat /proc/cpuinfo` processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz stepping : 3 microcode : 0x74 cpu MHz : 2400.000 cache size : 3072 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 2 apicid : 0 ini...
This is the output of
cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 78
model name : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
stepping : 3
microcode : 0x74
cpu MHz : 2400.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti retpoline intel_pt rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 78
model name : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
stepping : 3
microcode : 0x74
cpu MHz : 2400.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti retpoline intel_pt rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 78
model name : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
stepping : 3
microcode : 0x74
cpu MHz : 2400.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti retpoline intel_pt rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 78
model name : Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
stepping : 3
microcode : 0x74
cpu MHz : 2400.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti retpoline intel_pt rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 4800.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
Would I be right in assuming I have 4 processors that have 2 cores each?
Why then, are the core id's in each processor numbered 1, 0, 1, 0 as seen from the output of cat /proc/cpuinfo | grep 'core id'
?
core id : 0
core id : 1
core id : 0
core id : 1
Mahathi Vempati
(225 rep)
Mar 29, 2018, 12:13 PM
• Last activity: Apr 30, 2019, 09:08 PM
2
votes
1
answers
396
views
How to get one core of a quad-ARM running without an OS
I'm using a quad-core ARM CPU and I'm wondering if it is possible to get 3 cores running in Linux, and one core running without Linux?
I'm using a quad-core ARM CPU and I'm wondering if it is possible to get 3 cores running in Linux, and one core running without Linux?
yaya
(21 rep)
Apr 8, 2019, 09:10 AM
• Last activity: Apr 8, 2019, 10:09 PM
0
votes
1
answers
676
views
Using CPU affinity with taskset to speed up Node.js computation, blocked for extended chunks of time
I'm running online servers for a physics-based game with a backend in Node.js. Currently, the server can handle 10 games/worlds at a time before the next tick needs to be executed. With this in mind, I have tight scheduling using [nanotimer][1] so that the world steps don't step on each other's toes...
I'm running online servers for a physics-based game with a backend in Node.js. Currently, the server can handle 10 games/worlds at a time before the next tick needs to be executed.
With this in mind, I have tight scheduling using nanotimer so that the world steps don't step on each other's toes, but the ordering also means that I can only run on one processor, so my AWS instances are only sitting at about 55% CPU utilization.
I decided using taskset would be a good way of going about using both processors on my instance, but sometimes the worlds are blocked from processing for several seconds at times by what I imagine is another exterior process running.
How can I prevent this from happening? If I'm using two of the processors (without taskset), the one set of worlds runs just fine, but I'm only using 55% of my processor, which is really expensive.
DonutGaz
(111 rep)
Mar 17, 2019, 02:42 PM
• Last activity: Mar 18, 2019, 12:39 AM
4
votes
2
answers
4149
views
Finding the CPU id numbers to use with cpu sets
The man for [cpuset][1] doesn't seem to clearly list how to figure out which numbers map to which processing units. My current machine has two [Intel Xeon E5645][2]s, each of which has 6 cores and hyperthreading enabled, so I have 24 total processing units I can refer to with cpusets. My challenges...
The man for cpuset doesn't seem to clearly list how to figure out which numbers map to which processing units. My current machine has two Intel Xeon E5645 s, each of which has 6 cores and hyperthreading enabled, so I have 24 total processing units I can refer to with cpusets. My challenges are 1) determine which cpuset ID numbers map to which processor 2) determine which cpuset id numbers are paired (e.g. siblings on a core)
Are the numbers that
lscpu
outputs the same identifiers I should use to refer to cpu set processors? If so, it seems the numbers are alternated here, and this answers (1) with "evens are one processor, odds are the other processor", but I'm not sure if I'm reading it correctly.
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Stepping: 2
CPU MHz: 2393.964
BogoMIPS: 4788.01
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23
lstopo
from the hwloc
package seems to show me the answer to (2), and if I'm reading the man page correctly the P#...
bits are the identifier "used by the OS", which leads me to believe these are the ones I need to pass to cpu sets. So limiting a process to cpus 0 and 12 would be allowing use of two threads on the same core, while limiting it to cpus 0 and 2 would be two threads on two different cores. Does that seem correct?
$ lstopo
Machine (35GB)
NUMANode L#0 (P#0 18GB) + Socket L#0 + L3 L#0 (12MB)
L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#12)
L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#2)
PU L#3 (P#14)
L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#4)
PU L#5 (P#16)
L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#6)
PU L#7 (P#18)
L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#8)
PU L#9 (P#20)
L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#10)
PU L#11 (P#22)
NUMANode L#1 (P#1 18GB) + Socket L#1 + L3 L#1 (12MB)
L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#1)
PU L#13 (P#13)
L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#3)
PU L#15 (P#15)
L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
PU L#16 (P#5)
PU L#17 (P#17)
L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
PU L#18 (P#7)
PU L#19 (P#19)
L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
PU L#20 (P#9)
PU L#21 (P#21)
L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
PU L#22 (P#11)
PU L#23 (P#23)
HostBridge L#0
PCIBridge
PCI 14e4:163a
Net L#0 "eth0"
PCI 14e4:163a
Net L#1 "eth1"
PCIBridge
PCI 102b:0532
PCI 8086:2921
Block L#2 "sda"
PCI 8086:2926
Hamy
(295 rep)
Nov 1, 2014, 06:08 AM
• Last activity: Nov 16, 2018, 07:52 PM
0
votes
1
answers
114
views
Network addressing for multi-Processor Systems
I don't get how Network addressing works for Multi-processor systems. Check me out : Normal TCP/IP Networking uses The destination IP to designate the entire host/system, and the L4 (TCP/UDP) destination port to designate the target application. 1. I can (presumably) set the system to have one IP ad...
I don't get how Network addressing works for Multi-processor systems.
Check me out :
Normal TCP/IP Networking uses The destination IP to designate the entire host/system, and the L4 (TCP/UDP) destination port to designate the target application.
1. I can (presumably) set the system to have one IP address, and have
applications running on the different CPU's use different L4 port numbers.
Then, for each incoming packet the L4 port can be used to decide to which
CPU to send it. BUT - how does the system ensure applications on one CPU
will not re-use a L4 Port number being used on the other?
1. If the above is the same, it seems to mean I can't really use two
(or more) CPU's for one application. For example, if I am running a
web-server, all packets toward it will have a TCP port value of 80,
and how will the system then decide which packets to send top which
CPU?
2. It is possible, of course to give each CPU its own IP address - but
then again, I will not be able to run a multi-CPU application.
What am I missing? how does this work?
Michael Orr
(1 rep)
Aug 30, 2018, 06:41 AM
• Last activity: Sep 9, 2018, 08:47 PM
5
votes
2
answers
1226
views
How to find out how many cores (including emulated via HT and such) are avaliable to user?
I wonder how to get how many cores (including emulated via HT and such) are avaliable to user via bash shell-script command?
I wonder how to get how many cores (including emulated via HT and such) are avaliable to user via bash shell-script command?
myWallJSON
(1191 rep)
Jan 15, 2012, 08:42 AM
• Last activity: Apr 26, 2018, 03:05 AM
7
votes
1
answers
5037
views
Should I disable hyperthreading when concerned about performance of single-threaded applications?
I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that single-threaded applications only use a maximum of 25% of processing power, and I would rather have the...
I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that single-threaded applications only use a maximum of 25% of processing power, and I would rather have them max at 50% instead. Will disabling hyperthreading have adverse effects (this is a development machine and runs a desktop)? Am I even reading **
top
** properly?
tshepang
(67482 rep)
Aug 12, 2011, 01:30 PM
• Last activity: Mar 17, 2018, 11:52 PM
3
votes
1
answers
271
views
What is exactly happening in this process?
So I ran the following command and I accidentally saw the monitor and I wondered what is exactly happening. can someone explain to me what is exactly happening here? echo "`seq 100000000 -1 1`" > file [![system monitor screen-shot while running the command][1]][1] Since it happens many times that th...
So I ran the following command and I accidentally saw the monitor and I wondered what is exactly happening. can someone explain to me what is exactly happening here?
echo "
Since it happens many times that the images don't load for me I try to describe what I see in it. The memory usage grows linearly and the CPU1 usage is fixed at 100% then at the times t1(barely noticeable) t2 t3 t4 t5 the memory usage suddenly grows much faster for about half a second and CPU1 usage decreases and CPU2 usage increases all at the same time then memory growth rate comes back to it's linear form with the exact same slope and CPU1 usage grows towards 100% again and CPU2 usage drops to it's prior state in another half a second. The interesting points are that:
t(i)-t(i-1) = 2*(t(i+1)-t(i)) for every i
The amount of decrease in CPU1 usage doubles for each i
The time interval in which these things take seems to be the same for all i
The amount of sudden growth in memory usage also doubles for all i
The sum of the loads of CPU1 and CPU2 seem to be constant in time.
So I'm asking multiple questions in one question but I think this is the best way to do it. Rather than posting the same picture 5 times with almost the same explanations.
Can someone please explain to me Why exactly are these observations taking place?
**Memory info:**
$ sudo lshw -class memory
*-firmware
description: BIOS
vendor: LENOVO
physical id: 0
version: 9ECN31WW(V1.14)
date: 08/18/2014
size: 128KiB
capacity: 6592KiB
capabilities: pci upgrade shadowing cdboot bootselect edd int13floppynec int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 int9keyboard int10video acpi usb biosbootspecification uefi
*-cache:0
description: L1 cache
physical id: b
slot: L1 Cache
size: 32KiB
capacity: 32KiB
capabilities: synchronous internal write-back instruction
configuration: level=1
*-cache:1
description: L2 cache
physical id: c
slot: L2 Cache
size: 256KiB
capacity: 256KiB
capabilities: synchronous internal write-back unified
configuration: level=2
*-cache:2
description: L3 cache
physical id: d
slot: L3 Cache
size: 6MiB
capacity: 6MiB
capabilities: synchronous internal write-back unified
configuration: level=3
*-cache
description: L1 cache
physical id: a
slot: L1 Cache
size: 32KiB
capacity: 32KiB
capabilities: synchronous internal write-back data
configuration: level=1
*-memory
description: System Memory
physical id: 2a
slot: System board or motherboard
size: 8GiB
*-bank:0
description: SODIMM DDR3 Synchronous 1600 MHz (0.6 ns)
product: HMT451S6BFR8A-PB
vendor: Unknown
physical id: 0
serial: 1E52E0EA
slot: DIMM0
size: 4GiB
width: 64 bits
clock: 1600MHz (0.6ns)
*-bank:1
description: DIMM [empty]
product: Empty
vendor: Empty
physical id: 1
serial: Empty
slot: DIMM1
*-bank:2
description: SODIMM DDR3 Synchronous 1600 MHz (0.6 ns)
product: HMT451S6BFR8A-PB
vendor: Unknown
physical id: 2
serial: 1E82E0B8
slot: DIMM2
size: 4GiB
width: 64 bits
clock: 1600MHz (0.6ns)
*-bank:3
description: DIMM [empty]
product: Empty
vendor: Empty
physical id: 3
serial: Empty
slot: DIMM3
**CPU info:**
$ sudo lshw -class processor
*-cpu
description: CPU
product: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
vendor: Intel Corp.
physical id: 4
bus info: cpu@0
version: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
serial: To Be Filled By O.E.M.
slot: U3E1
size: 3285MHz
capacity: 3400MHz
width: 64 bits
clock: 100MHz
capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts cpufreq
configuration: cores=4 enabledcores=4 threads=8
**Kernel and OS info**
$ uname -a
Linux 4.10.0-42-generic #46~16.04.1-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
**Shell info:**
$ $SHELL --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
**Swap info:**
$ lsblk | grep SWAP
├─sda2 8:2 0 935M 0 part [SWAP]
seq 100000000 -1 1
" > file

yukashima huksay
(1164 rep)
Jan 7, 2018, 10:42 PM
• Last activity: Jan 10, 2018, 01:23 PM
Showing page 1 of 20 total questions