Sample Header Ad - 728x90

Minimizing disk usage with parallel calls to GCC

3 votes
0 answers
74 views
I am experimenting with testing GCC in parallel. My setup will run 96 tests before giving me the test report. If I run these tests sequentially it will invoke GCC once, run the executable, gather diagnostics and then repeat. However, when I try to run these tests in parallel the call to GCC takes much more time. My profiler tells me that (when averaged over the 96 tests) the calls to GCC account for 2% of total execution time when running the 96 tests in sequence. My machine has 8 cores, and when I run the same profiler over my program, but with 8 available threads (12 tests on each thread), the calls to GCC account for 12% of the total time. I theorized that perhaps the OS in this case is a shared resource, and tried to tell GCC to output the executable to a tmpfs location, but that only got me down to 11% of total time spent there. Can anyone help guide me a bit here? I am not too familiar with how the file system and IO writes work on Linux (Ubuntu 20). I don't think my testing code is necessarily erroneous, but I'll include it anyway. It is written in Haskell. This is the function that takes 2% of all time when the tests are run sequentially, but (now) 11% when the tests are run in parallel on 8 threads (I do have 8 available cores).
create_test_executable :: String -> State -> String -> IO ()
create_test_executable p s path = do -- p = the c program, s = unused here, path = where to write executable
    let process = proc "gcc" ["-xc", "-"]
    x@(Just sin, Just sout, Just serr, _) <-
        createProcess process { std_in  = CreatePipe
                              , std_err = CreatePipe
                              , std_out = CreatePipe
                              , cwd     = Just path
                              }
    hPutStr sin p
    hFlush sin
    hClose sin
    o <- hGetContents sout
    if o == "" then return () else return () -- force read from output pipe
    cleanupProcess x
To clarify: My suspicion is that GCC will write the executable to the disk, and that this contention for the disk IO is what is making each individual test slower. I tried to remedy this by writing the executable to a tmpfs location, but that barely made any difference. The write to this location is specified by the path I am passing in as a parameter to the function above,
Asked by Rewbert (131 rep)
Oct 19, 2022, 11:13 AM