Sample Header Ad - 728x90

Bash - cannot stat the last file created in a for...do loop

-1 votes
1 answer
234 views
In a bash script (on Ubuntu 22 if it matters) I have a function which, executed from within a for...do loop, is given a filename to execute some SQLplus .sql and store the results in .csv I added a little bit of a function to check success and to print out the file size of the csv it has just created. BUT...it runs through this for three queries and reports the file size in the success message no problem. But for the fourth and final one it throws an error saying "Cannot statx: no such file or directory". I can see it's there. As an alternative I make the whole process list the files in the directory and it appears. Any idea why it might not be doing the last one the same way. Here's the code for the run_query() and success_file() functions which I think is all you need to see what I am doing: [Edited to show the entire script file rather than just the snippet]
#!/bin/bash

source ~/overnights/scripts/.globals/.filenames
source ~/overnights/scripts/.globals/.creds
source ~/overnights/scripts/.globals/.commands.sh

echo "============================================================================="
echo "=  Starting to fetch Banner data                                            ="
echo "============================================================================="

# cleandrop() called from each sql script execution to finish the whole process if one file fails.
cleandrop () {
  echo "date: Problems connecting to or fetching from Oracle. Cleaning up and exiting."
  rm -f ${IN_BANNER}/*
  cd $SCRIPT_DIR
  exit 1
}

# Displays confirmations of which file we have just processed.
success_file () {
    size=$(stat -c%s "$2")
    echo "date: SQL script executed successfully for $1"
    echo "date: Output file saved to $2 (${size} bytes)."
    echo
}

# Given a query name from the array "queries" performs the query and saves the output.
run_query () {
    SQL="${BANNER_SCRIPTS}/$1.sql"
    CSV="${IN_BANNER}/$1.csv"
    echo "date: Beginning ${SQL} processing."
    sqlplus64 -S "${OUSER}/${OPASS}@//${OHOST}:${OPORT}/${OSID}" @${SQL}

# Check the exit status of sqlplus to see if the SQL script ran successfully
    if [[ $? -eq 0 ]]
    then
      success_file ${SQL} ${CSV}
    else
      cleandrop
    fi
}

# Try a connection just to make sure it works first otherwise exit
echo "exit" | sqlplus64 -L "${OUSER}/${OPASS}@//${OHOST}:${OPORT}/${OSID}" | grep Connected > /dev/null

if [[ $? -ne 0 ]]
then
   cleandrop
else
  echo "date: Connection checked okay, proceeding to fetch Banner data."
fi

# Remove files from previous runs if they exist.
cd $IN_BANNER

signal="${IN_BANNER}/DONE.txt"

if [[ -f "$signal" ]]
then
    echo "date: Removing signal file ${signal}"
    rm -f $signal
    echo "date: Removing any previous data files in ${IN_BANNER}"
    echo
    rm -f ${IN_BANNER}/*.csv
fi
rm -f *

# Set up the array of query names to execute with run_query()
queries=( ora_SRS_PEOPLE
          ora_SRS_COURSES
          ora_SRS_ENROLS
          ora_SRS_STAFF
          )

for query in "${queries[@]}"
do
    run_query ${query}
done

# drop in a signal file so other processes can tell if the downloads have completed.
echo "date: Creating completion signal file ${signal}"
touch ${signal}
echo
echo  "date: Listing of data files collected in ${IN_BANNER}"
ls -lh ${IN_BANNER}

cd ${SCRIPT_DIR}
echo
echo "date: overnights_oracle.sh completed"
exit 0
...and here it the output I see:
=============================================================================
=  Starting to fetch Banner data                                            =
=============================================================================
Sat Oct  7 15:55:53 BST 2023: Connection checked okay, proceeding to fetch Banner data.
Sat Oct  7 15:55:53 BST 2023: Removing signal file /home/p0071665/overnights/data/downloads/banner/DONE.txt
Sat Oct  7 15:55:53 BST 2023: Removing any previous data files in /home/p0071665/overnights/data/downloads/banner

Sat Oct  7 15:55:53 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_PEOPLE.sql processing.
Sat Oct  7 15:57:54 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_PEOPLE.sql
Sat Oct  7 15:57:54 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_PEOPLE.csv (10776253 bytes).

Sat Oct  7 15:57:54 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_COURSES.sql processing.
Sat Oct  7 15:58:04 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_COURSES.sql
Sat Oct  7 15:58:04 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_COURSES.csv (3992287 bytes).

Sat Oct  7 15:58:04 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_ENROLS.sql processing.
Sat Oct  7 16:01:07 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_ENROLS.sql
Sat Oct  7 16:01:07 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_ENROLS.csv (26717546 bytes).

Sat Oct  7 16:01:07 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_STAFF.sql processing.
Sat Oct  7 16:01:11 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_STAFF.sql
Sat Oct  7 16:01:11 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_STAFF.csv ( bytes).

Sat Oct  7 16:01:11 BST 2023: Creating completion signal file /home/p0071665/overnights/data/downloads/banner/DONE.txt

Sat Oct  7 16:01:11 BST 2023: Listing of data files collected in /home/p0071665/overnights/data/downloads/banner
total 41M
-rw-r--r-- 1 p0071665 p0071665    0 Oct  7 16:01 DONE.txt
-rw-r--r-- 1 p0071665 p0071665 3.9M Oct  7 15:58 ora_SRS_COURSES.csv
-rw-r--r-- 1 p0071665 p0071665  26M Oct  7 16:01 ora_SRS_ENROLS.csv
-rw-r--r-- 1 p0071665 p0071665  11M Oct  7 15:57 ora_SRS_PEOPLE.csv
-rw-r--r-- 1 p0071665 p0071665 1.2M Oct  7 16:01 ora_SRS_STAFF_ENROLS.csv
And here's what I get in the console when I run it:
stat: cannot statx '/home/p0071665/overnights/data/downloads/banner/ora_SRS_STAFF.csv': No such file or directory
And yet I know the file is there. I can move and copy it, it lists in an ls -al etc. I wondered if I had to do something to "close" the file before stat could find it, but I added a "touch" and "delete" a signalling file before it does the stat to no avail. It's not exactly critical but it's bugging me why it doesn't work on just the one element of the array of queries to run. Cheers, Jock
Asked by Jock Coats (11 rep)
Oct 7, 2023, 03:12 PM
Last activity: Oct 7, 2023, 08:02 PM