In a bash script (on Ubuntu 22 if it matters) I have a function which, executed from within a for...do loop, is given a filename to execute some SQLplus .sql and store the results in .csv
I added a little bit of a function to check success and to print out the file size of the csv it has just created.
BUT...it runs through this for three queries and reports the file size in the success message no problem. But for the fourth and final one it throws an error saying "Cannot statx: no such file or directory". I can see it's there. As an alternative I make the whole process list the files in the directory and it appears.
Any idea why it might not be doing the last one the same way.
Here's the code for the run_query() and success_file() functions which I think is all you need to see what I am doing:
[Edited to show the entire script file rather than just the snippet]
#!/bin/bash
source ~/overnights/scripts/.globals/.filenames
source ~/overnights/scripts/.globals/.creds
source ~/overnights/scripts/.globals/.commands.sh
echo "============================================================================="
echo "= Starting to fetch Banner data ="
echo "============================================================================="
# cleandrop() called from each sql script execution to finish the whole process if one file fails.
cleandrop () {
echo "date
: Problems connecting to or fetching from Oracle. Cleaning up and exiting."
rm -f ${IN_BANNER}/*
cd $SCRIPT_DIR
exit 1
}
# Displays confirmations of which file we have just processed.
success_file () {
size=$(stat -c%s "$2")
echo "date
: SQL script executed successfully for $1"
echo "date
: Output file saved to $2 (${size} bytes)."
echo
}
# Given a query name from the array "queries" performs the query and saves the output.
run_query () {
SQL="${BANNER_SCRIPTS}/$1.sql"
CSV="${IN_BANNER}/$1.csv"
echo "date
: Beginning ${SQL} processing."
sqlplus64 -S "${OUSER}/${OPASS}@//${OHOST}:${OPORT}/${OSID}" @${SQL}
# Check the exit status of sqlplus to see if the SQL script ran successfully
if [[ $? -eq 0 ]]
then
success_file ${SQL} ${CSV}
else
cleandrop
fi
}
# Try a connection just to make sure it works first otherwise exit
echo "exit" | sqlplus64 -L "${OUSER}/${OPASS}@//${OHOST}:${OPORT}/${OSID}" | grep Connected > /dev/null
if [[ $? -ne 0 ]]
then
cleandrop
else
echo "date
: Connection checked okay, proceeding to fetch Banner data."
fi
# Remove files from previous runs if they exist.
cd $IN_BANNER
signal="${IN_BANNER}/DONE.txt"
if [[ -f "$signal" ]]
then
echo "date
: Removing signal file ${signal}"
rm -f $signal
echo "date
: Removing any previous data files in ${IN_BANNER}"
echo
rm -f ${IN_BANNER}/*.csv
fi
rm -f *
# Set up the array of query names to execute with run_query()
queries=( ora_SRS_PEOPLE
ora_SRS_COURSES
ora_SRS_ENROLS
ora_SRS_STAFF
)
for query in "${queries[@]}"
do
run_query ${query}
done
# drop in a signal file so other processes can tell if the downloads have completed.
echo "date
: Creating completion signal file ${signal}"
touch ${signal}
echo
echo "date
: Listing of data files collected in ${IN_BANNER}"
ls -lh ${IN_BANNER}
cd ${SCRIPT_DIR}
echo
echo "date
: overnights_oracle.sh completed"
exit 0
...and here it the output I see:
=============================================================================
= Starting to fetch Banner data =
=============================================================================
Sat Oct 7 15:55:53 BST 2023: Connection checked okay, proceeding to fetch Banner data.
Sat Oct 7 15:55:53 BST 2023: Removing signal file /home/p0071665/overnights/data/downloads/banner/DONE.txt
Sat Oct 7 15:55:53 BST 2023: Removing any previous data files in /home/p0071665/overnights/data/downloads/banner
Sat Oct 7 15:55:53 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_PEOPLE.sql processing.
Sat Oct 7 15:57:54 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_PEOPLE.sql
Sat Oct 7 15:57:54 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_PEOPLE.csv (10776253 bytes).
Sat Oct 7 15:57:54 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_COURSES.sql processing.
Sat Oct 7 15:58:04 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_COURSES.sql
Sat Oct 7 15:58:04 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_COURSES.csv (3992287 bytes).
Sat Oct 7 15:58:04 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_ENROLS.sql processing.
Sat Oct 7 16:01:07 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_ENROLS.sql
Sat Oct 7 16:01:07 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_ENROLS.csv (26717546 bytes).
Sat Oct 7 16:01:07 BST 2023: Beginning /home/p0071665/overnights/scripts/sql/banner/ora_SRS_STAFF.sql processing.
Sat Oct 7 16:01:11 BST 2023: SQL script executed successfully for /home/p0071665/overnights/scripts/sql/banner/ora_SRS_STAFF.sql
Sat Oct 7 16:01:11 BST 2023: Output files saved to /home/p0071665/overnights/data/downloads/banner/ora_SRS_STAFF.csv ( bytes).
Sat Oct 7 16:01:11 BST 2023: Creating completion signal file /home/p0071665/overnights/data/downloads/banner/DONE.txt
Sat Oct 7 16:01:11 BST 2023: Listing of data files collected in /home/p0071665/overnights/data/downloads/banner
total 41M
-rw-r--r-- 1 p0071665 p0071665 0 Oct 7 16:01 DONE.txt
-rw-r--r-- 1 p0071665 p0071665 3.9M Oct 7 15:58 ora_SRS_COURSES.csv
-rw-r--r-- 1 p0071665 p0071665 26M Oct 7 16:01 ora_SRS_ENROLS.csv
-rw-r--r-- 1 p0071665 p0071665 11M Oct 7 15:57 ora_SRS_PEOPLE.csv
-rw-r--r-- 1 p0071665 p0071665 1.2M Oct 7 16:01 ora_SRS_STAFF_ENROLS.csv
And here's what I get in the console when I run it:
stat: cannot statx '/home/p0071665/overnights/data/downloads/banner/ora_SRS_STAFF.csv': No such file or directory
And yet I know the file is there. I can move and copy it, it lists in an ls -al etc.
I wondered if I had to do something to "close" the file before stat could find it, but I added a "touch" and "delete" a signalling file before it does the stat to no avail. It's not exactly critical but it's bugging me why it doesn't work on just the one element of the array of queries to run.
Cheers, Jock
Asked by Jock Coats
(11 rep)
Oct 7, 2023, 03:12 PM
Last activity: Oct 7, 2023, 08:02 PM
Last activity: Oct 7, 2023, 08:02 PM