Sample Header Ad - 728x90

create variables from CSV with varying number of fields

0 votes
4 answers
1869 views
Looking for some help turning a CSV into variables. I tried using IFS, but seems you need to define the number of fields. I need something that can handle varying number of fields. *I am modifying my original question with the current code I'm using (taken from the answer provided by hschou) which includes updated variable names using type instead of row, section etc. I'm sure you can tell by my code, but I am pretty green with scripting, so I am looking for help to determine if and how I should add another loop or take a different approach to parsing the typeC data because although they follow the same format, there is only one entry for each of the typeA and typeB data, and there can be between 1-15 entries for the typeC data. The goal being only 3 files, one for each of the data types. Data format: Container: PL[1-100] TypeA: [1-20].[1-100].[1-1000].[1-100]-[1-100] TypeB: [1-20].[1-100].[1-1000].[1-100]-[1-100] TypeC (1 to 15 entries): [1-20].[1-100].[1-1000].[1-100]-[1-100] *There is no header in the CSV, but if there were it would look like this (Container, typeA, and typeB data always being in position 1,2,3, and typeC data being all that follow): Container,typeA,typeB,typeC,tycpeC,typeC,typeC,typeC,.. CSV: PL3,12.1.4.5-77,13.6.4.5-20,17.3.577.9-29,17.3.779.12-33,17.3.802.12-60,17.3.917.12-45,17.3.956.12-63,17.3.993.12-42 PL4,12.1.4.5-78,13.6.4.5-21,17.3.577.9-30,17.3.779.12-34 PL5,12.1.4.5-79,13.6.4.5-22,17.3.577.9-31,17.3.779.12-35,17.3.802.12-62,17.3.917.12-47 PL6,12.1.4.5-80,13.6.4.5-23,17.3.577.9-32,17.3.779.12-36,17.3.802.12-63,17.3.917.12-48,17.3.956.12-66 PL7,12.1.4.5-81,13.6.4.5-24,17.3.577.9-33,17.3.779.12-37,17.3.802.12-64,17.3.917.12-49,17.3.956.12-67,17.3.993.12-46 PL8,12.1.4.5-82,13.6.4.5-25,17.3.577.9-34 Code: #!/bin/bash #Set input file _input="input.csv" # Pull variables in from csv # read file using while loop while read; do declare -a COL=( ${REPLY//,/ } ) echo -e "containerID=${COL}\ntypeA=${COL}\ntypeB=${COL}" >/tmp/typelist.txt idx=1 while [ $idx -lt 10 ]; do echo "typeC$idx=${COL[$((idx+2))]}" >>/tmp/typelist.txt let idx=idx+1 #whack off empty variables sed '/\=$/d' /tmp/typelist.txt > /tmp/typelist2.txt && mv /tmp/typelist2.txt /tmp/typelist.txt #set variables from temp file . /tmp/typelist.txt done sleep 1 #Parse data in this loop.# echo -e "\n" echo "Begin Processing for $container" #echo $typeA #echo $typeB #echo $typeC #echo -e "\n" #Strip - from sub data for extra parsing typeAsub="$(echo "$typeA" | sed 's/\-.*$//')" typeBsub="$(echo "$typeB" | sed 's/\-.*$//')" typeCsub1="$(echo "$typeC1" | sed 's/\-.*$//')" #strip out first two decimils for extra parsing typeAprefix="$(echo "$typeA" | cut -d "." -f1-2)" typeBprefix="$(echo "$typeB" | cut -d "." -f1-2)" typeCprefix1="$(echo "$typeC1" | cut -d "." -f1-2)" #echo $typeAsub #echo $typeBsub #echo $typeCsub1 #echo -e "\n" #echo $typeAprefix #echo $typeBprefix #echo $typeCprefix1 #echo -e "\n" echo "Getting typeA dataset for $typeA" #call api script to pull data ; echo out for test echo "API-gather -option -b "$typeAsub" -g all > "$container"typeA-dataset" sleep 1 echo "Getting typeB dataset for $typeB" #call api script to pull data ; echo out for test echo "API-gather -option -b "$typeBsub" -g all > "$container"typeB-dataset" sleep 1 echo "Getting typeC dataset for $typeC1" #call api script to pull data ; echo out for test echo "API-gather -option -b "$typeCsub" -g all > "$container"typeC-dataset" sleep 1 echo "Getting additional typeC datasets for $typeC2-15" #call api script to pull data ; echo out for test echo "API-gather -option -b "$typeCsub2-15" -g all >> "$container"typeC-dataset" sleep 1 echo -e "\n" done < "$_input" exit 0 Speed isnt a concern, but if I've done anything really stupid up there, feel free to slap me in the right direction. :)
Asked by Jdubyas (45 rep)
Jul 12, 2017, 05:09 AM
Last activity: Aug 6, 2025, 12:04 AM