Sample Header Ad - 728x90

bash: filter except the latest n records

0 votes
2 answers
70 views
I'm creating a small script that will delete indexes on an Elasticsearch cluster to prevent it for fill up all the storage with logstash data. I have a list of records, and I would like to keep the latest n records (for example 7) and delete all the others. I can get the list of the indexes with curl:
drakaris:~/ # curl -sL localhost:9200/_cat/indices/logstash-* | awk '{print $3;}' | sort
logstash-2022.12.30
logstash-2022.12.31
logstash-2023.01.01
logstash-2023.01.02
logstash-2023.01.03
logstash-2023.01.04
logstash-2023.01.05
logstash-2023.01.06
logstash-2023.01.07
logstash-2023.01.08
logstash-2023.01.09
In my script I would like to keep only the latest 7th indexes and delete all the others (logstash-2022.12.30, logstash-2022.12.31. logstash-2023.01.01, logstash-2023.01.02) using "curl -XDELETE localhost:9200/index". How can I get these records from an array like that in bash? Thanks --- [EDIT] I solved in this way, just in case someone find it useful
RETENTION=7
nbk=$(curl -sL localhost:9200/_cat/indices/logstash-* | awk '{print $3;}' | wc -l)
if [ $nbk -gt $RETENTION ]; then
    echo -e "======== Delete obsolete indexes (retention: $RETENTION)"
    let ntodel=$nbk-$RETENTION
    for efile in $(curl -sL localhost:9200/_cat/indices/logstash-* | awk '{print $3;}' | sort -r | /usr/bin/tail -$ntodel); do
        curl -XDELETE localhost:9200/$efile
        sleep 10
    done
fi
Asked by Tasslehoff Burrfoot (1 rep)
Jan 9, 2023, 02:58 PM
Last activity: Jan 10, 2023, 09:27 AM