Sample Header Ad - 728x90

How to know when a following tail moves from old file to the new one

1 vote
1 answer
207 views
So, I am working with tail -F ( or tail --follow=filename). Now it works as advertised and when a rollover occurs it will move to the new file. This is great and helps me keep track of my logs. The issue is I want to know when tail moves from old file to the new one. **The situation is as follows:** I have a huge log file that takes 15 mins to process. Now lets say at minute 5, a rollover occurs. Tail has the file description open, and uses it to complete tailing process and then moves to the new one. Now I keep a history of which file and which line I have last logged. I do this by increasing the number by the lines processes (its multiprocess program, but what else can I do?). The issue comes that after the rollover, the new file is starting from line 0, but my line_number has already been increase to 5 million. So, for this new file which has say 100 logs, I'll store the line number as 5,000,100. I used watchdog to find when the file rolls over to reset the line number to 0, but in case the rollover happens during the initial run, like say at 5minute mark of the 15 min run, then I still end up with a 3mil+ number. Since line_number is used to continue from where I last left in case the program died accidentally, this can mean loss of data. Just to note: I am running this command from python (CPython)
Asked by user607688 (13 rep)
Apr 8, 2024, 11:21 AM
Last activity: Apr 8, 2024, 12:10 PM