How to add date string to each line of a continuously written log file
Having a long running program that continuously writes to a logfile — how is it possible, disregarding any buffering issues, to add a date string to each line written to that file using a linux script? I would imagine something like this:
tail -f logfile | ADD_DATE_TO_EACH_LINE > logfile2 2011-06-16 18:30:59 abc 2011-06-16 18:31:00 def 2011-06-16 18:35:21 ghi 2011-06-16 18:40:15 jkl 8 Answers 8
Replace command with tail -f logfile for your specific example. Or, perhaps you could just redirect the original program’s stdout/stderr to the above pipe.
The benefit you get with this answer is that it doesn’t spawn a new date process with each line, but both work well.
Thanks! I don’t really know perl. I just cut and paste other crud from the internet. 🙂 Last time I used perl for real was over a decade ago! I usually use the gawk provided answer above, but included the perl one too in case the OP was not on linux.
@StevePrentice When I do the first command, it hangs. I’m running tail -f test-text.rtf 2>&1 | perl -pe ‘print scalar(localtime()), » «;’ > test-text.rtf Any idea why?
tail -f logfile | while read line; do echo `date` "$line" ; done +1 Just one note though: using date +»%y-%m-%d %H:%M:%S» instead of simply date would result in the exact output he expects; and you could also add the redirect to the output file in the end. (It’s actually two notes) 🙂
Thank you, that works. And if you add » signs to $line the shell will not expand * signs to the directory content 😉
the problem with this solution is : when the input buffer given to the ‘while read line’ loop is too big, some lines will be truncate.
cat /etc/motd | xargs -d"\n" -I <> date +"%Y-%m-%d %H:%M:%S <>" 2013-02-26 15:13:57 2013-02-26 15:13:57 The programs included with the Debian GNU/Linux system are free software; 2013-02-26 15:13:57 the exact distribution terms for each program are described in the 2013-02-26 15:13:57 individual files in /usr/share/doc/*/copyright. 2013-02-26 15:13:57 2013-02-26 15:13:57 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent 2013-02-26 15:13:57 permitted by applicable law.
@Vouze You can fix that with sed in between, or actually rather instead of the useless use of cat ; sed ‘s/%/%%/g’ /etc/motd | xargs .
A tool exists for that exact purpose, it’s ts (see man ts )
For example, with your logfile:
tail -f logfile | ts '%Y-%m-%d %H:%M:%S' Also works with any software writing on the standard output, of course:
If needed, you can add sub-second precision, with %.S instead of %S
A little lengthy, but here is what I came up with:
tail -f logfile | sed -u 's/%/%%/g' | xargs -I <> date +"%Y-%m-%d %H:%M:%S <>" That’s a very good solution. But what’s the sed -u ‘s/%/%%/g’ for? Do the % signs have a special meaning for xargs ?
@bmk This method works by putting the entire line into the format string for date , where % has a special meaning. For example if the line in the log contained %Y then it would get replaced by the year, but changing it to %%Y with sed will correctly output %Y .
I have one additional remark: If there are single quotes ( ‘ ) in the input stream xargs terminates with following error message: xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option therefore I changed the command to tail -f logfile | sed -u ‘s/%/%%/g’ | tr «\n» «\0» | xargs -0 -I <> date +»%Y-%m-%d %H:%M:%S <>» .
Can you configure the long running program to write it’s output to the standard output and not to the logfile? In this case it would be easy to pipe the output to a script that first writes the current timestamp and then the entry.
If that is impossible, it may help to periodically (e.g. every second) read the logfile content, copy each line to another file (adding the current timestamp) and then deleting the logfile. This may, however, impose losing logfile entries that are written between reading and deleting the file 🙁
I cannot configure the program to log to standard output because it is (resp. can be used as) an interactive program. But actually executing tail -f is in my opinion more or less the same as directly writing to stdout. Actually I’m looking for that «script that first writes the current timestamp and then the entry».
you could use a FIFO then. The Program will write everything to the named pipe (fifo) and you can read from it the same time, append the date and write to the real logfile.
How to get time range in a log file in linux? [duplicate]
I define some variables: date_now=date +»%Y-%m-%d %H:%M:%S,%3N» , date_minus_one=date -d «-1 hour» +»%Y-%m-%d %H:%M:%S,%3N» and date_minus_two=date -d «-2 hour» +»%Y-%m-%d %H:%M:%S,%3N» . I know the logic that you can get a per hour by making `date_now >= date_minus_one and date_now
2016-03-30|02:56:41,410|[WARNING] 2016-03-30|02:42:13,691|[UNKNOWN] 2016-03-30|02:53:16,356|[UNKNOWN] 5 Answers 5
grep -E '^2016-03-30\|02:1:3,' file.log hour_to_search=$(date '+%F\|%H') grep -E "^$:7:8," file.log Or use a speciic date-time as reference:
hour_to_search=$(date -d '30 Mar 2016 02 AM' '+%F\|%H') grep -E "^$:2:3," file.log $ cat file.log 2016-03-30|00:54:46,060|[WARNING] 2016-03-30|00:55:46,318|[OK] 2016-03-30|00:55:46,318|[OK] 2016-03-30|01:42:13,691|[UNKNOWN] 2016-03-30|01:53:16,356|[CRITICAL] 2016-03-30|02:56:41,410|[WARNING] 2016-03-30|02:42:13,691|[UNKNOWN] 2016-03-30|02:53:16,356|[UNKNOWN] 2016-03-30|03:56:41,410|[WARNING] $ hour_to_search=$(date -d '30 Mar 2016 02 AM' '+%F\|%H') $ echo "$hour_to_search" 2016-03-30\|02 $ grep -E "^$:1:3," file.txt 2016-03-30|02:56:41,410|[WARNING] 2016-03-30|02:42:13,691|[UNKNOWN] 2016-03-30|02:53:16,356|[UNKNOWN] How can i make it dynamic? that every hour I can get the data. No need to enter values of date and hour. Thanks,
This approach has the same shortcoming as Michael’s answer of being «aligned» on hours of the actual time, so in the worst case there will be almost no output because the hour you are grepping for has just started.
@heemayl : how about the hour? how can you make it dynamic? I’ve tried your syntax it works but I need to change the hour to get the past one hour data.
Nicely done. You can probably assume that after 2016-03-30|02 the format will be consistent and just grep for this part.
#To get the current date and hour hour=$( date +'%F|%H') #Filter the logs grep $hour log Note: Edited Based on suggestion in comment.
thanks for efficent way, I have incorporated your suggestions. however, earlier version was also providing users solution.
Make search pattern that includes the date and hour only ( «%Y-%m-%d|%H:» ) and use it for filtering with any one of the tools you’ve mentioned
I’ve tried awk: awk ‘$date_file >= $date_minus_one && $date_file
This approach will cause the output to be «aligned» on hours, so in the worst case you will get less than one minute of output, because the last hour has just started.
Yes, but this is exactly what was the purpose of this question: «I want to get an interval of one hour»
You might also try something like
csplit -zk sample.log '/2016-03-30|0'/ '' awk -F "|" '< print "date -d\""$1 " " $2"\" +%s.%N" "'\''|" $0 "'\''" >' mylog.file |\ bash |\ awk -F "|" -v OFS="|" \ -v startdate=$(date -d "2016-03-30 01:42:13,691" "+%s.%N") \ -v enddate=$(date -d "2016-03-30 02:53:16,356" "+%s.%N") \ '$1>=startdate && $1' Adapt the startdate and enddate to your needs, it should be seconds since EPCOH and nanoseconds: format is «+%s.%N»
First part with awk . just build a date command line and uses bash to execute it. The goal is to add at the beginning of each line the date as seconds since EPOCH and nanoseconds.
The final awk compares the dates in seconds.nanoseconds format and prints only the lines in the range: [startdate .. enddate]
$ awk -F "|" \ '< print "date -d\""$1 " " $2"\" +%s.%N" "'\''|" $0 "'\''" >' mylog.file |\ bash |\ awk -F "|" -v OFS="|" \ -v startdate=$(date -d "2016-03-30 01:42:13,691" "+%s.%N") \ -v enddate=$(date -d "2016-03-30 02:53:16,356" "+%s.%N") \ '$1>=startdate && $1' |2016-03-30|01:42:13,691|[UNKNOWN] |2016-03-30|01:53:16,356|[CRITICAL] |2016-03-30|02:42:13,691|[UNKNOWN] |2016-03-30|02:53:16,356|[UNKNOWN] append time to log filenames
I want to make log_files named as log_Jan_01_hh:mm:ss , so the variable time-stamp is defined as date +»%b_%d_%T» .
Now if I set the value to a variable:
Then $now variable does get value update at that moment, so all the time log files have same name. How can I do something so that $now gets it value refreshed at every moment?
I can do «log_$(date +»%b_%d_%T»)» but it would be easier if I have it in a variable so that I can just use log_$now
@muru I will define that variable in my ~/.cshrc file, so that when ever I need I can use that variable as log_$now whenever I need. I didn’t get the «easier» part in using or typing Whereever you’re using the filename, use «log_$(date +»%b_%d_%T»)» for all time
Making $now have an updated variable won’t be easy. Hence, not easier. If you want, create a function now and do log_$(now) .
2 Answers 2
If you just store a string in $now , and add backticks when you invoke it, you get the current date:
[~]% setenv now 'date +"%b_%d_%T"' [~]% echo $now date +"%b_%d_%T" [~]% echo `$now` "Dec_03_13:09:52" [~]% echo `$now` "Dec_03_13:09:54" This is effectively the same as doing:
There is no way to tell tcsh to re-evaluate the command on invoking the variable; you assign a string to $now , and tcsh has no way of knowing what the command was that generated this string.
Another (possible) solution might be using functions, but tcsh lacks those as well.