Retry a Bash command with timeout
How to retry a bash command until its status is ok or until a timeout is reached? My best shot (I’m looking for something simpler):
NEXT_WAIT_TIME=0 COMMAND_STATUS=1 until [ $COMMAND_STATUS -eq 0 || $NEXT_WAIT_TIME -eq 4 ]; do command COMMAND_STATUS=$? sleep $NEXT_WAIT_TIME let NEXT_WAIT_TIME=NEXT_WAIT_TIME+1 done 7 Answers 7
You can simplify things a bit by putting command right in the test and doing increments a bit differently. Otherwise the script looks fine:
NEXT_WAIT_TIME=0 until [ $NEXT_WAIT_TIME -eq 5 ] || command; do sleep $(( NEXT_WAIT_TIME++ )) done [ $NEXT_WAIT_TIME -lt 5 ] Good solution, only problem is after the last «command» fail, you’ll still have to sleep for 4 seconds. Not sure if that can be avoided and keep the code this compact.
@David Really? Will the || operator evaluate the right-hand side if the left-hand side returns a truthy value? (I don’t know much bash, I just know that in javascript it wouldn’t.)
@openCivilisation that is indeed the effect achieved. That seems to be what the question was asking for but I’ve sometimes wondered why this answer received so many votes, since it does not seem like functionality that would frequently be desired.
One line and shortest, and maybe the best approach:
timeout 12h bash -c 'until ssh root@mynewvm; do sleep 10; done' I think for most purposes this is the best answer. If you need more unwieldy code you can use functions. Yet it keeps the main thing your trying to achieve short enough to be easy understood.
Note that timeout is a GNU extension, part of GNU coreutils , so it’s in essence unavailable (by default) on BSD/macOS-based systems. For a POSIX-compliant approximation of this, see unix.stackexchange.com/questions/274564/…
Amazing, worked wonderfully for my usecase! Had to check until a healthcheck passed in a container running in docker-compose, ended up being timeout 10s bash -c «until docker-compose ps $CONTAINER_NAME | grep \(healthy\); do sleep 1; done»
retry function
#!/bin/bash # Retries a command on failure. # $1 - the max number of attempts # $2. - the command to run retry()
# example usage: foo() < #whatever you want do. >declare -fxr foo retry 3 timeout 60 bash -ce 'foo' retry timeout 3 ping google.com PING google.com (173.194.123.97): 56 data bytes 64 bytes from 173.194.123.97: icmp_seq=0 ttl=55 time=13.982 ms 64 bytes from 173.194.123.97: icmp_seq=1 ttl=55 time=44.857 ms 64 bytes from 173.194.123.97: icmp_seq=2 ttl=55 time=64.187 ms Before retry #1: sleeping 0.3 seconds PING google.com (173.194.123.103): 56 data bytes 64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=56.549 ms 64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=60.220 ms 64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=8.872 ms Before retry #2: sleeping 0.6 seconds PING google.com (173.194.123.103): 56 data bytes 64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=25.819 ms 64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=16.382 ms 64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=3.224 ms Before retry #3: sleeping 1.2 seconds PING google.com (173.194.123.103): 56 data bytes 64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=58.438 ms 64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=94.828 ms 64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=61.075 ms Before retry #4: sleeping 2.4 seconds PING google.com (173.194.123.103): 56 data bytes 64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=43.361 ms 64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=32.171 ms . Check exit status for ultimate pass/fail.
Bash: wait with timeout
I.e., launch two applications in the background, and give them 60 seconds to complete their work. Then, if they don’t finish within that interval, kill them.
Unfortunately, the above does not work, since timeout is an executable, while wait is a shell command. I tried changing it to:
timeout 60 bash -c wait $pidApp1 $pidApp2 But this still does not work, since wait can only be called on a PID launched within the same shell.
Linux Solutions
Solution 1 — Linux
Both your example and the accepted answer are overly complicated, why do you not only use timeout since that is exactly its use case? The timeout command even has an inbuilt option ( -k ) to send SIGKILL after sending the initial signal to terminate the command ( SIGTERM by default) if the command is still running after sending the initial signal (see man timeout ).
If the script doesn’t necessarily require to wait and resume control flow after waiting it’s simply a matter of
timeout -k 60s 60s app1 & timeout -k 60s 60s app2 & # [. ] If it does, however, that’s just as easy by saving the timeout PIDs instead:
pids=() timeout -k 60s 60s app1 & pids+=($!) timeout -k 60s 60s app2 & pids+=($!) wait "$ " # [. ] $ cat t.sh #!/bin/bash echo "$(date +%H:%M:%S): start" pids=() timeout 10 bash -c 'sleep 5; echo "$(date +%H:%M:%S): job 1 terminated successfully"' & pids+=($!) timeout 2 bash -c 'sleep 5; echo "$(date +%H:%M:%S): job 2 terminated successfully"' & pids+=($!) wait "$ " echo "$(date +%H:%M:%S): done waiting. both jobs terminated on their own or via timeout; resuming script" $ ./t.sh 08:59:42: start 08:59:47: job 1 terminated successfully 08:59:47: done waiting. both jobs terminated on their own or via timeout; resuming script Solution 2 — Linux
Write the PIDs to files and start the apps like this:
pidFile=. ( app ; rm $pidFile ; ) & pid=$! echo $pid > $pidFile ( sleep 60 ; if [[ -e $pidFile ]]; then killChildrenOf $pid ; fi ; ) & killerPid=$! wait $pid kill $killerPid That would create another process that sleeps for the timeout and kills the process if it hasn’t completed so far.
If the process completes faster, the PID file is deleted and the killer process is terminated.
killChildrenOf is a script that fetches all processes and kills all children of a certain PID. See the answers of this question for different ways to implement this functionality: https://stackoverflow.com/questions/392022/best-way-to-kill-all-child-processes
If you want to step outside of BASH, you could write PIDs and timeouts into a directory and watch that directory. Every minute or so, read the entries and check which processes are still around and whether they have timed out.
EDIT If you want to know whether the process has died successfully, you can use kill -0 $pid
EDIT2 Or you can try process groups. kevinarpe said: To get PGID for a PID(146322):
ps -fjww -p 146322 | tail -n 1 | awk '< print $4 >' In my case: 145974. Then PGID can be used with a special option of kill to terminate all processes in a group: kill — -145974
Solution 3 — Linux
Here’s a simplified version of Aaron Digulla’s answer, which uses the kill -0 trick that Aaron Digulla leaves in a comment:
app & pidApp=$! ( sleep 60 ; echo 'timeout'; kill $pidApp ) & killerPid=$! wait $pidApp kill -0 $killerPid && kill $killerPid In my case, I wanted to be both set -e -x safe and return the status code, so I used:
set -e -x app & pidApp=$! ( sleep 45 ; echo 'timeout'; kill $pidApp ) & killerPid=$! wait $pidApp status=$? (kill -0 $killerPid && kill $killerPid) || true exit $status An exit status of 143 indicates SIGTERM, almost certainly from our timeout.
Solution 4 — Linux
I wrote a bash function that will wait until PIDs finished or until timeout, that return non zero if timeout exceeded and print all the PIDs not finisheds.
function wait_timeout < local limit=$ local pids=$ local count=0 while true do local have_to_wait=false for pid in $ ; do if kill -0 $ &>/dev/null; then have_to_wait=true else pids=`echo $ | sed -e "s/$ //g"` fi done if $ && (( $count < $limit )); then count=$(( count + 1 )) sleep 1 else echo $ return 1 fi done return 0 > To use this is just wait_timeout $timeout $PID1 $PID2 .
Solution 5 — Linux
To put in my 2c, we can boild down Teixeira’s solution to:
try_wait() < # Usage: [PID]. for ((i = 0; i < $#; i += 1)); do kill -0 $@ && sleep 0.001 || return 0 done return 1 # timeout or no PIDs > &>/dev/null Bash’s sleep accepts fractional seconds, and 0.001s = 1 ms = 1 KHz = plenty of time. However, UNIX has no loopholes when it comes to files and processes. try_wait accomplishes very little.
$ cat & [1] 16574 $ try_wait %1 && echo 'exited' || echo 'timeout' timeout $ kill %1 $ try_wait %1 && echo 'exited' || echo 'timeout' exited We have to answer some hard questions to get further.
Why has wait no timeout parameter? Maybe because the timeout , kill -0 , wait and wait -n commands can tell the machine more precisely what we want.
Why is wait builtin to Bash in the first place, so that timeout wait PID is not working? Maybe only so Bash can implement proper signal handling.
$ timeout 30s cat & [1] 6680 $ jobs [1]+ Running timeout 30s cat & $ kill -0 %1 && echo 'running' running $ # now meditate a bit and then. $ kill -0 %1 && echo 'running' || echo 'vanished' bash: kill: (NNN) - No such process vanished Whether in the material world or in machines, as we require some ground on which to run, we require some ground on which to wait too.
- When kill fails you hardly know why. Unless you wrote the process, or its manual names the circumstances, there is no way to determine a reasonable timeout value.
- When you have written the process, you can implement a proper TERM handler or even respond to «Auf Wiedersehen!» send to it through a named pipe. Then you have some ground even for a spell like try_wait 🙂
Solution 6 — Linux
app1 & app2 & sleep 60 & wait -n Timeout a command in bash without unnecessary delay
This answer to Command line command to auto-kill a command after a certain amount of time proposes a 1-line method to timeout a long-running command from the bash command line:
( /path/to/slow command with options ) & sleep 5 ; kill $! - has a bash implementation (the other question already has Perl and C answers)
- will terminate at the earlier of the two: tlrbsf program termination, or timeout elapsed
- will not kill non-existing/non-running processes (or, optionally: will not complain about a bad kill)
- doesn’t have to be a 1-liner
- can run under Cygwin or Linux
- runs the tlrbsf command in the foreground
- any ‘sleep’ or extra process in the background
such that the stdin/stdout/stderr of the tlrbsf command can be redirected, same as if it had been run directly?
If so, please share your code. If not, please explain why.
I have spent awhile trying to hack the aforementioned example but I’m hitting the limit of my bash skills.
Another similar question: stackoverflow.com/questions/526782/… (but I think the ‘timeout3’ answer here is much better).
timeout is great! you can even use with multiple commands (multi-line script): stackoverflow.com/a/61888916/658497
24 Answers 24
You are probably looking for the timeout command in coreutils. Since it’s a part of coreutils, it is technically a C solution, but it’s still coreutils. info timeout for more details. Here’s an example:
timeout 5 /path/to/slow/command with options To clarify, the homebrew command you need on OSX/mac is brew install coreutils , and then you can use gtimeout .
I think this is precisely what you are asking for:
#!/bin/bash # # The Bash shell script executes a command with a time-out. # Upon time-out expiration SIGTERM (15) is sent to the process. If the signal # is blocked, then the subsequent SIGKILL (9) terminates it. # # Based on the Bash documentation example. # Hello Chet, # please find attached a "little easier" :-) to comprehend # time-out example. If you find it suitable, feel free to include # anywhere: the very same logic as in the original examples/scripts, a # little more transparent implementation to my taste. # # Dmitry V Golovashkin scriptName="$" declare -i DEFAULT_TIMEOUT=9 declare -i DEFAULT_INTERVAL=1 declare -i DEFAULT_DELAY=1 # Timeout. declare -i timeout=DEFAULT_TIMEOUT # Interval between checks if the process is still alive. declare -i interval=DEFAULT_INTERVAL # Delay between posting the SIGTERM signal and destroying the process by SIGKILL. declare -i delay=DEFAULT_DELAY function printUsage() < cat # Options. while getopts ":t:i:d:" option; do case "$option" in t) timeout=$OPTARG ;; i) interval=$OPTARG ;; d) delay=$OPTARG ;; *) printUsage; exit 1 ;; esac done shift $((OPTIND - 1)) # $# should be at least 1 (the command to execute), however it may be strictly # greater than 1 if the command itself has options. if (($# == 0 || interval 0)); do sleep $interval kill -0 $$ || exit 0 ((t -= interval)) done # Be nice, post SIGTERM first. # The 'exit 0' below will be executed if any preceeding command fails. kill -s SIGTERM $$ && kill -0 $$ || exit 0 sleep $delay kill -s SIGKILL $$ ) 2> /dev/null & exec "$@"