Why does system() fail with error code 127?
On a Linux system I am trying to call a program at runtime with the system() call. The system call exits with an return code not equal zero. Calling WEXITSTATUS on the error code gives «127». According to the man page of system this code indicates that /bin/sh could not be called: In case /bin/sh could not be executed, the exit status will be that of a command that does exit(127) . I checked: /bin/sh is a link to bash . bash is there. I can execute it from the shell. Now, how can I find out why /bin/sh could not be called ? Any kernel history or something? Edit: After the very helpful tip (see below) i strace -f -p
Process 16080 detached [pid 11779] ) = ? ERESTARTNOHAND (To be restarted) [pid 11774] [], 0, NULL) = 16080 [pid 11779] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 11779] rt_sigaction(SIGCHLD, , [pid 11774] rt_sigaction(SIGINT, , [pid 11779] , 8) = 0 [pid 11779] sendto(5, "a", 1, 0, NULL, 0 [pid 11774] NULL, 8) = 0 [pid 11779] ) = 1 [pid 11779] rt_sigreturn(0x2 [pid 11774] rt_sigaction(SIGQUIT, , [pid 11779] ) = -1 EINTR (Interrupted system call) [pid 11779] select(16, [9 15], [], NULL, NULL [pid 11774] NULL, 8) = 0 [pid 11774] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 11774] write(1, "Problems calling nvcc jitter: ex". 49) = 49 [pid 11774] rt_sigaction(SIGINT, , , 8) = 0 [pid 11774] rt_sigaction(SIGQUIT, , , 8) = 0 [pid 11774] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 [pid 11774] clone(Process 16081 attached (waiting for parent) Process 16081 resumed (parent 11774 ready) child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff0177ab68) = 16081 [pid 16081] rt_sigaction(SIGINT, , [pid 11774] wait4(16081, Process 11774 suspended [pid 16081] NULL, 8) = 0 [pid 16081] rt_sigaction(SIGQUIT, , NULL, 8) = 0 [pid 16081] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 16081] execve("/bin/sh", ["sh", "-c", 0xdda1d98], [/* 58 vars */]) = -1 EFAULT (Bad address) [pid 16081] exit_group(127) = ? Process 11774 resumed When it comes to the call to /bin/sh it says bad address. Why that ? Edit: Here the whole part that involves the failing system (here already the safe copy to a buffer is in place):
std::ostringstream jit_command; jit_command else if (WIFSIGNALED(ret)) < printf("killed by signal %d\n", WTERMSIG(ret)); >else if (WIFSTOPPED(ret)) < printf("stopped by signal %d\n", WSTOPSIG(ret)); >else if (WIFCONTINUED(ret)) < printf("continued\n"); >else < printf("not recognized\n"); >cout delete[] cmd; return true; /usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link bench_cudp_Oku2fm.cu -I$LIB_PATH/include -o bench_cudp_Oku2fm.o Problems calling nvcc jitter: exited, status=127 Checking shell.. ok! string gen = jit_command.str(); cout The complexity of the string creation is not the problem here. As strace shows a "bad address" is the problem. Its a legal string. A "bad address" should not occur. As far as i know the std::string::c_str() returns a const char * that might point to a scratch space of libc++ where a read only copy of the string might be kept. Unfortunately the error is not really reproduceable. The call to system succeeds several times before it fails. I don't want to be hasty but it smells like a bug in either in the kernel, libc or the hardware. Edit: I produced a more verbose strace output ( strace -f -v -s 2048 -e trace=process -p $! ) of the failing execve system call: First a succeeding call:
[pid 2506] execve("/bin/sh", ["sh", "-c", "/usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.cu -I$LIB_PATH/include -o /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.o"], ["MODULE_VERSION_STACK=3.2.8", . ]) = 0 [pid 17398] execve("/bin/sh", ["sh", "-c", 0x14595af0], ) = -1 EFAULT (Bad address) Here is identical. It seems its not the list of environment variables that cause the bad address. As Chris Dodd mentioned the 3rd argument to execve is the raw pointer 0x14595af0, which strace thinks (and the kernel agrees) is invalid. strace does not recognize it as a string (so it prints the hex value and not the string). Edit: I inserted print out of the pointer value cmd to see what's the value of this pointer in the parent process:
string gen = jit_command.str(); cout cmd = 0x14595af0 failed cmd = 0x14595af0 Problems calling nvcc jitter: exited, status=127 Checking shell.. ok! Its the same pointer value as the 3rd argument from strace . (I updated the strace output above). Regards the 32bit looking of the cmd pointer: I checked the value of the cmd pointer for a succeeding call. Can't see any difference in structure. That's one of the values of cmd when then system call succeeds:
Meanwhile let me post a workaround. Its so silly to be forced to implement something like that. but it works. So the following code block gets executed in case the system call fails. It allocates new command strings and retries until it succeeds (well not indefinitely).
list listPtr; int maxtry=1000; do < char* tmp = new(nothrow) char[gen.size()+1]; if (!tmp) __error_exit("no memory for jitter command"); strcpy(tmp,gen.c_str()); listPtr.push_back( tmp ); >while ((ret=system(listPtr.back())) && (--maxtry>0)); while(listPtr.size()) I just saw that this workaround in one particular run did not work. It went the whole way, 1000 attempts, all with newly allocated cmd command strings. All 1000 failed. Not only this. I tried on a different Linux host (same Linux/software configuration tho).
Taking this into account one would maybe exclude a hardware problem. (Must be on 2 physically different hosts then). Remains a kernel bug ??
torek, i will try and install a modified system call. Give me some time for that.
How to fix exit status 127?
I attempted to follow this guide to run a Node application as a service. However, it is failing to start, with exit code 127. Is there any way to fix this? This is the journal.
sudo journalctl --follow -u serviceName -- Logs begin at Tue 2017-08-08 16:27:10 GMT. -- Aug 08 17:06:57 raspberrypi systemd[1]: Started serviceName. Aug 08 17:06:57 raspberrypi app.js[7234]: [46B blob data] Aug 08 17:06:57 raspberrypi systemd[1]: serviceName.service: main process exited, code=exited, status=127/n/a Aug 08 17:06:57 raspberrypi systemd[1]: Unit serviceName.service entered failed state. Aug 08 17:06:57 raspberrypi systemd[1]: serviceName.service holdoff time over, scheduling restart. Aug 08 17:06:57 raspberrypi systemd[1]: Stopping serviceName. Aug 08 17:06:57 raspberrypi systemd[1]: Starting serviceName. Aug 08 17:06:57 raspberrypi systemd[1]: serviceName.service start request repeated too quickly, refusing to start. Aug 08 17:06:57 raspberrypi systemd[1]: Failed to start serviceName. Aug 08 17:06:57 raspberrypi systemd[1]: Unit serviceName.service entered failed state. [Unit] Description=ServiceName After=network.target [Service] ExecStart=/home/pi/projects/ServiceName/app.js Restart=always User=root Group=root Environment=PATH=/usr/bin:/usr/local/bin Environment=NODE_ENV=production WorkingDirectory=/home/pi/projects/ServiceName [Install] WantedBy=multi-user.target 1. node is /home/pi/.nvm/versions/node/v7.8.0/bin/node 2. Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 6166268 4446224 1383764 77% /
127 Return code from $?
Value 127 is returned by /bin/sh when the given command is not found within your PATH system variable and it is not a built-in shell command. In other words, the system doesn't understand your command, because it doesn't know where to find the binary you're trying to call.
You can try using which [program] to see which binary the OS is using. If it comes up empty, next step is checking execution bit and PATH.
@cr125rider, which is not particularly accurate -- it doesn't know about aliases, shell functions, PATH lookup memoization, or other factors internal to shell state. Much better to use type , a shell builtin which knows about all of those things.
This also happened to me with a file that had Windows line feeds. Correcting the line endings to unix format solved the problem
@MatthewKremer: Actually, I get 126 ( Permission denied ), not 127 when I attempt to invoke a non-executable file (irrespective of its contents); similarly, an attempt to execute a directory also results in 126 ( is a directory ).
but it can also mean that the command is found,
but a library that is required by the command is NOT found.
example: $caat The error message will
A shell convention is that a successful executable should exit with the value 0. Anything else can be interpreted as a failure of some sort, on part of bash or the executable you that just ran. See also $PIPESTATUS and the EXIT STATUS section of the bash man page:
For the shell’s purposes, a command which exits with a zero exit status has succeeded. An exit status of zero indicates success. A non-zero exit status indicates failure. When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status. If a command is not found, the child process created to execute it returns a status of 127. If a com- mand is found but is not executable, the return status is 126. If a command fails because of an error during expansion or redirection, the exit status is greater than zero. Shell builtin commands return a status of 0 (true) if successful, and non-zero (false) if an error occurs while they execute. All builtins return an exit status of 2 to indicate incorrect usage. Bash itself returns the exit status of the last command executed, unless a syntax error occurs, in which case it exits with a non-zero value. See also the exit builtin command below. How do I solve "Server terminated early with status 127" when running node.js on Linux?
I’m getting the below error when running my script (“Server terminated early with status 127”). I have verified that I can reach the URL in question using “wget”, so I can’t tell what else I need to make this thing work …
[davea@mydevbox mydir]$ node myscript.js Validation Complete /home/davea/node_modules/selenium-webdriver/lib/promise.js:654 throw error; ^ Error: Server terminated early with status 127 at Error (native) at /home/davea/node_modules/selenium-webdriver/remote/index.js:242:20 at ManagedPromise.invokeCallback_ (/home/davea/node_modules/selenium-webdriver/lib/promise.js:1343:14) at TaskQueue.execute_ (/home/davea/node_modules/selenium-webdriver/lib/promise.js:2868:14) at TaskQueue.executeNext_ (/home/davea/node_modules/selenium-webdriver/lib/promise.js:2851:21) at /home/davea/node_modules/selenium-webdriver/lib/promise.js:2730:27 at /home/davea/node_modules/selenium-webdriver/lib/promise.js:639:7 at process._tickCallback (internal/process/next_tick.js:103:7) From: Task: WebDriver.createSession() at acquireSession (/home/davea/node_modules/selenium-webdriver/lib/webdriver.js:107:22) at Function.createSession (/home/davea/node_modules/selenium-webdriver/lib/webdriver.js:337:12) at Driver (/home/davea/node_modules/selenium-webdriver/chrome.js:778:38) at Builder.build (/home/davea/node_modules/selenium-webdriver/builder.js:464:16) at Object. (/home/davea/mydir/js/Optimus.js:14:4) at Module._compile (module.js:413:34) at Object.Module._extensions..js (module.js:422:10) at Module.load (module.js:357:32) at Function.Module._load (module.js:314:12) at Module.require (module.js:367:17) From: Task: WebDriver.navigate().to(http://localhost:8081/myproject) at Driver.schedule (/home/davea/node_modules/selenium-webdriver/lib/webdriver.js:386:17) at Navigation.to (/home/davea/node_modules/selenium-webdriver/lib/webdriver.js:1029:25) at Driver.get (/home/davea/node_modules/selenium-webdriver/lib/webdriver.js:797:28) at Object.exports.Login.Page (/home/davea/mydir/js/Optimus.js:505:16) at Object.exports.smokeTest (/home/davea/mydir/js/Optimus.js:2442:19) at Object. (/home/davea/mydir/SkyNet.js:13:6) at Module._compile (module.js:413:34) at Object.Module._extensions..js (module.js:422:10) at Module.load (module.js:357:32) at Function.Module._load (module.js:314:12)