Why does system() fail with error code 127?
On a Linux system I am trying to call a program at runtime with the system() call. The system call exits with an return code not equal zero. Calling WEXITSTATUS on the error code gives «127». According to the man page of system this code indicates that /bin/sh could not be called: In case /bin/sh could not be executed, the exit status will be that of a command that does exit(127) . I checked: /bin/sh is a link to bash . bash is there. I can execute it from the shell. Now, how can I find out why /bin/sh could not be called ? Any kernel history or something? Edit: After the very helpful tip (see below) i strace -f -p
Process 16080 detached [pid 11779] ) = ? ERESTARTNOHAND (To be restarted) [pid 11774] [], 0, NULL) = 16080 [pid 11779] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 11779] rt_sigaction(SIGCHLD, , [pid 11774] rt_sigaction(SIGINT, , [pid 11779] , 8) = 0 [pid 11779] sendto(5, "a", 1, 0, NULL, 0 [pid 11774] NULL, 8) = 0 [pid 11779] ) = 1 [pid 11779] rt_sigreturn(0x2 [pid 11774] rt_sigaction(SIGQUIT, , [pid 11779] ) = -1 EINTR (Interrupted system call) [pid 11779] select(16, [9 15], [], NULL, NULL [pid 11774] NULL, 8) = 0 [pid 11774] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 11774] write(1, "Problems calling nvcc jitter: ex". 49) = 49 [pid 11774] rt_sigaction(SIGINT, , , 8) = 0 [pid 11774] rt_sigaction(SIGQUIT, , , 8) = 0 [pid 11774] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 [pid 11774] clone(Process 16081 attached (waiting for parent) Process 16081 resumed (parent 11774 ready) child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff0177ab68) = 16081 [pid 16081] rt_sigaction(SIGINT, , [pid 11774] wait4(16081, Process 11774 suspended [pid 16081] NULL, 8) = 0 [pid 16081] rt_sigaction(SIGQUIT, , NULL, 8) = 0 [pid 16081] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 16081] execve("/bin/sh", ["sh", "-c", 0xdda1d98], [/* 58 vars */]) = -1 EFAULT (Bad address) [pid 16081] exit_group(127) = ? Process 11774 resumed When it comes to the call to /bin/sh it says bad address. Why that ? Edit: Here the whole part that involves the failing system (here already the safe copy to a buffer is in place):
std::ostringstream jit_command; jit_command else if (WIFSIGNALED(ret)) < printf("killed by signal %d\n", WTERMSIG(ret)); >else if (WIFSTOPPED(ret)) < printf("stopped by signal %d\n", WSTOPSIG(ret)); >else if (WIFCONTINUED(ret)) < printf("continued\n"); >else < printf("not recognized\n"); >cout delete[] cmd; return true; /usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link bench_cudp_Oku2fm.cu -I$LIB_PATH/include -o bench_cudp_Oku2fm.o Problems calling nvcc jitter: exited, status=127 Checking shell.. ok! string gen = jit_command.str(); cout The complexity of the string creation is not the problem here. As strace shows a "bad address" is the problem. Its a legal string. A "bad address" should not occur. As far as i know the std::string::c_str() returns a const char * that might point to a scratch space of libc++ where a read only copy of the string might be kept. Unfortunately the error is not really reproduceable. The call to system succeeds several times before it fails. I don't want to be hasty but it smells like a bug in either in the kernel, libc or the hardware. Edit: I produced a more verbose strace output ( strace -f -v -s 2048 -e trace=process -p $! ) of the failing execve system call: First a succeeding call:
[pid 2506] execve("/bin/sh", ["sh", "-c", "/usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.cu -I$LIB_PATH/include -o /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.o"], ["MODULE_VERSION_STACK=3.2.8", . ]) = 0 [pid 17398] execve("/bin/sh", ["sh", "-c", 0x14595af0], ) = -1 EFAULT (Bad address) Here is identical. It seems its not the list of environment variables that cause the bad address. As Chris Dodd mentioned the 3rd argument to execve is the raw pointer 0x14595af0, which strace thinks (and the kernel agrees) is invalid. strace does not recognize it as a string (so it prints the hex value and not the string). Edit: I inserted print out of the pointer value cmd to see what's the value of this pointer in the parent process:
string gen = jit_command.str(); cout cmd = 0x14595af0 failed cmd = 0x14595af0 Problems calling nvcc jitter: exited, status=127 Checking shell.. ok! Its the same pointer value as the 3rd argument from strace . (I updated the strace output above). Regards the 32bit looking of the cmd pointer: I checked the value of the cmd pointer for a succeeding call. Can't see any difference in structure. That's one of the values of cmd when then system call succeeds:
Meanwhile let me post a workaround. Its so silly to be forced to implement something like that. but it works. So the following code block gets executed in case the system call fails. It allocates new command strings and retries until it succeeds (well not indefinitely).
list listPtr; int maxtry=1000; do < char* tmp = new(nothrow) char[gen.size()+1]; if (!tmp) __error_exit("no memory for jitter command"); strcpy(tmp,gen.c_str()); listPtr.push_back( tmp ); >while ((ret=system(listPtr.back())) && (--maxtry>0)); while(listPtr.size()) I just saw that this workaround in one particular run did not work. It went the whole way, 1000 attempts, all with newly allocated cmd command strings. All 1000 failed. Not only this. I tried on a different Linux host (same Linux/software configuration tho).
Taking this into account one would maybe exclude a hardware problem. (Must be on 2 physically different hosts then). Remains a kernel bug ??
torek, i will try and install a modified system call. Give me some time for that.
Exit status 127 linux
Таблица C-1. "Зарезервированные" коды завершения
| Код завершения | Смысл | Пример | Примечание |
|---|---|---|---|
| 1 | разнообразные ошибки | let "var1 = 1/0" | различные ошибки, такие как "деление на ноль" и пр. |
| 2 | согласно документации к Bash -- неверное использование встроенных команд | Встречаются довольно редко, обычно код завершения возвращается равным 1 | |
| 126 | вызываемая команда не может быть выполнена | возникает из-за проблем с правами доступа или когда вызван на исполнение неисполняемый файл | |
| 127 | "команда не найдена" | Проблема связана либо с переменной окружения $PATH, либо с неверным написанием имени команды | |
| 128 | неверный аргумент команды exit | exit 3.14159 | команда exit может принимать только целочисленные значения, в диапазоне 0 - 255 |
| 128+n | фатальная ошибка по сигналу "n" | kill -9 $PPID сценария | $? вернет 137 (128 + 9) |
| 130 | завершение по Control-C | Control-C -- это выход по сигналу 2, (130 = 128 + 2, см. выше) | |
| 255* | код завершения вне допустимого диапазона | exit -1 | exit может принимать только целочисленные значения, в диапазоне 0 - 255 |
Согласно этой таблице, коды завершения 1 - 2, 126 - 165 и 255 [1] имеют предопределенное значение, поэтому вам следует избегать употребления этих кодов для своих нужд. Завершение сценария с кодом возврата exit 127, может привести в замешательство при поиске ошибок в сценарии (действительно ли он означает ошибку "команда не найдена" ? Или это предусмотренный программистом код завершения?). В большинстве случаев, программисты вставляют exit 1, в качестве реакции на ошибку. Так как код завершения 1 подразумевает целый "букет" ошибок, то в данном случае трудно говорить о какой либо двусмысленности, хотя и об информативности -- тоже.
Не раз предпринимались попытки систематизировать коды завершения (см. /usr/include/sysexits.h), но эта систематизация предназначена для программистов, пишущих на языках C и C++. Автор документа предлагает ограничить коды завершения, определяемые пользователем, диапазоном 64 - 113 (и, само собой разумеется -- 0, для обозначения успешного завершения), в соответствии со стандартом C/C++. Это сделало бы поиск ошибок более простым.
Обращение к переменной $?, из командной строки, после завершения работы сценария, дает результат, в соответствии с таблицей, приведенной выше, но только для Bash или sh . Под управлением csh или tcsh значения могут в некоторых случаях отличаться.
Примечания
Указание кода завершения за пределами установленного диапазона, приводит к возврату ошибочных кодов. Например, exit 3809 вернет код завершения, равный 225 .
Can connect via SSH - but not via SFTP? Exit status 127
# Package generated configuration file # See the sshd_config(5) manpage for details # What ports, IPs and protocols we listen for Port 22 # Use these options to restrict which interfaces/protocols sshd will bind to #ListenAddress :: #ListenAddress 0.0.0.0 Protocol 2 # HostKeys for protocol version 2 HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_dsa_key #Privilege Separation is turned on for security UsePrivilegeSeparation yes # Lifetime and size of ephemeral version 1 server key KeyRegenerationInterval 3600 ServerKeyBits 768 # Logging SyslogFacility AUTH LogLevel INFO # Authentication: LoginGraceTime 120 PermitRootLogin no StrictModes yes RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile %h/.ssh/authorized_keys # Don't read the user's ~/.rhosts and ~/.shosts files IgnoreRhosts yes # For this to work you will also need host keys in /etc/ssh_known_hosts RhostsRSAAuthentication no # similar for protocol version 2 HostbasedAuthentication no # Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication #IgnoreUserKnownHosts yes # To enable empty passwords, change to yes (NOT RECOMMENDED) PermitEmptyPasswords no # Change to yes to enable challenge-response passwords (beware issues with # some PAM modules and threads) ChallengeResponseAuthentication no # Change to no to disable tunnelled clear text passwords PasswordAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosGetAFSToken no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes # GSSAPI options GSSAPIAuthentication no #GSSAPICleanupCredentials yes X11Forwarding no X11DisplayOffset 10 PrintMotd yes PrintLastLog yes TCPKeepAlive yes #UseLogin no UseDNS yes #MaxStartups 10:30:60 Banner no # Allow setting environment variables AcceptEnv LANG LC_* #Subsystem sftp /usr/lib/openssh/sftp-server Subsystem sftp /usr/lib/openssh/sftp-server UsePAM yes AllowUsers user@xx.xxx.xx.xxx user@xx.xxx.xx.xxx apt-dater@xx.xxx.xx.xxx Match User user PasswordAuthentication yes me:~ user$ sftp -v user@example.org OpenSSH_6.9p1, LibreSSL 2.1.8 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to example.org [xx.xxx.xx.xx] port 22. debug1: Connection established. debug1: identity file /Users/user/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa-cert type -1 debug1: identity file /Users/user/.ssh/id_dsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.9 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000 debug1: Authenticating to example.org:22 as 'user' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client chacha20-poly1305@openssh.com none debug1: kex: client->server chacha20-poly1305@openssh.com none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ssh-rsa SHA256:/RUeBQCA1w0nHUWT2OgLDhy7/bNQy4nWH2IVZatMLw4 debug1: Host 'example.org' is known and matches the RSA host key. debug1: Found key in /Users/user/.ssh/known_hosts:4 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/user/.ssh/id_rsa debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: Authentication succeeded (publickey). Authenticated to example.org ([xx.xxx.xx.xx]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env LANG = de_DE.UTF-8 debug1: Sending subsystem: sftp debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0 debug1: channel 0: free: client-session, nchannels 1 debug1: fd 0 clearing O_NONBLOCK Transferred: sent 3332, received 2796 bytes, in 0.2 seconds Bytes per second: sent 18890.9, received 15852.0 debug1: Exit status 127 Connection closed