Is there an error log file for linux that shows file open failures?
I just blew a few days tracking down a bug in a python script (unknown to me, web2py used a different root directory from what i was expecting, leading to a file read to fail silently. Thus when run from command line, code was fine, but when run from web, code failed). Having finally tracked down the culprit I can just fix the silent fail (which is in a library, openCV in this case) but a smarter detective would have seen the fail in some sort of system log , if such exists. Then , no matter where the silent fail is, I still see it and don’t have to painstakingly track down the fail.
So — is there some sort of global error logfile for linux that logs such things as file-read errors? And yes I know there is python specific error logging but the question still holds. e.g. if i have a complex project with some python, some C, some whatever, and someone somewhere is silently failing, a system-wide error log would be of immense help.
When your script is being run by a Web server, error messages will normally appear in the logs that the server maintains, so you need to look at how the server is configured. I know that this doesn’t exactly answer your question (which is why it’s in a comment 🙂 ).
2 Answers 2
This solution may not be performant enough for your needs, but regardless I wanted to report on some research I did that may lead you in the right direction.
First of all, there is logging in linux systems under /var/log. Of interest are the syslog and messages files, which log all kinds of system events. But file read «errors» are not logged, as explained below.
In the case of opening a file that doesn’t exist, we are ultimately looking for an open system call that fails (python’s open calls this). But there is no notion of an exception at this low level — if open fails it just returns a negative number. In C, you can open files that don’t exist all day long and still have your program return a 0 error code.
This means you have to do some work yourself to track this problem. I took your question to be, «How can I track these errors at a level below python’s exceptions?» For this you can use a combination of strace and grep. You attach strace per process and it logs all the system calls that take place.
So imagine we have a C program that looks like this:
By running strace ./test 2>&1 | grep ENOENT, we get:
open("nothere.txt", O_RDONLY) = -1 ENOENT (No such file or directory) You could of course run strace on a python process to achieve the same results.
- You have to attach this per process. If you don’t, we’re back to silent errors.
- Python generates a lot of system calls. Your log files might get big.
- There are a lot of IO errors out there. ENOENT is only one of them.
- You will need more complex string parsing to filter out system calls you don’t care about.