log network activity in ubuntu server
I administer an Ubunu Server exposed to the internet and have the need to monitor and keep track of all the network activity in a manner that allows me to analyze it afterwards. I already tried some tools, such as tshark or tcpdump, which give me too much detail, vnstat, which does not give me the detail I want (It shows only the bandwidth), and tcptrack, which is OK as a real time monitoring tool but gives me no logging option for further analysis. What I have in mind is something between tcptrack and vnstat: A daemon which logs every connection and, when needed, it provides me with a comprehensive report showing IPs, ports and timestamps of every established connection, and every connection attempt (so, it should also show the SYN packets of the connections dropped by iptables). Ideally (this is just a bonus point :), it would store information into some sql database, such as mysql or postgresql, which would allow to execute arbitrary select statements in order to obtain custom reports (for example, monitor all the activity coming from a single IP, or extract a list of all IPs using a specific service). I must say that I already tried combining some tools, like logging with tcpdump and then showing the results using tcptrack, but I it didn’t work as expected. So, is there any tool close to this «idea»?
Have you looked into the LOG jump point in iptables? For example iptables -I INPUT -j LOG will syslog all inbound packets. You can use the normal criteria for limiting the packets that match the logging rule (for example only new TCP connections, etc). It won’t be a report (hence a comment) but it will record this information.
I had a look at ntop and it looks as a good tool. However, few minutes after a connection is dropped it disappears from the list. Is there a way to get a historical report from it?
How to get a linux network log?
We have a java server running in linux at a specific port that accepts persistent connections for thousands and thousands of users. Recently our clients are not able to connect with a time out error. We suspect the traffic is getting too high but our java log actually shows that not many are connected in per second. We suspect that could it be that too many are trying at the same time and they are basically dropped at the OS level and therefore the java program never really gets a chance to accept the connection? Is there some sort of log in linux that can show someone trying to hit a socket?
5 Answers 5
iptables -I INPUT -p tcp —dport some_port -j LOG then
tail -f /var/log/messages
Afterwards, to see how much data has been hit by that rule: iptables -L -n -v
Or you could run tcpdump and grep out the ports.
+1. A slight modification might work better to catch just the new connection attempts: iptables -I INPUT -p tcp —dport some_port -m state —state NEW . Note that unless you’re interested in the details of each connection attempt, omitting -j LOG avoids spamming the system log file with lots of unneeded data.
Upon setting this up, prepare to see a stomach-churningly high number of supposedly non-malicious security crawlers connecting to your server, like implant-scanner-victims-will-be-notified.threatsinkhole.com , stretchoid.com , jobqueue-listener.jobqueue.netcraft.com-digitalocean , . good to know.
Linux: Looking for a network log for server connections
The only sure way to get a single centralized log of all TCP connections would be to add LOG rules to your iptables software firewall configuration that would log any packet with a SYN and ACK bits set, i.e. the second packet of any TCP connection. These would appear as kernel log messages in /var/log/messages by default.
Essentially you would want to add iptables rules like this:
iptables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK -j LOG --log-prefix "Inbound connection established: " iptables -A INPUT -p tcp --tcp-flags SYN,ACK SYN,ACK -j LOG --log-prefix "Outbound connection established: " (You may want to place these rules in your existing ruleset so that they won’t apply to the loopback interface, as that might generate two entries for each local connection between application processes within the host.)
And yes, the rule in OUTPUT chain records inbound connections and vice versa, because this will only log connections that are actually being responded to: recording just the SYN packets would also record connection attempts that will be rejected. Because of port scans executed by malware-ridden systems on the internet, that would often give you a lot of useless log entries: you would have to cross-check with other logs, only to find out that either you don’t have such a service running at all, or that the connection was rejected by another iptables rule or by the service in question.
For UDP protocols it’s trickier, because UDP is connectionless and basically just a platform on which an application-specific protocol can be built. So there is no easy way to detect «the first packet of a connection» or anything like that, because all that will be entirely application-specific.