I recently created a python script to analyze Nginx’s access log & a WAF generated error log. The preliminary result looks promising. The script can filter out most malicious & bot traffic with very low false positive on human traffic. I ran the IPs collected by this script against multiple blacklists:

More than 88% of the IPs picked up by the script are already in one of the blacklists, and most of IPs in the remaining 12% are likely to be malicious / bot as well.

I am working on a more in-depth behavior analysis for these two logs in order to detect more advanced bot traffic.