irulan.net

How to Extract Bot IPs From Nginx Access Log

Recently, I’ve noticed an aggressive uptick in bots crawling websites and applications I host. To update the firewall with undesirable IPs, you can run the following command:

sudo cat /var/log/nginx/access.log* | \
    grep MauiBot | \
    awk '{print $1}' | \
    sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | \
    uniq | \
    sudo tee -a badhosts.txt

Afterwards, merge badhosts.txt with your firewall settings.