weblogs
Web Access Logs (IIS)
Traffic generated
sourcetype="whatever" website="www.xxx.com" | timechart span=15m sum(bytes_in), sum(bytes_out)
List of page visits of www.xxx.com, including least popular user-agent and HTTP methods used
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | timechart span=1h count
List the least popular User-Agents
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | search user_agent!=*bot/* user_agent!=msnbot* user_agent!=Dalvik/* user_agent!=*spider* user_agent!=Feedfetcher-Google* | iplocation prefix=src_ip_ src_ip | transaction clientip,user_agent maxspan=30s maxpause=5s | rare limit=50 user_agent
List the used HTTP Methods
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | search user_agent!=*bot/* user_agent!=msnbot* | iplocation prefix=src_ip_ src_ip | transaction clientip,user_agent maxspan=30s maxpause=5s | top limit=50 http_method
List the friendly bots only
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | search user_agent!=*bot/* user_agent!=msnbot* user_agent!=*crawler* user_agent=*spider* user_agent!=adsbot* | iplocation prefix=src_ip_ src_ip | transaction clientip,user_agent maxspan=30s maxpause=5s | top limit=50 user_agent
List all unusual bots unlegit User-Agents
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | search user_agent!=Mozilla/* user_agent!=Opera/* user_agent!=*CFNetwork* user_agent!=*bot/* user_agent!=msnbot* user_agent!=*crawler* user_agent!=*spider* | iplocation prefix=src_ip_ src_ip | transaction clientip,user_agent maxspan=30s maxpause=5s | top limit=50 user_agent</query>
Traffic profile by 24h with 15 min buckets
sourcetype="whatever" website="www.xxx.com" | timechart span=15m sum(bytes_in), sum(bytes_out)
Events over last 24h with 1h buckets
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | timechart span=1h count
List the page visits of www.xxx.com, including least popular user-agents and HTTP methods used
sourcetype="whatever" website="www.xxx.com" | rex field=_raw "^(?<date>.+?) (?<time>.+?) (?<www_server_name>.+?) (?<www_server_sitename>.+?) (?<www_server_ip>.+?) (?<http_method>.+?) (?<uri>.+?) (?<uri_query>.+?) (?<dst_port>.+?) (?<user_name>.+?) (?<src_ip>.+?) HTTP\/(?<http_version>.+?) (?<user_agent>.+?) (?<referrer>.+?) (?<www_domain>.+?) (?<http_status>\d+?) (?<http_win32_status>\d+?) (?<bytes_sent>\d+?) (?<bytes_recd>\d+?) (?<time_taken>.+?)" | rex field=user_agent mode=sed "s/\+/ /g" | search user_agent!=*bot/* user_agent!=msnbot* user_agent!=*crawler* | iplocation prefix=src_ip_ src_ip | transaction clientip,user_agent maxspan=30s maxpause=5s | table _time, src_ip, src_ip_Country, website, http_method, uri, uri_query, user_agent