Suspicious activity in logs

These are just my personal notes on the log entries I have investigated as part of my ongoing IT security project.

54.211.196.160 - - [22/Apr/2018:02:54:46 -0400] "GET /robots.txt HTTP/1.1" 200 24 "-" ""
54.211.196.160 - - [22/Apr/2018:18:58:12 -0400] "GET /ads.txt HTTP/1.1" 200 58 "-" ""
54.211.196.160 - - [22/Apr/2018:18:58:12 -0400] "GET /robots.txt HTTP/1.1" 200 24 "-" ""
  • Browser: No user agent given
  • Location: IP resolves to AmazonAMS

Only retrieved crawler configuration files. Seems harmless enough.

123.125.67.221 - - [22/Apr/2018:15:34:21 -0400] "GET /robots.txt HTTP/1.1" 200 24 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
  • Browser: Chrome browser running on Windows 10. Browser outdated (Sept 2015)
  • Location: China Unicom Beijing province network

Looks like a manual retrieve of just robots – possibly seeking further information server configuration with an inconspicuous page request.

159.203.127.111 - - [31/Mar/2018:01:48:27 -0400] "GET /2017/12 HTTP/1.1" 301 - "-" "Go-http-client/1.1"
159.203.127.111 - - [31/Mar/2018:01:48:28 -0400] "GET /2017/12/ HTTP/1.1" 200 43764 "http://example.com/2017/12" "Go-http-client/1.1"
159.203.127.111 - - [19/Apr/2018:12:38:47 -0400] "GET /category/attachment HTTP/1.1" 301 - "-" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [19/Apr/2018:12:38:48 -0400] "GET /category/attachment/ HTTP/1.1" 200 46172 "http://example.com/category/attachment" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [21/Apr/2018:18:27:25 -0400] "GET /2017/04 HTTP/1.1" 301 - "http://example.com/2017/04" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [21/Apr/2018:18:27:27 -0400] "GET /2017/04/ HTTP/1.1" 200 44445 "https://example.com/2017/04" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [24/Apr/2018:15:45:30 -0400] "GET /robots.txt HTTP/1.1" 200 24 "http://example.com/robots.txt" "Go-http-client/2.0"
159.203.127.111 - - [24/Apr/2018:15:45:35 -0400] "GET /category/mof HTTP/1.1" 301 - "http://example.com/category/mof" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [24/Apr/2018:15:45:35 -0400] "GET /robots.txt HTTP/1.1" 200 24 "http://example.com/robots.txt" "Go-http-client/2.0"
159.203.127.111 - - [24/Apr/2018:15:45:36 -0400] "GET /category/mof/ HTTP/1.1" 200 40418 "https://example.com/category/mof" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [25/Apr/2018:20:39:39 -0400] "GET /2017/07/11/sa HTTP/1.1" 301 - "http://example.com/2017/07/11/sa" "VelenPublicWebCrawler (velen.io)"
159.203.127.111 - - [25/Apr/2018:20:39:39 -0400] "GET /robots.txt HTTP/1.1" 200 24 "http://example.com/robots.txt" "Go-http-client/2.0"
159.203.127.111 - - [25/Apr/2018:20:39:39 -0400] "GET /robots.txt HTTP/1.1" 200 24 "http://example.com/robots.txt" "Go-http-client/2.0"
159.203.127.111 - - [25/Apr/2018:20:39:40 -0400] "GET /2017/07/11/sa/ HTTP/1.1" 200 29649 "https://example.com/2017/07/11/sa" "VelenPublicWebCrawler (velen.io)"
  • Browser:  Go-http-client/2.0  & VelenPublicWebCrawler (velen.io)
  • IP: DigitalOcean

Go is a programming language and the user agent “Go-http-client” indicates someone using scripting to scrape undetermined web content. Other requests from the same IP give a botname VelenPublicWebCrawler and an explanation of their presence is available on their website – research and development. I do not want to donate my bandwidth so they will be added to robot.txt.