Question

Driving myself nuts. I am trying to get just the domain name (http://www.example.com) out of access.log. What the log looks like:

tail access.log 

Fri, 13 Jul 2012 20:32:03 -0700,INFO,6fgmd8fk,params,http://www.example.com/images/CIV-260.jpg|

I have tried many variations of this one-liner (with sed and awk):

tail -4 access.log |grep http |awk {'print $6'} |cut -c28- |awk '$1>".com"' |sort |uniq

http://www.example.com/2713-7807.jpg|
http://www.example.com/2713-7808.jpg|
http://barfoo.com/img/14616_20120711182527.jpg|
http://foobar.com/css/14616_20120713142151.css|

I am stuck.

Answer 1

Using grep:

grep -Po '(?<=http://)[^/]+' access.log | sort -u

If you want to have http:// as a part of domain name,

grep -Po 'http://[^/]+' access.log | sort -u