Ytdyklly

I have a huge log file (20 millions of lines) telling me if some url status are responding "200 OK" or not.

I'd like to extract all url with status "200 OK", plus the filename attached with it.

Input example:

Spider mode enabled. Check if remote file exists.

--2019-02-06 07:38:43--  https://www.example/download/123456789

Reusing existing connection to website.

HTTP request sent, awaiting response... 

  HTTP/1.1 200 OK

  Content-Type: application/zip

  Connection: keep-alive

  Status: 200 OK

  Content-Disposition: attachment; filename="myfile123.zip"

  Last-Modified: 2019-02-06 01:38:44 +0100

  Access-Control-Allow-Origin: *

  Cache-Control: private

  X-Runtime: 0.312890

  X-Frame-Options: SAMEORIGIN

  Access-Control-Request-Method: GET,OPTIONS

  X-Request-Id: 99920e01-d308-40ba-9461-74405e7df4b3

  Date: Wed, 06 Feb 2019 00:38:44 GMT 

  X-Powered-By: Phusion Passenger 5.1.11

  Server: nginx + Phusion Passenger 5.1.11

  X-Powered-By: cloud66

Length: unspecified [application/zip]

Last-modified header invalid -- time-stamp ignored.

Remote file exists.



Spider mode enabled. Check if remote file exists.

--2019-02-06 07:38:43--  https://www.example/download/234567890

Reusing existing connection to website.

HTTP request sent, awaiting response... 

  HTTP/1.1 404 Not Found

  Content-Type: text/html; charset=utf-8

  Connection: keep-alive

  Status: 404 Not Found

  Cache-Control: no-cache

  Access-Control-Allow-Origin: *

  X-Runtime: 0.020718

  X-Frame-Options: SAMEORIGIN

  Access-Control-Request-Method: GET,OPTIONS

  X-Request-Id: bc20626b-095f-4b28-8322-ad3f294e4ee2

  Date: Wed, 06 Feb 2019 00:37:42 GMT

  X-Powered-By: Phusion Passenger 5.1.11

  Server: nginx + Phusion Passenger 5.1.11

Remote file does not exist -- broken link!!!

Desired Output:

https://www.example/download/123456789 myfile123.zip

I had a friend who turned expert in doing this kind of extraction with grep and awk, and I'd love to finally understand the logic behind.

If I do this:

awk '/: 200 OK/{print $0}' file.log

I get all lines with Status: 200 OK but without the context.

If I do this:

grep -C4 "1 200 OK" file.log

I get the context but with "noise". I'd like to rearrange the output to get only the relevant information on one line.

You need to use awk as below. Store the URL first in a variable and then on the Status line if its OK get the filename from the subsequent line. It should work on GNU awk as the match() function would need the third argument to store the captured group in an array.

awk '/^--/{ url = $NF } 

    /^[[:space:]]+Status/ && $NF == "OK" { getline nextline; match(nextline, /filename="(.+)"/,arr); print url, arr[1] }' file

i=`awk '/Status: 200 OK/{x=NR+1}(NR<x){getline;print $NF}' filename | awk -F "=" '{print $NF}'| sed 's/"//g'`



awk '{a[++i]=$0}/Status: 200 OK/{for(x=NR-7;x<=NR;x++)print a[x]}' filename | awk -v i="$i" '/https:/{$1=$2="";print $0 " " i}'

output

https://www.example/download/123456789 myfile123.zip