Home/ Questions/Q 6763945
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T14:32:11+00:00

I’m setting up an ETL process for my company’s S3 buckets so we can

  • 0

I’m setting up an ETL process for my company’s S3 buckets so we can track our usage, and I’ve run into some trouble breaking up the columns of the S3 log file because Amazon uses spaces, double quotes, and square brackets to delimit columns.

I found this Regex: [^\\s\"']+|\"([^\"]*)\"|'([^']*)' on this SO post: Regex for splitting a string using space when not surrounded by single or double quotes and that’s gotten me pretty close. I just need help adjusting it to ignore single quotes and also ignore spaces between a “[” and a “]”

Here’s an example line from one of our files:

dd8d30dd085515d73b318a83f4946b26d49294a95030e4a7919de0ba6654c362 ourbucket.name.config [31/Oct/2011:17:00:04 +0000] 184.191.213.218 - 013259AC1A20DF37 REST.GET.OBJECT ourbucket.name.config.txt "GET /ourbucket.name.config.txt HTTP/1.1" 200 - 325 325 16 16 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" -

And here’s the format definition: http://s3browser.com/amazon-s3-bucket-logging-server-access-logs.php

Any help would be appreciated!

EDIT: in response to FaileDev, the output should be any string contained between two square brackets, e.g. [foo bar], two quotes, e.g. “foo bar” or spaces, e.g. foo bar (where both foo and bar would match individually. I’ve broken each match in the example line I provided into it’s own line in the following block:

dd8d30dd085515d73b318a83f4946b26d49294a95030e4a7919de0ba6654c362 
ourbucket.name.config 
[31/Oct/2011:17:00:04 +0000] 
184.191.213.218 
- 
013259AC1A20DF37 
REST.GET.OBJECT 
ourbucket.name.config.txt 
"GET /ourbucket.name.config.txt HTTP/1.1" 
200 
- 
325 
325 
16 
16 
"-" 
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" 
-
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You can’t do it using string.split, you need to iterate through all captures of the ‘column’ group (if you’re using C#)

    This matches a non-quoted, non-bracketed field: [^\s\"\[\]]+
This matches a bracketed field: \[[^\]\[]+\] 
This matches a quoted field: \"[^\"]+\"

    It’s easiest to leave the quotes and brackets on during matching, then strip them off using Trim(‘[‘,’\’,'”‘). 

    @"^((?<column>[^\s\"\[\]]+|\[[^\]\[]+\]|\"[^\"]+\")\s+)+$"
    • 0