Patrick Hoerter writes in:
I have a large firewall configuration file that I am working with. It comes from that vendor that likes to prepend each product they sell with the same "well defended" name. Each configuration item inside it is multiple lines starting with "edit" and ending with "next". I'm trying to extract only the configuration items that are in some way tied to a specific port, in this case "port10".
Sample Data:
edit "port10" set vdom "root" set ip 192.168.1.54 255.255.255.248 set allowaccess ping set type physical set sample-rate 400 set description "Other Firewall" set alias "fw-outside" set sflow-sampler enable next edit "192.168.0.0" set subnet 192.168.0.0 255.255.0.0 next edit "10.0.0.0" set subnet 10.0.0.0 255.0.0.0 next edit "172.16.0.0" set subnet 172.16.0.0 255.240.0.0 next edit "vpn-CandC-1" set associated-interface "port10" set subnet 10.254.153.0 255.255.255.0 next edit "vpn-CandC-2" set associated-interface "port10" set subnet 10.254.154.0 255.255.255.0 next edit "vpn-CandC-3" set associated-interface "port10" set subnet 10.254.155.0 255.255.255.0 next edit 92 set srcintf "port10" set dstintf "port1" set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3" set dstaddr "all" set action accept set schedule "always" set service "ANY" set logtraffic enable next
Sample Results:
edit "port10" set vdom "root" set ip 192.168.1.54 255.255.255.248 set allowaccess ping set type physical set sample-rate 400 set description "Other Firewall" set alias "fw-outside" set sflow-sampler enable next edit "vpn-CandC-1" set associated-interface "port10" set subnet 10.254.153.0 255.255.255.0 next edit "vpn-CandC-2" set associated-interface "port10" set subnet 10.254.154.0 255.255.255.0 next edit "vpn-CandC-3" set associated-interface "port10" set subnet 10.254.155.0 255.255.255.0 next edit 92 set srcintf "port10" set dstintf "port1" set srcaddr "vpn-CandC-1" "vpn-CandC-2" "vpn-CandC-3" set dstaddr "all" set action accept set schedule "always" set service "ANY" set logtraffic enable next
Patrick gave us the full text and the expected output. In short, he wants the text between "edit" and "next" if it contains the text "port10". To begin this task we need to first need get each of the edit/next chunks.
PS C:\> ((cat fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | select -ExpandProperty matches
This command will read the entire file fw.txt and combine it into one string. Normally, each line is treated as a separate object, but we are going to join them into a big string using the newline (`n) to join each line. Now that the text is one big string we can use Select-String with a regular expression to find all the matches. The regular expression will find text across line breaks and allows for very flexible searches so we can find our edit/next chunks. Here is a break down of the pieces of the regular expression:
Now that we have the chunks we use a Where-Object filter (alias ?) to find matching objects to pass down the pipeline.
PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | select -ExpandProperty matches | ? { $_.Captures | Select-String "port10" }
Inside the Where-Object filter we can check the Value property to see if it contains the text "port10". The Value property is piped into Select-String to look for the text "port10", and if it contains "port10" it continues down the pipeline, if not, it is dropped.
At this point, we have the objects we want, so all we need to do is display the results by expanding the Value and displaying it again. The expansion means that it just displays the text and no data or metadata associated with the parent object. Here is what the final command looks like.
PS C:\> ((cat .\fw.txt) -join "`n") | select-string "(?s)edit.*?next" -AllMatches | select -ExpandProperty matches | ? { $_.Value | Select-String "port10" } | select -ExpandProperty Value
Not so bad, but I have a feeling it is going to be worse for my friend Hal.
Old Hal uses some old tricksOh sure, I know what Tim's thinking here. "It's multi-line matching, and the Unix shell is lousy at that. Hal's in trouble now. Mwhahaha. The Command-Line Kung Fu title will finally be mine! Mine! Do you hear me?!? MINE!"
Uh-huh. Well how about this, old friend:
awk -v RS=next -v ORS=next '/port10/' fw.txt
While we're doing multi-line matching here, the blocks of text have nice regular delimiters. That means I can change the awk "record separator" ("RS") from newline to the string "next" and gobble up entire chunks at a time.
After that, it's smooth sailing. I just use awk's pattern-matching operator to match the "port10" strings. Since I don't have an action defined, "{print}" is assumed and we output the matching blocks of text.
The only tricky part is that I have to remember to change the "output record separator" ("ORS") to be "next". Otherwise, awk will use its default ORS value, which is newline. That would give me output like:
$ awk -v RS=next '/port10/' fw.txt edit "port10" set vdom "root" set ip 192.168.1.54 255.255.255.248 set allowaccess ping set type physical set sample-rate 400 set description "Other Firewall" set alias "fw-outside" set sflow-sampler enable edit "vpn-CandC-1" set associated-interface "port10" set subnet 10.254.153.0 255.255.255.0 edit "vpn-CandC-2" set associated-interface "port10" ...
The "next" terminators get left out and we get extra lines in the output. But when ORS is set properly, we get exactly what we were after:
$ awk -v RS=next -v ORS=next '/port10/' fw.txt edit "port10" set vdom "root" set ip 192.168.1.54 255.255.255.248 set allowaccess ping set type physical set sample-rate 400 set description "Other Firewall" set alias "fw-outside" set sflow-sampler enable next edit "vpn-CandC-1" set associated-interface "port10" set subnet 10.254.153.0 255.255.255.0 next edit "vpn-CandC-2" set associated-interface "port10" ...
So that wasn't bad at all. Sorry about that Tim. Maybe next time, old buddy.
Click to Open Code Editor