very fast ip and tcp layer decoding

[gfr10598]

This PR adds a small library that decodes ip and tcp layer by creating overlays on the underlying byte data, using the unsafe library.

It uses the gopacket library for supporting types and constants.

This level of decoding incurs significant memory allocation, but performance is superior to gopacket library, so far. When hooked into the pcap parser, the throughput per 15 cpu instance drops from roughly 1.07 GB/sec to roughly 660 MB/sec.

---

This change is 

[gfr10598]

File: etl_worker
Build ID: 77564c5e62367b71c386fbcb33cda445495a2195
Type: cpu
Time: Nov 25, 2021 at 10:20am (EST)
Duration: 30.21s, Total samples = 5.39mins (1069.73%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top50 -cum
Showing nodes accounting for 221.13s, 68.42% of 323.20s total
Dropped 530 nodes (cum <= 1.62s)
Showing top 50 nodes out of 126
      flat  flat%   sum%        cum   cum%
         0     0%     0%    292.03s 90.36%  github.com/m-lab/etl/active.(*GardenerAPI).RunAll.func1
         0     0%     0%    292.03s 90.36%  github.com/m-lab/etl/active.(*throttledRunnable).Run
         0     0%     0%    292.03s 90.36%  github.com/m-lab/etl/worker.ProcessGKETask
         0     0%     0%    292.03s 90.36%  golang.org/x/sync/errgroup.(*Group).Go.func1
         0     0%     0%    292.03s 90.36%  main.(*runnable).Run
         0     0%     0%    292.01s 90.35%  github.com/m-lab/etl/task.(*Task).ProcessAllTests
         0     0%     0%    292.01s 90.35%  github.com/m-lab/etl/worker.DoGKETask
         0     0%     0%    191.24s 59.17%  github.com/m-lab/etl/storage.(*GCSSource).NextTest
     0.02s 0.0062% 0.0062%    191.12s 59.13%  bytes.(*Buffer).ReadFrom
         0     0% 0.0062%    190.54s 58.95%  github.com/m-lab/etl/storage.(*GCSSource).nextData
         0     0% 0.0062%    190.45s 58.93%  io/ioutil.ReadAll (inline)
         0     0% 0.0062%    190.45s 58.93%  io/ioutil.readAll
     0.06s 0.019% 0.025%    177.24s 54.84%  compress/gzip.(*Reader).Read
     0.10s 0.031% 0.056%    175.79s 54.39%  compress/flate.(*decompressor).Read
    31.30s  9.68%  9.74%    169.66s 52.49%  compress/flate.(*decompressor).huffmanBlock
    63.52s 19.65% 29.39%    113.84s 35.22%  compress/flate.(*decompressor).huffSym
     0.02s 0.0062% 29.40%    100.73s 31.17%  github.com/m-lab/etl/parser.(*PCAPParser).ParseAndInsert
     3.15s  0.97% 30.37%    100.60s 31.13%  github.com/m-lab/etl/tcpip.GetPackets
    34.07s 10.54% 40.92%     80.09s 24.78%  bufio.(*Reader).ReadByte
     0.05s 0.015% 40.93%     59.76s 18.49%  bufio.(*Reader).fill
     0.01s 0.0031% 40.93%     59.68s 18.47%  archive/tar.(*Reader).Read
     0.04s 0.012% 40.95%     59.67s 18.46%  archive/tar.(*regFileReader).Read
     0.05s 0.015% 40.96%     59.14s 18.30%  runtime.systemstack
     8.41s  2.60% 43.56%     58.46s 18.09%  runtime.mallocgc
     0.02s 0.0062% 43.57%     58.16s 18.00%  compress/flate.(*decompressor).nextBlock
     2.84s  0.88% 44.45%     40.14s 12.42%  github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
     1.05s  0.32% 44.77%     38.70s 11.97%  runtime.makeslice
    15.62s  4.83% 49.61%     30.90s  9.56%  runtime.scanobject
     4.16s  1.29% 50.89%     30.32s  9.38%  github.com/m-lab/etl/tcpip.Wrap
         0     0% 50.89%     25.47s  7.88%  runtime.gcBgMarkWorker
         0     0% 50.89%     25.11s  7.77%  runtime.gcBgMarkWorker.func2
     0.55s  0.17% 51.06%     25.11s  7.77%  runtime.gcDrain
    20.11s  6.22% 57.29%     20.11s  6.22%  runtime.memmove
     0.58s  0.18% 57.47%     19.53s  6.04%  runtime.newobject
     0.61s  0.19% 57.65%     17.42s  5.39%  github.com/m-lab/etl/tcpip.WrapTCP
     2.10s  0.65% 58.30%     16.28s  5.04%  io.ReadFull (partial-inline)
     2.95s  0.91% 59.22%     14.18s  4.39%  io.ReadAtLeast
         0     0% 59.22%     13.87s  4.29%  bytes.(*Buffer).grow
     3.35s  1.04% 60.25%     13.77s  4.26%  compress/flate.(*decompressor).moreBits
     0.08s 0.025% 60.28%     13.28s  4.11%  runtime.(*mcache).nextFree
     0.01s 0.0031% 60.28%     12.96s  4.01%  runtime.(*mcache).refill
     6.68s  2.07% 62.35%     12.87s  3.98%  compress/flate.(*dictDecoder).tryWriteCopy
    10.12s  3.13% 65.48%     12.87s  3.98%  runtime.findObject
     0.69s  0.21% 65.69%     12.38s  3.83%  runtime.(*mcentral).cacheSpan
     5.83s  1.80% 67.50%     12.10s  3.74%  github.com/google/gopacket/pcapgo.(*Reader).readPacketHeader
     0.46s  0.14% 67.64%     11.40s  3.53%  runtime.(*mspan).sweep
     1.87s  0.58% 68.22%     11.36s  3.51%  runtime.bulkBarrierPreWrite
         0     0% 68.22%     11.04s  3.42%  runtime.(*mheap).alloc
     0.65s   0.2% 68.42%     10.93s  3.38%  runtime.typedmemmove
         0     0% 68.42%     10.75s  3.33%  bytes.makeSlice

[coveralls]

Pull Request Test Coverage Report for Build 6906

• 79 of 119 (66.39%) changed or added relevant lines in 4 files are covered.
• 17 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.3%) to 63.503%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
parser/pcap.go	1	2	50.0%
cmd/parse/parse.go	72	111	64.86%

Files with Coverage Reduction	New Missed Lines	%
parser/pcap.go	17	61.29%

Totals
Change from base Build 6882:	-0.3%
Covered Lines:	3847
Relevant Lines:	6058

---

💛 - Coveralls

[gfr10598]

With just indexing, GC is around 3.5%, and around 3-4 Mb/sec
With this fast packet decoding, GC is around 8.5% and 12 Mb/sec.

[gfr10598]

Between r1 and r2, discovered that code was making multiple copies of CaptureInfo, and was doing the tcpip.Wrap twice. After fixing these two inefficiencies, performance improved about 13%. Time spent decoding gzip grows from about 55% to 62%.
Time spent with packet interpretation, including gopacket code (and associated decompression of packet bytes), drops from 31% to 26.5%.

File: etl_worker
Build ID: 7fee663445098c45ed9223cb3a5d87cab6f2fb97
Type: cpu
Time: Nov 25, 2021 at 7:10pm (EST)
Duration: 30.29s, Total samples = 291.69s (962.99%)
Showing nodes accounting for 271.19s, 92.97% of 291.69s total
Dropped 511 nodes (cum <= 1.46s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%    268.63s 92.09%  github.com/m-lab/etl/active.(*GardenerAPI).RunAll.func1
         0     0%     0%    268.63s 92.09%  golang.org/x/sync/errgroup.(*Group).Go.func1
         0     0%     0%    268.62s 92.09%  github.com/m-lab/etl/active.(*throttledRunnable).Run
         0     0%     0%    268.62s 92.09%  github.com/m-lab/etl/worker.ProcessGKETask
         0     0%     0%    268.62s 92.09%  main.(*runnable).Run
         0     0%     0%    268.59s 92.08%  github.com/m-lab/etl/task.(*Task).ProcessAllTests
         0     0%     0%    268.59s 92.08%  github.com/m-lab/etl/worker.DoGKETask
     0.01s 0.0034% 0.0034%    190.89s 65.44%  github.com/m-lab/etl/storage.(*GCSSource).NextTest
     0.02s 0.0069%  0.01%    190.18s 65.20%  bytes.(*Buffer).ReadFrom
         0     0%  0.01%    189.71s 65.04%  github.com/m-lab/etl/storage.(*GCSSource).nextData
     0.02s 0.0069% 0.017%    189.59s 65.00%  io/ioutil.ReadAll (inline)
         0     0% 0.017%    189.57s 64.99%  io/ioutil.readAll
     0.07s 0.024% 0.041%    180.34s 61.83%  compress/gzip.(*Reader).Read
     0.07s 0.024% 0.065%    178.78s 61.29%  compress/flate.(*decompressor).Read
    30.91s 10.60% 10.66%    173.21s 59.38%  compress/flate.(*decompressor).huffmanBlock
    67.28s 23.07% 33.73%    117.43s 40.26%  compress/flate.(*decompressor).huffSym
    33.06s 11.33% 45.06%     81.49s 27.94%  bufio.(*Reader).ReadByte
     0.01s 0.0034% 45.06%     77.66s 26.62%  github.com/m-lab/etl/parser.(*PCAPParser).ParseAndInsert
     2.21s  0.76% 45.82%     77.51s 26.57%  github.com/m-lab/etl/tcpip.GetPackets
     0.02s 0.0069% 45.83%     62.31s 21.36%  compress/flate.(*decompressor).nextBlock
     0.11s 0.038% 45.87%     61.67s 21.14%  bufio.(*Reader).fill
     0.02s 0.0069% 45.87%     61.52s 21.09%  archive/tar.(*Reader).Read
     0.04s 0.014% 45.89%     61.49s 21.08%  archive/tar.(*regFileReader).Read
     6.22s  2.13% 48.02%     48.02s 16.46%  runtime.mallocgc
     0.17s 0.058% 48.08%     47.88s 16.41%  runtime.systemstack
     0.65s  0.22% 48.30%     32.88s 11.27%  runtime.makeslice
     2.39s  0.82% 49.12%     30.75s 10.54%  github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
     2.71s  0.93% 50.05%     21.95s  7.53%  github.com/m-lab/etl/tcpip.Wrap
     9.28s  3.18% 53.23%     20.34s  6.97%  runtime.scanobject
         0     0% 53.23%     16.53s  5.67%  runtime.gcBgMarkWorker
         0     0% 53.23%     16.36s  5.61%  runtime.gcBgMarkWorker.func2
     0.37s  0.13% 53.36%     16.36s  5.61%  runtime.gcDrain
    15.20s  5.21% 58.57%     15.20s  5.21%  runtime.memmove
     0.42s  0.14% 58.71%     14.96s  5.13%  runtime.newobject
     0.38s  0.13% 58.84%     13.80s  4.73%  github.com/m-lab/etl/tcpip.WrapTCP
     3.17s  1.09% 59.93%     13.68s  4.69%  compress/flate.(*decompressor).moreBits
     7.23s  2.48% 62.41%     13.28s  4.55%  compress/flate.(*dictDecoder).tryWriteCopy
     0.07s 0.024% 62.43%     12.72s  4.36%  runtime.(*mcache).nextFree
     0.03s  0.01% 62.44%     12.54s  4.30%  runtime.(*mcache).refill
     0.56s  0.19% 62.63%     12.22s  4.19%  runtime.(*mcentral).cacheSpan
     1.35s  0.46% 63.10%     11.71s  4.01%  io.ReadFull (partial-inline)
     0.34s  0.12% 63.21%     11.53s  3.95%  runtime.(*mspan).sweep
     1.56s  0.53% 63.75%     11.22s  3.85%  runtime.bulkBarrierPreWrite
     0.66s  0.23% 63.98%     11.02s  3.78%  runtime.typedmemmove
    10.99s  3.77% 67.74%     10.99s  3.77%  compress/flate.(*dictDecoder).writeByte
     8.48s  2.91% 70.65%     10.81s  3.71%  runtime.findObject
     0.08s 0.027% 70.68%     10.57s  3.62%  runtime.deductSweepCredit
        1s  0.34% 71.02%     10.55s  3.62%  runtime.sweepone
     2.21s  0.76% 71.78%     10.36s  3.55%  io.ReadAtLeast
     0.01s 0.0034% 71.78%     10.18s  3.49%  bytes.(*Buffer).grow
     0.02s 0.0069% 71.79%     10.06s  3.45%  runtime.(*mheap).freeSpan
     0.23s 0.079% 71.87%     10.04s  3.44%  runtime.(*mheap).freeSpan.func1
     0.01s 0.0034% 71.87%      9.76s  3.35%  runtime.(*mheap).alloc
         0     0% 71.87%      9.39s  3.22%  runtime.largeAlloc
         0     0% 71.87%      9.39s  3.22%  runtime.mallocgc.func1
         0     0% 71.87%      9.01s  3.09%  bytes.makeSlice

Reviewable status: 0 of 1 approvals obtained

[stephen-soltesz]

Closing old PR - and referencing the notes from this change in #1085