IP control loop

[maiqueb]

Introduce an IP reconciliation loop for whereabouts.

The controller monitor pod deletion events, and for every deleted pod it:

• reads the pod's network status annotations
• read the network attachment for every network referenced in the status
• extract the associated IP range of the attachment
• read the corresponding IP pool
• delete all existing attachments on that pool for the pod being deleted

The controller is unit tested, and e2e tests are introduced to check for stale IP addresses for both pods and stateful sets.

It is updating the allocations in the datastore via the whereabouts API, which uses leader election to allow concurrent updates to the IP pools.

It only reads the Pods / IPPools / Net-Attach-Defs from the informer cache, in order to minimize the load of the Kubernetes API.

Depends-on: #202
Fixes: #182

[maiqueb]

Rebased latest master & fixed vendoring in this forced push.

[coveralls]

Pull Request Test Coverage Report for Build 2155322022

• 424 of 511 (82.97%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+13.4%) to 53.88%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/reconciler/controlloop/entity_generators.go	91	93	97.85%
pkg/api/whereabouts.cni.cncf.io/v1alpha1/register.go	16	22	72.73%
pkg/config/config.go	31	51	60.78%
pkg/reconciler/controlloop/dummy_controller.go	79	102	77.45%
pkg/reconciler/controlloop/pod.go	207	243	85.19%

Totals
Change from base Build 2070941103:	13.4%
Covered Lines:	861
Relevant Lines:	1598

---

💛 - Coveralls

[crandles]

controller-runtime could be leveraged for this, as far as tooling for creating a controller with informers

[maiqueb]

controller-runtime could be leveraged for this, as far as tooling for creating a controller with informers

I've started a discussion item in #205 for this. I would definitely appreciate if you voiced your opinion there @crandles :)

[dougbtv]

fwiw, I went to go build it and run it locally and stopped when I ran into this index-out-of-bounds error when running ./hack/test-go.sh, https://pasteall.org/BkLq#L17,79 (don't feel like you have to debug it, but, I slowed down when I got here)

nevermind (it was an environmental problem)

[dougbtv]

Hey Miguel -- I attempted to use this interactively and overall -- I had great luck, and I'm really pleased.

One question I have is... I (by accident) had a pod running on my system which was not using whereabouts, but... the control loop appears to be picking it up?

E.g. Here's a snippet from the logs...

2022-04-01T18:51:40Z [verbose] deleted pod [default/samplepod]
2022-04-01T18:51:40Z [verbose] skipped net-attach-def for default network
2022-04-01T18:51:40Z [debug] pod's network status: {Name:default/macvlan-conf Interface:net1 IPs:[192.168.1.200] Mac:02:dc:79:11:4f:a6 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2022-04-01T18:51:40Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart": "192.168.1.200", "rangeEnd": "192.168.1.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.1.1" } }}
2022-04-01T18:51:40Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2022-04-01T18:51:40Z [verbose] result of garbage collecting pods: failed to create an IPAM configuration for the pod default/samplepod iface default/macvlan-conf: invalid CIDR : invalid CIDR address: 
2022-04-01T18:51:40Z [verbose] re-queuing IP address reconciliation request for pod default/samplepod; retry #: 0

Are we filtering down to the right pods with the controller? Or did I do something... weird (very well could be)

[maiqueb]

Hey Miguel -- I attempted to use this interactively and overall -- I had great luck, and I'm really pleased.

One question I have is... I (by accident) had a pod running on my system which was not using whereabouts, but... the control loop appears to be picking it up?

E.g. Here's a snippet from the logs...

2022-04-01T18:51:40Z [verbose] deleted pod [default/samplepod]
2022-04-01T18:51:40Z [verbose] skipped net-attach-def for default network
2022-04-01T18:51:40Z [debug] pod's network status: {Name:default/macvlan-conf Interface:net1 IPs:[192.168.1.200] Mac:02:dc:79:11:4f:a6 Default:false DNS:{Nameservers:[] Domain: Search:[] Options:[]} DeviceInfo:<nil>}
2022-04-01T18:51:40Z [verbose] the NAD's config: {{ "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart": "192.168.1.200", "rangeEnd": "192.168.1.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.1.1" } }}
2022-04-01T18:51:40Z [debug] Used defaults from parsed flat file config @ /host/etc/cni/net.d/whereabouts.d/whereabouts.conf
2022-04-01T18:51:40Z [verbose] result of garbage collecting pods: failed to create an IPAM configuration for the pod default/samplepod iface default/macvlan-conf: invalid CIDR : invalid CIDR address: 
2022-04-01T18:51:40Z [verbose] re-queuing IP address reconciliation request for pod default/samplepod; retry #: 0

Are we filtering down to the right pods with the controller? Or did I do something... weird (very well could be)

I'm honestly unsure on how ( if ?...) that can be done. Is there something on the pod that indicates it as a whereabouts pod ? Can I filter it somehow ? AFAIU, the best I can do is check if the pod's network IPAM configuration is of type whereabouts .
But doing that requires accessing the informer.

Do you have any suggestion on how to implement your proposal ? I agree it would be better to only send to the queue stuff we knew were whereabouts related.

[dougbtv]

re:

but... the control loop appears to be picking it up?

...I think this was user error on my part after all 👍

[dougbtv]

LGTM! Thank you Miguel!