Amazon Linux 2023.6.20241212 Client() returns KeyError: http+unix

[Dweller]

My ec2 nodes pylxd scripts don't work with the latest version of Amazon Linux 2023.6.20241212

Client()call returns KeyError: http+unix which is to day almost like the underlining module doesn't understand http+unix?.

If I go back a version of Amazon Linux then it still works with exactly the same cloud-init configuration.

Python 3.9.20 (main, Dec 11 2024, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylxd
>>> lxd=pylxd.Client()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/atcore/.local/lib/python3.9/site-packages/pylxd/client.py", line 472, in __init__
    response = self.api.get()
  File "/home/atcore/.local/lib/python3.9/site-packages/pylxd/client.py", line 268, in get
    response = self.session.get(self._api_endpoint, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 557, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 467, in send
    conn = self._get_connection(request, verify, proxies)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 338, in _get_connection
    conn = self.poolmanager.connection_from_host(
  File "/usr/lib/python3.9/site-packages/urllib3/poolmanager.py", line 240, in connection_from_host
    return self.connection_from_context(request_context)
  File "/usr/lib/python3.9/site-packages/urllib3/poolmanager.py", line 250, in connection_from_context
    pool_key_constructor = self.key_fn_by_scheme[scheme]
KeyError: 'http+unix'

However if I issue a curl request to the same port..

curl --unix-socket /var/snap/lxd/common/lxd/unix.socket http://localhost/1.0
{"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{"config":{},"api_extensions":["storage_zfs_remove_snapshots","container_host_shutdown_timeout","container_stop_priority","container_syscall_filtering","auth_pki","container_last_used_at","etag","patch","usb_devices","https_allowed_credentials","image_compression_algorithm","directory_manipulation","container_cpu_time","storage_zfs_use_refquota","storage_lvm_mount_options","network","profile_usedby","container_push","container_exec_recording","certificate_update","container_exec_signal_handling","gpu_devices","container_image_properties","migration_progress","id_map","network_firewall_filtering","network_routes","storage","file_delete","file_append","network_dhcp_expiry","storage_lvm_vg_rename","storage_lvm_thinpool_rename","network_vlan","image_create_aliases","container_stateless_copy","container_only_migration","storage_zfs_clone_copy","unix_device_rename","storage_lvm_use_thinpool","storage_rsync_bwlimit","network_vxlan_interface","storage_btrfs_mount_options","entity_description","image_force_refresh","storage_lvm_lv_resizing","id_map_base","file_symlinks","container_push_target","network_vlan_physical","storage_images_delete","container_edit_metadata","container_snapshot_stateful_migration","storage_driver_ceph","storage_ceph_user_name","resource_limits","storage_volatile_initial_source","storage_ceph_force_osd_reuse","storage_block_filesystem_btrfs","resources","kernel_limits","storage_api_volume_rename","network_sriov","console","restrict_devlxd","migration_pre_copy","infiniband","maas_network","devlxd_events","proxy","network_dhcp_gateway","file_get_symlink","network_leases","unix_device_hotplug","storage_api_local_volume_handling","operation_description","clustering","event_lifecycle","storage_api_remote_volume_handling","nvidia_runtime","container_mount_propagation","container_backup","devlxd_images","container_local_cross_pool_handling","proxy_unix","proxy_udp","clustering_join","proxy_tcp_udp_multi_port_handling","network_state","proxy_unix_dac_properties","container_protection_delete","unix_priv_drop","pprof_http","proxy_haproxy_protocol","network_hwaddr","proxy_nat","network_nat_order","container_full","backup_compression","nvidia_runtime_config","storage_api_volume_snapshots","storage_unmapped","projects","network_vxlan_ttl","container_incremental_copy","usb_optional_vendorid","snapshot_scheduling","snapshot_schedule_aliases","container_copy_project","clustering_server_address","clustering_image_replication","container_protection_shift","snapshot_expiry","container_backup_override_pool","snapshot_expiry_creation","network_leases_location","resources_cpu_socket","resources_gpu","resources_numa","kernel_features","id_map_current","event_location","storage_api_remote_volume_snapshots","network_nat_address","container_nic_routes","cluster_internal_copy","seccomp_notify","lxc_features","container_nic_ipvlan","network_vlan_sriov","storage_cephfs","container_nic_ipfilter","resources_v2","container_exec_user_group_cwd","container_syscall_intercept","container_disk_shift","storage_shifted","resources_infiniband","daemon_storage","instances","image_types","resources_disk_sata","clustering_roles","images_expiry","resources_network_firmware","backup_compression_algorithm","ceph_data_pool_name","container_syscall_intercept_mount","compression_squashfs","container_raw_mount","container_nic_routed","container_syscall_intercept_mount_fuse","container_disk_ceph","virtual-machines","image_profiles","clustering_architecture","resources_disk_id","storage_lvm_stripes","vm_boot_priority","unix_hotplug_devices","api_filtering","instance_nic_network","clustering_sizing","firewall_driver","projects_limits","container_syscall_intercept_hugetlbfs","limits_hugepages","container_nic_routed_gateway","projects_restrictions","custom_volume_snapshot_expiry","volume_snapshot_scheduling","trust_ca_certificates","snapshot_disk_usage","clustering_edit_roles","container_nic_routed_host_address","container_nic_ipvlan_gateway","resources_usb_pci","resources_cpu_threads_numa","resources_cpu_core_die","api_os","container_nic_routed_host_table","container_nic_ipvlan_host_table","container_nic_ipvlan_mode","resources_system","images_push_relay","network_dns_search","container_nic_routed_limits","instance_nic_bridged_vlan","network_state_bond_bridge","usedby_consistency","custom_block_volumes","clustering_failure_domains","resources_gpu_mdev","console_vga_type","projects_limits_disk","network_type_macvlan","network_type_sriov","container_syscall_intercept_bpf_devices","network_type_ovn","projects_networks","projects_networks_restricted_uplinks","custom_volume_backup","backup_override_name","storage_rsync_compression","network_type_physical","network_ovn_external_subnets","network_ovn_nat","network_ovn_external_routes_remove","tpm_device_type","storage_zfs_clone_copy_rebase","gpu_mdev","resources_pci_iommu","resources_network_usb","resources_disk_address","network_physical_ovn_ingress_mode","network_ovn_dhcp","network_physical_routes_anycast","projects_limits_instances","network_state_vlan","instance_nic_bridged_port_isolation","instance_bulk_state_change","network_gvrp","instance_pool_move","gpu_sriov","pci_device_type","storage_volume_state","network_acl","migration_stateful","disk_state_quota","storage_ceph_features","projects_compression","projects_images_remote_cache_expiry","certificate_project","network_ovn_acl","projects_images_auto_update","projects_restricted_cluster_target","images_default_architecture","network_ovn_acl_defaults","gpu_mig","project_usage","network_bridge_acl","warnings","projects_restricted_backups_and_snapshots","clustering_join_token","clustering_description","server_trusted_proxy","clustering_update_cert","storage_api_project","server_instance_driver_operational","server_supported_storage_drivers","event_lifecycle_requestor_address","resources_gpu_usb","clustering_evacuation","network_ovn_nat_address","network_bgp","network_forward","custom_volume_refresh","network_counters_errors_dropped","metrics","image_source_project","clustering_config","network_peer","linux_sysctl","network_dns","ovn_nic_acceleration","certificate_self_renewal","instance_project_move","storage_volume_project_move","cloud_init","network_dns_nat","database_leader","instance_all_projects","clustering_groups","ceph_rbd_du","instance_get_full","qemu_metrics","gpu_mig_uuid","event_project","clustering_evacuation_live","instance_allow_inconsistent_copy","network_state_ovn","storage_volume_api_filtering","image_restrictions","storage_zfs_export","network_dns_records","storage_zfs_reserve_space","network_acl_log","storage_zfs_blocksize","metrics_cpu_seconds","instance_snapshot_never","certificate_token","instance_nic_routed_neighbor_probe","event_hub","agent_nic_config","projects_restricted_intercept","metrics_authentication","images_target_project","cluster_migration_inconsistent_copy","cluster_ovn_chassis","container_syscall_intercept_sched_setscheduler","storage_lvm_thinpool_metadata_size","storage_volume_state_total","instance_file_head","instances_nic_host_name","image_copy_profile","container_syscall_intercept_sysinfo","clustering_evacuation_mode","resources_pci_vpd","qemu_raw_conf","storage_cephfs_fscache","network_load_balancer","vsock_api","instance_ready_state","network_bgp_holdtime","storage_volumes_all_projects","metrics_memory_oom_total","storage_buckets","storage_buckets_create_credentials","metrics_cpu_effective_total","projects_networks_restricted_access","storage_buckets_local","loki","acme","internal_metrics","cluster_join_token_expiry","remote_token_expiry","init_preseed","storage_volumes_created_at","cpu_hotplug","projects_networks_zones","network_txqueuelen","cluster_member_state","instances_placement_scriptlet","storage_pool_source_wipe","zfs_block_mode","instance_generation_id","disk_io_cache","amd_sev","storage_pool_loop_resize","migration_vm_live","ovn_nic_nesting","oidc","network_ovn_l3only","ovn_nic_acceleration_vdpa","cluster_healing","instances_state_total","auth_user","security_csm","instances_rebuild","numa_cpu_placement","custom_volume_iso","network_allocations","storage_api_remote_volume_snapshot_copy","zfs_delegate","operations_get_query_all_projects","metadata_configuration","syslog_socket","event_lifecycle_name_and_project","instances_nic_limits_priority","disk_initial_volume_configuration","operation_wait","cluster_internal_custom_volume_copy","disk_io_bus","storage_cephfs_create_missing","instance_move_config","ovn_ssl_config","init_preseed_storage_volumes","metrics_instances_count","server_instance_type_info","resources_disk_mounted","server_version_lts","oidc_groups_claim","loki_config_instance","storage_volatile_uuid","import_instance_devices","instances_uefi_vars","instances_migration_stateful","container_syscall_filtering_allow_deny_syntax","access_management","vm_disk_io_limits","storage_volumes_all","instances_files_modify_permissions","image_restriction_nesting","container_syscall_intercept_finit_module","device_usb_serial","network_allocate_external_ips","explicit_trust_token"],"api_status":"stable","api_version":"1.0","auth":"trusted","public":false,"auth_methods":["tls"],"auth_user_name":"atcore","auth_user_method":"unix","environment":{"addresses":[],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE-----\nMIICWDCCAd2gAwIBAgIQPgA0vZOcU8FlFq3A+/glCTAKBggqhkjOPQQDAzBKMQww\nCgYDVQQKEwNMWEQxOjA4BgNVBAMMMXJvb3RAaXAtMTkyLTE2OC0xMC0yMjIuZXUt\nd2VzdC0yLmNvbXB1dGUuaW50ZXJuYWwwHhcNMjQxMjE3MTY1OTI1WhcNMzQxMjE1\nMTY1OTI1WjBKMQwwCgYDVQQKEwNMWEQxOjA4BgNVBAMMMXJvb3RAaXAtMTkyLTE2\nOC0xMC0yMjIuZXUtd2VzdC0yLmNvbXB1dGUuaW50ZXJuYWwwdjAQBgcqhkjOPQIB\nBgUrgQQAIgNiAAR0PkqloGbmbTDqq7ECQ4+ZRzXCzOfseqcjL/l1/5AFoitILjG7\nMdwF8g8UEVreprZyysZ691foeoGXXBgGMGtVzvc0hHNSvvInHRxUaWA9Vv92DbwX\ndmWW5FWXaNXSHVujgYcwgYQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsG\nAQUFBwMBMAwGA1UdEwEB/wQCMAAwTwYDVR0RBEgwRoIsaXAtMTkyLTE2OC0xMC0y\nMjIuZXUtd2VzdC0yLmNvbXB1dGUuaW50ZXJuYWyHBH8AAAGHEAAAAAAAAAAAAAAA\nAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxANEo8LN1v0GRm2F+gU91GPRafU1zijDt\nSLj8XN30QhB9lCX6sPC+EbIYf5MYPJTRUwIxAInrlOGdO9jcjydw6Qt8sAvK57tH\nEWzmD7ZriziiMFXOE+pm2+f6DXAMw02DzY9ffg==\n-----END CERTIFICATE-----\n","certificate_fingerprint":"3e211c95821c27d747f521c60086fcee3cfd66fa1854e6cddf6dfdc8815a208e","driver":"lxc","driver_version":"6.0.0","instance_types":["container"],"firewall":"nftables","kernel":"Linux","kernel_architecture":"x86_64","kernel_features":{"idmapped_mounts":"true","netnsid_getifaddrs":"true","seccomp_listener":"true","seccomp_listener_continue":"true","uevent_injection":"true","unpriv_fscaps":"true"},"kernel_version":"6.1.119-129.201.amzn2023.x86_64","lxc_features":{"cgroup2":"true","core_scheduling":"true","devpts_fd":"true","idmapped_mounts_v2":"true","mount_injection_file":"true","network_gateway_device_route":"true","network_ipvlan":"true","network_l2proxy":"true","network_phys_macvlan_mtu":"true","network_veth_router":"true","pidfd":"true","seccomp_allow_deny_syntax":"true","seccomp_notify":"true","seccomp_proxy_send_notify_fd":"true"},"os_name":"Amazon Linux","os_version":"2023","project":"default","server":"lxd","server_clustered":false,"server_event_mode":"full-mesh","server_name":"ip-192-168-10-222.eu-west-2.compute.internal","server_pid":7280,"server_version":"5.21.2","server_lts":true,"storage":"dir","storage_version":"1","storage_supported_drivers":[{"Name":"lvm","Version":"2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.47.0","Remote":false},{"Name":"powerflex","Version":"1.16 (nvme-cli)","Remote":true},{"Name":"btrfs","Version":"5.16.2","Remote":false},{"Name":"ceph","Version":"17.2.7","Remote":true},{"Name":"cephfs","Version":"17.2.7","Remote":true},{"Name":"cephobject","Version":"17.2.7","Remote":true},{"Name":"dir","Version":"1","Remote":false}]}}}

[simondeziel]

@Dweller can you identify what changed between the working version and the broken one? Please check pylxd's version as well as all its dependencies.

[Dweller]

The pylxd version if 2.3.5 (I've downgraded it to 2.5.1 with no difference in the result).

I'll get a list of the depending packages later on.. I'm assuming its probably going to be urllib3 which is where the error is being raised from.. Possibly because this latest version of Amazon linux has boosted the python version and supporting libs..

Will confirm later on today

[Dweller]

I've not yet been able to confirm the version changes but a quick web search how's its possible it was a change made to requests-unix-sockets.. Apparently they have no intention of fixing this one.

psf/requests#6707

[Dweller]

OK so this if the installed python modules on Amazon Linux 2023.6.20241212 after doing a yum install python3-pip and then pip install pylxd. The python version is 3.9.20.

ws4py==0.5.1
requests-toolbelt==1.0.0
pylxd==2.3.5
selinux==3.4
dbus-python==1.2.18
ruamel.yaml.clib==0.1.2
rpm==4.16.1.3
awscrt==0.19.19
gpg==1.15.1
libcomps==0.1.20
MarkupSafe==1.1.1
netifaces==0.10.6
cffi==1.14.5
cryptography==36.0.1
pyrsistent==0.17.3
PyYAML==5.4.1
setools==4.4.1
systemd-python==235
setuptools==59.6.0
six==1.15.0
distro==1.5.0
python-dateutil==2.8.1
docutils==0.16
idna==2.10
ruamel.yaml==0.16.6
configobj==5.0.6
chevron==0.13.1
attrs==20.3.0
chardet==4.0.0
colorama==0.4.4
jmespath==0.10.0
jsonpointer==2.0
jsonpatch==1.21
lockfile==0.12.2
python-daemon==2.3.0
oauthlib==3.0.2
ply==3.11
pycparser==2.20
prettytable==0.7.2
jsonschema==3.2.0
pyserial==3.4
PySocks==1.7.1
urllib3==1.25.10
requests==2.25.1
pytz==2022.7.1
Babel==2.9.1
Jinja2==2.11.3
sepolicy==3.4
cloud-init==22.2.2
wcwidth==0.2.5
prompt-toolkit==3.0.24
awscli==2.17.18
aws-cfn-bootstrap==2.0
release-notification==1.2
support-info==1.0
ec2-hibinit-agent==1.0.8
pip==21.3.1

[Dweller]

And the previous version.... that works.. Amazon Linux release 2023.6.20241121

ws4py==0.5.1
requests-toolbelt==1.0.0
pylxd==2.3.5
selinux==3.4
dbus-python==1.2.18
ruamel.yaml.clib==0.1.2
rpm==4.16.1.3
awscrt==0.19.19
gpg==1.15.1
libcomps==0.1.20
MarkupSafe==1.1.1
netifaces==0.10.6
cffi==1.14.5
cryptography==36.0.1
pyrsistent==0.17.3
PyYAML==5.4.1
setools==4.4.1
systemd-python==235
pycurl==7.45.1
setuptools==59.6.0
six==1.15.0
distro==1.5.0
python-dateutil==2.8.1
docutils==0.16
idna==2.10
ruamel.yaml==0.16.6
configobj==5.0.6
chevron==0.13.1
attrs==20.3.0
chardet==4.0.0
colorama==0.4.4
jmespath==0.10.0
jsonpointer==2.0
jsonpatch==1.21
lockfile==0.12.2
python-daemon==2.3.0
oauthlib==3.0.2
ply==3.11
pycparser==2.20
prettytable==0.7.2
jsonschema==3.2.0
pyserial==3.4
PySocks==1.7.1
urllib3==1.25.10
requests==2.25.1
pytz==2022.7.1
Babel==2.9.1
Jinja2==2.11.3
sepolicy==3.4
cloud-init==22.2.2
wcwidth==0.2.5
prompt-toolkit==3.0.24
awscli==2.15.30
aws-cfn-bootstrap==2.0
release-notification==1.2
support-info==1.0
ec2-hibinit-agent==1.0.8
pip==21.3.1

[Dweller]

Which is to say no difference what so ever..

19d18
< pycurl==7.45.1
53c52
< awscli==2.15.30
---
> awscli==2.17.18

[Dweller]

Working version is running
Python 3.9.16

Broken version is Python 3.9.20

Is biggest difference I can find so far..

[Dweller]

Ok I've resolved the issue with pylxd by installing python3.12 on the platform...

So my conclusion is the issue is connected with python3.9.20