Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MoltenVK failing to resolve GetDeviceXxxMemoryRequirements #105

Open
pollend opened this issue Nov 12, 2024 · 22 comments
Open

MoltenVK failing to resolve GetDeviceXxxMemoryRequirements #105

pollend opened this issue Nov 12, 2024 · 22 comments
Labels
in progress Issue is in progress not bug Not a bug vk VK specific

Comments

@pollend
Copy link
Contributor

pollend commented Nov 12, 2024 •
edited
Loading

for the moment i was trying to use MoltenVK for my project. I'm unable to resolve these methods below I wonder if moltenVK provides these interfaces? just wanted to see if it would be possible to just use this but if worse comes to worse its more ideal to just support metal directly then using an intermediary api.

Screenshot 2024-11-11 at 10 47 01 PM 
    GET_DEVICE_CORE_OR_KHR_PROC(GetDeviceBufferMemoryRequirements);
    GET_DEVICE_CORE_OR_KHR_PROC(GetDeviceImageMemoryRequirements);

VUID-VkPhysicalDeviceGroupProperties-sType-sType(ERROR / SPEC): msgNum: -907153781 - Validation Error: [ VUID-VkPhysicalDeviceGroupProperties-sType-sType ] | MessageID = 0xc9edee8b | vkEnumeratePhysicalDeviceGroups(): pPhysicalDeviceGroupProperties[0].sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES. The Vulkan spec states: sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES (https://vulkan.lunarg.com/doc/view/1.3.290.0/mac/1.3-extensions/vkspec.html#VUID-VkPhysicalDeviceGroupProperties-sType-sType)
    Objects: 0
nri::INFO (DeviceVK.hpp:133) - VK::Unknown - Supported instance extensions:
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_device_group_creation (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_fence_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_memory_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_semaphore_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_get_physical_device_properties2 (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_get_surface_capabilities2 (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_surface (v25)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_debug_report (v10)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_debug_utils (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_headless_surface (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_layer_settings (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_metal_surface (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_surface_maintenance1 (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_swapchain_colorspace (v4)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_MVK_macos_surface (v3)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_portability_enumeration (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_LUNARG_direct_driver_loading (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_validation_features (v2)
VUID-VkPhysicalDeviceGroupProperties-sType-sType(ERROR / SPEC): msgNum: -907153781 - Validation Error: [ VUID-VkPhysicalDeviceGroupProperties-sType-sType ] | MessageID = 0xc9edee8b | vkEnumeratePhysicalDeviceGroups(): pPhysicalDeviceGroupProperties[0].sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES. The Vulkan spec states: sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES (https://vulkan.lunarg.com/doc/view/1.3.290.0/mac/1.3-extensions/vkspec.html#VUID-VkPhysicalDeviceGroupProperties-sType-sType)
    Objects: 0
nri::INFO (DeviceVK.hpp:179) - VK::Apple M3 - Supported device extensions:
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_16bit_storage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_8bit_storage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_bind_memory2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_buffer_device_address (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_calibrated_timestamps (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_copy_commands2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_create_renderpass2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_dedicated_allocation (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_deferred_host_operations (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_depth_stencil_resolve (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_descriptor_update_template (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_device_group (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_driver_properties (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_dynamic_rendering (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_fence (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_memory (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_semaphore (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_fragment_shader_barycentric (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_format_feature_flags2 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_get_memory_requirements2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_imageless_framebuffer (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_image_format_list (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_incremental_present (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance1 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance3 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_map_memory2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_multiview (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_portability_subset (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_push_descriptor (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_relaxed_block_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_sampler_mirror_clamp_to_edge (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_sampler_ycbcr_conversion (v14)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_separate_depth_stencil_layouts (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_draw_parameters (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_float_controls (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_float16_int8 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_integer_dot_product (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_non_semantic_info (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_subgroup_extended_types (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_spirv_1_4 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_storage_buffer_storage_class (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_swapchain (v70)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_swapchain_mutable_format (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_synchronization2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_timeline_semaphore (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_uniform_buffer_standard_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_variable_pointers (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_vertex_attribute_divisor (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_4444_formats (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_buffer_device_address (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_calibrated_timestamps (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_debug_marker (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_descriptor_indexing (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state3 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_external_memory_host (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_fragment_shader_interlock (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_hdr_metadata (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_host_image_copy (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_host_query_reset (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_image_robustness (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_inline_uniform_block (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_memory_budget (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_metal_objects (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_pipeline_creation_cache_control (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_pipeline_creation_feedback (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_post_depth_coverage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_private_data (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_robustness2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_sample_locations (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_scalar_block_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_separate_stencil_usage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_atomic_float (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_demote_to_helper_invocation (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_stencil_export (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_subgroup_ballot (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_subgroup_vote (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_viewport_index_layer (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_subgroup_size_control (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_swapchain_maintenance1 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_texel_buffer_alignment (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_texture_compression_astc_hdr (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_vertex_attribute_divisor (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_gpu_shader_half_float (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_negative_viewport_height (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_shader_image_load_store_lod (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_shader_trinary_minmax (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_IMG_format_pvrtc (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_INTEL_shader_integer_functions2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_GOOGLE_display_timing (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_NV_fragment_shader_barycentric (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_NV_glsl_shader (v1)
nri::ERROR (DeviceVK.hpp:1578) - VK::Apple M3 - ResolveDispatchTable(): Failed to get device function: 'vkGetDeviceBufferMemoryRequirements'
@pollend pollend changed the title moltenVK vulkan missing moltenVK vulkan failing to resolve GetDeviceBufferMemoryRequirements/GetDeviceImageMemoryRequirements Nov 12, 2024
@dzhdanNV
Copy link
Collaborator

I googled a bit and I have found that MoltenVK does support GetDeviceBufferMemoryRequirements/GetDeviceImageMemoryRequirement and even 2 variants.

@dzhdanNV
Copy link
Collaborator

For example:
KhronosGroup/MoltenVK#1250
KhronosGroup/MoltenVK#1216
Can it be a linkage issue? (like in 1216 from the above)

@pollend
Copy link
Contributor Author

pollend commented Nov 13, 2024

I tried a couple more things, i separated out the windowing logic so i don't get this double linking problem im seeing. the rendering code is loaded from another library but i don't see why this would be problem. this looks similar to what is happening but, im a little flumixed why this is not working.

KhronosGroup/MoltenVK#1708


WARNING-CreateInstance-status-message(INFO / SPEC): msgNum: 601872502 - Validation Information: [ WARNING-CreateInstance-status-message ] Object 0: handle = 0x12c853800, type = VK_OBJECT_TYPE_INSTANCE; | MessageID = 0x23dfd876 | vkCreateInstance():  Khronos Validation Layer Active:
    Settings File: Found at /Users/michaelpollind/.local/share/vulkan/settings.d/vk_layer_settings.txt specified by VkConfig application override.
    Current Enables: None.
    Current Disables: VK_VALIDATION_FEATURE_DISABLE_CORE_CHECKS_EXT, VK_VALIDATION_FEATURE_DISABLE_THREAD_SAFETY_EXT.

    Objects: 1
        [0] 0x12c853800, type: 1, name: NULL
VUID-VkPhysicalDeviceGroupProperties-sType-sType(ERROR / SPEC): msgNum: -907153781 - Validation Error: [ VUID-VkPhysicalDeviceGroupProperties-sType-sType ] | MessageID = 0xc9edee8b | vkEnumeratePhysicalDeviceGroups(): pPhysicalDeviceGroupProperties[0].sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES. The Vulkan spec states: sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES (https://vulkan.lunarg.com/doc/view/1.3.290.0/mac/1.3-extensions/vkspec.html#VUID-VkPhysicalDeviceGroupProperties-sType-sType)
    Objects: 0
nri::INFO (DeviceVK.hpp:133) - VK::Unknown - Supported instance extensions:
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_device_group_creation (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_fence_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_memory_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_external_semaphore_capabilities (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_get_physical_device_properties2 (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_get_surface_capabilities2 (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_surface (v25)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_debug_report (v10)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_debug_utils (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_headless_surface (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_layer_settings (v2)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_metal_surface (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_surface_maintenance1 (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_swapchain_colorspace (v4)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_MVK_macos_surface (v3)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_KHR_portability_enumeration (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_LUNARG_direct_driver_loading (v1)
nri::INFO (DeviceVK.hpp:135) - VK::Unknown -     VK_EXT_validation_features (v2)
WARNING-CreateInstance-status-message(INFO / SPEC): msgNum: 601872502 - Validation Information: [ WARNING-CreateInstance-status-message ] Object 0: handle = 0x12d03ae08, type = VK_OBJECT_TYPE_INSTANCE; | MessageID = 0x23dfd876 | vkCreateInstance():  Khronos Validation Layer Active:
    Settings File: Found at /Users/michaelpollind/.local/share/vulkan/settings.d/vk_layer_settings.txt specified by VkConfig application override.
    Current Enables: None.
    Current Disables: VK_VALIDATION_FEATURE_DISABLE_CORE_CHECKS_EXT, VK_VALIDATION_FEATURE_DISABLE_THREAD_SAFETY_EXT.

    Objects: 1
        [0] 0x12d03ae08, type: 1, name: NULL
VUID-VkPhysicalDeviceGroupProperties-sType-sType(ERROR / SPEC): msgNum: -907153781 - Validation Error: [ VUID-VkPhysicalDeviceGroupProperties-sType-sType ] | MessageID = 0xc9edee8b | vkEnumeratePhysicalDeviceGroups(): pPhysicalDeviceGroupProperties[0].sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES. The Vulkan spec states: sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES (https://vulkan.lunarg.com/doc/view/1.3.290.0/mac/1.3-extensions/vkspec.html#VUID-VkPhysicalDeviceGroupProperties-sType-sType)
    Objects: 0
nri::ERROR (DeviceVK.hpp:1248) - VK::Unknown - Validation Error: [ VUID-VkPhysicalDeviceGroupProperties-sType-sType ] | MessageID = 0xc9edee8b | vkEnumeratePhysicalDeviceGroups(): pPhysicalDeviceGroupProperties[0].sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES. The Vulkan spec states: sType must be VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_GROUP_PROPERTIES (https://vulkan.lunarg.com/doc/view/1.3.290.0/mac/1.3-extensions/vkspec.html#VUID-VkPhysicalDeviceGroupProperties-sType-sType)
nri::INFO (DeviceVK.hpp:179) - VK::Apple M3 - Supported device extensions:
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_16bit_storage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_8bit_storage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_bind_memory2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_buffer_device_address (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_calibrated_timestamps (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_copy_commands2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_create_renderpass2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_dedicated_allocation (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_deferred_host_operations (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_depth_stencil_resolve (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_descriptor_update_template (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_device_group (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_driver_properties (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_dynamic_rendering (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_fence (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_memory (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_external_semaphore (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_fragment_shader_barycentric (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_format_feature_flags2 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_get_memory_requirements2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_imageless_framebuffer (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_image_format_list (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_incremental_present (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance1 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_maintenance3 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_map_memory2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_multiview (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_portability_subset (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_push_descriptor (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_relaxed_block_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_sampler_mirror_clamp_to_edge (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_sampler_ycbcr_conversion (v14)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_separate_depth_stencil_layouts (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_draw_parameters (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_float_controls (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_float16_int8 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_integer_dot_product (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_non_semantic_info (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_shader_subgroup_extended_types (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_spirv_1_4 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_storage_buffer_storage_class (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_swapchain (v70)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_swapchain_mutable_format (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_synchronization2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_timeline_semaphore (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_uniform_buffer_standard_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_variable_pointers (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_KHR_vertex_attribute_divisor (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_4444_formats (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_buffer_device_address (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_calibrated_timestamps (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_debug_marker (v4)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_descriptor_indexing (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_extended_dynamic_state3 (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_external_memory_host (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_fragment_shader_interlock (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_hdr_metadata (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_host_image_copy (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_host_query_reset (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_image_robustness (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_inline_uniform_block (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_memory_budget (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_metal_objects (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_pipeline_creation_cache_control (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_pipeline_creation_feedback (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_post_depth_coverage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_private_data (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_robustness2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_sample_locations (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_scalar_block_layout (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_separate_stencil_usage (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_atomic_float (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_demote_to_helper_invocation (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_stencil_export (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_subgroup_ballot (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_subgroup_vote (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_shader_viewport_index_layer (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_subgroup_size_control (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_swapchain_maintenance1 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_texel_buffer_alignment (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_texture_compression_astc_hdr (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_EXT_vertex_attribute_divisor (v3)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_gpu_shader_half_float (v2)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_negative_viewport_height (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_shader_image_load_store_lod (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_AMD_shader_trinary_minmax (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_IMG_format_pvrtc (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_INTEL_shader_integer_functions2 (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_GOOGLE_display_timing (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_NV_fragment_shader_barycentric (v1)
nri::INFO (DeviceVK.hpp:181) - VK::Apple M3 -     VK_NV_glsl_shader (v1)
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 - Inserted device layer "VK_LAYER_KHRONOS_validation" (/usr/local/share/vulkan/explicit_layer.d/../../../lib/libVkLayer_khronos_validation.dylib)
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 - vkCreateDevice layer callstack setup to:
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -    <Application>
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -      ||
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -    <Loader>
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -      ||
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -    VK_LAYER_KHRONOS_validation
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -            Type: Explicit
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -            Manifest: /usr/local/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -            Library:  /usr/local/share/vulkan/explicit_layer.d/../../../lib/libVkLayer_khronos_validation.dylib
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -      ||
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -    <Device>
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 -        Using "Apple M3" with driver: "/usr/local/share/vulkan/icd.d/../../../lib/libMoltenVK.dylib"
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 - Vulkan semaphores using MTLEvent.
nri::INFO (DeviceVK.hpp:1248) - VK::Apple M3 - Created VkDevice to run on GPU Apple M3 with the following 10 Vulkan extensions enabled:
		VK_KHR_deferred_host_operations v4
		VK_KHR_dynamic_rendering v1
		VK_KHR_portability_subset v1
		VK_KHR_push_descriptor v2
		VK_KHR_swapchain v70
		VK_KHR_swapchain_mutable_format v1
		VK_KHR_synchronization2 v1
		VK_EXT_memory_budget v1
		VK_EXT_sample_locations v1
		VK_EXT_shader_atomic_float v1
nri::ERROR (DeviceVK.hpp:1576) - VK::Apple M3 - ResolveDispatchTable(): Failed to get device function: 'vkGetDeviceBufferMemoryRequirements'

@pollend pollend mentioned this issue Nov 13, 2024
Open
@pollend
Copy link
Contributor Author

pollend commented Nov 14, 2024 •
edited
Loading

@dzhdanNV so I managed to reproduce this on my macbook on the NRISamples. @vertver were you ever able to run nri on any kind of apple hardware or might it be specific to me or my setup? At worse I just give up on trying with vulkan and committing to the metal branch.

NVIDIAGameWorks/NRISamples#8

@pollend pollend mentioned this issue Nov 14, 2024
Merged
@dzhdanNV
Copy link
Collaborator

I'm happy to help, but I don't know how.

@vertver
Copy link
Contributor

vertver commented Nov 14, 2024

@vertver were you ever able to run nri on any kind of apple hardware or might it be specific to me or my setup?

Yeah, I can try to run it on my Macbook Air M1. Most likely it's an SDK version issue.

@pollend
Copy link
Contributor Author

pollend commented Nov 14, 2024

@dzhdanNV I got a response from moltenvk looks like maintenance 4 ext is not available KhronosGroup/MoltenVK#2394

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Nov 15, 2024 •
edited
Loading

Thanks for digging. I would say it's unexpectedly expected. Of course, I take the responsibility for migrating from vkGetXxxMemoryRequirements to vkGetDeviceXxxMemoryRequirements and friends. The reason was simple it's stupid to create an object beforehand and ask for memory requirements afterwards. It adds unnecessary complications and restrictions on use cases. And this is what has been fixed in VK_KHR_maintenance4. IMO, MoltenVK must get updated to the latest ASAP, but it's out of our control. So, the question is - how to proceed?

Solutions:

  • MoltenVK: wait/push for fixes
  • NRI: return back support for vkGetXxxMemoryRequirements, requiring a pre-created object (it's easy for me, but it will add restrictions on use cases obviously)
  • NRI: native Metal support (hope they have support for something similar to vkGetDeviceXxxMemoryRequirements)

Other suggestions?

@dzhdanNV dzhdanNV added vk VK specific not bug Not a bug in progress Issue is in progress labels Nov 15, 2024
@pollend
Copy link
Contributor Author

pollend commented Nov 15, 2024

Thanks for digging. I would say it's unexpectedly expected. Of course, I take the responsibility for migrating from vkGetXxxMemoryRequirements to vkGetDeviceXxxMemoryRequirements and friends. The reason was simple it's stupid to create an object beforehand and ask for memory requirements afterwards. It adds unnecessary complications and restrictions on use cases. And this is what has been fixed in VK_KHR_maintenance4. IMO, MoltenVK must get updated to the latest ASAP, but it's out of our control. So, the question is - how to proceed?

Solutions:

* MoltenVK: wait/push for fixes

* NRI: return back support for `vkGetXxxMemoryRequirements`, requiring a pre-created object (it's easy for me, but it will add restrictions on use cases obviously)

* NRI: native Metal support (hope they have support for something similar to `vkGetDeviceXxxMemoryRequirements`)

Other suggestions?

i might just work on the metal branch more might take a while before i get it to a working state.

@dzhdanNV dzhdanNV changed the title moltenVK vulkan failing to resolve GetDeviceBufferMemoryRequirements/GetDeviceImageMemoryRequirements MoltenVK failing to resolve GetDeviceXxxMemoryRequirements Nov 15, 2024
@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Nov 15, 2024 •
edited
Loading

Please, let me know if you want to get back vkGetXxxMemoryRequirements from VK 1.0. Many thanks for your work!

@pollend
Copy link
Contributor Author

pollend commented Nov 15, 2024 •
edited
Loading

here is the ticket to track for getting samples to work again:

ref:
KhronosGroup/MoltenVK#1930
KhronosGroup/MoltenVK#2116

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 6, 2025

NRI has become fully VK v1.3 conformant. Moreover, VK v1.4 is around the corner. But MoltenVK does still barely support VK v1.2. Hope the community will push to the success or we get a native Metal implementation instead.

@parsaiej
Copy link

parsaiej commented Jan 7, 2025 •
edited
Loading

Hi! I stumbled here after my OSX app reported this during VK extension enumeration:

nri::ERROR (DeviceVK.hpp:1301) - VK::Apple M3 Pro - loader_validate_device_extensions: Device extension VK_KHR_maintenance4 not supported by selected physical device or enabled layers.

If I understand correctly, NRI makes use of VK_KHR_maintenance4 which is currently not supported yet in MoltenVK. I recall using NRI sometime last year with MoltenVK without issue, so it seems the hard dependency on VK_KHR_maintenance4 recently added when NRI became VK 1.3 conformant?

If so, it seems like your "option 2":

  • NRI: return back support for vkGetXxxMemoryRequirements, requiring a pre-created object (it's easy for me, but it will add restrictions on use cases obviously)

May be the path of least resistance for NRI to work on OSX again, at least until they finish catching up MoltenVK to 1.3 :)
I am discouraged it will be soon, since the PR is collecting dust: KhronosGroup/MoltenVK#2116

Edit: I wound up just merging that stale maintenance4 MoltenVK branch with latest on main and deployed the new binaries to /usr/. NRI device creation now succeeds. I know I am asking for trouble by using an unfinished PR, but at least this let me proceed with development in the meantime.

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 7, 2025

Do you mind to look through README (which now includes the list of required VK extensions) and post here unsupported, please? Or is it only VK_KHR_maintenance4?

I recall using NRI sometime last year with MoltenVK without issue, so it seems the hard dependency on VK_KHR_maintenance4 recently added when NRI became VK 1.3 conformant?

Actually it's unrelated. I switched to vkGetDeviceXxxMemoryRequirements quite a while ago.

Statements:

  • I don't mind to return back vkGetDeviceXxxMemoryRequirements (and old corresponding NRI core functions) to become MoltenVK-friendly, while we are waiting for:
    • MoltenVK update
    • Metal native support in NRI
  • I would prefer to see NRI using cutting edge Vulkan (v1.4 is just around the corner, it doesn't make any sense to support v1.1 and v1.2 with a pile of extensions applied on top just because all vendors move towards modern VK, in many cases it means just a driver update. Of course, middleware, like MoltenVK, is different, so we have to be flexible)
  • As the worst case, more old-school functionality can be organized in a small extension

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 8, 2025 •
edited
Loading

@pollend @parsaiej @vertver, would it be enough to do this:

  • add isGetMemoryDesc2Supported to DeviceDesc (D3D: always supported, VK: requires v1.3 or VK_KHR_maintenance4)
    • i.e. no hard dependency on VK_KHR_maintenance4
  • rename currently available Get[Buffer/Texture]MemoryDesc to Get[Buffer/Texture]MemoryDesc2
  • return back Get[Buffer/Texture]MemoryDesc accepting a Buffer/Texture object instead of its desc
  • ensure that internally NRI uses non-2 functions (in Helper interface)
  • (optionally) fix a few basic samples to use non-2 functions

Thoughts?

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 8, 2025 •
edited
Loading

These extensions stays required:

  • for Vulkan 1.2:
    • VK_KHR_synchronization2
    • VK_KHR_dynamic_rendering
    • VK_KHR_copy_commands2
    • VK_EXT_extended_dynamic_state
  • for APPLE:
    • VK_KHR_portability_enumeration (instance extension)
    • VK_KHR_get_physical_device_properties2 (instance extension)
    • VK_KHR_portability_subset

Is it a problem for MoltenVK?

@vertver
Copy link
Contributor

vertver commented Jan 8, 2025

Yes, it's no problem for MoltenVK - KhronosGroup/MoltenVK#1930. The only extension that is currently not supported is VK_KHR_maintenance4, so that should be fine.

Other than that, I don't think the other changes are that critical (it's nice to have GetMemoryDesc without a handle, but I don't use it often).

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 8, 2025

I always weep when I do a breaking change, but it's for the better future. Hope these breaking changes are acceptable for users. The good news is that I don't see a reason why old-school "non-2" should be removed. These functions are handy in situations when you already have an object. It happens very often. It happens even more often in some use cases.

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 8, 2025

I already implemented this. I will commit tomorrow alongside with modified NRI samples (will switch most of them to non-2, add isGetMemoryDesc2Supported check to others).

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 9, 2025

The problem has been solved in the latest release (v1.160). Now Core API has this:

void (NRI_CALL *GetBufferMemoryDesc)(const NriRef(Buffer) buffer, Nri(MemoryLocation) memoryLocation, NriOut NriRef(MemoryDesc) memoryDesc);
void (NRI_CALL *GetTextureMemoryDesc)(const NriRef(Texture) texture, Nri(MemoryLocation) memoryLocation, NriOut NriRef(MemoryDesc) memoryDesc);
void (NRI_CALL *GetBufferMemoryDesc2)(const NriRef(Device) device, const NriRef(BufferDesc) bufferDesc, Nri(MemoryLocation) memoryLocation, NriOut NriRef(MemoryDesc) memoryDesc); // requires "isGetMemoryDesc2Supported"
void (NRI_CALL *GetTextureMemoryDesc2)(const NriRef(Device) device, const NriRef(TextureDesc) textureDesc, Nri(MemoryLocation) memoryLocation, NriOut NriRef(MemoryDesc) memoryDesc); // requires "isGetMemoryDesc2Supported"

struct DeviceDesc {
    ...
    uint32_t isGetMemoryDesc2Supported : 1; // D3D: always supported, VK: requires "maintenance4" support
    ...
};

Same logic has been applied to RayTracing interface. Helper and Streamer interfaces use GetMemoryDesc under the hood to maximize compatibility.

The last step - could someone try it on an APPLE device, please? (NRI itself or samples)

@parsaiej
Copy link

parsaiej commented Jan 9, 2025 •
edited
Loading

I only have time to test this in my own app at the moment (there seem to be some build issues with NRISamples for OSX), but device creation at least works like a charm now. Many thanks for this!

I can submit a PR to fix some of the build/compile issues I see for the samples on OSX, its a little tedious though since there are several issues through the submodules, so would need to do one for each submodule.

@dzhdanNV
Copy link
Collaborator

dzhdanNV commented Jan 9, 2025 •
edited
Loading

Thanks for testing! Hope someone fixes NRI samples on OSX (again, doh) :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
in progress Issue is in progress not bug Not a bug vk VK specific
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pollend @parsaiej @vertver @dzhdanNV