Affected module
Backend, Ingestion Framework
Describe the bug
When S3/storage containers are ingested with subdirectories (for example via openmetadata.json configured with dataPath and depth > 0), the container's relative path contains slashes (e.g., ba/entec/woveycur/actor). This results in an entity FQN containing slashes (e.g., S3.my-bucket.ba/entec/woveycur/actor).
This causes two related issues:
- Permissions API route failure (404 Not Found):
Calling GET /api/v1/permissions/container/name/{fqn} fails with 404 because PermissionsResource.java declares @Path("/{resource}/name/{name}") with the default single-segment regex [^/]+. When reverse proxies decode %2F into /, Jersey parses the remaining segments as extra unmatched path parts and returns 404.
- Search hydration failure in
es_search_container_by_path:
In ESMixin (_search_es_entity), search hits matched by Elasticsearch are hydrated by calling self.get_by_name(entity=entity_type, fqn=hit["_source"]["fullyQualifiedName"], fields=fields). Because the container FQN contains slashes, the backend call fails with 404. As a result, get_by_name returns None, and _search_es_entity skips the valid entity (if entity is None: continue), causing es_search_container_by_path to return None despite the container existing in both the database and Elasticsearch.
To Reproduce
- Ingest an S3 bucket with an
openmetadata.json containing:
{"entries":[{"dataPath":"ba","depth":3,"structureFormat":"json"}]}
- Ingestion creates a container entity with name
ba/entec/woveycur/actor and FQN S3.<bucket>.ba/entec/woveycur/actor.
- Call
GET /api/v1/permissions/container/name/S3.<bucket>.ba%2Fentec%2Fwoveycur%2Factor -> returns 404.
- Call
metadata.es_search_container_by_path(full_path="s3://<bucket>/ba/entec/woveycur/actor", fields="dataModel") -> returns None even though metadata.list_all_entities(entity=Container) shows the record exists.
Expected behavior
- The permissions endpoint should match slash-containing FQNs and return user permissions for the container.
es_search_container_by_path should successfully hydrate and return the container entity using its ID from the Elasticsearch search hit.
Additional context
Addressed in PR: #34234
Affected module
Backend, Ingestion Framework
Describe the bug
When S3/storage containers are ingested with subdirectories (for example via
openmetadata.jsonconfigured withdataPathanddepth > 0), the container's relative path contains slashes (e.g.,ba/entec/woveycur/actor). This results in an entity FQN containing slashes (e.g.,S3.my-bucket.ba/entec/woveycur/actor).This causes two related issues:
Calling
GET /api/v1/permissions/container/name/{fqn}fails with 404 becausePermissionsResource.javadeclares@Path("/{resource}/name/{name}")with the default single-segment regex[^/]+. When reverse proxies decode%2Finto/, Jersey parses the remaining segments as extra unmatched path parts and returns 404.es_search_container_by_path:In
ESMixin(_search_es_entity), search hits matched by Elasticsearch are hydrated by callingself.get_by_name(entity=entity_type, fqn=hit["_source"]["fullyQualifiedName"], fields=fields). Because the container FQN contains slashes, the backend call fails with 404. As a result,get_by_namereturnsNone, and_search_es_entityskips the valid entity (if entity is None: continue), causinges_search_container_by_pathto returnNonedespite the container existing in both the database and Elasticsearch.To Reproduce
openmetadata.jsoncontaining:{"entries":[{"dataPath":"ba","depth":3,"structureFormat":"json"}]}ba/entec/woveycur/actorand FQNS3.<bucket>.ba/entec/woveycur/actor.GET /api/v1/permissions/container/name/S3.<bucket>.ba%2Fentec%2Fwoveycur%2Factor-> returns 404.metadata.es_search_container_by_path(full_path="s3://<bucket>/ba/entec/woveycur/actor", fields="dataModel")-> returnsNoneeven thoughmetadata.list_all_entities(entity=Container)shows the record exists.Expected behavior
es_search_container_by_pathshould successfully hydrate and return the container entity using its ID from the Elasticsearch search hit.Additional context
Addressed in PR: #34234