Endpoint to conflate the submission with osm data #1594

Sujanadh · 2024-06-21T07:18:46Z

What type of PR is this? (check all applicable)

Related Issue

Issue No. Create endpoint for conflation of FMTM data with existing OSM data #1548

Describe this PR

This PR creates a new endpoint in order to conflate the submission data with osm features.

if both geometries (buildings for now) are equal or almost equal, then only their tags/properties are appended and returned for further validation
if both geometries overlaps to some extent then in that case both geometries are returned along with their properties and tags appended
is_duplicate and is_partial_overlap boolean flags (True or False) are appended, respectively.

Response

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              85.297789,
              27.723315
            ],
            [
              85.297842,
              27.723297
            ],
            [
              85.297856,
              27.723329
            ],
            [
              85.297908,
              27.723312
            ],
            [
              85.297924,
              27.723351
            ],
            [
              85.297819,
              27.723386
            ],
            [
              85.297789,
              27.723315
            ]
          ]
        ]
      },
      "properties": {
        "start": "2024-06-11T14:01:37.472+05:45",
        "end": "2024-06-11T14:02:11.013+05:45",
        "today": "2024-06-11",
        "phonenumber": null,
        "deviceid": "collect:N9J5nVWtA9ApE60K",
        "username": "localadmin",
        "email": "NOT IMPLEMENTED",
        "instructions": null,
        "task_id": null,
        "existing": "be3d424a-a5a4-4544-a0c5-3b38dd522dd7",
        "point": null,
        "digitisation_correct": "yes",
        "digitisation_problem": null,
        "digitisation_problem_other": null,
        "image": null,
        "form_category": "Unknown",
        "xid": "650020468",
        "status": "2",
        "category": "housing",
        "name": null,
        "building_material": null,
        "building_levels": null,
        "religion": null,
        "denomination": null,
        "muslim": null,
        "hindu": null,
        "religious": null,
        "service": null,
        "Shop": null,
        "tourism": null,
        "education_details": null,
        "link": null,
        "government": null,
        "medical": null,
        "food": null,
        "cuisine": null,
        "heritage": null,
        "emergency": null,
        "operator": null,
        "inscription": null,
        "housing": null,
        "provider": null,
        "rooms": null,
        "ref": null,
        "building_neighbor": null,
        "power": null,
        "generator_source": null,
        "generator": null,
        "water": null,
        "water_source": null,
        "pump_unit": null,
        "capacity": null,
        "age": null,
        "building_prefab": null,
        "building_floor": null,
        "building_roof": null,
        "condition": null,
        "access_roof": null,
        "levels_underground": null,
        "comment": null,
        "osm_id": 650020468,
        "osm_type": "ways_poly",
        "version": 2,
        "tags": {
          "building": "yes"
        },
        "changeset": 98649258,
        "timestamp": "2021-02-03T14:09:58",
        "is_duplicate": true,
        "is_partial_overlap": false
      }
    }
  ]
}

Alternative Approaches Considered

Did you attempt any other approaches that are not documented in code?

Review Guide

Notes for the reviewer. How to test this change?

Checklist before requesting a review

📖 Read the FMTM Contributing Guide: https://github.com/hotosm/fmtm/blob/main/CONTRIBUTING.md
📖 Read the HOT Code of Conduct: https://docs.hotosm.org/code-of-conduct
👷‍♀️ Create small PRs. In most cases, this will be possible.
✅ Provide tests for your changes.
📝 Use descriptive commit messages.
📗 Update any related documentation and include any relevant screenshots.

[optional] What gif best describes this PR or how it makes you feel?

spwoodcock · 2024-06-28T19:31:21Z

src/backend/app/db/postgis_utils.py

@@ -777,3 +785,160 @@ def parse_featcol(features: Union[Feature, FeatCol, MultiPolygon, Polygon]):
    elif isinstance(features, Feature):
        feat_col = geojson.FeatureCollection([feat_col])
    return feat_col
+
+
+def request_snapshot(geometry):


This function and the poll_task_status function can be replaced with osm_fieldwork.PostgresClient.
There is already functionality in place to get a data extract using provided config 👍
Example:

fmtm/src/backend/app/projects/project_crud.py

Line 449 in 319f65e

pg = PostgresClient(

spwoodcock · 2024-06-28T19:34:23Z

src/backend/app/submissions/submission_crud.py

@@ -524,3 +530,45 @@ async def get_submission_detail(
        odk_form.getSubmissions(project.odkid, db_xform.odk_form_id, submission_id)
    )
    return submission.get("value", [])[0]
+
+
+async def get_submission_geojson(


I noticed we already have central_crud.convert_odk_submission_json_to_geojson and submission_routes.download_submission_geojson that all basically do the same thing.

Could we refactor to only have the logic once, then reuse the code?

if i want geojson only for the conflation, then this function won't work since it returns submission in bytes. like you said, we can refactor the download_submission endpoint to use convert_odk_submission_json_to_geojson. Or we can update this function to return geojson and convert it into bytes wherever necessary instead of returning bytes from this function.

The second option sounds best in the spirit of DRY code.

A single function to return a geojson, then encode as bytes where specifically needed.

If absolutely necessary it's also possible to decode bytes back to geojson using .decode("utf--8") 👍

json.load would probably also do that

spwoodcock · 2024-06-28T19:34:45Z

src/backend/app/db/postgis_utils.py

+    )
+
+
+def conflate_features(


spwoodcock · 2024-07-02T15:20:33Z

We can merge this for now as initial functionality 👍

In the long term we should look into integrating this (after some good testing): osm-merge/osm-merge#9

…onflation

Sujanadh · 2024-07-04T11:41:04Z

updates:

used PostgresClient to extract features from osm.
used task outline instead of bbox of submitted feature.
updated the convert_odk_submission_json_to_geojson to return geojson and respective calls to convert geojson to bytes

spwoodcock · 2024-07-05T08:12:42Z

src/backend/app/db/postgis_utils.py

+        geometry,
+        extra_params={
+            "fileName": (
+                f"fmtm/{settings.FMTM_DOMAIN}/data_extract"


Minor thing - the folder structure starting with fmtm/... should only be used if we want to keep the data extract.

Raw data api has a rule if we use auth and a set structure, the file is retained for the project.

In this case we could remove this param as we only need a temp extract 👍

spwoodcock · 2024-07-05T08:13:32Z

src/backend/app/db/postgis_utils.py

+                if settings.RAW_DATA_API_AUTH_TOKEN
+                else "fmtm_extract"
+            ),
+            "outputType": "geojson",


I believe bind zip true and geojson are defaults on raw data api, but no harm being explicit 👍

I think its better to mention, as we won't know what are the default values.

…acts

Sujanadh · 2024-07-11T05:19:08Z

Updates:

bypass check for public project in wrap_check_access
removed filename parameter in extra params when requesting data extracts as we don't need it since it is only temp extracts and we don't have to save it in fmtm

spwoodcock · 2024-07-11T15:35:10Z

src/backend/app/auth/roles.py

@@ -242,6 +242,12 @@ async def wrap_check_access(
    role: ProjectRole,
 ) -> ProjectUserDict:
    """Wrap check_access call with HTTPException."""
+    # If project is public, skip permission check
+    if project.visibility == ProjectVisibility.PUBLIC:


Adding this logic here means that all users have access to all endpoints if the project is public.

So they could delete the project, modify anything, etc.

It's currently only present in the mapper permission, as we only want to bypass the auth if the endpoint requires mapper role only:

fmtm/src/backend/app/auth/roles.py

Line 327 in fd11d08

if project.visibility == ProjectVisibility.PUBLIC:

Was this logic needed for the code in this PR to work?

Everything else is all set to merge!

I did a minor update, fixing the return type on the mapper role if ProjectVisibility.PUBLIC.

I'm going to merge as this PR has been open a while - let me know if any issues!

Uh I totally missed that. Yes , you are right. I changed that in order to get ProjectUserDict.

…PUBLIC

feat: endpoint to conflate the submission with osm data

479c4b6

Sujanadh requested review from nrjadkry and spwoodcock June 21, 2024 07:18

Sujanadh self-assigned this Jun 21, 2024

Sujanadh temporarily deployed to test June 21, 2024 07:18 — with GitHub Actions Inactive

github-actions bot added the backend Related to backend code label Jun 21, 2024

Sujanadh temporarily deployed to test June 21, 2024 07:19 — with GitHub Actions Inactive

spwoodcock reviewed Jun 28, 2024

View reviewed changes

spwoodcock requested changes Jun 28, 2024

View reviewed changes

Sujanadh added 2 commits July 4, 2024 17:18

refactor: use postgresclient to get osm features using task geom

29a342c

Merge branch 'development' of github.com:hotosm/fmtm into feat/data-c…

ffc9a32

…onflation

Sujanadh temporarily deployed to test July 4, 2024 11:38 — with GitHub Actions Inactive

spwoodcock approved these changes Jul 5, 2024

View reviewed changes

fix: update wrap_check_access to bypass check for public project

4c7c55e

Sujanadh temporarily deployed to test July 11, 2024 05:16 — with GitHub Actions Inactive

refactor: remove filename from extra params when requesting data extr…

c4b73f5

…acts

Sujanadh temporarily deployed to test July 11, 2024 05:17 — with GitHub Actions Inactive

spwoodcock requested changes Jul 11, 2024

View reviewed changes

refactor: return DbUser and DbProject dict even if ProjectVisibility.…

1aceae4

…PUBLIC

spwoodcock temporarily deployed to test July 11, 2024 15:45 — with GitHub Actions Inactive

spwoodcock temporarily deployed to test July 11, 2024 15:46 — with GitHub Actions Inactive

spwoodcock merged commit 477f810 into development Jul 11, 2024
5 checks passed

spwoodcock deleted the feat/data-conflation branch July 11, 2024 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Endpoint to conflate the submission with osm data #1594

Endpoint to conflate the submission with osm data #1594

Sujanadh commented Jun 21, 2024

spwoodcock Jun 28, 2024

spwoodcock Jun 28, 2024

Sujanadh Jul 3, 2024 •

edited

Loading

spwoodcock Jul 3, 2024

spwoodcock Jun 28, 2024

spwoodcock commented Jul 2, 2024 •

edited

Loading

Sujanadh commented Jul 4, 2024

spwoodcock Jul 5, 2024

Sujanadh Jul 10, 2024

spwoodcock Jul 5, 2024

Sujanadh Jul 11, 2024

Sujanadh commented Jul 11, 2024 •

edited

Loading

spwoodcock Jul 11, 2024

spwoodcock Jul 11, 2024

spwoodcock Jul 11, 2024

Sujanadh Jul 11, 2024

Sujanadh Jul 11, 2024

Endpoint to conflate the submission with osm data #1594

Endpoint to conflate the submission with osm data #1594

Conversation

Sujanadh commented Jun 21, 2024

What type of PR is this? (check all applicable)

Related Issue

Describe this PR

Response

Alternative Approaches Considered

Review Guide

Checklist before requesting a review

[optional] What gif best describes this PR or how it makes you feel?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sujanadh Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spwoodcock commented Jul 2, 2024 • edited Loading

Sujanadh commented Jul 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sujanadh commented Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sujanadh Jul 3, 2024 •

edited

Loading

spwoodcock commented Jul 2, 2024 •

edited

Loading

Sujanadh commented Jul 11, 2024 •

edited

Loading