Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added composite tabular data file creation #107

Merged
merged 96 commits into from
Oct 8, 2024
Merged

Conversation

bbollen23
Copy link

@bbollen23 bbollen23 commented Sep 16, 2024

Does this PR close any open issues?

Closes #70 , #72 , #106 , #117 , #119 , #120

Give a longer description of what this PR addresses and why it's needed

  • Fixes several bugs/issues
  • Allows for experiment level aggregation

TODOs:

  • Generate composite tabular data csv
  • Fix models not being persistent
  • Fix DuckDB data location URL
  • Fix windows error logging when building
  • Decide on final segmentations folder path design
  • Add tags support when uploading data
  • Update the documentation on data format with the new combined tabular data attribute.
  • Switch to Pandas for CSV and save as Parquet file
  • Fix Uploading UI so that we can't navigate to uploading aside from going through stepper
  • Remove 'run-dev' flag and update documentation to reduce complexity.
  • Bug - UI reports experiment created even if failed (case: all data uploaded and processed, just finish experiment view failed)

EDIT 09/23/24:

To account for adding different scales of "cellMetadata" (i.e. experiment level, location level, etc), we will need to adjust the cellMetaDataStore.

Initial thoughts is that the datastructure should be an object which houses the entirity of the current cellMetaDataStore information so that we can easily access different cellMetaData. For example,

{
    "location-1": { }
    "location-2": { }
}

Then, when performing actions against the cell metadata store, we act on one (or more) of these objects. But mostly will be self contained (especially the initializing functions).

I do not think we will need more granularity. The "tagging" element we will eventually have will most likely be stored at the individual location level, so filtering based on this should be simple.

EDIT 09/27/24

After speaking with @Dev-Lan , I was misunderstanding what our goal is. The cell metadata store is mostly meant to just work with the actual cell metadata table. We will only ever load one table at once -- therefore, we do not need to necessarily split this out based on location. We may just have to remove some things from the store that are more general.

Also, we decided to add the "tagging" feature in this branch since it still pertains to the creation of the composite tabular data file.

@bbollen23 bbollen23 added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request Loon Migration Features that were in Loon, that must be added to Loonar backend Deals with Django backend labels Sep 21, 2024
Copy link

@Dev-Lan Dev-Lan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice job! I made some edits directly. There are many commits but all of them are small.

Just a few comments to resolve.

apps/client/src/components/upload/StepUploadStatus.vue Outdated Show resolved Hide resolved
apps/client/src/stores/componentStores/uploadStore.ts Outdated Show resolved Hide resolved
apps/server/api/migrations/0001_initial.py Outdated Show resolved Hide resolved
apps/client/src/App.scss Show resolved Hide resolved
apps/server/api/views.py Outdated Show resolved Hide resolved
@bbollen23 bbollen23 merged commit 657b8c1 into dev Oct 8, 2024
@bbollen23 bbollen23 deleted the 70-multi-location branch October 8, 2024 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architecture Deals with general architecture of application backend Deals with Django backend bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request Loon Migration Features that were in Loon, that must be added to Loonar
Projects
None yet
3 participants