Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/sim search b2f #77

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
24dd1f0
wip sim search integration
moilerat Dec 15, 2023
5a2ffd4
move feature extraction to import
emmaamblard Dec 19, 2023
c346100
- JsonDict -> List[Dict]
bea777515 Dec 20, 2023
d42cd2c
samples are not correct, we need to have a similar endpoint for simil…
moilerat Dec 22, 2023
694ed4b
typo in endpoint that doesn't explain the fact it is not reachable fr…
moilerat Dec 22, 2023
01aa2ae
add seed_object_id in filter to get them in api on back
moilerat Dec 22, 2023
c3905aa
I would rather put out of github the .idea
moilerat Dec 23, 2023
263a359
fix typo in endpoint
moilerat Dec 23, 2023
39c615f
first working version of unsupervised search in ecotaxa by lovnower, wip
moilerat Dec 23, 2023
b1c166f
cleaning
moilerat Dec 23, 2023
3d16a35
replace obj_cnn_features table with obj_cnn_features_vector
emmaamblard Jan 11, 2024
2b6e387
add basic similarity search as a job + test
emmaamblard Jan 23, 2024
a4d79f7
attempt to fix import bug in simsearch test
emmaamblard Jan 23, 2024
eb21e16
add filters to similarity search
emmaamblard Jan 24, 2024
ddcd5ab
remove print and broken test
emmaamblard Jan 24, 2024
f3d273a
launch feature extraction job before similarity search
emmaamblard Jan 25, 2024
e5a0c08
add similarity scores + test similarity search with filters on taxo
emmaamblard Jan 26, 2024
15c1786
fix filters bug
emmaamblard Jan 30, 2024
353ced8
replace function used in similarity search endpoint
emmaamblard Jan 30, 2024
4826154
remove limit to similarity search results set by the object_set query
emmaamblard Jan 31, 2024
b4a5abc
remove prints
emmaamblard Jan 31, 2024
d7e2311
add blank new line in README to test right to commit and config of lo…
Feb 15, 2024
d794cf5
first version script upgrade db with alchemy
Feb 16, 2024
405f99d
mention emma and lovnower for ss and cnnvector in license (only comme…
moilerat Feb 17, 2024
473d984
fix check that cnn_features are selected https://github.com/ecotaxa/e…
moilerat Feb 17, 2024
2111b96
trying to take into account @grololo remark https://github.com/ecotax…
moilerat Feb 17, 2024
7aeeb25
update test value for missing descriptors in test ss https://github.c…
Feb 17, 2024
3cd37d0
change version migration after following different advice
Feb 19, 2024
f61893b
using pgvector/pgvector:pg16 docker in NRT github actions tests after…
moilerat Feb 18, 2024
c5f88b9
remove blank line to force github actions that weren't trigge by chan…
moilerat Feb 19, 2024
9978ee5
remove useless migration create and drop linked to 'scories' with R i…
moilerat Feb 19, 2024
9b257ec
remove 'scories' due to different management of password in prod and …
moilerat Feb 19, 2024
02d6551
script migrate data cnnfeature vector
moilerat Feb 19, 2024
2181476
continue merge fix conflicts between @grololo commit on feat/sim_sear…
moilerat May 2, 2024
1259375
fix: upgrade script for new pgvector table
grololo06 Feb 19, 2024
4646c2f
remove useless endpoint ss
Feb 20, 2024
05c5293
protect some call in ss and change test to use existing endpoint and …
moilerat Apr 12, 2024
863ec11
use unique ss endpoint in nrt, didn't test but will look at tox
moilerat May 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"object_id" "object_lat" "object_lon" "object_date" "object_time" "object_link" "object_depth_min" "object_depth_max" "object_annotation_status" "object_annotation_person_name" "object_annotation_person_email" "object_annotation_date" "object_annotation_time" "object_annotation_category" "object_annotation_hierarchy" "complement_info" "object_by" "object_width" "object_height" "object_area" "object_mean" "object_major" "object_minor" "object_feret" "object_area_exc" "object_thickr" "object_esd" "object_elongation" "object_range" "object_meanpos" "object_centroids" "object_cv" "object_sr" "object_perimareaexc" "object_feretareaexc" "object_perimferet" "object_perimmajor" "object_circex" "object_cdexc" "object_kurt_mean" "object_skew_mean" "object_convperim_perim" "object_convarea_area" "object_symetrieh_area" "object_symetriev_area" "object_nb1_area" "object_nb2_area" "object_nb3_area" "object_nb1_range" "object_nb2_range" "object_nb3_range" "object_median_mean" "object_median_mean_range" "object_skeleton_area" "object_extra" "sample_id" "sample_dataportal_descriptor" "sample_project" "sample_ship" "sample_filename" "sample_samplinggear" "sample_volconc" "sample_comment" "sample_tot_vol" "process_id" "process_software" "process_start_date" "process_start_time" "process_analysed_color_channel" "process_background_method" "process_nb_images" "process_rolling" "process_pixel" "process_upper" "process_gamma_value" "process_vignette_color" "process_esd_min" "process_esd_max" "process_lut_offset" "process_lut_slope" "process_remove_objects_on_sides" "process_stop_n_images" "process_scale" "process_stop_after_m_objects" "process_nb_of_rawfile_images_in_folder" "process_grey_auto_adjust" "acq_id" "acq_instrument" "acq_cell_type" "acq_magnification" "acq_sub_part"
"m106_mn01_n1_sml_409" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "" "" "" "" "t001" "living>t001" "" 10262.0 46.0 94.0 516.0 241.039993286133 50.0999984741211 13.1000003814697 102.599998474121 4.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_821" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "" "" "" "" "Oncaeidae" "living>Oncaeidae" "" 9811.0 65.0 54.0 1401.0 202.039993286133 70.1999969482422 25.3999996185303 73.1999969482422 0.0 2.5 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_822" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "" "" "" "" "Cyclopoida" "living>Cyclopoida" "" 9866.0 33.0 51.0 733.0 200.25 48.7999992370605 19.1000003814697 51.5999984741211 0.0 2.82399988174438 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_850" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "" "" "" "" "other<living" "living>other" "" 10787.0 42.0 47.0 1583.0 222.789993286133 48.9000015258789 41.2000007629395 53.0999984741211 100.0 2.64299988746643 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_851" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "" "" "" "" "Oncaeidae" "living>Oncaeidae" "" 10823.0 47.0 69.0 1511.0 192.240005493164 60.7000007629395 31.7000007629395 70.5999984741211 3.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_409" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "t001" "living>t001" "" 10262.0 46.0 94.0 516.0 241.039993286133 50.0999984741211 13.1000003814697 102.599998474121 4.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_821" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "Oncaeidae" "living>Oncaeidae" "" 9811.0 65.0 54.0 1401.0 202.039993286133 70.1999969482422 25.3999996185303 73.1999969482422 0.0 2.5 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_822" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "Cyclopoida" "living>Cyclopoida" "" 9866.0 33.0 51.0 733.0 200.25 48.7999992370605 19.1000003814697 51.5999984741211 0.0 2.82399988174438 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_850" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "other<living" "living>other" "" 10787.0 42.0 47.0 1583.0 222.789993286133 48.9000015258789 41.2000007629395 53.0999984741211 100.0 2.64299988746643 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
"m106_mn01_n1_sml_851" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 600.0 1000.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "Oncaeidae" "living>Oncaeidae" "" 10823.0 47.0 69.0 1511.0 192.240005493164 60.7000007629395 31.7000007629395 70.5999984741211 3.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n1_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n1_sml" "net" "13.125" "xx" "" "zooprocess_m106_mn01_n1_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2243" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n1_sml" "epson" "200-500" "1" ""
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"object_id" "object_lat" "object_lon" "object_date" "object_time" "object_link" "object_depth_min" "object_depth_max" "object_annotation_status" "object_annotation_person_name" "object_annotation_person_email" "object_annotation_date" "object_annotation_time" "object_annotation_category" "object_annotation_hierarchy" "complement_info" "object_by" "object_width" "object_height" "object_area" "object_mean" "object_major" "object_minor" "object_feret" "object_area_exc" "object_thickr" "object_esd" "object_elongation" "object_range" "object_meanpos" "object_centroids" "object_cv" "object_sr" "object_perimareaexc" "object_feretareaexc" "object_perimferet" "object_perimmajor" "object_circex" "object_cdexc" "object_kurt_mean" "object_skew_mean" "object_convperim_perim" "object_convarea_area" "object_symetrieh_area" "object_symetriev_area" "object_nb1_area" "object_nb2_area" "object_nb3_area" "object_nb1_range" "object_nb2_range" "object_nb3_range" "object_median_mean" "object_median_mean_range" "object_skeleton_area" "object_extra" "sample_id" "sample_dataportal_descriptor" "sample_project" "sample_ship" "sample_filename" "sample_samplinggear" "sample_volconc" "sample_comment" "sample_tot_vol" "process_id" "process_software" "process_start_date" "process_start_time" "process_analysed_color_channel" "process_background_method" "process_nb_images" "process_rolling" "process_pixel" "process_upper" "process_gamma_value" "process_vignette_color" "process_esd_min" "process_esd_max" "process_lut_offset" "process_lut_slope" "process_remove_objects_on_sides" "process_stop_n_images" "process_scale" "process_stop_after_m_objects" "process_nb_of_rawfile_images_in_folder" "process_grey_auto_adjust" "acq_id" "acq_instrument" "acq_cell_type" "acq_magnification" "acq_sub_part"
"m106_mn01_n2_sml_1" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "" "" "" "" "detritus" "living>detritus" "" 14.0 25.0 26.0 413.0 175.529998779297 24.7000007629395 21.2999992370605 27.2999992370605 0.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
"m106_mn01_n2_sml_1081" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "" "" "" "" "detritus" "living>detritus" "" 9940.0 56.0 38.0 929.0 222.75 44.2000007629395 26.7999992370605 58.2000007629395 9.0 1.86000001430511 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
"m106_mn01_n2_sml_1111" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "" "" "" "" "egg<other" "living>egg" "" 10961.0 23.0 28.0 490.0 210.800003051758 27.0 23.1000003814697 29.2000007629395 0.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
"m106_mn01_n2_sml_1" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "detritus" "living>detritus" "" 14.0 25.0 26.0 413.0 175.529998779297 24.7000007629395 21.2999992370605 27.2999992370605 0.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
"m106_mn01_n2_sml_1081" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "detritus" "living>detritus" "" 9940.0 56.0 38.0 929.0 222.75 44.2000007629395 26.7999992370605 58.2000007629395 9.0 1.86000001430511 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
"m106_mn01_n2_sml_1111" 18.0 -24.4166666666667 "2014-04-20" "04:20:00" "" 300.0 600.0 "predicted" "Application Administrator" "administrator" "2015-11-11" "15:31:00" "egg<other" "living>egg" "" 10961.0 23.0 28.0 490.0 210.800003051758 27.0 23.1000003814697 29.2000007629395 0.0 2.0 "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "m106_mn01_n2_sml" "" "2014_m106_sml" "meteor" "m106_mn01_n2_sml" "net" "9.75" "xx" "" "zooprocess_m106_mn01_n2_sml" "zooprocess_pid_to_ecotaxa_7.23_2017/03/10" "20150829" "2256" "" "rolling_ball" "2" "20" "10.6" "245" "2" "b&w_no_background" "200" "1000" "0" "1" "yes" "" "1" "" "2" "" "generic_m106_mn01_n2_sml" "epson" "200-500" "1" ""
Loading
Loading