backport: Add layer of security and download p7zip-full lib on docker (…

…#643)

* bump: Bump custom proposal states (#599)

* Feat: custom sort for processes (#596)

* feat: add custom sort for processesdepending on new variable

* feat: update locales files

* feat: update seeds

* test: add controller tests for assemblies and processes

* chore: update i18n config for unused keys

* docs: update overrides

* refactor: update env variable after review

* Fix form initiatives (#600)

* fix: XSS vulnerability with img on initiative form and model

* test: add tests for new validation

* docs: update overrides section

* fix: interference from added extends with migration

* style: update with rubocop

* fix: ActiveRecord::NoDatabaseError

* fix: trying to fix again interference

* fix: update initiative fomr extends and modify admin initiative controller

* refactor: update with rubocop

* fix: validation in initiative_form extends and update test

* docs: update overrides section in overloads.md

* fix: Update OVERLOADS.md

---------

Co-authored-by: Quentin Champenois <[email protected]>

* bump: Decidim-Awesome to last commit (#607)

* feat: Bump decidim-awesome to last commit

* fix: Fix migration that has been changed since first implementation

* fix: Add Referrer-Policy to strict (#613)

* fix: Flash message on proposal limit per user reached (#609)

* fix error message displaying when reaching proposition add limit

* add test

* add keys in ignore_missing keys

* fix: Add block reported user task (#614)

* feat: Add module decidim-cleaner (#597)

* feat: Add decidim-cleaner

* refactor: Comment env var by default

---------

Co-authored-by: Quentin Champenois <[email protected]>

* Install GuestMeetingRegistration module (#615)

* Install GuestMeetingRegistration module

* Update registration module

* feat: Add Sendethics possibility to the sms gateway (#605)

Co-authored-by: Lucie Grau <[email protected]>

* Bump: Phone authorization handler module (#623)

* backport: remove sentry (#622)

* backport: Use cdn (#624)

* backport: self hosted cdn

* backport: update js files content

* fix: Missing image in survey question (#621)

* fix: update condition to not empty input value if image is present

* test: add system test to check for input value

* test: update check for image

* test: update img check again

* test: last update check img

* test: update

* test: another update

* test: if img is present

* test: update other test to avoid ambiguous selector error

* test: update to see if image is presnet after save

* fix: override editor js in decidim_awesome

* test: update system test

* fix: Questions order in survey export (#618)

* chore: update after pull

* fix: order questions by position in serializer

* test: add test for question order

* refactor: update test

* style: remove empty line

* bump: Guest meeting registration module (#625)

* Bump: Guest meeting registration module

* refactor(Gemfile): Remove ref reference in Gemfile

---------

Co-authored-by: Quentin Champenois <[email protected]>

* feat: add decypted private body to extra fields (#608)

* feat: add new column to proposal extra fields

* feat: add callback to proposal extra field model

* test: add test for proposal extra field model

* feat: add rake task to update existing data

* test: add test for new rake task

* feat: add proposal extra field model extends to config

* chore: update rubocop rules

* refactor: update task and test

* create the job file

* update rake task

* lint code

* add tests file

* fix rspec

* clean double specs

* update spec

* update spec

* update syntax test with a context instade of only 'it'

* lint code by removing useless line in job spec file

* update spec

* add more context in spec file

* lint code

* update rake tasks test

---------

Co-authored-by: barbara oliveira <[email protected]>
Co-authored-by: Lucie Grau <[email protected]>
Co-authored-by: Quentin Champenois <[email protected]>

* feat: Clear minio s3 bucket (#612)

* feat(Docker): Add minio service

* feat(rake): Add new tasks to cleanup s3 bucket

* fix: Add S3 purge rake task

* fix: S3 Bucket endpoint for docker local

* fix(rake): Active storage clear orphans job

* fix(sidekiq): Add sidekiq configuration

* fix: Logger for active_storage.rake job

* fix: Prevent duplicated ActiveRecord Query

* fix: Prevent error on PP sort with end_date nil (#626)

* fix: update to handle processes without start_date or end_date

* fix: update sort in controllers

* test: update test with process without end date

* refactor: optimize queries in controllers and update tests

* refactor: update sort processes in controllers

---------

Co-authored-by: Lucie Grau <[email protected]>

* bump: Module Spam Detection to 4.1.2 (#630)

* feat: Author notification on proposal publication  (#620)

* add notification with eventmanager

* base to watch the CI and see files on github

* fix translation key & notififaction displaying

* add send_pubication_notification to right file

* start test rspec

* continuing rspec

* potential final test file

* fix: Merge proposal command and anonymous proposals

* fix: Change ProposalPublishedEvent to SimpleEvent

* fix: Proposal Published Event

* fix: Push FR locales

* test: Add specs for proposal_published_event

* update methode & file name

* adjust trad key

* adjust test file & update name of test file

* lint

* lint fr trad key

* lint

* update trad key order

* lint

* correct trad key link in method

* correst rspec

* fix rspec

* fix: Notification small title

* lint(rubocop): Fix offenses

* ci: Exclude BeEq Rubocop rule

* clean

* fix CI

* delete test file

* add ignore trad key in i118n-tasks.yml to fix CI

* update text syntaxe via trad key

* add '' in fr trad key

---------

Co-authored-by: Quentin Champenois <[email protected]>

* revert: "fix: Flash message on proposal limit per user reached (#609)" (#634)

This reverts commit 28003b5.

* bump: Fix geocofing on homepage interactive map (#635)

* feat: Add module emitter (#633)

* feat: Addition of the emitter module

* fix: Fix failing specs

* Fix/backport decidim awesome slowness on proposals index page (#631)

* add env variable

* add UUID and IP to logs

* add weighted voting configuration

* add secrets for weighted voting

* update test to fix CI

* continue fix test file

* fix CI

* fix CI

* clean spec

* clean and add test

* backport: Addition of sortable scopes using drag and drop  (#632)

* backport: Backport the Drag & Drop on Scopes on BackOffice

* fix: Fix locales that were not normalized or misplaced

* test: Add some specs about the check_boxes_tree_helper sort

* test: Add specs releated to the backport scopes management

* fix: Fix wrong locales

* fix: Add the weight sorting on select input of scopes

* fix: Add missing locale

* feat: Allow to choose notification settings when attachment added (#627)

* fix: Override Create attachment admin command

* fix: Force email notification on attachment event

* fix: Override attachment form

* feat: Allow admins to toggle notification sending

* refactor: Remove override and add extend

* lint: Fix rubocop offenses

---------

Co-authored-by: Lucie Grau <[email protected]>

* fix: Remove caching from the geocoding elements to avoid map not reloading when refreshing (#638)

* backport: Reorder scopes in meetings (#639)

* fix: Scopes can't be updated in BO (#640)

Co-authored-by: Lucie Grau <[email protected]>

* backport: Add layer of security and download p7zip-full lib on docker

---------

Co-authored-by: stephanierousset <[email protected]>
Co-authored-by: Quentin Champenois <[email protected]>
Co-authored-by: ’Barbara Oliveira <[email protected]>
Co-authored-by: Quentin Champenois <[email protected]>
Co-authored-by: Alexandru Emil Lupu <[email protected]>
Co-authored-by: Lucie Grau <[email protected]>
Co-authored-by: barbara oliveira <[email protected]>

Dockerfile

@@ -6,7 +6,7 @@ ENV RAILS_ENV=production \
 WORKDIR /app
 
 RUN apt-get update && \
-    apt-get -y install libpq-dev curl git libicu-dev build-essential && \
+    apt-get -y install libpq-dev curl git libicu-dev build-essential p7zip-full && \
     curl https://deb.nodesource.com/setup_16.x | bash && \
     apt-get install -y nodejs  && \
     npm install --global yarn && \
@@ -41,7 +41,7 @@ ENV RAILS_ENV=production \
     RAILS_LOG_TO_STDOUT=true
 
 RUN apt update && \
-    apt install -y postgresql-client imagemagick libproj-dev proj-bin libjemalloc2 && \
+    apt install -y postgresql-client imagemagick libproj-dev proj-bin libjemalloc2 p7zip-full && \
     gem install bundler:2.4.9
 
 WORKDIR /app

Dockerfile.local

@@ -8,7 +8,7 @@ ENV RAILS_ENV=production \
 # Install common dependencies
 RUN apt-get update -q && \
     apt-get install -yq --no-install-recommends \
-    libpq-dev curl git libicu-dev build-essential openssl && \
+    libpq-dev curl git libicu-dev build-essential openssl p7zip-full && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -66,7 +66,7 @@ WORKDIR /app
 # Install runtime dependencies
 RUN apt-get update -q && \
     apt-get install -yq --no-install-recommends \
-    postgresql-client imagemagick libproj-dev proj-bin libjemalloc2 && \
+    postgresql-client imagemagick libproj-dev proj-bin libjemalloc2 p7zip-full && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 

app/services/decidim/download_your_data_exporter.rb

@@ -30,7 +30,7 @@ def export
       save_user_data(tmpdir, user_data)
       save_user_attachments(tmpdir, user_attachments)
 
-      SevenZipWrapper.compress_and_encrypt(filename: @path, password: @password, input_directory: tmpdir)
+      Decidim::SevenZipWrapper.compress_and_encrypt(filename: @path, password: @password, input_directory: tmpdir)
     end
 
     private
@@ -59,6 +59,9 @@ def save_user_data(tmpdir, user_data)
         next if exporter_data.read == "\n"
 
         file_name = File.join(tmpdir, "#{entity}-#{exporter_data.filename}")
+
+        dir_path = File.dirname(file_name)
+        FileUtils.mkdir_p(dir_path) unless Dir.exist?(dir_path)
         File.write(file_name, exporter_data.read)
       end
     end
@@ -70,7 +73,8 @@ def save_user_attachments(tmpdir, user_attachments)
 
           blobs = attachment.is_a?(ActiveStorage::Attached::One) ? [attachment.blob] : attachment.blobs
           blobs.each do |blob|
-            Dir.mkdir(File.join(tmpdir, entity.parameterize))
+            dir_path = File.join(tmpdir, entity.parameterize)
+            Dir.mkdir(dir_path) unless Dir.exist?(dir_path)
             file_name = File.join(tmpdir, entity.parameterize, blob.filename.to_s)
             blob.open do |blob_file|
               File.write(file_name, blob_file.read.force_encoding("UTF-8"))