fix mount ecs ec2

[daidokoro]

• [cds-724] re-add sync workflow (Add cloudtrail integration via cloudformation #10) * initial commit

• Example of receiving the S3 payload

• newline_pattern

• sending to coralogix

• sending as batch

• Put records in ExportLogsServiceRequest

• sent to coralogix working

• added gzip at source

• Limit batch size to 500 logs

• Adding integration_type

• Implement splitting with fancy regexes

• WIP Blocking Pattern

• Blocking Working

• wip sampling blocking_pattern

• WIP DynamicAppName

• WIP dyn app

• Dynamic application/subsystem name

• added cloudtrail support WIP

• Cloudwatch working

• WIP Severity

• Severity fix

• Severity Matcher

• Resource per app/subsystem

• git ignored cargo.lock

• cloudtrail working

• initial sns

• sns working

• Working SNS

• Split initial

• Split initial

• vpcflow working

• Cloudwatch work in progress

• Cloudwatch work in progress

• Refactor/split code

• initial template

• added sam makefile

• added workfllows

• updated triggers

• adjust workflow

• added requirements txt

• add ssh agent

• add proto

• add sudo

• change lambda name

• change name

• add license

• updated Author

• added version change check

• fix exit

• added job depenedencies

• adjust push

• [skip-version-check]

CoralogixRegionMap the log URL should actually be the domain and than we need to build CORALOGIX_URL from it with prefix/sufix the reason for it are 2 like in here
we wanted just one place to change if the URL changes (the domain is not)
the support of CustomDomain that i see currently missing from the setting
the URL should be https://ingress.coralogix.com:443 so currently prefix is https://ingress domain is coralogix.com sufix :443` (not sure if sufix is needed or can stay empty for now
CoralogixRegion should custom option and this should be the default setting unless someone select otherwise something similar to what we did for firehose
PrivateKey should be changed to ApiKey
not sure that we have MaxLength for application or subsystem names (i know we have it now but not sure we have limit on our backend)
we need to add ParameterGroups in the template (like we discuss so the template will be more readable) we also have it in some of the current templates
not sure about the labels list (i think we should remove athena,query,results)
Description i think should be Send logs to Coralogix from AWS since we will have also cloudwatch
not sure that we need BufferSize i dont think we have any
what about FunctionArchitecture we should give option to run on x86_64 or arm64 (default should be arm64)
the lambda description should be generic not like now
CustomResourceLambdaTriggerFunction should be only in case it is S3, we need to add some condition so when we will have cloudwatch this will not be trigger so need to think how to create it and also support the cloudwatch parmas i think it is only CloudWatchLogGroupName
for S3KeyPrefix/S3KeySuffix how do we set them per the use case? i.e in cloudtrail they should be set by default
missing SNSTopicArn param so if customer want to use his topic he should add the ARN and we need to register to this topic
missing SamplingRate
missing BlockingPattern
missing NewlinePattern which is different syntax than what we currently have like juan explained
why the python runtime for the CustomResourceLambdaTriggerFunction is 3.9 and not latest 3.11
we need probably a new param IntegrationType so they can set
S3/Cloudtarail/Cloudwatch/VPC-Flow/etc...

• [skip-version-check] fix variable declaration

• [skip-version-check] removed AWS::LanguageExtension transform

• [skip-version-check] fix dependson

• [skip-version-check] fix code uri

• template updated

• adjust template

• Cloudwatch working

• thread_id WIP

• [skip-version-check] fixed UseSNSTopicARM condition and updated function
  variables

• gitignore

• Add thread_id to log_record

• [skip-version-check] Added metadata for grouping parameters. Set the
  Newline Param to empty by default. Set the S3KeyPrefix var to empty by
  default.

• [skip-version-check] update lambdafunction key name in store workflow

• update sync workflow

• change workflow name

• add -f flag

• add signing -S

• [skip-version-check] updated suffix

• updated s3 suffix

• [skip-version-check] updated

• Send batches in parallel

• [skip-version-check] added rule assertion for parameter validation, updated metadata label grouping

• LogMode::Vanilla

• [skip-version-check]

• merge fix

• [skip-version-check] update descriptions

• [skip-version-check] update retention description

• WOIP

• Cloudwatch fix

• CW Fix

• fix errors in main

• fix

• Fix Cargo.lock

• dependency update

• add get_api_key_from_secrets_manager

• adjust main function to support fetching secret

• [skip-version-check] added SecretStore variable and fixed Assertion

• added planetscale workflow

• updated

• adjust repo

• added content write

• add sync call

• updated repo name

• dependencies

• fix api key

• [skip-version-check]

Fixed cloudwatchgroupname parameter
Fixed cloudwatch custom resource function parameters
Removed S3 trigger for SNS topic deployment
Added Stackid to Secret creation name

• addded custom sync

• remove separate sync workflow

• fix merge conflicts

• re-add sync workflow

• remove sync step

---

Co-authored-by: juan-coralogix [email protected]
Co-authored-by: Rafał Sumisławski [email protected]
Co-authored-by: juan-coralogix [email protected]

• Update endpoints (Update endpoints #91)
• Change the sync action[CDS-865] (Create replace_codeuri.sh #17) * add custom to the region mapping and change the sync action to replace the codeuri

Signed-off-by: guyrenny [email protected]

• update the tempalte.yaml to incloud custom in region mapping

Signed-off-by: guyrenny [email protected]

---

Signed-off-by: guyrenny [email protected]

• CDS-799 Feature/s3_csv (Add readme file and remove duplicates template files #18) * S3_CSV Feature added
• [skip-version-check] CDS-799 Fix Max Memory in template (Change integration name from yml to yaml #19)
• update sync action to work with rust lambda (update sync action to work with rust lambda[CDS-870] #92)
• Update template.yaml (Update README.md #21)
• Vpc support [skip-version-check] (Create sync action and replace_template file #23) * FEAT - Support for deploying into VPC

• FIX - Remove second LogsPerBatch

• FIX - Restore Lambda MaxValue to 10240

• [skip-version-check] CDS-799 SNS Text function (Change cloudwatch lambdafunctionSsm codeuri #22)

• [skip-version-check] CDS-799 SNS Text function

• Update src/main.rs

unwrap remove

Co-authored-by: Rafał Sumisławski [email protected]

• Update src/process.rs

Co-authored-by: Rafał Sumisławski [email protected]

• Fix clone issue

---

Co-authored-by: Rafał Sumisławski [email protected]

• add variable max_elapesed_time (made changes to test the sync action #25)

• FEAT - Add PrivateLink conditionals and validation

• CHORE - Update CHANGELOG.md

---

Co-authored-by: juan-coralogix [email protected]
Co-authored-by: Rafał Sumisławski [email protected]
Co-authored-by: juan-coralogix [email protected]

• Updated readme file with all params (Add resource-metadata to synchronizing.yaml #27)
• [skip-version-check] Update README.md (fix a bug in synchronizing action #28) * [skip-version-check] Update README.md

• Update README.md

• Update README.md

• Update synchronizing.yaml (fix sync action issue[CDS-870] #94)
• Sync tets[CDS-870] (add publish action and update the s3-archive module[CDS-502] #31) * Update template.yaml

• Update template.yaml

• [skip-version-check] CDS-719 add csvdelimiter variable (Ecs fargate support #35) * [skip-version-check] add csvdelimiter variable

• [cds-724] Update cloudformation template ([cds-305] Update ECS-EC2 cloudformation template #34) * [skip-version-check]

• s3 bucket name should not be mandatory need to update the regex probebly to validate it is minimum 3 latters

• Add note for prefix/sufix that it is not working with sns and add condition to validate it is empty when sns is used so it will not confuse customers

• change serverless function name to coralogix-aws-shipper

• S3KeySuffix for VPC flowlog default suffix set to .json.gz and default Prefix set to AWSLogs

• removed LogsPerBatch param

• added note to description saying this a beta release

• [skip-version-check] added badge for publish workflow

• [skip-version-check] adjust branch location

• [skip-version-check] updated s3 param

• [skip-version-check] adjust discription for s3bucket

• [skip-version-check] fix license badge

• [skip-version-check]removed default prefix for s3 integration

• [skip-version-check] adjust cargo version to match template

• [skip-version-check] add badged

• [skip-version-check] updated cloudwatch filter name

• [skip-version-check] re-add event on failure to cloudwatch lambda

• [skip-version-check] change badge name

• adjust badges (update sync action to work with firehose integration and trigger the action[CDS-307] #38)
• Readme add
• [cds-724] Update template syntax ([cds--619] Update cdot Image description #43) * [cds-724] Update cloudformation template

• Update template syntax

• [cds-724] Update template syntax

• [cds-724] Update readme (Fix firehose integration alignment #44)
• fix bug in replace_template.sh (fix bug in replace_template.sh #95)
• [skip-version-check] Add data source and user agent headers (Add Collector Metrics to ECS-EC2 otel implementation #45) * Add data source and user agent headers

• Adjust readme and CF/SAM template

• Fixed version validation

• [skip-version-check] CDS-719 - Blocking Fn for all S3 (log.file.path removal from coralogix exporter subsystem attributes #48) Blocking Fn for all S3
• remove CloudWatch_Metrics_JSON (remove CloudWatch_Metrics_JSON #96)
• [skip-version-check] CDS-719 Update Changelog.yaml (fix for bucketname issues and domain input #54) * Update Changelog.yaml
• [cds-724] Final checks (Guyrenny patch 1 #56)
• [cds-724] Update with small changed (add test line check #61) * Update README.md

• Update template.yaml

• Update CHANGELOG.md

• Update README.md

• Update config.rs

Added Sns support

• Filter the list of lambda functions which should be the subject of metadata collection#feature/NGSTN-464-metadata-lambda-filtering
• Feature/ngstn 464 fix readme table#feature/NGSTN-464-fix-readme-table
• [skip-version-check]change the prefix and suffix to the vpcflow[CDS-719] (added cf text change to trigger #64) * change the prefix and suffix to the vpcflow

• skip cahngelog

• [skip-version-check] CDS-719 Fix/lastone (Test Pr Sync  #66) * changed newline example

• added csv empty line fix

• [cds-719][skip-version-check] Readme update (Test pr branch #69) * Readme update
• [skip-version-check][cds-719] Hide newline and blocking pattern values if not set (Test pr branch #70) * [skip-version-check] added default empty string for NEWLINE_PATTERN and BLOCKING_PATTERN

• [skip-version-check] NEWLINE and BLOCKING pattern env vars made to not show if not set

• Update template.yaml

• Update CHANGELOG.md

---

Co-authored-by: royfur [email protected]

• update version badege
• readme
• [cds-719] Version Update (Update default ECS-EC2 image #1) * updated version

• removed test file

• align cargo.toml version with template version

• changelog

---

Co-authored-by: Concourse [email protected]

• Small fixes in Readme (Update default image name for ECS EC2 #2)
• CDS-897 Elasticsearch to Opensearch Reports#CDS-897-es-opensearch-reports
• Deprecate resource tags#deprecate-resource-tags
• [ecs-fargate] fix CloudFormation template ssm name, use nginx demo (Fix ECS Fargate CloudFormation template ssm name, use nginx demo #93)
• Stop resource-metadata from failing on issue with single lambda function#feature/NGSTN-477-improve-resource-metadata-error-handling
• V0.0.2 ([sc-12791] [skip-ci] adding lifecycle_policy #3) * fix zero file error

• add key on gz error

• changelog

• Update CHANGELOG.md

• variable set

• changlog update

• changed default logging level

• Added Rust Report Card

• Update version to 0.0.2

• Fix default LogLevel

• added file size in s3 logging

• align cargo.toml version with template version

---

Co-authored-by: royfur [email protected]
Co-authored-by: Concourse [email protected]

• [skip-version-check] Fix LogLevel Typo (Updated the Docker Image used to the correct one. #4) * fix typo in template

• changelog

• Fix issue with coralogixApiKey (Fix issue with coralogixApiKey [CDS-867] #97)
• [skip-version-check] fixed readme version link (update default otel conf #5)
• Update README.md (Update README.md #98)
• Correct conversion of tags to attributes#feature/NGSTN-480-fix-tag-to-attribute-conversion
• [cds-902] Integration Testing ([CDS-337] Updated ECS-EC2 template #6) * updated dependencies

• moved internal logic to lib.rs

• initial commit for integration tests

• removed unused imports

• added comments to test macro

• udpate s3 event

• changelog

• changelog

• add bad code

• bad code

• updated dependencies

• refactored integration tests

• Added mock s3 client and removed the need to call aws apis. Add local
  test files, refactored tests

• added formatting

• dependencies

• moved s3 key decoding into s3 handler function directly

• moved s3 key decoding

• added test_s3_event_handler

• move test_handle_s3_event to unit test

• add tests workflow

• add tag step

• Publish v0.0.3 (add test #11) * release v0.0.3

• align cargo.toml version with template version

• align changelog with version

---

Co-authored-by: Concourse [email protected]

• Add CF template for assigning EventBridge policy (Add CF template for assigning EventBridge policy #100)
• Remove platform from code owners (Remove platform from code owners #101)
• Update sync action to skip create_pr job instead of failing (Update sync action to skip create_pr job instead of failing #102)
• Update Image Parameter for Prometheus Auto Discovery template (Update Image Parameter for Prometheus Auto Discovery template #104)
• Bump tj-actions/changed-files from 37 to 41 in /.github/workflows (Bump tj-actions/changed-files from 37 to 41 in /.github/workflows #105)
• Update aws eventbridge template to use role (Update aws eventbridge template to use role #106)
• v0.0.5 (Add Fluentbit ECS Sidecar description #20) * v0.0.5

• align cargo.toml version with template version

---

Co-authored-by: Concourse [email protected]

• Fix eventbridge description (Fix eventbridge description #107)
• Update parameters to track in replace_template.sh script (Update parameters to track in replace_template.sh script #103)
• Add sqs to allowed values in integrationType (Update README.md #21) * add sqs to allowed values in integrationType

• align cargo.toml version with template version

• Update CHANGELOG.md

• Update Cargo.toml version

• align cargo.toml version with template version

---

Co-authored-by: Concourse [email protected]

• Update the bucket role permissions [PLTD-478] (Update the bucket role permissions [PLTD-478] #108)
• Release v0.0.7 (Update privateKey in opentelemetry #24) * Release v0.0.7

---

Co-authored-by: Concourse [email protected]

• Add role name eventbridge (Add role name eventbridge #109)
• Add option to run the action manually (Add option to run the action manually #110)
• Manual trigger bug fix for action publish and sync (Manual trigger bug fix for action publish and sync #111)
• Fix issue with only first member of a multi-member gz file being ungzipped (Add resource-metadata to synchronizing.yaml #27) * Fix issue with only first member of a multi-member gz file being ungzipped

• change version

• [cds-977] add pprof to default otel config ([cds-977] add pprof to default otel config #112)
• cds-979: Add integration test for ELB log (fix a bug in synchronizing action #28) * cargo

• initial test add

• removed metadata label for s3csv

• add elb gz log

• update loglevel

• Fix region typo US --> US1 (Create changelog action and update PR template[CDS-516] #33) * Fix typo in region selection option in template

• Update README.md

• Update CHANGELOG.md

• Update version

• align cargo.toml version with template version

• Update README.md

• Update README.md

• Update template.yaml

---

Co-authored-by: Concourse [email protected]

• S3 archive update role for metrics bucket [CDS-982] (S3 archive update role for metrics bucket [CDS-982] #113)
• Update sync action to add messages in the readme and template files in the CF ([cds-305] Update ECS-EC2 cloudformation template #34) * remove duplicated kinesis option

• Update sync.yml to private messages

• fix a bug with the sync (Ecs fargate support #35)
• [cds-740] Add MSK and Kafka Support (CF Firehose logs metrics #36) * add kafka to cargo.tom

• simplify comebined event serialization

• added kafka_logs process fn

• added kafka test

• template initial commit

• added MSK integration type

• changelog

• added kafka integration type test

• added integration tests for base64

• added support for msk and kafka to cfn template

• updated README with kafka and msk details

• added option to parse base64 kafka values

• added kafka integration type

• added base64 parsing to dependencies

• mdox formatting

• Update src/process.rs

Co-authored-by: Roman Timushev [email protected]

• add parameter group for kafka and msk

• added MSK and Kafka to list of support services, merged kafka and msk configiruation sections

• update dependencies

---

Co-authored-by: Roman Timushev [email protected]

• adjust rule condition ([cds--619] Update cdot Image description #43)
• cds-740 fix template (Fix firehose integration alignment #44) * update template version and adjust kafka/msk param grouping

• align cargo.toml version with template version

• adjust condition

• removed validation from comaseparated list parameters

---

Co-authored-by: Concourse [email protected]

• change kafka param types (Add Collector Metrics to ECS-EC2 otel implementation #45)
• v0.0.12 ([cds-683] remove ecsattribute filters fom ecs-ec2 embedded config #46)
• cds-740 fix kafka-event bug (log.file.path removal from coralogix exporter subsystem attributes #48) * re-added kinesis event type and adjusted kafka event handle

• adjusted all tests to ensure that they call the deserializer, removed internal reverence to underlying types.

• change log

• adjust comments

• Add default to subsystem param

• Updated version

• align cargo.toml version with template version

---

Co-authored-by: royfur [email protected]
Co-authored-by: Concourse [email protected]

• Adding Feature ECR Image Scan Report (Fix changelog for remove log.file.path #49) New ECR Image Scan Event Support added.

---

Co-authored-by: Rafał Sumisławski [email protected]
Co-authored-by: royfur [email protected]

• v0.0.14 release (Add resource-metadata to sync github action #50) * Update CHANGELOG.md

• Update template.yaml

• align cargo.toml version with template version

---

Co-authored-by: Concourse [email protected]

• Deprecation notice athena-s3#Node16-Deprecation
• adjust the name of the pr that the sync action create (adjust the name of the pr that the sync action create #114)
• Update Status to GA v1.0.0
• remove old integrations from the sync (remove old integrations from the sync #115)
• Deprecation Notice for all nodejs16 integration
• move the aws-shipper lambda in the sync to shared folder (move the aws-shipper lambda in the sync to shared folder #116)
• Remove jest and update dependencies in resource-metadata
• OTEL Agent security mount vol scope (OTEL Agent security mount vol scope #117)
• cds-719 fix rc workflow
• set default ecs-ec2 otel to log level warn (set default ecs-ec2 otel to log level warn #119)
• [CDS-1060] Add ses:sendmail policy to use sender identity
• [CDS-1060] semantic version missing
• cds-1050 - add x86 lambda support
• V 1.0.1
• allow sync to run manually from master
• Cds 863
• fix vulnerability in dependencies Update default image name for ECS EC2 #2
• Update README.md (Update README.md #120)
• fix ecs ec2 default config (fix ecs ec2 default config #121)
• changelog
• fix mount

Description

Fixes #

How Has This Been Tested?

Checklist:

• I have updated the relevant component changelog(s)
• This change does not affect any particular component (e.g. it's readme or docs change)