CamelHttpsSinkConnector version 4.4.2 , 4.4.3 , 4.8.0 do not support request redelivery

[KevinMeng4662]

hi @oscerd, @valdar or Team,

I am trying to use the CamelHttpsSinkConnector to fulfill that sync the Kafka events to the remote Endpoint.

There are some properties in the document https://camel.apache.org/camel-kafka-connector/4.0.x/user-guide/basic-configuration.html

camel.error.handler
camel.error.handler.max.redeliveries
camel.error.handler.redelivery.delay

I want to use these properties to implement the blocking retry that if the http request encounter exception will do redelivery and redelivery times are based on camel.error.handler.max.redeliveries value.

But the verification result not meeting our expectation, when the Http request encounter 400 error will not do redelivery.

My question is that whether this function is available for CamelHttpsSinkConnector or not?

Thanks

[oscerd]

What camel.error.handler are you setting?

[KevinMeng4662]

What camel.error.handler are you setting?

camel.error.handler: default, the whole configuration like below:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: camel-https-sink-connector
  labels:
    strimzi.io/cluster: kafka
spec:
  class: org.apache.camel.kafkaconnector.https.CamelHttpsSinkConnector
  tasksMax: 2
  config:
    topics: "xxxxxxxx"
    camel.sink.path.httpUri: XXXXXX

    camel.sink.endpoint.httpMethod: POST
    camel.sink.unmarshal: jackson
    camel.dataformat.jackson.content-type-header: false
    camel.map.headers: true
    camel.sink.endpoint.useSystemProperties: true

    camel.sink.endpoint.throwExceptionOnFailure: true
    
    camel.error.handler: default
    camel.error.handler.max.redeliveries: 3
    camel.error.handler.redelivery.delay: 2000

   
    errors.deadletterqueue.context.headers.enable: true
    errors.deadletterqueue.topic.name: camel-https-sink-connector-remote-retry1
    errors.deadletterqueue.topic.replication.factor: 1
    errors.log.enable: true
    errors.log.include.messages: true
    errors.retry.delay.max.ms: 10000
    errors.retry.timeout: 300000
    errors.tolerance :  none
    #errors.tolerance: all

[oscerd]

Try to use "no" instead of default

[oscerd]

Also what is the version you're using?

[KevinMeng4662]

Also what is the version you're using?

the latest version 4.4.2

[oscerd]

Try with "no"

[KevinMeng4662]

Try with "no"

Still not working as expectation, according to the configuration as least delivery 4 times, but actually only deliver once.
The log as below:

java.base\/java.lang.Thread.run(Thread.java:840)","exception_class":"org.apache.camel.http.base.HttpOperationFailedException","exception_message":"HTTP operation failed invoking https:XXXXXXXXXX\/ with statusCode: 400"},"source_host":"connect-1","method":"log","level":"ERROR","message":"Failed delivery for (MessageId: C5132132CFAE6BB-0000000000000000 on ExchangeId: C5132132CFAE6BB-0000000000000000). Exhausted after delivery attempt: 1 caught: org.apache.camel.http.base.HttpOperationFailedException: HTTP operation failed invoking https:\/\/xxxxxxxxx\/ with statusCode: 400\n\nMessage History (source location and message history is disabled)\n---------------------------------------------------------------------------------------------------------------------------------------\nSource ID Processor Elapsed (ms)\n route9\/route9 97\n\nStacktrace\n---------------------------------------------------------------------------------------------------------------------------------------","mdc":{"connector.context":"[camel-https-sink-connector|task-0] "},"@timestamp":"2024-07-18T08:09:29.942Z","file":"CamelLogger.java","line_number":"205","thread_name":"task-thread-camel-https-sink-connector-0","@version":1,"logger_name":"org.apache.camel.processor.errorhandler.DefaultErrorHandler","class":"org.apache.camel.spi.CamelLogger"}
{"@timestamp":"2024-07-18T08:09:29.942Z","source_host":"connect-1","file":"CamelSinkTask.java","method":"put","level":"WARN","line_number":"213","thread_name":"task-thread-camel-https-sink-connector-0","@version":1,"logger_name":"org.apache.camel.kafkaconnector.CamelSinkTask","message":"A delivery has failed and the error reporting is enabled. Sending record to the DLQ","class":"org.apache.camel.kafkaconnector.CamelSinkTask","mdc":{"connector.context":"[camel-https-sink-connector|task-0] "}} 

[oscerd]

I don't know when I'll have time to reproduce.

[KevinMeng4662]

I don't know when I'll have time to reproduce.

Ok, Thanks. Once confirm please update in here.

[KevinMeng4662]

@oscerd

How about your reproduce result?

[oscerd]

I don't have the time this period

[KevinMeng4662]

@oscerd Hope you are well, not sure whether you have time now to confirm redelivery function now.

[oscerd]

Hello, no I don't have time yet.

[KevinMeng4662]

@oscerd Just used the CamelHttpsSinkConnector version 3.21.0 to do the testing, we got the expectation result. (ERROR [CamelHttpsSinkConnector|task-0] Failed delivery for (MessageId: C66F50ECDC25A89-0000000000000000 on ExchangeId: C66F50ECDC25A89-0000000000000000). Exhausted after delivery attempt: 4 caught: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired
)

So the version 4.4.2 or 4.4.3 may be injected bug for that.

[KevinMeng4662]

@oscerd and Team,

Just did the verification for CamelHttpsSinkConnector version 4.8.0, still not work as expectation.
And there are so many CVE issues for older version 3.21.0 that work as expectation.

Could you please plan to fix this issue with high priority.

Thanks

[oscerd]

There are no high priority issues, we do what we can in the time we have. If you are in a hurry,fork the project and the required subproject, do your fixes and use that.