Incremental Co-operative Rebalancing Support for HDFS Connector

[ghost]

kafka-connect-hdfs/src/main/java/io/confluent/connect/hdfs/DataWriter.java

Line 486 in bad81d6

topicPartitionWriters.clear();

Hi Team,
@ConfluentJenkins
Just wanted to know if the HDFS connector has the support for the "Incremental Rebalance". As I can see that "TopicPartitionWriters" Map is being cleared on every close call, which is correct if we consider the stop the world scenario. But in Incremental mode there can be case that only few of the partitions have been revoked and task has some partitions with it to read from, In that scenario it will be throwing NullPointerException at the following part of the code.

kafka-connect-hdfs/src/main/java/io/confluent/connect/hdfs/DataWriter.java

Line 364 in bad81d6

topicPartitionWriters.get(tp).buffer(record);

[ychernysh]

Hi @kkonstantine @ewencp @maxzheng , any plans on this issue in the nearest releases?

[ychernysh]

Hey @kkonstantine @ewencp @maxzheng , I've opened a PR solving this issue: #711. Could you please take a look on it? I cannot add reviewers for some reason...
The 2 things I'm not sure about in this PR is:

• Should we bother about versions of Kafka that do not contain https://issues.apache.org/jira/browse/KAFKA-12487 fix? I haven't tested the fix on such versions
• The big @ewencp's comment saying that we must always close all the partition writers, which I simply ignored and removed in the PR. I didn't understand it honestly

I'd appreciate if you review it.
Thanks!

[yarocher]

I've opened another PR #712 from my personal account, where I signed the CLA. Please use this one.

@kkonstantine @ewencp @maxzheng , could you please take a look?