HDDS-16497. Support OM ThreadLocal context propagation for Ratis read

[ivandika3]

What changes were proposed in this pull request?

While working on HDDS-15089, encountered an NPE issue that was caused by null
Server.getRemoteUser() in OzoneManager#getS3VolumeContext. This reason is that Ratis would call StateMachine#query in a separate thread (under RaftServerImpl#clientExecutor). Therefore, the ThreadLocal in OzoneManager and the CurrCall which is used by Server.getRemoteUser() is not going to be propagated.

This affects most read requests submitted to Ratis (executed in OzoneManagerStateMachine#query) that require using ThreadLocal one way or another.

We need a way to propagate the Server#getCurrcall ThreadLocal context to the StateMachine#query.

Note this suggests that follower read feature is not production ready until this is resolved.

The main mechanism is the introduction of OMRatisRequestContext which is a context mechanism to pass the OM ThreadLocal info OMRequest UserInfo and allows the OzoneManagerStateMachine#query (that is executed in a separate thread) to create am artificial Server.Call with getRemoteUser and getHostInetAddress so that Server.getRemoteUser() will not return null. Additionally, this OmRatisRequestContext also helps to handle propagation to OzoneManager ThreadLocal like stsTokenIdentifier.

Note that OM might need to use the RaftServer#readOnlyAsync (apache/ratis#1448) that might either be executed directly in caller thread (if we use DEFAULT ReadOption) or executed in another thread (if we use LINEARIZABLE ReadOption). Therefore, OMRatisRequestContext should handle context propagation within a thread and across differentthreads (see previousCall and previousS3Context).

I chose this approach since it does not need to change every OM read implementation. However, any suggestions to improve this design is welcome since this is a core OM logic.

This patch also includes some refactoring on OMLockDetailsUtil and S3AuthenticationContext to reduce duplications.

Saw projects like https://github.com/alibaba/transmittable-thread-local to pass the ThreadLocal context across threads, but the current solution should be enough for now. In the future, we should also think whether this ThreadLocal passing mechanisms should be a first-class feature for Ratis (or we can support something like ScopedValues in Ratis).

Generated by: GPT 5.6

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16497

How was this patch tested?

UT and IT.

Clean CI: https://github.com/ivandika3/ozone/actions/runs/35331947876

[adoroszlai]

Please take a look at intermittent test failure, only seen in this PR so far.

[ERROR] Tests run: 64, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 51.18 s <<< FAILURE! -- in org.apache.hadoop.fs.ozone.TestOFS
[ERROR] org.apache.hadoop.fs.ozone.TestOFS.testGetFileStatusUsesSingleOmRpc -- Time elapsed: 0.028 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <500> but was: <501>
	at org.apache.hadoop.fs.ozone.AbstractRootedOzoneFileSystemTest.testGetFileStatusUsesSingleOmRpc(AbstractRootedOzoneFileSystemTest.java:1858)

[ivandika3]

@adoroszlai Thanks for checking this, I have attached the possible race on HDDS-16513 related to trash calling getFileStatus, which causes the assertion to fail.

Should not be directly related to this patch.

[peterxcli]

other lgtm, thanks for the update!

[peterxcli]

+1 again, thanks for the update!

[ivandika3]

Thanks @peterxcli for the review and @adoroszlai for helping with the merge conflicts.

[szetszwo]

@ivandika3 , @peterxcli , thanks for working on this! (and sorry that I was not able to reviw.)

[ivandika3]

@szetszwo Thanks for always thoughtfully reviewing the PRs. I imagine it's not easy (and time consuming) to review numerous PRs especially with recent increase in volume of contributions.