[Compatibility] Add hash expiration - HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HTTL, HPTTL, HEXPIRETIME, HPEXPIRETIME, HPERSIST and HCOLLECT #864
Conversation
|
@badrishc Can you quick look at this draft PR and let me know if I am going on the right path? Also, you will see a lot of TODO comments, I would like to get your opinion and help on some of those. Add your comments on those TODO items if possible. Thanks |
|
For maintaining the performance bar, we may need to: (1) Have a separate small PR to main, that adds BDN for more of the hash APIs for perf test (i.e., the ones that this PR might potentially regress). We only have one right now here: (2) Use those BDNs to compare main versus this PR, to ensure that there is no regression |
|
I think it's ready for review. Let the review comments flood it 😄 @badrishc I am working on creating a separate PR for adding more hash APIs in BDN |
main branch results as of commit
this PR branch results as of commit
Difference
For me, Variant looks minimal with the PR but I like to know your opinion. I will analyze why there is an additional allocation for HKeys, HGetAll, HVals commands |
|
I have to remove the This PR as of
Difference
For me, HMSet is the only one that looks outstanding. Update: Not sure what's going on in HMSet commands. Even after reverting everything back in HMSet commands flow, the performance is still slower than the main. But changing other parts of the code affects its performance like using |
|
Fixed all review comments except the one where we need to find the size of the new objects |
|
Please rereview this PR, closed all outstanding items from my end |
| @@ -1546,6 +1546,31 @@ | |||
| } | |||
| ] | |||
| }, | |||
| { | |||
| "Command": "HCOLLECT", | |||
There was a problem hiding this comment.
This should be added to GarnetCommandsInfo.json (in CommandInfoUpdater)
| @@ -2751,6 +2751,12 @@ | |||
| } | |||
| ] | |||
| }, | |||
| { | |||
There was a problem hiding this comment.
This should be added to GarnetCommandsDocs.json (in CommandInfoUpdater)
| if (!isExpired) | ||
| { | ||
| hash.Add(item, value); | ||
| InitializeExpirationStructures(); |
There was a problem hiding this comment.
Why call this method at every iteration? Have an _isExpirationStructuresInitialized field to indicate if this has been previously called.
| @@ -57,6 +58,33 @@ public void CanSetAndGetOnePair() | |||
| ClassicAssert.AreEqual("Tsavorite", r); | |||
| } | |||
|
|
|||
| [Test] | |||
There was a problem hiding this comment.
Since you've added serialization logic for a hash object with expiration values, it would be good to test taking a checkpoint of that object + test proper recovery
|
|
||
| private bool Remove(byte[] key, out byte[] value) | ||
| { | ||
| DeleteExpiredItems(); |
There was a problem hiding this comment.
Why call this at every remove, add etc? The overhead is not necessary at every call, right?
| SendAndReset(); | ||
| break; | ||
| default: | ||
| while (!RespWriteUtils.WriteError(CmdStrings.RESP_ERR_HCOLLECT_ALREADY_IN_PROGRESS, ref dcurr, dend)) |
There was a problem hiding this comment.
Where in the implementation do you return a different value?
| { | ||
| if (token.IsCancellationRequested) return; | ||
|
|
||
| if (objectStore is null) |
There was a problem hiding this comment.
This should be outside of the while loop
| @@ -2,6 +2,8 @@ | |||
| // Licensed under the MIT license. | |||
|
|
|||
| using System; | |||
| using System.Diagnostics; | |||
| @@ -17,6 +17,7 @@ | |||
|
|
|||
| namespace Garnet.server | |||
| { | |||
| using static System.Reflection.Metadata.BlobBuilder; | |||
| await Task.Delay(TimeSpan.FromSeconds(hashCollectFrequencySecs), token); | ||
| } | ||
| } | ||
| catch (TaskCanceledException ex) when (token.IsCancellationRequested) |
There was a problem hiding this comment.
Token cancellation does not mean we've reached an error state, it just means that the store wrapper was disposed
Adding support for field-specific expiration in hash set. This PR adds HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HTTL, HPTTL, HEXPIRETIME, HPEXPIRETIME, HPERSIST and HCOLLECT commands to garnet to support this feature.
Fixes: #857