-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PECO-969] Make sure that DBSQLOperation.fetchChunk returns chunks of requested size #200
Conversation
…ltsHelper into provider of TRowSet Signed-off-by: Levko Kravets <[email protected]>
…vider interface Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
…size Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implementation looks good, but I think that we should raise a ticket for fixing the server issue.
@nithinkdb I doubt it's possible to fix it on a backend - considering all the variation of the results formats (e.g. it's totally impossible for cloudfetch). There is a similar approach implemented for other drivers to ensure consistent batch size |
PECO-969
DBSQLOperation.fetchChunk/fetchAll
methods suport amaxRows
parameter which should define a chunk size returned fromfetchChunk
. This parameters is passed to server which uses it to decide how much rows to return. Unfortunately, sometimes (often?) server returns chunks which size doesn't match requested value (chunks could be even bigger). This behavior confuses users of the library, which expect that chunk size will be equal tomaxRows
(or probably less for a last chunk). We used to explain users how things works (e.g. #155 (comment)), but eventually we had to fix this behavior, especially considering that other connectors already do this right.The proposed solution is similar to the one implemented in Python connector. Instead of returning raw chunks, we collect records in buffer until we collect enough, and then slice and return a part of this buffer. Remaining records are kept for the next
fetchChunk
call.maxRows
option forDBSQLOperation.fetchChunk
DBSQLOperation.fetchAll
- since it collects all the data anyway, there's no need to use intermediate buffer. Using raw chunks will optimize memory consumption