System.OutOfMemoryException during Deletion

[podkolzzzin]

I know that OOM exception is exception that in every line of code, and often it falls somewhere else and not in real point of memory overusage.
Today I used LiteDB to put in FileStorage really big file (140MB) and then during attempt of deletion LiteDB falls with OOM.
I investigated the stack and found this line of code:
https://github.com/mbdavid/LiteDB/blob/697b81d64d1d230172769f0901db66af458a2734/LiteDB/Engine/Services/PageService.cs#L142

If I'm not mistaken you are loading all pages and all its content to single array and this operation really requires a huge amount of memory.

I'm sorry that i missed full StackTrace, if I reproduced it again all attach it.

[mbdavid]

Hi @podkolzzzin, yes, now it's using a single array for all pages deleted. And I know that could be use lot of memory if you are deleting lots of documents in a single Delete command.
But, in FileStorage this cloud not happing because each file are splited in many documents (each document has 256kb). So, if you delete a file it's could delete 256kb data pages, clear cache, delete more 256kb.... So, max memory usage must be 256kb". You can see here:

https://github.com/mbdavid/LiteDB/blob/master/LiteDB/Storage/LiteStorage.cs#L209

Can you write a simple console program to reproduce this problem (you could add #483 issue together)

[mbdavid]

Ok, I found an issue when writing file. When initialize LiteFileStream for write, if file already exists, LiteDB first delete all file content using "single-delete-operation", and this operation use all memory. I'm fixing.

[podkolzzzin]

Yes It's really my case!
My application very often rewrites files in FileStorage.

[podkolzzzin]

And it's wrapped by transaction....

[ednsinf]

Issue opened #2541