Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store and Expose RawNode in YTNode #826

Open
4 tasks done
redwoodlid opened this issue Dec 4, 2024 · 3 comments
Open
4 tasks done

Store and Expose RawNode in YTNode #826

redwoodlid opened this issue Dec 4, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@redwoodlid
Copy link

redwoodlid commented Dec 4, 2024

Describe your suggestion

Right now, it is impossible to deserialize and serialize any YTNode, because in most cases the RawNode data being provided get's discarded after the constructor.

If the RawNode used to initialize a YTNode is preserved, it becomes possible to store the RawNode to disk, then to resume the data from disk by passing the RawNode back to YTNode.

Suggestion

Add a public field in YTNode base class.

export class YTNode {

     readonly raw: RawNode;

     constructor(raw: RawNode) {
           this.raw = raw;
     }

    //....
}

Other details

No response

Checklist

  • I am running the latest version.
  • I checked the documentation and found no answer.
  • I have searched the existing issues and made sure this is not a duplicate.
  • I have provided sufficient information.
@redwoodlid redwoodlid added the enhancement New feature or request label Dec 4, 2024
@absidue
Copy link
Collaborator

absidue commented Dec 4, 2024

Instead of doubling the memory use for everyone by keeping two copies of the data in memory, you could call actions.execute yourself and then store the raw response data and pass it to Parser.parseResponse() any time you want to parse it. That also means you'll only have to serialise a single object, instead of each node.

@redwoodlid
Copy link
Author

This doesn't really solve the use case of the following (but let me know if there's alternative way of doing this).

A Playlist can have continuation data, ideally, we would want to store only the continuation token to avoid large data being written to disk. The continuation data can be easily stored/restored from RawNode, without the need to store the entire raw response data (which is really big!) right away. We can simply then use the restored continuation data and get the remaining playlist whenever we like it.

I believe with the current implementation there is no way of achieving this.

RawNode is a perfect media to deserialize and serialize each YTNode at will, instead of the need to deserialize / serialize the whole data every time.

It would be nice if we could manually capture RawNodes required for each individually YTNode if possible (and therefore making most of the entire library easy to pause / resume from disk), but the easiest way and less breaking change seems to be simply capture the raw data in YTNode.

Alternatively a bonus for this, is that it might allow better debugging experience and know what the YTNode actually had captured and parsed after the class has been created.

@redwoodlid
Copy link
Author

Alternatively, have this field on as an optional value for all YTNode.

For example, we can set a global context object if we want to preserve raw data or not.

The default behavior is to discard raw data as fast as possible for better memory footprint,
however, an option (such as an InnerTubeConfig option), can set it to preserve raw data, and only during parsing of that innertube instance will the raw data be preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants