Implement table component for display of more than 100 rows

[oyo]

Description

Currently the table component is based on the x-data-grid component which is limited to 100 data rows on one page. With such a limit an infinite scrolling mechanism can't be implemented. Additionally a lot of features from x-data-grid are useless in such a scenario - like filtering, sorting or searching the loaded items. Create a table component which can render unlimited data.

Acceptance Criteria

• Table renders a collection of more than 100 items
• In the infinite scroll scenario on scroll to bottom the table fetches additional pages of data from the backend and appends them to the end where the final number of items is higher than 100

Additional Information

Since we don't want to break existing code the recommended approach is to keep the existing table and introduce a new one with similar look and feel for the cases with more than 100 rows.

[shahmargi12]

Hello @oyo

I have a suggestion: since we have more than 100 data entries in our table, we could implement virtualization to enhance the user experience. This would ensure a smoother interaction with the component, regardless of the amount of data displayed.

[oyo]

I have a suggestion: since we have more than 100 data entries in our table, we could implement virtualization to enhance the user experience. This would ensure a smoother interaction with the component, regardless of the amount of data displayed.

@shahmargi12 I'm not sure about what you mean with virtualization in that context - can you explain your idea in more detail? In the past we have discussed ideas to improve the user experience. The most promising approach was caching all the data that is more often read than changed like apps, users, use cases, etc. in a Solr or Elasticsearch system and query much faster from there. On DB update that cache must be refreshed of course.

[shahmargi12]

Sure @oyo.

By virtualization, I meant optimizing the front-end rendering of large data sets. Without it, React can slow down when trying to render large amounts of data. Virtualization allows us to render only the portion of data that is visible on the screen at any given time, rather than rendering the entire dataset (which could be thousands of rows).

As the user scrolls down, new rows are loaded dynamically, and the rows that are no longer visible are removed. This approach reduces the workload on the browser's DOM, making the page faster and more responsive.

Many table or grid libraries, such as X-Data-Grid (from Material-UI, though it provides row virtualization only in the pro plan), AG Grid, or Material React Table, come with built-in virtualization features. These libraries are specifically designed to handle large datasets efficiently by only rendering what’s visible, avoiding performance issues caused by loading everything at once.

The main goal is to avoid performance bottlenecks caused by the browser trying to render all the data at once, especially when dealing with large datasets (over 100 or even thousands of records).

[oyo]

I support your idea. However I don't think we can do it with x-data-grid as our limitation is the free plan which supports pagination with a limit of 100 rows per page. As an Eclipse open source project we are not allowed to use the commercial pro or premium plans without that limit: https://mui.com/x/react-data-grid/#mit-license-free-forever

Possible options I see here are:

• going back to pagination without infinity scroll
• switching to the commercial plan in your fork only
• evaluating and eventually switch to a different data grid library
• write our own data grid component from scratch which implements the features we need only (infinity scroll, etc.).

Personally I'd vote for the last one: custom implementation. The effort should not be too high as we don't use a lot of features. Things like search, filter, sort, etc. are not required on FE side - they have to come from the backed anyway as it doesn't make sense to apply them on partially loaded data.

@shahmargi12 not sure if you are still on the project - what's your opinion here?

[MaximilianHauer]

@ybidois /@dorkacbox / @Cofinity-UX fyi

[shahmargi12]

Hello @oyo
I believe that evaluating and switching to a different library would be the best choice, as it can save developers significant time by offering a ready-made solution. Additionally, these libraries often include extra features that could be beneficial for future development.

However, if time is not a constraint and the project requirements are well-defined, building the data grid from scratch could be a viable option.

I’m not actively involved due to other priorities but am happy to assist where I can 😃 .

[MaximilianHauer]

as aligned with @Cofinity-UX cofinity will internally discuss if a switch of licenses is an option for cofinity .

[Cofinity-UX]

@MaximilianHauer we evaluated internally and decided to keep the table as it is. From user perspective, displaying more than 100 items in the table is a very very unlikely edgecase. Humans can't even process that amount of data.

In order to make the data usable, we should focus on well implemented table search, filtering and sorting. The goal of using a table is making sense of the displayed data. So it's about manipulating the content to the expected result instead of displaying everything. Of course the total number of items should be displayed and navigation between the pages should be seamless.

cc: @ybidois