[ForVic]

What changes were proposed in this pull request?

JIRA: https://issues.apache.org/jira/browse/CELEBORN-2499

Show the slots reserved on each worker for every Celeborn shuffle on the Celeborn tab of the Spark UI and the History Server. This is the "Shuffle Assignments" follow-up planned in CELEBORN-2415. The driver-side LifecycleManager already knows the assignment, so no executor instrumentation is needed.

• client: once a shuffle is registered, LifecycleManager reports the slots reserved on each worker to a ShuffleAssignmentListener, together with the Celeborn shuffle id and the application (Spark) shuffle id. With stage rerun enabled, the two ids differ. SparkShuffleManager.registerShuffleAssignmentListener hands the listener to its lazily created LifecycleManager.
• spark-ui:
  • CelebornPlugin registers the listener and posts a CelebornShuffleAssignmentEvent per shuffle, so applications without the plugin post nothing. Registration happens in registerMetrics because Spark 4 initializes the shuffle manager after the driver plugins' init.
  • CelebornListener persists two things, for both the live UI and History Server replay:
    • a compact per-shuffle summary, retaining up to spark.ui.retainedStages shuffles;
    • the shuffles and slots of each worker over all shuffles, which stay exact after old shuffles are dropped.
  • The tab gains a Shuffle Assignments summary line and two collapsed tables:
    • Shuffle Assignments: Spark and Celeborn shuffle ids, workers, reserved slots, and slots per worker as min / avg / max.
    • Worker Slots: the shuffles and reserved slots of each worker.

Why are the changes needed?

The Celeborn tab shows nothing about which workers serve an application's shuffles or how evenly their slots are spread, which is needed to understand worker hotspots for a specific application.

Does this PR resolve a correctness bug?

• Yes

Does this PR introduce any user-facing change?

• Yes

The Celeborn tab gains the Shuffle Assignments summary line and two collapsed tables. Nothing changes for applications that don't enable CelebornPlugin. Some cases are not covered:

• Slots that partition splits or revives reallocate later are not included.
• Shuffle managers that create their own LifecycleManager instead of using Celeborn's SparkShuffleManager (e.g. Gluten's) report no assignments.

Same workload and cluster (Spark 3.5.9, 3 workers, replication enabled).

Before (main):

After, with the collapsed sections expanded:

How was this patch tested?

• CelebornListenerSuite:
  • per-shuffle summaries and per-worker totals are persisted from the event;
  • per-worker totals stay exact beyond spark.ui.retainedStages;
  • the History Server KVStore serializer round trip works.
• CelebornPluginSuite: a local SparkContext with CelebornPlugin and SparkShuffleManager has the listener registered, and reported assignments reach the UI store.
• LifecycleManagerSuite: against a mini cluster, the listener receives both shuffle ids and primary plus replica slots for every partition.
• ./build/mvn -P<profile> -pl client-spark/spark-3-ui test passes for spark-3.0, spark-3.5 and spark-4.0.
• Manual: the screenshots above, from a local cluster with 3 Celeborn workers, Spark 3.5.9 standalone and the History Server, which shows the same data on replay.

[ForVic]

cc @zaynt4606 if you could take a look when you get the chance

[zaynt4606]

LGTM