RHELAI-2756 Adding Adaptive Throttling

[eshwarprasadS]

Why Adaptive Throttling?

The Problem:

If too many requests are sent concurrently to the server, it might:

• Run out of GPU memory (VRAM) or other resources.
• Fail to respond to requests within the timeout period.
• Reducing concurrency manually (batch_num_workers) is a workaround, but it's static and not adaptive to server conditions.

The Solution:

• Monitor the server's behavior (e.g., response success or failure).
• Dynamically increase or decrease concurrency based on server feedback.

Key Components in the Code

The AdaptiveThrottler Class

This class encapsulates the logic for adjusting concurrency:

Parameters:

        min_workers: Minimum number of concurrent workers (never go below this).
        max_workers: Maximum allowed concurrent workers, matches the num_cpus from instructlab core.
        initial_workers: Starting number of workers, 50% of max to begin with.
        tolerance: Fraction to reduce workers when errors occur.