Replica order should be randomized in token aware LBP

Description

Currently the start index is randomized but this means that whenever a failing node is picked as the first replica, the retries and speculative executions will always go towards the same nodes in the same order.

We can see this happening in our duration tests where one node gets double the throughput compared to the remaining alive node.

We should also consider removing hosts in the DOWN state from the query plan returned by this LBP.

Environment

None

Pull Requests

None

Assignee

Unassigned

Reporter

Joao Reis

Labels

None

PM Priority

None

Fix versions

None

External issue ID

None

Doc Impact

None

Reviewer

None

Pull Request

None

Epic Link

Sprint

Size

None

Priority

Major
Configure