Accept iterators/generators for execute_concurrent()
Description
Currently, execute_concurrent() and execute_concurrent_with_args() convert the passed in statements to a list. It would be better to handle iterators/generators naturally.
As suggested by the TODO comment in concurrent.py, special care needs to be taken to track the correct index for statements. This means results should be stored as a list of (index, result) tuples. Before returning the results, this list would be sorted and then processed to discard the index.
The other challenge mentioned in the TODO comment (figuring out when the last operation has finished) can be accomplished with a countdown latch. See the patch for CASSANDRA-7405 for an example of this.
Environment
Pull Requests
Activity
Great, hope it pans out!
Again, if I need to polish anything feel free to 'put me to work'
I wasn't 100% confident about the 'design' of the 'API'. It's nearly a drop in replacement, but I've introduced the silent keyword argument to allow calling execute_concurrent without the need to iterate over the results. If you have some feedback on this design, I'd be happy to turn this around in a pull request if that helps.
Cheers!
Test ideas:
should be able to generate some unit tests that cover most of this
make sure to test out-of-order result arrival
may be able to verify we're avoiding materializing inputs and outputs by generating large sequences and verifying peak memory usage
Reviewed. Nice design! Added some comments to the PR.
PR for tests is open here
