Accept iterators/generators for execute_concurrent()

Description

Currently, execute_concurrent() and execute_concurrent_with_args() convert the passed in statements to a list. It would be better to handle iterators/generators naturally.

As suggested by the TODO comment in concurrent.py, special care needs to be taken to track the correct index for statements. This means results should be stored as a list of (index, result) tuples. Before returning the results, this list would be sorted and then processed to discard the index.

The other challenge mentioned in the TODO comment (figuring out when the last operation has finished) can be accomplished with a countdown latch. See the patch for CASSANDRA-7405 for an example of this.

Environment

None

Pull Requests

None

Activity

Show:
Frens Jan Rumph
March 30, 2015, 2:14 PM

Great, hope it pans out!

Again, if I need to polish anything feel free to 'put me to work'

I wasn't 100% confident about the 'design' of the 'API'. It's nearly a drop in replacement, but I've introduced the silent keyword argument to allow calling execute_concurrent without the need to iterate over the results. If you have some feedback on this design, I'd be happy to turn this around in a pull request if that helps.

Cheers!

Adam Holmberg
July 14, 2015, 9:24 PM
Adam Holmberg
July 14, 2015, 9:26 PM

Test ideas:

  • should be able to generate some unit tests that cover most of this

  • make sure to test out-of-order result arrival

  • may be able to verify we're avoiding materializing inputs and outputs by generating large sequences and verifying peak memory usage

Michael Penick
July 20, 2015, 6:24 PM

Reviewed. Nice design! Added some comments to the PR.

Greg Bestland
July 21, 2015, 9:04 PM
Fixed

Assignee

Unassigned

Reporter

Tyler Hobbs

Fix versions

Labels

PM Priority

None

External issue ID

None

Doc Impact

None

Reviewer

None

Size

None

Pull Request

None

Priority

Major
Configure