asyncio message chunks can be processed discontinuously

Description

When a connection message is bigger than 4096 (Connection.out_buffer_size), it is split into multiple chunks and pushed in a asyncio Queue. This Queue is a specific non-blocking implementation for asyncio. For this reason, the message/chunks producer needs to be a coroutine and run in the loop.

The way current asyncio reactor is accomplishing this is basically to split the message into chunks and start a asyncio.Queue.put coroutine with run_coroutine_threadsafe. So, all chunks are pushed in the queue by a different coroutine. Due to the nature of an event loop, there is no guarantee that those coroutines will be executed sequentially.... and even less if there a lot of messages to process on that connection.

This results to bad messages sent to the server due to the wrong order of the chunks. (e.g. cannot decode utf8 string error for a insight message)

This bug was discovered by some tests and a big message (a DataStax Insight request). It is not that easily to reproduce unless we start to stress the asyncio loop with a lot of big messages.

Environment

None

Pull Requests

None

Status

Assignee

James Falcon

Reporter

Alan Boudreault

Fix versions

Labels

None

Reproduced in

None

PM Priority

None

External issue ID

None

Doc Impact

None

Reviewer

None

Size

None

Priority

Major
Configure