This is a really strange bug. I'm guessing it's something with the encoding that's sent to the server, but I can't figure out how.
To reproduce, you need two python files, one (importer.py) that imports the other (connect.py) file which contains the actual cassandra-driver code. They also need to be running inside a docker container. (For reasons I don't know. However this happens to be convenient for quickly and reliably reproducing)
My real setup is much more complicated than this, but I've reduced it as much as possible to help others debug it in a self contained environment. Note: I replaced my public IPs with $CASSIP1 through $CASSIP6
Here's a very simple Dockerfile that can reproduce the bug-- nothing special:
Here's the shortest cassandra connection code I could use with our cluster, to reproduce it:
To reproduce, put all these files in a directory, then build and run:
And finally. here's the error:
Now the REALLY strange part. I found a "fix" that makes no sense. Add this line to the top of importer.py:
And it works perfectly!
This bug was very ellusive to say the least. It doesn't happen outside of docker, as far as I can tell (despite having same kernel and python version). We've reproduced it on many different machines with different docker versions and on various docker images. (even tried on another network)
The first clue that led me to find that calling `"".decode("utf-8")` magically fixed it, was that if I imported tornado, pandas, or pika beforehand, it would work. I literally gutted pika and tornado until I found a single `re.compile(str)` which first decoded the string to utf-8. Commenting this out prevented the importing of tornado from fixing it.
Also, it's not unique to asyncore, we typically run on libevent. I just removed that and many other configurations from the equation while debugging.
If it would be helpful in debugging, I can create a github repository, which has everything ready to go to reproduce the error.
Reproduced in cassandra-driver 3.4.0 and 3.4.1. It seems that 3.3 and lower are unaffected.
Docker version 1.11.1, build 5604cbe
Python and kernel versions inside docker: