Segmentation fault when using DSE Python Driver

Description

I am running a script in a docker container that copies information from a 3rd party service into our DSE Graph. It is migrating approx 500 entities from one database to the other using the Datastax Python Driver, but I am encountering a segfault error during approximately 95% of the attempts. The segmentation faults are occurring at different times during each run, though it is the same dataset and sorted the same way. Occasionally a script makes it all the way to completion. Attached is the log output from one run that ended in a segfault. I had to break the log into pieces to get under your attachment size limit and have only included the last chunk before the segfault.

Environment

Running in a docker container running Alpine 3.3.3, Python 2.7.12, standard event loop, and cython enabled

Pull Requests

None

Activity

Show:
Owen Nelson
December 8, 2017, 4:35 AM

Seems like adjusting the stack size has sidestepped the issue for us.

Kenneth Lowe
December 8, 2017, 4:39 AM

Thanks Owen! That is correct, I've done a bunch of testing and have had no more segmentation faults since adding "import threading; threading.stack_size(4*80*1024)" to my script.

Jim Witschey
December 8, 2017, 6:21 AM

Ok, great – glad you've found a workaround. Thanks for sharing all this info as you debugged.

Unless I'm misunderstanding something, this seems like a bad interaction between Python's and musl's defaults. I don't think it looks like a driver issue, unless something in your core dumps demonstrates that some driver thread is behaving badly. (I don't believe the core you attached has debug symbols, so I could be missing some evidence of that).

I'm inclined to close this as NOT A PROBLEM – users who run into similar issues are likely to find this ticket, and I don't think we want to maintain and test code that works around this.

Owen Nelson
December 8, 2017, 6:35 AM

I think that sounds fair. We were looking for an alpine package that provides the debug symbols, but came up short. Next on the list for us was to compile python from source inside an alpine container (so we could get the symbols), but we found the workaround before we got there.

I'd leave it to you to decide if supporting alpine is important enough to follow through on this. As far as I'm concerned switching to another distro (such as debian) is a great alternative if others don't feel like tweaking the stack size.

Jim Witschey
December 8, 2017, 7:01 AM

Yeah, looks like we're on the same page. This ticket seems like documentation enough, but if we run into other folks having trouble like this in the future, we can consider more prominent docs.

Thanks for going back and forth with us on this!

Not a Problem

Assignee

Unassigned

Reporter

Kenneth Lowe

Fix versions

None

Labels

None

Reproduced in

None

PM Priority

None

External issue ID

None

Doc Impact

None

Reviewer

None

Size

None

Pull Request

None

Components

Priority

Major