Cannot use BatchQuery save methods when using ResponseFuture ?

Description

Hi,
this code is not working...

It stucks on first page of results until it reaches heartbeat timeout (stucks at b.execute()...)

If I set idle_heartbeat_interval at 0, disabling heartbeat, it stucks forever.

Could you see if and what I'm doing wrong? Any suggestions?
I can't even use tweet.save() method: same symptoms (Tweet is a CQL engine/ORM class representing a simple column family).

Also, I have to set executor_threads at 1 in all my python cassandra code (never worked with any other value). Is it really working in a C Python env (i.e. with GIL)?

Many thanks in advance.

Environment

cassandra-driver 3.18.0
Python 3.6, Ubuntu 18.04

Pull Requests

None

Activity

Show:
Alan Boudreault
June 20, 2019, 10:47 PM

2. Indeed, you cannot execute statements inside a query callback. I think this is the issue you are experiencing. You cannot execute statements in the handle_page function.

3. The problem is in your application code. You shouldn´t have to modify that value. Make you you don´t execute statements inside callbacks.

D. Nappo
June 20, 2019, 10:56 PM

Ok, I'm sure that's the issue: I'm executing queries inside a query callback.
Thanks!

Given that, how could I perform the task? Do you have any idea? resources which I can check?

I need to iterate through fetched results and update rows and the only thing that worked for me to iterate efficiently through millions of records was the async/ResponseFuture pattern.

Alan Boudreault
June 20, 2019, 11:17 PM

The important thing is to not call execute/execute_async in the callback, which is executed inside the event loop. So you could simply iterate the results and send them to another consumer (with a Queue in example).

D. Nappo
June 20, 2019, 11:25 PM

Thanks for details.

Indeed, that batch script was coded using Kafka...I wanted to avoid it for batch processing but I had wrong assumptions
I will try with a simple Python Queue as you suggested.

Cheers!

d.

Alan Boudreault
June 20, 2019, 11:38 PM

Great. Good luck!

Not a Problem

Assignee

Unassigned

Reporter

D. Nappo

Fix versions

None

Reproduced in

None

PM Priority

None

External issue ID

None

Doc Impact

None

Reviewer

None

Size

None

Pull Request

None

Priority

Major