Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Cloud integration tests appear to enter an infinite loop if sni_single_endpoint Docker image isn't available

Description

Looks like a lot of this kind of thing:

 

05:01:01 /home/jenkins/workspace/drivers_python_oss_master@tmp/durable-821af322/script.sh: line 7: 25813 Terminated EVENT_LOOP=${EVENT_LOOP} CLOUD_PROXY_PATH="${HOME}/proxy/" CASSANDRA_VERSION=${CCM_CASSANDRA_VERSION} MAPPED_CASSANDRA_VERSION=${MAPPED_CASSANDRA_VERSION} VERIFY_CYTHON=${CYTHON_ENABLED} nosetests -s -v --logging-format="[%(levelname)s] %(asctime)s %(thread)d: %(message)s" --with-ignore-docstrings --with-xunit --xunit-file=advanced_results.xml tests/integration/cloud/ 05:01:01 + true 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Sending interrupt signal to process 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 Waiting for container to be ready... 05:01:01 Error response from daemon: Container 042950621d00d3583dbf39b45b1763b50c8106cebc4f854137b9d5b068116162 is not running 05:01:01 ./run.sh: line 70: 7620 Terminated sleep 1 05:01:01 Waiting for container to be ready... 05:01:01 Terminated 05:01:01 ERROR 05:01:01 05:01:01 ====================================================================== 05:01:01 ERROR: test suite for <module 'tests.integration.cloud' from '/home/jenkins/workspace/drivers_python_oss_master/tests/integration/cloud/__init__.py'> 05:01:01 ---------------------------------------------------------------------- 05:01:01 Traceback (most recent call last): 05:01:01 File "/home/jenkins/.pyenv/versions/3.8.3/lib/python3.8/site-packages/nose/suite.py", line 210, in run 05:01:01 self.setUp() 05:01:01 File "/home/jenkins/.pyenv/versions/3.8.3/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp 05:01:01 self.setupContext(ancestor) 05:01:01 File "/home/jenkins/.pyenv/versions/3.8.3/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext 05:01:01 try_run(context, names) 05:01:01 File "/home/jenkins/.pyenv/versions/3.8.3/lib/python3.8/site-packages/nose/util.py", line 471, in try_run 05:01:01 return func() 05:01:01 File "/home/jenkins/workspace/drivers_python_oss_master/tests/integration/cloud/__init__.py", line 26, in setup_package 05:01:01 start_cloud_proxy() 05:01:01 File "/home/jenkins/workspace/drivers_python_oss_master/tests/integration/cloud/__init__.py", line 109, in start_cloud_proxy 05:01:01 CLOUD_PROXY_SERVER.start() 05:01:01 File "/home/jenkins/workspace/drivers_python_oss_master/tests/integration/cloud/__init__.py", line 75, in start 05:01:01 raise Exception("Error while starting proxy server") 05:01:01 Exception: Error while starting proxy server 05:01:01 -------------------- >> begin captured logging << -------------------- 05:01:01 [INFO] 2022-12-20 17:46:36,226 140633535332480: Using Cassandra version: 2.2.19 05:01:01 --------------------- >> end captured logging << --------------------- 05:01:01 05:01:01 ---------------------------------------------------------------------- 05:01:01 XML: /home/jenkins/workspace/drivers_python_oss_master/advanced_results.xml 05:01:01 ---------------------------------------------------------------------- 05:01:01 Ran 0 tests in 62064.933s 05:01:01 05:01:01 FAILED (errors=1)

 

which seems entirely reasonable if the image isn’t available. Problem is it looks like this just continues again and again.

Environment

None

Pull Requests

None

Activity

Bret McGuire 
December 21, 2022 at 8:33 PM

Upon further inspection I don’t think this is an issue with the build proper. I’m seeing a sequence of these results, but there is a pattern to the sequence: for a given C*/DSE version there are two versions with Python 3.8, then two with Python 3.7, and so on through Python 2.7… then we move on to the next C*/DSE version. This is consistent with what’s currently specified in the Jenkinsfile:

 

"FULL": [ "SERVER": ['2.1', '2.2', '3.0', '3.11', '4.0', 'dse-5.0', 'dse-5.1', 'dse-6.0', 'dse-6.7', 'dse-6.8'], "RUNTIME": ['2.7.18', '3.5.9', '3.6.10', '3.7.7', '3.8.3'], "CYTHON": ["True", "False"] ],

 

This leads to a sequence of 100 sub-builds: 10 C*/DSE versions * 5 Python versions * a boolean for Cython. That’s a lot of builds but isn’t really in the scope of this issue… that will be addressed elsewhere.

 

Once this sequence completes I see the runner simply stop. It doesn’t continue re-executing builds (which is what I initially thought) and the build is clearly failed. At the moment I don’t think we have an issue with the build (it’s likely more a CI issue) but we can revisit if these symptoms present themselves again.

Not a Problem

Details

Assignee

Reporter

Affects versions

Priority

Created December 21, 2022 at 5:02 PM
Updated December 21, 2022 at 8:33 PM
Resolved December 21, 2022 at 8:33 PM

Flag notifications