Skip to content

cleanup job sometimes fails

Dzisiaj zauważyłem że w częsci pipelinów failuje mi ostatni job z cleanupem:

2021-05-21 10:13:15,253 [INFO]: Cleaning up resources from job 287503
2021-05-21 10:13:15,253 [INFO]: Deleting instance
2021-05-21 10:13:15,459 [INFO]: Instance deleted
2021-05-21 10:13:15,460 [INFO]: Cleanup resources from job 287503 ended
2021-05-21 10:13:16,727 [INFO]: Cleaning up resources from job 287514
2021-05-21 10:13:16,728 [INFO]: Deleting instance
2021-05-21 10:13:16,932 [INFO]: Instance deleted
2021-05-21 10:13:16,932 [INFO]: Cleanup resources from job 287514 ended
2021-05-21 10:13:18,185 [INFO]: Cleaning up resources from job 287512
2021-05-21 10:13:18,186 [INFO]: Deleting instance
2021-05-21 10:13:18,406 [INFO]: Instance deleted
2021-05-21 10:13:18,407 [INFO]: Cleanup resources from job 287512 ended
2021-05-21 10:13:19,767 [INFO]: Cleaning up resources from job 287494
2021-05-21 10:13:19,768 [INFO]: Deleting instance
2021-05-21 10:13:20,005 [INFO]: Instance deleted
2021-05-21 10:13:20,005 [INFO]: Deleting floating IP
2021-05-21 10:13:21,675 [INFO]: Floating IP deleted
2021-05-21 10:13:21,675 [INFO]: Lock 140667660169120 acquired on /srv/vms/filelock.lock
2021-05-21 10:13:21,676 [INFO]: File "/srv/vms/287494" deleted
2021-05-21 10:13:21,676 [INFO]: Lock 140667660169120 released on /srv/vms/filelock.lock
2021-05-21 10:13:21,676 [INFO]: Cleanup resources from job 287494 ended
Traceback (most recent call last):
  File "/usr/local/bin/spawner", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/spawner/commander.py", line 10, in main
    service.run()
  File "/usr/local/lib/python3.9/site-packages/spawner/spawner.py", line 103, in run
    self.cleanup_pipeline()
  File "/usr/local/lib/python3.9/site-packages/spawner/spawner.py", line 210, in cleanup_pipeline
    self.cleanup_job(job_id)
  File "/usr/local/lib/python3.9/site-packages/spawner/spawner.py", line 187, in cleanup_job
    floating_ips = [floating_ip for floating_ip in self.cloud.network.ips() if job_id == floating_ip.tags[0].split(':')[1]]
  File "/usr/local/lib/python3.9/site-packages/spawner/spawner.py", line 187, in <listcomp>
    floating_ips = [floating_ip for floating_ip in self.cloud.network.ips() if job_id == floating_ip.tags[0].split(':')[1]]
IndexError: list index out of range
Cleaning up file based variables 00:01
ERROR: Job failed: exit code 1

tutaj link do tego joba.

Edited by Cyprian Kleist