Skip to content

Catching exceptions during program running

Description

For now autoupdater updates all project that it can find in GitLab. If for some reason an exeption will be raised during program run it will ends with error and projects which were in queue after project on which error apear will not be updated. We need some mechanism that will allow autoupdater to update every project that is possible even if exeption will be raised in some of them. It is possible that e.g. GitLab server will not respond or someone tries to use autoupdater in incorrect way and this should not be the reason of failnig whole run of program.

Possible solutions

  1. Add some overall catching of exeptions and put them in some list then put them in critical log message at the end of program run. Also if rocketchat webhook is defined, post some special informations which projetcs failed and why. Such solution will prevent from failing whole program, because one or more projects fails.

    🔴 Such solutions is not the best practice in python world, becouse catching all exeptions is wrong. But in some situations is allowed https://wiki.python.org/moin/HandlingExceptions#General_Error_Catching.

Some logs which proves that this problem exist

An internal error of GitLab server with 500 HTTP return code

2019-08-12 06:00:23,343 [INFO]: Package "python/numpy" is up-to-date: 1.17.0
08:00:23            2019-08-12 06:00:23,591 [INFO]: Update available for package "python/openstacksdk": 0.32.0 -> 0.33.0
08:00:23            2019-08-12 06:00:23,958 [INFO]: Package "python/Pillow" is up-to-date: 6.1.0
08:00:24            2019-08-12 06:00:24,236 [INFO]: Package "python/aiofiles" is up-to-date: 0.4.0
08:00:24            2019-08-12 06:00:24,572 [INFO]: Package "python/aiohttp" is up-to-date: 3.5.4
08:00:24            2019-08-12 06:00:24,844 [INFO]: Package "python/sanic" is up-to-date: 19.6.2
08:00:25            2019-08-12 06:00:25,131 [INFO]: Package "python/filetype" is up-to-date: 1.0.5
08:00:25            2019-08-12 06:00:25,325 [INFO]: Package "python/progress" is up-to-date: 1.5
08:00:25            2019-08-12 06:00:25,592 [INFO]: Package "python/pydicom" is up-to-date: 1.3.0
08:00:25            2019-08-12 06:00:25,809 [INFO]: Package "python/requests" is up-to-date: 2.22.0
08:00:26            2019-08-12 06:00:26,038 [INFO]: Package "python/mypy" is up-to-date: 0.720
08:00:27            2019-08-12 06:00:26,978 [INFO]: Reseting merge request "Resolve "Autoupdater update"" to updated source branch.
08:00:27            2019-08-12 06:00:26,999 [INFO]: Pushing changes to origin.
08:00:28            Traceback (most recent call last):
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/exceptions.py", line 259, in wrapped_f
08:00:28                return f(*args, **kwargs)
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/mixins.py", line 285, in update
08:00:28                return http_method(path, post_data=new_data, files=files, **kwargs)
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/__init__.py", line 703, in http_put
08:00:28                **kwargs
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/__init__.py", line 564, in http_request
08:00:28                response_body=result.content,
08:00:28            gitlab.exceptions.GitlabHttpError: 500: 500 Internal Server Error
08:00:28            
08:00:28            During handling of the above exception, another exception occurred:
08:00:28            
08:00:28            Traceback (most recent call last):
08:00:28              File "/usr/local/bin/autoupdater", line 10, in <module>
08:00:28                sys.exit(main())
08:00:28              File "/usr/local/lib/python3.7/site-packages/autoupdater/commander.py", line 34, in main
08:00:28                arguments.webhook
08:00:28              File "/usr/local/lib/python3.7/site-packages/autoupdater/projects/update.py", line 12, in update_projects
08:00:28                updated_projects = list(repository.update_projects(project_to_update, labels))
08:00:28              File "/usr/local/lib/python3.7/site-packages/autoupdater/gitlab/update.py", line 119, in update_projects
08:00:28                updated_project = project.update(labels)
08:00:28              File "/usr/local/lib/python3.7/site-packages/autoupdater/gitlab/update.py", line 210, in update
08:00:28                merge_request.save()
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/mixins.py", line 374, in save
08:00:28                server_data = self.manager.update(obj_id, updated_data, **kwargs)
08:00:28              File "/usr/local/lib/python3.7/site-packages/gitlab/exceptions.py", line 261, in wrapped_f
08:00:28                raise error(e.error_message, e.response_code, e.response_body)
08:00:28            gitlab.exceptions.GitlabUpdateError: 500: 500 Internal Server Error
08:00:29            Running cleanup - removing workspace /var/lib/rundeck/work/teamwork/681424
08:00:29            Result: 1
08:00:30            Failed: NonZeroResultCode: Result code was 1
08:00:30            Execution failed: 681424 in project teamwork: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [0.0.0.0: NonZeroResultCode: Result code was 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:0.0.0.0)=BaseDataContext{{exec={exitCode=0}}}, ContextView(node:0.0.0.0)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]}, Node failures: {0.0.0.0=[NonZeroResultCode: Result code was 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:0.0.0.0)=BaseDataContext{{exec={exitCode=0}}}, ContextView(node:0.0.0.0)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]}, status: failed]
Edited by Krzysztof Szymański