Deadlocks when setting iptables chains in prestart script
Summary
There are deadlocks when multiple Docker-based services start and execute their *_prestart.sh
scripts, setting iptables chains.
Logs
Multiple services starting (grafana, kapacitor, nginx):
lip 18 08:10:12 docker[8792]: Status: Image is up to date for kapacitor:1.5.2
lip 18 08:10:12 grafana_prestart.sh[9362]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
lip 18 08:10:12 systemd[1]: grafana.service: control process exited, code=exited status=4
lip 18 08:10:12 kapacitor_prestart.sh[9396]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
lip 18 08:10:12 systemd[1]: Failed to start Grafana - an open source, feature rich metrics dashboard and graph editor.
lip 18 08:10:12 nginx_prestart.sh[9379]: iptables: Bad rule (does a matching rule exist in that chain?).
lip 18 08:10:12 systemd[1]: Unit grafana.service entered failed state.
lip 18 08:10:12 nginx_prestart.sh[9379]: iptables: Bad rule (does a matching rule exist in that chain?).
lip 18 08:10:12 systemd[1]: grafana.service failed.
lip 18 08:10:12 systemd[1]: kapacitor.service: control process exited, code=exited status=4
lip 18 08:10:12 systemd[1]: Failed to start Kapacitor - open source framework for processing, monitoring, and alerting on time series data.
lip 18 08:10:12 systemd[1]: Unit kapacitor.service entered failed state.
lip 18 08:10:12 systemd[1]: kapacitor.service failed.
lip 18 08:10:12 systemd[1]: Started nginx service.
Grafana permanently unable to start:
lip 18 08:10:24 docker[16043]: Status: Image is up to date for grafana/grafana:6.2.5
lip 18 08:10:24 grafana_prestart.sh[16086]: iptables: Chain already exists.
lip 18 08:10:24 systemd[1]: grafana.service: control process exited, code=exited status=1
lip 18 08:10:24 systemd[1]: Failed to start Grafana - an open source, feature rich metrics dashboard and graph editor.
lip 18 08:10:24 systemd[1]: Unit grafana.service entered failed state.
lip 18 08:10:24 systemd[1]: grafana.service failed.