Quantcast
Channel: Distracted-IT
Viewing all articles
Browse latest Browse all 37

SystemD System-V (init.d) compatibility layer and auto-restarts

0
0
SystemD (eg. RHEL7) has a compatibility layer for init.d services. When you run '/etc/init.d/ start', it doesn't do what you think it does; there is some magic involved. Have a look at the TurnKey Linux blog for some insight around that.

One of the great things about System-D is that you can auto-restart failed services. In this post, I want to show you how you take a init.d script (provided via a third-party package), and add some System-D goodness for making sure it can restart on failure.

For this example, I shall be using Globus's GridFTP service, which I have been working with recently. I found that after some routine patching, perhaps due to other patching activity around DNS, failed to resolve some name and failed to start the service.

SystemD will start (by 'magic' compatibility layer) init.d services when the name of the service doesn't exist (eg. systemctl restart globus-gridftp-server.service will auto-resolve to /etc/rc.d/init.d/globus-gridftp-server when there is no SystemD unit otherwise defined.

To auto-restart init.d services that get started by SystemD, there’s a bit of other configuration that needs to be catered for; some of this will be typical for other similar init.d type of services (eg. PIDFile), and I suspect that RemainAfterExit is only relevant for this particular service (hint: the service starts with –no-detach, so doesn’t run as a traditional daemon process). I put this in using systemctl edit globus-gridftp-server.service

# /etc/systemd/system/globus-gridftp-server.service.d/override.conf
[Service]
Restart=always
PIDFile=/var/run/globus-gridftp-server.pid
RemainAfterExit=no (I suspect for most proper daemons this should remain as =yes, but here we need =no)

Testing this (remembering to 'systemctl daemon-reload; systemctl restart globus-gridftp-server.service' first); let's check the status, kill the process, and witness the restart.
 
[PROD] root@myvm:/etc
# systemctl status globus-gridftp-server.service
● globus-gridftp-server.service - LSB: Globus GridFTP Server
Loaded: loaded (/etc/rc.d/init.d/globus-gridftp-server; bad; vendor preset: disabled)
Drop-In: /etc/systemd/system/globus-gridftp-server.service.d
└─override.conf
Active: active (running) since Tue 2019-04-30 15:04:17 NZST; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 7698 ExecStop=/etc/rc.d/init.d/globus-gridftp-server stop (code=exited, status=0/SUCCESS)
Process: 7701 ExecStart=/etc/rc.d/init.d/globus-gridftp-server start (code=exited, status=0/SUCCESS)
Main PID: 7704 (globus-gridftp-)
CGroup: /system.slice/globus-gridftp-server.service
└─7704 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -C /etc/gridftp.d -pidfile /var/run/globus-gridftp-server.pid -no-detach -config-base-path /

Apr 30 15:04:16 myvm.example systemd[1]: globus-gridftp-server.service failed.
Apr 30 15:04:16 myvm.example systemd[1]: Starting LSB: Globus GridFTP Server...
Apr 30 15:04:17 myvm.example systemd[1]: Started LSB: Globus GridFTP Server.
Apr 30 15:04:17 myvm.example globus-gridftp-server[7701]: Starting globus-gridftp-server: OK

[PROD] root@myvm:/etc
# kill 7704

[PROD] root@myvm:/etc
# systemctl status globus-gridftp-server.service
● globus-gridftp-server.service - LSB: Globus GridFTP Server
Loaded: loaded (/etc/rc.d/init.d/globus-gridftp-server; bad; vendor preset: disabled)
Drop-In: /etc/systemd/system/globus-gridftp-server.service.d
└─override.conf
Active: active (running) since Tue 2019-04-30 15:04:28 NZST; 1s ago
Docs: man:systemd-sysv-generator(8)
Process: 7709 ExecStop=/etc/rc.d/init.d/globus-gridftp-server stop (code=exited, status=0/SUCCESS)
Process: 7712 ExecStart=/etc/rc.d/init.d/globus-gridftp-server start (code=exited, status=0/SUCCESS)
Main PID: 7714 (globus-gridftp-)
CGroup: /system.slice/globus-gridftp-server.service
└─7714 /usr/sbin/globus-gridftp-server -c /etc/gridftp.conf -C /etc/gridftp.d -pidfile /var/run/globus-gridftp-server.pid -no-detach -config-base-path /

Apr 30 15:04:28 myvm.example systemd[1]: Stopped LSB: Globus GridFTP Server.
Apr 30 15:04:28 myvm.example systemd[1]: Starting LSB: Globus GridFTP Server...
Apr 30 15:04:28 myvm.example globus-gridftp-server[7712]: Starting globus-gridftp-server: OK
Apr 30 15:04:28 myvm.example systemd[1]: Started LSB: Globus GridFTP Server.

Things to note:
  • In the above, we killed the running service and verified that it was restarted (new main PID)
  • You can use systemctl edit globus-gridftp-server.service to add overrides, which drops a file override.conf into /etc/systemd/system/.d/
  • man systemd.service for information on service directives.
  • Main PID should be listed in the status output (otherwise you don't have PIDFile= set in the service.
  • CGroup line should have a PID listed; if there is no second-line of the CGroup section (no process) after the kill has been issued, it may be that you need to set RemainAfterExit=no
  • RemainAfterExit normally defaults to no, but systemd-sysv-generator seems to set this to yes, which is wrong for this particular service, which doesn't start as a daemon oddly enough. In this case, if RemainAfterExit is not changed backed to =no, the Main PID will show that/how the process was killed, but the service won't be restarted.
Enjoy the pleasure of auto-immunity ... perhaps next time I should figure out how to get alerts for when stuff auto-restarts.



Viewing all articles
Browse latest Browse all 37

Latest Images

Trending Articles





Latest Images