Ansible is great as a configuration management tool. What I like most about it, is that you can make your own recipe on how to handle things.

Here we’ll talk about how can Ansible restart, sequentially, a service, on some servers, which are members of a load balancing poll.

Long story short, we have an F5 load balancer which distributes the syslog traffic to 2-3 servers. These servers run syslog-ng. We don’t want to restart syslog-ng at the same time on all the nodes. We’ll lose events.

The tricky part is how to call the handler in Ansible but not at the same time on all servers behind it.

I’ll use server1 and server2 as examples. Below the implementation:

• For each server, assign a variable (I’m calling it node_type) and a value, primary or secondary. We can expand this to more than 2 servers by making one primary and the rest secondaries.

#cat host_vars/server1
node_type: primary

#cat host_vars/server2
node_type: secondary

• Create the handler components:

- name: restart_syslog-ng_primary
  systemd:
    name: syslog-ng.service
    state: restarted
  listen: restart_syslog-ng
  when: node_type == "primary"
- name: restart_syslog-ng_secondary
  systemd:
    name: syslog-ng.service
    state: restarted
  listen: restart_syslog-ng
  when: node_type == "secondary"

• Use the handler inside a task:

- name: Install syslog-ng configuration
  copy:
    src: files/syslog-ng.conf
    dest: /etc/syslog-ng/syslog-ng.conf
    mode: 0644
    owner: root
    group: root
    backup: yes
  notify:
    - restart_syslog-ng

The most important part above is the notify section, which calls the listener, restart_syslog-ng, which triggers two handlers sequentially.

https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html