Reloading Prometheus on Debian with Ansible

Prometheus on Debian 9.9 doesn't spawn a PID and you can't force it to create one by editing its systemd unit file to add PIDFile= (though there are some other hacks), you're going to need to restart it with the http interface as mentioned here

Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a SIGHUP to the Prometheus process or sending a HTTP POST request to the /-/reload endpoint (when the --web.enable-lifecycle flag is enabled). This will also reload any configured rule files.

When adding more nodes to Prometheus scrape_config (so that it grabs the output from node_exporter on remote machines) you need to edit /etc/prometheus/prometheus.yml and add the node to the targets when using a static configuration. After you do that you need to have Prometheus reload its configuration file. I want to use Ansible to update /etc/prometheus/prometheus.yml to make these changes. I'm using it as a template, and the playbook to accomplish all of this is:

- hosts: monitoring
    - name: Update prometheus configuration file
      become: True
        src: "roles/prometheus/templates/prometheus.yml.j2" # path to template
        dest: /etc/prometheus/prometheus.yml
        owner: root
        group: root
        mode: 0755

    - name: Check prometheus config
      become: True
      command: "promtool check-config /etc/prometheus/prometheus.yml"
      register: promtool_result
      failed_when: "'FAILED' in promtool_result.stderr"
      notify: reload prometheus

    - name: reload prometheus
        warn: False
      command: "curl -X POST http://localhost:9090/-/reload"

Note: we have to suppress the warning from the curl command because ansible wants you to use get_url but since we need to send an HTTP POST request it doesn't work.

Show Comments