Prometheus on Debian 9.9 doesn't spawn a PID and you can't force it to create one by editing its systemd unit file to add
PIDFile= (though there are some other hacks), you're going to need to restart it with the http interface as mentioned here
Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a
SIGHUPto the Prometheus process or sending a HTTP POST request to the
/-/reloadendpoint (when the
--web.enable-lifecycleflag is enabled). This will also reload any configured rule files.
When adding more nodes to Prometheus
scrape_config (so that it grabs the output from node_exporter on remote machines) you need to edit
/etc/prometheus/prometheus.yml and add the node to the
targets when using a static configuration. After you do that you need to have Prometheus reload its configuration file. I want to use Ansible to update
/etc/prometheus/prometheus.yml to make these changes. I'm using it as a template, and the playbook to accomplish all of this is:
--- #update_prometheus.yml - hosts: monitoring tasks: - name: Update prometheus configuration file become: True template: src: "roles/prometheus/templates/prometheus.yml.j2" # path to template dest: /etc/prometheus/prometheus.yml owner: root group: root mode: 0755 - name: Check prometheus config become: True command: "promtool check-config /etc/prometheus/prometheus.yml" register: promtool_result failed_when: "'FAILED' in promtool_result.stderr" notify: reload prometheus handlers: - name: reload prometheus args: warn: False command: "curl -X POST http://localhost:9090/-/reload"
Note: we have to suppress the warning from the curl command because ansible wants you to use get_url but since we need to send an HTTP POST request it doesn't work.