System Layout
This document describes how the Hosted Graphite agent (hg-agent) works on your machine.
Contents
/opt/hg-agent/... (static files)
package/... (pyinstaller bundle)
collectors/... (Diamond collectors)
bin/... (symlinks to binaries in package bundle)
/etc/opt/hg-agent/... (configuration files)
/var/opt/hg-agent/ (spool or other state files)
/var/log/hg-agent/ (log files)
The agent is distributed as a PyInstaller bundle: this is a self-contained build of the whole agent system (including an embedded Python) which requires only a compatible libc.
You typically shouldn’t need to run anything in /opt/hg-agent/bin, since just editing /etc/opt/hg-agent/hg-agent.conf will result in an automatic configuration reload. If something seems to be stuck, restart the service as normal, e.g. for Ubuntu 14.04:
$ sudo service hg-agent restart
In /var/log/hg-agent, you can see the metrics generated by diamond in archive.log, as well as the various daemon logs supervisord.log, diamond.log, periodic.log, forwarder.log, and receiver.log.
The agent runs as a process tree:
supervisord ─┬──┬─ diamond ─┬─ diamond (collector)
│ │ ├─ diamond (collector)
│ │ └─ ...
│ └─ periodic
├──── metric receiver
│ ├─ UDP
│ └─ TCP
└──── metric forwarder
├─ Spool Reader
└─ Forwarder
This all runs as the non-privileged user hg-agent.
This popular metric collection daemon does the primary work of hg-agent: gathering system metrics and exporting them to Hosted Graphite.
A “sidecar” that manages periodic tasks for hg-agent, including (re)configuration of diamond and sending “heartbeat” metadata to Hosted Graphite.
Threads that listen for metrics via both UDP and TCP. On receipt of metrics we perform some validation, and write the data out to a spool file (/var/opt/hg-agent/spool/*.spool.<timestamp>), this write happens once a second. By default, these will listen for metrics on localhost:2003.
You may configure the ports for the UDP and TCP receivers to listen on by adding the following values to /etc/opt/hg-agent/hg-agent.conf:
udp_port: <port_num>
tcp_port: <port_num>
Tails spool files in /var/opt/hg-agent/spool and forwards metrics to Hosted Graphite’s HTTP API. Metrics are batched and, by default, a POST request is sent every 0.5 seconds or once the batch size has reached 250.
The agent maintains a progress file that tells us how much of each spool file has been forwarded. This allows us to recover in times when there may be some network connectivity issues between the host server and Hosted Graphite.
Agent configuration in /etc/opt/hg-agent/hg-agent.conf is used to generate a more complete configuration for diamond in /var/opt/hg-agent/diamond.conf.
On a representative low-power 4-core Ubuntu 14.04 machine, hg-agent currently takes up the following resources:
- ~13MiB of disk space for package files;
- a further 100MiB (by default: 10 * 10MB) for metric spool files;
- ~0.02 process CPU seconds/s
Last modified 1mo ago