System Layout

This document describes how our Hosted Graphite Agent (hg-agent) works on your machine.

Contents

• System Layout

  • Disk

  • Processes

    • supervisord

    • diamond

    • periodic

    • receiver

    • forwarder

  • Configuration

  • Resource requirements

Disk

The layout on disk is per FHS guidelines for add-on application software packages:

/opt/hg-agent/... (static files)
              package/... (pyinstaller bundle)
              collectors/... (Diamond collectors)
              bin/... (symlinks to binaries in package bundle)
/etc/opt/hg-agent/... (configuration files)
/var/opt/hg-agent/ (spool or other state files)
/var/log/hg-agent/ (log files)

The agent is distributed as a PyInstaller bundle: this is a self-contained build of the whole agent system (including an embedded Python) which requires only a compatible libc.

You typically shouldn’t need to run anything in /opt/hg-agent/bin, since just editing /etc/opt/hg-agent/hg-agent.conf will result in an automatic configuration reload. If something seems to be stuck, restart the service as normal, e.g. for Ubuntu 14.04:

$ sudo service hg-agent restart

In /var/log/hg-agent, you can see the metrics generated by diamond in archive.log, as well as the various daemon logs supervisord.log, diamond.log, periodic.log, forwarder.log, and receiver.log.

Processes

The agent runs as a process tree:

supervisord ─┬──┬─ diamond ─┬─ diamond (collector)
             │  │         ├─ diamond (collector)
             │  │         └─ ...
             │  └─ periodic
             ├──── metric receiver
             │                  ├─ UDP
             │                  └─ TCP
             └──── metric forwarder
                                ├─ Spool Reader
                                └─ Forwarder

This all runs as the non-privileged user hg-agent.

supervisord

A process manager which we use to manage the sub-processes that make up hg-agent.

diamond

This popular metric collection daemon does the primary work of hg-agent: gathering system metrics and exporting them to Hosted Graphite.

periodic

A “sidecar” that manages periodic tasks for hg-agent, including (re)configuration of diamond and sending “heartbeat” metadata to Hosted Graphite.

receiver

Threads that listen for metrics via both UDP and TCP. On receipt of metrics we perform some validation, and write the data out to a spool file (/var/opt/hg-agent/spool/*.spool.<timestamp>), this write happens once a second. By default, these will listen for metrics on localhost:2003.

You may configure the ports for the UDP and TCP receivers to listen on by adding the following values to /etc/opt/hg-agent/hg-agent.conf:

udp_port: <port_num>
tcp_port: <port_num>

forwarder

Tails spool files in /var/opt/hg-agent/spool and forwards metrics to Hosted Graphite’s HTTP API. Metrics are batched and, by default, a POST request is sent every 0.5 seconds or once the batch size has reached 250.

The agent maintains a progress file that tells us how much of each spool file has been forwarded. This allows us to recover in times when there may be some network connectivity issues between the host server and Hosted Graphite.

Configuration

Agent configuration in /etc/opt/hg-agent/hg-agent.conf is used to generate a more complete configuration for diamond in /var/opt/hg-agent/diamond.conf.

Resource requirements

On a representative low-power 4-core Ubuntu 14.04 machine, hg-agent currently takes up the following resources:

• ~13MiB of disk space for package files;

• a further 100MiB (by default: 10 * 10MB) for metric spool files;

• ~0.02 process CPU seconds/s

• ~150MiB physical memory (measured as unique set size)