Hosted Graphite Docs
Get StartedBook a Demo
  • Welcome to Hosted Graphite
  • Getting Started
  • HG-CLI
  • Sending Metrics
    • Supported Protocols
    • Graphite Tag Support
    • Metric Management
    • Metric APIs
  • Language Guide
    • Metric Libraries
    • .NET
    • Go
    • Java
    • Javascript
    • Node.js
    • PHP
    • Postman
    • Python 2.x
    • Python 3.x
    • Python Pickle
    • Ruby
    • Shell
    • TypeScript
  • Dashboard and Graphs
    • Primary Dashboards
    • Dashboard Library
    • Local Dashboard Integration
    • Worldmap Panel
    • Graphite Dashboard Guide
    • Graphite Graph Menu Reference
    • Other Dashboard Options
  • Alerting Guide
    • Alerting Overview
    • Alerts API
    • Notification Channels API
    • Scheduled Mutes API
    • Using Your Own Alerting
  • Agents Guide
    • The Hosted Graphite Agent
      • Base Metrics
      • System Layout
    • Telegraf
    • K8 Telegraf Daemon
    • OpenTelemetry
    • collectd Agent
    • StatsD Agent
    • Diamond
  • Add-Ons and Integrations Guide
    • AWS CloudWatch
    • Azure Monitor Metrics
    • GCP Metrics
    • Carbon-C-Relay
    • Circle CI
    • Cloudbees
    • Collectd Add Ons
    • GitHub
    • GitLab
    • Heroku
    • Hosted StatsD
    • New Relic
    • Papertrail
    • Pingdom
    • Sentry
    • Sitespeed
    • StatsD Add-on
    • Statuspage
  • Account Management
    • Access Keys
    • Account Diagnostics
    • Account Settings
    • Team Access: Limited Access Groups
    • SAML Authentication
    • Team Access
  • Additional Features
    • Aggregation Rules
    • Data Views
  • API Guides
    • Metrics API
    • Tag API
    • Graphite Render API
    • Render Variables API
    • Dashboard API
    • Annotations and Events API
    • Aggregation Rules API
    • Alerts APIs
  • FAQ
    • General
    • Business
    • Technical
    • Account Metrics and Limiting
    • Customization
    • Troubleshooting
    • Support
    • Changelog
Powered by GitBook
On this page
  • Creating Alerts
  • Composite Alerts
  • Searching Alerts
  • Obtain Information on a Single Alert
  • Updating Alerts
  • Deleting Alerts
  • Mute an Alert
  • Unmute an Alert
  • Check if an Alert is Muted
  • Mute Multiple Alert
  • Unmute Multiple Alerts
  • Obtain Alert History

Was this helpful?

  1. Alerting Guide

Alerts API

PreviousAlerting OverviewNextNotification Channels API

Last updated 4 months ago

Was this helpful?

Contents

POST /v2/alerts/

Path Parameters

Name
Type
Description

name*

String

A name that uniquely identifies this alert.

metric*

String

The graphite metric query to alert on.

alert_criteria*

String

The criteria for which the alert triggers.

additional_criteria

String

Any extra metrics that are to be included when evaluating the state of an alert, these are referred to as Composite Alerts. Note that any additional criteria must be referenced in the expression field and a maximum of 3 additional criteria is allowed. Example: {"f": {"type": "above", "above_value": 5, "metric": "additional.metric.to.alert.on"} where f is also referenced in the expression field. Defaults to {}.

expression

String

A conditional relationship between multiple metrics. It allows for the combining of multiple alert criteria. It’s represented in the format of a boolean expression where each individual alert criteria is denoted as a single letter. The letter a is reserved for the criteria defined in alert_criteria. Only required when additional alerting criteria are defined. Maximum number of 3 additional criteria allowed (4 including the default alert criteria). && takes higher precedence over || Examples: a && b, a || c. Defaults to a.

above_vale

String

The value above which the alert should trigger. Required for alert type above and outside_bounds.

below_value

String

The value below which the alert should trigger. Required for alert type below and outside_bounds.

time_period

String

The time period in minutes for which the threshold needs to be breached. Leave empty for an instantaneous evaluation. Required for alert type missing.

scheduled_mutes

String

List of scheduled mutes for this alert. Must be a list of scheduled mute IDs. Defaults to an empty list.

notification_channels

String

List of notification channels for this alert. This can be a combination of channel names and channel IDs. Defaults to Email me if none exists.

notification_type

String

The type of notification interval for this alert. Options are state_change (['state_change'] or 'state_change' is valid) or ['every', time_in_minutes]. This lets you control how often you want to be notified for an alert. Defaults to ['every', 60] if empty.

info

String

Alert message sent with notifications. Can contain an arbitrary string which may contain a description of the alert, steps to follow or references to documentation.

on_query_failure

String

Controls if a notification is delivered if the graphite function query fails. Only valid for Alerts that have graphite function in the metric field. Defaults to notify. Graphite function query can fail due to timeouts from matching too many metrics, being malformed or if it returns duplicate metrics due to aliasing.

HTTP/1.1 201 
Content-Type: application/json 
{
  "id": "<alert_id>",
  "url": "/v2/alerts/<alert_id>"
}

Bad Request Invalid format

Conflict Alert with the name already exists.

Curl example:

curl -H "Content-Type: application/json" -X POST -d \
'{"name": "alert1", "metric": "test.metric.1", "alert_criteria": {"type": "above", "above_value": 5, "time_period": 2}}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/"

Creates an alert named alert1 for metric test.metric.1 that triggers if the value is above 5 for 2 minutes with the (default) Email me notification type.

Request format:

{
  "name": "string",
  "metric": "string",
  "alert_criteria":
    {
      "type": "above" | "below" | "outside_bounds" | "missing",
      "above_value": 123,
      "below_value": 123,
      "time_period": 123,
    },
  "additional_criteria":
    {
     "b":
       {
        "type": "above" | "below" | "outside_bounds" | "missing",
        "above_value": 123,
        "below_value": 123,
        "time_period": 123,
        "metric": "string"
       }
    },
  "expression": "a && b",
  "scheduled_mutes": ["scheduled_mute_id1", "scheduled_mute_id2", "scheduled_mute_id3"],
  "notification_channels": ["channel_id1", "channel_name1", "channel_id2"],
  "notification_type": "state_change" | ["state_change"] | ["every", 123],
  "info": "string",
  "on_query_failure": null | "ignore" | "notify"
}

Alert criteria examples:

Create an alert that triggers if the metric is missing for 10 minutes.

"alert_criteria": {
  "type": "missing",
  "time_period": 10
}

Create an alert that triggers if the metric is above 5.6 ever.

"alert_criteria": {
  "type": "above",
  "above_value": 5.6
}

Create an alert that triggers if the metric is below 2.1 or above 5.6 ever.

"alert_criteria": {
  "type": "outside_bounds",
  "above_value": 5.6,
  "below_value": 2.1
}

Create an alert that triggers if the metric is below 2.1 or above 5.6 for 10 minutes.

"alert_criteria": {
  "type": "outside_bounds",
  "above_value": 5.6,
  "below_value": 2.1,
  "time_period": 10
}

Composite alerts can evaluate multiple metrics/values, and are defined in the additional_criteria and expression parameters in the request format.

Create an alert when one metric is above 80 and another is below 20.

"metric": "my.first.metric",
"alert_criteria": {
  "type": "above",
  "above_value": 80
},
"additional_criteria": {
  "b": {
    "type": "below",
    "below_value": 20,
    "metric": "my.second.metric"
  }
},
"expression": "a && b",

Create an alert when one metric is below 50 or another is below 30.

"metric": "my.first.metric",
  "alert_criteria": {
    "type": "above",
    "above_value": 50
  },
  "additional_criteria": {
    "c": {
      "type": "below",
      "below_value": 30,
      "metric": "my.second.metric"
    }
  },
  "expression": "a || c"

Create an alert when one metric (A) is below 50 and another (B) is above 60 or if a third (C) is above 120. Note that performing an && on two monitors gets preference over || so this alert expression will be evaluated as (A && B) || C

"metric": "my.first.metric",
"alert_criteria": {
  "type": "below",
  "below_value": 50
},
"additional_criteria": {
  "b": {
    "type": "above",
    "above_value": 60,
    "metric": "my.second.metric"
  },
  "c": {
    "type": "above",
    "above_value": 120,
    "metric": "my.third.metric"
  }
},
"expression": "A && B || C"

NOTE: - Our Alerts UI doesn’t fully support composites so composite alerts can only be created and updated via the API. The alert overview page (when you click the eye button on an alert), will only display one metric for the alert instead of all the metrics associated. However the alert notifications are working and will display the graph of the last metric that breached the alert threshold. So for example, if the alert is a && b, and a breaches the threshold, then a few minutes later b breaches it’s threshold, the alert notification will show the metric graph for b.


GET /v2/alerts/

Returns a JSON object containing information on alerts matching the search query. Returns all the alerts if the parameter is left empty: /v2/alerts/?

Path Parameters

Name
Type
Description

name

String

Format: &name=<alert_name1>&name=<alert_name2>The alert name to search for. Can specify multiple alert names to search for more than 1. Performs an OR operation so a query such as &name=<alert_name1>&name=<alert_name2> will return info on both alert_name1 and alert_name2

id

String

Format: &id=<id1>&id=<alert_id2>The alert id to search for. Can specify multiple ids to search for more than 1. Performs an OR operation so a query such as &id=<alert_id1>&id=<alert_id2> will return info on both alert_id1 and alert_id2.

search

String

Format: &search=<search_string>A more generic search that will match any alert whose alert name, metric name or notification channel name contains the search string.

page

String

Format: &page=1The page number to query. Used if there are more alerts than can be displayed in a single request. (default: 1)

max

String

Format: &max=50The maximum number of alerts to display. Limit of 100. (default: 100)

HTTP/1.1 201 
Content-Type: application/json 
{
  "alerts": [
    {
      "name": "Name of the alert",
      "id": "unique id of the alert",
      "alert_criteria": {
        "type": "type of alert",
        "above_value": 123,
        "below_value": 123,
        "time_period": 123
      },
      "notification_channels": [""],
      "notification_type": ["state_change", "every", 123],
      "scheduled_mutes": ["<list of scheduled mute ids>"],
      "currently_triggered_metrics": [],
      "muted": true,
      "status": "alerting",
      "info": "alert info",
      "on_query_failure": null
    }
  ],
  "next_page": false
}

Bad Request Invalid form data

Curl example:

curl -X GET "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/?"

Returns a list of all alerts

curl -X GET "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/?&search=example

Returns a list of alerts that match the search pattern "example".

Specifying both alert_name and alert_id performs an OR operation and returns alert that match either criteria. Including a search string performs an AND operation with the other criteria.


GET /v2/alerts/<alert_id>/

Returns information on a single alert

HTTP/1.1 200
Content-Type: application/json 
{
  "name": "Name of the alert",
  "id": "unique id of the alert",
  "alert_criteria": {
    "type": "threshold",
    "above_value": 123,
    "below_value": 123,
    "time_period": 123
  },
  "expression": "a",
  "notification_channels": ["email", "sms"],
  "scheduled_mutes": ["mute_id_1"],
  "currently_triggered_metrics": ["metric_1", "metric_2"],
  "notification_type": ["state_change"],
  "muted": false,
  "status": "alerting",
  "info": "Alert is active due to threshold breach.",
  "on_query_failure": "notify"
}

Not Found Alert doesn’t exist

Curl example:

curl -X GET "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-7890/"

Obtain information on the alert identified by the id 123-456-7890. The alert id can be found in the URL of the alert when viewed from within the Hosted Graphite Alerts UI. NOTE: a request must contain the forward slash character after the alert_id.


PUT /v2/alerts/<alert_id>/

Update attributes of an alert identified by the alert id

HTTP/1.1 200
Content-Type: application/json 

{
  "name": "string",
  "metric": "string",
  "info": "string",
  "alert_criteria": {
    "type": "above",
    "above_value": 123,
    "below_value": 123,
    "time_period": 123
  },
  "scheduled_mutes": ["scheduled_mute_id1", "scheduled_mute_id2", "scheduled_mute_id3"],
  "notification_channels": ["channel_id1", "channel_id2", "channel_id3"],
  "notification_type": ["state_change"]
}

Bad Request Invalid form data

Not Found Alert doesn’t exist.

Curl Example

curl -H "Content-Type: application/json" -X PUT -d \
'{"alert_criteria": {"time_period": 3, "type": "below", "below_value": 6}}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-789/"

Modify the alert identified by the id 123-456-789 to alert if the metric values are below 6 for 3 minutes. Each field is optional, only the specified fields are updated, and requests must contain the forward slash character after the alert_id.


DELETE /v2/alerts/<alert_id>/

Delete an alert identified by the alert id.

OK

Bad Request Alert doesn't exist

Curl Example

curl -X DELETE "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-789/"

Delete the alert identified by the id 123-456-789.


POST /v2/alerts/<alert_id>/muted/

Mute alerts identified by the alert id.

Path Parameters

Name
Type
Description

duration*

String

Time to mute in minutes.

HTTP/1.1 200
Content-Type: application/json 
{
  "duration": 60
}

Bad Request Invalid form data

Not Found Alert doesn't exist

Curl Example:

curl -X POST -d '{"duration":60}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-789/muted/"

Mute the alert identified by the id 123-456-789 for 60 minutes.


DELETE /v2/alerts/<alert_id>/muted/

Unmute an alert identified by the alert id.

OK

Not Found Alert doesn't exist

Curl Example:

curl -X DELETE "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-789/muted/"

Unmute the alert identified by the id 123-456-789`.


GET /v2/alerts/<alert_id>/muted/

Get the mute status of an alert identified by the alert id.

HTTP/1.1 200 
Content-Type: application/json 
{
  "id": "alert_id",
  "name": "alert_name",
  "muted": false,
  "duration": 30,
  "scheduled_mutes": ["mute_id1", "mute_id2"]
}

Not Found Alert doesn't exist

Curl Example:

curl -X GET "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/123-456-789/muted/"

Obtain information on the mute status of the alert identified by the id 123-456-789. Also provides a list of the scheduled mutes currently attached to the alert.


POST /v2/alerts/muted/

Path Parameters

Name
Type
Description

ids

String

List of alert ids to mute.

search

String

String to filter alerts. Every alert with an alert name, metric name, or notification channel name containing this string will be muted.

duration*

String

Time to mute in minutes.

HTTP/1.1 200 
Content-Type: application/json 
{
  "ids": [
    "alert_id1",
    "alert_id2"
  ],
  "duration": 60
}

Bad Request Invalid request

Not Found Alert doesn't exist

Curl examples:

curl -H "Content-Type: application/json" -X POST -d '{"ids": ["123-456-789", "321-654-987"], "duration":60}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/muted/"

Mute alerts 123-456-789 and 321-654-987 for 60 minutes.

curl -H "Content-Type: application/json" -X POST -d '{"search": "search_string", "duration":60}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/muted/"

Mute alerts with alert name, metric name or notification channel name containing search_string for 60 minutes. If both ids and search are empty, then all alerts will be muted.


DELETE /v2/alerts/muted/

Path Parameters

Name
Type
Description

ids

String

List of alert ids to unmute.

search

String

String to filter alerts. Every alert with an alert name, metric name, or notification channel name containing this string will be unmuted.

HTTP/1.1 200 
Content-Type: application/json 
{
  "ids": [
    "alert_id1",
    "alert_id2"
  ]
}

Bad Request Invalid request

Not Found Alert doesn't exist

Curl examples:

curl -H "Content-Type: application/json" -X DELETE -d '{"ids": ["123-456-789", "321-654-987"]}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/muted/"

Unmute alerts 123-456-789 and 321-654-987.

curl -H "Content-Type: application/json" -X DELETE -d '{"search": "search_string"}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/muted/"

Unmute alerts with alert name, metric name or notification channel name containing search_string. If both ids and search are empty, then all alerts will be unmuted.


GET /v2/alerts/history/?

Path Parameters

Name
Type
Description

id

String

The alert id to search for. Can specify multiple ids to search for more than 1. Returns history of all alerts if emptyFormat: &alert_id=<alert_id1>&alert_id=<alert_id2>

days

String

The number of days to obtain history of.Format: &days=1 (default: 3)

status

String

The status that was recorded. Cant be either ‘alerting’ or ‘recovered’.

value

String

The value at which the alert triggered or recovered. Value will be null if the alert was a missing metric alert.

time

String

Unix timestamp at which the alert triggered or recovered.

HTTP/1.1 200 
Content-Type: application/json 
{
  "alert_id_1": [
    {
      "status": "alerting",
      "value": 123,
      "time": "1672531200",
      "metric": "The metric that triggered this."
    }
  ],
  "alert_id_2": [
    {
      "status": "healthy",
      "value": null,
      "time": "1672531200",
      "metric": "The metric that triggered this."
    }
  ]
}

Bad Request Invalid query parameters

Curl example:

curl -X GET "https://YOUR-API-KEY@api.hostedgraphite.com/v2/alerts/history/?id=123-456-789

Obtain alert history of the alert identified by the id 123-456-789.

Alerts API
Creating Alerts
Composite Alerts
Searching Alerts
Obtain Information on a Single Alert
Updating Alerts
Deleting Alerts
Mute an Alert
Unmute an Alert
Check if an Alert is Muted
Mute Multiple Alerts
Unmute Multiple Alerts
Obtain Alert History
Creating Alerts
Composite Alerts
Searching Alerts
Obtain Information on a Single Alert
Updating Alerts
Deleting Alerts
Mute an Alert
Unmute an Alert
Check if an Alert is Muted
Mute Multiple Alert
Unmute Multiple Alerts
Obtain Alert History