I’ve been using Observium for a very long time. Looking through this
site recently, I realised that quite a few of my old Observium articles
are still online, but I’ve never really written down how I became
involved with the project in the first place.

It goes back to my time at eBay and eBay Classifieds Group, where I was
working on the network infrastructure behind businesses including
Marktplaats.nl.

At that scale, monitoring obviously wasn’t optional. We had a large
amount of network infrastructure and needed to know not only whether
devices were reachable, but what was actually happening inside the
network: interfaces, BGP sessions, processors, memory, storage, sensors
and all the other things that can tell you that something is going wrong
before it becomes visible to users.

I came across Observium, which at the time was still a relatively
young project being developed primarily by Adam Armstrong.

I got to know Adam quite well as we started using Observium more
seriously and became more involved with its development, and over the
years I’ve come to consider him a personal friend. At the time, though,
the reason I became interested in Observium was simply that I liked its
approach to network monitoring.

It was very good at automatically discovering what was actually present
on a device. Instead of spending huge amounts of time manually defining
every interface, sensor and BGP session, Observium could discover the
network and build a useful picture of it automatically.

We started using it, and I became increasingly involved with the
project.

One of the things that was missing at the time was a proper alerting
system.

Observium was already very good at showing you the state of a network,
but in a production environment we also needed it to actively tell us
when that state changed in a way that required attention. Monitoring
something and periodically looking at a graph is one thing; knowing
immediately that a BGP peer has gone down or that an important resource
has crossed a threshold is another.

Through eBay/Marktplaats, I sponsored development of Observium and
specifically had the original alerting functionality developed.

The idea was to build alerting around the entities Observium was already
discovering. Rather than maintaining a completely separate set of
monitoring definitions, alerts could operate on things Observium already
understood: devices, ports, BGP peers, processors, memory, storage and
sensors.

That eventually became the entity-based alerting system.

Once it existed, the next problem was actually configuring it properly.

The alerting system was flexible, but that also meant you needed to
understand its metrics, attributes, conditions and association rules.
There weren’t many ready-made examples available at the time, so in 2014
I wrote “Useful Observium alert checker examples” on this site.

That article contained examples for the things I thought most network
operators would actually want to monitor: devices going down, CPU and
memory utilisation, BGP sessions, interface errors and discards, storage
utilisation, traffic levels and sensors.

Looking at it again more than a decade later is quite interesting
because the terminology and implementation have evolved, but much of the
basic alerting model is still recognisable.

Alerting continued to develop after that.

One limitation of the original system was that it was poller based. If
the poller ran every five minutes, there were some events that you
either wouldn’t see immediately or couldn’t sensibly detect through
polling at all.

That led to another feature I was interested in: syslog alerting.

When that became available in Observium, I wrote another article
explaining how to use it. Syslog alerting made it possible to react in
real time to messages generated by network devices—for example OSPF
events, duplicate IP or MAC-address warnings and other conditions that
don’t fit naturally into normal SNMP polling.

Over the years I wrote several other Observium-related articles and bits
of documentation, and maintained some of the material as the software
changed.

I wasn’t one of Observium’s developers, and I don’t want to rewrite
history to suggest that I was. Adam and the other Observium developers
built the software. My involvement was from the other side: I was
running networks, using the software in a real production environment,
sponsoring development of functionality we needed and helping document
how some of it could actually be used.

The alerting system was probably the most substantial part of that
involvement.

Observium has developed enormously since then. The current system has
alert checkers, notification transports, scheduled maintenance, an API,
syslog alerting and considerably more functionality than the version we
were working with at the time.

It’s nice to see that something we needed for our network more than a
decade ago became part of a monitoring system that is still actively
developed and used today.

And it’s also why there are so many old Observium posts scattered around
this website.

I wasn’t writing them as blog content.

I was writing down how we got things to work.

My first professional job in IT was at Agrotel, in the 1990s.

Looking back at it now, it was a good moment to start working with
computers because the technology we were using was changing underneath
us. I started as a programmer in an environment that had been built
around Videotex, and during my time there the Internet and World Wide
Web were rapidly taking over.

Agrotel had been around before I joined. Its roots were in agricultural
electronic information services that predated the Web. Information was
distributed through Videotex, or VTX, allowing users to connect to
central services and retrieve pages of agricultural information
electronically.

By the time I was working there, however, it was obvious where things
were going.

The Internet was growing quickly and the Web provided a much better way
of distributing information. The problem was that there was already a
lot of useful information and data sitting in systems designed for the
old environment.

One of the things I worked on was software to automatically convert
information from the existing systems into HTML. One of my programs was
simply called DAT2HTML — a conversion program that took data
intended for the old information environment and generated webpages from
it.

Some of that program’s output has, somewhat accidentally, survived on my
website to this day.

There is an old test page in /grafiek/ whose original HTML title is
still “Agrotel Conversie programma DAT2HTML.” It generates a
graphical presentation of a series of values using HTML and tiny GIF
elements — very much the way these things were done before browsers
had SVG, Canvas, JavaScript charting libraries or any of the tools we’d
use for the same job today.

https://www.maartenmoerman.nl/grafiek/TEST.HTM

There is even an old WS_FTP.LOG sitting next to it. It shows the files
being transferred on 14 January 2004, first involving a backup under my
Windows profile and later being uploaded to /home/mgm/www/grafiek on
one of my machines.

The files themselves may be older — the log refers to them as coming
from a backup — so I don’t regard 2004 as the date I originally wrote
DAT2HTML. But it is a nice little timestamp that happened to survive
with the files.

https://www.maartenmoerman.nl/grafiek/WS_FTP.LOG

DAT2HTML is a good example of the sort of software I was writing at the
time. We already had information and data in systems built around an
earlier generation of online services, while the Web was becoming the
place where that information needed to be. Rather than manually
rebuilding everything as webpages, I wrote tools to automate as much of
that process as possible.

Looking at the output today is quite funny. Generating a graph using a
collection of tiny GIF files and HTML layout is obviously not how anyone
would approach the problem now. But browsers and HTML were considerably
more limited, and getting information from one system onto the Web often
meant simply inventing a practical way of doing it with what was
available.

The programming also brought me increasingly into contact with the
infrastructure underneath it. If we were going to provide these services
over the Internet, we needed the servers, connectivity, DNS, mail and
networking to actually run them.

So alongside writing software, I became involved in building an **ISP
environment**.

That was probably where my later career in network engineering really
started. I began as a programmer, but I found myself spending more and
more time with Unix systems, servers and networking.

I still have other remnants from those days online. There is, for
example, an old maarten.agroweb.nl Christmas page from 2000 sitting on
this domain, complete with my old @agroweb.nl email address.

It is a small surviving piece of a very different Internet.

Connections were slower, servers were physical machines you actually
knew, and if you wanted a service to exist there was a reasonable chance
you would end up configuring much of the infrastructure yourself.

Agrotel itself went through later organisational changes. One thing I do
want to make clear, because some online employment databases get this
wrong, is that I worked for Agrotel.

I did not work for the later Agroportal organisation. That came
afterwards.

After Agrotel my career moved further into Internet infrastructure. I
later worked at Solcon, where operating Internet infrastructure was
the actual business, and eventually moved almost entirely into network
engineering.

Years later I would be working on BGP, MPLS, data centres and network
automation for eBay and eBay Classifieds Group.

But a surprising amount of it started with trying to get agricultural
information out of an old information system and onto this new thing
called the World Wide Web.

At the time, of course, we didn’t think of it as Internet history.

It was just the new system we were building.

1 Minute polling for Observium, quite an interesting and useful feature if you ask me. Especially with large capacity interfaces such as 25/40/100gbit interfaces, there’s tiny differences which could be interesting to spot, and you just can’t see those when you are using 5 minute polling. Another interesting used case for example are your uplinks, and your core links in your network.

By default Observium is not setup to do this, and it requires a couple of changes. Now luckily, it’s not that hard to change this, as the configuration settings for the RRDs are a bit hidden, but setup in such a way, that you can actually change this.

First before we get into the details on how to actually do this there are a couple of things you’ll need/want to know:

N E E D   T O    K N O W :

• These changes are an all or nothing setup. Meaning, all your things will be polled on a 1 minute interval, that is sensors, ports, cpu’s, memory, everything. No possibility to do just a couple of things on 1 minute intervals
• You NEED to do this with an empty RRD directory / new installation, as RRD files need to be recreated with the proper ‘time brackets’ configured in them
• I have not tested it on an existing installation, so i don’t know what will happen, you could probably loose your data, or at least corrupt it. I simply don’t know enough of RRDtool what will happen.
• Make sure all your devices finish their polling within 1 minute
• It’s NOT, and i repeat NOT an officially supported thing. Observium (Adam Armstrong) recommends against it
• There will be a 5time increase in load on your server
• Your disk IO will increase by 5 times as well. Keep that in mind if your are for example runnning your RRD directory on an SSD, as this will wear out 5 times as fast

Configuration
Ok, let’s see how to configure this:

In the default configuration thats deployed once you have downloaded it, the RRD settings are configured for:

• 7 days of 5 minutes
• 62 days of 30 minutes
• 120 days of 2 hours
• 4 years of 1 day

Now these settings are perfectly acceptable for me, except the first one ofcourse. We’ll want that to change to 1 minute.

You’ll need to make changes in 2 places:

• Observium configuration file
• Crontab

Your configuration is stored in, and you can lookup them up in the file /opt/observium/includes/defaults.inc.php

It looks like this:


// Default Poller Interval (in seconds)
$config['rrd']['step'] = 300;
// 7 days of 5 min 62 days of 30 min 120 days of 2 hour 4 years of 1 day
$config['rrd']['rra'] = "RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:2976 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440 ";
$config['rrd']['rra'] .= " RRA:MIN:0.5:6:1440 RRA:MIN:0.5:96:360 RRA:MIN:0.5:288:1440 ";
$config['rrd']['rra'] .= " RRA:MAX:0.5:6:1440 RRA:MAX:0.5:96:360 RRA:MAX:0.5:288:1440 ";


Now you could modify the RRD settings in this file, but that’s not really helping if the default settings get adjusted when you upgrade your Observium instance. So change it in the file /opt/observium/config.php

Change 2 things in that piece of text:

• the rrd step size from 300 to 60
• the first RRA value from RRA:AVERAGE:0.5:1:2016 to RRA:AVERAGE:0.5:1:10080

Once this has been changed, change your cronjob (/etc/cron.d/observium) to run poller-wrapper / poller every minute

That should be it! Enjoy your 1 minute resolution in Observium!

Observium SYSLOG alerting

I maintain an updated version on Github: https://github.com/mgmoerman/docs/blob/master/observium-syslog-alerting.md

Observium has builtin syslog alerting since release r7970. This text will explain on what it does, how it works, and how you configure it.

Something that was missing in the entity based alerting system, is now fixed with this brand new feature of Observium.

You can now perform realtime alerting where as the conventional entity based alerting only would trigger every 5 minutes, because it is poller based.

Now this new feature is not going to replace the existing entity alerting, as it just serves a different purpose, as it just allows you to catch issues from your devices, which are not possible to catch with the existing entity based alerting system. Examples are: Duplicate IPs/MAC addresses, or OSPF messages.

Syslog alerting allows you to send out notifications from syslog messages that are produced by your devices. You can match on those with Regular Expressions

Syslog alerting in observium perfectly integrates with the existing contact system, so it allows you to notify via the usual channels, E-mail, Slack, Pagerduty, XMPP, webhook, etc.

For a complete overview of transport methods, see: Alerting Transports

How to set it up

First make sure you have configured syslog to integrate with Observium. The documentation for doing this, can be found here: Syslog Integration

If you are running r7970 or later you will find 2 new entries in the global menu:

• Syslog Alerts
• Syslog Rules

Let’s start with creating a useful syslog alert rule, that triggers an alert when there is a duplicate mac address found on a Cisco device:

• First click on “Syslog Rules” in the global menu
• Then click on “Add Syslog Rule”

You will then be presented with the following screen, where you have to configure the details of the syslog alert rule:

• Rule Name: This defines the name for the actual rule, this is just an administrative reference
• Message: This is the message that will end up in the notification send out
• Regular Expression: This is where you configure the actual rule to match syslog content against

Syslog Rules are built using standard PCRE regular expressions.

There are many online resources to help you learn and test regular expressions. Good resources include regex101.com, Debuggex Cheatsheet, regexr.com and Tutorials Point. There are many other sites with examples which can be found online.

A simple rule to match the word “duplicate” could look like:

/duplicate/

A more complex rule to match SSH authentication failures from PAM for the users root or adama might look like:

/pam.+\(sshd:auth\).+failure.+user\=(root|adama)/

Useful syslog alerts

Here are a couple of alerts you could implement which come in pretty handy:

Sending out notifications

To actually send out notifications, you will have to associate the syslog alert rule with the contact. To do this, edit the contact that you have configured and add the syslog rule association:

Select from the drop down a syslog alert rule, and click “+ Associate”. Once you have done this, the association is completed

If you associate it to an email contact, the notification will look like this:

Custom SNMP OIDs in Observium, how cool is that! Since release 7175 of Observium Professional, it is now possible to have these custom SNMP OIDs monitored and graphed. This is quite handy, as writing individual device support for devices that are not recognized by Observium itself can be hard for people, as you need to dive into PHP, and Observiums codebase, and the code you might turn out might not be as beautiful as the Observium Dev team likes it to see, so it won’t be merged.

Now some people can perfectly live with a device being added, and shown up as a “Generic Device”, and have all the basic funtionality of the standard MIBS. This works to a certain extend off course. Now what if you just want to add something weird (SNR, RF levels for example) that is not supported by Observium out of the box, wether it be a standard graph, or an entity type that is not supported, this is now possible!

So here is how it works:

Click on the “Custom OID” menu in the global menu (the one with the globe), and you’ll get an overview of the custom OIDs configured by you.

Then click on the “Add Custom OID” option in the right top corner to get the input fields:

There’s a couple of fields that you’d have to fill in:

– Numeric OID: This is the OID in SNMP that you want to monitor
– Text OID: Right now, this doesn’t do much, but it’s advisable to use the MIB translated name for this
– Value type: Specific wether the value you are retrieving is a GAUGE or a COUNTER (for information on what this is, i advise you to read the RRD documentation on the differences between the two.
– Description: This is what the name of the entry is in the overview
– Display Units: This is the text that will be displayed in the legend of the graph that’s being created

After you’re done with filling in these fields, submit it, and you’ll be returned to the overview of custom OIDs. By now, this isn’t doing anything yet.
What you have to do is associate the custom OID with a device of your choosing. This association is pretty nifty, as it allows you to only write a custom OID once, and the associate with an unlimited amount of devices:

Once this is done, and you have the correct OID being polled, and after 15minutes of polling, you should be able to see graphs in the Custom Graphs part of the device:

This page is outdated as of Feb 26 2015, as state sensors have been rewritten. I keep an updated document on:
https://github.com/mgmoerman/docs/blob/master/observium-alert-checkers.md

Observium straight out of the SVN repository (if you bought the subscription) doesn’t come with alert-checkers, which is unfortunate, as you need to figure out how this alerting system works by trial and error. Goal of this blog post is to give some examples of generic alert-checkers, and provide some more explanation on Metrics & Attributes, and some of the values that go with it. This document is off course not complete, and can always be improved. Please give me feedback to improve this.

Observium has a very powerful way of using entity types & check conditions to do alerting. But you do need to know how this is implemented.

There is some documentation on the Observium site itself, which is useful to read:

• http://www.observium.org/wiki/Creating_an_Alert_Checker
• http://www.observium.org/wiki/Alerting_Metrics_and_Attributes

Creating an alert checker

Let’s go through the steps that are involved to actually create/add an alert checker in Observium

Entity type

First of all when you create an alert,you’ll need to pick the ‘entity’ type for what you are building the alert for. An entity type is nothing more than a “thing” for which you would like to see alerts.

These are the ones that are available as of 12/12/2014:

• Device
• Memory
• Storage
• Processor
• BGP Peer
• Netscaler vServer
• Netscaler Service
• Toner
• Port
• Sensor

They kinda speak for them selves, if you want alerts on things that go on with ports, pick ports, if you want something that has to do with a sensor, pick that one. Device is a very generic one, and will just give you status things on wether it’s up/down and it’s uptime and the response time for ping/snmp, the entity type Device has nothing to do with Ports or Sensor on the device itself, for alerting for that, pick actually Ports or Sensor

Alert Checker details

Once you picked the entity type, there’s a couple of more things that need to be filled in but these are simple, pick a name for the alert, and pick a message you want to be included once an alert is sent out.

Use Alert Delay to set the amount of poller runs that a condition of your alert checker should persist until it actually starts alerting. This could be useful when for example you’re creating a check for processor usage, but you don’t want to be alerted on every CPU spike that is happening. If you set a delay of say, 2, it’ll take 2 poller runs for actually alerting (providing the condition for which you are checking hasn’t changed off course)

Send Recovery button is self explanatory, and the Severity is currently not in use

Checker Conditions

Then we come to the Checker Conditions, this is where you actually implement the check for a specific entity.

It’s important to know what Metrics & Attributes are, see the overview below for a complete list of Metrics & Attributes

When filling in the fields for Checker Conditions, you use the Metrics mentioned in this page.

These need to be single lined entries, you can put as much in there if you want but you usually have one to check for a single condition, or two, for example to check an upper and lower limit. Use the boolean to switch between ANY or ALL of these conditions to match.

A single line consists of three values:

• the actual metric
• a “test” (le, ge, lt, gt, ne, match and notmatch)
• a value

Associations

In these input fields you’ll create the first association rule, in other words, which subset of the entity type you selected needs alerting based on the conditions specified in the previous pane. When initially creating an alert checker, it allows for ony 1 association rule. Once it’s added, you can later on add more association rules to it.

These association rules are made from a “device association” and an “entity association”. First input field you’ll do your device matching, based on the attributes for devices. Second input field you’ll do your entity matching, using the attributes for the entity type you want to associate it with (this can off course be different then the condition you’re checking for)

This works in sort of the same way as the Checker Conditions. It uses the same line method (metric,test,value), however with some exceptions:

• instead of using metrics, you’ll be using attributes
• you can’t use a device attribute twice in the same association rule, so for example multiple “hostname match bla” statements with in the same association rule won’t work
• for a single device association line, you can have multiple entity association lines

That last exception allows for more specific filtering, for example, you would want to match against all sensor classes (sensor_class) that are of type “state”, but when that nets you to many results, you can add a match for it’s description (sensor_descr), or you’d want to match all ports of type (ifType) ethernetCsmacd, but you only want certain ones with a specific description (ifAlias)

Example alerts

If you scrolled down here to just copy/paste some alert-checkers, perfectly fine, but don’t complain if they don’t work, PLEASE read how these work above.

The following is a set of very useful alert checkers:

Per entity overview of Attributes , Metrics and their values (if any)

Device

Port

Memory

Processor

Storage

BGP Peer

Sensor

Toner

Netscaler vServers

Netscaler Services