Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
  • Locked thread
bolind
Jun 19, 2005



I have a scenario where we're trying to collect a bunch of data and graph it. Data comes in at a not alarming rate; somewhere between once a day and once a minute. Things should ideally run on Linux and have a html based interface for displaying the data. Currently we're running an RRDtool based solution, which is kinda OK, but not quite.

What I'd like to do:

  • Have something collect data and stuff it into a fitting database. RRDtool does this very well. Problem is that data sources can come and go. Maybe we track metric A, B and C one week, and then A, B and D. And then maybe back to A, B and C again two weeks later. This must be supported.
  • Have something else that can display this data in nice graphs, ideally with the ability to zoom dynamically etc. Something like this: http://www.highcharts.com/demo/line-time-series
  • Be able to configure custom "views", basically being able to bundle metrics in graphs at will. Ideally with some sort of hierarchical structure (folder containing some views/graphs.)
  • Data must be able to participate in more than one graph.

Does anything like this exist? I can' t be the first nerd on this here planet who has had that need.

Edit: researching a bit, it seems like RRDtool should definitely be the backend for this.

bolind fucked around with this message at Aug 16, 2016 around 10:53

Adbot
ADBOT LOVES YOU

Raskolnikov2089
Nov 3, 2006
Schizzy to the matic

D3.js should be able to do all of your front end without a hassle, but there's a steep learning curve.

Chart.js might be a little easier to get up and going, though it's not as full featured as D3.

akadajet
Sep 14, 2003



Slippery Tilde

You could glue all this together using Node/Java/Whatever and Highcharts (since you mentioned you like the presentation). But I guess you want some kind of ready-made product to do it all for you. Highcharts is relatively easy to use FWIW.

akadajet fucked around with this message at Oct 9, 2016 around 16:22

jre
Sep 2, 2011

To the cloud ?





bolind posted:

I have a scenario where we're trying to collect a bunch of data and graph it. Data comes in at a not alarming rate; somewhere between once a day and once a minute. Things should ideally run on Linux and have a html based interface for displaying the data. Currently we're running an RRDtool based solution, which is kinda OK, but not quite.

What I'd like to do:

  • Have something collect data and stuff it into a fitting database. RRDtool does this very well. Problem is that data sources can come and go. Maybe we track metric A, B and C one week, and then A, B and D. And then maybe back to A, B and C again two weeks later. This must be supported.
  • Have something else that can display this data in nice graphs, ideally with the ability to zoom dynamically etc. Something like this: http://www.highcharts.com/demo/line-time-series
  • Be able to configure custom "views", basically being able to bundle metrics in graphs at will. Ideally with some sort of hierarchical structure (folder containing some views/graphs.)
  • Data must be able to participate in more than one graph.

Does anything like this exist? I can' t be the first nerd on this here planet who has had that need.

Edit: researching a bit, it seems like RRDtool should definitely be the backend for this.


Use grafana for visualisation , then pick a backend from graphite / opentsdb / influx

If you are used to docker, you can quickly get containers for grafana , graphite and tsdb to test it out

adorai
Nov 2, 2002

10/27/04 Never forget

Grimey Drawer

ELK stack sounds about right for this.

Novo
May 13, 2003

Stercorem pro cerebro habes

Fun Shoe

seconding influxdb + grafana

rrdtool is fine but i don't know how to use it with grafana, which is a real pleasure once you get the hang of it

bolind
Jun 19, 2005



ANSWERS! Super cool, thanks people, I'll look into all of those.

ColonelDimak
May 1, 2007

Guardian of the Salsa

ELK stacks all day erry day.
For reals though, Elastic Search, Kibana and Logstash are what you want.
It'll take any input if you can parse it with JSON and there are a bunch of prebuilt stacks for specific use cases.

The two I use the most are,
SexiLog for VMware, windows events and Veeam SNMP logging
Graylog for netflow monitoring.

Highly recommended everyone check out elk stacks and if you have any questions let me know.

Walked
Apr 14, 2003



ColonelDimak posted:

ELK stacks all day erry day.
For reals though, Elastic Search, Kibana and Logstash are what you want.
It'll take any input if you can parse it with JSON and there are a bunch of prebuilt stacks for specific use cases.

The two I use the most are,
SexiLog for VMware, windows events and Veeam SNMP logging
Graylog for netflow monitoring.

Highly recommended everyone check out elk stacks and if you have any questions let me know.

It's been a while since I've looked at setting up ELK; but I recall a lot of documentation was a mishmash of out of date or inconsistent.

Any good relatively recent configuration guides for a Windows focused environment?

Nahrix
Mar 17, 2004

Can't afford to eat out

Sumo Logic sounds a lot like what you need.

https://www.sumologic.com/

Adbot
ADBOT LOVES YOU

jre
Sep 2, 2011

To the cloud ?





ColonelDimak posted:

ELK stacks all day erry day.
For reals though, Elastic Search, Kibana and Logstash are what you want.
It'll take any input if you can parse it with JSON and there are a bunch of prebuilt stacks for specific use cases.

The two I use the most are,
SexiLog for VMware, windows events and Veeam SNMP logging
Graylog for netflow monitoring.

Highly recommended everyone check out elk stacks and if you have any questions let me know.

ELK is great for event logging and pretty much useless for metric ( time series ) data despite that one elastic.co blog claiming otherwise. The OPs use case is for time series as they are already using rrd.

jre fucked around with this message at Jan 4, 2017 around 21:56

  • Locked thread