Re: Graph Utils (Open-Source)

[Max Pierson <nmaxpierson () gmail com>]

Hiya Jimmy!!

How has it been?

> For simple visualizations,  I think usually a 'native'  framework/API is
preferred, e.g. JGraph for java apps.

Unfortunately, I'm not savvy with Java at all, so the really cool viz API's
wont work for me (there's just something about Java ... I simply can't get
into it and I see alot of Java based apps that are resource hogs). I was
looking at mostly using some simple Perl + PHP (or even Python) for the
graph generation. My own cacti if you will, just not as feature filled
but template driven.

> If you need to paint a bunch of  arbitrary X and Y values   on a graph from
> an input file or based on simple equations,   gnuplot will happily
> oblige; it can
> handle chart types  rrdtool cannot, and you have more direct control of
output.

> If you want some 3D / surface graphs, RRDTool won't do it, anyways.
> Gnuplot's less expensive
> than Matlab / Maple.

This is why rrdtool won't work for me. I'll be inputting data sets for
charts that rrdtool doesn't understand and I would like to add a z-axis for
some data that would be better understood on a 3D plot. The Graph.pm has
hooks for xrt3d which looks pretty neat. I'm also looking at Grace as well
using that same module (xmgrace), and it also has gnuplot I can work with.
This way I can reuse code if I decide to use one over the other.

The other issue is that all of this data is already stored (at least most of
it) in mysql, and I don't need to waste cycles and chunks on riping data
from one DB and have to create another just to display a graph. Plus some of
the RRA's would be huge and therefore slow.


> Try plotting a test dataset with 500 million datapoints.
> there should normally be some form of averaging / smoothing / "selection of
points"  contemplated, if the dataset is huge

The few hundred million rows are spread out across different servers and are
not all the same table types. I believe it's 280-300 million rows by
first estimates, but they will be represented in different manors. I won't
be injecting the volume of inserts aggregate onto one plot over time.  I
haven't gone that mad yet :)   For measurements such as that, averaging will
be used for such trends.

Good talking to you again,
Max



On Sun, Feb 20, 2011 at 10:45 PM, Jimmy Hess <mysidia () gmail com> wrote:

> On Fri, Feb 18, 2011 at 1:13 PM, Max Pierson <nmaxpierson () gmail com>
> wrote:
> > Anyone out there using something other than rrdtool for creating graphs??
> I
> > have a project that will need a trend taken, and unfortunately rrdtool
> > doesn't fit the bill. All of the scripting, data collection,
> > database archival, etc will be custom written or is already done (with
> some
> > hacks of course :). So really what i'm looking for is something along the
> > lines of GNUplot. Has anyone used it before and would like to share
>
> I haven't heard of gnuplot used often with other software as a framework
> for graphing/visualizations. For simple visualizations,  I think usually a
> 'native'  framework/API is preferred, e.g. JGraph for java apps.
>
> I suspect one reason gnuplot is not used as widely as it could be otherwise
> is, its  licensing is not as  "friendly" as other graphics frameworks.
> gnuplot license is GPL incompatible and does not seem to even fully meet
> the open source definition,
>
> Because redistributing complete modified source code of gnuplot itself
> is not allowed by the license;  a clear reading of gnuplot license suggests
> only patches, unmodified source code, can be freely redistributed,
> redistributed binaries based on modified source have special rules).
>
>
> Aside from that caveat,  which most likely does not normally impair private
> use by a network operator: gnuplot is a really good tool.
> If you need to paint a bunch of  arbitrary X and Y values   on a graph from
> an input file or based on simple equations,   gnuplot will happily
> oblige; it can
> handle chart types  rrdtool cannot, and you have more direct control of
> output.
>
> If you want some 3D / surface graphs, RRDTool won't do it, anyways.
> Gnuplot's less expensive
> than Matlab / Maple.
>
> You can even set terminal type to "dumb" in gnuplot, and generate some
> fancy
> ASCII art graphs on stdout.
>
>
> In regards to scalability...
>
> About the millions of rows... err..
> Try plotting a test dataset with 500 million datapoints.    Chances
> are gnuplot won't
> necessarily scale that well either, and you need some method to be
> selective of which rows are
> provided as input to the plotting framework, in that case.
>
> If you have a  million datapoints on your X axis,  each X position is
> smaller than  1/1000 of
> a display pixel   (on a graph that fits on a display at say  1920x1080);
> displaying such high resolution of all datapoints at once on the
> unzoomed graph is beyond
> the display hardware capabilitiy.
>
> there should normally be some form of averaging / smoothing /
> "selection of points"  contemplated,
> if the dataset is huge
>
> > experiences?? Seems like it will be able to my plot data accordingly, but
> > wanted to see if there were any other popular tools I've yet to come
> across.
> > (Open-Source only please)
>
> --
> -JH