hadoop - Hadoop 计数器可视化

Question

我想知道是否有任何 Hadoop 计数器和作业可视化库/项目。例如，显示每个作业类型以及开始和结束时间的时间显示，随时间绘制。另一个例子是随着时间的推移绘制一个给定的计数器。我认为这可以用于操作洞察、监控和警报。

Answer 1

Note: Since this is my first post on SO I cannot post more than two URLs. Please have a look at this GitHub gist for my post with all links in Markdown format. All italic items below are actually hyperlinks. Sorry for the inconvenience. --Michael

The only open source visualization project that comes to my mind right now is Twitter Ambrose. From the various tools I mention in this reply you might want to have a look at Ambrose first. It supports the following features in its web UI:

[Ambrose web UI features]

• A table view of all the associated jobs, along with their current state
• Chord and graph diagrams to visualize job dependencies and current state
• An overall script progress bar

Apart from that my personal experience has been with offerings from commercial vendors. To name but two of them:

• Cloudera Manager for the CDH4 distribution
• MapR's Dashboard for the M3 and M5 distributions

Both products come with an API that allows you to extend them and integrate them with your own Ops tool set. Cloudera Manager requires an evaluation license whereas MapR's Dashboard is available in the free M3 distribution if you want to give it a spin. As usual there are pros and cons for each of them.

That said, you can also configure standard Hadoop to sent its metrics to a monitoring tool such as Ganglia (see live demo at UC Berkely Grid). Basically, you just dump metrics into Ganglia and the latter will take care of the visualization/plotting of the various metrics. There are several online guides available that describe how to configure Ganglia for a small Hadoop cluster. If you are running Hadoop 2.x have a look at What is Hadoop Metrics2 for how the metrics system in next-gen Hadoop works in general.

Finally albeit a bit unrelated to your direct question, you can also write custom monitors by calling Hadoop's Java API. It is usually straight-forward to write these custom monitors in a way that is compatible with other Ops infrastructure tools such as Nagios. For instance, one of our custom monitors connects to the JobTracker in order to detect any MapReduce jobs that run for longer than 24 hours (which is in 99% a tell-tale that a job is broken one way or another). Depending on the tool you dump the metrics into you will get visualizations/graphs for free (cf. Ganglia example above).

Hope this helps, Michael