大数据平台监控指标整理


hadoop metrics2

监控的内容:
1. yarn
2. jvm
3. rpc
4. rpcdetailed
5. metricssystem
6. mapred
7. dfs
8. ugi

已经提供的:

Source : org.apache.hadoop.metrics2.source.JvmMerticsorg.apache.hadoop.metrics2.source.JvmMetricsInfo

其他相关

FSOpDurations : org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSOpDurations

fairscheduler-op-durations context :

  @Metric("Duration for a continuous scheduling run")  MutableRate continuousSchedulingRun;
  @Metric("Duration to handle a node update")  MutableRate nodeUpdateCall;
  @Metric("Duration for a update thread run")  MutableRate updateThreadRun;
  @Metric("Duration for an update call")  MutableRate updateCall;
  @Metric("Duration for a preempt call") MutableRate preemptCall;

QueueMetrics : org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics

yarn context :

  @Metric("# of apps submitted") MutableCounterInt appsSubmitted;
  @Metric("# of running apps") MutableGaugeInt appsRunning;
  @Metric("# of pending apps") MutableGaugeInt appsPending;
  @Metric("# of apps completed") MutableCounterInt appsCompleted;
  @Metric("# of apps killed") MutableCounterInt appsKilled;
  @Metric("# of apps failed") MutableCounterInt appsFailed;
  @Metric("Allocated memory in MB") MutableGaugeInt allocatedMB;
  @Metric("Allocated CPU in virtual cores") MutableGaugeInt allocatedVCores;
  @Metric("# of allocated containers") MutableGaugeInt allocatedContainers;
  @Metric("Aggregate # of allocated containers") MutableCounterLong aggregateContainersAllocated;
  @Metric("Aggregate # of released containers") MutableCounterLong aggregateContainersReleased;
  @Metric("Available memory in MB") MutableGaugeInt availableMB;
  @Metric("Available CPU in virtual cores") MutableGaugeInt availableVCores;
  @Metric("Pending memory allocation in MB") MutableGaugeInt pendingMB;
  @Metric("Pending CPU allocation in virtual cores") MutableGaugeInt pendingVCores;
  @Metric("# of pending containers") MutableGaugeInt pendingContainers;
  @Metric("# of reserved memory in MB") MutableGaugeInt reservedMB;
  @Metric("Reserved CPU in virtual cores") MutableGaugeInt reservedVCores;
  @Metric("# of reserved containers") MutableGaugeInt reservedContainers;
  @Metric("# of active users") MutableGaugeInt activeUsers;
  @Metric("# of active applications") MutableGaugeInt activeApplications;

FSQueueMetrics : org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics

yarn context :

  @Metric("Fair share of memory in MB") MutableGaugeInt fairShareMB;
  @Metric("Fair share of CPU in vcores") MutableGaugeInt fairShareVCores;
  @Metric("Steady fair share of memory in MB") MutableGaugeInt steadyFairShareMB;
  @Metric("Steady fair share of CPU in vcores") MutableGaugeInt steadyFairShareVCores;
  @Metric("Minimum share of memory in MB") MutableGaugeInt minShareMB;
  @Metric("Minimum share of CPU in vcores") MutableGaugeInt minShareVCores;
  @Metric("Maximum share of memory in MB") MutableGaugeInt maxShareMB;
  @Metric("Maximum share of CPU in vcores") MutableGaugeInt maxShareVCores;

MetricsSystemImpl : org.apache.hadoop.metrics2.impl.MetricsSystemImpl

metricssystem context :

  @Metric({"Snapshot", "Snapshot stats"}) MutableStat snapshotStat;
  @Metric({"Publish", "Publishing stats"}) MutableStat publishStat;
  @Metric("Dropped updates by all sinks") MutableCounterLong droppedPubAll;

Sink : org.apache.hadoop.metrics2.sink.GraphiteSinkorg.apache.hadoop.metrics2.sink.FileSink 以及org.apache.hadoop.metrics2.sink.AbstractGangliaSink

metricsSystem : org.apache.hadoop.metrics2.lib.DefaultMetricsSystem

自我实现:

  1. Source : org.apache.hadoop.metrics2.MetricsSource
  2. Sink : org.apache.hadoop.metrics2.MetricsSink
  3. MetricsSystem : org.apache.hadoop.metrics2.MetricsSystem

使用方式:

$HADOOP_HOME/etc/hadoop/hadoop-metrics2.properties中配置就可以

spark metrics

可以获取到的内容主要有:
1. master
2. applications
3. worker
4. executor
5. driver

实现的Sink有

ConsoleSink : org.apache.spark.metrics.sink.ConsoleSink

CSVSink : org.apache.spark.metrics.sink.CSVSink

JmxSink : org.apache.spark.metrics.sink.JmxSink

MetricsServlet : org.apache.spark.metrics.sink.MetricsServlet

GraphiteSink : org.apache.spark.metrics.sink.GraphiteSink

Slf4jSink : org.apache.spark.metrics.sink.Slf4jSink

实现的Source有

JvmSource : org.apache.spark.metrics.source.JvmSource

ApplicationSource : org.apache.spark.deploy.master.ApplicationSource

  1. status
  2. runtime_ms
  3. cores

BlockManagerSource : org.apache.spark.storage.BlockManagerSource

  1. maxMem_MB
  2. remainingMem_MB
  3. memUsed_MB
  4. diskSpaceUsed_MB

DAGSchedulerSource : org.apache.spark.scheduler.DAGSchedulerSource

  1. failedStages
  2. runningStages
  3. waitingStages
  4. allJobs
  5. activeJobs

ExecutorAllocationManagerSource : package org.apache.spark.ExecutorAllocationManagerSource

  1. numberExecutorsToAdd
  2. numberExecutorsPendingToRemove
  3. numberAllExecutors
  4. numberTargetExecutors
  5. numberMaxNeededExecutors

ExecutorSource : org.apache.spark.executor.ExecutorSource

  1. activeTasks
  2. completeTasks
  3. currentPool_size
  4. maxPool_size
  5. read_bytes
  6. write_bytes
  7. read_ops
  8. largeRead_ops
  9. write_ops

MasterSource : org.apache.spark.deploy.master.MasterSource

  1. workers
  2. aliveWorkers
  3. apps
  4. waitingApps

MesosClusterSchedulerSource : org.apache.spark.scheduler.cluster.mesos.MesosClusterSchedulerSource

  1. waitingDrivers
  2. launchedDrivers
  3. retryDrivers

WorkerSource : org.apache.spark.deploy.worker.WorkerSource

  1. executors
  2. coresUsed
  3. memUsed_MB
  4. coresFree
  5. memFree_MB

注意!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系我们删除。



GreenPlum 大数据平台--web监控 大数据测试指标(二) 大数据测试指标 大数据框架整理 大数据平台的选择 大数据平台搭建 大数据平台创新 大数据平台搭建 浪潮大数据平台 大数据平台集群
 
粤ICP备14056181号  © 2014-2020 ITdaan.com