Windows>> hadoop-hdfs-fsimage-exporter>> 返回
项目作者： marcelmay

项目描述 ：
Exports Hadoop HDFS content statistics to Prometheus
高级语言： Java
项目地址: git://github.com/marcelmay/hadoop-hdfs-fsimage-exporter.git
创建时间： 2017-07-17T20:59:40Z
项目社区：https://github.com/marcelmay/hadoop-hdfs-fsimage-exporter
开源协议：Apache License 2.0
下载


Prometheus Hadoop HDFS FSImage Exporter

Apache License, Version 2.0, January 2004
Maven Central
Docker Pulls
ci
CodeQL

Exports Hadoop HDFS statistics to Prometheus monitoring including

  • total / per user / per group / per configured directory path / per set of paths
    • number of directories
    • number of files
    • file size and optionally size distribution
    • number of blocks
    • file replication (overall / per user summary)

The exporter parses the FSImage using the Hadoop FSImage Analysis library.
This approach has the advantage of

  • being fast (2.6 GB FSImage ~ 50s)
  • adding no heavy additional load to HDFS NameNode (no NameNode queries, you can run it on second NameNode)

The disadvantage is

  • no real time update, only when NameNode writes FSImage (interval of hours, see
    dfs.namenode.checkpoint.period).
    This should be sufficient for most cases (long-term trend, detecting HDFS small file abuses, user and group stats)
  • parsing takes 2x-3x FSImage size in heap space

FSImage Exporter overview

The exporter parses fsimage data in background thread which checks every 60s for fsimage changes.
This avoids blocking and long-running Prometheus scrapes and potentially stale metrics.

Grafana dashboards

Requirements

For building:

  • JDK 8
  • Maven 3.9.x
  • Docker (only required if building docker image)

For running:

  • JRE 8+ for running (JRE 17+ LTS recommended)
  • Access to Hadoop FSImage file
  • Docker (only required if building docker image)

Downloading

Available on Maven Central and as docker image on Docker Hub

Building

mvn clean install

You can test the exporter using run_example.sh after building.

For building including docker image, run:

mvn clean install -Pdocker

You can run the docker image via maven, too:

mvn clean install docker:run -Pdocker

Or directly using docker command line

  1. docker run -i -t -p 9709:9709 -v $PWD/src/test/resources:/fsimage-location \
  2.            -e "JAVA_OPTS=-server -XX:+UseG1GC -Xmx1024m" \
  3.            marcelmay/hadoop-hdfs-fsimage-exporter

When running the docker image via Maven, docker will mount the projects’ src/test/resources directory (with test fsimage) and expose the exporter on http://0.0.0.0:9709/ .

Installation and configuration

  • Install the JAR on a system where the FSImage is locally available (e.g. name node server).

  • Configure the exporter
    Create a yml file (see example.yml):

    1. # Path where HDFS NameNode stores the fsimage files
    2. # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.namenode.name.dir
    3. fsImagePath : 'src/test/resources'
    5. # Skip file size distribution for group based stats
    6. skipFileDistributionForGroupStats : true
    8. # Skip file size distribution for user based stats
    9. # Enable for figuring out who has too many small files.
    10. skipFileDistributionForUserStats : false
    12. # Compute per path stats
    13. # Supports regex matching per path part
    14. paths:
    15.   - '/tmp'
    16.   - '/datalak.?/a.*'
    17.   - '/hive_warehouse/.*/.*'
    18.   - '/user/m.*'
    20. # Skip file size distribution for path based stats
    21. skipFileDistributionForPathStats : true
    23. # Path sets are grouped paths by an identifier.
    24. # The exporter computes for each identifier the stats.
    25. # Compared to simple "paths" above, this allows to specify several paths for one stat computation.
    26. pathSets:
    27.   'userMmAndFooAndAsset1' : [
    28.     '/datalake/asset3',
    29.     '/user/mm',
    30.     '/user/foo'
    31.     '/user/b.*' # With regexp for path part
    32.     ]
    33.   'datalakeAsset1and2' : [
    34.     '/datalake/asset1',
    35.   '/datalake/asset2'
    36.   ]
    38. # Skip file size distribution for path sets based stats
    39. skipFileDistributionForPathSetStats : true

    Note that the flag toggling file size distribution switches between Summary (few time series)
    and Histogram (many time series)

  • Run the exporter

    1.   > java -jar target/fsimage-exporter.jar
    2.   Usage: WebServer [-Dlog.level=[WARN|INFO|DEBUG]] <hostname> <port> <yml configuration file>

    Example JVM opts (-Xmx max heap depends on your fsimage size): 

    1. > java -Xmx1024m -dsa -server -XX:+UseG1GC \
    2.        -jar target/fsimage-exporter-1.0-SNAPSHOT.jar \
    3.        0.0.0.0 9709 example.yml

    Note: Make sure to size the heap correctly. As a heuristic, you can use 3 * fsimage size.

    Note: Previous versions up to 1.3 use the default port 7772 instead of 9709

  • Test the exporter
    Open http://\:\/metrics or http://\:\/ (for configuration overview)

  • Add to prometheus

    1. - job_name: 'fsimage'
    2.     scrape_interval: 180m # Depends on how often the name node writes a fsimage file.
    3.     scrape_timeout:  200s # Depends on size
    4.     static_configs:
    5.       - targets: ['<hostname>:<port>']
    6.         labels:
    7.           ...

    Note:
    For Grafana, you want to sample more often with a scrape interval of minutes.
    The exporter caches previously parsed FSImage, so it is a fast operation.

Metrics

The metrics follow the naming and labelling pattern fsimage[AGG][NAME][METRIC TYPE SPECIFIC] where

  • AGG is the aggregation level

    • Empty aggregation for overall stats
      Example for total number of files: fsimage_fsize_count{}
    • user for by user aggregations
      Example for number of files for a user: fsimage_user_fsize_count{user_name="foo"}
    • group for by group aggregations
      Example for number of files for a group: fsimage_group_fsize_count{group_name="nobody"}
    • path for by configured path aggregations
      Example for total size in path: fsimage_path_fsize_sum{path="/datalake/asset2"}
    • path_set for by configured path set aggregations
      A path set aggregates for several configured paths and is identified by name
      Example for total number of files for given path set logs : fsimage_path_set_fsize_count{path_set="logs"}

  • NAME can be

    • fsize for tracking file size and file number
    • csize for tracking consumed/erasure-encoded file size and file number
    • replication for tracking file replication
    • blocks for number of file data blocks
    • dirs for tracking number of files
    • links for tracking number of symbolic links

  • METRIC TYPE SPECIFIC depends on metric type (Counter, Gauge, Histogram, Summary …)

Details

  • File size fsize and csize

    • Tracks number of files and file size
    • csize is the consumed/erasure-encoded size
    • Type: Depends on configuration flag skipFileDistribution[ForUser|ForGroup|ForPath|ForPathSets]Stats
      • Summary : when flag is false (default)
      • Histogram : when flag is true
        Note that you can also configure the buckets via fileSizeDistributionBuckets
    • fsimage[AGG]fsize_count : The total number of files
    • fsimage[AGG]fsize_sum : The total number of bytes
    • fsimage[AGG]fsize_bucket{le=”“} : The number of files in this bucket range

  • Replication replication

    • Tracks file replication
    • Type: Summary
    • fsimage[AGG]replication_count
    • fsimage[AGG]replication_sum

  • File blocks

    • Tracks number of file blocks
    • Type: Gauge
    • fsimage[AGG]blocks

  • Directory dirs

    • Tracks number of directories
    • Type: Gauge
    • fsimage[AGG]dirs

  • Directory links

    • Tracks number of filesystem links
    • Type: Gauge
    • fsimage[AGG]links

Exporter internal metrics

Metric Type Descriptions
fsimage_exporter_build_info{appVersion, buildTime, buildScmVersion, buildScmBranch} Info Build info
fsimage_compute_stats_duration_seconds[_count,_sum] Summary Time for computing stats for a loaded/parsed FSImage (after parsing)
fsimage_load_duration_seconds[_count,_sum] Summary Time for loading/parsing FSImage
fsimage_load_file_size_bytes Gauge Size of raw FSImage file parsed
fsimage_scrape_duration_seconds Gauge Exporter scrape request duration (does not include fsimage load/parsing/stats-computation)
fsimage_scrape_errors_total Count Count of failed scrapes
fsimage_scrape_requests_total Count Total number of scrape requests received
jvm_memory* Exporter JVM memory settings, see MemoryPoolsExports

Example output

Example home output

Home output

Example metrics

Here’s the example output for the test fsimage, using example.yml configuration:

  1. # HELP fsimage_exporter_build_info Hadoop FSImage exporter build info
  2. # TYPE fsimage_exporter_build_info gauge
  3. fsimage_exporter_build_info{appVersion="1.4.11-SNAPSHOT",buildTime="2025-02-17/18:04",buildScmVersion="d643236608f4508d401e663a71e3a8fc5a7e53c3",buildScmBranch="master",} 1.0
  4. # HELP fsimage_dirs Number of directories.
  5. # TYPE fsimage_dirs gauge
  6. fsimage_dirs 14.0
  7. # HELP fsimage_blocks Number of blocks.
  8. # TYPE fsimage_blocks gauge
  9. fsimage_blocks 17.0
  10. # HELP fsimage_links Number of sym links.
  11. # TYPE fsimage_links gauge
  12. fsimage_links 0.0
  13. # HELP fsimage_user_dirs Number of directories.
  14. # TYPE fsimage_user_dirs gauge
  15. fsimage_user_dirs{user_name="root",} 0.0
  16. fsimage_user_dirs{user_name="foo",} 0.0
  17. fsimage_user_dirs{user_name="mm",} 14.0
  18. # HELP fsimage_user_blocks Number of blocks.
  19. # TYPE fsimage_user_blocks gauge
  20. fsimage_user_blocks{user_name="root",} 1.0
  21. fsimage_user_blocks{user_name="foo",} 2.0
  22. fsimage_user_blocks{user_name="mm",} 14.0
  23. # HELP fsimage_user_links Number of sym links.
  24. # TYPE fsimage_user_links gauge
  25. fsimage_user_links{user_name="root",} 0.0
  26. fsimage_user_links{user_name="foo",} 0.0
  27. fsimage_user_links{user_name="mm",} 0.0
  28. # HELP fsimage_group_dirs Number of directories.
  29. # TYPE fsimage_group_dirs gauge
  30. fsimage_group_dirs{group_name="root",} 0.0
  31. fsimage_group_dirs{group_name="supergroup",} 14.0
  32. fsimage_group_dirs{group_name="nobody",} 0.0
  33. # HELP fsimage_group_blocks Number of blocks.
  34. # TYPE fsimage_group_blocks gauge
  35. fsimage_group_blocks{group_name="root",} 1.0
  36. fsimage_group_blocks{group_name="supergroup",} 13.0
  37. fsimage_group_blocks{group_name="nobody",} 3.0
  38. # HELP fsimage_group_links Number of sym links.
  39. # TYPE fsimage_group_links gauge
  40. fsimage_group_links{group_name="root",} 0.0
  41. fsimage_group_links{group_name="supergroup",} 0.0
  42. fsimage_group_links{group_name="nobody",} 0.0
  43. # HELP fsimage_path_dirs Number of directories.
  44. # TYPE fsimage_path_dirs gauge
  45. fsimage_path_dirs{path="/datalake/asset2",} 0.0
  46. fsimage_path_dirs{path="/datalake/asset3",} 2.0
  47. fsimage_path_dirs{path="/user/mm",} 0.0
  48. fsimage_path_dirs{path="/datalake/asset1",} 0.0
  49. # HELP fsimage_path_blocks Number of blocks.
  50. # TYPE fsimage_path_blocks gauge
  51. fsimage_path_blocks{path="/datalake/asset2",} 2.0
  52. fsimage_path_blocks{path="/datalake/asset3",} 3.0
  53. fsimage_path_blocks{path="/user/mm",} 0.0
  54. fsimage_path_blocks{path="/datalake/asset1",} 0.0
  55. # HELP fsimage_path_links Number of sym links.
  56. # TYPE fsimage_path_links gauge
  57. fsimage_path_links{path="/datalake/asset2",} 0.0
  58. fsimage_path_links{path="/datalake/asset3",} 0.0
  59. fsimage_path_links{path="/user/mm",} 0.0
  60. fsimage_path_links{path="/datalake/asset1",} 0.0
  61. # HELP fsimage_path_set_dirs Number of directories.
  62. # TYPE fsimage_path_set_dirs gauge
  63. fsimage_path_set_dirs{path_set="userMmAndFooAndAsset1",} 2.0
  64. fsimage_path_set_dirs{path_set="datalakeAsset1and2",} 0.0
  65. # HELP fsimage_path_set_blocks Number of blocks.
  66. # TYPE fsimage_path_set_blocks gauge
  67. fsimage_path_set_blocks{path_set="userMmAndFooAndAsset1",} 3.0
  68. fsimage_path_set_blocks{path_set="datalakeAsset1and2",} 2.0
  69. # HELP fsimage_path_set_links Number of sym links.
  70. # TYPE fsimage_path_set_links gauge
  71. fsimage_path_set_links{path_set="userMmAndFooAndAsset1",} 0.0
  72. fsimage_path_set_links{path_set="datalakeAsset1and2",} 0.0
  73. # HELP fsimage_fsize Overall file size distribution
  74. # TYPE fsimage_fsize histogram
  75. fsimage_fsize_bucket{le="0.0",} 0.0
  76. fsimage_fsize_bucket{le="1048576.0",} 4.0
  77. fsimage_fsize_bucket{le="3.3554432E7",} 13.0
  78. fsimage_fsize_bucket{le="6.7108864E7",} 14.0
  79. fsimage_fsize_bucket{le="1.34217728E8",} 15.0
  80. fsimage_fsize_bucket{le="1.073741824E9",} 16.0
  81. fsimage_fsize_bucket{le="1.073741824E10",} 16.0
  82. fsimage_fsize_bucket{le="+Inf",} 16.0
  83. fsimage_fsize_count 16.0
  84. fsimage_fsize_sum 3.56409344E8
  85. # HELP fsimage_csize Overall consumed file size distribution
  86. # TYPE fsimage_csize histogram
  87. fsimage_csize_bucket{le="0.0",} 0.0
  88. fsimage_csize_bucket{le="1048576.0",} 4.0
  89. fsimage_csize_bucket{le="3.3554432E7",} 13.0
  90. fsimage_csize_bucket{le="6.7108864E7",} 14.0
  91. fsimage_csize_bucket{le="1.34217728E8",} 14.0
  92. fsimage_csize_bucket{le="1.073741824E9",} 16.0
  93. fsimage_csize_bucket{le="1.073741824E10",} 16.0
  94. fsimage_csize_bucket{le="+Inf",} 16.0
  95. fsimage_csize_count 16.0
  96. fsimage_csize_sum 5.40762112E8
  97. # HELP fsimage_replication Overall file replication
  98. # TYPE fsimage_replication summary
  99. fsimage_replication_count 16.0
  100. fsimage_replication_sum 22.0
  101. # HELP fsimage_group_fsize Per group file size and file count
  102. # TYPE fsimage_group_fsize summary
  103. fsimage_group_fsize_count{group_name="root",} 1.0
  104. fsimage_group_fsize_sum{group_name="root",} 1024.0
  105. fsimage_group_fsize_count{group_name="supergroup",} 13.0
  106. fsimage_group_fsize_sum{group_name="supergroup",} 1.6766464E8
  107. fsimage_group_fsize_count{group_name="nobody",} 2.0
  108. fsimage_group_fsize_sum{group_name="nobody",} 1.8874368E8
  109. # HELP fsimage_group_csize Per group consumed file size and file count
  110. # TYPE fsimage_group_csize summary
  111. fsimage_group_csize_count{group_name="root",} 1.0
  112. fsimage_group_csize_sum{group_name="root",} 1024.0
  113. fsimage_group_csize_count{group_name="supergroup",} 13.0
  114. fsimage_group_csize_sum{group_name="supergroup",} 3.52017408E8
  115. fsimage_group_csize_count{group_name="nobody",} 2.0
  116. fsimage_group_csize_sum{group_name="nobody",} 1.8874368E8
  117. # HELP fsimage_user_fsize Per user file size distribution
  118. # TYPE fsimage_user_fsize histogram
  119. fsimage_user_fsize_bucket{user_name="root",le="0.0",} 0.0
  120. fsimage_user_fsize_bucket{user_name="root",le="1048576.0",} 1.0
  121. fsimage_user_fsize_bucket{user_name="root",le="3.3554432E7",} 1.0
  122. fsimage_user_fsize_bucket{user_name="root",le="6.7108864E7",} 1.0
  123. fsimage_user_fsize_bucket{user_name="root",le="1.34217728E8",} 1.0
  124. fsimage_user_fsize_bucket{user_name="root",le="1.073741824E9",} 1.0
  125. fsimage_user_fsize_bucket{user_name="root",le="1.073741824E10",} 1.0
  126. fsimage_user_fsize_bucket{user_name="root",le="+Inf",} 1.0
  127. fsimage_user_fsize_count{user_name="root",} 1.0
  128. fsimage_user_fsize_sum{user_name="root",} 1024.0
  129. fsimage_user_fsize_bucket{user_name="foo",le="0.0",} 0.0
  130. fsimage_user_fsize_bucket{user_name="foo",le="1048576.0",} 0.0
  131. fsimage_user_fsize_bucket{user_name="foo",le="3.3554432E7",} 0.0
  132. fsimage_user_fsize_bucket{user_name="foo",le="6.7108864E7",} 0.0
  133. fsimage_user_fsize_bucket{user_name="foo",le="1.34217728E8",} 0.0
  134. fsimage_user_fsize_bucket{user_name="foo",le="1.073741824E9",} 1.0
  135. fsimage_user_fsize_bucket{user_name="foo",le="1.073741824E10",} 1.0
  136. fsimage_user_fsize_bucket{user_name="foo",le="+Inf",} 1.0
  137. fsimage_user_fsize_count{user_name="foo",} 1.0
  138. fsimage_user_fsize_sum{user_name="foo",} 1.6777216E8
  139. fsimage_user_fsize_bucket{user_name="mm",le="0.0",} 0.0
  140. fsimage_user_fsize_bucket{user_name="mm",le="1048576.0",} 3.0
  141. fsimage_user_fsize_bucket{user_name="mm",le="3.3554432E7",} 12.0
  142. fsimage_user_fsize_bucket{user_name="mm",le="6.7108864E7",} 13.0
  143. fsimage_user_fsize_bucket{user_name="mm",le="1.34217728E8",} 14.0
  144. fsimage_user_fsize_bucket{user_name="mm",le="1.073741824E9",} 14.0
  145. fsimage_user_fsize_bucket{user_name="mm",le="1.073741824E10",} 14.0
  146. fsimage_user_fsize_bucket{user_name="mm",le="+Inf",} 14.0
  147. fsimage_user_fsize_count{user_name="mm",} 14.0
  148. fsimage_user_fsize_sum{user_name="mm",} 1.8863616E8
  149. # HELP fsimage_user_csize Per user consumed file size and file count
  150. # TYPE fsimage_user_csize summary
  151. fsimage_user_csize_count{user_name="root",} 1.0
  152. fsimage_user_csize_sum{user_name="root",} 1024.0
  153. fsimage_user_csize_count{user_name="foo",} 1.0
  154. fsimage_user_csize_sum{user_name="foo",} 1.6777216E8
  155. fsimage_user_csize_count{user_name="mm",} 14.0
  156. fsimage_user_csize_sum{user_name="mm",} 3.72988928E8
  157. # HELP fsimage_user_replication Per user file replication
  158. # TYPE fsimage_user_replication summary
  159. fsimage_user_replication_count{user_name="root",} 1.0
  160. fsimage_user_replication_sum{user_name="root",} 1.0
  161. fsimage_user_replication_count{user_name="foo",} 1.0
  162. fsimage_user_replication_sum{user_name="foo",} 1.0
  163. fsimage_user_replication_count{user_name="mm",} 14.0
  164. fsimage_user_replication_sum{user_name="mm",} 20.0
  165. # HELP fsimage_path_fsize Path specific file size and file count
  166. # TYPE fsimage_path_fsize summary
  167. fsimage_path_fsize_count{path="/datalake/asset2",} 2.0
  168. fsimage_path_fsize_sum{path="/datalake/asset2",} 2098176.0
  169. fsimage_path_fsize_count{path="/datalake/asset3",} 3.0
  170. fsimage_path_fsize_sum{path="/datalake/asset3",} 6291456.0
  171. fsimage_path_fsize_count{path="/user/mm",} 0.0
  172. fsimage_path_fsize_sum{path="/user/mm",} 0.0
  173. fsimage_path_fsize_count{path="/datalake/asset1",} 0.0
  174. fsimage_path_fsize_sum{path="/datalake/asset1",} 0.0
  175. # HELP fsimage_path_csize Path specific consumed file size and file count
  176. # TYPE fsimage_path_csize summary
  177. fsimage_path_csize_count{path="/datalake/asset2",} 2.0
  178. fsimage_path_csize_sum{path="/datalake/asset2",} 2098176.0
  179. fsimage_path_csize_count{path="/datalake/asset3",} 3.0
  180. fsimage_path_csize_sum{path="/datalake/asset3",} 6291456.0
  181. fsimage_path_csize_count{path="/user/mm",} 0.0
  182. fsimage_path_csize_sum{path="/user/mm",} 0.0
  183. fsimage_path_csize_count{path="/datalake/asset1",} 0.0
  184. fsimage_path_csize_sum{path="/datalake/asset1",} 0.0
  185. # HELP fsimage_path_set_fsize Path set specific file size and file count
  186. # TYPE fsimage_path_set_fsize summary
  187. fsimage_path_set_fsize_count{path_set="userMmAndFooAndAsset1",} 3.0
  188. fsimage_path_set_fsize_sum{path_set="userMmAndFooAndAsset1",} 6291456.0
  189. fsimage_path_set_fsize_count{path_set="datalakeAsset1and2",} 2.0
  190. fsimage_path_set_fsize_sum{path_set="datalakeAsset1and2",} 2098176.0
  191. # HELP fsimage_path_set_csize Path set specific consumed file size and file count
  192. # TYPE fsimage_path_set_csize summary
  193. fsimage_path_set_csize_count{path_set="userMmAndFooAndAsset1",} 3.0
  194. fsimage_path_set_csize_sum{path_set="userMmAndFooAndAsset1",} 6291456.0
  195. fsimage_path_set_csize_count{path_set="datalakeAsset1and2",} 2.0
  196. fsimage_path_set_csize_sum{path_set="datalakeAsset1and2",} 2098176.0
  197. # HELP fsimage_load_duration_seconds Time for loading/parsing FSImage
  198. # TYPE fsimage_load_duration_seconds summary
  199. fsimage_load_duration_seconds_count 1.0
  200. fsimage_load_duration_seconds_sum 0.105681675
  201. # HELP fsimage_compute_stats_duration_seconds Time for computing stats for a loaded FSImage
  202. # TYPE fsimage_compute_stats_duration_seconds summary
  203. fsimage_compute_stats_duration_seconds_count 1.0
  204. fsimage_compute_stats_duration_seconds_sum 0.049727828
  205. # HELP fsimage_load_file_size_bytes Size of raw FSImage
  206. # TYPE fsimage_load_file_size_bytes gauge
  207. fsimage_load_file_size_bytes 2420.0
  208. # HELP fsimage_scrape_duration_seconds Scrape duration
  209. # TYPE fsimage_scrape_duration_seconds gauge
  210. fsimage_scrape_duration_seconds 6.83354E-4
  211. # HELP fsimage_scrape_requests_total Exporter requests made
  212. # TYPE fsimage_scrape_requests_total counter
  213. fsimage_scrape_requests_total 2.0
  214. # HELP fsimage_scrape_errors_total Counts failed scrapes.
  215. # TYPE fsimage_scrape_errors_total counter
  216. fsimage_scrape_errors_total 0.0
  217. # HELP jvm_info VM version info
  218. # TYPE jvm_info gauge
  219. jvm_info{runtime="OpenJDK Runtime Environment",vendor="Amazon.com Inc.",version="1.8.0_442-b06",} 1.0
  220. # HELP jvm_threads_current Current thread count of a JVM
  221. # TYPE jvm_threads_current gauge
  222. jvm_threads_current 14.0
  223. # HELP jvm_threads_daemon Daemon thread count of a JVM
  224. # TYPE jvm_threads_daemon gauge
  225. jvm_threads_daemon 12.0
  226. # HELP jvm_threads_peak Peak thread count of a JVM
  227. # TYPE jvm_threads_peak gauge
  228. jvm_threads_peak 14.0
  229. # HELP jvm_threads_started_total Started thread count of a JVM
  230. # TYPE jvm_threads_started_total counter
  231. jvm_threads_started_total 16.0
  232. # HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
  233. # TYPE jvm_threads_deadlocked gauge
  234. jvm_threads_deadlocked 0.0
  235. # HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
  236. # TYPE jvm_threads_deadlocked_monitor gauge
  237. jvm_threads_deadlocked_monitor 0.0
  238. # HELP jvm_threads_state Current count of threads by state
  239. # TYPE jvm_threads_state gauge
  240. jvm_threads_state{state="NEW",} 0.0
  241. jvm_threads_state{state="TERMINATED",} 0.0
  242. jvm_threads_state{state="RUNNABLE",} 4.0
  243. jvm_threads_state{state="BLOCKED",} 0.0
  244. jvm_threads_state{state="WAITING",} 6.0
  245. jvm_threads_state{state="TIMED_WAITING",} 4.0
  246. jvm_threads_state{state="UNKNOWN",} 0.0
  247. # HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
  248. # TYPE jvm_buffer_pool_used_bytes gauge
  249. jvm_buffer_pool_used_bytes{pool="direct",} 8192.0
  250. jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
  251. # HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
  252. # TYPE jvm_buffer_pool_capacity_bytes gauge
  253. jvm_buffer_pool_capacity_bytes{pool="direct",} 8192.0
  254. jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
  255. # HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
  256. # TYPE jvm_buffer_pool_used_buffers gauge
  257. jvm_buffer_pool_used_buffers{pool="direct",} 1.0
  258. jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
  259. # HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
  260. # TYPE jvm_classes_currently_loaded gauge
  261. jvm_classes_currently_loaded 2377.0
  262. # HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
  263. # TYPE jvm_classes_loaded_total counter
  264. jvm_classes_loaded_total 2377.0
  265. # HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
  266. # TYPE jvm_classes_unloaded_total counter
  267. jvm_classes_unloaded_total 0.0
  268. # HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
  269. # TYPE jvm_memory_pool_allocated_bytes_total counter
  270. jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 2.319572E7
  271. jvm_memory_pool_allocated_bytes_total{pool="Code Cache",} 2825600.0
  272. jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 2.62144E7
  273. jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 4194304.0
  274. jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 1606296.0
  275. jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 1.40232E7
  276. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  277. # TYPE process_cpu_seconds_total counter
  278. process_cpu_seconds_total 0.800972
  279. # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
  280. # TYPE process_start_time_seconds gauge
  281. process_start_time_seconds 1.739815507449E9
  282. # HELP process_open_fds Number of open file descriptors.
  283. # TYPE process_open_fds gauge
  284. process_open_fds 17.0
  285. # HELP process_max_fds Maximum number of open file descriptors.
  286. # TYPE process_max_fds gauge
  287. process_max_fds 10240.0
  288. # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
  289. # TYPE jvm_gc_collection_seconds summary
  290. jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 1.0
  291. jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.005
  292. jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
  293. jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
  294. # HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
  295. # TYPE jvm_memory_objects_pending_finalization gauge
  296. jvm_memory_objects_pending_finalization 0.0
  297. # HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
  298. # TYPE jvm_memory_bytes_used gauge
  299. jvm_memory_bytes_used{area="heap",} 3.1584344E7
  300. jvm_memory_bytes_used{area="nonheap",} 1.9290016E7
  301. # HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
  302. # TYPE jvm_memory_bytes_committed gauge
  303. jvm_memory_bytes_committed{area="heap",} 5.36870912E8
  304. jvm_memory_bytes_committed{area="nonheap",} 2.0119552E7
  305. # HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
  306. # TYPE jvm_memory_bytes_max gauge
  307. jvm_memory_bytes_max{area="heap",} 1.073741824E9
  308. jvm_memory_bytes_max{area="nonheap",} -1.0
  309. # HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
  310. # TYPE jvm_memory_bytes_init gauge
  311. jvm_memory_bytes_init{area="heap",} 5.36870912E8
  312. jvm_memory_bytes_init{area="nonheap",} 2555904.0
  313. # HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
  314. # TYPE jvm_memory_pool_bytes_used gauge
  315. jvm_memory_pool_bytes_used{pool="Code Cache",} 3029888.0
  316. jvm_memory_pool_bytes_used{pool="Metaspace",} 1.4585064E7
  317. jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 1675816.0
  318. jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 4194304.0
  319. jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 4194304.0
  320. jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 2.3195736E7
  321. # HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
  322. # TYPE jvm_memory_pool_bytes_committed gauge
  323. jvm_memory_pool_bytes_committed{pool="Code Cache",} 3080192.0
  324. jvm_memory_pool_bytes_committed{pool="Metaspace",} 1.5204352E7
  325. jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 1835008.0
  326. jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 4.194304E7
  327. jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 4194304.0
  328. jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 4.90733568E8
  329. # HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
  330. # TYPE jvm_memory_pool_bytes_max gauge
  331. jvm_memory_pool_bytes_max{pool="Code Cache",} 2.5165824E8
  332. jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
  333. jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
  334. jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
  335. jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
  336. jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.073741824E9
  337. # HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
  338. # TYPE jvm_memory_pool_bytes_init gauge
  339. jvm_memory_pool_bytes_init{pool="Code Cache",} 2555904.0
  340. jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
  341. jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
  342. jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 2.8311552E7
  343. jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
  344. jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 5.0855936E8
  345. # HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
  346. # TYPE jvm_memory_pool_collection_used_bytes gauge
  347. jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
  348. jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 4194304.0
  349. jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 0.0
  350. # HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
  351. # TYPE jvm_memory_pool_collection_committed_bytes gauge
  352. jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 4.194304E7
  353. jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 4194304.0
  354. jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 0.0
  355. # HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
  356. # TYPE jvm_memory_pool_collection_max_bytes gauge
  357. jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
  358. jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
  359. jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.073741824E9
  360. # HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
  361. # TYPE jvm_memory_pool_collection_init_bytes gauge
  362. jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 2.8311552E7
  363. jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
  364. jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 5.0855936E8
  365. # HELP fsimage_compute_stats_duration_seconds_created Time for computing stats for a loaded FSImage
  366. # TYPE fsimage_compute_stats_duration_seconds_created gauge
  367. fsimage_compute_stats_duration_seconds_created 1.739815507662E9
  368. # HELP fsimage_csize_created Overall consumed file size distribution
  369. # TYPE fsimage_csize_created gauge
  370. fsimage_csize_created 1.739815507807E9
  371. # HELP fsimage_fsize_created Overall file size distribution
  372. # TYPE fsimage_fsize_created gauge
  373. fsimage_fsize_created 1.739815507807E9
  374. # HELP fsimage_group_csize_created Per group consumed file size and file count
  375. # TYPE fsimage_group_csize_created gauge
  376. fsimage_group_csize_created{group_name="root",} 1.739815507844E9
  377. fsimage_group_csize_created{group_name="supergroup",} 1.739815507813E9
  378. fsimage_group_csize_created{group_name="nobody",} 1.739815507844E9
  379. # HELP fsimage_group_fsize_created Per group file size and file count
  380. # TYPE fsimage_group_fsize_created gauge
  381. fsimage_group_fsize_created{group_name="root",} 1.739815507844E9
  382. fsimage_group_fsize_created{group_name="supergroup",} 1.739815507813E9
  383. fsimage_group_fsize_created{group_name="nobody",} 1.739815507844E9
  384. # HELP fsimage_load_duration_seconds_created Time for loading/parsing FSImage
  385. # TYPE fsimage_load_duration_seconds_created gauge
  386. fsimage_load_duration_seconds_created 1.739815507662E9
  387. # HELP fsimage_path_csize_created Path specific consumed file size and file count
  388. # TYPE fsimage_path_csize_created gauge
  389. fsimage_path_csize_created{path="/datalake/asset2",} 1.739815507849E9
  390. fsimage_path_csize_created{path="/datalake/asset3",} 1.739815507848E9
  391. fsimage_path_csize_created{path="/user/mm",} 1.739815507848E9
  392. fsimage_path_csize_created{path="/datalake/asset1",} 1.739815507849E9
  393. # HELP fsimage_path_fsize_created Path specific file size and file count
  394. # TYPE fsimage_path_fsize_created gauge
  395. fsimage_path_fsize_created{path="/datalake/asset2",} 1.739815507849E9
  396. fsimage_path_fsize_created{path="/datalake/asset3",} 1.739815507848E9
  397. fsimage_path_fsize_created{path="/user/mm",} 1.739815507848E9
  398. fsimage_path_fsize_created{path="/datalake/asset1",} 1.739815507849E9
  399. # HELP fsimage_path_set_csize_created Path set specific consumed file size and file count
  400. # TYPE fsimage_path_set_csize_created gauge
  401. fsimage_path_set_csize_created{path_set="userMmAndFooAndAsset1",} 1.739815507851E9
  402. fsimage_path_set_csize_created{path_set="datalakeAsset1and2",} 1.739815507851E9
  403. # HELP fsimage_path_set_fsize_created Path set specific file size and file count
  404. # TYPE fsimage_path_set_fsize_created gauge
  405. fsimage_path_set_fsize_created{path_set="userMmAndFooAndAsset1",} 1.739815507851E9
  406. fsimage_path_set_fsize_created{path_set="datalakeAsset1and2",} 1.739815507851E9
  407. # HELP fsimage_replication_created Overall file replication
  408. # TYPE fsimage_replication_created gauge
  409. fsimage_replication_created 1.739815507807E9
  410. # HELP fsimage_scrape_errors_created Counts failed scrapes.
  411. # TYPE fsimage_scrape_errors_created gauge
  412. fsimage_scrape_errors_created 1.739815507659E9
  413. # HELP fsimage_scrape_requests_created Exporter requests made
  414. # TYPE fsimage_scrape_requests_created gauge
  415. fsimage_scrape_requests_created 1.739815507659E9
  416. # HELP fsimage_user_csize_created Per user consumed file size and file count
  417. # TYPE fsimage_user_csize_created gauge
  418. fsimage_user_csize_created{user_name="root",} 1.739815507844E9
  419. fsimage_user_csize_created{user_name="foo",} 1.739815507844E9
  420. fsimage_user_csize_created{user_name="mm",} 1.739815507813E9
  421. # HELP fsimage_user_fsize_created Per user file size distribution
  422. # TYPE fsimage_user_fsize_created gauge
  423. fsimage_user_fsize_created{user_name="root",} 1.739815507844E9
  424. fsimage_user_fsize_created{user_name="foo",} 1.739815507844E9
  425. fsimage_user_fsize_created{user_name="mm",} 1.739815507813E9
  426. # HELP fsimage_user_replication_created Per user file replication
  427. # TYPE fsimage_user_replication_created gauge
  428. fsimage_user_replication_created{user_name="root",} 1.739815507844E9
  429. fsimage_user_replication_created{user_name="foo",} 1.739815507844E9
  430. fsimage_user_replication_created{user_name="mm",} 1.739815507813E9
  431. # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
  432. # TYPE jvm_memory_pool_allocated_bytes_created gauge
  433. jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.739815507851E9
  434. jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.739815507851E9
  435. jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.739815507851E9
  436. jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.739815507851E9
  437. jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.739815507851E9
  438. jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.739815507851E9

License

This Hadoop HDFS FSImage Exporter is released under the Apache 2.0 license.

  1. Copyright 2018+ Marcel May
  3. Licensed under the Apache License, Version 2.0 (the "License");
  4. you may not use this file except in compliance with the License.
  5. You may obtain a copy of the License at
  7. https://www.apache.org/licenses/LICENSE-2.0
  9. Unless required by applicable law or agreed to in writing, software
  10. distributed under the License is distributed on an "AS IS" BASIS,
  11. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12. See the License for the specific language governing permissions and
  13. limitations under the License.