DSBulk is already well instrumented, but it only exports its metrics to JMX and CSV files.
It would be good if it could export metrics to Prometheus as well.
I am not thinking about replacing Dropwizard with another metrics framework, I am only thinking about exposing the existing Dropwizard metrics to Prometheus. It turns out there is a Prometheus exporter for Dropwizard that will take care of all the heavy lifting.
Also we would need to think about how to expose the metrics. The usual pull model (scraping) is not always suitable for batch jobs like DSBulk, especially short-lived operations. Prometheus also offers a push-based model based on a PushGateway.
According to this doc, batch jobs should have a few important metrics pushed to a PushGateway (time elapsed, success/failure, etc.), but it is useful to also scrape them using pull-based monitoring. I propose that we offer both approaches and let the user decide which ones to enable.
┆Issue is synchronized with a Github issue by Unito ┆Repository Name: dsbulk ┆Issue Number: 405 ┆Milestone: 1.9.0
Major
DSBulk is already well instrumented, but it only exports its metrics to JMX and CSV files.
It would be good if it could export metrics to Prometheus as well.
I am not thinking about replacing Dropwizard with another metrics framework, I am only thinking about exposing the existing Dropwizard metrics to Prometheus. It turns out there is a Prometheus exporter for Dropwizard that will take care of all the heavy lifting.
Also we would need to think about how to expose the metrics. The usual pull model (scraping) is not always suitable for batch jobs like DSBulk, especially short-lived operations. Prometheus also offers a push-based model based on a PushGateway.
According to this doc, batch jobs should have a few important metrics pushed to a PushGateway (time elapsed, success/failure, etc.), but it is useful to also scrape them using pull-based monitoring. I propose that we offer both approaches and let the user decide which ones to enable.
┆Issue is synchronized with a Github issue by Unito
┆Repository Name: dsbulk
┆Issue Number: 405
┆Milestone: 1.9.0