If you environment is bound to Prometheus it is best we focus on this platform as a baseline.
We will need multiple aspects:
System Counters
Main metrics needed:
1. Machines CPU
2. Read and Write IOPS to disks
3. Memory utilization
4. Disk utilization
5. Incoming and outgoing network traffic
MongoDB Counters
For this task it is best to use the MongoDB exporter
There are also 3 Grafana dashboards provided that we can use to monitor the performance
MongoDB Overview;
MongoDB ReplSet;
MongoDB WiredTiger;
MongoDB ReplSet;
MongoDB WiredTiger;
These will be able to provide us key indicators such as locks, replicaset status,
MongoDB Slow Queries
It will also be wise to enable the MongoDB slow query w/ an initial limit of 100ms to gain some overview of the cluster slow queries:
db.setProfilingLevel(1,100)
