Prometheus operator monitoring pods in different namespaces

Tim Blackwell

2018-08-14 09:59:36 UTC

While ServiceMonitors must live in the same namespace as the Prometheus
resource, discovered targets may come from any namespace. This allows
cross-namespace monitoring use cases, for example, for meta-monitoring. Use
the namespaceSelector of the ServiceMonitorSpec to restrict the
namespaces from which Endpoints objects may be discovered.

From the generic service monitor
guid https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md#namespace-limitsthings-to-keep-in-mind

When running the examples in default namespace, all is good.
I tried to create the prometheus operator and prometheus stateful set in a
monitoring namespace but then it was unable to monitor the example app.
When running the prometheus / operator in monitoring namespace, I updated
the service account yaml accordingly.
Is it possible for prometheus / operator to monitor and collect metrics
from different namespaces? If so what should I change to get this behaviour?
# Default Namespace
## Operator
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-service-account.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-cluster-role-binding.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-deployment.yaml
## Prometheus
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-service-account.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-cluster-role.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-cluster-role-binding.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus.yaml
## Example app
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-deployment.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-service-monitor.yaml
kubectl create -f
https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-service.yaml
# Monitoring Namespace
kubectl create namespace monitoring
## Operator
kubectl --namespace monitoring create -f
example/rbac/prometheus-operator/prometheus-operator-service-account.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus-operator/prometheus-operator-cluster-role-binding.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus-operator/prometheus-operator-deployment.yaml
## Prometheus
kubectl --namespace monitoring create -f
example/rbac/prometheus/prometheus-service-account.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus/prometheus-cluster-role.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus/prometheus-cluster-role-binding.yaml
kubectl --namespace monitoring create -f
example/rbac/prometheus/prometheus.yaml
## Example app
kubectl create -f
example/user-guides/getting-started/example-app-deployment.yaml
kubectl create -f
example/user-guides/getting-started/example-app-service-monitor.yaml
kubectl create -f
example/user-guides/getting-started/example-app-service.yaml