Kubernetes/Monitoring - Revision history

Pio2pio: /* References */

2020-07-14T17:24:49Z

References

← Older revision		Revision as of 19:24, 14 July 2020
Line 243:		Line 243:
	*[https://kubernetes.io/docs/concepts/cluster-administration/logging/ Logging] K8s docs		*[https://kubernetes.io/docs/concepts/cluster-administration/logging/ Logging] K8s docs
	*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs Logs] K8s docs		*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs Logs] K8s docs
			*[https://github.com/VictoriaMetrics/VictoriaMetrics VictoriaMetrics] Github

Pio2pio: /* Metric-server */

2020-07-14T17:11:15Z

Metric-server

← Older revision		Revision as of 19:11, 14 July 2020
Line 18:		Line 18:


	EKS Installation [1]		[https://aws.amazon.com/premiumsupport/knowledge-center/eks-metrics-server/ EKS Installation] [1]
	<source lang=bash>		<source lang=bash>
	wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml -O metrics-server-0.3.6.yaml		wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml -O metrics-server-0.3.6.yaml
Line 56:		Line 56:
	kubectl top pods --containers # CPU,memory of the containers inside the pod		kubectl top pods --containers # CPU,memory of the containers inside the pod
	</source>		</source>


	[1] EKS errors if installed oob <code>unable to fully scrape metrics</code>:		[1] EKS errors if installed oob <code>unable to fully scrape metrics</code>:
Line 61:		Line 62:
	metrics-server-aaaaaaaaaa-h64c5 metrics-server E0714 15:59:42.204640 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-35-70-169.eu-west-1.compute.internal: unable to fetch metrics from Kubelet ip-10-35-70-169.eu-west-1.compute.internal (ip-10-35-70-169.dev.acme.com): Get https://ip-10-35-70-169.dev.acme.com:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-35-70-169.dev.acme.com on 172.20.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:		metrics-server-aaaaaaaaaa-h64c5 metrics-server E0714 15:59:42.204640 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-35-70-169.eu-west-1.compute.internal: unable to fetch metrics from Kubelet ip-10-35-70-169.eu-west-1.compute.internal (ip-10-35-70-169.dev.acme.com): Get https://ip-10-35-70-169.dev.acme.com:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-35-70-169.dev.acme.com on 172.20.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:
	</source>		</source>


	[2] Working metric-server scrapes metrics by default every 1 minute.		[2] Working metric-server scrapes metrics by default every 1 minute.

Pio2pio: /* Metric-server */

2020-07-14T16:56:56Z

Metric-server

← Older revision		Revision as of 18:56, 14 July 2020
Line 51:		Line 51:
	kubectl top pods # CPU,memory utilization of the pods in your cluster		kubectl top pods # CPU,memory utilization of the pods in your cluster
	kubectl top pods -A # CPU,memory of pods in all namespaces		kubectl top pods -A # CPU,memory of pods in all namespaces
			kubectl top pods -A --sort-by=memory
	kubectl top pod -l run=<label> # CPU and memory of pods with a label selector:		kubectl top pod -l run=<label> # CPU and memory of pods with a label selector:
	kubectl top pod <pod-name> # CPU,memory of a specific pod		kubectl top pod <pod-name> # CPU,memory of a specific pod

Pio2pio: /* Metric-server */

2020-07-14T16:48:43Z

Metric-server

@@ Line 1: / Line 1: @@
 = Monitor cluster resources =
 == Metric-server ==
-In order to get cluster resources you need a metric collector plugin. Popular one is <tt>heapster</tt> that exposes <tt>metric-server</tt> service. The below commands relay on its API to get data:
+In order to get cluster resources you need a metric collector plugin. Popular one was <tt>heapster</tt> now deprecated replaced by <tt>metric-server</tt>.
 Install metrics-server
 <source lang=bash>
 git clone https://github.com/kubernetes-incubator/metrics-server.git
 kubectl apply -f ~/metrics-server/deploy/1.8+/
 </source>
 Get metrics
 <source lang=bash>
 # verify metrics server API
 kubectl get --raw /apis/metrics.k8s.io/
 kubectl top node               # CPU,memory utilization of the nodes in your cluster
@@ Line 19: / Line 53: @@
 kubectl top pod -l run=<label> # CPU and memory of pods with a label selector:
 kubectl top pod <pod-name>     # CPU,memory of a specific pod
-kubectl top pods group-context --containers # CPU,memory of the containers inside the pod
+kubectl top pods --containers  # CPU,memory of the containers inside the pod
 </source>

Pio2pio: /* Kubernetes worker nodes docker log configuration */

2020-01-18T20:20:41Z

Kubernetes worker nodes docker log configuration

← Older revision		Revision as of 22:20, 18 January 2020
Line 109:		Line 109:


	== Kubernetes worker nodes [https://docs.docker.com/config/containers/logging/configure/ docker log configuration] ==		== Kubernetes worker nodes [https://docs.docker.com/config/containers/logging/configure/ docker log configuration] - size ==
	The Docker runtime log configuration is setup on each node in <code>/etc/docker/daemon.json</code>		The Docker runtime log configuration is setup on each node in <code>/etc/docker/daemon.json</code>
	<source lang=json>		<source lang=json>

Pio2pio: /* Kubernetes worker nodes docker log configuration */

2020-01-18T20:20:03Z

Kubernetes worker nodes docker log configuration

@@ Line 122: / Line 122: @@
    "max-concurrent-downloads": 10
+}
 </source>

Pio2pio: /* Logs */

2020-01-18T19:52:22Z

Logs

← Older revision		Revision as of 21:52, 18 January 2020
Line 108:		Line 108:
	</source>		</source>


			== Kubernetes worker nodes [https://docs.docker.com/config/containers/logging/configure/ docker log configuration] ==
			The Docker runtime log configuration is setup on each node in <code>/etc/docker/daemon.json</code>
			<source lang=json>
			{
			"bridge": "none",
			"log-driver": "json-file",
			"log-opts": {
			"max-size": "10m",
			"max-file": "10"
			},
			"live-restore": true,
			"max-concurrent-downloads": 10
			}
			</source>

	= Termination message =		= Termination message =

Pio2pio: /* Liveness and Readiness probes */

2019-10-13T21:52:39Z

Liveness and Readiness probes

← Older revision		Revision as of 23:52, 13 October 2019
Line 37:		Line 37:
	</source>		</source>

	= Liveness and Readiness probes =		= Liveness and Readiness [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-command probes] =
	Check this [https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes Visual explanation]		Check this [https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes Visual explanation]
	* <code>readinessProbe</code> - checks if a pod is ready to receive a client requests, when passed, then the pod is added to <code>endpoint</code>. When the probe fails - the pod is not restarted, instead removed from <code>endpoint</code>.		* <code>readinessProbe</code> - checks if a pod is ready to receive a client requests, when passed, then the pod is added to <code>endpoint</code>. When the probe fails - the pod is not restarted, instead removed from <code>endpoint</code>.

Pio2pio: /* Liveness and Readiness probes */

2019-10-13T21:50:11Z

Liveness and Readiness probes

← Older revision		Revision as of 23:50, 13 October 2019
Line 39:		Line 39:
	= Liveness and Readiness probes =		= Liveness and Readiness probes =
	Check this [https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes Visual explanation]		Check this [https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes Visual explanation]
			* <code>readinessProbe</code> - checks if a pod is ready to receive a client requests, when passed, then the pod is added to <code>endpoint</code>. When the probe fails - the pod is not restarted, instead removed from <code>endpoint</code>.
			* <code>livenessProbe</code> - when the probe fails, pod gets restarted
	Get service endpoints. Only healthy and ready pods will be added to the <code>endpoint</code>		Get service endpoints. Only healthy and ready pods will be added to the <code>endpoint</code>
	<source lang=bash>		<source lang=bash>
Line 57:		Line 58:
	name: main		name: main
	livenessProbe:		livenessProbe:
	httpGet:		httpGet: # exec: or tcpSocket:
	path: /healthz # not all containers have this endpoint		path: /healthz # not all containers have this endpoint
	port: 8081		port: 8081
Line 64:		Line 65:
	path: /		path: /
	port: 80		port: 80
			initialDelaySeconds: 5 # default, tell kubelet to wait 5 second after container starts, before performing the first probe
			periodSeconds: 5 # default, tell kueblet to run probe ever 5s
	</syntaxhighlightjs>		</syntaxhighlightjs>

Pio2pio: /* Liveness and Readiness probes */

2019-10-13T21:26:14Z

Liveness and Readiness probes

← Older revision		Revision as of 23:26, 13 October 2019
Line 47:		Line 47:

	Liveness and readiness probes in both Pod and Deployment manifests are at <code>.spec.containers.image</code> level		Liveness and readiness probes in both Pod and Deployment manifests are at <code>.spec.containers.image</code> level
	<~~source~~ lang=~~yml~~>		<syntaxhighlightjs lang=yaml>
	apiVersion: v1		apiVersion: v1
	kind: Pod		kind: Pod
Line 64:		Line 64:
	path: /		path: /
	port: 80		port: 80
	</~~source~~>		</syntaxhighlightjs>

	= Logs =		= Logs =