Kubernetes/Istio/Observability - Revision history

Pio2pio: /* Load Testing - fortio */

2021-04-14T07:07:54Z

Load Testing - fortio

← Older revision		Revision as of 09:07, 14 April 2021
Line 217:		Line 217:

	= Load Testing - [https://github.com/fortio/fortio fortio] =		= Load Testing - [https://github.com/fortio/fortio fortio] =
			Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio runs at a specified query per second (qps) and records an histogram of execution time and calculates percentiles (e.g. p99 ie the response time such as 99% of the requests take less than that number (in seconds, SI unit)).
	<source lang=bash>		<source lang=bash>
	# Install		# Install

Pio2pio: /* Kiali */

2021-04-04T23:19:54Z

Kiali

← Older revision		Revision as of 01:19, 5 April 2021
Line 215:		Line 215:

	* Kiali http://localhost:20001		* Kiali http://localhost:20001

			= Load Testing - [https://github.com/fortio/fortio fortio] =
			<source lang=bash>
			# Install
			VERSION=$(curl --silent "https://api.github.com/repos/fortio/fortio/releases/latest" \| jq -r .tag_name); echo $VERSION
			curl -L https://github.com/fortio/fortio/releases/download/${VERSION}/fortio-linux_x64-${VERSION#v}.tgz \
			\| sudo tar -C / -xvzpf -

			# or the debian package
			wget https://github.com/fortio/fortio/releases/download/${VERSION}/fortio_${VERSION#v}-1_amd64.deb
			dpkg -i fortio_${VERSION#v}-1_amd64.deb

			# or the rpm
			rpm -i https://github.com/fortio/fortio/releases/download/${VERSION}/fortio-${VERSION#v}-1.x86_64.rpm

			# Docker run
			docker run -p 8080:8080 -p 8079:8079 fortio/fortio server & # For the server
			docker run fortio/fortio load http://www.google.com/ # For a test run
			</source>

			= References =

Pio2pio: /* Prometheus */

2021-04-02T14:44:12Z

Prometheus

@@ Line 45: / Line 45: @@
 ;[https://istio.io/latest/docs/ops/integrations/prometheus/#option-1-metrics-merging Metrics merging]
 Envoy sidecar can merge Istio’s metrics with the application metrics, this is enabled by default <code>--set meshConfig.enablePrometheusMerge=true</code>. When enabled, appropriate prometheus.io annotations will be added to all data plane pods to set up scraping. The merged metrics will be scraped from <code>/stats/prometheus:15020</code>.
 = Grafana Istio dashboards =

Pio2pio: /* Prometheus */

2021-04-02T14:19:09Z

Prometheus

← Older revision		Revision as of 16:19, 2 April 2021
Line 9:		Line 9:
	kubectl -n prometheus port-forward statefulset/prometheus-prom-kube-prometheus-stack-prometheus 9090		kubectl -n prometheus port-forward statefulset/prometheus-prom-kube-prometheus-stack-prometheus 9090
	kubectl -n prometheus port-forward svc/prom-grafana 3000:80		kubectl -n prometheus port-forward svc/prom-grafana 3000:80

			# Get rules. The Prometheus rule configmap is actually stored as a secret and is updated through the operator configurations
			kubectl get secret -n prometheus prometheus-prom-kube-prometheus-stack-prometheus -o jsonpath="{.data['prometheus\.yaml\.gz']}" \| base64 -d \| gunzip
	</source>		</source>

Line 38:		Line 41:
	enabled: false		enabled: false
	</syntaxhighlightjs>		</syntaxhighlightjs>


			;[https://istio.io/latest/docs/ops/integrations/prometheus/#option-1-metrics-merging Metrics merging]
			Envoy sidecar can merge Istio’s metrics with the application metrics, this is enabled by default <code>--set meshConfig.enablePrometheusMerge=true</code>. When enabled, appropriate prometheus.io annotations will be added to all data plane pods to set up scraping. The merged metrics will be scraped from <code>/stats/prometheus:15020</code>.

	= Grafana Istio dashboards =		= Grafana Istio dashboards =

Pio2pio: /* Kiali */

2021-04-02T13:46:35Z

Kiali

← Older revision		Revision as of 15:46, 2 April 2021
Line 179:		Line 179:
	</syntaxhighlightjs>		</syntaxhighlightjs>

	* Kiali http://localhost:20001		Create ServiceAccount and ClusterRoleBinding for Token Auth

	~~Get a token~~
	<source lang=bash>		<source lang=bash>
	kubectl create serviceaccount kiali-dashboard -n istio-system		kubectl create serviceaccount kiali-dashboard -n istio-system
Line 189:		Line 187:
	kubectl get secret -n istio-system -o jsonpath="{.data.token}" $(kubectl get secret -n istio-system \| grep kiali-dashboard \| awk '{print $1}' ) \| base64 --decode		kubectl get secret -n istio-system -o jsonpath="{.data.token}" $(kubectl get secret -n istio-system \| grep kiali-dashboard \| awk '{print $1}' ) \| base64 --decode
	</source>		</source>


			* Kiali http://localhost:20001

Pio2pio: /* Kiali */

2021-04-02T13:45:03Z

Kiali

@@ Line 175: / Line 175: @@
 EOF
 ) --dry-run=server
 </syntaxhighlightjs>

Pio2pio: /* Setup Prometheus to scrape metrics */

2021-04-02T13:39:54Z

Setup Prometheus to scrape metrics

@@ Line 130: / Line 130: @@
        action: replace
        targetLabel: pod_name
 EOF
 ) --dry-run=server
 </syntaxhighlightjs>

Pio2pio: /* Setup Prometheus to scrape metrics */

2021-04-02T13:22:40Z

Setup Prometheus to scrape metrics

← Older revision		Revision as of 15:22, 2 April 2021
Line 94:		Line 94:
	kubectl apply -f <(cat <<EOF		kubectl apply -f <(cat <<EOF
	apiVersion: monitoring.coreos.com/v1		apiVersion: monitoring.coreos.com/v1
	kind: ~~ServiceMonitor~~		kind: PodMonitor
	metadata:		metadata:
	name: ~~istio~~-~~component~~-monitor		name: envoy-stats-monitor
	namespace: prometheus		namespace: prometheus
	labels:		labels:
	monitoring: istio-~~components~~		monitoring: istio-proxies
	release: prom		release: prom
	spec:		spec:
	~~jobLabel: istio~~
	~~targetLabels: [app]~~
	selector:		selector:
	matchExpressions:		matchExpressions:
	- {key: istio, operator: ~~In, values: [pilot]~~}		- {key: istio-prometheus-ignore, operator: DoesNotExist}
	namespaceSelector:		namespaceSelector:
	any: true		any: true
	~~endpoints~~:		jobLabel: envoy-stats
	- ~~port~~: ~~http-monitoring~~		podMetricsEndpoints:
			- path: /stats/prometheus
	interval: 15s		interval: 15s
			relabelings:
			- action: keep
			sourceLabels: [__meta_kubernetes_pod_container_name]
			regex: "istio-proxy"
			- action: keep
			sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
			- sourceLabels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
			action: replace
			regex: ([^:]+)(?::\d+)?;(\d+)
			replacement: $1:$2
			targetLabel: __address__
			- action: labeldrop
			regex: "__meta_kubernetes_pod_label_(.+)"
			- sourceLabels: [__meta_kubernetes_namespace]
			action: replace
			targetLabel: namespace
			- sourceLabels: [__meta_kubernetes_pod_name]
			action: replace
			targetLabel: pod_name
	EOF		EOF
	) --dry-run=server		) --dry-run=server
	</syntaxhighlightjs>		</syntaxhighlightjs>

Pio2pio: /* Prometheus */

2021-04-02T13:16:16Z

Prometheus

@@ Line 37: / Line 37: @@
 kubeProxy: ## Component scraping kube proxy
    enabled: false
 </syntaxhighlightjs>

Pio2pio: /* Prometheus */

2021-04-02T10:19:41Z

Prometheus

← Older revision		Revision as of 12:19, 2 April 2021
Line 5:		Line 5:
	helm repo update		helm repo update
	helm install prom prometheus-community/kube-prometheus-stack --version 13.13.1 -n prometheus -f values.yaml		helm install prom prometheus-community/kube-prometheus-stack --version 13.13.1 -n prometheus -f values.yaml

			# Dashboards
			kubectl -n prometheus port-forward statefulset/prometheus-prom-kube-prometheus-stack-prometheus 9090
			kubectl -n prometheus port-forward svc/prom-grafana 3000:80
	</source>		</source>


			Access now:
			* Prometheus - http://localhost:9090/
			* Grafana - http://localhost:3000/, username: admin, password: prom-operator


	The values.yaml file is default just with following components disabled:		The values.yaml file is default just with following components disabled: