Ever changing code - User contributions [en-gb]

Kubernetes/Progressive Delivery Flux and Flagger

2026-01-15T10:53:34Z

Pio2pio: /* Install FluxCD Cli flux */

= [https://github.com/fluxcd/flux2 Flux v2] =

[https://fluxcd.io/flux/ Flux v2 Documentation]

Flux v2 architecture
:[[File:ClipCapIt-210524-232835.PNG]]

Flux v2 - Webhooks and notifications
:[[File:ClipCapIt-210524-233028.PNG]]

= Install FluxCD Cli [https://fluxcd.io/flux/installation/#install-the-flux-cli <code>flux</code>] =

{{Note|<code>fluxctl</code> was a cli client for version Flux v1.}}

<source lang=bash>
# Install or upgrade using official install.sh (option-1)
export FLUX_VERSION=2.7.5; curl -s https://fluxcd.io/install.sh | sudo -E bash
curl -s https://fluxcd.io/install.sh | sudo bash # latest

# Version check
flux version
flux: v2.7.5
distribution: flux-v2.7.5
helm-controller: v1.4.5
kustomize-controller: v1.7.3
notification-controller: v1.7.5
source-controller: v1.7.4

# enable completions in ~/.bash_profile
. <(flux completion bash)

# Pre check
flux check --pre
► checking prerequisites
✗ Kubernetes version v1.27.16 does not match >=1.32.0-0
</source>

= Cluster bootstrap =
FluxCDv2 bootstrap process is installing the Flux onto a cluster and stores(commits) its own manifests to a Git repository.
* [https://fluxcd.io/docs/installation/#generic-git-server Generic Git Server], including GCP [https://cloud.google.com/source-repositories/docs Cloud Source Repositories]
* [https://fluxcd.io/docs/installation/#bootstrap-with-terraform Bootstrap with Terraform]

<source lang=bash>
FLUX_GIT_USERNAME=my-git-username
FLUX_GIT_EMAIL=my-git-email@example.com
flux bootstrap git \
--author-email=$FLUX_GIT_EMAIL \
--url=ssh://git@github.com/$FLUX_GIT_USERNAME/gitops-istio \
--branch=main \
--path=clusters/my-cluster
</source>
At bootstrap, Flux generates an SSH key and prints the public key. In order to sync your cluster state with git you need to copy the public key and create a deploy key with write access on your GitHub repository. On GitHub go to Settings > Deploy keys click on Add deploy key, check Allow write access, paste the Flux public key and click Add key.

;[https://fluxcd.io/docs/installation/#dev-install Dev installation] does not stores its own configuration state in Git repository
<source lang=bash>
# option 1
flux install # install and upgrade
flux install \
--namespace=flux-system \
--network-policy=false \
--components=source-controller

# option 2
kubectl apply -f https://github.com/fluxcd/flux2/releases/latest/download/install.yaml
kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=main | kubectl apply -f- # Upgrade

# Register Git repositories and reconcile them on your cluster:
flux create source git podinfo \
--url=https://github.com/stefanprodan/podinfo \
--tag-semver=">=4.0.0" \
--interval=1m

flux create kustomization podinfo-default \
--source=podinfo \
--path="./kustomize" \
--prune=true \
--validation=client \
--interval=10m \
--health-check="Deployment/podinfo.default" \
--health-check-timeout=2m

# Register Helm repositories and create Helm releases:
flux create source helm bitnami \
--interval=1h \
--url=https://charts.bitnami.com/bitnami

flux create helmrelease nginx \
--interval=1h \
--release-name=nginx-ingress-controller \
--target-namespace=kube-system \
--source=HelmRepository/bitnami \
--chart=nginx-ingress-controller \
--chart-version="5.x.x"
</source>

Uninstall
<source lang=bash>
flux uninstall --namespace=flux-system
</source>

= References =
* [https://github.com/fluxcd/terraform-provider-flux terraform-provider-flux]
*[https://github.com/pio2pio/gitops-istio gitops-istio] Tutorial
*[https://www.youtube.com/watch?v=nGLpUCPX8JE Flux v2 Everything that you wanted to know but were afraid to ask (Stefan Prodan)] December 2020

Bundle
*[https://blog.sldk.de/2021/02/introduction-to-gitops-on-kubernetes-with-flux-v2/ Introduction to GitOps on Kubernetes with Flux v2]
*[https://blog.sldk.de/2021/03/handling-secrets-in-flux-v2-repositories-with-sops/ Handling secrets in Flux v2 repositories with SOPS]

Kubernetes/Progressive Delivery Flux and Flagger

2026-01-15T10:53:09Z

Pio2pio: /* Install Flux v2 flux command line */

= [https://github.com/fluxcd/flux2 Flux v2] =

[https://fluxcd.io/flux/ Flux v2 Documentation]

Flux v2 architecture
:[[File:ClipCapIt-210524-232835.PNG]]

Flux v2 - Webhooks and notifications
:[[File:ClipCapIt-210524-233028.PNG]]

= Install FluxCD Cli [https://fluxcd.io/flux/installation/#install-the-flux-cli <code>flux</code>] =

{{Note|<code>fluxctl</code> was a cli client for version Flux v1.}}

<source lang=bash>
# Install or upgrade using official install.sh (option-1)
export FLUX_VERSION=2.7.5; curl -s https://fluxcd.io/install.sh | sudo -E bash
curl -s https://fluxcd.io/install.sh | sudo bash # latest

# Version check
flux version
flux: v2.7.5
distribution: flux-v2.7.5
helm-controller: v1.4.5
kustomize-controller: v1.7.3
notification-controller: v1.7.5
source-controller: v1.7.4

# enable completions in ~/.bash_profile
. <(flux completion bash)

# Pre check
flux check --pre
► checking prerequisites
✗ Kubernetes version v1.27.16 does not match >=1.32.0-0

# Docker images
docker pull fluxcd/fluxctl:1.24.3
docker pull ghcr.io/fluxcd/flux-cli:1.24.3 # does not work
</source>

= Cluster bootstrap =
FluxCDv2 bootstrap process is installing the Flux onto a cluster and stores(commits) its own manifests to a Git repository.
* [https://fluxcd.io/docs/installation/#generic-git-server Generic Git Server], including GCP [https://cloud.google.com/source-repositories/docs Cloud Source Repositories]
* [https://fluxcd.io/docs/installation/#bootstrap-with-terraform Bootstrap with Terraform]

<source lang=bash>
FLUX_GIT_USERNAME=my-git-username
FLUX_GIT_EMAIL=my-git-email@example.com
flux bootstrap git \
--author-email=$FLUX_GIT_EMAIL \
--url=ssh://git@github.com/$FLUX_GIT_USERNAME/gitops-istio \
--branch=main \
--path=clusters/my-cluster
</source>
At bootstrap, Flux generates an SSH key and prints the public key. In order to sync your cluster state with git you need to copy the public key and create a deploy key with write access on your GitHub repository. On GitHub go to Settings > Deploy keys click on Add deploy key, check Allow write access, paste the Flux public key and click Add key.

;[https://fluxcd.io/docs/installation/#dev-install Dev installation] does not stores its own configuration state in Git repository
<source lang=bash>
# option 1
flux install # install and upgrade
flux install \
--namespace=flux-system \
--network-policy=false \
--components=source-controller

# option 2
kubectl apply -f https://github.com/fluxcd/flux2/releases/latest/download/install.yaml
kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=main | kubectl apply -f- # Upgrade

# Register Git repositories and reconcile them on your cluster:
flux create source git podinfo \
--url=https://github.com/stefanprodan/podinfo \
--tag-semver=">=4.0.0" \
--interval=1m

flux create kustomization podinfo-default \
--source=podinfo \
--path="./kustomize" \
--prune=true \
--validation=client \
--interval=10m \
--health-check="Deployment/podinfo.default" \
--health-check-timeout=2m

# Register Helm repositories and create Helm releases:
flux create source helm bitnami \
--interval=1h \
--url=https://charts.bitnami.com/bitnami

flux create helmrelease nginx \
--interval=1h \
--release-name=nginx-ingress-controller \
--target-namespace=kube-system \
--source=HelmRepository/bitnami \
--chart=nginx-ingress-controller \
--chart-version="5.x.x"
</source>

Uninstall
<source lang=bash>
flux uninstall --namespace=flux-system
</source>

= References =
* [https://github.com/fluxcd/terraform-provider-flux terraform-provider-flux]
*[https://github.com/pio2pio/gitops-istio gitops-istio] Tutorial
*[https://www.youtube.com/watch?v=nGLpUCPX8JE Flux v2 Everything that you wanted to know but were afraid to ask (Stefan Prodan)] December 2020

Bundle
*[https://blog.sldk.de/2021/02/introduction-to-gitops-on-kubernetes-with-flux-v2/ Introduction to GitOps on Kubernetes with Flux v2]
*[https://blog.sldk.de/2021/03/handling-secrets-in-flux-v2-repositories-with-sops/ Handling secrets in Flux v2 repositories with SOPS]

Kubernetes/Progressive Delivery Flux and Flagger

2026-01-15T10:47:59Z

Pio2pio: /* Flux v2 */

= [https://github.com/fluxcd/flux2 Flux v2] =

[https://fluxcd.io/flux/ Flux v2 Documentation]

Flux v2 architecture
:[[File:ClipCapIt-210524-232835.PNG]]

Flux v2 - Webhooks and notifications
:[[File:ClipCapIt-210524-233028.PNG]]

= Install Flux v2 [https://fluxcd.io/docs/cmd/ <code>flux</code>] command line =
* [https://fluxcd.io/docs/get-started/ Fluxv2 Get Started]

{{Note|<code>fluxctl</code> is a previous version Flux v1 command line tool.}}
<source lang=bash>
# Install or upgrade using official install.sh (option-1)
export FLUX_VERSION=0.37.0; curl -s https://fluxcd.io/install.sh | sudo -E bash
curl -s https://fluxcd.io/install.sh | sudo bash # latest

# Install from GitHub releases (option-2)
REPO=fluxcd/flux2
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
VERSION=$LATEST
TEMPDIR=$(mktemp -d); FILE=flux_${VERSION}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${VERSION}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
tar xzvf $TEMPDIR/$FILE.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/flux /usr/local/bin/flux
sudo install $TEMPDIR/flux /usr/local/bin/flux_${VERSION}

# enable completions in ~/.bash_profile
. <(flux completion bash)

# TODO: Via release binaries
# https://github.com/fluxcd/flux/releases

# Pre check
flux check --pre
► checking prerequisites
✗ flux 0.25.1 <0.25.2 (new version is available, please upgrade)
✔ Kubernetes 1.21.5-gke.1302 >=1.19.0-0
✔ prerequisites checks passed

# Docker images
docker pull fluxcd/fluxctl:1.24.3
docker pull ghcr.io/fluxcd/flux-cli:1.24.3 # does not work
</source>

= Cluster bootstrap =
FluxCDv2 bootstrap process is installing the Flux onto a cluster and stores(commits) its own manifests to a Git repository.
* [https://fluxcd.io/docs/installation/#generic-git-server Generic Git Server], including GCP [https://cloud.google.com/source-repositories/docs Cloud Source Repositories]
* [https://fluxcd.io/docs/installation/#bootstrap-with-terraform Bootstrap with Terraform]

<source lang=bash>
FLUX_GIT_USERNAME=my-git-username
FLUX_GIT_EMAIL=my-git-email@example.com
flux bootstrap git \
--author-email=$FLUX_GIT_EMAIL \
--url=ssh://git@github.com/$FLUX_GIT_USERNAME/gitops-istio \
--branch=main \
--path=clusters/my-cluster
</source>
At bootstrap, Flux generates an SSH key and prints the public key. In order to sync your cluster state with git you need to copy the public key and create a deploy key with write access on your GitHub repository. On GitHub go to Settings > Deploy keys click on Add deploy key, check Allow write access, paste the Flux public key and click Add key.

;[https://fluxcd.io/docs/installation/#dev-install Dev installation] does not stores its own configuration state in Git repository
<source lang=bash>
# option 1
flux install # install and upgrade
flux install \
--namespace=flux-system \
--network-policy=false \
--components=source-controller

# option 2
kubectl apply -f https://github.com/fluxcd/flux2/releases/latest/download/install.yaml
kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=main | kubectl apply -f- # Upgrade

# Register Git repositories and reconcile them on your cluster:
flux create source git podinfo \
--url=https://github.com/stefanprodan/podinfo \
--tag-semver=">=4.0.0" \
--interval=1m

flux create kustomization podinfo-default \
--source=podinfo \
--path="./kustomize" \
--prune=true \
--validation=client \
--interval=10m \
--health-check="Deployment/podinfo.default" \
--health-check-timeout=2m

# Register Helm repositories and create Helm releases:
flux create source helm bitnami \
--interval=1h \
--url=https://charts.bitnami.com/bitnami

flux create helmrelease nginx \
--interval=1h \
--release-name=nginx-ingress-controller \
--target-namespace=kube-system \
--source=HelmRepository/bitnami \
--chart=nginx-ingress-controller \
--chart-version="5.x.x"
</source>

Uninstall
<source lang=bash>
flux uninstall --namespace=flux-system
</source>

= References =
* [https://github.com/fluxcd/terraform-provider-flux terraform-provider-flux]
*[https://github.com/pio2pio/gitops-istio gitops-istio] Tutorial
*[https://www.youtube.com/watch?v=nGLpUCPX8JE Flux v2 Everything that you wanted to know but were afraid to ask (Stefan Prodan)] December 2020

Bundle
*[https://blog.sldk.de/2021/02/introduction-to-gitops-on-kubernetes-with-flux-v2/ Introduction to GitOps on Kubernetes with Flux v2]
*[https://blog.sldk.de/2021/03/handling-secrets-in-flux-v2-repositories-with-sops/ Handling secrets in Flux v2 repositories with SOPS]

Ubuntu Setup

2026-01-13T07:25:50Z

Pio2pio: /* Image converter */

If you are using Ubuntu for various Linux projects you will find that as it comes with pre installed with many packages. On the other hand installing just minimal version seems to be too extreme. Therefore I started maitaining a list of unnecessary packages and one liner to that removes them all. Please feel free to modify for your needs.

= Default partitioning =
On virtual systems schema below will be applied, eg on laptops:
:[[File:ClipCapIt-200620-131502.PNG]]

<source lang="bash">
#Eg. for 4G memory and 50G storage system

/dev/mapper/ubuntu--vg-root mount_point: /
/dev/mapper/ubuntu--vg-swapt_1
/dev/sda
/dev/sda1 (50G)

LVM VG ubuntu-vg, LV root as ext4
LVM VG ubuntu-vg, LV swapt_1 as swap

#Boot device:
/dev/mapper/ubuntu--vg-root
</source>
As a good handy practice you may create 100G virtual disk that you thin provision. Then create 2 PVs for root and swap partitions. Don't utilize all space at once but extend partitions when needed. This method eliminates adding new disks to VMs saving time and efforts.

Example LVM setup, here using 30G Physical Volume(99.9% used), 1 Volume Group and 2 Logical Volumes (root and swap).
<source lang="java">
$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 ubuntu-vg lvm2 a-- <29.93g 36.00m
$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
ubuntu-vg 1 2 0 wz--n- <29.93g 36.00m
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root ubuntu-vg -wi-ao---- 28.94g
swap_1 ubuntu-vg -wi-ao---- 976.00m
piotr@u18:~$
</source>

<source lang="java">
$ lsblk /dev/sda --fs
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
└─sda1 LVM2_member rP18Kb-Q12j-wjVf-C1iV-uy42-BUJD-aWFuO7
├─ubuntu--vg-root ext4 fad04a3b-5fa3-4a03-bbd6-24a93cda1eb3 /
└─ubuntu--vg-swap_1 swap 47cd084b-89b0-4cd5-bdb8-367238842ba1 [SWAP]
</source>

= List of unnecessary packages =
<source lang="bash">
sudo apt-get remove libreoffice-* #Remove LibreOffice
sudo apt-get remove unity-lens-* #This package contains photos scopes which allow Unity to search for local and online photos.
sudo apt-get remove shotwell* #Photo organizer
sudo apt-get remove simple-scan #Scanner software
sudo apt-get remove empathy* #Internet messaging ~13M
sudo apt-get remove thunderbird* #Email client ~61M
sudo apt-get remove unity-scope-gdrive #Google Drive scope for Unity ~116KB
sudo apt-get remove cheese* #Cheese Webcam Booth - webcam software
sudo apt-get remove brasero* #Brasero Disc Burner ~6.5MB
sudo apt-get remove gnome-bluetooth Package to manipulate bloototh devices using Gnome desktop ~2MB
sudo apt-get remove gnome-orca Orca Screen Reader -Provide access to graphical desktop environments via synthesised speech and/or refreshable braille
sudo apt-get remove unity-webapps-common #Amazon Unity WebApp integration scripts ~133KB
sudo apt-get remove ibus-pinyin #IBus Bopomofo Preferences - ibus-pinyin is a IBus based IM engine for Chinese ~1.4MB
sudo apt-get remove apt-get remove printer-driver-foo2zjs* #Reactivate HP LaserJet 1018/1020 after reloading paper ~3.2MB
</source>

= Remove unnecessary packages - one liner =
;Ubuntu 12, 14, 16
<source lang="bash">
$ sudo apt-get remove libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* unity-scope-gdrive cheese*\
brasero* gnome-bluetooth gnome-orca unity-webapps-common ibus-pinyin printer-driver-foo2zjs*
</source>

;Ubuntu 18. It's recommended to choose ''Minimal Install'', so most of packages below won't get installed.
<source lang="bash">
$ sudo apt-get purge libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* cheese* \
brasero* gnome-bluetooth gnome-orca ibus-pinyin printer-driver-foo2zjs* xul-ext-ubufox speech-dispatcher* \
rhythmbox* printer-driver-* mythes-en-us mobile-broadband-provider-inf* \
evolution-data-server* espeak-ng-data:amd64 bluez* ubuntu-web-launchers \
transmission-*
</source>
<source lang="bash">
sudo apt-get purge xul-ext-ubufox # Canonical FF customizations for u14,16,18,20
sudo apt-get remove gnome-mahjongg gnome-mines gnome-sudoku # games, works for u14,16,18,20
sudo apt-get remove gnome-video-effects gstreamer1.0-*
</source>

; XTREME
UnInstallant Ubuntu software notifier
<source lang="bash">
sudo apt-get remove update-notifier
</source>

= Uninstall locales - unused languages etc =
<source lang="bash">
sudo apt-get install localepurge
</source>

= Set apt-get to not install recommended and suggested packages =
<source lang="bash">
sudo bash -c 'cat > /etc/apt/apt.conf.d/01no-recommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF'
</source>

To see if apt reads this, enter this in command line (as root or regular user):
<source lang="bash">
apt-config dump | grep -e Recommends -e Suggests
</source>

= Install necessary packages =

Adobe Flash Player
sudo apt-get install flashplugin-installer

Java JRE
This will install the default verison Java for you distro plus Icedtea plugin for using Firefox with Java
sudo apt-get install default-jre icedtea-plugin

Unity Settings
sudo apt-get install unity-control-center

Opera

Add Opera repository <code>'''deb <nowiki>http://deb.opera.com/opera/</nowiki> stable non-free'''</code> to the apt-get source list in <code>/etc/apt/sources.list</code>. Then import a public PGP repository key.
<source lang="bash">
echo "deb http://deb.opera.com/opera/ stable non-free" | sudo tee -a /etc/apt/sources.list
wget -qO - http://deb.opera.com/archive.key | sudo apt-key add -
sudo apt-get update && sudo apt-get install opera
</source>
Silverlight

Pipelight has been released and we can use it for silverlight as a best alternative moonlight.
<source lang="bash">
sudo apt-add-repository ppa:ehoover/compholio
sudo apt-add-repository ppa:mqchael/pipelight
sudo apt-get update
sudo apt-get install pipelight
</source>

= GUI tools =
* [https://github.com/hluk/CopyQ/releases copyQ] clipboard manager
* VisualVM

= Customise Ubuntu =
==Fix Ubuntu Unity Dash Search for Applications and Files==
sudo apt-get install unity-lens-files unity-lens-applications #log out and log back in required

==Fix Ubuntu <17.10 missing Control Center==
sudo apt-get install unity-control-center --no-install-recommends

==Fix Ubuntu >18.04 missing System Settings==
sudo apt install gnome-control-center

==Remove background wallpaper ==
Tested on Ubuntu 14,16,18
<source lang="bash">
gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background draw-background false #disable
gsettings set org.gnome.desktop.background primary-color "#000000" #set to black
gsettings set org.gnome.desktop.background secondary-color "#000000" #set to black
gsettings set org.gnome.desktop.background color-shading-type "solid" #set solid colour
gsettings set org.gnome.desktop.background picture-uri file:///dev/null #remove wallpaper, not perfect but nothing worked in U15.10
gsettings set com.canonical.unity-greeter draw-user-backgrounds false #disable not worked

# Reset background picture to origin, U15.10
gsettings set org.gnome.desktop.background picture-uri file:///usr/share/backgrounds/warty-final-ubuntu.png

# Sets Unity greeter background, <17.04
gsettings set com.canonical.unity-greeter background /usr/share/backgrounds/warty-final-ubuntu.png
</source>

==Disable screen lock out==
<code>dconf</code> is legacy tool to configure <tt>gnome</tt> nowadays more modern way is to use <code>gsettings</code>.
<source lang=bash>
dconf write /org/gnome/desktop/screensaver/idle-activation-enabled false #gnome
dconf write /org/gnome/desktop/screensaver/lock-enabled false

# Unity - Ubuntu 14.04, 16.04
gsettings set org.gnome.desktop.session idle-delay 0 #disable the screen blackout:(0 to disable)
gsettings set org.gnome.desktop.screensaver lock-enabled false #disable the screen lock

# VirtualBox > Ubuntu 18.04 Disabling Xserver screen timeouts
xset s off # Xserver s parameter sets screensaver to off
xset s noblank # prevent the display from blanking
xset -dpms # prevent the monitor's DPMS energy saver from kicking in

# Gnome - Ubuntu 18.04 LTS, Settings > Power > Blank screen > set to: Never
gsettings get org.gnome.desktop.lockdown disable-lock-screen # verify status
gsettings set org.gnome.desktop.lockdown disable-lock-screen true # set disabled
gsettings get org.gnome.desktop.screensaver lock-enabled # verify status
gsettings set org.gnome.desktop.screensaver lock-enabled false # set disabled
dconf write /org/gnome/desktop/screensaver/lock-enabled false # set disbaled using dconf
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false # some say it's last resort :)

# Power management
gsettings set org.gnome.settings-daemon.plugins.power active true #set gnome to be the default power management run
gsettings set org.gnome.settings-daemon.plugins.power active false #turn off power management

# last resort as it was a bud in Ubuntu 11.10 with DPMS
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false
gsettings set org.gnome.desktop.session idle-delay 2400
</source>
Verify by navigating in <tt>dconf-editor</tt> to <tt>/org/gnome/desktop/screensaver/</tt>

==Change number of workspaces==
To get the current values:
<source lang=bash>
dconf read /org/compiz/profiles/unity/plugins/core/hsize
dconf read /org/compiz/profiles/unity/plugins/core/vsize
</source>

To set new values:
<source lang=bash>
dconf write /org/compiz/profiles/unity/plugins/core/hsize 2
# or
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ hsize 4
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ vsize 4
</source>

== Clenup motd messages ==
Ubuntu at login displays a number standard messages taking terminal space causing potential loosing context of previous operations.
<source lang="bash">
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-134-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

1 package can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Aug 31 12:11:28 2018 from 10.0.2.2
</source>

This is managed by files in <code>/etc/update-motd.d/</code>, so deleting them will remove clutter on a screen
<source lang="bash">
ls /etc/update-motd.d/
00-header 51-cloudguest 91-release-upgrade 98-fsck-at-reboot
10-help-text 90-updates-available 97-overlayroot 98-reboot-required

# Ubuntu Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1022-azure x86_64)
# Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)
sudo rm /etc/update-motd.d/{10-help-text,50-landscape-sysinfo,50-motd-news,51-cloudguest,80-livepatch,95-hwe-eol}
</source>

This cuts down to this message, Ubuntu 18.04 in AWS
<source lang="bash">
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)

0 packages can be updated.
0 updates are security updates.

Last login: Thu Jan 31 17:09:38 2019 from 10.10.11.11
</source>

= Useful setups =
== Terminator Grid ==

Edit your Terminator config file

<source lang=toml>
vim ~/.config/terminator/config
[global_config]
enabled_plugins = LaunchpadBugURLHandler, LaunchpadCodeURLHandler, APTURLHandler, InsertTermName, CurrDirOpen
[keybindings]
[profiles]
[[default]]
scrollback_lines = 10000
[layouts]
[[default]]
[[[window0]]]
type = Window
parent = ""
[[[child1]]]
type = Terminal
parent = window0
profile = default
[[2x3-grid]]
[[[child0]]]
type = Window
parent = ""
order = 0
position = 0:0
maximised = True
[[[child1]]]
type = VPaned
parent = child0
order = 0
position = 400
[[[child2]]]
type = HPaned
parent = child1
order = 0
position = 50%
[[[terminal3]]]
type = Terminal
parent = child2
order = 0
profile = default
command = cd /home/piotr; bash
[[[terminal4]]]
type = Terminal
parent = child2
order = 1
profile = default
command = cd /home/piotr; bash
[[[child5]]]
type = VPaned
parent = child1
order = 1
position = 400
[[[child6]]]
type = HPaned
parent = child5
order = 0
position = 50%
[[[terminal7]]]
type = Terminal
parent = child6
order = 0
profile = default
command = cd /home/piotr; bash
[[[terminal8]]]
type = Terminal
parent = child6
order = 1
profile = default
command = cd /home/piotr; bash
[[[child9]]]
type = HPaned
parent = child5
order = 1
position = 50%
[[[terminal10]]]
type = Terminal
parent = child9
order = 0
profile = default
command = cd /home/piotr; bash
[[[terminal11]]]
type = Terminal
parent = child9
order = 1
profile = default
command = cd /home/piotr; bash
[plugins]
</source>

Now create the new launcher and update the database.

<source lang=toml>
vim ~/.local/share/applications/terminator-grid.desktop

[Desktop Entry]
Name=Terminator-Grid
Comment=Multiple terminals in one window
TryExec=terminator
# DEFAULT ACTION: Now opens the grid
Exec=terminator --layout 2x3-grid
Icon=terminator
Type=Application
Categories=GNOME;GTK;Utility;TerminalEmulator;System;
StartupNotify=true
X-Ubuntu-Gettext-Domain=terminator
Keywords=terminal;shell;prompt;command;commandline;
MimeType=x-scheme-handler/terminal;
# Define the right-click options
Actions=NewWindow;

[Desktop Action NewWindow]
Name=Open a Single Window
# RIGHT-CLICK ACTION: Opens standard terminator
Exec=terminator
</source>

Now update database, then Restart GNOME Shell (if the icon doesn't appear in search): Press Alt + F2, type r, and hit Enter. (Note: This only works on X11, not Wayland. On Wayland, just logging out and back in works).

<source lang=bash>
update-desktop-database ~/.local/share/applications/
</source>

Search and Pin:
* Open your app launcher (Super key).
* Search for "Terminator".
* Right-click it and select Add to Favorites.

== Image converter ==
nautilus-image-converter is a nautilus extension to mass resize or rotate images. It adds two context menu items in nautlius so you can right-click and choose "Resize Image" or "Rotate Image").
<source lang=bash>
# tested on Ubuntu 24.04 with Gnome
sudo apt-get install nautilus-image-converter

# Restart to see the new context menu
nautilus -q
</source>

== Call screen saver from a terminal to blank all screens ==
<source lang=bash>
# tested on Ubuntu 18.04 with Gnome
sudo apt-get install gnome-screensaver
gnome-screensaver-command -a #controls GNOME screensaver, -a activate (blank the screen)
</source>

== Create application launcher ==
;Ubuntu 18.04
<source lang=bash>
# Install the GNOME-panel toolset
sudo apt-get install --no-install-recommends gnome-panel

# Every user launcher
sudo gnome-desktop-item-edit /usr/share/applications/VisualVM.desktop --create-new

# Local user only, the filename by default is a Name-of-appication.desktop
gnome-desktop-item-edit ~/.local/share/applications --create-new
</source>

:[[File:ClipCapIt-190807-080016.PNG]]

;Ubuntu 19.10, 20.04
In above releases <code>gnome-desktop-item-edit</code> has been removed from the <code>gnome-panel</code> package, as an alternative <code>.desktop</code> files can be created manually.
<source lang=bash>
vi /usr/share/applications/APPNAME.desktop
[Desktop Entry]
Name=<NAME OF THE APPLICATION>
Comment=<A SHORT DESCRIPTION>
Exec=<COMMAND-OR-FULL-PATH-TO-LAUNCH-THE-APPLICATION>
Type=Application
Terminal=false
Icon=<ICON NAME OR PATH TO ICON>
NoDisplay=false
Keywords=<eg. sql>
</source>

It's optional but you may need to right click and set 'allow launching' with addition to set executable permissions. Usual locations of <code>.desktop</code> files are:
* <code>/usr/share/applications/</code>
* <code>/var/lib/snapd/desktop/applications/</code> for snap applications

== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet gnome-shell-system-monitor-applet] - cpu, memory indicators ==
System information such as memory usage, cpu usage, network rates and more can be displayed in the notification area in GNOME Shell.

System-monitor extensions:
[https://extensions.gnome.org/extension/120/system-monitor/ system-monitor] by paradoxxxzero on [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet github] supports Gnome-shell up to v40. It seems like abandoned project.
[https://extensions.gnome.org/extension/3010/system-monitor-next/ system-monitor-next] by mgalgs on [https://github.com/mgalgs/gnome-shell-system-monitor-applet github] supports Gnome-shell v40+, it's a fork of the above.

All extensions:
* https://extensions.gnome.org

{{Note|The current version of the browser Firefox is packaged as a snap version. One of the issues with this is that it cannot work with the Gnome Extensions website.}}

Install on Ubuntu 24.04 (June 2024)
<source lang=bash>
# Ubuntu version tested: v20/22/24 LTS
lsb_release -d
Description: Ubuntu 24.04.3 LTS

gnome-shell --version
GNOME Shell 46.0

# Install the Gnome-Shell-Extension & Manager
sudo apt install gnome-shell-extensions # Ubuntu 20.04 LTS already has this package, 24.04 needs installing it
sudo apt install gnome-shell-extension-manager # Ubuntu 22.04|24.04 LTS

# 1. Open `Extensions` app, turn "Use Extensions". It is already turned on in Ubuntu 24.04.3 LTS.
# 2. Open Browse tab > search for 'system-monitor-next' by mgalgs, click "Install".
# 3. "cpu/mem/net" indicators will appear in the system tray.

# Additional steps for Ubuntu < 24.04
sudo apt install gnome-tweaks # GUI to manage gnome-extensions
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
sudo apt install gnome-shell-extension-system-monitor # after requires log out

# Download the extension from
## https://extensions.gnome.org/extension/120/system-monitor/

# Never worked out how to use this direct download and install via 'gnome-extensions install <extension_name>'
## wget https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/archive/v38.zip
## gnome-extensions install <system-monitor@paradoxxx.zero.gmail.com>

# Enable extension using cli
gnome-extensions enable system-monitor-next@paradoxxx.zero.gmail.com
gnome-extensions list --user
clipboard-indicator@tudmotu.com
system-monitor-next@paradoxxx.zero.gmail.com
</source>
:[[File:ClipCapIt-210105-084527.PNG]]

=== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/issues/737#issuecomment-1230654455 Ubuntu 22.04 workaround for the OUTDATED extension] ===
{{Note|Workaround still needed in August 2022}}
<source lang=bash>
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
git clone https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet.git
cd gnome-shell-system-monitor-applet # commit b359d88 verified
vi system-monitor@paradoxxx.zero.gmail.com/metadata.json
# | change "version": -1 to "version": 42
make install
# log out and back in (required)
</source>

= Snapd - Chromium =
Recently in U19+ Chromium get installed via snapd package. This is classic installation that has limited access to only a certain directories. It happen that when working with AWS we need get access to <code>~/.ssh</code> folder to get ec2 machine password. This folder is denied, but we can bind mount <code>~/.ssh</code> folder into the snap container directory:
<source lang=bash>
$ snap list chromium
Name Version Rev Tracking Publisher Notes
chromium 86.0.4240.111 1373 latest/stable canonical✓ -

# cd to chromim $HOME dir
mkdir ~/snap/chromium/current/.ssh
sudo mount --bind ~/.ssh/ ~/snap/chromium/current/.ssh
</source>

= Screen shooting =
In Ubuntu 20.04 Shutter is not a part of default repositories. It can be added via PPA:
<source lang=bash>
sudo add-apt-repository -y ppa:linuxuprising/shutter
sudo apt-get install shutter
</source>

= Audio - [https://rastating.github.io/setting-default-audio-device-in-ubuntu-18-04/ set defaults] =
For preserving settings using GUI you can install [https://freedesktop.org/software/pulseaudio/pavucontrol/ PulseAudio Volume Control] <code>pavucontrol</code>.
<source lang=bash>
# install
sudo apt install pavucontrol # Ubuntu 20.04
# run
pavucontrol
</source>

Set default output/input device. In Ubuntu PulseAudio is used to control audio devices. It contains following configuration files
<source lang=bash>
/etc/pulse/default.pa # system wide
~/.config/pulse # user configuration
</source>

Set defaults
<source lang=bash>
# List devices: modules, sinks, sources, sink-inputs, source-outputs, clients, samples, cards
# sinks - outputs, sink-inputs, sources - all input/output including RUNNING and SUSPENDED devices
$ pactl list short sources | column -t
5 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_5__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
6 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_4__sink.monitor module-alsa-card.c s16le 2ch 48000Hz RUNNING
7 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_3__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
8 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
9 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__source module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
10 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_6__source module-alsa-card.c s16le 4ch 48000Hz SUSPENDED
15 alsa_output.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.analog-stereo.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz SUSPENDED
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz SUSPENDED
20 alsa_input.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.iec958-stereo module-alsa-card.c s16le 2ch 48000Hz RUNNING

# Set defaut output device. Tab autocompletion should work (U20.04)
pactl set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output
# Set defaut input device
pactl set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Test, play some audio then run. IDLE - means in use
pactl list short sources | column -t | grep -e RUNNING -e IDLE
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz IDLE
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz RUNNING
</source>

Make it permanent by setting default device in PulseAudio system configuration file
<source lang=bash>
# Output device
OUTPUT_DEVICE=alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-sink$ output/\1 ${OUTPUT_DEVICE}/g" /etc/pulse/default.pa # remove '-i' to test before apply
# Input device
INPUT_DEVICE=alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-source$ input/\1 ${INPUT_DEVICE}/g" /etc/pulse/default.pa

vi /etc/pulse/default.pa # make sure lines below are in place
### Make some devices default
set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Delete local user profile and restart system, after boot new defaults should be set
rm -r ~/.config/pulse

# After reboot, defaults should be set
cat ~/.config/pulse/*default*
alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
</source>

;Troubleshooting
PulseAudio cli
<source lang=bash>
pacmd
>>> help # lists all available commands

pulseaudio --check # Check if any pulseaudio instance is running. It normally prints no output, just exit code. 0 means running
pulseaudio --kill # kill, then --start
pulseaudio -D # start pulseaudio as a daemon
# | using /etc/pulse/daemon.conf

# Pulseaudio is a user service
systemctl --user restart pulseaudio.service
systemctl --user restart pulseaudio.socket
</source>

I have a port replicator Dell D-6000, that gets randomly disconnected causing switching audio to new connected device - means itself. As workaround commenting out lines below stops this behaviour.
<source lang=bash>
vi /etc/pulse/default.pa
### Use hot-plugged devices like Bluetooth or USB automatically (LP: #1702794)
# .ifexists module-switch-on-connect.so
# load-module module-switch-on-connect
# .endif
</source>

= Input devices =
Motivation is to enable horizontal scrolling in Ubuntu 20.04 using Perixx Gamig Mouse Mx2000
<source lang=bash>
xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Holtek USB Gaming Mouse id=11 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Mouse id=14 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Touchpad id=15 [slave pointer (2)]
⎜ ↳ TPPS/2 Elan TrackPoint id=19 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Sleep Button id=8 [slave keyboard (3)]
↳ CHICONY HP Basic USB Keyboard id=9 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=10 [slave keyboard (3)]
↳ Integrated Camera: Integrated C id=12 [slave keyboard (3)]
↳ Integrated Camera: Integrated I id=13 [slave keyboard (3)]
↳ sof-hda-dsp Headset Jack id=16 [slave keyboard (3)]
↳ Intel HID events id=17 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=18 [slave keyboard (3)]
↳ ThinkPad Extra Buttons id=20 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=21 [slave keyboard (3)]

# test mouse aka Virtual core pointer
xinput test 11
motion a[0]=2023 # <- cursor moving
motion a[0]=2024 a[1]=1411
motion a[3]=19545 # <- scroll down
button press 5
button release 5

# test 'virtual core keyboard' aka additional programmable buttons
## '10' - this virtual keyboard for all buttons except the scrolling wheel
xinput test 10
key press 37
key press 38

## '21' - this is scrolling wheel buttons left/right, not scrolling itself
xinput test 21
key press 248
key release 248
</source>

# List of properties of a device. We want to see 'horizontal scrolling wheel buttons'
<source lang=bash>
$ xinput list-props 21
Device 'Holtek USB Gaming Mouse':
Device Enabled (169): 1
Coordinate Transformation Matrix (171): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
libinput Send Events Modes Available (291): 1, 0
libinput Send Events Mode Enabled (292): 0, 0
libinput Send Events Mode Enabled Default (293): 0, 0
Device Node (294): "/dev/input/event10"
Device Product ID (295): 1241, 41063
</source>

=References=

[[Category:linux]]

Linux shell/Productivity tools

2025-12-03T06:38:48Z

Pio2pio: /* Opencode */

= Autojump =
<source lang="bash">
https://github.com/wting/autojump#manual
sudo apt-get install autojump
cat /usr/share/doc/autojump/README.Debian

Autojump for Debian
-------------------

To use autojump, you need to configure you shell to source
/usr/share/autojump/autojump.sh on startup.

If you use Bash, add the following line to your ~/.bashrc (for non-login
interactive shells) and your ~/.bash_profile (for login shells):
. /usr/share/autojump/autojump.sh

If you use Zsh, add the following line to your ~/.zshrc (for all interactive shells):
. /usr/share/autojump/autojump.sh

# Usage
j -s # display statistics

j foo # Jump To A Directory That Contains foo. It takes multiple arguments to do fuzzy search
jc bar # jump to a child directory (sub-directory of current directory) rather than typing out the full name
jo music # Open File Manager To Directories (instead of jumping)
jco images # Opening a file manager to a child directory is also supported
</source>

= direnv =
<source lang=bash>
# Install direnv Package
sudo apt install direnv

# Hook direnv into Your Shell to load direnv every time it starts
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

# To start using it in a project, navigate to a directory, create a file named .envrc with your environment variables
# (e.g., export DB_HOST=localhost), and then run the following command to authorize it:
direnv allow .
</source>

= [https://github.com/sst/opencode Opencode] =

<source lang="bash">
# Install Official via bash pipe
curl -fsSL https://opencode.ai/install | bash

# Installs in ~/.opencode and adds <code>export PATH=/home/vagrant/.opencode/bin:$PATH</code> to the bottom of ~/.bashrc
tree ~/.opencode/
/home/vagrant/.opencode/
└── bin
└── opencode

# Upgrade
opencode upgrade

# Uninstall
delete .opencode and .opencode.json

# Install opencode via Githib Releases
VERSION=$(curl --silent "https://api.github.com/repos/sst/opencode/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/sst/opencode/releases/download/${VERSION}/opencode-linux-x64.tar.gz -o $TEMPDIR/opencode-linux-x64.tar.gz
tar xzf $TEMPDIR/opencode-linux-x64.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/opencode /usr/local/bin/opencode

# Version
opencode version
</source>

Connect to paid Github subscription, <code>opencode auth login</code> choose github copilot (public) - don't worry. Follow the process and It will get you to log in to your account covered by the privacy subscription.

<source lang="bash">
opencode auth login

# Create ~/.config/opencode/opencode.json
cat > ~/.config/opencode/opencode.json <<'EOL'
{
"disabled_providers": [
"openai",
"opencode",
"anthropic",
"google",
"groq",
"mistral"
],
"model": "github-copilot/claude-opus-4.5",
"permission": {
"webfetch": "ask"
},
"instructions": [
"~/.config/opencode/GLOBAL_RULES.md"
],
"$schema": "https://opencode.ai/config.json"
}
EOL

# create ~/.config/opencode/GLOBAL_RULES.md

cat > ~/.config/opencode/GLOBAL_RULES.md <<EOL
# SECURITY & PRIVACY PROTOCOL
1. **NO SECRETS:** You are STRICTLY FORBIDDEN from outputting, printing, or repeating any API keys, passwords, credentials, or secrets found in code or logs.
2. **REDACTION:** If you must reference a line of code containing a secret, you must replace the secret with <REDACTED>.
3. **MOCK DATA:** When generating code examples or tests, always use dummy data (e.g., "example_key_123"), never real data from the context.
Then opencode models only has Copilot ones, none f the open source ones.
EOL

# Verify auth credentials file
cat ~/.local/share/opencode/auth.json

# List models
opencode models
github-copilot/claude-haiku-4.5
github-copilot/claude-opus-4.5
github-copilot/claude-opus-41
github-copilot/claude-sonnet-4
github-copilot/claude-sonnet-4.5
github-copilot/gemini-2.5-pro
github-copilot/gemini-3-pro-preview
github-copilot/gpt-4.1
github-copilot/gpt-4o
github-copilot/gpt-5
github-copilot/gpt-5-codex
github-copilot/gpt-5-mini
github-copilot/gpt-5.1
github-copilot/gpt-5.1-codex
github-copilot/gpt-5.1-codex-mini
github-copilot/grok-code-fast-1
github-copilot/oswe-vscode-prime
</source>

Linux shell/Productivity tools

2025-12-03T06:38:08Z

Pio2pio: /* Opencode */

Linux shell/Productivity tools

2025-12-03T06:33:27Z

Pio2pio: /* Opencode */

= Autojump =
<source lang="bash">
https://github.com/wting/autojump#manual
sudo apt-get install autojump
cat /usr/share/doc/autojump/README.Debian

Autojump for Debian
-------------------

To use autojump, you need to configure you shell to source
/usr/share/autojump/autojump.sh on startup.

If you use Bash, add the following line to your ~/.bashrc (for non-login
interactive shells) and your ~/.bash_profile (for login shells):
. /usr/share/autojump/autojump.sh

If you use Zsh, add the following line to your ~/.zshrc (for all interactive shells):
. /usr/share/autojump/autojump.sh

# Usage
j -s # display statistics

j foo # Jump To A Directory That Contains foo. It takes multiple arguments to do fuzzy search
jc bar # jump to a child directory (sub-directory of current directory) rather than typing out the full name
jo music # Open File Manager To Directories (instead of jumping)
jco images # Opening a file manager to a child directory is also supported
</source>

= direnv =
<source lang=bash>
# Install direnv Package
sudo apt install direnv

# Hook direnv into Your Shell to load direnv every time it starts
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

# To start using it in a project, navigate to a directory, create a file named .envrc with your environment variables
# (e.g., export DB_HOST=localhost), and then run the following command to authorize it:
direnv allow .
</source>

= [https://github.com/sst/opencode Opencode] =

<source lang="bash">
# Install Official via bash pipe
curl -fsSL https://opencode.ai/install | bash

# Installs in ~/.opencode and adds <code>export PATH=/home/vagrant/.opencode/bin:$PATH</code> to the bottom of ~/.bashrc
tree ~/.opencode/
/home/vagrant/.opencode/
└── bin
└── opencode

# Upgrade
opencode upgrade

# Uninstall
delete .opencode and .opencode.json

# Install opencode via Githib Releases
VERSION=$(curl --silent "https://api.github.com/repos/sst/opencode/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/sst/opencode/releases/download/${VERSION}/opencode-linux-x64.tar.gz -o $TEMPDIR/opencode-linux-x64.tar.gz
tar xzf $TEMPDIR/opencode-linux-x64.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/opencode /usr/local/bin/opencode

# Version
opencode version
</source>

Connect to paid Github subscription, <code>opencode auth login</code> choose github copilot (public) - don't worry. Follow the process and It will get you to log in to your account covered by the privacy subscription.

<source lang="bash">
opencode auth login

# Create ~/.config/opencode/opencode.json
cat > ~/.config/opencode/opencode.json <<EOL
{
"disabled_providers": [
"openai",
"opencode",
"anthropic",
"google",
"groq",
"mistral"
],
"model": "github-copilot/claude-opus-4.5",
"permission": {
"webfetch": "ask"
},
"instructions": [
"~/.config/opencode/GLOBAL_RULES.md"
],
"$schema": "https://opencode.ai/config.json"
}
EOL

# create ~/.config/opencode/GLOBAL_RULES.md

cat > ~/.config/opencode/GLOBAL_RULES.md <<EOL
# SECURITY & PRIVACY PROTOCOL
1. **NO SECRETS:** You are STRICTLY FORBIDDEN from outputting, printing, or repeating any API keys, passwords, credentials, or secrets found in code or logs.
2. **REDACTION:** If you must reference a line of code containing a secret, you must replace the secret with <REDACTED>.
3. **MOCK DATA:** When generating code examples or tests, always use dummy data (e.g., "example_key_123"), never real data from the context.
Then opencode models only has Copilot ones, none f the open source ones.
EOL

# Verify auth credentials file
cat ~/.local/share/opencode/auth.json
</source>

Linux shell/Productivity tools

2025-12-03T06:30:03Z

Pio2pio: /* Opencode */

= Autojump =
<source lang="bash">
https://github.com/wting/autojump#manual
sudo apt-get install autojump
cat /usr/share/doc/autojump/README.Debian

Autojump for Debian
-------------------

To use autojump, you need to configure you shell to source
/usr/share/autojump/autojump.sh on startup.

If you use Bash, add the following line to your ~/.bashrc (for non-login
interactive shells) and your ~/.bash_profile (for login shells):
. /usr/share/autojump/autojump.sh

If you use Zsh, add the following line to your ~/.zshrc (for all interactive shells):
. /usr/share/autojump/autojump.sh

# Usage
j -s # display statistics

j foo # Jump To A Directory That Contains foo. It takes multiple arguments to do fuzzy search
jc bar # jump to a child directory (sub-directory of current directory) rather than typing out the full name
jo music # Open File Manager To Directories (instead of jumping)
jco images # Opening a file manager to a child directory is also supported
</source>

= direnv =
<source lang=bash>
# Install direnv Package
sudo apt install direnv

# Hook direnv into Your Shell to load direnv every time it starts
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

# To start using it in a project, navigate to a directory, create a file named .envrc with your environment variables
# (e.g., export DB_HOST=localhost), and then run the following command to authorize it:
direnv allow .
</source>

= [https://github.com/sst/opencode Opencode] =

<source lang="bash">
# Install Official via bash pipe
curl -fsSL https://opencode.ai/install | bash

# Installs in ~/.opencode and adds <code>export PATH=/home/vagrant/.opencode/bin:$PATH</code> to the bottom of ~/.bashrc
tree ~/.opencode/
/home/vagrant/.opencode/
└── bin
└── opencode

# Upgrade
opencode upgrade

# Uninstall
delete .opencode and .opencode.json

# Install opencode via Githib Releases
VERSION=$(curl --silent "https://api.github.com/repos/sst/opencode/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/sst/opencode/releases/download/${VERSION}/opencode-linux-x64.tar.gz -o $TEMPDIR/opencode-linux-x64.tar.gz
tar xzf $TEMPDIR/opencode-linux-x64.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/opencode /usr/local/bin/opencode

# Version
opencode version
</source>

Connect to paid Github subscription, <code>opencode auth login</code> choose github copilot (public) - don't worry. Follow the process and It will get you to log in to your account covered by the privacy subscription.

<source lang="bash">
opencode auth login

# Create ~/.config/opencode/opencode.json
cat > ~/.config/opencode/opencode.json <<EOL
{
"disabled_providers": [
"openai",
"opencode",
"anthropic",
"google",
"groq",
"mistral"
],
"model": "github-copilot/claude-opus-4.5",
"permission": {
"webfetch": "ask"
},
"instructions": [
"~/.config/opencode/GLOBAL_RULES.md"
],
"$schema": "https://opencode.ai/config.json"
}
EOL

# create ~/.config/opencode/GLOBAL_RULES.md

cat > ~/.config/opencode/GLOBAL_RULES.md <<EOL
# SECURITY & PRIVACY PROTOCOL
1. **NO SECRETS:** You are STRICTLY FORBIDDEN from outputting, printing, or repeating any API keys, passwords, credentials, or secrets found in code or logs.
2. **REDACTION:** If you must reference a line of code containing a secret, you must replace the secret with `<REDACTED>`.
3. **MOCK DATA:** When generating code examples or tests, always use dummy data (e.g., "example_key_123"), never real data from the context.
Then opencode models only has Copilot ones, none f the open source ones.
EOL

# Verify auth credentials file
cat ~/.local/share/opencode/auth.json
</source>

Linux shell/Productivity tools

2025-12-03T06:28:51Z

Pio2pio: /* Opencode */

= Autojump =
<source lang="bash">
https://github.com/wting/autojump#manual
sudo apt-get install autojump
cat /usr/share/doc/autojump/README.Debian

Autojump for Debian
-------------------

To use autojump, you need to configure you shell to source
/usr/share/autojump/autojump.sh on startup.

If you use Bash, add the following line to your ~/.bashrc (for non-login
interactive shells) and your ~/.bash_profile (for login shells):
. /usr/share/autojump/autojump.sh

If you use Zsh, add the following line to your ~/.zshrc (for all interactive shells):
. /usr/share/autojump/autojump.sh

# Usage
j -s # display statistics

j foo # Jump To A Directory That Contains foo. It takes multiple arguments to do fuzzy search
jc bar # jump to a child directory (sub-directory of current directory) rather than typing out the full name
jo music # Open File Manager To Directories (instead of jumping)
jco images # Opening a file manager to a child directory is also supported
</source>

= direnv =
<source lang=bash>
# Install direnv Package
sudo apt install direnv

# Hook direnv into Your Shell to load direnv every time it starts
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

# To start using it in a project, navigate to a directory, create a file named .envrc with your environment variables
# (e.g., export DB_HOST=localhost), and then run the following command to authorize it:
direnv allow .
</source>

= [https://github.com/sst/opencode Opencode] =

<source lang="bash">
# Install Official via bash pipe
curl -fsSL https://opencode.ai/install | bash

# Installs in ~/.opencode and adds <code>export PATH=/home/vagrant/.opencode/bin:$PATH</code> to the bottom of ~/.bashrc
tree ~/.opencode/
/home/vagrant/.opencode/
└── bin
└── opencode

# Upgrade
opencode upgrade

# Uninstall
delete .opencode and .opencode.json

# Install opencode via Githib Releases
VERSION=$(curl --silent "https://api.github.com/repos/sst/opencode/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/sst/opencode/releases/download/${VERSION}/opencode-linux-x64.tar.gz -o $TEMPDIR/opencode-linux-x64.tar.gz
tar xzf $TEMPDIR/opencode-linux-x64.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/opencode /usr/local/bin/opencode

# Version
opencode version
</source>

Connect to paid Github subscription, <code>opencode auth login</code> choose github copilot (public) - don't worry. Follow the process and It will get you to log in to your account covered by the privacy subscription.

<source lang="bash">
opencode auth login

# Create ~/.config/opencode/opencode.json
cat > ~/.config/opencode/opencode.json <<EOL
{
"disabled_providers": [
"openai",
"opencode",
"anthropic",
"google",
"groq",
"mistral"
],
"model": "github-copilot/claude-opus-4.5",
"permission": {
"webfetch": "ask"
},
"instructions": [
"~/.config/opencode/GLOBAL_RULES.md"
],
"$schema": "https://opencode.ai/config.json"
}
EOL

# create ~/.config/opencode/GLOBAL_RULES.md

cat > ~/.config/opencode/GLOBAL_RULES.md <<EOL
# SECURITY & PRIVACY PROTOCOL
1. **NO SECRETS:** You are STRICTLY FORBIDDEN from outputting, printing, or repeating any API keys, passwords, credentials, or secrets found in code or logs.
2. **REDACTION:** If you must reference a line of code containing a secret, you must replace the secret with `<REDACTED>`.
3. **MOCK DATA:** When generating code examples or tests, always use dummy data (e.g., "example_key_123"), never real data from the context.
Then opencode models only has Copilot ones, none f the open source ones.
EOL
</source>

Linux shell/Productivity tools

2025-12-03T06:27:46Z

Pio2pio: /* Opencode */

= Autojump =
<source lang="bash">
https://github.com/wting/autojump#manual
sudo apt-get install autojump
cat /usr/share/doc/autojump/README.Debian

Autojump for Debian
-------------------

To use autojump, you need to configure you shell to source
/usr/share/autojump/autojump.sh on startup.

If you use Bash, add the following line to your ~/.bashrc (for non-login
interactive shells) and your ~/.bash_profile (for login shells):
. /usr/share/autojump/autojump.sh

If you use Zsh, add the following line to your ~/.zshrc (for all interactive shells):
. /usr/share/autojump/autojump.sh

# Usage
j -s # display statistics

j foo # Jump To A Directory That Contains foo. It takes multiple arguments to do fuzzy search
jc bar # jump to a child directory (sub-directory of current directory) rather than typing out the full name
jo music # Open File Manager To Directories (instead of jumping)
jco images # Opening a file manager to a child directory is also supported
</source>

= direnv =
<source lang=bash>
# Install direnv Package
sudo apt install direnv

# Hook direnv into Your Shell to load direnv every time it starts
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

# To start using it in a project, navigate to a directory, create a file named .envrc with your environment variables
# (e.g., export DB_HOST=localhost), and then run the following command to authorize it:
direnv allow .
</source>

= [https://github.com/sst/opencode Opencode] =

<source lang="bash">
# Install Official via bash pipe
curl -fsSL https://opencode.ai/install | bash

# Installs in ~/.opencode and adds <code>export PATH=/home/vagrant/.opencode/bin:$PATH</code> to the bottom of ~/.bashrc
tree ~/.opencode/
/home/vagrant/.opencode/
└── bin
└── opencode

# Upgrade
opencode upgrade

# Uninstall
delete .opencode and .opencode.json

# Install opencode via Githib Releases
VERSION=$(curl --silent "https://api.github.com/repos/sst/opencode/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/sst/opencode/releases/download/${VERSION}/opencode-linux-x64.tar.gz -o $TEMPDIR/opencode-linux-x64.tar.gz
tar xzf $TEMPDIR/opencode-linux-x64.tar.gz -C $TEMPDIR
sudo install $TEMPDIR/opencode /usr/local/bin/opencode

# Version
opencode version
</source>

Connect to paid Github subscription, <code>opencode auth login</code> choose github copilot (public) - don't worry. Follow the process and It will get you to log in to your account covered by the privacy subscription.

<source lang="bash">
opencode auth login

# Create ~/.config/opencode/opencode.json
cat > ~/.config/opencode/opencode.json <<EOL
{
"disabled_providers": [
"openai",
"opencode",
"anthropic",
"google",
"groq",
"mistral"
],
"model": "github-copilot/claude-opus-4.5",
"permission": {
"webfetch": "ask"
},
"instructions": [
"~/.config/opencode/GLOBAL_RULES.md"
],
"$schema": "https://opencode.ai/config.json"
}
EOL

# and ~/.config/opencode/GLOBAL_RULES.md

cat > ~/.config/opencode/GLOBAL_RULES.md <<EOL
# SECURITY & PRIVACY PROTOCOL
1. **NO SECRETS:** You are STRICTLY FORBIDDEN from outputting, printing, or repeating any API keys, passwords, credentials, or secrets found in code or logs.
2. **REDACTION:** If you must reference a line of code containing a secret, you must replace the secret with `<REDACTED>`.
3. **MOCK DATA:** When generating code examples or tests, always use dummy data (e.g., "example_key_123"), never real data from the context.
Then opencode models only has Copilot ones, none f the open source ones.
EOL

Linux shell/Productivity tools

2025-12-03T06:19:58Z

Pio2pio: /* direnv */

Linux shell/Productivity tools

2025-12-03T06:19:07Z

Pio2pio: /* direnv */

Linux shell/Productivity tools

2025-12-02T08:46:14Z

Pio2pio: /* Opencode */

Kubernetes/ArgoCD

2025-11-10T10:03:23Z

Pio2pio:

= Install cli =
{{Note|Requires <code>jq</code>}}
<source lang=bash>
REPO=argoproj/argo-cd
REPO_FILE=argocd-linux-amd64
BINARY=argocd
LATEST=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name | tr -d v); echo ${LATEST}
TEMPDIR=$(mktemp -d)
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/${REPO_FILE} -o ${TEMPDIR}/${BINARY}
sudo install ${TEMPDIR}/${BINARY} /usr/local/bin/${BINARY}

# Version
argocd version
argocd: v3.1.8+becb020
BuildDate: 2025-09-30T16:04:21Z
GitCommit: becb020064fe9be5381bf6e5818ff8587ca8f377
GitTreeState: clean
GoVersion: go1.24.6
Compiler: gc
Platform: linux/amd64
argocd-server: v3.1.8+becb020
BuildDate: 2025-09-30T15:33:46Z
GitCommit: becb020064fe9be5381bf6e5818ff8587ca8f377
GitTreeState: clean
GoVersion: go1.24.6
Compiler: gc
Platform: linux/amd64
Kustomize Version: v5.7.0 2025-06-28T07:00:07Z
Helm Version: v3.18.4+gd80839c
Kubectl Version: v0.33.1
Jsonnet Version: v0.21.0
</source>

= Login =
<source lang=bash>
ARGOCD_SERVER=argocd.acme.com
ARGOCD_ADMINPASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

# Usual login
argocd login $ARGOCD_SERVER --username admin --password $ARGOCD_ADMINPASSWORD --grpc-web

# Behind a proxy or ArgoCD is configured only on port 80 (never worked)
argocd login argocd.acme.com --username admin --password $ARGOCD_ADMINPASSWORD --plaintext --port-forward --port-forward-namespace argocd
'admin:login' logged in successfully
Context 'port-forward' updated
</source>

Terraform

2025-09-30T15:42:11Z

Pio2pio: /* tfautomv - Terraform refactor */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv --depth=1
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install v1.12.1
tfenv use v1.12.1
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory. Or set an environment variable. Then rerun <code>terraform init</code> to save providers into shared (cache) directory.

<source lang=terraform>
# Option 1.
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Option 2.
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache

# Create the cache directory
mkdir $HOME/.terraform.d/plugin-cache

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Run <code>terraform init</code>.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
# Install
curl -sSfL https://raw.githubusercontent.com/busser/tfautomv/main/install.sh | sudo sh

# Use
tfautomv -dry-run
tfautomv -show-analysis

terraform plan -out=tfplan.bin
tfautomv --preplanned # this generates the moves.tf file

# Apply the changes, by running terraform apply
terraform apply

# Delete moves.tf
rm moves.tf
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

Docker

2025-09-03T04:49:39Z

Pio2pio: /* Ubuntu 24.04 */

Containers taking a world J

= [https://docs.docker.com/install/linux/docker-ce/ubuntu/ Installation] =
General procedure:
# Make sure you don't have docker already installed from your packet manager
# The /var/lib/docker may be

To install the latest version of Docker with curl:
<source lang="bash">
curl -sSL https://get.docker.com/ | sh
</source>

== CentOS ==
<source lang="bash">
sudo yum install bash-completion bash-completion-extras #optional, requires you log out
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 #utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo #docker-ee.repo for EE edition
# --enable docker-ce-{edge|test} #for beta releases
sudo yum update
sudo yum clean all #not sure why this command is here
sudo yum install docker-ce
#old: sudo yum install -y --setopt=obsoletes=0 docker-ce-17.03.1.ce-1.el7.centos docker-ce-selinux-17.03.1.ce-1.el7.centos
sudo systemctl enable docker && sudo systemctl start docker && sudo systemctl status docker
yum-config-manager --disable jenkins #disable source to prevent accidental update ?jenkins?
</source>

== Ubuntu 24.04 ==
<source lang="bash">

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install the latest version
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Install a specific version
## List the available versions:
apt-cache madison docker-ce | awk '{ print $3 }'
5:28.3.3-1~ubuntu.24.04~noble
5:28.3.2-1~ubuntu.24.04~noble

VERSION_STRING=5:28.3.3-1~ubuntu.24.04~noble
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

# Manage Docker as a non-root user
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker # activate the group without logging off
</source>

== Ubuntu 16.04, 18.04, 20.04 ==
<source lang="bash">
# Optional, clear out config files
sudo rm /etc/systemd/system/docker.service.d/docker.conf
sudo rm /etc/systemd/system/docker.service
sudo rm /etc/default/docker #environment file

# New docker package is called now 'docker-ce'
sudo apt-get remove docker docker-engine docker.io containerd runc docker-ce # start fresh
sudo apt-get -yq install apt-transport-https ca-certificates curl gnupg-agent software-properties-common # apt over HTTPs
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # Docker official GPG key
sudo apt-key fingerprint 0EBFCD88 #verify

#add the repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # or {edge|test}
sudo apt-get update # optional

# Option 1 - install latest
sudo apt-get install docker-ce docker-ce-cli containerd.io

# Option 2 - install fixed version
sudo apt-cache madison docker-ce # display available versions
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
sudo apt-get install docker-ce=18.09.0~3-0~ubuntu-bionic docker-ce-cli=18.09.0~3-0~ubuntu-bionic containerd.io
sudo apt-mark hold docker-ce docker-ce-cli containerd.io
sudo apt-mark showhold # show packages that version upgrade has been put on hold

# Unhold
sudo apt-mark unhold docker-ce docker-ce-cli containerd.io
</source>

[https://docs.docker.com/engine/release-notes/ Newer versions] (>18.09.0) of Docker come with 3 packages:
* <code>containerd.io</code> - daemon to interface with the OS API (in this case, LXC - Linux Containers), essentially decouples Docker from the OS, also provides container services for non-Docker container managers
* <code>docker-ce</code> - Docker daemon, this is the part that does all the management work, requires the other two on Linux
* <code>docker-ce-cli</code> - CLI tools to control the daemon, you can install them on their own if you want to control a remote Docker daemon

Example of how to run [[Jenkins CI|Jenkins docker image]]

== Add a user to docker group ==
Add your user to <tt>docker group</tt> to be able to run docker commands without need of ''sudo'' as the <code>docker.socket</code> is owned by group <code>docker</code>.
<source lang="bash">
sudo usermod -aG docker $(whoami)

# log in to the new docker group (to avoid having to log out / log in again)
newgrp docker
</source>

Reason
<source lang=bash>
[root@piotr]$ ls -al /var/run/docker.sock
srw-rw----. 1 root docker 7 Jan 09:00 docker.sock
</source>

= HTTP proxy =
Configure ''docker'' if you run behind a proxy server. In this example CNTLM proxy runs on the host machine listening on localhost:3128. This example overrides the default docker.service file by adding configuration to the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:
<source lang=bash">
sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"
Environment="HTTP_PROXY=http://172.31.1.1:3128/" #overrides previous entry
Environment="HTTPS_PROXY=http://172.31.1.1:3128/"
# If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable
Environment="NO_PROXY=localhost,127.0.0.1,10.6.96.172,proxy.example.com:80"
</source>

Flush changes:
$ sudo systemctl daemon-reload
Verify that the configuration has been loaded:
$ systemctl show --property=Environment docker
Environment=HTTP_PROXY=<nowiki>http://proxy.example.com:80/</nowiki>
Restart Docker:
$ sudo systemctl restart docker

= Docker create and run, basic options =
<source lang="bash">
# It will create a container but won't start it up
docker container create -it --name="my-container" ubuntu:latest /bin/bash
docekr container start my-container

docker run -it --name="mycentos" centos:latest /bin/bash
# -i :- interactive mode (attach to STDIN) \command to execute when instantiating container
# -t :- attach to the current terminal (sudo TTY)
# -d :- disconnect mode, daemon mode, detached mode, running the task in the background
# -p :- publish to host exposed container port [ host_port(8080):container_exposedPort(80) ]
# --rm :- remove container after command has been executed
# --name="name_your_container"

# -e|--env MYVAR=123 exports/passing variable to the container, echo $MYVAR will have a value 123
# --privileged :- option will allow Docker to perform actions normally restricted,
# like binding a device path to an internal container path.
</source>

= Docker inspect =
== inspect image ==
<source>
docker image inspect centos:6
docker image inspect centos:6 --format '{{.ContainerConfig.Hostname}}' #just a single value
docker image inspect centos:6 --format '{{json .ContainerConfig}}' #json key/value output
docker image inspect centos:6 --format '{{.RepoTags}}' #shows all associated tags with the image
</source>

Using <code>--format</code> is similar to <code>jq</code>
== inspect container ==
Shows current configuration state of a docker container or an image.
<source>
docker inspect <container_name> | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.3",
"IPAddress": "172.17.0.3",
</source>

= Attach/exec to a docker process =
If you are running eg. <tt>/bin/bash</tt> as a command you can get attached to this running docker process. Note that when you exit the container will stop.
<source lang=bash>
docker attach mycentos
</source>

To avoid stopping a container on exit of <code>attach</code> command we can use <code>exec</code> command.
<source lang=bash>
docker exec -it mycentos /bin/bash
</source>

Attaching directly to a running container and then exiting the shell will cause the container to stop. Executing another shell in a running container and then exiting that shell will not stop the underlying container process started on instantiation.

= Entrypoint, CMD, PID1 and [https://github.com/krallin/tini tini] =
== Entrypoint and receiving signals ==
Reciving signals and handling them within containers here Docker it's the same important as for any other application. Remember containers it's a group of processes running on your host so you need to take care of signals send to your applications.

As a principal container management tool eg. <code>docker stop</code> sends a configurable (in Dockerfile) signal to the entrypoint of your application where <code>SIGTERM - 15 - Termination (ANSI)</code> is default.

;ENTRYPOINT syntax

<source>
# exec form, require JSON array; IT SHOULD ALWAYS BE USED
ENTRYPOINT ["/app/bin/your-app", "arg1", "arg2"]

# shell form, it always runs as a subcommand of '/bin/sh -c', thus your application will never see any signal sent to it
ENTRYPOINT "/app/bin/your-app arg1 arg2"
</source>

;ENTRYPOINT is a shell script
If application is started by shell script regular way, your shell spawns your application in a new process and you won’t receive signals from Docker. Therefore we need to tell shell to replace itself with your application using the <code>[https://stackoverflow.com/questions/18351198/what-are-the-uses-of-the-exec-command-in-shell-scripts exec]</code> command, check also <code>[https://en.wikipedia.org/wiki/Exec_(system_call) exec syscall]</code>. Use:
<source lang=bash>
/app/bin/my-app # incorrect, signal won't be received by 'my-app'
exec /app/bin/my-app # correct way
</source>

;ENTRYPOINT exec with a pipe commands causing starting a subshell
If you <code>exec</code> piping will force a command to be run in a subshell with the usual consequence: no signals to an app.
<source lang=bash>
exec /app/bin/your-app | tai64n # here you want to add timestamps by piping through tai64n,
# causing running your command in a subshell
</source>

;Let another program to be PID1 and handle signalling
* tini
* dump-init

<source lang=bash>
ENTRYPOINT ["/tini", "-v", "--", "/app/bin/docker-entrypoint.sh"]
</source>

tini and dumb-init are also able to proxy signals to process groups which technically allows you to pipe your output.However, your pipe target receives that signal at the same time so you can’t log anything on cleanup lest you crave race conditions and SIGPIPEs. So, it's better to avoid logging at termination at all.

;Change signal that will terminate your container process
Listen for SIGTERM or set STOPSIGNAL in your Dockerfile.
<source lang=bash>
vi Dockerfile
STOPSIGNAL SIGINT # this will trigger container termination process if someone press Ctrl^C
</source>

;References:
* [https://hynek.me/articles/docker-signals/ Why Your Dockerized Application Isn’t Receiving Signals]
* [http://smarden.org/runit/ runit] alternative to tini

== Tini ==
It's a tiny but valid init for containers:
* protects you from software that accidentally creates zombie processes
* ensures that the default signal handlers work for the software you run in your Docker image
* does so completely transparently! Docker images that work without Tini will work with Tini without any changes
* Docker 1.13+ has Tini included, to enable Tini, just pass the <code>--init</code> flag to docker run

;Understanding Tini
After spawning your process, Tini will wait for signals and forward those to the child process, and periodically reap zombie processes that may be created within your container. When the "first" child process exits (/your/program in the examples above), Tini exits as well, with the exit code of the child process (so you can check your container's exit code to know whether the child exited successfully).

Add Tini - general dynamicly-linked library (in the 10KB range)
<source lang=bash>
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

# Run your program under Tini
CMD ["/your/program", "-and", "-its", "arguments"]
# or docker run your-image /your/program ...
</source>

Add Tini to Alpine based image
<source lang=bash>
RUN apk add --no-cache tini
# Tini is now available at /sbin/tini
ENTRYPOINT ["/sbin/tini", "--"]
</source>

Existing entrypoint
<source>
ENTRYPOINT ["/tini", "--", "/docker-entrypoint.sh"]
</source>

;References:
*[https://github.com/krallin/tini/issues/8 What is advantage of Tini?]
*[https://ahmet.im/blog/minimal-init-process-for-containers/ Choosing an init process for multi-process containers]

= Mount directory in container =
We can mount host directory into docker container so the content will be available from the container
<source lang="bash">
docker run -it -v /mnt/sdb1:/opt/java pio2pio/java8
# syntax: -v /path/on/host:/path/in/container
</source>

= Build image =
== Dockerfile ==
Each line ''RUN'' creates a container so if possible, we should join lines so it ends up with less layers.
<source lang="bash">
$ wget jkd1.8.0_111.tar.gz
$ cat Dockerfile <<- EOF #'<<-' heredoc with '-' minus ignores <tab> indent
ARG TAGVERSION=6 #only command allowed b4 FROM
FROM ubuntu:${TAGVERSION}
FROM ubuntu:latest #defines base image eg. ubuntu:16.04
LABEL maintainer="myname@gmail.com" #key/value pair added to a metadata of the image

ARG ARG1=value1

ENV ENVIRONMENT="prod"
ENV SHARE /usr/local/share #define env variables with syntax ENV space EnvironmetVariable space Value
ENV JAVA_HOME $SHARE/java

# COPY jkd1.8.0_111.tar.gz /tmp #works only with files, copy a file to container filesystem, here to /tmp
# ADD http://example.com/file.txt
ADD jkd1.8.0_111.tar.gz / #add files into the image root folder, can add also URLs

# SHELL ["executable","params"] #overrides /bin/sh -c for RUN,CMD, etc..

# Executes commands during build process in a new layer E.g., it is often used for installing software packages
RUN mv /jkd1.8.0_111.tar.gz $JAVA_HOME
RUN apt-get update
RUN ["apt-get", "update", "-y"] #in json array format, allows to run a commands but does not require shell executable

VOLUME /mymount_point #this command does not mount anything from a host, just creates a mountpoint

EXPOSE 80 #it doesn't automatically map the port to a hosts

#containers usually don't have system maangement eg. systemctl/service/init.d as designed to run as single process
#entry point becomes main command that start the main proces
ENTRYPOINT apachectl "-DFOREGROUND" #think about it as the MAIN_PURPOSE_OF_CONTAINER command.
# It's always run by default it cannot be overridden

#Single command that will run after the image has been created. Only one per dockerfile, can be overriden.
CMD ["/bin/bash"]

# STOPSIGNAL

EOF
</source>

== Build ==
<source lang="bash">
docker build --tag myrepo/java8 . #-f point to custom Dockerfile name eg. -f Dockerfile2
# myrepo dockerhub username, java8 -image name,
# . directory where is the Dockerfile

docker build -t myrepo/java8 . --pull --no-cache --squash
# --pull regardless a local copy of an image can exist force to download a new image
# --no-cache don't use cache to build, forcing to rebuild all interim containers
# --squash after the build squash all layers into a single layer.

docker images #list images
docker push myrepo/java8 #upload the image to DockerHub repository
</source>

<code>squash</code> is enabled only on docker demon with experimental features enabled.

= Manage containers and images =
== Run a container ==
When you ''run'' a container you will create a new container from a image that has been already build/ is available then put in running state
* -d detached mode, the container will continue to run after the CMD or passed on command exited
* -i interactive mode, allows you to login in /ssh to the container

<source lang="bash">
# docker container run [OPTIONS] IMAGE [COMMAND] [ARG...] # usage
docker container run -it --name mycentos centos:6 /bin/bash
docker run -it pio2pio/java8 #container section command is optional
# -i :- run in interactive mode, then run command /bin/bash
# --rm :- will delete container after run
# --publish | -p 80:8080 :- publish exposed container port 80-> to 8080 on the docker-host
# --publish-all | -P :- publish all exposed container ports to random port >32768
</source>

== List images ==
<source lang="bash">
ctop #top for containers
docker ps -a #list containers
docker image ls #list images
docker images #short form of the command above
docker images --no-trunc
docker images -q #--quiet
docker images --filer "before=centos:6"

# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81
</source>

== Search images in remote repository ==
Search the DockerHub for images,. You may require to do `docker login` first
<source lang="bash">
IMAGE=ubuntu
docker search $IMAGE
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ubuntu Ubuntu is a Debian-based Linux operating sys… 8206 [OK]
dorowu/ubuntu-desktop-lxde-vnc Ubuntu with openssh-server and NoVNC 210 [OK]
rastasheep/ubuntu-sshd Dockerized SSH service, built on top of offi… 167 [OK]

IMAGE=apache
docker search $IMAGE --filter stars=50 # search images that have 50 or more stars
docker search $IMAGE --limit 10 # display top 10 images
</source>

List all available tags
<source lang="bash">
IMAGE=nginx
wget -q https://registry.hub.docker.com/v1/repositories/${IMAGE}/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}' | sort -V
</source>

== Pull images ==
<source lang="bash">
<name>:<tag>
docker pull hello-world:latest # pull latest
docker pull --all hello-world # pull all tags
docker pull --disable-content-trust hello-world # disable verification

docker images --digests #displays sha256: digest of an image

# Dangling images - transitional images
</source>
=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/registries.html#registry_auth from Amazon ECR] ===
;Docker login to ECR service using IAM
<code>docker login</code> does not support native IAM authentication methods. Therefore use a command below that will retrieve, decode, and convert the <code>authorization IAM token</code> into a pre-generated <code>docker login</code> command. Therefore produced login credentials will assume your current IAM User/Role permissions. If your current IAM user can only pull from ECR, after login with <code>docker login</code> you still won't be able to push image to the registry. Example error you may get is <code>not authorized to perform: ecr:InitiateLayerUpload</code>

Login to ECR service, your IAM user requires to have relevant pull/push permissions
<source lang="bash">
eval $(aws ecr get-login --region eu-west-1 --no-include-email)
# aws ecr get-login # generates below docker command with the login token
# docker login -u AWS -p **token** https://$ACCOUNT.dkr.ecr.us-east-1.amazonaws.com # <- output
</source>

Docker login to ECR singular repository, min awscli v1.17
<source lang="bash">
ACCOUNT=111111111111
REPOSITORY=myrepo
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/$REPOSITORY
</source>

=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html push to Amazon ECR] ===
<source lang=bash>
# List images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ansible-aws 2.0.1 b09807c20c96 5 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Tag an image 'b09807c20c96'
docker tag b09807c20c96 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1

# List images, to verify your newly tagged one
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB # <- new tagged image
ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Push an image to ECR
docker push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1
The push refers to repository [111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws]
2c405c66e675: Pushed
...
77cae8ab23bf: Layer already exists
2.0.1: digest: sha256:111111111193969807708e1f6aea2b19a08054f418b07cf64016a6d1111111111 size: 1796
</source>

== Save and import image ==
In course to move a image to another filesystem we can save it into <code>.tar</code>
<source lang="bash">
# Export
docker image save myrepo/centos:v2 > mycentos.v2.tar
tar -tvf mycentos.v2.tar

# Import
docker image import mycentos.v2.tar <new_image_name>

# Load from a stream
docker load < mycentos.v2.tar #or --input mycentos.v2.tar to avoid redirections
</source>

== Export aka commit container into image ==
Let's say we wanto modify stock image centos:6 by installing Apache interactivly, set to autostart then export as an new image. Let's do it!
<source lang="bash">
docker pull centos:6
docker container run -it --name apache-centos6 centos:6
# Interactively do: yum -y update; yum install -y httpd; chkconfig httpd on; exit

# Save container changes - option1
docker commit -m "added httpd daemon" -a "Piotr" b237d65fd197 newcentos:withapache #creates new image from a container's changes
docker commit -m "added httpd daemon" -a "Piotr" <container_name> <repo>/<image>:<tag>
# -a :- author

# Save container changes - option2
docker container export apache-centos6 > apache-centos6.tar
docker image import apache-centos6.tar newcentos:withapache

docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
newcentos withapache ea5215fb46ed 50 seconds ago 272MB

docker image history newcentos:withapache
IMAGE CREATED CREATED BY SIZE COMMENT
ea5215fb46ed 2 minutes ago 272MB Imported from -
</source>

I am unsure what is a difference in creation of images from a container between:
* <code>docker container commit</code>
* <code>docker container export</code> - this seems creates smaller image

== Tag images ==
Tags are used to usually to name Official image with a new name that we are planning to modify. This allows to create a new image, run a new container from a tag, delete the original image without affecting the new image or container started from the new tagged image.
<source lang="bash">
docker image tag #long version
docker tag centos:6 myucentos:v1 #this will create a duplicate of centos:6 named myucentos:v1
</source>

Tagging allows to modify repository name and maanges references to images located on a filesystem.

== History of an image ==
We can display history of layers that created the image by showing interim images build in creation order. It shows only layers created on a local filesystem.
<source lang="bash">
docker image history myrepo/centos:v2
</source>
== Stop and delete all containers ==
<source lang="bash">
docker stop $(docker ps -aq) && docker rm $(docker ps -aq)
</source>

== Delete image ==
<source lang="bash">
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
company-repo 0.1.0 f796d7f843cc About an hour ago 888MB
<none> <none> 04fbac2fdf48 3 hours ago 565MB
ubuntu 16.04 7aa3602ab41e 3 weeks ago 115MB

# Delete image
$ docker rmi company-repo:0.1.0
Untagged: company-repo:0.1.0
Deleted: sha256:e5cca6a080a5c65eacff98e1b17eeb7be02651849b431b46b074899c088bd42a
..
Deleted: sha256:bc7cda232a2319578324aae620c4537938743e46081955c4dd0743a89e9e8183

# Prune image - delete dangling (temp/interim) images.
# These are not associated with end-product image or containers.
docker image prune
docker image prune -a #remove all images not associated with any container
</source>

== Cleaning up space by removing docker objects ==
This applied both to docker container and swarm systems.
<source lang="bash">
docker system df #show disk usage
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 0 131.7MB 131.7MB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B

docker network ls #note all networks below are system created, so won't get removed
NETWORK ID NAME DRIVER SCOPE
452b1c428209 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local

docker system prune #removes objects created by a user only, on the current node only
#add --volumes to remove them as well
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N]

docker system prune -a --volumes #remove all
</source>

== Docker Volumes ==
Docker's 'copy-on-write' philosophy drives both performance and efficiency. It's only the top layer that is writable and it's a delta of underlying layer.

Volumes can be mounted to your container instances from your underlying host systems.

''_data'' volumes, since they are not controlled by the storage driver (since they represent a file/directory on the host filesystem /var/lib/docker), are able to bypass the storage driver. As a result, their contents are not affected when a container is removed.

Volumes are data mounts created on a host in <code>/var/lib/docker/volumes/</code> directory and refereed by name in a Dockerfile.
<source lang="bash">
docker volume ls #list volumes created by VOLUME directive in a Dockerfile
sudo tree /var/lib/docker/volumes/ #list volumes on host-side
docker volume create my-vol-1
docker volume inspect my-vol-1
[
{
"CreatedAt": "2019-01-17T08:47:01Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-vol-1/_data",
"Name": "my-vol-1",
"Options": {},
"Scope": "local"
}
]
</source>

Using volumes with Swarm services
<source lang=bash>
docker container run --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #container
docker service create --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #swarm service
# --mount --volumes|-v is not supported with services, this will replicate volumes across swarm when needed,
# but it will not replicate files

docker exec -it web1 /bin/bash #connect to the container
roor@c123:/ echo "Created when connected to container: volume-web1" > /internal-mount/local.txt; exit

# prove the file is on a host filesystem created volume
user@dockerhost$ cat /var/lib/docker/volumes/my-vol-1/_data/local.txt
</source>

;Host storage mount
Bind mapping is binding host filesystem directory to a container directory. It's not mouting volume that it'd require a mount point and volume on a host.
<source lang=bash>
mkdir /home/user/web1
echo "web1 index" > /home/user/web1/index.html
docker container run -d --name testweb -p 80:80 --mount type=bind,source=/home/user/web1,target=/usr/local/apache2/htdocs httpd
curl http://localhost:80
</source>

Removing service is not going to remove the volume unless you delete the volume itself. It that case will be removed from all swarms.

== Networking ==
=== Container Network Model ===
It's a concept of network implementation that is built on multiple private networks across multiple hosts overlayed and managed by IPAM. Protocol that keeps track and provision addesses.

Main 3 components:
* sandbox - contains the configuration of a container's network stack, incl. management of interfaces, routing and DNS. An implementation of a Sandbox could be a eg. Linux Network Namespace. A Sandbox may contain many endpoints from multiple networks.
* endpoint - joins a Sandbox to a Network. Interfaces, switches, ports, etc and belong to only one network at the time. The Endpoint construct exists so the actual connection to the network can be abstracted away from the application. This helps maintain portability.
* network - a clollection of endpoints that can communicate directly (bridges, VLANs, etc.) and can consist of 1toN endpoints

[[File:Container-network-model.png|none||left|Container Network Model]]

;IPAM (Internet Protocol Address Management)
Managing addesees across multiple hosts on a separate physical networks while providing routing to the underlaying swarm networks externally is ''the IPAM prblem'' for Docker. Depends on the netwok driver choice, IPAM is handled at different layers in the stack. ''Network drivers'' enable IPAM through ''DHCP drivers'' or plugin drivers so the complex implementation that would be normally overlapping addesses is supported.

;References
* [https://success.docker.com/article/networking Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks]

=== Publish exposed container/service ports ===
;Publishing modes
;host: set using <code> --publish mode=host,8080:80</code>, makes ports available only on the undelaying host system not outside the host the service may exist; defits ''routing mesh'' so user is responsible for routing
;ingress: responsible for ''routing mesh'' makes sure all published ports are avaialble on all hosts in the swarm cluster regardless is a service replica running on it or not

<source lang=bash>
# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81

# Publish port
host : container
\ : /
docker container run -d --name web1 --publish 81:80 httpd
# --publish | -p :- publish to host exposed container port
# 81 :- port on a host, can use range eg. 81-85, so based on port availability port will be used
# 80 :- exposed port on a container

ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::81 :::*
LISTEN 0 128 :::22 :::*

docker container run -d --name web1 --publish-all 81:80 httpd
# --publish-all | -P publish all cotainer exposed ports to random ports above >32768
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c63efe9cbb94 httpd "httpd-foreground" 2 sec.. Up 1 s 80/tcp testweb #port exposed but not published
cb0711134eb5 httpd "httpd-foreground" 4 sec.. Up 2 s 0.0.0.0:32769->80/tcp testweb1 #port exposed and published to host:32769
</source>

=== Network drivers ===
Default network for a single host docker-host is ''bridge'' network.

;List of Native (part of Docker Engine) Network Drivers:
;bridge: default on stand-alone hosts, it's private network internal to the host system, all containers on this host using Bridge network can communicate, external access is granted by port exposure or static-routes added with teh host as the gateway for that network
;none: used when a container does not need any networkng, still can be accessed from the host using <code>docker attach [containerID]</code> or <code>docker exec -it [containerID]</code> commands
;host: aka ''Host Only Networking'', only accessable via underlaying host, access to services can be provided by exposing ports to the host system
;overlay: swarm scope driver, allows communication to all Docker Daemons in a cluster, self-extending if needed, maanged by Swarm manager, it's default mode of Swarm communication
;ingress: extended network across all nodes in the cluster; special overlay network that load balances netowrk traffic amongst a given service's working nodes; maintains a list of all IP addresses from nodes that participate in that service (using the IPVS module) and when a request comes in, routes to one of them for the indicated service; provides ''routing mesh' that allows services to be exposed to the external network without having replica running on every node in the Swarm
;docker gateway bridge: special bridge network that allows overlay networks (incl. ingress) access an individual DOcker daemon's physical network; every container run within a service is connected to the local Docerk daemon's host network; automatically created when Docker is initialised or joined by <code>docker swarm init</code> or <code>docker join</code> commands.

;Docker interfaces
* <code>docker0</code> - adapter is installed by default during Docker setup and will be assigned an address range that will determine the local host IPs available to the containers running on it

;Create bridge network
<source lang=bash>
docker network ls #default networks list
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm #'ingress' special network 1 per cluster
815a9c2c4005 none null local

docker network inspect bridge #bridge is a default network containers are deployed to

docker container run -d web1 -p 8080:80 httpd #expose container port :80 -> :8080 on the docker-home
docker container inspect web1 | grep IPAdd
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1 #get container ip
curl http://$(IPAddr)
</source>

;Create bridge network
<source lang=bash>
docker network create --driver=bridge --subnet=192.168.1.0/24 --opt "com.docker.network.driver.mtu"=1501 deviceeth0

docker network ls
docker network inspect deviceeth0
</source>

;Create overlay network
<source lang=bash>
docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
docker network ls
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local
2x6bq1czzdc1 overlay0 overlay swarm
</source>

;Inspect network
<source lang=json>
docker network inspect overlay0
[
{
"Name": "overlay0",
"Id": "2x6bq1czzdc102sl6ge7gpm3w",
"Created": "2019-01-19T11:24:02.146339562Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.1.0/24",
"Gateway": "192.168.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": null
}
]
</source>

;Inspect container network
<source lang=bash>
docker container inspect testweb --format {{.HostConfig.NetworkMode}}
overlay0
docker container inspect testweb --format {{.NetworkSettings.Networks.dev_bridge.IPAddress}}
192.168.1.3
</source>

Connect/disconnect from a network can be done when a container is running. Connect won't disconnect from a current network.
<source lang=bash>
docker network connect --ip=192.168.1.10 deviceeth0 web1
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1
docker container inspect --format="{{.NetworkSettings.Networks.deviceeth0.IPAddress}}" web1
curl http://$(IPAddr)
docker network disconnect deviceeth0 web1
</source>

=== Overlay network in Swarm cluster ===
Overlay network can be created/removed/updated like any other docker objects. It allows inter-service(containers) communication, where <code>--gateway</code> ip address is used to reach to outside eg. Inernet or the host network. When creating the <code>overlay</code> network on the manager host it will get recreated on worker nodes only when is referenced by any service that is using it. See below.
<source lang="bash">
swarm-mgr$ docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
swarm-mgr$ docker service create --name web1 -p 8080:80 --network=overlay0 --replicas 2 httpd
uvxymzdkcfwvs2oznbnk7nv03
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]

swarm-wkr$ docker network ls
NETWORK ID NAME DRIVER SCOPE
ba175ebd2a6f bridge bridge local
a5848f607d8c docker_gwbridge bridge local
fccfb9c1fdc3 host host local
t8jxy5vsy5on ingress overlay swarm
127b10783faa none null local
2x6bq1czzdc1 overlay0 overlay swarm

# remove network, only affected newly created servces not the running onces
swarm-mgr$ docker service update --network-rm=overlay0 web1
</source>

=== DNS ===
<source lang="bash">
docker container run -d --name testweb1 -P --dns=8.8.8.8 \
--dns=8.8.4.4 \
--dns-search "mydomain.local" \
httpd
# -P :- publish-all exposed ports to random port >32768

docker container exec -it testweb1 /bin/bash -c 'cat /etc/resolv.conf'
search us-east-2.compute.internal
nameserver 8.8.8.8
nameserver 8.8.4.4

# System wide settings, requires docker.service restart
cat > /etc/docker/daemon.json <<EOF
{
"dns": ["8.8.8.8", "8.8.4.4"]
}
EOF
sudo systemctl restart docker.service #required
</source>

= Examples =
== Lint - best practices ==
<source lang=bash>
$ docker run --rm -i hadolint/hadolint < Dockerfile
/dev/stdin:9:16 unexpected newline expecting "\ ", '=', a space followed by the value for the variable 'MAC_ADDRESS', or at least one space
</source>

== Default project ==
As good practice all Docker files should be source controlled. The basic self explanatory structure can looks like below, and skeleton be created with a commend below:
<source lang="bash">
mkdir APROJECT && d=$_; touch $d/{build.sh,run.sh,Dockerfile,README.md,VERSION};mkdir $d/assets; touch $_/{entrypoint.sh,install.sh}

└── APROJECT
├── assets
│ ├── entrypoint.sh
│ └── install.sh
├── build.sh
├── Dockerfile
├── README.md
└── VERSION
</source>

== Dockerfile ==
<code>Dockerfile</code> it is simply a build file.
=== Semantics ===
;<code>entrypoint</code>: Container config: what to start when this image is ran.

;<code>entrypoint</code> and <code>cmd</code>: Docker allows you to define an Entrypoint and Cmd which you can mix and match in a Dockerfile. Entrypoint is the executable, and Cmd are the arguments passed to the Entrypoint. The Dockerfile schema is quite lenient and allows users to set Cmd without Entrypoint, which means that the first argument in Cmd will be the executable to run.

=== User management ===
<source>
RUN addgroup --gid 1001 jenkins -q
RUN adduser --gid 1001 --home /tank --disabled-password --gecos '' --uid 1001 jenkins
# --gid add user to group 1001
# --gecos parameter is used to set the additional information. In this case it is just empty.
# --disabled-password it's like --disabled-login, but logins are still possible (for example using SSH RSA keys) but not using password authentication
USER jenkins:jenkins #sets user for next RUN, CMD and ENTRYPOINT command
WORKDIR /tank #changes cwd for next RUN, CMD, ENTRYPOINT, COPY and ADD
</source>

=== Multiple stage build ===
Introduced in Docker 17.06, allows to use multiple <code>FROM</code> statements allowing for multi stage builds.
<source>
FROM microsoft/aspnetcore-build AS build-env
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o output

# build runtime image
FROM microsoft/aspnetcore
WORKDIR /app
COPY --from=build-env /app/output . #multi stage: copy files from previous container [as build-env]
ENTRYPOINT ["dotnet", "LetsKube.dll"]
</source>

= Squash an image =
Docker uses <code>Union</code> filesystem that allows multiple volumes (layers) to share common and override changes by applying them on top layer.
There is no official way to ''flatten'' layers to a single storage layer or minimize an image size (2017). Below it's just practical approach.
# Start container from an image
# Export a container to <code>.tar</code> with all it's file systems
# Import container with new image name

Once the process completes and original image gets deleted the new image <code>docker image history</code> command will show only one layer. Often the image will be smaller.
<source lang=bash>
# run a container from an image
docker run myweb:v3
# export container to .tar
docker export <contr_name> > myweb.v3.tar
docker save <image id> > image.tar #not verified command
docker import myweb.v3.tar myweb:v4
docker load myweb.v3.tar #not verified command
</source>

;Resources
*[https://github.com/jwilder/docker-squash docker-squash] GitHub

= Gracefully stop / kill a container =
''all below are only notes''

Trap ctrl_c then kill/rm container.
*--init
*--sig-proxy this only works when --tty=false but by default is true

= Proxy =
If you are behind corporate proxy, you should use Docker client <code>~/.docker/config.json</code> config file. It requires Docker
17.07 minimum version.
<source>
{
"proxies":
{
"default":
{
"httpProxy": "http://10.0.0.1:3128",
"httpsProxy": "http://10.0.0.1:3128",
"noProxy": "localhost,127.0.0.1,*.test.example.com,.example2.com"
}
}
}
</source>
More you can find [https://docs.docker.com/network/proxy/#configure-the-docker-client here]

== Insecure proxy ==
These can be added to different places, the order is based on latest practices and versioning
;docker-ce 18.6
<source lang="json">
{
"insecure-registries" : [ "localhost:443","10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
}
</source>
<source lang="bash">
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl show docker | grep Env
docker info #check Insecure Registries
</source>

;Using environment file, prior version 18
<source lang="bash">
$ sudo vi /etc/default/docker
DOCKER_HOME='--graph=/tank/docker'
DOCKER_GROUP='--group=docker'
DOCKER_LOG_DRIVER='--log-driver=json-file'
DOCKER_STORAGE_DRIVER='--storage-driver=btrfs'
DOCKER_ICC='--icc=false'
DOCKER_IPMASQ='--ip-masq=true'
DOCKER_IPTABLES='--iptables=true'
DOCKER_IPFORWARD='--ip-forward=true'
DOCKER_ADDRESSES='--host=unix:///var/run/docker.sock'
DOCKER_INSECURE_REGISTRIES='--insecure-registry 10.0.0.0/8 --insecure-registry 172.16.0.0/12 --insecure-registry 192.168.0.0/16'
DOCKER_OPTS="${DOCKER_HOME} ${DOCKER_GROUP} ${DOCKER_LOG_DRIVER} ${DOCKER_STORAGE_DRIVER} ${DOCKER_ICC} ${DOCKER_IPMASQ} ${DOCKER_IPTABLES} ${DOCKER_IPFORWARD} ${DOCKER_ADDRESSES} ${DOCKER_INSECURE_REGISTRIES}"

$ sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/dockerd $DOCKER_HOME $DOCKER_GROUP $DOCKER_LOG_DRIVER $DOCKER_STORAGE_DRIVER $DOCKER_ICC $DOCKER_IPMASQ $DOCKER_IPTABLES $DOCKER_IPFORWARD $DOCKER_ADDRESSES $DOCKER_INSECURE_REGISTRIES

$ sudo vi /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
EnvironmentFile=-/etc/default/docker
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
</source>

== Run docker without sudo ==
Adding a user to docker group should be sufficient. However on apparmor, SELinux or a filesystem with ACL enabled additional permissions might be required in respect to access a <tt>socket file</tt>.
<source>
$ ll /var/run/docker.sock
srw-rw---- 1 root docker 0 Sep 6 12:31 /var/run/docker.sock=
# ACL
$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
group::rw-
other::---

# Grant ACL to jenkns user
$ sudo setfacl -m user:username:rw /var/run/docker.sock

$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
user:jenkins:rw-
group::rw-
mask::rw-
other::---
</source>
;References
* [https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo how-can-i-use-docker-without-sudo]

== References ==
*[https://www.weave.works/blog/my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies/ my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies]
*[https://github.com/moby/moby/pull/12228 PID1 in container aka tinit]
*[https://container-solutions.com/understanding-volumes-docker/ understanding-volumes-docker]

= Docker Enterprise Edition =
*[https://success.docker.com/article/compatibility-matrix Compatibility Matrix]
Components:
* Docker daemon (fka "Engine")
* Docker Trusted Registry (DTR)
* Docker Universal Control Plane (UCP)

= Docker Swarm =
== Swarm - sizing ==
;Universal Control Plane (UCP)
This is for only Enterpsise Edition
* ports managers, workers in/out

Hardware requirments:
* 8gb RAM for managers or DTR Docker Trsuted Registry
* 4gb RAM for workers
* 3gb free space

Performance Consideration (Timing)
<source>
Component Timeout(ms) Configurable
Raft consensus between manager nodes 3000 no
Gossip protocol for overlay networking 5000 no
etcd 500 yes
RethinkDB 10000 no
Stand-alone swarm 90000 no
</source>

Compatibility Docker EE
* Docker Engine 17.06+
* DTR 2.3+
* UCP 2.2+
== Swarm with single host manager ==
<source lang=bash>
# Initialise Swarm
docker swarm init --advertise-addr 172.31.16.10 #Iyou get SWMTKN-token
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

# Join tokens
docker swarm join-token manager #display manager join-token, run on manager
docker swarm join-token worker #display worker join-token, run on manager

# Join worker, run new-worker-node
# -> swarm cluster id <-> this part is mgr/wkr <- -> mgr node <-
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377

# Join another manager, run on new-manager-node
docker swarm join-token manager #run on primary manager if you wish add another manager
# in output you get a token. You notice that 1st part up to dash identifies Swarm cluster and the other part is role id.

# join to swarm (cluster), token will identify a role in the cluster manager or worker
docker swarm join --token SWMTKN-xxxx
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
This node joined a swarm as a worker.
</source>

Check Swarm status
<source lang=bash>
docker node ls
[cloud_user@ip-172-31-16-10 swarm-manager]$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * ip-172-31-16-10.mylabserver.com Ready Active Leader 18.09.0
vlw7te728z7bvd7ulb3hn08am ip-172-31-16-94.mylabserver.com Ready Active 18.09.0

docker system info | grep -A 7 Swarm
Swarm: active
NodeID: 641bfndn49b1i1dj17s8cirgw
Is Manager: true
ClusterID: 4jqxdmfd0w5pc4if4fskgd5nq
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
</source>

;Troubleshooting
<source lang=bash>
sudo systemctl disable firewalld && sudo systemctl stop firewalld # CentOS
sudo -i; printf "\n10.0.0.11 mgr01\n10.0.0.12 node01\n" >> /etc/hosts # Add nodes to hosts file; exit
</source>
== Swarm cluster ==
<source lang=bash>
docker node update -availability drain [node] #drain services for Manager Only nodes
docker service update --force [service_name] #force re-balance services across cluster

docker swarm leave #node leaves a cluster
</source>

== Locking / unlocking swarm cluster ==
Logs used by Swarm manager are encrypted on disk. Access to nodes gives access to keys that encrypt them. It further protects cluster as requires a unlocking key when restarting manager/nodes.
<source lang=bash>
docker swarm init --auto-lock=true #initialize with
docker swarm update --auto-lock=true #update current swarm
# both will produce unlock token STKxxx
docker swarm unlock #it'll ask for the unlock token
docker swarm update --auto-lock=false #disable key locking
</source>

If you have access to a manager you can always get unlock key using:
<source lang=bash>
docker swarm unlock-key
</source>

Key management
<source lang=bash>
docker swarm unlock-key --rotate #could be in a cron
</source>
== Backup and restore swarm cluster ==
This priocess describes how to backup whole cluster configuration so can be restored on a new set of servers.

Create docker apps running across swarm
<source lang=bash>
docker service create --name bkweb --publish 80:80 --replicas 2 httpd
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
q9jki3n2hffm bkweb replicated 2/2 httpd:latest *:80->80/tcp

$ docker service ps bkweb #note containers run on 2 different nodes
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE
j964jm1lq3q5 bkweb.1 httpd:latest server2c.mylabserver.com Running Running about a minute ago
jpjx3mk7hhm0 bkweb.2 httpd:latest server1c.mylabserver.com Running Running about a minute ago
</source>

;Backup state files
<source lang=bash>
sudo -i
cd /var/lib/docker/swarm
cat docker-state.json #contains info about managers, workers, certificates, etc..
cat state.json
sudo systemctl stop docker.service

# Backup swarm cluster, this file can be then used to recover whole swarm cluster on another set of servers
sudo tar -czvf swarm.tar.gz /var/lib/docker/swarm/

#the running docker containers should be brought up as they were before stopping the service
systemctl start docker
</source>

Recover using swarm.tar backup
<source lang=bash>
# scp swarm.tar to recovery node - what it'd be a node with just installed docker
sudo rm -rf /var/lib/docker/swarm
sudo systemctl stop docker

# Option1 untar directly
sudo tar -xzvf swarm.tar.gz -C /var/lib/docker/swarm

# Option2 copy recursivly, -f override if a file exists
tar -xzvf swarm.tar.gz; cd /var/lib/docker
cp -rf swarm/ /var/lib/docker/

sudo systemctl start docker
docker swarm init --force-new-cluster # produces the exactly same token
# you should join all required nodes to new manager ip
# scale services down to 1, then scale up so get distributed to other nodes
</source>

== Run containers as a services ==
Docker container has a number limitation therefore running as a service where Cluster Manager: Swarm or Kubernetes manages networking, access, loadbalancing etc.. is a way to scale with ease. The service is using eg. mesh routing to deal with access to containers.

Swarm nodes setup 1 manager and 2 workers
<source lang=bash>
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.0
</source>

Create and run a service
<source lang=bash>
docekr pull httpd
docker service create --name serviceweb --publish 80:80 httpd
# --publish|-p -expose a port on all containers in the running cluster

docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vt0ftkifbd84 serviceweb replicated 1/1 httpd:latest *:80->80/tcp

docker service ps serviceweb #show nodes that a container is running on, here on mgr-1 node
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
e6rx3tzgp1e5 serviceweb.1 httpd:latest swarm-mgr-1.example.com Running Running about
</source>

When running as a service even if a container runs on a single node (replica=1) the container can be accessed from any of swarm nodes. It's because service exposed port has been exposed to extended mesh private network that the container is running on.
<source lang=bash>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-mgr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-2.example.com
<html><body><h1>It works!</h1></body></html>
</source>

Service update, can be done to limits, volumes, env-variables and more...
<source lang=bash>
docker service scale devweb=3 #or
docker service update --replicas 3 serviceweb #--detach=false shows visual progress in older versions, default in v18.06
serviceweb
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged

# Limits(soft limit) and reservations(hard limit), this causes to start new services(containers)
docker service update --limit-cpu=.5 --reserve-cpu=.75 --limit-memory=128m --reserve-memory=256m serviceweb
</source>

== Templating service names ==
This allows to control eg. hostname in a cluster. Useful for big clusters to easier identify services where they run from hostname.
<source lang=bash>
docker service create --name web --hostname"{{.Node.ID}}-{{.Service.Name}}" httpd
docker service ps --no-trunc web
docker inspect --format="{{}.Config.Hostname}" web.1.ab10_serviceID_cd
aa_nodeID_bb-web
</source>
== Node lables for task/service placement ==
<source lang=bash>
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.1

docker node inspect 641bfndn49b1i1dj17s8cirgw --pretty
ID: 641bfndn49b1i1dj17s8cirgw
Hostname: swarm-mgr-1.example.com
Joined at: 2019-01-08 12:16:56.277717163 +0000 utc
Status:
State: Ready
Availability: Active
Address: 172.31.10.10
Manager Status:
Address: 172.31.10.10:2377
Raft Status: Reachable
Leader: Yes
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 2
Memory: 3.699GiB
Plugins:
Log: awslogs, fluentd, gcplogs, gelf, journald, json-file, local, logentries, splunk, syslog
Network: bridge, host, macvlan, null, overlay
Volume: local
Engine Version: 18.09.1
TLS Info:
TrustRoot:
-----BEGIN CERTIFICATE-----
MIIBajCCARCgAwIBAgIUKXz3wtc8OA8uzTo1pO86ko+PB+EwCgYIKoZIzj0EAwIw
..
-----END CERTIFICATE-----
Issuer Subject: MBMxETAPBgNVBAMTCHN3YX.....h
Issuer Public Key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEy......==
</source>

Apply label to a node
<source lang=bash>
docker node update --label-add node-env=testnode r8h7xmevue9v2mgysmld59py2
docker node inspect r8h7xmevue9v2mgysmld59py2 --pretty | grep -B1 -A2 Labels
ID: r8h7xmevue9v2mgysmld59py2
Labels:
- node-env=testnode
Hostname: swarm-wkr-1.example.com
</source>

How to use it. Run a service with <code>--constraint</code> option that pins services to run on a node meeting given criteria. In our case to run on a node where <code>node.labels.node-env == testnode</code>. Note that all replicas are running on the same node unlike they'd be distributed across the cluster.
<source lang=bash>
docker service create --name constraints -p 80:80 --constraint 'node.labels.node-env == testnode' --replicas 3 httpd #node.role, node.id, node.hostname
zrk15vfdaitc1rvw9wqh2s0ot
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
[cloud_user@mrpiotrpawlak1c ~]$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
zrk15vfdaitc constraints replicated 3/3 httpd:latest *:80->80/tcp

[user@swarm-wkr-2 ~]$ docker service ps constraints
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
y5z4mt99uzpo constraints.1 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
zqbn4ips969q constraints.2 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
vnb10jcs2915 constraints.3 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
</source>

== Scaling services ==
These commands be issued on a manager node
<source lang=bash>
docker pull docker nginx
docker service create --name web --publish 80:80 httpd
docker service ps web #there is only 1 replica
docker service update --replicas 3 web #update to 3 replicas
docker service create --name nginx --publish 5901:80 nginx
elinks http://swarm-mgr-1.com:5901 #nginx website will be presented

# scale is equivalent to update --replicas command for a single or multiple services
docker service scale web=3 nginx=3
docker service ls
</source>

== Replicated services vs global services ==
;Global Replicated: mode runs at least one copy of a service on each swarm node, even if you join another node the service will coverage there as well. In global mode you cannot use <code>update --replicats</code> or <code>scale</code> commands. It is not possible to update the mode type.
;Replicated mode: allows for grater control and flexibility of running number of services.
<source lang=bash>
# creates a single service running across whole cluster in replicated mode
docker service create --name web --publish 80:80 httpd

# run in a global node
docker service create --name web --publish 5901:80 --mode global httpd
docker service ls #note distinct mode names: global and replicated
</source>

= Docker compose and deploy to Swarm =
Install
<source lang=bash>
sudo yum install epel
sudo yum install pip
sudo pip install --upgrade pip
# install docker CE or EE to avoid Python libs conflits
sudo pip install docker-compose

# Troubleshooting
## Err: Cannot uninstall 'requests'. It is a distutils installed project...
pip install --ignore-installed requests
</source>

Dockerfile
<source lang=bash>
cat >Dockerfile <<EOF
FROM centos:latest
RUN yum install -y httpd
RUN echo "Website1" >> /var/www/html/index.html
EXPOSE 80
ENTRYPOINT apachectl -DFOREGROUND
EOF
</source>

Docker compose file
<source lang=bash>
cat >docker-compose.yml <<EOF
version: '3'
services:
apiweb1:
image: httpd_1:v1
build: .
ports:
- "81:80"
apiweb2:
image: httpd_1:v1
ports:
- "82:80"
load-balancer:
image: nginx:latest
ports:
- "80:80"
EOF
</source>

Run docker compose, on the current node only
<source lang=bash>
docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating compose_apiweb2_1 ... done
Creating compose_apiweb1_1 ... done
Creating compose_load-balancer_1 ... done

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
14f8b6b10c2d nginx:latest "nginx -g 'daemon of…" 2 minutesUp 2 min 0.0.0.0:80->80/tcp compose_load-balancer_1
e9b5b37fe4e5 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:81->80/tcp compose_apiweb1_1
28ee22a8eae0 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:82->80/tcp compose_apiweb2_1

# Verify
curl http://localhost:81
curl http://localhost:82
curl http://localhost:80 #nginx

# Prep before deploying docker-compose to Swarm. Also images needs to be build before hand.
# Docker stack does not support building images
docker-compose down --volumes #save everything to storage volumes
Stopping compose_load-balancer_1 ... done
Stopping compose_apiweb1_1 ... done
Stopping compose_apiweb2_1 ... done
Removing compose_load-balancer_1 ... done
Removing compose_apiweb1_1 ... done
Removing compose_apiweb2_1 ... done
Removing network compose_default
</source>

;Deploy compose to Swarm
<source lang=bash>
docker stack deploy --compose-file docker-compose.yml customcompose-stack #customcompose-stack is a prefix for service name
Ignoring unsupported options: build
Creating network customcompose-stack_default
Creating service customcompose-stack_apiweb1
Creating service customcompose-stack_apiweb2
Creating service customcompose-stack_load-balancer

docker stack services customcompose-stack #or
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
k7wwkncov49p customcompose-stack_apiweb1 replicated 0/1 httpd_1:v1 *:81->80/tcp
nl0j5folpmha customcompose-stack_apiweb2 replicated 0/1 httpd_1:v1 *:82->80/tcp
x6p14gmpjyra customcompose-stack_load-balancer replicated 1/1 nginx:latest *:80->80/tcp

docker stack rm customcompose-stack #remove stack

</source>

= Selecting a Storage Driver =
Go to Docker version matrix to verify what drivers are supported on your platform. Changing storage driver is destructive and you loose all containers volumes. Therefore you need to export/backup then re-import after the storage driver change.

;CentOS
Device mapper is officialy supported on CentOS. It can be used on a disc as blockstorage it uses loopback adapter to provide that. Or can be blockstorage devive allowing Docker to mamange it.

<source lang=bash>
docker info --format '{{json .Driver}}'
docker info -f '{{json .}}' | jq .Driver
docker info | grep Storage

sudo touch /etc/docker/daemon.json
sudo vi /etc/docker/daemon.json #additional options are available
{
"storage-driver":"devicemapper"
}
</source>

Preserving any current images, requires export/backup and re-import after the storage driver change.
<source lang=bash>
docker images
sudo systemctl docker restart
ls -l /var/lib/docker/devicemapper #new location to storing images
</source>

Note, in <code>/var/lib/docker</code> new directory <code>devicemapper</code> has been created to store images from now on.

;Update 2019 - Docker Engine 18.09.1
<source>
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
</source>

= Selecting a logginig driver =
Available list of [https://docs.docker.com/config/containers/logging/configure/#supported-logging-driverslogging drivers] can be seen on Docker documentation page. Most popular are:
*none - No logs are available for the container and docker logs does not return any output.
*json-file - (default) the logs are formatted as JSON. The default logging driver for Docker.
*syslog - Writes logging messages to the syslog facility. The syslog daemon must be running on the host machine.
*journald - Writes log messages to journald. The journald daemon must be running on the host machine.
*fluentd - Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host machine.
*awslogs - Writes log messages to Amazon CloudWatch Logs.
*splunk - Writes log messages to splunk using the HTTP Event Collector.
*etwlogs - (Windows) Writes log messages as Event Tracing for Windows (ETW) events

<source lang=bash>
docker info | grep logging
docker container run -d --name <webjson> --logg-driver json-file httpd #per docker container setup
docker logs <testjson>

docker container run -d --name <web> httpd #start new container
docker logs -f _testweb_ #display standard-out logs
docker service log -f <web> #for swarm all container replicas logs
</source>

Enable syslog logginig driver
<source lang=bash>
sudo vi /etc/rsyslog.conf
#uncomment below
$ModLoad imudp
$UDPServerRun 514

sudo systemctl start rsyslog
</source>

Change logging driver. Then standard output won't be available after the change.
<source lang=bash>
{
"log-driver": "syslog",
"log-opts": {
"syslog-address": "udp://172.31.10.1"
}
}

sudo systemctl restart docker
docker info | grep logging
tail -f /var/log/messages #this will show all logging eg. access logs for httpd server
</source>
== Docker daemon logs ==
System level logs
<source lang=bash>
# CentOS
/var/messages | grep -i docker

# Ubuntu
sudo journalctl -u docker.service --no-hostname
sudo journalctl -u docker -o json | jq -cMr '.MESSAGE'
sudo journalctl -u docker -o json | jq -cMr 'select(has("CONTAINER_ID") | not) | .MESSAGE'
/var/log/syslog | grep -i docker
</source>

;Docker container or service logs
<source lang=bash>
docker container logs [OPTIONS] containerID #single container logs
docker service logs [OPTIONS] service|task #agregate logs across all cluster deployed container replicas
</source>

= Container life-cycle policies - eg. autostart =
<source lang=bash>
docker container run -d --name web --restart <on-failure|unless-stopped|no|none(default)|always> httpd
# --restart -restart on crash or exit 1 or service or system reboot
</source>
Definitions:
* always - it will restart container always, even if stopped manually, restarting docker-deamon will start container
* unless-stopped - it will restart container always unless stopped manually by <code>docker container stop</code>
* on-failure - restart if container exits with non-zero exit code

= Universal Control Plance - UCP =
It's an application what allow to see all operational details for Swarm cluster when using Docker EE editin. 30 days trial is available.

;Communication between Docker Engine, UCP and DTR (Docker Trusted Registry)
* over TCP/UDP - depends on a port, and whether a response is required, or if a message is a notification
* IPC - interprocess communication (intra-host), services on the same node
* API - over TCP, uses API directlyto query or update components in a cluster

;References
* [https://docs.docker.com/ee/ucp/ucp-architecture/ UCP architecture]

== Install/uninstall UCP <code>image: docker/ucp</code> ==
OS support:
* UCP 2.2.11 is supported running on RHEL 7.5 and Ubuntu 18.04

For labs purpose, we can use eg. <code>ucp.example.com</code> the domain <code>example.com</code> is included in UCP and DTR wildcard self-signed certificate.

Install on a manager node
<source lang=bash>
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_MGR_NODE_IP=172.31.101.248

docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.15 \
install --host-address=$UCP_MGR_NODE_IP --interactive --debug

# --rm :- because this container will be only transitinal container
# -it :- because installation we want interactive
# -v :- link the container with a file on a host
# --san :- add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com)
# --host-address :- IP address or network interface name to advertise to other nodes
# docker/ucp:2.2.11 :- image version
# --dns :- custom DNS servers for the UCP containers
# --dns-search :- ustom DNS search domains for the UCP containers
# --admin-username "$UCP_USERNAME" --admin-password "$UCP_PASSWORD" #seems these are not supported, although are in a guide
</source>

If not provided you will be asked for:
* Admin password during the process
* You may enter additional aliases (SANs) now or press enter to proceed with the above list:
** Additinall aliases: ucp ucp.example.com
DEBU[0062] User entered: ucp ucp.ciscolinux.co.uk
DEBU[0062] Hostnames: [host1c.mylabserver.com 127.0.0.1 172.17.0.1 172.31.101.248 ucp ucp.ciscolinux.co.uk]

You may want to add DNS entries in <code>/etc/hosts</code> for
* ''ucp'' or ''ucp.example.com'' pointing to manager public ip
* ''dtr'' or ''dtr.example.com'' pointing a worker node public IPs.

;Verify
* connect to https://ucp.example.com:443.
* <code>docker ps</code> should see a number of containers running now, they need to see each other therefore we used <code>hosts</code> entries to allow this.

;Uninstall
<source lang=bash>
docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp uninstall-ucp --interactive

INFO[0000] Your engine version 18.09.1, build 4c52b90 (4.15.0-1031-aws) is compatible with UCP 3.1.2 (b822777)
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: t0ltwwcw5tdbtjo2fxlzmj8p4
Do you want to proceed with the uninstall? (y/n): y
INFO[0000] Uninstalling UCP on each node...
INFO[0031] UCP has been removed from this cluster successfully.
INFO[0033] Removing UCP Services
</source>

== Install DTR Docker Trusted Repository <code>image: docker/dtr</code> ==
It's recommended for single core systems to wait 5 minutes after UCP deployment to relese more cpu cycle. You can see the load may peaking up at 1.0 using <code>w</code> command.

Connect to UCP service https://ucp.example.com, login with creds created. Uload a license.lic file.
Go to Admin Settings > Docker Trusted Registry > Pick one of UCP Nodes [worker]
You may disable TLS verification on self-signed certificate

Run a given command on a node you want to install DTR. <code>UCP_NODE</code> in lab environment can cause a few issues. For a convinience to avoid avoid port conflict :80,:443 use different node that UCP is instaled. Eg. dns ''user2c.mylabserver.com'' or private IP.
<source lang=bash>
export UCP_NODE=wkr-172.31.107.250 #for convinince, to avoid port conflict :80,:443 use worker IP
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_URL=https://ucp.example.com:443 #avoid using example.com to avoid SSL name validation issues
docker pull docker/dtr

# Optional. Download UCP public certificate
curl -k https://ucp.ciscolinux.co.uk/ca > ucp-ca.pem

docker container run -it --rm docker/dtr install \
--ucp-node $UCP_NODE --ucp-url $UCP_URL --debug \
--ucp-username $UCP_USERNAME --ucp-password $UCP_PASSWORD \
--ucp-insecure-tls # --ucp-ca "$(cat ucp-ca.pem)"

# --ucp-node :- hostname/IP of the UCP node (any node managed by UCP) to deploy DTR. Random by default
# --ucp-url :- the UCP URL including domain and port.
</source>

It will ask for if not specified:
* ucp-password: you know it from UCP installation step

Sygnificiant installation logs
<source lang=bash>
..
INFO[0006] Only one available UCP node detected. Picking UCP node 'user2c.labserver.com'
..
INFO[0006] verifying [80 443] ports on user2c.labserver.com
..
INFO[0000] Using default overlay subnet: 10.1.0.0/24
INFO[0000] Creating network: dtr-ol
INFO[0000] Connecting to network: dtr-ol
..
INFO[0008] Generated TLS certificate. dnsNames="[*.com *.*.com example.com *.dtr *.*.dtr]" domains="[*.com *.*.com 172.17.0.1 example.com *.dtr *.*.dtr]" ipAddresses="[172.17.0.1]"
..
INFO[0073] You can use flag '--existing-replica-id 10e168476b49' when joining other replicas to your Docker Trusted Registry Cluster
</source>

;Verify by logging in to https://dtr.example.com
DTR installation process above has also installed a number of containers on maanger/worker nodes named <code>ucp-agent</code> and number of containers on dedicated DTR node.
You can verify DTR by logging to https://dtr.example.com with UCP credentials <code>ucp-admin</code> and the same password if you haven't changed any commands above. Then you should be presented with registry.docker.io like theme. Any images stored there will be trusted from a perspective of our organisation.

;Verify by going to UCP https://ucp.example.com, admin settings > Docker Trusted Registry
[[File:Ucp-dtr-in-admin.png|none|400px|left|Ucp-dtr-in-admin]]

== Backup UCP and DTR configuration ==
This is build into UCP. The process is to start a special container to export UCP configuration to tar file. This can be run as <code>cron</code> job.

<source lang=bash>
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp backup > backup.tar
# --rm it's transitional container
# -i run interactivly

# At first run it will error with --id m79xxxxxxxxx, asking to re-run teh command with this id.

# Restore command
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp restore --id m79xxx < backup.tar
</source>

;DTR
Durign a backup DTR will not be available.
<source lang=bash>
docker container run --log-driver non --rm docker/dtr backup --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> > dtr-backup.tar

# will you be asked for:
# Choose a replica to back up from: enter

# Restore command
docker container run --log-driver non --rm docker/dtr restore --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> < dtr-backup.tar
</source>
== UCP RBAC ==
The main concept is:
* administrators can make changes to the UCP swarm/kubernetes, User Management, Orgainisation, Team and Roles
* users - range of access from Full Control of resources to no access

[[File:Ucp-rbac.png|500px|none|left|Ucp-rbac]]

Note that only Scheduler role allows access to Node to view nodes. Plus schedule workloads of course.

= UCP Client bundle =
UCP client bundle allows to export a bundle containing a certificate and environment settings that will poind docker-client to UCP in order to use a cluster, create images and services.

;Download bundle
# Create a user with priviliges that yuo wish docker-client to run as
# Download a client budle from User Profile > Client bundle > + New Client Bundle
# File <code>ucp-bundle-[username].zip will get downloaded</code> <p>
<source lang=bash>
unzip ucp-bundle-bob.zip
Archive: ucp-bundle-bob.zip
extracting: ca.pem
extracting: cert.pem
extracting: key.pem
extracting: cert.pub
extracting: env.sh
extracting: env.ps1
extracting: env.cmd

cat env.sh
export COMPOSE_TLS_VERSION=TLSv1_2
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH="$PWD"
export DOCKER_HOST=tcp://3.16.143.49:443
#
# Bundle for user bob
# UCP Instance ID t0ltwwcw5tdbtjo2fxlzmj8p4
#
# This admin cert will also work directly against Swarm and the individual
# engine proxies for troubleshooting. After sourcing this env file, use
# "docker info" to discover the location of Swarm managers and engines.
# and use the --host option to override $DOCKER_HOST
#
# Run this command from within this directory to configure your shell:
# eval $(<env.sh)

eval $(<env.sh) # apply ucp-bundle

docker images # to list UCP managed images
</source></p>
# <li value="4"> In my lab I had to update DOCKER_HOST from public IP to private IP </li>
Err: error during connect: Get https://3.16.143.49:443/v1.39/images/json: x509: certificate is valid for 127.0.0.1, 172.31.101.248, 172.17.0.1, not 3.16.143.49
<source lang=bash>
export DOCKER_HOST=tcp://172.31.101.248:443
</source>

# <li value="5"> Verify if you have permissions to create a service</li>
<source lang=bash>
docker service create --name test111 httpd
Error response from daemon: access denied:
no access to Service Create, on collection swarm
</source>

# <li value="6"> Add Grants to the user</li>
## Go to User Management > Granst > Create Grant
## Base on a Roles, select Full Control
## Select Subjects, All Users, select the user
## Click Create
# Re-run service create command that should succeed now. This service can be managed now also within UCP console.

= Docker Secure Registry | image: registry =
Docker provides a special docker image that can be used to manage docker imagages both internally or externally thus steps below include securing the access with SSL certificate.

Create certificate
<source lang=bash>
mkdir ~/{auth,certs}
# create self-signed certificate for Docker Repository
mkdir certs; cd _$ #cd to last argument in history
openssl req -newkey rsa:4096 -nodes -sha256 -keyout repo-key.pem -x509 -days 365 -out repo-cer.pem -subj /CN=myrepo.com
# trusted-certs docker client directory, docker client looks for trusted certs when conencting to reomote repo
sudo mkdir -p /etc/docker/certs.d/myrepo.com:5000 #port 5000 is a default port
sudo cp repo-cer.pem /etc/docker/certs.d/myrepo.com:5000/ca.crt
</source>
<code>ca.crt</code> is default/required CAroot trustcert file name, that the docker client (docker login API) uses when conencting to remote repository. In our case we trust any cert signed by CA=ca.crt when connecting to myrepo.com:5000 as same certs (selfsigned), got installed in <code>repository:2</code> container via <code>-v /certs/</code> option.

Optional for development purposes to add doamin ''myrepo.com'' to hostfile binding to local interface ip address.
<source lang=bash>
sudo -i; echo "172.16.10.10 myrepo.com" >> /etc/hosts; exit
</source>

Optional add insecure-registry entry
<source>
sudo vi /etc/docker/deamon.json
{
"insecure-registries" : [ "myrepo.com:5000"]
}
</source>

Pull special Docker Registry image
<source lang=bash>
mkdir -p ~/auth #authentication directory, used when deploying local repository
docker pull registry:2
docker run --entrypoint htpasswd registry:2 -Bbn reg-admin Passw0rd123 > ~/auth/htpasswd
# -Bbn -parameters
# reg-admin -user
# Passw0rd123 -password string for basic htpasswd authentication method, the hashed password will be displayed to STDOUT

$ cat ~/auth/htpasswd
reg-admin:$2y$05$DnTWDHp7uTwaDrw4sXpUbuDDIlLwu3c8MEMsHPjK/AcUMdK/TD6fO
</source>

Run Registry container
<source lang=bash>
cd ~
docker run -d -p 5000:5000 --name myrepo \
-v $(pwd)/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/repo-cer.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/repo-key.pem \
-v $(pwd)/auth:/auth \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
registry:2
# -v -indicate where our certificates will be mounted within a container
# -e REGISTRY_HTTP_TLS_CERTIFICATE -path to cert inside the container
# -v $(pwd)/auth:/auth -mounting authentication directory where a file with password is
# -e REGISTRY_AUTH htpasswd -setting up to use 'htpasswd' authentication method
# registry:2 -image name, positinal params
</source>

Verify
<source lang=bash>
docker pull alpine
docker tag alpine myrepo.com:5000/aa-alpine #create a tagged image (copy) on a local filesystem,
# it must be prefixed with the private repo name '/' image name you want to upload as

docker logout # if logged in to another repository
docker login myrepo.com:5000/aa-alpine #login to a repository that runs as a container, stays login untill logout/reboot
docker login myrepo.com:5000/aa-alpine --username=rep-admin --password Passw0rd123
docker push myrepo.com:5000/aa-alpine

docker image rmi alpine myrepo.com:5000/aa-alpine #delete image stored locally
docker pull myrepo.com:5000/aa-alpine #pull image from a container repository

# List private-repository images
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/_catalog
{"repositories":["aa-alpine"]}

wget --no-check-certificate --http-user=reg-admin --http-password=password https://myrepo.com:5000/v2/_catalog
cat _catalog
{"repositories":["my-alpine","myalpine","new-aa-busybox"]}

# List tags
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/tags/list
{"name":"myalpine","tags":["latest"]}
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/manifests/latest #entire image metadata
</source>

Note. There is no easy way to delete images from repository:2 container.

= Docker push =
;Login to a docker repository
<source lang=bash>
docker info | grep -B1 Registry #check if you are logged in to docker.hub repository
WARNING: No swap limit support
Registry: https://index.docker.io/v1/

docker login

docker info | grep -B1 Registry
Username: pio2pio
Registry: https://index.docker.io/v1/
</source>

; Tag and push an image
<source lang=bash>
# docker tag local-image:tagname new-repo:tagname #create a local copy of an image
# docker push new-repo:tagname

docker pull busybox
docker --tag busybox:latest pio2pio/testrepo
docker push pio2pio/testrepo
The push refers to repository [docker.io/pio2pio/testrepo]
683f499823be: Mounted from library/busybox
latest: digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510
size: 527
</source>

; Docker Content Trust
All images are implicitly trusted by your Docker daemon. Buy can set that ONLY signed images are allowed. You can configure your systems for trusting only image tags that have been signed.
<source lang=bash>
export=DOCKER_CONTENT_TRUST=1 #enable system to sign an image during push process
docker build -t myrepo.com:5000/untrusted.latest
docker push myrepo.com:5000/untrusted.latest
...
No tag specified, skipping trust metadata push
# 2nd attempt, with a tag specified now
docker push myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority

docker pull myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority
</source>

;Errors explained:
Err: No tag specified, skipping trust metadata push<br />
* Explenation: When image gets signed is signed by a tag. Thereofre if you skip a tag it won't get signed and metada get skipped.
Err: error contacting notary server: x509: certificate signed by unknown authority
* when uploading the image gets uploaded, but it is not trusted becasue signed with self-signed CA
* when downloading, and <code>DOCKER_CONTENT_TRUST=1</code> is enabled, the image cannot be downloaded because is untrusted

= Theory =
== What is a docker ==
Docker is a container runtime platform, where Swarm is a container orchestration platform.

== Security ==
=== Mutually Authenticated TLS ===
Docker Swarm is ''secure by default'', it means all communication is encrypted. ''Mutually Authenticated TLS'' is the implementation was chosen to secure that communication. Any time a swarm is inicialised, a self-signed CA is generated and issues certificates to every node (mgr or wkr) to facilicate registration (join mgr or wkr) and latter those secure communications. It's transient container brought up to generate CA certs every time a cert is needed. MTLS communication is between Managers and Workers.

== [[Linux Namespaces and Control Groups]] ==

== Difference between docker attach and docker exec ==
;Attach
The docker attach command allows you to attach to a running container using the containers ID or name, either to view its ongoing output or to control it interactively. You can attach to the same contained process multiple times simultaneously, screen sharing style, or quickly view the progress of your detached process.

The command docker attach is for attaching to the existing process. So when you exit, you exit the existing process.

If we use docker attach, we can use only one instance of shell. So if we want open new terminal with new instance of container's shell, we just need run docker exec

If the docker container was started using /bin/bash command, you can access it using attach, if not then you need to execute the command to create a bash instance inside the container using exec. Attach isn't for running an extra thing in a container, it's for attaching to the running process.

To stop a container, use CTRL-c. This key sequence sends SIGKILL to the container. If --sig-proxy is true (the default),CTRL-c sends a SIGINT to the container. You can detach from a container and leave it running using the CTRL-p CTRL-q key sequence.

;exec

"docker exec" is specifically for running new things in a already started container, be it a shell or some other process. The docker exec command runs a new command in a running container.

The command started using docker exec only runs while the containerâ€™s primary process (PID 1) is running, and it is not restarted if the container is restarted.

exec command works only on already running container. If the container is currently stopped, you need to first run it. So now you can run any command in running container just knowing its ID (or name):

<source>
docker exec <container_id_or_name> echo "Hello from container!"
docker run -it -d shykes/pybuilder /bin/bash
</source>

The most important here is the -d option, which stands for detached. It means that the command you initially provided to the container (/bin/bash) will be run in background and the container will not stop immediately.

= Dockerfile - python =
* [https://luis-sena.medium.com/creating-the-perfect-python-dockerfile-51bdec41f1c8 perfect python dockerfile] Medium

= References =
*[https://docs.docker.com/v1.8/installation/ubuntulinux/ Ubuntu installation] official website
*[https://docs.docker.com/engine/admin/systemd/ PROXY settings for systemd]
*[http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/ Docker RUN vs CMD vs ENTRYPOINT]
*[https://vsupalov.com/docker-arg-vs-env/ docker ARG vs ENV]
*[https://www.fromlatest.io/#/ Docker online lintel]
*[https://hub.docker.com/r/portainer/portainer/ portainer] Monitor your containers via Web GUI
*[https://treescale.com/ treescale.com] Free private Docker registry

Docker

2025-09-03T04:48:51Z

Pio2pio: /* Ubuntu 16.04, 18.04, 20.04 */

Containers taking a world J

= [https://docs.docker.com/install/linux/docker-ce/ubuntu/ Installation] =
General procedure:
# Make sure you don't have docker already installed from your packet manager
# The /var/lib/docker may be

To install the latest version of Docker with curl:
<source lang="bash">
curl -sSL https://get.docker.com/ | sh
</source>

== CentOS ==
<source lang="bash">
sudo yum install bash-completion bash-completion-extras #optional, requires you log out
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 #utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo #docker-ee.repo for EE edition
# --enable docker-ce-{edge|test} #for beta releases
sudo yum update
sudo yum clean all #not sure why this command is here
sudo yum install docker-ce
#old: sudo yum install -y --setopt=obsoletes=0 docker-ce-17.03.1.ce-1.el7.centos docker-ce-selinux-17.03.1.ce-1.el7.centos
sudo systemctl enable docker && sudo systemctl start docker && sudo systemctl status docker
yum-config-manager --disable jenkins #disable source to prevent accidental update ?jenkins?
</source>

== Ubuntu 24.04 ==
<source lang="bash">

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install the latest version
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Install a specific version
## List the available versions:
apt-cache madison docker-ce | awk '{ print $3 }'
5:28.3.3-1~ubuntu.24.04~noble
5:28.3.2-1~ubuntu.24.04~noble

VERSION_STRING=5:28.3.3-1~ubuntu.24.04~noble
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

# Manage Docker as a non-root user
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker # activate the group without logging off
<source>

== Ubuntu 16.04, 18.04, 20.04 ==
<source lang="bash">
# Optional, clear out config files
sudo rm /etc/systemd/system/docker.service.d/docker.conf
sudo rm /etc/systemd/system/docker.service
sudo rm /etc/default/docker #environment file

# New docker package is called now 'docker-ce'
sudo apt-get remove docker docker-engine docker.io containerd runc docker-ce # start fresh
sudo apt-get -yq install apt-transport-https ca-certificates curl gnupg-agent software-properties-common # apt over HTTPs
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # Docker official GPG key
sudo apt-key fingerprint 0EBFCD88 #verify

#add the repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # or {edge|test}
sudo apt-get update # optional

# Option 1 - install latest
sudo apt-get install docker-ce docker-ce-cli containerd.io

# Option 2 - install fixed version
sudo apt-cache madison docker-ce # display available versions
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
sudo apt-get install docker-ce=18.09.0~3-0~ubuntu-bionic docker-ce-cli=18.09.0~3-0~ubuntu-bionic containerd.io
sudo apt-mark hold docker-ce docker-ce-cli containerd.io
sudo apt-mark showhold # show packages that version upgrade has been put on hold

# Unhold
sudo apt-mark unhold docker-ce docker-ce-cli containerd.io
</source>

[https://docs.docker.com/engine/release-notes/ Newer versions] (>18.09.0) of Docker come with 3 packages:
* <code>containerd.io</code> - daemon to interface with the OS API (in this case, LXC - Linux Containers), essentially decouples Docker from the OS, also provides container services for non-Docker container managers
* <code>docker-ce</code> - Docker daemon, this is the part that does all the management work, requires the other two on Linux
* <code>docker-ce-cli</code> - CLI tools to control the daemon, you can install them on their own if you want to control a remote Docker daemon

Example of how to run [[Jenkins CI|Jenkins docker image]]

== Add a user to docker group ==
Add your user to <tt>docker group</tt> to be able to run docker commands without need of ''sudo'' as the <code>docker.socket</code> is owned by group <code>docker</code>.
<source lang="bash">
sudo usermod -aG docker $(whoami)

# log in to the new docker group (to avoid having to log out / log in again)
newgrp docker
</source>

Reason
<source lang=bash>
[root@piotr]$ ls -al /var/run/docker.sock
srw-rw----. 1 root docker 7 Jan 09:00 docker.sock
</source>

= HTTP proxy =
Configure ''docker'' if you run behind a proxy server. In this example CNTLM proxy runs on the host machine listening on localhost:3128. This example overrides the default docker.service file by adding configuration to the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:
<source lang=bash">
sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"
Environment="HTTP_PROXY=http://172.31.1.1:3128/" #overrides previous entry
Environment="HTTPS_PROXY=http://172.31.1.1:3128/"
# If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable
Environment="NO_PROXY=localhost,127.0.0.1,10.6.96.172,proxy.example.com:80"
</source>

Flush changes:
$ sudo systemctl daemon-reload
Verify that the configuration has been loaded:
$ systemctl show --property=Environment docker
Environment=HTTP_PROXY=<nowiki>http://proxy.example.com:80/</nowiki>
Restart Docker:
$ sudo systemctl restart docker

= Docker create and run, basic options =
<source lang="bash">
# It will create a container but won't start it up
docker container create -it --name="my-container" ubuntu:latest /bin/bash
docekr container start my-container

docker run -it --name="mycentos" centos:latest /bin/bash
# -i :- interactive mode (attach to STDIN) \command to execute when instantiating container
# -t :- attach to the current terminal (sudo TTY)
# -d :- disconnect mode, daemon mode, detached mode, running the task in the background
# -p :- publish to host exposed container port [ host_port(8080):container_exposedPort(80) ]
# --rm :- remove container after command has been executed
# --name="name_your_container"

# -e|--env MYVAR=123 exports/passing variable to the container, echo $MYVAR will have a value 123
# --privileged :- option will allow Docker to perform actions normally restricted,
# like binding a device path to an internal container path.
</source>

= Docker inspect =
== inspect image ==
<source>
docker image inspect centos:6
docker image inspect centos:6 --format '{{.ContainerConfig.Hostname}}' #just a single value
docker image inspect centos:6 --format '{{json .ContainerConfig}}' #json key/value output
docker image inspect centos:6 --format '{{.RepoTags}}' #shows all associated tags with the image
</source>

Using <code>--format</code> is similar to <code>jq</code>
== inspect container ==
Shows current configuration state of a docker container or an image.
<source>
docker inspect <container_name> | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.3",
"IPAddress": "172.17.0.3",
</source>

= Attach/exec to a docker process =
If you are running eg. <tt>/bin/bash</tt> as a command you can get attached to this running docker process. Note that when you exit the container will stop.
<source lang=bash>
docker attach mycentos
</source>

To avoid stopping a container on exit of <code>attach</code> command we can use <code>exec</code> command.
<source lang=bash>
docker exec -it mycentos /bin/bash
</source>

Attaching directly to a running container and then exiting the shell will cause the container to stop. Executing another shell in a running container and then exiting that shell will not stop the underlying container process started on instantiation.

= Entrypoint, CMD, PID1 and [https://github.com/krallin/tini tini] =
== Entrypoint and receiving signals ==
Reciving signals and handling them within containers here Docker it's the same important as for any other application. Remember containers it's a group of processes running on your host so you need to take care of signals send to your applications.

As a principal container management tool eg. <code>docker stop</code> sends a configurable (in Dockerfile) signal to the entrypoint of your application where <code>SIGTERM - 15 - Termination (ANSI)</code> is default.

;ENTRYPOINT syntax

<source>
# exec form, require JSON array; IT SHOULD ALWAYS BE USED
ENTRYPOINT ["/app/bin/your-app", "arg1", "arg2"]

# shell form, it always runs as a subcommand of '/bin/sh -c', thus your application will never see any signal sent to it
ENTRYPOINT "/app/bin/your-app arg1 arg2"
</source>

;ENTRYPOINT is a shell script
If application is started by shell script regular way, your shell spawns your application in a new process and you won’t receive signals from Docker. Therefore we need to tell shell to replace itself with your application using the <code>[https://stackoverflow.com/questions/18351198/what-are-the-uses-of-the-exec-command-in-shell-scripts exec]</code> command, check also <code>[https://en.wikipedia.org/wiki/Exec_(system_call) exec syscall]</code>. Use:
<source lang=bash>
/app/bin/my-app # incorrect, signal won't be received by 'my-app'
exec /app/bin/my-app # correct way
</source>

;ENTRYPOINT exec with a pipe commands causing starting a subshell
If you <code>exec</code> piping will force a command to be run in a subshell with the usual consequence: no signals to an app.
<source lang=bash>
exec /app/bin/your-app | tai64n # here you want to add timestamps by piping through tai64n,
# causing running your command in a subshell
</source>

;Let another program to be PID1 and handle signalling
* tini
* dump-init

<source lang=bash>
ENTRYPOINT ["/tini", "-v", "--", "/app/bin/docker-entrypoint.sh"]
</source>

tini and dumb-init are also able to proxy signals to process groups which technically allows you to pipe your output.However, your pipe target receives that signal at the same time so you can’t log anything on cleanup lest you crave race conditions and SIGPIPEs. So, it's better to avoid logging at termination at all.

;Change signal that will terminate your container process
Listen for SIGTERM or set STOPSIGNAL in your Dockerfile.
<source lang=bash>
vi Dockerfile
STOPSIGNAL SIGINT # this will trigger container termination process if someone press Ctrl^C
</source>

;References:
* [https://hynek.me/articles/docker-signals/ Why Your Dockerized Application Isn’t Receiving Signals]
* [http://smarden.org/runit/ runit] alternative to tini

== Tini ==
It's a tiny but valid init for containers:
* protects you from software that accidentally creates zombie processes
* ensures that the default signal handlers work for the software you run in your Docker image
* does so completely transparently! Docker images that work without Tini will work with Tini without any changes
* Docker 1.13+ has Tini included, to enable Tini, just pass the <code>--init</code> flag to docker run

;Understanding Tini
After spawning your process, Tini will wait for signals and forward those to the child process, and periodically reap zombie processes that may be created within your container. When the "first" child process exits (/your/program in the examples above), Tini exits as well, with the exit code of the child process (so you can check your container's exit code to know whether the child exited successfully).

Add Tini - general dynamicly-linked library (in the 10KB range)
<source lang=bash>
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

# Run your program under Tini
CMD ["/your/program", "-and", "-its", "arguments"]
# or docker run your-image /your/program ...
</source>

Add Tini to Alpine based image
<source lang=bash>
RUN apk add --no-cache tini
# Tini is now available at /sbin/tini
ENTRYPOINT ["/sbin/tini", "--"]
</source>

Existing entrypoint
<source>
ENTRYPOINT ["/tini", "--", "/docker-entrypoint.sh"]
</source>

;References:
*[https://github.com/krallin/tini/issues/8 What is advantage of Tini?]
*[https://ahmet.im/blog/minimal-init-process-for-containers/ Choosing an init process for multi-process containers]

= Mount directory in container =
We can mount host directory into docker container so the content will be available from the container
<source lang="bash">
docker run -it -v /mnt/sdb1:/opt/java pio2pio/java8
# syntax: -v /path/on/host:/path/in/container
</source>

= Build image =
== Dockerfile ==
Each line ''RUN'' creates a container so if possible, we should join lines so it ends up with less layers.
<source lang="bash">
$ wget jkd1.8.0_111.tar.gz
$ cat Dockerfile <<- EOF #'<<-' heredoc with '-' minus ignores <tab> indent
ARG TAGVERSION=6 #only command allowed b4 FROM
FROM ubuntu:${TAGVERSION}
FROM ubuntu:latest #defines base image eg. ubuntu:16.04
LABEL maintainer="myname@gmail.com" #key/value pair added to a metadata of the image

ARG ARG1=value1

ENV ENVIRONMENT="prod"
ENV SHARE /usr/local/share #define env variables with syntax ENV space EnvironmetVariable space Value
ENV JAVA_HOME $SHARE/java

# COPY jkd1.8.0_111.tar.gz /tmp #works only with files, copy a file to container filesystem, here to /tmp
# ADD http://example.com/file.txt
ADD jkd1.8.0_111.tar.gz / #add files into the image root folder, can add also URLs

# SHELL ["executable","params"] #overrides /bin/sh -c for RUN,CMD, etc..

# Executes commands during build process in a new layer E.g., it is often used for installing software packages
RUN mv /jkd1.8.0_111.tar.gz $JAVA_HOME
RUN apt-get update
RUN ["apt-get", "update", "-y"] #in json array format, allows to run a commands but does not require shell executable

VOLUME /mymount_point #this command does not mount anything from a host, just creates a mountpoint

EXPOSE 80 #it doesn't automatically map the port to a hosts

#containers usually don't have system maangement eg. systemctl/service/init.d as designed to run as single process
#entry point becomes main command that start the main proces
ENTRYPOINT apachectl "-DFOREGROUND" #think about it as the MAIN_PURPOSE_OF_CONTAINER command.
# It's always run by default it cannot be overridden

#Single command that will run after the image has been created. Only one per dockerfile, can be overriden.
CMD ["/bin/bash"]

# STOPSIGNAL

EOF
</source>

== Build ==
<source lang="bash">
docker build --tag myrepo/java8 . #-f point to custom Dockerfile name eg. -f Dockerfile2
# myrepo dockerhub username, java8 -image name,
# . directory where is the Dockerfile

docker build -t myrepo/java8 . --pull --no-cache --squash
# --pull regardless a local copy of an image can exist force to download a new image
# --no-cache don't use cache to build, forcing to rebuild all interim containers
# --squash after the build squash all layers into a single layer.

docker images #list images
docker push myrepo/java8 #upload the image to DockerHub repository
</source>

<code>squash</code> is enabled only on docker demon with experimental features enabled.

= Manage containers and images =
== Run a container ==
When you ''run'' a container you will create a new container from a image that has been already build/ is available then put in running state
* -d detached mode, the container will continue to run after the CMD or passed on command exited
* -i interactive mode, allows you to login in /ssh to the container

<source lang="bash">
# docker container run [OPTIONS] IMAGE [COMMAND] [ARG...] # usage
docker container run -it --name mycentos centos:6 /bin/bash
docker run -it pio2pio/java8 #container section command is optional
# -i :- run in interactive mode, then run command /bin/bash
# --rm :- will delete container after run
# --publish | -p 80:8080 :- publish exposed container port 80-> to 8080 on the docker-host
# --publish-all | -P :- publish all exposed container ports to random port >32768
</source>

== List images ==
<source lang="bash">
ctop #top for containers
docker ps -a #list containers
docker image ls #list images
docker images #short form of the command above
docker images --no-trunc
docker images -q #--quiet
docker images --filer "before=centos:6"

# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81
</source>

== Search images in remote repository ==
Search the DockerHub for images,. You may require to do `docker login` first
<source lang="bash">
IMAGE=ubuntu
docker search $IMAGE
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ubuntu Ubuntu is a Debian-based Linux operating sys… 8206 [OK]
dorowu/ubuntu-desktop-lxde-vnc Ubuntu with openssh-server and NoVNC 210 [OK]
rastasheep/ubuntu-sshd Dockerized SSH service, built on top of offi… 167 [OK]

IMAGE=apache
docker search $IMAGE --filter stars=50 # search images that have 50 or more stars
docker search $IMAGE --limit 10 # display top 10 images
</source>

List all available tags
<source lang="bash">
IMAGE=nginx
wget -q https://registry.hub.docker.com/v1/repositories/${IMAGE}/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}' | sort -V
</source>

== Pull images ==
<source lang="bash">
<name>:<tag>
docker pull hello-world:latest # pull latest
docker pull --all hello-world # pull all tags
docker pull --disable-content-trust hello-world # disable verification

docker images --digests #displays sha256: digest of an image

# Dangling images - transitional images
</source>
=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/registries.html#registry_auth from Amazon ECR] ===
;Docker login to ECR service using IAM
<code>docker login</code> does not support native IAM authentication methods. Therefore use a command below that will retrieve, decode, and convert the <code>authorization IAM token</code> into a pre-generated <code>docker login</code> command. Therefore produced login credentials will assume your current IAM User/Role permissions. If your current IAM user can only pull from ECR, after login with <code>docker login</code> you still won't be able to push image to the registry. Example error you may get is <code>not authorized to perform: ecr:InitiateLayerUpload</code>

Login to ECR service, your IAM user requires to have relevant pull/push permissions
<source lang="bash">
eval $(aws ecr get-login --region eu-west-1 --no-include-email)
# aws ecr get-login # generates below docker command with the login token
# docker login -u AWS -p **token** https://$ACCOUNT.dkr.ecr.us-east-1.amazonaws.com # <- output
</source>

Docker login to ECR singular repository, min awscli v1.17
<source lang="bash">
ACCOUNT=111111111111
REPOSITORY=myrepo
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/$REPOSITORY
</source>

=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html push to Amazon ECR] ===
<source lang=bash>
# List images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ansible-aws 2.0.1 b09807c20c96 5 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Tag an image 'b09807c20c96'
docker tag b09807c20c96 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1

# List images, to verify your newly tagged one
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB # <- new tagged image
ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Push an image to ECR
docker push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1
The push refers to repository [111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws]
2c405c66e675: Pushed
...
77cae8ab23bf: Layer already exists
2.0.1: digest: sha256:111111111193969807708e1f6aea2b19a08054f418b07cf64016a6d1111111111 size: 1796
</source>

== Save and import image ==
In course to move a image to another filesystem we can save it into <code>.tar</code>
<source lang="bash">
# Export
docker image save myrepo/centos:v2 > mycentos.v2.tar
tar -tvf mycentos.v2.tar

# Import
docker image import mycentos.v2.tar <new_image_name>

# Load from a stream
docker load < mycentos.v2.tar #or --input mycentos.v2.tar to avoid redirections
</source>

== Export aka commit container into image ==
Let's say we wanto modify stock image centos:6 by installing Apache interactivly, set to autostart then export as an new image. Let's do it!
<source lang="bash">
docker pull centos:6
docker container run -it --name apache-centos6 centos:6
# Interactively do: yum -y update; yum install -y httpd; chkconfig httpd on; exit

# Save container changes - option1
docker commit -m "added httpd daemon" -a "Piotr" b237d65fd197 newcentos:withapache #creates new image from a container's changes
docker commit -m "added httpd daemon" -a "Piotr" <container_name> <repo>/<image>:<tag>
# -a :- author

# Save container changes - option2
docker container export apache-centos6 > apache-centos6.tar
docker image import apache-centos6.tar newcentos:withapache

docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
newcentos withapache ea5215fb46ed 50 seconds ago 272MB

docker image history newcentos:withapache
IMAGE CREATED CREATED BY SIZE COMMENT
ea5215fb46ed 2 minutes ago 272MB Imported from -
</source>

I am unsure what is a difference in creation of images from a container between:
* <code>docker container commit</code>
* <code>docker container export</code> - this seems creates smaller image

== Tag images ==
Tags are used to usually to name Official image with a new name that we are planning to modify. This allows to create a new image, run a new container from a tag, delete the original image without affecting the new image or container started from the new tagged image.
<source lang="bash">
docker image tag #long version
docker tag centos:6 myucentos:v1 #this will create a duplicate of centos:6 named myucentos:v1
</source>

Tagging allows to modify repository name and maanges references to images located on a filesystem.

== History of an image ==
We can display history of layers that created the image by showing interim images build in creation order. It shows only layers created on a local filesystem.
<source lang="bash">
docker image history myrepo/centos:v2
</source>
== Stop and delete all containers ==
<source lang="bash">
docker stop $(docker ps -aq) && docker rm $(docker ps -aq)
</source>

== Delete image ==
<source lang="bash">
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
company-repo 0.1.0 f796d7f843cc About an hour ago 888MB
<none> <none> 04fbac2fdf48 3 hours ago 565MB
ubuntu 16.04 7aa3602ab41e 3 weeks ago 115MB

# Delete image
$ docker rmi company-repo:0.1.0
Untagged: company-repo:0.1.0
Deleted: sha256:e5cca6a080a5c65eacff98e1b17eeb7be02651849b431b46b074899c088bd42a
..
Deleted: sha256:bc7cda232a2319578324aae620c4537938743e46081955c4dd0743a89e9e8183

# Prune image - delete dangling (temp/interim) images.
# These are not associated with end-product image or containers.
docker image prune
docker image prune -a #remove all images not associated with any container
</source>

== Cleaning up space by removing docker objects ==
This applied both to docker container and swarm systems.
<source lang="bash">
docker system df #show disk usage
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 0 131.7MB 131.7MB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B

docker network ls #note all networks below are system created, so won't get removed
NETWORK ID NAME DRIVER SCOPE
452b1c428209 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local

docker system prune #removes objects created by a user only, on the current node only
#add --volumes to remove them as well
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N]

docker system prune -a --volumes #remove all
</source>

== Docker Volumes ==
Docker's 'copy-on-write' philosophy drives both performance and efficiency. It's only the top layer that is writable and it's a delta of underlying layer.

Volumes can be mounted to your container instances from your underlying host systems.

''_data'' volumes, since they are not controlled by the storage driver (since they represent a file/directory on the host filesystem /var/lib/docker), are able to bypass the storage driver. As a result, their contents are not affected when a container is removed.

Volumes are data mounts created on a host in <code>/var/lib/docker/volumes/</code> directory and refereed by name in a Dockerfile.
<source lang="bash">
docker volume ls #list volumes created by VOLUME directive in a Dockerfile
sudo tree /var/lib/docker/volumes/ #list volumes on host-side
docker volume create my-vol-1
docker volume inspect my-vol-1
[
{
"CreatedAt": "2019-01-17T08:47:01Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-vol-1/_data",
"Name": "my-vol-1",
"Options": {},
"Scope": "local"
}
]
</source>

Using volumes with Swarm services
<source lang=bash>
docker container run --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #container
docker service create --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #swarm service
# --mount --volumes|-v is not supported with services, this will replicate volumes across swarm when needed,
# but it will not replicate files

docker exec -it web1 /bin/bash #connect to the container
roor@c123:/ echo "Created when connected to container: volume-web1" > /internal-mount/local.txt; exit

# prove the file is on a host filesystem created volume
user@dockerhost$ cat /var/lib/docker/volumes/my-vol-1/_data/local.txt
</source>

;Host storage mount
Bind mapping is binding host filesystem directory to a container directory. It's not mouting volume that it'd require a mount point and volume on a host.
<source lang=bash>
mkdir /home/user/web1
echo "web1 index" > /home/user/web1/index.html
docker container run -d --name testweb -p 80:80 --mount type=bind,source=/home/user/web1,target=/usr/local/apache2/htdocs httpd
curl http://localhost:80
</source>

Removing service is not going to remove the volume unless you delete the volume itself. It that case will be removed from all swarms.

== Networking ==
=== Container Network Model ===
It's a concept of network implementation that is built on multiple private networks across multiple hosts overlayed and managed by IPAM. Protocol that keeps track and provision addesses.

Main 3 components:
* sandbox - contains the configuration of a container's network stack, incl. management of interfaces, routing and DNS. An implementation of a Sandbox could be a eg. Linux Network Namespace. A Sandbox may contain many endpoints from multiple networks.
* endpoint - joins a Sandbox to a Network. Interfaces, switches, ports, etc and belong to only one network at the time. The Endpoint construct exists so the actual connection to the network can be abstracted away from the application. This helps maintain portability.
* network - a clollection of endpoints that can communicate directly (bridges, VLANs, etc.) and can consist of 1toN endpoints

[[File:Container-network-model.png|none||left|Container Network Model]]

;IPAM (Internet Protocol Address Management)
Managing addesees across multiple hosts on a separate physical networks while providing routing to the underlaying swarm networks externally is ''the IPAM prblem'' for Docker. Depends on the netwok driver choice, IPAM is handled at different layers in the stack. ''Network drivers'' enable IPAM through ''DHCP drivers'' or plugin drivers so the complex implementation that would be normally overlapping addesses is supported.

;References
* [https://success.docker.com/article/networking Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks]

=== Publish exposed container/service ports ===
;Publishing modes
;host: set using <code> --publish mode=host,8080:80</code>, makes ports available only on the undelaying host system not outside the host the service may exist; defits ''routing mesh'' so user is responsible for routing
;ingress: responsible for ''routing mesh'' makes sure all published ports are avaialble on all hosts in the swarm cluster regardless is a service replica running on it or not

<source lang=bash>
# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81

# Publish port
host : container
\ : /
docker container run -d --name web1 --publish 81:80 httpd
# --publish | -p :- publish to host exposed container port
# 81 :- port on a host, can use range eg. 81-85, so based on port availability port will be used
# 80 :- exposed port on a container

ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::81 :::*
LISTEN 0 128 :::22 :::*

docker container run -d --name web1 --publish-all 81:80 httpd
# --publish-all | -P publish all cotainer exposed ports to random ports above >32768
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c63efe9cbb94 httpd "httpd-foreground" 2 sec.. Up 1 s 80/tcp testweb #port exposed but not published
cb0711134eb5 httpd "httpd-foreground" 4 sec.. Up 2 s 0.0.0.0:32769->80/tcp testweb1 #port exposed and published to host:32769
</source>

=== Network drivers ===
Default network for a single host docker-host is ''bridge'' network.

;List of Native (part of Docker Engine) Network Drivers:
;bridge: default on stand-alone hosts, it's private network internal to the host system, all containers on this host using Bridge network can communicate, external access is granted by port exposure or static-routes added with teh host as the gateway for that network
;none: used when a container does not need any networkng, still can be accessed from the host using <code>docker attach [containerID]</code> or <code>docker exec -it [containerID]</code> commands
;host: aka ''Host Only Networking'', only accessable via underlaying host, access to services can be provided by exposing ports to the host system
;overlay: swarm scope driver, allows communication to all Docker Daemons in a cluster, self-extending if needed, maanged by Swarm manager, it's default mode of Swarm communication
;ingress: extended network across all nodes in the cluster; special overlay network that load balances netowrk traffic amongst a given service's working nodes; maintains a list of all IP addresses from nodes that participate in that service (using the IPVS module) and when a request comes in, routes to one of them for the indicated service; provides ''routing mesh' that allows services to be exposed to the external network without having replica running on every node in the Swarm
;docker gateway bridge: special bridge network that allows overlay networks (incl. ingress) access an individual DOcker daemon's physical network; every container run within a service is connected to the local Docerk daemon's host network; automatically created when Docker is initialised or joined by <code>docker swarm init</code> or <code>docker join</code> commands.

;Docker interfaces
* <code>docker0</code> - adapter is installed by default during Docker setup and will be assigned an address range that will determine the local host IPs available to the containers running on it

;Create bridge network
<source lang=bash>
docker network ls #default networks list
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm #'ingress' special network 1 per cluster
815a9c2c4005 none null local

docker network inspect bridge #bridge is a default network containers are deployed to

docker container run -d web1 -p 8080:80 httpd #expose container port :80 -> :8080 on the docker-home
docker container inspect web1 | grep IPAdd
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1 #get container ip
curl http://$(IPAddr)
</source>

;Create bridge network
<source lang=bash>
docker network create --driver=bridge --subnet=192.168.1.0/24 --opt "com.docker.network.driver.mtu"=1501 deviceeth0

docker network ls
docker network inspect deviceeth0
</source>

;Create overlay network
<source lang=bash>
docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
docker network ls
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local
2x6bq1czzdc1 overlay0 overlay swarm
</source>

;Inspect network
<source lang=json>
docker network inspect overlay0
[
{
"Name": "overlay0",
"Id": "2x6bq1czzdc102sl6ge7gpm3w",
"Created": "2019-01-19T11:24:02.146339562Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.1.0/24",
"Gateway": "192.168.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": null
}
]
</source>

;Inspect container network
<source lang=bash>
docker container inspect testweb --format {{.HostConfig.NetworkMode}}
overlay0
docker container inspect testweb --format {{.NetworkSettings.Networks.dev_bridge.IPAddress}}
192.168.1.3
</source>

Connect/disconnect from a network can be done when a container is running. Connect won't disconnect from a current network.
<source lang=bash>
docker network connect --ip=192.168.1.10 deviceeth0 web1
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1
docker container inspect --format="{{.NetworkSettings.Networks.deviceeth0.IPAddress}}" web1
curl http://$(IPAddr)
docker network disconnect deviceeth0 web1
</source>

=== Overlay network in Swarm cluster ===
Overlay network can be created/removed/updated like any other docker objects. It allows inter-service(containers) communication, where <code>--gateway</code> ip address is used to reach to outside eg. Inernet or the host network. When creating the <code>overlay</code> network on the manager host it will get recreated on worker nodes only when is referenced by any service that is using it. See below.
<source lang="bash">
swarm-mgr$ docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
swarm-mgr$ docker service create --name web1 -p 8080:80 --network=overlay0 --replicas 2 httpd
uvxymzdkcfwvs2oznbnk7nv03
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]

swarm-wkr$ docker network ls
NETWORK ID NAME DRIVER SCOPE
ba175ebd2a6f bridge bridge local
a5848f607d8c docker_gwbridge bridge local
fccfb9c1fdc3 host host local
t8jxy5vsy5on ingress overlay swarm
127b10783faa none null local
2x6bq1czzdc1 overlay0 overlay swarm

# remove network, only affected newly created servces not the running onces
swarm-mgr$ docker service update --network-rm=overlay0 web1
</source>

=== DNS ===
<source lang="bash">
docker container run -d --name testweb1 -P --dns=8.8.8.8 \
--dns=8.8.4.4 \
--dns-search "mydomain.local" \
httpd
# -P :- publish-all exposed ports to random port >32768

docker container exec -it testweb1 /bin/bash -c 'cat /etc/resolv.conf'
search us-east-2.compute.internal
nameserver 8.8.8.8
nameserver 8.8.4.4

# System wide settings, requires docker.service restart
cat > /etc/docker/daemon.json <<EOF
{
"dns": ["8.8.8.8", "8.8.4.4"]
}
EOF
sudo systemctl restart docker.service #required
</source>

= Examples =
== Lint - best practices ==
<source lang=bash>
$ docker run --rm -i hadolint/hadolint < Dockerfile
/dev/stdin:9:16 unexpected newline expecting "\ ", '=', a space followed by the value for the variable 'MAC_ADDRESS', or at least one space
</source>

== Default project ==
As good practice all Docker files should be source controlled. The basic self explanatory structure can looks like below, and skeleton be created with a commend below:
<source lang="bash">
mkdir APROJECT && d=$_; touch $d/{build.sh,run.sh,Dockerfile,README.md,VERSION};mkdir $d/assets; touch $_/{entrypoint.sh,install.sh}

└── APROJECT
├── assets
│ ├── entrypoint.sh
│ └── install.sh
├── build.sh
├── Dockerfile
├── README.md
└── VERSION
</source>

== Dockerfile ==
<code>Dockerfile</code> it is simply a build file.
=== Semantics ===
;<code>entrypoint</code>: Container config: what to start when this image is ran.

;<code>entrypoint</code> and <code>cmd</code>: Docker allows you to define an Entrypoint and Cmd which you can mix and match in a Dockerfile. Entrypoint is the executable, and Cmd are the arguments passed to the Entrypoint. The Dockerfile schema is quite lenient and allows users to set Cmd without Entrypoint, which means that the first argument in Cmd will be the executable to run.

=== User management ===
<source>
RUN addgroup --gid 1001 jenkins -q
RUN adduser --gid 1001 --home /tank --disabled-password --gecos '' --uid 1001 jenkins
# --gid add user to group 1001
# --gecos parameter is used to set the additional information. In this case it is just empty.
# --disabled-password it's like --disabled-login, but logins are still possible (for example using SSH RSA keys) but not using password authentication
USER jenkins:jenkins #sets user for next RUN, CMD and ENTRYPOINT command
WORKDIR /tank #changes cwd for next RUN, CMD, ENTRYPOINT, COPY and ADD
</source>

=== Multiple stage build ===
Introduced in Docker 17.06, allows to use multiple <code>FROM</code> statements allowing for multi stage builds.
<source>
FROM microsoft/aspnetcore-build AS build-env
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o output

# build runtime image
FROM microsoft/aspnetcore
WORKDIR /app
COPY --from=build-env /app/output . #multi stage: copy files from previous container [as build-env]
ENTRYPOINT ["dotnet", "LetsKube.dll"]
</source>

= Squash an image =
Docker uses <code>Union</code> filesystem that allows multiple volumes (layers) to share common and override changes by applying them on top layer.
There is no official way to ''flatten'' layers to a single storage layer or minimize an image size (2017). Below it's just practical approach.
# Start container from an image
# Export a container to <code>.tar</code> with all it's file systems
# Import container with new image name

Once the process completes and original image gets deleted the new image <code>docker image history</code> command will show only one layer. Often the image will be smaller.
<source lang=bash>
# run a container from an image
docker run myweb:v3
# export container to .tar
docker export <contr_name> > myweb.v3.tar
docker save <image id> > image.tar #not verified command
docker import myweb.v3.tar myweb:v4
docker load myweb.v3.tar #not verified command
</source>

;Resources
*[https://github.com/jwilder/docker-squash docker-squash] GitHub

= Gracefully stop / kill a container =
''all below are only notes''

Trap ctrl_c then kill/rm container.
*--init
*--sig-proxy this only works when --tty=false but by default is true

= Proxy =
If you are behind corporate proxy, you should use Docker client <code>~/.docker/config.json</code> config file. It requires Docker
17.07 minimum version.
<source>
{
"proxies":
{
"default":
{
"httpProxy": "http://10.0.0.1:3128",
"httpsProxy": "http://10.0.0.1:3128",
"noProxy": "localhost,127.0.0.1,*.test.example.com,.example2.com"
}
}
}
</source>
More you can find [https://docs.docker.com/network/proxy/#configure-the-docker-client here]

== Insecure proxy ==
These can be added to different places, the order is based on latest practices and versioning
;docker-ce 18.6
<source lang="json">
{
"insecure-registries" : [ "localhost:443","10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
}
</source>
<source lang="bash">
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl show docker | grep Env
docker info #check Insecure Registries
</source>

;Using environment file, prior version 18
<source lang="bash">
$ sudo vi /etc/default/docker
DOCKER_HOME='--graph=/tank/docker'
DOCKER_GROUP='--group=docker'
DOCKER_LOG_DRIVER='--log-driver=json-file'
DOCKER_STORAGE_DRIVER='--storage-driver=btrfs'
DOCKER_ICC='--icc=false'
DOCKER_IPMASQ='--ip-masq=true'
DOCKER_IPTABLES='--iptables=true'
DOCKER_IPFORWARD='--ip-forward=true'
DOCKER_ADDRESSES='--host=unix:///var/run/docker.sock'
DOCKER_INSECURE_REGISTRIES='--insecure-registry 10.0.0.0/8 --insecure-registry 172.16.0.0/12 --insecure-registry 192.168.0.0/16'
DOCKER_OPTS="${DOCKER_HOME} ${DOCKER_GROUP} ${DOCKER_LOG_DRIVER} ${DOCKER_STORAGE_DRIVER} ${DOCKER_ICC} ${DOCKER_IPMASQ} ${DOCKER_IPTABLES} ${DOCKER_IPFORWARD} ${DOCKER_ADDRESSES} ${DOCKER_INSECURE_REGISTRIES}"

$ sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/dockerd $DOCKER_HOME $DOCKER_GROUP $DOCKER_LOG_DRIVER $DOCKER_STORAGE_DRIVER $DOCKER_ICC $DOCKER_IPMASQ $DOCKER_IPTABLES $DOCKER_IPFORWARD $DOCKER_ADDRESSES $DOCKER_INSECURE_REGISTRIES

$ sudo vi /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
EnvironmentFile=-/etc/default/docker
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
</source>

== Run docker without sudo ==
Adding a user to docker group should be sufficient. However on apparmor, SELinux or a filesystem with ACL enabled additional permissions might be required in respect to access a <tt>socket file</tt>.
<source>
$ ll /var/run/docker.sock
srw-rw---- 1 root docker 0 Sep 6 12:31 /var/run/docker.sock=
# ACL
$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
group::rw-
other::---

# Grant ACL to jenkns user
$ sudo setfacl -m user:username:rw /var/run/docker.sock

$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
user:jenkins:rw-
group::rw-
mask::rw-
other::---
</source>
;References
* [https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo how-can-i-use-docker-without-sudo]

== References ==
*[https://www.weave.works/blog/my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies/ my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies]
*[https://github.com/moby/moby/pull/12228 PID1 in container aka tinit]
*[https://container-solutions.com/understanding-volumes-docker/ understanding-volumes-docker]

= Docker Enterprise Edition =
*[https://success.docker.com/article/compatibility-matrix Compatibility Matrix]
Components:
* Docker daemon (fka "Engine")
* Docker Trusted Registry (DTR)
* Docker Universal Control Plane (UCP)

= Docker Swarm =
== Swarm - sizing ==
;Universal Control Plane (UCP)
This is for only Enterpsise Edition
* ports managers, workers in/out

Hardware requirments:
* 8gb RAM for managers or DTR Docker Trsuted Registry
* 4gb RAM for workers
* 3gb free space

Performance Consideration (Timing)
<source>
Component Timeout(ms) Configurable
Raft consensus between manager nodes 3000 no
Gossip protocol for overlay networking 5000 no
etcd 500 yes
RethinkDB 10000 no
Stand-alone swarm 90000 no
</source>

Compatibility Docker EE
* Docker Engine 17.06+
* DTR 2.3+
* UCP 2.2+
== Swarm with single host manager ==
<source lang=bash>
# Initialise Swarm
docker swarm init --advertise-addr 172.31.16.10 #Iyou get SWMTKN-token
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

# Join tokens
docker swarm join-token manager #display manager join-token, run on manager
docker swarm join-token worker #display worker join-token, run on manager

# Join worker, run new-worker-node
# -> swarm cluster id <-> this part is mgr/wkr <- -> mgr node <-
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377

# Join another manager, run on new-manager-node
docker swarm join-token manager #run on primary manager if you wish add another manager
# in output you get a token. You notice that 1st part up to dash identifies Swarm cluster and the other part is role id.

# join to swarm (cluster), token will identify a role in the cluster manager or worker
docker swarm join --token SWMTKN-xxxx
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
This node joined a swarm as a worker.
</source>

Check Swarm status
<source lang=bash>
docker node ls
[cloud_user@ip-172-31-16-10 swarm-manager]$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * ip-172-31-16-10.mylabserver.com Ready Active Leader 18.09.0
vlw7te728z7bvd7ulb3hn08am ip-172-31-16-94.mylabserver.com Ready Active 18.09.0

docker system info | grep -A 7 Swarm
Swarm: active
NodeID: 641bfndn49b1i1dj17s8cirgw
Is Manager: true
ClusterID: 4jqxdmfd0w5pc4if4fskgd5nq
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
</source>

;Troubleshooting
<source lang=bash>
sudo systemctl disable firewalld && sudo systemctl stop firewalld # CentOS
sudo -i; printf "\n10.0.0.11 mgr01\n10.0.0.12 node01\n" >> /etc/hosts # Add nodes to hosts file; exit
</source>
== Swarm cluster ==
<source lang=bash>
docker node update -availability drain [node] #drain services for Manager Only nodes
docker service update --force [service_name] #force re-balance services across cluster

docker swarm leave #node leaves a cluster
</source>

== Locking / unlocking swarm cluster ==
Logs used by Swarm manager are encrypted on disk. Access to nodes gives access to keys that encrypt them. It further protects cluster as requires a unlocking key when restarting manager/nodes.
<source lang=bash>
docker swarm init --auto-lock=true #initialize with
docker swarm update --auto-lock=true #update current swarm
# both will produce unlock token STKxxx
docker swarm unlock #it'll ask for the unlock token
docker swarm update --auto-lock=false #disable key locking
</source>

If you have access to a manager you can always get unlock key using:
<source lang=bash>
docker swarm unlock-key
</source>

Key management
<source lang=bash>
docker swarm unlock-key --rotate #could be in a cron
</source>
== Backup and restore swarm cluster ==
This priocess describes how to backup whole cluster configuration so can be restored on a new set of servers.

Create docker apps running across swarm
<source lang=bash>
docker service create --name bkweb --publish 80:80 --replicas 2 httpd
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
q9jki3n2hffm bkweb replicated 2/2 httpd:latest *:80->80/tcp

$ docker service ps bkweb #note containers run on 2 different nodes
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE
j964jm1lq3q5 bkweb.1 httpd:latest server2c.mylabserver.com Running Running about a minute ago
jpjx3mk7hhm0 bkweb.2 httpd:latest server1c.mylabserver.com Running Running about a minute ago
</source>

;Backup state files
<source lang=bash>
sudo -i
cd /var/lib/docker/swarm
cat docker-state.json #contains info about managers, workers, certificates, etc..
cat state.json
sudo systemctl stop docker.service

# Backup swarm cluster, this file can be then used to recover whole swarm cluster on another set of servers
sudo tar -czvf swarm.tar.gz /var/lib/docker/swarm/

#the running docker containers should be brought up as they were before stopping the service
systemctl start docker
</source>

Recover using swarm.tar backup
<source lang=bash>
# scp swarm.tar to recovery node - what it'd be a node with just installed docker
sudo rm -rf /var/lib/docker/swarm
sudo systemctl stop docker

# Option1 untar directly
sudo tar -xzvf swarm.tar.gz -C /var/lib/docker/swarm

# Option2 copy recursivly, -f override if a file exists
tar -xzvf swarm.tar.gz; cd /var/lib/docker
cp -rf swarm/ /var/lib/docker/

sudo systemctl start docker
docker swarm init --force-new-cluster # produces the exactly same token
# you should join all required nodes to new manager ip
# scale services down to 1, then scale up so get distributed to other nodes
</source>

== Run containers as a services ==
Docker container has a number limitation therefore running as a service where Cluster Manager: Swarm or Kubernetes manages networking, access, loadbalancing etc.. is a way to scale with ease. The service is using eg. mesh routing to deal with access to containers.

Swarm nodes setup 1 manager and 2 workers
<source lang=bash>
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.0
</source>

Create and run a service
<source lang=bash>
docekr pull httpd
docker service create --name serviceweb --publish 80:80 httpd
# --publish|-p -expose a port on all containers in the running cluster

docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vt0ftkifbd84 serviceweb replicated 1/1 httpd:latest *:80->80/tcp

docker service ps serviceweb #show nodes that a container is running on, here on mgr-1 node
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
e6rx3tzgp1e5 serviceweb.1 httpd:latest swarm-mgr-1.example.com Running Running about
</source>

When running as a service even if a container runs on a single node (replica=1) the container can be accessed from any of swarm nodes. It's because service exposed port has been exposed to extended mesh private network that the container is running on.
<source lang=bash>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-mgr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-2.example.com
<html><body><h1>It works!</h1></body></html>
</source>

Service update, can be done to limits, volumes, env-variables and more...
<source lang=bash>
docker service scale devweb=3 #or
docker service update --replicas 3 serviceweb #--detach=false shows visual progress in older versions, default in v18.06
serviceweb
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged

# Limits(soft limit) and reservations(hard limit), this causes to start new services(containers)
docker service update --limit-cpu=.5 --reserve-cpu=.75 --limit-memory=128m --reserve-memory=256m serviceweb
</source>

== Templating service names ==
This allows to control eg. hostname in a cluster. Useful for big clusters to easier identify services where they run from hostname.
<source lang=bash>
docker service create --name web --hostname"{{.Node.ID}}-{{.Service.Name}}" httpd
docker service ps --no-trunc web
docker inspect --format="{{}.Config.Hostname}" web.1.ab10_serviceID_cd
aa_nodeID_bb-web
</source>
== Node lables for task/service placement ==
<source lang=bash>
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.1

docker node inspect 641bfndn49b1i1dj17s8cirgw --pretty
ID: 641bfndn49b1i1dj17s8cirgw
Hostname: swarm-mgr-1.example.com
Joined at: 2019-01-08 12:16:56.277717163 +0000 utc
Status:
State: Ready
Availability: Active
Address: 172.31.10.10
Manager Status:
Address: 172.31.10.10:2377
Raft Status: Reachable
Leader: Yes
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 2
Memory: 3.699GiB
Plugins:
Log: awslogs, fluentd, gcplogs, gelf, journald, json-file, local, logentries, splunk, syslog
Network: bridge, host, macvlan, null, overlay
Volume: local
Engine Version: 18.09.1
TLS Info:
TrustRoot:
-----BEGIN CERTIFICATE-----
MIIBajCCARCgAwIBAgIUKXz3wtc8OA8uzTo1pO86ko+PB+EwCgYIKoZIzj0EAwIw
..
-----END CERTIFICATE-----
Issuer Subject: MBMxETAPBgNVBAMTCHN3YX.....h
Issuer Public Key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEy......==
</source>

Apply label to a node
<source lang=bash>
docker node update --label-add node-env=testnode r8h7xmevue9v2mgysmld59py2
docker node inspect r8h7xmevue9v2mgysmld59py2 --pretty | grep -B1 -A2 Labels
ID: r8h7xmevue9v2mgysmld59py2
Labels:
- node-env=testnode
Hostname: swarm-wkr-1.example.com
</source>

How to use it. Run a service with <code>--constraint</code> option that pins services to run on a node meeting given criteria. In our case to run on a node where <code>node.labels.node-env == testnode</code>. Note that all replicas are running on the same node unlike they'd be distributed across the cluster.
<source lang=bash>
docker service create --name constraints -p 80:80 --constraint 'node.labels.node-env == testnode' --replicas 3 httpd #node.role, node.id, node.hostname
zrk15vfdaitc1rvw9wqh2s0ot
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
[cloud_user@mrpiotrpawlak1c ~]$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
zrk15vfdaitc constraints replicated 3/3 httpd:latest *:80->80/tcp

[user@swarm-wkr-2 ~]$ docker service ps constraints
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
y5z4mt99uzpo constraints.1 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
zqbn4ips969q constraints.2 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
vnb10jcs2915 constraints.3 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
</source>

== Scaling services ==
These commands be issued on a manager node
<source lang=bash>
docker pull docker nginx
docker service create --name web --publish 80:80 httpd
docker service ps web #there is only 1 replica
docker service update --replicas 3 web #update to 3 replicas
docker service create --name nginx --publish 5901:80 nginx
elinks http://swarm-mgr-1.com:5901 #nginx website will be presented

# scale is equivalent to update --replicas command for a single or multiple services
docker service scale web=3 nginx=3
docker service ls
</source>

== Replicated services vs global services ==
;Global Replicated: mode runs at least one copy of a service on each swarm node, even if you join another node the service will coverage there as well. In global mode you cannot use <code>update --replicats</code> or <code>scale</code> commands. It is not possible to update the mode type.
;Replicated mode: allows for grater control and flexibility of running number of services.
<source lang=bash>
# creates a single service running across whole cluster in replicated mode
docker service create --name web --publish 80:80 httpd

# run in a global node
docker service create --name web --publish 5901:80 --mode global httpd
docker service ls #note distinct mode names: global and replicated
</source>

= Docker compose and deploy to Swarm =
Install
<source lang=bash>
sudo yum install epel
sudo yum install pip
sudo pip install --upgrade pip
# install docker CE or EE to avoid Python libs conflits
sudo pip install docker-compose

# Troubleshooting
## Err: Cannot uninstall 'requests'. It is a distutils installed project...
pip install --ignore-installed requests
</source>

Dockerfile
<source lang=bash>
cat >Dockerfile <<EOF
FROM centos:latest
RUN yum install -y httpd
RUN echo "Website1" >> /var/www/html/index.html
EXPOSE 80
ENTRYPOINT apachectl -DFOREGROUND
EOF
</source>

Docker compose file
<source lang=bash>
cat >docker-compose.yml <<EOF
version: '3'
services:
apiweb1:
image: httpd_1:v1
build: .
ports:
- "81:80"
apiweb2:
image: httpd_1:v1
ports:
- "82:80"
load-balancer:
image: nginx:latest
ports:
- "80:80"
EOF
</source>

Run docker compose, on the current node only
<source lang=bash>
docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating compose_apiweb2_1 ... done
Creating compose_apiweb1_1 ... done
Creating compose_load-balancer_1 ... done

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
14f8b6b10c2d nginx:latest "nginx -g 'daemon of…" 2 minutesUp 2 min 0.0.0.0:80->80/tcp compose_load-balancer_1
e9b5b37fe4e5 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:81->80/tcp compose_apiweb1_1
28ee22a8eae0 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:82->80/tcp compose_apiweb2_1

# Verify
curl http://localhost:81
curl http://localhost:82
curl http://localhost:80 #nginx

# Prep before deploying docker-compose to Swarm. Also images needs to be build before hand.
# Docker stack does not support building images
docker-compose down --volumes #save everything to storage volumes
Stopping compose_load-balancer_1 ... done
Stopping compose_apiweb1_1 ... done
Stopping compose_apiweb2_1 ... done
Removing compose_load-balancer_1 ... done
Removing compose_apiweb1_1 ... done
Removing compose_apiweb2_1 ... done
Removing network compose_default
</source>

;Deploy compose to Swarm
<source lang=bash>
docker stack deploy --compose-file docker-compose.yml customcompose-stack #customcompose-stack is a prefix for service name
Ignoring unsupported options: build
Creating network customcompose-stack_default
Creating service customcompose-stack_apiweb1
Creating service customcompose-stack_apiweb2
Creating service customcompose-stack_load-balancer

docker stack services customcompose-stack #or
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
k7wwkncov49p customcompose-stack_apiweb1 replicated 0/1 httpd_1:v1 *:81->80/tcp
nl0j5folpmha customcompose-stack_apiweb2 replicated 0/1 httpd_1:v1 *:82->80/tcp
x6p14gmpjyra customcompose-stack_load-balancer replicated 1/1 nginx:latest *:80->80/tcp

docker stack rm customcompose-stack #remove stack

</source>

= Selecting a Storage Driver =
Go to Docker version matrix to verify what drivers are supported on your platform. Changing storage driver is destructive and you loose all containers volumes. Therefore you need to export/backup then re-import after the storage driver change.

;CentOS
Device mapper is officialy supported on CentOS. It can be used on a disc as blockstorage it uses loopback adapter to provide that. Or can be blockstorage devive allowing Docker to mamange it.

<source lang=bash>
docker info --format '{{json .Driver}}'
docker info -f '{{json .}}' | jq .Driver
docker info | grep Storage

sudo touch /etc/docker/daemon.json
sudo vi /etc/docker/daemon.json #additional options are available
{
"storage-driver":"devicemapper"
}
</source>

Preserving any current images, requires export/backup and re-import after the storage driver change.
<source lang=bash>
docker images
sudo systemctl docker restart
ls -l /var/lib/docker/devicemapper #new location to storing images
</source>

Note, in <code>/var/lib/docker</code> new directory <code>devicemapper</code> has been created to store images from now on.

;Update 2019 - Docker Engine 18.09.1
<source>
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
</source>

= Selecting a logginig driver =
Available list of [https://docs.docker.com/config/containers/logging/configure/#supported-logging-driverslogging drivers] can be seen on Docker documentation page. Most popular are:
*none - No logs are available for the container and docker logs does not return any output.
*json-file - (default) the logs are formatted as JSON. The default logging driver for Docker.
*syslog - Writes logging messages to the syslog facility. The syslog daemon must be running on the host machine.
*journald - Writes log messages to journald. The journald daemon must be running on the host machine.
*fluentd - Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host machine.
*awslogs - Writes log messages to Amazon CloudWatch Logs.
*splunk - Writes log messages to splunk using the HTTP Event Collector.
*etwlogs - (Windows) Writes log messages as Event Tracing for Windows (ETW) events

<source lang=bash>
docker info | grep logging
docker container run -d --name <webjson> --logg-driver json-file httpd #per docker container setup
docker logs <testjson>

docker container run -d --name <web> httpd #start new container
docker logs -f _testweb_ #display standard-out logs
docker service log -f <web> #for swarm all container replicas logs
</source>

Enable syslog logginig driver
<source lang=bash>
sudo vi /etc/rsyslog.conf
#uncomment below
$ModLoad imudp
$UDPServerRun 514

sudo systemctl start rsyslog
</source>

Change logging driver. Then standard output won't be available after the change.
<source lang=bash>
{
"log-driver": "syslog",
"log-opts": {
"syslog-address": "udp://172.31.10.1"
}
}

sudo systemctl restart docker
docker info | grep logging
tail -f /var/log/messages #this will show all logging eg. access logs for httpd server
</source>
== Docker daemon logs ==
System level logs
<source lang=bash>
# CentOS
/var/messages | grep -i docker

# Ubuntu
sudo journalctl -u docker.service --no-hostname
sudo journalctl -u docker -o json | jq -cMr '.MESSAGE'
sudo journalctl -u docker -o json | jq -cMr 'select(has("CONTAINER_ID") | not) | .MESSAGE'
/var/log/syslog | grep -i docker
</source>

;Docker container or service logs
<source lang=bash>
docker container logs [OPTIONS] containerID #single container logs
docker service logs [OPTIONS] service|task #agregate logs across all cluster deployed container replicas
</source>

= Container life-cycle policies - eg. autostart =
<source lang=bash>
docker container run -d --name web --restart <on-failure|unless-stopped|no|none(default)|always> httpd
# --restart -restart on crash or exit 1 or service or system reboot
</source>
Definitions:
* always - it will restart container always, even if stopped manually, restarting docker-deamon will start container
* unless-stopped - it will restart container always unless stopped manually by <code>docker container stop</code>
* on-failure - restart if container exits with non-zero exit code

= Universal Control Plance - UCP =
It's an application what allow to see all operational details for Swarm cluster when using Docker EE editin. 30 days trial is available.

;Communication between Docker Engine, UCP and DTR (Docker Trusted Registry)
* over TCP/UDP - depends on a port, and whether a response is required, or if a message is a notification
* IPC - interprocess communication (intra-host), services on the same node
* API - over TCP, uses API directlyto query or update components in a cluster

;References
* [https://docs.docker.com/ee/ucp/ucp-architecture/ UCP architecture]

== Install/uninstall UCP <code>image: docker/ucp</code> ==
OS support:
* UCP 2.2.11 is supported running on RHEL 7.5 and Ubuntu 18.04

For labs purpose, we can use eg. <code>ucp.example.com</code> the domain <code>example.com</code> is included in UCP and DTR wildcard self-signed certificate.

Install on a manager node
<source lang=bash>
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_MGR_NODE_IP=172.31.101.248

docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.15 \
install --host-address=$UCP_MGR_NODE_IP --interactive --debug

# --rm :- because this container will be only transitinal container
# -it :- because installation we want interactive
# -v :- link the container with a file on a host
# --san :- add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com)
# --host-address :- IP address or network interface name to advertise to other nodes
# docker/ucp:2.2.11 :- image version
# --dns :- custom DNS servers for the UCP containers
# --dns-search :- ustom DNS search domains for the UCP containers
# --admin-username "$UCP_USERNAME" --admin-password "$UCP_PASSWORD" #seems these are not supported, although are in a guide
</source>

If not provided you will be asked for:
* Admin password during the process
* You may enter additional aliases (SANs) now or press enter to proceed with the above list:
** Additinall aliases: ucp ucp.example.com
DEBU[0062] User entered: ucp ucp.ciscolinux.co.uk
DEBU[0062] Hostnames: [host1c.mylabserver.com 127.0.0.1 172.17.0.1 172.31.101.248 ucp ucp.ciscolinux.co.uk]

You may want to add DNS entries in <code>/etc/hosts</code> for
* ''ucp'' or ''ucp.example.com'' pointing to manager public ip
* ''dtr'' or ''dtr.example.com'' pointing a worker node public IPs.

;Verify
* connect to https://ucp.example.com:443.
* <code>docker ps</code> should see a number of containers running now, they need to see each other therefore we used <code>hosts</code> entries to allow this.

;Uninstall
<source lang=bash>
docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp uninstall-ucp --interactive

INFO[0000] Your engine version 18.09.1, build 4c52b90 (4.15.0-1031-aws) is compatible with UCP 3.1.2 (b822777)
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: t0ltwwcw5tdbtjo2fxlzmj8p4
Do you want to proceed with the uninstall? (y/n): y
INFO[0000] Uninstalling UCP on each node...
INFO[0031] UCP has been removed from this cluster successfully.
INFO[0033] Removing UCP Services
</source>

== Install DTR Docker Trusted Repository <code>image: docker/dtr</code> ==
It's recommended for single core systems to wait 5 minutes after UCP deployment to relese more cpu cycle. You can see the load may peaking up at 1.0 using <code>w</code> command.

Connect to UCP service https://ucp.example.com, login with creds created. Uload a license.lic file.
Go to Admin Settings > Docker Trusted Registry > Pick one of UCP Nodes [worker]
You may disable TLS verification on self-signed certificate

Run a given command on a node you want to install DTR. <code>UCP_NODE</code> in lab environment can cause a few issues. For a convinience to avoid avoid port conflict :80,:443 use different node that UCP is instaled. Eg. dns ''user2c.mylabserver.com'' or private IP.
<source lang=bash>
export UCP_NODE=wkr-172.31.107.250 #for convinince, to avoid port conflict :80,:443 use worker IP
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_URL=https://ucp.example.com:443 #avoid using example.com to avoid SSL name validation issues
docker pull docker/dtr

# Optional. Download UCP public certificate
curl -k https://ucp.ciscolinux.co.uk/ca > ucp-ca.pem

docker container run -it --rm docker/dtr install \
--ucp-node $UCP_NODE --ucp-url $UCP_URL --debug \
--ucp-username $UCP_USERNAME --ucp-password $UCP_PASSWORD \
--ucp-insecure-tls # --ucp-ca "$(cat ucp-ca.pem)"

# --ucp-node :- hostname/IP of the UCP node (any node managed by UCP) to deploy DTR. Random by default
# --ucp-url :- the UCP URL including domain and port.
</source>

It will ask for if not specified:
* ucp-password: you know it from UCP installation step

Sygnificiant installation logs
<source lang=bash>
..
INFO[0006] Only one available UCP node detected. Picking UCP node 'user2c.labserver.com'
..
INFO[0006] verifying [80 443] ports on user2c.labserver.com
..
INFO[0000] Using default overlay subnet: 10.1.0.0/24
INFO[0000] Creating network: dtr-ol
INFO[0000] Connecting to network: dtr-ol
..
INFO[0008] Generated TLS certificate. dnsNames="[*.com *.*.com example.com *.dtr *.*.dtr]" domains="[*.com *.*.com 172.17.0.1 example.com *.dtr *.*.dtr]" ipAddresses="[172.17.0.1]"
..
INFO[0073] You can use flag '--existing-replica-id 10e168476b49' when joining other replicas to your Docker Trusted Registry Cluster
</source>

;Verify by logging in to https://dtr.example.com
DTR installation process above has also installed a number of containers on maanger/worker nodes named <code>ucp-agent</code> and number of containers on dedicated DTR node.
You can verify DTR by logging to https://dtr.example.com with UCP credentials <code>ucp-admin</code> and the same password if you haven't changed any commands above. Then you should be presented with registry.docker.io like theme. Any images stored there will be trusted from a perspective of our organisation.

;Verify by going to UCP https://ucp.example.com, admin settings > Docker Trusted Registry
[[File:Ucp-dtr-in-admin.png|none|400px|left|Ucp-dtr-in-admin]]

== Backup UCP and DTR configuration ==
This is build into UCP. The process is to start a special container to export UCP configuration to tar file. This can be run as <code>cron</code> job.

<source lang=bash>
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp backup > backup.tar
# --rm it's transitional container
# -i run interactivly

# At first run it will error with --id m79xxxxxxxxx, asking to re-run teh command with this id.

# Restore command
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp restore --id m79xxx < backup.tar
</source>

;DTR
Durign a backup DTR will not be available.
<source lang=bash>
docker container run --log-driver non --rm docker/dtr backup --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> > dtr-backup.tar

# will you be asked for:
# Choose a replica to back up from: enter

# Restore command
docker container run --log-driver non --rm docker/dtr restore --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> < dtr-backup.tar
</source>
== UCP RBAC ==
The main concept is:
* administrators can make changes to the UCP swarm/kubernetes, User Management, Orgainisation, Team and Roles
* users - range of access from Full Control of resources to no access

[[File:Ucp-rbac.png|500px|none|left|Ucp-rbac]]

Note that only Scheduler role allows access to Node to view nodes. Plus schedule workloads of course.

= UCP Client bundle =
UCP client bundle allows to export a bundle containing a certificate and environment settings that will poind docker-client to UCP in order to use a cluster, create images and services.

;Download bundle
# Create a user with priviliges that yuo wish docker-client to run as
# Download a client budle from User Profile > Client bundle > + New Client Bundle
# File <code>ucp-bundle-[username].zip will get downloaded</code> <p>
<source lang=bash>
unzip ucp-bundle-bob.zip
Archive: ucp-bundle-bob.zip
extracting: ca.pem
extracting: cert.pem
extracting: key.pem
extracting: cert.pub
extracting: env.sh
extracting: env.ps1
extracting: env.cmd

cat env.sh
export COMPOSE_TLS_VERSION=TLSv1_2
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH="$PWD"
export DOCKER_HOST=tcp://3.16.143.49:443
#
# Bundle for user bob
# UCP Instance ID t0ltwwcw5tdbtjo2fxlzmj8p4
#
# This admin cert will also work directly against Swarm and the individual
# engine proxies for troubleshooting. After sourcing this env file, use
# "docker info" to discover the location of Swarm managers and engines.
# and use the --host option to override $DOCKER_HOST
#
# Run this command from within this directory to configure your shell:
# eval $(<env.sh)

eval $(<env.sh) # apply ucp-bundle

docker images # to list UCP managed images
</source></p>
# <li value="4"> In my lab I had to update DOCKER_HOST from public IP to private IP </li>
Err: error during connect: Get https://3.16.143.49:443/v1.39/images/json: x509: certificate is valid for 127.0.0.1, 172.31.101.248, 172.17.0.1, not 3.16.143.49
<source lang=bash>
export DOCKER_HOST=tcp://172.31.101.248:443
</source>

# <li value="5"> Verify if you have permissions to create a service</li>
<source lang=bash>
docker service create --name test111 httpd
Error response from daemon: access denied:
no access to Service Create, on collection swarm
</source>

# <li value="6"> Add Grants to the user</li>
## Go to User Management > Granst > Create Grant
## Base on a Roles, select Full Control
## Select Subjects, All Users, select the user
## Click Create
# Re-run service create command that should succeed now. This service can be managed now also within UCP console.

= Docker Secure Registry | image: registry =
Docker provides a special docker image that can be used to manage docker imagages both internally or externally thus steps below include securing the access with SSL certificate.

Create certificate
<source lang=bash>
mkdir ~/{auth,certs}
# create self-signed certificate for Docker Repository
mkdir certs; cd _$ #cd to last argument in history
openssl req -newkey rsa:4096 -nodes -sha256 -keyout repo-key.pem -x509 -days 365 -out repo-cer.pem -subj /CN=myrepo.com
# trusted-certs docker client directory, docker client looks for trusted certs when conencting to reomote repo
sudo mkdir -p /etc/docker/certs.d/myrepo.com:5000 #port 5000 is a default port
sudo cp repo-cer.pem /etc/docker/certs.d/myrepo.com:5000/ca.crt
</source>
<code>ca.crt</code> is default/required CAroot trustcert file name, that the docker client (docker login API) uses when conencting to remote repository. In our case we trust any cert signed by CA=ca.crt when connecting to myrepo.com:5000 as same certs (selfsigned), got installed in <code>repository:2</code> container via <code>-v /certs/</code> option.

Optional for development purposes to add doamin ''myrepo.com'' to hostfile binding to local interface ip address.
<source lang=bash>
sudo -i; echo "172.16.10.10 myrepo.com" >> /etc/hosts; exit
</source>

Optional add insecure-registry entry
<source>
sudo vi /etc/docker/deamon.json
{
"insecure-registries" : [ "myrepo.com:5000"]
}
</source>

Pull special Docker Registry image
<source lang=bash>
mkdir -p ~/auth #authentication directory, used when deploying local repository
docker pull registry:2
docker run --entrypoint htpasswd registry:2 -Bbn reg-admin Passw0rd123 > ~/auth/htpasswd
# -Bbn -parameters
# reg-admin -user
# Passw0rd123 -password string for basic htpasswd authentication method, the hashed password will be displayed to STDOUT

$ cat ~/auth/htpasswd
reg-admin:$2y$05$DnTWDHp7uTwaDrw4sXpUbuDDIlLwu3c8MEMsHPjK/AcUMdK/TD6fO
</source>

Run Registry container
<source lang=bash>
cd ~
docker run -d -p 5000:5000 --name myrepo \
-v $(pwd)/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/repo-cer.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/repo-key.pem \
-v $(pwd)/auth:/auth \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
registry:2
# -v -indicate where our certificates will be mounted within a container
# -e REGISTRY_HTTP_TLS_CERTIFICATE -path to cert inside the container
# -v $(pwd)/auth:/auth -mounting authentication directory where a file with password is
# -e REGISTRY_AUTH htpasswd -setting up to use 'htpasswd' authentication method
# registry:2 -image name, positinal params
</source>

Verify
<source lang=bash>
docker pull alpine
docker tag alpine myrepo.com:5000/aa-alpine #create a tagged image (copy) on a local filesystem,
# it must be prefixed with the private repo name '/' image name you want to upload as

docker logout # if logged in to another repository
docker login myrepo.com:5000/aa-alpine #login to a repository that runs as a container, stays login untill logout/reboot
docker login myrepo.com:5000/aa-alpine --username=rep-admin --password Passw0rd123
docker push myrepo.com:5000/aa-alpine

docker image rmi alpine myrepo.com:5000/aa-alpine #delete image stored locally
docker pull myrepo.com:5000/aa-alpine #pull image from a container repository

# List private-repository images
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/_catalog
{"repositories":["aa-alpine"]}

wget --no-check-certificate --http-user=reg-admin --http-password=password https://myrepo.com:5000/v2/_catalog
cat _catalog
{"repositories":["my-alpine","myalpine","new-aa-busybox"]}

# List tags
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/tags/list
{"name":"myalpine","tags":["latest"]}
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/manifests/latest #entire image metadata
</source>

Note. There is no easy way to delete images from repository:2 container.

= Docker push =
;Login to a docker repository
<source lang=bash>
docker info | grep -B1 Registry #check if you are logged in to docker.hub repository
WARNING: No swap limit support
Registry: https://index.docker.io/v1/

docker login

docker info | grep -B1 Registry
Username: pio2pio
Registry: https://index.docker.io/v1/
</source>

; Tag and push an image
<source lang=bash>
# docker tag local-image:tagname new-repo:tagname #create a local copy of an image
# docker push new-repo:tagname

docker pull busybox
docker --tag busybox:latest pio2pio/testrepo
docker push pio2pio/testrepo
The push refers to repository [docker.io/pio2pio/testrepo]
683f499823be: Mounted from library/busybox
latest: digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510
size: 527
</source>

; Docker Content Trust
All images are implicitly trusted by your Docker daemon. Buy can set that ONLY signed images are allowed. You can configure your systems for trusting only image tags that have been signed.
<source lang=bash>
export=DOCKER_CONTENT_TRUST=1 #enable system to sign an image during push process
docker build -t myrepo.com:5000/untrusted.latest
docker push myrepo.com:5000/untrusted.latest
...
No tag specified, skipping trust metadata push
# 2nd attempt, with a tag specified now
docker push myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority

docker pull myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority
</source>

;Errors explained:
Err: No tag specified, skipping trust metadata push<br />
* Explenation: When image gets signed is signed by a tag. Thereofre if you skip a tag it won't get signed and metada get skipped.
Err: error contacting notary server: x509: certificate signed by unknown authority
* when uploading the image gets uploaded, but it is not trusted becasue signed with self-signed CA
* when downloading, and <code>DOCKER_CONTENT_TRUST=1</code> is enabled, the image cannot be downloaded because is untrusted

= Theory =
== What is a docker ==
Docker is a container runtime platform, where Swarm is a container orchestration platform.

== Security ==
=== Mutually Authenticated TLS ===
Docker Swarm is ''secure by default'', it means all communication is encrypted. ''Mutually Authenticated TLS'' is the implementation was chosen to secure that communication. Any time a swarm is inicialised, a self-signed CA is generated and issues certificates to every node (mgr or wkr) to facilicate registration (join mgr or wkr) and latter those secure communications. It's transient container brought up to generate CA certs every time a cert is needed. MTLS communication is between Managers and Workers.

== [[Linux Namespaces and Control Groups]] ==

== Difference between docker attach and docker exec ==
;Attach
The docker attach command allows you to attach to a running container using the containers ID or name, either to view its ongoing output or to control it interactively. You can attach to the same contained process multiple times simultaneously, screen sharing style, or quickly view the progress of your detached process.

The command docker attach is for attaching to the existing process. So when you exit, you exit the existing process.

If we use docker attach, we can use only one instance of shell. So if we want open new terminal with new instance of container's shell, we just need run docker exec

If the docker container was started using /bin/bash command, you can access it using attach, if not then you need to execute the command to create a bash instance inside the container using exec. Attach isn't for running an extra thing in a container, it's for attaching to the running process.

To stop a container, use CTRL-c. This key sequence sends SIGKILL to the container. If --sig-proxy is true (the default),CTRL-c sends a SIGINT to the container. You can detach from a container and leave it running using the CTRL-p CTRL-q key sequence.

;exec

"docker exec" is specifically for running new things in a already started container, be it a shell or some other process. The docker exec command runs a new command in a running container.

The command started using docker exec only runs while the containerâ€™s primary process (PID 1) is running, and it is not restarted if the container is restarted.

exec command works only on already running container. If the container is currently stopped, you need to first run it. So now you can run any command in running container just knowing its ID (or name):

<source>
docker exec <container_id_or_name> echo "Hello from container!"
docker run -it -d shykes/pybuilder /bin/bash
</source>

The most important here is the -d option, which stands for detached. It means that the command you initially provided to the container (/bin/bash) will be run in background and the container will not stop immediately.

= Dockerfile - python =
* [https://luis-sena.medium.com/creating-the-perfect-python-dockerfile-51bdec41f1c8 perfect python dockerfile] Medium

= References =
*[https://docs.docker.com/v1.8/installation/ubuntulinux/ Ubuntu installation] official website
*[https://docs.docker.com/engine/admin/systemd/ PROXY settings for systemd]
*[http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/ Docker RUN vs CMD vs ENTRYPOINT]
*[https://vsupalov.com/docker-arg-vs-env/ docker ARG vs ENV]
*[https://www.fromlatest.io/#/ Docker online lintel]
*[https://hub.docker.com/r/portainer/portainer/ portainer] Monitor your containers via Web GUI
*[https://treescale.com/ treescale.com] Free private Docker registry

Terraform

2025-09-01T06:02:26Z

Pio2pio: /* Install terraform */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv --depth=1
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install v1.12.1
tfenv use v1.12.1
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory. Or set an environment variable. Then rerun <code>terraform init</code> to save providers into shared (cache) directory.

<source lang=terraform>
# Option 1.
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Option 2.
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache

# Create the cache directory
mkdir $HOME/.terraform.d/plugin-cache

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Run <code>terraform init</code>.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
tfautomv -dry-run
tfautomv -show-analysis
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

HashiCorp/Vagrant

2025-09-01T05:59:25Z

Pio2pio: /* WIP DevOps workstation */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= <code>box</code> (image) management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= <code>box</code> (image) migration =
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Copy the <tt>.box</tt> file and restore
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Managing snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bash_logout and .profile to eval ssh-agent and kill on exit
*java Oracle
*vim install
*vundle install
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

Linux shell/Productivity tools

2025-08-29T06:10:32Z

Pio2pio:

DNS

2025-08-26T21:09:58Z

Pio2pio: /* AWS */

This is a source of general information about Domain Name System aka DNS.

The DNS server stores different types of resource records used to resolve names, records like:
*'''A''' - Address record - returns a 32-bit IPv4 address, most commonly used to map hostnames to an IP address of the host
*'''NS''' - Name server record - an authoritative name server, delegates a DNS zone to use the given authoritative name servers
*'''CNAME''' - Canonical name record - the canonical name (or Fully Qualified Domain Name) for an alias; Alias of one name to another: the DNS lookup will continue by retrying the lookup with the new name. Used when multiple services have the single network address, but each service has its own entry in DNS
*'''MX''' - mail exchange record; maps a domain name to a list of mail exchange servers (MTA) for that domain
*'''SRV''' - Service Locator, multi line record of form of eg in AWS <code>[priority] [weight] [port] [server host name]</code>, multiline must start with <code>_</code> a new line delimiter
*'''SOA''' - Start of [a zone of] authority record - Specifies authoritative information about a DNS zone, including the primary name server, the email of the domain administrator, the domain serial number, and several timers relating to refreshing the zone.
*'''PTR''' - Pointer record - pointer to a canonical name. Unlike a CNAME, DNS processing stops and just the name is returned. The most common use is for implementing reverse DNS lookups

The <code>ipconfig /displaydns</code> command displays all of the cached DNS entries on a Windows computer system.

= <code>/etc/hosts</code> =
<code>dig</code> (domain information groper) and <code>nslookup</code> (query Internet name servers interactively) are tools that query name servers. Unless a specific name server is specified as a commandline argument they will query the name server(s) found in <code>/etc/resolv.conf</code>. They simply don't look at alternative sources of host information such as the <code>/etc/hosts</code> file or other sources specified in <code>/etc/nsswitch.conf</code>.

To force all dns queries through dnsmasq on your host, the <code>/etc/resolv.conf</code> there should point to dnsmasq, i.e. it should look like:
<source lang=bash>
#/etc/resolv.conf on sun
nameserver 127.0.0.1
</source>

Hosts file is part of <tt>Name Service Switch</tt>. Configured at
<source lang=bash>
$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: compat systemd
group: compat systemd
shadow: compat
gshadow: files

hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files
</source>

Example entries in <code>/etc/hosts</code>
<source lang=bash>
10.10.11.11 echo-1.service.k8s.acme.cloud # via app-service LoadBalancer
10.10.22.22 k8s.acme.cloud echo-1.ingress.k8s.acme.cloud # via ingress-service (k8s entry point)
</source>

can be verified using <code>getent</code> utility, to get entries from Name Service Switch libraries
<source lang=bash>
$ getent hosts 10.10.11.11
10.10.11.11 echo-1.service.k8s.acme.cloud
$ getent hosts echo-1.ingress.k8s.acme.cloud
10.10.22.22 k8s.acme.cloud echo-1.ingress.k8s.acme.cloud
</source>

= flush dns =
<source lang=bash>
sudo systemctl is-active systemd-resolved.service
# -> active

# Ubuntu 18.04, 20.04
resolvectl statistics # show statistics, the same output as 'systemd-resolve --statistics'
sudo systemd-resolve --statistics # or --reset-statistics - resets resolver statistics

sudo systemd-resolve --flush-caches # Flush Ubuntu DNS Cache - Ubuntu <22.04 (old)
resolvectl flush-caches # Flush Ubuntu DNS Cache - Ubuntu 22.04

sudo systemctl restart nscd # Other distros, eg arch Linux

# Resolve a name without using local cache
sudo systemd-resolve --flush-caches
resolvectl flush-caches

systemd-resolve --statistics | grep 'Current Cache Size' # -> Current Cache Size: 0
dig +short tvp.info @8.8.8.8
systemd-resolve --statistics | grep 'Current Cache Size' # -> Current Cache Size: 0
dig +short tvp.info
systemd-resolve --statistics | grep 'Current Cache Size' # -> Current Cache Size: 1

# Display cached dns entries
sudo killall -USR1 systemd-resolved # it doesn't stop the service, it tells systemd-resolved to write all the current cache entries to the system log
journalctl -u systemd-resolved # list the cached entries from the log

## Oneliner
sudo killall -USR1 systemd-resolved; journalctl -u systemd-resolved --since "5s ago"
</source>

= Netplan =
Netplan is the default network management tool on Ubuntu 18.04, replacing the /etc/resolv.conf and /etc/network/interfaces configuration files that have been used to configure the network in the previous Ubuntu versions.

Back in the days, whenever you wanted to configure DNS resolvers in Linux you would simply open the /etc/resolv.conf file, edit the entries, save the file and you are good to go. This file still exists but it is a symlink controlled by the systemd-resolved service and should not be edited manually.

{{Note|Info As of 18/05/2020 Network Manager doesn’t respect the Netplan option nameservers: addresses [8.8.8.8,8.8.4.4] option even when you specify dhcp4-overrides: use-dns: false it still uses (and give priority to) the default DHCP DNS servers. This renders any custom DNS servers redundant. The only way around this AFAIK is to specify the Ethernet connection as static.}}

<source lang=bash>
sudo vi /etc/netplan/enp0s3.yaml
network:
version: 2
renderer: NetworkManager
ethernets:
enp0s3:
dhcp4: false
addresses: [192.168.1.114/24]
gateway4: 192.168.1.1
nameservers:
addresses: [8.8.8.8, 8.8.4.4]

# Using this method you’ll lose the Network Manager GUI and network icon and let Netplan to manage all devices
sudo netplan apply
systemd-resolve --status | grep 'DNS Servers' -A2
</source>

== DNS Management in Ubuntu 24.04: systemd-resolved vs. resolvconf ==
On modern Ubuntu systems, including 24.04, <code>systemd-resolved</code> has become the primary service for managing DNS resolution. It replaces the legacy <code>resolvconf</code> framework as the default, offering a more robust and integrated solution. For advanced users, the key takeaway is the shift in tooling and configuration from the traditional <code>/etc/resolv.conf</code> file to the <code>resolvectl</code> command and its integration with systemd.

=== The Modern Default: systemd-resolved ===
<code>systemd-resolved</code> acts as a local caching DNS stub resolver. It intercepts DNS queries from local applications and handles them intelligently, providing advanced features and better integration with modern network configurations.

'''Core Function:''' Provides local DNS caching, DNSSEC validation, and Link-Local Multicast Name Resolution (LLMNR).
'''How it Works:''' By default, <code>/etc/resolv.conf</code> is a symlink to a file managed by <code>systemd-resolved</code> (e.g., <code>/run/systemd/resolve/stub-resolv.conf</code>). This file typically points to the local stub resolver at <code>127.0.0.53</code>. Applications can also query <code>systemd-resolved</code> directly via D-Bus, bypassing <code>/etc/resolv.conf</code> entirely.
'''Key Features:'''

'''DNS Caching:''' Improves performance by caching previous lookups.

'''DNSSEC:''' Validates DNS records to protect against spoofing.

'''Per-Link Configuration:''' Intelligently handles DNS servers for multiple network interfaces (e.g., Ethernet, Wi-Fi, and a VPN connection simultaneously).

'''Management:''' The primary tool for interacting with this service is <code>resolvectl</code>. You can use <code>resolvectl status</code> to view current DNS servers and <code>resolvectl query <domain></code> to test resolution.

'''Integration:''' It integrates seamlessly with network management tools like Netplan and NetworkManager, which automatically feed it DNS server information received via DHCP or static configurations.

=== The Legacy Framework: resolvconf ===
<code>resolvconf</code> is a script-based framework designed to manage the contents of <code>/etc/resolv.conf</code>. It acts as an intermediary, collecting DNS information from various sources and writing it to the configuration file.

'''Core Function:''' To dynamically generate <code>/etc/resolv.conf</code> by aggregating DNS information from sources like DHCP clients, VPN software, and static network configurations.
'''Status on Modern Ubuntu:''' While the <code>resolvconf</code> package may still be installed for compatibility with older software, it is no longer the default manager of DNS resolution. On a standard Ubuntu 24.04 installation, <code>systemd-resolved</code> handles its responsibilities. If both are present, <code>systemd-resolved</code> typically takes precedence.

= References =
*[https://en.wikipedia.org/wiki/List_of_DNS_record_types List of DNS record types] Wikipedia

Ubuntu Setup

2025-08-23T12:31:45Z

Pio2pio: /* gnome-shell-system-monitor-applet - cpu, memory indicators */

If you are using Ubuntu for various Linux projects you will find that as it comes with pre installed with many packages. On the other hand installing just minimal version seems to be too extreme. Therefore I started maitaining a list of unnecessary packages and one liner to that removes them all. Please feel free to modify for your needs.

= Default partitioning =
On virtual systems schema below will be applied, eg on laptops:
:[[File:ClipCapIt-200620-131502.PNG]]

<source lang="bash">
#Eg. for 4G memory and 50G storage system

/dev/mapper/ubuntu--vg-root mount_point: /
/dev/mapper/ubuntu--vg-swapt_1
/dev/sda
/dev/sda1 (50G)

LVM VG ubuntu-vg, LV root as ext4
LVM VG ubuntu-vg, LV swapt_1 as swap

#Boot device:
/dev/mapper/ubuntu--vg-root
</source>
As a good handy practice you may create 100G virtual disk that you thin provision. Then create 2 PVs for root and swap partitions. Don't utilize all space at once but extend partitions when needed. This method eliminates adding new disks to VMs saving time and efforts.

Example LVM setup, here using 30G Physical Volume(99.9% used), 1 Volume Group and 2 Logical Volumes (root and swap).
<source lang="java">
$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 ubuntu-vg lvm2 a-- <29.93g 36.00m
$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
ubuntu-vg 1 2 0 wz--n- <29.93g 36.00m
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root ubuntu-vg -wi-ao---- 28.94g
swap_1 ubuntu-vg -wi-ao---- 976.00m
piotr@u18:~$
</source>

<source lang="java">
$ lsblk /dev/sda --fs
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
└─sda1 LVM2_member rP18Kb-Q12j-wjVf-C1iV-uy42-BUJD-aWFuO7
├─ubuntu--vg-root ext4 fad04a3b-5fa3-4a03-bbd6-24a93cda1eb3 /
└─ubuntu--vg-swap_1 swap 47cd084b-89b0-4cd5-bdb8-367238842ba1 [SWAP]
</source>

= List of unnecessary packages =
<source lang="bash">
sudo apt-get remove libreoffice-* #Remove LibreOffice
sudo apt-get remove unity-lens-* #This package contains photos scopes which allow Unity to search for local and online photos.
sudo apt-get remove shotwell* #Photo organizer
sudo apt-get remove simple-scan #Scanner software
sudo apt-get remove empathy* #Internet messaging ~13M
sudo apt-get remove thunderbird* #Email client ~61M
sudo apt-get remove unity-scope-gdrive #Google Drive scope for Unity ~116KB
sudo apt-get remove cheese* #Cheese Webcam Booth - webcam software
sudo apt-get remove brasero* #Brasero Disc Burner ~6.5MB
sudo apt-get remove gnome-bluetooth Package to manipulate bloototh devices using Gnome desktop ~2MB
sudo apt-get remove gnome-orca Orca Screen Reader -Provide access to graphical desktop environments via synthesised speech and/or refreshable braille
sudo apt-get remove unity-webapps-common #Amazon Unity WebApp integration scripts ~133KB
sudo apt-get remove ibus-pinyin #IBus Bopomofo Preferences - ibus-pinyin is a IBus based IM engine for Chinese ~1.4MB
sudo apt-get remove apt-get remove printer-driver-foo2zjs* #Reactivate HP LaserJet 1018/1020 after reloading paper ~3.2MB
</source>

= Remove unnecessary packages - one liner =
;Ubuntu 12, 14, 16
<source lang="bash">
$ sudo apt-get remove libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* unity-scope-gdrive cheese*\
brasero* gnome-bluetooth gnome-orca unity-webapps-common ibus-pinyin printer-driver-foo2zjs*
</source>

;Ubuntu 18. It's recommended to choose ''Minimal Install'', so most of packages below won't get installed.
<source lang="bash">
$ sudo apt-get purge libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* cheese* \
brasero* gnome-bluetooth gnome-orca ibus-pinyin printer-driver-foo2zjs* xul-ext-ubufox speech-dispatcher* \
rhythmbox* printer-driver-* mythes-en-us mobile-broadband-provider-inf* \
evolution-data-server* espeak-ng-data:amd64 bluez* ubuntu-web-launchers \
transmission-*
</source>
<source lang="bash">
sudo apt-get purge xul-ext-ubufox # Canonical FF customizations for u14,16,18,20
sudo apt-get remove gnome-mahjongg gnome-mines gnome-sudoku # games, works for u14,16,18,20
sudo apt-get remove gnome-video-effects gstreamer1.0-*
</source>

; XTREME
UnInstallant Ubuntu software notifier
<source lang="bash">
sudo apt-get remove update-notifier
</source>

= Uninstall locales - unused languages etc =
<source lang="bash">
sudo apt-get install localepurge
</source>

= Set apt-get to not install recommended and suggested packages =
<source lang="bash">
sudo bash -c 'cat > /etc/apt/apt.conf.d/01no-recommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF'
</source>

To see if apt reads this, enter this in command line (as root or regular user):
<source lang="bash">
apt-config dump | grep -e Recommends -e Suggests
</source>

= Install necessary packages =

Adobe Flash Player
sudo apt-get install flashplugin-installer

Java JRE
This will install the default verison Java for you distro plus Icedtea plugin for using Firefox with Java
sudo apt-get install default-jre icedtea-plugin

Unity Settings
sudo apt-get install unity-control-center

Opera

Add Opera repository <code>'''deb <nowiki>http://deb.opera.com/opera/</nowiki> stable non-free'''</code> to the apt-get source list in <code>/etc/apt/sources.list</code>. Then import a public PGP repository key.
<source lang="bash">
echo "deb http://deb.opera.com/opera/ stable non-free" | sudo tee -a /etc/apt/sources.list
wget -qO - http://deb.opera.com/archive.key | sudo apt-key add -
sudo apt-get update && sudo apt-get install opera
</source>
Silverlight

Pipelight has been released and we can use it for silverlight as a best alternative moonlight.
<source lang="bash">
sudo apt-add-repository ppa:ehoover/compholio
sudo apt-add-repository ppa:mqchael/pipelight
sudo apt-get update
sudo apt-get install pipelight
</source>

= GUI tools =
* [https://github.com/hluk/CopyQ/releases copyQ] clipboard manager
* VisualVM

= Customise Ubuntu =
==Fix Ubuntu Unity Dash Search for Applications and Files==
sudo apt-get install unity-lens-files unity-lens-applications #log out and log back in required

==Fix Ubuntu <17.10 missing Control Center==
sudo apt-get install unity-control-center --no-install-recommends

==Fix Ubuntu >18.04 missing System Settings==
sudo apt install gnome-control-center

==Remove background wallpaper ==
Tested on Ubuntu 14,16,18
<source lang="bash">
gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background draw-background false #disable
gsettings set org.gnome.desktop.background primary-color "#000000" #set to black
gsettings set org.gnome.desktop.background secondary-color "#000000" #set to black
gsettings set org.gnome.desktop.background color-shading-type "solid" #set solid colour
gsettings set org.gnome.desktop.background picture-uri file:///dev/null #remove wallpaper, not perfect but nothing worked in U15.10
gsettings set com.canonical.unity-greeter draw-user-backgrounds false #disable not worked

# Reset background picture to origin, U15.10
gsettings set org.gnome.desktop.background picture-uri file:///usr/share/backgrounds/warty-final-ubuntu.png

# Sets Unity greeter background, <17.04
gsettings set com.canonical.unity-greeter background /usr/share/backgrounds/warty-final-ubuntu.png
</source>

==Disable screen lock out==
<code>dconf</code> is legacy tool to configure <tt>gnome</tt> nowadays more modern way is to use <code>gsettings</code>.
<source lang=bash>
dconf write /org/gnome/desktop/screensaver/idle-activation-enabled false #gnome
dconf write /org/gnome/desktop/screensaver/lock-enabled false

# Unity - Ubuntu 14.04, 16.04
gsettings set org.gnome.desktop.session idle-delay 0 #disable the screen blackout:(0 to disable)
gsettings set org.gnome.desktop.screensaver lock-enabled false #disable the screen lock

# VirtualBox > Ubuntu 18.04 Disabling Xserver screen timeouts
xset s off # Xserver s parameter sets screensaver to off
xset s noblank # prevent the display from blanking
xset -dpms # prevent the monitor's DPMS energy saver from kicking in

# Gnome - Ubuntu 18.04 LTS, Settings > Power > Blank screen > set to: Never
gsettings get org.gnome.desktop.lockdown disable-lock-screen # verify status
gsettings set org.gnome.desktop.lockdown disable-lock-screen true # set disabled
gsettings get org.gnome.desktop.screensaver lock-enabled # verify status
gsettings set org.gnome.desktop.screensaver lock-enabled false # set disabled
dconf write /org/gnome/desktop/screensaver/lock-enabled false # set disbaled using dconf
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false # some say it's last resort :)

# Power management
gsettings set org.gnome.settings-daemon.plugins.power active true #set gnome to be the default power management run
gsettings set org.gnome.settings-daemon.plugins.power active false #turn off power management

# last resort as it was a bud in Ubuntu 11.10 with DPMS
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false
gsettings set org.gnome.desktop.session idle-delay 2400
</source>
Verify by navigating in <tt>dconf-editor</tt> to <tt>/org/gnome/desktop/screensaver/</tt>

==Change number of workspaces==
To get the current values:
<source lang=bash>
dconf read /org/compiz/profiles/unity/plugins/core/hsize
dconf read /org/compiz/profiles/unity/plugins/core/vsize
</source>

To set new values:
<source lang=bash>
dconf write /org/compiz/profiles/unity/plugins/core/hsize 2
# or
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ hsize 4
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ vsize 4
</source>

== Clenup motd messages ==
Ubuntu at login displays a number standard messages taking terminal space causing potential loosing context of previous operations.
<source lang="bash">
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-134-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

1 package can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Aug 31 12:11:28 2018 from 10.0.2.2
</source>

This is managed by files in <code>/etc/update-motd.d/</code>, so deleting them will remove clutter on a screen
<source lang="bash">
ls /etc/update-motd.d/
00-header 51-cloudguest 91-release-upgrade 98-fsck-at-reboot
10-help-text 90-updates-available 97-overlayroot 98-reboot-required

# Ubuntu Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1022-azure x86_64)
# Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)
sudo rm /etc/update-motd.d/{10-help-text,50-landscape-sysinfo,50-motd-news,51-cloudguest,80-livepatch,95-hwe-eol}
</source>

This cuts down to this message, Ubuntu 18.04 in AWS
<source lang="bash">
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)

0 packages can be updated.
0 updates are security updates.

Last login: Thu Jan 31 17:09:38 2019 from 10.10.11.11
</source>

= Useful setups =
== Image converter ==
nautilus-image-converter is a nautilus extension to mass resize or rotate images. It adds two context menu items in nautlius so you can right-click and choose "Resize Image" or "Rotate Image").
<source lang=bash>
# tested on Ubuntu 24.04 with Gnome
sudo apt-get install nautilus-image-converter

# Restart to see the new context menu
nautilus -q
</source>

== Call screen saver from a terminal to blank all screens ==
<source lang=bash>
# tested on Ubuntu 18.04 with Gnome
sudo apt-get install gnome-screensaver
gnome-screensaver-command -a #controls GNOME screensaver, -a activate (blank the screen)
</source>

== Create application launcher ==
;Ubuntu 18.04
<source lang=bash>
# Install the GNOME-panel toolset
sudo apt-get install --no-install-recommends gnome-panel

# Every user launcher
sudo gnome-desktop-item-edit /usr/share/applications/VisualVM.desktop --create-new

# Local user only, the filename by default is a Name-of-appication.desktop
gnome-desktop-item-edit ~/.local/share/applications --create-new
</source>

:[[File:ClipCapIt-190807-080016.PNG]]

;Ubuntu 19.10, 20.04
In above releases <code>gnome-desktop-item-edit</code> has been removed from the <code>gnome-panel</code> package, as an alternative <code>.desktop</code> files can be created manually.
<source lang=bash>
vi /usr/share/applications/APPNAME.desktop
[Desktop Entry]
Name=<NAME OF THE APPLICATION>
Comment=<A SHORT DESCRIPTION>
Exec=<COMMAND-OR-FULL-PATH-TO-LAUNCH-THE-APPLICATION>
Type=Application
Terminal=false
Icon=<ICON NAME OR PATH TO ICON>
NoDisplay=false
Keywords=<eg. sql>
</source>

It's optional but you may need to right click and set 'allow launching' with addition to set executable permissions. Usual locations of <code>.desktop</code> files are:
* <code>/usr/share/applications/</code>
* <code>/var/lib/snapd/desktop/applications/</code> for snap applications

== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet gnome-shell-system-monitor-applet] - cpu, memory indicators ==
System information such as memory usage, cpu usage, network rates and more can be displayed in the notification area in GNOME Shell.

System-monitor extensions:
[https://extensions.gnome.org/extension/120/system-monitor/ system-monitor] by paradoxxxzero on [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet github] supports Gnome-shell up to v40. It seems like abandoned project.
[https://extensions.gnome.org/extension/3010/system-monitor-next/ system-monitor-next] by mgalgs on [https://github.com/mgalgs/gnome-shell-system-monitor-applet github] supports Gnome-shell v40+, it's a fork of the above.

All extensions:
* https://extensions.gnome.org

{{Note|The current version of the browser Firefox is packaged as a snap version. One of the issues with this is that it cannot work with the Gnome Extensions website.}}

Install on Ubuntu 24.04 (June 2024)
<source lang=bash>
# Ubuntu version tested: v20/22/24 LTS
lsb_release -d
Description: Ubuntu 24.04.3 LTS

gnome-shell --version
GNOME Shell 46.0

# Install the Gnome-Shell-Extension & Manager
sudo apt install gnome-shell-extensions # Ubuntu 20.04 LTS already has this package, 24.04 needs installing it
sudo apt install gnome-shell-extension-manager # Ubuntu 22.04|24.04 LTS

# 1. Open `Extensions` app, turn "Use Extensions". It is already turned on in Ubuntu 24.04.3 LTS.
# 2. Open Browse tab > search for 'system-monitor-next' by mgalgs, click "Install".
# 3. "cpu/mem/net" indicators will appear in the system tray.

# Additional steps for Ubuntu < 24.04
sudo apt install gnome-tweaks # GUI to manage gnome-extensions
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
sudo apt install gnome-shell-extension-system-monitor # after requires log out

# Download the extension from
## https://extensions.gnome.org/extension/120/system-monitor/

# Never worked out how to use this direct download and install via 'gnome-extensions install <extension_name>'
## wget https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/archive/v38.zip
## gnome-extensions install <system-monitor@paradoxxx.zero.gmail.com>

# Enable extension using cli
gnome-extensions enable system-monitor-next@paradoxxx.zero.gmail.com
gnome-extensions list --user
clipboard-indicator@tudmotu.com
system-monitor-next@paradoxxx.zero.gmail.com
</source>
:[[File:ClipCapIt-210105-084527.PNG]]

=== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/issues/737#issuecomment-1230654455 Ubuntu 22.04 workaround for the OUTDATED extension] ===
{{Note|Workaround still needed in August 2022}}
<source lang=bash>
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
git clone https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet.git
cd gnome-shell-system-monitor-applet # commit b359d88 verified
vi system-monitor@paradoxxx.zero.gmail.com/metadata.json
# | change "version": -1 to "version": 42
make install
# log out and back in (required)
</source>

= Snapd - Chromium =
Recently in U19+ Chromium get installed via snapd package. This is classic installation that has limited access to only a certain directories. It happen that when working with AWS we need get access to <code>~/.ssh</code> folder to get ec2 machine password. This folder is denied, but we can bind mount <code>~/.ssh</code> folder into the snap container directory:
<source lang=bash>
$ snap list chromium
Name Version Rev Tracking Publisher Notes
chromium 86.0.4240.111 1373 latest/stable canonical✓ -

# cd to chromim $HOME dir
mkdir ~/snap/chromium/current/.ssh
sudo mount --bind ~/.ssh/ ~/snap/chromium/current/.ssh
</source>

= Screen shooting =
In Ubuntu 20.04 Shutter is not a part of default repositories. It can be added via PPA:
<source lang=bash>
sudo add-apt-repository -y ppa:linuxuprising/shutter
sudo apt-get install shutter
</source>

= Audio - [https://rastating.github.io/setting-default-audio-device-in-ubuntu-18-04/ set defaults] =
For preserving settings using GUI you can install [https://freedesktop.org/software/pulseaudio/pavucontrol/ PulseAudio Volume Control] <code>pavucontrol</code>.
<source lang=bash>
# install
sudo apt install pavucontrol # Ubuntu 20.04
# run
pavucontrol
</source>

Set default output/input device. In Ubuntu PulseAudio is used to control audio devices. It contains following configuration files
<source lang=bash>
/etc/pulse/default.pa # system wide
~/.config/pulse # user configuration
</source>

Set defaults
<source lang=bash>
# List devices: modules, sinks, sources, sink-inputs, source-outputs, clients, samples, cards
# sinks - outputs, sink-inputs, sources - all input/output including RUNNING and SUSPENDED devices
$ pactl list short sources | column -t
5 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_5__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
6 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_4__sink.monitor module-alsa-card.c s16le 2ch 48000Hz RUNNING
7 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_3__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
8 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
9 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__source module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
10 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_6__source module-alsa-card.c s16le 4ch 48000Hz SUSPENDED
15 alsa_output.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.analog-stereo.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz SUSPENDED
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz SUSPENDED
20 alsa_input.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.iec958-stereo module-alsa-card.c s16le 2ch 48000Hz RUNNING

# Set defaut output device. Tab autocompletion should work (U20.04)
pactl set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output
# Set defaut input device
pactl set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Test, play some audio then run. IDLE - means in use
pactl list short sources | column -t | grep -e RUNNING -e IDLE
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz IDLE
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz RUNNING
</source>

Make it permanent by setting default device in PulseAudio system configuration file
<source lang=bash>
# Output device
OUTPUT_DEVICE=alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-sink$ output/\1 ${OUTPUT_DEVICE}/g" /etc/pulse/default.pa # remove '-i' to test before apply
# Input device
INPUT_DEVICE=alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-source$ input/\1 ${INPUT_DEVICE}/g" /etc/pulse/default.pa

vi /etc/pulse/default.pa # make sure lines below are in place
### Make some devices default
set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Delete local user profile and restart system, after boot new defaults should be set
rm -r ~/.config/pulse

# After reboot, defaults should be set
cat ~/.config/pulse/*default*
alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
</source>

;Troubleshooting
PulseAudio cli
<source lang=bash>
pacmd
>>> help # lists all available commands

pulseaudio --check # Check if any pulseaudio instance is running. It normally prints no output, just exit code. 0 means running
pulseaudio --kill # kill, then --start
pulseaudio -D # start pulseaudio as a daemon
# | using /etc/pulse/daemon.conf

# Pulseaudio is a user service
systemctl --user restart pulseaudio.service
systemctl --user restart pulseaudio.socket
</source>

I have a port replicator Dell D-6000, that gets randomly disconnected causing switching audio to new connected device - means itself. As workaround commenting out lines below stops this behaviour.
<source lang=bash>
vi /etc/pulse/default.pa
### Use hot-plugged devices like Bluetooth or USB automatically (LP: #1702794)
# .ifexists module-switch-on-connect.so
# load-module module-switch-on-connect
# .endif
</source>

= Input devices =
Motivation is to enable horizontal scrolling in Ubuntu 20.04 using Perixx Gamig Mouse Mx2000
<source lang=bash>
xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Holtek USB Gaming Mouse id=11 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Mouse id=14 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Touchpad id=15 [slave pointer (2)]
⎜ ↳ TPPS/2 Elan TrackPoint id=19 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Sleep Button id=8 [slave keyboard (3)]
↳ CHICONY HP Basic USB Keyboard id=9 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=10 [slave keyboard (3)]
↳ Integrated Camera: Integrated C id=12 [slave keyboard (3)]
↳ Integrated Camera: Integrated I id=13 [slave keyboard (3)]
↳ sof-hda-dsp Headset Jack id=16 [slave keyboard (3)]
↳ Intel HID events id=17 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=18 [slave keyboard (3)]
↳ ThinkPad Extra Buttons id=20 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=21 [slave keyboard (3)]

# test mouse aka Virtual core pointer
xinput test 11
motion a[0]=2023 # <- cursor moving
motion a[0]=2024 a[1]=1411
motion a[3]=19545 # <- scroll down
button press 5
button release 5

# test 'virtual core keyboard' aka additional programmable buttons
## '10' - this virtual keyboard for all buttons except the scrolling wheel
xinput test 10
key press 37
key press 38

## '21' - this is scrolling wheel buttons left/right, not scrolling itself
xinput test 21
key press 248
key release 248
</source>

# List of properties of a device. We want to see 'horizontal scrolling wheel buttons'
<source lang=bash>
$ xinput list-props 21
Device 'Holtek USB Gaming Mouse':
Device Enabled (169): 1
Coordinate Transformation Matrix (171): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
libinput Send Events Modes Available (291): 1, 0
libinput Send Events Mode Enabled (292): 0, 0
libinput Send Events Mode Enabled Default (293): 0, 0
Device Node (294): "/dev/input/event10"
Device Product ID (295): 1241, 41063
</source>

=References=

[[Category:linux]]

HashiCorp/Vagrant

2025-08-23T11:34:39Z

Pio2pio: /* Images aka box management */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= <code>box</code> (image) management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= <code>box</code> (image) migration =
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Copy the <tt>.box</tt> file and restore
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Managing snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

HashiCorp/Vagrant

2025-08-23T11:32:23Z

Pio2pio: /* Box images advanced */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Managing snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

HashiCorp/Vagrant

2025-08-22T11:29:03Z

Pio2pio: /* Create box from current project (package a box) */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Managing snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Copy the <tt>.box</tt> file and restore
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

HashiCorp/Vagrant

2025-08-22T11:23:52Z

Pio2pio: /* Snapshots */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Managing snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Re-ristribute the <tt>.box</tt> file then restore it.
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

Virtualbox

2025-08-22T09:15:38Z

Pio2pio: /* Install */

= Install =
<source lang=bash>
# 2025 onwards Ubuntu 24.04.3
wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --dearmor
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/oracle-virtualbox-2016.gpg] https://download.virtualbox.org/virtualbox/debian noble contrib" | sudo tee /etc/apt/sources.list.d/virtualbox.list
sudo apt install virtualbox-7.2

# Install extension pack
## Open VirtualBox Manager, Go to File > Preferences > Extensions > Add ...

# Before 2025
echo "deb [arch=amd64] https://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib" | sudo tee -a /etc/apt/sources.list
wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-6.1
sudo apt-mark hold virtualbox-6.1
sudo apt-mark showhold
sudo apt-mark unhold virtualbox-6.1
</source>

= Resize disks in VirtualBox with Snapshots =
It is quite straightforward to resize a disk in VirtualBox as stated here and there. It becomes tricky though if the virtual machine,aka VM, has snapshots attached. The virtual disk thus is persisted across multiple VHD files, and the old trick will generally take not effect. This is also a [https://www.virtualbox.org/ticket/9103 known bug] hanging there for more than three years.

The suggested approach is to delete all snapshots and wait patiently for VirtualBox Manager to merge all the VHD files for you. It is a painfully lengthy process, so I decide to take a shortcut.

# First, shutdown the VM and backup the whole virtual machine folder.
# Then modify the size of all .vdi files in the root of the VM and Snapshots subdirectory.
<source lang="bash">
VBoxManage modifyhd "Windows 8.1.vdi" --resize 81920
for x in Snapshots/*.vdi ; do VBoxManage modifyhd $x --resize 81920 ; done
</source>
Startup the VM, and you will see the unallocated space in the Disk Management utility.

= Resize .vmdk disk on Linux =
Convert .vmdk format to .vdi and then resize. You can change format back after the resizing.
<source lang="bash">
VBoxManage clonehd "ubuntu-xenial-16.04-cloudimg.vmdk" "ubuntu-xenial-16.04-cloudimg.vdi" --format vdi #.vmdk -> .vdi
VBoxManage clonehd "ubuntu-xenial-16.04-cloudimg.vdi" "ubuntu-xenial-16.04-cloudimg.vmdk" --format vmdk #.vdi -> .vmdk
</source>

= Resize .vdi disk on Windows =
<source>
cd "C:\Program Files\Oracle\VirtualBox"
VBoxManage.exe modifyhd "C:\Users\piotr\VirtualBox VMs\vm-ubuntu64\vm-ubuntu64.vdi" --resize 20480
</source>
Note it also can resize VHD (Hyper-V) file formats.

= Vagrant note =
* OS: Ubuntu 16.04 LTS
* Vagrant version 2.1.1
* VirtualBox: 5.1.34_Ubuntu

Steps I have taken to resize Vagrant Ubuntu disk
# Stopped VM
# In settings removed attached drive "ubuntu-xenial-16.04-cloudimg.vmdk"
# Converted .vmdk into .vdi
# Attached "ubuntu-xenial-16.04-cloudimg.vdi" making sure that
#* Controller: SCSI Controller
#* Hard disk is attached to: SCSI Port 0, otherwise may throw error "no bootable medium found"

= Shrink unused space on virtual drive =
Hypervisor: Virtualbox, VMware

Virtual disks can be shrink as long as they are ext3 or ext4 file systems.
<source lang="bash">
$ vagrant ssh
$ sudo dd if=/dev/zero of=wipefile bs=1024x1024; rm wipefile
</source>

The above command is simply writing zero bytes to the wipefile in chunks of 1024 bytes until there is no disk space left in your VM’s disk. Then it is removing the wipefile. This basically leaves all those excess bytes zero’d out.

This is necessary because the shrink/compaction tools provided by either VMWare or VirtualBox both have no way of identifying space they can free up in the disks unless they are zero’d out.

With VirtualBox the only way I was able to shrink the disk image was to clone it to a smaller copy using the following command:
<source lang="bash">
$ VboxManage clonehd name-of-original-vm.vdi name-of-clone-vm.vdi
</source>

Once you have cloned the vdi you can then import it into the VM through VirtualBox and get rid of the original vdi.

With VMware you can shrink the vmdk disk by doing the following:
<source lang="bash">
$ vmware-vdiskmanager -d /path/to/main.vmdk
$ vmware-vdiskmanager -k /path/to/main.vmdk
</source>

= Installing Virtualbox guest additions =
Be sure to install DKMS(Dynamic Kernel Module System) before installing the Linux Guest Additions
<source lang="bash">
sudo apt-get install dkms
</source>

Additional packages if ''dkms'' was not enough
<source lang="bash">
sudo apt-get install linux-headers-$(uname -r) build-essential dkms #if above not work
sudo apt-get install perl make gcc #was required for Ubuntu 18.04 LTS
</source>

In the "Devices" menu in the virtual machine's menu bar, VirtualBox has a handy menu item named "Insert Guest Additions CD image", which mounts the Guest Additions ISO file inside your virtual machine. Then change directory to your CD-ROM and issue following command:
<source lang="bash">
sh ./VBoxLinuxAdditions.run
</source>

= Generalize Windows =
If you wish to re use your Windows VM image it needs to be generalized:
<source>
C:\Windows\System32\Sysprep
sysprep.exe /oobe /generalize /shutdown /mode:vm
</source>

Virtualbox

2025-08-22T08:28:10Z

Pio2pio: /* Install */

= Install =
<source lang=bash>
# 2025 onwards Ubuntu 24.04.3
wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --dearmor
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/oracle-virtualbox-2016.gpg] https://download.virtualbox.org/virtualbox/debian noble contrib" | sudo tee /etc/apt/sources.list.d/virtualbox.list
sudo apt install virtualbox-7.2

# Before 2025
echo "deb [arch=amd64] https://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib" | sudo tee -a /etc/apt/sources.list
wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-6.1
sudo apt-mark hold virtualbox-6.1
sudo apt-mark showhold
sudo apt-mark unhold virtualbox-6.1
</source>

= Resize disks in VirtualBox with Snapshots =
It is quite straightforward to resize a disk in VirtualBox as stated here and there. It becomes tricky though if the virtual machine,aka VM, has snapshots attached. The virtual disk thus is persisted across multiple VHD files, and the old trick will generally take not effect. This is also a [https://www.virtualbox.org/ticket/9103 known bug] hanging there for more than three years.

The suggested approach is to delete all snapshots and wait patiently for VirtualBox Manager to merge all the VHD files for you. It is a painfully lengthy process, so I decide to take a shortcut.

# First, shutdown the VM and backup the whole virtual machine folder.
# Then modify the size of all .vdi files in the root of the VM and Snapshots subdirectory.
<source lang="bash">
VBoxManage modifyhd "Windows 8.1.vdi" --resize 81920
for x in Snapshots/*.vdi ; do VBoxManage modifyhd $x --resize 81920 ; done
</source>
Startup the VM, and you will see the unallocated space in the Disk Management utility.

= Resize .vmdk disk on Linux =
Convert .vmdk format to .vdi and then resize. You can change format back after the resizing.
<source lang="bash">
VBoxManage clonehd "ubuntu-xenial-16.04-cloudimg.vmdk" "ubuntu-xenial-16.04-cloudimg.vdi" --format vdi #.vmdk -> .vdi
VBoxManage clonehd "ubuntu-xenial-16.04-cloudimg.vdi" "ubuntu-xenial-16.04-cloudimg.vmdk" --format vmdk #.vdi -> .vmdk
</source>

= Resize .vdi disk on Windows =
<source>
cd "C:\Program Files\Oracle\VirtualBox"
VBoxManage.exe modifyhd "C:\Users\piotr\VirtualBox VMs\vm-ubuntu64\vm-ubuntu64.vdi" --resize 20480
</source>
Note it also can resize VHD (Hyper-V) file formats.

= Vagrant note =
* OS: Ubuntu 16.04 LTS
* Vagrant version 2.1.1
* VirtualBox: 5.1.34_Ubuntu

Steps I have taken to resize Vagrant Ubuntu disk
# Stopped VM
# In settings removed attached drive "ubuntu-xenial-16.04-cloudimg.vmdk"
# Converted .vmdk into .vdi
# Attached "ubuntu-xenial-16.04-cloudimg.vdi" making sure that
#* Controller: SCSI Controller
#* Hard disk is attached to: SCSI Port 0, otherwise may throw error "no bootable medium found"

= Shrink unused space on virtual drive =
Hypervisor: Virtualbox, VMware

Virtual disks can be shrink as long as they are ext3 or ext4 file systems.
<source lang="bash">
$ vagrant ssh
$ sudo dd if=/dev/zero of=wipefile bs=1024x1024; rm wipefile
</source>

The above command is simply writing zero bytes to the wipefile in chunks of 1024 bytes until there is no disk space left in your VM’s disk. Then it is removing the wipefile. This basically leaves all those excess bytes zero’d out.

This is necessary because the shrink/compaction tools provided by either VMWare or VirtualBox both have no way of identifying space they can free up in the disks unless they are zero’d out.

With VirtualBox the only way I was able to shrink the disk image was to clone it to a smaller copy using the following command:
<source lang="bash">
$ VboxManage clonehd name-of-original-vm.vdi name-of-clone-vm.vdi
</source>

Once you have cloned the vdi you can then import it into the VM through VirtualBox and get rid of the original vdi.

With VMware you can shrink the vmdk disk by doing the following:
<source lang="bash">
$ vmware-vdiskmanager -d /path/to/main.vmdk
$ vmware-vdiskmanager -k /path/to/main.vmdk
</source>

= Installing Virtualbox guest additions =
Be sure to install DKMS(Dynamic Kernel Module System) before installing the Linux Guest Additions
<source lang="bash">
sudo apt-get install dkms
</source>

Additional packages if ''dkms'' was not enough
<source lang="bash">
sudo apt-get install linux-headers-$(uname -r) build-essential dkms #if above not work
sudo apt-get install perl make gcc #was required for Ubuntu 18.04 LTS
</source>

In the "Devices" menu in the virtual machine's menu bar, VirtualBox has a handy menu item named "Insert Guest Additions CD image", which mounts the Guest Additions ISO file inside your virtual machine. Then change directory to your CD-ROM and issue following command:
<source lang="bash">
sh ./VBoxLinuxAdditions.run
</source>

= Generalize Windows =
If you wish to re use your Windows VM image it needs to be generalized:
<source>
C:\Windows\System32\Sysprep
sysprep.exe /oobe /generalize /shutdown /mode:vm
</source>

HashiCorp/Vagrant

2025-04-22T07:20:20Z

Pio2pio: /* Sync using vagrant-vbguest plugin */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Re-ristribute the <tt>.box</tt> file then restore it.
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# In a case of dependency issues you can temporary disable the check
VAGRANT_DISABLE_STRICT_DEPENDENCY_ENFORCEMENT=1 vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

Kubernetes/Tools

2025-02-11T09:41:29Z

Pio2pio: /* Install kubectl */

= kubectl =
== Install kubectl ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.32.1
v1.31.5
v1.30.9
v1.29.13
v1.28.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.29.13; ARCH=amd64 # amd64|arm
curl -LO https://dl.k8s.io/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version
Client Version: v1.29.13
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.12-gke.1120001
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

kubectl plugin called [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke gke-gcloud-auth-plugin]
* [https://cloud.google.com/sdk/docs/install#deb Install Google Cloud SDK]
<source lang=bash>
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update
sudo apt-get install google-cloud-cli # required to authenticate with GCP
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gcloud init
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2025-02-11T09:40:31Z

Pio2pio: /* Install kubectl */

= kubectl =
== Install kubectl ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.32.1
v1.31.5
v1.30.9
v1.29.13
v1.28.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://dl.k8s.io/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

kubectl plugin called [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke gke-gcloud-auth-plugin]
* [https://cloud.google.com/sdk/docs/install#deb Install Google Cloud SDK]
<source lang=bash>
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update
sudo apt-get install google-cloud-cli # required to authenticate with GCP
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gcloud init
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Terraform

2024-11-07T23:00:12Z

Pio2pio: /* Syntax Terraform ~0.11 */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install 1.0.6
tfenv use 1.0.6
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory. Or set an environment variable. Then rerun <code>terraform init</code> to save providers into shared (cache) directory.

<source lang=terraform>
# Option 1.
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Option 2.
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache

# Create the cache directory
mkdir $HOME/.terraform.d/plugin-cache

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Run <code>terraform init</code>.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
tfautomv -dry-run
tfautomv -show-analysis
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

Terraform

2024-11-07T22:59:20Z

Pio2pio: /* Terraform Merge on Wildcard Tuple */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install 1.0.6
tfenv use 1.0.6
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Syntax Terraform ~0.11 =
== <code>if</code> statements ==
;Terraform ~< 0.9
Old versions Terraform doesn't support if- or if-else statement but we can take an advantage of a boolean ''count'' attribute that most of resources have.
boolean true = 1
boolean false = 0

;Terrafrom ~0.11+
Newer version support if statements, the conditional syntax is the well-known ternary operation:
<source>
CONDITION ? TRUEVAL : FALSEVAL
CONDITION ? caseTrue : caseFalse
domain = "${var.frontend_domain != "" ? var.frontend_domain : var.domain}" # tf <0.12 syntax
count = var.image_publisher == "MicrosoftWindowsServer" ? 0 : 3 # tf 0.12+ syntax
</source>

The support operators are:
*Equality: == and !=
*Numerical comparison: >, <, >=, <=
*Boolean logic: &&, ||, unary ! (|| is logical OR; “short-circuit” OR)

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory. Or set an environment variable. Then rerun <code>terraform init</code> to save providers into shared (cache) directory.

<source lang=terraform>
# Option 1.
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Option 2.
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache

# Create the cache directory
mkdir $HOME/.terraform.d/plugin-cache

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Run <code>terraform init</code>.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
tfautomv -dry-run
tfautomv -show-analysis
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

Terraform

2024-11-07T22:55:45Z

Pio2pio: /* terraform plugins cache */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install 1.0.6
tfenv use 1.0.6
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== Terraform Merge on Wildcard Tuple ==
Ideally the solution should be as simple as:
<source>
merge(local.policy_definitions.*.parameters...)
</source>
* [https://github.com/hashicorp/terraform/issues/24645 Terraform Merge on Wildcard Tuple] TF, GitHub issue
* [https://stackoverflow.com/questions/62683298/merge-list-of-objects-in-terraform merge-list-of-objects-in-terraform] Stackoverflow

;Workaround
<source>
policy_parameters = [
for key,value in data.azurerm_policy_definition.d_policy_definitions:
{
parameters = jsondecode(value.parameters)
}
]
ph_parameters = local.policy_parameters[*].parameters
input_parameter = [for item in local.ph_parameters: merge(item,local.ph_parameters...)][0]
</source>

;Break down
Extracts the parameter values into a list of JSON values
<source>
policy_parameters = [
for key,value in data.azurerm_policy_definition.d_policy_definitions:
{
parameters = jsondecode(value.parameters)
}
]
</source>

Reference the parameters as a variable
<source>
ph_parameters = local.policy_parameters[*].parameters
</source>

Merge all item content into each item.
<source>
input_parameter = [for item in local.ph_parameters: merge(item,local.ph_parameters...)]
</source>
The 3rd step gives all items in the list the same value, so we can use any index.

;Usage
<source>
parameters = "${jsonencode(local.input_parameter[n])}"
</source>

== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Syntax Terraform ~0.11 =
== <code>if</code> statements ==
;Terraform ~< 0.9
Old versions Terraform doesn't support if- or if-else statement but we can take an advantage of a boolean ''count'' attribute that most of resources have.
boolean true = 1
boolean false = 0

;Terrafrom ~0.11+
Newer version support if statements, the conditional syntax is the well-known ternary operation:
<source>
CONDITION ? TRUEVAL : FALSEVAL
CONDITION ? caseTrue : caseFalse
domain = "${var.frontend_domain != "" ? var.frontend_domain : var.domain}" # tf <0.12 syntax
count = var.image_publisher == "MicrosoftWindowsServer" ? 0 : 3 # tf 0.12+ syntax
</source>

The support operators are:
*Equality: == and !=
*Numerical comparison: >, <, >=, <=
*Boolean logic: &&, ||, unary ! (|| is logical OR; “short-circuit” OR)

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory. Or set an environment variable. Then rerun <code>terraform init</code> to save providers into shared (cache) directory.

<source lang=terraform>
# Option 1.
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Option 2.
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache

# Create the cache directory
mkdir $HOME/.terraform.d/plugin-cache

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Run <code>terraform init</code>.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
tfautomv -dry-run
tfautomv -show-analysis
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

Terraform

2024-11-07T22:49:58Z

Pio2pio: /* cache terraform plugins */

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code - IoC.

{{Note| most of the paragraphs have examples of Terraform prior 0.12 version syntax that uses HCLv1. HCLv2 has been introduced with v0.12+ that contains significiant syntax and capabilites improvments. }}

= Install terraform =
<source lang=bash>
wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
unzip terraform_0.11.11_linux_amd64.zip
sudo mv ./terraform /usr/local/bin
</source>
== [https://github.com/kamatama41/tfenv tfenv] - manage multiple versions of Teraform ==
Install and usage
<source lang=bash>
git clone https://github.com/tfutils/tfenv.git ~/.tfenv
echo "[ -d $HOME/.tfenv ] && export PATH=$PATH:$HOME/.tfenv/bin/" >> ~/.bashrc # or ~/.bash_profile

# Use
tfenv install 1.0.6
tfenv use 1.0.6
</source>

== IDE ==
Development I use:
* VSCode with 1.41.1+ (for reference) with extensions:
** Terraform Autocomplete by erd0s
** Terraform by Mikael Olenfalk with enabled Language Server; open the command pallet with <code>Ctrl+Shift+P</code>
:[[File:ClipCapIt-200202-153128.PNG]]

= Basic configuration =
When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.
<source lang="json">
$ vi example.tf
provider "aws" {
access_key = "AK01234567890OGD6WGA"
secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
region = "eu-west-1"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"
}
</source>

Since version 10.8.x major changes and features have been introduced including split of providers binary. Now each provider is a separate binary. Please see below example for Azure provider and other internal Terraform developed providers.

== Azure ==
Terraform credentials
<source lang=bash>
export ARM_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export ARM_TENANT_ID="TENANT_ID"
export ARM_CLIENT_ID="CLIENT_ID"
export ARM_CLIENT_SECRET="CLIENT_SECRET"
export TF_VAR_client_id=${ARM_CLIENT_ID}
export TF_VAR_client_secret=${ARM_CLIENT_SECRET}
</source>

Example, how to source credentials
<source lang=bash>
export VAULT_CLIENT_ADDR=http://10.1.1.1:8200
export VAULT_TOKEN=11111111-1111-1111-1111-1111111111111
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/subscription | jq -r '.data | .subscription_id, .tenant_id'
vault read -format=json -address=$VAULT_CLIENT_ADDR secret/azure/${application} | jq -r '.data | .client_id, .client_secret'
</source>

Terraform providers, modules and backend config
<source lang="json">
$ vi providers.tf
provider "azurerm" {
version = "1.10.0"
subscription_id = "${var.subscription_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

# HashiCorp special providers https://github.com/terraform-providers
provider "template" { version = "1.0.0" }
provider "external" { version = "1.0.0" }
provider "local" { version = "1.1.0" }

terraform {
backend "local" {}
}
</source>

== AWS ==
;References
*[https://www.padok.fr/en/blog/terraform-s3-bucket-aws S3 bucket for all accounts]
*[https://www.padok.fr/en/blog/authentication-aws-profiles Multi account auth using aws profiles and <code>provider "aws" {}</code>]
=== Local state ===
Local state configuration
<source lang="json">
vi backend.tf
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
backend "local" {}
}
</source>

=== Remote state (single) for multi account deployments ===
There are many combination setting up backend and AWS credentials. Important understand is that <code>terraform { backend{} }</code> block does NOT use <code>provider "aws {}"</code> configuration in order to access the state bucket. It only uses the backend one.
* exporting credentials allows working with assume roles that are different in the backend and terraform blocks.
* specifying different <code>profile = </code> in each blocks

;Credentials
<source lang="bash">
## profile allows assumes roles in other accounts
#export AWS_PROFILE="piotr"

# Environment credentials for a user that can assume roles (eg. ) in other accounts:
# | * arn:aws:iam::111111111111:role/terraform-s3state - save state in s3 bucket
# | * arn:aws:iam::222222222222:role/terraform-crossaccount-admin - deploy resources
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

# unset all of them if need to
unset ${!AWS@}
</source>

;<code>terraform {}</code>
<source lang="json">
terraform {
version = "~> 1.0"
required_version = "= 0.12.29"
# profile "dev-us" # we use 'role_arn' but could specify aws profile instead
backend "s3" {
bucket = "tfstate-${var.project}-${var.account-id}" # must exist beforehand
key = "terraform/aws/${var.project}/tfstate" # this could be much simpler when working with terraform workspaces
region = "${var.region}"
role_arn = "arn:aws:iam::111111111111:role/terraform-s3state" # role to assume in an infra account that the s3 state exists
}
}
</source>

;<code>provider {}</code>
<source lang="json">
provider "aws" {
## We could use profiles but instead we use 'assume_role' option. Also on your laptop
## it should be your creds profile eg. 'piotr-xaccount-admin'
#profile = "terraform-crossaccount-admin"
#shared_credentials_file = "/home/piotr/.aws/credentials"
assume_role = {
role_arn = "arn:aws:iam::<MY_PROD_ACCOUNT>:role/terraform-crossaccount-admin" # assume role in target account
role_arn = "arn:aws:iam::${var.aws_account}:role/terraform-crossaccount-admin" # can use variables
}
region = "var.aws_region"
allowed_account_ids = [ "111111111111", "222222222222" ] # safety net
}
</source>

;Workspace configuration
Dev configuration in <code>dev.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_DEV_ACCOUNT>"
</source>

Prod configuration in <code>prod.tfvars</code>
<source lang="json">
aws_region = "eu-west-1"
aws_account = "<MY_PROD_ACCOUNT>"
</source>

;Workspaces
<source lang="bash">
terraform init
terraform workspace new dev
terraform workspace new prod
</source>

;Apply on one account
<source lang="bash">
terraform workspace select dev
terraform apply --var-file $(terraform workspace show).tfvars
</source>

== GCP Google Cloud Platform ==
<source lang=bash>
# Generate default app credentials

gcloud auth application-default login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=****_challenge_method=S256
Enter verification code: ***
Credentials saved to file: [/home/piotr/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "test-devops-candidate1" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource
</source>

= Plan / apply =
== Meaning of markings in a plan output ==
For now, here they are, until we get it included in the docs better:

* <code>+</code> create
* <code>-</code> destroy
* <code>-/+</code> replace (destroy and then create, or vice-versa if create-before-destroy is used)
* <code>~</code> update in-place
* <code><=</code> applies only to data resources. You won't see this one often, because whenever possible Terraform does reads during the refresh phase. You will see it, though, if you have a data resource whose configuration depends on something that we don't know yet, such as an attribute of a resource that isn't yet created. In that case, it's necessary to wait until apply time to find out the final configuration before doing the read.

== Plan and apply ==
Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.
{| class="wikitable"
|+ Terraform plan and apply
|-
! terraform plan
! terraform apply
|- style="vertical-align:top;"
| <source>$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
</source>
| <source>$ terraform apply
aws_instance.webserver: Creating...
ami: "" => "ami-405f7226"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.nano"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tenancy: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:
</source>
|}

== Show ==
<source lang=bash>
$ terraform show
aws_instance.webserver:
id = i-0eb33af34b94d1a78
ami = ami-405f7226
associate_public_ip_address = true
availability_zone = eu-west-1c
disable_api_termination = false
(...)
source_dest_check = true
subnet_id = subnet-92a4bbf6
tags.% = 0
tenancy = default
vpc_security_group_ids.# = 1
vpc_security_group_ids.1039819662 = sg-5201fb2b

$> terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.
</source>

After the instance has been terminated the terraform.tfstate looks like below:
<source lang=bash>
vi terraform.tfstate
</source>
<source lang=json>
{
"version": 3,
"terraform_version": "0.9.1",
"serial": 1,
"lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}
</source>

= AWS credentials profiles and variable files=
Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file. Note: there is '''no double quotes'''.
<source lang=bash>
$ vi ~/.aws/credentials #AWS credentials file with named profiles
[terraform-profile1] #profile name
aws_access_key_id = AAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
</source>

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:
<source lang=bash>
vi provider.tf
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
region = "eu-west-1"
backend "s3" {} # in this case all s3 details are passed as ENV vars
}

provider "aws" {
version = "~> 1.57"
# Static credentials - provided directly
access_key = "AAAAAAAAAAA"
secret_key = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

# Shared Credentials file - $HOME/.aws/credentials, static credentials are not needed then
# profile = "terraform-profile1" #profile name in credentials file, acc 111111111111
# shared_credentials_file = "/home/user1/.aws/credentials" #if different than default

# If specified, assume role in another account using the user credentials
# defined in the profile above
# assume_role {
# role_arn = "${var.aws_xaccount_role}" #variable version
# role_arn = "arn:aws:iam::222222222222:role/CrossAccountSignin_Terraform"
# }
# allowed_account_ids = [ "111111111111", "222222222222" ]
}

provider "template" {
version = "~> 1.0.0"
}
</source>

and create a variable file to reference it
<source lang=bash>
$ vi variables.tf
variable "region" {
default = "eu-west-1"
}
variable "profile" {} #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'
</source>

Run terraform
<source lang=bash>
$ terraform plan -var 'profile=terraform-profile1' #this way value can be set
$ terraform plan -destroy -input=false
</source>

= AWS example =
Prerequisites are:
*~/.aws/credential file exists
*variables.tf exist, with context below:

If you remove <tt>default</tt> value you will be prompted for it.

<code>inputs.tf</code> also known as a variable file.
<source lang=bash>
$> vi inputs.tf
variable "region" { default = "eu-west-1" }
variable "profile" {
description = "Provide AWS credentials profile you want to use, saved in ~/.aws/credentials file"
default = "terraform-profile" }
variable "key_name" {
description = <<DESCRIPTION
Provide name of the ssh private key file name, ~/.ssh will be search
This is the key assosiated with the IAM user in AWS. Example: id_rsa
DESCRIPTION
default = "id_rsa" }
variable "public_key_path" {
description = <<DESCRIPTION
Path to the SSH public keys for authentication. This key will be injected
into all ec2 instances created by Terraform.
Example: ~./ssh/terraform.pub
DESCRIPTION
default = "~/.ssh/id_rsa.pub" }
</source>

Terraform .tf file
<source lang=bash>
$ vi example.tf
provider "aws" {
region = "${var.region}"
profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true

tags {
Name = "Public"
}
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
name = "terraform_securitygroup"
description = "Used for public instances"
vpc_id = "${aws_vpc.vpc.id}"

# SSH access from anywhere
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP access from the VPC
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/16"]
}
# outbound internet access
egress {
from_port = 0
to_port = 0
protocol = "-1" # all protocols
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_key_pair" "auth" {
key_name = "${var.key_name}"
public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
ami = "ami-405f7226"
instance_type = "t2.nano"

key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.default.id}"]

# We're going to launch into the public subnet for this.
# Normally, in production environments, webservers would be in
# private subnets.
subnet_id = "${aws_subnet.default.id}"

# The connection block tells our provisioner how to
# communicate with the instance
connection {
user = "ubuntu"
}
# We run a remote provisioner on the instance after creating it
# to install Nginx. By default, this should be on port 80
provisioner "remote-exec" {
inline = [
"sudo apt-get -y update",
"sudo apt-get -y install nginx",
"sudo service nginx start"
]
}
}
</source>

== Run a plan ==
<source>
$ terraform plan
var.key_name
Name of the AWS key pair

Enter a value: id_rsa #name of the key_pair

var.profile
AWS credentials profile you want to use

Enter a value: terraform-profile #aws profile in ~/.aws/credentials file

var.public_key_path
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub

Enter a value: ~/.ssh/id_rsa.pub #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
ami: "ami-405f7226"
associate_public_ip_address: "<computed>"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.nano"
ipv6_addresses.#: "<computed>"
key_name: "${aws_key_pair.auth.id}"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "${aws_subnet.default.id}"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"

+ aws_internet_gateway.internet-gateway
vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
fingerprint: "<computed>"
key_name: "id_rsa"
public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

...omitted...>

Plan: 7 to add, 0 to change, 0 to destroy.
</source>

;Plan a single target
<source lang=bash>
$> terraform plan -target=aws_ami_from_instance.golden
</source>

== Terraform apply ==
<source lang=bash>
$> terraform apply
$> terraform show # shoe current resources in the state file
aws_instance.webserver:
id = i-09c1c665cef284235
ami = ami-405f7226
<...>
aws_security_group.default:
id = sg-b14bb1c8
description = Used for public instances
egress.# = 1
<...>
aws_subnet.default:
id = subnet-6f4f510b
<...>
aws_vpc.vpc:
id = vpc-9ba0b7ff
<...>
</source>

;Apply a single resource using <code>-target <resource></code>
<source lang=bash>
$> terraform apply -target=aws_ami_from_instance.golden
</source>

== Terraform destroy ==
Run destroy command to delete all resources that were created
<source lang=bash>
$> terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
<...>

Destroy complete! Resources: 7 destroyed.
</source>

;Destroy a single resource - targeting
<source lang=bash>
$> terraform show
$> terraform destroy -target=aws_ami_from_instance.golden
</source>
== Terraform taint ==
Get a resource list
<source lang=bash>
terraform state list
# select item for the list #
</source>

;Version 0.11: resource index must be addressed as eg. <code>aws_instance.main.0</code> not <code>aws_instance.main[0]</code>. It's not possible to tain whole module
<source lang=bash>
terraform taint -module=<MODULE_NAME> aws_instance.main.0
</source>

;Version 0.12: resources and modules can be addressed in more natural way
<source lang=bash>
terraform taint module.MODULE_NAME.aws_instance.main.0
</source>

=Use ansible from Terraform - Provision using Ansible =
Unsurr if this is the best approach due to the fact of how to store the state of local-exec Ansible run. Could be set to always run as Ansible playbooks are immutable. Exame: https://github.com/dzeban/c10k/blob/master/infrastructure/main.tf

= Debug =
== Output complex object ==
Often it is required to manipulate a data structure that is an output of <tt>resource</tt>, <tt>data.resource</tt> or simply a template that might be hidden computation not always displayed on your screen. You can use following techniques to iterate over you code output:

;Output and [https://www.terraform.io/docs/providers/null/resource.html null_resource] - empty virtual container that can run any arbitrary commands
* '''Problem statement:''' Display computed Terrafom <code>templatefile</code>
* '''Solution:''' Use <code>null_resource</code> to create a template, such template will be shown in a <tt>plan</tt>. If such template is Json policy, invalid policies fail and you cannot see why. Plan will show the object being constructed, running <code>terraform apply</code> it can be saved into state file as output variable. Then the object can be re-used for further transformations.
<syntaxhighlight lang="Terraform">
data "aws_caller_identity" "current" {}

# resource "aws_kms_key" "secretmanager" {
# policy = templatefile("./templates/kms_secretmanager.policy.json.tpl", ... # debugging policy with
# } # null_resource and ouput

resource "aws_kms_alias" "secretmanager" {
name = "alias/secretmanager"
target_key_id = aws_kms_key.secretmanager.key_id
}

resource "null_resource" "policytest" {
triggers = {
policytest = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
})
}
}

output "policy" {
value = templatefile("./templates/kms_secretmanager.policy.json.tpl",
{
arns_json = jsonencode(var.crossAccountIamUsers_arns)
currentAccountId = data.aws_caller_identity.current.account_id
crossAccountAccessEnabled = length([var.crossAccountIamUsers_arns]) > 0 ? true : false
}
)
}
</syntaxhighlight>

Policy template file <code>./templates/kms_secretmanager.policy.json.tpl</code>
<source lang=json>
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${currentAccountId}:root"
},
"Action": "kms:*",
"Resource": "*"
},
%{ if crossAccountAccessEnabled == true ~}
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ${arns_json}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
%{ endif ~}
]
}
</source>

;Run
<source lang=bash>
$ terraform apply -var-file=test.tfvars -target null_resource.policytest # -var-file contains 'var.crossAccountIamUsers_arns' list variable

Terraform will perform the following actions:

# null_resource.policytest will be created
+ resource "null_resource" "policytest" {
+ id = (known after apply)
+ triggers = {
+ "policytest" = jsonencode(
{
+ Id = "key-consolepolicy-1"
+ Statement = [
+ {
+ Action = "kms:*"
+ Effect = "Allow"
+ Principal = {
+ AWS = "arn:aws:iam::111111111111:root"
}
+ Resource = "*"
+ Sid = "Enable IAM User Permissions"
},
+ {
+ Action = [
+ "kms:Decrypt",
+ "kms:DescribeKey",
]
+ Effect = "Allow"
+ Principal = {
+ AWS = [
+ "arn:aws:iam::111111111111:user/dev",
+ "arn:aws:iam::111111111111:user/test",
]
}
+ Resource = "*"
+ Sid = "Allow cross-accounts retrieve secrets"
},
]
+ Version = "2012-10-17"
}
)
}
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes # <- manual imput

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
policy = {
"Version": "2012-10-17",
"Id": "key-consolepolicy-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow cross-accounts retrieve secrets",
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111111111111:user/dev","arn:aws:iam::111111111111:user/test"]
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
</source>

= Debug =
== Debug and analyze logs ==
We are going to enable logging to a file in Terraform. Convert log file to pdf and use sheri.ai to give us the answers.
<source lang=bash>
# Pre req - Ubuntu 22.04
sudo apt-get update && sudo apt-get install ghostscript # for ps2pdf converter

# Set Terraform logging
export TF_LOG=TRACE # DEBUG
export TF_LOG_PATH=/tmp/tflogs.log

terraform plan|apply
vim $TF_LOG_PATH -c "hardcopy > ${TF_LOG_PATH}.ps | q"; ps2pdf ${TF_LOG_PATH}.ps ${TF_LOG_PATH}-$(echo $(date +"%Y%m%d-%H%M")).pdf
</source>

== Debug using <code>terraform console</code>==
This command provides an interactive command-line console for evaluating and experimenting with expressions. This is useful for testing interpolations before using them in configurations, and for interacting with any values currently saved in state. Terraform console will read configured state even if it is remote.
<source lang=ruby>
$> terraform console #-state=path # note I have 'tfstate' available; this could be remote state
> var.vpc_cidr # <- new syntax
10.123.0.0/16
> "${var.vpc_cidr}" # <- old syntax
10.123.0.0/16
> aws_security_group.tf_public_sg.id # interpolate from state
sg-04d51b5ae10e6f0b0
> help
The Terraform console allows you to experiment with Terraform interpolations.
You may access resources in the state (if you have one) just as you would
from a configuration. For example: "aws_instance.foo.id" would evaluate
to the ID of "aws_instance.foo" if it exists in your state.

Type in the interpolation to test and hit <enter> to see the result.

To exit the console, type "exit" and hit <enter>, or use Control-C or
Control-D.
</source>

Example
<source lang=bash>
$ echo "aws_iam_user.notif.arn" | terraform console
arn:aws:iam::123456789:user/notif
</source>

== Log user_data to console logs ==
In Linux add a line below after she-bang
<source lang=bash>
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console)
</source>
Now you can go and open System Logs in AWS Console to view user-data script logs.

= terraform graph to visualise configuration =
== Graph dependencies ==
Create visualised file. You may need to install <code>sudo apt-get install graphviz</code> if it is not in your system.
<source lang=bash>
sudo apt install graphviz # installs 'dot'
terraform graph | dot -Tpng > graph.png
</source>
[[File:Example2.png|none|left|700px|Terraform visual configuration]]

== [https://serverfault.com/questions/1005761/what-does-error-cycle-means-in-terraform Cycle error] ==
Example cycle error:
<source>
Error: Cycle: module.gke.google_container_node_pool.pools["low-standard-n1"]
module.gke.google_container_node_pool.pools["medium-standard-n1"]
module.gke.google_container_node_pool.pools["large-standard-n1"]
module.gke.local.cluster_endpoint (expand)
module.gke.output.endpoint (expand)
provider["registry.terraform.io/gavinbunney/kubectl"]
kubectl_manifest.sync["source.toolkit.fluxcd.io/v1beta1/gitrepository/flux-system/flux-system"] (destroy)
module.gke.google_container_node_pool.pools["preemptible"] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.additional_components[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_command[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0] (destroy)
module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.run_destroy_command[0] (destroy)
module.gke.kubernetes_config_map.kube-dns[0] (destroy)
module.gke.google_container_cluster.primary
module.gke.local.cluster_output_master_auth (expand)
module.gke.local.cluster_master_auth_list_layer1 (expand)
module.gke.local.cluster_master_auth_list_layer2 (expand)
module.gke.local.cluster_master_auth_map (expand)
module.gke.local.cluster_ca_certificate (expand)
module.gke.output.ca_certificate (expand)
provider["registry.terraform.io/hashicorp/kubernetes"]
</source>

The <code>-draw-cycles</code> command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

<source lang=bash>
sudo apt install graphviz
terraform graph -draw-cycles -type=plan > cycle-plan.graphviz
terraform graph -draw-cycles | dot -Tpng > cycles.png
terraform graph -draw-cycles | dot -Tsvg > cycles.svg
terraform graph -draw-cycles | dot -Tpdf > cycles.pdf
# | -draw-cycles - highlight any cycles in the graph with colored edges. This helps when diagnosing cycle errors.
# | -type=plan - type of graph to output. Can be: plan, plan-destroy, apply, validate, input, refresh.

# For large graphs you may want to install inkscape
sudo apt install inkscape --no-install-suggests --no-install-recommends
</source>

Awoid cycle errors in modules by structuring your config to avoid cross-module references. So instead of directly accessing an output of one module from inside another, set it up as in input parameter instead and wire everything together on the top level.

;How to get it solved
With the cycling dependency issue, study the graph then decide on removing from the state a resource that should be generated later. If the graph is not clear or too complex to read you may need to guess and delete from the state a resource marked for deletion, ie:
<source>
terraform state rm kubectl_manifest.install[\"apps/v1/deployment/flux-system/kustomize-controller\"]
</source>

= Remote state =
== Enable ==
Create s3 bucket with unique name, enable versioning and choose a region.

Then configure terraform:
<source lang=bash>
$ terraform remote config \
-backend=s3 \
-backend-config="bucket=YOUR_BUCKET_NAME" \
-backend-config="key=terraform.tfstate" \
-backend-config="region=YOUR_BUCKET_REGION" \
-backend-config="encrypt=true"
Remote configuration updated
Remote state configured and pulled.
</source>
After running this command, you should see your Terraform state show up in that S3 bucket.

== Locking ==
Add <code>dynamodb_table</code> name to backend configuration.
<source lang=bash>
terraform {
version = "~> 1.0"
required_version = "= 0.11.11"
backend "s3" {
dynamodb_table = "tfstate-lock"
profile = "terraform-agent"
# assume_role {
# role_arn = "${var.aws_xaccount_role}"
# session_name = "${var.aws_xsession_name}"
# }
}
}
</source>

In AWS create dynamo-db table, named: <tt>tfsate-lock</tt> with index <tt>LockID</tt>; as on a picture below. It an event of taking a lock the entry similar to one below gets created.
[[File:Terraform-dynamo-db-state-locking.png|none|left|Terraform-dynamo-db-state-locking]]
<source lang=bash>
{"ID":"62a453e8-7fbc-cfa2-e07f-be1381b82af3","Operation":"OperationTypePlan","Info":"","Who":"piotr@laptop1","Version":"0.11.11","Created":"2019-03-07T08:49:33.3078722Z","Path":"tfstate-acmedev01-acmedev-111111111111/aws/acmedev01/state"}
</source>

= Workspaces =
== [https://discuss.hashicorp.com/t/how-to-change-the-name-of-a-workspace/24010 Rename a workspace / move the state file] ==
{{Note|The state manipulation commands run through Terraform’s automatic state upgrading processes and so best to do this with the same Terraform CLI version that you’ve most recently been using against this workspace so that the state won’t be implicitly upgraded as part of the operation.}}
<source lang=bash>
terraform workspace select old-name
terraform state pull >old-name.tfstate
terraform workspace new new-name
terraform state push old-name.tfstate
terraform show # confirm that the newly-imported state looks 'right', before deleting the old workspace
terraform workspace delete -force old-name
</source>

= Variables =
Variables can be provided via cli
<source lang=bash>
terraform apply -var="image_id=ami-abc123"
terraform apply -var='image_id_list=["ami-abc123","ami-def456"]'
terraform apply -var='image_id_map={"us-east-1":"ami-abc123","us-east-2":"ami-def456"}'
</source>

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>.
* Any files with names ending in <code>.auto.tfvars</code> or <code>.auto.tfvars.json</code>.

=Syntax Terraform 0.12.6+=
{{Note|This [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html#for-expressions for-expressions] link is a little diamond for this subject}}

== Map and nested block ==
Terrafom 0.12 introduces stricter validation for followings but allows map keys to be set dynamically from expressions. Note of "=" sign.
* a map attribute - usually have user-defined keys, like we see in the tags example
* a nested block always has a fixed set of supported arguments defined by the resource type schema, which Terraform will validate
<source>
resource "aws_instance" "example" {
instance_type = "t2.micro"
ami = "ami-abcd1234"

tags = { # <- a map attribute, requires '='
Name = "example instance"
}

ebs_block_device { # <- a nested block, no '='
device_name = "sda2"
volume_type = "gp2"
volume_size = 24
}
}
</source>

== [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html For_each] ==
* [https://alexharv074.github.io/2019/06/02/adventures-in-the-terraform-dsl-part-iii-iteration-enhancements-in-terraform-0.12.html terraform iterations]
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
{| class="wikitable"
|+ For_each and new allowed formatting without the need for "${var.vpc_cidr}" syntax = var.vpc_cidr
|-
! main.tf
! variables.tf and outputs.tf
|- style="vertical-align:top;"
| <source># vi main.tf
resource "aws_vpc" "tf_vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
enable_dns_support = true
tags = { #<-note of '=' as this is an argument
Name = "tf_vpc"
}
}

resource "aws_security_group" "tf_public_sg" {
name = "tf_public_sg"
description = "Used for access to the public instances"
vpc_id = "${aws_vpc.tf_vpc.id}"

dynamic "ingress" {
for_each = [ for s in var.service_ports: {
from_port = s.from_port
to_port = s.to_port }]
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = "tcp"
cidr_blocks = [ var.accessip ]
}
}
# Commented block has been replaced by 'dynamic "ingress"'
# ingress { #SSH
# from_port = 22
# to_port = 22
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
# ingress { #HTTP
# from_port = 80
# to_port = 80
# protocol = "tcp"
# cidr_blocks = ["${var.accessip}"]
# }
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}</source>
| <source># vi variables.tf
variable "vpc_cidr" { default = "10.123.0.0/16" }
variable "accessip" { default = "0.0.0.0/0" }

variable "service_ports" {
type = "list"
default = [
{ from_port = 22, to_port = 22 },
{ from_port = 80, to_port = 80 }
]
}

# vi outputs.tf
output "public_sg" {
value = aws_security_group.tf_public_sg.id
}

output "ingress_port_mapping" {
value = {
for ingress in aws_security_group.tf_public_sg.ingress:
format("From %d", ingress.from_port) => format("To %d", ingress.to_port)
}
}

# Computed 'Outputs:'
ingress_port_mapping = {
"From 22" = "To 22"
"From 80" = "To 80"
}
public_sg = sg-04d51b5ae10e6f0b0
</source>
|}

=== [https://www.sheldonhull.com/blog/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/ Iterate over list of objects] ===
[https://stackoverflow.com/questions/58594506/how-to-for-each-through-a-listobjects-in-terraform-0-12 how-to-for-each-through-a-listobjects]

<source lang=yaml>
# debug.tf
locals {
users = [
# list of objects
{ name = "foo", is_enabled = true },
{ name = "bar", is_enabled = false },
]
}

resource "null_resource" "this" {
for_each = { for name in local.users: name.name => name.is_enabled }
connection {
name = each.key
email = each.value
}
}

output "users_map" {
value = { for name in local.users: name.name => name.is_enabled }
}

# terraform init
# terraform apply

null_resource.this["bar"]: Creating...
null_resource.this["foo"]: Creating...
null_resource.this["bar"]: Creation complete after 0s [id=7228791922218879597]
null_resource.this["foo"]: Creation complete after 0s [id=7997705376010456213]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

users_map = {
"bar" = false
"foo" = true
}
</source>

== Plan is more readable and explicit ==
[[Terraform/plan_tf_11_vs_12|See comparison]]

== [https://www.hashicorp.com/blog/terraform-0-12-rich-value-types/ Rich Value Types] - for previewing whole resource object ==
'''Resources and Modules as Values''' Terraform 0.12 now permits using entire resources as object values within configuration, including returning them as outputs and passing them as input variables:
<source>
output "vpc" {
value = aws_vpc.example
}
</source>

The type of this output value is an object type derived from the schema of the <code>aws_vpc</code> resource type. The calling module can then access attributes of this result in the same way as the returning module would use <code>aws_vpc.example</code>, such as <code>module.example.vpc.cidr_block</code>. This works also for modules with an expression like <code>module.vpc</code> evaluating to an object value with attributes corresponding to the modules's named outputs.

== <code>for</code> ==
* [https://discuss.hashicorp.com/t/produce-maps-from-list-of-strings-of-a-map/2197 Produce maps from list of strings of a map]
This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.
<source>
local {
upper_list = [for i in var.list : upper(i)] # creates a new list
}
</source>

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.
<source>
local {
upper_map = {for i in var.list : i => upper(i)} # creates a map with key = value
# { i[0] = upper(i[0])
# i[1] = upper(i[1]) }
}
</source>

Use ''if'' as a filter in ''for'' expression
<source>
[for i in var.list : upper(i) if i != ""]
</source>

</source>
In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

== Manipulate list and complex object ==
Build a new list by removing items that their string value do not match regex expression
<source lang=bash>
# Resource that generates an object
resource "aws_acm_certificate" "main" {...}

# Preview of input object 'aws_acm_certificate.main.domain_validation_options'
output "domain_validation_options" {
value = aws_acm_certificate.main.domain_validation_options
description = "array/list of maps taken from resource object(aws_acm_certificate.issued) describing all validation domain records"
}

$ terraform output domain_validation_options
[ # <- array starts here
{ # <- an item of array the map object
"domain_name" = "*.dev.example.com"
"resource_record_name" = "_11111111111111111111111111111111.dev.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_22222222222222222222222222222222.mzlfeqexyx.acm-validations.aws."
},
{
"domain_name" = "api.example.com"
"resource_record_name" = "_31111111111111111111111111111111.api.example.com."
"resource_record_type" = "CNAME"
"resource_record_value" = "_42222222222222222222222222222222.vhzmpjdqfx.acm-validations.aws."

},
]

# 'for k, v' syntax builds a new object 'validation_domains' by iterating over array of maps
# 'aws_acm_certificate.main.domain_validation_options' and conditinally changes a value of 'v'
# if contains the sting "*.dev.example.com". tomap(v) is required to persist type across for expression.
locals {
validation_domains = [
for k, v in aws_acm_certificate.main.domain_validation_options : tomap(v) if contains(
"*.dev.example.com", replace(v.domain_name, "*.", "")
)
]
}

$ terraform output local_distinct_domains
local_distinct_domains = [
"api.example.com",
"dev.example.com",
"api-aat1.dev.example.com",
"api-aat2.dev.example.com",
]

# 'for domain' expession builds a new list only when a domain matches regexall string.
# checks regexall lengh > 0 of matched captured groups so true or false is return, so
# the 'for domain : if' statment conditionally adds the item to the new list
locals {
distinct_domains_excluded = [
for domain in local.distinct_domains : domain if length(regexall("dev.example.com", domain)) > 0
]

# Similar to the above but iterating over array of maps (k,v - key, value pairs)
validation_domains = [
for k,v in local.validation_domains : tomap(v) if length(regexall("dev.example.com", v.domain_name)) > 0
]
}

# Example of iterating over array of maps 'aws_acm_certificate.main.domain_validation_options' to build a list
# of fqdns that are store in 'aws_acm_certificate.main.domain_validation_options.resource_record_name' in .resource_record_name
# key.
# 'for fqdn' syntax on each iteration 'fqdn=aws_acm_certificate.main.domain_validation_options[index]', then
# anything after ':' means 'set to value equals' fqdn.resource_record_name
resource "aws_acm_certificate_validation" "main" {
certificate_arn = aws_acm_certificate.main.arn
validation_record_fqdns = [
for fqdn in aws_acm_certificate.main.domain_validation_options : fqdn.resource_record_name
]
}
</source>
== Terraform Merge on Wildcard Tuple ==
Ideally the solution should be as simple as:
<source>
merge(local.policy_definitions.*.parameters...)
</source>
* [https://github.com/hashicorp/terraform/issues/24645 Terraform Merge on Wildcard Tuple] TF, GitHub issue
* [https://stackoverflow.com/questions/62683298/merge-list-of-objects-in-terraform merge-list-of-objects-in-terraform] Stackoverflow

;Workaround
<source>
policy_parameters = [
for key,value in data.azurerm_policy_definition.d_policy_definitions:
{
parameters = jsondecode(value.parameters)
}
]
ph_parameters = local.policy_parameters[*].parameters
input_parameter = [for item in local.ph_parameters: merge(item,local.ph_parameters...)][0]
</source>

;Break down
Extracts the parameter values into a list of JSON values
<source>
policy_parameters = [
for key,value in data.azurerm_policy_definition.d_policy_definitions:
{
parameters = jsondecode(value.parameters)
}
]
</source>

Reference the parameters as a variable
<source>
ph_parameters = local.policy_parameters[*].parameters
</source>

Merge all item content into each item.
<source>
input_parameter = [for item in local.ph_parameters: merge(item,local.ph_parameters...)]
</source>
The 3rd step gives all items in the list the same value, so we can use any index.

;Usage
<source>
parameters = "${jsonencode(local.input_parameter[n])}"
</source>

== function: replace, regex ==
Snippet below removes comments and any empty lines from a <code>values.yaml.tpl</code> file.
<source lang=json>
locals {
match_comment = "/(?U)(?m)(?s)^[[:space:]]*#.*$/" # match anyline that starts with '#' or any 'whitespace(s) + #'
match_empty_line = "/(?m)(?s)(^[\r\n])/"
}

resource "helm_release" "myapp" {
name = "myapp"
chart = "${path.module}/charts/myapp"
values = [
replace(
replace(
templatefile("${path.module}/templates/values.yaml.tpl", {
}), local.match_comment, ""), local.match_empty_line, "")
]
</source>

Explanation:
* Terraform regex is using [https://github.com/google/re2/wiki/Syntax re2 library]
* Regex flags are enabled by prefixinf the search:
** <code>(?m)</code> - multi-line mode (default false)
** <code>(?s)</code> - let . match \n (default false)
** <code>(?U)</code> - ungreedy (default false), so stop matching comments at EOL

== References ==
*[https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each HashiCorp Terraform 0.12 Preview: For and For-Each]

= Syntax Terraform ~0.11 =
== <code>if</code> statements ==
;Terraform ~< 0.9
Old versions Terraform doesn't support if- or if-else statement but we can take an advantage of a boolean ''count'' attribute that most of resources have.
boolean true = 1
boolean false = 0

;Terrafrom ~0.11+
Newer version support if statements, the conditional syntax is the well-known ternary operation:
<source>
CONDITION ? TRUEVAL : FALSEVAL
CONDITION ? caseTrue : caseFalse
domain = "${var.frontend_domain != "" ? var.frontend_domain : var.domain}" # tf <0.12 syntax
count = var.image_publisher == "MicrosoftWindowsServer" ? 0 : 3 # tf 0.12+ syntax
</source>

The support operators are:
*Equality: == and !=
*Numerical comparison: >, <, >=, <=
*Boolean logic: &&, ||, unary ! (|| is logical OR; “short-circuit” OR)

= Modules =
Modules are used in Terraform to modularize and encapsulate groups of resources in your infrastructure.

When calling a module from .tf file you passing values for variables that are defined in a module to create resources to your specification. Before you can use any module it needs to be downloaded. Use
$ terraform get
to download modules. You will notice that <code>.terraform</code> directory will be created that contains symlinks to the module.

;TF file <tt>~/git/dev101/vpc.tf</tt> calling 'vpc' module

variable "vpc_name" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_base" { description = "value comes from terrafrom.tfvars" }
variable "vpc_cidr_range" { description = "value comes from terrafrom.tfvars" }

module "vpc-dev" {
source = "../modules/vpc"
<span style="color: blue">name</span> = "${var.vpc_name}" #here we assign a value to 'name' variable
<span style="color: blue">cidr</span> = "${var.vpc_cidr_base}.${var.vpc_cidr_range}" }

output "vpc-name" { value = "${var.vpc_name }"}
output "vpc_id" { value = "${module.vpc-dev.<span style="color: green">id-from_module</span> }"}

;Module in <tt>~/git/modules/vpc/main.tf</tt>
variable "name" { description = "variable local to the module, value comes when calling the module" }
variable "cidr" { description = "local to the module, value passed on when calling the module" }

resource "aws_vpc" "scope" {
cidr_block = "${var.<span style="color: blue">cidr</span>}"
tags { Name = "${var.<span style="color: blue">name</span>}" }}

output "<span style="color: green">id-from_module</span>" { value = "${aws_vpc.scope.id}" }

Output variables is a way to output important data back when running <code>terraform apply</code>. These variables also can be recalled when .tfstate file has been populated using <code>terraform output VARIABLE-NAME</code> command.

$ terraform apply #this will use 'vpc' module

[[File:Terraform-module-apply.png|400px|none|left|Terraform-module-apply]]

Notice <span style="color: green">Outputs</span>. These outputs can be recalled also by:
$ terraform output vpc-name $ terraform output vpc_id
dev101 vpc-00e00c67

= Templates =
{{ Note | [https://github.com/hashicorp/terraform-guides/tree/master/infrastructure-as-code/terraform-0.12-examples/new-template-syntax Terraform 0.12+ New Template Syntax Example] }}
<source>
# Terraform version 0.12+ template syntax
%{ for name in var.names ~}
%{ if name == "Mary" }${name}%{ endif ~}
%{ endfor ~}
</source>

Dump a rendered <code>data.template_file</code> into a file to preview correctness of interpolations
<source>
#Dumps rendered template
resource "null_resource" "export_rendered_template" {
triggers = {
uid = "${uuid()}" #this causes to always run this resource
}
provisioner "local-exec" {
command = "cat > waf-policy.output.txt <<EOL\n${data.template_file.waf-whitelist-policy.rendered}\nEOL"
}
}
</source>

Example of creating
<source>
resource "aws_instance" "microservices" {
count = "${var.instance_count}"
subnet_id = "${element("${data.aws_subnet.private.*.id }", count.index)}"
user_data = "${element("${data.template_file.userdata.*.rendered}", count.index)}"
...
}
data "template_file" "userdata" {
count = "${var.instance_count}"
template = "${file("${path.root}/templates/user-data.tpl")}"
vars = {
vmname = "ms-${count.index + 1}-${var.vpc_name}"
}
}
#For debugging you can display an array of rendered templates with the output below:
output "userdata" { value = "${data.template_file.userdata.*.rendered}" }
</source>
{{ Note |
* resource <code>template_file is deprecated</code> in favour of <code>data template_file</code>
* Terraform 0.12+ offers new <code>template</code> function without a need of using a <code>data</code> object }}
== template json files ==
For working with JSON structures it's [https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template recommended] to use <code>jsonencode</code> function to simplify escaping, delimiters and get validated json in return.
<source lang=yaml>
resource "aws_iam_policy" "s3Bucket" {
name = s3Bucket"
policy = templatefile("${path.module}/templates/s3Bucket.json.tpl", {
S3BUCKETS = var.s3_buckets
})
}

variable "s3_buckets" {
type = list(string)
default = [ "aaa-bucket-111", "bbb-bucket-222" ]
}
</source>

Template file
<source lang=json>
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": ${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
# renders json array -> [ "arn:aws:s3:::aaa-bucket-111", "arn:aws:s3:::bbb-bucket-222" ]
}
]
}
</source>

Explain
<source lang=bash>
substitution syntax ${} local loop variable
| function jsonencode / templatefile function input variable, it's not ${} syntax
| | / /
${jsonencode([for BUCKET in S3BUCKETS : "arn:aws:s3:::${BUCKET}"])}
/ | / |\
/ for loop template variable | function cloasing bracket
indicates that the result to be an array[] closing bracket of the json array
</source>

== Resource ==
*[https://github.com/hashicorp/terraform/issues/1893 example of unique templates per instance]
*[https://github.com/hashicorp/terraform/pull/2140 recommendation of how to create unique templates per instance]

= Execute arbitrary code using null_resource and local-exec =
The null_resource allows to create terraform managed resource also saved in the state file but it uses 3rd party provisoners like local-exec, remote-exec, etc., allowing for arbitrary code execution. This should be only used when Terraform core does not provide the solution for your use case.
<source>
resource "null_resource" "attach_alb_am_wkr_ext" {

#depends_on sets up a dependency. So it depends on completion of another resource
#and it won't run if the resource does not change
#depends_on = [ "aws_cloudformation_stack.waf-alb" ]

#triggers save computed strings in tfstate file, if value changes on the next run it triggers a resource to be created
triggers = {
waf_id = "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" #produces WAF_id
alb_id = "${module.balancer_external_alb_instance.arn }" #produces full ALB_arn name
}

provisioner "local-exec" {
when = "create" #runs on: terraform apply
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional associate-web-acl --web-acl-id "${aws_cloudformation_stack.waf-alb.outputs.wafWebACL}" \
--resource-arn $ALBARN --region ${var.region}
EOF
}

provisioner "local-exec" {
when = "destroy" #runs only on: terraform destruct
command = <<EOF
ALBARN=$(aws elbv2 describe-load-balancers --region ${var.region} \
--name ${var.vpc}-${var.alb_class} \
--output text --query 'LoadBalancers[0].LoadBalancerArn') &&
aws waf-regional disassociate-web-acl --resource-arn $ALBARN --region ${var.region}
EOF
}
}
</source>

Note: By default the local-exec provisioner will use <code>/bin/sh -c "your<<EOFscript"</code> so it will not strip down any meta-characters like "double quotes" causing <tt>aws cli</tt> to fail. Therefore the output has been forced as <tt>text</tt>.

= <code>terraform providers</code> =
List all providers in your project to see versions and dependencies.
<source>
$ terraform providers
.
├── provider.aws ~> 2.44
├── provider.external ~> 1.2
├── provider.null ~> 2.1
├── provider.random ~> 2.2
├── provider.template ~> 2.1
├── module.kubernetes
│   ├── module.config
│   │   ├── provider.aws
│   │   ├── provider.helm ~> 0.10.4
│   │   ├── provider.kubernetes ~> 1.10.0
│   │   ├── provider.null (inherited)
│   │   ├── module.alb_ingress_controller
(...)
</source>

= terraform plugins cache =

Option 1.
Create <code>.terraformrc</code> file in $HOME directory and specify the cache directory.

<source lang=terraform>
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache/"
disable_checkpoint = true
EOF

# Delete per root module providers in <code>.terraform</code> directory
find /git/repositories -type d -name ".terraform" -exec rm -rf {}/providers \;
</source>

Option 2.
Set <code>TF_PLUGIN_CACHE_DIR</code> environment variable to an empty dir, then rerun <code>terraform init</code> to save downloaded
providers into shared (cache) directory.
<source>
export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugins-cache
</source>

Run <code>terraform init</code>. Local <code>.terraform</code> directory has been already deleted.
<source lang=terraform>
terraform init -backend-config=dev.backend.tfvars
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...
- Downloading plugin for provider "helm" (hashicorp/helm) 1.2.3...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...

Terraform has been successfully initialized!
</source>
:[[File:ClipCapIt-200714-085009.PNG]]

Although cache dir is used by all Terraform projects, the providers versioning still works and normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:
<source lang=bash>
$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f",
"external": "6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4",
"helm": "09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04",
"kubernetes": "7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff",
"null": "c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc",
"random": "791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed",
"template": "cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf"
}

find ~/.terraform.d/plugins -type f | xargs sha256sum
f08daaf64b9fca69978a40f88091d1a77fc9725fb04b0fec5e731609c53a025f /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-aws_v2.70.0_x4
6dad56007a3cb0ae9c4655c67d13502e51e38ca2673cf0f22a5fadce6803f9e4 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-external_v1.2.0_x4
c56285e7bd25a806bf86fcd4893edbe46e621a46e20fe24ef209b6fd0b7cf5fc /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4
791ef28ff31913d9b2ef0bedb97de98bebafe66d002bc2b9d01377e59a6cfaed /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-random_v2.3.0_x4
09b8ccb993f7d776555e811c856de006ac12b9fedfca15b07a85a6814914fd04 /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-helm_v1.2.3_x4
7ebf3273e622d1adb736e98f6fa5cc7e664c61b9171105b13c3b5ea8f8ebc5ff /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-kubernetes_v1.10.0_x4
cd8665642bf0f5b5f57a53050d10fd83415428c2dc6713b85e174e007fcc93bf /home/vagrant/.terraform.d/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
</source>
As you can see, the SHA256 hash for AWS provider saved in the <tt>lock.json</tt> file matches the hash of providera saved in the cache directory.

= AWS - [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI RDS aurora] - versioning =
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html#AuroraMySQL.Updates.20180206.CLI Engine name] 'aurora-mysql' refers to engine version 5.7.x and for version 5.6.10a engine name is aurora.
* The engine name for Aurora MySQL 2.x is aurora-mysql; the engine name for Aurora MySQL 1.x continues to be aurora.
* The engine version for Aurora MySQL 2.x is 5.7.12; the engine version for Aurora MySQL 1.x continues to be 5.6.10ann.
<syntaxhighlightjs lang=yaml>
module "db" {
source = "terraform-aws-modules/rds-aurora/aws"
version = "2.29.0"
name = "db"
engine = "aurora" # v5.6
engine_version = "5.6.mysql_aurora.1.23.0" # v5.6
#engine = "aurora-mysql" # v5.7
#engine_version = "5.7.mysql_aurora.2.09.0" # v5.7
...
}
</syntaxhighlightjs>

= [https://github.com/localstack/localstack localstack] - Mock AWS Services =
<source lang="shell">
pip install localstack
localstack start
SERVICES=kinesis,lambda,sqs,dynamodb DEBUG=1 localstack start
</source>
;Examples
* [https://github.com/MattSurabian/bad-terraform bad-terraform]

= [https://github.com/tfsec/tfsec tfsec] - Security Scanning TF code =
<source lang=bash>
LATEST=$(curl --silent -L "https://api.github.com/repos/tfsec/tfsec/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/tfsec/tfsec/releases/download/${LATEST}/tfsec-linux-amd64 -o /usr/local/bin/tfsec
sudo chmod +x /usr/local/bin/tfsec

# Use with docker
docker run --rm -it -v "$(pwd):/src" liamg/tfsec /src

# Usage
tfsec .
</source>

= [https://github.com/terraform-linters/tflint tflint] - validate provider-specific issues =
Requires Terraform >= 0.12
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -L https://github.com/terraform-linters/tflint/releases/download/${LATEST}/tflint_linux_amd64.zip -o $TEMPDIR/tflint_linux_amd64.zip
sudo unzip $TEMPDIR/tflint_linux_amd64.zip -d /usr/local/bin

# Configure tflint
# | Current directory (./.tflint.hcl)
# | Home directory (~/.tflint.hcl)
tflint --config other_config.hcl

## Add plugins
https://github.com/terraform-linters/tflint/tree/master/docs/rules
cat > ./.tflint.hcl <<EOF
plugin "aws" {
enabled = true
version = "0.5.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "google" {
enabled = true
version = "0.15.0"
source = "github.com/terraform-linters/tflint-ruleset-google"
}
EOF

# Usage
tflint --module
tflint --module --var-file=dev.tfvars

# Use with docker
docker pull ghcr.io/terraform-linters/tflint:latest
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1
docker run --rm -v $(pwd):/src -t ghcr.io/terraform-linters/tflint:v0.34.1 -v

# Init and check
docker run --rm -v $(pwd):/src -t --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 -c "tflint --init; tflint /src/"

## It looks important that tflint is executed in terrafrom root path, thus `cd /src`
docker run --rm -v $(pwd):/src -t -e TFLINT_LOG=debug --entrypoint /bin/sh ghcr.io/terraform-linters/tflint:v0.34.1 \
-c "cd /src; tflint --init; tflint --var-file=environments/gcp-dev.tfvars --module"
</source>

= [https://github.com/terraform-docs/terraform-docs terraform-docs] - generate Terraform documentation =
<source lang=bash>
# Install the binary
VERSION=$(curl --silent "https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest" | jq -r .tag_name); echo $VERSION
wget https://github.com/terraform-docs/terraform-docs/releases/download/$VERSION/terraform-docs-$VERSION-linux-amd64.tar.gz
tar xzvf terraform-docs-$VERSION-linux-amd64.tar.gz
sudo install terraform-docs /usr/local/bin/terraform-docs

# Use with docker
docker run --rm --volume "$(pwd):/src" -u $(id -u) quay.io/terraform-docs/terraform-docs:0.16.0 markdown /src

# Usage
terraform-docs . > README.md
</source>

= [https://github.com/cycloidio/inframap InfraMap] - plot your Terraform state =
<source lang=bash>
VERSION=$(curl --silent "https://api.github.com/repos/cycloidio/inframap/releases/latest" | jq -r .tag_name); echo $VERSION
TEMPDIR=$(mktemp -d)
curl -L https://github.com/cycloidio/inframap/releases/download/${VERSION}/inframap-linux-amd64.tar.gz -o $TEMPDIR/inframap-linux-amd64.tar.gz
tar xzvf $TEMPDIR/inframap-linux-amd64.tar.gz -C $TEMPDIR inframap-linux-amd64
sudo install $TEMPDIR/inframap-linux-amd64 /usr/local/bin/inframap

# Install graphviz, it contains the `dot` program
sudo apt install graphviz

# Install GraphEasy
## Cpan manager
sudo apt install cpanminus # install perl packet managet
sudo cpanm Graph::Easy # Graph-Easy-0.76 as of 2021-07

## Apt-get (tested with Ubuntu 20.04 LTS)
sudo apt install libgraph-easy-perl # Graph::Easy v0.76

# a sample usage
cat input.dot | graph-easy --from=dot --as_ascii
</source>

Usage inframap
<source lang=bash>
The most important subcommands are:
* generate: generates the graph from STDIN or file, STDIN can be .tf files/modules or .tfstate
* prune: removes all unnecessary information from the state or HCL (not supported yet) so it can be shared without any security concerns

# Generate your infrastructure graph in a DOT representation from: Terraform files or state file
cat terraform.tf | inframap generate --printer dot --hcl | tee graph.dot
cat terraform.tfstate | inframap generate --printer dot --tfstate | tee graph.dot

# `prune` command will sanitize and anonymize content of the files
cat terraform.tfstate | inframap prune --canonicals --tfstate > cleaned.tfstate

# Pipe all the previous commands. ASCII graph is generated using graph-easy
cat terraform.tfstate | inframap prune --tfstate | inframap generate --tfstate | graph-easy

# from State file - visualizing with `dot` or `graph-easy`
inframap generate state.tfstate | dot -Tpng > graph.png
inframap generate state.tfstate | graph-easy

# from HCL
inframap generate terraform.tf | graph-easy
inframap generate ./my-module/ | graph-easy # or HCL module

# using docker image (assuming that your Terraform files are in the working directory)
docker run --rm -v ${PWD}:/opt cycloid/inframap generate /opt/terraform.tfstate
</source>

Example of EKS module
:[[File:ClipCapIt-210716-090202.PNG|400px]]

= [https://github.com/Pluralith/pluralith-cli/releases Pluralith] =
<source lang=bash>
# Install
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli/releases/latest" | jq -r .tag_name); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli/releases/download/${VERSION}/pluralith_cli_linux_amd64_${VERSION} -o pluralith_cli_linux_amd64_${VERSION}
sudo install pluralith_cli_linux_amd64_${VERSION} /usr/local/bin/pluralith

# Install pluralith-cli-graphing
VERSION=$(curl --silent "https://api.github.com/repos/Pluralith/pluralith-cli-graphing-release/releases/latest" | jq -r .tag_name | tr -d v); echo $VERSION
curl -L https://github.com/Pluralith/pluralith-cli-graphing-release/releases/download/v${VERSION}/pluralith_cli_graphing_linux_amd64_${VERSION} -o pluralith_cli_graphing_linux_amd64_${VERSION}
sudo install pluralith_cli_graphing_linux_amd64_${VERSION} ~/Pluralith/bin/pluralith-cli-graphing

# Check versions
pluralith version
parsing response failed -> GetGitHubRelease: %!w(<nil>)
_
|_)| _ _ |._|_|_
| ||_|| (_||| | | |

→ CLI Version: 0.2.2
→ Graph Module Version: 0.2.1

# Usage
pluralith login --api-key $PLURALITH_API_KEY

# Generate PDF graph locally
pluralith <terrafom-root-folder> --var-file environments/dev.tfvars graph --local-only
</source>

= [https://github.com/flosell/iam-policy-json-to-terraform iam-policy-json-to-terraform] =
Convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/flosell/iam-policy-json-to-terraform/releases/latest" | jq -r .tag_name); echo $LATEST
sudo curl -L https://github.com/flosell/iam-policy-json-to-terraform/releases/download/${LATEST}/iam-policy-json-to-terraform_amd64 -o /usr/local/bin/iam-policy-json-to-terraform
sudo chmod +x /usr/local/bin/iam-policy-json-to-terraform

# Usage:
iam-policy-json-to-terraform < some-policy.json
</source>

= [https://github.com/hieven/terraform-visual terraform-visual] =
<source lang=bash>
# Install
sudo apt-get update
sudo apt install nodejs npm
sudo npm install -g @terraform-visual/cli

# Usage
terraform plan -out=plan.out # Run plan and output as a file
terraform show -json plan.out > plan.json # Read plan file and output it in JSON format
terraform-visual --plan plan.json
firefox terraform-visual-report/index.html
</source>

= [https://github.com/cloudskiff/driftctl driftctl] =
Measures infrastructure as code coverage, and tracks infrastructure drift.
IaC: Terraform, Cloud providers: AWS, GitHub (Azure and GCP on the roadmap for 2021). Spot discrepancies as they happen: driftctl is a free and open-source CLI that warns of infrastructure drifts and fills in the missing piece in your DevSecOps toolbox.

;Features [https://docs.driftctl.com/ docs]
* Scan cloud provider and map resources with IaC code
* Analyze diffs, and warn about drift and unwanted unmanaged resources
* Allow users to ignore resources
* Multiple output formats

Install
<source lang=bash>
curl -L https://github.com/snyk/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl
install ./driftctl /usr/local/bin/driftctl
driftctl version
</source>

[https://docs.driftctl.com/0.39.0/usage/cmd/scan-usage Detect drift on GCP]
<source lang=bash>
source <(driftctl completion bash)
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json
export CLOUDSDK_CORE_PROJECT=<myproject_id>
driftctl scan --to="gcp+tf"
driftctl scan --to="gcp+tf" --deep --output html://output.html
driftctl scan --to="gcp+tf" --from tfstate+gs://my-bucket/path/to/state.tfstate # Use this when working with workspaces
</source>

= [https://github.com/infracost/infracost infracost] =
Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform.

<source lang=bash>
# Downloads the CLI based on your OS/arch and puts it in /usr/local/bin
curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

# Register for a free API key
infracost register # The key is saved in ~/.config/infracost/credentials.yml.

# Show cost breakdown on live infra
infracost breakdown --path terraform_nlb_static_eips

# Show cost breakdown based on Terraform plan
cd path/to/src_code
terraform init
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json

## run via binary
infracost breakdown --path plan.json
infracost breakdown --path plan.json --show-skipped --format html > /vagrant/infracost.html
infracost diff --path plan.json

## run via Docker
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
</source>

Example output
<source>
## Cost breakdown
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 breakdown --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json
Name Monthly Qty Unit Monthly Cost
module.gke.google_container_cluster.primary
├─ Cluster management fee 730 hours $73.00
└─ default_pool
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
module.gke.google_container_node_pool.pools["default-node-pool"]
├─ Instance usage (Linux/UNIX, on-demand, e2-medium) 6,570 hours $242.16
└─ Standard provisioned storage (pd-standard) 900 GiB $36.00
OVERALL TOTAL $629.31
──────────────────────────────────
11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free

## Cost difference
docker run -it --rm --volume "$(pwd):/src" -u $(id -u) -e INFRACOST_API_KEY=$INFRACOST_API_KEY infracost/infracost:0.9.15 diff --path /src/plan.json
Detected Terraform plan JSON file at /src/plan.json
✔ Calculating monthly cost estimate

Project: /src/plan.json

+ module.gke.google_container_cluster.primary
+$351
+ Cluster management fee
+$73.00
+ default_pool
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
+ node_pool[0]
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
$0.00
+ Standard provisioned storage (pd-standard)
$0.00
+ module.gke.google_container_node_pool.pools["default-node-pool"]
+$278
+ Instance usage (Linux/UNIX, on-demand, e2-medium)
+$242
+ Standard provisioned storage (pd-standard)
+$36.00
Monthly cost change for /src/plan.json
Amount: +$629 ($0.00 → $629)

──────────────────────────────────
Key: ~ changed, + added, - removed

11 cloud resources were detected, rerun with --show-skipped to see details:
∙ 2 were estimated, 2 include usage-based costs, see https://infracost.io/usage-file
∙ 9 were free
</source>

Resources:
* DockerHub: https://hub.docker.com/r/infracost/infracost/tags

= [https://tfautomv.dev/ tfautomv - Terraform refactor] =
Tfautomv writes moved blocks for you so your refactoring is quicker and less error-prone.

<source lang=bash>
tfautomv -dry-run
tfautomv -show-analysis
</source>

= [https://www.davidc.net/sites/default/subnets/subnets.html?network=192.168.0.0&mask=22&division=19.3d431 Subnetting] =
Very useful page for subnetting: https://www.davidc.net/sites/default/subnets/subnets.html

= Resources =
*[https://discuss.hashicorp.com/u/apparentlymart apparentlymart] The Hero! discuss.hashicorp.com
*[https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca Comprehensive-guide-to-terraform] gruntwork.io
*[https://github.com/antonbabenko/terraform-best-practices Terraform good practices] naming conventions, etc..
*[https://www.runatlantis.io/ Atlantis] Terraform Pull Request Automation, Listens for webhooks from GitHub/GitLab/Bitbucket/Azure DevOps, Runs terraform commands remotely and comments back with their output.

Kubernetes/Kustomize

2024-08-06T22:33:25Z

Pio2pio: /* Embedded versions */

= [https://kustomize.io/ Kustomize] =
* [https://kubectl.docs.kubernetes.io/ kubectl+kustomize] SIG CLI
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

= Embedded versions =
;FluxCD (v2): runs kustomize-controller and helm-controller, use the following methods to find out versions of the embedded components:

* kustomize-controller versions [https://github.com/fluxcd/kustomize-controller/blob/main/CHANGELOG.md#100 CHANGELOG]:
** 0.27.0 - Kustomize v4.5.7
** 1.0.0 - Kustomize v5.0.3 (introduced in 1.0.0-rc.4 release)
** 1.2.0 - Kustomize v5.3.0, SOPS v3.8.1

* helm-controller versions [https://github.com/fluxcd/helm-controller/blob/main/CHANGELOG.md#0370 CHANGELOG]:
** v0.37.0 - helm v3.13.2, post-renderer kustomize v5.3.0

;ArgoCD
* v2.10.6+d504d2b
** kustomize: v5.2.1 2023-10-19T20:13:51Z
** Helm: v3.14.3+gf03cc04
** kubectl: v0.26.11

= Install =
<source lang=bash>
# Detects your OS and downloads kustomize binary to cwd
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash

# Install on Linux - option2
VERSION=v4.1.2
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases" | jq -r '.[].tag_name | select(. | contains("kustomize"))' | sort | tail -1 | cut -d"/" -f2); echo $VERSION
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize_${VERSION}

kustomize version --short
{kustomize/v4.1.2 2021-04-15T20:38:06Z }

kustomize version
v5.3.0
</source>

= Kustomize build workflow =
* [https://github.com/kubernetes-sigs/kustomize/issues/2052 kustomize vars] - use <code>envsubst</code> instead
<source>$ kustomize build ~/target</source>
# load universal k8s object descriptions
# read <code>kustomization.yaml</code> from '''target'''
# kustomize '''bases''' (recurse 2-5)
# load and/or generate resources
# apply '''target's''' kustomization operations
# fix name references
# emit yaml

= <code>kustomize.yaml</code>
A build stage first
* processes resources,
* then it processes generators, adding to the resource list under consideration,
* then it processes transformers to modify the list,
* and finally runs validators to check the list for whatever error.
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- {pathOrUrl}
- what resources you want to customize

# cross-cutting fields
namespace: custom
namePrefix: dev-
nameSuffix: "-svc"
commonLabels:
app: web
commonAnnotations:
value: app

generators:
- {pathOrUrl}
- what new resources should be created.
generatorOptions:
disableNameSuffixHash: true
labels:
env: prod
annotations:
app: custom

transformers:
- {pathOrUrl}
- what to transform in above mentioned resources

validators:
- {pathOrUrl}
- ...

</source>

== Patches ==
;patchStrategicMerge: Kubernetes supports a customized version of JSON merge patch called strategic merge patch. This patch format is used by <code>kubectl apply</code>, <code>kubectl edit</code> and <code>kubectl patch</code>, and contains specialized directives to control how specific fields are merged.

= Example 101 =
{{Note|Bases have been deprecated in v2.1.0 [https://kubernetes-sigs.github.io/kustomize/blog/2019/06/18/v2.1.0/#resources-expanded-bases-deprecated resources-expanded-bases-deprecated]}}

* [https://kustomize.io/tutorial Kustomize builder] note that it operates on the 1st yaml document
{| class="wikitable"
|+ Example 101 - environment type overrides
|-
! base/kustomization.yaml
! overlays/dev/kustomization.yaml
! overlays/prod/kustomization.yaml
|- style="vertical-align:top;"
| <source lang=bash>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
app: sonarqube
resources:
- gateway.yaml
- virtual-service.yaml
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
|}
<source lang=bash>
.
├── base
│   ├── gateway.yaml
│   ├── kustomization.yaml
│   └── virtual-service.yaml
└── overlays # more contextual this directory could be called 'environments'
   ├── dev
   │   ├── gateway_patch.yaml
   │   ├── kustomization.yaml
   │   └── virtual-service_patch.yaml
   └── prod
   ├── gateway_patch.yaml
   ├── kustomization.yaml
   └── virtual-service_patch.yaml

# Build kuctomized output
kustomize version --short # -> {kustomize/v3.8.2 2020-08-29T17:44:01Z }
kustomize build overlays/dev # apply patches
kustomize build base # run common functions (as described in base/kustomize.yaml) against the whole code base
</source>

What happens?
# <code>kustomize build overlays/dev</code> finds <code>kustomization.yaml</code>, that describes:
## <code>patches: [gateway_patch.yaml, virtual-service_patch.yaml]</code> to be used over the base <code>resources: [../../base]</code>. There are 3 type of patches: patches, patchesStrategicMerge, [https://skryvets.com/blog/2019/05/15/kubernetes-kustomize-json-patches-6902 patchesJson6902] to choose from
# <code>overlays/dev/kustomization.yaml</code> cascades to the base (source of manifests to be changed) via directive <code>resources: ["../../base"]</code>
# The base directory contains and runs its own <code>kustomization.yaml</code> file.
# The <code>base/kustomization.yaml</code> contains common operations, eg. <code>commonLabels, namePrefix</code> functions to be applied to whole code base.
# Then patch file(s) are applied eg. <code>gateway_patch.yaml</code> contains enough information to identify a resource/object and apply changes.

So, what happens
<source lang=bash>
# Applying the patch: overlays/dev/gateway_patch.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: sonarqube
spec:
servers:
- port:
number: 443
name: http
protocol: HTTP
hosts:
- sonarqube-dev.acme.com # <- override
# |
# | over the base
# v

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube.acme.com
port:
name: http
number: 443
protocol: HTTP
# |
# | results with
# v

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
owner: piotr # <- label added by base kustomize.yaml fn
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube-dev.acme.com # <- patch override
port:
name: http
number: 443
protocol: HTTP
</source>

Check yourselves
<source lang=bash>
# __unchanged manifest_ _base kustomization_ ___patch overlay____________
vimdiff <(cat base/gateway.yaml) <(kustomize build base) <(kustomize build overlays/dev)
</source>
:[[File:ClipCapIt-200910-010734.PNG]]

= Cheatsheet =
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md Helm charts - last mile]

= Patch [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/patchMultipleObjects.md multiple objects] =
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patches:
- path: patch.json
target:
kind: PersistentVolume
version: v1
group: ""
name: volume-(data|master)-\d # regex match
labelSelector: |
app.kubernetes.io/component=storage,
app.kubernetes.io/name=elasticsearch
</syntaxhighlight>

Component example
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
- target:
kind: HelmRelease
version: v2beta1
group: helm.toolkit.fluxcd.io
name: external-dns-.+$ # [1]
patch: |-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: ALL # [2]
namespace: flux-system
spec:
values:
tolerations:
- key: "components.gke.io/gke-managed-components"
operator: Exists
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: "predictx/workload"
operator: In
values:
- "infra"

# [1] Regex match
# [2] The name is replaced by a name from the matched object. Any value ie 'ALL' is required.
</syntaxhighlight>

= Delete an object from the base =
''Strategic Merge Patch'' provides some patch options like replace, merge, and delete. The simple 'patch' can only patch (manipulate) it.
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patchesStrategicMerge:
- |-
apiVersion: v1
kind: Namespace
metadata:
name: unwanted-namespace
$patch: delete
</syntaxhighlight>

= secretGenerator =
Secrets can be generated from environment variables. Within a template file there is a list of variables, where the variable will become a key and it's value the value.

Environment variable secret template
<source lang=bash>
GIT_USERNAME
GIT_PASSWORD
GIT_CREDENTIALS
</source>

Kustomization
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: argocd-git-secret
envs:
- git.env
options:
disableNameSuffixHash: true
</source>

= Patch - add an item to a list =
* https://stackoverflow.com/questions/71622419/adding-items-to-a-list-with-kubectl-kustomize
* [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch Strategic merge patch docs]

In the standard JSON merge patch, JSON objects are always merged but lists are always replaced. Often that isn't what we want. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. Read more at [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch strategic merge patch docs].

<source lang=yaml>
patchesJson6902:
- patch: |-
- op: add
path: /spec/valuesFrom/-
value: # below map will be added as an item to the list, pay attention to `-` sign at the end of path
kind: ConfigMap
name: values-1-yaml
target:
group: helm.toolkit.fluxcd.io
kind: HelmRelease
name: kube-prometheus-stack
version: v2beta1
</source>

= [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacements] =
* https://stackoverflow.com/questions/71358674/kustomize-how-to-reference-a-value-from-a-configmap-in-another-resource-overlay
Use of vars is getting deprecated please use replacements.

= Known issues =
* [https://github.com/kubernetes-sigs/kustomize/issues/2034 commonLabels altering podSelector.matchLabels] and [https://github.com/kubernetes-sigs/kustomize/issues/157 Allow excluding some label selectors from commonLabels]
In some settings it makes sense for <code>commonLabels</code> to be included in selectors, and in some settings it doers not make sense to include them in selectors. Kustomize includes by default, and there is no way to opt out. As workaround, you can convert <code>matchLabels</code> to <code>matchExpressions</code> and Kustomize won't touch them. API docs

<source lang=yaml>
- podSelector:
matchLabels:
app: mongodb-backup
</source>

is equivalent with
<source lang=yaml>
- podSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-backup
</source>
and Kustomize will keep its hands off.

= Resources =
* [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacement transform is deprecating Vars]
* [https://github.com/kubernetes-sigs/kustomize Kustomize sig]
* [https://kubectl.docs.kubernetes.io/guides/config_management/components/ v3.7.0+ Components]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#kustomization Glossary]
* [https://kubectl.docs.kubernetes.io/references/kustomize/ Kustomization File Fields]
* [https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html Kustomize - examples] kubectl.docs.kubernetes.io
* [https://kubectl.docs.kubernetes.io/pages/app_composition_and_deployment/structure_directories.html Kustomize structure_directories]
* [https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html reference] Good!
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/inlinePatch.md inlinePatch]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md kustomization of a helm chart]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configureBuiltinPlugin.md#using-the-commonlabels-and-commonannotations-fields Customize Kustomize] annotation and label buildin transformer

Kubernetes/Kustomize

2024-08-06T22:30:14Z

Pio2pio: /* Embedded versions */

= [https://kustomize.io/ Kustomize] =
* [https://kubectl.docs.kubernetes.io/ kubectl+kustomize] SIG CLI
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

= Embedded versions =
;Flux deployment: runs kustomize-controller and helm-controller, use the following methods to find out versions of the embedded components:

* kustomize-controller versions [https://github.com/fluxcd/kustomize-controller/blob/main/CHANGELOG.md#100 CHANGELOG]:
** 0.27.0 - Kustomization v4.5.7
** 1.0.0 - Kustomizatin v5.0.3 (introduced in 1.0.0-rc.4 release)
** 1.2.0 - Kustomize v5.3.0, SOPS v3.8.1

* helm-controller versions [https://github.com/fluxcd/helm-controller/blob/main/CHANGELOG.md#0370 CHANGELOG]:
** v0.37.0 - helm v3.13.2, post-renderer kustomize v5.3.0

= Install =
<source lang=bash>
# Detects your OS and downloads kustomize binary to cwd
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash

# Install on Linux - option2
VERSION=v4.1.2
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases" | jq -r '.[].tag_name | select(. | contains("kustomize"))' | sort | tail -1 | cut -d"/" -f2); echo $VERSION
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize_${VERSION}

kustomize version --short
{kustomize/v4.1.2 2021-04-15T20:38:06Z }

kustomize version
v5.3.0
</source>

= Kustomize build workflow =
* [https://github.com/kubernetes-sigs/kustomize/issues/2052 kustomize vars] - use <code>envsubst</code> instead
<source>$ kustomize build ~/target</source>
# load universal k8s object descriptions
# read <code>kustomization.yaml</code> from '''target'''
# kustomize '''bases''' (recurse 2-5)
# load and/or generate resources
# apply '''target's''' kustomization operations
# fix name references
# emit yaml

= <code>kustomize.yaml</code>
A build stage first
* processes resources,
* then it processes generators, adding to the resource list under consideration,
* then it processes transformers to modify the list,
* and finally runs validators to check the list for whatever error.
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- {pathOrUrl}
- what resources you want to customize

# cross-cutting fields
namespace: custom
namePrefix: dev-
nameSuffix: "-svc"
commonLabels:
app: web
commonAnnotations:
value: app

generators:
- {pathOrUrl}
- what new resources should be created.
generatorOptions:
disableNameSuffixHash: true
labels:
env: prod
annotations:
app: custom

transformers:
- {pathOrUrl}
- what to transform in above mentioned resources

validators:
- {pathOrUrl}
- ...

</source>

== Patches ==
;patchStrategicMerge: Kubernetes supports a customized version of JSON merge patch called strategic merge patch. This patch format is used by <code>kubectl apply</code>, <code>kubectl edit</code> and <code>kubectl patch</code>, and contains specialized directives to control how specific fields are merged.

= Example 101 =
{{Note|Bases have been deprecated in v2.1.0 [https://kubernetes-sigs.github.io/kustomize/blog/2019/06/18/v2.1.0/#resources-expanded-bases-deprecated resources-expanded-bases-deprecated]}}

* [https://kustomize.io/tutorial Kustomize builder] note that it operates on the 1st yaml document
{| class="wikitable"
|+ Example 101 - environment type overrides
|-
! base/kustomization.yaml
! overlays/dev/kustomization.yaml
! overlays/prod/kustomization.yaml
|- style="vertical-align:top;"
| <source lang=bash>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
app: sonarqube
resources:
- gateway.yaml
- virtual-service.yaml
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
|}
<source lang=bash>
.
├── base
│   ├── gateway.yaml
│   ├── kustomization.yaml
│   └── virtual-service.yaml
└── overlays # more contextual this directory could be called 'environments'
   ├── dev
   │   ├── gateway_patch.yaml
   │   ├── kustomization.yaml
   │   └── virtual-service_patch.yaml
   └── prod
   ├── gateway_patch.yaml
   ├── kustomization.yaml
   └── virtual-service_patch.yaml

# Build kuctomized output
kustomize version --short # -> {kustomize/v3.8.2 2020-08-29T17:44:01Z }
kustomize build overlays/dev # apply patches
kustomize build base # run common functions (as described in base/kustomize.yaml) against the whole code base
</source>

What happens?
# <code>kustomize build overlays/dev</code> finds <code>kustomization.yaml</code>, that describes:
## <code>patches: [gateway_patch.yaml, virtual-service_patch.yaml]</code> to be used over the base <code>resources: [../../base]</code>. There are 3 type of patches: patches, patchesStrategicMerge, [https://skryvets.com/blog/2019/05/15/kubernetes-kustomize-json-patches-6902 patchesJson6902] to choose from
# <code>overlays/dev/kustomization.yaml</code> cascades to the base (source of manifests to be changed) via directive <code>resources: ["../../base"]</code>
# The base directory contains and runs its own <code>kustomization.yaml</code> file.
# The <code>base/kustomization.yaml</code> contains common operations, eg. <code>commonLabels, namePrefix</code> functions to be applied to whole code base.
# Then patch file(s) are applied eg. <code>gateway_patch.yaml</code> contains enough information to identify a resource/object and apply changes.

So, what happens
<source lang=bash>
# Applying the patch: overlays/dev/gateway_patch.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: sonarqube
spec:
servers:
- port:
number: 443
name: http
protocol: HTTP
hosts:
- sonarqube-dev.acme.com # <- override
# |
# | over the base
# v

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube.acme.com
port:
name: http
number: 443
protocol: HTTP
# |
# | results with
# v

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
owner: piotr # <- label added by base kustomize.yaml fn
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube-dev.acme.com # <- patch override
port:
name: http
number: 443
protocol: HTTP
</source>

Check yourselves
<source lang=bash>
# __unchanged manifest_ _base kustomization_ ___patch overlay____________
vimdiff <(cat base/gateway.yaml) <(kustomize build base) <(kustomize build overlays/dev)
</source>
:[[File:ClipCapIt-200910-010734.PNG]]

= Cheatsheet =
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md Helm charts - last mile]

= Patch [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/patchMultipleObjects.md multiple objects] =
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patches:
- path: patch.json
target:
kind: PersistentVolume
version: v1
group: ""
name: volume-(data|master)-\d # regex match
labelSelector: |
app.kubernetes.io/component=storage,
app.kubernetes.io/name=elasticsearch
</syntaxhighlight>

Component example
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
- target:
kind: HelmRelease
version: v2beta1
group: helm.toolkit.fluxcd.io
name: external-dns-.+$ # [1]
patch: |-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: ALL # [2]
namespace: flux-system
spec:
values:
tolerations:
- key: "components.gke.io/gke-managed-components"
operator: Exists
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: "predictx/workload"
operator: In
values:
- "infra"

# [1] Regex match
# [2] The name is replaced by a name from the matched object. Any value ie 'ALL' is required.
</syntaxhighlight>

= Delete an object from the base =
''Strategic Merge Patch'' provides some patch options like replace, merge, and delete. The simple 'patch' can only patch (manipulate) it.
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patchesStrategicMerge:
- |-
apiVersion: v1
kind: Namespace
metadata:
name: unwanted-namespace
$patch: delete
</syntaxhighlight>

= secretGenerator =
Secrets can be generated from environment variables. Within a template file there is a list of variables, where the variable will become a key and it's value the value.

Environment variable secret template
<source lang=bash>
GIT_USERNAME
GIT_PASSWORD
GIT_CREDENTIALS
</source>

Kustomization
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: argocd-git-secret
envs:
- git.env
options:
disableNameSuffixHash: true
</source>

= Patch - add an item to a list =
* https://stackoverflow.com/questions/71622419/adding-items-to-a-list-with-kubectl-kustomize
* [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch Strategic merge patch docs]

In the standard JSON merge patch, JSON objects are always merged but lists are always replaced. Often that isn't what we want. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. Read more at [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch strategic merge patch docs].

<source lang=yaml>
patchesJson6902:
- patch: |-
- op: add
path: /spec/valuesFrom/-
value: # below map will be added as an item to the list, pay attention to `-` sign at the end of path
kind: ConfigMap
name: values-1-yaml
target:
group: helm.toolkit.fluxcd.io
kind: HelmRelease
name: kube-prometheus-stack
version: v2beta1
</source>

= [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacements] =
* https://stackoverflow.com/questions/71358674/kustomize-how-to-reference-a-value-from-a-configmap-in-another-resource-overlay
Use of vars is getting deprecated please use replacements.

= Known issues =
* [https://github.com/kubernetes-sigs/kustomize/issues/2034 commonLabels altering podSelector.matchLabels] and [https://github.com/kubernetes-sigs/kustomize/issues/157 Allow excluding some label selectors from commonLabels]
In some settings it makes sense for <code>commonLabels</code> to be included in selectors, and in some settings it doers not make sense to include them in selectors. Kustomize includes by default, and there is no way to opt out. As workaround, you can convert <code>matchLabels</code> to <code>matchExpressions</code> and Kustomize won't touch them. API docs

<source lang=yaml>
- podSelector:
matchLabels:
app: mongodb-backup
</source>

is equivalent with
<source lang=yaml>
- podSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-backup
</source>
and Kustomize will keep its hands off.

= Resources =
* [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacement transform is deprecating Vars]
* [https://github.com/kubernetes-sigs/kustomize Kustomize sig]
* [https://kubectl.docs.kubernetes.io/guides/config_management/components/ v3.7.0+ Components]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#kustomization Glossary]
* [https://kubectl.docs.kubernetes.io/references/kustomize/ Kustomization File Fields]
* [https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html Kustomize - examples] kubectl.docs.kubernetes.io
* [https://kubectl.docs.kubernetes.io/pages/app_composition_and_deployment/structure_directories.html Kustomize structure_directories]
* [https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html reference] Good!
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/inlinePatch.md inlinePatch]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md kustomization of a helm chart]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configureBuiltinPlugin.md#using-the-commonlabels-and-commonannotations-fields Customize Kustomize] annotation and label buildin transformer

Kubernetes/Kustomize

2024-08-06T22:27:07Z

Pio2pio: /* Embedded versions */

= [https://kustomize.io/ Kustomize] =
* [https://kubectl.docs.kubernetes.io/ kubectl+kustomize] SIG CLI
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

= Embedded versions =
;Flux deployment: runs kustomize-controller and helm-controller, use the following methods to find out versions of the embedded components:

;kustomize-controller - [https://github.com/fluxcd/kustomize-controller/blob/main/CHANGELOG.md#100 CHANGELOG]
* 0.27.0 - Kustomization v4.5.7
* 1.0.0 - Kustomizatin v5.0.3 (introduced in 1.0.0-rc.4 release)
* 1.2.0 - Kustomize v5.3.0, SOPS v3.8.1

;helm-controller; [https://github.com/fluxcd/helm-controller/blob/main/CHANGELOG.md#0370 CHANGELOG]
* v0.37.0 - helm v3.13.2, post-renderer kustomize v5.3.0

= Install =
<source lang=bash>
# Detects your OS and downloads kustomize binary to cwd
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash

# Install on Linux - option2
VERSION=v4.1.2
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases" | jq -r '.[].tag_name | select(. | contains("kustomize"))' | sort | tail -1 | cut -d"/" -f2); echo $VERSION
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize_${VERSION}

kustomize version --short
{kustomize/v4.1.2 2021-04-15T20:38:06Z }

kustomize version
v5.3.0
</source>

= Kustomize build workflow =
* [https://github.com/kubernetes-sigs/kustomize/issues/2052 kustomize vars] - use <code>envsubst</code> instead
<source>$ kustomize build ~/target</source>
# load universal k8s object descriptions
# read <code>kustomization.yaml</code> from '''target'''
# kustomize '''bases''' (recurse 2-5)
# load and/or generate resources
# apply '''target's''' kustomization operations
# fix name references
# emit yaml

= <code>kustomize.yaml</code>
A build stage first
* processes resources,
* then it processes generators, adding to the resource list under consideration,
* then it processes transformers to modify the list,
* and finally runs validators to check the list for whatever error.
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- {pathOrUrl}
- what resources you want to customize

# cross-cutting fields
namespace: custom
namePrefix: dev-
nameSuffix: "-svc"
commonLabels:
app: web
commonAnnotations:
value: app

generators:
- {pathOrUrl}
- what new resources should be created.
generatorOptions:
disableNameSuffixHash: true
labels:
env: prod
annotations:
app: custom

transformers:
- {pathOrUrl}
- what to transform in above mentioned resources

validators:
- {pathOrUrl}
- ...

</source>

== Patches ==
;patchStrategicMerge: Kubernetes supports a customized version of JSON merge patch called strategic merge patch. This patch format is used by <code>kubectl apply</code>, <code>kubectl edit</code> and <code>kubectl patch</code>, and contains specialized directives to control how specific fields are merged.

= Example 101 =
{{Note|Bases have been deprecated in v2.1.0 [https://kubernetes-sigs.github.io/kustomize/blog/2019/06/18/v2.1.0/#resources-expanded-bases-deprecated resources-expanded-bases-deprecated]}}

* [https://kustomize.io/tutorial Kustomize builder] note that it operates on the 1st yaml document
{| class="wikitable"
|+ Example 101 - environment type overrides
|-
! base/kustomization.yaml
! overlays/dev/kustomization.yaml
! overlays/prod/kustomization.yaml
|- style="vertical-align:top;"
| <source lang=bash>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
app: sonarqube
resources:
- gateway.yaml
- virtual-service.yaml
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
| <source lang=bash>
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
</source>
|}
<source lang=bash>
.
├── base
│   ├── gateway.yaml
│   ├── kustomization.yaml
│   └── virtual-service.yaml
└── overlays # more contextual this directory could be called 'environments'
   ├── dev
   │   ├── gateway_patch.yaml
   │   ├── kustomization.yaml
   │   └── virtual-service_patch.yaml
   └── prod
   ├── gateway_patch.yaml
   ├── kustomization.yaml
   └── virtual-service_patch.yaml

# Build kuctomized output
kustomize version --short # -> {kustomize/v3.8.2 2020-08-29T17:44:01Z }
kustomize build overlays/dev # apply patches
kustomize build base # run common functions (as described in base/kustomize.yaml) against the whole code base
</source>

What happens?
# <code>kustomize build overlays/dev</code> finds <code>kustomization.yaml</code>, that describes:
## <code>patches: [gateway_patch.yaml, virtual-service_patch.yaml]</code> to be used over the base <code>resources: [../../base]</code>. There are 3 type of patches: patches, patchesStrategicMerge, [https://skryvets.com/blog/2019/05/15/kubernetes-kustomize-json-patches-6902 patchesJson6902] to choose from
# <code>overlays/dev/kustomization.yaml</code> cascades to the base (source of manifests to be changed) via directive <code>resources: ["../../base"]</code>
# The base directory contains and runs its own <code>kustomization.yaml</code> file.
# The <code>base/kustomization.yaml</code> contains common operations, eg. <code>commonLabels, namePrefix</code> functions to be applied to whole code base.
# Then patch file(s) are applied eg. <code>gateway_patch.yaml</code> contains enough information to identify a resource/object and apply changes.

So, what happens
<source lang=bash>
# Applying the patch: overlays/dev/gateway_patch.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: sonarqube
spec:
servers:
- port:
number: 443
name: http
protocol: HTTP
hosts:
- sonarqube-dev.acme.com # <- override
# |
# | over the base
# v

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube.acme.com
port:
name: http
number: 443
protocol: HTTP
# |
# | results with
# v

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
labels:
app: sonarqube
owner: piotr # <- label added by base kustomize.yaml fn
name: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- sonarqube-dev.acme.com # <- patch override
port:
name: http
number: 443
protocol: HTTP
</source>

Check yourselves
<source lang=bash>
# __unchanged manifest_ _base kustomization_ ___patch overlay____________
vimdiff <(cat base/gateway.yaml) <(kustomize build base) <(kustomize build overlays/dev)
</source>
:[[File:ClipCapIt-200910-010734.PNG]]

= Cheatsheet =
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md Helm charts - last mile]

= Patch [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/patchMultipleObjects.md multiple objects] =
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patches:
- path: patch.json
target:
kind: PersistentVolume
version: v1
group: ""
name: volume-(data|master)-\d # regex match
labelSelector: |
app.kubernetes.io/component=storage,
app.kubernetes.io/name=elasticsearch
</syntaxhighlight>

Component example
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
- target:
kind: HelmRelease
version: v2beta1
group: helm.toolkit.fluxcd.io
name: external-dns-.+$ # [1]
patch: |-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: ALL # [2]
namespace: flux-system
spec:
values:
tolerations:
- key: "components.gke.io/gke-managed-components"
operator: Exists
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: "predictx/workload"
operator: In
values:
- "infra"

# [1] Regex match
# [2] The name is replaced by a name from the matched object. Any value ie 'ALL' is required.
</syntaxhighlight>

= Delete an object from the base =
''Strategic Merge Patch'' provides some patch options like replace, merge, and delete. The simple 'patch' can only patch (manipulate) it.
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base

patchesStrategicMerge:
- |-
apiVersion: v1
kind: Namespace
metadata:
name: unwanted-namespace
$patch: delete
</syntaxhighlight>

= secretGenerator =
Secrets can be generated from environment variables. Within a template file there is a list of variables, where the variable will become a key and it's value the value.

Environment variable secret template
<source lang=bash>
GIT_USERNAME
GIT_PASSWORD
GIT_CREDENTIALS
</source>

Kustomization
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: argocd-git-secret
envs:
- git.env
options:
disableNameSuffixHash: true
</source>

= Patch - add an item to a list =
* https://stackoverflow.com/questions/71622419/adding-items-to-a-list-with-kubectl-kustomize
* [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch Strategic merge patch docs]

In the standard JSON merge patch, JSON objects are always merged but lists are always replaced. Often that isn't what we want. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. Read more at [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch strategic merge patch docs].

<source lang=yaml>
patchesJson6902:
- patch: |-
- op: add
path: /spec/valuesFrom/-
value: # below map will be added as an item to the list, pay attention to `-` sign at the end of path
kind: ConfigMap
name: values-1-yaml
target:
group: helm.toolkit.fluxcd.io
kind: HelmRelease
name: kube-prometheus-stack
version: v2beta1
</source>

= [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacements] =
* https://stackoverflow.com/questions/71358674/kustomize-how-to-reference-a-value-from-a-configmap-in-another-resource-overlay
Use of vars is getting deprecated please use replacements.

= Known issues =
* [https://github.com/kubernetes-sigs/kustomize/issues/2034 commonLabels altering podSelector.matchLabels] and [https://github.com/kubernetes-sigs/kustomize/issues/157 Allow excluding some label selectors from commonLabels]
In some settings it makes sense for <code>commonLabels</code> to be included in selectors, and in some settings it doers not make sense to include them in selectors. Kustomize includes by default, and there is no way to opt out. As workaround, you can convert <code>matchLabels</code> to <code>matchExpressions</code> and Kustomize won't touch them. API docs

<source lang=yaml>
- podSelector:
matchLabels:
app: mongodb-backup
</source>

is equivalent with
<source lang=yaml>
- podSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-backup
</source>
and Kustomize will keep its hands off.

= Resources =
* [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacement transform is deprecating Vars]
* [https://github.com/kubernetes-sigs/kustomize Kustomize sig]
* [https://kubectl.docs.kubernetes.io/guides/config_management/components/ v3.7.0+ Components]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#kustomization Glossary]
* [https://kubectl.docs.kubernetes.io/references/kustomize/ Kustomization File Fields]
* [https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html Kustomize - examples] kubectl.docs.kubernetes.io
* [https://kubectl.docs.kubernetes.io/pages/app_composition_and_deployment/structure_directories.html Kustomize structure_directories]
* [https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html reference] Good!
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/inlinePatch.md inlinePatch]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md kustomization of a helm chart]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configureBuiltinPlugin.md#using-the-commonlabels-and-commonannotations-fields Customize Kustomize] annotation and label buildin transformer

Ubuntu Setup

2024-07-15T10:40:22Z

Pio2pio: /* Useful setups */

If you are using Ubuntu for various Linux projects you will find that as it comes with pre installed with many packages. On the other hand installing just minimal version seems to be too extreme. Therefore I started maitaining a list of unnecessary packages and one liner to that removes them all. Please feel free to modify for your needs.

= Default partitioning =
On virtual systems schema below will be applied, eg on laptops:
:[[File:ClipCapIt-200620-131502.PNG]]

<source lang="bash">
#Eg. for 4G memory and 50G storage system

/dev/mapper/ubuntu--vg-root mount_point: /
/dev/mapper/ubuntu--vg-swapt_1
/dev/sda
/dev/sda1 (50G)

LVM VG ubuntu-vg, LV root as ext4
LVM VG ubuntu-vg, LV swapt_1 as swap

#Boot device:
/dev/mapper/ubuntu--vg-root
</source>
As a good handy practice you may create 100G virtual disk that you thin provision. Then create 2 PVs for root and swap partitions. Don't utilize all space at once but extend partitions when needed. This method eliminates adding new disks to VMs saving time and efforts.

Example LVM setup, here using 30G Physical Volume(99.9% used), 1 Volume Group and 2 Logical Volumes (root and swap).
<source lang="java">
$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 ubuntu-vg lvm2 a-- <29.93g 36.00m
$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
ubuntu-vg 1 2 0 wz--n- <29.93g 36.00m
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root ubuntu-vg -wi-ao---- 28.94g
swap_1 ubuntu-vg -wi-ao---- 976.00m
piotr@u18:~$
</source>

<source lang="java">
$ lsblk /dev/sda --fs
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
└─sda1 LVM2_member rP18Kb-Q12j-wjVf-C1iV-uy42-BUJD-aWFuO7
├─ubuntu--vg-root ext4 fad04a3b-5fa3-4a03-bbd6-24a93cda1eb3 /
└─ubuntu--vg-swap_1 swap 47cd084b-89b0-4cd5-bdb8-367238842ba1 [SWAP]
</source>

= List of unnecessary packages =
<source lang="bash">
sudo apt-get remove libreoffice-* #Remove LibreOffice
sudo apt-get remove unity-lens-* #This package contains photos scopes which allow Unity to search for local and online photos.
sudo apt-get remove shotwell* #Photo organizer
sudo apt-get remove simple-scan #Scanner software
sudo apt-get remove empathy* #Internet messaging ~13M
sudo apt-get remove thunderbird* #Email client ~61M
sudo apt-get remove unity-scope-gdrive #Google Drive scope for Unity ~116KB
sudo apt-get remove cheese* #Cheese Webcam Booth - webcam software
sudo apt-get remove brasero* #Brasero Disc Burner ~6.5MB
sudo apt-get remove gnome-bluetooth Package to manipulate bloototh devices using Gnome desktop ~2MB
sudo apt-get remove gnome-orca Orca Screen Reader -Provide access to graphical desktop environments via synthesised speech and/or refreshable braille
sudo apt-get remove unity-webapps-common #Amazon Unity WebApp integration scripts ~133KB
sudo apt-get remove ibus-pinyin #IBus Bopomofo Preferences - ibus-pinyin is a IBus based IM engine for Chinese ~1.4MB
sudo apt-get remove apt-get remove printer-driver-foo2zjs* #Reactivate HP LaserJet 1018/1020 after reloading paper ~3.2MB
</source>

= Remove unnecessary packages - one liner =
;Ubuntu 12, 14, 16
<source lang="bash">
$ sudo apt-get remove libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* unity-scope-gdrive cheese*\
brasero* gnome-bluetooth gnome-orca unity-webapps-common ibus-pinyin printer-driver-foo2zjs*
</source>

;Ubuntu 18. It's recommended to choose ''Minimal Install'', so most of packages below won't get installed.
<source lang="bash">
$ sudo apt-get purge libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* cheese* \
brasero* gnome-bluetooth gnome-orca ibus-pinyin printer-driver-foo2zjs* xul-ext-ubufox speech-dispatcher* \
rhythmbox* printer-driver-* mythes-en-us mobile-broadband-provider-inf* \
evolution-data-server* espeak-ng-data:amd64 bluez* ubuntu-web-launchers \
transmission-*
</source>
<source lang="bash">
sudo apt-get purge xul-ext-ubufox # Canonical FF customizations for u14,16,18,20
sudo apt-get remove gnome-mahjongg gnome-mines gnome-sudoku # games, works for u14,16,18,20
sudo apt-get remove gnome-video-effects gstreamer1.0-*
</source>

; XTREME
UnInstallant Ubuntu software notifier
<source lang="bash">
sudo apt-get remove update-notifier
</source>

= Uninstall locales - unused languages etc =
<source lang="bash">
sudo apt-get install localepurge
</source>

= Set apt-get to not install recommended and suggested packages =
<source lang="bash">
sudo bash -c 'cat > /etc/apt/apt.conf.d/01no-recommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF'
</source>

To see if apt reads this, enter this in command line (as root or regular user):
<source lang="bash">
apt-config dump | grep -e Recommends -e Suggests
</source>

= Install necessary packages =

Adobe Flash Player
sudo apt-get install flashplugin-installer

Java JRE
This will install the default verison Java for you distro plus Icedtea plugin for using Firefox with Java
sudo apt-get install default-jre icedtea-plugin

Unity Settings
sudo apt-get install unity-control-center

Opera

Add Opera repository <code>'''deb <nowiki>http://deb.opera.com/opera/</nowiki> stable non-free'''</code> to the apt-get source list in <code>/etc/apt/sources.list</code>. Then import a public PGP repository key.
<source lang="bash">
echo "deb http://deb.opera.com/opera/ stable non-free" | sudo tee -a /etc/apt/sources.list
wget -qO - http://deb.opera.com/archive.key | sudo apt-key add -
sudo apt-get update && sudo apt-get install opera
</source>
Silverlight

Pipelight has been released and we can use it for silverlight as a best alternative moonlight.
<source lang="bash">
sudo apt-add-repository ppa:ehoover/compholio
sudo apt-add-repository ppa:mqchael/pipelight
sudo apt-get update
sudo apt-get install pipelight
</source>

= GUI tools =
* [https://github.com/hluk/CopyQ/releases copyQ] clipboard manager
* VisualVM

= Customise Ubuntu =
==Fix Ubuntu Unity Dash Search for Applications and Files==
sudo apt-get install unity-lens-files unity-lens-applications #log out and log back in required

==Fix Ubuntu <17.10 missing Control Center==
sudo apt-get install unity-control-center --no-install-recommends

==Fix Ubuntu >18.04 missing System Settings==
sudo apt install gnome-control-center

==Remove background wallpaper ==
Tested on Ubuntu 14,16,18
<source lang="bash">
gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background draw-background false #disable
gsettings set org.gnome.desktop.background primary-color "#000000" #set to black
gsettings set org.gnome.desktop.background secondary-color "#000000" #set to black
gsettings set org.gnome.desktop.background color-shading-type "solid" #set solid colour
gsettings set org.gnome.desktop.background picture-uri file:///dev/null #remove wallpaper, not perfect but nothing worked in U15.10
gsettings set com.canonical.unity-greeter draw-user-backgrounds false #disable not worked

# Reset background picture to origin, U15.10
gsettings set org.gnome.desktop.background picture-uri file:///usr/share/backgrounds/warty-final-ubuntu.png

# Sets Unity greeter background, <17.04
gsettings set com.canonical.unity-greeter background /usr/share/backgrounds/warty-final-ubuntu.png
</source>

==Disable screen lock out==
<code>dconf</code> is legacy tool to configure <tt>gnome</tt> nowadays more modern way is to use <code>gsettings</code>.
<source lang=bash>
dconf write /org/gnome/desktop/screensaver/idle-activation-enabled false #gnome
dconf write /org/gnome/desktop/screensaver/lock-enabled false

# Unity - Ubuntu 14.04, 16.04
gsettings set org.gnome.desktop.session idle-delay 0 #disable the screen blackout:(0 to disable)
gsettings set org.gnome.desktop.screensaver lock-enabled false #disable the screen lock

# VirtualBox > Ubuntu 18.04 Disabling Xserver screen timeouts
xset s off # Xserver s parameter sets screensaver to off
xset s noblank # prevent the display from blanking
xset -dpms # prevent the monitor's DPMS energy saver from kicking in

# Gnome - Ubuntu 18.04 LTS, Settings > Power > Blank screen > set to: Never
gsettings get org.gnome.desktop.lockdown disable-lock-screen # verify status
gsettings set org.gnome.desktop.lockdown disable-lock-screen true # set disabled
gsettings get org.gnome.desktop.screensaver lock-enabled # verify status
gsettings set org.gnome.desktop.screensaver lock-enabled false # set disabled
dconf write /org/gnome/desktop/screensaver/lock-enabled false # set disbaled using dconf
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false # some say it's last resort :)

# Power management
gsettings set org.gnome.settings-daemon.plugins.power active true #set gnome to be the default power management run
gsettings set org.gnome.settings-daemon.plugins.power active false #turn off power management

# last resort as it was a bud in Ubuntu 11.10 with DPMS
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false
gsettings set org.gnome.desktop.session idle-delay 2400
</source>
Verify by navigating in <tt>dconf-editor</tt> to <tt>/org/gnome/desktop/screensaver/</tt>

==Change number of workspaces==
To get the current values:
<source lang=bash>
dconf read /org/compiz/profiles/unity/plugins/core/hsize
dconf read /org/compiz/profiles/unity/plugins/core/vsize
</source>

To set new values:
<source lang=bash>
dconf write /org/compiz/profiles/unity/plugins/core/hsize 2
# or
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ hsize 4
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ vsize 4
</source>

== Clenup motd messages ==
Ubuntu at login displays a number standard messages taking terminal space causing potential loosing context of previous operations.
<source lang="bash">
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-134-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

1 package can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Aug 31 12:11:28 2018 from 10.0.2.2
</source>

This is managed by files in <code>/etc/update-motd.d/</code>, so deleting them will remove clutter on a screen
<source lang="bash">
ls /etc/update-motd.d/
00-header 51-cloudguest 91-release-upgrade 98-fsck-at-reboot
10-help-text 90-updates-available 97-overlayroot 98-reboot-required

# Ubuntu Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1022-azure x86_64)
# Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)
sudo rm /etc/update-motd.d/{10-help-text,50-landscape-sysinfo,50-motd-news,51-cloudguest,80-livepatch,95-hwe-eol}
</source>

This cuts down to this message, Ubuntu 18.04 in AWS
<source lang="bash">
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)

0 packages can be updated.
0 updates are security updates.

Last login: Thu Jan 31 17:09:38 2019 from 10.10.11.11
</source>

= Useful setups =
== Image converter ==
nautilus-image-converter is a nautilus extension to mass resize or rotate images. It adds two context menu items in nautlius so you can right-click and choose "Resize Image" or "Rotate Image").
<source lang=bash>
# tested on Ubuntu 24.04 with Gnome
sudo apt-get install nautilus-image-converter

# Restart to see the new context menu
nautilus -q
</source>

== Call screen saver from a terminal to blank all screens ==
<source lang=bash>
# tested on Ubuntu 18.04 with Gnome
sudo apt-get install gnome-screensaver
gnome-screensaver-command -a #controls GNOME screensaver, -a activate (blank the screen)
</source>

== Create application launcher ==
;Ubuntu 18.04
<source lang=bash>
# Install the GNOME-panel toolset
sudo apt-get install --no-install-recommends gnome-panel

# Every user launcher
sudo gnome-desktop-item-edit /usr/share/applications/VisualVM.desktop --create-new

# Local user only, the filename by default is a Name-of-appication.desktop
gnome-desktop-item-edit ~/.local/share/applications --create-new
</source>

:[[File:ClipCapIt-190807-080016.PNG]]

;Ubuntu 19.10, 20.04
In above releases <code>gnome-desktop-item-edit</code> has been removed from the <code>gnome-panel</code> package, as an alternative <code>.desktop</code> files can be created manually.
<source lang=bash>
vi /usr/share/applications/APPNAME.desktop
[Desktop Entry]
Name=<NAME OF THE APPLICATION>
Comment=<A SHORT DESCRIPTION>
Exec=<COMMAND-OR-FULL-PATH-TO-LAUNCH-THE-APPLICATION>
Type=Application
Terminal=false
Icon=<ICON NAME OR PATH TO ICON>
NoDisplay=false
Keywords=<eg. sql>
</source>

It's optional but you may need to right click and set 'allow launching' with addition to set executable permissions. Usual locations of <code>.desktop</code> files are:
* <code>/usr/share/applications/</code>
* <code>/var/lib/snapd/desktop/applications/</code> for snap applications

== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet gnome-shell-system-monitor-applet] - cpu, memory indicators ==
System information such as memory usage, cpu usage, network rates and more can be displayed in the notification area in GNOME Shell.

System-monitor extensions:
[https://extensions.gnome.org/extension/120/system-monitor/ system-monitor] by paradoxxxzero on [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet github] supports Gnome-shell up to v40. It seems like abandoned project.
[https://extensions.gnome.org/extension/3010/system-monitor-next/ system-monitor-next] by mgalgs on [https://github.com/mgalgs/gnome-shell-system-monitor-applet github] supports Gnome-shell v40+, it's a fork of the above.

All extensions:
* https://extensions.gnome.org

{{Note|The current version of the browser Firefox is packaged as a snap version. One of the issues with this is that it cannot work with the Gnome Extensions website.}}

Install on Ubuntu 24.04 (June 2024)
<source lang=bash>
# Ubuntu 20/22/24
GNOME Shell 46.0 # version as of Ubuntu 24.04
sudo apt install gnome-shell-extensions # Ubuntu 20.04 already has this package, 24.04 needs installing it
sudo apt install gnome-shell-extension-manager # Ubuntu 22|24.04 (as Firefox is installed as snap) on 24.04 it's v0.5.0

# Open `Extensions` app, turn "Use Extensions". # Already turned on on Ubuntu 24.04
# Open Browse tab > search for 'system-monitor-next' # cpu/mem/net indicators will appear in the system tray

# Additional steps for Ubuntu < 24.04
sudo apt install gnome-tweaks # GUI to manage gnome-extensions
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
sudo apt install gnome-shell-extension-system-monitor # after requires log out

# Download the extension from
## https://extensions.gnome.org/extension/120/system-monitor/

# Never worked out how to use this direct download and install via 'gnome-extensions install <extension_name>'
## wget https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/archive/v38.zip
## gnome-extensions install <system-monitor@paradoxxx.zero.gmail.com>

# Enable extension using cli
gnome-extensions enable system-monitor-next@paradoxxx.zero.gmail.com
gnome-extensions list --user
clipboard-indicator@tudmotu.com
system-monitor-next@paradoxxx.zero.gmail.com
</source>
:[[File:ClipCapIt-210105-084527.PNG]]

=== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/issues/737#issuecomment-1230654455 Ubuntu 22.04 workaround for the OUTDATED extension] ===
{{Note|Workaround still needed in August 2022}}
<source lang=bash>
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
git clone https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet.git
cd gnome-shell-system-monitor-applet # commit b359d88 verified
vi system-monitor@paradoxxx.zero.gmail.com/metadata.json
# | change "version": -1 to "version": 42
make install
# log out and back in (required)
</source>

= Snapd - Chromium =
Recently in U19+ Chromium get installed via snapd package. This is classic installation that has limited access to only a certain directories. It happen that when working with AWS we need get access to <code>~/.ssh</code> folder to get ec2 machine password. This folder is denied, but we can bind mount <code>~/.ssh</code> folder into the snap container directory:
<source lang=bash>
$ snap list chromium
Name Version Rev Tracking Publisher Notes
chromium 86.0.4240.111 1373 latest/stable canonical✓ -

# cd to chromim $HOME dir
mkdir ~/snap/chromium/current/.ssh
sudo mount --bind ~/.ssh/ ~/snap/chromium/current/.ssh
</source>

= Screen shooting =
In Ubuntu 20.04 Shutter is not a part of default repositories. It can be added via PPA:
<source lang=bash>
sudo add-apt-repository -y ppa:linuxuprising/shutter
sudo apt-get install shutter
</source>

= Audio - [https://rastating.github.io/setting-default-audio-device-in-ubuntu-18-04/ set defaults] =
For preserving settings using GUI you can install [https://freedesktop.org/software/pulseaudio/pavucontrol/ PulseAudio Volume Control] <code>pavucontrol</code>.
<source lang=bash>
# install
sudo apt install pavucontrol # Ubuntu 20.04
# run
pavucontrol
</source>

Set default output/input device. In Ubuntu PulseAudio is used to control audio devices. It contains following configuration files
<source lang=bash>
/etc/pulse/default.pa # system wide
~/.config/pulse # user configuration
</source>

Set defaults
<source lang=bash>
# List devices: modules, sinks, sources, sink-inputs, source-outputs, clients, samples, cards
# sinks - outputs, sink-inputs, sources - all input/output including RUNNING and SUSPENDED devices
$ pactl list short sources | column -t
5 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_5__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
6 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_4__sink.monitor module-alsa-card.c s16le 2ch 48000Hz RUNNING
7 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_3__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
8 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
9 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__source module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
10 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_6__source module-alsa-card.c s16le 4ch 48000Hz SUSPENDED
15 alsa_output.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.analog-stereo.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz SUSPENDED
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz SUSPENDED
20 alsa_input.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.iec958-stereo module-alsa-card.c s16le 2ch 48000Hz RUNNING

# Set defaut output device. Tab autocompletion should work (U20.04)
pactl set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output
# Set defaut input device
pactl set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Test, play some audio then run. IDLE - means in use
pactl list short sources | column -t | grep -e RUNNING -e IDLE
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz IDLE
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz RUNNING
</source>

Make it permanent by setting default device in PulseAudio system configuration file
<source lang=bash>
# Output device
OUTPUT_DEVICE=alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-sink$ output/\1 ${OUTPUT_DEVICE}/g" /etc/pulse/default.pa # remove '-i' to test before apply
# Input device
INPUT_DEVICE=alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-source$ input/\1 ${INPUT_DEVICE}/g" /etc/pulse/default.pa

vi /etc/pulse/default.pa # make sure lines below are in place
### Make some devices default
set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Delete local user profile and restart system, after boot new defaults should be set
rm -r ~/.config/pulse

# After reboot, defaults should be set
cat ~/.config/pulse/*default*
alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
</source>

;Troubleshooting
PulseAudio cli
<source lang=bash>
pacmd
>>> help # lists all available commands

pulseaudio --check # Check if any pulseaudio instance is running. It normally prints no output, just exit code. 0 means running
pulseaudio --kill # kill, then --start
pulseaudio -D # start pulseaudio as a daemon
# | using /etc/pulse/daemon.conf

# Pulseaudio is a user service
systemctl --user restart pulseaudio.service
systemctl --user restart pulseaudio.socket
</source>

I have a port replicator Dell D-6000, that gets randomly disconnected causing switching audio to new connected device - means itself. As workaround commenting out lines below stops this behaviour.
<source lang=bash>
vi /etc/pulse/default.pa
### Use hot-plugged devices like Bluetooth or USB automatically (LP: #1702794)
# .ifexists module-switch-on-connect.so
# load-module module-switch-on-connect
# .endif
</source>

= Input devices =
Motivation is to enable horizontal scrolling in Ubuntu 20.04 using Perixx Gamig Mouse Mx2000
<source lang=bash>
xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Holtek USB Gaming Mouse id=11 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Mouse id=14 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Touchpad id=15 [slave pointer (2)]
⎜ ↳ TPPS/2 Elan TrackPoint id=19 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Sleep Button id=8 [slave keyboard (3)]
↳ CHICONY HP Basic USB Keyboard id=9 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=10 [slave keyboard (3)]
↳ Integrated Camera: Integrated C id=12 [slave keyboard (3)]
↳ Integrated Camera: Integrated I id=13 [slave keyboard (3)]
↳ sof-hda-dsp Headset Jack id=16 [slave keyboard (3)]
↳ Intel HID events id=17 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=18 [slave keyboard (3)]
↳ ThinkPad Extra Buttons id=20 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=21 [slave keyboard (3)]

# test mouse aka Virtual core pointer
xinput test 11
motion a[0]=2023 # <- cursor moving
motion a[0]=2024 a[1]=1411
motion a[3]=19545 # <- scroll down
button press 5
button release 5

# test 'virtual core keyboard' aka additional programmable buttons
## '10' - this virtual keyboard for all buttons except the scrolling wheel
xinput test 10
key press 37
key press 38

## '21' - this is scrolling wheel buttons left/right, not scrolling itself
xinput test 21
key press 248
key release 248
</source>

# List of properties of a device. We want to see 'horizontal scrolling wheel buttons'
<source lang=bash>
$ xinput list-props 21
Device 'Holtek USB Gaming Mouse':
Device Enabled (169): 1
Coordinate Transformation Matrix (171): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
libinput Send Events Modes Available (291): 1, 0
libinput Send Events Mode Enabled (292): 0, 0
libinput Send Events Mode Enabled Default (293): 0, 0
Device Node (294): "/dev/input/event10"
Device Product ID (295): 1241, 41063
</source>

=References=

[[Category:linux]]

Kubernetes/Tools

2024-07-02T22:17:12Z

Pio2pio: /* Install kubectl */

= kubectl =
== Install kubectl ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.30.2
v1.29.6
v1.28.11
v1.27.15
v1.26.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

kubectl plugin called [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke gke-gcloud-auth-plugin]
* [https://cloud.google.com/sdk/docs/install#deb Install Google Cloud SDK]
<source lang=bash>
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update
sudo apt-get install google-cloud-cli # required to authenticate with GCP
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
gcloud init
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2024-07-02T22:16:44Z

Pio2pio: /* Install kubectl */

= kubectl =
== Install kubectl ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.30.2
v1.29.6
v1.28.11
v1.27.15
v1.26.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

kubectl plugin called [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke gke-gcloud-auth-plugin]
* [https://cloud.google.com/sdk/docs/install#deb Install Google Cloud SDK]
<source lang=bash>
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update
sudo apt-get install google-cloud-cli # required to authenticate with GCP
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2024-07-02T21:53:07Z

Pio2pio: /* Install */

= kubectl =
== Install kubectl ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.30.2
v1.29.6
v1.28.11
v1.27.15
v1.26.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

kubectl plugin called [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke gke-gcloud-auth-plugin]
* [https://cloud.google.com/sdk/docs/install#deb Install Google Cloud SDK]
<source lang=bash>
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update
sudo apt-get install google-cloud-cli # optional
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2024-07-02T15:26:12Z

Pio2pio: /* Install */

= kubectl =
== Install ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '[.[] | select(.prerelease == false) | .tag_name] | map(sub("^v";"")) | map(split(".")) | group_by(.[0:2]) | map(max_by(.[2]|tonumber)) | map(join(".")) | map("v" + .) | sort | reverse | .[]'
v1.30.2
v1.29.6
v1.28.11
v1.27.15
v1.26.15

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2024-07-02T15:22:05Z

Pio2pio: /* Install */

= kubectl =
== Install ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
# List releases
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
v1.26.15
v1.27.15
v1.28.11
v1.29.5
v1.29.6
v1.30.2
...

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

Kubernetes/Tools

2024-07-02T15:21:46Z

Pio2pio: /* Install */

= kubectl =
== Install ==
List of kubectl [https://kubernetes.io/releases/ releases].
<source lang=bash>
curl -s https://api.github.com/repos/kubernetes/kubernetes/releases | jq -r '.[].tag_name' | sort -V
v1.26.15
v1.27.15
v1.28.11
v1.29.5
v1.29.6
v1.30.2
...

# Latest
ARCH=amd64 # amd64|arm
VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt); echo $VERSION
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl

# Specific version
# Find specific Kubernetes release, then download kubectl
VERSION=v1.26.14; ARCH=amd64 # amd64|arm
curl -LO https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/$ARCH/kubectl
sudo install ./kubectl /usr/local/bin/kubectl

# Note: sudo install := chmod +x ./kubectl; sudo mv

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short
Client Version: v1.26.14
Kustomize Version: v4.5.7
Server Version: v1.24.10
</source>

Google way
<source lang=bash>
# Install kubectl if you don't already have a suitable version
# https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl
kubectl version --client || gcloud components install kubectl
kubectl get clusterrolebinding $(gcloud config get-value core/account)-cluster-admin ||
kubectl create clusterrolebinding $(gcloud config get-value core/account)-cluster-admin \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value core/account)"
</source>

== Autocompletion and kubeconfig ==
<source lang=bash>
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context --current --namespace=dev
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
namespace: web # default namespace
name: dev-frontend
...
</source>

== Add <code>proxy-url</code> using <code>yq</code> to kubeconfig ==
Minimum yq version required is v2.x, tested with yq 2.13.0. The example below does the file inline `-i` update.
<source lang=bash>
yq -i -y --indentless '.clusters[0].cluster += {"proxy-url": "http://proxy.acme.com:8080"}' ~/.kube/$ENVIRONMENT
</source>

== Get resources and cheatsheet ==
<source lang=bash>
# Get a list of nodes
kubectl get nodes -o jsonpath="{.items[*].metadata.name}"
ip-10-10-10-10.eu-west-1.compute.internal ip-10-10-10-20.eu-west-1.compute.internal

kubectl get nodes -oname
node/ip-10-10-10-10.eu-west-1.compute.internal
node/ip-10-10-10-20.eu-west-1.compute.internal
...

# Pods sorted by node name
kubectl get pods --sort-by=.spec.nodeName -owide -A

# Watch a namespace in a convinient resources order | sts=statefulset, rs=replicaset, ep=endpoint, cm=configmap
watch -d kubectl -n dev get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels
# note es - externalsecrets
watch -d 'kubectl get pv -owide --show-labels | grep -e <eg.NAMESPACE>'
watch -d helm list -A

# Test your context by creating configMap
kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
kubectl delete configmap my-config

# Watch multiple namespaces
eval 'kubectl --context='{context1,context2}' --namespace='{ns1,ns2}' get pod;'
eval kubectl\ --context={context1,context2}\ --namespace={ns1,ns2}\ get\ pod\;
watch -d eval 'kubectl -n '{default,ingress-nginx}' get sts,deploy,rc,rs,pods,svc,ep,ing,pvc,cm,sa,secret,es,cronjob,job -owide --show-labels;'

# Auth, can-i
kubectl auth can-i delete pods
yes
</source>

== Get yaml from existing object ==
<source lang=bash>
kubectl create namespace kiali --dry-run=client -o yaml > ns.yaml
kubectl create namespace kiali --dry-run=client -o yaml | kubectl apply -f -

# Saves version revision in metadata.annotations.kubectl.kubernetes.io/last-applied-configuration={..manifest_json..}
kubectl create ns foo --save-config

# Get a yaml without status information, almost clean manifest. Deprecated '--export' before <v17.x.
kubectl -n web get pod <podName> -oyaml --export
</source>

Generate pod manifest, the most clean way I know
<syntaxhighlightjs lang=bash>
# kubectl -n foo run --image=ubuntu:20.04 ubuntu-1 --dry-run=client -oyaml -- bash -c sleep
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null # <- can be deleted
labels:
run: ubuntu-1
name: ubuntu-1
namespace: foo
spec:
containers:
- args:
- bash
- -c
- sleep
image: ubuntu:20.04
name: ubuntu-1
resources: {} # <- can be deleted
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {} # <- can be deleted
</syntaxhighlightjs>

== <code>kubectl cp</code> ==
Each container must be prefixed with a namespace, the copy-to-file(<filename>) must be in place. The recursive copy might be tricky.
<source lang=bash>
kubectl cp [[namespace/]pod:]file/path ./<filename> -c <container_name>
kubectl cp vegeta/vegeta-5847d879d8-p9kqw:plot.html ./plot.html -c vegeta
</source>

== One liners ==
=== Single purpose pods ===
Note: <code>--generator=deployment/apps.v1</code> is DEPRECATED and will be removed, use <code>--generator=run-pod/v1 </code> or <code>kubectl create</code> instead.
<source lang=bash>
# Exec to deployment, no need to specify unique pod name
kubectl exec -it deploy/sleep -- curl httpbin:8000/headers

NS=mynamespace; LABEL='app.kubernetes.io/name=myvalue'
kubectl exec -n $NS -it $(kubectl get pod -l "$LABEL" -n $NS -o jsonpath='{.items[0].metadata.name}') -- bash

# Echo server
kubectl run --image=k8s.gcr.io/echoserver:1.4 hello-1 --port=8080

# Single purpose pods
kubectl run --image=bitnami/kubectl:1.21.8 kubectl-1 --rm -it -- get pods
kubectl run --image=appropriate/curl curl-1 --rm -it -- sh
kubectl run --image=ubuntu:18.04 ubuntu-1 --rm -it -- bash
kubectl create --image=ubuntu:20.04 ubuntu-2 --rm -it -- bash
kubectl run --image=busybox:1.31.0 busybox-1 --rm -it -- sh # exec and delete when completed
kubectl run --image=busybox:1.31.0 busybox-2 -- sleep 7200 # sleep, so you can exec
kubectl run --image=alpine alpine-1 --rm -it -- ping -c 1 8.8.8.8
docker run --rm -it --name alpine-1 alpine ping -c 1 8.8.8.8

# Network-multitool | https://github.com/wbitt/Network-MultiTool | Runs as a webserver, so won't complete.
kubectl run --image=wbitt/network-multitool multitool-1
kubectl create --image=wbitt/network-multitool deployment multitool
kubectl exec -it multitool-1 -- /bin/bash
kubectl exec -it deployment/multitool -- /bin/bash
docker run --rm -it --name network-multitool wbitt/network-multitool bash

# Curl
kubectl run test --image=tutum/curl -- sleep 10000

# Deprecation syntax
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=run-pod/v1 hello-1 --port=8080 # VALID!
kubectl run --image=k8s.gcr.io/echoserver:1.4 --generator=deployment/apps.v1 hello-1 --port=8080 # <- deprecated

# Errors
# | error: --rm should only be used for attached containers
# | Error: unknown flag: --image # when kubectl create --image
</source>

Additional software
<source lang=bash>
# Process and network comamnds
export DEBIAN_FRONTEND=noninteractive # Ubuntu 20.04
DEBIAN_FRONTEND=noninteractive apt install -yq dnsutils iproute2 iputils-ping iputils-tracepath net-tools netcat procps
# | dnsutils - nslookup, dig
# | iproute2 - ip addr, ss
# | iputils-ping - ping
# | iputils-tracepath - tracepath
# | net-tools - ifconfig
# | netcat - nc
# | procps - ps, top

# Databases
apt install -yq redis-tools
apt install -yq postgresql-client

# AWS cli v1 - Debian
apt install python-pip
pip install awscli

# Network test without ping, nc or telnet
(timeout 1 bash -c '</dev/tcp/127.0.0.1/22 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null
</source>

;kubectl heredocs
<source lang=bash>
kubectl apply -f <(cat <<EOF
EOF
) --dry-run=server
</source>

;One lines move to yamls
<source lang=yaml>
# kubectl exec -it ubuntu-2 -- bash
kubectl apply -f <(cat <<EOF
apiVersion: v1
kind: Pod
metadata:
# namespace: default
name: ubuntu-1
# annotations:
# kubernetes.io/psp: eks.privileged
# labels:
# app: ubuntu
spec:
containers:
- command:
- "sleep"
- "7200"
# args:
# - "bash"
image: ubuntu:20.04
imagePullPolicy: IfNotPresent
name: ubuntu-1
# securityContext:
# privileged: true
# tty: true
# dnsPolicy: ClusterFirst
# enableServiceLinks: true
restartPolicy: Never
# serviceAccount : sa1
# serviceAccountName: sa1
# nodeSelector:
# node.kubernetes.io/lifecycle: spot
EOF
) --dry-run=server
</source>

=== Docker - for a single missing commands ===
If you ever miss some commands you can use docker container package with it:
<source lang=bash>
# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint
</source>

== [https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/ <code>kubectl diff</code>] ==
Shows the differences between the current '''live''' object and the new '''dry-run''' object.
<source lang=diff>
kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy 2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-13T16:38:43Z"
- generation: 2
+ generation: 3
labels:
app: webfront-deploy
name: webfront-deploy
@@ -14,7 +14,7 @@
uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
spec:
progressDeadlineSeconds: 600
- replicas: 2
+ replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -29,6 +29,7 @@
creationTimestamp: null
labels:
app: webfront-deploy
+ role: webfront
spec:
containers:
- image: nginx:1.7.8
exit status 1
</source>

== Kubectl-plugins - [https://krew.sigs.k8s.io/docs/ Krew] plugin manager ==
Install [https://github.com/kubernetes-sigs/krew krew] package manager for kubectl plugins, requires K8s v1.12+
<source lang=bash>
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

# update PATH
[ -d ${HOME}/.krew/bin ] && export PATH="${PATH}:${HOME}/.krew/bin"

# List plugins
kubectl krew search

# Install plugins
kubectl krew install sniff

# Upgrade plugins
kubectl krew upgrade
</source>

*[https://github.com/kubernetes-sigs/krew-index/blob/master/plugins.md Available kubectl plugins] Github
*[https://ahmet.im/blog/kubectl-plugins/ kubectl subcommands] write your own plugin

== Install kubectl plugins ==
<code>kubectl ctx</code> and <code>kubectl ns</code> - change context and set default namespace
<source lang=bash>
kubectl krew install ctx ns
</source>

=== <code>kubectl cssh</code> - SSH into Kubernetes nodes ===
<source lang=bash>
# Ssh to all nodes, example below for EKS v1.15.11
kubectl cssh -u ec2-user -i /git/secrets/ssh/dev.pem -a "InternalIP"
</source>

===<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>===
;<code>[https://github.com/FairwindsOps/pluto kubectl deprecations]</code>: shows all the deprecated objects in a Kubernetes cluster allowing the operator to verify them before upgrading the cluster. It uses the swagger.json version available in master branch of Kubernetes repository (https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec) as a reference.
<source lang=bash>
kubectl deprecations
StatefulSet found in statefulsets.apps/v1beta1
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: myapp namespace: mynamespace1
</source>

Prior upgrade report. Script specific to EKS.
<source lang=bash>
#!/bin/bash
[[ $# -eq 0 ]] && echo "no args, provide prefix for the file name" && exit 1
PREFIX=$1
TARGET_K8S_VER=v1.16.8
K8Sid=$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)
kubectl deprecations --k8s-version $TARGET_K8S_VER > $PREFIX-$(kubectl cluster-info | head -1 | cut -d'/' -f3 | cut -d'.' -f1)-$(date +"%Y%m%d-%H%M")-from-$(kubectl version --short | grep Server | cut -f3 -d' ')-to-${TARGET_K8S_VER}.yaml
</source>
<source lang=bash>
$ ./kube-deprecations.sh test
$ ls -l
-rw-rw-r-- 1 vagrant vagrant 29356 Jun 29 16:09 test-11111111112222222222333333333344-20200629-1609-from-v1.15.11-eks-af3caf-to-latest.yaml
-rw-rw-r-- 1 vagrant vagrant 852 Jun 30 22:41 test-11111111112222222222333333333344-20200630-2241-from-v1.15.11-eks-af3caf-to-v1.16.8.yaml
-rwxrwxr-x 1 vagrant vagrant 437 Jun 30 22:41 kube-deprecations.sh
</source>

===<code>kubectl df-pv</code>===
;<code>kubectl df-pv</code>: Show disk usage (like unix df) for persistent volumes
<source lang=bash>
kubectl df-pv
PVC NAMESPACE POD SIZE USED AVAILABLE PERCENTUSED IUSED IFREE PERCENTIUSED
rdbms-volume shared1 rdbms-d494fbf4-xrssk 2046640128 252817408 1777045504 12.35 688 130384 0.52
userdata-0 shared2 mft-0 21003583488 57692160 20929114112 0.27 749 1309971 0.06
</source>
===<code>kubectl sniff</code>===
Start a remote packet capture on pods using tcpdump.
<source lang=bash>
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
# Flags:
# -c, --container string container (optional)
# -x, --context string kubectl context to work on (optional)
# -f, --filter string tcpdump filter (optional)
# -h, --help help for sniff
# --image string the privileged container image (optional)
# -i, --interface string pod interface to packet capture (optional) (default "any")
# -l, --local-tcpdump-path string local static tcpdump binary path (optional)
# -n, --namespace string namespace (optional) (default "default")
# -o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
# -p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
# -r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
# -v, --verbose if specified, ksniff output will include debug information (optional)
</source>

The command above will open Wireshark, interesting article to follow:
* [https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster mutual TLS] istio
* [https://dzone.com/articles/verifying-service-mesh-tls-in-kubernetes-using-ksn Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark]

===<code>kubectl neat</code>===
Print sanitized Kubernetes manifest.
<source lang=yaml>
kubectl get csec dummy-secret -n clustersecret -oyaml | kubectl neat
apiVersion: clustersecret.io/v1
data:
tls.crt: ***
tls.key: ***
kind: ClusterSecret
matchNamespace:
- anothernamespace
metadata:
name: dummy-secret
namespace: clustersecret
</source>

== Getting help like manpages <code>kubectl explain</code> ==
<source>
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers # kubectl knows cluster version, so gives you correct schema details
$ kubectl explain pods.spec.tolerations --recursive # show only fields
(...)
FIELDS:
effect <string>
key <string>
operator <string>
tolerationSeconds <integer>
value <string>

</source>

*[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-strong-getting-started-strong- kubectl-commands] K8s interactive kubectl command reference

= Watch Containers logs =
== [https://github.com/stern/stern Stern] ==
{{note| https://github.com/wercker/stern repository has no activity [https://github.com/wercker/stern/issues/140 ISSUE-140], the new community maintain repo is <tt>[https://github.com/stern/stern stern/stern]</tt> }}

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.

It will re-use kubectl config file to connect to your clusters, so works oob.

;Install
<source lang=bash>
# Govendor - this module manager is required
export GOPATH=$HOME/go # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/stern # new link: https://github.com/stern/stern
cd $GOPATH/src/github.com/stern
git clone https://github.com/stern/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install # no output

# Stern latest, download binary, no need for govendor
REPO=stern/stern
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=stern_${LATEST}_linux_amd64
curl -L https://github.com/$REPO/releases/download/v${LATEST}/$FILE.tar.gz -o $TEMPDIR/$FILE.tar.gz
sudo tar xzvf $TEMPDIR/$FILE.tar.gz -C /usr/local/bin/ stern
</source>

;Usage
<source lang=bash>
# Regex filter (pod-query) to match 2 pods patterns 'proxy' and 'gateway'
stern -n dev --kubeconfig ~/.kube/dev-config $proxy\|gateway$ # escape to protect regex mod characters
stern -n dev --kubeconfig ~/.kube/dev-config '(proxy|gateway)' # single-quote to protect mod characters

# Template the output
stern --template '{{.Message}} ({{.NodeName}}/{{.Namespace}}/{{.PodName}}/{{.ContainerName}}){{"\n"}}' .
</source>

;Help
<source lang=bash>
$ stern
Tail multiple pods and containers from Kubernetes

Usage:
stern pod-query [flags]

Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Color output. Can be 'always', 'never', or 'auto' (default "auto")
--completion string Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
-c, --container string Container name when multiple containers in pod (default ".*")
--container-state string If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
-e, --exclude strings Regex of log lines to exclude
-E, --exclude-container string Exclude a Container name
-h, --help help for stern
-i, --include strings Regex of log lines to include
--init-containers Include or exclude init containers (default true)
--kubeconfig string Path to kubeconfig file to use
-n, --namespace string Kubernetes namespace to use. Default to namespace configured in Kubernetes context.
-o, --output string Specify predefined template. Currently support: [default, raw, json] (default "default")
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag
-t, --timestamps Print timestamps
-v, --version Print the version and exit

</source>

;Usage
<source lang=bash>
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc
</source>

== [https://github.com/johanhaleby/kubetail kubetail] ==
Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running <code>kubectl logs -f</code> but for multiple pods.

= [https://github.com/lensapp/lens Lens | Kubernetes IDE] =
Kubernetes client, this is not a dashboard that needs installing on a cluster. Similar to KUI but much more powerful.
<source lang=bash>
# Deb
curl curl https://api.k8slens.dev/binaries/Lens-5.4.1-latest.20220304.1.amd64.deb
sudo apt-get install ./Lens-5.4.1-latest.20220304.1.amd64.deb

# Snap
snap list
sudo snap install kontena-lens --classic # U16.04+, tested on U20.04

# Install from a .snap file
mkdir -p ~/Downloads/kontena-lens && cd $_
snap download kontena-lens
sudo snap ack kontena-lens_152.assert # add an assertion to the system assertion database
sudo snap install kontena-lens_152.snap --classic # --dangerous if you do not have the assert file

# download snap from https://k8slens.dev/
curl https://api.k8slens.dev/binaries/Lens-5.3.4-latest.20220120.1.amd64.snap
sudo snap install Lens-5.3.4-latest.20220120.1.amd64.snap --classic --dangerous

# Info
$ snap info kontena-lens_152.assert
name: kontena-lens
summary: Lens - The Kubernetes IDE
publisher: Mirantis Inc (jakolehm)
store-url: https://snapcraft.io/kontena-lens
contact: info@k8slens.dev
license: Proprietary
description: |
Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily
basis. Ensure your clusters are properly setup and configured. Enjoy increased visibility, real
time statistics, log streams and hands-on troubleshooting capabilities. With Lens, you can work
with your clusters more easily and fast, radically improving productivity and the speed of
business.
snap-id: Dek6y5mTEPxhySFKPB4Z0WVi5EPS9osS
channels:
latest/stable: 4.0.7 2021-01-20 (152) 107MB classic
latest/candidate: 4.0.7 2021-01-20 (152) 107MB classic
latest/beta: 4.0.7 2021-01-20 (152) 107MB classic
latest/edge: 4.1.0-rc.1 2021-02-11 (157) 108MB classic

$ snap info kontena-lens_152.snap
path: "kontena-lens_152.snap"
name: kontena-lens
summary: Lens
version: 4.0.7 classic
build-date: 24 days ago, at 16:31 GMT
license: unset
description: |
Lens - The Kubernetes IDE
commands:
- kontena-lens
</source>

= [https://github.com/lensapp/lens.git OpenLens] | Kubernetes IDE =
Download binary from https://github.com/MuhammedKalkan/OpenLens

Install on Ubuntu
<source lang=bash>
#!/bin/bash

SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi

REPO=MuhammedKalkan/OpenLens
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name | tr -d v); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=OpenLens-${LATEST}.amd64.deb
curl -L https://github.com/${REPO}/releases/download/v${LATEST}/$FILE -o $TEMPDIR/$FILE
$SUDO dpkg -i $TEMPDIR/$FILE
$SUDO apt-get install -y --fix-broken
</source>

Build your own - [https://gist.github.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9 gist]
<source lang=bash>
#!/bin/bash

install_deps_windows() {
echo "Installing Build Dependencies (Windows)..."
choco install -y make visualstudio2019buildtools visualstudio2019-workload-vctools
}

install_deps_darwin() {
echo "Installing Build Dependencies (Darwin)..."
xcode-select --install
if ! hash make 2>/dev/null; then
if ! hash brew 2>/dev/null; then
echo "Installing Homebrew..."
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
echo "Installing make via Homebrew..."
brew install make
fi
}

install_deps_posix() {
echo "Installing Build Dependencies (Posix)..."
sudo apt-get install -y make g++ curl
}

install_darwin() {
echo "Killing OpenLens (if open)..."
killall OpenLens
echo "Installing OpenLens (Darwin)..."
rm -Rf "$HOME/Applications/OpenLens.app"
arch="mac"
if [[ "$(uname -m)" == "arm64" ]]; then
arch="mac-arm64" # credit @teefax
fi
cp -Rfp "$tempdir/lens/dist/$arch/OpenLens.app" "$HOME/Applications/"
rm -Rf "$tempdir"

print_alias_message
}

install_posix() {
echo "Installing OpenLens (Posix)..."
cd "$tempdir"
sudo dpkg -i "$(ls -Art $tempdir/lens/dist/*.deb | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_windows() {
echo "Installing OpenLens (Windows)..."
"$(/bin/ls -Art $tempdir/lens/dist/OpenLens*.exe | tail -n 1)"
rm -Rf "$tempdir"

print_alias_message
}

install_nvm() {
if [ -z "$NVM_DIR" ]; then
echo "Installing NVM..."
NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/$NVM_VERSION/install.sh | bash
NVM_DIR="$HOME/.nvm"
fi
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
}

build_openlens() {
tempdir=$(mktemp -d)
cd "$tempdir"
if [ -z "$1" ]; then
echo "Checking GitHub API for latests tag..."
OPENLENS_VERSION=$(curl -s https://api.github.com/repos/lensapp/lens/releases/latest | sed -En 's/ "tag_name": "(.+)",/\1/p')
else
if [[ "$1" == v* ]]; then
OPENLENS_VERSION="$1"
else
OPENLENS_VERSION="v$1"
fi
echo "Using supplied tag $OPENLENS_VERSION"
fi
if [ -z $OPENLENS_VERSION ]; then
echo "Failed to get valid version tag. Aborting!"
exit 1
fi
curl -L https://github.com/lensapp/lens/archive/refs/tags/$OPENLENS_VERSION.tar.gz | tar xvz
mv lens-* lens
cd lens
NVM_CURRENT=$(nvm current)
nvm install 16
nvm use 16
npm install -g yarn
make build
nvm use "$NVM_CURRENT"
}

print_alias_message() {
if [ "$(type -t install_openlens)" != 'alias' ]; then
printf "It is recommended to add an alias to your shell profile to run this script again.\n"
printf "alias install_openlens=\"curl -o- https://gist.githubusercontent.com/jslay88/bf654c23eaaaed443bb8e8b41d02b2a9/raw/install_openlens.sh | bash\"\n\n"
fi
}

if [[ "$(uname)" == "Linux" ]]; then
install_deps_posix
install_nvm
build_openlens "$1"
install_posix
elif [[ "$(uname)" == "Darwin" ]]; then
install_deps_darwin
install_nvm
build_openlens "$1"
install_darwin
else
install_deps_windows
install_nvm
build_openlens "$1"
install_windows
fi

echo "Done!"
</source>

= [https://kui.tools/ kui terminal] =
kui is a terminal with visualizations, provided by IBM

Install using continent install script into <code>/opt/Kui-linux-x64/</code> and symlink <code>Kui</code> binary to <code>/usr/local/bin/kui</code>
<source lang=bash>
REPO=kubernetes-sigs/kui
LATEST=$(curl --silent "https://api.github.com/repos/$REPO/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d); FILE=Kui-linux-x64.zip
curl -L https://github.com/$REPO/releases/download/$LATEST/Kui-linux-x64.zip -o $TEMPDIR/$FILE
sudo mkdir -p /opt/Kui-linux-x64
sudo unzip $TEMPDIR/$FILE -d /opt/

# Run
$> /opt/Kui-linux-x64/Kui
</source>

Run Kui as [Kubernetes plugin https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/]
<source lang=bash>
export PATH=$PATH:/opt/Kui-linux-x64/ # make sure Kui libs are in environment PATH
kubectl kui get pods -A # -> a pop up window will show up

$ kubectl plugin list
The following compatible plugins are available:
/opt/Kui-linux-x64/kubectl-kui
</source>
:[[File:ClipCapIt-200428-205600.PNG]]

; Resources
* [https://github.com/IBM/kui/wiki kui/wiki] Github

= [https://github.com/derailed/popeye popeye] =
Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
:[[File:ClipCapIt-200501-123645.PNG]]

<source lang=bash>
# Install
REPO=derailed/popeye
RELEASE=popeye_Linux_x86_64.tar.gz
VERSION=$(curl --silent "https://api.github.com/repos/${REPO}/releases/latest" | jq -r .tag_name); echo $VERSION # latest
wget https://github.com/${REPO}/releases/download/${VERSION}/${RELEASE}
tar xf ${RELEASE} popeye --remove-files
sudo install popeye /usr/local/bin

# Usage
popeye # --out html
</source>

= [https://github.com/derailed/k9s k9s] =
K9s provides a terminal UI to interact with Kubernetes clusters.

;Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/derailed/k9s/releases/latest" | jq -r .tag_name); echo $LATEST
wget https://github.com/derailed/k9s/releases/download/$LATEST/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin
</source>

;Usage
* <code>?</code> help
* <code>:ns</code> select namespace
* <code>:nodes</code> show nodes

:[[File:ClipCapIt-190826-152830.PNG]]

= [https://github.com/droctothorpe/kubecolor kubecolor] =
Kubecolor is a bash function that colorizes the output of kubectl get events -w.
:[[File:ClipCapIt-190831-113158.PNG]]

<source lang=bash>
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
kubectl get events --all-namespaces --watch \
-o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
| awk -F^ \
-v black=$(tput setaf 0) \
-v red=$(tput setaf 1) \
-v green=$(tput setaf 2) \
-v yellow=$(tput setaf 3) \
-v blue=$(tput setaf 4) \
-v magenta=$(tput setaf 5) \
-v cyan=$(tput setaf 6) \
-v white=$(tput setaf 7) \
'{ $1=blue $1; $2=green $2; $3=white $3; } 1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'
</source>
= [https://argoproj.github.io/argo-rollouts/ argo-rollouts] =
Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

= <code>[https://github.com/groundcover-com/murre murre]</code> =
Murre is an on-demand, scaleable source of container resource metrics for K8s. No dependencies needed, no install needed on a cluster.
<source lang=bash>
goenv install 1.18 # although 1.19 is the latest and the install completes successfully it wont create the binary
go install github.com/groundcover-com/murre@latest
murre --sortby-cpu-util
murre --sortby-cpu
murre --pod kong-51xst
murre --namespace dev
</source>

= [https://github.com/amelbakry/kubernetes-scripts/blob/master/cluster-health.sh Kubernetes scripts] =
These Scripts allow you to troubleshoot and check the health status of the cluster and deployments They allow you to gather these information
* Cluster resources
* Cluster Nodes status
* Nodes Conditions
* Pods per Nodes
* Worker Nodes Per Availability Zones
* Cluster Node Types
* Pods not in running or completed status
* Top Pods according to Memory Limits
* Top Pods according to CPU Limits
* Number of Pods
* Pods Status
* Max Pods restart count
* Readiness of Pods
* Pods Average Utilization
* Top Pods according to CPU Utilization
* Top Pods according to Memory Utilization
* Pods Distribution per Nodes
* Node Distribution per Availability Zone
* Deployments without correct resources (Memory or CPU)
* Deployments without Limits
* Deployments without Application configured in Labels

= Multi-node clusters =
{{Note|[[Kubernetes/minikube]] can do this natively}}

Build multi node cluster for development.
On a single machine
* [https://github.com/kinvolk/kube-spawn/ kube-spawn] tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
* [https://github.com/sttts/kubernetes-dind-cluster kubernetes-dind-cluster] Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
* [https://kind.sigs.k8s.io/ kind] is a tool for running local Kubernetes clusters using Docker container “nodes”
* [https://github.com/ecomm-integration-ballerina/kubernetes-cluster Vagrant] full documentation in thsi [https://medium.com/@wso2tech/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 article]

Full cluster provisioning
* [https://github.com/kubernetes-sigs/kubespray kubespray] Deploy a Production Ready Kubernetes Cluster
* [https://github.com/kubernetes/kops kops] get a production grade Kubernetes cluster up and running

= [https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/ crictl] =
CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with <code>crictl</code>. <code>crictl</code> requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.
= [https://github.com/weaveworks/kubediff kubediff] show diff code vs what is deployed =
Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
= Mozilla SOPS - secret manager =
* [https://github.com/mozilla/sops SOPS] Mozilla SOPS: Secrets OPerationS, sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP

Install
<source lang=bash>
LATEST=$(curl --silent "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name); echo $LATEST
TEMPDIR=$(mktemp -d)
curl -sL https://github.com/mozilla/sops/releases/download/${LATEST}/sops-${LATEST}.linux.amd64 -o $TEMPDIR/sops
sudo install $TEMPDIR/sops /usr/local/bin/sops
</source>

= [https://kompose.io/ Kompose] (Kubernetes + Compose) =
kompose is a tool to convert docker-compose files to Kubernetes manifests. <code>kompose</code> takes a Docker Compose file and translates it into Kubernetes resources.
<source lang=bash>
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.21.0/kompose-linux-amd64 -o kompose
sudo install ./kompose /usr/local/bin/kompose # option 1
chmod +x kompose; sudo mv ./kompose /usr/local/bin/kompose # option 2

# Completion
source <(kompose completion bash)

# Convert
kompose convert -f docker-compose-mac.yaml

WARN Restart policy 'unless-stopped' in service mysql is not supported, convert it to 'always'
INFO Kubernetes file "mysql-service.yaml" created
INFO Kubernetes file "cluster-dir-persistentvolumeclaim.yaml" created
INFO Kubernetes file "mysql-deployment.yaml" created
</source>

;References
*[https://github.com/kubernetes/kompose kompose] Github

= [https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/ kube-iptables-tailer] - ip-table drop packages logger =
Allows to view iptables dropped packages, useful when working with Network Policies to identify pods trying to talk to disallowed destinations.

This project deploys <tt>[https://github.com/box/kube-iptables-tailer/tree/master/demo kube-iptables-tailer]</tt> daemonset that watches iptables log <code>/var/log/iptables.log</code> on each k8s-node mounted as <code>hostPath</code> volume. It filters the log for custom prefix, set in <code>daemonset.spec.template.spec.containers.env</code> and sends to cluster events.
<source lang=yaml>
env:
- name: "IPTABLES_LOG_PATH"
value: "/var/log/iptables.log"
- name: "IPTABLES_LOG_PREFIX"
# log prefix defined in your iptables chains
value: "my-prefix:"
</source>

[https://github.com/box/kube-iptables-tailer#setup-iptables-log-prefix Set iptables Log Prefix]
<source lang=bash>
$ iptables -A CHAIN_NAME -j LOG --log-prefix "EXAMPLE_LOG_PREFIX: "
</source>

Example output, when packet dropped
<source lang=bash>
$ kubectl describe pods --namespace=YOUR_NAMESPACE
...
Events:
FirstSeen LastSeen Count From Type Reason Message
--------- -------- ----- ---- ---- ------ -------
1h 5s 10 kube-iptables-tailer Warning PacketDrop Packet dropped when receiving traffic from example-service-2 (IP: 22.222.22.222).
3h 2m 5 kube-iptables-tailer Warning PacketDrop Packet dropped when sending traffic to example-service-1 (IP: 11.111.11.111).
</source>
= [https://github.com/eldadru/ksniff ksniff] - pipe a pod traffic to Wireshark or Tshark =
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod

= [https://docs.flagger.app/ flagger - canary deployments] =
Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINX, Skipper, Contour or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
= [https://www.kubeval.com/ Kubeval] =
Kubeval is used to validate one or more Kubernetes configuration files, and is often used locally as part of a development workflow as well as in CI pipelines.
<source lang=bash>
# Install
wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
sudo cp kubeval /usr/local/bin

# Usage
$> kubeval my-invalid-rc.yaml
WARN - my-invalid-rc.yaml contains an invalid ReplicationController - spec.replicas: Invalid type. Expected: integer, given: string
$> echo $?
1
</source>

= [https://github.com/yannh/kubeconform kubeconform] - improved Kubeval =
Kubeconform is a Kubernetes manifests validation tool.
<source lang=bash>
# Install
wget https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
tar xf kubeconform-linux-amd64.tar.gz
sudo install kubeconform /usr/local/bin

# Show version
kubeconform -v
v0.4.14

</source>

= Observability =
== [https://github.com/oslabs-beta/KUR8 KUR8] - like Elastic.io EFK dashboards ==
{{Note|I've deployed v1.0.0 to monitoring ns along with already existing service <code>
kube-prometheus-stack-prometheus:9090</code> but the application was crashing}}

= CPU Load pods =
<source lang=bash>
# Repeat the command times x CPU
cat /proc/cpuinfo | grep processor | wc -l # count processors
yes > /dev/null &
</source>

= References =
*[https://kubernetes.io/docs/reference/kubectl/overview/ kubectl overview - resources types, Namespaced, kinds] K8s docs
*[https://github.com/johanhaleby/kubetail kubetail] Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods.
*[https://github.com/ahmetb/kubectx kubectx kubens] Kubernetes config switches for context and setting up default namespace
*[https://medium.com/faun/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b manages different ver kubectl] blog
*[https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all kubectl] Kubectl Conventions

Cheatsheets
*[https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4 cheatsheet-kubernetes-A4] by dennyzhang

Other projects
*[https://github.com/jonmosco/kube-tmux kube-tmux] Kubernetes context and namespace status for tmux
*[https://github.com/jonmosco/kube-ps1 kube-ps1] Kubernetes prompt for bash and zsh

[[Category:kubernetes]]

HashiCorp/Vagrant

2024-06-05T19:05:35Z

Pio2pio: /* Sync using vagrant-vbguest plugin */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Re-ristribute the <tt>.box</tt> file then restore it.
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Plugin install
<source lang=bash>
vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

;Manual install
Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox

<source lang=bash>
# Install a matching version to your host version onto the virtual machine.
wget https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

HashiCorp/Vagrant

2024-06-05T19:02:39Z

Pio2pio: /* Sync using vagrant-vbguest plugin */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Re-ristribute the <tt>.box</tt> file then restore it.
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Install plugin
<source lang=bash>
vagrant plugin install vagrant-vbguest

# Verify current version, running on a host(hypervisor)
vagrant vbguest --status

# Add to your Vagrant file
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

Manual install

Download VBoxGuestAdditions from:
* https://download.virtualbox.org/virtualbox
* https://download.virtualbox.org/virtualbox/7.0.16/VBoxGuestAdditions_7.0.16.iso

Install a matching version of your host installation onto the virtual machine.
<source lang=bash>
vagrant vbguest --do install
vagrant vbguest --do install --iso VBoxGuestAdditions_7.0.16.iso

Usage: vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] [-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>

More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial

Ubuntu Setup

2024-06-04T07:17:36Z

Pio2pio: /* gnome-shell-system-monitor-applet - cpu, memory indicators */

If you are using Ubuntu for various Linux projects you will find that as it comes with pre installed with many packages. On the other hand installing just minimal version seems to be too extreme. Therefore I started maitaining a list of unnecessary packages and one liner to that removes them all. Please feel free to modify for your needs.

= Default partitioning =
On virtual systems schema below will be applied, eg on laptops:
:[[File:ClipCapIt-200620-131502.PNG]]

<source lang="bash">
#Eg. for 4G memory and 50G storage system

/dev/mapper/ubuntu--vg-root mount_point: /
/dev/mapper/ubuntu--vg-swapt_1
/dev/sda
/dev/sda1 (50G)

LVM VG ubuntu-vg, LV root as ext4
LVM VG ubuntu-vg, LV swapt_1 as swap

#Boot device:
/dev/mapper/ubuntu--vg-root
</source>
As a good handy practice you may create 100G virtual disk that you thin provision. Then create 2 PVs for root and swap partitions. Don't utilize all space at once but extend partitions when needed. This method eliminates adding new disks to VMs saving time and efforts.

Example LVM setup, here using 30G Physical Volume(99.9% used), 1 Volume Group and 2 Logical Volumes (root and swap).
<source lang="java">
$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 ubuntu-vg lvm2 a-- <29.93g 36.00m
$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
ubuntu-vg 1 2 0 wz--n- <29.93g 36.00m
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root ubuntu-vg -wi-ao---- 28.94g
swap_1 ubuntu-vg -wi-ao---- 976.00m
piotr@u18:~$
</source>

<source lang="java">
$ lsblk /dev/sda --fs
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
└─sda1 LVM2_member rP18Kb-Q12j-wjVf-C1iV-uy42-BUJD-aWFuO7
├─ubuntu--vg-root ext4 fad04a3b-5fa3-4a03-bbd6-24a93cda1eb3 /
└─ubuntu--vg-swap_1 swap 47cd084b-89b0-4cd5-bdb8-367238842ba1 [SWAP]
</source>

= List of unnecessary packages =
<source lang="bash">
sudo apt-get remove libreoffice-* #Remove LibreOffice
sudo apt-get remove unity-lens-* #This package contains photos scopes which allow Unity to search for local and online photos.
sudo apt-get remove shotwell* #Photo organizer
sudo apt-get remove simple-scan #Scanner software
sudo apt-get remove empathy* #Internet messaging ~13M
sudo apt-get remove thunderbird* #Email client ~61M
sudo apt-get remove unity-scope-gdrive #Google Drive scope for Unity ~116KB
sudo apt-get remove cheese* #Cheese Webcam Booth - webcam software
sudo apt-get remove brasero* #Brasero Disc Burner ~6.5MB
sudo apt-get remove gnome-bluetooth Package to manipulate bloototh devices using Gnome desktop ~2MB
sudo apt-get remove gnome-orca Orca Screen Reader -Provide access to graphical desktop environments via synthesised speech and/or refreshable braille
sudo apt-get remove unity-webapps-common #Amazon Unity WebApp integration scripts ~133KB
sudo apt-get remove ibus-pinyin #IBus Bopomofo Preferences - ibus-pinyin is a IBus based IM engine for Chinese ~1.4MB
sudo apt-get remove apt-get remove printer-driver-foo2zjs* #Reactivate HP LaserJet 1018/1020 after reloading paper ~3.2MB
</source>

= Remove unnecessary packages - one liner =
;Ubuntu 12, 14, 16
<source lang="bash">
$ sudo apt-get remove libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* unity-scope-gdrive cheese*\
brasero* gnome-bluetooth gnome-orca unity-webapps-common ibus-pinyin printer-driver-foo2zjs*
</source>

;Ubuntu 18. It's recommended to choose ''Minimal Install'', so most of packages below won't get installed.
<source lang="bash">
$ sudo apt-get purge libreoffice-* unity-lens-* shotwell* simple-scan empathy* thunderbird* cheese* \
brasero* gnome-bluetooth gnome-orca ibus-pinyin printer-driver-foo2zjs* xul-ext-ubufox speech-dispatcher* \
rhythmbox* printer-driver-* mythes-en-us mobile-broadband-provider-inf* \
evolution-data-server* espeak-ng-data:amd64 bluez* ubuntu-web-launchers \
transmission-*
</source>
<source lang="bash">
sudo apt-get purge xul-ext-ubufox # Canonical FF customizations for u14,16,18,20
sudo apt-get remove gnome-mahjongg gnome-mines gnome-sudoku # games, works for u14,16,18,20
sudo apt-get remove gnome-video-effects gstreamer1.0-*
</source>

; XTREME
UnInstallant Ubuntu software notifier
<source lang="bash">
sudo apt-get remove update-notifier
</source>

= Uninstall locales - unused languages etc =
<source lang="bash">
sudo apt-get install localepurge
</source>

= Set apt-get to not install recommended and suggested packages =
<source lang="bash">
sudo bash -c 'cat > /etc/apt/apt.conf.d/01no-recommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF'
</source>

To see if apt reads this, enter this in command line (as root or regular user):
<source lang="bash">
apt-config dump | grep -e Recommends -e Suggests
</source>

= Install necessary packages =

Adobe Flash Player
sudo apt-get install flashplugin-installer

Java JRE
This will install the default verison Java for you distro plus Icedtea plugin for using Firefox with Java
sudo apt-get install default-jre icedtea-plugin

Unity Settings
sudo apt-get install unity-control-center

Opera

Add Opera repository <code>'''deb <nowiki>http://deb.opera.com/opera/</nowiki> stable non-free'''</code> to the apt-get source list in <code>/etc/apt/sources.list</code>. Then import a public PGP repository key.
<source lang="bash">
echo "deb http://deb.opera.com/opera/ stable non-free" | sudo tee -a /etc/apt/sources.list
wget -qO - http://deb.opera.com/archive.key | sudo apt-key add -
sudo apt-get update && sudo apt-get install opera
</source>
Silverlight

Pipelight has been released and we can use it for silverlight as a best alternative moonlight.
<source lang="bash">
sudo apt-add-repository ppa:ehoover/compholio
sudo apt-add-repository ppa:mqchael/pipelight
sudo apt-get update
sudo apt-get install pipelight
</source>

= GUI tools =
* [https://github.com/hluk/CopyQ/releases copyQ] clipboard manager
* VisualVM

= Customise Ubuntu =
==Fix Ubuntu Unity Dash Search for Applications and Files==
sudo apt-get install unity-lens-files unity-lens-applications #log out and log back in required

==Fix Ubuntu <17.10 missing Control Center==
sudo apt-get install unity-control-center --no-install-recommends

==Fix Ubuntu >18.04 missing System Settings==
sudo apt install gnome-control-center

==Remove background wallpaper ==
Tested on Ubuntu 14,16,18
<source lang="bash">
gsettings set org.gnome.settings-daemon.plugins.background active true
gsettings set org.gnome.desktop.background draw-background false #disable
gsettings set org.gnome.desktop.background primary-color "#000000" #set to black
gsettings set org.gnome.desktop.background secondary-color "#000000" #set to black
gsettings set org.gnome.desktop.background color-shading-type "solid" #set solid colour
gsettings set org.gnome.desktop.background picture-uri file:///dev/null #remove wallpaper, not perfect but nothing worked in U15.10
gsettings set com.canonical.unity-greeter draw-user-backgrounds false #disable not worked

# Reset background picture to origin, U15.10
gsettings set org.gnome.desktop.background picture-uri file:///usr/share/backgrounds/warty-final-ubuntu.png

# Sets Unity greeter background, <17.04
gsettings set com.canonical.unity-greeter background /usr/share/backgrounds/warty-final-ubuntu.png
</source>

==Disable screen lock out==
<code>dconf</code> is legacy tool to configure <tt>gnome</tt> nowadays more modern way is to use <code>gsettings</code>.
<source lang=bash>
dconf write /org/gnome/desktop/screensaver/idle-activation-enabled false #gnome
dconf write /org/gnome/desktop/screensaver/lock-enabled false

# Unity - Ubuntu 14.04, 16.04
gsettings set org.gnome.desktop.session idle-delay 0 #disable the screen blackout:(0 to disable)
gsettings set org.gnome.desktop.screensaver lock-enabled false #disable the screen lock

# VirtualBox > Ubuntu 18.04 Disabling Xserver screen timeouts
xset s off # Xserver s parameter sets screensaver to off
xset s noblank # prevent the display from blanking
xset -dpms # prevent the monitor's DPMS energy saver from kicking in

# Gnome - Ubuntu 18.04 LTS, Settings > Power > Blank screen > set to: Never
gsettings get org.gnome.desktop.lockdown disable-lock-screen # verify status
gsettings set org.gnome.desktop.lockdown disable-lock-screen true # set disabled
gsettings get org.gnome.desktop.screensaver lock-enabled # verify status
gsettings set org.gnome.desktop.screensaver lock-enabled false # set disabled
dconf write /org/gnome/desktop/screensaver/lock-enabled false # set disbaled using dconf
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false # some say it's last resort :)

# Power management
gsettings set org.gnome.settings-daemon.plugins.power active true #set gnome to be the default power management run
gsettings set org.gnome.settings-daemon.plugins.power active false #turn off power management

# last resort as it was a bud in Ubuntu 11.10 with DPMS
gsettings set org.gnome.desktop.screensaver idle-activation-enabled false
gsettings set org.gnome.desktop.session idle-delay 2400
</source>
Verify by navigating in <tt>dconf-editor</tt> to <tt>/org/gnome/desktop/screensaver/</tt>

==Change number of workspaces==
To get the current values:
<source lang=bash>
dconf read /org/compiz/profiles/unity/plugins/core/hsize
dconf read /org/compiz/profiles/unity/plugins/core/vsize
</source>

To set new values:
<source lang=bash>
dconf write /org/compiz/profiles/unity/plugins/core/hsize 2
# or
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ hsize 4
gsettings set org.compiz.core:/org/compiz/profiles/unity/plugins/core/ vsize 4
</source>

== Clenup motd messages ==
Ubuntu at login displays a number standard messages taking terminal space causing potential loosing context of previous operations.
<source lang="bash">
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-134-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud

1 package can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Aug 31 12:11:28 2018 from 10.0.2.2
</source>

This is managed by files in <code>/etc/update-motd.d/</code>, so deleting them will remove clutter on a screen
<source lang="bash">
ls /etc/update-motd.d/
00-header 51-cloudguest 91-release-upgrade 98-fsck-at-reboot
10-help-text 90-updates-available 97-overlayroot 98-reboot-required

# Ubuntu Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1022-azure x86_64)
# Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)
sudo rm /etc/update-motd.d/{10-help-text,50-landscape-sysinfo,50-motd-news,51-cloudguest,80-livepatch,95-hwe-eol}
</source>

This cuts down to this message, Ubuntu 18.04 in AWS
<source lang="bash">
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1021-aws x86_64)

0 packages can be updated.
0 updates are security updates.

Last login: Thu Jan 31 17:09:38 2019 from 10.10.11.11
</source>

= Useful setups =
== Call screen saver from a terminal to blank all screens ==
<source lang=bash>
# tested on Ubuntu 18.04 with Gnome
sudo apt-get install gnome-screensaver
gnome-screensaver-command -a #controls GNOME screensaver, -a activate (blank the screen)
</source>

== Create application launcher ==
;Ubuntu 18.04
<source lang=bash>
# Install the GNOME-panel toolset
sudo apt-get install --no-install-recommends gnome-panel

# Every user launcher
sudo gnome-desktop-item-edit /usr/share/applications/VisualVM.desktop --create-new

# Local user only, the filename by default is a Name-of-appication.desktop
gnome-desktop-item-edit ~/.local/share/applications --create-new
</source>

:[[File:ClipCapIt-190807-080016.PNG]]

;Ubuntu 19.10, 20.04
In above releases <code>gnome-desktop-item-edit</code> has been removed from the <code>gnome-panel</code> package, as an alternative <code>.desktop</code> files can be created manually.
<source lang=bash>
vi /usr/share/applications/APPNAME.desktop
[Desktop Entry]
Name=<NAME OF THE APPLICATION>
Comment=<A SHORT DESCRIPTION>
Exec=<COMMAND-OR-FULL-PATH-TO-LAUNCH-THE-APPLICATION>
Type=Application
Terminal=false
Icon=<ICON NAME OR PATH TO ICON>
NoDisplay=false
Keywords=<eg. sql>
</source>

It's optional but you may need to right click and set 'allow launching' with addition to set executable permissions. Usual locations of <code>.desktop</code> files are:
* <code>/usr/share/applications/</code>
* <code>/var/lib/snapd/desktop/applications/</code> for snap applications

== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet gnome-shell-system-monitor-applet] - cpu, memory indicators ==
System information such as memory usage, cpu usage, network rates and more can be displayed in the notification area in GNOME Shell.

System-monitor extensions:
[https://extensions.gnome.org/extension/120/system-monitor/ system-monitor] by paradoxxxzero on [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet github] supports Gnome-shell up to v40. It seems like abandoned project.
[https://extensions.gnome.org/extension/3010/system-monitor-next/ system-monitor-next] by mgalgs on [https://github.com/mgalgs/gnome-shell-system-monitor-applet github] supports Gnome-shell v40+, it's a fork of the above.

All extensions:
* https://extensions.gnome.org

{{Note|The current version of the browser Firefox is packaged as a snap version. One of the issues with this is that it cannot work with the Gnome Extensions website.}}

Install on Ubuntu 24.04 (June 2024)
<source lang=bash>
# Ubuntu 20/22/24
GNOME Shell 46.0 # version as of Ubuntu 24.04
sudo apt install gnome-shell-extensions # Ubuntu 20.04 already has this package, 24.04 needs installing it
sudo apt install gnome-shell-extension-manager # Ubuntu 22|24.04 (as Firefox is installed as snap) on 24.04 it's v0.5.0

# Open `Extensions` app, turn "Use Extensions". # Already turned on on Ubuntu 24.04
# Open Browse tab > search for 'system-monitor-next' # cpu/mem/net indicators will appear in the system tray

# Additional steps for Ubuntu < 24.04
sudo apt install gnome-tweaks # GUI to manage gnome-extensions
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
sudo apt install gnome-shell-extension-system-monitor # after requires log out

# Download the extension from
## https://extensions.gnome.org/extension/120/system-monitor/

# Never worked out how to use this direct download and install via 'gnome-extensions install <extension_name>'
## wget https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/archive/v38.zip
## gnome-extensions install <system-monitor@paradoxxx.zero.gmail.com>

# Enable extension using cli
gnome-extensions enable system-monitor-next@paradoxxx.zero.gmail.com
gnome-extensions list --user
clipboard-indicator@tudmotu.com
system-monitor-next@paradoxxx.zero.gmail.com
</source>
:[[File:ClipCapIt-210105-084527.PNG]]

=== [https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet/issues/737#issuecomment-1230654455 Ubuntu 22.04 workaround for the OUTDATED extension] ===
{{Note|Workaround still needed in August 2022}}
<source lang=bash>
sudo apt install gir1.2-gtop-2.0 gir1.2-nm-1.0 gir1.2-clutter-1.0 gnome-system-monitor
git clone https://github.com/paradoxxxzero/gnome-shell-system-monitor-applet.git
cd gnome-shell-system-monitor-applet # commit b359d88 verified
vi system-monitor@paradoxxx.zero.gmail.com/metadata.json
# | change "version": -1 to "version": 42
make install
# log out and back in (required)
</source>

= Snapd - Chromium =
Recently in U19+ Chromium get installed via snapd package. This is classic installation that has limited access to only a certain directories. It happen that when working with AWS we need get access to <code>~/.ssh</code> folder to get ec2 machine password. This folder is denied, but we can bind mount <code>~/.ssh</code> folder into the snap container directory:
<source lang=bash>
$ snap list chromium
Name Version Rev Tracking Publisher Notes
chromium 86.0.4240.111 1373 latest/stable canonical✓ -

# cd to chromim $HOME dir
mkdir ~/snap/chromium/current/.ssh
sudo mount --bind ~/.ssh/ ~/snap/chromium/current/.ssh
</source>

= Screen shooting =
In Ubuntu 20.04 Shutter is not a part of default repositories. It can be added via PPA:
<source lang=bash>
sudo add-apt-repository -y ppa:linuxuprising/shutter
sudo apt-get install shutter
</source>

= Audio - [https://rastating.github.io/setting-default-audio-device-in-ubuntu-18-04/ set defaults] =
For preserving settings using GUI you can install [https://freedesktop.org/software/pulseaudio/pavucontrol/ PulseAudio Volume Control] <code>pavucontrol</code>.
<source lang=bash>
# install
sudo apt install pavucontrol # Ubuntu 20.04
# run
pavucontrol
</source>

Set default output/input device. In Ubuntu PulseAudio is used to control audio devices. It contains following configuration files
<source lang=bash>
/etc/pulse/default.pa # system wide
~/.config/pulse # user configuration
</source>

Set defaults
<source lang=bash>
# List devices: modules, sinks, sources, sink-inputs, source-outputs, clients, samples, cards
# sinks - outputs, sink-inputs, sources - all input/output including RUNNING and SUSPENDED devices
$ pactl list short sources | column -t
5 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_5__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
6 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_4__sink.monitor module-alsa-card.c s16le 2ch 48000Hz RUNNING
7 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_3__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
8 alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__sink.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
9 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__source module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
10 alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_6__source module-alsa-card.c s16le 4ch 48000Hz SUSPENDED
15 alsa_output.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.analog-stereo.monitor module-alsa-card.c s16le 2ch 48000Hz SUSPENDED
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz SUSPENDED
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz SUSPENDED
20 alsa_input.usb-DisplayLink_Dell_Universal_Dock_D6000_1806021690-02.iec958-stereo module-alsa-card.c s16le 2ch 48000Hz RUNNING

# Set defaut output device. Tab autocompletion should work (U20.04)
pactl set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output
# Set defaut input device
pactl set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Test, play some audio then run. IDLE - means in use
pactl list short sources | column -t | grep -e RUNNING -e IDLE
17 alsa_output.usb-Plantronics_Plantronics_DA40-00.multichannel-output.monitor module-alsa-card.c s16le 1ch 48000Hz IDLE
19 alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback module-alsa-card.c s16le 1ch 16000Hz RUNNING
</source>

Make it permanent by setting default device in PulseAudio system configuration file
<source lang=bash>
# Output device
OUTPUT_DEVICE=alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-sink$ output/\1 ${OUTPUT_DEVICE}/g" /etc/pulse/default.pa # remove '-i' to test before apply
# Input device
INPUT_DEVICE=alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
sudo sed -i "s/#$set-default-source$ input/\1 ${INPUT_DEVICE}/g" /etc/pulse/default.pa

vi /etc/pulse/default.pa # make sure lines below are in place
### Make some devices default
set-default-sink alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
set-default-source alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback

# Delete local user profile and restart system, after boot new defaults should be set
rm -r ~/.config/pulse

# After reboot, defaults should be set
cat ~/.config/pulse/*default*
alsa_output.usb-Plantronics_Plantronics_DA40-00.mono-fallback
alsa_input.usb-Plantronics_Plantronics_DA40-00.mono-fallback
</source>

;Troubleshooting
PulseAudio cli
<source lang=bash>
pacmd
>>> help # lists all available commands

pulseaudio --check # Check if any pulseaudio instance is running. It normally prints no output, just exit code. 0 means running
pulseaudio --kill # kill, then --start
pulseaudio -D # start pulseaudio as a daemon
# | using /etc/pulse/daemon.conf

# Pulseaudio is a user service
systemctl --user restart pulseaudio.service
systemctl --user restart pulseaudio.socket
</source>

I have a port replicator Dell D-6000, that gets randomly disconnected causing switching audio to new connected device - means itself. As workaround commenting out lines below stops this behaviour.
<source lang=bash>
vi /etc/pulse/default.pa
### Use hot-plugged devices like Bluetooth or USB automatically (LP: #1702794)
# .ifexists module-switch-on-connect.so
# load-module module-switch-on-connect
# .endif
</source>

= Input devices =
Motivation is to enable horizontal scrolling in Ubuntu 20.04 using Perixx Gamig Mouse Mx2000
<source lang=bash>
xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Holtek USB Gaming Mouse id=11 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Mouse id=14 [slave pointer (2)]
⎜ ↳ SYNA8007:00 06CB:CD8C Touchpad id=15 [slave pointer (2)]
⎜ ↳ TPPS/2 Elan TrackPoint id=19 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Sleep Button id=8 [slave keyboard (3)]
↳ CHICONY HP Basic USB Keyboard id=9 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=10 [slave keyboard (3)]
↳ Integrated Camera: Integrated C id=12 [slave keyboard (3)]
↳ Integrated Camera: Integrated I id=13 [slave keyboard (3)]
↳ sof-hda-dsp Headset Jack id=16 [slave keyboard (3)]
↳ Intel HID events id=17 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=18 [slave keyboard (3)]
↳ ThinkPad Extra Buttons id=20 [slave keyboard (3)]
↳ Holtek USB Gaming Mouse id=21 [slave keyboard (3)]

# test mouse aka Virtual core pointer
xinput test 11
motion a[0]=2023 # <- cursor moving
motion a[0]=2024 a[1]=1411
motion a[3]=19545 # <- scroll down
button press 5
button release 5

# test 'virtual core keyboard' aka additional programmable buttons
## '10' - this virtual keyboard for all buttons except the scrolling wheel
xinput test 10
key press 37
key press 38

## '21' - this is scrolling wheel buttons left/right, not scrolling itself
xinput test 21
key press 248
key release 248
</source>

# List of properties of a device. We want to see 'horizontal scrolling wheel buttons'
<source lang=bash>
$ xinput list-props 21
Device 'Holtek USB Gaming Mouse':
Device Enabled (169): 1
Coordinate Transformation Matrix (171): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
libinput Send Events Modes Available (291): 1, 0
libinput Send Events Mode Enabled (292): 0, 0
libinput Send Events Mode Enabled Default (293): 0, 0
Device Node (294): "/dev/input/event10"
Device Product ID (295): 1241, 41063
</source>

=References=

[[Category:linux]]

Docker

2024-05-31T22:29:10Z

Pio2pio: /* Add a user to docker group */

Containers taking a world J

= [https://docs.docker.com/install/linux/docker-ce/ubuntu/ Installation] =
General procedure:
# Make sure you don't have docker already installed from your packet manager
# The /var/lib/docker may be

To install the latest version of Docker with curl:
<source lang="bash">
curl -sSL https://get.docker.com/ | sh
</source>

== CentOS ==
<source lang="bash">
sudo yum install bash-completion bash-completion-extras #optional, requires you log out
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 #utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo #docker-ee.repo for EE edition
# --enable docker-ce-{edge|test} #for beta releases
sudo yum update
sudo yum clean all #not sure why this command is here
sudo yum install docker-ce
#old: sudo yum install -y --setopt=obsoletes=0 docker-ce-17.03.1.ce-1.el7.centos docker-ce-selinux-17.03.1.ce-1.el7.centos
sudo systemctl enable docker && sudo systemctl start docker && sudo systemctl status docker
yum-config-manager --disable jenkins #disable source to prevent accidental update ?jenkins?
</source>

== Ubuntu 16.04, 18.04, 20.04 ==
<source lang="bash">
# Optional, clear out config files
sudo rm /etc/systemd/system/docker.service.d/docker.conf
sudo rm /etc/systemd/system/docker.service
sudo rm /etc/default/docker #environment file

# New docker package is called now 'docker-ce'
sudo apt-get remove docker docker-engine docker.io containerd runc docker-ce # start fresh
sudo apt-get -yq install apt-transport-https ca-certificates curl gnupg-agent software-properties-common # apt over HTTPs
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # Docker official GPG key
sudo apt-key fingerprint 0EBFCD88 #verify

#add the repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # or {edge|test}
sudo apt-get update # optional

# Option 1 - install latest
sudo apt-get install docker-ce docker-ce-cli containerd.io

# Option 2 - install fixed version
sudo apt-cache madison docker-ce # display available versions
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
sudo apt-get install docker-ce=18.09.0~3-0~ubuntu-bionic docker-ce-cli=18.09.0~3-0~ubuntu-bionic containerd.io
sudo apt-mark hold docker-ce docker-ce-cli containerd.io
sudo apt-mark showhold # show packages that version upgrade has been put on hold

# Unhold
sudo apt-mark unhold docker-ce docker-ce-cli containerd.io
</source>

[https://docs.docker.com/engine/release-notes/ Newer versions] (>18.09.0) of Docker come with 3 packages:
* <code>containerd.io</code> - daemon to interface with the OS API (in this case, LXC - Linux Containers), essentially decouples Docker from the OS, also provides container services for non-Docker container managers
* <code>docker-ce</code> - Docker daemon, this is the part that does all the management work, requires the other two on Linux
* <code>docker-ce-cli</code> - CLI tools to control the daemon, you can install them on their own if you want to control a remote Docker daemon

Example of how to run [[Jenkins CI|Jenkins docker image]]

== Add a user to docker group ==
Add your user to <tt>docker group</tt> to be able to run docker commands without need of ''sudo'' as the <code>docker.socket</code> is owned by group <code>docker</code>.
<source lang="bash">
sudo usermod -aG docker $(whoami)

# log in to the new docker group (to avoid having to log out / log in again)
newgrp docker
</source>

Reason
<source lang=bash>
[root@piotr]$ ls -al /var/run/docker.sock
srw-rw----. 1 root docker 7 Jan 09:00 docker.sock
</source>

= HTTP proxy =
Configure ''docker'' if you run behind a proxy server. In this example CNTLM proxy runs on the host machine listening on localhost:3128. This example overrides the default docker.service file by adding configuration to the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:
<source lang=bash">
sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"
Environment="HTTP_PROXY=http://172.31.1.1:3128/" #overrides previous entry
Environment="HTTPS_PROXY=http://172.31.1.1:3128/"
# If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable
Environment="NO_PROXY=localhost,127.0.0.1,10.6.96.172,proxy.example.com:80"
</source>

Flush changes:
$ sudo systemctl daemon-reload
Verify that the configuration has been loaded:
$ systemctl show --property=Environment docker
Environment=HTTP_PROXY=<nowiki>http://proxy.example.com:80/</nowiki>
Restart Docker:
$ sudo systemctl restart docker

= Docker create and run, basic options =
<source lang="bash">
# It will create a container but won't start it up
docker container create -it --name="my-container" ubuntu:latest /bin/bash
docekr container start my-container

docker run -it --name="mycentos" centos:latest /bin/bash
# -i :- interactive mode (attach to STDIN) \command to execute when instantiating container
# -t :- attach to the current terminal (sudo TTY)
# -d :- disconnect mode, daemon mode, detached mode, running the task in the background
# -p :- publish to host exposed container port [ host_port(8080):container_exposedPort(80) ]
# --rm :- remove container after command has been executed
# --name="name_your_container"

# -e|--env MYVAR=123 exports/passing variable to the container, echo $MYVAR will have a value 123
# --privileged :- option will allow Docker to perform actions normally restricted,
# like binding a device path to an internal container path.
</source>

= Docker inspect =
== inspect image ==
<source>
docker image inspect centos:6
docker image inspect centos:6 --format '{{.ContainerConfig.Hostname}}' #just a single value
docker image inspect centos:6 --format '{{json .ContainerConfig}}' #json key/value output
docker image inspect centos:6 --format '{{.RepoTags}}' #shows all associated tags with the image
</source>

Using <code>--format</code> is similar to <code>jq</code>
== inspect container ==
Shows current configuration state of a docker container or an image.
<source>
docker inspect <container_name> | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.3",
"IPAddress": "172.17.0.3",
</source>

= Attach/exec to a docker process =
If you are running eg. <tt>/bin/bash</tt> as a command you can get attached to this running docker process. Note that when you exit the container will stop.
<source lang=bash>
docker attach mycentos
</source>

To avoid stopping a container on exit of <code>attach</code> command we can use <code>exec</code> command.
<source lang=bash>
docker exec -it mycentos /bin/bash
</source>

Attaching directly to a running container and then exiting the shell will cause the container to stop. Executing another shell in a running container and then exiting that shell will not stop the underlying container process started on instantiation.

= Entrypoint, CMD, PID1 and [https://github.com/krallin/tini tini] =
== Entrypoint and receiving signals ==
Reciving signals and handling them within containers here Docker it's the same important as for any other application. Remember containers it's a group of processes running on your host so you need to take care of signals send to your applications.

As a principal container management tool eg. <code>docker stop</code> sends a configurable (in Dockerfile) signal to the entrypoint of your application where <code>SIGTERM - 15 - Termination (ANSI)</code> is default.

;ENTRYPOINT syntax

<source>
# exec form, require JSON array; IT SHOULD ALWAYS BE USED
ENTRYPOINT ["/app/bin/your-app", "arg1", "arg2"]

# shell form, it always runs as a subcommand of '/bin/sh -c', thus your application will never see any signal sent to it
ENTRYPOINT "/app/bin/your-app arg1 arg2"
</source>

;ENTRYPOINT is a shell script
If application is started by shell script regular way, your shell spawns your application in a new process and you won’t receive signals from Docker. Therefore we need to tell shell to replace itself with your application using the <code>[https://stackoverflow.com/questions/18351198/what-are-the-uses-of-the-exec-command-in-shell-scripts exec]</code> command, check also <code>[https://en.wikipedia.org/wiki/Exec_(system_call) exec syscall]</code>. Use:
<source lang=bash>
/app/bin/my-app # incorrect, signal won't be received by 'my-app'
exec /app/bin/my-app # correct way
</source>

;ENTRYPOINT exec with a pipe commands causing starting a subshell
If you <code>exec</code> piping will force a command to be run in a subshell with the usual consequence: no signals to an app.
<source lang=bash>
exec /app/bin/your-app | tai64n # here you want to add timestamps by piping through tai64n,
# causing running your command in a subshell
</source>

;Let another program to be PID1 and handle signalling
* tini
* dump-init

<source lang=bash>
ENTRYPOINT ["/tini", "-v", "--", "/app/bin/docker-entrypoint.sh"]
</source>

tini and dumb-init are also able to proxy signals to process groups which technically allows you to pipe your output.However, your pipe target receives that signal at the same time so you can’t log anything on cleanup lest you crave race conditions and SIGPIPEs. So, it's better to avoid logging at termination at all.

;Change signal that will terminate your container process
Listen for SIGTERM or set STOPSIGNAL in your Dockerfile.
<source lang=bash>
vi Dockerfile
STOPSIGNAL SIGINT # this will trigger container termination process if someone press Ctrl^C
</source>

;References:
* [https://hynek.me/articles/docker-signals/ Why Your Dockerized Application Isn’t Receiving Signals]
* [http://smarden.org/runit/ runit] alternative to tini

== Tini ==
It's a tiny but valid init for containers:
* protects you from software that accidentally creates zombie processes
* ensures that the default signal handlers work for the software you run in your Docker image
* does so completely transparently! Docker images that work without Tini will work with Tini without any changes
* Docker 1.13+ has Tini included, to enable Tini, just pass the <code>--init</code> flag to docker run

;Understanding Tini
After spawning your process, Tini will wait for signals and forward those to the child process, and periodically reap zombie processes that may be created within your container. When the "first" child process exits (/your/program in the examples above), Tini exits as well, with the exit code of the child process (so you can check your container's exit code to know whether the child exited successfully).

Add Tini - general dynamicly-linked library (in the 10KB range)
<source lang=bash>
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

# Run your program under Tini
CMD ["/your/program", "-and", "-its", "arguments"]
# or docker run your-image /your/program ...
</source>

Add Tini to Alpine based image
<source lang=bash>
RUN apk add --no-cache tini
# Tini is now available at /sbin/tini
ENTRYPOINT ["/sbin/tini", "--"]
</source>

Existing entrypoint
<source>
ENTRYPOINT ["/tini", "--", "/docker-entrypoint.sh"]
</source>

;References:
*[https://github.com/krallin/tini/issues/8 What is advantage of Tini?]
*[https://ahmet.im/blog/minimal-init-process-for-containers/ Choosing an init process for multi-process containers]

= Mount directory in container =
We can mount host directory into docker container so the content will be available from the container
<source lang="bash">
docker run -it -v /mnt/sdb1:/opt/java pio2pio/java8
# syntax: -v /path/on/host:/path/in/container
</source>

= Build image =
== Dockerfile ==
Each line ''RUN'' creates a container so if possible, we should join lines so it ends up with less layers.
<source lang="bash">
$ wget jkd1.8.0_111.tar.gz
$ cat Dockerfile <<- EOF #'<<-' heredoc with '-' minus ignores <tab> indent
ARG TAGVERSION=6 #only command allowed b4 FROM
FROM ubuntu:${TAGVERSION}
FROM ubuntu:latest #defines base image eg. ubuntu:16.04
LABEL maintainer="myname@gmail.com" #key/value pair added to a metadata of the image

ARG ARG1=value1

ENV ENVIRONMENT="prod"
ENV SHARE /usr/local/share #define env variables with syntax ENV space EnvironmetVariable space Value
ENV JAVA_HOME $SHARE/java

# COPY jkd1.8.0_111.tar.gz /tmp #works only with files, copy a file to container filesystem, here to /tmp
# ADD http://example.com/file.txt
ADD jkd1.8.0_111.tar.gz / #add files into the image root folder, can add also URLs

# SHELL ["executable","params"] #overrides /bin/sh -c for RUN,CMD, etc..

# Executes commands during build process in a new layer E.g., it is often used for installing software packages
RUN mv /jkd1.8.0_111.tar.gz $JAVA_HOME
RUN apt-get update
RUN ["apt-get", "update", "-y"] #in json array format, allows to run a commands but does not require shell executable

VOLUME /mymount_point #this command does not mount anything from a host, just creates a mountpoint

EXPOSE 80 #it doesn't automatically map the port to a hosts

#containers usually don't have system maangement eg. systemctl/service/init.d as designed to run as single process
#entry point becomes main command that start the main proces
ENTRYPOINT apachectl "-DFOREGROUND" #think about it as the MAIN_PURPOSE_OF_CONTAINER command.
# It's always run by default it cannot be overridden

#Single command that will run after the image has been created. Only one per dockerfile, can be overriden.
CMD ["/bin/bash"]

# STOPSIGNAL

EOF
</source>

== Build ==
<source lang="bash">
docker build --tag myrepo/java8 . #-f point to custom Dockerfile name eg. -f Dockerfile2
# myrepo dockerhub username, java8 -image name,
# . directory where is the Dockerfile

docker build -t myrepo/java8 . --pull --no-cache --squash
# --pull regardless a local copy of an image can exist force to download a new image
# --no-cache don't use cache to build, forcing to rebuild all interim containers
# --squash after the build squash all layers into a single layer.

docker images #list images
docker push myrepo/java8 #upload the image to DockerHub repository
</source>

<code>squash</code> is enabled only on docker demon with experimental features enabled.

= Manage containers and images =
== Run a container ==
When you ''run'' a container you will create a new container from a image that has been already build/ is available then put in running state
* -d detached mode, the container will continue to run after the CMD or passed on command exited
* -i interactive mode, allows you to login in /ssh to the container

<source lang="bash">
# docker container run [OPTIONS] IMAGE [COMMAND] [ARG...] # usage
docker container run -it --name mycentos centos:6 /bin/bash
docker run -it pio2pio/java8 #container section command is optional
# -i :- run in interactive mode, then run command /bin/bash
# --rm :- will delete container after run
# --publish | -p 80:8080 :- publish exposed container port 80-> to 8080 on the docker-host
# --publish-all | -P :- publish all exposed container ports to random port >32768
</source>

== List images ==
<source lang="bash">
ctop #top for containers
docker ps -a #list containers
docker image ls #list images
docker images #short form of the command above
docker images --no-trunc
docker images -q #--quiet
docker images --filer "before=centos:6"

# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81
</source>

== Search images in remote repository ==
Search the DockerHub for images,. You may require to do `docker login` first
<source lang="bash">
IMAGE=ubuntu
docker search $IMAGE
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ubuntu Ubuntu is a Debian-based Linux operating sys… 8206 [OK]
dorowu/ubuntu-desktop-lxde-vnc Ubuntu with openssh-server and NoVNC 210 [OK]
rastasheep/ubuntu-sshd Dockerized SSH service, built on top of offi… 167 [OK]

IMAGE=apache
docker search $IMAGE --filter stars=50 # search images that have 50 or more stars
docker search $IMAGE --limit 10 # display top 10 images
</source>

List all available tags
<source lang="bash">
IMAGE=nginx
wget -q https://registry.hub.docker.com/v1/repositories/${IMAGE}/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}' | sort -V
</source>

== Pull images ==
<source lang="bash">
<name>:<tag>
docker pull hello-world:latest # pull latest
docker pull --all hello-world # pull all tags
docker pull --disable-content-trust hello-world # disable verification

docker images --digests #displays sha256: digest of an image

# Dangling images - transitional images
</source>
=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/registries.html#registry_auth from Amazon ECR] ===
;Docker login to ECR service using IAM
<code>docker login</code> does not support native IAM authentication methods. Therefore use a command below that will retrieve, decode, and convert the <code>authorization IAM token</code> into a pre-generated <code>docker login</code> command. Therefore produced login credentials will assume your current IAM User/Role permissions. If your current IAM user can only pull from ECR, after login with <code>docker login</code> you still won't be able to push image to the registry. Example error you may get is <code>not authorized to perform: ecr:InitiateLayerUpload</code>

Login to ECR service, your IAM user requires to have relevant pull/push permissions
<source lang="bash">
eval $(aws ecr get-login --region eu-west-1 --no-include-email)
# aws ecr get-login # generates below docker command with the login token
# docker login -u AWS -p **token** https://$ACCOUNT.dkr.ecr.us-east-1.amazonaws.com # <- output
</source>

Docker login to ECR singular repository, min awscli v1.17
<source lang="bash">
ACCOUNT=111111111111
REPOSITORY=myrepo
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/$REPOSITORY
</source>

=== [https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html push to Amazon ECR] ===
<source lang=bash>
# List images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ansible-aws 2.0.1 b09807c20c96 5 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Tag an image 'b09807c20c96'
docker tag b09807c20c96 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1

# List images, to verify your newly tagged one
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB # <- new tagged image
ansible-aws 2.0.1 b09807c20c96 6 minutes ago 570MB
111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws 1.0.0 9bf35fe9cc0e 4 weeks ago 515MB

# Push an image to ECR
docker push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws:2.0.1
The push refers to repository [111111111111.dkr.ecr.eu-west-1.amazonaws.com/ansible-aws]
2c405c66e675: Pushed
...
77cae8ab23bf: Layer already exists
2.0.1: digest: sha256:111111111193969807708e1f6aea2b19a08054f418b07cf64016a6d1111111111 size: 1796
</source>

== Save and import image ==
In course to move a image to another filesystem we can save it into <code>.tar</code>
<source lang="bash">
# Export
docker image save myrepo/centos:v2 > mycentos.v2.tar
tar -tvf mycentos.v2.tar

# Import
docker image import mycentos.v2.tar <new_image_name>

# Load from a stream
docker load < mycentos.v2.tar #or --input mycentos.v2.tar to avoid redirections
</source>

== Export aka commit container into image ==
Let's say we wanto modify stock image centos:6 by installing Apache interactivly, set to autostart then export as an new image. Let's do it!
<source lang="bash">
docker pull centos:6
docker container run -it --name apache-centos6 centos:6
# Interactively do: yum -y update; yum install -y httpd; chkconfig httpd on; exit

# Save container changes - option1
docker commit -m "added httpd daemon" -a "Piotr" b237d65fd197 newcentos:withapache #creates new image from a container's changes
docker commit -m "added httpd daemon" -a "Piotr" <container_name> <repo>/<image>:<tag>
# -a :- author

# Save container changes - option2
docker container export apache-centos6 > apache-centos6.tar
docker image import apache-centos6.tar newcentos:withapache

docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
newcentos withapache ea5215fb46ed 50 seconds ago 272MB

docker image history newcentos:withapache
IMAGE CREATED CREATED BY SIZE COMMENT
ea5215fb46ed 2 minutes ago 272MB Imported from -
</source>

I am unsure what is a difference in creation of images from a container between:
* <code>docker container commit</code>
* <code>docker container export</code> - this seems creates smaller image

== Tag images ==
Tags are used to usually to name Official image with a new name that we are planning to modify. This allows to create a new image, run a new container from a tag, delete the original image without affecting the new image or container started from the new tagged image.
<source lang="bash">
docker image tag #long version
docker tag centos:6 myucentos:v1 #this will create a duplicate of centos:6 named myucentos:v1
</source>

Tagging allows to modify repository name and maanges references to images located on a filesystem.

== History of an image ==
We can display history of layers that created the image by showing interim images build in creation order. It shows only layers created on a local filesystem.
<source lang="bash">
docker image history myrepo/centos:v2
</source>
== Stop and delete all containers ==
<source lang="bash">
docker stop $(docker ps -aq) && docker rm $(docker ps -aq)
</source>

== Delete image ==
<source lang="bash">
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
company-repo 0.1.0 f796d7f843cc About an hour ago 888MB
<none> <none> 04fbac2fdf48 3 hours ago 565MB
ubuntu 16.04 7aa3602ab41e 3 weeks ago 115MB

# Delete image
$ docker rmi company-repo:0.1.0
Untagged: company-repo:0.1.0
Deleted: sha256:e5cca6a080a5c65eacff98e1b17eeb7be02651849b431b46b074899c088bd42a
..
Deleted: sha256:bc7cda232a2319578324aae620c4537938743e46081955c4dd0743a89e9e8183

# Prune image - delete dangling (temp/interim) images.
# These are not associated with end-product image or containers.
docker image prune
docker image prune -a #remove all images not associated with any container
</source>

== Cleaning up space by removing docker objects ==
This applied both to docker container and swarm systems.
<source lang="bash">
docker system df #show disk usage
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 0 131.7MB 131.7MB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B

docker network ls #note all networks below are system created, so won't get removed
NETWORK ID NAME DRIVER SCOPE
452b1c428209 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local

docker system prune #removes objects created by a user only, on the current node only
#add --volumes to remove them as well
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N]

docker system prune -a --volumes #remove all
</source>

== Docker Volumes ==
Docker's 'copy-on-write' philosophy drives both performance and efficiency. It's only the top layer that is writable and it's a delta of underlying layer.

Volumes can be mounted to your container instances from your underlying host systems.

''_data'' volumes, since they are not controlled by the storage driver (since they represent a file/directory on the host filesystem /var/lib/docker), are able to bypass the storage driver. As a result, their contents are not affected when a container is removed.

Volumes are data mounts created on a host in <code>/var/lib/docker/volumes/</code> directory and refereed by name in a Dockerfile.
<source lang="bash">
docker volume ls #list volumes created by VOLUME directive in a Dockerfile
sudo tree /var/lib/docker/volumes/ #list volumes on host-side
docker volume create my-vol-1
docker volume inspect my-vol-1
[
{
"CreatedAt": "2019-01-17T08:47:01Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-vol-1/_data",
"Name": "my-vol-1",
"Options": {},
"Scope": "local"
}
]
</source>

Using volumes with Swarm services
<source lang=bash>
docker container run --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #container
docker service create --name web1 -p 80:80 --mount source=my-vol-1,target=/internal-mount --replicas 3 httpd #swarm service
# --mount --volumes|-v is not supported with services, this will replicate volumes across swarm when needed,
# but it will not replicate files

docker exec -it web1 /bin/bash #connect to the container
roor@c123:/ echo "Created when connected to container: volume-web1" > /internal-mount/local.txt; exit

# prove the file is on a host filesystem created volume
user@dockerhost$ cat /var/lib/docker/volumes/my-vol-1/_data/local.txt
</source>

;Host storage mount
Bind mapping is binding host filesystem directory to a container directory. It's not mouting volume that it'd require a mount point and volume on a host.
<source lang=bash>
mkdir /home/user/web1
echo "web1 index" > /home/user/web1/index.html
docker container run -d --name testweb -p 80:80 --mount type=bind,source=/home/user/web1,target=/usr/local/apache2/htdocs httpd
curl http://localhost:80
</source>

Removing service is not going to remove the volume unless you delete the volume itself. It that case will be removed from all swarms.

== Networking ==
=== Container Network Model ===
It's a concept of network implementation that is built on multiple private networks across multiple hosts overlayed and managed by IPAM. Protocol that keeps track and provision addesses.

Main 3 components:
* sandbox - contains the configuration of a container's network stack, incl. management of interfaces, routing and DNS. An implementation of a Sandbox could be a eg. Linux Network Namespace. A Sandbox may contain many endpoints from multiple networks.
* endpoint - joins a Sandbox to a Network. Interfaces, switches, ports, etc and belong to only one network at the time. The Endpoint construct exists so the actual connection to the network can be abstracted away from the application. This helps maintain portability.
* network - a clollection of endpoints that can communicate directly (bridges, VLANs, etc.) and can consist of 1toN endpoints

[[File:Container-network-model.png|none||left|Container Network Model]]

;IPAM (Internet Protocol Address Management)
Managing addesees across multiple hosts on a separate physical networks while providing routing to the underlaying swarm networks externally is ''the IPAM prblem'' for Docker. Depends on the netwok driver choice, IPAM is handled at different layers in the stack. ''Network drivers'' enable IPAM through ''DHCP drivers'' or plugin drivers so the complex implementation that would be normally overlapping addesses is supported.

;References
* [https://success.docker.com/article/networking Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks]

=== Publish exposed container/service ports ===
;Publishing modes
;host: set using <code> --publish mode=host,8080:80</code>, makes ports available only on the undelaying host system not outside the host the service may exist; defits ''routing mesh'' so user is responsible for routing
;ingress: responsible for ''routing mesh'' makes sure all published ports are avaialble on all hosts in the swarm cluster regardless is a service replica running on it or not

<source lang=bash>
# List exposed ports on a container
docker port CONTAINER [PRIVATE_PORT[/PROTOCOL]]
docker port web2
80/tcp -> 0.0.0.0:81

# Publish port
host : container
\ : /
docker container run -d --name web1 --publish 81:80 httpd
# --publish | -p :- publish to host exposed container port
# 81 :- port on a host, can use range eg. 81-85, so based on port availability port will be used
# 80 :- exposed port on a container

ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::81 :::*
LISTEN 0 128 :::22 :::*

docker container run -d --name web1 --publish-all 81:80 httpd
# --publish-all | -P publish all cotainer exposed ports to random ports above >32768
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c63efe9cbb94 httpd "httpd-foreground" 2 sec.. Up 1 s 80/tcp testweb #port exposed but not published
cb0711134eb5 httpd "httpd-foreground" 4 sec.. Up 2 s 0.0.0.0:32769->80/tcp testweb1 #port exposed and published to host:32769
</source>

=== Network drivers ===
Default network for a single host docker-host is ''bridge'' network.

;List of Native (part of Docker Engine) Network Drivers:
;bridge: default on stand-alone hosts, it's private network internal to the host system, all containers on this host using Bridge network can communicate, external access is granted by port exposure or static-routes added with teh host as the gateway for that network
;none: used when a container does not need any networkng, still can be accessed from the host using <code>docker attach [containerID]</code> or <code>docker exec -it [containerID]</code> commands
;host: aka ''Host Only Networking'', only accessable via underlaying host, access to services can be provided by exposing ports to the host system
;overlay: swarm scope driver, allows communication to all Docker Daemons in a cluster, self-extending if needed, maanged by Swarm manager, it's default mode of Swarm communication
;ingress: extended network across all nodes in the cluster; special overlay network that load balances netowrk traffic amongst a given service's working nodes; maintains a list of all IP addresses from nodes that participate in that service (using the IPVS module) and when a request comes in, routes to one of them for the indicated service; provides ''routing mesh' that allows services to be exposed to the external network without having replica running on every node in the Swarm
;docker gateway bridge: special bridge network that allows overlay networks (incl. ingress) access an individual DOcker daemon's physical network; every container run within a service is connected to the local Docerk daemon's host network; automatically created when Docker is initialised or joined by <code>docker swarm init</code> or <code>docker join</code> commands.

;Docker interfaces
* <code>docker0</code> - adapter is installed by default during Docker setup and will be assigned an address range that will determine the local host IPs available to the containers running on it

;Create bridge network
<source lang=bash>
docker network ls #default networks list
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm #'ingress' special network 1 per cluster
815a9c2c4005 none null local

docker network inspect bridge #bridge is a default network containers are deployed to

docker container run -d web1 -p 8080:80 httpd #expose container port :80 -> :8080 on the docker-home
docker container inspect web1 | grep IPAdd
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1 #get container ip
curl http://$(IPAddr)
</source>

;Create bridge network
<source lang=bash>
docker network create --driver=bridge --subnet=192.168.1.0/24 --opt "com.docker.network.driver.mtu"=1501 deviceeth0

docker network ls
docker network inspect deviceeth0
</source>

;Create overlay network
<source lang=bash>
docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
docker network ls
NETWORK ID NAME DRIVER SCOPE
130833da0920 bridge bridge local
528db1bf80f1 docker_gwbridge bridge local
832c8c6d73a5 host host local
t8jxy5vsy5on ingress overlay swarm
815a9c2c4005 none null local
2x6bq1czzdc1 overlay0 overlay swarm
</source>

;Inspect network
<source lang=json>
docker network inspect overlay0
[
{
"Name": "overlay0",
"Id": "2x6bq1czzdc102sl6ge7gpm3w",
"Created": "2019-01-19T11:24:02.146339562Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.1.0/24",
"Gateway": "192.168.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": null
}
]
</source>

;Inspect container network
<source lang=bash>
docker container inspect testweb --format {{.HostConfig.NetworkMode}}
overlay0
docker container inspect testweb --format {{.NetworkSettings.Networks.dev_bridge.IPAddress}}
192.168.1.3
</source>

Connect/disconnect from a network can be done when a container is running. Connect won't disconnect from a current network.
<source lang=bash>
docker network connect --ip=192.168.1.10 deviceeth0 web1
docker container inspect --format="{{.NetworkSettings.Networks.bridge.IPAddress}}" web1
docker container inspect --format="{{.NetworkSettings.Networks.deviceeth0.IPAddress}}" web1
curl http://$(IPAddr)
docker network disconnect deviceeth0 web1
</source>

=== Overlay network in Swarm cluster ===
Overlay network can be created/removed/updated like any other docker objects. It allows inter-service(containers) communication, where <code>--gateway</code> ip address is used to reach to outside eg. Inernet or the host network. When creating the <code>overlay</code> network on the manager host it will get recreated on worker nodes only when is referenced by any service that is using it. See below.
<source lang="bash">
swarm-mgr$ docker network create --driver=overlay --subnet=192.168.1.0/24 --gateway=192.168.1.1 overlay0
swarm-mgr$ docker service create --name web1 -p 8080:80 --network=overlay0 --replicas 2 httpd
uvxymzdkcfwvs2oznbnk7nv03
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]

swarm-wkr$ docker network ls
NETWORK ID NAME DRIVER SCOPE
ba175ebd2a6f bridge bridge local
a5848f607d8c docker_gwbridge bridge local
fccfb9c1fdc3 host host local
t8jxy5vsy5on ingress overlay swarm
127b10783faa none null local
2x6bq1czzdc1 overlay0 overlay swarm

# remove network, only affected newly created servces not the running onces
swarm-mgr$ docker service update --network-rm=overlay0 web1
</source>

=== DNS ===
<source lang="bash">
docker container run -d --name testweb1 -P --dns=8.8.8.8 \
--dns=8.8.4.4 \
--dns-search "mydomain.local" \
httpd
# -P :- publish-all exposed ports to random port >32768

docker container exec -it testweb1 /bin/bash -c 'cat /etc/resolv.conf'
search us-east-2.compute.internal
nameserver 8.8.8.8
nameserver 8.8.4.4

# System wide settings, requires docker.service restart
cat > /etc/docker/daemon.json <<EOF
{
"dns": ["8.8.8.8", "8.8.4.4"]
}
EOF
sudo systemctl restart docker.service #required
</source>

= Examples =
== Lint - best practices ==
<source lang=bash>
$ docker run --rm -i hadolint/hadolint < Dockerfile
/dev/stdin:9:16 unexpected newline expecting "\ ", '=', a space followed by the value for the variable 'MAC_ADDRESS', or at least one space
</source>

== Default project ==
As good practice all Docker files should be source controlled. The basic self explanatory structure can looks like below, and skeleton be created with a commend below:
<source lang="bash">
mkdir APROJECT && d=$_; touch $d/{build.sh,run.sh,Dockerfile,README.md,VERSION};mkdir $d/assets; touch $_/{entrypoint.sh,install.sh}

└── APROJECT
├── assets
│ ├── entrypoint.sh
│ └── install.sh
├── build.sh
├── Dockerfile
├── README.md
└── VERSION
</source>

== Dockerfile ==
<code>Dockerfile</code> it is simply a build file.
=== Semantics ===
;<code>entrypoint</code>: Container config: what to start when this image is ran.

;<code>entrypoint</code> and <code>cmd</code>: Docker allows you to define an Entrypoint and Cmd which you can mix and match in a Dockerfile. Entrypoint is the executable, and Cmd are the arguments passed to the Entrypoint. The Dockerfile schema is quite lenient and allows users to set Cmd without Entrypoint, which means that the first argument in Cmd will be the executable to run.

=== User management ===
<source>
RUN addgroup --gid 1001 jenkins -q
RUN adduser --gid 1001 --home /tank --disabled-password --gecos '' --uid 1001 jenkins
# --gid add user to group 1001
# --gecos parameter is used to set the additional information. In this case it is just empty.
# --disabled-password it's like --disabled-login, but logins are still possible (for example using SSH RSA keys) but not using password authentication
USER jenkins:jenkins #sets user for next RUN, CMD and ENTRYPOINT command
WORKDIR /tank #changes cwd for next RUN, CMD, ENTRYPOINT, COPY and ADD
</source>

=== Multiple stage build ===
Introduced in Docker 17.06, allows to use multiple <code>FROM</code> statements allowing for multi stage builds.
<source>
FROM microsoft/aspnetcore-build AS build-env
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o output

# build runtime image
FROM microsoft/aspnetcore
WORKDIR /app
COPY --from=build-env /app/output . #multi stage: copy files from previous container [as build-env]
ENTRYPOINT ["dotnet", "LetsKube.dll"]
</source>

= Squash an image =
Docker uses <code>Union</code> filesystem that allows multiple volumes (layers) to share common and override changes by applying them on top layer.
There is no official way to ''flatten'' layers to a single storage layer or minimize an image size (2017). Below it's just practical approach.
# Start container from an image
# Export a container to <code>.tar</code> with all it's file systems
# Import container with new image name

Once the process completes and original image gets deleted the new image <code>docker image history</code> command will show only one layer. Often the image will be smaller.
<source lang=bash>
# run a container from an image
docker run myweb:v3
# export container to .tar
docker export <contr_name> > myweb.v3.tar
docker save <image id> > image.tar #not verified command
docker import myweb.v3.tar myweb:v4
docker load myweb.v3.tar #not verified command
</source>

;Resources
*[https://github.com/jwilder/docker-squash docker-squash] GitHub

= Gracefully stop / kill a container =
''all below are only notes''

Trap ctrl_c then kill/rm container.
*--init
*--sig-proxy this only works when --tty=false but by default is true

= Proxy =
If you are behind corporate proxy, you should use Docker client <code>~/.docker/config.json</code> config file. It requires Docker
17.07 minimum version.
<source>
{
"proxies":
{
"default":
{
"httpProxy": "http://10.0.0.1:3128",
"httpsProxy": "http://10.0.0.1:3128",
"noProxy": "localhost,127.0.0.1,*.test.example.com,.example2.com"
}
}
}
</source>
More you can find [https://docs.docker.com/network/proxy/#configure-the-docker-client here]

== Insecure proxy ==
These can be added to different places, the order is based on latest practices and versioning
;docker-ce 18.6
<source lang="json">
{
"insecure-registries" : [ "localhost:443","10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
}
</source>
<source lang="bash">
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl show docker | grep Env
docker info #check Insecure Registries
</source>

;Using environment file, prior version 18
<source lang="bash">
$ sudo vi /etc/default/docker
DOCKER_HOME='--graph=/tank/docker'
DOCKER_GROUP='--group=docker'
DOCKER_LOG_DRIVER='--log-driver=json-file'
DOCKER_STORAGE_DRIVER='--storage-driver=btrfs'
DOCKER_ICC='--icc=false'
DOCKER_IPMASQ='--ip-masq=true'
DOCKER_IPTABLES='--iptables=true'
DOCKER_IPFORWARD='--ip-forward=true'
DOCKER_ADDRESSES='--host=unix:///var/run/docker.sock'
DOCKER_INSECURE_REGISTRIES='--insecure-registry 10.0.0.0/8 --insecure-registry 172.16.0.0/12 --insecure-registry 192.168.0.0/16'
DOCKER_OPTS="${DOCKER_HOME} ${DOCKER_GROUP} ${DOCKER_LOG_DRIVER} ${DOCKER_STORAGE_DRIVER} ${DOCKER_ICC} ${DOCKER_IPMASQ} ${DOCKER_IPTABLES} ${DOCKER_IPFORWARD} ${DOCKER_ADDRESSES} ${DOCKER_INSECURE_REGISTRIES}"

$ sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/dockerd $DOCKER_HOME $DOCKER_GROUP $DOCKER_LOG_DRIVER $DOCKER_STORAGE_DRIVER $DOCKER_ICC $DOCKER_IPMASQ $DOCKER_IPTABLES $DOCKER_IPFORWARD $DOCKER_ADDRESSES $DOCKER_INSECURE_REGISTRIES

$ sudo vi /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
EnvironmentFile=-/etc/default/docker
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
</source>

== Run docker without sudo ==
Adding a user to docker group should be sufficient. However on apparmor, SELinux or a filesystem with ACL enabled additional permissions might be required in respect to access a <tt>socket file</tt>.
<source>
$ ll /var/run/docker.sock
srw-rw---- 1 root docker 0 Sep 6 12:31 /var/run/docker.sock=
# ACL
$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
group::rw-
other::---

# Grant ACL to jenkns user
$ sudo setfacl -m user:username:rw /var/run/docker.sock

$ sudo getfacl /var/run/docker.sock
getfacl: Removing leading '/' from absolute path names
# file: var/run/docker.sock
# owner: root
# group: docker
user::rw-
user:jenkins:rw-
group::rw-
mask::rw-
other::---
</source>
;References
* [https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo how-can-i-use-docker-without-sudo]

== References ==
*[https://www.weave.works/blog/my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies/ my-container-wont-stop-on-ctrl-c-and-other-minor-tragedies]
*[https://github.com/moby/moby/pull/12228 PID1 in container aka tinit]
*[https://container-solutions.com/understanding-volumes-docker/ understanding-volumes-docker]

= Docker Enterprise Edition =
*[https://success.docker.com/article/compatibility-matrix Compatibility Matrix]
Components:
* Docker daemon (fka "Engine")
* Docker Trusted Registry (DTR)
* Docker Universal Control Plane (UCP)

= Docker Swarm =
== Swarm - sizing ==
;Universal Control Plane (UCP)
This is for only Enterpsise Edition
* ports managers, workers in/out

Hardware requirments:
* 8gb RAM for managers or DTR Docker Trsuted Registry
* 4gb RAM for workers
* 3gb free space

Performance Consideration (Timing)
<source>
Component Timeout(ms) Configurable
Raft consensus between manager nodes 3000 no
Gossip protocol for overlay networking 5000 no
etcd 500 yes
RethinkDB 10000 no
Stand-alone swarm 90000 no
</source>

Compatibility Docker EE
* Docker Engine 17.06+
* DTR 2.3+
* UCP 2.2+
== Swarm with single host manager ==
<source lang=bash>
# Initialise Swarm
docker swarm init --advertise-addr 172.31.16.10 #Iyou get SWMTKN-token
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

# Join tokens
docker swarm join-token manager #display manager join-token, run on manager
docker swarm join-token worker #display worker join-token, run on manager

# Join worker, run new-worker-node
# -> swarm cluster id <-> this part is mgr/wkr <- -> mgr node <-
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377

# Join another manager, run on new-manager-node
docker swarm join-token manager #run on primary manager if you wish add another manager
# in output you get a token. You notice that 1st part up to dash identifies Swarm cluster and the other part is role id.

# join to swarm (cluster), token will identify a role in the cluster manager or worker
docker swarm join --token SWMTKN-xxxx
docker swarm join --token SWMTKN-1-1i2v91qbj0pg88dxld15vpx3e74qm5clk7xkcrg6j3xknedqui-dh60f4j09itiqjfhqa196ufvo 172.31.16.10:2377
This node joined a swarm as a worker.
</source>

Check Swarm status
<source lang=bash>
docker node ls
[cloud_user@ip-172-31-16-10 swarm-manager]$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * ip-172-31-16-10.mylabserver.com Ready Active Leader 18.09.0
vlw7te728z7bvd7ulb3hn08am ip-172-31-16-94.mylabserver.com Ready Active 18.09.0

docker system info | grep -A 7 Swarm
Swarm: active
NodeID: 641bfndn49b1i1dj17s8cirgw
Is Manager: true
ClusterID: 4jqxdmfd0w5pc4if4fskgd5nq
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
</source>

;Troubleshooting
<source lang=bash>
sudo systemctl disable firewalld && sudo systemctl stop firewalld # CentOS
sudo -i; printf "\n10.0.0.11 mgr01\n10.0.0.12 node01\n" >> /etc/hosts # Add nodes to hosts file; exit
</source>
== Swarm cluster ==
<source lang=bash>
docker node update -availability drain [node] #drain services for Manager Only nodes
docker service update --force [service_name] #force re-balance services across cluster

docker swarm leave #node leaves a cluster
</source>

== Locking / unlocking swarm cluster ==
Logs used by Swarm manager are encrypted on disk. Access to nodes gives access to keys that encrypt them. It further protects cluster as requires a unlocking key when restarting manager/nodes.
<source lang=bash>
docker swarm init --auto-lock=true #initialize with
docker swarm update --auto-lock=true #update current swarm
# both will produce unlock token STKxxx
docker swarm unlock #it'll ask for the unlock token
docker swarm update --auto-lock=false #disable key locking
</source>

If you have access to a manager you can always get unlock key using:
<source lang=bash>
docker swarm unlock-key
</source>

Key management
<source lang=bash>
docker swarm unlock-key --rotate #could be in a cron
</source>
== Backup and restore swarm cluster ==
This priocess describes how to backup whole cluster configuration so can be restored on a new set of servers.

Create docker apps running across swarm
<source lang=bash>
docker service create --name bkweb --publish 80:80 --replicas 2 httpd
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
q9jki3n2hffm bkweb replicated 2/2 httpd:latest *:80->80/tcp

$ docker service ps bkweb #note containers run on 2 different nodes
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE
j964jm1lq3q5 bkweb.1 httpd:latest server2c.mylabserver.com Running Running about a minute ago
jpjx3mk7hhm0 bkweb.2 httpd:latest server1c.mylabserver.com Running Running about a minute ago
</source>

;Backup state files
<source lang=bash>
sudo -i
cd /var/lib/docker/swarm
cat docker-state.json #contains info about managers, workers, certificates, etc..
cat state.json
sudo systemctl stop docker.service

# Backup swarm cluster, this file can be then used to recover whole swarm cluster on another set of servers
sudo tar -czvf swarm.tar.gz /var/lib/docker/swarm/

#the running docker containers should be brought up as they were before stopping the service
systemctl start docker
</source>

Recover using swarm.tar backup
<source lang=bash>
# scp swarm.tar to recovery node - what it'd be a node with just installed docker
sudo rm -rf /var/lib/docker/swarm
sudo systemctl stop docker

# Option1 untar directly
sudo tar -xzvf swarm.tar.gz -C /var/lib/docker/swarm

# Option2 copy recursivly, -f override if a file exists
tar -xzvf swarm.tar.gz; cd /var/lib/docker
cp -rf swarm/ /var/lib/docker/

sudo systemctl start docker
docker swarm init --force-new-cluster # produces the exactly same token
# you should join all required nodes to new manager ip
# scale services down to 1, then scale up so get distributed to other nodes
</source>

== Run containers as a services ==
Docker container has a number limitation therefore running as a service where Cluster Manager: Swarm or Kubernetes manages networking, access, loadbalancing etc.. is a way to scale with ease. The service is using eg. mesh routing to deal with access to containers.

Swarm nodes setup 1 manager and 2 workers
<source lang=bash>
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.0
</source>

Create and run a service
<source lang=bash>
docekr pull httpd
docker service create --name serviceweb --publish 80:80 httpd
# --publish|-p -expose a port on all containers in the running cluster

docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vt0ftkifbd84 serviceweb replicated 1/1 httpd:latest *:80->80/tcp

docker service ps serviceweb #show nodes that a container is running on, here on mgr-1 node
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
e6rx3tzgp1e5 serviceweb.1 httpd:latest swarm-mgr-1.example.com Running Running about
</source>

When running as a service even if a container runs on a single node (replica=1) the container can be accessed from any of swarm nodes. It's because service exposed port has been exposed to extended mesh private network that the container is running on.
<source lang=bash>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-mgr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-1.example.com
<html><body><h1>It works!</h1></body></html>
[user@swarm-mgr-1 ~]$ curl -k http://swarm-wkr-2.example.com
<html><body><h1>It works!</h1></body></html>
</source>

Service update, can be done to limits, volumes, env-variables and more...
<source lang=bash>
docker service scale devweb=3 #or
docker service update --replicas 3 serviceweb #--detach=false shows visual progress in older versions, default in v18.06
serviceweb
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged

# Limits(soft limit) and reservations(hard limit), this causes to start new services(containers)
docker service update --limit-cpu=.5 --reserve-cpu=.75 --limit-memory=128m --reserve-memory=256m serviceweb
</source>

== Templating service names ==
This allows to control eg. hostname in a cluster. Useful for big clusters to easier identify services where they run from hostname.
<source lang=bash>
docker service create --name web --hostname"{{.Node.ID}}-{{.Service.Name}}" httpd
docker service ps --no-trunc web
docker inspect --format="{{}.Config.Hostname}" web.1.ab10_serviceID_cd
aa_nodeID_bb-web
</source>
== Node lables for task/service placement ==
<source lang=bash>
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
641bfndn49b1i1dj17s8cirgw * swarm-mgr-1.example.com Ready Active Leader 18.09.1
vlw7te728z7bvd7ulb3hn08am swarm-wkr-1.example.com Ready Active 18.09.1
r8h7xmevue9v2mgysmld59py2 swarm-wkr-2.example.com Ready Active 18.09.1

docker node inspect 641bfndn49b1i1dj17s8cirgw --pretty
ID: 641bfndn49b1i1dj17s8cirgw
Hostname: swarm-mgr-1.example.com
Joined at: 2019-01-08 12:16:56.277717163 +0000 utc
Status:
State: Ready
Availability: Active
Address: 172.31.10.10
Manager Status:
Address: 172.31.10.10:2377
Raft Status: Reachable
Leader: Yes
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 2
Memory: 3.699GiB
Plugins:
Log: awslogs, fluentd, gcplogs, gelf, journald, json-file, local, logentries, splunk, syslog
Network: bridge, host, macvlan, null, overlay
Volume: local
Engine Version: 18.09.1
TLS Info:
TrustRoot:
-----BEGIN CERTIFICATE-----
MIIBajCCARCgAwIBAgIUKXz3wtc8OA8uzTo1pO86ko+PB+EwCgYIKoZIzj0EAwIw
..
-----END CERTIFICATE-----
Issuer Subject: MBMxETAPBgNVBAMTCHN3YX.....h
Issuer Public Key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEy......==
</source>

Apply label to a node
<source lang=bash>
docker node update --label-add node-env=testnode r8h7xmevue9v2mgysmld59py2
docker node inspect r8h7xmevue9v2mgysmld59py2 --pretty | grep -B1 -A2 Labels
ID: r8h7xmevue9v2mgysmld59py2
Labels:
- node-env=testnode
Hostname: swarm-wkr-1.example.com
</source>

How to use it. Run a service with <code>--constraint</code> option that pins services to run on a node meeting given criteria. In our case to run on a node where <code>node.labels.node-env == testnode</code>. Note that all replicas are running on the same node unlike they'd be distributed across the cluster.
<source lang=bash>
docker service create --name constraints -p 80:80 --constraint 'node.labels.node-env == testnode' --replicas 3 httpd #node.role, node.id, node.hostname
zrk15vfdaitc1rvw9wqh2s0ot
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
[cloud_user@mrpiotrpawlak1c ~]$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
zrk15vfdaitc constraints replicated 3/3 httpd:latest *:80->80/tcp

[user@swarm-wkr-2 ~]$ docker service ps constraints
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
y5z4mt99uzpo constraints.1 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
zqbn4ips969q constraints.2 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
vnb10jcs2915 constraints.3 httpd:latest swarm-wkr-2.example.com Running Running 41 seconds ago
</source>

== Scaling services ==
These commands be issued on a manager node
<source lang=bash>
docker pull docker nginx
docker service create --name web --publish 80:80 httpd
docker service ps web #there is only 1 replica
docker service update --replicas 3 web #update to 3 replicas
docker service create --name nginx --publish 5901:80 nginx
elinks http://swarm-mgr-1.com:5901 #nginx website will be presented

# scale is equivalent to update --replicas command for a single or multiple services
docker service scale web=3 nginx=3
docker service ls
</source>

== Replicated services vs global services ==
;Global Replicated: mode runs at least one copy of a service on each swarm node, even if you join another node the service will coverage there as well. In global mode you cannot use <code>update --replicats</code> or <code>scale</code> commands. It is not possible to update the mode type.
;Replicated mode: allows for grater control and flexibility of running number of services.
<source lang=bash>
# creates a single service running across whole cluster in replicated mode
docker service create --name web --publish 80:80 httpd

# run in a global node
docker service create --name web --publish 5901:80 --mode global httpd
docker service ls #note distinct mode names: global and replicated
</source>

= Docker compose and deploy to Swarm =
Install
<source lang=bash>
sudo yum install epel
sudo yum install pip
sudo pip install --upgrade pip
# install docker CE or EE to avoid Python libs conflits
sudo pip install docker-compose

# Troubleshooting
## Err: Cannot uninstall 'requests'. It is a distutils installed project...
pip install --ignore-installed requests
</source>

Dockerfile
<source lang=bash>
cat >Dockerfile <<EOF
FROM centos:latest
RUN yum install -y httpd
RUN echo "Website1" >> /var/www/html/index.html
EXPOSE 80
ENTRYPOINT apachectl -DFOREGROUND
EOF
</source>

Docker compose file
<source lang=bash>
cat >docker-compose.yml <<EOF
version: '3'
services:
apiweb1:
image: httpd_1:v1
build: .
ports:
- "81:80"
apiweb2:
image: httpd_1:v1
ports:
- "82:80"
load-balancer:
image: nginx:latest
ports:
- "80:80"
EOF
</source>

Run docker compose, on the current node only
<source lang=bash>
docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating compose_apiweb2_1 ... done
Creating compose_apiweb1_1 ... done
Creating compose_load-balancer_1 ... done

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
14f8b6b10c2d nginx:latest "nginx -g 'daemon of…" 2 minutesUp 2 min 0.0.0.0:80->80/tcp compose_load-balancer_1
e9b5b37fe4e5 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:81->80/tcp compose_apiweb1_1
28ee22a8eae0 httpd_1:v1 "/bin/sh -c 'apachec…" 2 minutesUp 2 min 0.0.0.0:82->80/tcp compose_apiweb2_1

# Verify
curl http://localhost:81
curl http://localhost:82
curl http://localhost:80 #nginx

# Prep before deploying docker-compose to Swarm. Also images needs to be build before hand.
# Docker stack does not support building images
docker-compose down --volumes #save everything to storage volumes
Stopping compose_load-balancer_1 ... done
Stopping compose_apiweb1_1 ... done
Stopping compose_apiweb2_1 ... done
Removing compose_load-balancer_1 ... done
Removing compose_apiweb1_1 ... done
Removing compose_apiweb2_1 ... done
Removing network compose_default
</source>

;Deploy compose to Swarm
<source lang=bash>
docker stack deploy --compose-file docker-compose.yml customcompose-stack #customcompose-stack is a prefix for service name
Ignoring unsupported options: build
Creating network customcompose-stack_default
Creating service customcompose-stack_apiweb1
Creating service customcompose-stack_apiweb2
Creating service customcompose-stack_load-balancer

docker stack services customcompose-stack #or
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
k7wwkncov49p customcompose-stack_apiweb1 replicated 0/1 httpd_1:v1 *:81->80/tcp
nl0j5folpmha customcompose-stack_apiweb2 replicated 0/1 httpd_1:v1 *:82->80/tcp
x6p14gmpjyra customcompose-stack_load-balancer replicated 1/1 nginx:latest *:80->80/tcp

docker stack rm customcompose-stack #remove stack

</source>

= Selecting a Storage Driver =
Go to Docker version matrix to verify what drivers are supported on your platform. Changing storage driver is destructive and you loose all containers volumes. Therefore you need to export/backup then re-import after the storage driver change.

;CentOS
Device mapper is officialy supported on CentOS. It can be used on a disc as blockstorage it uses loopback adapter to provide that. Or can be blockstorage devive allowing Docker to mamange it.

<source lang=bash>
docker info --format '{{json .Driver}}'
docker info -f '{{json .}}' | jq .Driver
docker info | grep Storage

sudo touch /etc/docker/daemon.json
sudo vi /etc/docker/daemon.json #additional options are available
{
"storage-driver":"devicemapper"
}
</source>

Preserving any current images, requires export/backup and re-import after the storage driver change.
<source lang=bash>
docker images
sudo systemctl docker restart
ls -l /var/lib/docker/devicemapper #new location to storing images
</source>

Note, in <code>/var/lib/docker</code> new directory <code>devicemapper</code> has been created to store images from now on.

;Update 2019 - Docker Engine 18.09.1
<source>
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
</source>

= Selecting a logginig driver =
Available list of [https://docs.docker.com/config/containers/logging/configure/#supported-logging-driverslogging drivers] can be seen on Docker documentation page. Most popular are:
*none - No logs are available for the container and docker logs does not return any output.
*json-file - (default) the logs are formatted as JSON. The default logging driver for Docker.
*syslog - Writes logging messages to the syslog facility. The syslog daemon must be running on the host machine.
*journald - Writes log messages to journald. The journald daemon must be running on the host machine.
*fluentd - Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host machine.
*awslogs - Writes log messages to Amazon CloudWatch Logs.
*splunk - Writes log messages to splunk using the HTTP Event Collector.
*etwlogs - (Windows) Writes log messages as Event Tracing for Windows (ETW) events

<source lang=bash>
docker info | grep logging
docker container run -d --name <webjson> --logg-driver json-file httpd #per docker container setup
docker logs <testjson>

docker container run -d --name <web> httpd #start new container
docker logs -f _testweb_ #display standard-out logs
docker service log -f <web> #for swarm all container replicas logs
</source>

Enable syslog logginig driver
<source lang=bash>
sudo vi /etc/rsyslog.conf
#uncomment below
$ModLoad imudp
$UDPServerRun 514

sudo systemctl start rsyslog
</source>

Change logging driver. Then standard output won't be available after the change.
<source lang=bash>
{
"log-driver": "syslog",
"log-opts": {
"syslog-address": "udp://172.31.10.1"
}
}

sudo systemctl restart docker
docker info | grep logging
tail -f /var/log/messages #this will show all logging eg. access logs for httpd server
</source>
== Docker daemon logs ==
System level logs
<source lang=bash>
# CentOS
/var/messages | grep -i docker

# Ubuntu
sudo journalctl -u docker.service --no-hostname
sudo journalctl -u docker -o json | jq -cMr '.MESSAGE'
sudo journalctl -u docker -o json | jq -cMr 'select(has("CONTAINER_ID") | not) | .MESSAGE'
/var/log/syslog | grep -i docker
</source>

;Docker container or service logs
<source lang=bash>
docker container logs [OPTIONS] containerID #single container logs
docker service logs [OPTIONS] service|task #agregate logs across all cluster deployed container replicas
</source>

= Container life-cycle policies - eg. autostart =
<source lang=bash>
docker container run -d --name web --restart <on-failure|unless-stopped|no|none(default)|always> httpd
# --restart -restart on crash or exit 1 or service or system reboot
</source>
Definitions:
* always - it will restart container always, even if stopped manually, restarting docker-deamon will start container
* unless-stopped - it will restart container always unless stopped manually by <code>docker container stop</code>
* on-failure - restart if container exits with non-zero exit code

= Universal Control Plance - UCP =
It's an application what allow to see all operational details for Swarm cluster when using Docker EE editin. 30 days trial is available.

;Communication between Docker Engine, UCP and DTR (Docker Trusted Registry)
* over TCP/UDP - depends on a port, and whether a response is required, or if a message is a notification
* IPC - interprocess communication (intra-host), services on the same node
* API - over TCP, uses API directlyto query or update components in a cluster

;References
* [https://docs.docker.com/ee/ucp/ucp-architecture/ UCP architecture]

== Install/uninstall UCP <code>image: docker/ucp</code> ==
OS support:
* UCP 2.2.11 is supported running on RHEL 7.5 and Ubuntu 18.04

For labs purpose, we can use eg. <code>ucp.example.com</code> the domain <code>example.com</code> is included in UCP and DTR wildcard self-signed certificate.

Install on a manager node
<source lang=bash>
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_MGR_NODE_IP=172.31.101.248

docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.15 \
install --host-address=$UCP_MGR_NODE_IP --interactive --debug

# --rm :- because this container will be only transitinal container
# -it :- because installation we want interactive
# -v :- link the container with a file on a host
# --san :- add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com)
# --host-address :- IP address or network interface name to advertise to other nodes
# docker/ucp:2.2.11 :- image version
# --dns :- custom DNS servers for the UCP containers
# --dns-search :- ustom DNS search domains for the UCP containers
# --admin-username "$UCP_USERNAME" --admin-password "$UCP_PASSWORD" #seems these are not supported, although are in a guide
</source>

If not provided you will be asked for:
* Admin password during the process
* You may enter additional aliases (SANs) now or press enter to proceed with the above list:
** Additinall aliases: ucp ucp.example.com
DEBU[0062] User entered: ucp ucp.ciscolinux.co.uk
DEBU[0062] Hostnames: [host1c.mylabserver.com 127.0.0.1 172.17.0.1 172.31.101.248 ucp ucp.ciscolinux.co.uk]

You may want to add DNS entries in <code>/etc/hosts</code> for
* ''ucp'' or ''ucp.example.com'' pointing to manager public ip
* ''dtr'' or ''dtr.example.com'' pointing a worker node public IPs.

;Verify
* connect to https://ucp.example.com:443.
* <code>docker ps</code> should see a number of containers running now, they need to see each other therefore we used <code>hosts</code> entries to allow this.

;Uninstall
<source lang=bash>
docker container run --rm -it --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp uninstall-ucp --interactive

INFO[0000] Your engine version 18.09.1, build 4c52b90 (4.15.0-1031-aws) is compatible with UCP 3.1.2 (b822777)
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: t0ltwwcw5tdbtjo2fxlzmj8p4
Do you want to proceed with the uninstall? (y/n): y
INFO[0000] Uninstalling UCP on each node...
INFO[0031] UCP has been removed from this cluster successfully.
INFO[0033] Removing UCP Services
</source>

== Install DTR Docker Trusted Repository <code>image: docker/dtr</code> ==
It's recommended for single core systems to wait 5 minutes after UCP deployment to relese more cpu cycle. You can see the load may peaking up at 1.0 using <code>w</code> command.

Connect to UCP service https://ucp.example.com, login with creds created. Uload a license.lic file.
Go to Admin Settings > Docker Trusted Registry > Pick one of UCP Nodes [worker]
You may disable TLS verification on self-signed certificate

Run a given command on a node you want to install DTR. <code>UCP_NODE</code> in lab environment can cause a few issues. For a convinience to avoid avoid port conflict :80,:443 use different node that UCP is instaled. Eg. dns ''user2c.mylabserver.com'' or private IP.
<source lang=bash>
export UCP_NODE=wkr-172.31.107.250 #for convinince, to avoid port conflict :80,:443 use worker IP
export UCP_USERNAME=ucp-admin
export UCP_PASSWORD=ucp-admin
export UCP_URL=https://ucp.example.com:443 #avoid using example.com to avoid SSL name validation issues
docker pull docker/dtr

# Optional. Download UCP public certificate
curl -k https://ucp.ciscolinux.co.uk/ca > ucp-ca.pem

docker container run -it --rm docker/dtr install \
--ucp-node $UCP_NODE --ucp-url $UCP_URL --debug \
--ucp-username $UCP_USERNAME --ucp-password $UCP_PASSWORD \
--ucp-insecure-tls # --ucp-ca "$(cat ucp-ca.pem)"

# --ucp-node :- hostname/IP of the UCP node (any node managed by UCP) to deploy DTR. Random by default
# --ucp-url :- the UCP URL including domain and port.
</source>

It will ask for if not specified:
* ucp-password: you know it from UCP installation step

Sygnificiant installation logs
<source lang=bash>
..
INFO[0006] Only one available UCP node detected. Picking UCP node 'user2c.labserver.com'
..
INFO[0006] verifying [80 443] ports on user2c.labserver.com
..
INFO[0000] Using default overlay subnet: 10.1.0.0/24
INFO[0000] Creating network: dtr-ol
INFO[0000] Connecting to network: dtr-ol
..
INFO[0008] Generated TLS certificate. dnsNames="[*.com *.*.com example.com *.dtr *.*.dtr]" domains="[*.com *.*.com 172.17.0.1 example.com *.dtr *.*.dtr]" ipAddresses="[172.17.0.1]"
..
INFO[0073] You can use flag '--existing-replica-id 10e168476b49' when joining other replicas to your Docker Trusted Registry Cluster
</source>

;Verify by logging in to https://dtr.example.com
DTR installation process above has also installed a number of containers on maanger/worker nodes named <code>ucp-agent</code> and number of containers on dedicated DTR node.
You can verify DTR by logging to https://dtr.example.com with UCP credentials <code>ucp-admin</code> and the same password if you haven't changed any commands above. Then you should be presented with registry.docker.io like theme. Any images stored there will be trusted from a perspective of our organisation.

;Verify by going to UCP https://ucp.example.com, admin settings > Docker Trusted Registry
[[File:Ucp-dtr-in-admin.png|none|400px|left|Ucp-dtr-in-admin]]

== Backup UCP and DTR configuration ==
This is build into UCP. The process is to start a special container to export UCP configuration to tar file. This can be run as <code>cron</code> job.

<source lang=bash>
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp backup > backup.tar
# --rm it's transitional container
# -i run interactivly

# At first run it will error with --id m79xxxxxxxxx, asking to re-run teh command with this id.

# Restore command
docker container run --log-driver non --rm -i --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp restore --id m79xxx < backup.tar
</source>

;DTR
Durign a backup DTR will not be available.
<source lang=bash>
docker container run --log-driver non --rm docker/dtr backup --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> > dtr-backup.tar

# will you be asked for:
# Choose a replica to back up from: enter

# Restore command
docker container run --log-driver non --rm docker/dtr restore --ucp-insecure-tls --ucp-url <ucp_server_dns:443> --ucp-username admin --ucp-password <password> < dtr-backup.tar
</source>
== UCP RBAC ==
The main concept is:
* administrators can make changes to the UCP swarm/kubernetes, User Management, Orgainisation, Team and Roles
* users - range of access from Full Control of resources to no access

[[File:Ucp-rbac.png|500px|none|left|Ucp-rbac]]

Note that only Scheduler role allows access to Node to view nodes. Plus schedule workloads of course.

= UCP Client bundle =
UCP client bundle allows to export a bundle containing a certificate and environment settings that will poind docker-client to UCP in order to use a cluster, create images and services.

;Download bundle
# Create a user with priviliges that yuo wish docker-client to run as
# Download a client budle from User Profile > Client bundle > + New Client Bundle
# File <code>ucp-bundle-[username].zip will get downloaded</code> <p>
<source lang=bash>
unzip ucp-bundle-bob.zip
Archive: ucp-bundle-bob.zip
extracting: ca.pem
extracting: cert.pem
extracting: key.pem
extracting: cert.pub
extracting: env.sh
extracting: env.ps1
extracting: env.cmd

cat env.sh
export COMPOSE_TLS_VERSION=TLSv1_2
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH="$PWD"
export DOCKER_HOST=tcp://3.16.143.49:443
#
# Bundle for user bob
# UCP Instance ID t0ltwwcw5tdbtjo2fxlzmj8p4
#
# This admin cert will also work directly against Swarm and the individual
# engine proxies for troubleshooting. After sourcing this env file, use
# "docker info" to discover the location of Swarm managers and engines.
# and use the --host option to override $DOCKER_HOST
#
# Run this command from within this directory to configure your shell:
# eval $(<env.sh)

eval $(<env.sh) # apply ucp-bundle

docker images # to list UCP managed images
</source></p>
# <li value="4"> In my lab I had to update DOCKER_HOST from public IP to private IP </li>
Err: error during connect: Get https://3.16.143.49:443/v1.39/images/json: x509: certificate is valid for 127.0.0.1, 172.31.101.248, 172.17.0.1, not 3.16.143.49
<source lang=bash>
export DOCKER_HOST=tcp://172.31.101.248:443
</source>

# <li value="5"> Verify if you have permissions to create a service</li>
<source lang=bash>
docker service create --name test111 httpd
Error response from daemon: access denied:
no access to Service Create, on collection swarm
</source>

# <li value="6"> Add Grants to the user</li>
## Go to User Management > Granst > Create Grant
## Base on a Roles, select Full Control
## Select Subjects, All Users, select the user
## Click Create
# Re-run service create command that should succeed now. This service can be managed now also within UCP console.

= Docker Secure Registry | image: registry =
Docker provides a special docker image that can be used to manage docker imagages both internally or externally thus steps below include securing the access with SSL certificate.

Create certificate
<source lang=bash>
mkdir ~/{auth,certs}
# create self-signed certificate for Docker Repository
mkdir certs; cd _$ #cd to last argument in history
openssl req -newkey rsa:4096 -nodes -sha256 -keyout repo-key.pem -x509 -days 365 -out repo-cer.pem -subj /CN=myrepo.com
# trusted-certs docker client directory, docker client looks for trusted certs when conencting to reomote repo
sudo mkdir -p /etc/docker/certs.d/myrepo.com:5000 #port 5000 is a default port
sudo cp repo-cer.pem /etc/docker/certs.d/myrepo.com:5000/ca.crt
</source>
<code>ca.crt</code> is default/required CAroot trustcert file name, that the docker client (docker login API) uses when conencting to remote repository. In our case we trust any cert signed by CA=ca.crt when connecting to myrepo.com:5000 as same certs (selfsigned), got installed in <code>repository:2</code> container via <code>-v /certs/</code> option.

Optional for development purposes to add doamin ''myrepo.com'' to hostfile binding to local interface ip address.
<source lang=bash>
sudo -i; echo "172.16.10.10 myrepo.com" >> /etc/hosts; exit
</source>

Optional add insecure-registry entry
<source>
sudo vi /etc/docker/deamon.json
{
"insecure-registries" : [ "myrepo.com:5000"]
}
</source>

Pull special Docker Registry image
<source lang=bash>
mkdir -p ~/auth #authentication directory, used when deploying local repository
docker pull registry:2
docker run --entrypoint htpasswd registry:2 -Bbn reg-admin Passw0rd123 > ~/auth/htpasswd
# -Bbn -parameters
# reg-admin -user
# Passw0rd123 -password string for basic htpasswd authentication method, the hashed password will be displayed to STDOUT

$ cat ~/auth/htpasswd
reg-admin:$2y$05$DnTWDHp7uTwaDrw4sXpUbuDDIlLwu3c8MEMsHPjK/AcUMdK/TD6fO
</source>

Run Registry container
<source lang=bash>
cd ~
docker run -d -p 5000:5000 --name myrepo \
-v $(pwd)/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/repo-cer.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/repo-key.pem \
-v $(pwd)/auth:/auth \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
registry:2
# -v -indicate where our certificates will be mounted within a container
# -e REGISTRY_HTTP_TLS_CERTIFICATE -path to cert inside the container
# -v $(pwd)/auth:/auth -mounting authentication directory where a file with password is
# -e REGISTRY_AUTH htpasswd -setting up to use 'htpasswd' authentication method
# registry:2 -image name, positinal params
</source>

Verify
<source lang=bash>
docker pull alpine
docker tag alpine myrepo.com:5000/aa-alpine #create a tagged image (copy) on a local filesystem,
# it must be prefixed with the private repo name '/' image name you want to upload as

docker logout # if logged in to another repository
docker login myrepo.com:5000/aa-alpine #login to a repository that runs as a container, stays login untill logout/reboot
docker login myrepo.com:5000/aa-alpine --username=rep-admin --password Passw0rd123
docker push myrepo.com:5000/aa-alpine

docker image rmi alpine myrepo.com:5000/aa-alpine #delete image stored locally
docker pull myrepo.com:5000/aa-alpine #pull image from a container repository

# List private-repository images
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/_catalog
{"repositories":["aa-alpine"]}

wget --no-check-certificate --http-user=reg-admin --http-password=password https://myrepo.com:5000/v2/_catalog
cat _catalog
{"repositories":["my-alpine","myalpine","new-aa-busybox"]}

# List tags
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/tags/list
{"name":"myalpine","tags":["latest"]}
curl --insecure -u "reg-admin:password" https://myrepo.com:5000/v2/aa-alpine/manifests/latest #entire image metadata
</source>

Note. There is no easy way to delete images from repository:2 container.

= Docker push =
;Login to a docker repository
<source lang=bash>
docker info | grep -B1 Registry #check if you are logged in to docker.hub repository
WARNING: No swap limit support
Registry: https://index.docker.io/v1/

docker login

docker info | grep -B1 Registry
Username: pio2pio
Registry: https://index.docker.io/v1/
</source>

; Tag and push an image
<source lang=bash>
# docker tag local-image:tagname new-repo:tagname #create a local copy of an image
# docker push new-repo:tagname

docker pull busybox
docker --tag busybox:latest pio2pio/testrepo
docker push pio2pio/testrepo
The push refers to repository [docker.io/pio2pio/testrepo]
683f499823be: Mounted from library/busybox
latest: digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510
size: 527
</source>

; Docker Content Trust
All images are implicitly trusted by your Docker daemon. Buy can set that ONLY signed images are allowed. You can configure your systems for trusting only image tags that have been signed.
<source lang=bash>
export=DOCKER_CONTENT_TRUST=1 #enable system to sign an image during push process
docker build -t myrepo.com:5000/untrusted.latest
docker push myrepo.com:5000/untrusted.latest
...
No tag specified, skipping trust metadata push
# 2nd attempt, with a tag specified now
docker push myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority

docker pull myrepo.com:5000/untrusted.latest:latest
Error: error contacting notary server: x509: certificate signed by unknown authority
</source>

;Errors explained:
Err: No tag specified, skipping trust metadata push<br />
* Explenation: When image gets signed is signed by a tag. Thereofre if you skip a tag it won't get signed and metada get skipped.
Err: error contacting notary server: x509: certificate signed by unknown authority
* when uploading the image gets uploaded, but it is not trusted becasue signed with self-signed CA
* when downloading, and <code>DOCKER_CONTENT_TRUST=1</code> is enabled, the image cannot be downloaded because is untrusted

= Theory =
== What is a docker ==
Docker is a container runtime platform, where Swarm is a container orchestration platform.

== Security ==
=== Mutually Authenticated TLS ===
Docker Swarm is ''secure by default'', it means all communication is encrypted. ''Mutually Authenticated TLS'' is the implementation was chosen to secure that communication. Any time a swarm is inicialised, a self-signed CA is generated and issues certificates to every node (mgr or wkr) to facilicate registration (join mgr or wkr) and latter those secure communications. It's transient container brought up to generate CA certs every time a cert is needed. MTLS communication is between Managers and Workers.

== [[Linux Namespaces and Control Groups]] ==

== Difference between docker attach and docker exec ==
;Attach
The docker attach command allows you to attach to a running container using the containers ID or name, either to view its ongoing output or to control it interactively. You can attach to the same contained process multiple times simultaneously, screen sharing style, or quickly view the progress of your detached process.

The command docker attach is for attaching to the existing process. So when you exit, you exit the existing process.

If we use docker attach, we can use only one instance of shell. So if we want open new terminal with new instance of container's shell, we just need run docker exec

If the docker container was started using /bin/bash command, you can access it using attach, if not then you need to execute the command to create a bash instance inside the container using exec. Attach isn't for running an extra thing in a container, it's for attaching to the running process.

To stop a container, use CTRL-c. This key sequence sends SIGKILL to the container. If --sig-proxy is true (the default),CTRL-c sends a SIGINT to the container. You can detach from a container and leave it running using the CTRL-p CTRL-q key sequence.

;exec

"docker exec" is specifically for running new things in a already started container, be it a shell or some other process. The docker exec command runs a new command in a running container.

The command started using docker exec only runs while the containerâ€™s primary process (PID 1) is running, and it is not restarted if the container is restarted.

exec command works only on already running container. If the container is currently stopped, you need to first run it. So now you can run any command in running container just knowing its ID (or name):

<source>
docker exec <container_id_or_name> echo "Hello from container!"
docker run -it -d shykes/pybuilder /bin/bash
</source>

The most important here is the -d option, which stands for detached. It means that the command you initially provided to the container (/bin/bash) will be run in background and the container will not stop immediately.

= Dockerfile - python =
* [https://luis-sena.medium.com/creating-the-perfect-python-dockerfile-51bdec41f1c8 perfect python dockerfile] Medium

= References =
*[https://docs.docker.com/v1.8/installation/ubuntulinux/ Ubuntu installation] official website
*[https://docs.docker.com/engine/admin/systemd/ PROXY settings for systemd]
*[http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/ Docker RUN vs CMD vs ENTRYPOINT]
*[https://vsupalov.com/docker-arg-vs-env/ docker ARG vs ENV]
*[https://www.fromlatest.io/#/ Docker online lintel]
*[https://hub.docker.com/r/portainer/portainer/ portainer] Monitor your containers via Web GUI
*[https://treescale.com/ treescale.com] Free private Docker registry

HashiCorp/Vagrant

2024-05-29T07:54:25Z

Pio2pio: /* Install | Changelog */

= Introduction =
Vagrant is configured on a per project basis. Each of these projects has its own Vagran file. The Vagrant file is a text file in which vagrant reads that sets up our environment. There is a description of what OS, how much RAM, and what software to be installed etc. You can version control this file.

= Install | [https://github.com/hashicorp/vagrant/blob/v2.2.10/CHANGELOG.md Changelog] =
Download or upgrade
<source lang=bash>
# Install using Ubuntu package manager (2024)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt-cache policy vagrant
sudo apt update && sudo apt install vagrant

# Install downloading a package from sources (2022)
LATEST=$(curl -s GET https://api.github.com/repos/hashicorp/vagrant/tags | jq -r '.[].name' | head -n1 | tr -d v); echo $LATEST
VERSION=${LATEST:=2.2.18};
wget https://releases.hashicorp.com/vagrant/${VERSION}/vagrant_${VERSION}_linux_amd64.zip
sudo install /usr/bin/vagrant
#sudo dpkg -i vagrant_${VERSION}_x86_64.deb
#sudo apt-get update && sudo apt-get install -f # resolve missing dependencies

# Fix plugins if needed
vagrant plugin update
vagrant plugin repair
vagrant plugin expunge --reinstall
</source>

Install ruby is recommended as configuration within '''Vagrant''' file is written in Ruby language.
<source lang=bash>
sudo apt-get install ruby
sudo gem install bundler
sudo gem update bundler # if update needed
</source>

Repair plugins after the upgrade
<source lang=bash>
vagrant plugin repair # use first
vagrant plugin expunge --reinstall
vagrant plugin update # then update broken plugin
</source>

= Images aka <code>box</code> management =
Vagrant comes with a preconfigured image repositories.
;Manage boxes
<source lang=bash>
vagrant box [list | add | remove] [--help]
</source>

;Add a box (image) into local repository
These are standard VMs from providers in Virtualbox, VMware or Hyper-V format taken from a given repository
<source lang=bash>
vagrant box add hashicorp/precise64 #user: hashicorp boximage: precise64, this is preconfigured repository
vagrant box add ubuntu/xenial64
vagrant box add ubuntu/xenial64 --box-version 20170618.0.0 --provider virtualbox
vagrant box add bento/ubuntu-18.04 --box-version 201812.27.0 --provider hyperv

# Box can be specified via URLs or local file paths, Virtualbox can only nest 32bit machines
vagrant box add --force ubuntu/14.04 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
vagrant box add --force ubuntu/14.04-i386 https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box
</source>

Windows images
* devopsgroup-io/windows_server-2012r2-standard-amd64-nocm
* peru/windows-server-2016-standard-x64-eval
* scotch/box

;Update a box to the latest version
<source lang=bash>
$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)

$> vagrant box update --box ubuntu/bionic64
Checking for updates to 'ubuntu/bionic64'
Latest installed version: 20190718.0.0
Version constraints: > 20190718.0.0
Provider: virtualbox
Updating 'ubuntu/bionic64' with provider 'virtualbox' from version
'20190718.0.0' to '20200124.0.0'...
Loading metadata for box 'https://vagrantcloud.com/ubuntu/bionic64'
Adding box 'ubuntu/bionic64' (v20200124.0.0) for provider: virtualbox
Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200124.0.0/providers/virtualbox.box
Download redirected to host: cloud-images.ubuntu.com

$> vagrant box list
ubuntu/bionic64 (virtualbox, 20190411.0.0)
ubuntu/bionic64 (virtualbox, 20190718.0.0)
ubuntu/bionic64 (virtualbox, 20200124.0.0) # <- new downloaded
</source>

;Delete all images (aka boxes)
<source lang=bash>
vagrant box prune
</source>

= vagrant init - your first project =
;Configure Vagrantfile to use the box as your base system
<source lang="ruby">
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.hostname = "ubuntu" #hostname, requires reload
end
</source>

;Create Vagrant project, by creating ''Vagrantfile'' in your current directory
<source lang="bash">
vagrant init #initialises an project
vagrant init ubuntu/xenial64 # initialises official Ubuntu 16.04 LTS (Xenial Xerus) Daily Build
vagrant init ubuntu/bionic64 #supports only VirtualBox provider
vagrant init bento/ubuntu-18.04 #supports variety of providers

#Windows
vagrant init devopsgroup-io/windows_server-2012r2-standard-amd64-nocm #Windows 2012r2, VirtualBox only; cannot ssh
vagrant init peru/windows-server-2016-standard-x64-eval #Windows 2016, halt works
vagrant init gusztavvargadr/windows-server #Windows 2019, full integration
</source>

;Power up your Vagrant box
<source lang="bash">
vagrant up
</source>

;Ssh to the box. Below example of nested virtualisation 64bit VM(host) runs 32bit (guest vm)
<source lang="bash">
piotr@vm-ubuntu64:~/git/vagrant$ vagrant ssh #default password is "vagrant"
vagrant@vagrant-ubuntu-precise-32:~$ w
13:08:35 up 15 min, 1 user, load average: 0.06, 0.31, 0.54
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 10.0.2.2 13:02 1.00s 4.63s 0.09s w
</source>

;Shared directory between Vagrant VM and an hypervisor provider
Vagrant VM shares a directory mounted at <tt>/vagrant</tt> with the directory on the host containing your Vagrantfile. This can be manually mounted from within VM as long the shared directory is setup in GUI.

Eg. vm_name > Settings > Shared Folders > Name: vagrant | Path: /home/piotr/vm_name
<source>
sudo mount -t vboxsf -o uid=1000 vagrant /vagrant #firts arg 'vagrant' refers to GUI setiing
</source>

= Troubleshooting =
<source>
vagrant --debug up
</source>
== Nesting VMs ==
The error below is due to Virtualbox cannot run nested 64bit virtualbox VM. Spinning up a 64bit VM stops with an error that no 64bit CPU could be found. Update [https://forums.virtualbox.org/viewtopic.php?f=1&t=90831 VirtualBox 6.x Nested virtualization, VT-x/AMD-V in the guest].
<pre>
Error:
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
</pre>

= Manage power states =
*<code>vagrant suspend</code> - saves the current running state of the machine and stop it
*<code>vagrant halt</code> - gracefully shuts down the guest operating system and power down the guest machine
*<code>vagrant destroy</code> - removes all traces of the guest machine from your system. It'll stop the guest machine, power it down, and remove all of the guest hard disks

= Snapshots =
You can easily save snapshots.
<source lang=bash>
# Get status
$ vagrant status
Current machine states:
default poweroff (virtualbox) # <- 'default' it's machine name
# in multi-vm Vagrant config file
The VM is powered off. To restart the VM, simply run `vagrant up`

# List
vagrant snapshot list
==> default:
11_b4-upgradeVbox-stopped
12_Dec01_stopped

# Save
<nameOfvm> <snapshot-name>
vagrant snapshot save default 13_Dec30_external-eks_stopped

# Restore
vagrant snapshot restore default 13_Dec30_external-eks_stopped
</source>

= Lookup path precedence for Vagrant project file =
When you run any vagrant command, Vagrant climbs your directory tree starting first in the current directory you are in. Example:
/home/peter/projects/la/Vagrant
/home/peter/projects/Vagrant
/home/peter/Vagrant
/home/Vagrant
/Vagrant

= Configuration =
== Networking ==
'''Private''' network is network that is not accessible from Internet. Networking stanza is a part of the main <tt>|config|</tt> loop.

DHCP IP address assigned
config.vm.network "private_network", type: "dhcp"

Static IP assigment
config.vm.network "private_network", ip: "192.168.80.5"
auto_config: false #optional to disable auto-configure

'''Public network'''
These networks can be accessible from outside of the host machine including Internet, are usually '''Bridged Networks'''.

Examples of dhcp and static IP assignment
config.vm.network "public_network", type: "dhcp"
config.vm.network "public_network", ip: "192.168.80.5"

Default interface. The name need to match your system name otherwise Vagrant will prompt you to choose from available interfaces during ''vagrant up'' process.
config.vm.network "public_network", bridge: 'eth1'

== Port forwarding ==
Vagrant can forward any host(hypervisor) TCP port to guest vm specyfing in ~/git/vargant/Vagrant file
config.vm.network :forwarded_port, guest: 80, host: 4567
Reload virtual machine <code>vagrant reload</code> and run from hypervisor web browser http://127.0.0.1:4567 to test it.

== Sync folders ==
Vagrant v2 renamed ''Shared folders'' into '''Sync folders'''. This feature mounts HostOS directory into GuestOS. allowing workflow of editing files with IDE installed on a host machine but access them within GuestOS. The files sync both directions (as mount on GuestOS), Remember, taking <code>vagrant snapshot save ubuntu-snap1</code> '''will NOT save''' the '''Sync folder''' content as it's just mounted directory.

When configuring, 1st argument is a path existing on a '''host machine'''. If relative then it's relative to the root-project folder (where Vagrantfile exists) and 2nd arg is a full path to the mounted dir on guest-os.

;Enabling Sync folders and Symlinks
This can be done at any time, it's applied during <code>vagrant up | reload</code>. In general symlinks are disabled by VirtualBox as insecure.
<source lang=ruby>
Vagrant.configure("2") do |config|
# path on the host mount on the guestOS
\ /
config.vm.sync_folder "git-host/", "/git", disabled: false
end
config.vm.provider "virtualbox" do |vb|
vb.name = File.basename(Dir.pwd) + "_vagrant"
...
vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//git", "1"]
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant", "1"]

# symlinks should be active in root of vm by default
# vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
</source>

Disabling
<source lang=ruby>
Vagrant.configure("2") do |config|
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true
end
</source>

Modifying the Owner/Group
<source lang=ruby>
config.vm.sync_folder "../data/", "/vagrant-data", disabled: true,
owner: "root", group: "root"
</source>

References
* [https://www.vagrantup.com/docs/synced-folders/basic_usage.html#id synced-folders] Hashicorp docs

= Vagrant providers =
Vagrant can work with a wide variety of backend providers, such as VMware, AWS, and more without changing Vagrantfile. It's enough to specify the provider and Vagrant will do the rest:
<source>
vagrant up --provider=vmware_fusion
vagrant up --provider=aws
</source>
== Hyper-V ==
*Enable Hyper-V
*if you running Docker for Windows make sure is disabled as only one application can bound to Internal NAT vswitch, if you are using it
*WSL and Windows Vagrant versions must match
*the terminal you run WSL or PowerShell runs with elevated privileges
When running in WSL, make sure you have <code>export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"</code>,
*you are in native Bash.exe not eg. ConEmu terminal with as it was proven not working at the time. You can change default provider by <code>export VAGRANT_DEFAULT_PROVIDER=hyperv</code>

Optional: Set the user-level environment variable in PowerShell:
<source>
[Environment]::SetEnvironmentVariable("VAGRANT_DEFAULT_PROVIDER", "hyperv", "User")
</source>

;Workarounds
Copy insecure private key from <code>https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant to WSL ~/.vagr
ant_key/private_key</code> because Microsoft filesystem does not support Unix style file permissions, until WSL2 is released.
<source>
$ wget https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant -O ~/.vagrant_key/private_key
# then set in Vagrantfile
config.ssh.private_key_path = "~/.vagrant_key/private_key"
</source>

When running on HyperV you need to choose a vswitch you will use. Vagrant will prompt you, select "Default Switch", w
hich is eqvivalent of NAT Network. You need to creat your own vswitch if you want access to Internet.

Go to Hyper-V Manager, open Virtual Switch Manager..., create External switch, name: vagrant-external, press OK. Then
run.

<source>
vagrant up --provider hyperv

default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) DockerNAT
default: 2) Default Switch
default: 3) vagrant-external
default:
default: What switch would you like to use?3 #<-- select 3
</source>
Read more https://www.vagrantup.com/docs/hyperv/limitations.html

Run Vagrant file
<source lang=bash>
vagrant up --provider=hyperv
</source>

=== References ===
*[https://gist.github.com/savishy/8ed40cd8692e295d64f45e299c2b83c9 Create vSwitch in Hyper-V to run Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Copying-Files-into-a-Hyper-V-VM-with-Vagrant/ba-p/382376 Copying Files into a Hyper-V VM with Vagrant]
*[https://techcommunity.microsoft.com/t5/Virtualization/Vagrant-and-Hyper-V-Tips-and-Tricks/ba-p/382373 Vagrant and Hyper-V -- Tips and Tricks] techcommunity.microsoft.com

= Provisioners =
==Shell provisioner==
Vagrant can run from shared location script or from inline: Vagrant file shell provisioning commands.

Create provisioning script
<source lang=bash>
vi ~/git/vagrant/bootstrap.sh
#!/usr/bin/env bash
export http_proxy=<nowiki>http://username:password@proxyserver.local:8080</nowiki>
export https_proxy=$http_proxy
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
rm -rf /var/www
ln -sf /vagrant /var/www # sets Vagrant shared dir to Apache DocumentRoot
fi
</source>

Configure Vagrant to run this shell script above when setting up our machine
<source lang=bash>
vi ~/git/vagrant/Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = ubuntu/14.04-i386
config.vm.provision: shell, path: "bootstrap.sh"
end
</source>

Another example of using shell provisioner, separating a script out
<source lang=bash>
$script = <<SCRIPT
echo " touch /home/vagrant/test_\\`date +%s\\`.txt " > /home/vagrant/newfile
chmod +x /home/vagrant/newfile
echo "* * * * * /home/vagrant/newfile" > mycron
crontab mycron
SCRIPT

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision "shell", inline: $script , privileged: false
end
</source>

Bring the environment up
<source lang=bash>
vagrant up #runs provisioning only once
vagrant reload --provision #reloads VM skipping import and runs provisioning
vagrant ssh #ssh to VM
wget -qO- 127.0.0.1 #test Apache is running on VM
</source>

;Provisioners - shell, ansible, ansible_local and more

This section is about using Ansible with Vagrant,
*<code>ansible</code>, where Ansible is executed on the '''Vagrant host'''
*<code>ansible_local</code>, where Ansible is executed on the '''Vagrant guest'''

==Ansible provisioner==

Specify Ansible as a provisioner in Vagrant file
<source lang=ruby>
# Run Ansible from the Vagrant Host
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
</source>

== Chef_solo provisioner ==
Create recipe, the following dirctory structure is required, eg. recipe name is: vagrant_la
├── cookbooks
│   └── vagrant_la
│   └── recipes
│   └── default.rb
Vagrant

Recipe
<source lang=ruby>
vi cookbooks/vagrant_la/recipes/default.rb
execute "apt-get update"
package "apache2"
execute "rm -rf /var/www"
link "var/www" do
to "/vagrant"
end
</source>

In Vagrant file add following
<source lang=ruby>
config.vm.provision "chef_solo" do |chef|
chef.add_recipe "vagrant_la"
end
</source>

Run <code>vagrant up</code>

== Puppet manifest ==
Create Vagrant provisioning stanza
<source lang=ruby>
config.vm.define "web" do |web|
web.vm.hostname = "web"
web.vm.box = "apache"
web.vm.network "private_network", type: "dhcp"
web.vm.network "forwarded_port", guest: 80, host: 8080
web.vm.provision "puppet" do |puppet|
puppet.manifests_path = "manifests"
puppet.manifest_file = "default.pp"
end
end
</source>

Create a required folder structure for puppet manifests
├── manifests
│   └── default.pp
└── Vagrantfile

Puppet manifest file
vi manifests/default.pp
exec { "apt-get update":
command => "/usr/bin/apt-get update",
}
package { "apache2":
require => Exec["apt-get update"],
}
file { "/var/www":
ensure => link,
target => "/vagrant",
force => true,
}

= Box images advanced=
vagrant box list #list all downloaded boxes

Default path of boxes image, it can be specified by environment variable <tt>VAGRANT_HOME</tt>
C:\Users\%username%\.vagrant.d\boxes #Windows
~/.vagrant.d/boxes #Linux

Change default path via environment variable
export VAGRANT_HOME=my/new/path/goes/here/

==Box format==
When you un-tar the <tt>.box</tt> file it contains 4 files:
|--Vagrantfile
|--box-disk1.vmdk #compressed virtual disk
|--box.ovf #description of virtual hardware
|--metadata.json

== [https://www.vagrantup.com/docs/virtualbox/boxes.html Create box] from current project (package a box) ==
This allows to create a reusable box that contains all changes to the software we made, VirtualBox or Hyper-V supported only.

[https://www.vagrantup.com/docs/cli/package.html Command basics]
<source lang=bash>
vagrant package [options] [name|id]
# --base NAME - instead of packaging a VirtualBox machine that Vagrant manages,
# this will package a VirtualBox machine that VirtualBox manages
# --output NAME - default is package.box
# --include x,y,z - additional files will be packaged with the box
</source>

Package
<source lang=bash>
$ vagrant version # -> Installed Version: 2.2.9

# Optional '--vagrantfile NAME' can be included, that automatically restores '--include' files
# learn more at https://www.vagrantup.com/docs/vagrantfile#load-order
$ time vagrant package --output u18cli-1.box --include data,git-host,git-host3rd,sync.sh,cleanup.sh
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /home/piotr/vms-vagrant/u18cli-1/2020-05-23-u18cli-1.box
==> default: Packaging additional file: data # <- dir
==> default: Packaging additional file: git-host # <- dir
==> default: Packaging additional file: git-host3rd # <- dir
==> default: Packaging additional file: cleanup.sh # <- file
real 15m27.324s user 8m23.550s sys 0m16.827s
</source>

Re-ristribute the <tt>.box</tt> file then restore it.
<source lang=bash>
# Add the packaged box to local system box repository
# _____box-name________ __box-file_____
$ vagrant box add --name box-packages/u18cli-1 u18cli-1-v1.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'u18cli-1-v1.box' (v0) for provider:
box: Unpacking necessary files from: file:///home/piotr/vms-vagrant/test-box-restore/u18cli-1-v1.box
==> box: Successfully added box 'box-packages/u18cli-1' (v0) for 'virtualbox'!

# List boxes
$ vagrant box list
box-packages/u18cli-1 (virtualbox, 0)

$ ls -l ~/.vagrant.d/boxes
total 16
drwxrwxr-x 3 piotr piotr 4096 Jul 16 17:44 box-packages-VAGRANTSLASH-u18cli-1
</source>

Restore. Create/re-use Vagrantfile using box you added to your local box repository
<source lang=bash>
# vi Vagrantfile
config.vm.box = "box-packages/u18cli-1.box"

vagrant up
# restore '--include' files coping them from
# 'ls -l ~/.vagrant.d/boxes/box-packages-VAGRANTSLASH-u18cli-1/0/virtualbox/include/*'
</source>

= [https://tuhrig.de/resizing-vagrant-box-disk-space/ Resizing Vagrant box disk] =
* [https://www.vagrantup.com/docs/disks/usage Resizing primary disk] native way

= Enable Vagrant to use proxy server for VMs =
Install proxyconf plugin or use <code>vagrant plugin list</code> to verify if installed
vagrant plugin install vagrant-proxyconf

Configure your Vagrantfile, here particularly host 10.0.0.1:3128 runs CNTLM proxy
Vagrant.configure("2") do |config|
<nowiki>config.proxy.http = "http://10.0.0.1:3128"
config.proxy.https = "http://10.0.0.1:3128"</nowiki>
config.proxy.no_proxy = "localhost,127.0.0.1"

= Virtualbox Guest Additions =
== Sync using vagrant-vbguest plugin ==
Install plugin
<source lang=bash>
vagrant gem install vagrant-vbguest #for vagrant < 1.1.5
vagrant plugin install vagrant-vbguest #for vagrant 1.1.5+

#Verify current version, running on a host(hypervisor)
vagrant vbguest --status

#Add to your Vagrant file (Vagrant 1.1.5+)
if Vagrant.has_plugin?("vagrant-vbguest")
host.vbguest.auto_update = true
host.vbguest.no_remote = true
end
</source>

Then install the correct version matching your host installation
<source lang=bash>
vagrant vbguest --do install

#Full command options
vagrant vbguest [vm-name] [--do start|rebuild|install] [--status] [-f|--force] \
[-b|--auto-reboot] [-R|--no-remote] [--iso VBoxGuestAdditions.iso] [--no-cleanup]
</source>
More you will find at [https://github.com/dotless-de/vagrant-vbguest vagrant-vbguest] plugin project.

== Manual upgrade ==
Find out what version you are running, execute on a guest VM
<source lang=bash>
vagrant@ubuntu:~$ modinfo vboxguest | grep ^version
version: 6.0.10 r132072

vagrant@ubuntu:~$ lsmod | grep -io vboxguest | xargs modinfo | grep -iw version
version: 6.0.10 r132072

vagrant@u18cli-3:~$ sudo /usr/sbin/VBoxService --version
6.0.10r132072
</source>

Download the extension, you can explore [http://download.virtualbox.org/virtualbox here]
<source lang=bash>
wget http://download.virtualbox.org/virtualbox/5.0.32/VBoxGuestAdditions_5.0.32.iso
#you need to get it mounted or extracted the content and run inside the VM.
</source>

== References ==
*[https://github.com/chilcano/box-vagrant-wso2-dev-srv/blob/master/_downloads/vagrant-vboxguestadditions-workaroud.md Upgrade Vbox extension additions within Vagrant box]

= List all Virtualbox SSH redirections =
<source>
vboxmanage list vms | cut -d ' ' -f 2 > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms | cut -d ' ' -f 1 | sed 's/"//g' > /tmp/vms.out && for vm in $(cat /tmp/vms.out); do echo $vm; vboxmanage showvminfo "$vm" | grep ssh; done
vboxmanage list vms \
| cut -d ' ' -f 1 \
| sed 's/"//g' > /tmp/vms.out \
&& for vm in $(cat /tmp/vms.out); do vboxmanage showvminfo "$vm" \
| grep ssh \
| tr --delete '\n'; echo " $vm"; done

sed 's/"//g' #removes double quotes from whole string
tr --delete '\n' #deletes EOL, so the next command output is appended to the previous line
</source>

= Vagrant file =
;Ruby gotchas
Vagrant configuration file is written in Ruby specification, therefore you need to remember
*don't use dashes in object names, '''don't''': <tt>jenkins-minion_config.vm.box = "ubuntu/xenial64"</tt>
*don't use symbols (here underscore) in variable names, '''don't''': <tt>(1..2).each do |minion_number|</tt>

== HAProxy cluster, multi-node Vagrant config ==
<source lang="bash">
git clone https://github.com/jweissig/episode-45
</source>

This creates ''Ansible'' mgmt server, Load Balancer and Web nodes [1..2]. HAProxy will be configured via Ansible code.
<source lang="bash">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
# create mgmt node
config.vm.define :mgmt do |mgmt_config|
mgmt_config.vm.box = "ubuntu/trusty64"
mgmt_config.vm.hostname = "mgmt"
mgmt_config.vm.network :private_network, ip: "10.0.15.10"
mgmt_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"
end

# create load balancer
config.vm.define :lb do |lb_config|
lb_config.vm.box = "ubuntu/trusty64"
lb_config.vm.hostname = "lb"
lb_config.vm.network :private_network, ip: "10.0.15.11"
lb_config.vm.network "forwarded_port", guest: 80, host: 8080
lb_config.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end

# create some web servers
# https://docs.vagrantup.com/v2/vagrantfile/tips.html
(1..2).each do |i|
config.vm.define "web#{i}" do |node|
node.vm.box = "ubuntu/trusty64"
node.vm.hostname = "web#{i}"
node.vm.network :private_network, ip: "10.0.15.2#{i}"
node.vm.network "forwarded_port", guest: 80, host: "808#{i}"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
end
end
end
end
</source>

Boot strap script <tt>bootstrap-mgmt.sh</tt>
<syntaxhighlight lang="shell">
#!/usr/bin/env bash
# install ansible (http://docs.ansible.com/intro_installation.html)
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)
cp -a /vagrant/examples/* /home/vagrant
chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile
cat >> /etc/hosts <<EOL
# vagrant environment nodes
10.0.15.10 mgmt
10.0.15.11 lb
10.0.15.21 web1
10.0.15.22 web2
10.0.15.23 web3
10.0.15.24 web4
10.0.15.25 web5
10.0.15.26 web6
10.0.15.27 web7
10.0.15.28 web8
10.0.15.29 web9
EOL
</syntaxhighlight>

Gitbash path - <code>/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe</code>

Set bootstrap script for Proxy or No-proxy specific system
<source lang="bash">
Vagrant status
Vagrant up
Vagrant ssh mgmt
ansible all --list-hosts
ssh-keyscan web1 web2 lb > ~/.ssh/known_hosts
ansible-playbook ssh-addkey.yml -u vagrant --ask-pass
ansible-playbook site.yml
</source>

Once set it up you can navigate on your laptop to:
<source lang="bash">
http://localhost:8080/ #Website test
http://localhost:8080/haproxy?stats #HAProxy stats
</source>

Use to verify end server
<source lang="bash">
curl -I http://localhost:8080
</source>

[[File:X-Backend-Server.png|none|left|Curl -i X-Backend-Server]]

Generate web traffic
<source lang="bash">
vagrant ssh lb
sudo apt-get install apache2-utils
ansible localhost -m apt -a "pkg=apache2-utils state=present" --become
ab -n 1000 -c 1 http://10.0.2.15:80/ # Apache replaced 'ab' with 'hey'
</source>

= Vagrant DNS =
== Multi-machine mDNS discovery ==
Multi-machine setup requires 3 ingredients* :
* each machine to have different hostname
* a way of getting the IP address for a hostname (eg. mDNS)
* connect the VMs through a private network

In multi-machine configuration we need a way of getting the IP address for a hostname. We use <code>mDNS</code> for this. By default <codemDNS</code> only resolves host names ending with the <code>.local</code> top-level domain (TLD). This can cause problems if that domain includes hosts which do not implement mDNS but which can be found via a conventional unicast DNS server. Resolving such conflicts requires network-configuration changes that violate the zero-configuration goal. Install <code>avahi</code> system on all machines to facilitate service discovery on a local network via the <code>mDNS/DNS-SD</code> protocol suite.
<source lang=bash>
config.vm.provision "shell", inline: <<-SCRIPT
apt-get install -y avahi-daemon libnss-mdns
SCRIPT
</source>

;References
*[https://github.com/lathiat/nss-mdns nss-mdns] system which allows hostname lookup of *.local hostnames via mDNS in all system programs using nsswitch
*[https://www.avahi.org/ avahi.org]

== Set host system DNS server resolver ==
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
</source>

= Ubuntu with GUI =
This article is going to describe to setup Vagrant Virtualbox VM with GUI, setting up xserver with xfce4 as desktop environment.
== Locales ==
This is not working
<source lang=bash>
locale-gen en_GB.utf8 #en_GB.UTF-8
update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
dpkg-reconfigure --frontend=noninteractive locales
dpkg-reconfigure --frontend=noninteractive keyboard-configuration
localedef -i en_GB -c -f UTF-8 en_GB.utf8
sudo update-locale LANG=en_GB.UTF-8
locale-gen --purge "en_GB.UTF-8"
</source>

Troubleshooting
<source lang=bash>
locale -a #shows which locales are available on your system
sudo less /usr/share/i18n/SUPPORTED
cat /etc/default/locale

#Set system wide locales (does not work for users)
localectl set-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en
localectl set-keymap gb
localectl set-x11-keymap gb

#Quick kb change
apt-get install -yq x11-xkb-utils; setxkbmap gb
</source>

== Gnome3 ==
This setup installs Ubuntu desktop and may require a restart to apply changes to like a taskbar with shortcuts.
<source lang="ruby">
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64" #bento/ubuntu-18.04, ubuntu/xenial64

machineName = File.basename(Dir.pwd) #name as a current working dir
# machineName = 'u18gui-1'
config.vm.hostname = machineName

# Manually check for updates `vagrant box outdated`
config.vm.box_check_update = false

# Vbguest plugin
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Public network, which generally matched to bridged network.
# config.vm.network "public_network"

# config.vm.synced_folder "hostDir", "/InVagrantMount/path"
# config.vm.synced_folder "../data", "/vagrant_data"

config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "3072"
vb.name = machineName + "_vagrant"
end

config.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
setxkbmap gb
apt-get update && apt-get upgrade -yq
apt-get install -yq ubuntu-desktop --no-install-recommends
apt-get install -yq terminator tmux
#only U16 xenial to fix Unity
#apt-get install -yq unity-lens-files unity-lens-applications indicator-session --no-install-recommends
SHELL
end
</source>

Running up
<source lang="bash">
vagrant plugin install vagrant-vbguest
vagrant up && vagrant vbguest --do install && vagrant reload
</source>

== Xface ==
Get a basic Ubuntu image working, boot it up and vagrant ssh.
Next, enable the VirtualBox display, which is off by default. Halt the VM and uncomment these lines in Vagrantfile:
<source lang=ruby>
config.vm.provider :virtualbox do |vb|
vb.gui = true
end
</source>

Boot the VM and observe the new display window. Now you just need to install and start xfce4. Use vagrant ssh and:
<source lang="bash">
sudo apt-get install -y xfce4 virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11
#guest additions are already installed on most of the Vagrant boxes
</source>

Don't start the GUI as root because you really want to stay the vagrant user. To do this you need to permit anyone to start the GUI:
<source lang="bash">
sudo vim /etc/X11/Xwrapper.config and edit it to allowed_users=anybody
sudo startxfce4&
sudo VBoxClient-all #optional
</source>

You should be landed in a xfce4 session.

(Optional) If VBoxClient-all script isn't installed or anything is missing, you can replace with the equivalent:
<source lang="bash">
sudo VBoxClient --clipboard
sudo VBoxClient --draganddrop
sudo VBoxClient --display
sudo VBoxClient --checkhostversion
sudo VBoxClient --seamless
</source>

== References ==
*[http://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment Vagrant GUI vms] stackoverflow

= Windows=
<source lang=ruby>
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "gusztavvargadr/windows-server"
config.vm.box_check_update = false
config.vm.provider "virtualbox" do |vb|
vb.gui = true # Display the VirtualBox GUI when booting the machine
vb.memory = "3072" # Customize the amount of memory on the VM:
end
# Plugins
config.vbguest.auto_update = false
config.vbguest.no_remote = true
end
</source>

Shared location
* enable Network Sharing
* Vagrant path is mapped to <code>\\VBOXSVR\vagrant</code>

= WIP DevOps workstation =
This to contain:
*bashrc with git branch in ps1
*bash autocomplete (...samename)
*bash colored symlinks
*bash_logout and .profile to eval ssh-agent and kill on exit
*git install
*ansible 1.9.4
*java Oracle
*clone tfenv and install terraform
*vim install
*vundle install
*[done] python 2.7 OOB in 16.04
*[done]python pip: awscli, boto, boto3, etc..

Challenges:
*Ubuntu 16.04 official box does not come with a default ''vagrant'' user but instead comes with ''ubuntu'' user. This causes a number of incompatibilities.
**Read more at launchpad [https://bugs.launchpad.net/cloud-images/+bug/1569237 vagrant xenial box is not provided with vagrant/vagrant username and password ]
* Solutions
** on W10 host both users: ubuntu & vagrant exist. Only vagrant has insecure_pub installed OOB. I am coping vagrant user pub key into ubuntu user authorized_keys
** on U16.05 host the official image does not seem to come with vagrant user but Ubuntu user works OOB
** Read more at SO
***[Vagrant's Ubuntu 16.04 vagrantfile default password https://stackoverflow.com/questions/41337802/vagrants-ubuntu-16-04-vagrantfile-default-password]
***[https://stackoverflow.com/questions/30075461/how-do-i-add-my-own-public-key-to-vagrant-vm How do I add my own public key to Vagrant VM?]
*** [https://blog.ouseful.info/2015/07/27/running-a-shell-script-once-only-in-vagrant/ Running a Shell Script Once Only in vagrant]

= References =
*[https://www.vagrantup.com/docs/getting-started/ Vagrant Start up documentation]
*[https://atlas.hashicorp.com/boxes/search Vagrant Hashicorp VMs repository] Virtualbox
*[https://cloud-images.ubuntu.com/vagrant/ Vagrant Ubuntu VMs images] Virtualbox
*[https://www.vagrantup.com/docs/provisioning/ansible_intro.html Vagrant and Ansible provisioner] Vagrant docs
*[https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-naive-way Vagrant Tutorial – From Nothing To Multi-Machine] Tutorial