Json queries with JMEPath and jq - Revision history

Pio2pio: Pio2pio moved page JMEPath - JSON Queries to Json queries with JMEPath and jq without leaving a redirect

2020-10-18T15:48:16Z

Pio2pio moved page JMEPath - JSON Queries to Json queries with JMEPath and jq without leaving a redirect

← Older revision	Revision as of 17:48, 18 October 2020
(No difference)

Pio2pio: /* References */

2020-10-18T15:47:04Z

References

← Older revision		Revision as of 17:47, 18 October 2020
Line 283:		Line 283:
	*[https://stedolan.github.io/jq/manual/ Jq manual] Include functions		*[https://stedolan.github.io/jq/manual/ Jq manual] Include functions
	*[https://jqplay.org/ jqplay] Jq playground online querying tool		*[https://jqplay.org/ jqplay] Jq playground online querying tool
			*[https://oncletom.io/2016/pipelining-http/ Processing HTTP to HTTP with bash, curl and jq]

Pio2pio: /* Display only Keys that mach value */

2018-11-10T23:49:27Z

Display only Keys that mach value

← Older revision		Revision as of 01:49, 11 November 2018
Line 85:		Line 85:
	aws ec2 describe-volumes --region us-east-1 --profile prod > prod_volumes.json		aws ec2 describe-volumes --region us-east-1 --profile prod > prod_volumes.json
	cat prod_volumes.json \		cat prod_volumes.json \
	\| jq --raw-output '.Volumes[] \| [ (.Tags[]? \| select(.Key=="Name").Value, select(.Key=="Backup").Value), .VolumeId, .State, .SnapshotId, .CreateTime, .Size ] \| @csv' \		\| cat ftr_volumes_all.json \
			\| jq --raw-output '.Volumes[] \| [ (.Tags[]? \| select(.Key=="Name").Value, select(.Key=="Backup").Value), .VolumeId, .State, .SnapshotId, .Attachments[].InstanceId //"notAttached", .CreateTime, .Size ] \| @csv' \
			> ftr_volumes_all.csv \
	> prod_volumes.csv		> prod_volumes.csv
	</source>		</source>
	* <code>.Tags[]?</code> means that the list/object is optional, prevents ''jq: error - Cannot iterate over null (null)'' error		* <code>.Tags[]?</code> means that the list/object is optional, prevents ''jq: error - Cannot iterate over null (null)'' error
	* <code>(.Tags[]? \| select(.Key=="Name").Value, select(.Key=="Backup").Value)</code> taking expression in brackets groups multiple filters into one, so it is not affecting next filter. Remember <code>,</code> coma feeds date through next filter here it's <code>.VolumeId</code>. This produces first 2 items on in the list, see below in results.		* <code>(.Tags[]? \| select(.Key=="Name").Value, select(.Key=="Backup").Value)</code> taking expression in brackets groups multiple filters into one, so it is not affecting next filter. Remember <code>,</code> coma feeds date through next filter here it's <code>.VolumeId</code>. This produces first 2 items on in the list, see below in results.
			* <code>.Attachments[].InstanceId //"notAttached"</code> 2 forward slashes set default value


Line 104:		Line 107:
	"Attachments": [],		"Attachments": [],
	"Tags": [		"Tags": [
	{
	~~"Value": "True",~~
	~~"Key": "Backup"~~
	},
	{		{
	"Value": "Comms Scheduler",		"Value": "Comms Scheduler",
Line 123:		Line 122:
	},		},
	{		{
	"AvailabilityZone": "us-east-1b",		"AvailabilityZone": "us-east-1b",
	"Attachments": [],		"Attachments": [
			{
			"AttachTime": "2018-06-27T11:13:30.000Z",
			"InstanceId": "i-44441a50fc245555",
			"VolumeId": "vol-77770ff240e67777",
			"State": "attached",
			"DeleteOnTermination": true,
			"Device": "/dev/sda1"
			}
			],
	"Tags": [		"Tags": [
	{		{
	"Value": "~~Comms Scheduler~~",		"Value": "apache",
	"Key": "Name"		"Key": "Name"
	},		},
	{		{
	"Value": "~~True~~",		"Value": "False",
	"Key": "Backup"		"Key": "Backup"
	}		}
	],		],
	"Encrypted": ~~false~~,		"Encrypted": true,
	"VolumeType": "gp2",		"VolumeType": "gp2",
	"VolumeId": "vol-~~22222664d04133333~~",		"VolumeId": "vol-123123ff240e62222",
	"State": "~~available~~",		"State": "in-use",
	"~~Iops~~": ~~150~~,		"KmsKeyId": "arn:aws:kms:us-west-1:0123481158111:key/40222222-4455-7777-8888-c06f27d59999",
	"SnapshotId": "snap-~~22222315bf2b33333~~",		"SnapshotId": "snap-999947cb7448d8888",
	"CreateTime": "2018-06-~~20T15~~:29:32.~~224Z~~",		"Iops": 100,
	"Size": 50		"CreateTime": "2018-06-27T11:13:30.582Z",
			"Size": 8
	}		}
	]		]
Line 149:		Line 158:
	\|style="vertical-align:top;" \|<source lang="json">		\|style="vertical-align:top;" \|<source lang="json">
	[		[
	~~"True",~~
	"Comms Scheduler",		"Comms Scheduler",
			"backupFalse",
	"vol-1111cd2b12e622222",		"vol-1111cd2b12e622222",
	"available",		"available",
	"snap-1111810031fa22222",		"snap-1111810031fa22222",
			"notAttached",
	"2018-06-20T15:29:32.333Z",		"2018-06-20T15:29:32.333Z",
	100		100
	]		]
	[		[
	"~~Comms Scheduler~~",		"apache",
	"~~True~~",		"backupFalse",
	"vol-~~22222664d04133333~~",		"False",
	"~~available~~",		"vol-123123ff240e62222",
	"snap-~~22222315bf2b33333~~",		"in-use",
	"2018-06-~~20T15~~:29:32.~~224Z~~",		"snap-999947cb7448d8888",
	50		"i-44441a50fc245555",
			"2018-06-27T11:13:30.582Z",
			8
	]		]
	</source>		</source>
	\|}		\|}

	== Filtered list of instance name, type, virtualization type and instance id ==		== Filtered list of instance name, type, virtualization type and instance id ==
	At the beginning my simple convention for Json is <tt>[]</tt> is an array and <tt>{}</tt> is an object.		At the beginning my simple convention for Json is <tt>[]</tt> is an array and <tt>{}</tt> is an object.

Pio2pio: /* Filtered list of instance name, type, virtualization type and instance id */

2018-11-10T22:22:11Z

Filtered list of instance name, type, virtualization type and instance id

@@ Line 80: / Line 80: @@
 </source>
 == Filtered list of instance name, type, virtualization type and instance id ==
 At the beginning my simple convention for Json is <tt>[]</tt> is an array and <tt>{}</tt> is an object.

Pio2pio: /* Turn JSON into CSV */

2018-11-10T21:44:34Z

Turn JSON into CSV

← Older revision		Revision as of 23:44, 10 November 2018
Line 40:		Line 40:
	== Turn JSON into CSV ==		== Turn JSON into CSV ==
	<source lang="bash">		<source lang="bash">
	.Snapshots[] \| [.Description,.Encrypted,.VolumeId,.State,.VolumeSize,.StartTime,.Progress,.OwnerId,.SnapshotId] \| @csv		aws ec2 describe-snapshots --owner-ids 777781152222 --region us-east-1 --profile prod > prod_snapshots.json
			cat prod_snapshots.json \
			\| jq -r '.Snapshots[] \| [.Description,.Encrypted,.VolumeId,.State,.VolumeSize,.StartTime,.Progress,.OwnerId,.SnapshotId] \| @csv' \
			> prod_snapshots.csv
	# by using filter '.Snapshots[]' you select "Snapshots" array every object, then		# by using filter '.Snapshots[]' you select "Snapshots" array every object, then
	# create an array with [.key1,.key2] as array is required data structure to turn into CSV		# create an array with [.key1,.key2] as array is required data structure to turn into CSV

Pio2pio: /* Turn JSON into CSV */

2018-11-10T10:25:54Z

Turn JSON into CSV

← Older revision		Revision as of 12:25, 10 November 2018
Line 43:		Line 43:
	# by using filter '.Snapshots[]' you select "Snapshots" array every object, then		# by using filter '.Snapshots[]' you select "Snapshots" array every object, then
	# create an array with [.key1,.key2] as array is required data structure to turn into CSV		# create an array with [.key1,.key2] as array is required data structure to turn into CSV
			# by default double quote will be escaped ' \" ' using '-r' prevents doing it making easier to open in Excel.
	</source>		</source>

Pio2pio: /* jq */

2018-11-10T10:22:46Z

@@ Line 38: / Line 38: @@
 = jq =
 == Filtered list of instance name, type, virtualization type and instance id ==
 At the beginning my simple convention for Json is <tt>[]</tt> is an array and <tt>{}</tt> is an object.

Pio2pio: /* What is JSON */

2018-11-10T09:55:09Z

What is JSON

← Older revision		Revision as of 11:55, 10 November 2018
Line 5:		Line 5:
	* An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.		* An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.

	;An object: is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).		;An object: is an unordered set of name/value pairs. An object begins with <code>{</code> (left brace) and ends with <code>}</code> (right brace). Each name is followed by <code>:</code> (colon) and the name/value pairs are separated by <code>,</code> (comma).
	;An array: is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).		;An array: is an ordered collection of values. An array begins with <code>[</code> (left bracket) and ends with <code>]</code> (right bracket). Values are separated by <code>,</code> (comma).
	;A value: can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.		;A value: can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
	;A string: is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. A character is represented as a single character string. A string is very much like a C or Java string.		;A string: is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. A character is represented as a single character string. A string is very much like a C or Java string.
	;A number: is very much like a C or Java number, except that the octal and hexadecimal formats are not used.		;A number: is very much like a C or Java number, except that the octal and hexadecimal formats are not used.

	= Interactive tools =		= Interactive tools =
	== [https://jqplay.org/ jqplay] online ==		== [https://jqplay.org/ jqplay] online ==

Pio2pio: /* References */

2018-11-10T09:52:47Z

References

← Older revision		Revision as of 11:52, 10 November 2018
Line 138:		Line 138:
	*[https://programminghistorian.org/en/lessons/json-and-jq jq and json training]		*[https://programminghistorian.org/en/lessons/json-and-jq jq and json training]
	*[https://stedolan.github.io/jq/manual/ Jq manual] Include functions		*[https://stedolan.github.io/jq/manual/ Jq manual] Include functions
			*[https://jqplay.org/ jqplay] Jq playground online querying tool

Pio2pio: /* Interactive tools */

2018-11-10T09:51:44Z

Interactive tools

← Older revision		Revision as of 11:51, 10 November 2018
Line 11:		Line 11:
	;A number: is very much like a C or Java number, except that the octal and hexadecimal formats are not used.		;A number: is very much like a C or Java number, except that the octal and hexadecimal formats are not used.
	= Interactive tools =		= Interactive tools =
			== [https://jqplay.org/ jqplay] online ==
			Online jq playground.
			[[File:Jqplay.png\|700px\|none\|left\|Jq playground]]
	== Jiq ==		== Jiq ==
	=== Install Jiq ===		=== Install Jiq ===