Illustrated jq tutorial

The concept of pipes

Unix pipelines were invented in 1973 by Douglas McIlroy as a novel way of stringing together programs, where the output of one program is the input of the next one; It's a way of creating a new program out of combining basic building blocks, McIlroy describes it by analogy as 'screwing together data streams like a garden hose'. This approach quickly became the UNIX philosophy of programming described by McIlroy as follows: 'Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.' Lets say you want to know what are the most common words occurring within a text, the following pipeline will order the words of a text by frequency of usage: 
cat README.md  | tr " " "\n" | tr  -d '[:punct:]'  | sort   | uniq  -c | sort -n -k 1
This example is a bit like functional programming: in each step of the pipeline the output depends only on the input received via the preceding pipe, each step acts on that text input only and produces its output without writing any files, that is without side effects.

Fast forward to the 21st century

Parsing of text streams is the Unix way of doing things and it has proven to be very versatile as is, however many application call for the exchange of structured data; There are more and more programs that use json as the format for exchanging structured data, examples are:

jq - a tool for manipulating structured data

jq is a very versatile tool for working with structured information in json format, the command syntax of jq is also structured by means of a processing pipeline, similar to that of a unix shell, again each processing step acts as a filter/modifier of the input received from the preceding stage. Again on might look at each of these stages as functions in a functional program. This tutorial tries to explain jq in terms of example pipelines; each example comes with links that show you the intermediate results for each stage of the processing pipeline; i think this makes it easier to understand each of the building blocks involved. You can click either on any one of the commands to show the command and how it transforms the input json structure into the output json, each pipe symbol is also a link that will show you the information that flows through it. The html for this tutorial is generated by this script

The tutorial

Get a single scalar values


cat s1.json | jq ' .spec.replicas '

Get a single scalar values (different form, as a pipeline)


cat s1.json | jq ' .spec | .replicas '

Get two scalar values


cat s1.json | jq ' .spec.replicas, .kind '

Get two scalar values and concatenate/format them into a single string


cat s1.json | jq ' "replicas: " + (.spec.replicas | tostring) + " kind: " + .kind '

Select an object from an array of object based on one of the names


cat dep.json | jq ' .status.conditions | map(select(.type == "Progressing")) '

Select a single key value pair from a json object


cat ann.json | jq ' .metadata.annotations | to_entries | map(select(.key == "label1")) | from_entries '

Select a single key value pair from a json object - short form


cat ann.json | jq ' .metadata.annotations | with_entries(select(.key == "label1")) '

Select two key value pairs from a json object


cat ann.json | jq ' .metadata.annotations | to_entries | map(select(.key == "label1" or .key == "label2")) | from_entries '

Select two key value pairs from a json object (second version)


cat ann.json | jq ' .metadata.annotations | to_entries | map(select(.key == ("label1", "label2"))) | from_entries '

Select all key value pairs from a json object where the name contains substring "label"


cat ann.json | jq ' .metadata.annotations | to_entries | map(select(.key | contains("label"))) | from_entries '

Select all key value pairs from a json object where the name matches the regular expression label[1-9]


cat ann.json | jq ' .metadata.annotations | to_entries | map(select(.key | test("label[1-9]"))) | from_entries '

Select all key value pairs from a json object where the name contains substring "label" (short form)


cat ann.json | jq ' .metadata.annotations | with_entries(select(.key | contains("label"))) '

Add another key value pair to a json object


cat ann.json | jq ' .metadata.annotations += { "label4" : "two" } '

Set all values in a json object


cat ann.json | jq ' .metadata.annotations | to_entries | map_values(.value="override-value") | from_entries '

Set all values of subset of keys in a json object


cat ann.json | jq ' .metadata.annotations | to_entries | map(if .key | contains("label") then .value="kuku" else . end) | from_entries '