Sample Header Ad - 728x90

Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

4 votes
2 answers
2385 views
jq: Printing multiple values from multiple arrays at once
The default functionality of `jq` is to send each object from an array one at a time, though the `join` operator can merge those values. My problem is in trying to print all the values from multiple arrays at once. Taking this example: ```json { "key1": { "list1": [ "val1", "val2", "val3" ] }, "key2...
The default functionality of jq is to send each object from an array one at a time, though the join operator can merge those values. My problem is in trying to print all the values from multiple arrays at once. Taking this example:
{
    "key1": {
        "list1": [
            "val1",
            "val2",
            "val3"
        ]
    },
    "key2": {
        "list1": [
            "val4",
            "val5"
        ]
    },
    "key3": {
        "list1": [
            "val6"
        ]
    }
}
I'd like to print:
val1 val2 val3 val4 val5 val6
And so far have this:
jq -r 'to_entries[] | { list: .value.list1 } | .list | join(" ")' test.json
*(More verbose than necessary to help reviewers.)* which gives:
val1 val2 val3
val4 val5
val6
Is there a way to gather all the values together in one command?
T145 (223 rep)
Apr 27, 2024, 07:33 PM • Last activity: Jul 21, 2025, 12:35 PM
-1 votes
3 answers
4204 views
How to get the json data used in each curl statement while using xargs and maps it to its corresponding result?
I have text file that has arguments of curl command. This is how the file looks ``` 'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username2",}' 'https://example.com/tl/' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username3",}' 'htt...
I have text file that has arguments of curl command. This is how the file looks
'https://example.com/tl/ ' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username2",}'
'https://example.com/tl/ ' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username3",}'
'https://example.com/tl/ ' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username4",}'
'https://example.com/tl/ ' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username5",}'
'https://example.com/tl/ ' -X POST -H 'Content-Type: application/json' --data-raw '{"email":"username6",}'
This is the command I use
/AbsolutePath/Inputfile.txt | xargs -P 10000 -n 10 curl -s | jq '.message'
I'm using jq , to parse json in command line What I want is, 1) Pipe or send output of above command to another command, so that if message has certain text, grab the email **value** used in that corresponding curl command and write to a log file or create a filename with usernameX.txt For example, only if username2 and username5 cURL command's message = 'success', these two usernames should be written to a log file or two files username2.txt and username5.txt should be created.
sofs1 (117 rep)
Jan 4, 2022, 11:26 AM • Last activity: Jun 26, 2025, 01:07 AM
5 votes
2 answers
510 views
Is there an equivalent of python's enumerate in jq?
I want to get the value *after* a selected item in a list in `jq`. Something like this seems like it would work: ``` jq '. as $list | select(.focused == true) | .id as $id | $list | select(????) ``` One way of doing this would be to enumerate entries like Python - is there a way of doing this. In py...
I want to get the value *after* a selected item in a list in jq. Something like this seems like it would work:
jq '. as $list | select(.focused == true) | .id as $id | $list | select(????)
One way of doing this would be to enumerate entries like Python - is there a way of doing this. In python enumerate takes [x0, x1, x2, x3, ...] -> [(0, x0), (1, x1), ...] . Another way would be an equivalent of itertools itertools.dropwhile
Att Righ (1412 rep)
May 3, 2025, 07:36 PM • Last activity: May 5, 2025, 03:03 PM
3 votes
3 answers
4652 views
Extract json array element based on a subelement value
We have the following example file ( very long file , this is short example ) "request_status" : "FAILED" { "href" : "http://localhost:8080/api/v1/clusters/sys41/requests/333", "Requests" : { "cluster_name" : "sys41", "id" : 333, "request_status" : "COMPLETED" } }, { "href" : "http://localhost:8080/...
We have the following example file ( very long file , this is short example ) "request_status" : "FAILED" { "href" : "http://localhost:8080/api/v1/clusters/sys41/requests/333", "Requests" : { "cluster_name" : "sys41", "id" : 333, "request_status" : "COMPLETED" } }, { "href" : "http://localhost:8080/api/v1/clusters/sys41/requests/334", "Requests" : { "cluster_name" : "sys41", "id" : 334, "request_status" : "FAILED" } }, { "href" : "http://localhost:8080/api/v1/clusters/sys41/requests/335", "Requests" : { "cluster_name" : "sys41", "id" : 335, "request_status" : "FAILED" } }, { "href" : "http://localhost:8080/api/v1/clusters/sys41/requests/336", "Requests" : { "cluster_name" : "sys41", "id" : 336, "request_status" : "COMPLETED" } } how to print the line after the line that matches "id" : $num e.g. for num=335 how to get the line after "id" : $num Expected output "request_status" : "FAILED"
yael (13936 rep)
Mar 5, 2018, 05:09 PM • Last activity: May 4, 2025, 08:48 AM
1 votes
1 answers
1962 views
Defining functions and using map in jq
Can you explain to me how the following two differ? ``` jq ´def addvalue(f): . + [f]; map(addvalue(.[0]))´ [[1,2],[10,20]] => [[1,2,1], [10,20,10]] jq ´def addvalue(f): f as $x | map(. + $x); addvalue(.[0])´ [[1,2],[10,20]] => [[1,2,1,2], [10,20,1,2]] ``` Thanks.
Can you explain to me how the following two differ?
jq ´def addvalue(f): . + [f]; map(addvalue(.))´

[[1,2],[10,20]]
=> [[1,2,1], [10,20,10]]

jq ´def addvalue(f): f as $x | map(. + $x); addvalue(.)´
[[1,2],[10,20]]
=> [[1,2,1,2], [10,20,1,2]]
Thanks.
Logan Lee (249 rep)
Oct 24, 2021, 05:06 AM • Last activity: Apr 20, 2025, 11:08 PM
3 votes
2 answers
9407 views
parse JSON data using jq and convert into csv
I am downloading api json data and converting into my final result as a csv file. The sample data is in the below form: { "content": [{ "Title": "abc", "brand": "xyz", "information": { "c1": "101", "c2": "11111", "c3": "a,b,c,d,e:abc." } }, { "Title": "RX100", "brand": "Yamaha", "information": { "c1...
I am downloading api json data and converting into my final result as a csv file. The sample data is in the below form: { "content": [{ "Title": "abc", "brand": "xyz", "information": { "c1": "101", "c2": "11111", "c3": "a,b,c,d,e:abc." } }, { "Title": "RX100", "brand": "Yamaha", "information": { "c1": "102", "c2": "22222", "c3": "a." } }, { "Title": "victor", "brand": "TVS", "information": { "c1": "103", "c2": "33333", "c3": "a,b,c" } }, { "Title": "R15", "brand": "Yamaha", "information": { "c1": "104", "c2": "44444", "c3": "a,b" } } ] } I have successfully downloaded and converted into multiple csv on the basis of number of Title. Data looks like in the below form after converting in multiple csv file. Headers-> c1,c2,c3 csv1--> 101,11111,a,b,c,d,e:abc. csv2--> 102,22222,a. csv3--> 103,33333,a,b,c. csv4--> 104,44444,a,b. but I want the above data in below form. Headers-> c1,c2,c3,c4,c5,c6,c7 csv1--> 101,11111,a,b,c,d,e:abc. csv2--> 102,22222,a. csv3--> 103,33333,a,b,c. csv4--> 104,44444,a,b. Is it possible to divide the c3 into different number of columns on the basis of "," present in my json file using json. The column c3 will get divided on the basis of number of elements are present and will be the maximum value of c3 data.
sam (191 rep)
Mar 21, 2019, 05:08 AM • Last activity: Mar 10, 2025, 08:10 PM
2 votes
1 answers
82 views
Append to matching children of an arbitrarily deep array
I am attempting to edit an OpenAPI specification by changing all parameters to be nullable. A parameter definition looks like this: ```json { "name": "foo", "required": true, "type": "string" } ``` They are contained in a `parameters` array that can be anywhere in the document. What I need is to app...
I am attempting to edit an OpenAPI specification by changing all parameters to be nullable. A parameter definition looks like this:
{
    "name": "foo",
    "required": true,
    "type": "string"
}
They are contained in a parameters array that can be anywhere in the document. What I need is to append "x-nullable": true to any parameter containing a type property. Sample data:
{
    "parameters": [
        {"notype": "x"},
        {"type": "x"}
    ],
    "foo": {
        "parameters": [
            {"type": "y"}
        ]
    },
    "bar": {
        "baz": {
            "parameters": [
                {"type": "z"}
            ]
        }
    }
}
Desired output:
{
    "parameters": [
        {"notype": "x"},
        {
            "type": "x",
            "x-nullable": true
        }
    ],
    "foo": {
        "parameters": [
            {
                "type": "y",
                "x-nullable": true
            }
        ]
    },
    "bar": {
        "baz": {
            "parameters": [
                {
                    "type": "z",
                    "x-nullable": true
                }
            ]
        }
    }
}
The closest I could get was [this](https://play.jqlang.org/s/Xx-ep42wRLGfU8R) :
.. | (.parameters[] | select(.type)) += {"x-nullable":true}
It successfully changes one of the items in my test document, but the results are inconsistent and seem to be based on the structure I choose for sample data.
miken32 (588 rep)
Feb 28, 2025, 05:12 PM • Last activity: Feb 28, 2025, 07:59 PM
3 votes
2 answers
492 views
Updating JSON document with JSON object embedded in part of JSON string
I have something like the following ```json { "t": "set foo='{\"mode\":1}'" } ``` And I'd like to transform it into something like ```json { "t": "set foo='{\"mode\":1}'", "mode": 1 } ``` ... where the key `mode` and the value `1` are taken from the embedded JSON in the value of the `t` key. I am ex...
I have something like the following
{
  "t": "set foo='{\"mode\":1}'"
}
And I'd like to transform it into something like
{
  "t": "set foo='{\"mode\":1}'",
  "mode": 1
}
... where the key mode and the value 1 are taken from the embedded JSON in the value of the t key. I am executing several commands to make this happen, but I would like to see if putting it into one jq call is possible.
winmutt (131 rep)
Dec 7, 2018, 07:57 PM • Last activity: Feb 25, 2025, 11:28 AM
0 votes
2 answers
857 views
jq command to retrieve titles from JSON embedded in JSON string
I have this JSON file: ```json "[{\"id\":188457,\"title\":\"Painted Dragon Tea Mug with Lid\",\"slug\":\"painted-dragon-tea-mug-with-lid\",\"image\":\"https:\\\/\\\/thievemedia.imgix.net\\\/2018\\\/09\\\/painted-dragon-tea-mug-with-lid-featured-1.jpg\",\"full_image_url\":\"https:\\\/\\\/s3.amazonaws...
I have this JSON file:
"[{\"id\":188457,\"title\":\"Painted Dragon Tea Mug with Lid\",\"slug\":\"painted-dragon-tea-mug-with-lid\",\"image\":\"https:\\\/\\\/thievemedia.imgix.net\\\/2018\\\/09\\\/painted-dragon-tea-mug-with-lid-featured-1.jpg\",\"full_image_url\":\"https:\\\/\\\/s3.amazonaws.com\\\/thieve-media\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/painted-dragon-tea-mug-with-lid-featured-1.jpg\",\"large_image_url\":\"https:\\\/\\\/s3.amazonaws.com\\\/thieve-media\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/painted-dragon-tea-mug-with-lid-featured-1.jpg\",\"medium_image_url\":\"https:\\\/\\\/s3.amazonaws.com\\\/thieve-media\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/painted-dragon-tea-mug-with-lid-featured-1.jpg\",\"original_price\":\"19.78\",\"price\":\"13.65\",\"discount\":\"30.99\",\"rating\":\"4.7\",\"total_orders\":\"10\",\"promotion_url\":\"https:\\\/\\\/api.thieve.co\\\/view?id=MzI4NTk0Njg0MDE=\",\"product_like_count\":\"100\",\"product_user_likes\"}]
How can I extract only the values of the title keys with jq?
Kerim Güleç (11 rep)
Oct 9, 2018, 04:16 PM • Last activity: Feb 25, 2025, 11:23 AM
1 votes
2 answers
314 views
Updating and removing values in JSON array based on matching with placeholder values
I search for an object with a specific property and then update another property. Given this input: ```json [ { "replacements": [ { "newValue": "0", "yamlPath": "k8s-helm-templates.deployment.containers.abc.image.tag" }, { "newValue": "0", "yamlPath": "k8s-helm-templates.deployment.containers.def.im...
I search for an object with a specific property and then update another property. Given this input:
[
  {
    "replacements": [
      {
        "newValue": "0",
        "yamlPath": "k8s-helm-templates.deployment.containers.abc.image.tag"
      },
      {
        "newValue": "0",
        "yamlPath": "k8s-helm-templates.deployment.containers.def.image.tag"
      },
      {
        "newValue": "0",
        "yamlPath": "k8s-helm-templates.deployment.containers.ghi.image.tag"
      }
    ],
    "yamlFilePath": "k8s/helm/Dev/us-east-1/values.yaml"
  }
]
And I have the placeholders available as:
[
  {"name":"abc-image-tag","value":"123"},
  {"name":"def-image-tag","value":"456"}
]
Here, abc-image-tag corresponds to k8s-helm-templates.deployment.containers.abc.image.tag, etc. This should result in a result where the given values are correctly replaced and 0-values are filtered, such as this:
[
  {
    "replacements": [
      {
        "newValue": "123",
        "yamlPath": "k8s-helm-templates.deployment.containers.abc.image.tag"
      },
      {
        "newValue": "456",
        "yamlPath": "k8s-helm-templates.deployment.containers.def.image.tag"
      }
    ],
    "yamlFilePath": "k8s/helm/Dev/us-east-1/values.yaml"
  }
]
I tried several tips and dug into the documentation but couldn't get it to work. Is this even possible with jq? Additional steps with bash would be ok.
Jazzschmidt (353 rep)
Dec 14, 2023, 02:35 PM • Last activity: Feb 25, 2025, 10:34 AM
3 votes
1 answers
1247 views
Extract data from JSON document embedded in another JSON document
I have a JSON document that embeds another JSON document. I need to extract data from the embedded document, but I'm not well versed with JSON or with `jq`. The following is my input (`guard.json`): ```json { "_index": "test-2021.06.02", "_type": "servrd", "_id": "ZWUxMDU5MjItOGY2MC00MGI5LWJhZTEtODR...
I have a JSON document that embeds another JSON document. I need to extract data from the embedded document, but I'm not well versed with JSON or with jq. The following is my input (guard.json):
{
  "_index": "test-2021.06.02",
  "_type": "servrd",
  "_id": "ZWUxMDU5MjItOGY2MC00MGI5LWJhZTEtODRhYWQ1YTZhOGIy",
  "_version": 1,
  "_score": null,
  "_source": {
    "stream": "stdout",
    "time": "2021-10-02T03:13:52.496705721Z",
    "docker": {
      "container_id": "392923402349320329432h3432k4kj32kfks9s9sdfksdfjkdsjfsh3939322342"
    },
    "kubernetes": {
      "container_name": "test",
      "namespace_name": "dev",
      "pod_name": "test-dev-v004-9s885",
      "container_image": "localhost:80/pg/test:1.19.0",
      "container_image_id": "docker-pullable://localhost:80/pg/test@sha256:2sfdsfsfsfsfdsr3wrewrewc235e728ad1b29baf5af3dfad30c7706e5eda38b6109",
      "pod_id": "ssfsfds-dsfdsfs-fs-sfsfs-sfdsfsdfsewrw",
      "host": "test-jup-node008",
      "labels": {
        "app": "test",
        "cluster": "test-dev",
        "load-balancer-test-dev": "true",
        "stack": "dev",
        "app_kubernetes_io/managed-by": "spinnaker",
        "app_kubernetes_io/name": "test",
        "moniker_spinnaker_io/sequence": "4"
      },
      "min_url": "https://100.400.0.22:443/api ",
      "namespace_id": "jajdjsdf-dfse-dsf-koksls-sksjf030292",
      "namespace_labels": {
        "name": "dev"
      }
    },
    "elapsedTime": 39013,
    "message": "TransactionLog",
    "membersId": "TEST_0233203203_030202020303",
    "payload": "{\"serviceId\":\"00343\",\"AccessKey\":\"testdfsolpGS\",\"trackID\":\"KOLPSLSLSLL99029283\",\"membersId\":\"TEST_0233203203_030202020303\",\"shopperInfo\":{\"emailAddress\":\"test.ooo4@yahoo.com\",\"ipAddress\":\"localhost\"},\"parkid\":{\"parkssID\":\"carrier-park\"},\"cartinfo\":{\"checkType\":\"preorder\",\"detailsmetis\":\"card\",\"currency\":\"US\",\"grosscount\":\"10\",\"reedeem\":\".00\",\"Discount\":\".00\",\"tokenvalue\":{\"token\":\"11102020392023920920393993\",\"Bin\":\"00000\",\"digit\":\"0000\",\"expirationDate\":\"202509\",\"price\":\"10\"}},\"cartdetails\":[{\"dones\":[{\"donesnames\":\"test\",\"price\":\"003\",\"quan\":\"1\"}]}]}",
    "requestDate": "2021-10-02T03:13:12.541207804Z",
    "requestId": "12321321wes-sfsfdsf-snnm79887-029299",
    "finalToClient": "{\"finalType\":\"ok\",\"evaluationMessage\":\"Accept\",\"subMessage\":\"testcallled\",\"score\":0}",
    "serviceId": 00343,
    "timestamp": "2021-10-02T03:13:51.951+00:00",
    "@timestamp": "2021-10-02T03:13:52.621643399+00:00"
  },
  "fields": {
    "@timestamp": [
      "2021-10-02T03:13:52.621Z"
    ],
    "requestDate": [
      "2021-10-02T03:13:12.541Z"
    ],
    "timestamp": [
      "2021-10-02T03:13:51.951Z"
    ]
  },
  "highlight": {
    "kubernetes.labels.app": [
      "@kibana-highlighted-field@test@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1654139632621
  ]
}
I need output in CSV format, similar to this:
serviceId, trackID, currency, grosscount
00343,KOLPSLSLSLL99029283,US,10
SRash (111 rep)
Jun 2, 2022, 07:39 AM • Last activity: Feb 25, 2025, 07:26 AM
8 votes
3 answers
797 views
How can I make jq assign values to environment variables in one command?
Say, I provide `jq` with the following JSON body as input: {"person1": {"name": "foo"}, "person2": {"name": "bar"}} Is it possible to have `jq` assign `person1.name` to environment variable `PERSON_1_NAME` and `person2.name` to environment variable `PERSON_2_NAME`, so that I don't have to run `jq` m...
Say, I provide jq with the following JSON body as input: {"person1": {"name": "foo"}, "person2": {"name": "bar"}} Is it possible to have jq assign person1.name to environment variable PERSON_1_NAME and person2.name to environment variable PERSON_2_NAME, so that I don't have to run jq multiple times with the same input, e.g.: INPUT='{"person1": {"name": "foo"}, "person2": {"name": "bar"}}' export PERSON_1_NAME=$(echo $INPUT | jq -r .person1) ?
Shuzheng (4931 rep)
Feb 18, 2025, 07:39 PM • Last activity: Feb 20, 2025, 07:35 PM
6 votes
2 answers
14950 views
How to search and replace multiple values in an array using jq?
In the following json file, { "contacts": [ { "name": "John", "phone": "1234" }, { "name": "Jane", "phone": "5678" } ] } I need to update both phone numbers based on the name and store the whole json in a new file. I tried stuff like: jq '.contacts[] | select(.name == "John") | .phone = "4321"' upda...
In the following json file, { "contacts": [ { "name": "John", "phone": "1234" }, { "name": "Jane", "phone": "5678" } ] } I need to update both phone numbers based on the name and store the whole json in a new file. I tried stuff like: jq '.contacts[] | select(.name == "John") | .phone = "4321"' updated_contacts.json But then I don't know how to go back to the parent node and change Jane's one, nor retrieve the whole json. I tried to store to root node in a variable with as, but it keeps unchanged. As a temporary workaround I'm just doing this: jq '.contacts.number = "4321" | .contacts.number = "4321"' updated_contacts.json But I should not rely on array indexes, but names, as the original json may change. Any idea how could I do it using jq command?
txigreman (163 rep)
Aug 24, 2021, 09:10 AM • Last activity: Feb 20, 2025, 10:56 AM
2 votes
2 answers
1155 views
Conditionally add values to JSON arrays
I have some JSON objects that I need to add a value to if it doesn't already exist. Each object will be the following except the `contact_group` inside each array (1-5) will vary: ```json { "contact_groups": { "1": [ "/contact_group/78" ], "2": [ "/contact_group/79" ], "3": [], "4": [], "5": [] } }...
I have some JSON objects that I need to add a value to if it doesn't already exist. Each object will be the following except the contact_group inside each array (1-5) will vary:
{
  "contact_groups": {
    "1": [
      "/contact_group/78"
    ],
    "2": [
      "/contact_group/79"
    ],
    "3": [],
    "4": [],
    "5": []
  }
}
I want to add /contact_group/109 to each array if it doesn't already exist, so the above would become:
{
  "contact_groups": {
    "1": [
      "/contact_group/78",
      "/contact_group/109"
    ],
    "2": [
      "/contact_group/79",
      "/contact_group/109"
    ],
    "3": [
      "/contact_group/109"
    ],
    "4": [
      "/contact_group/109"
    ],
    "5": [
      "/contact_group/109"
    ]
  }
}
I'm pretty sure jq can do this, but I'm inexperienced with it, so I don't know where to begin. Does anyone know how/if this can be done?
jesse_b (41447 rep)
Jul 16, 2020, 05:06 PM • Last activity: Feb 20, 2025, 09:38 AM
7 votes
3 answers
2583 views
Trim trailing zeroes off a number extracted by jq
The following command achieve my goal by grepping `BTC` price from specific exchange. curl -sS https://api.binance.com/api/v1/ticker/price?symbol=BTCUSDT | jq -r '.price' the output will be for the moment `7222.25000000` but i would like to get it `7222.25`
The following command achieve my goal by grepping BTC price from specific exchange. curl -sS https://api.binance.com/api/v1/ticker/price?symbol=BTCUSDT | jq -r '.price' the output will be for the moment 7222.25000000 but i would like to get it 7222.25
αԋɱҽԃ αмєяιcαη (1267 rep)
May 12, 2019, 06:31 AM • Last activity: Feb 19, 2025, 08:29 PM
1 votes
1 answers
1948 views
Variable in curl adds backslashes to string
I am trying to use curl based on some variables to create customers in Stripe, but when I assign the token to a variable it is giving me an error on Stripe saying that it does not exist. However, if I put the text in directly it works. How can I use the `$TOKEN` variable, is there something changing...
I am trying to use curl based on some variables to create customers in Stripe, but when I assign the token to a variable it is giving me an error on Stripe saying that it does not exist. However, if I put the text in directly it works. How can I use the $TOKEN variable, is there something changing the value that I don't realize? Michael$ curl https://api.stripe.com/v1/customers -u $access_token: -d source=tok_1CjvRiDZ5DqZ0yaUVWXXXXXX { "error": { "code": "token_already_used", "doc_url": "https://stripe.com/docs/error-codes/token-already-used ", "message": "You cannot use a Stripe token more than once: tok_1CjvRiDZ5DqZ0yaUVWXXXXXX.", "type": "invalid_request_error" } } Michael$ curl https://api.stripe.com/v1/customers -u $access_token: -d source=$TOKEN { "error": { "code": "resource_missing", "doc_url": "https://stripe.com/docs/error-codes/resource-missing ", "message": "No such token: \"tok_1CjvRiDZ5DqZ0yaUVWXXXXXX\"", "param": "source", "type": "invalid_request_error" } } $TOKEN is assigned like this OUTPUT="$(curl https://api.stripe.com/v1/tokens -u $access_token: -d customer=$external_customer_id)" TOKEN="$(echo $OUTPUT | jq .id)"
Michael St Clair (113 rep)
Jul 3, 2018, 09:11 PM • Last activity: Feb 14, 2025, 12:17 PM
5 votes
2 answers
366 views
Merging multiple JSON data blocks into a single entity
I'm using an API (from [SyncroMSP](https://api-docs.syncromsp.com/)) that returns paginated JSON data. I can obtain the number of pages, and I can obtain the data with a tool such as `curl`. Each chunk is valid JSON but it only contains a subset of the total data that I need. Using `jq` or otherwise...
I'm using an API (from [SyncroMSP](https://api-docs.syncromsp.com/)) that returns paginated JSON data. I can obtain the number of pages, and I can obtain the data with a tool such as curl. Each chunk is valid JSON but it only contains a subset of the total data that I need. Using jq or otherwise how can I merge the tickets[] elements of these paginated data chunks back into a single JSON document? Here are three example chunks. The tickets[] arrays are heavily edited for this question and in reality contain up to 25 entries, and each ticket entry contains many more elements including at least a couple of arrays. JSON example block 1 (part_1.json) { "tickets": [ { "number": 4445, "subject": "Your mailbox is almost full" }, { "number": 4444, "subject": "Cannot VPN" } ], "meta": { "total_pages": 3, "page": 1 } } JSON example block 2 (part_2.json) { "tickets": [ { "number": 4395, "subject": "Trados Studio issue" }, { "number": 4394, "subject": "Daily Backup Report(No Errors)" } ], "meta": { "total_pages": 3, "page": 2 } } JSON example block 3 (part_3.json) { "tickets": [ { "number": 4341, "subject": "Daily Backup Report(No Errors)" }, { "number": 4340, "subject": "Windows Updates on VMs" } ], "meta": { "total_pages": 3, "page": 3 } } In this case the expected result would be something like this: { "tickets": [ { "number": 4445, "subject": "Your mailbox is almost full" }, { "number": 4444, "subject": "Cannot VPN" }, { "number": 4395, "subject": "Trados Studio issue" }, { "number": 4394, "subject": "Daily Backup Report(No Errors)" }, { "number": 4341, "subject": "Daily Backup Report(No Errors)" }, { "number": 4340, "subject": "Windows Updates on VMs" } ] } The output could include the meta hash too, as I'd just ignore it, and it wouldn't matter which meta.page value was carried forward. You can assume that tickets[].number is unique and that you do not need to preserve any ordering at that tickets[] level. There's enough complexity in the real data that I don't want to have to declare the full JSON structure in any resulting code. This is my current attempt, but I'm not particularly strong with jq. Is there a better way - for example, not calling jq twice, or being able to generalise the code so that I don't need to specify the name of the top level array (tickets)? cat part_{1,2,3}.json | jq '.tickets[]' | jq -n '{ tickets:[ inputs ] }'
Chris Davies (126603 rep)
Dec 13, 2024, 03:18 PM • Last activity: Feb 14, 2025, 05:44 AM
2 votes
1 answers
66 views
Transforming nested arrays in Office 365 logs into table
I want to release a simple tool based on jq to transform some common Office 365 Unified Audit Log events into a tabular report format, but having challenges with the way certain key arrays are nested. In particular, when i get down to Folders[] that contain sets of Ids, Paths, and FolderItems[] that...
I want to release a simple tool based on jq to transform some common Office 365 Unified Audit Log events into a tabular report format, but having challenges with the way certain key arrays are nested. In particular, when i get down to Folders[] that contain sets of Ids, Paths, and FolderItems[] that contain rows of message IDs and sizes, I can't figure out a way to make the related values from the arrays stay in sync / collate - instead I am getting massive combinations of every value as though I'm unintentionally iterating through them. Here's some sample data:
{"CreationTime":"2024-02-06T12:13:14","Id":"abcdabcd-1234-1234-5555-888888888888","Operation":"MailItemsAccessed","ResultStatus":"Succeeded","UserId":"admin@example.com","ClientIPAddress":"5.5.5.5","Folders":[{"FolderItems":[{"InternetMessageId":"","SizeInBytes":12345},{"InternetMessageId":"","SizeInBytes":11122},{"InternetMessageId":"","SizeInBytes":88888}],"Id":"EEEEEEEE","Path":"\\Outbox"},{"FolderItems":[{"InternetMessageId":"","SizeInBytes":44444},{"InternetMessageId":"","SizeInBytes":100000},{"InternetMessageId":"","SizeInBytes":109000},{"InternetMessageId":"","SizeInBytes":22000},{"InternetMessageId":"","SizeInBytes":333333}],"Id":"FFFFFFFFFFFFFFFFFAB","Path":"\\Inbox"}]}
{"CreationTime":"2024-02-06T20:00:00","Id":"abcdabcd-1234-1234-6666-9999999999999","Operation":"MailItemsAccessed","ResultStatus":"Succeeded","UserId":"other@other.cc","ClientIPAddress":"7.7.7.7","Folders":{"FolderItems":[{"InternetMessageId":"","SizeInBytes":77777},{"InternetMessageId":"","SizeInBytes":888888},{"InternetMessageId":"","SizeInBytes":99999}],"Id":"12341234","Path":"\\Temp"}}
Desired output: | CreationTime | Id | UserId | ClientIPAddress | FolderId | FolderPath | InternetMessageId | SizeInBytes | | ------------------- | ------------------------------------ | ----------------- | --------------- | ------------------- | ---------- | ------------------------------ | ----------- | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | EEEEEEEE | \\Outbox | abc1234@aaabbbccc.whatever.com | 12345 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | EEEEEEEE | \\Outbox | cccccc@aaabbbccc.whatever.com | 11122 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | EEEEEEEE | \\Outbox | final@host.gmail.com | 88888 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | FFFFFFFFFFFFFFFFFAB | \\Inbox | otherid@host.com | 44444 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | FFFFFFFFFFFFFFFFFAB | \\Inbox | furtherhost@xyz.host2.com | 100000 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | FFFFFFFFFFFFFFFFFAB | \\Inbox | id7@bbb.outlook.com | 109000 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | FFFFFFFFFFFFFFFFFAB | \\Inbox | other@hosthost.com | 22000 | | 2024-02-06T12:13:14 | abcdabcd-1234-1234-5555-888888888888 | admin@example.com | 5.5.5.5 | FFFFFFFFFFFFFFFFFAB | \\Inbox | junk@b8b8b8.newhost.com | 333333 | | 2024-02-06T20:00:00 | 12341234 | other@other.cc | 7.7.7.7 | 12341234 | \\Temp | example@aaabbbccc.whatever.com | 77777 | | 2024-02-06T20:00:00 | 12341234 | other@other.cc | 7.7.7.7 | 12341234 | \\Temp | cccccc4@aaabbbccc.whatever.com | 888888 | | 2024-02-06T20:00:00 | 12341234 | other@other.cc | 7.7.7.7 | 12341234 | \\Temp | final3@host.gmail.com | 99999 | Note that the .Folders element can sometimes come in string format but that I was able to easily conditionally load using fromjson. For example:
[...]"Folders": "[{\"FolderItems\":[{\"InternetMessageId\":\""Fo\",\"SizeInBytes\":12345},[...]
Code so far:
cat | jq '
    if has("Folders") then
        if(.Folders | type=="string") and .Folders != "" then .Folders |= fromjson  end |
        if(.Folders | type=="string") and .Folders == "" then .Folders = null end
    end | .' |     # works up to here at least
    jq '
if has("Item") then .Item |= (if type=="string" and .!="" then fromjson else {} end) else .Item|={}  end |
    if has("Item") then
            if .Item | has("Id") then .ItemId = .Item.Id else .ItemId={} end |
            if .Item | has("ParentFolder") then
                .ItemParentFolderId=.Item.ParentFolder.Id? |
                    .ItemParentFolderPath=.Item.ParentFolder.Path? |
                    .ItemParentFolderName=.Item.ParentFolder.Name?
            end
        end | . ' | cat # works up to here at least
    jq '
    if has("Folders") then
        if (.Folders | select(type=="array")) then
            .Folders[].Id? |
            .FoldersPath=.Folders[].Path? |
            .FoldersFolderItems=.Folders[].FolderItems?
        else . end
    end
    ' |
jq -r '. | (.TimeGenerated // .CreationTime) as $EventTime |
.ClientIP = if .ClientIP == "" then null else .ClientIP end |
.ClientIP_ = if .ClientIP_ == "" then null else .ClientIP_ end |
.Client_IPAddress = if .Client_IPAddress == "" then null else .Client_IPAddress end |
.ClientIPAddress = if .ClientIPAddress == "" then null else .ClientIPAddress end |
.ActorIpAddress = if .ActorIpAddress == "" then null else .ActorIpAddress end |
(.ClientIP // .ClientIP_ // .Client_IPAddress // .ClientIPAddress // .ActorIpAddress) as $IPAddress |
(.UserId // .UserId_) as $LogonUser |
.FFIIMI as $InternetMessageId |
.FFISIB as $SizeInBytes |
{EventTime: $EventTime, IPAddress: $IPAddress, LogonUser: $LogonUser, InternetMessageId: $InternetMessageId, SizeInBytes: $SizeInBytes} + . |
[.Id, .EventTime, .IPAddress, .LogonUser, .MailboxOwnerUPN, .Operation, .InternetMessageId, .SizeInBytes] | @csv'
whitepaws (121 rep)
Feb 11, 2024, 06:31 AM • Last activity: Feb 8, 2025, 12:06 AM
5 votes
2 answers
3747 views
How to convert timestamp value to number and then convert to human readable
I have a string timestamp key I need to convert to a number because `strftime` throws an error that it needs to be a number. `journalctl -n1 --output=json | jq '.__REALTIME_TIMESTAMP | tonumber |= (. / 1000 | strftime("%Y-%m-%d")), .MESSAGE'` but I get invalid path expression errors. I assume I don'...
I have a string timestamp key I need to convert to a number because strftime throws an error that it needs to be a number. journalctl -n1 --output=json | jq '.__REALTIME_TIMESTAMP | tonumber |= (. / 1000 | strftime("%Y-%m-%d")), .MESSAGE' but I get invalid path expression errors. I assume I don't have the syntax down right. I ultimately want to display key __REALTIME_TIMESTAMP in human readable format and key MESSAGE.
user168989
Nov 11, 2023, 09:43 PM • Last activity: Feb 4, 2025, 12:24 PM
4 votes
2 answers
8509 views
Convert json to csv with headers in jq
Is it possible to convert this json: ```json [ { "bytes": 276697, "checked": false }, { "bytes": 276697, "checked": false } ] ``` to a table WITH headers in jq? I've tried: ```sh cat file.json | jq '.[] | join(",")' ``` but it omits headers: ``` "276697,false" "276697,false" ``` it should be: ``` "b...
Is it possible to convert this json:
[
    {
        "bytes": 276697,
        "checked": false
    },
    {
        "bytes": 276697,
        "checked": false
    }
]
to a table WITH headers in jq? I've tried:
cat file.json | jq '.[] | join(",")'
but it omits headers:
"276697,false"
"276697,false"
it should be:
"bytes,checked"
"276697,false"
"276697,false"
I hoped that you can just run two commands:
cat file.json | jq '.[] | keys, .[] | join(",")'
but the second one fails:
"bytes,checked"
jq: error (at :64): Cannot iterate over null (null)
Ideally it would be simpler than this .
Daniel Krajnik (371 rep)
Aug 25, 2023, 05:23 PM • Last activity: Feb 3, 2025, 11:25 PM
Showing page 1 of 20 total questions