JSON parsing tips & tricks
Parsing large unknown JSON files
“Let’s say I want a list of usernames of all the people who starred a project on Github via Github API.”
- Let’s call the API using curl and find the number of elements in the JSON returned
curl "https://api.github.com/repos/tomnomnom/gron/stargazers" -s | jq length
The JSON is farly large. Let’s get to parsing it.
- Use Gron to understand the structure of the JSON file.
gron "https://api.github.com/repos/tomnomnom/gron/stargazers" -s
json = [];
json[0] = {};
json[0].avatar_url = "https://avatars.githubusercontent.com/u/369020?v=4";
json[0].events_url = "https://api.github.com/users/iamthemovie/events{/privacy}";
json[0].followers_url = "https://api.github.com/users/iamthemovie/followers";
json[0].following_url = "https://api.github.com/users/iamthemovie/following{/other_user}";
json[0].gists_url = "https://api.github.com/users/iamthemovie/gists{/gist_id}";
json[0].gravatar_id = "";
json[0].html_url = "https://github.com/iamthemovie";
json[0].id = 369020;
json[0].login = "iamthemovie";
The login field seems to have the username of the stargazer. Let’s now use the output of gron with jq to extract all the usernames.
This is the beauty of Gron. It makes JSON greppable and also it gives you a flat visual structure of what the JSON looks like. You can simply copy paste the flat representation of a specific element to get the exact elements and their values in large JSON files
- Replace the
jsonkeyword with. - Remove the index numerical to get all the usernames
curl "https://api.github.com/repos/tomnomnom/gron/stargazers" -s | jq ".[].login"
Let’s do a similar activity. Using Known Exploited Vulnerabilities Catalog JSON, list the names of all the vendors whose products are being attacked actively in the wild.
gron https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
curl https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json | jq ".vulnerabilities[].vendorProject" | sort -u