Add account data export scripts based on osc-gif scripts

osc-export/README.md

@@ -0,0 +1,20 @@
+# OSC track export
+
+Scripts that export all tracks, pictures, and data associated with them (except for road sign detections) from an account on OpenStreetCam.
+
+**Tutorial for how to use them is below.**
+
+Log into your account to access unprocessed tracks (if you have uploaded since ~2/16/2020 and need information from your tracks).
+
+Big thanks to the original authors of the osc-gif scripts, those helped a lot.
+
+## Tutorial
+
+1. Figure out the access token when you're logged into your OSC account. In Firefox, you can do that by logging into your OSC account, going to the network tab, then finding the request to `api.openstreetcam.org/1.0/user/details`. Go to Params on the right hand side and copy the access token:
+![Network tab](tutorial-image-ff-network-tab.png)
+
+Put that in `osc-export-track.sh`.
+
+2. In Firefox, while you're logged into your OSC account, in the console, run `[...document.links].map(l => l.href).filter((a_url) => a_url.startsWith("https://openstreetcam.org/details")).toString()` and put the output (including quotes in the URLs) in the `URLs` variable in `osc-export-account.sh`. (An example is provided in that file.)
+
+3. In a terminal, type `chmod +x osc-export-account.sh` and run `./osc-export-account.sh`. Wait about 2 hours for the whole thing to finish depending on your internet speed and how many images you have uploaded to your account.

osc-export/osc-export-account.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+ # When logged into OSC account, in a JavaScript console, run:
+ # [...document.links].map(l => l.href).filter((a_url) => a_url.startsWith("https://openstreetcam.org/details")).toString()
+ # (used startsWith to remove instances of javascript:void(0)) to get this:
+
+ # (this can be automated by doing cURL and extracting the browser cookie or API access token in the header to get both processed and unprocessed tracks for the logged in  account)
+
+ # Result (output):
+ # "https://openstreetcam.org/details/2084134,https://openstreetcam.org/details/2084130,[...]"
+ # Paste the URLs you get, with the quotes at the beginning and end, in this variable (an example is shown below):
+
+ #URLs="https://openstreetcam.org/details/2084134,https://openstreetcam.org/details/2084130,https://openstreetcam.org/details/123456"
+
+ URLs=""
+
+ # What the rest of this script does:
+ # find and replace https://openstreetcam.org/details/ to nothing once to get track IDs:
+ # 2084134,2084130,[...]
+
+ # find and replace again to get spaces instead of commas:
+ # 2084134 2084130 [...]
+
+ TracksSpaceSepd="${URLs//https:\/\/openstreetcam.org\/details\//}"
+ TracksSpaceSepd="${TracksSpaceSepd//,/ }"
+
+ for track in $TracksSpaceSepd; do
+    ./osc-export-track-into-folder.sh $track;
+ done

osc-export/osc-export-track-into-folder.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+./osc-get-raw-images.sh $1
+mkdir $1/
+mv *.jpg $1/
+# the collected JSON files containing the image data can also go inside the track folder
+mv osc_$1.json $1/

osc-export/osc-export-track.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+url="https://openstreetcam.org/details"
+# need to log in to account to get the access token and view "unproccessed" OSC tracks.
+# find the access token (in params tab of network in firefox) for api.openstreetcam.org (looking for api.openstreetcam.org is OK because the request is just getting the list of images from openstreetcam.org, and all it needs is some API key in the header). Replace ACCESS_TOKEN_HERE with the hex string that you found.
+header="access_token=ACCESS_TOKEN_HERE&id=$1"
+# also gets the JSON data of more data (such as elevation etc) collected in the images in case you want to view it (interesting)
+curl $url --data $header -o osc_$1.json
+# download full quality images on the server (not blurry thumbnails) with 100 threads for fast processing
+cat osc_$1.json | jq ".osv.photos[].name" | xargs -I{} echo "www.openstreetcam.org/"{} >>urls_$1.txt 
+cat urls_$1.txt | xargs -n 1 -P 100 wget
+# to avoid redownloading the same image URLs if we re-run this script with the same track ID.
+mv urls_$1.txt urls_$1_.txt