Skip to content

Commit 6cde028

Browse files
committed
Prevent newlines in JSONL
If a set of 100 ids are hydrated and not one of them is still available the Hydrator was writing a blank line to the jsonl file. This could cause problems for downstream users of the jsonl that are expecting each line to contain a JSON object. Refs #48
1 parent 40bc882 commit 6cde028

File tree

2 files changed

+5
-3
lines changed

2 files changed

+5
-3
lines changed

app/utils/twitter.js

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,10 @@ export async function getUserSettings(auth) {
5656
export async function hydrateToStream(ids, out, auth, event, datasetId, tries=0) {
5757
try {
5858
const tweets = await fetchTweets(ids, auth)
59-
const text = tweets.map(t => JSON.stringify(t)).join('\n')
60-
out.write(text + "\n")
59+
if (tweets.length > 0) {
60+
const text = tweets.map(t => JSON.stringify(t)).join('\n')
61+
out.write(text + "\n")
62+
}
6163
event.sender.send(UPDATE_PROGRESS, {
6264
datasetId: datasetId,
6365
idsRead: ids.length,

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "hydrator",
3-
"version": "0.0.11",
3+
"version": "0.0.12",
44
"description": "A desktop utility to hydrate Twitter ID datasets.",
55
"main": "init.js",
66
"author": {

0 commit comments

Comments
 (0)