twiffle

Tracking of twitter terms to make a funny raffle

twiffle comes from twitter + raffle. It's a service written in Python to track twitter terms (mainly hashtags) in order to retrieve matching tweets and, afterwards, extract unique usernames, perhaps for a funny raffle 🎉.

Installation

$ git clone https://github.com/sdelquin/twiffle.git
$ cd twiffle
$ python3.9 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Configuration

A .env file must be present for twiffle to run properly. Write your Twitter API credentials here:

API_CONSUMER_KEY = '<your-api-token-here>'
API_CONSUMER_SECRET = '<your-api-token-here>'
API_ACCESS_TOKEN = '<your-api-token-here>'
API_ACCESS_TOKEN_SECRET = '<your-api-token-here>'

Manage here your Twitter API Tokens.

Settings

twiffle expects to have a data/settings.yml file where the behaviour of the service is configured.

Example

label: tabularconf
track:
  keywords:
    - "#TabularConf"
    - "#TabularConf2021"
dump_users:
  books:
    since: "2021-01-25 07:00:00"
    until: "2021-01-29 22:59:00"
    retweets: false
    excluded_users:
      - "@TabularConf"
      - sdelquin
    must_include:
      - "#SuperGift"
      - "@pandas_dev"
  rpi:
    must_include:
      - "#iwantrpi"
      - "@Raspberry_Pi"

• label: Give a unique name for the whole configuration. *
• track: Tracking service settings. *
  • keywords: List of keywords to be tracked. It's an "OR" among all these terms. Hashtags and accounts must include double quotes. *
• dump_users: Settings when dumping users. It contains "blocks" to define dumping features:
  • since: Users will be extracted since this value. Datetime (as string) in ISO-format. It must include quotes. [Default: beginning of time]
  • until: Users will be extracted until this value. Datetime (as string) in ISO-format. It must include quotes. [Default: current datetime]
  • retweets: Boolean value indicating if retweets are included in the dump. [Default: true]
  • excluded_users: List of Twitter usernames to be excluded on dump. If @ is prefixed, the text must be double-quoted. [Default: empty list]
  • must_include: List of terms which the matching tweets must include. It's an "AND" among all these terms. If hashtags or accounts are added, they must be double-quoted. [Default: empty list]

* Required fields.

Usage

Tracking service

To run the tracking service, use the following command:

$ ./main.py track

It will use the track section defined in data/settings.yml to read the proper parameters. Output will be a sqlite file in data/tabularconf.db.

You can provide a custom settings file using:

$ ./main.py track -c custom-settings.yml

Dumping users

To dump unique usernames from captured tweets (tracking service), use the following command:

$ ./main.py dump-users

It will use the dump_users section in data/settings.yml to read the proper parameters.

You can provide a custom settings file using:

$ ./main.py dump-users -c custom-settings.yml

DUMP SINGLE BLOCK

Instead of dumping all the existing blocks from settings file, you can dump a single one using:

$ ./main.py dump-users books

DUMP TO FILE

Instead of dumping users to stdout, you can redirect them to output files:

$ ./main.py dump-users -f

It will use the dump_users section in data/settings.yml to read the proper parameters. Output will create these two files:

• data/tabularconf-books.dump
• data/tabularconf-rpi.dump

A single block can be passed to the command.

Dumping database

To dump the whole contents of table (database), use the following command:

$ ./main.py dump-db

✨  A beautiful table is shown!

You can slice results with the -s option:

$ ./main.py dump-db -s 10  # show first 10 records
$ ./main.py dump-db -s -7  # show last 7 records

You can also provide a custom settings file using:

$ ./main.py dump-db -c custom-settings.yml

Logging

Operations performed by ./main.py track are logged to twiffle.log with a default file rotation.

Logging behaviour can be overwrite using the following keys in the .env file:

• LOGFILE_NAME: filename of the logfile. [Default: twiffle.log]
• LOGFILE_ROTATION: size of the logfile. [Default: 10MB]
• LOGFILE_RETENTION: number of rotating logfiles. [Default: 5]