Wilfred Ng

Twitter Web Scraper

📖 Table of Contents

Table of Contents

1. ➤ About The Project
2. ➤ Dependencies
3. ➤ Folder Structure

📝 About The Project

This project implements a twitter web scraper that automates the login process and scraping of twitter cards. It stores the collected information into a .CSV file which can be used for analysis.

🌵 Dependencies

• if you're using chrome - https://chromedriver.chromium.org/downloads
• if you're using edge - https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

pip install selenium

Note:

1. the chromedriver.exe might not match your chrome's version, if it does not match, please download the appropriate chrome driver from the link above
2. the lines of code below can be commented out if you did not encounter an additional username verification on twitter

# verification of username 
verify_username = WebDriverWait(chrome_driver, 10).until(
    EC.element_to_be_clickable((By.CSS_SELECTOR, '[name="text"]'))
)
verify_username.send_keys('twitterUsername')
verify_login_button = chrome_driver.find_element(By.XPATH,'/html/body/div[1]/div/div/div[1]/div/div/div/div/div/div/div[2]/div[2]/div/div/div[2]/div[2]/div[2]/div/div/div/div')

# Wait for 10 seconds before clicking the login button
chrome_driver.execute_script("arguments[0].click();", verify_login_button)

3. Remember to change the usernameEmail , password and twitterUsername to your own respective login details.

🔷 How To Run

python TwitterScraper.py