A full-stack web application that allows users to enter a URL and download the website's cookies as a .csv file. The frontend is hosted on GitHub Pages, and the backend is a containerized Flask and Selenium service deployed on Render.
- Frontend Interface: https://melogabriel.github.io/cookie-scraper/
- Backend Health Check: https://cookie-scrapper.onrender.com/
Note: The backend is hosted on Render's free tier and will "go to sleep" after 15 minutes of inactivity. The first request may take up to a minute to "wake up" the service.
- User-Friendly Web Interface: A clean, simple frontend for entering a target URL.
- Server-Side Scraping: Utilizes a powerful backend to handle browser automation.
- Headless Chrome Automation: Uses Selenium to run a headless Chrome instance in a Docker container.
- CSV Export: Dynamically generates and serves a
.csvfile of the captured cookies. - Robust Error Handling: Provides clear feedback to the user if the server times out or encounters an error.
- Containerized & Scalable: Uses Docker for a consistent and reproducible deployment environment.
- Backend: Python, Flask, Gunicorn
- Web Scraping: Selenium, webdriver-manager
- Data Handling: pandas
- Frontend: HTML, Tailwind CSS, JavaScript (
fetchAPI) - Deployment: Docker, GitHub Pages, Render
This project uses a client-server model, separating the user interface from the heavy processing work.
- Frontend (Client): A static
index.htmlfile hosted on GitHub Pages. It captures the user's input and uses JavaScript to make an API call to the backend. - Backend (Server): A Flask application running inside a Docker container on Render. It exposes a
/scrapeAPI endpoint that receives a URL, launches a headless Chrome browser using Selenium, scrapes the cookies, and sends back a CSV file.
To deploy your own version of this application, follow these steps:
-
Fork this Repository: Click the "Fork" button at the top right of this page to create your own copy.
-
Deploy the Backend to Render: a. Create a new "Web Service" on Render and connect it to your forked repository. b. Crucially, set the "Environment" to
Docker. c. Render will automatically detect yourDockerfile. Use the default settings for the service name and branch. d. Click "Create Web Service". The first build may take several minutes as it installs Google Chrome inside the container. -
Configure GitHub Pages for the Frontend: a. In your forked repository, go to Settings > Pages. b. Under "Build and deployment," select the source as
Deploy from a branch. c. Set the branch tomainand the folder to/ (root). Click Save. -
Connect Frontend to Backend: a. Once your Render service is live, copy its URL (e.g.,
https://your-app-name.onrender.com). b. In your GitHub repository, edit theindex.htmlfile. c. Find theapiUrlvariable in the<script>section and replace the placeholder URL with your live Render URL.// Change this line in index.html const apiUrl = 'https://your-app-name.onrender.com/scrape';
d. Commit and push this change. Your GitHub Pages site will update automatically.
"Network Error: Could not connect to the server." This is the most common issue. It means your browser timed out waiting for a response.
- Cause: The free server on Render "goes to sleep." The first request has to wake it up, which can take over 30 seconds.
- Solution: First, visit your backend URL (e.g.,
https://your-app-name.onrender.com/) to wake the server up. Once you see the{"status":"healthy"}message, go back to your frontend and try again.
"Error: Could not process the URL..." This means the backend started but crashed while running Selenium.
- Cause: The target website might be slow, or there could be an issue with the Chrome driver.
- Solution: Check the "Logs" tab for your service on Render. The traceback will provide details on the specific error. The backend code includes a 120-second timeout to handle most cases.
This project is licensed under the MIT License. See the LICENSE file for details.
