📊 Real-Time Cryptocurrency Data Pipeline

End-to-end data pipeline for real-time cryptocurrency market intelligence

---

📌 Project Overview

This project implements an end-to-end data pipeline for real-time cryptocurrency data. It scrapes cryptocurrency market data every 15 minutes, processes and loads it into PostgreSQL, transforms it with DBT, and visualizes trends in Power BI dashboards.

---

🏗️ Architecture

---

⚡ Pipeline Flow

1. Web Scraping (Python + BeautifulSoup)

• Scrapes title, prefix, price, 24h change, volume, market cap
• Runs every 15 minutes
• Cleans values ($, %, M, B) → numeric format

2. Data Storage (PostgreSQL)

• crypto_currencies → latest snapshot
• crypto_currencies_old → previous snapshot
• Ensures historical tracking of prices

3. Transformations (DBT)

• Combines crypto_currencies + crypto_currencies_old
• Produces a view with:
  • Current Price
  • Maximum Price
  • Minimum Price
  • Market Cap, Volume, 24h Change

4. Visualization (Power BI)

• Connects to DBT view
• Dashboards include:
  • Current market snapshot
  • Historical price highs/lows
  • Trends over time

---

🚀 How to Run on Local Machine

Prerequisites

• Docker installed with proper configurations
• PostgreSQL database setup

Setup Instructions

1. Database Setup: Create PostgreSQL database using create_schema.sql file
2. Docker Configuration: Ensure all files and folders are mounted in Docker with correct configs
3. AI Assistant: Share the DAG.py file with any AI for detailed instructions
4. Testing Only: To test scraping mechanism only, run the Scrape.py script

---

📊 Power BI Dashboard

Dashboard Features

• Snapshot View → Current market prices, volumes, caps
• Trend Analysis → Min/max price tracking across time

---

🎥 Project Demo

Click to watch the complete project walkthrough

---

📈 Project Outcomes

Achievement	Description
🔄 Automated Pipeline	Built end-to-end automated data processing system
⏰ Real-time Monitoring	Enables continuous cryptocurrency market tracking
📚 Historical Data	Preserves complete price history for trend analysis
📊 Business Intelligence	Provides insightful dashboards for decision-making

---

Built with modern data engineering practices for cryptocurrency market intelligence