first commit
This commit is contained in:
parent
f56b2fcf8c
commit
b4f6a83da0
580
README.md
580
README.md
|
|
@ -1,290 +1,290 @@
|
|||
# Website Change Detection Monitor
|
||||
|
||||
A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering.
|
||||
|
||||
## 🚀 Features
|
||||
|
||||
### MVP (Current Implementation)
|
||||
- ✅ User authentication (register/login)
|
||||
- ✅ Monitor creation and management
|
||||
- ✅ Automatic page checking at configurable intervals
|
||||
- ✅ Change detection with hash comparison
|
||||
- ✅ Email alerts on changes
|
||||
- ✅ History timeline with snapshots
|
||||
- ✅ Smart noise filtering (timestamps, cookie banners)
|
||||
- ✅ Manual check triggering
|
||||
- ✅ Plan-based limits (free/pro/business)
|
||||
|
||||
### Coming Soon
|
||||
- Keyword-based alerts
|
||||
- Element-specific monitoring
|
||||
- Visual diff viewer
|
||||
- Slack/Discord integrations
|
||||
- AI-powered change summaries
|
||||
- Team collaboration
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
- Node.js 18+ and npm
|
||||
- PostgreSQL 15+
|
||||
- Redis 7+
|
||||
- Docker (optional, for local development)
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
### 1. Clone the repository
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd website-monitor
|
||||
```
|
||||
|
||||
### 2. Start database services (Docker)
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
This starts:
|
||||
- PostgreSQL on port 5432
|
||||
- Redis on port 6379
|
||||
|
||||
### 3. Setup Backend
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npm install
|
||||
|
||||
# Copy environment file
|
||||
cp .env.example .env
|
||||
# Edit .env with your settings
|
||||
|
||||
# Run migrations
|
||||
npm run migrate
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Backend will run on http://localhost:3001
|
||||
|
||||
### 4. Setup Frontend
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Frontend will run on http://localhost:3000
|
||||
|
||||
## 🗄️ Database Schema
|
||||
|
||||
The database includes:
|
||||
- `users` - User accounts and authentication
|
||||
- `monitors` - Website monitors configuration
|
||||
- `snapshots` - Historical page snapshots
|
||||
- `alerts` - Alert records
|
||||
|
||||
Run migrations:
|
||||
```bash
|
||||
cd backend
|
||||
npm run migrate
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Backend (.env)
|
||||
```env
|
||||
PORT=3001
|
||||
DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor
|
||||
REDIS_URL=redis://localhost:6379
|
||||
JWT_SECRET=your-secret-key
|
||||
SMTP_HOST=smtp.sendgrid.net
|
||||
SMTP_PORT=587
|
||||
SMTP_USER=apikey
|
||||
SMTP_PASS=your-api-key
|
||||
```
|
||||
|
||||
### Frontend (.env.local)
|
||||
```env
|
||||
NEXT_PUBLIC_API_URL=http://localhost:3001
|
||||
```
|
||||
|
||||
## 📖 Usage
|
||||
|
||||
### Register an Account
|
||||
1. Go to http://localhost:3000
|
||||
2. Click "Sign up"
|
||||
3. Enter email and password
|
||||
4. You'll be logged in automatically
|
||||
|
||||
### Create a Monitor
|
||||
1. Go to Dashboard
|
||||
2. Click "+ Add Monitor"
|
||||
3. Enter URL and select frequency
|
||||
4. Click "Create Monitor"
|
||||
5. First check happens immediately
|
||||
|
||||
### View History
|
||||
1. Click "History" on any monitor
|
||||
2. See timeline of all checks
|
||||
3. View changes and errors
|
||||
|
||||
## 🔑 API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `POST /api/auth/register` - Register new user
|
||||
- `POST /api/auth/login` - Login user
|
||||
|
||||
### Monitors (Requires Auth)
|
||||
- `GET /api/monitors` - List all monitors
|
||||
- `POST /api/monitors` - Create monitor
|
||||
- `GET /api/monitors/:id` - Get monitor details
|
||||
- `PUT /api/monitors/:id` - Update monitor
|
||||
- `DELETE /api/monitors/:id` - Delete monitor
|
||||
- `POST /api/monitors/:id/check` - Trigger manual check
|
||||
- `GET /api/monitors/:id/history` - Get check history
|
||||
|
||||
## 📊 Plan Limits
|
||||
|
||||
### Free Plan
|
||||
- 5 monitors
|
||||
- Minimum 60-minute frequency
|
||||
- 7-day history retention
|
||||
- Email alerts only
|
||||
|
||||
### Pro Plan
|
||||
- 50 monitors
|
||||
- Minimum 5-minute frequency
|
||||
- 90-day history retention
|
||||
- All alert channels
|
||||
|
||||
### Business Plan
|
||||
- 200 monitors
|
||||
- Minimum 1-minute frequency
|
||||
- 1-year history retention
|
||||
- API access
|
||||
- Team features
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
### Backend
|
||||
- **Express + TypeScript** - API server
|
||||
- **PostgreSQL** - Relational database
|
||||
- **Redis + BullMQ** - Job queue (coming soon)
|
||||
- **Nodemailer** - Email alerts
|
||||
- **Axios** - HTTP requests
|
||||
- **Cheerio** - HTML parsing
|
||||
- **Diff** - Change detection
|
||||
|
||||
### Frontend
|
||||
- **Next.js 14** - React framework
|
||||
- **TypeScript** - Type safety
|
||||
- **Tailwind CSS** - Styling
|
||||
- **React Query** - Data fetching
|
||||
- **Axios** - API client
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
```bash
|
||||
# Backend tests
|
||||
cd backend
|
||||
npm test
|
||||
|
||||
# Frontend tests
|
||||
cd frontend
|
||||
npm test
|
||||
```
|
||||
|
||||
## 📝 Development
|
||||
|
||||
### Running in Development
|
||||
```bash
|
||||
# Terminal 1: Start databases
|
||||
docker-compose up
|
||||
|
||||
# Terminal 2: Backend
|
||||
cd backend
|
||||
npm run dev
|
||||
|
||||
# Terminal 3: Frontend
|
||||
cd frontend
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Building for Production
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
npm run build
|
||||
npm start
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Backend Deployment
|
||||
- Deploy to Railway, Render, or AWS
|
||||
- Set environment variables
|
||||
- Run migrations
|
||||
- Start with `npm start`
|
||||
|
||||
### Frontend Deployment
|
||||
- Deploy to Vercel or Netlify
|
||||
- Set `NEXT_PUBLIC_API_URL`
|
||||
- Build with `npm run build`
|
||||
|
||||
### Database
|
||||
- Use managed PostgreSQL (AWS RDS, Railway, Supabase)
|
||||
- Run migrations before deploying
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Database connection error
|
||||
- Check PostgreSQL is running
|
||||
- Verify DATABASE_URL is correct
|
||||
- Run migrations: `npm run migrate`
|
||||
|
||||
### Frontend can't connect to backend
|
||||
- Check API_URL in .env.local
|
||||
- Ensure backend is running
|
||||
- Check CORS settings
|
||||
|
||||
### Email alerts not working
|
||||
- Configure SMTP settings in backend .env
|
||||
- For development, use a service like Mailtrap
|
||||
- For production, use SendGrid or similar
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
See `/docs` folder for:
|
||||
- `spec.md` - Complete feature specifications
|
||||
- `task.md` - Development roadmap
|
||||
- `actions.md` - User workflows
|
||||
- `claude.md` - Project context
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Commit your changes
|
||||
4. Push to the branch
|
||||
5. Open a Pull Request
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT License - see LICENSE file for details
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Built with:
|
||||
- Next.js
|
||||
- Express
|
||||
- PostgreSQL
|
||||
- Tailwind CSS
|
||||
- TypeScript
|
||||
# Website Change Detection Monitor
|
||||
|
||||
A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering.
|
||||
|
||||
## 🚀 Features
|
||||
|
||||
### MVP (Current Implementation)
|
||||
- ✅ User authentication (register/login)
|
||||
- ✅ Monitor creation and management
|
||||
- ✅ Automatic page checking at configurable intervals
|
||||
- ✅ Change detection with hash comparison
|
||||
- ✅ Email alerts on changes
|
||||
- ✅ History timeline with snapshots
|
||||
- ✅ Smart noise filtering (timestamps, cookie banners)
|
||||
- ✅ Manual check triggering
|
||||
- ✅ Plan-based limits (free/pro/business)
|
||||
|
||||
### Coming Soon
|
||||
- Keyword-based alerts
|
||||
- Element-specific monitoring
|
||||
- Visual diff viewer
|
||||
- Slack/Discord integrations
|
||||
- AI-powered change summaries
|
||||
- Team collaboration
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
- Node.js 18+ and npm
|
||||
- PostgreSQL 15+
|
||||
- Redis 7+
|
||||
- Docker (optional, for local development)
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
### 1. Clone the repository
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd website-monitor
|
||||
```
|
||||
|
||||
### 2. Start database services (Docker)
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
This starts:
|
||||
- PostgreSQL on port 5432
|
||||
- Redis on port 6379
|
||||
|
||||
### 3. Setup Backend
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npm install
|
||||
|
||||
# Copy environment file
|
||||
cp .env.example .env
|
||||
# Edit .env with your settings
|
||||
|
||||
# Run migrations
|
||||
npm run migrate
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Backend will run on http://localhost:3001
|
||||
|
||||
### 4. Setup Frontend
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Frontend will run on http://localhost:3000
|
||||
|
||||
## 🗄️ Database Schema
|
||||
|
||||
The database includes:
|
||||
- `users` - User accounts and authentication
|
||||
- `monitors` - Website monitors configuration
|
||||
- `snapshots` - Historical page snapshots
|
||||
- `alerts` - Alert records
|
||||
|
||||
Run migrations:
|
||||
```bash
|
||||
cd backend
|
||||
npm run migrate
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Backend (.env)
|
||||
```env
|
||||
PORT=3001
|
||||
DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor
|
||||
REDIS_URL=redis://localhost:6379
|
||||
JWT_SECRET=your-secret-key
|
||||
SMTP_HOST=smtp.sendgrid.net
|
||||
SMTP_PORT=587
|
||||
SMTP_USER=apikey
|
||||
SMTP_PASS=your-api-key
|
||||
```
|
||||
|
||||
### Frontend (.env.local)
|
||||
```env
|
||||
NEXT_PUBLIC_API_URL=http://localhost:3001
|
||||
```
|
||||
|
||||
## 📖 Usage
|
||||
|
||||
### Register an Account
|
||||
1. Go to http://localhost:3000
|
||||
2. Click "Sign up"
|
||||
3. Enter email and password
|
||||
4. You'll be logged in automatically
|
||||
|
||||
### Create a Monitor
|
||||
1. Go to Dashboard
|
||||
2. Click "+ Add Monitor"
|
||||
3. Enter URL and select frequency
|
||||
4. Click "Create Monitor"
|
||||
5. First check happens immediately
|
||||
|
||||
### View History
|
||||
1. Click "History" on any monitor
|
||||
2. See timeline of all checks
|
||||
3. View changes and errors
|
||||
|
||||
## 🔑 API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `POST /api/auth/register` - Register new user
|
||||
- `POST /api/auth/login` - Login user
|
||||
|
||||
### Monitors (Requires Auth)
|
||||
- `GET /api/monitors` - List all monitors
|
||||
- `POST /api/monitors` - Create monitor
|
||||
- `GET /api/monitors/:id` - Get monitor details
|
||||
- `PUT /api/monitors/:id` - Update monitor
|
||||
- `DELETE /api/monitors/:id` - Delete monitor
|
||||
- `POST /api/monitors/:id/check` - Trigger manual check
|
||||
- `GET /api/monitors/:id/history` - Get check history
|
||||
|
||||
## 📊 Plan Limits
|
||||
|
||||
### Free Plan
|
||||
- 5 monitors
|
||||
- Minimum 60-minute frequency
|
||||
- 7-day history retention
|
||||
- Email alerts only
|
||||
|
||||
### Pro Plan
|
||||
- 50 monitors
|
||||
- Minimum 5-minute frequency
|
||||
- 90-day history retention
|
||||
- All alert channels
|
||||
|
||||
### Business Plan
|
||||
- 200 monitors
|
||||
- Minimum 1-minute frequency
|
||||
- 1-year history retention
|
||||
- API access
|
||||
- Team features
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
### Backend
|
||||
- **Express + TypeScript** - API server
|
||||
- **PostgreSQL** - Relational database
|
||||
- **Redis + BullMQ** - Job queue (coming soon)
|
||||
- **Nodemailer** - Email alerts
|
||||
- **Axios** - HTTP requests
|
||||
- **Cheerio** - HTML parsing
|
||||
- **Diff** - Change detection
|
||||
|
||||
### Frontend
|
||||
- **Next.js 14** - React framework
|
||||
- **TypeScript** - Type safety
|
||||
- **Tailwind CSS** - Styling
|
||||
- **React Query** - Data fetching
|
||||
- **Axios** - API client
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
```bash
|
||||
# Backend tests
|
||||
cd backend
|
||||
npm test
|
||||
|
||||
# Frontend tests
|
||||
cd frontend
|
||||
npm test
|
||||
```
|
||||
|
||||
## 📝 Development
|
||||
|
||||
### Running in Development
|
||||
```bash
|
||||
# Terminal 1: Start databases
|
||||
docker-compose up
|
||||
|
||||
# Terminal 2: Backend
|
||||
cd backend
|
||||
npm run dev
|
||||
|
||||
# Terminal 3: Frontend
|
||||
cd frontend
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Building for Production
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
npm run build
|
||||
npm start
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Backend Deployment
|
||||
- Deploy to Railway, Render, or AWS
|
||||
- Set environment variables
|
||||
- Run migrations
|
||||
- Start with `npm start`
|
||||
|
||||
### Frontend Deployment
|
||||
- Deploy to Vercel or Netlify
|
||||
- Set `NEXT_PUBLIC_API_URL`
|
||||
- Build with `npm run build`
|
||||
|
||||
### Database
|
||||
- Use managed PostgreSQL (AWS RDS, Railway, Supabase)
|
||||
- Run migrations before deploying
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Database connection error
|
||||
- Check PostgreSQL is running
|
||||
- Verify DATABASE_URL is correct
|
||||
- Run migrations: `npm run migrate`
|
||||
|
||||
### Frontend can't connect to backend
|
||||
- Check API_URL in .env.local
|
||||
- Ensure backend is running
|
||||
- Check CORS settings
|
||||
|
||||
### Email alerts not working
|
||||
- Configure SMTP settings in backend .env
|
||||
- For development, use a service like Mailtrap
|
||||
- For production, use SendGrid or similar
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
See `/docs` folder for:
|
||||
- `spec.md` - Complete feature specifications
|
||||
- `task.md` - Development roadmap
|
||||
- `actions.md` - User workflows
|
||||
- `claude.md` - Project context
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Commit your changes
|
||||
4. Push to the branch
|
||||
5. Open a Pull Request
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT License - see LICENSE file for details
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Built with:
|
||||
- Next.js
|
||||
- Express
|
||||
- PostgreSQL
|
||||
- Tailwind CSS
|
||||
- TypeScript
|
||||
|
|
|
|||
Loading…
Reference in New Issue