tknuth
/
website-monitor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

first commit

This commit is contained in:
knuthtimo-lab 2026-01-19 08:31:06 +01:00
parent f56b2fcf8c
commit b4f6a83da0
1 changed files with 290 additions and 290 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

580
README.md
View File

@ -1,290 +1,290 @@
# Website Change Detection Monitor # Website Change Detection Monitor
A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering. A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering.
## 🚀 Features ## 🚀 Features
### MVP (Current Implementation) ### MVP (Current Implementation)
- ✅ User authentication (register/login) - ✅ User authentication (register/login)
- ✅ Monitor creation and management - ✅ Monitor creation and management
- ✅ Automatic page checking at configurable intervals - ✅ Automatic page checking at configurable intervals
- ✅ Change detection with hash comparison - ✅ Change detection with hash comparison
- ✅ Email alerts on changes - ✅ Email alerts on changes
- ✅ History timeline with snapshots - ✅ History timeline with snapshots
- ✅ Smart noise filtering (timestamps, cookie banners) - ✅ Smart noise filtering (timestamps, cookie banners)
- ✅ Manual check triggering - ✅ Manual check triggering
- ✅ Plan-based limits (free/pro/business) - ✅ Plan-based limits (free/pro/business)
### Coming Soon ### Coming Soon
- Keyword-based alerts - Keyword-based alerts
- Element-specific monitoring - Element-specific monitoring
- Visual diff viewer - Visual diff viewer
- Slack/Discord integrations - Slack/Discord integrations
- AI-powered change summaries - AI-powered change summaries
- Team collaboration - Team collaboration
## 📋 Prerequisites ## 📋 Prerequisites
- Node.js 18+ and npm - Node.js 18+ and npm
- PostgreSQL 15+ - PostgreSQL 15+
- Redis 7+ - Redis 7+
- Docker (optional, for local development) - Docker (optional, for local development)
## 🛠️ Installation ## 🛠️ Installation
### 1. Clone the repository ### 1. Clone the repository
```bash ```bash
git clone <repository-url> git clone <repository-url>
cd website-monitor cd website-monitor
``` ```
### 2. Start database services (Docker) ### 2. Start database services (Docker)
```bash ```bash
docker-compose up -d docker-compose up -d
``` ```
This starts: This starts:
- PostgreSQL on port 5432 - PostgreSQL on port 5432
- Redis on port 6379 - Redis on port 6379
### 3. Setup Backend ### 3. Setup Backend
```bash ```bash
cd backend cd backend
npm install npm install
# Copy environment file # Copy environment file
cp .env.example .env cp .env.example .env
# Edit .env with your settings # Edit .env with your settings
# Run migrations # Run migrations
npm run migrate npm run migrate
# Start development server # Start development server
npm run dev npm run dev
``` ```
Backend will run on http://localhost:3001 Backend will run on http://localhost:3001
### 4. Setup Frontend ### 4. Setup Frontend
```bash ```bash
cd frontend cd frontend
npm install npm install
# Start development server # Start development server
npm run dev npm run dev
``` ```
Frontend will run on http://localhost:3000 Frontend will run on http://localhost:3000
## 🗄️ Database Schema ## 🗄️ Database Schema
The database includes: The database includes:
- `users` - User accounts and authentication - `users` - User accounts and authentication
- `monitors` - Website monitors configuration - `monitors` - Website monitors configuration
- `snapshots` - Historical page snapshots - `snapshots` - Historical page snapshots
- `alerts` - Alert records - `alerts` - Alert records
Run migrations: Run migrations:
```bash ```bash
cd backend cd backend
npm run migrate npm run migrate
``` ```
## 🔧 Configuration ## 🔧 Configuration
### Backend (.env) ### Backend (.env)
```env ```env
PORT=3001 PORT=3001
DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor
REDIS_URL=redis://localhost:6379 REDIS_URL=redis://localhost:6379
JWT_SECRET=your-secret-key JWT_SECRET=your-secret-key
SMTP_HOST=smtp.sendgrid.net SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587 SMTP_PORT=587
SMTP_USER=apikey SMTP_USER=apikey
SMTP_PASS=your-api-key SMTP_PASS=your-api-key
``` ```
### Frontend (.env.local) ### Frontend (.env.local)
```env ```env
NEXT_PUBLIC_API_URL=http://localhost:3001 NEXT_PUBLIC_API_URL=http://localhost:3001
``` ```
## 📖 Usage ## 📖 Usage
### Register an Account ### Register an Account
1. Go to http://localhost:3000 1. Go to http://localhost:3000
2. Click "Sign up" 2. Click "Sign up"
3. Enter email and password 3. Enter email and password
4. You'll be logged in automatically 4. You'll be logged in automatically
### Create a Monitor ### Create a Monitor
1. Go to Dashboard 1. Go to Dashboard
2. Click "+ Add Monitor" 2. Click "+ Add Monitor"
3. Enter URL and select frequency 3. Enter URL and select frequency
4. Click "Create Monitor" 4. Click "Create Monitor"
5. First check happens immediately 5. First check happens immediately
### View History ### View History
1. Click "History" on any monitor 1. Click "History" on any monitor
2. See timeline of all checks 2. See timeline of all checks
3. View changes and errors 3. View changes and errors
## 🔑 API Endpoints ## 🔑 API Endpoints
### Authentication ### Authentication
- `POST /api/auth/register` - Register new user - `POST /api/auth/register` - Register new user
- `POST /api/auth/login` - Login user - `POST /api/auth/login` - Login user
### Monitors (Requires Auth) ### Monitors (Requires Auth)
- `GET /api/monitors` - List all monitors - `GET /api/monitors` - List all monitors
- `POST /api/monitors` - Create monitor - `POST /api/monitors` - Create monitor
- `GET /api/monitors/:id` - Get monitor details - `GET /api/monitors/:id` - Get monitor details
- `PUT /api/monitors/:id` - Update monitor - `PUT /api/monitors/:id` - Update monitor
- `DELETE /api/monitors/:id` - Delete monitor - `DELETE /api/monitors/:id` - Delete monitor
- `POST /api/monitors/:id/check` - Trigger manual check - `POST /api/monitors/:id/check` - Trigger manual check
- `GET /api/monitors/:id/history` - Get check history - `GET /api/monitors/:id/history` - Get check history
## 📊 Plan Limits ## 📊 Plan Limits
### Free Plan ### Free Plan
- 5 monitors - 5 monitors
- Minimum 60-minute frequency - Minimum 60-minute frequency
- 7-day history retention - 7-day history retention
- Email alerts only - Email alerts only
### Pro Plan ### Pro Plan
- 50 monitors - 50 monitors
- Minimum 5-minute frequency - Minimum 5-minute frequency
- 90-day history retention - 90-day history retention
- All alert channels - All alert channels
### Business Plan ### Business Plan
- 200 monitors - 200 monitors
- Minimum 1-minute frequency - Minimum 1-minute frequency
- 1-year history retention - 1-year history retention
- API access - API access
- Team features - Team features
## 🏗️ Architecture ## 🏗️ Architecture
### Backend ### Backend
- **Express + TypeScript** - API server - **Express + TypeScript** - API server
- **PostgreSQL** - Relational database - **PostgreSQL** - Relational database
- **Redis + BullMQ** - Job queue (coming soon) - **Redis + BullMQ** - Job queue (coming soon)
- **Nodemailer** - Email alerts - **Nodemailer** - Email alerts
- **Axios** - HTTP requests - **Axios** - HTTP requests
- **Cheerio** - HTML parsing - **Cheerio** - HTML parsing
- **Diff** - Change detection - **Diff** - Change detection
### Frontend ### Frontend
- **Next.js 14** - React framework - **Next.js 14** - React framework
- **TypeScript** - Type safety - **TypeScript** - Type safety
- **Tailwind CSS** - Styling - **Tailwind CSS** - Styling
- **React Query** - Data fetching - **React Query** - Data fetching
- **Axios** - API client - **Axios** - API client
## 🧪 Testing ## 🧪 Testing
```bash ```bash
# Backend tests # Backend tests
cd backend cd backend
npm test npm test
# Frontend tests # Frontend tests
cd frontend cd frontend
npm test npm test
``` ```
## 📝 Development ## 📝 Development
### Running in Development ### Running in Development
```bash ```bash
# Terminal 1: Start databases # Terminal 1: Start databases
docker-compose up docker-compose up
# Terminal 2: Backend # Terminal 2: Backend
cd backend cd backend
npm run dev npm run dev
# Terminal 3: Frontend # Terminal 3: Frontend
cd frontend cd frontend
npm run dev npm run dev
``` ```
### Building for Production ### Building for Production
```bash ```bash
# Backend # Backend
cd backend cd backend
npm run build npm run build
npm start npm start
# Frontend # Frontend
cd frontend cd frontend
npm run build npm run build
npm start npm start
``` ```
## 🚀 Deployment ## 🚀 Deployment
### Backend Deployment ### Backend Deployment
- Deploy to Railway, Render, or AWS - Deploy to Railway, Render, or AWS
- Set environment variables - Set environment variables
- Run migrations - Run migrations
- Start with `npm start` - Start with `npm start`
### Frontend Deployment ### Frontend Deployment
- Deploy to Vercel or Netlify - Deploy to Vercel or Netlify
- Set `NEXT_PUBLIC_API_URL` - Set `NEXT_PUBLIC_API_URL`
- Build with `npm run build` - Build with `npm run build`
### Database ### Database
- Use managed PostgreSQL (AWS RDS, Railway, Supabase) - Use managed PostgreSQL (AWS RDS, Railway, Supabase)
- Run migrations before deploying - Run migrations before deploying
## 🐛 Troubleshooting ## 🐛 Troubleshooting
### Database connection error ### Database connection error
- Check PostgreSQL is running - Check PostgreSQL is running
- Verify DATABASE_URL is correct - Verify DATABASE_URL is correct
- Run migrations: `npm run migrate` - Run migrations: `npm run migrate`
### Frontend can't connect to backend ### Frontend can't connect to backend
- Check API_URL in .env.local - Check API_URL in .env.local
- Ensure backend is running - Ensure backend is running
- Check CORS settings - Check CORS settings
### Email alerts not working ### Email alerts not working
- Configure SMTP settings in backend .env - Configure SMTP settings in backend .env
- For development, use a service like Mailtrap - For development, use a service like Mailtrap
- For production, use SendGrid or similar - For production, use SendGrid or similar
## 📚 Documentation ## 📚 Documentation
See `/docs` folder for: See `/docs` folder for:
- `spec.md` - Complete feature specifications - `spec.md` - Complete feature specifications
- `task.md` - Development roadmap - `task.md` - Development roadmap
- `actions.md` - User workflows - `actions.md` - User workflows
- `claude.md` - Project context - `claude.md` - Project context
## 🤝 Contributing ## 🤝 Contributing
1. Fork the repository 1. Fork the repository
2. Create a feature branch 2. Create a feature branch
3. Commit your changes 3. Commit your changes
4. Push to the branch 4. Push to the branch
5. Open a Pull Request 5. Open a Pull Request
## 📄 License ## 📄 License
MIT License - see LICENSE file for details MIT License - see LICENSE file for details
## 🙏 Acknowledgments ## 🙏 Acknowledgments
Built with: Built with:
- Next.js - Next.js
- Express - Express
- PostgreSQL - PostgreSQL
- Tailwind CSS - Tailwind CSS
- TypeScript - TypeScript