website-monitor/README.md

# Website Change Detection Monitor

A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering.

## 🚀 Features

### MVP (Current Implementation)
- ✅ User authentication (register/login)
- ✅ Monitor creation and management
- ✅ Automatic page checking at configurable intervals
- ✅ Change detection with hash comparison
- ✅ Email alerts on changes
- ✅ History timeline with snapshots
- ✅ Smart noise filtering (timestamps, cookie banners)
- ✅ Manual check triggering
- ✅ Plan-based limits (free/pro/business)

### Coming Soon
- Keyword-based alerts
- Element-specific monitoring
- Visual diff viewer
- Slack/Discord integrations
- AI-powered change summaries
- Team collaboration

## 📋 Prerequisites

- Node.js 18+ and npm
- PostgreSQL 15+
- Redis 7+
- Docker (optional, for local development)

## 🛠️ Installation

### 1. Clone the repository
```bash
git clone <repository-url>
cd website-monitor
```

### 2. Start database services (Docker)
```bash
docker-compose up -d
```

This starts:
- PostgreSQL on port 5432
- Redis on port 6379

### 3. Setup Backend

```bash
cd backend
npm install

# Copy environment file
cp .env.example .env
# Edit .env with your settings

# Run migrations
npm run migrate

# Start development server
npm run dev
```

Backend will run on http://localhost:3001

### 4. Setup Frontend

```bash
cd frontend
npm install

# Start development server
npm run dev
```

Frontend will run on http://localhost:3000

## 🗄️ Database Schema

The database includes:
- `users` - User accounts and authentication
- `monitors` - Website monitors configuration
- `snapshots` - Historical page snapshots
- `alerts` - Alert records

Run migrations:
```bash
cd backend
npm run migrate
```

## 🔧 Configuration

### Backend (.env)
```env
PORT=3001
DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor
REDIS_URL=redis://localhost:6379
JWT_SECRET=your-secret-key
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASS=your-api-key
LANDING_ONLY_MODE=false
ADMIN_PASSWORD=change-me-for-admin-waitlist
```

### Frontend (.env.local)
```env
NEXT_PUBLIC_API_URL=http://localhost:3001
NEXT_PUBLIC_LANDING_ONLY_MODE=false
```

### Landing-only mode (Waitlist rollout)

To expose only landing pages and waitlist APIs, enable both flags:

- Backend `.env`: `LANDING_ONLY_MODE=true`
- Frontend `.env.local`: `NEXT_PUBLIC_LANDING_ONLY_MODE=true`

When enabled:
- Public pages: `/`, `/blog`, `/privacy`, `/admin`
- All other frontend routes redirect to `/` with HTTP `307`
- Backend only allows `/api/waitlist/*`, `POST /api/tools/meta-preview`, and `/health`

## 📖 Usage

### Register an Account
1. Go to http://localhost:3000
2. Click "Sign up"
3. Enter email and password
4. You'll be logged in automatically

### Create a Monitor
1. Go to Dashboard
2. Click "+ Add Monitor"
3. Enter URL and select frequency
4. Click "Create Monitor"
5. First check happens immediately

### View History
1. Click "History" on any monitor
2. See timeline of all checks
3. View changes and errors

## 🔑 API Endpoints

### Authentication
- `POST /api/auth/register` - Register new user
- `POST /api/auth/login` - Login user

### Monitors (Requires Auth)
- `GET /api/monitors` - List all monitors
- `POST /api/monitors` - Create monitor
- `GET /api/monitors/:id` - Get monitor details
- `PUT /api/monitors/:id` - Update monitor
- `DELETE /api/monitors/:id` - Delete monitor
- `POST /api/monitors/:id/check` - Trigger manual check
- `GET /api/monitors/:id/history` - Get check history

## 📊 Plan Limits

### Free Plan
- 5 monitors
- Minimum 60-minute frequency
- 7-day history retention
- Email alerts only

### Pro Plan
- 50 monitors
- Minimum 5-minute frequency
- 90-day history retention
- All alert channels

### Business Plan
- 200 monitors
- Minimum 1-minute frequency
- 1-year history retention
- API access
- Team features

## 🏗️ Architecture

### Backend
- **Express + TypeScript** - API server
- **PostgreSQL** - Relational database
- **Redis + BullMQ** - Job queue (coming soon)
- **Nodemailer** - Email alerts
- **Axios** - HTTP requests
- **Cheerio** - HTML parsing
- **Diff** - Change detection

### Frontend
- **Next.js 14** - React framework
- **TypeScript** - Type safety
- **Tailwind CSS** - Styling
- **React Query** - Data fetching
- **Axios** - API client

## 🧪 Testing

```bash
# Backend tests
cd backend
npm test

# Frontend tests
cd frontend
npm test
```

## 📝 Development

### Running in Development
```bash
# Terminal 1: Start databases
docker-compose up

# Terminal 2: Backend
cd backend
npm run dev

# Terminal 3: Frontend
cd frontend
npm run dev
```

### Building for Production
```bash
# Backend
cd backend
npm run build
npm start

# Frontend
cd frontend
npm run build
npm start
```

## 🚀 Deployment

### Backend Deployment
- Deploy to Railway, Render, or AWS
- Set environment variables
- Run migrations
- Start with `npm start`

### Frontend Deployment
- Deploy to Vercel or Netlify
- Set `NEXT_PUBLIC_API_URL`
- Build with `npm run build`

### Database
- Use managed PostgreSQL (AWS RDS, Railway, Supabase)
- Run migrations before deploying

## 🐛 Troubleshooting

### Database connection error
- Check PostgreSQL is running
- Verify DATABASE_URL is correct
- Run migrations: `npm run migrate`

### Frontend can't connect to backend
- Check API_URL in .env.local
- Ensure backend is running
- Check CORS settings

### Email alerts not working
- Configure SMTP settings in backend .env
- For development, use a service like Mailtrap
- For production, use SendGrid or similar

## 📚 Documentation

See `/docs` folder for:
- `spec.md` - Complete feature specifications
- `task.md` - Development roadmap
- `actions.md` - User workflows
- `claude.md` - Project context

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a Pull Request

## 📄 License

MIT License - see LICENSE file for details

## 🙏 Acknowledgments

Built with:
- Next.js
- Express
- PostgreSQL
- Tailwind CSS
- TypeScript