tknuth
/
website-monitor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
website-monitor/README.md

6.2 KiB
Raw Blame History

Website Change Detection Monitor

A SaaS platform that monitors web pages for changes and sends smart alerts with noise filtering.

🚀 Features

MVP (Current Implementation)

  • ✅ User authentication (register/login)
  • ✅ Monitor creation and management
  • ✅ Automatic page checking at configurable intervals
  • ✅ Change detection with hash comparison
  • ✅ Email alerts on changes
  • ✅ History timeline with snapshots
  • ✅ Smart noise filtering (timestamps, cookie banners)
  • ✅ Manual check triggering
  • ✅ Plan-based limits (free/pro/business)

Coming Soon

  • Keyword-based alerts
  • Element-specific monitoring
  • Visual diff viewer
  • Slack/Discord integrations
  • AI-powered change summaries
  • Team collaboration

📋 Prerequisites

  • Node.js 18+ and npm
  • PostgreSQL 15+
  • Redis 7+
  • Docker (optional, for local development)

🛠️ Installation

1. Clone the repository

git clone <repository-url>
cd website-monitor

2. Start database services (Docker)

docker-compose up -d

This starts:

  • PostgreSQL on port 5432
  • Redis on port 6379

3. Setup Backend

cd backend
npm install

# Copy environment file
cp .env.example .env
# Edit .env with your settings

# Run migrations
npm run migrate

# Start development server
npm run dev

Backend will run on http://localhost:3001

4. Setup Frontend

cd frontend
npm install

# Start development server
npm run dev

Frontend will run on http://localhost:3000

🗄️ Database Schema

The database includes:

  • users - User accounts and authentication
  • monitors - Website monitors configuration
  • snapshots - Historical page snapshots
  • alerts - Alert records

Run migrations:

cd backend
npm run migrate

🔧 Configuration

Backend (.env)

PORT=3001
DATABASE_URL=postgresql://admin:admin123@localhost:5432/website_monitor
REDIS_URL=redis://localhost:6379
JWT_SECRET=your-secret-key
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASS=your-api-key
LANDING_ONLY_MODE=false
ADMIN_PASSWORD=change-me-for-admin-waitlist

Frontend (.env.local)

NEXT_PUBLIC_API_URL=http://localhost:3001
NEXT_PUBLIC_LANDING_ONLY_MODE=false

Landing-only mode (Waitlist rollout)

To expose only landing pages and waitlist APIs, enable both flags:

  • Backend .env: LANDING_ONLY_MODE=true
  • Frontend .env.local: NEXT_PUBLIC_LANDING_ONLY_MODE=true

When enabled:

  • Public pages: /, /blog, /privacy, /admin
  • All other frontend routes redirect to / with HTTP 307
  • Backend only allows /api/waitlist/*, POST /api/tools/meta-preview, and /health

📖 Usage

Register an Account

  1. Go to http://localhost:3000
  2. Click "Sign up"
  3. Enter email and password
  4. You'll be logged in automatically

Create a Monitor

  1. Go to Dashboard
  2. Click "+ Add Monitor"
  3. Enter URL and select frequency
  4. Click "Create Monitor"
  5. First check happens immediately

View History

  1. Click "History" on any monitor
  2. See timeline of all checks
  3. View changes and errors

🔑 API Endpoints

Authentication

  • POST /api/auth/register - Register new user
  • POST /api/auth/login - Login user

Monitors (Requires Auth)

  • GET /api/monitors - List all monitors
  • POST /api/monitors - Create monitor
  • GET /api/monitors/:id - Get monitor details
  • PUT /api/monitors/:id - Update monitor
  • DELETE /api/monitors/:id - Delete monitor
  • POST /api/monitors/:id/check - Trigger manual check
  • GET /api/monitors/:id/history - Get check history

📊 Plan Limits

Free Plan

  • 5 monitors
  • Minimum 60-minute frequency
  • 7-day history retention
  • Email alerts only

Pro Plan

  • 50 monitors
  • Minimum 5-minute frequency
  • 90-day history retention
  • All alert channels

Business Plan

  • 200 monitors
  • Minimum 1-minute frequency
  • 1-year history retention
  • API access
  • Team features

🏗️ Architecture

Backend

  • Express + TypeScript - API server
  • PostgreSQL - Relational database
  • Redis + BullMQ - Job queue (coming soon)
  • Nodemailer - Email alerts
  • Axios - HTTP requests
  • Cheerio - HTML parsing
  • Diff - Change detection

Frontend

  • Next.js 14 - React framework
  • TypeScript - Type safety
  • Tailwind CSS - Styling
  • React Query - Data fetching
  • Axios - API client

🧪 Testing

# Backend tests
cd backend
npm test

# Frontend tests
cd frontend
npm test

📝 Development

Running in Development

# Terminal 1: Start databases
docker-compose up

# Terminal 2: Backend
cd backend
npm run dev

# Terminal 3: Frontend
cd frontend
npm run dev

Building for Production

# Backend
cd backend
npm run build
npm start

# Frontend
cd frontend
npm run build
npm start

🚀 Deployment

Backend Deployment

  • Deploy to Railway, Render, or AWS
  • Set environment variables
  • Run migrations
  • Start with npm start

Frontend Deployment

  • Deploy to Vercel or Netlify
  • Set NEXT_PUBLIC_API_URL
  • Build with npm run build

Database

  • Use managed PostgreSQL (AWS RDS, Railway, Supabase)
  • Run migrations before deploying

🐛 Troubleshooting

Database connection error

  • Check PostgreSQL is running
  • Verify DATABASE_URL is correct
  • Run migrations: npm run migrate

Frontend can't connect to backend

  • Check API_URL in .env.local
  • Ensure backend is running
  • Check CORS settings

Email alerts not working

  • Configure SMTP settings in backend .env
  • For development, use a service like Mailtrap
  • For production, use SendGrid or similar

📚 Documentation

See /docs folder for:

  • spec.md - Complete feature specifications
  • task.md - Development roadmap
  • actions.md - User workflows
  • claude.md - Project context

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Commit your changes
  4. Push to the branch
  5. Open a Pull Request

📄 License

MIT License - see LICENSE file for details

🙏 Acknowledgments

Built with:

  • Next.js
  • Express
  • PostgreSQL
  • Tailwind CSS
  • TypeScript