# Website Change Detection Monitor - Technical Specification

## Product Overview

A SaaS platform that monitors web pages for changes and alerts users when meaningful updates occur. The core value proposition is "signal over noise" - detecting real changes while filtering out irrelevant updates.

## Core Value Propositions

- **Easy to understand**: "I watch pages so you don't have to"
- **Smart filtering**: Automatically ignores timestamps, cookie banners, and noise
- **Keyword intelligence**: Alert on specific content appearing/disappearing
- **SEO-friendly**: Captures long-tail keywords (e.g., "monitor job posting changes")

---

## Feature Specification by Phase

### MVP Features (Launch Fast)

#### 1. URL Monitoring
- **Track URLs**: Add any public web page by URL
- **Frequency options**: 5min / 30min / 6hr / 24hr intervals
- **Change detection methods**:
  - Content hash comparison (fast, binary change detection)
  - Text diff (character/line level differences)
- **Storage**: Last 5-10 snapshots per URL

#### 2. Alert System
- **Email notifications**: Send alert when change detected
- **Alert content**:
  - Timestamp of change
  - Link to diff view
  - Change severity indicator
- **Basic throttling**: Max 1 alert per check interval

#### 3. Change Viewing
- **History timeline**: Chronological list of all checks
  - Status (changed/unchanged/error)
  - Timestamp
  - Response code
- **Diff viewer**:
  - Side-by-side or unified view
  - Highlighted additions/deletions
  - Character-level diff for precision
- **Snapshot storage**: Full HTML snapshots for history

#### 4. Reliability
- **Retry logic**: 2-3 attempts on timeout/5xx errors
- **Error alerts**: Notify if page becomes unavailable
- **Status tracking**:
  - HTTP response codes
  - Timeout detection
  - Robot/CAPTCHA blocking detection
- **Run logs**: Detailed history of each check attempt

---

### V1 Features (People Pay)

#### 5. Noise Reduction (Differentiator)
- **Automatic filtering**:
  - Cookie consent banners
  - Timestamps and "last updated" text
  - Current date/time displays
  - Session IDs in URLs
  - Rotating content (recommendations, ads)
- **Custom ignore rules**:
  - Text pattern matching (regex)
  - CSS selector exclusion
  - Ignore numeric changes except in specific areas
  - Whitelist/blacklist mode

#### 6. Selective Monitoring
- **Element monitoring**: Track only specific page sections
  - Visual point-and-click selector
  - CSS selector input (power users)
  - XPath support
- **Multiple elements**: Track different sections separately
- **Element naming**: Label tracked elements for clarity

#### 7. Keyword-Based Alerts (High Value)
- **Keyword rules**:
  - Alert when keyword appears
  - Alert when keyword disappears
  - Alert when keyword count changes
  - Threshold-based alerts ("less than 5 items")
- **Regex support**: Advanced pattern matching
- **Multiple keywords**: AND/OR logic combinations
- **Case sensitivity options**

#### 8. Advanced Alerting
- **Digest mode**: Daily/weekly summary of all changes
- **Quiet hours**: No alerts during specified times
- **Alert throttling**: Configurable limits
  - Max alerts per hour/day
  - Cooldown periods
- **Severity filtering**: Only alert on major changes
- **Multiple channels per monitor**: Email + Slack, etc.

---

### V2 Features (Market Winner)

#### 9. Visual Change Detection
- **Screenshot capture**: Full-page screenshots
- **Image diff**: Visual highlighting of changed areas
- **Pixel-perfect detection**: Layout shift detection
- **Before/After carousel**: Easy visual comparison
- **Screenshot retention**: Based on plan tier

#### 10. AI-Powered Summaries
- **Change summarization**: Natural language description
  - "Price changed from $29.99 to $24.99"
  - "New 'Out of Stock' banner added"
- **Change classification**:
  - Price change
  - Availability change
  - Policy/text update
  - Layout-only change
- **Smart alerts**: Only notify for meaningful changes
- **Summary in alert**: No need to view diff for simple changes

#### 11. Complex Page Support
- **JavaScript rendering**: Headless browser mode
- **Authentication**:
  - Basic auth
  - Session cookies
  - Login flow automation
- **Dynamic content**: Wait for AJAX/lazy loading
- **SPA support**: Monitor client-side rendered apps

#### 12. Integrations
- **Slack**: Channel notifications
- **Discord**: Webhook alerts
- **Microsoft Teams**: Connector integration
- **Webhook**: Generic POST for automation tools
- **RSS feed**: Per-monitor or global feed
- **Zapier/Make**: Pre-built integrations
- **API**: Programmatic access to monitors and history

---

### Power User Features (Teams & Scale)

#### 13. Organization
- **Folders/Projects**: Hierarchical organization
- **Tags**: Multi-dimensional categorization
- **Search**: Full-text search across monitors and history
- **Filters**: Status, tags, frequency, last changed
- **Bulk operations**:
  - Import URLs from CSV
  - Export history
  - Bulk pause/resume
  - Bulk delete

#### 14. Collaboration
- **Team workspaces**: Shared monitor collections
- **Role-based access**:
  - Admin: Full control
  - Editor: Create/edit monitors
  - Viewer: Read-only access
- **Assignment**: Assign monitors to team members
- **Comments**: Annotate changes
- **Audit log**: Track team actions

#### 15. Advanced Scheduling
- **Custom schedules**: Cron-like expressions
- **Business hours only**: Skip nights/weekends
- **Timezone-aware**: Different times per monitor
- **Geo-distributed checks**: Monitor from multiple regions
- **Adaptive frequency**: Check more often during active periods

---

## Technical Architecture

### Core Components

#### Frontend
- **Tech stack**: React/Next.js + TypeScript
- **UI components**: Tailwind CSS + shadcn/ui
- **State management**: React Query for API data
- **Real-time**: WebSocket for live updates (optional)

#### Backend
- **API**: Node.js/Express or Python/FastAPI
- **Database**: PostgreSQL for relational data
- **Queue system**: Redis + Bull/BullMQ for job scheduling
- **Storage**: S3-compatible for snapshots and screenshots

#### Monitoring Engine
- **Fetcher**: Axios/Got for simple pages
- **Browser**: Puppeteer/Playwright for JS-heavy sites
- **Differ**: jsdiff or custom algorithm
- **Scheduler**: Distributed job queue with priority
- **Rate limiting**: Per-domain backoff

#### Alert System
- **Email**: SendGrid/Postmark
- **Queue**: Separate alert queue for reliability
- **Templates**: Customizable alert formats
- **Delivery tracking**: Open/click tracking

### Data Models

#### Monitor
```
id, user_id, url, name, frequency, element_selector,
ignore_rules, keyword_rules, alert_settings, status,
created_at, last_checked_at, last_changed_at
```

#### Snapshot
```
id, monitor_id, html_content, screenshot_url,
content_hash, http_status, error_message,
created_at, changed_from_previous
```

#### Alert
```
id, monitor_id, snapshot_id, alert_type,
delivered_at, delivery_status, channels
```

---

## Monetization & Plan Gating

### Free Tier
- 5 monitors
- 1-hour minimum frequency
- 7-day history retention
- Email alerts only
- Basic noise filtering

### Pro Tier ($19-29/month)
- 50 monitors
- 5-minute frequency
- 90-day history
- All alert channels
- Advanced filtering + keywords
- Screenshot snapshots

### Business Tier ($99-149/month)
- 200 monitors
- 1-minute frequency
- 1-year history
- API access
- Team collaboration (5 seats)
- Priority support
- JS rendering included

### Enterprise Tier (Custom)
- Unlimited monitors
- Custom frequency
- Unlimited history
- Dedicated infrastructure
- SLA guarantees
- SSO/SAML
- Custom integrations

### Add-ons
- Extra monitors: $5 per 10
- Extended history: $10/month
- Additional team seats: $15/seat
- JS rendering credits: $20/100 pages

---

## Success Metrics

### Product KPIs
- Monitors created per user
- Check success rate (>99%)
- False positive rate (<5%)
- Alert open rate (>40%)
- User retention (D7, D30, M3)

### Business KPIs
- Free → Paid conversion (target >10%)
- Churn rate (target <5% monthly)
- Average monitors per paid user (target >15)
- CAC < 3x MRR
- Net promoter score (target >50)

---

## Security & Compliance

### Security
- **Authentication**: JWT + refresh tokens
- **2FA**: TOTP support
- **Encryption**: At rest (database) and in transit (TLS)
- **API keys**: Scoped, revocable
- **Rate limiting**: Per user and IP

### Privacy
- **GDPR**: Data export, deletion, consent management
- **Data retention**: Configurable, automatic cleanup
- **No tracking**: Don't store personal data from monitored pages
- **Anonymization**: Strip cookies/sessions from snapshots

### Reliability
- **Uptime SLA**: 99.9% for paid plans
- **Status page**: Public incident tracking
- **Backups**: Daily encrypted backups
- **Disaster recovery**: 24-hour RTO

---

## Competitive Differentiation

### vs. Visualping/ChangeTower
- **Better noise filtering**: AI-powered content classification
- **Smarter alerts**: Keyword-based + summarization
- **Better UX**: Cleaner UI, faster setup

### vs. Distill.io
- **Team features**: Built for collaboration from day one
- **More integrations**: Wider ecosystem support
- **Better pricing**: More generous free tier

### vs. Wachete
- **Modern tech**: Faster, more reliable
- **Visual diff**: Screenshot comparison
- **API-first**: Better for automation