website-monitor/task.md

# Website Change Detection Monitor - Development Tasks

> **Updated:** 2026-01-18 based on competitive market research (see findings_market.md)
>
> **Key Changes:**
> - 🔥 Webhooks moved to MVP (Week 3) - critical for workflow integration
> - 🔥 Slack integration moved to V1 (Week 7-8) - strongly demanded by market
> - ⚠️ Basic JS-Rendering moved to V1 (Week 7-8) - modern sites need this
> - Pricing model under review: considering "checks/month" instead of "monitors + frequency"

---

## Project Setup

### Environment & Infrastructure
- [ ] Initialize Git repository
- [ ] Set up project structure (monorepo or separate repos)
- [ ] Configure development environment
  - [ ] Node.js/npm or Python/pip
  - [ ] Docker for local development
  - [ ] Environment variables management
- [ ] Set up CI/CD pipeline
  - [ ] GitHub Actions or GitLab CI
  - [ ] Automated testing
  - [ ] Deployment automation
- [ ] Provision cloud infrastructure
  - [ ] Database (PostgreSQL)
  - [ ] Redis for queuing
  - [ ] Object storage (S3)
  - [ ] Application hosting

### Development Tools
- [ ] Set up linting and formatting (ESLint/Prettier or Black/Flake8)
- [ ] Configure TypeScript/type checking
- [ ] Set up testing frameworks (Jest/Pytest)
- [ ] Configure logging and monitoring
- [ ] Set up local development database

---

## MVP Phase (Launch Fast)

### 1. Backend Core - Week 1-2

#### Database Schema
- [ ] Design and create database schema
  - [ ] Users table
  - [ ] Monitors table
  - [ ] Snapshots table
  - [ ] Alerts table
  - [ ] Add indexes for performance
- [ ] Set up database migrations
- [ ] Create seed data for development

#### Authentication & User Management
- [ ] Implement user registration
- [ ] Implement user login (email/password)
- [ ] JWT token generation and validation
- [ ] Password reset flow
- [ ] Email verification (optional for MVP)
- [ ] Basic user profile endpoints

#### Monitor Management API
- [ ] POST /monitors - Create new monitor
- [ ] GET /monitors - List user's monitors
- [ ] GET /monitors/:id - Get monitor details
- [ ] PUT /monitors/:id - Update monitor
- [ ] DELETE /monitors/:id - Delete monitor
- [ ] Input validation and sanitization
- [ ] URL validation and normalization

### 2. Monitoring Engine - Week 2-3

#### Page Fetching
- [ ] Implement HTTP fetcher (Axios/Got)
- [ ] Handle different response types (HTML, JSON, text)
- [ ] Implement timeout handling
- [ ] Add retry logic with exponential backoff
- [ ] User-agent rotation
- [ ] Respect robots.txt (optional)
- [ ] Rate limiting per domain

#### Change Detection
- [ ] Implement content hash comparison
- [ ] Implement text diff algorithm
  - [ ] Character-level diff
  - [ ] Line-level diff
- [ ] Store snapshots in database
- [ ] Calculate change percentage
- [ ] Determine if change is significant

#### Job Scheduling
- [ ] Set up Redis and Bull/BullMQ
- [ ] Create monitor check job
- [ ] Implement job scheduler
  - [ ] 5-minute interval
  - [ ] 30-minute interval
  - [ ] 6-hour interval
  - [ ] 24-hour interval
- [ ] Handle job failures and retries
- [ ] Job priority management
- [ ] Monitor queue health

### 3. Alert System - Week 3

#### Email Alerts
- [ ] Set up email service (SendGrid/Postmark)
- [ ] Create email templates
  - [ ] Change detected template
  - [ ] Error alert template

#### Webhook Integration (MOVED UP from V2) 🔥
- [ ] Implement generic webhook POST endpoint
- [ ] Webhook configuration per user
- [ ] Webhook delivery with retry logic (3 attempts)
- [ ] Webhook logs and delivery status
- [ ] HMAC signature for security
- [ ] Test webhook delivery
- [ ] Implement alert sending logic
- [ ] Track alert delivery status
- [ ] Alert throttling (max 1 per check)
- [ ] Unsubscribe functionality

#### Alert Management
- [ ] GET /alerts - List user's alerts
- [ ] Mark alerts as read
- [ ] Alert preferences per monitor

### 4. Frontend Core - Week 3-4

#### Setup & Layout
- [ ] Initialize Next.js project
- [ ] Set up Tailwind CSS
- [ ] Install and configure shadcn/ui
- [ ] Create main layout component
  - [ ] Navigation
  - [ ] User menu
  - [ ] Responsive design
- [ ] Set up React Query

#### Authentication Pages
- [ ] Login page
- [ ] Registration page
- [ ] Password reset page
- [ ] Protected route handling

#### Dashboard
- [ ] Dashboard layout
- [ ] Monitor list view
  - [ ] Status indicators
  - [ ] Last checked time
  - [ ] Last changed time
- [ ] Empty state (no monitors)
- [ ] Loading states
- [ ] Error handling

#### Monitor Management
- [ ] Create monitor form
  - [ ] URL input
  - [ ] Frequency selector
  - [ ] Name/description
- [ ] Edit monitor page
- [ ] Delete monitor confirmation
- [ ] Form validation

#### History & Diff Viewer
- [ ] History timeline component
  - [ ] Paginated list
  - [ ] Filter by status
- [ ] Diff viewer component
  - [ ] Side-by-side view
  - [ ] Unified view
  - [ ] Syntax highlighting
  - [ ] Line numbers
- [ ] Snapshot viewer (raw HTML)

### 5. Testing & Polish - Week 4

- [ ] Write unit tests for core functions
- [ ] Write API integration tests
- [ ] Write E2E tests for critical flows
- [ ] Performance testing
  - [ ] Page fetch speed
  - [ ] Diff calculation speed
- [ ] Security audit
  - [ ] SQL injection prevention
  - [ ] XSS prevention
  - [ ] CSRF protection
- [ ] Accessibility audit (WCAG AA)
- [ ] Browser compatibility testing

### 6. Deployment - Week 4

- [ ] Set up production environment
- [ ] Configure domain and SSL
- [ ] Deploy backend API
- [ ] Deploy frontend
- [ ] Set up monitoring and logging
  - [ ] Application logs
  - [ ] Error tracking (Sentry)
  - [ ] Uptime monitoring
- [ ] Create status page
- [ ] Load testing

---

## V1 Phase (People Pay)

### 7. Noise Reduction - Week 5-6

#### Automatic Filtering
- [ ] Build filter engine
- [ ] Implement cookie banner detection
  - [ ] CSS selector patterns
  - [ ] Common text patterns
- [ ] Implement timestamp detection
  - [ ] Date/time regex patterns
  - [ ] "Last updated" patterns
- [ ] Implement session ID filtering
- [ ] Test filter accuracy on common sites

#### Custom Ignore Rules
- [ ] Design ignore rule UI
- [ ] Implement regex-based text filtering
- [ ] Implement CSS selector exclusion
- [ ] Add rule testing/preview
- [ ] Save rules with monitor config
- [ ] Apply rules during diff calculation

### 8. Selective Monitoring - Week 6

#### Element Selection
- [ ] Implement CSS selector monitoring
- [ ] Build visual element picker
  - [ ] Inject selection overlay
  - [ ] Click-to-select interface
  - [ ] Show element path
- [ ] XPath support
- [ ] Multiple element monitoring
- [ ] Element naming/labeling

### 9. Keyword Alerts - Week 7

#### Keyword Engine
- [ ] Keyword matching logic
  - [ ] Appears/disappears detection
  - [ ] Count tracking
  - [ ] Threshold checks
- [ ] Regex support
- [ ] Multiple keyword rules per monitor
- [ ] AND/OR logic combinations
- [ ] Case sensitivity options

#### UI & Alerts
- [ ] Keyword rule builder UI
- [ ] Keyword match highlighting in diffs
- [ ] Keyword-specific alert templates
- [ ] Alert only on keyword match option

### 10. Advanced Alerting - Week 7-8

#### Digest Mode
- [ ] Aggregate changes into digest
- [ ] Daily digest scheduling
- [ ] Weekly digest scheduling
- [ ] Digest email template

#### Smart Throttling
- [ ] Quiet hours configuration
- [ ] Max alerts per hour/day
- [ ] Cooldown period settings
- [ ] Alert settings UI

#### Severity System
- [ ] Calculate change severity
  - [ ] Small/medium/large changes
  - [ ] Price changes = high severity
  - [ ] Layout-only = low severity
- [ ] Severity-based filtering
- [ ] Visual severity indicators

#### Slack Integration (MOVED UP from V2) 🔥
- [ ] Set up Slack OAuth flow
- [ ] Store Slack workspace tokens
- [ ] Channel selection UI
- [ ] Post to Slack on change
- [ ] Rich message formatting with buttons
- [ ] Configure per-monitor or global
- [ ] Test Slack message delivery

#### Basic JS-Rendering (MOVED UP from V2) ⚠️
- [ ] Add Puppeteer/Playwright toggle option
- [ ] Headless browser mode for JS-heavy sites
- [ ] Wait for network idle
- [ ] Wait for specific elements (optional)
- [ ] JS-rendering usage tracking (for billing)
- [ ] Performance optimization (reuse browser instances)

### 11. Billing & Subscriptions - Week 8-9

#### Stripe Integration
- [ ] Set up Stripe account
- [ ] Create products and prices
- [ ] Implement checkout flow
- [ ] Subscription management
  - [ ] Upgrade/downgrade
  - [ ] Cancel subscription
  - [ ] Resume subscription
- [ ] Payment method management
- [ ] Invoice history

#### Plan Gating
- [ ] Implement usage tracking
  - [ ] Monitor count
  - [ ] Check frequency
  - [ ] History retention
- [ ] Enforce plan limits
- [ ] Usage dashboard for users
- [ ] Upgrade prompts
- [ ] Billing alerts

### 12. Onboarding & UX - Week 9

- [ ] Welcome tour
- [ ] Example monitors (pre-configured)
- [ ] Monitor templates
  - [ ] Job postings
  - [ ] Price tracking
  - [ ] Product availability
- [ ] Improved empty states
- [ ] Contextual help tooltips
- [ ] Documentation site

---

## V2 Phase (Market Winner)

### 13. Visual Change Detection - Week 10-11

#### Screenshot System
- [ ] Set up Puppeteer/Playwright
- [ ] Implement screenshot capture
  - [ ] Full page screenshots
  - [ ] Specific element screenshots
- [ ] Screenshot storage optimization
- [ ] Screenshot viewer UI

#### Image Diff
- [ ] Implement pixel-based comparison
- [ ] Highlight changed regions
- [ ] Before/After carousel
- [ ] Visual diff overlay
- [ ] Layout shift detection

### 14. AI Summaries - Week 11-12

#### LLM Integration
- [ ] Set up OpenAI/Anthropic API
- [ ] Implement change summarization
  - [ ] Prompt engineering
  - [ ] Token optimization
- [ ] Change classification
  - [ ] Price change detection
  - [ ] Availability change
  - [ ] Policy update
  - [ ] Layout change
- [ ] Summary caching

#### Smart Alerts
- [ ] Summary in alert emails
- [ ] Classification-based filtering
- [ ] Confidence scoring
- [ ] Summary viewer UI

### 15. Complex Page Support - Week 12-13

#### JS Rendering
- [ ] Headless browser mode toggle
- [ ] Wait for network idle
- [ ] Wait for specific elements
- [ ] Dynamic content detection
- [ ] SPA support

#### Authentication
- [ ] Basic auth support
- [ ] Cookie storage
- [ ] Login flow automation
  - [ ] Record login steps
  - [ ] Replay login
- [ ] Session management
- [ ] Credential encryption

### 16. Integrations - Week 13-15

#### Webhook System
- [ ] Generic webhook POST
- [ ] Webhook testing
- [ ] Retry logic
- [ ] Webhook logs

#### Third-Party Services
- [ ] Slack integration
  - [ ] OAuth flow
  - [ ] Channel selection
  - [ ] Message formatting
- [ ] Discord webhooks
- [ ] Microsoft Teams connector
- [ ] RSS feed generation
  - [ ] Per-monitor feeds
  - [ ] Global feed
  - [ ] Authentication

#### API
- [ ] Design REST API
- [ ] API key management
- [ ] API documentation
  - [ ] OpenAPI spec
  - [ ] Interactive docs
- [ ] Rate limiting
- [ ] Webhooks for API users

---

## Power User Phase

### 17. Organization Features - Week 16

- [ ] Folders/projects system
- [ ] Tagging system
- [ ] Full-text search
- [ ] Advanced filtering
- [ ] Bulk operations
  - [ ] CSV import
  - [ ] Export history
  - [ ] Bulk edit
  - [ ] Bulk delete

### 18. Collaboration - Week 17-18

#### Team Workspaces
- [ ] Create workspace model
- [ ] Team invitation system
- [ ] Role-based permissions
- [ ] Shared monitors
- [ ] Audit logging

#### Communication
- [ ] Comments on changes
- [ ] Assignment system
- [ ] Team notifications
- [ ] Activity feed

### 19. Advanced Scheduling - Week 18-19

- [ ] Custom cron schedules
- [ ] Business hours only
- [ ] Timezone configuration
- [ ] Geo-distributed checks
  - [ ] Multiple check locations
  - [ ] Location comparison
- [ ] Adaptive frequency
  - [ ] Check more during changes
  - [ ] Back off when stable

---

## Ongoing Tasks

### Marketing & Growth
- [/] Create landing page
  - [x] Design premium color palette and update global styles
  - [x] Refactor Hero, Features, and History sections with new design
  - [x] Implement realistic "Live" uptime graph animation
  - [x] Update pricing section to use new color variables
  - [x] Switch primary CTA to "Join the Waitlist"
  - [ ] Add final detailed copy (pending user input)
- [ ] Write blog posts for SEO
  - [ ] "How to monitor job postings"
  - [ ] "Track competitor prices"
  - [ ] "Monitor website availability"
- [ ] Create comparison pages (vs. competitors)
- [ ] Set up analytics (PostHog/Mixpanel)
- [ ] Email marketing campaigns
- [ ] Create demo video
- [ ] Social media presence

### Support & Documentation
- [ ] Help center/knowledge base
- [ ] API documentation
- [ ] Video tutorials
- [ ] Customer support system
  - [ ] Email support
  - [ ] Chat support (Intercom/Crisp)
- [ ] FAQ page
- [ ] Troubleshooting guides

### Optimization & Maintenance
- [ ] Performance monitoring
- [ ] Database optimization
- [ ] Query performance tuning
- [ ] Cost optimization
  - [ ] Cloud resource usage
  - [ ] Email sending costs
  - [ ] Storage costs
- [ ] Security updates
- [ ] Dependency updates
- [ ] Bug fixes and patches

### Analytics & Iteration
- [ ] User behavior tracking
- [ ] Feature usage analytics
- [ ] A/B testing framework
- [ ] Customer feedback collection
- [ ] NPS surveys
- [ ] Churn analysis
- [ ] Conversion funnel optimization

---

## Risk Management

### Technical Risks
- **Risk**: Blocked by anti-bot measures
  - **Mitigation**: Residential proxies, browser fingerprinting, rate limiting
- **Risk**: Scaling issues with thousands of checks
  - **Mitigation**: Distributed queue, horizontal scaling, caching
- **Risk**: High false positive rate
  - **Mitigation**: Better filtering, user feedback loop, ML training

### Business Risks
- **Risk**: High infrastructure costs
  - **Mitigation**: Efficient caching, optimize storage, tiered plans
- **Risk**: Low conversion rate
  - **Mitigation**: Strong onboarding, free tier value, quick wins
- **Risk**: Competitive market
  - **Mitigation**: Focus on noise reduction, better UX, integrations

---

## Launch Checklist

### Pre-Launch
- [ ] MVP features complete and tested
- [ ] Landing page live
- [ ] Pricing page finalized
- [ ] Terms of Service and Privacy Policy
- [ ] Payment processing tested
- [ ] Email deliverability tested
- [ ] Error monitoring configured
- [ ] Backup system tested

### Launch Day
- [ ] Deploy to production
- [ ] Monitor error rates
- [ ] Watch server resources
- [ ] Test critical user flows
- [ ] Announce on social media
- [ ] Submit to directories (Product Hunt, etc.)
- [ ] Email early access list

### Post-Launch
- [ ] Daily metric review (signups, monitors, errors)
- [ ] User feedback collection
- [ ] Bug triage and fixes
- [ ] Performance optimization
- [ ] Feature prioritization based on feedback