tknuth
/
website-monitor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
website-monitor/task.md

15 KiB
Raw Blame History

Website Change Detection Monitor - Development Tasks

Updated: 2026-01-18 based on competitive market research (see findings_market.md)

Key Changes:

  • 🔥 Webhooks moved to MVP (Week 3) - critical for workflow integration
  • 🔥 Slack integration moved to V1 (Week 7-8) - strongly demanded by market
  • ⚠️ Basic JS-Rendering moved to V1 (Week 7-8) - modern sites need this
  • Pricing model under review: considering "checks/month" instead of "monitors + frequency"

Project Setup

Environment & Infrastructure

  • Initialize Git repository
  • Set up project structure (monorepo or separate repos)
  • Configure development environment
    • Node.js/npm or Python/pip
    • Docker for local development
    • Environment variables management
  • Set up CI/CD pipeline
    • GitHub Actions or GitLab CI
    • Automated testing
    • Deployment automation
  • Provision cloud infrastructure
    • Database (PostgreSQL)
    • Redis for queuing
    • Object storage (S3)
    • Application hosting

Development Tools

  • Set up linting and formatting (ESLint/Prettier or Black/Flake8)
  • Configure TypeScript/type checking
  • Set up testing frameworks (Jest/Pytest)
  • Configure logging and monitoring
  • Set up local development database

MVP Phase (Launch Fast)

1. Backend Core - Week 1-2

Database Schema

  • Design and create database schema
    • Users table
    • Monitors table
    • Snapshots table
    • Alerts table
    • Add indexes for performance
  • Set up database migrations
  • Create seed data for development

Authentication & User Management

  • Implement user registration
  • Implement user login (email/password)
  • JWT token generation and validation
  • Password reset flow
  • Email verification (optional for MVP)
  • Basic user profile endpoints

Monitor Management API

  • POST /monitors - Create new monitor
  • GET /monitors - List user's monitors
  • GET /monitors/:id - Get monitor details
  • PUT /monitors/:id - Update monitor
  • DELETE /monitors/:id - Delete monitor
  • Input validation and sanitization
  • URL validation and normalization

2. Monitoring Engine - Week 2-3

Page Fetching

  • Implement HTTP fetcher (Axios/Got)
  • Handle different response types (HTML, JSON, text)
  • Implement timeout handling
  • Add retry logic with exponential backoff
  • User-agent rotation
  • Respect robots.txt (optional)
  • Rate limiting per domain

Change Detection

  • Implement content hash comparison
  • Implement text diff algorithm
    • Character-level diff
    • Line-level diff
  • Store snapshots in database
  • Calculate change percentage
  • Determine if change is significant

Job Scheduling

  • Set up Redis and Bull/BullMQ
  • Create monitor check job
  • Implement job scheduler
    • 5-minute interval
    • 30-minute interval
    • 6-hour interval
    • 24-hour interval
  • Handle job failures and retries
  • Job priority management
  • Monitor queue health

3. Alert System - Week 3

Email Alerts

  • Set up email service (SendGrid/Postmark)
  • Create email templates
    • Change detected template
    • Error alert template

Webhook Integration (MOVED UP from V2) 🔥

  • Implement generic webhook POST endpoint
  • Webhook configuration per user
  • Webhook delivery with retry logic (3 attempts)
  • Webhook logs and delivery status
  • HMAC signature for security
  • Test webhook delivery
  • Implement alert sending logic
  • Track alert delivery status
  • Alert throttling (max 1 per check)
  • Unsubscribe functionality

Alert Management

  • GET /alerts - List user's alerts
  • Mark alerts as read
  • Alert preferences per monitor

4. Frontend Core - Week 3-4

Setup & Layout

  • Initialize Next.js project
  • Set up Tailwind CSS
  • Install and configure shadcn/ui
  • Create main layout component
    • Navigation
    • User menu
    • Responsive design
  • Set up React Query

Authentication Pages

  • Login page
  • Registration page
  • Password reset page
  • Protected route handling

Dashboard

  • Dashboard layout
  • Monitor list view
    • Status indicators
    • Last checked time
    • Last changed time
  • Empty state (no monitors)
  • Loading states
  • Error handling

Monitor Management

  • Create monitor form
    • URL input
    • Frequency selector
    • Name/description
  • Edit monitor page
  • Delete monitor confirmation
  • Form validation

History & Diff Viewer

  • History timeline component
    • Paginated list
    • Filter by status
  • Diff viewer component
    • Side-by-side view
    • Unified view
    • Syntax highlighting
    • Line numbers
  • Snapshot viewer (raw HTML)

5. Testing & Polish - Week 4

  • Write unit tests for core functions
  • Write API integration tests
  • Write E2E tests for critical flows
  • Performance testing
    • Page fetch speed
    • Diff calculation speed
  • Security audit
    • SQL injection prevention
    • XSS prevention
    • CSRF protection
  • Accessibility audit (WCAG AA)
  • Browser compatibility testing

6. Deployment - Week 4

  • Set up production environment
  • Configure domain and SSL
  • Deploy backend API
  • Deploy frontend
  • Set up monitoring and logging
    • Application logs
    • Error tracking (Sentry)
    • Uptime monitoring
  • Create status page
  • Load testing

V1 Phase (People Pay)

7. Noise Reduction - Week 5-6

Automatic Filtering

  • Build filter engine
  • Implement cookie banner detection
    • CSS selector patterns
    • Common text patterns
  • Implement timestamp detection
    • Date/time regex patterns
    • "Last updated" patterns
  • Implement session ID filtering
  • Test filter accuracy on common sites

Custom Ignore Rules

  • Design ignore rule UI
  • Implement regex-based text filtering
  • Implement CSS selector exclusion
  • Add rule testing/preview
  • Save rules with monitor config
  • Apply rules during diff calculation

8. Selective Monitoring - Week 6

Element Selection

  • Implement CSS selector monitoring
  • Build visual element picker
    • Inject selection overlay
    • Click-to-select interface
    • Show element path
  • XPath support
  • Multiple element monitoring
  • Element naming/labeling

9. Keyword Alerts - Week 7

Keyword Engine

  • Keyword matching logic
    • Appears/disappears detection
    • Count tracking
    • Threshold checks
  • Regex support
  • Multiple keyword rules per monitor
  • AND/OR logic combinations
  • Case sensitivity options

UI & Alerts

  • Keyword rule builder UI
  • Keyword match highlighting in diffs
  • Keyword-specific alert templates
  • Alert only on keyword match option

10. Advanced Alerting - Week 7-8

Digest Mode

  • Aggregate changes into digest
  • Daily digest scheduling
  • Weekly digest scheduling
  • Digest email template

Smart Throttling

  • Quiet hours configuration
  • Max alerts per hour/day
  • Cooldown period settings
  • Alert settings UI

Severity System

  • Calculate change severity
    • Small/medium/large changes
    • Price changes = high severity
    • Layout-only = low severity
  • Severity-based filtering
  • Visual severity indicators

Slack Integration (MOVED UP from V2) 🔥

  • Set up Slack OAuth flow
  • Store Slack workspace tokens
  • Channel selection UI
  • Post to Slack on change
  • Rich message formatting with buttons
  • Configure per-monitor or global
  • Test Slack message delivery

Basic JS-Rendering (MOVED UP from V2) ⚠️

  • Add Puppeteer/Playwright toggle option
  • Headless browser mode for JS-heavy sites
  • Wait for network idle
  • Wait for specific elements (optional)
  • JS-rendering usage tracking (for billing)
  • Performance optimization (reuse browser instances)

11. Billing & Subscriptions - Week 8-9

Stripe Integration

  • Set up Stripe account
  • Create products and prices
  • Implement checkout flow
  • Subscription management
    • Upgrade/downgrade
    • Cancel subscription
    • Resume subscription
  • Payment method management
  • Invoice history

Plan Gating

  • Implement usage tracking
    • Monitor count
    • Check frequency
    • History retention
  • Enforce plan limits
  • Usage dashboard for users
  • Upgrade prompts
  • Billing alerts

12. Onboarding & UX - Week 9

  • Welcome tour
  • Example monitors (pre-configured)
  • Monitor templates
    • Job postings
    • Price tracking
    • Product availability
  • Improved empty states
  • Contextual help tooltips
  • Documentation site

V2 Phase (Market Winner)

13. Visual Change Detection - Week 10-11

Screenshot System

  • Set up Puppeteer/Playwright
  • Implement screenshot capture
    • Full page screenshots
    • Specific element screenshots
  • Screenshot storage optimization
  • Screenshot viewer UI

Image Diff

  • Implement pixel-based comparison
  • Highlight changed regions
  • Before/After carousel
  • Visual diff overlay
  • Layout shift detection

14. AI Summaries - Week 11-12

LLM Integration

  • Set up OpenAI/Anthropic API
  • Implement change summarization
    • Prompt engineering
    • Token optimization
  • Change classification
    • Price change detection
    • Availability change
    • Policy update
    • Layout change
  • Summary caching

Smart Alerts

  • Summary in alert emails
  • Classification-based filtering
  • Confidence scoring
  • Summary viewer UI

15. Complex Page Support - Week 12-13

JS Rendering

  • Headless browser mode toggle
  • Wait for network idle
  • Wait for specific elements
  • Dynamic content detection
  • SPA support

Authentication

  • Basic auth support
  • Cookie storage
  • Login flow automation
    • Record login steps
    • Replay login
  • Session management
  • Credential encryption

16. Integrations - Week 13-15

Webhook System

  • Generic webhook POST
  • Webhook testing
  • Retry logic
  • Webhook logs

Third-Party Services

  • Slack integration
    • OAuth flow
    • Channel selection
    • Message formatting
  • Discord webhooks
  • Microsoft Teams connector
  • RSS feed generation
    • Per-monitor feeds
    • Global feed
    • Authentication

API

  • Design REST API
  • API key management
  • API documentation
    • OpenAPI spec
    • Interactive docs
  • Rate limiting
  • Webhooks for API users

Power User Phase

17. Organization Features - Week 16

  • Folders/projects system
  • Tagging system
  • Full-text search
  • Advanced filtering
  • Bulk operations
    • CSV import
    • Export history
    • Bulk edit
    • Bulk delete

18. Collaboration - Week 17-18

Team Workspaces

  • Create workspace model
  • Team invitation system
  • Role-based permissions
  • Shared monitors
  • Audit logging

Communication

  • Comments on changes
  • Assignment system
  • Team notifications
  • Activity feed

19. Advanced Scheduling - Week 18-19

  • Custom cron schedules
  • Business hours only
  • Timezone configuration
  • Geo-distributed checks
    • Multiple check locations
    • Location comparison
  • Adaptive frequency
    • Check more during changes
    • Back off when stable

Ongoing Tasks

Marketing & Growth

  • [/] Create landing page
    • Design premium color palette and update global styles
    • Refactor Hero, Features, and History sections with new design
    • Implement realistic "Live" uptime graph animation
    • Update pricing section to use new color variables
    • Switch primary CTA to "Join the Waitlist"
    • Add final detailed copy (pending user input)
  • Write blog posts for SEO
    • "How to monitor job postings"
    • "Track competitor prices"
    • "Monitor website availability"
  • Create comparison pages (vs. competitors)
  • Set up analytics (PostHog/Mixpanel)
  • Email marketing campaigns
  • Create demo video
  • Social media presence

Support & Documentation

  • Help center/knowledge base
  • API documentation
  • Video tutorials
  • Customer support system
    • Email support
    • Chat support (Intercom/Crisp)
  • FAQ page
  • Troubleshooting guides

Optimization & Maintenance

  • Performance monitoring
  • Database optimization
  • Query performance tuning
  • Cost optimization
    • Cloud resource usage
    • Email sending costs
    • Storage costs
  • Security updates
  • Dependency updates
  • Bug fixes and patches

Analytics & Iteration

  • User behavior tracking
  • Feature usage analytics
  • A/B testing framework
  • Customer feedback collection
  • NPS surveys
  • Churn analysis
  • Conversion funnel optimization

Risk Management

Technical Risks

  • Risk: Blocked by anti-bot measures
    • Mitigation: Residential proxies, browser fingerprinting, rate limiting
  • Risk: Scaling issues with thousands of checks
    • Mitigation: Distributed queue, horizontal scaling, caching
  • Risk: High false positive rate
    • Mitigation: Better filtering, user feedback loop, ML training

Business Risks

  • Risk: High infrastructure costs
    • Mitigation: Efficient caching, optimize storage, tiered plans
  • Risk: Low conversion rate
    • Mitigation: Strong onboarding, free tier value, quick wins
  • Risk: Competitive market
    • Mitigation: Focus on noise reduction, better UX, integrations

Launch Checklist

Pre-Launch

  • MVP features complete and tested
  • Landing page live
  • Pricing page finalized
  • Terms of Service and Privacy Policy
  • Payment processing tested
  • Email deliverability tested
  • Error monitoring configured
  • Backup system tested

Launch Day

  • Deploy to production
  • Monitor error rates
  • Watch server resources
  • Test critical user flows
  • Announce on social media
  • Submit to directories (Product Hunt, etc.)
  • Email early access list

Post-Launch

  • Daily metric review (signups, monitors, errors)
  • User feedback collection
  • Bug triage and fixes
  • Performance optimization
  • Feature prioritization based on feedback