Issue Summary
| Category | Count | Status |
|---|---|---|
| CRITICAL Blockers | 0 | NONE |
| Resolved Issues | 18 | RESOLVED |
| Open (Non-blocking) | 1 | OPEN |
| Workarounds in Place | 3 | WORKAROUND |
| Items Being Monitored | 4 | MONITORING |
Previously Critical: Now Resolved
Resolved: n8n Webhook Caching Issue
Original Problem
Added a "Trigger Drip Campaign" HTTP Request node to the Confirmation workflow, but webhook used cached old workflow version.
Resolution (January 19, 2026)
The webhook path change approach worked:
- Changed webhook path from
listmonk/confirmtolistmonk/confirm-temp - Changed it back to
listmonk/confirm - Deactivated and reactivated the workflow
Key insight: n8n API PUT requests fail with "must NOT have additional properties" unless you only include {name, nodes, connections, settings}.
Current Status
Full flow now works: Confirmation → Mark Verified → Create Contest Entry → Trigger Drip Campaign
Resolved: January 19, 2026
Active Workarounds
Confirmation Workflow Webhook Not Auto-Registering on Startup
Problem
The 3. Confirmation workflow (ID: eoW1DejeZlfPA3l1) is not registering its webhook when n8n starts, even though the database shows active: true.
Evidence
# n8n startup log shows only 4 workflows activated: Start Active Workflows: Activated workflow "2. Email Capture" (ID: AeiltZEKb4YvAbQG) Activated workflow "4. Story & Reward Drip" (ID: YSzXpljWCgHFQo0g) Activated workflow "5. Monthly Winner Selection" (ID: cKATiI2Kh5UhIBIX) Activated workflow "1. Scan Intake" (ID: XjqHOdsNzN1Gg4n6) # Missing: "3. Confirmation" workflow! # Result when webhook is called: Received request for unknown webhook: The requested webhook "POST listmonk/confirm" is not registered.
Impact
- After n8n restart, confirmation flow breaks
- Users click confirm link but n8n doesn't process it
- Manual intervention required after each restart
Workaround
After each n8n restart, manually toggle the workflow via API:
# Deactivate curl -s -X POST -H "X-N8N-API-KEY: $API_KEY" \ "http://localhost:5678/api/v1/workflows/eoW1DejeZlfPA3l1/deactivate" # Wait 1 second, then activate curl -s -X POST -H "X-N8N-API-KEY: $API_KEY" \ "http://localhost:5678/api/v1/workflows/eoW1DejeZlfPA3l1/activate"
Potential Permanent Fixes
| Option | Approach | Complexity |
|---|---|---|
| A (Recommended) | Add startup script to docker-compose that toggles workflow after n8n starts | Low |
| B | Recreate workflow from scratch as "3. Confirmation v2" | Medium |
| C | Investigate why n8n skips this specific workflow on startup | Medium |
| D | Upgrade n8n version and test if issue persists | Medium |
Status
Discovered: January 20, 2026
Current State: Workaround in place - E2E tests passing
Priority: MEDIUM - System works with manual intervention
Issue #15: Listmonk Confirmation Not Triggering n8n
Original Problem
Listmonk has no built-in webhook for subscriber confirmations - opt-in flow was entirely internal.
Solution Implemented
Added custom JavaScript to Listmonk's appearance.public.custom_js that:
- Detects when confirmation success page loads
- Extracts subscriber UUID from URL
- POSTs to n8n webhook to trigger confirmation workflow
Result
Confirmation webhook now fires correctly. Subscriber verification and contest entry creation work. Only remaining issue is the auto-drip trigger (see blocker above).
Status
Resolved: January 18, 2026
External Webhook Intermittent Response (Partially Resolved)
Problem
External requests to the scan webhook (/webhook/scan) via public IP (76.150.65.61) intermittently return 200 OK with empty body instead of the expected 307 redirect with token.
Observed Behavior
| Access Method | Result | Reliability |
|---|---|---|
| Direct to n8n (localhost:5678) | 307 Redirect + Token | 100% |
| Via Traefik internally | 307 or 200 | ~80% |
| Via public IP (76.150.65.61) | 307 or 200 | ~50% |
| Rapid successive requests | 307 Redirect | ~95% |
Impact When 200 OK Returned
- No scan logged to database
- No token generated
- Workflow does NOT execute
- User cannot proceed with form submission
E2E Test Results (2026-01-16)
Internal E2E Test: PASSED ✓
Token: hzOzw1HtObTPW7GhDnibDWPm
Email: e2e-existing-1768532211@test.example.com
Response: {"success":true,"message":"Thanks for entering!..."}
Subscriber created: ✓
Token marked used: ✓
Possible Root Causes
- n8n webhook registration timing - Race condition between workflow activation and webhook availability
- Connection pooling differences - HTTP/1.1 vs HTTP/2 handling through proxy chain
- Proxy caching - NGINX Proxy Manager or Traefik may be caching empty responses
- Keep-alive behavior - First request on new connection may behave differently
- Workflow execution throttling - n8n may be rate-limiting webhook executions
Diagnostic Commands
# Test from internal server (should always work) ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \ 'curl -si http://localhost:5678/webhook/scan?qr_id=test | head -3' # Test via Traefik with Host header ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \ 'curl -si -H "Host: hooks.mad-monkey-creations.com" http://localhost:80/webhook/scan?qr_id=test | head -3' # Test from remote/external server ssh -i ~/.ssh/id_ed25519_tackleadmin tackleadmin@100.125.87.114 \ 'curl -s --resolve hooks.mad-monkey-creations.com:443:76.150.65.61 -D - -o /dev/null "https://hooks.mad-monkey-creations.com/webhook/scan?qr_id=test" | head -3' # Check recent n8n logs for webhook errors ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \ 'docker logs mm-n8n --tail 50 2>&1 | grep -iE "(webhook|error|scan)"'
Investigation Steps
- Check n8n container logs during failed requests
- Verify Traefik routing configuration
- Test with connection reuse disabled (
--no-keepalive) - Check NGINX Proxy Manager proxy settings for hooks domain
- Monitor n8n memory/CPU during requests
- Consider adding health check endpoint that doesn't use webhooks
Potential Fixes
| Option | Approach | Complexity |
|---|---|---|
| A | Restart n8n container and test stability | Low |
| B | Adjust Traefik timeout/retry settings | Medium |
| C | Add retry logic in proxy configuration | Medium |
| D | Switch webhook to POST method (may have different behavior) | Medium |
| E | Add middleware to force consistent response handling | High |
Status
Discovered: January 16, 2026
Current State: BLOCKING - External users cannot reliably use the system
Internal E2E: PASSING - Core workflow functions correctly
Priority: HIGH - Must diagnose proxy chain behavior
Resolved Issues
Issue #1: Domain Strategy - Multiple Domains vs Single Domain
Problem
Initial architecture used multiple subdomains under mmc.fit domain:
list.mmc.fit- Email list managementbenny.mmc.fit- Landing pagemmlnk.us- Short URLs
This created confusion and complexity in DNS management, SSL certificates, and branding.
Impact
- Confusing user experience with multiple domains
- Complex SSL certificate management
- Harder to maintain brand consistency
- Additional DNS configuration required
Solution
Consolidated to two-domain strategy:
- mmlnk.us - Short URL domain (clean, dedicated, public-facing)
- mad-monkey-creations.com - Main brand domain with subpaths:
/benny- Landing pagehooks.subdomain - n8n webhooks (internal)mail.subdomain - Email server
Timeline
Identified domain complexity issue
Evaluated alternatives and decided on two-domain strategy
Updated documentation and architecture diagrams
Status
Resolution Date: December 17, 2024
Outcome: Simplified architecture improves user experience and reduces operational complexity
Issue #2: Email Provider Selection - Brevo vs ForwardMail.net
Problem
Initial plan was to use Brevo (formerly Sendinblue) as SMTP relay:
- Free tier: 300 emails/day
- Well-documented API
- Direct integration with Listmonk
However, during implementation discovered limitations and switched to ForwardMail.net
Comparison
| Feature | Brevo | ForwardMail.net |
|---|---|---|
| Free Tier | 300 emails/day | More generous limits |
| Setup Complexity | Moderate | Simple |
| Deliverability | Good | Excellent |
| SMTP Support | Yes | Yes |
Solution
Switched to ForwardMail.net for improved deliverability and simpler configuration
Status
Resolution Date: December 18, 2024
Outcome: ForwardMail.net selected as primary SMTP provider
Issue #3: Landing Page Hosting - Separate Container vs Subpath
Problem
Uncertainty about whether to host landing page as:
- Separate Docker container on 10.0.0.154
- Subpath on existing website (
/benny)
Decision Factors
| Approach | Pros | Cons |
|---|---|---|
| Separate Container | Isolated, scalable, independent deployment | More complexity, additional infrastructure |
| Subpath on Main Site | Simpler, unified domain, easier SSL | Coupled to main website deployment |
Solution
Use mad-monkey-creations.com/benny subpath approach for simplicity
Can migrate to separate container later if traffic demands it
Status
Resolution Date: December 19, 2024
Outcome: Landing page will be hosted at /benny subpath
Issue #4: QR Code Format - SVG vs PNG
Problem
Decision needed on QR code image format for printing on PVC cards
Analysis
| Format | Pros | Cons | Best Use |
|---|---|---|---|
| PNG | Universal support, simple | Fixed resolution, larger files | Web display, standard print |
| SVG | Infinite scaling, smaller files | Some printers need rasterization | Professional printing, flexible sizes |
Solution
Generate both formats:
- SVG - Primary format for print production (scalable to any size)
- PNG - Backup format at 300 DPI for compatibility
Status
Resolution Date: December 20, 2024
Outcome: Dual-format approach ensures maximum flexibility
Issue #5: Card Design - Move Code Placement
Problem
Initial design placed move code on front of traveler cards, but this caused confusion about whether it's required for nest cards
Impact
- Visual clutter on nest cards if included
- User confusion about when to use move code
- Potential for accidental move code usage on nest cards
Solution
Final Design:
- Nest Cards (Back):
- QR code
- Short URL
- Simple instructions: "Scan. Tell us where. Confirm email. Monthly winner!"
- Traveler Cards (Back):
- QR code
- Short URL
- Instructions + move code section
- Move code clearly labeled: "MOVE CODE: XXXX-XXXX"
Status
Resolution Date: December 21, 2024
Outcome: Separate card designs for nest vs traveler cards improve clarity
Open Blockers
Blocker #1: Port 25 (SMTP) Blocked by ISP
Problem
Comcast blocks outbound SMTP connections on port 25, preventing direct email delivery from self-hosted mail server
Discovery
# Port scan results Port 25: FILTERED (blocked by ISP) Port 465: OPEN (SMTPS) Port 587: OPEN (Submission) Port 993: OPEN (IMAPS)
Impact
- CRITICAL: Cannot send emails directly without SMTP relay
- Prevents self-hosted email server from functioning independently
- Requires third-party SMTP relay service
Workaround
Use ForwardMail.net as SMTP relay on ports 465 or 587
Configure Listmonk to send via ForwardMail.net instead of direct SMTP
Potential Solutions
- Continue with relay (RECOMMENDED): Use ForwardMail.net indefinitely
- Business ISP: Upgrade to business internet plan (unblocks port 25)
- VPS/Cloud: Move mail server to cloud provider (AWS, Digital Ocean, etc.)
- ISP Request: Contact Comcast to request port 25 unblock (rarely successful)
Status
Discovered: December 22, 2024
Current State: Workaround in place, not blocking development
Priority: Monitor but not urgent (relay solution works)
Issue #6: Listmonk API Credentials Configuration
Problem
Listmonk API access was not configured for n8n integration. Initial attempts to use admin credentials failed.
Discovery
Listmonk v6 uses a different authentication model for API access:
- API users have type='api' in the users table
- API tokens are stored as plaintext (not bcrypt hashed)
- API authentication format:
api_user:token
Solution
Created dedicated API user for n8n integration:
INSERT INTO users (username, password_login, password, email, name, type, user_role_id, status)
VALUES ('n8n_api', false, 'madmonkey-api-token-2026', 'api@mad-monkey-creations.com', 'n8n API User', 'api', 1, 'enabled');
Updated n8n Listmonk credential from admin:password to n8n_api:madmonkey-api-token-2026
Status
Resolved: January 8, 2026
Outcome: Listmonk API access working, n8n can create subscribers
Issue #7: SSL Proxy Migration from 10.0.0.3 to 10.0.0.251
Problem
SSL Proxy needed to be migrated from old host (10.0.0.3) to new docker host (10.0.0.251)
Solution
Successfully migrated NGINX Proxy Manager:
- Restored backup (npm-backup-2026-01-08_0342.tgz) to new host
- Created docker-compose.yml with jlesage/nginx-proxy-manager image
- Preserved all 27 proxy hosts and 5 redirection hosts
- Updated all documentation references from 10.0.0.3 to 10.0.0.251
Status
Resolved: January 8, 2026
Outcome: All services accessible through new SSL proxy at 10.0.0.251
Issue #8: Shlink URLs Pointing to Wrong Path
Problem
Shlink short URLs (T01-T05) were pointing to /scan but NPM proxy configuration expected /webhook/scan
Solution
Updated all Shlink short URLs via API:
curl -X PATCH "http://10.0.0.250:8081/rest/v3/short-urls/T01" \
-H "X-Api-Key: 71870798-7566-4fce-bff3-055a3a076710" \
-H "Content-Type: application/json" \
-d '{"longUrl": "https://hooks.mad-monkey-creations.com/webhook/scan?qr_id=T01"}'
Status
Resolved: January 8, 2026
Outcome: All short URLs now correctly redirect through n8n webhooks
Issue #13: External Access Timing Out
Problem
External access via 76.150.65.61 was timing out. Internal access through SSL proxy (10.0.0.251) works correctly.
Original Symptoms
- Internal access to 10.0.0.251:80, :81, :443 works
- Internal proxy through SSL to 10.0.0.250 services works
- External access via 76.150.65.61 times out
Resolution
Port forwarding was configured correctly. External access now works, but with intermittent webhook issues (see Blocker #2 above).
Current State (2026-01-16)
- ✓ External HTTPS access works
- ✓ All services reachable via public IP
- ⚠ Webhook responses are intermittent (separate issue)
Status
Discovered: January 8, 2026
Resolved: January 16, 2026
Note: Webhook intermittency tracked separately as Blocker #2
Active Workarounds
Workaround #1: SMTP Relay via ForwardMail.net
Original Issue
Port 25 blocked by Comcast ISP (see Blocker #1 above)
Workaround Details
Instead of direct SMTP on port 25, configure Listmonk to relay through ForwardMail.net:
# Listmonk SMTP Configuration [smtp] host = smtp.forwardmail.net port = 587 username = hello@mad-monkey-creations.com password = [API key or password] from_email = hello@mad-monkey-creations.com tls = true auth_protocol = login
Limitations
- Dependent on third-party service availability
- Subject to ForwardMail.net rate limits
- Additional point of failure in email delivery
- May incur costs if exceeding free tier
Benefits
- Better deliverability (established sender reputation)
- Automatic SPF/DKIM signing
- No need to manage IP reputation
- Works around ISP port blocking
Status
Implemented: Pending (waiting on credentials)
Long-term Plan: Keep using relay (benefits outweigh negatives)
Items Being Monitored
Monitor #1: Database Performance
Description
PostgreSQL performance with expected load of:
- 1,000-5,000 scans per month
- 200-1,000 verified entries per month
- Multiple concurrent n8n workflow executions
Monitoring Plan
- Track query execution times in PostgreSQL logs
- Monitor connection pool usage
- Watch for slow queries (>500ms)
- Review index usage and add as needed
Thresholds
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Query Time | >500ms | >2s | Add indexes, optimize queries |
| Connection Pool | >80% used | >95% used | Increase pool size |
| Disk Usage | >70% | >85% | Expand storage |
Status
Current State: Not yet in production, baseline metrics TBD
Review Frequency: Weekly during pilot, monthly after full launch
Monitor #2: Email Deliverability & Spam Score
Description
Monitor email deliverability rates and spam scores to ensure emails reach subscribers
Key Metrics
- Delivery Rate: Target >95%
- Bounce Rate: Target <5%
- Spam Complaints: Target <0.1%
- Open Rate: Target >20%
- Click Rate: Target >3%
- Spam Score: Target 10/10 on mail-tester.com
Monitoring Tools
- Listmonk built-in analytics
- ForwardMail.net delivery reports
- Periodic mail-tester.com checks
- MXToolbox blacklist monitoring
Action Items if Issues Detected
- Review email content for spam triggers
- Verify SPF/DKIM/DMARC records
- Check sender IP reputation
- Review subscriber engagement (re-engagement campaign for inactive)
- Implement double opt-in strictly (already planned)
Status
Current State: Pending email system deployment
Review Frequency: Daily during pilot, weekly after launch
Monitor #3: Dynamic External IP Changes
Description
External IP (76.150.65.61) is dynamic and may change without notice, breaking DNS resolution
Impact
- All external services become unreachable when IP changes
- DNS records need manual update
- Downtime until DNS propagates (up to 24 hours)
- Email delivery may be affected
Monitoring Plan
- Daily check of current external IP vs DNS records
- Automated alert when IP mismatch detected
- Health checks for all external endpoints
- Monitor email deliverability metrics
Response Procedure
When external IP changes:
- Detect new IP:
curl ifconfig.me - Follow DNS-UPDATE-PROCESS.md
- Update A records for all domains
- Verify DNS propagation:
dig +short domain.com - Test all services (n8n, Shlink, Listmonk, Traefik)
- Update documentation with new IP
Mitigation Options
| Option | Cost | Complexity | Status |
|---|---|---|---|
| Dynamic DNS (DDNS) | Free | Low | Recommended for pilot |
| Static IP from ISP | $5-15/mo | Low | Consider for production |
| Move to VPS/Cloud | $10-50/mo | High | Future consideration |
Status
Current IP: 76.150.65.61 (as of January 1, 2026)
Last Change: Unknown
Review Frequency: Daily automated check
Monitor #4: Stuck Card Rate
Description
Monitor percentage of cards that go inactive (no scans for extended periods)
Definitions
- Stuck Card: No scans in 10+ days
- Stagnant Card: No scans in 45+ days
Target Thresholds
| Metric | Target | Warning | Action |
|---|---|---|---|
| Stuck Card Rate | <10% | >20% | Re-engagement campaign, venue check |
| Stagnant Card Rate | <5% | >10% | Retrieve and redeploy, contact venue |
Automated Alerts
n8n workflow will run daily to:
- Identify stuck cards (10+ days)
- Identify stagnant cards (45+ days)
- Send email report to admin
- Trigger rescue mission campaign for stagnant cards
Status
Current State: Workflow not yet built
Review Frequency: Daily automated check, weekly manual review
Lessons Learned
Key Takeaways from Issues
- Simplicity wins: Two-domain strategy is cleaner than multiple subdomains
- Research early: ISP port blocking should have been checked before planning self-hosted email
- Use managed services where it makes sense: Email relay services provide better deliverability than self-hosted
- Document decisions: Capturing why decisions were made helps future troubleshooting
- Plan for flexibility: Dual-format QR codes and modular architecture allow for changes
Recently Resolved (January 2026)
Issue #14: Duplicate Opt-in Emails Sent
Problem
When a user submitted the entry form, they received TWO opt-in emails:
- Custom Template 10 email from n8n workflow - styled but opt-in URL was BROKEN
- Default Listmonk opt-in email - worked correctly
Root Cause
n8n workflow "2. Email Capture" had a "Send Welcome Email" node that sent Template 10 via Listmonk transactional API, AND Listmonk's built-in app.send_optin_confirmation=true also sent its default opt-in email.
Solution
Removed the "Send Welcome Email" node from the n8n workflow entirely. Now only Listmonk's built-in opt-in email is sent (via app.send_optin_confirmation=true).
Status
Discovered: January 14, 2026
Resolved: January 18, 2026
Outcome: Users now receive only ONE opt-in email with working confirmation link
Issue #13: External Webhook Intermittent Response
Problem
External requests to the scan webhook via public IP intermittently returned 200 OK with empty body instead of 307 redirect.
Solution
Updated NPM proxy configuration at /config/nginx/proxy_host/25.conf with:
proxy_redirect offproxy_buffering offproxy_http_version 1.1proxy_cache offproxy_cache_bypass 1proxy_no_cache 1
Note: n8n webhook registration can still become stale. Workaround: deactivate/reactivate workflows via API to re-register webhooks.
Status
Discovered: January 16, 2026
Resolved: January 18, 2026 (partially - workaround in place)
Issue #9: Email Capture Workflow UUID Error
Problem
Listmonk returning "Invalid UUID(s)" error when users clicked opt-in links in Template 10 emails.
Root Cause
Manual opt-in URL construction in workflow was using incorrect format/data.
Solution
Switched to using Listmonk transactional email (Template 10) which automatically generates correct opt-in URLs using proper template variables.
Status
Resolved: January 11, 2026
Note: This fix led to the discovery of the duplicate email issue (current blocker)
Issue #10: Drip Campaign Nodes Disconnected
Problem
Drip campaign workflow had all nodes created but no connections between them.
Solution
Connected all nodes in proper sequence: Check Day → Wait 24 Hours → Send Day 1 Story → Wait 48 Hours → Send Day 3 Reward
Status
Resolved: January 11, 2026
Issue #11: Monthly Winner Using Old Tables
Problem
SQL queries in Monthly Winner workflow were referencing deprecated entries and users tables.
Solution
Updated all SQL queries to use new subscribers and contest_entries tables.
Status
Resolved: January 11, 2026
Issue #12: Missing qrIdInput JavaScript Variable
Problem
Landing page form submission failing because qrIdInput variable was not defined in JavaScript.
Solution
Added the missing variable definition in /home/bennybeen/mad-monkey/landing-page/e/index.html.
Status
Resolved: January 14, 2026