Last Updated: January 20, 2026 at 01:40 UTC

Issue Summary

Category Count Status
CRITICAL Blockers 0 NONE
Resolved Issues 18 RESOLVED
Open (Non-blocking) 1 OPEN
Workarounds in Place 3 WORKAROUND
Items Being Monitored 4 MONITORING

Previously Critical: Now Resolved

RESOLVED WAS CRITICAL

Resolved: n8n Webhook Caching Issue

Original Problem

Added a "Trigger Drip Campaign" HTTP Request node to the Confirmation workflow, but webhook used cached old workflow version.

Resolution (January 19, 2026)

The webhook path change approach worked:

  1. Changed webhook path from listmonk/confirm to listmonk/confirm-temp
  2. Changed it back to listmonk/confirm
  3. Deactivated and reactivated the workflow

Key insight: n8n API PUT requests fail with "must NOT have additional properties" unless you only include {name, nodes, connections, settings}.

Current Status

Full flow now works: Confirmation → Mark Verified → Create Contest Entry → Trigger Drip Campaign

Resolved: January 19, 2026

Active Workarounds

WORKAROUND IN PLACE MEDIUM SEVERITY

Confirmation Workflow Webhook Not Auto-Registering on Startup

Problem

The 3. Confirmation workflow (ID: eoW1DejeZlfPA3l1) is not registering its webhook when n8n starts, even though the database shows active: true.

Evidence

# n8n startup log shows only 4 workflows activated:
Start Active Workflows:
Activated workflow "2. Email Capture" (ID: AeiltZEKb4YvAbQG)
Activated workflow "4. Story & Reward Drip" (ID: YSzXpljWCgHFQo0g)
Activated workflow "5. Monthly Winner Selection" (ID: cKATiI2Kh5UhIBIX)
Activated workflow "1. Scan Intake" (ID: XjqHOdsNzN1Gg4n6)

# Missing: "3. Confirmation" workflow!

# Result when webhook is called:
Received request for unknown webhook: The requested webhook "POST listmonk/confirm" is not registered.

Impact

  • After n8n restart, confirmation flow breaks
  • Users click confirm link but n8n doesn't process it
  • Manual intervention required after each restart

Workaround

After each n8n restart, manually toggle the workflow via API:

# Deactivate
curl -s -X POST -H "X-N8N-API-KEY: $API_KEY" \
  "http://localhost:5678/api/v1/workflows/eoW1DejeZlfPA3l1/deactivate"

# Wait 1 second, then activate
curl -s -X POST -H "X-N8N-API-KEY: $API_KEY" \
  "http://localhost:5678/api/v1/workflows/eoW1DejeZlfPA3l1/activate"

Potential Permanent Fixes

Option Approach Complexity
A (Recommended) Add startup script to docker-compose that toggles workflow after n8n starts Low
B Recreate workflow from scratch as "3. Confirmation v2" Medium
C Investigate why n8n skips this specific workflow on startup Medium
D Upgrade n8n version and test if issue persists Medium

Status

Discovered: January 20, 2026

Current State: Workaround in place - E2E tests passing

Priority: MEDIUM - System works with manual intervention

RESOLVED HIGH SEVERITY

Issue #15: Listmonk Confirmation Not Triggering n8n

Original Problem

Listmonk has no built-in webhook for subscriber confirmations - opt-in flow was entirely internal.

Solution Implemented

Added custom JavaScript to Listmonk's appearance.public.custom_js that:

  • Detects when confirmation success page loads
  • Extracts subscriber UUID from URL
  • POSTs to n8n webhook to trigger confirmation workflow

Result

Confirmation webhook now fires correctly. Subscriber verification and contest entry creation work. Only remaining issue is the auto-drip trigger (see blocker above).

Status

Resolved: January 18, 2026

WORKAROUND IN PLACE MEDIUM SEVERITY

External Webhook Intermittent Response (Partially Resolved)

Problem

External requests to the scan webhook (/webhook/scan) via public IP (76.150.65.61) intermittently return 200 OK with empty body instead of the expected 307 redirect with token.

Observed Behavior

Access Method Result Reliability
Direct to n8n (localhost:5678) 307 Redirect + Token 100%
Via Traefik internally 307 or 200 ~80%
Via public IP (76.150.65.61) 307 or 200 ~50%
Rapid successive requests 307 Redirect ~95%

Impact When 200 OK Returned

  • No scan logged to database
  • No token generated
  • Workflow does NOT execute
  • User cannot proceed with form submission

E2E Test Results (2026-01-16)

Internal E2E Test: PASSED ✓

Token: hzOzw1HtObTPW7GhDnibDWPm
Email: e2e-existing-1768532211@test.example.com
Response: {"success":true,"message":"Thanks for entering!..."}
Subscriber created: ✓
Token marked used: ✓

Possible Root Causes

  1. n8n webhook registration timing - Race condition between workflow activation and webhook availability
  2. Connection pooling differences - HTTP/1.1 vs HTTP/2 handling through proxy chain
  3. Proxy caching - NGINX Proxy Manager or Traefik may be caching empty responses
  4. Keep-alive behavior - First request on new connection may behave differently
  5. Workflow execution throttling - n8n may be rate-limiting webhook executions

Diagnostic Commands

# Test from internal server (should always work)
ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \
  'curl -si http://localhost:5678/webhook/scan?qr_id=test | head -3'

# Test via Traefik with Host header
ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \
  'curl -si -H "Host: hooks.mad-monkey-creations.com" http://localhost:80/webhook/scan?qr_id=test | head -3'

# Test from remote/external server
ssh -i ~/.ssh/id_ed25519_tackleadmin tackleadmin@100.125.87.114 \
  'curl -s --resolve hooks.mad-monkey-creations.com:443:76.150.65.61 -D - -o /dev/null "https://hooks.mad-monkey-creations.com/webhook/scan?qr_id=test" | head -3'

# Check recent n8n logs for webhook errors
ssh -i ~/.ssh/id_ed25519_250 bennybeen@10.0.0.250 \
  'docker logs mm-n8n --tail 50 2>&1 | grep -iE "(webhook|error|scan)"'

Investigation Steps

  1. Check n8n container logs during failed requests
  2. Verify Traefik routing configuration
  3. Test with connection reuse disabled (--no-keepalive)
  4. Check NGINX Proxy Manager proxy settings for hooks domain
  5. Monitor n8n memory/CPU during requests
  6. Consider adding health check endpoint that doesn't use webhooks

Potential Fixes

Option Approach Complexity
A Restart n8n container and test stability Low
B Adjust Traefik timeout/retry settings Medium
C Add retry logic in proxy configuration Medium
D Switch webhook to POST method (may have different behavior) Medium
E Add middleware to force consistent response handling High

Status

Discovered: January 16, 2026

Current State: BLOCKING - External users cannot reliably use the system

Internal E2E: PASSING - Core workflow functions correctly

Priority: HIGH - Must diagnose proxy chain behavior

Resolved Issues

RESOLVED HIGH SEVERITY

Issue #1: Domain Strategy - Multiple Domains vs Single Domain

Problem

Initial architecture used multiple subdomains under mmc.fit domain:

  • list.mmc.fit - Email list management
  • benny.mmc.fit - Landing page
  • mmlnk.us - Short URLs

This created confusion and complexity in DNS management, SSL certificates, and branding.

Impact

  • Confusing user experience with multiple domains
  • Complex SSL certificate management
  • Harder to maintain brand consistency
  • Additional DNS configuration required

Solution

Consolidated to two-domain strategy:

  • mmlnk.us - Short URL domain (clean, dedicated, public-facing)
  • mad-monkey-creations.com - Main brand domain with subpaths:
    • /benny - Landing page
    • hooks. subdomain - n8n webhooks (internal)
    • mail. subdomain - Email server

Timeline

Dec 15, 2024

Identified domain complexity issue

Dec 16, 2024

Evaluated alternatives and decided on two-domain strategy

Dec 17, 2024

Updated documentation and architecture diagrams

Status

Resolution Date: December 17, 2024

Outcome: Simplified architecture improves user experience and reduces operational complexity

RESOLVED MEDIUM SEVERITY

Issue #2: Email Provider Selection - Brevo vs ForwardMail.net

Problem

Initial plan was to use Brevo (formerly Sendinblue) as SMTP relay:

  • Free tier: 300 emails/day
  • Well-documented API
  • Direct integration with Listmonk

However, during implementation discovered limitations and switched to ForwardMail.net

Comparison

Feature Brevo ForwardMail.net
Free Tier 300 emails/day More generous limits
Setup Complexity Moderate Simple
Deliverability Good Excellent
SMTP Support Yes Yes

Solution

Switched to ForwardMail.net for improved deliverability and simpler configuration

Status

Resolution Date: December 18, 2024

Outcome: ForwardMail.net selected as primary SMTP provider

RESOLVED LOW SEVERITY

Issue #3: Landing Page Hosting - Separate Container vs Subpath

Problem

Uncertainty about whether to host landing page as:

  • Separate Docker container on 10.0.0.154
  • Subpath on existing website (/benny)

Decision Factors

Approach Pros Cons
Separate Container Isolated, scalable, independent deployment More complexity, additional infrastructure
Subpath on Main Site Simpler, unified domain, easier SSL Coupled to main website deployment

Solution

Use mad-monkey-creations.com/benny subpath approach for simplicity

Can migrate to separate container later if traffic demands it

Status

Resolution Date: December 19, 2024

Outcome: Landing page will be hosted at /benny subpath

RESOLVED LOW SEVERITY

Issue #4: QR Code Format - SVG vs PNG

Problem

Decision needed on QR code image format for printing on PVC cards

Analysis

Format Pros Cons Best Use
PNG Universal support, simple Fixed resolution, larger files Web display, standard print
SVG Infinite scaling, smaller files Some printers need rasterization Professional printing, flexible sizes

Solution

Generate both formats:

  • SVG - Primary format for print production (scalable to any size)
  • PNG - Backup format at 300 DPI for compatibility

Status

Resolution Date: December 20, 2024

Outcome: Dual-format approach ensures maximum flexibility

RESOLVED MEDIUM SEVERITY

Issue #5: Card Design - Move Code Placement

Problem

Initial design placed move code on front of traveler cards, but this caused confusion about whether it's required for nest cards

Impact

  • Visual clutter on nest cards if included
  • User confusion about when to use move code
  • Potential for accidental move code usage on nest cards

Solution

Final Design:

  • Nest Cards (Back):
    • QR code
    • Short URL
    • Simple instructions: "Scan. Tell us where. Confirm email. Monthly winner!"
  • Traveler Cards (Back):
    • QR code
    • Short URL
    • Instructions + move code section
    • Move code clearly labeled: "MOVE CODE: XXXX-XXXX"

Status

Resolution Date: December 21, 2024

Outcome: Separate card designs for nest vs traveler cards improve clarity

Open Blockers

OPEN CRITICAL

Blocker #1: Port 25 (SMTP) Blocked by ISP

Problem

Comcast blocks outbound SMTP connections on port 25, preventing direct email delivery from self-hosted mail server

Discovery

# Port scan results
Port 25:  FILTERED (blocked by ISP)
Port 465: OPEN (SMTPS)
Port 587: OPEN (Submission)
Port 993: OPEN (IMAPS)

Impact

  • CRITICAL: Cannot send emails directly without SMTP relay
  • Prevents self-hosted email server from functioning independently
  • Requires third-party SMTP relay service

Workaround

Use ForwardMail.net as SMTP relay on ports 465 or 587

Configure Listmonk to send via ForwardMail.net instead of direct SMTP

Potential Solutions

  1. Continue with relay (RECOMMENDED): Use ForwardMail.net indefinitely
  2. Business ISP: Upgrade to business internet plan (unblocks port 25)
  3. VPS/Cloud: Move mail server to cloud provider (AWS, Digital Ocean, etc.)
  4. ISP Request: Contact Comcast to request port 25 unblock (rarely successful)

Status

Discovered: December 22, 2024

Current State: Workaround in place, not blocking development

Priority: Monitor but not urgent (relay solution works)

RESOLVED HIGH SEVERITY

Issue #6: Listmonk API Credentials Configuration

Problem

Listmonk API access was not configured for n8n integration. Initial attempts to use admin credentials failed.

Discovery

Listmonk v6 uses a different authentication model for API access:

  • API users have type='api' in the users table
  • API tokens are stored as plaintext (not bcrypt hashed)
  • API authentication format: api_user:token

Solution

Created dedicated API user for n8n integration:

INSERT INTO users (username, password_login, password, email, name, type, user_role_id, status)
VALUES ('n8n_api', false, 'madmonkey-api-token-2026', 'api@mad-monkey-creations.com', 'n8n API User', 'api', 1, 'enabled');

Updated n8n Listmonk credential from admin:password to n8n_api:madmonkey-api-token-2026

Status

Resolved: January 8, 2026

Outcome: Listmonk API access working, n8n can create subscribers

RESOLVED MEDIUM SEVERITY

Issue #7: SSL Proxy Migration from 10.0.0.3 to 10.0.0.251

Problem

SSL Proxy needed to be migrated from old host (10.0.0.3) to new docker host (10.0.0.251)

Solution

Successfully migrated NGINX Proxy Manager:

  • Restored backup (npm-backup-2026-01-08_0342.tgz) to new host
  • Created docker-compose.yml with jlesage/nginx-proxy-manager image
  • Preserved all 27 proxy hosts and 5 redirection hosts
  • Updated all documentation references from 10.0.0.3 to 10.0.0.251

Status

Resolved: January 8, 2026

Outcome: All services accessible through new SSL proxy at 10.0.0.251

RESOLVED MEDIUM SEVERITY

Issue #8: Shlink URLs Pointing to Wrong Path

Problem

Shlink short URLs (T01-T05) were pointing to /scan but NPM proxy configuration expected /webhook/scan

Solution

Updated all Shlink short URLs via API:

curl -X PATCH "http://10.0.0.250:8081/rest/v3/short-urls/T01" \
  -H "X-Api-Key: 71870798-7566-4fce-bff3-055a3a076710" \
  -H "Content-Type: application/json" \
  -d '{"longUrl": "https://hooks.mad-monkey-creations.com/webhook/scan?qr_id=T01"}'

Status

Resolved: January 8, 2026

Outcome: All short URLs now correctly redirect through n8n webhooks

RESOLVED HIGH SEVERITY

Issue #13: External Access Timing Out

Problem

External access via 76.150.65.61 was timing out. Internal access through SSL proxy (10.0.0.251) works correctly.

Original Symptoms

  • Internal access to 10.0.0.251:80, :81, :443 works
  • Internal proxy through SSL to 10.0.0.250 services works
  • External access via 76.150.65.61 times out

Resolution

Port forwarding was configured correctly. External access now works, but with intermittent webhook issues (see Blocker #2 above).

Current State (2026-01-16)

  • ✓ External HTTPS access works
  • ✓ All services reachable via public IP
  • ⚠ Webhook responses are intermittent (separate issue)

Status

Discovered: January 8, 2026

Resolved: January 16, 2026

Note: Webhook intermittency tracked separately as Blocker #2

Active Workarounds

WORKAROUND MEDIUM SEVERITY

Workaround #1: SMTP Relay via ForwardMail.net

Original Issue

Port 25 blocked by Comcast ISP (see Blocker #1 above)

Workaround Details

Instead of direct SMTP on port 25, configure Listmonk to relay through ForwardMail.net:

# Listmonk SMTP Configuration
[smtp]
host = smtp.forwardmail.net
port = 587
username = hello@mad-monkey-creations.com
password = [API key or password]
from_email = hello@mad-monkey-creations.com
tls = true
auth_protocol = login

Limitations

  • Dependent on third-party service availability
  • Subject to ForwardMail.net rate limits
  • Additional point of failure in email delivery
  • May incur costs if exceeding free tier

Benefits

  • Better deliverability (established sender reputation)
  • Automatic SPF/DKIM signing
  • No need to manage IP reputation
  • Works around ISP port blocking

Status

Implemented: Pending (waiting on credentials)

Long-term Plan: Keep using relay (benefits outweigh negatives)

Items Being Monitored

MONITORING LOW SEVERITY

Monitor #1: Database Performance

Description

PostgreSQL performance with expected load of:

  • 1,000-5,000 scans per month
  • 200-1,000 verified entries per month
  • Multiple concurrent n8n workflow executions

Monitoring Plan

  • Track query execution times in PostgreSQL logs
  • Monitor connection pool usage
  • Watch for slow queries (>500ms)
  • Review index usage and add as needed

Thresholds

Metric Warning Critical Action
Query Time >500ms >2s Add indexes, optimize queries
Connection Pool >80% used >95% used Increase pool size
Disk Usage >70% >85% Expand storage

Status

Current State: Not yet in production, baseline metrics TBD

Review Frequency: Weekly during pilot, monthly after full launch

MONITORING MEDIUM SEVERITY

Monitor #2: Email Deliverability & Spam Score

Description

Monitor email deliverability rates and spam scores to ensure emails reach subscribers

Key Metrics

  • Delivery Rate: Target >95%
  • Bounce Rate: Target <5%
  • Spam Complaints: Target <0.1%
  • Open Rate: Target >20%
  • Click Rate: Target >3%
  • Spam Score: Target 10/10 on mail-tester.com

Monitoring Tools

  • Listmonk built-in analytics
  • ForwardMail.net delivery reports
  • Periodic mail-tester.com checks
  • MXToolbox blacklist monitoring

Action Items if Issues Detected

  1. Review email content for spam triggers
  2. Verify SPF/DKIM/DMARC records
  3. Check sender IP reputation
  4. Review subscriber engagement (re-engagement campaign for inactive)
  5. Implement double opt-in strictly (already planned)

Status

Current State: Pending email system deployment

Review Frequency: Daily during pilot, weekly after launch

MONITORING MEDIUM SEVERITY

Monitor #3: Dynamic External IP Changes

Description

External IP (76.150.65.61) is dynamic and may change without notice, breaking DNS resolution

Impact

  • All external services become unreachable when IP changes
  • DNS records need manual update
  • Downtime until DNS propagates (up to 24 hours)
  • Email delivery may be affected

Monitoring Plan

  • Daily check of current external IP vs DNS records
  • Automated alert when IP mismatch detected
  • Health checks for all external endpoints
  • Monitor email deliverability metrics

Response Procedure

When external IP changes:

  1. Detect new IP: curl ifconfig.me
  2. Follow DNS-UPDATE-PROCESS.md
  3. Update A records for all domains
  4. Verify DNS propagation: dig +short domain.com
  5. Test all services (n8n, Shlink, Listmonk, Traefik)
  6. Update documentation with new IP

Mitigation Options

Option Cost Complexity Status
Dynamic DNS (DDNS) Free Low Recommended for pilot
Static IP from ISP $5-15/mo Low Consider for production
Move to VPS/Cloud $10-50/mo High Future consideration

Status

Current IP: 76.150.65.61 (as of January 1, 2026)

Last Change: Unknown

Review Frequency: Daily automated check

MONITORING LOW SEVERITY

Monitor #4: Stuck Card Rate

Description

Monitor percentage of cards that go inactive (no scans for extended periods)

Definitions

  • Stuck Card: No scans in 10+ days
  • Stagnant Card: No scans in 45+ days

Target Thresholds

Metric Target Warning Action
Stuck Card Rate <10% >20% Re-engagement campaign, venue check
Stagnant Card Rate <5% >10% Retrieve and redeploy, contact venue

Automated Alerts

n8n workflow will run daily to:

  • Identify stuck cards (10+ days)
  • Identify stagnant cards (45+ days)
  • Send email report to admin
  • Trigger rescue mission campaign for stagnant cards

Status

Current State: Workflow not yet built

Review Frequency: Daily automated check, weekly manual review

Lessons Learned

Key Takeaways from Issues

  1. Simplicity wins: Two-domain strategy is cleaner than multiple subdomains
  2. Research early: ISP port blocking should have been checked before planning self-hosted email
  3. Use managed services where it makes sense: Email relay services provide better deliverability than self-hosted
  4. Document decisions: Capturing why decisions were made helps future troubleshooting
  5. Plan for flexibility: Dual-format QR codes and modular architecture allow for changes

Recently Resolved (January 2026)

RESOLVED CRITICAL SEVERITY

Issue #14: Duplicate Opt-in Emails Sent

Problem

When a user submitted the entry form, they received TWO opt-in emails:

  1. Custom Template 10 email from n8n workflow - styled but opt-in URL was BROKEN
  2. Default Listmonk opt-in email - worked correctly

Root Cause

n8n workflow "2. Email Capture" had a "Send Welcome Email" node that sent Template 10 via Listmonk transactional API, AND Listmonk's built-in app.send_optin_confirmation=true also sent its default opt-in email.

Solution

Removed the "Send Welcome Email" node from the n8n workflow entirely. Now only Listmonk's built-in opt-in email is sent (via app.send_optin_confirmation=true).

Status

Discovered: January 14, 2026

Resolved: January 18, 2026

Outcome: Users now receive only ONE opt-in email with working confirmation link

RESOLVED HIGH SEVERITY

Issue #13: External Webhook Intermittent Response

Problem

External requests to the scan webhook via public IP intermittently returned 200 OK with empty body instead of 307 redirect.

Solution

Updated NPM proxy configuration at /config/nginx/proxy_host/25.conf with:

  • proxy_redirect off
  • proxy_buffering off
  • proxy_http_version 1.1
  • proxy_cache off
  • proxy_cache_bypass 1
  • proxy_no_cache 1

Note: n8n webhook registration can still become stale. Workaround: deactivate/reactivate workflows via API to re-register webhooks.

Status

Discovered: January 16, 2026

Resolved: January 18, 2026 (partially - workaround in place)

RESOLVED HIGH SEVERITY

Issue #9: Email Capture Workflow UUID Error

Problem

Listmonk returning "Invalid UUID(s)" error when users clicked opt-in links in Template 10 emails.

Root Cause

Manual opt-in URL construction in workflow was using incorrect format/data.

Solution

Switched to using Listmonk transactional email (Template 10) which automatically generates correct opt-in URLs using proper template variables.

Status

Resolved: January 11, 2026

Note: This fix led to the discovery of the duplicate email issue (current blocker)

RESOLVED MEDIUM SEVERITY

Issue #10: Drip Campaign Nodes Disconnected

Problem

Drip campaign workflow had all nodes created but no connections between them.

Solution

Connected all nodes in proper sequence: Check Day → Wait 24 Hours → Send Day 1 Story → Wait 48 Hours → Send Day 3 Reward

Status

Resolved: January 11, 2026

RESOLVED MEDIUM SEVERITY

Issue #11: Monthly Winner Using Old Tables

Problem

SQL queries in Monthly Winner workflow were referencing deprecated entries and users tables.

Solution

Updated all SQL queries to use new subscribers and contest_entries tables.

Status

Resolved: January 11, 2026

RESOLVED LOW SEVERITY

Issue #12: Missing qrIdInput JavaScript Variable

Problem

Landing page form submission failing because qrIdInput variable was not defined in JavaScript.

Solution

Added the missing variable definition in /home/bennybeen/mad-monkey/landing-page/e/index.html.

Status

Resolved: January 14, 2026