Troubleshooting Guide

Comprehensive troubleshooting guide and runbook for the Intercom integration.

Quick Diagnostics

Health Check Checklist

Cron Job Running
```
doctl apps list | grep sync-intercom
```

Environment Variables Set

# Check in DigitalOcean dashboard
# Settings → Environment Variables
# Verify: INTERCOM_ACCESS_TOKEN, SUPABASE_URL, SUPABASE_KEY, MYSQL_*

Recent Sync Success

doctl apps logs <app-id> --component sync-intercom | grep "Intercom Daily Sync Summary"

Webhook Active

# Check in Intercom dashboard
# Settings → Developers → Webhooks
# Verify: Active, URL correct, Topics selected

Platform Server Running

curl https://platform.natca.org/api/health

Common Issues

1. Daily Sync Not Running

Symptoms:

No recent logs in sync-intercom component
Contacts not being updated
Stale member data in Intercom

Diagnosis:

# Check cron job status
doctl apps list
 
# View recent deployments
doctl apps list-deployments <app-id>
 
# Check cron schedule
cat .do/app.yaml | grep -A 5 "sync-intercom"

Solutions:

A. Cron Job Disabled

# Re-enable in .do/app.yaml
jobs:
- name: sync-intercom
  # ... existing config
  schedule: "30 2 * * *"
 
# Deploy
git add .do/app.yaml
git commit -m "Re-enable Intercom sync cron"
git push origin main

B. Environment Variables Missing

# Add via CLI
doctl apps update <app-id> --spec .do/app.yaml
 
# Or via dashboard
# Settings → Environment Variables → Add Variable

C. Deployment Failed

# Check deployment logs
doctl apps logs <app-id> --component sync-intercom
 
# Force rebuild
doctl apps create-deployment <app-id> --force-rebuild

D. Wrong Schedule

# Verify schedule (should be "30 2 * * *" for 2:30 AM UTC)
# Update in .do/app.yaml and redeploy

Verification:

# Manually trigger sync
npm run sync:intercom
 
# Or wait for scheduled run and check logs
doctl apps logs <app-id> --component sync-intercom --follow

2. High Failure Rate

Symptoms:

Sync summary shows high "Failed" count
Logs show repeated errors for same members
Specific members not syncing

Diagnosis:

# Check recent sync logs
doctl apps logs <app-id> --component sync-intercom | grep "Failed to sync"
 
# Look for patterns
doctl apps logs <app-id> --component sync-intercom | grep "error" | sort | uniq -c

Common Causes & Solutions:

A. Blocked Contacts (FIXED - Now Automatic)

Error: 400 Bad Request: User has been blocked and is not restorable

What This Means:

Contact was marked as spam/blocked in Intercom
Cannot be unarchived without unblocking first
Previously caused sync failures

Current Behavior (Automatic Recovery): The sync now automatically handles blocked contacts:

Automatic flow:
1. Try to upsert → 409 conflict (archived contact)
2. Try to unarchive → 400 not_restorable (contact is blocked)
3. Unblock contact → 200 success
4. Unarchive contact → 200 success
5. Update contact → 200 success

Verification:

Check logs for "Successfully unblocked contact" messages
Verify contact is updated in Intercom
No manual intervention required

If Still Failing:

Verify INTERCOM_ACCESS_TOKEN has correct permissions (Read + Write)
Check Intercom API status: https://status.intercom.com (opens in a new tab)
Review error logs for unblock API failures

B. Invalid Phone Numbers

Error: 422 Validation Error: phone must be in E.164 format

Solution:

Phone formatter should catch this

Check member data in Supabase:

SELECT membernumber, phone FROM members WHERE membernumber = 12345;

Update phone or skip in sync:

// Phone formatter returns null for invalid - field is omitted
const formattedPhone = formatPhoneForIntercom(member.phone);
if (formattedPhone) {
  contactData.phone = formattedPhone;
}

C. Duplicate Email Conflicts

Error: 409 Conflict: A contact matching those details already exists

Solution:

Duplicate resolution should handle this automatically

If persisting, run audit script:

node sync/intercom/audit.js --dry-run --limit=100

Look for the specific email in audit log:

cat /tmp/intercom_audit.log | grep "email@example.com"

D. Intercom API Errors

Error: 500 Internal Server Error or 503 Service Unavailable

Solution:

Check Intercom status: https://status.intercom.com (opens in a new tab)

Implement retry logic (if not present):

async function syncWithRetry(member, retryCount = 0) {
  try {
    return await syncMember(member);
  } catch (error) {
    if (retryCount < 3 && (error.message.includes('500') || error.message.includes('503'))) {
      await sleep(Math.pow(2, retryCount) * 1000); // Exponential backoff
      return syncWithRetry(member, retryCount + 1);
    }
    throw error;
  }
}

E. Rate Limit Exceeded

Error: 429 Too Many Requests

Solution:

Rate limiting should prevent this

Check rate limit logs:

doctl apps logs <app-id> --component sync-intercom | grep "Rate limit"

Reduce requests per window if needed:

// In lib/client.js
this.maxRequestsPer10Seconds = 150; // Reduce from 166

Verification:

# Re-run sync and check failure rate
node sync/intercom/daily-sync.js --limit=100
 
# Should see lower failure count
# Failed: 0 (or < 5 for 100 members)

3. Duplicates Not Resolving

Symptoms:

Multiple contacts with same email in Intercom
Duplicate resolution logs but duplicates remain
Sync/audit script reports "duplicates resolved" but still exist

Diagnosis:

# Check for duplicate resolution logs
doctl apps logs <app-id> --component sync-intercom | grep "duplicate"
 
# Audit log shows resolution attempts
cat /tmp/intercom_audit.log | grep "DUPLICATE"
 
# Search Intercom for email to verify duplicates
# Intercom → Contacts → Search "email@example.com"

Common Causes & Solutions:

A. Archive Permission Issue

Error: 403 Forbidden: Cannot archive contact

Solution:

Verify Intercom API token has "Write" permission
Regenerate token with correct permissions:
1. Intercom → Settings → Developers → Authentication
2. Generate new token
3. Select "Read" and "Write" for Contacts
4. Update INTERCOM_ACCESS_TOKEN environment variable

B. Eventual Consistency Delay

Issue: Search returns 1 contact but 409 error says duplicates exist

Solution:

Already handled with retry logic:

if (allContactsWithEmail.length <= 1 && retryCount === 0) {
  // Wait 2 seconds for eventual consistency
  await sleep(2000);
  return await handleDuplicateContact(email, retryCount + 1);
}

If still failing, increase retry delay:

await sleep(5000); // 5 seconds instead of 2

C. Contacts Recreated Between Sync Runs

Issue: Duplicates resolved but reappear

Solution:

Check for multiple sources creating contacts:
- Webhook auto-creation
- Intercom inbox auto-creation
- Other integrations
Disable auto-contact creation in Intercom:
1. Intercom → Settings → Messenger
2. Disable "Auto-create contacts"

Ensure webhook doesn't create duplicates:

// Use upsertContact (create or update by email)
await intercomClient.upsertContact(email, contactData);

Verification:

# Run audit with dry-run
node sync/intercom/audit.js --dry-run --limit=100
 
# Check if duplicates are detected
cat /tmp/intercom_audit.log | grep "Found.*contacts with email"
 
# Run audit for real
node sync/intercom/audit.js
 
# Verify in Intercom (should be resolved)

4. Webhook Not Enriching Contacts

Symptoms:

Members message support but contact not enriched
No webhook logs in Platform
Contact remains without member_number

Diagnosis:

# Check Platform logs for webhook events
doctl apps logs <app-id> --component platform | grep "Intercom webhook"
 
# Test webhook with curl
curl -X POST https://platform.natca.org/api/intercom/webhook \
  -H "Content-Type: application/json" \
  -d '{"topic":"conversation.user.replied","data":{"item":{"id":"123"}}}'
 
# Should return: {"received":true}

Common Causes & Solutions:

A. Webhook Not Configured in Intercom

Solution:

Intercom → Settings → Developers → Webhooks
Create new webhook:
- URL: https://platform.natca.org/api/intercom/webhook
- Topics: conversation.user.created, conversation.user.replied
Enable webhook (toggle "Active")

B. Webhook URL Incorrect

Error: Intercom shows failed deliveries

Solution:

Verify URL in Intercom dashboard
Correct URL: https://platform.natca.org/api/intercom/webhook
NOT: http://... (must be HTTPS)
NOT: .../api/webhook (missing /intercom)

C. Platform Server Not Receiving Requests

Solution:

# Check Platform is running
curl https://platform.natca.org/api/health
 
# Check route exists
curl -X POST https://platform.natca.org/api/intercom/webhook \
  -H "Content-Type: application/json" \
  -d '{"topic":"test"}'
 
# Should return 200 OK with {"received":true}

D. Member Not Found by Email

Issue: Webhook processes but member not in MySQL

Solution:

Check MySQL connection in webhook logs:

doctl apps logs <app-id> --component platform | grep "Member not found"

Verify member exists in MySQL:

SELECT * FROM member m
LEFT JOIN emailinformation e ON m.id = e.memberid
WHERE e.email = 'email@example.com' AND e.isprimary = 1;

If member exists but not found, check email case sensitivity:

-- Case-insensitive search
WHERE LOWER(e.email) = LOWER('email@example.com')

Verification:

# Trigger test webhook in Intercom dashboard
# Settings → Developers → Webhooks → Send test webhook
 
# Check Platform logs
doctl apps logs <app-id> --component platform --follow
 
# Should see: "Intercom webhook received"
# Then: "Updated Intercom contact with member data" (if member found)

5. Only 1,000 Members Being Synced (FIXED)

Symptoms:

Sync completes but only ~1,000 members are updated
Total member count in Intercom doesn't match Active/Retired count in database
Some members never get synced

Diagnosis:

# Check sync logs for total member count
doctl apps logs <app-id> --component sync-intercom | grep "Total members to fetch"
 
# Should show full count (20,000+), not 1,000
# Example: "Total members to fetch: 20123 (fetching in pages of 1000)"
 
# Old behavior (FIXED):
# "Found 1000 members with email addresses"
 
# New behavior:
# "Total members to fetch: 20123"
# "Fetched page 1 (1000 members so far...)"
# "Fetched page 2 (2000 members so far...)"
# ...
# "Found 20123 members with email addresses"

Root Cause (FIXED in current version):

Supabase has a default row limit of 1,000
Previous implementation didn't use pagination
Only first 1,000 members were fetched and synced

Current Solution: The sync now uses automatic pagination:

// Implemented in fetchMembers() method
const PAGE_SIZE = 1000;
let allMembers = [];
let page = 0;
 
while (hasMore) {
  const { data, error, count } = await query
    .range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1);
 
  allMembers = allMembers.concat(data);
  page++;
  hasMore = data.length === PAGE_SIZE;
}

Verification:

# Check logs for pagination messages
doctl apps logs <app-id> --component sync-intercom | grep "Fetched page"
 
# Should see multiple pages:
# "Fetched page 1 (1000 members so far...)"
# "Fetched page 2 (2000 members so far...)"
# etc.
 
# Check final count matches expected member count
doctl apps logs <app-id> --component sync-intercom | grep "Found.*members with email"
 
# Should match your Active + Retired member count (e.g., 20,123)

If Still Only Seeing 1,000 Members:

Verify you're running the latest version of daily-sync.js
Check for pagination code in fetchMembers() method
Ensure no --limit flag is being passed to the sync command

6. Tag-Based Re-Sync Not Working

Symptoms:

Added "mynatca-sync" tag but contact not updated
Tag not being removed after adding
Member number provided but enrichment failed
Contact remains as LEAD instead of USER

Diagnosis:

# Check Platform logs for tag webhook events
doctl apps logs <app-id> --component platform | grep "tag-based sync"
 
# Check for tag processing errors
doctl apps logs <app-id> --component platform | grep "mynatca-sync"
 
# Verify conversation attribute
# In Intercom: Open conversation → Check "Member Number Verification" field
 
# Verify webhook is configured
# Intercom → Settings → Developers → Webhooks
# Should include: conversation_part.tag.created

Common Causes & Solutions:

A. Conversation Attribute Not Set

Issue: Tag added but no member_number_verification attribute

Solution:

Support team must fill in "Member Number Verification" attribute BEFORE adding tag
Attribute name must match exactly: "member_number_verification" (lowercase with underscores in API)
Verify attribute exists in Intercom: Settings → Data → Conversations

B. Webhook Topic Not Enabled

Issue: conversation_part.tag.created webhook not configured

Solution:

Go to Intercom → Settings → Developers → Webhooks
Edit webhook for platform.natca.org
Enable topic: conversation_part.tag.created
Save and test webhook

C. Member Not Found

Issue: Member number doesn't match Active/Retired member

Solution:

Verify member number is correct (ask member to confirm)

Check member status in database:

SELECT membernumber, status FROM member WHERE membernumber = '12345';

Only Active and Retired members are synced
If member is Suspended/Terminated, tag-based sync will skip

D. Tag Processing Error

Issue: System error during tag processing

Solution:

Check Platform logs for errors:

doctl apps logs <app-id> --component platform | grep -A 5 "Tag-based sync failed"

Common errors:
- Database connection timeout → Retry by removing and re-adding tag
- Intercom API error → Check Intercom status: https://status.intercom.com (opens in a new tab)
- Permission error → Verify INTERCOM_ACCESS_TOKEN has write permissions
If error persists, contact technical team

E. Tag Not Being Removed

Issue: Contact updated but tag still on conversation

Solution:

Check if tag removal API call succeeded:

doctl apps logs <app-id> --component platform | grep "Removed mynatca-sync tag"

If tag removal failed (404 error):
- Tag may have been manually removed during processing
- Refresh conversation in Intercom to verify
If tag persists after successful update:
- Manually remove tag
- Check webhook logs for removal API errors

F. Duplicate Contact Issues

Issue: 409 conflict when creating/updating contact

Solution:

System should automatically handle duplicates

Check logs for duplicate resolution:

doctl apps logs <app-id> --component platform | grep "duplicate"

If duplicates not resolved:
- Search Intercom for member email
- Note duplicate contact IDs
- Contact technical team for manual cleanup

Verification:

# Test tag-based sync workflow
# 1. Create test conversation in Intercom
# 2. Add member_number_verification attribute with test member number
# 3. Add mynatca-sync tag
# 4. Monitor logs
doctl apps logs <app-id> --component platform --follow
 
# Should see:
# "Processing tag-based sync for conversation"
# "Updated Intercom contact from tag-based sync"
# "Removed mynatca-sync tag from conversation"
 
# 5. Verify in Intercom:
# - Tag is removed
# - Contact has member data
# - Contact type is USER

7. Positions Not Syncing

Symptoms:

Contact has member_number but positions field empty
Positions field shows old/stale data
Specific members missing positions

Diagnosis:

# Check Supabase positions table
# Supabase → Table Editor → positions
# Filter: membernumber = 12345
 
# Check sync logs
doctl apps logs <app-id> --component sync-intercom | grep "positions"
 
# Test getMemberPositions directly
node -e "
const { getMemberPositions } = require('./lib/mysqlHelpers');
const supabase = require('./lib/supabase');
getMemberPositions(supabase, '12345').then(pos => console.log(pos));
"

Common Causes & Solutions:

A. Positions Not Synced to Supabase

Solution:

Ensure positions sync runs before Intercom sync

Check positions sync schedule:

# .do/app.yaml
- name: sync-positions
  schedule: "15 2 * * *"  # 2:15 AM (before Intercom at 2:30 AM)

Manually trigger positions sync:
```
npm run sync:positions
```

B. Supabase Connection Failure

Error: Failed to get positions: fetch failed

Solution:

Retry logic should handle this (3 retries)
Check Supabase status: https://status.supabase.com (opens in a new tab)

Verify SUPABASE_URL and SUPABASE_KEY:

curl -H "apikey: $SUPABASE_KEY" "$SUPABASE_URL/rest/v1/positions?membernumber=eq.12345"

C. Position Code Mapping Missing

Issue: Positions show as codes (e.g., "facrep") instead of names

Solution:

Add missing position code to mapping:

// In lib/mysqlHelpers.js
const POSITION_CODE_TO_NAME = {
  'pres': 'NATCA President',
  'evp': 'Executive Vice President',
  // ... existing codes
  'newcode': 'New Position Name'  // Add new mapping
};

D. Empty Positions Array

Issue: Member has positions in MySQL but not in Supabase

Solution:

Check positions sync logs:

doctl apps logs <app-id> --component sync-positions | grep "12345"

Re-sync positions for member:
```
npm run sync:positions
```

Verification:

# Test sync for specific member
node -e "
const sync = require('./sync/intercom/daily-sync');
const instance = new sync({ limit: 1 });
// Test with member number 12345
"
 
# Check Intercom contact
# Should have positions: "Facility President, Area Representative"

7. UUID External IDs Not Cleaned Up

Symptoms:

Contacts have UUID in external_id field
Old WordPress UUIDs not replaced with member numbers
Audit script skips UUID cleanup

Diagnosis:

# Search Intercom for contacts with UUID external_id
# Intercom → Contacts → Filter → external_id contains "-"
 
# Check audit log for UUID cleanup
cat /tmp/intercom_audit.log | grep "UUID external_id"
 
# Test UUID detection
node -e "
const { isUUID } = require('./sync/intercom/audit');
console.log(isUUID('a1b2c3d4-e5f6-7890-abcd-ef1234567890')); // true
console.log(isUUID('12345')); // false
"

Common Causes & Solutions:

A. Audit Script Not Run

Solution:

# Run audit script to clean up UUIDs
node sync/intercom/audit.js
 
# Or preview first
node sync/intercom/audit.js --dry-run --limit=100

B. UUID Validation Failed

Issue: UUID detected but member match validation failed

Solution:

Review validation logs:

cat /tmp/intercom_audit.log | grep "validation failed"

Adjust validation logic if needed (require only 1 match for UUID):

// In audit.js validateMemberMatch()
if (isExternalIdValidation && nameMatch && matchCount >= 1) {
  return { isValid: true, matchCount, matches: matchDetails };
}

C. Contact Has No Email/Name

Issue: UUID contact has only external_id, no email or name

Solution:

Already handled in validation:

if (isExternalIdValidation && !contact.email && !contact.name) {
  matchDetails.validation_note = 'Accepted: external_id only (authenticated source)';
  return { isValid: true, matchCount: 0, matches: matchDetails };
}

If still failing, check member lookup:

SELECT * FROM member WHERE membernumber = 'uuid-value'; -- Should return 0 rows

D. Manual Cleanup Needed

Solution:

# Create manual mappings file
echo '[
  { "contact_id": "67eb0a6e8f72a828ba21c342", "member_number": "12345" },
  { "contact_id": "66a13ba93686d4c0e4fdb04e", "member_number": "67890" }
]' > uuid-mappings.json
 
# Run manual update script
node sync/intercom/manual-update.js uuid-mappings.json

Verification:

# Search Intercom for UUIDs
# Should return 0 results
 
# Check specific contact
curl -H "Authorization: Bearer $INTERCOM_ACCESS_TOKEN" \
  https://api.intercom.io/contacts/67eb0a6e8f72a828ba21c342
 
# external_id should be member number, not UUID

Runbook

Daily Sync Failure Recovery

Scenario: Daily sync failed, members not updated

Steps:

Check Failure Reason:

doctl apps logs <app-id> --component sync-intercom | tail -100

Identify Issue (see Common Issues above)
Fix Issue (environment variables, rate limits, etc.)

Re-run Sync Manually:

# Test with limited members first
npm run sync:intercom -- --limit=100
 
# If successful, run full sync
npm run sync:intercom

Verify Success:

# Check summary logs
# ✅ Created: X, 🔄 Updated: Y, ❌ Failed: 0

Monitor Next Scheduled Run:

# Wait for 2:30 AM UTC cron run
# Check logs next morning

Duplicate Contact Cleanup

Scenario: Multiple contacts with same email need cleanup

Steps:

Identify Duplicates:

# Search Intercom for email
# Note contact IDs and created_at timestamps

Run Audit Script (handles duplicates automatically):
```
node sync/intercom/audit.js --dry-run
```

Review Planned Actions:

cat /tmp/intercom_audit.log | grep "DUPLICATE"
# Should show which contacts will be archived

Execute Cleanup:
```
node sync/intercom/audit.js
```

Verify Cleanup:

# Search Intercom again
# Should show only 1 contact for email

Tag-Based Re-Sync Troubleshooting

Scenario: Support team added tag but contact not enriched

Steps:

Verify Prerequisites:
- Conversation has "Member Number Verification" attribute filled
- Member number is valid Active/Retired member
- "mynatca-sync" tag exists in Intercom

Check Webhook Configuration:

# Verify webhook includes conversation_part.tag.created topic
# Intercom → Settings → Developers → Webhooks

Monitor Tag Processing:

# Watch for tag webhook events
doctl apps logs <app-id> --component platform --follow | grep "tag-based sync"

Check for Errors:

# View tag processing errors
doctl apps logs <app-id> --component platform | grep -A 10 "Tag-based sync failed"

Common Fixes:
- Missing attribute: Add "Member Number Verification" attribute first
- Invalid member: Verify member number and status
- Webhook not enabled: Enable conversation_part.tag.created topic
- Database error: Retry by removing and re-adding tag
Verify Success:
- Tag should be automatically removed
- Contact should have member_number, region, facility, positions
- Contact type should be USER (not LEAD)

Manual Intervention (if needed):

# If tag processing fails repeatedly:
# 1. Remove tag manually
# 2. Verify member number is correct
# 3. Check Platform logs for specific error
# 4. Contact technical team with conversation ID and member number

Webhook Troubleshooting

Scenario: Webhooks not enriching contacts

Steps:

Verify Webhook Active:
- Intercom → Settings → Developers → Webhooks
- Check "Active" status

Test Webhook Endpoint:

curl -X POST https://platform.natca.org/api/intercom/webhook \
  -H "Content-Type: application/json" \
  -d '{"topic":"test","data":{"item":{"id":"123"}}}'
 
# Should return: {"received":true}

Check Platform Logs:

doctl apps logs <app-id> --component platform --follow

Trigger Test Webhook:
- Intercom → Webhooks → Send test webhook
- Check Platform logs for "Intercom webhook received"
Test Real Conversation:
- Message Intercom support from test email
- Check if contact is enriched with member data

Debug Member Lookup:

# Test email lookup endpoint
curl -X POST https://platform.natca.org/api/intercom/lookup-email \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com"}'
 
# Should return member data if found

Emergency: Stop Sync

Scenario: Sync is causing issues, need to stop immediately

Steps:

Disable Cron Job:

# Comment out in .do/app.yaml
# jobs:
# - name: sync-intercom
#   ...
 
# Or change schedule to never run
schedule: "0 0 31 2 *"  # Feb 31st (never)

Deploy Change:

git add .do/app.yaml
git commit -m "Emergency: Disable Intercom sync"
git push origin main

Kill Running Job (if needed):

# In DigitalOcean dashboard
# Apps → Jobs → sync-intercom → Cancel

Investigate Issue:
- Review logs
- Identify root cause
- Fix issue

Re-enable Sync:

# Restore in .do/app.yaml
jobs:
- name: sync-intercom
  schedule: "30 2 * * *"

Diagnostic Commands

Check Sync Health

# View recent sync summary
doctl apps logs <app-id> --component sync-intercom | grep "Intercom Daily Sync Summary" | tail -1
 
# Count failures in last sync
doctl apps logs <app-id> --component sync-intercom | grep "Failed:" | tail -1
 
# View error details
doctl apps logs <app-id> --component sync-intercom | grep "Failed to sync" | tail -10

Check Webhook Health

# View recent webhook events
doctl apps logs <app-id> --component platform | grep "Intercom webhook received" | tail -10
 
# View webhook processing errors
doctl apps logs <app-id> --component platform | grep "Webhook processing failed" | tail -10
 
# Count webhook events per hour
doctl apps logs <app-id> --component platform | grep "Intercom webhook" | awk '{print $1}' | cut -d: -f1 | uniq -c

Check Data Quality

# Test phone formatting
node -e "
const { formatPhoneForIntercom } = require('./lib/phoneFormatter');
console.log(formatPhoneForIntercom('(765) 465-7031')); // Should output: +17654657031
console.log(formatPhoneForIntercom('invalid')); // Should output: null
"
 
# Test member type mapping
node -e "
const { getMemberTypeForIntercom } = require('./lib/memberTypeHelper');
console.log(getMemberTypeForIntercom('Active', 6)); // Should output: Current Member
console.log(getMemberTypeForIntercom('Retired', 6)); // Should output: Retired Member
"
 
# Test position lookup
node -e "
const { getMemberPositions } = require('./lib/mysqlHelpers');
const supabase = require('./lib/supabase');
getMemberPositions(supabase, '12345').then(pos => console.log(pos));
"

Monitoring Best Practices

Daily Checks

Check Sync Status (after 2:30 AM UTC):

doctl apps logs <app-id> --component sync-intercom | grep "Intercom Daily Sync Summary" | tail -1

Review Failure Rate:
- Target: < 0.1% (< 20 failures for 20,000 members)
- Alert if > 100 failures
Check Duplicate Resolution:
- Target: Decreasing over time
- Alert if > 50 duplicates resolved daily (after initial cleanup)

Weekly Checks

Review Audit Log:

cat /tmp/intercom_audit.log | jq 'select(.level == "error")' | wc -l

Verify Intercom Contact Count:
- Should match Active + Retired member count
- Check for orphaned contacts
Test Webhook:
- Send test message to Intercom
- Verify contact is enriched

Monthly Checks

Run Full Audit:
```
node sync/intercom/audit.js --force
```
Review Data Quality:
- Check for missing positions
- Verify region/facility codes
- Check phone number formats
Token Rotation:
- Rotate INTERCOM_ACCESS_TOKEN every 90 days
- Update environment variable

Getting Help

Logs to Collect

When reporting an issue, include:

Sync Logs:

doctl apps logs <app-id> --component sync-intercom --since 1h > sync-logs.txt

Webhook Logs:

doctl apps logs <app-id> --component platform --since 1h | grep intercom > webhook-logs.txt

Audit Log:

cat /tmp/intercom_audit.log > audit-log.txt

Environment Info:

echo "Node version: $(node --version)"
echo "App ID: <app-id>"
echo "Component: sync-intercom"

Contact Information

Platform Team: platform@natca.org
IT Support: it@natca.org
Intercom Support: https://www.intercom.com/help (opens in a new tab)

Troubleshooting Guide

Quick Diagnostics

Health Check Checklist

Common Issues

1. Daily Sync Not Running

A. Cron Job Disabled

B. Environment Variables Missing

C. Deployment Failed

D. Wrong Schedule

2. High Failure Rate

A. Blocked Contacts (FIXED - Now Automatic)

B. Invalid Phone Numbers

C. Duplicate Email Conflicts

D. Intercom API Errors

E. Rate Limit Exceeded

3. Duplicates Not Resolving

A. Archive Permission Issue

B. Eventual Consistency Delay

C. Contacts Recreated Between Sync Runs

4. Webhook Not Enriching Contacts

A. Webhook Not Configured in Intercom

B. Webhook URL Incorrect

C. Platform Server Not Receiving Requests

D. Member Not Found by Email

5. Only 1,000 Members Being Synced (FIXED)

6. Tag-Based Re-Sync Not Working

A. Conversation Attribute Not Set

B. Webhook Topic Not Enabled

C. Member Not Found

D. Tag Processing Error

E. Tag Not Being Removed

F. Duplicate Contact Issues

7. Positions Not Syncing

A. Positions Not Synced to Supabase

B. Supabase Connection Failure

C. Position Code Mapping Missing

D. Empty Positions Array

7. UUID External IDs Not Cleaned Up

A. Audit Script Not Run

B. UUID Validation Failed

C. Contact Has No Email/Name

D. Manual Cleanup Needed

Runbook

Daily Sync Failure Recovery

Duplicate Contact Cleanup

Tag-Based Re-Sync Troubleshooting

Webhook Troubleshooting

Emergency: Stop Sync

Diagnostic Commands

Check Sync Health

Check Webhook Health

Check Data Quality

Monitoring Best Practices

Daily Checks

Weekly Checks

Monthly Checks

Getting Help

Logs to Collect

Contact Information

Related Documentation