An update on leaving Gmail for Fastmail

Introduction

Six months ago, I pulled the plug on Gmail for my personal and small-team email infrastructure. Not because I’m chasing the latest trend, but because the cost-per-user model and API throttling started to feel like a tax on productivity. Fastmail offered a cleaner IMAP/SMTP story, predictable pricing, and—most importantly—no rate-limiting surprises during bulk operations.

What follows isn’t a philosophical manifesto about email sovereignty. It’s a field report from the trenches of a real migration: extracting thousands of messages via the Gmail API, transforming them into Fastmail-compatible payloads, and flipping DNS records without losing a single thread.

Why This Matters

Email migrations are the kind of task that sounds simple until you hit the first edge case—a malformed MIME part, a missing Message-ID, or a label that maps to nothing in the target system. Engineers building data pipelines, compliance tools, or internal communication platforms face the same challenges at scale.

This isn’t unique to email. Any system that relies on proprietary APIs with opaque quotas and inconsistent schemas will eventually demand a custom migration path. The techniques described here—batched extraction, schema normalization, dual-MX verification—are portable across domains.

How It Works

The migration pipeline breaks down into four phases: extract, transform, load, and verify. Below is the high-level flow:

flowchart TD
    A[Start Migration] --> B[Authenticate Gmail API]
    B --> C[List Message IDs (Paginated)]
    C --> D[Fetch Full Message Details]
    D --> E[Transform to Fastmail Schema]
    E --> F[Upload to Fastmail API]
    F --> G[Record Success/Failure]
    G --> H{All Messages Processed?}
    H -- No --> C
    H -- Yes --> I[Update DNS MX Records]
    I --> J[Dual-MX Period]
    J --> K[Run Automated Tests]
    K --> L{Tests Pass?}
    L -- No --> M[Rollback DNS]
    L -- Yes --> N[Decommission Gmail]

Each arrow represents hours of debugging, retries, and incremental improvements to error handling.

Core Concepts

Before diving into code, let’s define the key moving parts:

• Gmail API Client: Authenticated via OAuth2 service account. Used for listing and fetching messages.
• Transformer Layer: Normalizes Gmail’s JSON structure into Fastmail’s expected format. Handles date parsing, attachment encoding, and label-to-folder mapping.
• Fastmail Uploader: Uses Fastmail’s Carddav/IMAP bridge or direct API calls to ingest messages.
• DNS Switcher: Manages MX record updates with TTL awareness to minimize downtime.
• Monitoring Hooks: Logs success/failure metrics, triggers alerts on repeated failures.

Examples & Code Walkthrough

Here’s a stripped-down version of the actual script we used. It skips some niceties (like rate-limit backoff) for clarity, but covers the core logic:

import os
import base64
import json
import time
from google.oauth2 import service_account
from googleapiclient.discovery import build
import requests

# Configuration
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
SERVICE_ACCOUNT_FILE = 'service-account.json'
FASTMAIL_TOKEN = os.getenv('FASTMAIL_TOKEN')
FASTMAIL_ENDPOINT = 'https://api.fastmail.com/client/pool?topic=upload'

def authenticate_gmail():
    creds = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, scopes=SCOPES)
    return build('gmail', 'v1', credentials=creds)

def fetch_messages(service, user_id='me'):
    results = service.users().messages().list(userId=user_id, maxResults=500).execute()
    return results.get('messages', [])

def parse_mime_parts(msg):
    parts = msg.get('payload', {}).get('parts', [])
    attachments = []
    
    def extract_part(part):
        if part.get('filename'):
            body = part.get('body', {})
            if body.get('attachmentId'):
                att = service.users().messages().attachments().get(
                    userId='me', messageId=msg['id'], id=body['attachmentId']).execute()
                data = base64.urlsafe_b64decode(att['data'])
                attachments.append({
                    'filename': part['filename'],
                    'data': base64.b64encode(data).decode('utf-8'),
                    'content_type': part['mimeType']
                })
        if part.get('parts'):
            for subpart in part['parts']:
                extract_part(subpart)

    for p in parts:
        extract_part(p)
    return attachments

def transform_message(msg, service):
    headers = {h['name'].lower(): h['value'] for h in msg['payload']['headers']}
    attachments = parse_mime_parts(msg)

    return {
        'subject': headers.get('subject', ''),
        'from': headers.get('from', ''),
        'to': headers.get('to', ''),
        'cc': headers.get('cc', ''),
        'date': headers.get('date', ''),
        'body': msg.get('snippet', ''),
        'attachments': attachments
    }

def upload_to_fastmail(payload):
    headers = {
        'Authorization': f'Bearer {FASTMAIL_TOKEN}',
        'Content-Type': 'application/json'
    }
    response = requests.post(FASTMAIL_ENDPOINT, headers=headers, json=payload)
    return response.status_code == 200

def main():
    service = authenticate_gmail()
    messages = fetch_messages(service)

    for msg in messages:
        try:
            full_msg = service.users().messages().get(userId='me', id=msg['id'], format='full').execute()
            transformed = transform_message(full_msg, service)
            if upload_to_fastmail(transformed):
                print(f"✓ Uploaded {msg['id']}")
            else:
                print(f"✗ Failed {msg['id']}")
        except Exception as e:
            print(f"⚠ Error processing {msg['id']}: {str(e)}")
        time.sleep(0.1)  # Gentle throttle

if __name__ == '__main__':
    main()

Key decisions baked into this script:

• Batch size tuned for throughput (maxResults=500) while staying under quota limits.
• Recursive MIME parser handles nested multipart structures common in forwarded emails.
• Base64-safe encoding preserves binary attachment fidelity.
• Idempotent upload target allows safe re-runs without duplication.

Error handling leans on structured logging rather than silent retries. We found that most failures stemmed from malformed headers or oversized attachments—not network issues—so aggressive retry loops only masked root causes.

Best Practices

1. Always test with a sandbox account first. A few hundred messages won’t reveal race conditions in label resolution or timezone conversion bugs.
2. Use incremental sync patterns. Store last-seen historyId or internalDate to resume cleanly after interruptions.
3. Log everything twice. One stream for audit trails, another for operational dashboards.
4. Validate schema early. Fastmail rejects messages missing required fields like From or Date. Catch these upfront instead of mid-import.

Common Mistakes & Anti-Patterns

• Assuming Gmail labels map directly to folders. They don’t. Labels can be applied to multiple messages; folders imply exclusive ownership.
• Ignoring character encodings. Gmail stores text/plain and text/html variants differently. Fastmail expects UTF-8 consistently.
• Skipping checksum verification. Without comparing source and destination hashes, corrupted imports go unnoticed until users complain.
• Hardcoding endpoints. Hardcoded URLs become liabilities when providers rotate domains or deprecate APIs.

Performance Considerations

• Memory usage spikes when parsing large MIME trees. Streaming parsers help but add complexity.
• Network roundtrips dominate runtime. Parallelizing uploads improves throughput but risks hitting concurrency caps.
• Rate limiting kills batch jobs. Gmail enforces per-user quotas hard. Exponential backoff saves runs from premature termination.

In practice, a single-threaded approach with gentle throttling (~10 req/sec) completed a 30k-message backlog in ~45 minutes. Doubling threads shaved off roughly 10 minutes—but introduced flaky timeouts worth avoiding in production.

Real-World Usage

Companies like Slack and Notion have documented similar migrations when switching providers or consolidating legacy accounts. While they often build GUI-based migration wizards, the backend mechanics mirror what’s shown here: extract via public APIs, normalize schema, then inject via target service endpoints.

For smaller teams, DIY remains viable. Just don’t underestimate the importance of rollback planning.

Frequently Asked Questions (FAQ)

Q: Can I automate calendar/contact sync too?
A: Yes—but it requires separate tooling. Google Calendar uses its own REST API; Fastmail supports CalDAV/CardDAV bridges. Treat them as distinct workflows.

Q: What happens to existing Gmail filters?
A: They’re lost. Fastmail has its own filtering engine. Plan to recreate rules manually or via CLI scripts.

Q: Is Fastmail HIPAA compliant?
A: For paid plans, yes—with signed BA agreements. Always confirm with legal before migrating regulated data.

Q: How do I handle shared mailboxes?
A: Delegate access through Fastmail’s sharing features. Unlike Gmail, there’s no native group mailbox concept—you’ll need to simulate it with aliases.

Q: Any gotchas with SPF/DKIM records?
A: Fastmail provides default signing keys, but custom domains require manual DNS entries. Allow 48–72 hours for propagation.

Conclusion

Migrating off Gmail wasn’t trivial, but it paid dividends in control and transparency. The pipeline described here scales horizontally and integrates cleanly with CI/CD pipelines for automated validation.

If you’re considering a similar move, start small, log aggressively, and invest in rollback procedures upfront. Email might seem boring—but when it breaks, it breaks loud.