Email / SMTP Troubleshooting

Email Debugging Guide for Layer5 Cloud 🔗

This guide explains how to diagnose email sending issues in Layer5 Cloud deployments using the enhanced debug logging and testing features.

Overview 🔗

Email issues in Layer5 Cloud can occur due to various reasons including SMTP configuration problems, template errors, recipient validation issues, or network connectivity problems. This guide provides comprehensive debugging tools and techniques.

Debug Log Levels 🔗

To enable email debugging, set the LOG_LEVEL environment variable to 5 (Debug) or 6 (Trace):

# In config.env or environment variables
LOG_LEVEL=5

Testing Email Configuration 🔗

1. Email Configuration Test Endpoint (Provider Admin Only) 🔗

Check that the four SMTP_* values are configured, without sending an email. Both verbs of this endpoint require authentication and the provider admin role, so the GET must carry a credential. It validates configuration only - it does not dial the SMTP server.

curl -X GET "https://your-domain.com/api/system/email/test" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

Expected Response (Success):

{
  "status": "success",
  "message": "Email configuration is valid",
  "timestamp": "1695312000"
}

The response carries the verdict only. It does not report smtp_host, smtp_port or smtp_username: those are the deployment’s own relay settings, and SMTP_USERNAME is an email address. Read the configured values from your deployment configuration instead.

Expected Response (Error): 500 Internal Server Error, as plain text rather than JSON. The body is the error’s short description alone - the full error, which names the relay, stays in the log:

SMTP configuration error for field 'SMTP_HOST'

Expected Response (Unauthenticated): 401 Unauthorized

{
  "message": "user must be logged in to perform this operation"
}

Expected Response (Authenticated, not a provider admin): 403 Forbidden

{
  "message": "user [email protected] must be Provider Admin to perform this operation"
}

2. Authenticated Email Send Test (Provider Admin Only) 🔗

Send an actual test email to verify end-to-end email functionality. This endpoint requires authentication and provider admin role:

curl -X POST "https://cloud.layer5.io/api/system/email/test" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{
    "to": "[email protected]",
    "subject": "Layer5 Cloud Email Test"
  }'

Request Body:

{
  "to": "[email protected]",
  "subject": "Layer5 Cloud Email Test (optional)"
}

Expected Response (Success):

{
  "status": "success",
  "message": "Test email sent successfully",
  "timestamp": "1695312000",
  "sentTo": "[email protected]"
}

Expected Response (Unauthenticated / not a provider admin): the same 401 and 403 JSON bodies as the GET above - both verbs carry the same provider-admin gate.

Expected Response (Error - Invalid Email): 400 Bad Request, as plain text:

Invalid email address format

Expected Response (Error - Email Configuration): 500 Internal Server Error, as plain text. The configuration is checked before anything is sent, so this is the same verdict the GET gives:

SMTP configuration error for field 'SMTP_HOST'

Expected Response (Error - the send itself failed): 500 Internal Server Error, as plain text, carrying the short description only. The relay’s own reply, the resolved endpoint and the SMTP username stay in the log:

Failed to send email with subject 'Layer5 Cloud Email Test' to recipients: [email protected]

3. Required Environment Variables 🔗

Either set all four values or leave all four empty. A deployment that leaves them empty is a supported state - it simply sends no mail - but a deployment that sets some of them and not others does not start.

# Required SMTP Configuration
SMTP_HOST=smtp.example.com     # must resolve to a publicly routable address
SMTP_PORT=587                  # one of 25, 465, 587, 2525
SMTP_USERNAME[email protected]   # a full address: it becomes the From: header
SMTP_PASSWORD=your-app-password

SMTP_PORT also decides how the session is encrypted, because the four keys carry no mode of their own: 465 means implicit TLS, and every other permitted port takes STARTTLS. Read Constraints on the shared mail server before you change either value.

Constraints on the Shared Mail Server 🔗

Since v1.0.253, mail is routed per Organization: an Organization that has brought its own mail server sends through that server, and everyone else sends through the shared mail server your deployment configures. Both now go through one send primitive, so the shared mail server is screened exactly as a tenant’s is.

That screening is new. The previous send path applied none of it, so a deployment that has been sending mail happily can be refused after an upgrade. Layer5’s own hosted environments use smtp.gmail.com:587 and are unaffected; a self-hosted install that points SMTP_HOST somewhere else can be.

What is refused 🔗

Every address SMTP_HOST resolves to is screened, not just the one a connection happens to use. A host publishing both a public and a private record is refused.

Port 465 is the exception to STARTTLS: it speaks TLS from the first byte, and the deployment selects that mode automatically from the port. You do not configure the encryption mode separately.

What happens at startup 🔗

The server validates the shared mail server once, at startup, rather than waiting for somebody’s password reset to fail silently. Only the first row below stops the deployment:

The two non-fatal failures are still failures: no mail leaves over the shared mail server until they clear. Treat 3302 and 3304 at startup as a page, not a note. They are separate codes because the remedy is different - 3302 is the network, 3304 is the relay’s certificate.

A refusal is fatal on purpose. The alternative is a deployment that comes up green and drops every message silently, which is the failure this check exists to convert into a refusal to start.

The startup check does not authenticate and sends no message. A rotated password is not a reason a deployment will not start, and an authentication failure is not one of the verdicts this check acts on.

Debug Log Examples 🔗

When LOG_LEVEL=5, you’ll see detailed debug logs for email operations.

1. The Shared Mail Server, at Startup 🔗

This is where the relay is described now, once per process:

INFO The shared mail relay answered on smtp.example.com:587 over starttls

When none of the four values is set, the deployment starts and says so:

WARN The shared mail relay is not configured (SMTP_HOST is unset). No transactional or identity mail can be sent over it; organizations with their own mail server are unaffected

A refusal names the constraint that refused it and the process exits; see What happens at startup.

2. Template Processing 🔗

DEBUG Email template parsing template_paths=[email-templates/meshery-cloud/email.body.gotmpl, ...] template_count=5 base_template_path=email-templates/meshery-cloud/email.body.gotmpl
DEBUG Email template parsed template_name=email.body.gotmpl
DEBUG Executing email template email_type=welcome [email protected] has_org_vars=true org_name=MyOrg
DEBUG Email template executed body_length=2048 email_type=welcome

3. Sending 🔗

DEBUG Email sending attempt initiated subject="Welcome to Layer5 Cloud" recipient_count=1 cc_count=0
INFO Email sent [email protected] subject="Welcome to Layer5 Cloud"

4. Error Scenarios 🔗

A failed send is logged as its structured error, carrying a code:

ERROR meshery_cloud-1145: SMTP send mail error

For a message that belonged to an Organization with its own mail server, the failure is recorded against that Organization as well - see Which mail server a message left through.

Flow Emails (Kratos Integration) 🔗

Flow emails are rendered by the same process and logged through the same structured logger. They used to be written to standard output with a [DEBUG] prefix, echoing the relay host and the SMTP username past the logger at a level nothing could turn down; those lines no longer exist.

Only two Kratos template types are dispatched: verification_code_valid and recovery_code_valid. Every environment runs the code strategy for recovery and verification, so those are the only two Kratos emits - and the unknown-recipient notices (recovery_code_invalid, verification_code_invalid) are deliberately not sent. Any other type is acknowledged with a 200 and logged as unsupported rather than mailed; a 200 for one of those means the request was accepted and no message was sent.

DEBUG Flow email attempt.  subject_template:  email-templates/valid/email-recover-subject.body.gotmpl , body_template:  email-templates/valid/email-recover.body.gotmpl , organization_id:  0e6b7ba0-... , recipient:  [email protected]
DEBUG Flow email delivery failed.  error_type:  *errors.Error , template_type:  recovery_code_valid , organization_id:  0e6b7ba0-... , recipient:  [email protected]
INFO Flow email sent.  template_type:  recovery_code_valid , recipient:  [email protected]

Which Mail Server a Message Left Through 🔗

Every message this deployment sends is routed by the reader’s Organization. An Organization that has configured its own mail server, verified its from domain and turned it on sends through that server; every other message takes the shared mail server configured by SMTP_HOST.

That routing produces its own log lines, and a self-hosted operator will meet them without having configured anything themselves:

• The Organization’s mail server was used and the message was accepted. No operator action; the verdict is recorded against that Organization’s delivery health, which its administrators read on the Email tab.

• The Organization’s mail server refused the message and fallback is on. The message is re-sent over the shared mail server, arriving from this deployment’s address with the Organization’s name and a (via ...) suffix. Logged as meshery_cloud-3299.

• The Organization’s mail server refused the message and fallback is off. The message is dropped, logged as meshery_cloud-3300 at Critical. This is that Organization’s own setting working as documented, not a fault in this deployment - but a dropped verification or recovery message locks a user out of their account, so it is worth alerting on.

• The Organization’s mail server took the whole message and never answered. Delivery is unknown, so the message is deliberately not re-sent over the shared mail server - re-sending it could deliver it twice. Logged as meshery_cloud-3303 at Critical.

• The Organization’s stored configuration could not be read - a database read failed, or a stored password would not decrypt, usually after CLOUD_CREDENTIAL_ENCRYPTION_KEY was rotated. The message takes the shared mail server whatever that Organization’s fallback setting says, because the fault is this deployment’s rather than a delivery policy the tenant chose. Logged as meshery_cloud-3298.

The full behavior an Organization administrator sees is documented in Bring Your Own Mail Server.

Common Issues and Solutions 🔗

1. SMTP Configuration Errors 🔗

Issue: SMTP configuration error: SMTP_HOST is empty

Solution:

• Verify all SMTP environment variables are set
• Check that environment variables are properly loaded in your deployment
• Use the test endpoint to validate configuration

2. Authentication Failures 🔗

Issue: SMTP authentication was refused by the mail server

Solution:

• Verify SMTP username and password are correct
• For Gmail, use App Passwords instead of regular passwords
• Check if 2FA is enabled and properly configured

The identity that was refused and the endpoint it was presented to are in the log’s long description (authenticating as <username> at <host:port>), never in the response - an SMTP username is an email address.

3. Template Errors 🔗

Issue: Email template missing or inaccessible

Solution:

• Verify email template files exist in config/email-templates/
• Check file permissions
• Validate template syntax and required variables

4. Recipient Validation Errors 🔗

Issue: Email recipient validation failed

Solution:

• Verify email addresses are valid and properly formatted
• Check for empty recipient lists
• Validate email addresses contain @ symbol
• Check for a carriage return or line feed in an address. An address carrying one is refused, not sanitized: recipient addresses are written as SMTP protocol verbs and into the header block, so a line break in one would start a header of its own. This applies to Cc addresses as well as To

Subjects are the deliberate asymmetry: a line break in a subject is stripped rather than refused, because a subject is display text and dropping a notification over a design name containing a newline would turn a cosmetic problem into a lost message.

5. Network Connectivity Issues 🔗

Issue: meshery_cloud-3302 at startup, or dial tcp: lookup smtp.example.com: no such host

Solution:

• Check network connectivity to SMTP server
• Verify egress from this cluster to the relay’s submission port is allowed
• Test DNS resolution for SMTP host

No mail leaves over the shared mail server while this is outstanding, even though the deployment started.

6. The Server Will Not Start After an Upgrade 🔗

Issue: meshery_cloud-3301 at Error, then exit code 1.

Solution: the shared mail server fails one of the constraints. The logged cause names which screen refused it:

7. The Server’s Certificate Is Not Trusted 🔗

Issue: meshery_cloud-3304 at Error; the deployment starts.

Solution: the relay answered, and its certificate did not verify against the trusted roots - it has expired, was issued for a different host name, or is self-signed or from an issuer this server does not trust. Fix the certificate; this is not a network fault and not something egress rules will clear. Every send fails identically until it verifies.

Development Mode 🔗

In development environment (ENVIRONMENT=development), the rendered email content is logged in addition to being sent, so the body can be read without opening the recipient’s mailbox:

INFO Development mode - Email details [email protected] subject="Test Email" body="<html>...</html>"

The message still goes out over the shared mail server, which still has to satisfy the constraints above. Pointing a development deployment at a local catch-all mailbox on port 1025 does not work.

Error Codes Reference 🔗

Mail routing and the shared mail server 🔗

These codes arrived with per-Organization mail routing in v1.0.253. The four marked send path describe a message that belonged to an Organization with its own mail server; the rest describe the shared mail server this deployment configures.

Monitoring and Alerting 🔗

Consider setting up monitoring for email-related metrics:

1. Email Send Success Rate: Monitor successful vs failed email sends
2. SMTP Response Times: Track SMTP server response times
3. Template Processing Time: Monitor email template rendering performance
4. Configuration Validation: Regular health checks for email configuration

Best Practices 🔗

1. Use Debug Logs Sparingly: Only enable debug logging when troubleshooting
2. Secure Credentials: Never log SMTP passwords in plaintext
3. Regular Testing: Use the test endpoint to validate configuration regularly
4. Monitor Quotas: Keep track of email service provider quotas and limits
5. Template Validation: Test email templates thoroughly before deployment

Troubleshooting Checklist 🔗

• Check LOG_LEVEL is set to 5 or 6 for debug logging
• Verify all SMTP environment variables are configured
• Confirm SMTP_PORT is one of 25, 465, 587, 2525, and that SMTP_USERNAME is a full address
• Confirm SMTP_HOST resolves only to publicly routable addresses, and that the relay offers STARTTLS (or is on 465)
• Read the startup log for meshery_cloud-3301, -3302 or -3304 before looking anywhere else
• Test email configuration using the provider-admin-only /api/system/email/test endpoint, authenticating the request
• Check network connectivity to SMTP server
• Validate email template files exist and are accessible
• Verify recipient email addresses are valid
• Check SMTP server logs for additional error details
• Monitor email service provider quotas and limits

Related Reading

• Customizing Layer5 Cloud Deployment with Helm
• Customizing Webhooks
• Deploying Layer5 Cloud
• Identity Services
• Monitoring
• Networking