Skip to main content

Troubleshooting

A consolidated reference for diagnosing and resolving common issues with Cirata Symphony.

Service Issues​

Symphony Won't Start​

Check the logs:

# Linux
sudo journalctl -u symphony --since "10 min ago"

# Docker
docker compose logs symphony --tail 50

# Kubernetes
kubectl logs deployment/symphony --tail=50

Verify configuration is valid JSON:

python3 -m json.tool /var/lib/symphony/symphony.config

Check required ports are available:

PortServiceProtocol
8080UI and REST APIHTTP
4222NATS clientTCP
9222NATS WebSocketWS
ss -tlnp | grep -E '8080|4222|9222'

Verify identity keys are intact:

The symphony.jwt, symphony.seed, symphony.signingseed, operator.jwt, operator.seed, and operator.signingseed fields in symphony.config must be present and valid. If corrupted, restore from backup.

Symphony Starts But UI Shows Error​

  • Clear browser cache and cookies
  • Check that the OIDC provider is reachable from both the browser and the Symphony server
  • Verify oidc.issuer, oidc.clientid, and oidc.uiclientid in symphony.config
  • Check the browser console for JavaScript errors

Authentication Issues​

OIDC Login Fails​

  • Verify the OIDC issuer URL is correct and reachable: curl https://<issuer>/.well-known/openid-configuration
  • Confirm the OIDC client IDs match those configured in your identity provider
  • Check that the redirect URI in the identity provider includes your Symphony URL
  • Review Symphony logs for OIDC validation errors

API Token Returns 401 Unauthorized​

  • Verify the token is being sent correctly: Authorization: Bearer <token> (not Authentication)
  • Check if the API key has been revoked—list keys at Account → API Keys
  • Tokens include the server location—ensure the token was created for this Symphony instance
  • Check that the storage.salt in symphony.config has not changed (would invalidate all tokens)

Cannot Access Admin Interface​

  • Verify admin_subject or admin_group is configured in symphony.config
  • Confirm your user subject matches admin_subject exactly
  • Confirm your OIDC groups include the admin_group value
  • Roles are cached at login—sign out and back in after configuration changes
  • Check logs for role resolution errors

Extension Issues​

Extension Won't Connect​

  1. Verify the extension's API token is valid: 
    curl -H "Authorization: Bearer <token>" https://your-symphony-instance.com/api/v1/apikey
    Replace <token> with the extension's token. A successful response confirms the token is recognised and returns the associated API key details.
  2. Check network connectivity to Symphony on port 9222 (WebSocket) or 4222 (NATS)
  3. Review extension logs for connection errors
  4. Verify the SYMPHONY_TOKEN environment variable is set correctly

Extension Connected But Not Visible​

  • Check that the extension has registered menu items (not all extensions add menus)
  • Verify the extension's API key has subscribe permissions for cirata.extensions.<prefix>.>
  • If RBAC is enabled, verify your role's Subscribe Allow includes cirata.extensions.<prefix>.> for the extension—extensions are only shown to users whose subscribe permissions cover the extension's subject namespace
  • If RBAC is enabled, also check that your role includes the extension in its visible_extensions list
  • Refresh the browser—extension registration is immediate but the UI may need a reload

Extension Cannot Publish to Subject​

  • Check the extension's API key capabilities include the subject in pub.allow
  • Verify infrastructure subjects like _INBOX.> are included
  • Check logs for NATS permission denied errors: 
    sudo journalctl -u symphony | grep "permission"

Extension Service Calls Timeout​

  • Ensure the extension handler always calls respond() or error()—handlers that don't respond cause timeouts
  • Check that the extension is still connected (it may have crashed)
  • Verify the subject name matches exactly (subjects are case-sensitive)

RBAC Issues​

API Key Creation Fails with "Exceeds Role Permissions"​

  • View your effective permissions: curl -H "Authorization: Bearer <token>" https://your-symphony-instance.com/api/v1/roles (no CLI equivalent—use the UI at Account → Roles or curl)
  • Ensure requested capabilities are subsets of your role's allowed subjects
  • Contact your administrator to expand your role

User Has Wrong Roles​

  • Check role assignments for the user's LDAP/AD groups at Administration → Assignments
  • Verify the OIDC groups claim contains the expected group names
  • Roles are cached at login—the user must sign out and back in after assignment changes
  • Check logs for the role resolution process

Bootstrap Mode Warning Persists​

  • Configure admin_subject or admin_group in symphony.config to exit bootstrap mode
  • Or set "bootstrap": true in the rbac configuration to acknowledge intentional bootstrap mode

Performance Issues​

Slow API Responses​

  1. Check server resource utilization (CPU, memory, disk I/O)
  2. Query metrics via the Observability Extension (cirata observability query_scalar_metrics) for request latency distribution
  3. Check JetStream storage usage—approaching limits causes slowdowns
  4. Consider increasing jetstream.max_mem in nats.config

High Memory Usage​

  • Check JetStream memory usage against configured jetstream.max_mem in nats.config
  • Review extension processes for memory leaks
  • Consider reducing jetstream.max_mem if set higher than available system memory

WebSocket Disconnections​

  • Check that your reverse proxy has adequate timeouts (>60s for WebSocket connections)
  • For nginx, set proxy_read_timeout 86400; on the WebSocket location
  • Check for network interruptions between the client and Symphony

Storage Issues​

JetStream Storage Full​

  • Check current usage: nats_jetstream_storage_bytes metric
  • Increase jetstream.max_file in nats.config and restart
  • Identify large buckets and consider cleaning up old data

Data Appears Inconsistent​

  • In clustered deployments, check NATS cluster health and synchronization
  • Verify all Symphony instances are running the same version
  • Check extension behavior for data corruption issues
  • Consider restoring from a recent backup if corruption is confirmed

Getting Help​

If you cannot resolve an issue using this guide:

  1. Collect relevant logs (sudo journalctl -u symphony --since "1 hour ago" > symphony-logs.txt)
  2. Note the Symphony version (cirata info or curl https://your-symphony-instance.com/api/v1/symphonyinfo)
  3. Document the steps to reproduce the issue
  4. Contact support with the collected information

See Also​