Staging -> Production: changes since May 2025

.cursorindexingignore

@@ -0,0 +1,3 @@
+
+# Don't index SpecStory auto-save files, but allow explicit context inclusion via @ references
+.specstory/**

.docs/EMAIL_SERVICE.md

@@ -0,0 +1,103 @@
+# Email Service Configuration
+
+This document explains how to set up and use the email service with Microsoft Graph API and the Mail.Send permission.
+
+## Requirements
+
+To use the email service with Microsoft Graph API, you need:
+
+1. An Azure AD application registration with the following delegated permissions:
+   - `Mail.Send`
+   - `User.Read`
+
+2. The following environment variables:
+   - `MS_FROM_EMAIL`: The email address that will be used as the sender
+   - `EMAIL_SERVICE_TYPE`: The type of email service to use (default: `ms_graph`)
+
+## Configuration
+
+### Setting Up the Email Service
+
+The application uses a factory pattern to create the appropriate email service. By default, it uses the Microsoft Graph API with the `Mail.Send` permission.
+
+```python
+# The factory creates the appropriate email service based on the environment variables
+from src.services.email_factory import create_email_service
+
+# Create an email service instance
+email_service = create_email_service()
+```
+
+### Environment Variables
+
+Configure the following environment variables:
+
+```bash
+# Required for Microsoft Graph Email Service
+MS_FROM_EMAIL=your-sender-email@example.com
+EMAIL_SERVICE_TYPE=ms_graph  # Options: ms_graph, sendgrid
+```
+
+## Usage Examples
+
+### Sending a Simple Email
+
+```python
+from src.services.email_factory import create_email_service
+
+# Create an email service instance
+email_service = create_email_service()
+
+# Send an email
+await email_service.send_email(
+    to_emails=["recipient@example.com"],
+    subject="Test Subject",
+    content="This is the email content",
+    content_type="text/plain"  # or "text/html" for HTML content
+)
+```
+
+### Sending a Templated Notification Email
+
+```python
+from src.services.email_factory import create_email_service
+
+# Create an email service instance
+email_service = create_email_service()
+
+# Send a notification email using a template
+await email_service.send_notification_email(
+    to_email="recipient@example.com",
+    subject="Notification Subject",
+    template_id="welcome-template",
+    dynamic_data={
+        "name": "John Doe",
+        "organization": "UNDP",
+        "role": "Admin"
+    }
+)
+```
+
+## Testing the Email Service
+
+You can test the email service by running the provided test script:
+
+```bash
+# Make the script executable
+chmod +x test_mail_send.py
+
+# Run the test script
+./test_mail_send.py
+```
+
+The script will prompt you to enter a recipient email address and will send a test email to verify that the `Mail.Send` permission is working correctly.
+
+## Troubleshooting
+
+If you encounter issues with sending emails:
+
+1. Verify that the Azure AD application has the required permissions (Mail.Send and User.Read)
+2. Ensure that the permissions have been admin-consented
+3. Check that the MS_FROM_EMAIL environment variable is set correctly
+4. Check the application logs for detailed error messages
+5. Verify that the DefaultAzureCredential is properly configured 

.env.example

@@ -1,40 +1,17 @@
-# Authentication
-TENANT_ID="<microsoft-entra-tenant-id>"
-CLIENT_ID="<app-id>"
-API_KEY="<strong-password>" # for accessing "public" endpoints
-
-# Database and Storage
-DB_CONNECTION="postgresql://<user>:<password>@<host>:5432/<staging|production>"
-SAS_URL="https://<account-name>.blob.core.windows.net/<container-name>?<sas-token>"
-
-# Azure OpenAI, only required for `/signals/generation`
-AZURE_OPENAI_ENDPOINT="https://<subdomain>.openai.azure.com/"
-AZURE_OPENAI_API_KEY="<api-key>"
-
-# Testing, only required to run tests, must be a valid token of a regular user
-API_JWT="<json-token>"
 # Email Configuration
 MS_FROM_EMAIL=futureofdevelopment@undp.org
 EMAIL_SERVICE_TYPE=ms_graph
 
-# SendGrid Configuration (if using SendGrid email service)
-SENDGRID_API_KEY=
-SENDGRID_FROM_EMAIL=
-
-# Azure Authentication
+# Authentication
 TENANT_ID=
 CLIENT_ID=
-
-# API Authentication
 API_KEY=
 API_JWT=
 
-# Database Connection
+# Database and Storage
 DB_CONNECTION=
-
-# Azure Storage
 SAS_URL=
 
 # Azure OpenAI Configuration
 AZURE_OPENAI_ENDPOINT=
-AZURE_OPENAI_API_KEY= 
+AZURE_OPENAI_API_KEY=

.gitignore

@@ -149,3 +149,9 @@ Taskfile.yml
 /logs
 webapp_logs.zip
 /.schemas
+app_logs.zip
+/deployments
+/LogFiles
+/.exports
+schema.sql
+schema.dbml

docs/azure_graph_mail_send_setup.md

@@ -0,0 +1,55 @@
+# Enabling Automated Email Sending via Microsoft Graph
+
+To allow the Future of Development platform to send emails automatically (e.g., for digests, notifications) without manual authentication, you must configure Microsoft Graph **Application permissions** for your Azure AD app registration.
+
+## Why Application Permissions?
+- **Delegated permissions** require a user to be logged in interactively—this is not suitable for scheduled/automated jobs.
+- **Application permissions** allow your backend/server to send emails as a service account using only a client ID and secret.
+
+## Steps for Admin
+
+1. **Go to Azure Portal > Azure Active Directory > App registrations > [Your App]**
+2. **API permissions**:
+    - Click **Add a permission** > **Microsoft Graph** > **Application permissions**
+    - Search for and add **Mail.Send** (Application)
+3. **Grant admin consent**:
+    - Click **Grant admin consent for [Your Org]**
+4. **Verify**:
+    - You (or your admin) can run:
+      ```sh
+      az ad app permission list --id <client-id>
+      ```
+      - You should see a `"type": "Role"` for Mail.Send.
+
+## Template Email/Message to Admin
+
+```
+Subject: Request: Grant Application Mail.Send Permission to Azure App for Automated Email Sending
+
+Hi [Admin],
+
+We need to enable automated email sending from the "Future of Development" app (Client ID: 4b179bfc-6621-409a-a1ed-ad141c12eb11) using Microsoft Graph.
+
+**Please:**
+1. Go to Azure Portal > Azure Active Directory > App registrations > "Future of Development".
+2. Under **API permissions**, click **Add a permission** > **Microsoft Graph** > **Application permissions**.
+3. Add **Mail.Send** (Application).
+4. Click **Grant admin consent for [Your Org]**.
+
+This will allow our backend to send emails on a schedule without manual login.
+
+Thank you!
+```
+
+## After Admin Consent
+- You can now use the client ID, tenant ID, and client secret to send emails via Microsoft Graph API using `/users/{user_id}/sendMail`.
+- No manual login will be required for scheduled jobs.
+
+---
+
+**If you need to check the current permissions or verify setup, use:**
+```sh
+az ad app permission list --id <client-id>
+```
+
+--- 

docs/email_digest_delivery_methods.md

@@ -0,0 +1,82 @@
+# Email Digest Delivery Methods: Summary & Lessons Learned
+
+This document summarizes all the methods we have tried (and considered) for sending automated email digests from the Future Trends & Signals platform, including their outcomes, blockers, and references to official documentation.
+
+---
+
+## 1. Microsoft Graph API (Recommended, but Blocked)
+
+- **Approach:** Use Microsoft Graph API with Application permissions to send as `futureofdevelopment@undp.org`.
+- **Status:** **Blocked** (admin consent for Application permissions not yet granted).
+- **What we did:**
+  - Registered the app in Azure AD.
+  - Attempted to use `/users/{user_id}/sendMail` endpoint with client credentials.
+  - Only Delegated permissions are currently granted; Application permissions are missing.
+- **Blocker:**
+  - Cannot send as a service account without `Mail.Send` Application permission and admin consent.
+- **Reference:**
+  - See [azure_graph_mail_send_setup.md](./azure_graph_mail_send_setup.md) for detailed setup and admin request template.
+  - [Microsoft Docs: Send mail as any user](https://learn.microsoft.com/en-us/graph/api/user-sendmail?view=graph-rest-1.0&tabs=http)
+
+---
+
+## 2. Microsoft Graph API (Delegated Permissions)
+
+- **Approach:** Use Microsoft Graph API with Delegated permissions, logging in as the sender.
+- **Status:** **Not suitable for automation**
+- **What we did:**
+  - Successfully authenticated as a user and sent test emails using `/me/sendMail`.
+- **Blocker:**
+  - Requires interactive login; not suitable for scheduled/automated jobs.
+
+---
+
+## 3. SMTP (Office 365/Exchange Online)
+
+- **Approach:** Use SMTP to send as `futureofdevelopment@undp.org` via `smtp.office365.com`.
+- **Status:** **Blocked** (SMTP AUTH is disabled for the tenant).
+- **What we did:**
+  - Created a script (`send_digest_smtp.py`) to send the digest via SMTP.
+  - Attempted to authenticate with valid credentials.
+  - Received error: `SMTPAuthenticationError: 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant.`
+- **Blocker:**
+  - SMTP AUTH is disabled for all users by default in modern Microsoft 365 tenants for security reasons.
+  - Would require IT to enable SMTP AUTH for the sending account.
+- **Reference:**
+  - [Enable or disable SMTP AUTH in Exchange Online](https://aka.ms/smtp_auth_disabled)
+
+---
+
+---
+
+## 5. Distribution List/Group Delivery
+
+- **Approach:** Send the digest to a mail-enabled group (`futures.curator@undp.org`).
+- **Status:** **Group is mail-enabled and can receive mail**
+- **What we did:**
+  - Verified the group exists and is mail-enabled in Azure AD.
+  - All sending methods above (if working) can target this group.
+- **Blocker:**
+  - Blocked by the same issues as above (Graph permissions or SMTP AUTH).
+
+---
+
+## **Summary Table**
+
+| Method                | Automation | Current Status         | Blocker/Notes                       |
+|-----------------------|------------|-----------------------|--------------------------------------|
+| MS Graph (App perms)  | Yes        | Blocked               | Need admin to grant permissions      |
+| MS Graph (Delegated)  | No         | Works (manual only)   | Not suitable for automation          |
+| SMTP (O365)           | Yes        | Blocked               | SMTP AUTH disabled for tenant        |
+| Distribution List     | Yes        | Ready                 | Blocked by above sending method      |
+
+---
+
+## **Next Steps**
+- Await admin action to grant Application permissions for Microsoft Graph (see [azure_graph_mail_send_setup.md](./azure_graph_mail_send_setup.md)).
+- Alternatively, request IT to enable SMTP AUTH for the sending account (less secure, not recommended).
+- Consider third-party relay if allowed by policy.
+
+---
+
+**This document should be updated as our setup or permissions change.**