Is Your Microsoft 365 Tenant Secure? Audit It with Microsoft ORCA

Microsoft ORCA performs a comprehensive security assessment of an organization’s Microsoft 365 email security configuration. Rather than scanning endpoints or user mailboxes, ORCA focuses on evaluating Exchange Online and Microsoft Defender for Office 365 settings against Microsoft’s security best practices. During the assessment, ORCA reviews multiple security categories to identify configuration weaknesses, insecure policies, and opportunities to strengthen the tenant’s overall security posture.

Exchange Online Protection (EOP)

ORCA analyzes the core email protection configuration, including Anti-Spam policies, Anti-Malware policies, Quarantine settings, and Outbound Spam protection. These controls help ensure that malicious emails are detected, filtered, and prevented from reaching users or leaving the organization.

Microsoft Defender for Office 365

The assessment validates advanced protection features such as Safe Links, Safe Attachments, Anti-Phishing policies, Mailbox Intelligence, and Anti-Spoofing protection. These features defend users against phishing campaigns, malicious attachments, spoofed identities, and business email compromise (BEC) attacks.

Email Authentication

ORCA verifies that email authentication mechanisms are properly configured by checking SPF, DKIM, ARC, and MX records. Correct configuration of these technologies significantly reduces the risk of email spoofing and improves the organization’s email reputation.

Mail Flow Configuration

The tool reviews Exchange mail flow settings, including Connectors, Transport Rules, and Domain Allow Lists. Misconfigured mail flow rules or overly permissive allow lists can weaken email security and allow malicious messages to bypass protection.

Outlook Configuration

ORCA checks Outlook security settings such as External Email Tags. Displaying banners for emails originating outside the organization helps users quickly recognize potentially suspicious messages and reduces the likelihood of phishing attacks.

Threat Protection

The assessment also evaluates automated protection mechanisms, including Zero-Hour Auto Purge (ZAP) and Microsoft Defender Protection Alerts. These features automatically remove malicious emails after delivery and notify administrators about important security events.

Security Baselines

Finally, ORCA compares the tenant configuration against Microsoft’s Standard and Strict Preset Security Policies. This allows administrators to quickly identify configuration gaps and determine whether their environment aligns with Microsoft’s recommended security baseline.

By reviewing these security areas, ORCA provides administrators with a comprehensive overview of the organization’s Microsoft 365 security posture and generates actionable recommendations to improve email security, strengthen protection against modern threats, and reduce the overall attack surface.

installing Microsoft ORCA

The first step is to install the ORCA PowerShell module from the PowerShell Gallery. ORCA is distributed as a PowerShell module and can be installed directly using the following command:

Install-Module -Name ORCA

What happens during the installation?

When running the installation command for the first time, PowerShell may prompt you to install the NuGet Provider, which is required to download packages from the PowerShell Gallery.

NuGet provider is required to continue

Select Y (Yes) to install the provider.

Next, PowerShell displays a warning indicating that the PowerShell Gallery (PSGallery) is not marked as a trusted repository.

Untrusted repository

Choose Y (Yes) to continue the installation.

Once completed, ORCA will be installed under the Windows PowerShell Modules directory.

Verify the Installation

After the installation finishes, verify that ORCA has been installed successfully by running:

Get-Module ORCA -ListAvailable

If the installation was successful, PowerShell returns information similar to the following:

This confirms that the ORCA module is available on the system.

Import the ORCA Module

Before using any ORCA cmdlets, the module must be imported into the current PowerShell session.

Import-Module ORCA

During the first attempt, PowerShell may display the following error:

running scripts is disabled on this system

This occurs because the current PowerShell Execution Policy prevents scripts from running.

Temporarily Allow Script Execution

Instead of permanently changing the execution policy, a safer approach is to temporarily bypass the restriction for the current PowerShell session.

Set-ExecutionPolicy -Scope Process Bypass

This change only affects the active PowerShell session. Once PowerShell is closed, the execution policy automatically returns to its previous state.

After applying the temporary execution policy, import the ORCA module again:

Import-Module ORCA

Verify Available ORCA Commands

To display all available ORCA cmdlets, run:

Get-Command -Module ORCA

Typical output includes:

• Get-ORCAReport
• Get-ORCAReportEmbeddedConfig
• Invoke-ORCA

These cmdlets are used to generate reports and execute ORCA security assessments.

Generate the Security Assessment Report

To start the assessment and generate an HTML report, run:

Get-ORCAReport -Output HTML

During the first execution, PowerShell may display a security warning related to Invoke-WebRequest.

Security Warning: Script Execution Risk

This warning is expected because ORCA checks for module updates and downloads required components.

If prompted, select Y (Yes) to continue.

Installing Exchange Online Management

If the ExchangeOnlineManagement module is not already installed, ORCA automatically detects the missing dependency and installs it.

Exchange Online Management module is missing

Accept the installation prompt by selecting Y (Yes).

Once installed, ORCA uses the Exchange Online Management module to establish a secure connection with Microsoft 365 using Modern Authentication.

Connecting to Microsoft 365

During execution, ORCA connects to several Microsoft 365 services, including:

• Exchange Online
• Security & Compliance Center (SCC)

These connections allow ORCA to collect the tenant’s security configuration before starting the analysis.

Running the Security Assessment

Once connected successfully, ORCA begins collecting and analyzing security configurations across the Microsoft 365 tenant.

The assessment includes:

• Exchange Online Protection (EOP)
• Microsoft Defender for Office 365
• Anti-Spam Policies
• Anti-Phishing Policies
• Safe Links
• Safe Attachments
• DKIM Configuration
• SPF Records
• Connectors
• Mail Flow Rules
• Outlook External Tags
• Zero-Hour Auto Purge (ZAP)
• Protection Alerts
• Security Preset Policies

Each configuration is compared against Microsoft’s recommended security best practices.

Report Generation

After completing the analysis, ORCA automatically generates a comprehensive HTML report.

The report is stored locally under:

C:\Users\<Username>\AppData\Local\Microsoft\ORCA\

Example:

ORCA-Fayoumy@blalala.net-202606171117.html

The report contains:

• Security Findings
• Configuration Gaps
• Risk Levels
• Microsoft Recommendations
• Remediation Guidance

This report provides administrators with a detailed overview of the tenant’s email security posture and serves as a valuable reference for identifying and remediating Microsoft 365 security misconfigurations.