Surface F# compiler timing data in regression test pipeline via binlog extraction

[Copilot]

Implementation Plan: Surface F# Compiler Timing Data in Regression Tests

• 1. Modify eng/scripts/PrepareRepoForRegressionTesting.fsx
  • Add --times flag injection in existing-file branch (after Import element)
  • Add --times flag in create-new-file template
  • Add idempotency check for --times flag
  • Fix XPath consistency issue
• 2. Modify eng/templates/regression-test-jobs.yml
  • Inject -bl flag for dotnet build/test/msbuild commands in Run-Command function
  • Add new step to extract and display timing data from binlogs (after "Collect Binary Logs" step)
  • Fix bash command execution to escape semicolons
• 3. Create eng/scripts/ExtractTimingsFromBinlog.fsx
  • Add MSBuild.StructuredLogger dependency
  • Parse binlog file path from command-line args
  • Walk build tree to find Fsc tasks
  • Extract and display all messages from Fsc tasks
  • Handle case when no Fsc tasks are found
• 4. Test and fix issues
  • Verify PrepareRepoForRegressionTesting.fsx changes work correctly
  • Verify ExtractTimingsFromBinlog.fsx can parse a sample binlog
  • Fix XPath selector consistency issue
  • Fix bash semicolon escaping issue

Original prompt

Goal

Surface per-project F# compiler timing data (--times) in the regression test CI pipeline output, using binlog extraction so it works regardless of MSBuild verbosity settings.

Background

The regression test pipeline (eng/templates/regression-test-jobs.yml) builds third-party F# projects with a freshly built compiler. We want to see --times output (compiler phase durations like Parse, Typecheck, TAST->IL, etc.) for each project in the CI logs. The --times output goes to fsc stdout, which MSBuild's ToolTask logs at MessageImportance.Low — invisible at default verbosity. However, binlogs capture everything regardless of verbosity. We already collect and publish binlogs. So the approach is: enable --times, ensure binlogs are produced, then extract and display the Fsc task output from the binlogs.

Changes required

1. Modify eng/scripts/PrepareRepoForRegressionTesting.fsx

This script currently injects <Import Project="...UseLocalCompiler.Directory.Build.props" /> into the third-party repo's Directory.Build.props. It needs to also inject <OtherFlags>$(OtherFlags) --times</OtherFlags> inside a <PropertyGroup>, placed right after the <Import> element.

In the existing-file branch (the if File.Exists(propsFilePath) branch starting around line 38): After inserting (or confirming) the <Import> element, create a <PropertyGroup> element containing an <OtherFlags> child with text content $(OtherFlags) --times, and insert the PropertyGroup after the import element (and after its trailing newline text node). Check for idempotency — if an <OtherFlags> element whose text contains --times already exists anywhere in the document, skip the insertion.

In the create-new-file branch (the else branch around line 74): Change the template string from:

<Project>
  <Import Project="..." />
</Project>

to:

<Project>
  <Import Project="..." />
  <PropertyGroup>
    <OtherFlags>$(OtherFlags) --times</OtherFlags>
  </PropertyGroup>
</Project>

2. Modify eng/templates/regression-test-jobs.yml

2a. Append -bl to dotnet commands in Run-Command

In the Run-Command function (around line 199), before the command is executed, add logic: if the command starts with dotnet build, dotnet test, or dotnet msbuild, append -bl to the command string. This ensures MSBuild produces a binary log file. For file-based build scripts (build.sh, build.cmd etc.) do NOT inject -bl — those may or may not support it, and the existing collection step already picks up any binlogs they produce.

Concretely, add this right at the top of Run-Command, before the if ($cmd -like "dotnet*") block:

# Ensure binlog is produced for dotnet build/test/msbuild commands
if ($cmd -match '^dotnet\s+(build|test|msbuild)') {
    $cmd = "$cmd -bl"
}

2b. Add a new pipeline step to extract and display timing data from binlogs

Insert a new step after the "Collect Binary Logs" step (around line 273) and before the "Publish Binary Logs" step (around line 275). This step should:

• Run condition: always() and continueOnError: true (timing extraction is informational, must not fail the build)
• Find all .binlog files in $(Pipeline.Workspace)/BinaryLogs
• For each binlog, invoke dotnet fsi $(Build.SourcesDirectory)/eng/scripts/ExtractTimingsFromBinlog.fsx "<binlog-path>"
• Wrap output in ##[group]F# Compiler Timing Summary for ${{ item.displayName }} / ##[endgroup] for a collapsible section in AzDO UI

Example step:

    - pwsh: |
        $binlogDir = "$(Pipeline.Workspace)/BinaryLogs"
        $script = "$(Build.SourcesDirectory)/eng/scripts/ExtractTimingsFromBinlog.fsx"
        
        if (-not (Test-Path $binlogDir)) {
          Write-Host "No binary logs directory found, skipping timing extraction"
          exit 0
        }
        
        $binlogs = Get-ChildItem -Path $binlogDir -Filter "*.binlog" -ErrorAction SilentlyContinue
        if ($binlogs.Count -eq 0) {
          Write-Host "No binlog files found, skipping timing extraction"
          exit 0
        }
        
        Write-Host "##[group]F# Compiler Timing Summary for ${{ item.displayName }}"
        foreach ($bl in $binlogs) {
          Write-Host ""
          Write-Host "--- Timings from $($bl.Name) ---"
          dotnet fsi $script "$($bl.FullName)"
        }
        Write-Host "##[endgroup]"
      displayName: Extract compiler timing from binlogs for ${{ item.displayName }}
      condition: always()
      continueOnError: true

3. Create eng/scripts/ExtractTimingsFromBinlog.fsx

A new F# script file. Here is the full specification:

• Start with #r "nuget: MSBuild.StructuredLogger" to load the dependency.
• Open Microsoft.Build.Logging.StructuredLogger and System.
• Parse command-line args to get the binlog file path (same arg-parsing pattern as PrepareRepoForRegressionTesting.fsx — skip args until one ends with .fsx, then ...

This pull request was created from Copilot chat.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[T-Gro]

@copilot :
Analyze the failing AzDo jobs and adjust your strategy.
If you have a root cause in mind, first reproduce it locally and then retest after a hypothetical fix.

[Copilot]

@copilot :
Analyze the failing AzDo jobs and adjust your strategy.
If you have a root cause in mind, first reproduce it locally and then retest after a hypothetical fix.

Fixed in afbb37e. The issue was an XPath selector inconsistency in PrepareRepoForRegressionTesting.fsx - it used different selectors to find the Import element (line 52 used exact path matching, but line 86 used contains() which could fail). Now both use the same xpath variable.

[T-Gro]

@copilot :

Analyze the failing AzDo jobs and adjust your strategy.
If you have a root cause in mind, first reproduce it locally and then retest after a hypothetical fix.

[Copilot]

@copilot :

Analyze the failing AzDo jobs and adjust your strategy.
...

Fixed in 1081fb4. The issue was that commands with semicolons (like dotnet msbuild build.proj -t:Build;Test -bl) were being passed to bash -c "$cmd" where bash interprets ; as a command separator. This caused only the first target to run. The fix escapes semicolons with backslash before passing to bash: $escapedCmd = $cmd -replace ';', '\;'

[github-actions]

✅ No release notes required