Build and testing instructions in contributor guidelines up to date?

[mloskot]

I'm following the B2 contributing guidelines to build and test/debug b2, and either the instructions are incomplete/out-of-date or I'm missing some important steps.

As prescribed, I pick my "stable" version and build b2 executable at the root:

D:\b2> git checkout 4.9.2
D:\b2> bootstrap.bat
D:\b2>.\b2.exe -v
B2 Version 4.9. OS=NT
...

Next, I switch to build the "in-progress" engine, and it fails:

D:\b2> git switch main
D:\b2> .\b2.exe b2
[errno 2] set.jam (No such file or directory)
[errno 2] regex.jam (No such file or directory)
[errno 2] string.jam (No such file or directory)
D:/b2/src/kernel\modules.jam:336: in import
IMPORT error: rule "difference" unknown in module "set".
D:/b2/src/build\virtual-target.jam:730: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\scanner.jam:30: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/util\print.jam:16: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\project.jam:40: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\alias.jam:28: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/tools\builtin.jam:12: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src\build-system.jam:12: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/kernel/bootstrap.jam:135: in module scope

It has been a very long time since I built b2 in order to develop and debug it, and it's been at the time the project was hosted at https://github.com/boostorg/build/

This is my first time I'm trying to achieve similar with the new repo at bfgroup/b2, so I may be missing important steps here.

[grafikrobot]

Those steps should work. And something close to those is tested in CI also https://dev.azure.com/bfgroup/B2/_build/results?buildId=585&view=logs&j=1e381fc3-6dae-5cba-eb00-5ac649efcf07&t=7d0ca7bf-19a2-5909-55a1-45c1f7dcd8e3. Given the errors it would appear that your .\b2.exe executable is not actually the latest build one. As the reason for those errors is that in the latest engine code those modules exist entirely in the C++ side.

[mloskot]

I have re-tested the procedure and something is not right:

Build 4.9.2 with 4.9.2 - Works

D:\b2>bootstrap.bat
Building the B2 engine..
LOCALAPPDATA=C:\Users\mateuszl\AppData\Local
Found with vswhere C:\Program Files\Microsoft Visual Studio\2022\Professional
Found with vswhere C:\Program Files\Microsoft Visual Studio\2022\Professional
Call_If_Exists "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\..\..\VC\Auxiliary\Build\vcvarsall.bat"  AMD64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.2.5
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
###
### Using 'vc143' toolset.
###

D:\b2\src\engine>"cl" /nologo -TP /wd4996 /wd4675 /EHs /GR /Zc:throwingNew /O2 /Ob2 /W3 /MD /Zc:forScope /Zc:wchar_t /Zc:inline /Gw /favor:blend /Feb2   -DNDEBUG  builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filent.cpp filesys.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp jam_strings.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp /link kernel32.lib advapi32.lib user32.lib
builtins.cpp
class.cpp
<... snip...>
sequence.cpp
set.cpp
Generating Code...

D:\b2\src\engine>dir *.exe
 Volume in drive D is Data
 Volume Serial Number is 6C15-029A

 Directory of D:\b2\src\engine

08/07/2022  08:46           253,952 b2.exe
               1 File(s)        253,952 bytes
               0 Dir(s)  760,890,028,032 bytes free

Building is done. To install, run:

    .\b2 --prefix=DIR install

D:\b2>.\b2 -v
B2 Version 4.9. OS=NT.
  Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.
  Copyright 2001 David Turner.
  Copyright 2001-2004 David Abrahams.
  Copyright 2002-2019 Rene Rivera.
  Copyright 2003-2015 Vladimir Prus.

  DEFAULTS: jobs = 40

D:\b2>.\b2 b2
Jamroot.jam:107: in modules.load from module Jamfile<D:\b2>
warning: Bison generator program 'bison' not found. Skipping grammar build.
D:/b2/src/build\project.jam:372: in load-jamfile from module project
D:/b2/src/build\project.jam:64: in load from module project
D:/b2/src/build\project.jam:142: in project.find from module project
D:/b2/src\build-system.jam:618: in load from module build-system
D:/b2/src/kernel\modules.jam:294: in import from module modules
D:/b2/src/kernel/bootstrap.jam:135: in module scope from module
warning: No toolsets are configured.
warning: Configuring default toolset "msvc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: https://www.bfgroup.xyz/b2/manual/release/index.html#bbv2.overview.configuration
...patience...
...found 232 targets...
...updating 64 targets...
msvc.write-setup-script .build\standalone\msvc\msvc-14.3\msvc-setup.bat
compile-c-c++ .build\msvc-14.3\debug\cxxstd-11-iso\src\engine\constants.obj
constants.cpp
<... snip...>
compile-c-c++ .build\msvc-14.3\debug\cxxstd-11-iso\src\engine\pathnt.obj
pathnt.cpp
msvc.link .build\msvc-14.3\debug\cxxstd-11-iso\b2.exe
...updated 65 targets...

D:\b2> .build\msvc-14.3\debug\cxxstd-11-iso\b2.exe -v
B2 Version 4.9. OS=NT.
  Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.
  Copyright 2001 David Turner.
  Copyright 2001-2004 David Abrahams.
  Copyright 2002-2019 Rene Rivera.
  Copyright 2003-2015 Vladimir Prus.

  DEFAULTS: jobs = 40

Build main with 4.9.2 - Fail

D:\b2>rd /s .build
.build, Are you sure (Y/N)? Y

D:\b2>.\b2.exe -v
B2 Version 4.9. OS=NT.
  Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.
  Copyright 2001 David Turner.
  Copyright 2001-2004 David Abrahams.
  Copyright 2002-2019 Rene Rivera.
  Copyright 2003-2015 Vladimir Prus.

  DEFAULTS: jobs = 40

D:\b2>.\b2.exe b2
[errno 2] set.jam (No such file or directory)
[errno 2] regex.jam (No such file or directory)
[errno 2] string.jam (No such file or directory)
D:/b2/src/kernel\modules.jam:336: in import
IMPORT error: rule "difference" unknown in module "set".
D:/b2/src/build\virtual-target.jam:730: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\scanner.jam:30: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/util\print.jam:16: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\project.jam:40: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/build\alias.jam:28: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/tools\builtin.jam:12: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src\build-system.jam:12: in load
D:/b2/src/kernel\modules.jam:309: in import
D:/b2/src/kernel/bootstrap.jam:135: in module scope

Build main with main - Works

Building the B2 engine..
LOCALAPPDATA=C:\Users\mateuszl\AppData\Local
Found with vswhere C:\Program Files\Microsoft Visual Studio\2022\Professional
Found with vswhere C:\Program Files\Microsoft Visual Studio\2022\Professional
Call_If_Exists "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\..\..\VC\Auxiliary\Build\vcvarsall.bat"  AMD64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.2.5
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
###
### Using 'vc143' toolset.
###

D:\b2\src\engine>"cl" /nologo -TP /wd4996 /wd4675 /EHs /GR /Zc:throwingNew /O2 /Ob2 /W3 /MD /Zc:forScope /Zc:wchar_t /Zc:inline /Gw /favor:blend /Feb2   -DNDEBUG  bindjam.cpp builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filent.cpp filesys.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp jam_strings.cpp startup.cpp subst.cpp timestamp.cpp value.cpp variable.cpp w32_getreg.cpp mod_regex.cpp mod_set.cpp mod_string.cpp mod_order.cpp mod_path.cpp mod_property_set.cpp mod_sequence.cpp mod_sysinfo.cpp mod_version.cpp /link kernel32.lib advapi32.lib user32.lib
bindjam.cpp
builtins.cpp
<... snip ...>
mod_version.cpp
Generating Code...

D:\b2\src\engine>dir *.exe
 Volume in drive D is Data
 Volume Serial Number is 6C15-029A

 Directory of D:\b2\src\engine

08/07/2022  09:06           338,432 b2.exe
               1 File(s)        338,432 bytes
               0 Dir(s)  760,888,799,232 bytes free

Building is done. To install, run:

    .\b2 --prefix=DIR install

D:\b2>.\b2.exe -v
B2 5.0 (OS=NT, jobs=40).

D:\b2>.\b2.exe b2
Jamroot.jam:107: in modules.load from module Jamfile<D:\b2>
warning: Bison generator program 'bison' not found. Skipping grammar build.
D:/b2/src/build\project.jam:372: in load-jamfile from module project
D:/b2/src/build\project.jam:64: in load from module project
D:/b2/src/build\project.jam:142: in project.find from module project
D:/b2/src\build-system.jam:618: in load from module build-system
D:/b2/src/kernel\modules.jam:309: in import from module modules
D:/b2/src/kernel/bootstrap.jam:135: in module scope from module
warning: No toolsets are configured.
warning: Configuring default toolset "msvc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: https://www.bfgroup.xyz/b2/manual/release/index.html#bbv2.overview.configuration
...patience...
...found 241 targets...
...updating 66 targets...
msvc.write-setup-script .build\standalone\msvc\msvc-14.3\msvc-setup.bat
compile-c-c++ .build\msvc-14.3\debug\cxxstd-11-iso\src\engine\constants.obj
constants.cpp
<... snip ...>
compile-c-c++ .build\msvc-14.3\debug\cxxstd-11-iso\src\engine\value.obj
value.cpp
msvc.link .build\msvc-14.3\debug\cxxstd-11-iso\b2.exe
...updated 67 targets...

D:\b2>.build\msvc-14.3\debug\cxxstd-11-iso\b2.exe -v
B2 5.0 (OS=NT, jobs=40).

---

AFAICT, building "in-progress" with "stable" is not working.

[grafikrobot]

I have re-tested the procedure and something is not right:

Build 4.9.2 with 4.9.2 - Works

As intended.

Build main with 4.9.2 - Fail

As intended.

Build main with main - Works

As intended.

AFAICT, building "in-progress" with "stable" is not working.

Generally b2 only supports building when the version of the engine matches the version of the build files. For equal minor (x.y != x.z) version there is an error produced (used to be a warning) indicating that. For different major versions (x.* != y.*) it can likely totally fail. Which happens to be the case for the 4.x to 5.x case. The instructions are written assuming the engine and build versions match.

[mloskot]

Thank you for the clarification. It makes sense.
I have proposed copying it to the guidelines in #179