Proposed installation management

[zbeekman]

Hi all, I just want to elaborate more on how I suggest managing multiple compilers and installed packages. As you all know there is no standard Fortran ABI, and on many systems it is critical to support multiple Fortran compilers. .mod files can even be incompatible between different releases from the same compiler vendor. Often users then want to build their own programs against libraries that are installed. FLATPack should make it easy for users to link against and use Fortran packages managed by FLATPack.

I think FLATPack needs to know about compilers installed on the system via some configuration file, and optionally how to activate them. For example some DoD HPCs use GNU Modules to activate different compilers. Also, there may be MPI compiler wrapper scripts etc. A configuration file will be needed to tell FLATPack which compilers are available and how to use/activate them.

A method like

activate [.]

can be used to make available all packages installed on the system on standard compiler search paths. That way users only need to add a line like -ljsonfortran and the compiler can find libjsonfortran.a and json_module.mod since the proper versions of the files have been symlinked into compiler search paths by the activate method, and any previously activated packages from other compilers are unlinked.

$ tree \<prefix\>
<prefix>                                # This would be something like /usr/local/ or ~/
└── FLATPack                            # other FLATPack code could live in subdirectories, like Homebrew
    ├── formulae                        # holds python snippets containing build instructions kept under git version control
    └── packages
        ├── Intel
        │   └── 16.0.0
        │       ├── deployed            # Components of activated packages are linked into here   
        │       │   ├── bin
        │       │   ├── include
        │       │   ├── lib
        │       │   └── share
        │       └── json-fortran
        │           └── 4.1.1
        │               ├── bin
        │               ├── include
        │               ├── lib
        │               └── share
        ├── NAG
        ├── PGI
        └── gfortran
            ├── 4.9.2
            │   ├── deployed            # Components of activated packages are linked into here
            │   │   ├── bin
            │   │   ├── include
            │   │   ├── lib
            │   │   └── share
            │   └── json-fortran
            │       └── 4.1.1
            │           ├── bin
            │           ├── include
            │           ├── lib
            │           └── share
            └── 5.1.0
                ├── deployed            # Components of activated packages are linked into here
                │   ├── bin
                │   ├── include
                │   ├── lib
                │   └── share
                └── json-fortran
                    └── 4.1.1
                        ├── bin
                        ├── include
                        ├── lib
                        └── share

the activate method will first unlink all entries in /usr/local/{bin,lib,include,share} pointing to matching entries in the currently activated compiler’s deployed directory. It will then link each item in the requested compiler’s deployed directory into /usr/local/…

When building a new package the build can be performed with each compiler available on the system which the package has instructions for in its formula. This way the user can manage which compiler is currently active, and which packages and which versions of the packages are currently active.

[zbeekman]

This is essentially a complete rip off of Homebrew’s structure with an added level of linking to deal with Fortran’s lack of ABI and module compatibility.

[cmacmackin]

This looks excellent. However, I'm not quite clear on what the json-fortran directory is for. Is that where the compiling is done? Would there really be any need to keep that, once the binaries/.mods/documentation are copied into deployed. Or am I misunderstanding, and deployed contains links to the originals?

[cmacmackin]

Also, I've been putting some thought into how to deal with compilers. As you say, we'd need a configuration file. I was thinking this would be some JSON containing the following pieces of information:

• command: the command invoked to run the compiler
• vendor: the name of the compiler's vendor (e.g. GNU, Intel, NAG, etc.)
• version: the version number

As you say, there should be information on how to make the compiler available in case it is stored in a module. FLATPack would read this in and create a Compiler object. Between runs, this information could be cached in a pickled file and would only be updated in the event that the JSON is newer than the pickle.

As a slightly longer term goal, it might be nice to provide a script which can prompt for compiler information and generate the JSON. In the long run we might even be able to provide a utility which can probe for compilers and figure out their versions.

[szaghi]

I think that deployed should contain links to compiled objects that are in json-fortran. A clean way to have deployed exposing the production objects while having the possibility to have different versions of json-fortran library compiled not directly exposed to your path. The gnu module acts similarly: change your env rather than using sym links, but the logic is similar, different compiked objects of the same package that you could load on demand.

[zbeekman]

@cmacmackin I like the idea of json configuration files.

Yes, @szaghi is correct. deployed contains links to the originals which are installed in compiler-name/compiler-version/package-name/package-version/{bin,lib,share,include} This gives the user fine grained control over which packages are activated, allowing multiple versions to be installed and ready to go on the system, or packages with conflicts to be installed. Let’s look at an example usage:

Deploy a package:

deploy the default version of a package:

# for each compiler supported by json-fortran and for which json-fortran has been built take the
# following actions: in each compilers ‘deployed’ directory first remove any links pointing to any
# version of json-fortran, then create symlinks pointing to the default version 
$ FLATPack link json-fortran

deploy the specified version of the package:

# for each compiler supported by json-fortran and for which json-fortran has been built take the
# following actions: in each compilers ‘deployed’ directory first remove any links pointing to any
# version of json-fortran, then create symlinks pointing to the user specified version 
$ FLATPack link json-fortran HEAD

HEAD can be specified if the formula contains info on the HEAD, otherwise you need a version, like 4.1.1 instead of HEAD

Ultimately package authors are responsible for which compilers their package supports. Because of this, we may have conflicting packages installed for one compiler but not another. An optional command line switch could restrict the deployment operation to a list of compilers.

Activate a compiler:

activate a compiler so that the user can hack on a project that uses installed packages:

# activate the default Intel compiler
# first look in system folders (/usr/local/{bin,lib,include,share}) and unlink anything pointing to
# /usr/local/FLATPack/packages/…
# then create links in /usr/local/{bin,lib,share,include} pointing to the corresponding links in 
# /usr/local/FLATPack/packages/Intel/16.0.0/deployed/{bin,lib,share,include}.
# This will create a link like: /usr/local/lib/libjsonfortran.a —> /usr/local/FLATPack/packages/Intel/16.0.0/deployed/lib/libjsonfortran.a
# where /usr/local/FLATPack/packages/Intel/16.0.0/deployed/lib/libjsonfortran.a —> ../../json-fortran/4.1.1/lib/libjsonfortran.a
FLATPack activate Intel