The directory structure of installed Mercury libraries

# Introduction

Both "mmake" and "mmc --make" have mechanisms for installing a library to a target given directory. The name of this target directory is often called the install prefix, because the full pathname of all the installed files will include this name as a prefix.

The install process will copy files with the following suffixes to subdirectories of the install target directory. (In the rest of this document, we will refer to this list as the "library suffix list".)

- .int0
- .int
- .int2
- .int3
- .opt
- .module_dep
- .analysis
- .mh (for C grades)
- .mih (for C grades)
- .init (for C grades)
- .LE (for C grades, if --library-extension is .LE)
- .SLE (for C grades, if --shared-library-extension is .SLE)
- .dll (for C# grades)
- .jar (for Java grades)

Note that common values of --library-extension include .a and .lib, while common values of --shared-library-extension include .so, .dylib and .dll.

This document is intended to describe where (i.e. in which directories) files with each of the above suffixes are, or should be, installed. Specifically,

- where they are installed, by Mercury system versions before 2024 September 15, and
- where they should be installed, in a simplified and rationalized system that I (zs) think we should switch to.

The rest of this document will identify these two "scenarios" as LEGACY and PROPOSED respectively.

But before we can get to either of those, we need some background on where these files are stored in workspaces.

# The directory structure of Mercury workspaces

When the Mercury project started, we always stored all compiler-generated files, including all files with a suffix in the library suffix list, in the directory that stored the Mercury source files they were derived from. We refer to this directory as the "current directory", since that is the logical place to have as your current directory (in the Unix sense) when working on that code.

As the number of modules in e.g. the compiler directory grew, and as the number of files with different suffixes associated with a given module also grew, the number of files in the current directory became a problem. One problem was simply aesthetics: the output of "ls" in the current directory contained lots of "clutter". Another problem was more serious. Some of the Unix systems we used then had algorithms for filename lookups whose complexity was linear in the number of entries in a directory. This meant that the runtimes of tasks such as generating the .int3 file for each module in the directory was half-quadratic in the number of modules: for each of N modules, each of which had M associated files, the task of looking up that module's directory entry had to scan (on average) at least (N/2)*M other directory entries.

## The --use-subdirs option

We implemented the --use-subdirs option to solve both problems. (The second problem has since gone away, since pretty much all modern versions of Unix, and other OSs, have file lookup algorithms whose complexity is logarithmic in the number of entries in the directory, achieved by using data structures such as B-trees. The first problem still remains.) When --use-subdirs is specified, the compiler still puts the kinds of files that are explicitly intended to be looked at by the programmer, such .err files and HLDS/MLDS dumps, into the current directory, but it puts all other kinds of files into another directory.

The compiler associates with each extension an extension directory name. It originally created this nam from the extension itself, by deleting the initial dot and adding a final "s". For example, for the .int3 extension, the extension directory name is "int3s". The final "s" is intended to denote the plural form of the extension. This is because with --use-subdirs, the compiler stores all files with e.g. the .int3 extension in the directory named "Mercury/int3s".

We have since created some exceptions from the above rule. For example, for the .class extension, the extension directory name is "classes", replacing the old, ugly ".classs" (sic).

The general rule is: if ExtDir is the extension directory name for the extension .Ext, then for a module whose fully qualified module name is Fqmn, the compiler will put its .Ext file into "Mercury/ExtDir/Fqmn.Ext".

This scheme works well if you do all your work in the same grade. If you want to work on the same code in two or more different grades, e.g. because you want to switch to a debug grade to hunt down a bug, or if you want to switch back to the original grade once you have found the bug, you have a problem: you have to nuke the Mercury directory and rebuild it from scratch every time you want to switch grade. This takes a nontrivial amount of time, and therefore usually takes you any out of any "flow" state you had. The --use-subdirs option does not help with this problem at all.

## The --use-grade-subdirs option

We therefore added another option, --use-grade-subdirs, to solve this problem.

There are only three legal combinations of the settings of these options:

- --no-use-subdirs and --no-use-grade-subdirs (globals.m calls this combination use_cur_dir.)
- --use-subdirs and --no-use-grade-subdirs (globals.m calls this combination use_cur_ngs_dir.)
- --use-subdirs and --use-grade-subdirs (globals.m calls this combination use_cur_ngs_gs_dir.)

The fourth, illegal combination, --no-use-subdirs and --use-grade-subdirs, is actually unreachable, because --use-grade-subdirs implicitly implies --use-subdirs.

We classify extensions into two sets:

- the non-grade-specific extensions, which are grade-independent, and
- the grade-specific extensions, which are grade-dependent.

We usually abbreviate these two classes as ngs and gs respectively.

For the non-grade-specific extensions, the compiler puts their files

- into . (the current directory), with --no-use-subdirs, or
- into Mercury/ExtDir with --use-subdirs

LEGACY For the grade-specific extensions, the compiler puts their files

- into . (the current directory), with --no-use-subdirs, or
- into Mercury/ExtDir with --use-subdirs but --no-use-grade-subdirs, or
- into Mercury/Grade/TargetArch/Mercury/ExtDir with --use-subdirs and --use-grade-subdirs,

where Grade is a canonical representation of the grade and TargetArch is a canonical representation of the target ISA (instruction set architecture), with both being in a form that is suitable as a directory name component.

Note that while some kinds of files with grade-specific extensions, such as .o, are dependent on the target ISA, files with other grade-specific extensions, such as .mih, are NOT, so including the TargetArch component in their path names is overkill.

PROPOSED For the grade-specific extensions, the compiler should put their files

- into . (the current directory), with --no-use-subdirs, or
- into MercurySystem/ExtDir with --use-subdirs but --no-use-grade-subdirs, or
- into MercurySystem/ExtDir/Grade with --use-subdirs and --use-grade-subdirs, if files with the extension are not ISA specific
- into MercurySystem/ExtDir/Grade/TargetArch with --use-subdirs and --use-grade-subdirs, if files with the extension are ISA specific

The point of using "MercurySystem" instead of "Mercury" is to allow LEGACY and PROPOSED directory structures to coexist for the transition period. After that period is over, we can go back to using just "Mercury".

Note that to prepare for the implementation of the PROPOSED install structure, compiler/file_names.m makes a distinction between non-architecture-specific and architecture-specific extensions. It redefines the term grade-specific (gs) to refer to only the former, and introduces the term grade-and-architecture-specific (gas) to refer to the latter.

# Where files of various kinds are installed

The following shows, for each kind of file we install for a library, where those files are installed both in the LEGACY and in the PROPOSED install directory structures.

.int0

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/int0s

              PROPOSED
              Prefix/MercurySystem/int0s

.int

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/ints

              PROPOSED
              Prefix/MercurySystem/int1s

.int2

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/int2s

              PROPOSED
              Prefix/MercurySystem/int2s

.int3

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/int3s

              PROPOSED
              Prefix/MercurySystem/int3s

.opt

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/opts
              Prefix/lib/mercury/ints/Grade
              Prefix/lib/mercury/ints/Grade/Mercury/opts

              PROPOSED
              Prefix/MercurySystem/opts/Grade

  NOTE: LEGACY installs .opt files, which are grade-specific, to non-grade-specific directories. The non-grade-specific directory will end up with the .opt file for the default grade. If this directory is before the grade-specific directory in the search path, searches in non-default grades may nevertheless find the .opt file for the default grade.

.module_dep

              LEGACY
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/module_deps

              PROPOSED
              Prefix/MercurySystem/module_deps/Grade

.analysis

              LEGACY
              Prefix/lib/mercury/ints/Grade
              Prefix/lib/mercury/ints/Grade/Mercury/analyses

              PROPOSED
              Prefix/MercurySystem/analyses/Grade

.mh

              LEGACY
              Prefix/lib/mercury/inc
              Prefix/lib/mercury/ints
              Prefix/lib/mercury/ints/Mercury/mhs

              PROPOSED
              Prefix/lib/mercury/inc           (possibly, for backward compatibility)
              Prefix/MercurySystem/mhs

.mih

              LEGACY
              Prefix/lib/mercury/lib/Grade/inc
              Prefix/lib/mercury/lib/Grade/inc/Mercury/mihs
              Prefix/lib/mercury/lib/ints
              Prefix/lib/mercury/lib/ints/Mercury/mihs

              PROPOSED
              Prefix/lib/mercury/inc/Grade/inc (possibly, for backward compatibility)
              Prefix/MercurySystem/mihs/Grade

  NOTE: LEGACY installs .mih files, which are grade-specific, to non-grade-specific directories. The non-grade-specific directory will end up with the .mih file for the grade that is installed last. (The install process differs between .opt and .mih files.) If the non-grade-specific directory is before the grade-specific directory in the search path, searches in not-installed-last grades may nevertheless find the .opt file for the installed-last grade.

  Note that the install locations marked as "possible" for backward compatibility are not yet implemented.

.init

              LEGACY
              Prefix/lib/mercury/modules/Grade

              PROPOSED
              Prefix/MercurySystem/inits/Grade

.LE and .SLE

              LEGACY
              Prefix/lib/mercury/lib/Grade

              PROPOSED
              Prefix/MercurySystem/lib/Grade/TargetArch

.dll

              LEGACY
              Prefix/lib/mercury/lib/Grade

              PROPOSED
              Prefix/MercurySystem/lib/Grade

.jar

              LEGACY
              Prefix/lib/mercury/lib/Grade

              PROPOSED
              Prefix/MercurySystem/lib/Grade

# The PROPOSED directory structure of installed libraries

The following shows the overall structure of a library install directory under the PROPOSED rules. This table uses main_module to refer to the top module of the library, and moduleN to refer to any module of the library, whether it is the top module or not. As above, LE and SLE stand for the values of the --library-extension and --shared-library-extension options.

    // non-grade-specific files
    Prefix/MercurySystem/int0s/         moduleN.int0
    Prefix/MercurySystem/int1s/         moduleN.int
    Prefix/MercurySystem/int2s/         moduleN.int2
    Prefix/MercurySystem/int3s/         moduleN.int3
    Prefix/MercurySystem/mhs/           moduleN.mh

    // grade-specific but not architecture-specific files
    Prefix/MercurySystem/opts/          Grade/moduleN.opt
    Prefix/MercurySystem/trans_opts/    Grade/moduleN.trans_opt
    Prefix/MercurySystem/module_deps/   Grade/moduleN.module_dep
    Prefix/MercurySystem/analyses/      Grade/moduleN.analysis
    Prefix/MercurySystem/mihs/          Grade/moduleN.mih

    Prefix/MercurySystem/inits/         Grade/main_module.init
    Prefix/MercurySystem/lib/           Grade/main_module.jar
    Prefix/MercurySystem/lib/           Grade/main_module.dll

    // grade-specific and architecture-specific files
    Prefix/MercurySystem/lib/           Grade/TargetArch/main_module.LE
    Prefix/MercurySystem/lib/           Grade/TargetArch/main_module.SLE

# The principles of the PROPOSED directory structure

The principles governing the PROPOSED directory structure are the following.

- Each installed file should occur in the install library structure just once. (For backward compatibility, we may want to make an exception for .mh and .mih files, but the rest of this discussion assumes that we don't.)
- The relative path from Prefix to the installed location of the file Fqmn.Ext in an installed library should be the same as the relative path from . (the current directory in a workspace) to the location of Fqmn.Ext in (or under) that same directory, assuming a workspace with --use-grade-subdirs.
- The first directory name component after MercurySystem would dictate the structure of the subtree rooted at that directory. The structure would be one of
  - containing only regular files (all of which are non-grade-specific);
  - containing only subdirectories that identify grades, each of which contains only regular files (all of which are grade-specific but not architecture-specific); or
  - containing only subdirectories that identify grades, each of which contains only subdirectories that identify architectures, each of which contains only regular files (all of which are both grade-specific and architecture-specific).
- As a consequence of the above, no directory in the installed library would contain both directories and regular (non-directory) files.

  XXX The above structure violates this for Prefix/MercurySystem/lib/Grade, since it would contain both e.g. main_module.dll, a regular file, and TargetArch, a directory.

  XXX Should we replace Grade/TargetArch with e.g. Grade:TargetArch? Or should we install .jar and .dll files to Prefix/MercurySystem/jars/Grade/main_module.jar and Prefix/MercurySystem/dlls/Grade/main_module.dll respectively? The latter solution seems cleaner.

- As a consequence of the above, the installed library would contain no symbolic links, and especially no symbolic links that point to an ancestor directory :-(

Based on the above, it should be possible to use a single new mmc option, an accumulating option named something like --mercury-library-dir, that can take as arguments the names of both installed libraries and other directories in the current workspace.

For any directory Dir named in such an argument, when looking for Fqmn.Ext, mmc would search

- Dir/MercurySystem/ExtDir/Grade
- Dir/MercurySystem/ExtDir
- Dir

in that order if .Ext files are grade-specific but not architecture-specific, while it would search

- Dir/MercurySystem/ExtDir
- Dir

in that order if .Ext files are not grade-specific. (mmc never searches for architecture-specific files.) As a possible optimization, we could have a variant of this option, named maybe --workspace-library-dir, that is specialized for the use case where

- the named directory is another directory in the same workspace, and
- the named directory and the current directory use the same settings for --use-subdirs and --use-grade-subdirs.

This option variant would not do any search. Instead of searching in e.g.

- Dir/MercurySystem/ExtDir/Grade
- Dir/MercurySystem/ExtDir
- Dir

it would simply look at the one entry on that list that corresponds to the current settings of the --use-subdirs and --use-grade-subdirs options. This would make mmc slightly faster. However, the increased speed may be too small to be worth both

- the risk to users of them specifying this option even when the assumptions it needs are not satisfied, and
- the cost to us of developing a mechanism that integrates this with the base option. (A command line that contains both --mercury-library-dir and --workspace-library-dir options in some interleaved order would require those directories to be searched in that order, which means that the two options would need to maintain a common list of directories, even though different elements of that list would be need to be treated differently.)

  A simple way to do this would be for the --workspace-library-dir option to be a special option that adds its argument to both the list maintained by --mercury-library-dir and to a second, separate list. After getopt has done its job, we would convert the second list to a set. Thereafter, we would search the --mercury-library-dir list almost as usual, with the one difference being that we would treat any entry on that list that is also in this set as being an argument of --workspace-library-dir.

# System components that know the structure of install directories

The Mercury system components that know about the structure of library install directories, and which therefore must be updated if that structure ever changes, are the following. Most concern installs via mmake, but the last one concerns installs via mmc --make.

  scripts/Mmake.rules
This file contains most of the rules that install a library that has been built in a workspace into a chosen target directory (also called the prefix directory above). Specifically, these are all the static rules for library installs. As of 2024 Sep 16, these rules are in the last section of the file.

  scripts/Mmake.vars.in
The make entries in scripts/Mmake.rules refer to (and depend on) the values of make variables whose definitions are here. As of 2024 Sep 16, these rules are towards the end of this file, but not at the very end.

  compiler/generate_mmakefile_fragments.m
While script/Mmake.rules contains all the static rules for library installs, this file contains the code to generate the dynamic make rules for library installs. Being dynamic allows these rules to be specialized for the individual needs of a given library. The relevant code is in the generate_dep_file_install_targets predicate.

  Mmakefile
The Mmakefile in the top directory of each workspace contains some of the code to install the Mercury system itself. This requires installing several libraries (some in Mercury, some not) as well as several executables. As of 2024 Sep 16, these rules are in the second last section of the file.

  */Mmakefile
The Mmakefiles in the top-level subdirectories of each Mercury workspace contain the rest of the code to install the Mercury system itself. These files define the targets that the top level Mmakefile invokes.

XXX INSTALL Many of these files contain almost-duplicate copies of the same entries. It should be possible to factor out their commonalities.

  compiler/make.library_install.m
This module contains all the code that installs libraries for mmc --make.

XXX INSTALL There may be other files, such as bindist/bindist.Makefile.in.

# The structure of the LEGACY mmake library install targets

The following description specifies, for each mmake target, the following info:

- The name of the target, with "%" standing in for the name of the library. This is the first item on the first line.
- The location of the rule: either the standard Mmake.rules file, or the .dep file generated for the library (and therefore named after its main module). This is the second item on the first line.
- The job done by the actions of the rule. This is within angle brackets on the second line.
- The list of prerequisites of the target that are also part of the install system. The prereqs that require a file to be built before we try to install it are not shown.
- For the few rules whose actions explicitly call mmake recursively we show the (install-related) targets of those recursive invocations.

Note: some of the targets install mainly or exclusively grade-specific files, but their names do not contain the word "grade". The structure description below extends those names with extra text containing that word, with the extensions being distinguished by being inside square brackets.

    lib%.install                        (scripts/Mmake.rules)

        prereq  lib%.install_ints
        prereq  lib%.install_hdrs
        prereq  lib%.install_library[_grade_files]

        invokes lib%.install_grades

    lib%.install_grade_init             (scripts/Mmake.rules)

        prereq  install_grade_dirs      (which invokes only mkdir)

    lib%.install_library[_grade_files]  (scripts/Mmake.rules)

        prereq  install_grade_dirs      (which invokes only mkdir)
        prereq  lib%.install_grade_hdrs
        prereq  lib%.install_[grade]_opts
        prereq  lib%.install_grade_init

    lib%.install_grades                 (scripts/Mmake.rules)

        no prereqs here; all prereqs are on the invoker, lib%.install

        invokes lib%.install_library[_grade_files] FOR EACH LIBGRADE

    lib%.install_ints                   (main_module.dep)

        prereq  install_lib_dirs        (which invokes only mkdir)

    lib%.install_hdrs                   (main_module.dep)

        XXX We never install them if current grade is NOT C
        prereq  install_lib_dirs        (which invokes only mkdir)

    lib%.install_[grade]_opts           (main_module.dep)

        prereq  install_grade_dirs      (which invokes only mkdir)

    lib%.install_grade_hdrs             (main_module.dep)

        prereq  install_grade_dirs      (which invokes only mkdir)

# The structure of the PROPOSED mmake library install targets

The PROPOSED structure of mmake rules is much simpler than the LEGACY structure. It would consist of only three mmake targets:

    lib%.install_all_files
    lib%.install_ngs_pgs_files
    lib%.install_gs_gas_files

The job of the first one would be

- to invoke lib%.install_ngs_pgs_files once, and
- to invoke lib%.install_gs_gas_files for each install grade.

The last two rules divide the set of files to be installed into four categories.

- The non-grade-specific file category consists of .int0, .int, .int2 and .int3 files.
- The pseudo-grade-specific file category consists of .mh files. These files are not grade-specific, but they can be installed only when the library is built in a grade that targets C. So their install time is grade-specific, though their install target directory is not.
- The grade-specific file category consists of .mih, .opt, .trans_opt, .module_dep, .analysis, .init, .jar and .dll files.
- The grade-and-architecture-specific file category consists of .LE and .SLE files, whose suffixes are given by the values of the --library-extension and --shared-library-extension options.

Since we support the use of mmake only in grades that target C, the library will initially be built in such a grade, which means that we can install pgs files as well as ngs files in that state.

The code of the first rule could include code that tests the target language of each library grade, and invokes mmc --make instead of a recursive mmake if that language is Java or C#. This would enable users to include Java and/or C# grades in the list of grades in which to install a library. This is a capability that, as far as I (zs) know, does not currently exist.

The proposed three mmake targets, lib%.install_all_files, lib%.install_ngs_pgs_files and lib%.install_gs_gas_files, would all be in the library's main module's .dep file. This would have two main advantages. The first concerns only Mercury developers; the second concerns mostly Mercury users.

- With the LEGACY structure, any update that affects both a target in Mmake.rules and in the compiler-generated main_module.dep file has to contend with the fact that updating the Mmake.rules file and the compiler does not automatically update any previously-generated .dep file, and that an mmake invocation that uses both an updated Mmake.rules file and a non-updated .dep file will probably generate strange and hard-to-reproduce errors. With the PROPOSED rules for all targets being in the same file, such problems cannot happen.
- The targets being in the same file, and there being only three of them, their actions should also be easier to read and to understand, for any Mercury programmer who wants to understand them. This should be especially so because the code of the rules would be specialized for the actual setup of the library, in that e.g. there would be no targets or actions for installing .opt files if --intermodule-optimization is not specified for the library.