Skip to content

Optimize distributions #51

@jonahbeckford

Description

@jonahbeckford

Right now all the values and all the slots in the value stores are downloaded when a distribution is referenced. When a module is cross-compiled without a precommand optimizations (like CommonsBase_Std.S7z.MacLinux7zExe), and has N slots, that means O(N^2) entries are present in the value store. That is incredibly slow downloading and wasted storage and network traffic.

But typically we only want the final build values (O(1)).

So separate valuestore.zip into:

  1. assets.valuestore.zip: just the assets for a --assets-only build mode, where a build is rebuilt from scratch
  2. public.valuestore.zip: this is the default, and only the public: precommands of the distribute targets and their public: dependencies are packed
  3. build.valuestore.zip: this is every value in the value store, similar to today except the general modification below

General modification:

  • Only the values that belong to the distribute package should be in the valuestore.zip, plus any distributions (value/dist_json) plus any accepted licenses (add this to tracestore).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions