1. Topics
  2. C++ (and others) Package Management Quick Survey

C++ (and others) Package Management Quick Survey

Aloys Baillet
 

Hi,
 
Here is a quick pass on researching potential package managers to represent the VFX Platform packages.
You can also find some additional and recent thoughts from the author of cget: http://pfultz2.com/blog/2017/10/27/universal-package-manager/
 
 
cget:
Pros:
Cons:
  • Maintained by a single user
  • No artifact server
 
Conda:
https://docs.conda.io/en/latest/ (part of Anaconda data science distro)
Pros:
Cons:
  • Main goal is packages to work with Anaconda
  • Single repo(?)
  • Verbose package specs
  • Single linux x64 arch: no way to specify VFX platforms
 
Conan:
Pros:
  • Decentralised artifact servers
  • Package and tests packages built-in
  • “settings” can express VFX platform versions quite appropriately
  • Integrations available with autotools and cmake
  • Well maintained and documented by JFrog, developers of artifactory
Cons:
  • Developers only: no end-user deployment
 
Rez:
Pros:
  • Well known by VFX Studios
  • Both build recipes and end-user deployments
Cons:
  • Not easy to represent multiple VFX platforms
  • Single main developer, documentation is WIP
  • Artifacts are mostly assumed to be available on the file system
 
Hunter:
Pros:
  • Cmake-based
Cons:
  • Cmake-based?
  • No artifact management
 
Buckaroo
Pros:
  • Single executable for developer install… (but buck build seems Java?)
Cons:
  • Seems to be also the “build” system
 
My personal favourite option would be a hybrid of Conan and Rez... I'm thinking that Conan could be used as a package provider for Rez going forward.
Would be great to migrate this page to a working document format that we can all contribute to and keep alive maybe?
 
 
Cheers,
 
Aloys
Daniel Heckenberg
 

Hi Aloys, all,

Thanks again to Aloys for preparing this survey.  The interest in the package management topic was emphasized at our recent TAC meeting and forum, so I wanted to bump this thread and encourage everyone to look through these options in preparation for our imminent CI working group meeting.

Cheers,
Daniel
jordanbrandes@...
 

I'd go Conda personally. It's really easy to use and used almost everywhere.

Thanks,

Jordan
Jeff Bradley
 

Could you elaborate on the Rez "Con" of "Not easy to represent multiple VFX platforms?"  We've been using an internal approach with rez for so long, I suspect we've done some work-around. e.g. Internally, we don't use cmake, so if it's a problem during the actual build process, we may not hit it.

fwiw, we have packages using weak package references for boost, Qt, etc... that represent platforms for almost each major application, since many of the applications we use aren't aligned to the vfx platforms. It's mostly working well for both our build and runtime rez environments.
Aloys Baillet
 

Hi,

Here is a great talk about the state of C++ package management from a few days ago at the StockholmCpp conf: https://www.youtube.com/watch?v=C1m-Uy4nuQA
This would steer me a bit more towards Conan but as the presenter says he is biased :-)
I wasn't aware that vcpkg now supports linux and mac, but the fact that all the versions are locked and centrally managed is a big no-no for VFX Platform compliance.

Regarding Conda as I mentioned before it feels a lot more geared towards the scientific community and is a huge download to get started... I don't think it would fit us.

Re: Jeff about rez and multiple platforms, I could go into some of the details of our recent upgrade to VFX Platform 2018, where we had to rebuild about 500 rez packages I'll do a short version.
We had the following rez packages:
CentOS-6.4 <- stdlib-4.8.3 <- gcc-4.8.3
And all our C++ packages used to have a runtime dependency on the stdlib used at build time ( stdlib-4.8.3+<5 ) where stdlib contained the libstdc++.so from the matching gcc.

Our upgrade to VFX-2018 we switched to the official devtoolset-6.1 instead of using our own build of gcc as the crazy/awesome hacks made by RedHad to the linkerscript in devtoolset to allow C++14 to work against the system's libstdc++ version means that we had to get rid of our stdlib package, so here's our current structure:
CentOS-7.4 <- devtoolset-6.1 <- gcc-6.3.1
And all C++ packages now depend on CentOS-7.4 only.
Looks both simpler and better, right?

BUT!

We have no idea how we can start using C++14 in some of our packages and propagate this requirement down the rez dependency chain! So we know that once we can really start using C++14 in large upstream C++ libraries such as USD we know we will have to rebuild all downstream C++ packages with the right flags, and this is not something rez will be able to help us with. Rez has no idea about C++ flavours. We could try to use variants but variants are going to make some problems very hard: if you do a "rez env somePythonPackageThatDependsOnUSD" then which C++ would you get? Rez makes no guarantees there...

Also another issue with rez is a very similar issue: debug builds vs. release builds. We can't have rez variants for debug builds... And USD (again) debug builds are not ABI compatible with non-debug builds... that's where Conan would help a lot. Hope this clarifies some of the limitations of rez? And I would love to hear if you have a solution for the C++ version or the debug build issue?

Cheers,

Aloys
Guillaume Papin
 

Hello,

Interesting and complex subject, as there is no single/perfect tool yet, which makes C++ sometimes more difficult to work compared to other languages.
Like jordanbrandes above, I believe conda is an interesting tool.

I'd like to comment on some of the cons listed previously, as they do not reflect what has been my experience with the tool so far.

Cons:
- Single repo(?)
 
I'm not sure what is meant by single repo, but in conda you can have multiple channels/repositories of packages.
The mechanism is flexible enough to have multiple per-user, per-site and global repositories of packages.
You can use package you built locally, or use one of the official channels or create your own channel at repo.continuum.io.
But file:// URLs are also supported, so packages can easily be shared on an NFS shared for example.

Cons:
- Verbose package specs
 
Regarding the verbosity of package specs, I'm not sure it's better or worse than conan or vcpkg.
Conda has : meta.yaml, build.sh (Linux and macOS) and build.bat (Windows)
Conan : build.py, conanbuild.py
vcpkg: port.cmake and CONTROL
nixpgs: default.nix(?)

Basically, the complexity of these files reflects the complexity of the dependencies, options and idiosyncrasy of the package.

Cons:
- Single linux x64 arch: no way to specify VFX platforms

Conda works well on all 3 majors distributions: Windows, macOS, Linux.
As an anecdotal example, but I think CI was talked about in this list, on Azure Pipelines there is built-in integration of conda and it works similarly on all 3 major platforms:
Regarding the specification of the VFX platforms, I'm not sure what is the best way to specify it with conda, but I'm pretty sure there are multiple possibilities :
  1. Create a meta-package "vfx-$year", which brings in all the dependencies.
  2. A channel per VFX platform.
  3. Specify the package version you are interested directly in your dependencies, useful if a package is itself a member of the "meta-package" vfx-$year.
In any way, building the set of packages for each VFX reference platforms is theorically easy (once the package recipes have been written).

You can describe the VFX platform in a YAML file, and then use conda-build to make a build matrix of all the packages.

The workflow could be along these lines :

$ cat vfx2018.yaml

python:

- 2.7.5

qt:

- 5.6

openexr:

- 2.2

openvdb:

- 5

...

$ conda build --variant-config-file vfx2018.yaml

$ cat vfx2019.yaml

python:

- 2.7.5

- 3.6

qt:

- 5.12

openexr:

- 2.3

openvdb:

- 6

...

$ conda build --variant-config-file vfx2019.yaml



 Regarding Conda as I mentioned before it feels a lot more geared towards the scientific community and is a huge download to get started... I don't think it would fit us.

I think the scientific communities uses similar technologies to VFX such as Python and C++, so conda handle reasonably well these sort of packages.
Anaconda the commercial product/scientific distribution may be a big download, but have never downloaded it.
There is miniconda, which is just the tools but not the default selection of package, which is what I expect people would like to use here.
For example, I have used miniconda on CI (Gitlab CI), and it's fairly easy to do, and not what takes time during the build.
Fetching the dependencies takes more time.


Regarding the issues you mention with C++14, debug/release and API/ABI compatibility in general, I'm not sure if conda is better or worse than Rez in this regard.
At least for debug/release, they seems to recommend the use of "features", and I think the C++ standard could be applied here as well:

Regarding conan, one of the reason I'm not fond of it, is the official guidelines regarding the CMake integration.
Unlike vcpkg or conda, it seems intrusive but I'm confident they could fix this.
In comparison, vcpkg provides a toolchain file :

cmake -DCMAKE_TOOLCHAIN_FILE=.../vcpkg.cmake

And for conda, if CMake is provided by the conda environment, there is nothing to do, because CMake will find packages relative to there installations. However, if cmake is not installed via conda, you usually just have to specify the CMAKE_PREFIX_PATH like this: cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..

 
A few other things I like about conda:
  • package can describe their ABI compatibility guarantees: package can declare whether it is following semantic versioning or something else (I believe there was a discussion about the versioning scheme on this list, conda should adapt to most of them)
  • packages are merged in a single prefix, which means activation can be fast and just be adding one element to PATH: $CONDA_PREFIX/bin (it's possible for packages to export other environment variable though)
    Also, this single prefix approach, means you don't have PYTHONPATH explosion as everything is is merged in a single prefix.
    For Python, like CMake, if you have it in your environment, you don't even have to touch PYTHONPATH or similar, it is found automatically.
  • the automatic stuff conda build does for package relocatability worked well for me so far, it knows how to patch things like installation prefix embedded in binaries:
    https://docs.conda.io/projects/conda-build/en/latest/source/resources/make-relocatable.html
  • not limited to C++ or CMake, it's a general purpose package manager, Python is well supported
  • good community, lots of packages and recipes already available

Hope that helps,
Guillaume

On Wed, Mar 6, 2019 at 7:56 AM Aloys Baillet <aloys.baillet@...> wrote:
Hi,

Here is a great talk about the state of C++ package management from a few days ago at the StockholmCpp conf: https://www.youtube.com/watch?v=C1m-Uy4nuQA
This would steer me a bit more towards Conan but as the presenter says he is biased :-)
I wasn't aware that vcpkg now supports linux and mac, but the fact that all the versions are locked and centrally managed is a big no-no for VFX Platform compliance.

Regarding Conda as I mentioned before it feels a lot more geared towards the scientific community and is a huge download to get started... I don't think it would fit us.

Re: Jeff about rez and multiple platforms, I could go into some of the details of our recent upgrade to VFX Platform 2018, where we had to rebuild about 500 rez packages I'll do a short version.
We had the following rez packages:
CentOS-6.4 <- stdlib-4.8.3 <- gcc-4.8.3
And all our C++ packages used to have a runtime dependency on the stdlib used at build time ( stdlib-4.8.3+<5 ) where stdlib contained the libstdc++.so from the matching gcc.

Our upgrade to VFX-2018 we switched to the official devtoolset-6.1 instead of using our own build of gcc as the crazy/awesome hacks made by RedHad to the linkerscript in devtoolset to allow C++14 to work against the system's libstdc++ version means that we had to get rid of our stdlib package, so here's our current structure:
CentOS-7.4 <- devtoolset-6.1 <- gcc-6.3.1
And all C++ packages now depend on CentOS-7.4 only.
Looks both simpler and better, right?

BUT!

We have no idea how we can start using C++14 in some of our packages and propagate this requirement down the rez dependency chain! So we know that once we can really start using C++14 in large upstream C++ libraries such as USD we know we will have to rebuild all downstream C++ packages with the right flags, and this is not something rez will be able to help us with. Rez has no idea about C++ flavours. We could try to use variants but variants are going to make some problems very hard: if you do a "rez env somePythonPackageThatDependsOnUSD" then which C++ would you get? Rez makes no guarantees there...

Also another issue with rez is a very similar issue: debug builds vs. release builds. We can't have rez variants for debug builds... And USD (again) debug builds are not ABI compatible with non-debug builds... that's where Conan would help a lot. Hope this clarifies some of the limitations of rez? And I would love to hear if you have a solution for the C++ version or the debug build issue?

Cheers,

Aloys
Konstantin Dmitriev
 


ср, 6 мар. 2019 г. в 19:14, Guillaume Papin <guillaume.papin@...>:
Regarding conan, one of the reason I'm not fond of it, is the official guidelines regarding the CMake integration.
Unlike vcpkg or conda, it seems intrusive but I'm confident they could fix this.


Yes, this can be fixed with something like this:

if (EXISTS ${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
 include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
 conan_basic_setup()


Best Regards,
Konstantin
Jeff Bradley
 

Focusing on just the rez, compiler, C++ language flags, and debug/non-debug aspects of this thread...

We do something like what Guillaume mentioned. We encapsulate interesting packages from the VFX platform into platform packages that might have a dependency list in rez that looks like this:

name = 'refplat'
 
version = 'vfx2018.1.0.0'
 
requires = [
    'gcc-4.8',
    '~alembic-1.5.8.x.4',
    '~boost-1.61.0.x.1',
    '~ilmbase-2.2.0.x.3',
    '~openexr-2.2.0.x.3',
    '~python-2.7',
    '~qt-5.6.1.x.1',
    '~tbb-4',
    '~mkl-11.3',
]


The lines preceded by tilde's are "weak references," which tells rez, "If a package requests Qt, here's the version spec I require." And one DreamWorks specific note: we add internal versioning to the canonical versions, separated by a 'x'.  This lets us clearly denote the canonical version, while allowing us make changes that could break compatibility with deployed rez packages. e.g. remove a variant, change compiler flags, namespaces,...

I fully expect we'll hit more challenges in the future, but given the choices we've made to simplify this crazy software world, this approach seems to be working well.

Hope this helps clarify where my questions are coming from.
- Jeff
Allan Johns
 

Just to chip in briefly on rez:

Current limitations of variants (in the sense that they can only describe extra dependencies, and these are used to construct the installation path) are a problem. As Aloys mentioned, it isn't really possible to manage debug/release builds with rez. You can hack it, by providing and varianting on stub packages that represent different built types (eg `build-debug`, `build-release`) but that's pretty clunky and would result in some fairly bizarre installations, as well as forcing selection of build type across all packages in a resolve.

The solution to these problems IMO is the introduction of more expressive variants (allowing for, eg, build-only requirements that are variant-specific); and, funnily enough, "features", pretty much as described by the Conda docs provided earlier. I envision an extension to the requirements syntax, where packages and user both can specify something like `rez-env my_cpp_pkg-5@debug=true`.

A






On Thu, Mar 7, 2019 at 6:34 AM Jeff Bradley <jeff.bradley@...> wrote:
Focusing on just the rez, compiler, C++ language flags, and debug/non-debug aspects of this thread...

We do something like what Guillaume mentioned. We encapsulate interesting packages from the VFX platform into platform packages that might have a dependency list in rez that looks like this:

name = 'refplat'
 
version = 'vfx2018.1.0.0'
 
requires = [
    'gcc-4.8',
    '~alembic-1.5.8.x.4',
    '~boost-1.61.0.x.1',
    '~ilmbase-2.2.0.x.3',
    '~openexr-2.2.0.x.3',
    '~python-2.7',
    '~qt-5.6.1.x.1',
    '~tbb-4',
    '~mkl-11.3',
]


The lines preceded by tilde's are "weak references," which tells rez, "If a package requests Qt, here's the version spec I require." And one DreamWorks specific note: we add internal versioning to the canonical versions, separated by a 'x'.  This lets us clearly denote the canonical version, while allowing us make changes that could break compatibility with deployed rez packages. e.g. remove a variant, change compiler flags, namespaces,...

I fully expect we'll hit more challenges in the future, but given the choices we've made to simplify this crazy software world, this approach seems to be working well.

Hope this helps clarify where my questions are coming from.
- Jeff
1 - 9 of 9