1. Main
  2. Topics

ASWF CI Infrastructure: VFX Reference Platform Dependencies

Daniel Heckenberg
 


Hello everyone,

As we start to adopt projects into the ASWF, I would like to turn attention to the CI setup and especially the way that project dependencies are built and managed.

There have been many discussions here and elsewhere which I shall ambitiously summarize as, "We should make the ASWF CI dependencies a practical realization of the VFX Reference Platform."   This would entail establishing build configurations for the relevant packages to be built from source and storage of the results as reusable artifacts.  As all parts of this chain will be open, accessible and reproducible this offers a great opportunity for contributions from the community, and for the configurations and artifacts to be used to facilitate software development and OSS software use.

Of course, there are lots of gaps and questions with this, such as "what do we do for non-Linux builds"?  I'd like to encourage the discussion here as we embark on the effort.

Some links follow with excellent Linux Foundation documentation of the CI infrastructure.

Thanks,
Daniel Heckenberg
ASWF TAC Chair

References: 
ci-management repo here:
CI docs are available here:
     https://docs.releng.linuxfoundation.org/en/latest/

As a background into the current ASWF CI infrastructure, Thanh Ha gave a great overview back in July:

CI Infrastructure Usage Details

Thanh Ha
Jul 31  

Hi Everyone,
 
The infrastructure for ASWF is ready to go and has the following components:
 
 
I will step through each of those components one by one below:
 
GitHub Organization
 
The current organization has 4 repos: ci-management, opencolorio, openexr, and openvdb.
 
The 3 project repos are just forks of their respective upstream projects. The "ci-management" repo contains the code for the CI system, mainly all of the Jenkins build scripts. It contains 3 important directories:
 
1. jenkins-config: contains Jenkins global configuration, usually not too important for contributors and is typically maintained by the releng team
2. jjb: contains the project build scripts, usually maintained by the releng team & projects
3. packer: contains the vm image build scripts, we try to make reproducible CI so that any org can take the same packer scripts and build the same VM Images in their own environments.
 
Projects typically will be most interested in the "JJB" directory and detailed documentation on Jenkins Job Builder (JJB) / Jenkins usage are available at these 2 links:
 
 
Jenkins Job Builder is a tool that allows projects to manage their Jenkins Job configuration using YAML configuration files typically stored in a Git repo so that it can be source controlled the same way as code.
 
When a Pull Request is merged in the ci-management repo a Jenkins job will kick off and automatically update all of the Jenkins jobs in the production system with the updated build configuration.
 
Jenkins / Jenkins Sandbox
 
We provide 2 Jenkins servers for projects to utilize the first is the production Jenkins system located at https://jenkins.aswf.io/ which is where all the production builds are run (the jobs that report back to GitHub). We do not allow developers to login and manage Jenkins in the production system, this is purely managed via the ci-management repo mentioned previously.
 
The 2nd system is the Jenkins Sandbox located at https://jenkins.aswf.io/sandbox is for projects to develop new job types and test their jobs before merging the patch into the ci-management repo and making it live to GitHub.
 
Detailed documentation on how to use the Jenkins Sandbox is available here:
 
 
Nexus
 
Sonatype Nexus located at https://nexus.aswf.io/ is where all of our build artifacts / binaries are released to. Typically a "staging" job will produce a staging repository in Nexus containing a Release Candidate build. Projects can then test this repository and once approved will be released to a "release" repository in Nexus to make it widely available.
 
Jenkins jobs typically manages the staging repository creation and LF Helpdesk will manage the release (we are working on some automation here for the future).
 
Jira
 
Jira located at http://jira.aswf.io/ can be used to track bugs and new feature development. We've created Jira projects for the 4 repo projects already ci-management, opencolorio, openexr, and openvdb.
 
Jenkins Jobs
 
With that said I'd like to discuss some Jenkins jobs we've rolled out already for the projects. For every project we have 2 job types "verify" and "stage".
 
The "verify" jobs are used to verify code contributions from contributors by running test build of the project. Typically it will also run any unit tests etc... as configured by the project for the build. This job also watches GitHub for keywords so if someone leaves a comment "recheck" in a PR comment the job will automatically retrigger for that PR. This is useful for rerunning the verify tests in case of intermittent issues.
 
As an example we've configured the OpenEXR project to run 2 verify builds for incoming PRs. One job runs autoconf build and the other job runs CMake build, this allows us to test both supported build methods simultaneously for all incoming PRs.
 
Example:
 
 
The "stage" job type produces release candidate builds. The idea is that this job can run periodically and a project can decide to release any particular day's "staged release" as the final release. The job stores a commit hash of the build so that the developer can tag the appropriate commit as was built by Jenkins after approving a staged release.
 
For example:
 
 
This job produced a staging repo "opencolorio-1002" here:
 
 
This is a temporary repository containing the artifacts for the release. Once a project decides to release this repo it will be copied to the final release repository here:
 
 
Making the artifacts in that staging repo available generally to the world.
 
This was a quick walkthrough of the current CI infra. It is a lot to cover all at once but please feel free to reach out with any questions about this system.
 


Dan Bailey
 

Hi Daniel,

Thanks for sending this around, this is a great overview.

I know vendor dependencies were discussed in a past meeting, but skimming the meeting notes, I can't seem to find the outcome of those discussions. Wondering what the plan is to build against libraries for Houdini, Maya, Nuke, etc in terms of EULAs and exactly how this will be exposed in Nexus?

Regarding CI for OpenVDB specifically, I don't see a problem with sticking to using VFX Reference Platform versions of dependencies. I suggest one of the next steps will be to match the existing build matrix used with Travis CI (https://github.com/AcademySoftwareFoundation/openvdb/blob/master/.travis.yml) and the major component missing in ASWF Jenkins currently is building the Houdini plugins. 

Thanks,
Dan
Daniel Heckenberg
 


Hi Dan,

Thanks for the supportive feedback!

We have established vendor-approved mechanisms to automate download and build for the Maya SDK and for Houdini Apprentice which additionally allows for running tests.  The Foundry have indicated some support but that discussion is still in progress.

Some details, which I encourage others to correct to add to...

The Maya SDK (aka devkit) is directly downloadable. 

The Houdini download will require authentication but we plan to use Jenkins credential storage for an ASWF or OpenVDB project account.

Now that the OpenVDB project has been adopted and transferred into the ASWF we should be ready to start augmenting the CI with these components. 

Please lead the way, Dan!

Thanks,
Daniel



On Thu, Oct 25, 2018 at 7:08 AM +1100, "Dan Bailey" <danbailey@...> wrote:

Hi Daniel,

Thanks for sending this around, this is a great overview.

I know vendor dependencies were discussed in a past meeting, but skimming the meeting notes, I can't seem to find the outcome of those discussions. Wondering what the plan is to build against libraries for Houdini, Maya, Nuke, etc in terms of EULAs and exactly how this will be exposed in Nexus?

Regarding CI for OpenVDB specifically, I don't see a problem with sticking to using VFX Reference Platform versions of dependencies. I suggest one of the next steps will be to match the existing build matrix used with Travis CI (https://clicktime.symantec.com/a/1/7tI6fHsbeM79__DcOASJDJAxeOY9HSyVRbMO_nVOliw=?d=Z2KnFKxTJZI42zFyv8tNuxVINfbdxWtF1KYXk1uTmXGq5TQpUfofb9d4D1Dau_uJZXqQfItlx_E7xJkNFzXW9-yxC8vJXxyfGfW67po55JUlYT_BmgSKU1Yf8T6i6RBCGtakx_C6kkY3gELO2Ea1uOA14ioopLvHoXK62EvVszdo1qVYTvj145NNN5rXZTMmGdgxq06L7L_xH-jKsZh9lQlwVYglkArio_54CDTZeui2dxHx7C9Xaoj10Q0s4RAt5v622YA624-D1JV-WmSeDVb4gHo_DZOC5rbEkQvwI7nvMjwBRyrBKc4BqXzUFeDlgsBp266sjJ6tZchpP_lMRG5LElN-p9QRL3XAM0cWdK8pWT6zx-yh2f0No1EbhEeo-fVAGeuIF2_MzvFhKFWX3IRFyF9tvaFy-V7vK5ywYt-Z5TmfLXlbpFSC7RssBNA%3D&u=https%3A%2F%2Fgithub.com%2FAcademySoftwareFoundation%2Fopenvdb%2Fblob%2Fmaster%2F.travis.yml) and the major component missing in ASWF Jenkins currently is building the Houdini plugins. 

Thanks,
Dan
--
Daniel Heckenberg
R&D Supervisor - Graphics

T: +61 2 9383 4800 (main)
D: +61 2 8322 3123 (direct)
E: Daniel.Heckenberg@...

Building 54 / FSA #19, Fox Studios Australia, 38 Driver Avenue
Moore Park, NSW 2021
AUSTRALIA

  LinkedIn  Facebook  Twitter  Instagram

Animal Logic

www.animallogic.com

CONFIDENTIALITY AND PRIVILEGE NOTICE
This email is intended only to be read or used by the addressee. It is confidential and may contain privileged information. If you are not the intended recipient, any use, distribution, disclosure or copying of this email is strictly prohibited. Confidentiality and legal privilege attached to this communication are not waived or lost by reason of the mistaken delivery to you. If you have received this email in error, please delete it and notify us immediately by telephone or email.
Daniel Heckenberg
 


As discussed at the last TAC meeting, we'll set up a working group to progress this, and I'd like to encourage involvement from the community.

For pragmatic reasons, I'll propose the same timeslot as the TAC meeting, on alternate weeks.  That would place our first meeting on Wednesday 31 at 1pm (US/ EDT).  Please indicate your interest and ability to attend at this time in the following doodle poll:    https://doodle.com/poll/ig8zr6mtszq6r35p

Thanks,
Daniel
Daniel Heckenberg
 


Thanks for those of you answered the poll.  There's definitely enough interest to kick off the working group.

As you all probably realised, I made an error in my mental timezone juggling between Australian DST, US PDT and EDT (what's 2000 miles and 4 hours?!) so I've updated the poll to the correct time (same as the fortnightly TAC, but offset on alternate weeks) as originally planned.  A meeting invite and calendar update will come through on the TAC list shortly.

Thanks,
Daniel
Phil Parsonage
 

All,

I'd love Foundry to be involved in this - we've spend some time on this sort of problem across all three operating systems this year, and have had some good results we'd be happy to share (though our infrastructure is slightly different to what's current in the foundation it's not a million miles off). I'm afraid I'll be in transmit tomorrow evening but happy a) react to any notes, or b) write up some notes of what we're currently doing to continue the discussion is helpful?

Phil

On Mon, Oct 29, 2018 at 11:59 AM Daniel Heckenberg <daniel.heckenberg@...> wrote:

As discussed at the last TAC meeting, we'll set up a working group to progress this, and I'd like to encourage involvement from the community.

For pragmatic reasons, I'll propose the same timeslot as the TAC meeting, on alternate weeks.  That would place our first meeting on Wednesday 31 at 1pm (US/ EDT).  Please indicate your interest and ability to attend at this time in the following doodle poll:    https://doodle.com/poll/ig8zr6mtszq6r35p

Thanks,
Daniel

1 - 6 of 6