Date   
Re: lets remove some stuff!

Brian Cipriano
 

This sounds like a great idea!


On Wed, Jun 5, 2019 at 3:59 PM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:


6. ActiveMQ / JMS

Remove it for now but but bring it back as an external plugin. With spring, as long as you package the jar properly, you can load load external plugins at runtime. These plugins would have to be packaged as Spring Boot starters, which is basically just a file in src/main/resources that tells Spring where to search for beans.

Sounds interesting though I'm a little less clear on the details here. It sounds like it will simplify the Cuebot config by letting us ditch some of the "enable this feature" config values we recently added to help Cuebot work better in new environments.


ActiveMQ is hard coded as a dependency to OpenCue, however ActiveMQ is just one of the message bus systems in use in the M/E industry.  I actually think RabbitMQ is quite popular as well.

Maybe we should explore  using the local pub/sub system that is part of Google Guava to feed potentially any external message bus.  Then we just have plugins for Google PubSub, ActiveMQ, RabbitMQ, etc.  The plugins would be separate stand-alone jars available on the docker image that can be activated with an environment variable, something like that.

-Matt

Reviewing chained changes

Brian Cipriano
 

Hi Matt, all,

Big thanks for all the work you've been doing so far. Lots of good stuff in these PRs you've sent.

Listing them here for easy reference:
https://github.com/AcademySoftwareFoundation/OpenCue/pull/354

It looks like these changes are "chained", i.e. each is dependent on the one that came before it. Having done this before myself I certainly get why it's done this way :)  But it does make each change hard to review on its own, since each PR also contains the changelist from the ones before it.

So we should probably follow a process like:
- Start with the first PR, iterate on it until it's ready to go.
- Merge it to master.
- Move to the second PR, merge master into that branch, resolve conflicts as needed.
- Review second PR until it's ready to go.
- And so on.

Does that make sense? If anyone has suggestions for better ways to do this, either on the PR-creation side or the review side, I'm all ears.

Thanks,
- brian

New Repository Location

Brian Cipriano
 

Hi all,

As part of OpenCue's transfer to the Academy Software Foundation, the OpenCue Github repo can now be found at a new URL:


If you're running OpenCue directly from a code checkout, you'll want to update your checkout to use the new URL.

$ git remote set-url origin git@...:AcademySoftwareFoundation/OpenCue.git

(Note a couple things about this command:

1. The name of your Git remote might be different. Here I'm using "origin" but if you have your own fork it might be named "upstream" or something like that.

2. This assumes you're using SSH to check out from Github and not HTTP.)

Also, the ASWF requires any code contributors to have signed the OpenCue CLA. Any Pull Requests to the OpenCue repo will trigger an automatic check to make sure the author (or the author's company) has this on file. If not, the ASWF CLA tool will post a message to the Pull Request walking you through the process.

Let us know if you have any questions or run into any problems!

Thanks,
- brian

Re: lets remove some stuff!

Matt Chambers
 



6. ActiveMQ / JMS

Remove it for now but but bring it back as an external plugin. With spring, as long as you package the jar properly, you can load load external plugins at runtime. These plugins would have to be packaged as Spring Boot starters, which is basically just a file in src/main/resources that tells Spring where to search for beans.

Sounds interesting though I'm a little less clear on the details here. It sounds like it will simplify the Cuebot config by letting us ditch some of the "enable this feature" config values we recently added to help Cuebot work better in new environments.


ActiveMQ is hard coded as a dependency to OpenCue, however ActiveMQ is just one of the message bus systems in use in the M/E industry.  I actually think RabbitMQ is quite popular as well.

Maybe we should explore  using the local pub/sub system that is part of Google Guava to feed potentially any external message bus.  Then we just have plugins for Google PubSub, ActiveMQ, RabbitMQ, etc.  The plugins would be separate stand-alone jars available on the docker image that can be activated with an environment variable, something like that.

-Matt

Invitation: OpenCue TSC (F2F) @ Wed Jul 31, 2019 7pm - 8pm (EDT) (opencue-dev@lists.aswf.io)

John Mertic
 

You have been invited to the following event.

OpenCue TSC (F2F)

When
Wed Jul 31, 2019 7pm – 8pm Eastern Time - New York
Where
JW Marriott, Olympic 3 (map)
Calendar
opencue-dev@...
Who
jmertic@... - organizer
Greg Denton
Larry Gritz
greg.gewickey@...
mvchambers@...
tac@...
estrauss@...
Benjamin Dines
eolin@...
kthurston@...
james.l.jeffers@...
Brian Cipriano
opencue-dev@...

Going (opencue-dev@...)?   Yes - Maybe - No    more options »

Invitation from Google Calendar

You are receiving this courtesy email at the account opencue-dev@... because you are an attendee of this event.

To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.

Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn More.

Re: dev environment setup

Donal McMullan
 

Thanks lads - docker compose does seem like the lowest friction way to get someone started.

Re: lets remove some stuff!

Brian Cipriano
 

Thanks Matt! Good stuff. Will address points inline, and others feel free to chime in if you have some opinion on any of these.

On Fri, May 31, 2019 at 6:41 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:

I have OpenCue building and running with OpenJDK11, SpringBoot 2.1.5, Gradle 5.4.1 locally and within Docker!  Was easier than I thought it would be.

That's great! Feel free to send a Pull Request if you're ready, or you can also send a Draft Pull Request if you're just looking for general feedback before it's ready for real review.
 

I was thinking that, before we look at the scheduler/dispatcher, it might be beneficial to simplify/cleanup as much as possible.

1. Track-It stuff.

Can the track-it data source and just about everything associated with it be removed for now in favor of a future external prioritization interface? The scheduler could hit an external service to grab similar data and we’ll publish a REST interface people would implement in order to build a prioritization service.  We won’t even need to store it in tables anymore, so that stuff can go as well.

2.  Can we just delete everything related to Oracle?

Postgres is totally fine for this (better IMHO), and has no problems with a NetApp or any decent NFS implementation as backend storage.  If people need enterprise support to help out with disaster recovery on-prem, there is the Postgres based Enterprise DB.  Backups and HA are easily configurable for Postgres in Google Cloud SQL as well.

I'd be happy to ditch both of these things. We should collect Sony's opinion first.


3. All the XML files

Burn them with fire and switch to annotation based configuration.

Also agreed. It's pretty hard to find docs online that reference the XML formats -- switching to annotations should decrease barrier to entry.
 

4. Logging stuff.

Switch to LogBack which, will be a giant pull request where the import for org.apache.log4j.Logger changes to something else, though that something is API compatible.  Logback is the default for spring boot, much easier to configure stuff like log rotation, external logging aggregators, etc.

We have a somewhat related Issue in https://github.com/imageworks/OpenCue/issues/72.

In particular we're interested in providing the option to send logs to external log storage (e.g. Google Cloud Logging). The primary reason is, it would be one step closer to removing the need to have a common mount point that all components (RQD/CueGUI/etc) have access to. Setting up such a mount point can be fairly challenging in a hybrid on-prem/cloud environment.

But there are other benefits to doing this as well -- external logging services typically provide nice tools for analyzing logs, for example.

It looks like there's a Logback plugin for Google Cloud Logging at least, so I think this could work.
 

5. Java boiler plate.

I think all the getters/setters can all be removed form the service/dao beans as is right now and IoC will still be able to wire up the app.  We can also move to constructor injection which lets you finalize all those dependencies as well.

Hm, I seem to remember that the getters/setters were required under the current setup? If I'm wrong about that I don't have a problem ditching them.
 

6. ActiveMQ / JMS

Remove it for now but but bring it back as an external plugin. With spring, as long as you package the jar properly, you can load load external plugins at runtime. These plugins would have to be packaged as Spring Boot starters, which is basically just a file in src/main/resources that tells Spring where to search for beans.

Sounds interesting though I'm a little less clear on the details here. It sounds like it will simplify the Cuebot config by letting us ditch some of the "enable this feature" config values we recently added to help Cuebot work better in new environments.
 

I can submit a bunch of pulls for these 6 things plus the JDK11/SpringBoot 2.1.5 stuff which is mainly the gradle build file and Docker file.

-Matt


lets remove some stuff!

Matt Chambers
 

I have OpenCue building and running with OpenJDK11, SpringBoot 2.1.5, Gradle 5.4.1 locally and within Docker! Was easier than I thought it would be.

I was thinking that, before we look at the scheduler/dispatcher, it might be beneficial to simplify/cleanup as much as possible.

1. Track-It stuff.

Can the track-it data source and just about everything associated with it be removed for now in favor of a future external prioritization interface? The scheduler could hit an external service to grab similar data and we’ll publish a REST interface people would implement in order to build a prioritization service. We won’t even need to store it in tables anymore, so that stuff can go as well.

2. Can we just delete everything related to Oracle?

Postgres is totally fine for this (better IMHO), and has no problems with a NetApp or any decent NFS implementation as backend storage. If people need enterprise support to help out with disaster recovery on-prem, there is the Postgres based Enterprise DB. Backups and HA are easily configurable for Postgres in Google Cloud SQL as well.

3. All the XML files

Burn them with fire and switch to annotation based configuration.

4. Logging stuff.

Switch to LogBack which, will be a giant pull request where the import for org.apache.log4j.Logger changes to something else, though that something is API compatible. Logback is the default for spring boot, much easier to configure stuff like log rotation, external logging aggregators, etc.

5. Java boiler plate.

I think all the getters/setters can all be removed form the service/dao beans as is right now and IoC will still be able to wire up the app. We can also move to constructor injection which lets you finalize all those dependencies as well.

6. ActiveMQ / JMS

Remove it for now but but bring it back as an external plugin. With spring, as long as you package the jar properly, you can load load external plugins at runtime. These plugins would have to be packaged as Spring Boot starters, which is basically just a file in src/main/resources that tells Spring where to search for beans.

I can submit a bunch of pulls for these 6 things plus the JDK11/SpringBoot 2.1.5 stuff which is mainly the gradle build file and Docker file.

-Matt

Re: dev environment setup

Matt Chambers
 

Hi Donal!

I try to run everything through docker.

We could make a docker-compose file that runs Postgres, a Cuebot, and one or two RQDs.  

Something like this (I left a lot out) at the root of the project.  With this file, you wouldn’t even need to build anything to get a simple queue running.  It would pull the latest dev images from docker-hub, or use your local images.

postgres:
    image: “postgres:9.6.9”
    ports:
        - “5432:5432”
    environment:
        POSTGRES_PASSWORD: cuebot
        POSTGRES_USER: cuebot
        POSTGRES_DB: cuebot
        PG_DATA: /var/lib/postgresql/data/pgdata

cuebot:
   links:
      - “postgres”
   image: cuebot
   build:
       context: cuebot
       cache_from:
            - OpenCue/cuebot:latest 
            - cuebot
   ports:
       - 8080:8080
   volumes:
       - “/cuebot/target:/cuebot/lib”

rqd1:
   build:
       context: rqd

In this case, you would do:

docker-compose build
docker-compose up

This way if someone just want to do UI work, they can quickly spin up a cluster.

-Matt


On May 29, 2019, at 6:54 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Donal,

Thanks for sharing this!

How do you set up your development environments?

My personal setup depends on what I'm working on.

I run a local Postgres instance, installed via Homebrew.

If I'm doing work on Cuebot, I've got IntelliJ set up with a Run Configuration to run CuebotApplication, with some Program arguments to point it to my local postgres instance.

If I'm working on any of the Python components, my process looks like:
- Run Cuebot either a) via IntelliJ as above or b) (more typically) via Docker
- Activate whichever virtualenv I plan to use
- Execute the Python code directly from my local checkout, using the CUEBOT_HOSTS env var to point it to my local Cuebot instance.
- I use PyCharm locally and will sometimes set up Run configurations in there to execute the module in question, which lets me use things like PyCharm's built-in debugger.

Happy to go into more detail on any of this if desired.

- brian


On Sun, May 26, 2019 at 6:46 AM Donal McMullan <donal.mcmullan@...> wrote:
I added some scripts to help myself get started with development on the repo:

Specifically I wanted some bootstraps to create a development environment for new users, and then to launch the applications directly from that environment without any intermediate step (like running installers, etc).

How do you set up your development environments?

Thanks

DJM



Re: dev environment setup

Brian Cipriano
 

Hi Donal,

Thanks for sharing this!

How do you set up your development environments?

My personal setup depends on what I'm working on.

I run a local Postgres instance, installed via Homebrew.

If I'm doing work on Cuebot, I've got IntelliJ set up with a Run Configuration to run CuebotApplication, with some Program arguments to point it to my local postgres instance.

If I'm working on any of the Python components, my process looks like:
- Run Cuebot either a) via IntelliJ as above or b) (more typically) via Docker
- Activate whichever virtualenv I plan to use
- Execute the Python code directly from my local checkout, using the CUEBOT_HOSTS env var to point it to my local Cuebot instance.
- I use PyCharm locally and will sometimes set up Run configurations in there to execute the module in question, which lets me use things like PyCharm's built-in debugger.

Happy to go into more detail on any of this if desired.

- brian


On Sun, May 26, 2019 at 6:46 AM Donal McMullan <donal.mcmullan@...> wrote:
I added some scripts to help myself get started with development on the repo:

Specifically I wanted some bootstraps to create a development environment for new users, and then to launch the applications directly from that environment without any intermediate step (like running installers, etc).

How do you set up your development environments?

Thanks

DJM

Re: I'd like to contribute

Matt Chambers
 


Forgot to add that, I totally accept the reasons for wanting to stick with Java.  I’ve done so myself in the past vs using Scala, Clojure, Groovy, etc.  I’m not trying to brute force it into the project or anything.  I’ve been living in a bubble where every Java programmer I know is switching projects over to Kotlin, but yeah that is a small group of people compared to the legions of Java programmers out there. I can still contribute with Java.

I checked out the roadmap, looks awesome!

-Matt

On May 24, 2019, at 8:33 PM, Matt Chambers <mvchambers@...> wrote:


On May 24, 2019, at 6:42 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:


Since this is an open list I'm attaching a PDF export of that doc, in case anyone else is interested. In the future we'll try to adopt a workflow for docs to make them easier to share publicly.

To address the points you raised:

- convert all the code to Kotlin

By "all code" I'm assuming you mean all Cuebot code, right?
Yes, just the cuebot

I'm not familiar with Kotlin. I've heard of it but don't really know anything about it other than it is mostly Java compatible. TBH I'm personally skeptical of a language switch (though I understand it's not nearly as drastic as say, rewriting Cuebot in Go). But it will incur a learning curve for basically all other developers involved, and lack of popularity compared to Java could end up dissuading some folks from contributing. Given everything else that's on the roadmap I don't think it would rank very highly for me unless there's some very large benefit.

Could you tell us more about the pros of switching? Interested to hear more about it given my unfamiliarity.

I guess the main benefit is less keyboard strokes to express the same thing.  I’d find it very hard to go back to Java.

Here is a comparison if you want to check it out.

IntelliJ will covert the Java code to Kotlin.  This is like a 95% solution, it might fail to convert a few things but it usually does very well.  The other 5% is dealing with null.   This is the part with a slight learning curve.  You have to declare if a variable can be null, and if it can you’re forced to handle the fact it could be null when referencing the variable.  

So if you do this particular  2 lines of code:

var foo : String?  = “bar”
println(foo.length)

That is a compiler error because foo could be null.  In other words, potential NPEs are compiler errors and don’t often turn into runtime errors.  There are a handful of ways to handle that compiler error, so you gotta choose the best one for the situation, or just override the compiler like this:

println(foo!!.length)

If you’re familiar with C#, it has similar concepts.


- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Totally supportive of these. Upgrading spring boot was already on our list I believe.

Out of curiosity are there particular features of this upgraded tech you're interested in using?

Oracle wants everyone off 8 and onto 11, I guess they’ll be ending support for it kinda quickly here compared to other versions.

In Spring 2, there is micrometer integration (http://micrometer.io/) and an endpoint that feeds prometheus (or various other RRD systems), which in turn can feed Grafana.  Having good stats and visualizations is great for scheduler development.  It’s also great for monitoring API calls and JVM health.


Then after that I’d like to start replacing the scheduler.

I'm supportive of this too; it's also on the roadmap.

From the Google/Sony side, I think there's good consensus that the current scheduler, based nearly entirely in SQL procedures and complex queries, is difficult to understand, debug, and maintain. And it ultimately isn't very scalable -- you can run additional Cuebots to increase scheduling speed but this in turn puts more load on your database. HA/load balanced Postgres is possible of course but those setups are difficult to maintain for anyone who isn't a DBA.

This is true, and one of the main items I wanted to fix, not because of scalability but better scheduler decision making. Traditionally, the scheduler takes up far less DB resources than serving the UI, job launching, and artist's experimental python scripts.  Since tasks tend to run a while, and cause no scheduler load while they run, it doesn’t take long to get everything booked up.   Plus, you can’t start too many tasks at one time with NFS, it can end up locking up filer, so there should be a governor in place to stop it form going too fast.  Everyone learns that one the hard way, it was a rite of passage at SPI. Mainly I want to improve bin backing and add tiered backfill strategies, but to do that more logic needs to be moved up into the application.

Could you go into more detail on your thoughts here too? If you want you could start a doc or something like that, if it will be easier to describe than via email.

Thanks ill do that.


Thanks,
- brian

On Thu, May 23, 2019 at 4:40 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hey Brian,

Is the draft roadmap posted somewhere?

-Matt

On May 21, 2019, at 10:35 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone, 

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1, 
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt














<OpenCue Roadmap [Draft].pdf>


dev environment setup

Donal McMullan
 

I added some scripts to help myself get started with development on the repo:

Specifically I wanted some bootstraps to create a development environment for new users, and then to launch the applications directly from that environment without any intermediate step (like running installers, etc).

How do you set up your development environments?

Thanks

DJM

Re: I'd like to contribute

Matt Chambers
 


On May 24, 2019, at 6:42 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:


Since this is an open list I'm attaching a PDF export of that doc, in case anyone else is interested. In the future we'll try to adopt a workflow for docs to make them easier to share publicly.

To address the points you raised:

- convert all the code to Kotlin

By "all code" I'm assuming you mean all Cuebot code, right?
Yes, just the cuebot

I'm not familiar with Kotlin. I've heard of it but don't really know anything about it other than it is mostly Java compatible. TBH I'm personally skeptical of a language switch (though I understand it's not nearly as drastic as say, rewriting Cuebot in Go). But it will incur a learning curve for basically all other developers involved, and lack of popularity compared to Java could end up dissuading some folks from contributing. Given everything else that's on the roadmap I don't think it would rank very highly for me unless there's some very large benefit.

Could you tell us more about the pros of switching? Interested to hear more about it given my unfamiliarity.

I guess the main benefit is less keyboard strokes to express the same thing.  I’d find it very hard to go back to Java.

Here is a comparison if you want to check it out.

IntelliJ will covert the Java code to Kotlin.  This is like a 95% solution, it might fail to convert a few things but it usually does very well.  The other 5% is dealing with null.   This is the part with a slight learning curve.  You have to declare if a variable can be null, and if it can you’re forced to handle the fact it could be null when referencing the variable.  

So if you do this particular  2 lines of code:

var foo : String?  = “bar”
println(foo.length)

That is a compiler error because foo could be null.  In other words, potential NPEs are compiler errors and don’t often turn into runtime errors.  There are a handful of ways to handle that compiler error, so you gotta choose the best one for the situation, or just override the compiler like this:

println(foo!!.length)

If you’re familiar with C#, it has similar concepts.


- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Totally supportive of these. Upgrading spring boot was already on our list I believe.

Out of curiosity are there particular features of this upgraded tech you're interested in using?

Oracle wants everyone off 8 and onto 11, I guess they’ll be ending support for it kinda quickly here compared to other versions.

In Spring 2, there is micrometer integration (http://micrometer.io/) and an endpoint that feeds prometheus (or various other RRD systems), which in turn can feed Grafana.  Having good stats and visualizations is great for scheduler development.  It’s also great for monitoring API calls and JVM health.


Then after that I’d like to start replacing the scheduler.

I'm supportive of this too; it's also on the roadmap.

From the Google/Sony side, I think there's good consensus that the current scheduler, based nearly entirely in SQL procedures and complex queries, is difficult to understand, debug, and maintain. And it ultimately isn't very scalable -- you can run additional Cuebots to increase scheduling speed but this in turn puts more load on your database. HA/load balanced Postgres is possible of course but those setups are difficult to maintain for anyone who isn't a DBA.

This is true, and one of the main items I wanted to fix, not because of scalability but better scheduler decision making. Traditionally, the scheduler takes up far less DB resources than serving the UI, job launching, and artist's experimental python scripts.  Since tasks tend to run a while, and cause no scheduler load while they run, it doesn’t take long to get everything booked up.   Plus, you can’t start too many tasks at one time with NFS, it can end up locking up filer, so there should be a governor in place to stop it form going too fast.  Everyone learns that one the hard way, it was a rite of passage at SPI. Mainly I want to improve bin backing and add tiered backfill strategies, but to do that more logic needs to be moved up into the application.

Could you go into more detail on your thoughts here too? If you want you could start a doc or something like that, if it will be easier to describe than via email.

Thanks ill do that.


Thanks,
- brian

On Thu, May 23, 2019 at 4:40 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hey Brian,

Is the draft roadmap posted somewhere?

-Matt

On May 21, 2019, at 10:35 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone, 

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1, 
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt














<OpenCue Roadmap [Draft].pdf>

Re: I'd like to contribute

Brian Cipriano
 


Since this is an open list I'm attaching a PDF export of that doc, in case anyone else is interested. In the future we'll try to adopt a workflow for docs to make them easier to share publicly.

To address the points you raised:

- convert all the code to Kotlin

By "all code" I'm assuming you mean all Cuebot code, right?

I'm not familiar with Kotlin. I've heard of it but don't really know anything about it other than it is mostly Java compatible. TBH I'm personally skeptical of a language switch (though I understand it's not nearly as drastic as say, rewriting Cuebot in Go). But it will incur a learning curve for basically all other developers involved, and lack of popularity compared to Java could end up dissuading some folks from contributing. Given everything else that's on the roadmap I don't think it would rank very highly for me unless there's some very large benefit.

Could you tell us more about the pros of switching? Interested to hear more about it given my unfamiliarity.

- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Totally supportive of these. Upgrading spring boot was already on our list I believe.

Out of curiosity are there particular features of this upgraded tech you're interested in using?

Then after that I’d like to start replacing the scheduler.

I'm supportive of this too; it's also on the roadmap.

From the Google/Sony side, I think there's good consensus that the current scheduler, based nearly entirely in SQL procedures and complex queries, is difficult to understand, debug, and maintain. And it ultimately isn't very scalable -- you can run additional Cuebots to increase scheduling speed but this in turn puts more load on your database. HA/load balanced Postgres is possible of course but those setups are difficult to maintain for anyone who isn't a DBA.

Could you go into more detail on your thoughts here too? If you want you could start a doc or something like that, if it will be easier to describe than via email.

Thanks,
- brian

On Thu, May 23, 2019 at 4:40 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hey Brian,

Is the draft roadmap posted somewhere?

-Matt

On May 21, 2019, at 10:35 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone,

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt












[Action Needed] New OpenCue mailing lists

Brian Cipriano <cipr...@...>
 

Hi all,

As recently announced, OpenCue has been approved to join the Academy Software Foundation (ASWF) as a hosted project. OpenCue's new technical steering committee (TSC) is working with the Linux Foundation to transition the project's infrastructure to the ASWF.

As part of this transfer, we're going to move our email lists to the Foundation's list system, to match what other projects (OpenVDB, OpenColorIO) have done.

All subscribers will be required to sign up at lists.aswf.io to continue receiving messages. The existing Google group will remain online for the duration of the migration, and we'll send notice to the current list one week prior to its shutdown.

Subscribe to the new opencue-dev list here: https://lists.aswf.io/g/opencue-dev

Thanks all for your continued interest in OpenCue!

Brian Cipriano
OpenCue TSC Chairperson

Re: I'd like to contribute

Matt Chambers
 

Hey Brian,

Is the draft roadmap posted somewhere?

-Matt

On May 21, 2019, at 10:35 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone,

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt












Re: I'd like to contribute

Brian Cipriano
 

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone,

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt











I'd like to contribute

Matt Chambers
 

Hi everyone,

I’m still very new here and I haven’t read all the materials yet, mostly just poked around the code for some fun memories. I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests. So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt

Re: py3

Brian Cipriano <cipr...@...>
 

Yeah, to confirm all of the code we've migrated so far has maintained both python 2 and 3 compatibility. Tests are always run against both python 2 and 3 environments.

- brian

From: Donal McMullan <donal....@...>
Date: Tue, May 14, 2019 at 3:57 AM
To: opencue-dev

Hey Pierre - just as you suggest, this effort will not break python-2 compatibility.

Thanks

DJM

On Tuesday, 14 May 2019 11:23:32 UTC+1, Pierre wrote:
Apologies for my lack of knowledge regarding the OpenCue project architecture, but you might want to maintain compatibility with python 2.7 if the DCC (maya, nuke, ...) plugins are referencing RQD code.
So in that case, a "straddling mode" (a subset of features from py2.7 and py3.7) should be considered. Packages such as six or futurize would still be used.

Pierre

On Tuesday, May 14, 2019 at 1:40:19 AM UTC+2, Brian Cipriano wrote:
Yes, agreed. This is a blocker for getting full Windows support so I'm sure it would be much appreciated by lots of folks :)

I've been using Futurize so far to upgrade our code with pretty good results.


Happy to help either here or on the Github issue if I can.

- brian


From: 'Greg Denton' via opencue-dev <op...@...>
Date: Mon, May 13, 2019 at 4:34 PM
To: Donal McMullan
Cc: opencue-dev

Hi Donal,

It'd be great to have your help on getting RQD python3 ready. I've gone ahead and created https://github.com/imageworks/OpenCue/issues/315 for this work. 
Brian will need to add your github id to the project before we can assign you the issue, but in the meantime feel free to comment on the issue if you'd like to take it on.
I see we've received your CLA, thank you for that!

If you have any questions that come up, let me know.

Thanks,
Greg

From: Donal McMullan <do...@...>
Date: Mon, May 13, 2019 at 10:30 AM
To: opencue-dev

I thought I might have a look at updating rqd to address some of the python3 syntax incompatibility. I don't see any related issues in the tracker - is that a reasonable section of code to work on just now?

Thanks

DJM

--
You received this message because you are subscribed to the Google Groups "opencue-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to op...@....
To post to this group, send email to op...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/opencue-dev/fa99f43e-c496-49c6-8091-92da4ed69cca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "opencue-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to op...@....
To post to this group, send email to op...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/opencue-dev/CAKW9UCBHPryb_zitaYHBRGiQhB2-4%2BZhf6hDP%2Be%3D3exmPf_NUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "opencue-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openc...@....
To post to this group, send email to openc...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/opencue-dev/b0950c19-b31d-441e-92f0-6cdad84dd857%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: py3

Donal McMullan <donal....@...>
 

Hey Pierre - just as you suggest, this effort will not break python-2 compatibility.

Thanks

DJM


On Tuesday, 14 May 2019 11:23:32 UTC+1, Pierre wrote:
Apologies for my lack of knowledge regarding the OpenCue project architecture, but you might want to maintain compatibility with python 2.7 if the DCC (maya, nuke, ...) plugins are referencing RQD code.
So in that case, a "straddling mode" (a subset of features from py2.7 and py3.7) should be considered. Packages such as six or futurize would still be used.

Pierre

On Tuesday, May 14, 2019 at 1:40:19 AM UTC+2, Brian Cipriano wrote:
Yes, agreed. This is a blocker for getting full Windows support so I'm sure it would be much appreciated by lots of folks :)

I've been using Futurize so far to upgrade our code with pretty good results.


Happy to help either here or on the Github issue if I can.

- brian


From: 'Greg Denton' via opencue-dev <op...@...>
Date: Mon, May 13, 2019 at 4:34 PM
To: Donal McMullan
Cc: opencue-dev

Hi Donal,

It'd be great to have your help on getting RQD python3 ready. I've gone ahead and created https://github.com/imageworks/OpenCue/issues/315 for this work. 
Brian will need to add your github id to the project before we can assign you the issue, but in the meantime feel free to comment on the issue if you'd like to take it on.
I see we've received your CLA, thank you for that!

If you have any questions that come up, let me know.

Thanks,
Greg

From: Donal McMullan <do...@...>
Date: Mon, May 13, 2019 at 10:30 AM
To: opencue-dev

I thought I might have a look at updating rqd to address some of the python3 syntax incompatibility. I don't see any related issues in the tracker - is that a reasonable section of code to work on just now?

Thanks

DJM

--
You received this message because you are subscribed to the Google Groups "opencue-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to op...@....
To post to this group, send email to op...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/opencue-dev/fa99f43e-c496-49c6-8091-92da4ed69cca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "opencue-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to op...@....
To post to this group, send email to op...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/opencue-dev/CAKW9UCBHPryb_zitaYHBRGiQhB2-4%2BZhf6hDP%2Be%3D3exmPf_NUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.