Re: I'd like to contribute

Matt Chambers
 


Forgot to add that, I totally accept the reasons for wanting to stick with Java.  I’ve done so myself in the past vs using Scala, Clojure, Groovy, etc.  I’m not trying to brute force it into the project or anything.  I’ve been living in a bubble where every Java programmer I know is switching projects over to Kotlin, but yeah that is a small group of people compared to the legions of Java programmers out there. I can still contribute with Java.

I checked out the roadmap, looks awesome!

-Matt

On May 24, 2019, at 8:33 PM, Matt Chambers <mvchambers@...> wrote:


On May 24, 2019, at 6:42 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:


Since this is an open list I'm attaching a PDF export of that doc, in case anyone else is interested. In the future we'll try to adopt a workflow for docs to make them easier to share publicly.

To address the points you raised:

- convert all the code to Kotlin

By "all code" I'm assuming you mean all Cuebot code, right?
Yes, just the cuebot

I'm not familiar with Kotlin. I've heard of it but don't really know anything about it other than it is mostly Java compatible. TBH I'm personally skeptical of a language switch (though I understand it's not nearly as drastic as say, rewriting Cuebot in Go). But it will incur a learning curve for basically all other developers involved, and lack of popularity compared to Java could end up dissuading some folks from contributing. Given everything else that's on the roadmap I don't think it would rank very highly for me unless there's some very large benefit.

Could you tell us more about the pros of switching? Interested to hear more about it given my unfamiliarity.

I guess the main benefit is less keyboard strokes to express the same thing.  I’d find it very hard to go back to Java.

Here is a comparison if you want to check it out.

IntelliJ will covert the Java code to Kotlin.  This is like a 95% solution, it might fail to convert a few things but it usually does very well.  The other 5% is dealing with null.   This is the part with a slight learning curve.  You have to declare if a variable can be null, and if it can you’re forced to handle the fact it could be null when referencing the variable.  

So if you do this particular  2 lines of code:

var foo : String?  = “bar”
println(foo.length)

That is a compiler error because foo could be null.  In other words, potential NPEs are compiler errors and don’t often turn into runtime errors.  There are a handful of ways to handle that compiler error, so you gotta choose the best one for the situation, or just override the compiler like this:

println(foo!!.length)

If you’re familiar with C#, it has similar concepts.


- move to spring boot 2.1,
- move docker image to something like 11-jdk-slim

Totally supportive of these. Upgrading spring boot was already on our list I believe.

Out of curiosity are there particular features of this upgraded tech you're interested in using?

Oracle wants everyone off 8 and onto 11, I guess they’ll be ending support for it kinda quickly here compared to other versions.

In Spring 2, there is micrometer integration (http://micrometer.io/) and an endpoint that feeds prometheus (or various other RRD systems), which in turn can feed Grafana.  Having good stats and visualizations is great for scheduler development.  It’s also great for monitoring API calls and JVM health.


Then after that I’d like to start replacing the scheduler.

I'm supportive of this too; it's also on the roadmap.

From the Google/Sony side, I think there's good consensus that the current scheduler, based nearly entirely in SQL procedures and complex queries, is difficult to understand, debug, and maintain. And it ultimately isn't very scalable -- you can run additional Cuebots to increase scheduling speed but this in turn puts more load on your database. HA/load balanced Postgres is possible of course but those setups are difficult to maintain for anyone who isn't a DBA.

This is true, and one of the main items I wanted to fix, not because of scalability but better scheduler decision making. Traditionally, the scheduler takes up far less DB resources than serving the UI, job launching, and artist's experimental python scripts.  Since tasks tend to run a while, and cause no scheduler load while they run, it doesn’t take long to get everything booked up.   Plus, you can’t start too many tasks at one time with NFS, it can end up locking up filer, so there should be a governor in place to stop it form going too fast.  Everyone learns that one the hard way, it was a rite of passage at SPI. Mainly I want to improve bin backing and add tiered backfill strategies, but to do that more logic needs to be moved up into the application.

Could you go into more detail on your thoughts here too? If you want you could start a doc or something like that, if it will be easier to describe than via email.

Thanks ill do that.


Thanks,
- brian

On Thu, May 23, 2019 at 4:40 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hey Brian,

Is the draft roadmap posted somewhere?

-Matt

On May 21, 2019, at 10:35 PM, Brian Cipriano via Lists.Aswf.Io <cipriano=google.com@...> wrote:

Hi Matt,

Of course we'd love to have you contribute!

Let's talk more during the TSC meeting tomorrow. Some of the items you listed are already on our draft roadmap but others I'd be interested to hear more about.

- brian



On Tue, May 21, 2019 at 5:35 AM Matt Chambers via Lists.Aswf.Io <mvchambers=me.com@...> wrote:
Hi everyone, 

I’m still very new here and I haven’t read all the materials yet,  mostly just poked around the code for some fun memories.   I’d like to do some OpenCue development, but first I’d want to change a bunch of stuff and there would be a few massive pull requests.  So, before I start I’d like to figure out if they would even be accepted.

- convert all the code to Kotlin
- move to spring boot 2.1, 
- move docker image to something like 11-jdk-slim

Then after that I’d like to start replacing the scheduler.

-Matt














<OpenCue Roadmap [Draft].pdf>


Join opencue-dev@lists.aswf.io to automatically receive all group messages.