preparing port:qt5-kde step 2 : the qt5 PortGroup(s)

[RJVB (René Bertin)]

We're in the middle of preparing the terrain for the submission of port:qt5-kde and I think another ticket is in order, given the trac[k] record of how my 2+ years worth of effort on Qt5 (and ultimately, KF5) have been trickling into MacPorts but mostly brushed aside.

I'll be keeping this (relatively) short and put most in the comments because those I can edit post-hoc if necessary.

The topic at hand is making the port:qt5 and port:qt5-kde as much interchangeable as possible as seen from dependent ports. I have already bootstrapped the discussion about this on the devel ML ("port:libressl vs port:openssl, path-style variants and prebuilt binaries").
That is the context: ultimately port:qt5 and port:qt5-kde should be exchangeable at user discretion, and ports depending on Qt5 should work regardless which of those ports is installed and without having to implement complex depspecs themselves. The situation is however quite a bit more complex because port:qt5 and port:qt5-kde both have subports which shouldn't mingle, and it is thus important that we come to an agreement how to get this right before I will consider committing or submitting port:qt5-kde. And it's crucial that we stick to that agreement.
If port:qt5-kde is to have the interest it should have, and, ultimately, if there are to be official, feature-rich KF5 ports anytime soon.

FWIW I won't claim that the solution I currently have in mind is the only or the best, but it does everything I want and it's been reliable in my testing. I *can* claim that Marko and I are the only ones who have experience using qt5-kde and KF5 (among the current CC'ees) as well as with publishing a select few Qt5-dependent ports developed against qt5-kde and apparently working fine with port:qt5 (as intended).

[RJVB (René Bertin)]

A work about the main hurdle we've been dealing with: communication with the current Qt5 maintainer. Without claiming anything as to the underlying reasons (and trying hard not to be judgemental), this is what has mostly been happening in those past 2 years or so: I keep everyone involved in the loop the issues I face and the solutions I found for them (e.g. qt4/qt5 co-installability, and now qt5/qt5-kde exchangeability) but without getting constructive feedback from the main other person involved. Then at some point there appears to be a wake-up call, and things get done .... differently. Recent examples: a boolean state variable that is intended to be set once/read-only has been implemented as an options variable (trivial, but unnecessary complexity) and the relatively simple depspec procedure I proposed that would work for both ports has been morphed into something much more complicated using a component table that I simply cannot justify for port:qt5-kde
This is what made Marko and I decide to implement port:qt5-kde in the first place.

We can't do that at the level of the PortGroups (qt5 and likely also qmake5), not if dependent ports are to be agnostic as to which Qt5 flavour is installed.

[RJVB (René Bertin)]

A word about complying with MacPorts principles: I have exposed my solution to comply with the reproducible build principle (and avoid getting the wrong binary builds from the bots) in the aforementioned ML thread. AFAIK only Mojca replied and described a scenario that is almost to the letter identical to what I proposed. In absence of contradicting feedback I will take that as sufficient indication that the solution is in fact not in violation of any crucial guidelines (neither "reproducible builds" nor the idea of having adaptive dependency declarations).

[RJVB (René Bertin)]

As said, the main topic at hand is making port:qt5 and port:qt5-kde as much exchangeable as possible. As a bonus, we also get exchangeability of port:qt5, port:qt55, a potential future port:qt56, etc. (with the difference that a priori the choice among those ports depends on the OS version and not on user discretion).

For this to work, a port depending on Qt5 that currently does

PortGroup qt5 1.0

must continue to do so, and must build and install regardless of which Qt5 port is installed. The same applies to KF5 ports: I do not intend to support using them with port:qt5, but they should build (for the most part at least) and run (to some extent). So KF5 ports will express only a preference on port:qt5-kde, not a hard dependency (see below for what that means).

A crucial point: for most Qt5-dependent, non-KDE-related ports it ultimately wouldn't really matter if they function only with port:qt5 (say, Qt Creator) but there are a few ports that are dependencies of KF5 (port:qca-qt5, the future port:qgpgme, port:opencv+qt5, port:poppler-qt5 to name a few).

The main difficulty here was the breaking up of port:qt5 into Qt5 component subports. With port:qt5-kde I split off only the huge QtWebEngine and kept the rest bundled like port:qt4-mac is; port:qt5 was later converted to provide each component in subports. This has been solved by the introduction of a qt5.depends_component procedure that ports can use to declare dependencies on those subports, and which will translate into "adaptive depspecs".

My initial solution was to have a common "header" PG that would contain the logic to decide which actual implementation PortGroup needs to be loaded, as well as some common code like qt5.depends_component. For that reason I kept that procedure relatively simple. The nice aspect as I see it is that this would allow both Marcus and me to maintain our PG files as we see fit, relying on the fact that "our" code would always be run when appropriate thanks to that header PG. And evidently port:qt5 would be the default so that in essence nothing would change for Marcus nor anyone not interested in KF5 (nothing except the name of the PG file containing the actual implementation).

In short: port:qt5 and its (renamed) PG would be used, unless

• port:qt5-kde was already installed
• no Qt5 port was installed and we were installing a port indicating a *preference* on port:qt5-kde . Using port:qt5-kde meant that the qt5-kde-1.0.tcl PG would be loaded.

Dependency preference is a novel thing, but nothing seems to forbid it in the guidelines (as long as it's done right and the idea of adaptive depspecs isn't itself in violation of some written rule). It's just an extension of principles already used by regular vs. -devel ports, and as discussed on the ML, could well be used also to formalise the user choice for LibreSSL or OpenSSL.

This solution is still being maintained in my personal macstrop repo: ​https://github.com/RJVB/macstrop/tree/master/_resources/port1.0/group

[RJVB (René Bertin)]

In my current solution, ports that prefer port:qt5-kde would do

set qt5.prefer_kde 1
PortGroup qt5 1.0

while (rare) ports that really shouldn't be used with port:qt5-kde could do

set qt5.prefer_default 1
PortGroup qt5 1.0

If port:qt5-kde is being used, and thus the qt5-kde PG is loaded, that PG does

variant qt5kde description {default variant set for port:qt5-kde* and ports that depend on them} {}
default_variants +qt5kde

in other words, ports built against port:qt5-kde get what I call an auto-default variant to label them.

It would be possible to use such a variant to control the choice of port:qt5 vs port:qt5-kde; setting it in variants.conf would not be very different from the current approach for user who are interested in Qt5 mainly because of an interest in KF5. The action would need to be documented somewhere (the question is *where*!?).
The main issue I see is that it would make reverting to port:qt5 cumbersome for users, because installed ports with +qt5kde set will require reinstalling them manually without the variant set because there is no way to unset a variant programmatically. Not even remove it from default_variants, apparently.
On the contrary, and AFAIK, an auto-default variant that is no longer defined when port:qt5-kde is no longer used should disappear out of its own.

[RJVB (René Bertin)]

The current implementation of esp. qt5.depends_component makes this approach using 3 PGs (a header and 2 implementation files) a bit less logical though we can still use a header that only transfers control to the appropriate implementation PG as a function of *some* criterium.

That criterium would preferably simply be which port is installed, using simple file exists checks rather than registry calls. I put some stress on simplicity because these are operations that are potentially going to be executed many times in the dependency graph calculations; many KF5 ports have dependencies on many KF5 frameworks that each have multiple inter-dependencies. Even on my 2.7Ghz i7 I notice the cost of that.

The alternative I see uses only 2 PortGroups, but that means each PortGroup has to include some logic that makes it transfer control to the other PortGroup. And that means that the port:qt5-kde maintainers will need to have a say over a tiny section (and only that section!) in the port:qt5 PortGroup, and vice-versa.

If that's something we can agree on (bindingly, much as I hate the word), then we can have the following implementation, equivalent to what I proposed above:

• qt5-1.0.tcl transfers control to qt5-kde-1.0.tcl if

1. port:qt5-kde is installed, or
2. no Qt5 is installed and the *variable* qt5.prefer_kde exists

• qt5-kde-1.0.tcl transfers control to qt5-1.0.tcl if

1. port:qt5 is installed, or
2. no Qt5 is installed and the variable qt5.prefer_default exists (something that should never happen, but who knows).

Error conditions are to be detected and handled in the respective PG files, *after* the check.

The preference variables should be regular Tcl variables so that ports can use the simple Tcl set command; there is no justification for using options variables here. This approach would however make at least the qt5.prefer_kde variable redundant: a simple

PortGroup qt5-kde 1.0

would already express that preference, and would still be equivalent to PortGroup qt5 1.0 if port:qt5 is already installed.

This approach does have a certain appeal: it saves 1 file, and also seems to give me a bit more certitude that my KF5 ports will use the qt5-kde PortGroup unless the user installed port:qt5 (a choice I decided to respect anyway).

[RJVB (René Bertin)]

There will undoubtedly be some differences to account for for ports that now do

PortGroup qmake5 1.0

i.e. some changes to make in qmake5-1.0.tcl (like shared and specific sections), but that's probably step 3 (or 2.5).

[RJVB (René Bertin)]

Whew, I think I've covered everything (completed my rounds as we say here).

Apologies for such a slew of messages, but I think this was an appropriate way to address the issues that are bugging me, hopefully without annoying anyone (too much).

The reason I did this is also that I think that

• the port:qt5-kde and KF5 maintainers have some right of say over (PortGroup) implementation details and choices that affect their ports (and engage themselves to respect their explicit choice not to interfere with port:qt5 functionality)
• certain others on the CC list have much more say over all implementation details than any of the Qt5 port maintainers individually (the Qt5 port maintainers' combined say hopefully carries a bit more weight ;) )

I know that Clemens, Larry and Mojca will not hesitate to express their opinion (veto...) when they don't agree somewhere but in this case it would probably be useful if those opinions come earlier rather than later. Clemens and esp. Larry have been almost suspiciously silent in the exchanges I've tried to have related to qt5-kde.

BTW, I realise I've forgotten to add @devans to the CC list (he's indirectly concerned as the port:poppler maintainer).

[neverpanic (Clemens Lang)]

Sorry, I do not have the time to read this wall of text, and I lack enough interest in KDE to care to invest.

Please remember that this is an open source project, and while that's frustrating for you, you do not have a right to my time to get your stuff reviewed.

[RJVB (René Bertin)]

I don't mind that, but in that case please don't wake up at some later point to impose invasive changes that we should have known about earlier.

[dbevans (David B. Evans)]

Replying to RJVB:

I know that Clemens, Larry and Mojca will not hesitate to express their opinion (veto...) when they don't agree somewhere but in this case it would probably be useful if those opinions come earlier rather than later. Clemens and esp. Larry have been almost suspiciously silent in the exchanges I've tried to have related to qt5-kde.

BTW, I realise I've forgotten to add @devans to the CC list (he's indirectly concerned as the port:poppler maintainer).

So without wading through all the details above can you summarize what the impact will be on poppler and it's dependents?

[danielluke (Daniel J. Luke)]

Replying to RJVB:

I don't mind that, but in that case please don't wake up at some later point to impose invasive changes that we should have known about earlier.

If you want attention/review early in the process, you should optimize for the time of the reviewer(s) (ie, make things easy for them).

[RJVB (René Bertin)]

Replying to dbevans:

So without wading through all the details above can you summarize what the impact will be on poppler and it's dependents?

The goal is that the impact is 0 at the functional level. For poppler I'm quite certain as I'm using the stock port with my own ports.

Users who replace port:qt5 with port:qt5-kde will not in principle have to reinstall all their non-KF5 Qt5 dependent ports immediately, but upgrades and new installs will be from source. That's a necessary concession to the reproducible build principle. Similarly, users who decide to install KF5 ports against port:qt5 will have to install from source. Both are the result of conscious decisions so should be acceptable.

I have tried to be concise but there's a limit to how far I can condense explanations or background without becoming ambiguous or simply copying bits of code.

[ctreleaven (Craig Treleaven)]

As in comment 11, what will be the impact on my mythtv-core.28 port? In addition to qt5-qtbase, I rely on:

• qt5-mysql-plugin
• qt5-qtscript
• qt5-qtwebkit

webkit then brings in:

• qt5-qtxmlpatterns
• qt5-qtdeclarative
• qt5-qtlocation
• qt5-qtmultimedia
• qt5-qtsensors
• qt5-qtwebchannel

I've tried to follow the discussion but little is clear to me. May I also be assured that there will be no impact?

[RJVB (René Bertin)]

If you rewrite your port to use qt5.depends_components you should only see the impact cited above, i.e. installing from source when port:qt5-kde is installed. Your dependencies would translate *automatically* to

• qt5-kde (includes everything except qtwebkit)
• qt5-kde-qtwebkit

We don't want to risk unforeseen issues by allowing to install say qt5-qtwebkit against qt5-kde, so qt5.depends_components will introduce this concept of adaptive depspecs.

[RJVB (René Bertin)]

Craig, I think you hit paydirt by having a bit more specific requirements than usual. You'd need

qt5.depends_component \
    qtscript \
    qtwebkit \
    mysql-plugin

if {[variant_isset qt5kde]} {
   register_active_variants qt5-kde mariadb55 mysql55
} else {
   register_active_variants qt5-mysql-plugin mariadb55 mysql55
}

the extra complexity is because qt5-kde-mysql-plugin is a stub port for something provided by qt5-kde itself. Even if the stub was added to the dep. list I'm not sure that requiring it to have the indicated variants would have the intended effect.

It is not yet clear if [variant_isset qt5kde] will be the preferred check to be made by dependent ports, or [tbool qt5.using_kde].

[RJVB (René Bertin)]

PortGroup control transfer draft from qmake5 to qmake5-kde

[RJVB (René Bertin)]

Corresponding draft code in qmake5-kde-1.0.tcl :

if {![tbool qt5.using_kde]} {
    ui_warn "The qmake5-kde PortGroup shouldn't be called directly"
    # transfer control if qt5.using_kde isn't set
    PortGroup                   qmake5 1.0
    return
}

# if we're here, that means port:qt5-kde is installed, qt5.using_kde is set and
# qmake5-1.0.tcl transferred control to us.

# include qt5-kde only once from here
PortGroup                       qt5-kde 1.0

[mascguy (Christopher Nielsen)]

Add keyword portgroup, to pg-related tickets