R-tiledb @0.21.0 broken on macOS 12 and earlier when built with Clang: sh: line 1: 4733 Illegal instruction: 4

[barracuda156]

Upstream issue: ​https://github.com/TileDB-Inc/TileDB-R/issues/589

/opt/local/bin/clang++-mp-15 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -arch x86_64 -o tiledb.so RcppExports.o arrowio.o batched.o column_buffer.o deprecation.o durations.o libtiledb.o nanoarrow.o nullable.o shmem.o utilities.o -L/opt/local/lib -ltiledb -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/00LOCK-tiledb/00new/tiledb/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
sh: line 1:  4733 Illegal instruction: 4  R_TESTS= '/opt/local/Library/Frameworks/R.framework/Resources/bin/R' --no-save --no-restore --no-echo 2>&1 < '/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/.tmp/RtmpzRMoeB/fileefe9731c18'

 *** caught illegal operation ***
address 0x10723c8de, cause 'illegal opcode'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps, exclude, include.only)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return && !quietly)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE)
 9: withCallingHandlers(expr, packageStartupMessage = function(c) tryInvokeRestart("muffleMessage"))
10: suppressPackageStartupMessages(library(pkg_name, lib.loc = lib,     character.only = TRUE, logical.return = TRUE))
11: doTryCatch(return(expr), name, parentenv, handler)
12: tryCatchOne(expr, names, parentenv, handlers[[1L]])
13: tryCatchList(expr, classes, parentenv, handlers)
14: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call, nlines = 1L)        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        sm <- strsplit(conditionMessage(e), "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && isTRUE(getOption("show.error.messages"))) {        cat(msg, file = outFile)        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
15: try(suppressPackageStartupMessages(library(pkg_name, lib.loc = lib,     character.only = TRUE, logical.return = TRUE)))
16: tools:::.test_load_package("tiledb", "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/00LOCK-tiledb/00new")
An irrecoverable exception occurred. R is aborting now ...
ERROR: loading failed
* removing ‘/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/tiledb’
Command failed:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/tiledb" && /opt/local/bin/R CMD INSTALL . --library=/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library 
Exit code: 1

[barracuda156]

At the same time, strangely, on CI it built fine:

  /opt/local/bin/clang++-mp-15 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -arch x86_64 -o tiledb.so RcppExports.o arrowio.o batched.o column_buffer.o deprecation.o durations.o libtiledb.o nanoarrow.o nullable.o shmem.o utilities.o -L/opt/local/lib -ltiledb -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
  ld: warning: -undefined dynamic_lookup may not work with chained fixups
  installing to /opt/local/var/macports/build/_Users_runner_work_macports-ports_macports-ports_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/00LOCK-tiledb/00new/tiledb/libs
  ** R
  ** inst
  ** byte-compile and prepare package for lazy loading
  ** help
  *** installing help indices
  ** building package indices
  ** installing vignettes
  ** testing if installed package can be loaded from temporary location
  ** checking absolute paths in shared objects and dynamic libraries
  ** testing if installed package can be loaded from final location
  ** testing if installed package keeps a record of temporary installation path
  * DONE (tiledb)

​https://github.com/macports/macports-ports/actions/runs/6192771093/job/16813338148?pr=20432

[ryandesign (Ryan Carsten Schmidt)]

Both buildbot (where it failed) and CI (where it succeeded) are running macOS 12.6.8 and are building R-tiledb using MacPorts clang 15.0.7. However, I see that CI has Xcode 14.2 and CLT 14.2 whereas buildbot has Xcode 13.4.1 and CLT 13.4. Is it possible that some part of the build is happening using Xcode/CLT clang instead of MacPorts clang 15 and that this accounts for the difference? My recollection is that I deliberately downgraded Xcode on the macOS 12 buildbot workers to the last 13.x version because 14.x crashed when building vital ports like llvm but I cannot find the ticket right now.

I see that the crashing program is R itself. I initially wondered if perhaps R itself had been compiled with different compilers on CI vs buildbot but that shouldn't be the case, since it too is supposed to build with MacPorts clang 15 and CI should be getting binaries for dependencies from the packages server which is populated by the buildbot.

[ryandesign (Ryan Carsten Schmidt)]

I guess an obvious thing to point out is that the build is using -undefined dynamic_lookup and there is a warning from the linker stating ld: warning: -undefined dynamic_lookup may not work with chained fixups. Have you tried using --linkopt=-Wl,-no_fixup_chains to disable the chained fixups feature? Chained fixups are a new Xcode feature; you may need to use this flag only conditionally. See [b2f2ec1887f32b769364fb761d326708d69076e8/macports-ports] for an example.

[ryandesign (Ryan Carsten Schmidt)]

And are you able to reproduce this problem on your own system or only on the buildbot? Ideally, we would not want to commit a bunch of attempted fixes just to see if it fixes the build on the buildbot; ideally you would be able to reproduce on your own system and verify a fix before committing.

[barracuda156]

Replying to ryandesign:

And are you able to reproduce this problem on your own system or only on the buildbot? Ideally, we would not want to commit a bunch of attempted fixes just to see if it fixes the build on the buildbot; ideally you would be able to reproduce on your own system and verify a fix before committing.

The error does not happen with GCC, at least on the system I have access to atm (10.6.8 Rosetta). I am away from my normal hardware until about the end of month, so neither native PPC nor my testing Intel systems are available.

[barracuda156]

Replying to ryandesign:

I see that the crashing program is R itself. I initially wondered if perhaps R itself had been compiled with different compilers on CI vs buildbot but that shouldn't be the case, since it too is supposed to build with MacPorts clang 15 and CI should be getting binaries for dependencies from the packages server which is populated by the buildbot.

This is strange, since this is the only such instance, I believe. We use R 4.3.1 for quite a while, everything is working fine otherwise.

16. S. Upstream requirement for 10.14 is a result of a fallacy of equating Xcode compiler with the best available compiler, apparently. tiledb itself not only builds, but passes all tests on 10.6.8 x86_64; I think R-tiledb would pass all tests too (as it relies on tiledb).

[ryandesign (Ryan Carsten Schmidt)]

Crash log from macOS 12 x86_64 buildbot worker

[ryandesign (Ryan Carsten Schmidt)]

It does not fail for me on my Monterey system:

/opt/local/bin/clang++-mp-15 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -arch x86_64 -o tiledb.so RcppExports.o arrowio.o batched.o column_buffer.o deprecation.o durations.o libtiledb.o nanoarrow.o nullable.o shmem.o utilities.o -L/opt/local/lib -ltiledb -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-tiledb/R-tiledb/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/00LOCK-tiledb/00new/tiledb/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (tiledb)

I see that the warning about chained fixups did not occur, however since we also see the problem on buildbot workers back to macOS 10.14 where chained fixups were not a thing, they're probably not related.

% sw_vers
ProductName:	macOS
ProductVersion:	12.6.7
BuildVersion:	21G651

% xcodebuild -version
Xcode 13.2.1
Build version 13C100

% clang --version
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Volumes/BigSur/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I have attached the crash log from the Monterey x86_64 buildbot worker.