10.12 buildbot worker is offline

[mascguy (Christopher Nielsen)]

Viewing the available files at ​http://packages.macports.org/rust/, I'm expecting to see a binary for version 1.44.0_0 and macOS 10.12 (Darwin 16), but none exists. Ditto for the various mirrors, so it's not isolated to 'packages.macports.org'.

Binaries 'rust-1.44.0_0.darwin_15.x86_64.tbz2', 'rust-1.44.0_0.darwin_17.x86_64.tbz2', 'rust-1.44.0_0.darwin_18.x86_64.tbz2', and 'rust-1.44.0_0.darwin_19.x86_64.tbz2' all exist (plus their respective checksum files), but a Darwin 16 variant does not. (Though said binary does exist, for earlier Rust versions.)

Based on the various build info I've reviewed, it doesn't appear that the 10.12 compilation failed for 1.44.0_0. Nor could I find any applicable bugs, apart from #60670 - which involves building Rust from source, on macOS Yosemite.

Is something failing in the build or publish, that's preventing the binary from being copied and mirrored? Or is there something else I'm missing?

p.s. The rust source build - which is huge, and takes an eternity - also fails for me, so that's not an acceptable workaround. And would prefer not to have to dive into that, given that the binary should arguably exist.

[mascguy (Christopher Nielsen)]

Of note, reviewing the build log at ​https://ports.macports.org/port/rust/builds, it appears that there hasn't been a build for version 1.44.0.0 on 10.12.

The four other successful MacOS builds (10.11, 10.13, 10.14, 10.15) for 1.44.0.0, occurred on 6/14/2020. But 10.12 is notably absent, which would explain the lack of a binary...

[mascguy (Christopher Nielsen)]

The last build attempt for any version of Rust on 10.12, occurred on 4/26... and was successful.

[kencu (Ken)]

The 10.12 builder is currently offline <​https://build.macports.org/waterfall> and I'm not sure when the new hardware it needs is coming -- I think Ryan said an SSD was on order.

[kencu (Ken)]

By the way -- not sure how motivated you are -- but I am absolutely certain Ryan would be happy to receive a bit of Patreon or other support for the incredible amount of personal work and personal cash he invests in that buildbot infrastructure. His motivation would continue to stay much higher with some encouragement!!!!!

[ryandesign (Ryan Carsten Schmidt)]

Two of the three SSDs I bought for the buildbot hardware in 2016 have failed this year, so completely that the data on them could not be recovered. The third is well past its life expectancy and will undoubtedly fail soon too.

I bought one replacement SSD of a newer model to test it out. It's working fine. (The 10.7, 10.10 and 10.13 buildbot workers are on this disk.) I will buy two more SSDs soon.

Of the 7 buildbot virtual machines that went offline as a result of the SSD failures, 6 have already been brought back online (some on the new SSD, some temporarily on hard disks); the 10.12 builder is the last one to get back online (until the third SSD fails, at which point we'll have 3 more to deal with, but I've taken recent complete backups of them so restoring them should be simple).

[ryandesign (Ryan Carsten Schmidt)]

Replying to mascguy:

Based on the various build info I've reviewed, it doesn't appear that the 10.12 compilation failed for 1.44.0_0.

p.s. The rust source build - which is huge, and takes an eternity - also fails for me,

Well those are contradictory statements. :) If it fails to build for you, maybe it fails for everyone else too, and will fail on the buildbot once it's back online. If you attach your compressed main.log file, we could take a look at what's going on.

[ryandesign (Ryan Carsten Schmidt)]

I think we've worked out in ​the mailing list discussion that the build failure was due to #56351 so I'll make this ticket about getting the 10.12 buildbot worker back online.

[mascguy (Christopher Nielsen)]

Bless you folks for the prompt help and response!

Once I'm finally employed again (the pandemic has made things difficult), I'll happily make a monetary contribution to the project. Cheers guys!

[ryandesign (Ryan Carsten Schmidt)]

I'm bringing the 10.12 worker back online on a hard disk while I wait for the new SSD to arrive. OS is installed, time machine restored, Xcode and CLT reinstalled.

I'm waiting for #59022 to be resolved before turning the buildworker back on.

I should delete failcache entries for anything depending on numpy before turning the buildworker back on.

The ​last successful build was for stellarium in [3b70980fcfc852f3d06c3ac5f668efb84975ad47/macports-ports].

Looks like ​the first thing it'll try to build is pihpsdr in [8973b9f89f0acb28ae3efca3622ca59bd9a0bb51/macports-ports].

I may need to reschedule builds for any ports modified between those two commits.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I should delete failcache entries for anything depending on numpy before turning the buildworker back on.

Here are the ports that depend on numpy:

EGSimulation Gyoto HORTON MyPaint NanoVNASaver alps boost caffe cantera digital_rf ecCodes gds gildas gnudatalanguage gnuradio gr-ofdm grass7 hpddm inkscape inkscape-devel inkscape-gtk3-devel lal lalapps lalburst lalframe lalinference lalinspiral lalmetaio lalpulsar lalsimulation lscsoft-deps macos-fortress mathgl nds2-client-swig ola ompl opencv openexr openimageio openvdb pdal plplot plplot510 py-GridDataFormats py-MDAnalysis py-acor py-algopy py-altair py-apache-beam py-aplpy py-arviz py-astlib py-astroML py-astroML_addons py-astrolibcoords py-astroplan py-astropy py-astropy-healpix py-atpy py-barnaba py-bcolz py-beancount-import py-biggles py-biopython py-blaze py-blis py-bokeh py-bottleneck py-bqplot py-cartopy py-cftime py-chaco py-chainer py-clawpack py-colormath py-colour py-commpy py-control py-corner py-dap py-datashape py-descartes py-dice3ds py-dicom py-dijitso py-dolfin py-dynd py-eli5 py-emcee py-fabio py-fiat py-filterpy py-fipy py-gacode py-game py-gammapy py-gdal py-gensim py-geopandas py-gnuplot py-gpilab-framework py-graph-tool py-gsd py-gsl py-gwpy py-h5py py-hcluster py-hdbscan py-healpy py-htseq py-imageio py-iminuit py-imread py-instrumentkit py-isce2 py-kapteyn py-keras py-keras_applications py-keras_preprocessing py-kombine py-langid py-librosa py-llvmmath py-lmfit py-logilab-hmm py-lscsoft-glue py-magpy py-mahotas py-mapclassify py-matplotlib py-matplotlib-basemap py-mayavi py-mdp-toolkit py-mdtraj py-milk py-minecraft-overviewer py-mlpy py-molmod py-montage py-netcdf4 py-networkit py-nibabel py-nio py-nltk py-nmrglue py-numba py-numdifftools py-numexpr py-obspy py-odo py-onnx py-opengl-accelerate py-openopt py-pacal py-pandas py-parakeet py-param py-patsy py-pebl py-petsc4py py-photutils py-plumed py-polygon py-pyFAI py-pyasdf py-pycg py-pycluster py-pydicom py-pyepl py-pyfftw py-pygpu py-pygrib py-pygtk py-pymc3 py-pymca py-pyml py-pymvpa py-pynds py-pyne py-pynifti py-pyopencl py-pyorick py-pyphant py-pyqwt py-pyregion py-pysal py-pyside2 py-pysparse py-pytools py-pytorch py-pyviennacl py-pyviz_comms py-pyvttbl py-pywavelets py-qimage2ndarray py-quantities py-qutip py-rasa-nlu py-rasterio py-regions py-reproject py-resampy py-scientific py-scikit-image py-scikit-learn py-scikits-ann py-scikits-audiolab py-scikits-bootstrap py-scikits-bvp_solver py-scikits-samplerate py-scikits-talkbox py-scikits-timeseries py-scipy py-scitools py-seaborn py-sentence-transformers py-sfepy py-slepc4py py-snuggs py-sounddevice py-soundfile py-spaCy py-spacepy py-spectral py-spyder py-spyder-devel py-srsly py-statsmodels py-stfio py-sunpy py-svipc py-tables py-tensorboard py-tensorboard1 py-tensorflow py-tensorflow1 py-theano py-thinc py-tpot py-traits py-transformers py-trep py-tropycal py-tstables py-uncertainties py-usadel1 py-word2vec py-xarray py-yt pymol relax root6 rsgislib shogun shogun-devel stimfit uhd vapor vigra xraylib

Here are the ones of those that had failcache entries that I deleted:

Gyoto HORTON MyPaint alps caffe cantera gds gildas gnudatalanguage gr-ofdm grass7 inkscape-devel lalburst mathgl ola openvdb plplot510 py-mayavi py-onnx py-param py-pyviz_comms pymol root6 rsgislib shogun shogun-devel uhd vapor vigra

Somehow I thought it would be more.

I'm waiting for #59022 to be resolved before turning the buildworker back on.

It's resolved. The builder is back online and will work through the backlog over the coming weeks. I'll migrate it to the SSD when it arrives.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Here are the ones of those that had failcache entries that I deleted:

Gyoto HORTON MyPaint alps caffe cantera gds gildas gnudatalanguage gr-ofdm grass7 inkscape-devel lalburst mathgl ola openvdb plplot510 py-mayavi py-onnx py-param py-pyviz_comms pymol root6 rsgislib shogun shogun-devel uhd vapor vigra

Somehow I thought it would be more.

I forgot to account for subports. Now deleted these additional failcache entries:

boost-numpy cantera-devel gnuradio-devel gnuradio-next nds2-client-octave octave-lal octave-lalburst octave-lalframe octave-lalinference octave-lalinspiral octave-lalmetaio octave-lalpulsar octave-lalsimulation plplot-single plplot510-single py27-astropy-healpix py27-bcolz py27-biggles py27-cartopy py27-game py27-gensim py27-graph-tool py27-gyoto py27-petsc4py py27-pycg py27-pyepl py27-pyilmbase py27-pymvpa py27-pyside2 py27-qutip py27-spyder py27-tables py27-tensorflow1 py35-astropy py35-astropy-healpix py35-bcolz py35-blis py35-bottleneck py35-cartopy py35-cftime py35-fabio py35-game py35-gdal py35-graph-tool py35-gsl py35-gyoto py35-h5py py35-healpy py35-iminuit py35-imread py35-mahotas py35-matplotlib py35-matplotlib-basemap py35-mayavi py35-montage py35-numba py35-numexpr py35-opengl-accelerate py35-pandas py35-petsc4py py35-pygrib py35-pyilmbase py35-pymvpa py35-qutip py35-rasterio py35-scikit-learn py35-scipy py35-spyder py35-svipc py36-astropy py36-astropy-healpix py36-bcolz py36-blis py36-bottleneck py36-cartopy py36-cftime py36-dijitso py36-fabio py36-game py36-gdal py36-graph-tool py36-gsd py36-gsl py36-gyoto py36-h5py py36-htseq py36-iminuit py36-imread py36-mahotas py36-matplotlib py36-mayavi py36-montage py36-numba py36-numexpr py36-opengl-accelerate py36-pandas py36-param py36-petsc4py py36-pygrib py36-pyilmbase py36-pymvpa py36-pyside2 py36-pytorch py36-pywavelets py36-qutip py36-rasterio py36-scikit-learn py36-scipy py36-spaCy py36-spyder py36-spyder-devel py36-svipc py37-astropy py37-astropy-healpix py37-bcolz py37-biggles py37-blis py37-bottleneck py37-cftime py37-fabio py37-gacode py37-game py37-gdal py37-graph-tool py37-gsd py37-gsl py37-gyoto py37-h5py py37-htseq py37-iminuit py37-imread py37-mahotas py37-matplotlib py37-mayavi py37-networkit py37-numba py37-numexpr py37-opengl-accelerate py37-pandas py37-param py37-petsc4py py37-pyilmbase py37-pymvpa py37-pyside2 py37-pytorch py37-pywavelets py37-rasterio py37-scikit-learn py37-scipy py37-spaCy py37-spyder py37-spyder-devel py37-svipc py38-astropy py38-bcolz py38-bottleneck py38-cftime py38-gdal py38-gsl py38-gyoto py38-h5py py38-iminuit py38-imread py38-matplotlib py38-nds2-client py38-numba py38-numexpr py38-opengl-accelerate py38-pandas py38-param py38-petsc4py py38-pyilmbase py38-pyside2 py38-pywavelets py38-rasterio py38-scipy py38-spaCy uhd-39lts uhd-devel

After the backlog has been worked through I should probably force another build of everything that depends on numpy and/or all python modules to make sure nothing was skipped.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I'll migrate it to the SSD when it arrives.

It's on the new SSD now and also on a different Xserve whose CPUs weren't being used heavily.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

After the backlog has been worked through I should probably force another build of everything that depends on numpy

​https://build.macports.org/builders/ports-10.12_x86_64-watcher/builds/34904

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

and/or all python modules to make sure nothing was skipped.

​https://build.macports.org/builders/ports-10.12_x86_64-watcher/builds/34970

[jmroot (Joshua Root)]

Is there any other folllow-up that needs to be done here, Ryan, or can we close this?