Opened 2 years ago
Last modified 18 months ago
#65799 assigned defect
mupdf @1.20.3: builds fail across-the-board: expected ';' after top level declarator
Reported by: | mascguy (Christopher Nielsen) | Owned by: | essandess (Steve Smith) |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | ports | Version: | 2.7.2 |
Keywords: | Cc: | mopihopi, gctwnl, harciga, ryandesign (Ryan Carsten Schmidt) | |
Port: | mupdf |
Description
Details:
generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c:597528:67: error: expected ';' after top level declarator "\x1c\x5e\x7c\x0a\x47\xb4\x18\x40\x49\x23\x55\x2c\x69\x08\x9b\xfb" ^ ;
Attachments (1)
Change History (24)
comment:1 follow-up: 2 Changed 2 years ago by mascguy (Christopher Nielsen)
comment:2 Changed 2 years ago by mascguy (Christopher Nielsen)
Replying to mascguy:
Per the following bug report, Clang is most likely running out of memory:
https://bugs.ghostscript.com/show_bug.cgi?id=699317
The suggested fix is to disable optimization when compiling generated resources. And given how large these files are, that approach sounds appropriate.
Sadly, disabling optimization via -O0
doesn't help: It almost looks as if we might be hitting some type of internal Clang limit.
I've also tried switching from Xcode Clang (version 12 on my 10.15 install), to MacPorts Clang 14 - but no love.
Toolchain folks, apart from tweaking the resources - which are essentially monstrous C files, with binary data encoded in strings - is there any way to increase Clang limits? Or do we need to consider other options, like compiling these via GCC?
Still digging through the project, to get a better idea of what the flow is. And perhaps there's some option to split files. But no luck so far.
Thoughts/suggestions appreciated!
comment:3 follow-up: 4 Changed 2 years ago by jmroot (Joshua Root)
Some of the build logs have clang segfaulting or reporting an internal error, and those are certainly a bug in clang. Maybe it's related to memory pressure, or maybe not. But with the latest Xcode clang on macOS 12, the error is as in the ticket description, and that doesn't look like the compiler hitting a limit, but just invalid code.
comment:4 Changed 2 years ago by essandess (Steve Smith)
Replying to jmroot:
Some of the build logs have clang segfaulting or reporting an internal error, and those are certainly a bug in clang. Maybe it's related to memory pressure, or maybe not. But with the latest Xcode clang on macOS 12, the error is as in the ticket description, and that doesn't look like the compiler hitting a limit, but just invalid code.
The C code is created at build with the script scripts/hexdump.sh
. It appears to be valid C code to me, with a terminating ;
, but does have over 1.5 million lines of font hex strings. This must be the issue.
wc -l /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_graphics_mupdf/mupdf/work/mupdf-1.20.3-source//generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c 1550676 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_graphics_mupdf/mupdf/work/mupdf-1.20.3-source//generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c
SourceHanSerif-Regular.ttc.c
:
// This is an automatically generated file. Do not edit. const unsigned char _binary_SourceHanSerif_Regular_ttc[] = "\x74\x74\x63\x66\x00\x01\x00\x00\x00\x00\x00\x04\x00\x00\x00\x1c" "\x00\x00\x01\x18\x00\x00\x02\x14\x00\x00\x03\x10\x4f\x54\x54\x4f" … "\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70" "\x03\x70\x03\x70" "" ; const unsigned int _binary_SourceHanSerif_Regular_ttc_size = sizeof(_binary_SourceHanSerif_Regular_ttc) - 1;
Changed 2 years ago by essandess (Steve Smith)
Attachment: | main.log.xz added |
---|
comment:5 Changed 2 years ago by essandess (Steve Smith)
FWIW, here's a successful build log on an x86_64
box where this works. I also have another x86_64
box where clang
segfaults. Exact same macOS and Xcode versions and builds.
I've tried to compare differences between /etc/sysctl.conf
and sudo launchctl limits
and don't see any differences there.
comment:6 Changed 2 years ago by essandess (Steve Smith)
More troubleshooting references. Using MacPorts emacs
on the C file also segfaults (!!) Using vi
works. This is only a 113 MB file, so there's no way that should happen. Related issue?
/opt/local/bin/emacs /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_graphics_mupdf/mupdf/work/mupdf-1.20.3-source//generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c Fatal error 11: Segmentation fault Backtrace: 0 emacs-28.1 0x000000010009e0e5 emacs_backtrace + 152 1 emacs-28.1 0x000000010016be2a terminate_due_to_signal + 125 2 emacs-28.1 0x000000010009f314 deliver_thread_signal + 0 3 emacs-28.1 0x000000010009f386 init_bignum + 0 4 emacs-28.1 0x000000010009df44 maybe_fatal_sig + 0 5 libsystem_platform.dylib 0x00007ff80887fdfd _sigtramp + 29 6 ??? 0x0000000000000000 0x0 + 0 7 emacs-28.1 0x0000000100005161 update_frame + 254 8 emacs-28.1 0x000000010001ab6d echo_area_display + 513 9 emacs-28.1 0x000000010001a892 message3_nolog + 519 10 emacs-28.1 0x000000010001a5b2 message3 + 393 11 emacs-28.1 0x00000001000ee109 Fmessage + 67 12 emacs-28.1 0x00000001000f6ae0 Ffuncall + 559 13 startup-bbc6ea72-9c04c94d.eln 0x000000010252762c F646973706c61792d737461727475702d6563686f2d617265612d6d657373616765_display_startup_echo_area_message_0 + 1084 Segmentation fault: 11
comment:7 Changed 2 years ago by mopihopi
Cc: | mopihopi added |
---|
comment:8 Changed 2 years ago by essandess (Steve Smith)
Still an issue on one box after recent macOS/Xcode updates:
macOS 12.6 21G115 x86_64 Xcode 14.0 14A309
comment:9 Changed 2 years ago by mascguy (Christopher Nielsen)
Worst-case, are there any libs/utilities out there, to embed an external binary blob in an object file?
Alternatively, does this project support dynamically loading such files into memory at runtime, to sidestep this? (While I haven't looked at the code yet, it's not clear why this approach is necessary...)
comment:10 Changed 2 years ago by essandess (Steve Smith)
It’s weird. It compiles successfully on a 128 GB Mac Pro, but fails on a 64 GB Mac Mini, exact same configuration otherwise, and I’ve sanity checked launchctl limits
and specific sysctl
settings.
I’m speculating on the side of a clang
bug.
comment:11 Changed 23 months ago by gctwnl
Is there anything being done about this? Might we for instance as a workaround exclude this font in the config? And not only this font but also others, as mine now fails (clang crash) on NotoSansOldPersian-Regular.otf
. Maybe exclude the Noto fonts.
comment:12 Changed 23 months ago by gctwnl
From StackExchange:
One possible solution to this issue is to tell the mupdf configure script not to compile the NotoSansOldPersian-Regular.otf font. You can do this by passing the --without-font-truetype
option to the ./configure script when building mupdf. This will prevent mupdf from trying to compile the font, which should allow the compilation process to complete successfully.
comment:13 Changed 22 months ago by jmroot (Joshua Root)
Sounds like a reasonable workaround for the short term. Has anyone opened an issue with LLVM?
comment:14 Changed 20 months ago by gctwnl
I am returning to this issue (I really need mupdf
and I can't upgrade to Ventura in case that would fix anything)
I am trying to see what I have to change, but I'm getting lost. I have no idea how to implement "You can do this by passing the --without-font-truetype
option to the ./configure script when building mupdf." if only because use_configure
is set to no
and I see only a single configure
script in the entire distribution.
comment:15 Changed 20 months ago by ryandesign (Ryan Carsten Schmidt)
Cc: | gctwnl harciga ryandesign added |
---|
Yeah I don't understand that either. This project does not have a configure script. (It includes bundled copies of other libraries, some of which have configure scripts, but that's irrelevant.) Maybe some Linux package manager has added a configure script for mupdf, and it has this option.
I was able to overcome the clang crash with 1.20.3 by disabling parallel building; try this:
sudo port install mupdf build.jobs=1
I am also preparing an update to 1.21.1 in which I don't see the clang crash even with parallel building.
comment:16 Changed 20 months ago by gctwnl
Indeed, disabling parallel builds is a workaround. Great!
comment:17 Changed 20 months ago by ryandesign (Ryan Carsten Schmidt)
comment:18 Changed 18 months ago by markmentovai (Mark Mentovai)
Given that 8c3eac6324ce (2023-02-26) disabled parallel building, is this now considered fixed?
Comment 15 also says that even without parallel building, mupdf 1.21.1 doesn’t have this problem. 9cd1afbaa7d2 (2023-04-04) updated mupdf to 1.21.1. Is it advisable to revert 8c3eac6324ce now?
comment:19 follow-up: 22 Changed 18 months ago by ryandesign (Ryan Carsten Schmidt)
Yes, my intention was to remove the parallel building restriction and mark this ticket resolved when updating to 1.21.1 but you beat me to it with the update. I don't remember what else I was still working on with my 1.21.1 update or why I hadn't committed it already... I'll go check.
comment:20 Changed 18 months ago by ryandesign (Ryan Carsten Schmidt)
Looks like I was making extensive modifications to the port. mupdf comes bundled with many libraries that we already have in MacPorts and I was trying to make the port use the existing MacPorts copies instead of its bundled ones.
comment:21 Changed 18 months ago by markmentovai (Mark Mentovai)
Is that work in progress something you can share so we can iterate collaboratively?
comment:22 Changed 18 months ago by mascguy (Christopher Nielsen)
Replying to ryandesign:
Yes, my intention was to remove the parallel building restriction and mark this ticket resolved when updating to 1.21.1 but you beat me to it with the update. I don't remember what else I was still working on with my 1.21.1 update or why I hadn't committed it already... I'll go check.
Just for giggles, I tried re-enabling parallel builds for v1.21.1, and still see the failure locally. (At least on 10.13, with Xcode 9.4.1.)
So unless I'm missing something, it looks like it's still problematic in some circumstances.
comment:23 Changed 18 months ago by gctwnl
Given the extreme size of code statements (hard coded huge data elements) and the proven instability with some compilers, disabling parallel builds may be a good default for this one.
Per the following bug report, Clang is most likely running out of memory:
https://bugs.ghostscript.com/show_bug.cgi?id=699317
The suggested fix is to disable optimization when compiling generated resources. And given how large these files are, that approach sounds appropriate.