emacs-mac-app-devel: Unable to fetch from repo: fatal: the remote end hung up unexpectedly

[George9000]

There's a 404 when attempting to fetch from the repository:

:msg:archivefetch --->  Fetching archive for emacs-mac-app-devel
:debug:archivefetch Executing org.macports.archivefetch (emacs-mac-app-devel)
:debug:archivefetch euid/egid changed to: 0/0
:debug:archivefetch chowned /opt/local/var/macports/incoming to macports
:debug:archivefetch euid/egid changed to: 502/501
:info:archivefetch --->  emacs-mac-app-devel-20240325_0+imagemagick+metal+nativecomp+rsvg+treesitter.darwin_23.arm64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
:msg:archivefetch --->  Attempting to fetch emacs-mac-app-devel-20240325_0+imagemagick+metal+nativecomp+rsvg+treesitter.darwin_23.arm64.tbz2 from https://packages.macports.org/emacs-mac-app-devel
:debug:archivefetch Fetching archive failed: The requested URL returned error: 404
:msg:archivefetch --->  Attempting to fetch emacs-mac-app-devel-20240325_0+imagemagick+metal+nativecomp+rsvg+treesitter.darwin_23.arm64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/emacs-mac-app-devel
:debug:archivefetch Fetching archive failed: The requested URL returned error: 404
:msg:archivefetch --->  Attempting to fetch emacs-mac-app-devel-20240325_0+imagemagick+metal+nativecomp+rsvg+treesitter.darwin_23.arm64.tbz2 from http://mirror.fcix.net/macports/packages/emacs-mac-app-devel
:debug:archivefetch Fetching archive failed: The requested URL returned error: 404
:debug:archivefetch Privilege de-escalation not attempted as not running as root.
:debug:fetch fetch phase started at Fri Apr 12 12:28:13 EDT 2024
:notice:fetch --->  Fetching distfiles for emacs-mac-app-devel
:debug:fetch Executing org.macports.fetch (emacs-mac-app-devel)
:debug:fetch Executing: /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
:debug:fetch system: /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
:info:fetch Cloning into '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0'...
:info:fetch fatal: the remote end hung up unexpectedly
:info:fetch Command failed: /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
:info:fetch Exit code: 128
:error:fetch Failed to fetch emacs-mac-app-devel: Git clone failed
:debug:fetch Error code: NONE
:debug:fetch Backtrace: Git clone failed
:debug:fetch     while executing
:debug:fetch "gitfetch"
:debug:fetch     (procedure "portfetch::fetch_main" line 14)
:debug:fetch     invoked from within
:debug:fetch "$procedure $targetname"
:error:fetch See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/main.log for details.

[ryandesign (Ryan Carsten Schmidt)]

There's a 404 when attempting to fetch from the repository:

No, there isn't. There's a 404 when trying to fetch precompiled archives. This is not a bug. (See #68869.) Since precompiled archives were not available, MacPorts tried to fetch the source, in this case from a git repository. It failed with this error (which is not a 404):

fatal: the remote end hung up unexpectedly

I had noticed this too on my macOS 12 machine but hoped that it was just a transient problem, but I guess it isn't. I had worked around it temporarily by removing --shallow-since=2024-03-25T00:00:00 from the Portfile. I guess maybe we have to remove that part permanently if bitbucket's git servers no longer accept this option. That results in fetching the entire history of the repository, however, which is large and takes a lot of time and disk space. Maybe there is a different flag we can use that would similarly limit the fetch to only what we need. Or maybe there is a way that we can fetch a distfile instead of fetching from the repo.

[George9000]

Ok. Thanks. Following your suggestion, I used vim to remove that chunk from the Portfile.

sudo vim /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/aqua/emacs-mac-app/Portfile

on line 98 of the Portfile, remove

--shallow-since=${date}T00:00:00

The port then built without issue.

[amake (Aaron Madlon-Kay)]

I'm not seeing this. On my machine:

% sudo port -d fetch emacs-mac-app-devel

...

--->  Fetching distfiles for emacs-mac-app-devel
DEBUG: Executing org.macports.fetch (emacs-mac-app-devel)
DEBUG: Executing: /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
DEBUG: system: /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
Cloning into '/opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0'...
remote: Enumerating objects: 5236, done.        
remote: Counting objects: 100% (5236/5236), done.        
remote: Compressing objects: 100% (4815/4815), done.        
remote: Total 5236 (delta 564), reused 1657 (delta 257), pack-reused 0        
Receiving objects: 100% (5236/5236), 48.96 MiB | 6.28 MiB/s, done.
Resolving deltas: 100% (564/564), done.
Updating files: 100% (4954/4954), done.
DEBUG: Executing GIT_DIR=/opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0/.git GIT_WORK_TREE=/opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 /opt/local/bin/git checkout -q 0386c590892066c4b58388848c2c93c61a505b31 2>&1
DEBUG: system: GIT_DIR=/opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0/.git GIT_WORK_TREE=/opt/local/var/macports/build/_Users_amake_Code_MacPorts_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 /opt/local/bin/git checkout -q 0386c590892066c4b58388848c2c93c61a505b31 2>&1
DEBUG: Checking time since last reclaim run

[ryandesign (Ryan Carsten Schmidt)]

Some more detail after enabling verbose mode like this:

/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/aqua/emacs-mac-app/Portfile

@@ -101 +101 @@
     # --shallow-since needs a newer version of git than on some older systems,
     # so use MacPorts version
     depends_fetch-append port:git
-    git.cmd         ${prefix}/bin/git
+    git.cmd         GIT_CURL_VERBOSE=1 ${prefix}/bin/git
 
     conflicts       emacs-mac-app
 

--->  Fetching distfiles for emacs-mac-app-devel
DEBUG: Executing org.macports.fetch (emacs-mac-app-devel)
DEBUG: Executing: GIT_CURL_VERBOSE=1 /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
DEBUG: system: GIT_CURL_VERBOSE=1 /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
Cloning into '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0'...
22:09:31.856345 http.c:843              == Info: Uses proxy env variable NO_PROXY == '*.local,169.254/16'
22:09:31.856784 http.c:843              == Info: Couldn't find host bitbucket.org in the .netrc file; using defaults
22:09:31.884007 http.c:843              == Info: Host bitbucket.org:443 was resolved.
22:09:31.884021 http.c:843              == Info: IPv6: (none)
22:09:31.884025 http.c:843              == Info: IPv4: 104.192.141.1
22:09:31.884083 http.c:843              == Info:   Trying 104.192.141.1:443...
22:09:31.902648 http.c:843              == Info: Connected to bitbucket.org (104.192.141.1) port 443
22:09:31.919530 http.c:843              == Info: found 147 certificates in /opt/local/share/curl/curl-ca-bundle.crt
22:09:31.919617 http.c:843              == Info: GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
22:09:31.919657 http.c:843              == Info: ALPN: curl offers h2,http/1.1
22:09:31.989797 http.c:843              == Info: SSL connection using TLS1.3 / ECDHE_RSA_AES_128_GCM_SHA256
22:09:31.991819 http.c:843              == Info:   server certificate verification OK
22:09:31.991832 http.c:843              == Info:   server certificate status verification SKIPPED
22:09:31.991978 http.c:843              == Info:   common name: bitbucket.org (matched)
22:09:31.991988 http.c:843              == Info:   server certificate expiration date OK
22:09:31.991994 http.c:843              == Info:   server certificate activation date OK
22:09:31.992007 http.c:843              == Info:   certificate public key: EC/ECDSA
22:09:31.992013 http.c:843              == Info:   certificate version: #3
22:09:31.992070 http.c:843              == Info:   subject: jurisdictionOfIncorporationCountryName=US,jurisdictionOfIncorporationStateOrProvinceName=Delaware,businessCategory=Private Organization,serialNumber=3928449,C=US,ST=California,L=San Francisco,O=Atlassian US\, Inc.,CN=bitbucket.org
22:09:31.992451 http.c:843              == Info:   start date: Wed, 13 Mar 2024 00:00:00 GMT
22:09:31.992472 http.c:843              == Info:   expire date: Sun, 13 Apr 2025 23:59:59 GMT
22:09:31.992529 http.c:843              == Info:   issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA
22:09:31.992571 http.c:843              == Info: ALPN: server accepted h2
22:09:31.992718 http.c:843              == Info: using HTTP/2
22:09:31.992789 http.c:843              == Info: [HTTP/2] [1] OPENED stream for https://bitbucket.org/mituharu/emacs-mac.git/info/refs?service=git-upload-pack
22:09:31.992796 http.c:843              == Info: [HTTP/2] [1] [:method: GET]
22:09:31.992800 http.c:843              == Info: [HTTP/2] [1] [:scheme: https]
22:09:31.992804 http.c:843              == Info: [HTTP/2] [1] [:authority: bitbucket.org]
22:09:31.992810 http.c:843              == Info: [HTTP/2] [1] [:path: /mituharu/emacs-mac.git/info/refs?service=git-upload-pack]
22:09:31.992814 http.c:843              == Info: [HTTP/2] [1] [user-agent: git/2.44.0]
22:09:31.992818 http.c:843              == Info: [HTTP/2] [1] [accept: */*]
22:09:31.992822 http.c:843              == Info: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd]
22:09:31.992827 http.c:843              == Info: [HTTP/2] [1] [accept-language: en-US, *;q=0.9]
22:09:31.992831 http.c:843              == Info: [HTTP/2] [1] [pragma: no-cache]
22:09:31.992834 http.c:843              == Info: [HTTP/2] [1] [git-protocol: version=2]
22:09:31.994066 http.c:790              => Send header, 0000000248 bytes (0x000000f8)
22:09:31.994082 http.c:802              => Send header: GET /mituharu/emacs-mac.git/info/refs?service=git-upload-pack HTTP/2
22:09:31.994087 http.c:802              => Send header: Host: bitbucket.org
22:09:31.994090 http.c:802              => Send header: User-Agent: git/2.44.0
22:09:31.994093 http.c:802              => Send header: Accept: */*
22:09:31.994096 http.c:802              => Send header: Accept-Encoding: deflate, gzip, br, zstd
22:09:31.994099 http.c:802              => Send header: Accept-Language: en-US, *;q=0.9
22:09:31.994102 http.c:802              => Send header: Pragma: no-cache
22:09:31.994106 http.c:802              => Send header: Git-Protocol: version=2
22:09:31.994108 http.c:802              => Send header:
22:09:31.994123 http.c:843              == Info: Request completely sent off
22:09:32.095939 http.c:790              <= Recv header, 0000000013 bytes (0x0000000d)
22:09:32.095968 http.c:802              <= Recv header: HTTP/2 200
22:09:32.095995 http.c:790              <= Recv header, 0000000015 bytes (0x0000000f)
22:09:32.096003 http.c:802              <= Recv header: server: envoy
22:09:32.096014 http.c:790              <= Recv header, 0000000025 bytes (0x00000019)
22:09:32.096019 http.c:802              <= Recv header: cache-control: no-cache
22:09:32.096029 http.c:790              <= Recv header, 0000000059 bytes (0x0000003b)
22:09:32.096033 http.c:802              <= Recv header: content-type: application/x-git-upload-pack-advertisement
22:09:32.096041 http.c:790              <= Recv header, 0000000032 bytes (0x00000020)
22:09:32.096045 http.c:802              <= Recv header: x-b3-traceid: b9ff3d6331c8188e
22:09:32.096053 http.c:790              <= Recv header, 0000000073 bytes (0x00000049)
22:09:32.096057 http.c:802              <= Recv header: strict-transport-security: max-age=31536000; includeSubDomains; preload
22:09:32.096064 http.c:790              <= Recv header, 0000000037 bytes (0x00000025)
22:09:32.096068 http.c:802              <= Recv header: date: Mon, 15 Apr 2024 03:09:32 GMT
22:09:32.096075 http.c:790              <= Recv header, 0000000024 bytes (0x00000018)
22:09:32.096081 http.c:802              <= Recv header: x-server: db8ddea36019
22:09:32.096088 http.c:790              <= Recv header, 0000000034 bytes (0x00000022)
22:09:32.096093 http.c:802              <= Recv header: x-envoy-upstream-service-time: 1
22:09:32.096099 http.c:790              <= Recv header, 0000000023 bytes (0x00000017)
22:09:32.096104 http.c:802              <= Recv header: x-version: a18a18c76f
22:09:32.096110 http.c:790              <= Recv header, 0000000002 bytes (0x00000002)
22:09:32.096114 http.c:802              <= Recv header:
22:09:32.096156 http.c:843              == Info: Connection #0 to host bitbucket.org left intact
22:09:32.118200 http.c:843              == Info: Uses proxy env variable NO_PROXY == '*.local,169.254/16'
22:09:32.118243 http.c:843              == Info: Couldn't find host bitbucket.org in the .netrc file; using defaults
22:09:32.118253 http.c:843              == Info: Found bundle for host: 0x600001938450 [can multiplex]
22:09:32.118268 http.c:843              == Info: Re-using existing connection with host bitbucket.org
22:09:32.118360 http.c:843              == Info: [HTTP/2] [3] OPENED stream for https://bitbucket.org/mituharu/emacs-mac.git/git-upload-pack
22:09:32.118368 http.c:843              == Info: [HTTP/2] [3] [:method: POST]
22:09:32.118372 http.c:843              == Info: [HTTP/2] [3] [:scheme: https]
22:09:32.118375 http.c:843              == Info: [HTTP/2] [3] [:authority: bitbucket.org]
22:09:32.118378 http.c:843              == Info: [HTTP/2] [3] [:path: /mituharu/emacs-mac.git/git-upload-pack]
22:09:32.118381 http.c:843              == Info: [HTTP/2] [3] [user-agent: git/2.44.0]
22:09:32.118384 http.c:843              == Info: [HTTP/2] [3] [accept-encoding: deflate, gzip, br, zstd]
22:09:32.118387 http.c:843              == Info: [HTTP/2] [3] [content-type: application/x-git-upload-pack-request]
22:09:32.118391 http.c:843              == Info: [HTTP/2] [3] [accept: application/x-git-upload-pack-result]
22:09:32.118394 http.c:843              == Info: [HTTP/2] [3] [accept-language: en-US, *;q=0.9]
22:09:32.118396 http.c:843              == Info: [HTTP/2] [3] [content-length: 195]
22:09:32.118491 http.c:790              => Send header, 0000000295 bytes (0x00000127)
22:09:32.118503 http.c:802              => Send header: POST /mituharu/emacs-mac.git/git-upload-pack HTTP/2
22:09:32.118506 http.c:802              => Send header: Host: bitbucket.org
22:09:32.118508 http.c:802              => Send header: User-Agent: git/2.44.0
22:09:32.118510 http.c:802              => Send header: Accept-Encoding: deflate, gzip, br, zstd
22:09:32.118513 http.c:802              => Send header: Content-Type: application/x-git-upload-pack-request
22:09:32.118515 http.c:802              => Send header: Accept: application/x-git-upload-pack-result
22:09:32.118517 http.c:802              => Send header: Accept-Language: en-US, *;q=0.9
22:09:32.118520 http.c:802              => Send header: Content-Length: 195
22:09:32.118522 http.c:802              => Send header:
22:09:32.118526 http.c:843              == Info: upload completely sent off: 195 bytes
22:09:32.246759 http.c:790              <= Recv header, 0000000013 bytes (0x0000000d)
22:09:32.246805 http.c:802              <= Recv header: HTTP/2 200
22:09:32.246855 http.c:790              <= Recv header, 0000000015 bytes (0x0000000f)
22:09:32.246866 http.c:802              <= Recv header: server: envoy
22:09:32.246877 http.c:790              <= Recv header, 0000000025 bytes (0x00000019)
22:09:32.246884 http.c:802              <= Recv header: cache-control: no-cache
22:09:32.246900 http.c:790              <= Recv header, 0000000052 bytes (0x00000034)
22:09:32.246907 http.c:802              <= Recv header: content-type: application/x-git-upload-pack-result
22:09:32.246917 http.c:790              <= Recv header, 0000000032 bytes (0x00000020)
22:09:32.246924 http.c:802              <= Recv header: x-b3-traceid: cc65d033e2043f6e
22:09:32.246933 http.c:790              <= Recv header, 0000000073 bytes (0x00000049)
22:09:32.246949 http.c:802              <= Recv header: strict-transport-security: max-age=31536000; includeSubDomains; preload
22:09:32.246962 http.c:790              <= Recv header, 0000000037 bytes (0x00000025)
22:09:32.246969 http.c:802              <= Recv header: date: Mon, 15 Apr 2024 03:09:32 GMT
22:09:32.246981 http.c:790              <= Recv header, 0000000024 bytes (0x00000018)
22:09:32.246991 http.c:802              <= Recv header: x-server: 036f0c34af38
22:09:32.247004 http.c:790              <= Recv header, 0000000035 bytes (0x00000023)
22:09:32.247012 http.c:802              <= Recv header: x-envoy-upstream-service-time: 42
22:09:32.247023 http.c:790              <= Recv header, 0000000023 bytes (0x00000017)
22:09:32.247029 http.c:802              <= Recv header: x-version: a18a18c76f
22:09:32.247042 http.c:790              <= Recv header, 0000000019 bytes (0x00000013)
22:09:32.247052 http.c:802              <= Recv header: content-length: 0
22:09:32.247068 http.c:790              <= Recv header, 0000000002 bytes (0x00000002)
22:09:32.247076 http.c:802              <= Recv header:
22:09:32.247131 http.c:843              == Info: Connection #0 to host bitbucket.org left intact
fatal: the remote end hung up unexpectedly
Command failed: GIT_CURL_VERBOSE=1 /opt/local/bin/git clone --progress --shallow-since=2024-03-25T00:00:00 --branch work https://bitbucket.org/mituharu/emacs-mac.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_emacs-mac-app/emacs-mac-app-devel/work/emacs-29.1-mac-10.0 2>&1
Exit code: 128

[ryandesign (Ryan Carsten Schmidt)]

I'm not sure why this fails. It could be a problem specific to bitbucket and/or the path our network traffic takes to their server.

But maybe we don't need to investigate why --shallow-since doesn't work and instead focus on avoiding the need to use it. No other port uses this flag.

You're presumably using it to limit network traffic and disk space so that you don't fetch the entire repository when you need just one commit.

Why not fetch just that one commit then? ​git has this capability now. It didn't used to, back when MacPorts git fetching commands were implemented.

It should be fixed in MacPorts base, then. A problem is that the git (and hg) fetching code does things manually instead of using the typical command_exec method that's used elsewhere in MacPorts base.

A patch to fix git (and hg) fetching to use command_exec was submitted in #29093. If we get that fixed, then switching the git fetching procedure to fetch just the required commit when one is specified should be fairly easy. That method does work to fetch the emacs-mac-app-devel repository on my system, when done manually on the command line.

However, ideally, the port should not fetch from git at all; it should fetch from a distfile. Taking a quick look at the bitbucket portgroup it seems like it should support that just as github and gitlab do but I see at least one problem with the bitbucket portgroup and will look at it more later.