12020-12-12T00:00:06 *** r2wj <r2wj!~r2wj@185.204.1.185> has joined #bitcoin-core-dev
22020-12-12T00:02:35 <achow101> I'll try an updated native_cctools first
32020-12-12T00:02:58 <sipa> cool
42020-12-12T00:03:15 <sipa> it's also possible that jonasschnelli's codesign tool is too old!
52020-12-12T00:03:28 <sipa> (as native_cctools was updated for 0.21)
62020-12-12T00:03:41 <achow101> Well it's worked for the previous rc
72020-12-12T00:04:29 <sipa> it may be that some alignment thing changed that doesn't matter in every situation
82020-12-12T00:06:27 *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has quit IRC (Remote host closed the connection)
92020-12-12T00:07:45 <sipa> maybe we should just make some arbitrary binary-changing commits for rc4, and hope that that fixes it
102020-12-12T00:07:48 <sipa> *hides*
112020-12-12T00:09:37 *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Quit: Konversation terminated!)
122020-12-12T00:09:57 *** mol_ <mol_!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
132020-12-12T00:12:49 <sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1
142020-12-12T00:13:07 <sipa> achow101: what version is your codesigning tool on macos?
152020-12-12T00:13:22 <achow101> no idea
162020-12-12T00:13:47 <achow101> there's no version command for codesign or codesign_allocate
172020-12-12T00:14:13 *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 265 seconds)
182020-12-12T00:14:59 <sipa> this line in codesign_allocate changed in the latest cctools:
192020-12-12T00:15:10 <sipa> object->seg_linkedit->vmsize = rnd32(object->seg_linkedit->filesize, get_segalign_from_flag(&arch_signs[i].arch_flag));
202020-12-12T00:15:39 <achow101> where do you see this?
212020-12-12T00:17:10 <sipa> https://github.com/tpoechtrager/cctools-port/commit/466063c7f7486762a5a8f0d98c48dc5ffbe42f74#diff-c0274ccf413d9c63c15dc20f765faa1dd750ae7a74b65dfda9afbe1e91ea137cR711
222020-12-12T00:17:22 <sipa> this is in the version we're using in the gitian build
232020-12-12T00:17:48 <sipa> but given that it recently changed (may 2020), perhaps somehow jonasschnelli is using an apple equivalent that is older
242020-12-12T00:17:57 <sipa> *march 2020
252020-12-12T00:19:22 <sipa> actually, no
262020-12-12T00:19:30 <sipa> rnd -> rnd32 doesn't change anything
272020-12-12T00:21:08 <achow101> eh.. they don't make tags for new versions.. great
282020-12-12T00:22:41 *** dermoth_ <dermoth_!~dermoth@unaffiliated/dermoth> has joined #bitcoin-core-dev
292020-12-12T00:23:01 *** dermoth <dermoth!~dermoth@unaffiliated/dermoth> has quit IRC (Disconnected by services)
302020-12-12T00:23:03 *** dermoth_ is now known as dermoth
312020-12-12T00:24:18 <sipa> got it!
322020-12-12T00:24:34 <sipa> cctools v921 uses 4k as default alignment for x86_64
332020-12-12T00:24:44 <sipa> oh, no, i misread :(
342020-12-12T00:28:01 <sipa> what is the full output from that x86_64-apple-darwin-otool -l Bitcoin-Qt command?
352020-12-12T00:28:08 *** proofofkeags <proofofkeags!~proofofke@174-16-212-53.hlrn.qwest.net> has joined #bitcoin-core-dev
362020-12-12T00:29:52 <achow101> of which version of Bitcoin-Qt
372020-12-12T00:30:11 <sipa> no matter
382020-12-12T00:31:01 <achow101> https://0bin.net/paste/ArdAXxJ3#67qz6LxpqPGqIXWODToWwyFxorHcouHH1CzJGxdBB2I
392020-12-12T00:31:04 <gribble> https://github.com/bitcoin/bitcoin/issues/67 | does not compile with boost_1_45_0 on OS X 10.6 · Issue #67 · bitcoin/bitcoin · GitHub
402020-12-12T00:31:15 <achow101> bad bot
412020-12-12T00:35:47 <sipa> the __LINKEDIT "filesize" entry is the same for the codesign-output-binary and the gitian-reconstructed one, right?
422020-12-12T00:35:51 <sipa> just vmsize is different
432020-12-12T00:35:57 <achow101> yes
442020-12-12T00:36:06 <sipa> yeah, if it wasnt't, more than 1 byte would differ
452020-12-12T00:38:17 <sipa> literally every version of cctools i can find uses 0x1000 as segment alignment for X86_64
462020-12-12T00:38:51 <sipa> so the correct rounding of filesize 500144 to vmsize is 0x7b000
472020-12-12T00:39:15 <sipa> why is jonas' tool rounding it to 0x7c000 (like he's using a segment alignment of 0x2000 or 0x4000)
482020-12-12T00:39:47 <sipa> now, libstuff uses 0x2000 for unknown CPU architectures
492020-12-12T00:40:43 <sipa> so maybe somehow the wrong architecture is passed down?
502020-12-12T00:41:24 <sipa> see https://opensource.apple.com/source/cctools/cctools-949.0.1/libstuff/arch.c.auto.html look for "{ CPU_TYPE_X86_64, LITTLE_ENDIAN_BYTE_SEX, 0x7fff5fc00000LL, 0x1000 },"
512020-12-12T00:41:59 <achow101> updating cctools (specifically codesign_allocate) did not make a difference
522020-12-12T00:42:33 *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 260 seconds)
532020-12-12T00:42:54 <achow101> jonasschnelli: are you using the same signing machine as before?
542020-12-12T00:43:00 <sipa> and in https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html look for "object->seg_linkedit->vmsize ="
552020-12-12T00:43:22 <achow101> IIRC he has one of the M1 macs for testing stuff, maybe he used that for signing?
562020-12-12T00:44:36 *** Eagle[TM] <Eagle[TM]!~EagleTM@unaffiliated/eagletm> has quit IRC (Ping timeout: 240 seconds)
572020-12-12T00:50:47 <sipa> i suspect that if we add 1024 bytes to the sig size in codesign_allocate it may work
582020-12-12T00:50:56 <sipa> if you do:
592020-12-12T00:51:05 <sipa> codesign_allocate -i <unsigned Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <some temp file>
602020-12-12T00:51:08 <sipa> on linux
612020-12-12T00:51:28 <sipa> it may just work, with the existing signature
622020-12-12T00:51:41 <achow101> I'm wondering why this size makes a difference to how the signature validates
632020-12-12T00:51:50 <achow101> istm it shouldn't be covered by the signature itself
642020-12-12T00:52:06 <sipa> i believe the code signature covers the entire binary, with the signature itself wiped out (this is a guess...)
652020-12-12T00:52:11 <sipa> so its size matters
662020-12-12T00:52:25 <sipa> that would explain why a tool like codesign_allocate exists in the first place
672020-12-12T00:52:34 <sipa> as its behavior is nominal for the signing process
682020-12-12T00:52:40 <achow101> but the size of the signature isn't known until the signature is made
692020-12-12T00:53:13 <sipa> i assume that the codesign tool does now the signature size beforehand
702020-12-12T00:53:16 <sipa> *know
712020-12-12T00:53:26 <sipa> or it pads it to make sure it does
722020-12-12T00:53:39 <achow101> i suppose it just needs to fall within the page size
732020-12-12T00:53:46 <achow101> and that's probably a big enough range
742020-12-12T00:53:51 <sipa> right
752020-12-12T00:54:24 <achow101> so apple makes codesign_allocate open source, but not codesign itself?
762020-12-12T00:54:47 <sipa> apparently
772020-12-12T00:54:56 <sipa> but codesign invokes codesign_allocate
782020-12-12T00:55:09 <achow101> supposedly
792020-12-12T00:55:18 <sipa> and you can override which codesign_allocate to use... as long as it's signed by apple
802020-12-12T00:56:44 <sipa> if you run "codesign_allocate -i <gitian Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <newname>" in linux, does the resulting binary work in macos?
812020-12-12T00:57:06 <sipa> oh, nvm
822020-12-12T00:57:11 <sipa> nope, won't work
832020-12-12T00:57:21 <sipa> that'll actually make space in the binary
842020-12-12T00:58:12 *** chjj_ <chjj_!~chjj@unaffiliated/chjj> has joined #bitcoin-core-dev
852020-12-12T01:00:01 <achow101> what if I set arch to ARM64 so it uses 0x4000 as the page size?
862020-12-12T01:00:59 *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
872020-12-12T01:03:33 <achow101> and codesign_allocate doesn't like it
882020-12-12T01:08:19 *** chjj <chjj!~chjj@unaffiliated/chjj> has quit IRC (Quit: WeeChat 2.2)
892020-12-12T01:08:26 <sipa> yeah i don't expect that'd work
902020-12-12T01:08:56 <sipa> achow101: but if you run codesign_allocate with 225312, on macos, you also get the 0x7c000 vmsize?
912020-12-12T01:09:54 *** chjj_ <chjj_!~chjj@unaffiliated/chjj> has quit IRC (Quit: WeeChat 2.9)
922020-12-12T01:10:16 *** chjj <chjj!~chjj@unaffiliated/chjj> has joined #bitcoin-core-dev
932020-12-12T01:11:14 <achow101> Ah, yes
942020-12-12T01:12:41 <sipa> so perhaps... the codesign tool isn't invoking it with size 225312
952020-12-12T01:12:53 <sipa> mayne it invokes it twice
962020-12-12T01:13:20 <sipa> once with a prospective size, to get the to-be-signed binary, and then again with the real size
972020-12-12T01:13:39 <achow101> Now, I think the installation of codesign_allocate is different from the open source one
982020-12-12T01:13:54 <sipa> that'd explain the filesize of LINKEDIT being correct, but vmsize isn't
992020-12-12T01:14:01 <achow101> Because it matches jonas's, not gitian
1002020-12-12T01:17:13 *** mj_node <mj_node!~mj_node@122.0.25.130> has quit IRC (Quit: Leaving)
1012020-12-12T01:19:15 <sipa> right
1022020-12-12T01:23:43 <sipa> if you on linux run codesign_allocate with a larger size first, and then again withnthe correct size, does the __linkedit stuff end up being correct?
1032020-12-12T01:33:56 <achow101> what's a big enough size?
1042020-12-12T01:35:02 <sipa> $((225312+1024))
1052020-12-12T01:39:03 *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
1062020-12-12T01:39:21 <achow101> nope. also still 7b
1072020-12-12T01:40:36 <sipa> after both commands, or even after the first one?
1082020-12-12T01:40:41 <achow101> both
1092020-12-12T01:40:44 <sipa> oh, i'm wrong
1102020-12-12T01:40:46 *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
1112020-12-12T01:40:54 <achow101> well both as in both both commands and first one
1122020-12-12T01:40:55 <sipa> 0x1000 is not 1024, but 4096
1132020-12-12T01:41:04 <sipa> so $((225312+4024))
1142020-12-12T01:41:10 <sipa> so $((225312+4096))
1152020-12-12T01:41:12 <achow101> oh right
1162020-12-12T01:43:36 <achow101> that works
1172020-12-12T01:43:57 <achow101> although I will say that this is not a suitable workaround
1182020-12-12T01:44:55 <sipa> we could produce in the gitian output the information on what size to resize to
1192020-12-12T01:45:11 <sipa> it can be deterministically inferred from the macos-signed binary
1202020-12-12T01:46:25 <achow101> i suppose
1212020-12-12T01:46:58 <achow101> it feels like there should be a better way
1222020-12-12T01:47:17 <sipa> yes
1232020-12-12T01:47:37 <sipa> there is clearly a discrepancy between the macos codesign_allocate tool and the opensource one
1242020-12-12T01:47:54 <sipa> and i'm confused why that isn't a problem for other projects
1252020-12-12T01:48:47 <achow101> maybe no one has tried to make a new release recently?
1262020-12-12T01:49:14 <achow101> we didn't see this a few weeks ago for rc2
1272020-12-12T01:52:38 <sipa> yeah, baffling
1282020-12-12T01:54:17 <sipa> lol @ actively {-beating our heads against Apple's walled garden wall-} debugging this issue
1292020-12-12T01:56:41 <sipa> there is a macports version of cctools, presumably using the open source code
1302020-12-12T01:56:53 <sipa> if you run that on macos, what behavior does it have?
1312020-12-12T01:57:22 <achow101> I don't have macports installed
1322020-12-12T02:01:14 <achow101> I found an old version of codesign source on apple's opensource site: https://opensource.apple.com/source/security_systemkeychain/security_systemkeychain-55191.1/src/codesign.cpp.auto.html
1332020-12-12T02:01:26 <achow101> it disappears eventually, but that might just mean it got moved to somewhere else
1342020-12-12T02:02:12 *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has quit IRC ()
1352020-12-12T02:02:21 <sipa> that one doesn't invoke codesign_allocate
1362020-12-12T02:02:52 <sipa> as far i can see, but it may be in another source file
1372020-12-12T02:12:42 *** EagleTM <EagleTM!~EagleTM@unaffiliated/eagletm> has joined #bitcoin-core-dev
1382020-12-12T02:18:08 *** ConsciousFlesh <ConsciousFlesh!~cFlesh@unaffiliated/consciousflesh> has joined #bitcoin-core-dev
1392020-12-12T02:21:34 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1402020-12-12T02:22:06 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Read error: Connection reset by peer)
1412020-12-12T02:24:27 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1422020-12-12T02:24:53 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1432020-12-12T02:27:22 <achow101> sipa: macports cctools uses 0x7b000
1442020-12-12T02:31:27 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1452020-12-12T02:31:54 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1462020-12-12T02:32:21 <ConsciousFlesh> Are there any devs here who were involved with, or around when the `RPCHelpMan()` function replaced throwing a runtime error to display JSON-RPC help text?
1472020-12-12T02:33:00 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1482020-12-12T02:33:19 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1492020-12-12T02:33:43 <sipa> achow101: great, so the apple codesign_allocate just behaves differently from their published source code
1502020-12-12T02:48:42 *** ConsciousFlesh <ConsciousFlesh!~cFlesh@unaffiliated/consciousflesh> has left #bitcoin-core-dev ("If anyone was involved in the RPCHelpMan update I mentioned could you pm me? Had some questions about it's implementation.")
1512020-12-12T02:49:27 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1522020-12-12T02:49:54 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1532020-12-12T02:54:57 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1542020-12-12T02:55:21 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
1552020-12-12T02:58:58 *** EagleTM <EagleTM!~EagleTM@unaffiliated/eagletm> has quit IRC (Ping timeout: 246 seconds)
1562020-12-12T02:59:11 <achow101> i guess we shouldn't be that surprised that they do different things
1572020-12-12T03:24:32 *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
1582020-12-12T03:26:43 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
1592020-12-12T03:32:25 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
1602020-12-12T03:52:33 <achow101> sipa: adding the run codesign_allocate twice trick to detached-sig-apply.sh does make rc3 pass signature validation
1612020-12-12T03:52:53 <sipa> great
1622020-12-12T03:53:27 <sipa> so we just need something to take a size from the code-signatures.tgz
1632020-12-12T03:53:41 <sipa> and modify jonas' script to compute what the right pre-allocate size is
1642020-12-12T03:53:54 <achow101> I wonder if maybe it would be easier to codesign_allocate the unsigned dmg with a larger than necessary but fixed sized chunk
1652020-12-12T03:54:29 <achow101> the gitian signer's codesign allocate shrinks it, but I don't think effects the vmsize
1662020-12-12T03:54:47 <sipa> looking at the codesign_allocate tool, it just only ever increments the vmsize
1672020-12-12T03:55:08 <sipa> which is going to be the rounded-up version of the largest value the filesize ever had
1682020-12-12T03:55:37 <sipa> and whatever we do, we need to replicate the apple tool's behavior
1692020-12-12T03:55:38 <achow101> right, so we could always overshoot by a lot when we make the unsigned dmg
1702020-12-12T03:55:47 <sipa> yup
1712020-12-12T03:55:52 <sipa> that's a possibility
1722020-12-12T03:56:24 <luke-jr> [20:46:44] <jonasschnelli> is there a way to speeup gitians "Upgrading system, may take a while (log in var/install.log)"? <-- I just run the base VM and update it
1732020-12-12T03:56:25 <achow101> I'm going to go through history and see what the largest sigs we've had before is
1742020-12-12T03:57:48 <achow101> any idea if an overly large vmsize has any negative effects?
1752020-12-12T03:59:48 <sipa> i suspect if we crank up the codesign --pagesize value the sig will be much smaller
1762020-12-12T04:00:57 <sipa> it signs every page separately
1772020-12-12T04:01:11 <achow101> oh, is that why the sig is so big and varies a lot
1782020-12-12T04:12:30 *** davec <davec!~davec@072-183-054-196.res.spectrum.com> has joined #bitcoin-core-dev
1792020-12-12T04:43:27 *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has joined #bitcoin-core-dev
1802020-12-12T04:56:11 *** mj_node <mj_node!~mj_node@122.0.25.130> has joined #bitcoin-core-dev
1812020-12-12T05:00:00 *** Emcy <Emcy!~Emcy@unaffiliated/emcy> has quit IRC (Read error: Connection reset by peer)
1822020-12-12T05:01:48 *** Emcy <Emcy!~Emcy@unaffiliated/emcy> has joined #bitcoin-core-dev
1832020-12-12T05:05:41 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1842020-12-12T05:05:41 <bitcoin-git> [bitcoin] pox opened pull request #20635: fix misleading comment about call to non-existing function (master...master) https://github.com/bitcoin/bitcoin/pull/20635
1852020-12-12T05:05:42 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1862020-12-12T05:08:36 *** dhruvm <dhruvm!~noreply@c-73-158-59-66.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 240 seconds)
1872020-12-12T05:17:26 *** r2wj <r2wj!~r2wj@185.204.1.185> has quit IRC (Remote host closed the connection)
1882020-12-12T05:18:15 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
1892020-12-12T05:23:52 *** flag <flag!~flag@net-93-66-71-105.cust.vodafonedsl.it> has quit IRC (Quit: leaving)
1902020-12-12T05:25:43 *** sledges <sledges!~sledges@185.163.110.125> has joined #bitcoin-core-dev
1912020-12-12T05:31:28 *** flag <flag!~flag@net-93-66-71-105.cust.vodafonedsl.it> has joined #bitcoin-core-dev
1922020-12-12T05:33:03 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 260 seconds)
1932020-12-12T06:22:26 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
1942020-12-12T06:44:30 *** mj_node <mj_node!~mj_node@122.0.25.130> has quit IRC (Remote host closed the connection)
1952020-12-12T06:44:50 *** mj_node <mj_node!~mj_node@122.0.25.130> has joined #bitcoin-core-dev
1962020-12-12T07:00:19 <jonasschnelli> sipa, achow101: I'm reading the backlog... for your info,... I haven't changed anything on my signing computer since signing rc2 (which was successful).
1972020-12-12T07:01:12 <achow101> jonasschnelli: good to know. the backlog is quite a journey..
1982020-12-12T07:01:21 <jonasschnelli> AFAIK detached-sig-create.sh under the hood, it uses the macOS codesign tool (which I haven't changed)
1992020-12-12T07:01:32 <jonasschnelli> achow101: yeah.. halfway through.
2002020-12-12T07:01:35 <jonasschnelli> Very interesting.
2012020-12-12T07:03:11 <jonasschnelli> 2020-12-11T23:57:47 <sipa> an alternative is making jonasschnelli downgrade his codesign_allocate tool to one that's compatible with the native_ccttols we use, and then tell his codesign tool to use that
2022020-12-12T07:03:24 <jonasschnelli> ^ as said,... I haven't upgraded anything since rc2
2032020-12-12T07:04:03 <jonasschnelli> 2020-12-12T00:12:49 <sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1
2042020-12-12T07:05:02 <jonasschnelli> I guess sipa wants to know what clang version I have on my signing mac. Why would this be relevant? IMO codesign is not part of the llvm suite
2052020-12-12T07:05:14 <jonasschnelli> And cctools on the gitian host should be irrelevant
2062020-12-12T07:05:15 <achow101> cctools is separate from clang
2072020-12-12T07:05:27 <achow101> it's part of xcode's developer tools
2082020-12-12T07:06:04 <achow101> jonasschnelli: the tl;dr is that codesign_allocate as distributed by apple is different from codesign_allocate compiled from source
2092020-12-12T07:06:28 <achow101> codesign uses codesign_allocate under the hood somewhere and that's causing us problems
2102020-12-12T07:06:40 <jonasschnelli> achow101: thats very interesting...
2112020-12-12T07:07:46 <sipa> jonasschnelli: can you try whag happens if you pass --pagesize 0 to the codesign tool?
2122020-12-12T07:08:28 <sipa> that's unrelated, but it may result in smaller ans more predictable signature sizes
2132020-12-12T07:08:58 <jonasschnelli> sipa: okay... I executed "./detached-sig-create.sh -s "Bitcoin" --pagesize 0"
2142020-12-12T07:09:14 <sipa> no no, to the codesign tool
2152020-12-12T07:09:24 <sipa> so inside detached-sig-create.sh
2162020-12-12T07:09:28 <achow101> sipa: I've figured out what that codesign source I found doesn't have a call to codesign_allocate - the actual signing is way deeper in the stack and isn't open source. the code I found is just the wrapper to expose things as a command line tool
2172020-12-12T07:09:31 <jonasschnelli> sipa: okay
2182020-12-12T07:09:39 <sipa> or does it pass everything through?
2192020-12-12T07:09:45 <achow101> passing extra args to detached-sig-create should pass through to codesign
2202020-12-12T07:09:52 <sipa> ok
2212020-12-12T07:10:39 <sipa> achow101: yeah, i tried browsing but i couldn't even find any place where the pagesize argument was used
2222020-12-12T07:11:23 <jonasschnelli> https://bitcoin.jonasschnelli.ch/signature-osx_pagesize_0.tar.gz
2232020-12-12T07:11:46 <jonasschnelli> I changed L26 to ${CODESIGN} --pagesize 0 -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
2242020-12-12T07:12:25 <jonasschnelli> but why was rc2 fine and rc3 isn't?
2252020-12-12T07:12:31 <achow101> beats me
2262020-12-12T07:12:39 <achow101> I think we've found an edge case
2272020-12-12T07:12:49 <jonasschnelli> as I suspected
2282020-12-12T07:13:04 <achow101> it's weird that this is not present in the open soure build though
2292020-12-12T07:13:11 <sipa> jonasschnelli: it appears that the macos tool uses a different segment alignment
2302020-12-12T07:13:32 <sipa> as your signed result has an alignment of 8192 or 16384 bytes
2312020-12-12T07:13:45 <sipa> while the gitian result is 4096 byte aligned
2322020-12-12T07:14:00 <jonasschnelli> okay
2332020-12-12T07:14:11 <sipa> but the apple (open source) definitely uses 4096
2342020-12-12T07:14:29 <sipa> it may be that rc2 just accidentally was already 8192 aligned
2352020-12-12T07:14:37 <sipa> we could check
2362020-12-12T07:14:53 <sipa> jonasschnelli: how big is the .sign file now?
2372020-12-12T07:15:13 <achow101> sipa: I checked earlier, it's 7b0000
2382020-12-12T07:15:54 <sipa> ok, i have no explanation then :)
2392020-12-12T07:16:04 <jonasschnelli> sipa: jonasschnelli staff 18496 12 Dez 08:10 Bitcoin-Qt.sign
2402020-12-12T07:17:52 <sipa> much better!
2412020-12-12T07:18:12 <sipa> the previous ones were 200 kB or so
2422020-12-12T07:18:35 <jonasschnelli> I can try to gbuild the macOS signer with that
2432020-12-12T07:18:54 <sipa> it's worth a try
2442020-12-12T07:19:13 <sipa> if it was a weird edge case due to alignments, maybe this is enough to fix it
2452020-12-12T07:26:09 *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has quit IRC (Remote host closed the connection)
2462020-12-12T07:26:59 *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
2472020-12-12T07:38:59 <achow101> --pagesize 0 might actually make it possible for us to hack together a tool for codesigning
2482020-12-12T07:39:07 <achow101> that doesn't require macOS
2492020-12-12T07:39:20 <jonasschnelli> pagesize 0 seems also not to work
2502020-12-12T07:39:20 <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed_pagesize0.dmg
2512020-12-12T07:41:38 <achow101> I was going to suggest that we try using the macports codesign_allocate, but the manpage for codesign says "The system will not accept such substitutes unless they are specially signed (by Apple)." on that topic, so I don't think it will work
2522020-12-12T07:42:02 <sipa> indeed
2532020-12-12T07:42:25 <sipa> jonasschnelli: same problem?
2542020-12-12T07:42:39 <sipa> achow101: is it again the vmsize being off?
2552020-12-12T07:42:53 <achow101> checking
2562020-12-12T07:43:11 <sipa> jonasschnelli: is that file the signed binary that comes out of the codesign tool?
2572020-12-12T07:44:05 <jonasschnelli> sipa: no... that file above is the gitian signer result with using the signature.tar.gz that was built with pagesize 0
2582020-12-12T07:44:20 <achow101> i need to compare against the result of the codesign tool
2592020-12-12T07:44:27 <sipa> jonasschnelli: we'll also need the codesign output
2602020-12-12T07:44:41 <jonasschnelli> okay.. one sec
2612020-12-12T07:46:33 <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-pagesize-0.zip
2622020-12-12T07:46:38 *** alko89 <alko89!~alko89@unaffiliated/alko89> has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
2632020-12-12T07:46:48 *** alko89 <alko89!~alko89@unaffiliated/alko89> has joined #bitcoin-core-dev
2642020-12-12T07:47:47 <achow101> yes same problem
2652020-12-12T07:48:03 <achow101> codesign_allocate really likes 7c000 for some reason
2662020-12-12T07:48:05 <sipa> achow101: what are the vmsize values now?
2672020-12-12T07:48:18 <sipa> wait what
2682020-12-12T07:48:38 <sipa> what is the filesize field and the vmsize field in both?
2692020-12-12T07:48:42 <achow101> gitian is 0x4800 and codesign is 7c000
2702020-12-12T07:49:08 <sipa> dafuq
2712020-12-12T07:49:11 <achow101> fileoff and filesize are the same
2722020-12-12T07:49:21 <achow101> both are otherwise identical except for vmsize
2732020-12-12T07:49:28 <sipa> and what is that filesize?
2742020-12-12T07:49:30 * jonasschnelli throws away his mac
2752020-12-12T07:49:42 <achow101> filesize 293328
2762020-12-12T07:49:44 <sipa> jonasschnelli: a just decision
2772020-12-12T07:50:07 <achow101> signature is 18496
2782020-12-12T07:50:40 <sipa> 0x48000 is the correct vmsize
2792020-12-12T07:51:05 *** alko89 <alko89!~alko89@unaffiliated/alko89> has quit IRC (Client Quit)
2802020-12-12T07:51:22 <sipa> well, "correct", from my understanding of what the opensource allocate tool does
2812020-12-12T07:51:28 <achow101> yes
2822020-12-12T07:51:34 <sipa> and that's no surprise, because that's also what it did
2832020-12-12T07:51:52 *** alko89 <alko89!~alko89@unaffiliated/alko89> has joined #bitcoin-core-dev
2842020-12-12T07:51:55 <sipa> but so this means it is not a rounding issue at all
2852020-12-12T07:52:11 <sipa> unless it's rounding up to a multiple of 0x7c000 !
2862020-12-12T07:53:29 <sipa> if you on macos run codesign_allocate infile -a x86_64 18496 -o outfile on the unsigned binary, does it also produce 0x7c000 as vmsize?
2872020-12-12T07:54:11 <jonasschnelli> let me try
2882020-12-12T07:55:29 <sipa> jonasschnelli: did you accidentally codesign an already-signed binary?
2892020-12-12T07:55:52 <jonasschnelli> sipa:.... hmm.. that could be for the latest just done build
2902020-12-12T07:55:56 <sipa> vmsize is not expected to go down in that case
2912020-12-12T07:56:13 <sipa> so you have to start with the unsigned binary
2922020-12-12T07:56:13 *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has quit IRC (Read error: Connection reset by peer)
2932020-12-12T07:56:36 <achow101> vmsize 0x0000000000043188
2942020-12-12T07:56:36 *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has joined #bitcoin-core-dev
2952020-12-12T07:56:43 <jonasschnelli> I try it again with pagesize 0 and make sure it's not signed already
2962020-12-12T07:56:49 <achow101> that's not a multple of 0x1000, 0x2000, or 0x4000
2972020-12-12T07:56:58 <sipa> achow101: wut
2982020-12-12T07:57:41 <achow101> ah wrong file
2992020-12-12T07:57:42 <achow101> vmsize 0x0000000000048000
3002020-12-12T07:57:52 <achow101> that's correct and what we expect
3012020-12-12T07:58:37 <sipa> achow101: with the macos tool?
3022020-12-12T07:58:42 <achow101> yes
3032020-12-12T07:59:34 <sipa> and just to be sure... if you pass 225312 as size?
3042020-12-12T08:00:08 <sipa> (i know we did that before, but perhaps that before we knew that history of sig sizes the binary has been through mattered)
3052020-12-12T08:00:11 <achow101> as before vmsize 0x000000000007c000
3062020-12-12T08:00:17 <sipa> ok
3072020-12-12T08:00:54 <sipa> well, if it's an alignment issue, 0x48000 is 0x4000 aligned, so unlikely to cause problem
3082020-12-12T08:01:08 <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-again-pagesize0.zip <--- signed with pagesize 0, check that is was not already codesigned
3092020-12-12T08:01:08 <sipa> so i expect this will be a solution
3102020-12-12T08:01:12 <jonasschnelli> (gitian output follows)
3112020-12-12T08:02:50 <achow101> sipa: it's aligning against 0x2000
3122020-12-12T08:03:37 <sipa> achow101: you found that by trying more sizes?
3132020-12-12T08:03:45 <achow101> if I use 4096 as the size, I get 46000. same with 8192
3142020-12-12T08:04:04 <jonasschnelli> and if anyone wants to test 0.20.2rc1, I just created the detached signature (no gitian build yet): https://bitcoin.jonasschnelli.ch/signature-osx_0.20.2rc1.tar.gz
3152020-12-12T08:04:05 <sipa> ok
3162020-12-12T08:04:18 <sipa> achow101: we could patch out cctools to do the same
3172020-12-12T08:04:21 <sipa> *our
3182020-12-12T08:04:32 <sipa> it's a tiny change
3192020-12-12T08:04:47 <achow101> I'm not optimistic that that's a lasting solution
3202020-12-12T08:05:55 <sipa> yeah, agree
3212020-12-12T08:06:27 <sipa> but it's simpler than trying to hack around the issue with multiple preallocate calls and passing the size through
3222020-12-12T08:07:19 <achow101> https://github.com/achow101/bitcoin/commit/26f8ebcca26eb74b6546b5d551f3ec468e2878be is my current proposal
3232020-12-12T08:07:27 <achow101> if we combine with pagesize 0, then it'll probably always work
3242020-12-12T08:08:10 <achow101> the largest signature since 0.12.1 is 407424
3252020-12-12T08:10:55 <sipa> achow101: if the gitian preallocate takes it to a large *odd* multiple of 0x1000, will the macos sign tool not increase it further to a multiple of 0x2000 ?
3262020-12-12T08:11:23 <sipa> if it's anything like the opensource version, it won't
3272020-12-12T08:11:39 <sipa> but who knows what it does
3282020-12-12T08:12:20 <jonasschnelli> oh!
3292020-12-12T08:12:30 <jonasschnelli> pagesize 0 result in a valid signatue (through gitian)
3302020-12-12T08:12:43 <jonasschnelli> I didn't before because I signed an already signed binary
3312020-12-12T08:13:07 <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed_pagesize0_clean.dmg
3322020-12-12T08:13:33 <jonasschnelli> (used detach signature from https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-again-pagesize0.zip )
3332020-12-12T08:13:52 <achow101> I think that is currently the expected result
3342020-12-12T08:14:12 <sipa> jonasschnelli: yeah, matches our understanding
3352020-12-12T08:14:47 <achow101> sipa: it does not appear to shrink the vmsize at all
3362020-12-12T08:15:07 <sipa> achow101: yes, but it may increase it further
3372020-12-12T08:15:09 *** tripleslash <tripleslash!~triplesla@unaffiliated/imsaguy> has quit IRC (Ping timeout: 260 seconds)
3382020-12-12T08:15:26 <sipa> (i don't expect it to, but we should check i think)
3392020-12-12T08:15:50 <achow101> i'll try against the macports cctools
3402020-12-12T08:16:08 *** tripleslash <tripleslash!~triplesla@unaffiliated/imsaguy> has joined #bitcoin-core-dev
3412020-12-12T08:19:30 <achow101> it preserves it
3422020-12-12T08:20:15 <sipa> great
3432020-12-12T08:20:52 <sipa> 500000 is probably overkill if we're going to use pagesize 0
3442020-12-12T08:21:21 <sipa> but that does like a pretty simple fix
3452020-12-12T08:21:38 <achow101> probably, but I also have no idea what goes into the signature size
3462020-12-12T08:22:00 <achow101> i'll open a pr when i wake up
3472020-12-12T08:22:13 <jonasschnelli> thanks achow101!
3482020-12-12T08:22:15 <jonasschnelli> and sipa
3492020-12-12T08:22:51 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
3502020-12-12T08:22:53 <bitcoin-git> [bitcoin] MarcoFalke pushed 6 commits to master: https://github.com/bitcoin/bitcoin/compare/6a4806367177...ffc4d0499020
3512020-12-12T08:22:53 <bitcoin-git> bitcoin/master 5aaeb6c Russell Yanofsky: MOVEONLY: Move IsBDBFile, IsSQLiteFile, and ListWalletDir
3522020-12-12T08:22:54 <bitcoin-git> bitcoin/master 6ee9cbd Russell Yanofsky: refactor: Replace ListWalletDir() function with ListDatabases()
3532020-12-12T08:22:55 <bitcoin-git> bitcoin/master 6a7a636 Russell Yanofsky: refactor: Drop call to GetWalletEnv in wallet salvage code
3542020-12-12T08:22:57 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
3552020-12-12T08:23:02 <jonasschnelli> I knew that when I'm going to wake up that you guys will have a solution
3562020-12-12T08:23:09 <achow101> lol
3572020-12-12T08:23:11 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
3582020-12-12T08:23:11 <bitcoin-git> [bitcoin] MarcoFalke merged pull request #20275: wallet: List all wallets in non-SQLite and non-BDB builds (master...pr/exist) https://github.com/bitcoin/bitcoin/pull/20275
3592020-12-12T08:23:12 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
3602020-12-12T08:24:13 <sipa> achow101: number of pages, and "resources" are counted separately
3612020-12-12T08:24:39 <sipa> with pagesize=0 the executable code is always 1 page
3622020-12-12T08:27:02 <sipa> so i expect it to be mostly constant with pagesize=0
3632020-12-12T08:34:00 <jonasschnelli> I'm also getting an invalid signature for 0.20.2rc1
3642020-12-12T08:34:12 <jonasschnelli> shall I codesign with --pagesize 0?
3652020-12-12T08:35:12 <sipa> jonasschnelli: that may work (50% chance)
3662020-12-12T08:36:17 <sipa> jonasschnelli: basically any pagesize you try (has to be zero or a power of 2) has an independent 50% chance of being succesful with the current tooling
3672020-12-12T08:36:28 <sipa> with achow's fix it should always work
3682020-12-12T08:36:41 <sipa> better just wait; it can be backported too
3692020-12-12T08:38:07 <jonasschnelli> ok
3702020-12-12T09:40:34 *** mj_node <mj_node!~mj_node@122.0.25.130> has quit IRC (Quit: Leaving)
3712020-12-12T09:49:25 *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 264 seconds)
3722020-12-12T09:52:51 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:759a:a9e9:8a19:69d2> has quit IRC (Remote host closed the connection)
3732020-12-12T10:04:53 *** kexkey <kexkey!~kexkey@static-198-54-132-158.cust.tzulo.com> has quit IRC (Ping timeout: 260 seconds)
3742020-12-12T10:19:26 *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
3752020-12-12T10:37:28 *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has quit IRC (Ping timeout: 246 seconds)
3762020-12-12T10:42:38 *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
3772020-12-12T11:07:42 *** jonatack <jonatack!~jon@213.152.161.244> has quit IRC (Ping timeout: 256 seconds)
3782020-12-12T11:08:12 *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
3792020-12-12T11:18:28 *** Kenny51Hand <Kenny51Hand!~Kenny51Ha@static.57.1.216.95.clients.your-server.de> has joined #bitcoin-core-dev
3802020-12-12T11:23:34 *** Kenny51Hand <Kenny51Hand!~Kenny51Ha@static.57.1.216.95.clients.your-server.de> has quit IRC (Ping timeout: 272 seconds)
3812020-12-12T11:24:04 *** jonatack <jonatack!~jon@88.124.242.136> has joined #bitcoin-core-dev
3822020-12-12T11:29:37 *** jonatack <jonatack!~jon@88.124.242.136> has quit IRC (Ping timeout: 264 seconds)
3832020-12-12T11:30:03 *** jonatack <jonatack!~jon@213.152.161.40> has joined #bitcoin-core-dev
3842020-12-12T11:35:49 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
3852020-12-12T11:35:51 <bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/ffc4d0499020...b18978066d87
3862020-12-12T11:35:51 <bitcoin-git> bitcoin/master fad68af MarcoFalke: p2p: Ignore non-version msgs before version msg
3872020-12-12T11:35:52 <bitcoin-git> bitcoin/master faaad1b MarcoFalke: p2p: Ignore version msgs after initial version msg
3882020-12-12T11:35:53 <bitcoin-git> bitcoin/master b189780 MarcoFalke: Merge #20079: p2p: Treat handshake misbehavior like unknown message
3892020-12-12T11:35:55 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
3902020-12-12T11:36:09 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
3912020-12-12T11:36:09 <bitcoin-git> [bitcoin] MarcoFalke merged pull request #20079: p2p: Treat handshake misbehavior like unknown message (master...2010-p2pNoVersion) https://github.com/bitcoin/bitcoin/pull/20079
3922020-12-12T11:36:10 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
3932020-12-12T11:40:11 *** sledges <sledges!~sledges@185.163.110.125> has quit IRC (Remote host closed the connection)
3942020-12-12T11:58:27 *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
3952020-12-12T12:11:27 *** Eagle[TM] <Eagle[TM]!~EagleTM@unaffiliated/eagletm> has joined #bitcoin-core-dev
3962020-12-12T12:18:50 *** Talkless <Talkless!~Talkless@mail.dargis.net> has joined #bitcoin-core-dev
3972020-12-12T12:21:30 *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Client Quit)
3982020-12-12T12:21:46 *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
3992020-12-12T12:35:03 *** Talkless <Talkless!~Talkless@mail.dargis.net> has joined #bitcoin-core-dev
4002020-12-12T12:35:10 *** slidercrank1 <slidercrank1!~slidercra@s91904426.blix.com> has joined #bitcoin-core-dev
4012020-12-12T12:40:47 *** rex4539 <rex4539!~rex4539@balticom-197-78.balticom.lv> has joined #bitcoin-core-dev
4022020-12-12T13:08:34 *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
4032020-12-12T13:23:58 *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has joined #bitcoin-core-dev
4042020-12-12T13:24:44 *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has quit IRC (Client Quit)
4052020-12-12T13:27:04 *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
4062020-12-12T13:30:14 *** mol_ <mol_!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 260 seconds)
4072020-12-12T13:35:50 *** vasild_ <vasild_!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
4082020-12-12T13:35:51 *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Disconnected by services)
4092020-12-12T13:35:53 *** vasild_ is now known as vasild
4102020-12-12T13:49:00 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has joined #bitcoin-core-dev
4112020-12-12T13:53:27 *** slidercrank1 <slidercrank1!~slidercra@s91904426.blix.com> has quit IRC (Remote host closed the connection)
4122020-12-12T13:55:59 *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
4132020-12-12T13:56:42 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
4142020-12-12T13:57:21 *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has quit IRC (Ping timeout: 272 seconds)
4152020-12-12T13:58:01 *** rh0nj <rh0nj!~rh0nj@88.99.167.175> has quit IRC (Remote host closed the connection)
4162020-12-12T13:59:08 *** rh0nj <rh0nj!~rh0nj@88.99.167.175> has joined #bitcoin-core-dev
4172020-12-12T14:06:27 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Remote host closed the connection)
4182020-12-12T14:06:53 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
4192020-12-12T14:08:44 *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has joined #bitcoin-core-dev
4202020-12-12T14:12:46 *** veox <veox!~veox@185.163.110.125> has joined #bitcoin-core-dev
4212020-12-12T14:18:47 *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has quit IRC (Remote host closed the connection)
4222020-12-12T14:19:09 *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
4232020-12-12T14:20:26 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Ping timeout: 264 seconds)
4242020-12-12T14:27:37 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
4252020-12-12T14:36:44 *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo_)
4262020-12-12T14:38:37 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Ping timeout: 246 seconds)
4272020-12-12T14:44:14 *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzzâ¦)
4282020-12-12T14:45:35 *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
4292020-12-12T14:48:12 *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has joined #bitcoin-core-dev
4302020-12-12T14:55:36 *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 240 seconds)
4312020-12-12T15:04:03 *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
4322020-12-12T15:09:02 *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
4332020-12-12T15:22:54 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
4342020-12-12T15:27:02 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
4352020-12-12T15:29:30 *** darosior <darosior!~darosior@194.36.189.246> has quit IRC (Remote host closed the connection)
4362020-12-12T15:30:03 *** darosior <darosior!~darosior@194.36.189.246> has joined #bitcoin-core-dev
4372020-12-12T15:32:04 *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has joined #bitcoin-core-dev
4382020-12-12T15:36:31 *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has joined #bitcoin-core-dev
4392020-12-12T15:42:13 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
4402020-12-12T15:46:36 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
4412020-12-12T15:48:43 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
4422020-12-12T15:48:50 *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzzâ¦)
4432020-12-12T15:49:17 *** miketwenty1 <miketwenty1!~miketwent@136.55.84.49> has joined #bitcoin-core-dev
4442020-12-12T15:52:38 *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
4452020-12-12T15:53:35 *** miketwenty1 <miketwenty1!~miketwent@136.55.84.49> has quit IRC (Ping timeout: 258 seconds)
4462020-12-12T15:56:02 *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
4472020-12-12T15:57:17 *** belcher_ <belcher_!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
4482020-12-12T15:59:27 *** tryphe <tryphe!~tryphe@unaffiliated/tryphe> has quit IRC (Read error: Connection reset by peer)
4492020-12-12T15:59:35 *** tryphe_ <tryphe_!~tryphe@unaffiliated/tryphe> has joined #bitcoin-core-dev
4502020-12-12T16:00:02 *** belcher <belcher!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 256 seconds)
4512020-12-12T16:04:25 *** jonatack <jonatack!~jon@213.152.161.40> has quit IRC (Ping timeout: 240 seconds)
4522020-12-12T16:06:09 <prusnak> do you know if https://github.com/shesek is on IRC?
4532020-12-12T16:06:14 *** belcher_ is now known as belcher
4542020-12-12T16:06:23 *** jonatack <jonatack!~jon@109.232.227.138> has joined #bitcoin-core-dev
4552020-12-12T16:06:36 <belcher> prusnak yes hes called shesek
4562020-12-12T16:06:54 <prusnak> shesek: no such nick
4572020-12-12T16:07:27 <prusnak> thanks, i was asking because he's offline most probably
4582020-12-12T16:25:55 <achow101> sipa: I found the codesign source. It's in a library called "Security": https://opensource.apple.com/source/Security/Security-59306.140.5/OSX/libsecurity_codesigning/lib/
4592020-12-12T16:29:41 <achow101> Would it be worthwhile to write a tool that lets us code sign macOS from linux? Given that we can read the original code signing source, I don't think it would be too difficult to write one.
4602020-12-12T16:30:27 *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has quit IRC (Read error: Connection reset by peer)
4612020-12-12T16:32:04 *** rex4539_ <rex4539_!~rex4539@8.40.29.10> has joined #bitcoin-core-dev
4622020-12-12T16:32:06 *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has joined #bitcoin-core-dev
4632020-12-12T16:33:27 *** rex4539 <rex4539!~rex4539@balticom-197-78.balticom.lv> has quit IRC (Read error: Connection reset by peer)
4642020-12-12T16:36:49 *** rex4539_ <rex4539_!~rex4539@8.40.29.10> has quit IRC (Ping timeout: 260 seconds)
4652020-12-12T16:40:55 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
4662020-12-12T16:42:59 *** jonatack <jonatack!~jon@109.232.227.138> has quit IRC (Quit: jonatack)
4672020-12-12T16:49:44 *** dhruv <dhruv!~dhruv@165.227.49.220> has joined #bitcoin-core-dev
4682020-12-12T16:50:35 *** dhruv <dhruv!~dhruv@165.227.49.220> has quit IRC (Client Quit)
4692020-12-12T16:51:09 *** dhruvm <dhruvm!~dhruv@165.227.49.220> has joined #bitcoin-core-dev
4702020-12-12T16:58:08 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
4712020-12-12T16:58:08 <bitcoin-git> [bitcoin] achow101 opened pull request #20638: Mac codesign fixed alloc (master...mac-codesign-fixed-alloc) https://github.com/bitcoin/bitcoin/pull/20638
4722020-12-12T16:58:09 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
4732020-12-12T17:01:11 *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
4742020-12-12T17:05:54 *** jonatack <jonatack!~jon@88.124.242.136> has joined #bitcoin-core-dev
4752020-12-12T17:08:40 *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
4762020-12-12T17:10:16 *** jonatack <jonatack!~jon@88.124.242.136> has quit IRC (Ping timeout: 240 seconds)
4772020-12-12T17:10:42 *** jonatack <jonatack!~jon@213.152.162.84> has joined #bitcoin-core-dev
4782020-12-12T17:17:43 *** dermoth_ <dermoth_!~dermoth@unaffiliated/dermoth> has joined #bitcoin-core-dev
4792020-12-12T17:18:02 *** dermoth <dermoth!~dermoth@unaffiliated/dermoth> has quit IRC (Disconnected by services)
4802020-12-12T17:18:04 *** dermoth_ is now known as dermoth
4812020-12-12T17:25:49 *** davterra <davterra!~davterra@107.182.237.18> has joined #bitcoin-core-dev
4822020-12-12T17:30:17 *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
4832020-12-12T17:38:35 <sipa> achow101: maybe, depending on how hard it is to compile that for linux
4842020-12-12T17:40:24 *** owowo <owowo!~ovovo@unaffiliated/ovovo> has quit IRC (Ping timeout: 256 seconds)
4852020-12-12T17:40:58 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 256 seconds)
4862020-12-12T17:45:22 *** owowo <owowo!~ovovo@unaffiliated/ovovo> has joined #bitcoin-core-dev
4872020-12-12T17:58:58 <wumpus> achow101: nice!
4882020-12-12T18:01:43 <achow101> sipa: I don't think we can compile it ourselves. It would have to be a completely separate implementation.
4892020-12-12T18:02:31 <sipa> achow101: that sounds nontrivial
4902020-12-12T18:02:58 <wumpus> assuming, of course, it's the same as the binary they ship
4912020-12-12T18:03:39 <sipa> the benefit is that if someone actually makes that work, we may be able to replace it with multiparty RSA or s
4922020-12-12T18:03:40 <achow101> it appears to be nontrivial. the code is a bit hard to understand
4932020-12-12T18:03:42 <sipa> o
4942020-12-12T18:04:05 <achow101> I just wanted to make it use rfc6979
4952020-12-12T18:04:07 <wumpus> if it's RSA in the first place
4962020-12-12T18:04:21 <achow101> the cert is rsa
4972020-12-12T18:05:43 <achow101> it's amazing how hard it is to actually find the function that does the cryptographic signature
4982020-12-12T18:07:01 <sipa> i assume the signature contais a certnl chain too
4992020-12-12T18:07:08 <wumpus> you'd say if you're going to do something like per-page signing you'd use an algorithm with short signatures :)
5002020-12-12T18:07:16 <sipa> which may complicate thihgs
5012020-12-12T18:07:39 <sipa> and the code for deciding exactly is being signed may be nontrivial too
5022020-12-12T18:07:58 <wumpus> that explains why no one else made a portable signer i guess
5032020-12-12T18:08:29 <achow101> I'm surprised no one has put in the effort to make one given that it's actually open source
5042020-12-12T18:08:31 <sipa> i wonder how many people even care about doing codesigning macos binaries on a non-macos platform
5052020-12-12T18:09:15 <wumpus> true, not that many projects are doing deterministic builds
5062020-12-12T18:10:25 <wumpus> though even for non-deterministic builds it can often be useful to do all builds on a single build host platform
5072020-12-12T18:11:10 *** mol_ <mol_!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
5082020-12-12T18:12:32 <achow101> even with per-page signing, shouldn't the signature be way larger than 200k?
5092020-12-12T18:12:35 <achow101> given our binary size
5102020-12-12T18:13:55 <sipa> what is the actual binary size without resources?
5112020-12-12T18:14:24 <sipa> i think jonasschnelli's tgz has a file that lists all the resources
5122020-12-12T18:14:28 *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 265 seconds)
5132020-12-12T18:15:12 <achow101> 26891600 Dec 12 11:40 Bitcoin-Qt
5142020-12-12T18:16:51 <sipa> do you know how many bits the RSA key is?
5152020-12-12T18:17:48 <achow101> hmm. the cert is not in contrib/macdeploy.. I thought it was
5162020-12-12T18:17:49 <sipa> achow101: perhaps you can find in that apple source code what the default page size is
5172020-12-12T18:19:34 <sipa> perhaps the default page size is 32k or 64k
5182020-12-12T18:19:35 <achow101> turns out codesign has a lot of --verbose options
5192020-12-12T18:19:42 <achow101> the page size is 4096 in the rc3 sig
5202020-12-12T18:19:56 <sipa> hmm
5212020-12-12T18:20:02 <sipa> that makes little sense
5222020-12-12T18:21:01 <achow101> here is the full verbose output of codesign -d: https://0bin.net/paste/2gi9BEF9#YOEGkrrhbpToojZbGKJhHvcYp5H1zV3MfO+skMBRotg
5232020-12-12T18:21:07 <sipa> well that"s 35 bytes per sig
5242020-12-12T18:21:13 <achow101> it looks like it's actually just a ton of hashes and the signature is over all of them?
5252020-12-12T18:21:20 <achow101> it says Signature size=8973
5262020-12-12T18:21:21 <sipa> ah!
5272020-12-12T18:21:35 <sipa> it's a merkle tree over the binary?
5282020-12-12T18:21:54 <sipa> that makes sens
5292020-12-12T18:22:24 <achow101> couldn't say. but 6463 * 32 is in the ballpark of the signature size
5302020-12-12T18:23:00 <achow101> and there's 6432 hashes. that also fits with the page size and binary size, 6432 * 4096 is near the binary size
5312020-12-12T18:23:50 <achow101> *6463 hashes
5322020-12-12T18:24:26 <sipa> yup
5332020-12-12T18:24:34 <sipa> i suspect that's it exactly
5342020-12-12T18:24:54 <achow101> making hashes of each page seems excessive though
5352020-12-12T18:25:06 <sipa> it allows for fast per-page validation
5362020-12-12T18:25:27 <sipa> without needing to hash all the leaves every time
5372020-12-12T18:28:29 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
5382020-12-12T18:28:29 <bitcoin-git> [bitcoin] jbampton opened pull request #20639: doc: fix case of GitHub (master...fix-case-of-github) https://github.com/bitcoin/bitcoin/pull/20639
5392020-12-12T18:28:30 *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
5402020-12-12T18:32:48 <achow101> I don't think it's a merkle tree. I'm pretty sure it's just a flat list of hashes
5412020-12-12T18:33:01 <achow101> that's what gets hashes again and signed
5422020-12-12T18:35:50 <wumpus> that makes sense
5432020-12-12T18:36:36 <wumpus> a hash would be much more efficient to verify per page, after all, and the signature check can be done once at load time for the hashes section
5442020-12-12T18:37:39 <wumpus> per page (or bunch-of-pages) makes sense as it decreases the granularity of loading, a hash over the entire binary means the entire binary has to be read
5452020-12-12T18:38:17 <achow101> so our trick to do a single page may not be ideal
5462020-12-12T18:38:32 <wumpus> well if it works as a workaround
5472020-12-12T18:38:50 <wumpus> but not in the long term no i guess
5482020-12-12T18:39:28 <achow101> well we now know mostly how big the signature will be, so we could compute an overestimate instead
5492020-12-12T18:40:08 <sipa> achow101: why not? do we care about not loading all pages all at once?
5502020-12-12T18:40:13 <wumpus> if someone wants to reverse engineer this i'd prefer having a signature *verifier* first, so the gitian process can detect if it went wrong
5512020-12-12T18:40:51 <wumpus> sipa: i guess so? there's always a lot of data in a binary that's not 'hot', might never be accessed at all, especially as we static link qt
5522020-12-12T18:41:17 <achow101> i dunno
5532020-12-12T18:41:29 <achow101> it seems like there's a default for a reason
5542020-12-12T18:41:51 <achow101> and it's clearly architecture optimized
5552020-12-12T18:41:53 <wumpus> that said, given the memory requiremetns i'm not sure we care about paging much
5562020-12-12T18:42:24 <achow101> verification of the signature itself should only happen once
5572020-12-12T18:42:41 <sipa> oh you think it's re-verifying the pages every time they get loaded again from disk
5582020-12-12T18:42:43 <wumpus> yes, but the hash might be done more often, say if a page is paged out and in again?
5592020-12-12T18:42:44 <achow101> I would guess this is some kind of runtime integrity check
5602020-12-12T18:42:44 <sipa> that does make sense
5612020-12-12T18:42:59 <sipa> but just the hash, not the full sig
5622020-12-12T18:43:03 <achow101> yeah
5632020-12-12T18:43:23 <wumpus> yes, the sig verification for the hashes section is likely done only once
5642020-12-12T18:43:30 <sipa> but that means that the list of hashes must be loaded at all times
5652020-12-12T18:43:43 <wumpus> yes
5662020-12-12T18:44:45 <wumpus> so it'd be a compromise--too small page size and the list of hashes is huge, too big page size and it has very large paging granularity
5672020-12-12T18:46:23 <wumpus> and i'm not sure--if it wouldn't check every time a page is re-loaded from disk it might open up some kinds of exploits, espeically for network file systems if manipulation of the data can happen without the OS knowing
5682020-12-12T18:47:27 <wumpus> but that's not certain
5692020-12-12T18:48:05 <sipa> if it was a Merkle tree instead they wouldn't need to keep the hash lish loaded, but it'd be twice the size
5702020-12-12T18:52:09 <achow101> I think we can produce a good enough overestimate to allow for the default page size
5712020-12-12T18:53:09 <achow101> (((unsigned_size / 4096) + 1) * 32) + 50000
5722020-12-12T18:53:30 <sipa> yup
5732020-12-12T18:53:38 <wumpus> good!
5742020-12-12T18:53:39 <achow101> the sig is 9k, and there's an additional 10k of overhead. with + 50k, we get 30k of extra
5752020-12-12T18:54:21 <achow101> i'll update the pr
5762020-12-12T18:55:45 <phantomcircuit> it seems to be using https://en.wikipedia.org/wiki/Cryptographic_Message_Syntax
5772020-12-12T18:56:17 <achow101> oh!
5782020-12-12T18:56:26 <achow101> so openssl can be used?
5792020-12-12T18:56:55 *** adiabat <adiabat!~adiabat@63.209.32.102> has quit IRC (Ping timeout: 272 seconds)
5802020-12-12T18:57:13 *** adiabat <adiabat!~adiabat@63.209.32.102> has joined #bitcoin-core-dev
5812020-12-12T18:59:08 <phantomcircuit> achow101, well to quote a comment 'Sign a Mach-O binary, using liberal dollops of that special Mach-O magic sauce.'
5822020-12-12T18:59:09 <phantomcircuit> so
5832020-12-12T18:59:23 <achow101> when I read that, I was not encouraged
5842020-12-12T18:59:57 <sipa> how to construct the Xcode extract for the 0.20 and 0.19 builds?
5852020-12-12T19:00:21 <achow101> sipa: see contrib/macdeploy/README in the 0.20 branch
5862020-12-12T19:00:26 <achow101> it explains how for both
5872020-12-12T19:00:59 <wumpus> if the download is still available on the apple site, hopefully
5882020-12-12T19:01:14 <sipa> https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-mac-os-sdk.md seems to refer to very old stuff
5892020-12-12T19:01:24 <sipa> i guess it should be updated or deleted?
5902020-12-12T19:01:33 <sipa> or changed into a link to the macdeploy stuff?
5912020-12-12T19:01:39 <achow101> I think it should link to macdeploy
5922020-12-12T19:02:08 <wumpus> yea the information on a branch should probably only describe for that branch
5932020-12-12T19:02:47 <achow101> for some reason the 0.19 branch doesn't describe the extract process, but 0.20 does as "the old way"
5942020-12-12T19:03:29 <phantomcircuit> achow101, yeha i cant figure out where it's actually signing either
5952020-12-12T19:03:52 <phantomcircuit> im sure if i could download that entire directory from git or as a tar ball i could
5962020-12-12T19:04:02 <phantomcircuit> but no
5972020-12-12T19:04:13 *** Zenton <Zenton!~user@unaffiliated/vicenteh> has quit IRC (Read error: Connection reset by peer)
5982020-12-12T19:04:13 <phantomcircuit> classic apple
5992020-12-12T19:04:56 <achow101> phantomcircuit: mirror of it here https://github.com/apple-opensource/Security/
6002020-12-12T19:05:11 <achow101> slightly older version, but close enough
6012020-12-12T19:06:06 <achow101> I got to SecCodeSigner::Signer::signCodeDirectory in signer.cpp before I got super lost
6022020-12-12T19:06:45 <phantomcircuit> achow101, yeah that's definitely the right function
6032020-12-12T19:07:22 <phantomcircuit> i got to CMSEncoderAddSigners but can't tell what it's actually doing
6042020-12-12T19:07:29 <phantomcircuit> at least i think that's a decent place to look
6052020-12-12T19:08:07 <phantomcircuit> also, if(hashDict != NULL) {assert(hashDict != NULL); they must not trust their compiler very much
6062020-12-12T19:08:17 *** Zenton <Zenton!~user@unaffiliated/vicenteh> has joined #bitcoin-core-dev
6072020-12-12T19:08:27 <achow101> CMSEncoderCopyEncodedContent claims to do the actual signing, but I find no evidence that supports that. it just kinda goes into a while function stack that I can't understand
6082020-12-12T19:08:37 <achow101> *wild
6092020-12-12T19:09:45 <achow101> CMSEncoderAddSigners just adds some kind of signer object to a list
6102020-12-12T19:10:08 *** Zenton <Zenton!~user@unaffiliated/vicenteh> has quit IRC (Read error: Connection reset by peer)
6112020-12-12T19:11:03 <phantomcircuit> achow101, CMSEncoderUpdateContent
6122020-12-12T19:13:30 <phantomcircuit> wait this looks like it's generating hashes and then encrypting them???
6132020-12-12T19:13:46 <achow101> I think they use encrypt and sign interchangeably
6142020-12-12T19:14:13 *** joelklabo <joelklabo!~textual@108-196-216-127.lightspeed.sntcca.sbcglobal.net> has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
6152020-12-12T19:14:43 <phantomcircuit> achow101, https://github.com/apple-opensource/Security/blob/3aaa9af09496be26b5c15ff011e3d65ca75821ac/libsecurity_smime/lib/cmsencode.c#L389
6162020-12-12T19:14:51 <phantomcircuit> iono kinda looks like it's actually encrypting things
6172020-12-12T19:15:25 <phantomcircuit> of course there's about 10 levels of indirection though so who can even tell
6182020-12-12T19:15:41 <achow101> I think it's just some more generic code that can also handle encryption?
6192020-12-12T19:15:44 <achow101> honstly it's hard to know
6202020-12-12T19:16:31 <sipa> for RSA encrypting and signing are kind-of interchangeable (but the devil is in the details)
6212020-12-12T19:18:32 *** joelklabo <joelklabo!~textual@108-196-216-127.lightspeed.sntcca.sbcglobal.net> has joined #bitcoin-core-dev
6222020-12-12T19:20:20 <phantomcircuit> sipa, just gotta figure out where the ciphcx object is set.. which is uhhh
6232020-12-12T19:21:31 *** Zenton <Zenton!~user@unaffiliated/vicenteh> has joined #bitcoin-core-dev
6242020-12-12T19:29:18 <sipa> ok, downloading xocde 10.2.1
6252020-12-12T19:29:21 <sipa> *xcode
6262020-12-12T19:57:50 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has quit IRC (Remote host closed the connection)
6272020-12-12T20:04:50 <wumpus> can you see what hash algorithm they use? if so it might be most straightforward to check if the hashes are encrypted or otherwise obfuscating by hashing pages and looking if they appear in the signature section of the file
6282020-12-12T20:06:04 <wumpus> pages are bound to have an address aligned to the start of the file, otherwise it'd be wildly inefficient
6292020-12-12T20:08:31 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has joined #bitcoin-core-dev
6302020-12-12T20:13:18 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has quit IRC (Ping timeout: 260 seconds)
6312020-12-12T20:15:43 <achow101> they use sha256, so it shouldn't be hard to check
6322020-12-12T20:21:24 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
6332020-12-12T20:27:02 *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has quit IRC (Read error: Connection reset by peer)
6342020-12-12T20:27:26 *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has joined #bitcoin-core-dev
6352020-12-12T20:37:36 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
6362020-12-12T20:42:33 <achow101> The hashes are definitely embedded
6372020-12-12T20:43:05 <achow101> From the 2nd page to the end of the file, the hashes in the signed binary match the unsigned one
6382020-12-12T20:43:26 <achow101> the 2nd page contains the alloc thing, so it's different, but if the alloc is done, it matches
6392020-12-12T20:43:55 <achow101> only the first page doesn't match. and then there are 3 "special" hashes that are of... something else
6402020-12-12T20:45:50 <wumpus> achow101: oh that's great!
6412020-12-12T20:46:07 <achow101> actually the first page matches, it just doesn't match the unsigned one
6422020-12-12T20:47:19 <wumpus> maybe it updates something in the header first
6432020-12-12T20:50:03 <sipa> well, we know it does :)
6442020-12-12T20:50:12 <sipa> if it didn't, this whole problem wouldn't exist
6452020-12-12T20:50:27 <sipa> the vmsize/filesize entries on the __LINKEDIT section are changed
6462020-12-12T20:50:56 <wumpus> yeah then it makes sense for the hash to match the new state, not the original one
6472020-12-12T20:51:01 <wumpus> right, true :)
6482020-12-12T20:58:47 <achow101> yep, first 2 pages differ from unsigned because of the addition of the code signing section
6492020-12-12T20:59:21 <achow101> so the hashes in the sig are 4096 byte pages from beginning of the signed binary to the beginning of the code signature
6502020-12-12T20:59:29 <achow101> this is something we can verify ourselves
6512020-12-12T21:07:58 *** jonatack <jonatack!~jon@213.152.162.84> has quit IRC (Ping timeout: 256 seconds)
6522020-12-12T21:08:58 <sipa> ah, neat
6532020-12-12T21:09:57 <achow101> i figured out 2 of the 3 special hashes
6542020-12-12T21:10:51 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:25fd:4b36:9c2b:340> has joined #bitcoin-core-dev
6552020-12-12T21:10:51 *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:25fd:4b36:9c2b:340> has quit IRC (Client Quit)
6562020-12-12T21:17:41 <wumpus> that's fast :)
6572020-12-12T21:18:06 <sipa> last one may be some metadata with the signature/hashing parameters?
6582020-12-12T21:18:56 <achow101> looking at the source, I think it's "entitlements"
6592020-12-12T21:18:59 <achow101> but I don't know what that means
6602020-12-12T21:19:26 <sipa> the codesign tool has a way to provide something called entitlements to it
6612020-12-12T21:21:14 <achow101> one of the hashes is of the file Bitcoin-Qt.app/Contents/_CodeSignature/CodeResources, and the other is Bitcoin-Qt.app/Contents/Info.plist
6622020-12-12T21:22:02 <achow101> but this last hash is not any of the files in the bundle
6632020-12-12T21:23:24 <wumpus> if we don't pass "entitlements", maybe it's a hash of empty data?
6642020-12-12T21:23:49 <achow101> I don't think so
6652020-12-12T21:24:00 <achow101> The hash is Bitcoin-Qt.app/Contents
6662020-12-12T21:24:08 <achow101> 1761bc7a0f37e9809acd22c53f42ff21622ea02093c14058e7daa3a466c95d97
6672020-12-12T21:29:01 <sipa> is this hash the same for other releases?
6682020-12-12T21:29:13 <sipa> if it's just some constant we don't actually care what it's a hash of
6692020-12-12T21:34:20 *** astrolince <astrolince!astrolince@gateway/shell/matrix.org/x-pefoseecitekmcnv> has joined #bitcoin-core-dev
6702020-12-12T21:34:29 <wumpus> right a proto-verifyer could just compare it against the fixed hash and raise alarm if it doesn't match
6712020-12-12T21:34:44 <achow101> it's the same in 0.20.1, 0.21.0rc2, and 0.21.0rc3
6722020-12-12T21:35:05 <achow101> afaik we don't have any entitlements so it should just be the hash of an empty something
6732020-12-12T21:35:54 <sipa> google is of no help
6742020-12-12T21:39:33 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
6752020-12-12T21:40:43 <wumpus> "
6762020-12-12T21:40:45 <wumpus> The specific entitlements will be embedded in the signature of an application. If you are having trouble, it can help to look at what the signature actually says about the entitlements: $ codesign -d --entitlements - Example.app will show an XML property list similar to the one above.
6772020-12-12T21:41:04 <achow101> it gives nothing so there are no entitlements
6782020-12-12T21:41:07 <wumpus> so it's a hash of something that is in the signature itself, that sounds strange but...
6792020-12-12T21:41:22 <sipa> maybe it's just <xml></xml> or something
6802020-12-12T21:41:31 <wumpus> I'd expect so
6812020-12-12T21:42:04 <achow101> the source code has it as a datatype of CFDataRef which I think just means it's a void *
6822020-12-12T21:42:10 <sipa> wumpus: sounds similar to bitcoin's use of a sighash byte
6832020-12-12T21:42:18 <sipa> part of the signature itself, and also signed
6842020-12-12T21:43:11 <wumpus> sipa: ah yes
6852020-12-12T21:53:40 <achow101> is there such a thing as a generic DER decoder?
6862020-12-12T21:53:54 <sipa> yes
6872020-12-12T21:55:22 <sipa> https://github.com/pornin/DDer
6882020-12-12T21:55:34 *** owowo <owowo!~ovovo@unaffiliated/ovovo> has quit IRC (Ping timeout: 256 seconds)
6892020-12-12T21:55:44 <sipa> ha, thomas pornin
6902020-12-12T21:55:46 <sipa> i've met him
6912020-12-12T21:57:41 <achow101> hmm, openssl should be able to decode this if it was properly CMS DER, but it doesn't like it
6922020-12-12T21:58:07 <wumpus> it apparently DiskRep::component(cdEntitlementSlot) gets the data that is hashed to verify the entitlement slot, there are different implementations of DispRep though, but in diskrep this is the file "CodeEntitlements"
6932020-12-12T21:58:20 <wumpus> filediskrep*
6942020-12-12T21:59:30 *** abetusk <abetusk!~abe@68.175.128.91> has joined #bitcoin-core-dev
6952020-12-12T22:00:17 *** owowo <owowo!~ovovo@unaffiliated/ovovo> has joined #bitcoin-core-dev
6962020-12-12T22:00:38 <wumpus> in CodeDirectory::slotIsPresent a non-zero digest in the slot means that is is present
6972020-12-12T22:02:38 <wumpus> i guess i should be looking at BundleDiskRep::component(cdEntitlementSlot) as it can fetch components from the executable itself
6982020-12-12T22:02:44 <achow101> I found a bunch of ASN.1 templates, this means it should be possible to figure the serialization, right?
6992020-12-12T22:03:04 <sipa> achow101: can you show me the DER data?
7002020-12-12T22:03:39 <achow101> sipa: in the bitcoin-detached-sigs repo osx/dist/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt.sign
7012020-12-12T22:03:45 <achow101> that's the code signature. in theory it's DER
7022020-12-12T22:03:59 <achow101> (checkout 0.21 branch)
7032020-12-12T22:03:59 <sipa> you don't actually need the schema to decode DER, it's self-descriptive
7042020-12-12T22:04:09 <sipa> the scheme just helps with naming stuff
7052020-12-12T22:04:30 <achow101> I'm not convinced it's actually DER
7062020-12-12T22:04:59 <wumpus> *i have no clue what i'm doing dog image*
7072020-12-12T22:05:08 <wumpus> this looks interesting: https://chromium.googlesource.com/experimental/chromium/src/+/refs/wip/bajones/webvr/chrome/browser/safe_browsing/signature_evaluator_mac.mm
7082020-12-12T22:06:16 <sipa> it's not
7092020-12-12T22:06:25 <wumpus> ok
7102020-12-12T22:07:01 <sipa> $ file Bitcoin-Qt.sign
7112020-12-12T22:07:01 <sipa> Bitcoin-Qt.sign: Mac OS X Detached Code Signature (non-executable) - 216249 bytes
7122020-12-12T22:07:23 <achow101> wait file recognizes this???
7132020-12-12T22:09:16 <sipa> hmm where does file keeps its magic definitions these days?
7142020-12-12T22:09:28 <sipa> i only find a precompiled magic.mgc file
7152020-12-12T22:09:46 <achow101> I think that's it
7162020-12-12T22:09:51 <achow101> gotta find the source for file
7172020-12-12T22:09:54 <wumpus> from MachORep::compoenent to EmbeddedSignatureBlob::component now
7182020-12-12T22:10:31 <sipa> https://github.com/file/file/blob/master/magic/Magdir/apple
7192020-12-12T22:11:05 <wumpus> "An EmbeddedSignatureBlob is a SuperBlob indexed by component slot number." of coursee
7202020-12-12T22:11:53 <wumpus> class EmbeddedSignatureBlob : public SuperBlobCore<EmbeddedSignatureBlob, 0xfade0cc0, uint32_t>
7212020-12-12T22:13:13 <sipa> _0x0ff: belong0xfade0cc1Mac OS X Detached Code Signature
7222020-12-12T22:13:48 <sipa> 0 belong 0xfade0cc1 Mac OS X Detached Code Signature
7232020-12-12T22:14:31 <wumpus> typedef SuperBlob<0xfade0cc1> DetachedSignatureBlob; // indexed by main architecture
7242020-12-12T22:14:52 <wumpus> superblobs, all of them
7252020-12-12T22:15:09 <sipa> turtles all the way down
7262020-12-12T22:15:15 <wumpus> hehe
7272020-12-12T22:15:24 <achow101> blobloblob
7282020-12-12T22:17:58 <wumpus> / as a genuine Blob (e.g. for insertion into a SuperBlob).
7292020-12-12T22:18:22 <wumpus> a superblob is a map of integers to blobs
7302020-12-12T22:18:46 <sipa> getting deeper
7312020-12-12T22:19:27 *** yunier2002 <yunier2002!~yunier200@c-73-85-126-91.hsd1.fl.comcast.net> has joined #bitcoin-core-dev
7322020-12-12T22:19:44 <wumpus> "// echoes from parent BlobCore (the C++ type system is too restrictive here)" uh
7332020-12-12T22:21:50 <wumpus> tbh seems like it's actually all fairly straightforward C++ code
7342020-12-12T22:22:19 <wumpus> the comments are way scarier than the code itself
7352020-12-12T22:24:41 <sipa> wumpus: you're biased
7362020-12-12T22:24:44 <sipa> :)
7372020-12-12T22:25:45 <wumpus> i... suppose hehe
7382020-12-12T22:26:30 <achow101> i've somehow wandered into some code in this libsecurity that's Copyright Mozilla
7392020-12-12T22:44:25 *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has quit IRC (Quit: Going offline, see ya! (www.adiirc.com))
7402020-12-12T22:54:51 <sipa> so do we know how to extract the blobs from the EmbeddedSignatureBlob ?
7412020-12-12T23:00:51 <achow101> nope?
7422020-12-12T23:02:31 *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Quit: Konversation terminated!)
7432020-12-12T23:04:18 <achow101> That unknown hash earlier is apparently "internal requirements", not entitlements
7442020-12-12T23:11:38 <wumpus> kind of, it's u32 magic (BE), u32 (BE) size, then u32 (LE) number of blobs, then per blob u32 (LE) blob type, u32 LE offset
7452020-12-12T23:11:47 <wumpus> then the data
7462020-12-12T23:12:47 <wumpus> blob type is the slot id, so 5 for the cdEntitlementSlot, but apprantly we're no longer looking for that
7472020-12-12T23:13:41 <sipa> ok so 3 blobs
7482020-12-12T23:14:26 <sipa> first blob is type 0, length 0x24
7492020-12-12T23:14:40 <wumpus> which is good as it doesn't have that one
7502020-12-12T23:14:47 <sipa> ah no, not length; offset
7512020-12-12T23:15:01 <sipa> the individual blobs don't have a length?
7522020-12-12T23:15:16 <wumpus> sipa: that seems correct?
7532020-12-12T23:15:28 <wumpus> yes, offset
7542020-12-12T23:15:33 <wumpus> the length is in the blob itself
7552020-12-12T23:15:37 <sipa> the first blob has magic fa de 0c 02
7562020-12-12T23:15:46 <wumpus> blobs are all <magic> <length>
7572020-12-12T23:15:57 <sipa> got it
7582020-12-12T23:16:12 <achow101> list of magics in OSX/libsecurity_codesigning/lib/CSCommonPriv.h
7592020-12-12T23:16:51 <sipa> Mac OS X Code Directory is the first one
7602020-12-12T23:16:55 <wumpus> so yes we have 0 at offset 0x24, 2 at offset 0x2f7a4 and 0x100 at offset 0x2f864
7612020-12-12T23:17:16 <achow101> sipa: yes. this contains our list of hashes
7622020-12-12T23:17:19 <wumpus> (all integers are BE, it seemed from the code that they'd be LE but apparently they're swapped somewhere)
7632020-12-12T23:18:04 <achow101> it looks like the structure is the class members
7642020-12-12T23:18:08 <achow101> in order as listed
7652020-12-12T23:18:09 <sipa> second one has magic fa de 0c 01, Mac OS X Code Requirement Set
7662020-12-12T23:18:45 <wumpus> there's your requirement set to hash i guess
7672020-12-12T23:18:54 <sipa> last blob has magic fa de 0b 01
7682020-12-12T23:19:57 <wumpus> class BlobWrapper : public Blob<BlobWrapper, 0xfade0b01>
7692020-12-12T23:20:01 <sipa> the first blob has magic fa de 0c 02
7702020-12-12T23:20:05 <sipa> https://gist.github.com/nmoinvaz/cd8651ffb3659161423534b29824510a
7712020-12-12T23:20:22 <achow101> oh nice
7722020-12-12T23:21:33 <achow101> hmm, it can't be the structure as listed because I run into some strings before I'm supposed to
7732020-12-12T23:21:38 <achow101> but it's pretty close
7742020-12-12T23:21:59 <wumpus> euhh the third's slot id is 0x10000 which is cdSignatureSlot = 0x10000,| | | // CMS signature
7752020-12-12T23:22:19 *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has quit IRC (Remote host closed the connection)
7762020-12-12T23:22:24 <achow101> that might be DER encoded
7772020-12-12T23:22:26 <wumpus> which is wrapped in a BlobWrapper which is simply a blob header with arbitrary binary dat
7782020-12-12T23:22:45 *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has joined #bitcoin-core-dev
7792020-12-12T23:22:51 *** miketwenty1 <miketwenty1!~miketwent@ec2-34-202-224-110.compute-1.amazonaws.com> has joined #bitcoin-core-dev
7802020-12-12T23:22:58 <sipa> this has so many layers, i'd be surprised if it isn't somehow exploitable...
7812020-12-12T23:24:40 <achow101> In class CodeDirectory, everything in the class decodes in order up to spare3
7822020-12-12T23:25:00 <achow101> then we get to the offset for identifier string and the identifer string begins there, as expected
7832020-12-12T23:25:31 <wumpus> otoh the blob format is super simple, there could be a mistake of course, but yes e.g. DER parsers are notoriously exploitable
7842020-12-12T23:26:53 *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has quit IRC (Remote host closed the connection)
7852020-12-12T23:27:19 *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has joined #bitcoin-core-dev
7862020-12-12T23:30:09 <wumpus> the CMS signature is produced in SecCodeSigner::Signer::signCodeDirectory in signer.cpp
7872020-12-12T23:30:27 *** veox <veox!~veox@185.163.110.125> has quit IRC (Remote host closed the connection)
7882020-12-12T23:30:39 <achow101> it seems like the message that is signed is the CodeDirectory blob
7892020-12-12T23:31:48 <wumpus> that's what the comment says too "Generate the CMS signature for a (finished) CodeDirectory.", in addition it takes a hashDict and hashList
7902020-12-12T23:32:13 *** Barno7 <Barno7!~bob@93.55.242.131> has quit IRC (Remote host closed the connection)
7912020-12-12T23:38:05 <wumpus> where hashDict is apparently "v2" and hashList for "v1"
7922020-12-12T23:40:43 <wumpus> the encoding is indeed the Cryptographic Message Syntax what phantomcircuit linked to with apple specific tags
7932020-12-12T23:49:31 <phantomcircuit> <sipa> this has so many layers, i'd be surprised if it isn't somehow exploitable...
7942020-12-12T23:49:33 <phantomcircuit> indeed
7952020-12-12T23:53:08 <achow101> wumpus: that Requirement Set blob is indeed the unknown hash.
7962020-12-12T23:55:19 *** miketwenty1 <miketwenty1!~miketwent@ec2-34-202-224-110.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
7972020-12-12T23:56:43 *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has joined #bitcoin-core-dev
7982020-12-12T23:56:56 *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
7992020-12-12T23:58:00 *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev