12020-12-11T00:00:20  *** jonatack <jonatack!~jon@213.152.186.40> has joined #bitcoin-core-dev
  22020-12-11T00:09:52  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@171.5.29.209> has joined #bitcoin-core-dev
  32020-12-11T00:10:25  *** belcher <belcher!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 264 seconds)
  42020-12-11T00:10:33  *** belcher_ <belcher_!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
  52020-12-11T00:14:18  <luke-jr> my gitians all match, pushed
  62020-12-11T00:15:03  *** chri_eb <chri_eb!~chris@gateway/tor-sasl/chrieb/x-28824719> has quit IRC (Ping timeout: 240 seconds)
  72020-12-11T00:15:23  *** sipa <sipa!~pw@gateway/tor-sasl/sipa1024> has quit IRC (Ping timeout: 240 seconds)
  82020-12-11T00:15:24  *** sdaftuar <sdaftuar!~sdaftuar@gateway/tor-sasl/sdaftuar> has quit IRC (Ping timeout: 240 seconds)
  92020-12-11T00:15:24  *** virtu <virtu!~virtu@gateway/tor-sasl/virtu> has quit IRC (Ping timeout: 240 seconds)
 102020-12-11T00:15:24  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
 112020-12-11T00:15:24  *** yanmaani <yanmaani!~yanmaani@gateway/tor-sasl/yanmaani> has quit IRC (Ping timeout: 240 seconds)
 122020-12-11T00:15:43  *** andrewtoth_ <andrewtoth_!~andrewtot@gateway/tor-sasl/andrewtoth> has quit IRC (Ping timeout: 240 seconds)
 132020-12-11T00:15:44  *** az0re <az0re!~az0re@gateway/tor-sasl/az0re> has quit IRC (Ping timeout: 240 seconds)
 142020-12-11T00:15:44  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has quit IRC (Ping timeout: 240 seconds)
 152020-12-11T00:15:44  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
 162020-12-11T00:15:44  *** k3tan <k3tan!~pi@gateway/tor-sasl/k3tan> has quit IRC (Ping timeout: 240 seconds)
 172020-12-11T00:16:03  *** ghost43 <ghost43!~daer@gateway/tor-sasl/daer> has quit IRC (Ping timeout: 240 seconds)
 182020-12-11T00:17:06  *** theStack <theStack!~honeybadg@vps1648322.vs.webtropia-customer.com> has quit IRC (Ping timeout: 272 seconds)
 192020-12-11T00:17:36  *** theStack <theStack!~honeybadg@vps1648322.vs.webtropia-customer.com> has joined #bitcoin-core-dev
 202020-12-11T00:20:31  *** belcher_ <belcher_!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 246 seconds)
 212020-12-11T00:23:16  *** chri_eb <chri_eb!~chris@gateway/tor-sasl/chrieb/x-28824719> has joined #bitcoin-core-dev
 222020-12-11T00:23:24  *** k3tan <k3tan!~pi@gateway/tor-sasl/k3tan> has joined #bitcoin-core-dev
 232020-12-11T00:24:04  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
 242020-12-11T00:24:04  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has joined #bitcoin-core-dev
 252020-12-11T00:24:04  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
 262020-12-11T00:24:06  *** ghost43 <ghost43!~daer@gateway/tor-sasl/daer> has joined #bitcoin-core-dev
 272020-12-11T00:24:42  *** yanmaani <yanmaani!~yanmaani@gateway/tor-sasl/yanmaani> has joined #bitcoin-core-dev
 282020-12-11T00:26:00  *** sdaftuar <sdaftuar!~sdaftuar@gateway/tor-sasl/sdaftuar> has joined #bitcoin-core-dev
 292020-12-11T00:27:34  *** sipa <sipa!~pw@gateway/tor-sasl/sipa1024> has joined #bitcoin-core-dev
 302020-12-11T00:29:01  *** kristapsk <kristapsk!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
 312020-12-11T00:32:20  *** virtu <virtu!~virtu@gateway/tor-sasl/virtu> has joined #bitcoin-core-dev
 322020-12-11T00:32:23  *** az0re <az0re!~az0re@gateway/tor-sasl/az0re> has joined #bitcoin-core-dev
 332020-12-11T00:36:46  <achow101> why did we keep changing the sdk?
 342020-12-11T00:36:51  <achow101> *macOS sdk
 352020-12-11T00:37:09  <achow101> 0.19 is 10.11, 0.20 is 10.14, and 0.21 is long_xcode_name
 362020-12-11T00:38:12  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
 372020-12-11T00:38:23  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Ping timeout: 258 seconds)
 382020-12-11T00:42:43  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
 392020-12-11T00:45:05  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@171.5.29.209> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
 402020-12-11T00:46:15  <fanquake> achow: just to piss everyone off
 412020-12-11T00:46:52  <fanquake> sipa: welcome back to the gitian building world
 422020-12-11T00:48:42  <sipa> :)
 432020-12-11T00:49:17  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
 442020-12-11T00:49:18  <bitcoin-git> [bitcoin] dongcarl opened pull request #20619: guix: Quality of life improvements (master...2020-12-guix-fixups) https://github.com/bitcoin/bitcoin/pull/20619
 452020-12-11T00:49:19  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
 462020-12-11T00:50:07  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo_)
 472020-12-11T00:57:51  *** promag_ <promag_!~promag@92.250.103.147> has joined #bitcoin-core-dev
 482020-12-11T00:59:13  *** promag <promag!~promag@188.250.84.129> has quit IRC (Ping timeout: 260 seconds)
 492020-12-11T01:01:21  *** promag <promag!~promag@188.250.84.129> has joined #bitcoin-core-dev
 502020-12-11T01:01:22  *** promag_ <promag_!~promag@92.250.103.147> has quit IRC (Read error: Connection reset by peer)
 512020-12-11T01:06:08  *** promag_ <promag_!~promag@188.250.84.129> has joined #bitcoin-core-dev
 522020-12-11T01:09:58  *** promag <promag!~promag@188.250.84.129> has quit IRC (Ping timeout: 256 seconds)
 532020-12-11T01:18:54  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@mx-ll-171.5.29-209.dynamic.3bb.co.th> has joined #bitcoin-core-dev
 542020-12-11T01:20:16  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Read error: Connection reset by peer)
 552020-12-11T01:20:23  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
 562020-12-11T01:30:50  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
 572020-12-11T01:34:13  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 260 seconds)
 582020-12-11T01:36:37  *** larryruane_ <larryruane_!uid473749@gateway/web/irccloud.com/x-ucsaxcwppqjkhsjq> has quit IRC (Quit: Connection closed for inactivity)
 592020-12-11T01:46:08  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
 602020-12-11T01:46:09  <bitcoin-git> [bitcoin] fanquake pushed 4 commits to master: https://github.com/bitcoin/bitcoin/compare/da957cd62ecc...736eb4d80838
 612020-12-11T01:46:10  <bitcoin-git> bitcoin/master c5e3e74 Hennadii Stepanov: sync: Improve CheckLastCritical()
 622020-12-11T01:46:11  <bitcoin-git> bitcoin/master cb23fe0 Hennadii Stepanov: [skip ci] sync: Check precondition in LEAVE_CRITICAL_SECTION() macro
 632020-12-11T01:46:12  <bitcoin-git> bitcoin/master e1e68b6 Hennadii Stepanov: test: Fix inconsistent lock order in wallet_tests/CreateWallet
 642020-12-11T01:46:16  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
 652020-12-11T01:46:33  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
 662020-12-11T01:46:33  <bitcoin-git> [bitcoin] fanquake merged pull request #19982: test: Fix inconsistent lock order in wallet_tests/CreateWallet (master...200920-leave-cs) https://github.com/bitcoin/bitcoin/pull/19982
 672020-12-11T01:46:34  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
 682020-12-11T01:46:59  <fanquake> I assume now that we are using cirrus, adding things like [skip ci] to commit messages is pointless ?
 692020-12-11T01:47:12  <fanquake> I never really saw the point of doing it anyways
 702020-12-11T01:47:50  <andytoshi> travis still runs some lints that check every commit
 712020-12-11T01:48:44  <fanquake> Yea. Although hopefully that is going away shortly: #20467
 722020-12-11T01:48:45  <gribble> https://github.com/bitcoin/bitcoin/issues/20467 | Move travis linter job to cirrus · Issue #20467 · bitcoin/bitcoin · GitHub
 732020-12-11T01:49:08  <fanquake> I think someone else could jump in and  take that on if they wanted, given it's been 2 weeks
 742020-12-11T01:50:19  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
 752020-12-11T01:50:31  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@mx-ll-171.5.29-209.dynamic.3bb.co.th> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
 762020-12-11T01:57:44  *** dr-orlovsky <dr-orlovsky!~dr-orlovs@31.14.40.19> has quit IRC (Ping timeout: 256 seconds)
 772020-12-11T02:04:52  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@171.5.29.209> has joined #bitcoin-core-dev
 782020-12-11T02:05:40  *** az0re <az0re!~az0re@gateway/tor-sasl/az0re> has quit IRC (Quit: Leaving)
 792020-12-11T02:17:37  *** miketwen_ <miketwen_!~miketwent@ec2-34-202-224-110.compute-1.amazonaws.com> has quit IRC (Ping timeout: 264 seconds)
 802020-12-11T02:35:23  <fanquake> Someone could also port the Travis check in the gitian.sigs repo to Cirrus (if we want to keep it). That should be very easy
 812020-12-11T02:54:36  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has joined #bitcoin-core-dev
 822020-12-11T02:56:57  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
 832020-12-11T02:57:21  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
 842020-12-11T02:58:24  *** Eagle[TM] <Eagle[TM]!~EagleTM@unaffiliated/eagletm> has joined #bitcoin-core-dev
 852020-12-11T03:00:38  *** EagleTM <EagleTM!~EagleTM@unaffiliated/eagletm> has quit IRC (Ping timeout: 256 seconds)
 862020-12-11T03:14:26  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Ping timeout: 264 seconds)
 872020-12-11T03:19:37  *** twistedline <twistedline!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
 882020-12-11T03:23:53  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo_)
 892020-12-11T03:42:16  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Read error: Connection reset by peer)
 902020-12-11T03:44:34  *** twistedline <twistedline!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
 912020-12-11T04:01:11  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has quit IRC (Ping timeout: 272 seconds)
 922020-12-11T04:02:45  *** k3tan <k3tan!~pi@gateway/tor-sasl/k3tan> has quit IRC (Remote host closed the connection)
 932020-12-11T04:03:16  *** k3tan <k3tan!~pi@gateway/tor-sasl/k3tan> has joined #bitcoin-core-dev
 942020-12-11T04:49:22  *** JackSparrow <JackSparrow!~JackSparr@s91904426.blix.com> has quit IRC (Remote host closed the connection)
 952020-12-11T05:09:17  *** gac410 <gac410!~gac410@178.162.212.214> has joined #bitcoin-core-dev
 962020-12-11T05:33:50  *** ryanofsky <ryanofsky!russ@jumpy.yanofsky.org> has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
 972020-12-11T05:35:18  *** harding <harding!quassel@2600:3c03::f03c:91ff:fe7b:78d1> has quit IRC (Remote host closed the connection)
 982020-12-11T05:35:21  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@171.5.29.209> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
 992020-12-11T05:36:03  *** ryanofsky <ryanofsky!~russ@jumpy.yanofsky.org> has joined #bitcoin-core-dev
1002020-12-11T05:39:27  *** harding <harding!~quassel@newmail.dtrt.org> has joined #bitcoin-core-dev
1012020-12-11T06:00:31  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has quit IRC (Remote host closed the connection)
1022020-12-11T06:01:02  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has joined #bitcoin-core-dev
1032020-12-11T06:05:38  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has joined #bitcoin-core-dev
1042020-12-11T07:30:38  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 260 seconds)
1052020-12-11T07:38:38  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
1062020-12-11T07:40:03  *** virtu <virtu!~virtu@gateway/tor-sasl/virtu> has quit IRC (Ping timeout: 240 seconds)
1072020-12-11T07:43:41  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has joined #bitcoin-core-dev
1082020-12-11T07:46:40  *** virtu <virtu!~virtu@gateway/tor-sasl/virtu> has joined #bitcoin-core-dev
1092020-12-11T08:17:19  <jonasschnelli> macOS gitian build on master seems to fail: https://bitcoin.jonasschnelli.ch/gitian/build/359
1102020-12-11T08:17:38  <jonasschnelli> Fetching v2.1.1.tar.gz from https://github.com/al45tair/mac_alias/archive/ (stalls)
1112020-12-11T08:33:16  <wumpus> achow101: I noticed that the sqlite wallet backend recreates the prepared queries for every batch, is there a specific reason for this? I guess it has to do with multi-threading?
1122020-12-11T08:34:07  *** mj_node <mj_node!7a001982@122.0.25.130> has joined #bitcoin-core-dev
1132020-12-11T08:35:51  <mj_node> Folks, I was 95% done syncing my node, and unplugged my external SSD accidently, I tried to restart bitcoind, but it's resuming from block 0, can anyone help? I tried to re-builld, etc.. the [blocks] folder has all the raw blk.dat but I just cant get it to use them.
1142020-12-11T08:36:37  <sipa> mj_node: run with -reindex
1152020-12-11T08:37:24  <sipa> it'll still restart the validation from 0, but it won't redownload everything
1162020-12-11T08:41:16  <wumpus> normally unless a really high dbcache value is used it shouldn't go back that far on a crash, though unplugging storage while running can cause all kinds of corruption so it's hard to say-in any case a reindex is all you can do, try to keep the device plugged this time :-)
1172020-12-11T08:42:22  <mj_node> I tried that -reindex doesnt work, its started to re-download blocks, I launched again re-index it spent hours going through the blocks but still says "loadBlockIndexDB: last block file = 136", when I have over 2000 block files
1182020-12-11T08:42:40  <wumpus> ok, wipe everything and start over then
1192020-12-11T08:42:43  <mj_node> somehow I think block\index content is the problem
1202020-12-11T08:42:53  <mj_node> Im on 1mbps conneciton ,ehehe
1212020-12-11T08:42:54  <wumpus> it's corrupted beyond repair
1222020-12-11T08:42:58  <mj_node> ouch
1232020-12-11T08:43:17  <mj_node> just to understand, is it because the blk.dat files are unique/
1242020-12-11T08:43:37  <mj_node> and custom to each ?
1252020-12-11T08:44:58  <sipa> mj_node: presumably when it started redownloading it started overwriting the block files you already had
1262020-12-11T08:45:33  <mj_node> sipa: yes exactly that's what happened, so even a command like 'reconsider block' doesnt work?
1272020-12-11T08:45:34  <wumpus> not necessarily, they contain public information after all, though the blocks don't come in in sequental order so they'll be in different orders in the files on different nodes, the "block index" database contains pointers to where every block is
1282020-12-11T08:46:02  <mj_node> @wumpus hmm so surely I should be able to rebuild the block index from my raw blocks on the SDD no?
1292020-12-11T08:46:11  <wumpus> yes, that's what reindex does
1302020-12-11T08:46:22  <sipa> mj_node: redownloading corrupted the blocks you already had
1312020-12-11T08:46:31  <wumpus> but not if the first blocks were overwritten as sipa says
1322020-12-11T08:46:54  <mj_node> got it -thanks guys, appreciate the explaination, another 20 days i guess.
1332020-12-11T08:46:55  <sipa> so you have a few good blocks at the beginning, then overwritten garbage, and then all old good blocks
1342020-12-11T08:47:05  <sipa> 20?!
1352020-12-11T08:47:15  <mj_node> slow wifi....
1362020-12-11T08:47:16  <sipa> what hardware is this?
1372020-12-11T08:47:28  <sipa> ow
1382020-12-11T08:47:38  <mj_node> ehehhehe, I'm in quarantine with shitty connection
1392020-12-11T08:47:49  <sipa> if network is the bottleneck i can't help you
1402020-12-11T08:48:15  <sipa> mj_node: here is something you couldnl try (it's a moonshot, though)
1412020-12-11T08:48:17  <wumpus> you might want to copy the block files from someone else on physical storage
1422020-12-11T08:48:38  <sipa> start downloading again in another directory
1432020-12-11T08:48:58  <sipa> until you have the first few block files (as many as you possibly had overwritten)
1442020-12-11T08:49:22  <wumpus> in any case, this isn't a support channel, unless you're doing development and having questions about the code for that reason this is not the place, use #bitcoin next time
1452020-12-11T08:49:23  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
1462020-12-11T08:49:40  <sipa> then copy those over the same-named ones in your real dir
1472020-12-11T08:49:46  <sipa> and then do a reindex
1482020-12-11T08:49:52  <sipa> also, yeah, what wumpus said
1492020-12-11T08:49:55  <mj_node> @wumpus sorry for that, and I will make sure to go to #bitcoin-core-dev
1502020-12-11T08:50:02  <mj_node> #bitcoin i mean...
1512020-12-11T08:50:02  *** Pavlenex <Pavlenex!~Thunderbi@185.245.85.251> has joined #bitcoin-core-dev
1522020-12-11T08:50:26  *** mj_node <mj_node!7a001982@122.0.25.130> has left #bitcoin-core-dev
1532020-12-11T08:57:08  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
1542020-12-11T09:13:03  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
1552020-12-11T09:13:25  <wumpus> support for rescanning with missing block files, and in general for disjointed block directories and downloading only missing blocks would be an interesting feature, and likely required for more eleborate pruning strategies, but I think there's a few things prventing this from working right now
1562020-12-11T09:20:27  <vasild> the way blocks and undo are stored now in blk and rev files would make this messy
1572020-12-11T09:20:29  <vasild> put everything in sqlite!
1582020-12-11T09:21:21  <wumpus> I don't think the storage format is the problem, I think this kind of corruption is easier to handle the simpler your data format is
1592020-12-11T09:21:50  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has joined #bitcoin-core-dev
1602020-12-11T09:22:38  <wumpus> heh :)
1612020-12-11T09:23:21  <wumpus> ideally it would get the longest headers chain from P2P *then* start reconstructing
1622020-12-11T09:23:42  <wumpus> it's easier to puzzle what fits where then
1632020-12-11T09:24:14  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1642020-12-11T09:24:15  <bitcoin-git> [bitcoin] MarcoFalke pushed 5 commits to master: https://github.com/bitcoin/bitcoin/compare/736eb4d80838...6a4806367177
1652020-12-11T09:24:15  <bitcoin-git> bitcoin/master 91d6195 Suhas Daftuar: Simplify and clarify extra outbound peer counting
1662020-12-11T09:24:16  <bitcoin-git> bitcoin/master 3cc8a7a Suhas Daftuar: Use conn_type to identify block-relay peers, rather than m_tx_relay == nul...
1672020-12-11T09:24:16  <bitcoin-git> bitcoin/master daffaf0 Suhas Daftuar: Periodically make block-relay connections and sync headers
1682020-12-11T09:24:18  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1692020-12-11T09:24:34  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1702020-12-11T09:24:34  <bitcoin-git> [bitcoin] MarcoFalke merged pull request #19858: Periodically make block-relay connections and sync headers (master...2020-08-blocks-only-rotation) https://github.com/bitcoin/bitcoin/pull/19858
1712020-12-11T09:24:36  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1722020-12-11T09:30:14  <wumpus> instead of having 'import blocks' as a discrete initialization phase, consider the unplaced but known blocks already on disk as another block source like P2P (thinking of it I suppose it *almost* works that way already, after reindex-chainstate two-phase reindex process)
1732020-12-11T09:31:23  <wumpus> it'd be a lot of un-fun fiddling around with broken block directories and such to develop and test this, doesen't sound like fun :)
1742020-12-11T09:33:24  *** promag_ is now known as promag
1752020-12-11T09:33:27  <promag> cfields: ping?
1762020-12-11T09:40:24  <wumpus> at the least it'd need to rescan *all* `blk*` files, not stop when one is missing and delete the rest, e.g. get rid of this sledgehammer: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L658
1772020-12-11T09:43:38  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 260 seconds)
1782020-12-11T09:46:13  *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has quit IRC (Remote host closed the connection)
1792020-12-11T09:49:59  <gwillen> wumpus: one thing that occurred to me, when I had a corrupted block directory myself and was waiting to see if core could recover (it could not), is that you have to be slightly careful
1802020-12-11T09:50:30  <gwillen> you don't want any risk of ending up in a situation where your block files on disk actually contain two slightly different copies of the same block
1812020-12-11T09:50:42  <gwillen> and you checked one of them, but one or more indices ends up pointing at the other one
1822020-12-11T09:51:08  <gwillen> this seems like the sort of thing that could slip in, by being more liberal in what one accepts from the block files
1832020-12-11T09:52:23  <gwillen> (especially considered that it is Generally Regarded As Safe to grab this stuff from someone else to bootstrap, trusting that core will check it on startup before using it)
1842020-12-11T09:53:03  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
1852020-12-11T09:54:59  <aj> isn't that what `-loadblock=<file>` is for?
1862020-12-11T10:02:23  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1872020-12-11T10:02:24  <bitcoin-git> [bitcoin] Psyruss77 opened pull request #20623: Create msbuild.yml (master...master) https://github.com/bitcoin/bitcoin/pull/20623
1882020-12-11T10:02:24  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1892020-12-11T10:03:09  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1902020-12-11T10:03:09  <bitcoin-git> [bitcoin] fanquake closed pull request #20623: Create msbuild.yml (master...master) https://github.com/bitcoin/bitcoin/pull/20623
1912020-12-11T10:03:10  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1922020-12-11T10:05:08  *** kexkey_ <kexkey_!~kexkey@static-198-54-132-93.cust.tzulo.com> has quit IRC (Ping timeout: 260 seconds)
1932020-12-11T10:12:52  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
1942020-12-11T10:16:48  *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has joined #bitcoin-core-dev
1952020-12-11T10:49:51  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
1962020-12-11T10:49:51  <bitcoin-git> [bitcoin] jnewbery opened pull request #20624: net processing: Remove nStartingHeight check from block relay (master...2020-12-remove-starting-height) https://github.com/bitcoin/bitcoin/pull/20624
1972020-12-11T10:49:52  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
1982020-12-11T10:54:35  <aj> jnewbery: is the logic for 20624 right? we never go back into IBD without stopping and restarting, but could conceivably get more than two weeks behind (eg if we've got an unattended node whose network was down)
1992020-12-11T10:59:46  <jnewbery> aj: potentially, yes. We could get out of IBD, lose connectivity and fall behind, and then start catching up again and relaying old blocks to peers. That's mostly also the case now (the 2000 blocks behind nStartingHeight won't stop us from relaying old blocks to peers if we've been connected to them for a while)
2002020-12-11T11:00:53  <aj> jnewbery: presuming we were connected a while, they should be in the same state as us (either both current, or both out of date), so relaying is probably okay; if the network was down, when it came back up, we'd reconnect and choose new starting heights though?
2012020-12-11T11:01:15  <jnewbery> I think the worst case is this: we get out of IBD, lose connectivity and fall behind, and then connect to new peers that are on the best tip. We start catching up and then relay headers to peers that are ahead of us. The worst case is 80 bytes for each header, and that peer wouldn't download the block.
2022020-12-11T11:01:53  *** belcher <belcher!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
2032020-12-11T11:02:54  <aj> jnewbery: i guess my impression is that if this works okay if you fall behind, it should work okay if you're still in IBD?
2042020-12-11T11:04:34  <jnewbery> the 'this' working ok being 'checking that you're within 2000 blocks of the peer's starting height'?
2052020-12-11T11:05:13  <aj> jnewbery: jnewbery: "sending headers, relaying blocks" -- if it's okay to do when you've been out of IBD but are 2001 blocks behind, it should also be okay if you're in IBD?
2062020-12-11T11:06:19  <aj> jnewbery: (the "?" there is doing a lot of work...)
2072020-12-11T11:07:40  <jnewbery> maybe. I think it's best that we just avoid relaying any inventory when we're in IBD
2082020-12-11T11:08:45  <aj> jnewbery: so i think the argument against doing it in IBD is you could be on a relatively low-work but high-height fork from genesis, and end up spamming lots of headers/blocks and waste lots of b/w before everyone realises there's a higher-work fork and switches to that
2092020-12-11T11:10:48  <jnewbery> I
2102020-12-11T11:11:17  <aj> jnewbery: doing a high-height fork from 2000 blocks ago is expensive, though; and maybe finishing IBD generally is good enough there -- worst case there's a long, low-work chain from a year or two ago, _and_ you've been running but disconnected for that time, but the number of nodes doing that are going to be small, even compared to the number of nodes syncing from genesis at any point in time?
2112020-12-11T11:11:18  <jnewbery> I'm confused. Are you arguing against the change in 20624?
2122020-12-11T11:11:36  <aj> jnewbery: no, i'm trying to understand it
2132020-12-11T11:14:31  <jnewbery> I think worst worst case is you send headers from that long chain from a year or two ago. Two years of headers is ~8MB
2142020-12-11T11:15:01  <sdaftuar> aj: in practice, if you're out of IBD, you'll be syncing headers with all your peers shortly after the connection is established
2152020-12-11T11:15:32  <sdaftuar> aj: because the criteria for doing that lines up pretty closely (iirc). While in IBD, we purposely sync headers from only one peer to avoid duplication.
2162020-12-11T11:16:10  <sdaftuar> so i think if we are announcing blocks to peers while catching up after coming back online (so IBD is false, but we're in fact behind), it's no big deal, because we ought to know our peers' header chain anyway
2172020-12-11T11:16:21  <sdaftuar> and therefore won't actually be announcing to them, since we'll know they know the blocks already
2182020-12-11T11:16:40  *** filchef <filchef!~filchef@89.253.179.83> has joined #bitcoin-core-dev
2192020-12-11T11:17:15  *** filchef <filchef!~filchef@89.253.179.83> has quit IRC (Client Quit)
2202020-12-11T11:18:14  <sdaftuar> i guess it's worth testing that there's not some slippage at the beginning of a connection, if they are slow to respond to our getheaders and we are connecting blocks, maybe we'd blast them with useless data?  not sure how likely that is
2212020-12-11T11:18:36  *** Neoma33Crooks <Neoma33Crooks!~Neoma33Cr@static.57.1.216.95.clients.your-server.de> has joined #bitcoin-core-dev
2222020-12-11T11:20:37  *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has quit IRC (Ping timeout: 264 seconds)
2232020-12-11T11:20:51  *** provoostenator <provoostenator!~quassel@provoostenator.sprovoost.nl> has quit IRC (Remote host closed the connection)
2242020-12-11T11:20:55  <jnewbery> sdaftuar: are you talking about after removing the nStartingHeight check? I think if we're out of IBD, we can't be connecting that many blocks?
2252020-12-11T11:21:23  <sdaftuar> (just realized "while in ibd, we purposely sync headers from only one peer" is a bit imprecise -- what i meant was that we wait until we have completed header sync from one peer before syncing with others, so we could be connecting blocks whiel still doing header download, and not yet have tried syncing headers with our other peers)
2262020-12-11T11:21:31  *** provoostenator <provoostenator!~quassel@provoostenator.sprovoost.nl> has joined #bitcoin-core-dev
2272020-12-11T11:21:50  <sdaftuar> jnewbery: yes i'm talking about that check (well your PR), trying to explain the IBD vs nonIBD distinction better
2282020-12-11T11:23:16  *** Neoma33Crooks <Neoma33Crooks!~Neoma33Cr@static.57.1.216.95.clients.your-server.de> has quit IRC (Ping timeout: 256 seconds)
2292020-12-11T11:23:47  <sdaftuar> hm! the behavior around exactly how we starting syncing headers witha  peer is not quite what i remember.
2302020-12-11T11:25:40  <sdaftuar> it still may not matter much, but it looks like if we fall behind a bunch (say because we lost our network for a while, and then it came back up) that we would only sync headers from one peer until our tip is close to current, before syncing with new ones
2312020-12-11T11:28:01  <sdaftuar> anyway, i think that's a long-winded way of me thinking that not-relay in IBD is the right behavior, and relaying while not in IBD even if we happen to be a bit behind is probably ok.
2322020-12-11T11:30:52  <jnewbery> I need to understand the "start block sync" and FindNextBlocksToDownload() logic better
2332020-12-11T11:32:25  <sdaftuar> the basic idea is that when we start up, we pick a first peer to start syncing headers from. as soon as we have headers that indicate there is a tip >= work of our current tip, we start downloading towards it from any peer that has it.  also, once our headers chain is close to current (time within a day of current time), we sync headers from all peers.
2342020-12-11T11:33:15  <sdaftuar> and then as those peers respond with their headers (which should be quick, if our headers chain is the correct one -- a single header with their best tip is typical) we'll download blocks from them as well, since we'll know they have the blocks we need.
2352020-12-11T11:36:38  <jnewbery> That certainly makes sense conceptually. I just gind that mapping that design to the various bits of logic and state in SendMessages() and elsewhere is a bit tricky
2362020-12-11T11:38:35  <aj> jnewbery: "gind" ?
2372020-12-11T11:38:51  <aj> find
2382020-12-11T11:39:22  <jnewbery> *find
2392020-12-11T12:00:05  <michaelfolkson> I'm looking at bech32 https://bitcoin.stackexchange.com/questions/100508/can-you-break-down-what-data-is-encoded-into-a-bech32-address
2402020-12-11T12:00:33  <michaelfolkson> bech32 addresses start with bc representing Bitcoin and then 1 which is just a separator
2412020-12-11T12:00:52  <michaelfolkson> But regtest bech32 start bcrt
2422020-12-11T12:01:11  <michaelfolkson> Does anyone know what the `rt` represent?
2432020-12-11T12:01:35  <wumpus> regtest
2442020-12-11T12:02:34  <michaelfolkson> Huh, that was easy. Why does regtest have a rt separator yet testnet and signet both start tb1 an don't need a separator?
2452020-12-11T12:02:51  <wumpus> this comes from https://github.com/bitcoin/bitcoin/blob/master/src/chainparams.cpp#L446
2462020-12-11T12:03:39  <aj> michaelfolkson: 1 is always the separator in bech32
2472020-12-11T12:04:10  <wumpus> both testnet and signet start with 'tb' i don't know why it was chosen to use the same there, probably because they are both test networks and there can potentially already be many of them
2482020-12-11T12:04:12  <michaelfolkson> Oh so regtest always starts bcrt1?
2492020-12-11T12:04:29  <michaelfolkson> bcrt is human readable and 1 is the separator
2502020-12-11T12:04:36  <wumpus> yes
2512020-12-11T12:05:33  <michaelfolkson> Ok thanks
2522020-12-11T12:06:21  <michaelfolkson> This is the context on why testnet and signet both start tb https://github.com/bitcoin/bitcoin/pull/18267#discussion_r491150895
2532020-12-11T12:06:32  <wumpus> this is the best reference of course: https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki
2542020-12-11T12:07:05  <wumpus> michaelfolkson: thanks for looking it up so it was as i guessed
2552020-12-11T12:08:06  <michaelfolkson> Presumably non-default signets will be encouraged to not start tb
2562020-12-11T12:08:27  <michaelfolkson> Although can't force them
2572020-12-11T12:08:34  <aj> michaelfolkson: nah, that would require patching the code
2582020-12-11T12:08:48  <michaelfolkson> Ohhh non-default signets will also start tb?
2592020-12-11T12:08:57  <aj> michaelfolkson: there used to be config options for it, -signet_hrp= or so
2602020-12-11T12:09:38  <wumpus> seems from that discussion that all new test networks start with tb, as it's for testing, address overlap is not critical
2612020-12-11T12:09:52  <wumpus> that regtest has its own is a historical artifact then
2622020-12-11T12:09:56  <aj> yeah, and makes it easier to port wallets to different testnets
2632020-12-11T12:10:03  <wumpus> exactly
2642020-12-11T12:10:15  *** peterrizzo <peterrizzo!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
2652020-12-11T12:10:15  <aj> meanwhile regtest is kind-of bitcoin-core specific, and there's no point having wallets work with it
2662020-12-11T12:10:47  *** mj_node <mj_node!~mj_node@122.0.25.130> has joined #bitcoin-core-dev
2672020-12-11T12:11:05  <michaelfolkson> Yeah treating regtest differently to testnet, signet makes sense as not meant to be a public network
2682020-12-11T12:11:42  <michaelfolkson> Though choosing tb versus bcrt is a bit peculiar
2692020-12-11T12:11:56  <aj> i noticed today that apparently when p2sh was being deployed, regtest didn't even exist
2702020-12-11T12:12:04  <wumpus> the public signet is a public network
2712020-12-11T12:12:34  <michaelfolkson> And presumably the non-default signets would also be public(ish) networks
2722020-12-11T12:12:42  <michaelfolkson> Otherwise use regtest
2732020-12-11T12:13:09  <wumpus> yes definitely easier to share
2742020-12-11T12:15:07  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has joined #bitcoin-core-dev
2752020-12-11T12:16:01  *** mj_node is now known as symbiosis
2762020-12-11T12:17:01  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has quit IRC (Remote host closed the connection)
2772020-12-11T12:17:16  *** gac410 <gac410!~gac410@178.162.212.214> has quit IRC (Remote host closed the connection)
2782020-12-11T12:17:48  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has joined #bitcoin-core-dev
2792020-12-11T12:20:55  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
2802020-12-11T12:22:19  *** kristapsk <kristapsk!~KK@gateway/tor-sasl/kristapsk> has quit IRC (Remote host closed the connection)
2812020-12-11T12:22:20  *** ghost43 <ghost43!~daer@gateway/tor-sasl/daer> has quit IRC (Read error: Connection reset by peer)
2822020-12-11T12:22:21  *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
2832020-12-11T12:22:29  *** ghost43_ <ghost43_!~daer@gateway/tor-sasl/daer> has joined #bitcoin-core-dev
2842020-12-11T12:23:05  *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 240 seconds)
2852020-12-11T12:29:36  *** Pavlenex <Pavlenex!~Thunderbi@185.245.85.251> has quit IRC (Ping timeout: 240 seconds)
2862020-12-11T12:37:01  <jonatack> "// Download if this is a nice peer, or we have no nice peers and this one might do."
2872020-12-11T12:37:11  <jonatack> love some of these comments :)
2882020-12-11T12:37:59  *** AmberJ_ <AmberJ_!~AmberJ_@178.239.168.171> has joined #bitcoin-core-dev
2892020-12-11T12:43:21  *** dr-orlovsky <dr-orlovsky!~dr-orlovs@31.14.40.19> has joined #bitcoin-core-dev
2902020-12-11T12:46:23  <wumpus> it's cute
2912020-12-11T12:58:20  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
2922020-12-11T13:09:29  <michaelfolkson> What you looking at jonatack?
2932020-12-11T13:10:31  <jonatack> michaelfolkson: grep the codebase ;)
2942020-12-11T13:11:26  <michaelfolkson> So demanding of my typing fingers
2952020-12-11T13:12:01  *** kali1 <kali1!~kali@i16-les01-ntr-31-36-36-40.sfr.lns.abo.bbox.fr> has joined #bitcoin-core-dev
2962020-12-11T13:12:29  *** kali1 <kali1!~kali@i16-les01-ntr-31-36-36-40.sfr.lns.abo.bbox.fr> has left #bitcoin-core-dev
2972020-12-11T13:17:37  <jonatack> you'd be forgiven for thinking that line was from a jane austen novel instead
2982020-12-11T13:21:16  <aj> "It is a truth universally acknowledged, that a high-bandwidth archive node must be in want of an inbound connection" ?
2992020-12-11T13:24:03  *** symbiosis <symbiosis!~mj_node@122.0.25.130> has quit IRC (Quit: Leaving)
3002020-12-11T13:25:36  <vasild> "uhoh, different"
3012020-12-11T13:30:37  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has joined #bitcoin-core-dev
3022020-12-11T13:34:49  *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has joined #bitcoin-core-dev
3032020-12-11T13:35:44  *** vasild_ <vasild_!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
3042020-12-11T13:35:45  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Disconnected by services)
3052020-12-11T13:35:45  *** vasild_ is now known as vasild
3062020-12-11T13:39:11  *** vincenzopalazzo <vincenzopalazzo!~vincent@2001:b07:6474:9d49:849d:db24:7f93:fb8a> has joined #bitcoin-core-dev
3072020-12-11T13:40:02  *** mol_ <mol_!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
3082020-12-11T13:40:05  *** owowo <owowo!~ovovo@unaffiliated/ovovo> has quit IRC (Ping timeout: 240 seconds)
3092020-12-11T13:42:41  *** molz_ <molz_!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 258 seconds)
3102020-12-11T13:44:55  *** owowo <owowo!~ovovo@179.43.152.51> has joined #bitcoin-core-dev
3112020-12-11T13:46:50  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d> has quit IRC (Ping timeout: 264 seconds)
3122020-12-11T13:49:58  *** peterrizzo <peterrizzo!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo)
3132020-12-11T14:03:26  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
3142020-12-11T14:10:39  *** Pavlenex <Pavlenex!~Thunderbi@178.220.68.122> has joined #bitcoin-core-dev
3152020-12-11T14:24:23  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
3162020-12-11T14:34:48  *** Pavlenex1 <Pavlenex1!~Thunderbi@185.244.212.67> has joined #bitcoin-core-dev
3172020-12-11T14:36:26  *** davterra <davterra!~davterra@107.182.237.18> has joined #bitcoin-core-dev
3182020-12-11T14:37:25  *** Pavlenex <Pavlenex!~Thunderbi@178.220.68.122> has quit IRC (Ping timeout: 264 seconds)
3192020-12-11T14:37:26  *** Pavlenex1 is now known as Pavlenex
3202020-12-11T14:54:45  <wumpus> jonasschnelli: strange, you have a mismatch for the gitian macos signed build
3212020-12-11T14:55:57  <wumpus> but the unsigned one matches
3222020-12-11T15:00:26  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
3232020-12-11T15:07:14  <jonasschnelli> wumpus: that’s really strange.
3242020-12-11T15:07:21  <jonasschnelli> Let me do it again
3252020-12-11T15:11:47  *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has joined #bitcoin-core-dev
3262020-12-11T15:14:14  *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has quit IRC (Read error: Connection reset by peer)
3272020-12-11T15:14:39  *** sr_gi <sr_gi!~sr_gi@80.174.218.168.dyn.user.ono.com> has joined #bitcoin-core-dev
3282020-12-11T15:15:57  <wumpus> it shouldn't even be possible, a difference would invalidate the code-signing right?
3292020-12-11T15:16:07  <wumpus> or is there scope for malleability
3302020-12-11T15:18:11  *** Pavlenex1 <Pavlenex1!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
3312020-12-11T15:18:54  <jonasschnelli> The only reason I could think of is that I have built it before the sigs where pushed (then it would have took the rc2 detached sig).
3322020-12-11T15:19:18  <jonasschnelli> But I very much doubt that I did this
3332020-12-11T15:20:37  *** Pavlenex <Pavlenex!~Thunderbi@185.244.212.67> has quit IRC (Ping timeout: 265 seconds)
3342020-12-11T15:20:37  *** Pavlenex1 is now known as Pavlenex
3352020-12-11T15:22:21  <wumpus> it doesn't verify what it is attaching?
3362020-12-11T15:25:21  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
3372020-12-11T15:27:40  <jonasschnelli> I don’t think so. It just takes the newest signature from the 0.21 branch (signature repository)
3382020-12-11T15:28:06  <jonasschnelli> A check against the release/tag should probably be added.
3392020-12-11T15:28:30  <jonasschnelli> I investigate as soon as when I’m back on my desk
3402020-12-11T15:28:47  <wumpus> I mean a check that the signature checks out against what it's attached to
3412020-12-11T15:29:23  <wumpus> the windows one has a check like that IIRC
3422020-12-11T15:31:13  <jonasschnelli> wumpus: that’s not the case AFAIK. I don’t know if you can verify the signature on Linux. Probably possible but maybe complicated to add.
3432020-12-11T15:31:45  <wumpus> right, thanks
3442020-12-11T15:33:49  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
3452020-12-11T15:38:36  *** mol_ <mol_!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 240 seconds)
3462020-12-11T15:46:46  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
3472020-12-11T15:48:49  <luke-jr> is the CI failing to merge with master before running? or did we lose a bunch of Cirrus instances?
3482020-12-11T15:49:17  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
3492020-12-11T15:49:17  <bitcoin-git> [bitcoin] ryanofsky closed pull request #19195: wallet: ScanForWalletTransactions cleanup (master...2020-06-06-scanforwallettransactions-cleanup) https://github.com/bitcoin/bitcoin/pull/19195
3502020-12-11T15:49:18  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
3512020-12-11T15:50:12  <luke-jr> also does this error make sense to anyone? only happens on bitcoinbuilds :/ https://bitcoinbuilds.org/index.php?ansilog=98434a62-d0b5-4596-bfde-5e5504998bc3.log
3522020-12-11T15:53:10  *** belcher <belcher!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 272 seconds)
3532020-12-11T15:54:17  <wumpus> luke-jr: it's passing two arguments to a function that takes one?
3542020-12-11T15:54:33  <wumpus> e.g. "CreateChainParams(gArgs, gArgs.GetChainName());"
3552020-12-11T15:54:35  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Read error: Connection reset by peer)
3562020-12-11T15:54:57  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
3572020-12-11T15:55:10  *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
3582020-12-11T15:55:31  *** miketwen_ <miketwen_!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
3592020-12-11T15:56:54  *** reallll <reallll!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
3602020-12-11T15:58:45  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Ping timeout: 240 seconds)
3612020-12-11T16:00:45  *** reallll is now known as belcher
3622020-12-11T16:00:58  <wumpus> oh the two-arg version is correct in master, is this some rebase/merge problem maybe?
3632020-12-11T16:01:25  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
3642020-12-11T16:01:53  <wumpus> as somehow it ends up with an old chainparams.h with single-argument CreateChainParams
3652020-12-11T16:13:27  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
3662020-12-11T16:22:21  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
3672020-12-11T16:25:05  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Remote host closed the connection)
3682020-12-11T16:27:47  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
3692020-12-11T16:34:07  *** Bob_ <Bob_!~bob@93.55.242.131> has joined #bitcoin-core-dev
3702020-12-11T16:35:01  <jonasschnelli> wumpus: I did the build again and getting now the “correct” hash.
3712020-12-11T16:35:09  <jonasschnelli> New build: https://bitcoin.jonasschnelli.ch/gitian/build/360
3722020-12-11T16:35:40  <jonasschnelli> Old build (the one I pushed to gitian.signs): https://bitcoin.jonasschnelli.ch/gitian/build/358
3732020-12-11T16:37:01  <wumpus> jonasschnelli: happy to hear that! time to upload the rc binaries then
3742020-12-11T16:37:01  <luke-jr> wumpus: I see. So that confirms a CI issue too >_<
3752020-12-11T16:37:05  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Quit: = "")
3762020-12-11T16:37:14  <luke-jr> jonasschnelli: bitcoinbuilds is apparently not merging master before running CI
3772020-12-11T16:39:54  <jonasschnelli> luke-jr: AFAIK is does
3782020-12-11T16:40:27  <jonasschnelli> Someone, my old (invalid) build has different bitcoin-osx-unsigned.tar.gz as input
3792020-12-11T16:40:30  <jonasschnelli> I don't know why
3802020-12-11T16:40:43  <jonasschnelli> (old invalid build): https://bitcoin.jonasschnelli.ch/gitian/builds/360/bitcoin-dmg-signer-build.assert
3812020-12-11T16:41:25  <jonasschnelli> https://bitcoin.jonasschnelli.ch/gitian/builds/358/bitcoin-dmg-signer-build.assert
3822020-12-11T16:41:42  <jonasschnelli> very strange
3832020-12-11T16:42:19  <luke-jr> jonasschnelli: see https://bitcoinbuilds.org/index.php?ansilog=98434a62-d0b5-4596-bfde-5e5504998bc3.log
3842020-12-11T16:42:36  <jonasschnelli> I was just checking and came to the same conclusion luke-jr... hmm..
3852020-12-11T16:44:09  <luke-jr> jonasschnelli: if I fix my PR will it mess up your ability to troubleshoot?
3862020-12-11T16:44:17  <jonasschnelli> luke-jr: your right... so yes. bitcoinbuilds currently builds PR branches as they are (not merged on master)
3872020-12-11T16:44:41  <jonasschnelli> luke-jr: no. I'll know how to fix it.
3882020-12-11T16:44:43  <luke-jr> k
3892020-12-11T16:48:27  *** Bob_ <Bob_!~bob@93.55.242.131> has left #bitcoin-core-dev
3902020-12-11T16:49:43  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
3912020-12-11T16:49:56  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Client Quit)
3922020-12-11T16:50:38  *** kljasdfvv <kljasdfvv!~flack@p200300d46f24de007f9b1b51e45d0773.dip0.t-ipconnect.de> has quit IRC (Quit: Konversation terminated!)
3932020-12-11T16:52:21  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has joined #bitcoin-core-dev
3942020-12-11T16:54:16  *** Barno7 <Barno7!~bob@93.55.242.131> has joined #bitcoin-core-dev
3952020-12-11T16:55:06  *** harrigan <harrigan!~harrigan@ptr-93-89-242-235.ip.airwire.ie> has quit IRC (Ping timeout: 256 seconds)
3962020-12-11T16:55:13  <sipa> wumpus: treating the block data as an import sounds like a good idea, and not too hard
3972020-12-11T17:02:25  <achow101> wumpus: yes, the prepared statement for each batch is so that we don't have issues where there are multiple batches for the same database. Although I'm pretty sure that we never have multiple batches writing to the same db at the same time anyeays.
3982020-12-11T17:03:37  <wumpus> sipa: great!
3992020-12-11T17:04:06  *** mj_node <mj_node!~mj_node@122.0.25.130> has joined #bitcoin-core-dev
4002020-12-11T17:04:41  <wumpus> achow101: right, thanks
4012020-12-11T17:06:30  *** potato <potato!~Thunderbi@240d:1a:3d4:7d00:6195:91a4:15a2:31e8> has joined #bitcoin-core-dev
4022020-12-11T17:11:26  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has joined #bitcoin-core-dev
4032020-12-11T17:13:59  *** alko89 <alko89!~alko89@unaffiliated/alko89> has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
4042020-12-11T17:14:42  *** alko89 <alko89!~alko89@unaffiliated/alko89> has joined #bitcoin-core-dev
4052020-12-11T17:28:59  *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has quit IRC (Quit: freebsd 12.2 update)
4062020-12-11T17:31:29  *** Pavlenex <Pavlenex!~Thunderbi@141.98.103.251> has quit IRC (Quit: Pavlenex)
4072020-12-11T17:34:19  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
4082020-12-11T17:38:08  *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has joined #bitcoin-core-dev
4092020-12-11T17:38:27  <wumpus> rc3 binaries up https://bitcoincore.org/bin/bitcoin-core-0.21.0/test.rc3/
4102020-12-11T17:57:36  <achow101> for some reason gverify is giving a mismatch for darosior's 0.20.2rc1 win and osx builds, but manual inspection does not show a difference
4112020-12-11T17:57:58  <luke-jr> O.o
4122020-12-11T18:00:09  <luke-jr> achow101: where do you see it?
4132020-12-11T18:00:28  <achow101> locally
4142020-12-11T18:01:04  <luke-jr> I also see no difference
4152020-12-11T18:01:25  <roconnor> does gverify look at filename dates or other file attributes?
4162020-12-11T18:02:00  <darosior> Hmm, gverify passes on my end
4172020-12-11T18:02:16  <luke-jr> achow101: could the signature be invalid/rejected for some reason?
4182020-12-11T18:02:27  <achow101> luke-jr: i don't think so
4192020-12-11T18:02:33  <darosior> Oh no, it does not
4202020-12-11T18:02:46  <darosior> Not for Windows, good catch..
4212020-12-11T18:03:00  <achow101> roconnor: iirc it expects the same filenames
4222020-12-11T18:03:32  <luke-jr> oh, found it
4232020-12-11T18:03:37  *** samuel-pedraza <samuel-pedraza!a5169307@gateway/web/cgi-irc/kiwiirc.com/ip.165.22.147.7> has joined #bitcoin-core-dev
4242020-12-11T18:03:42  <luke-jr> - release: v0.20.2rc1-win-unsigned
4252020-12-11T18:03:47  <luke-jr> I bet tha'ts it
4262020-12-11T18:04:03  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
4272020-12-11T18:04:09  <achow101> ugh
4282020-12-11T18:04:20  <achow101> ok that doesn't matter
4292020-12-11T18:04:39  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
4302020-12-11T18:05:02  <prusnak> Big Sur says Bitcoin Core installed from bitcoin-0.21.0rc3-osx.dmg is broken and should be moved to the bin; that said this happens on M1 system, so maybe on x86-64 it's fine
4312020-12-11T18:05:13  <prusnak> maybe jonasschnelli can confirm? ^
4322020-12-11T18:05:33  <achow101> prusnak: is it the notarization warning?
4332020-12-11T18:05:57  <prusnak> no, it's different than the error with rc2 (which i assume was notarization error)
4342020-12-11T18:07:11  <luke-jr> where did the dmg come from?
4352020-12-11T18:07:23  <luke-jr> could we have ended up with a signature that doesn't match a non-deterministic bin?
4362020-12-11T18:08:28  <dongcarl> Just tested on x86_64, same error
4372020-12-11T18:08:50  <luke-jr> :/
4382020-12-11T18:09:02  <luke-jr> what about the unsigned dmg?
4392020-12-11T18:09:02  <achow101> any more detail on that error?
4402020-12-11T18:10:48  <dongcarl> https://nextcloud.carl.homeserver.net/s/WXdFHLwwTygXZW4
4412020-12-11T18:11:33  <luke-jr> dongcarl: what is that and why does it demand I run JS?
4422020-12-11T18:12:04  <dongcarl> Direct link: https://nextcloud.carl.homeserver.net/s/WXdFHLwwTygXZW4/preview
4432020-12-11T18:12:43  <dongcarl> It's just the screenshot of the error when I open from the DMG (didn't drag to Applications)
4442020-12-11T18:12:49  <luke-jr> does that question mark on the bottom left tell anything more?
4452020-12-11T18:13:27  <achow101> hmm, damaged?
4462020-12-11T18:13:47  <sipa> damaged could mean the signature check fails, i guess
4472020-12-11T18:13:50  <dongcarl> https://nextcloud.carl.homeserver.net/s/7YTDtmrgaqPCYde/preview
4482020-12-11T18:14:01  <dongcarl> code signing went wrong
4492020-12-11T18:14:05  <luke-jr> sounds like it
4502020-12-11T18:14:11  <luke-jr> dongcarl: can you verify the unsigned DMG is okay?
4512020-12-11T18:14:24  <luke-jr> ping jonasschnelli
4522020-12-11T18:14:48  <dongcarl> luke-jr: that's not uploaded yet I don't think
4532020-12-11T18:15:08  <dongcarl> Happy to test if anyone has one
4542020-12-11T18:15:15  <achow101> dongcarl: uploading it soon
4552020-12-11T18:17:02  <achow101> https://github.com/achow101/bitcoin/releases/tag/v0.21.0rc3
4562020-12-11T18:17:45  *** samuel-pedraza <samuel-pedraza!a5169307@gateway/web/cgi-irc/kiwiirc.com/ip.165.22.147.7> has quit IRC (Quit: Connection closed)
4572020-12-11T18:18:36  <dongcarl> achow101: works
4582020-12-11T18:19:21  <prusnak> same here (on m1 system)
4592020-12-11T18:20:02  <luke-jr> sounds like jonasschnelli signed the wrong file, or committed the wrong sig
4602020-12-11T18:20:19  <luke-jr> or our combining process is suddnely broke
4612020-12-11T18:21:20  <achow101> i think he signed the wrong file
4622020-12-11T18:23:33  <roconnor> maybe I should keep my nose out of this, but what are the consequences of signing the wrong thing?  AFAIK publish signatures are unrevokable.
4632020-12-11T18:23:43  <roconnor> *published
4642020-12-11T18:23:55  <dongcarl> who does the windows codesigning?
4652020-12-11T18:23:58  <achow101> me
4662020-12-11T18:24:09  <dongcarl> roconnor: You might be thinking of notarization instead of codesigning?
4672020-12-11T18:24:23  <roconnor> maybe.  What's the difference?
4682020-12-11T18:24:35  <achow101> roconnor: in theory there shouldn't be any consequences because code signing is only done to shut up os warnings.
4692020-12-11T18:24:46  <roconnor> ah.
4702020-12-11T18:24:56  <achow101> but in practice, it both doesn't do that anymore, and it does convey some level of "this software is trusted" because users don't quite understand why we do that
4712020-12-11T18:25:22  <luke-jr> maybe we should just stop signing then?
4722020-12-11T18:25:24  <achow101> the consequence is that the software that is signed is malicious and some user is tricked into signing it
4732020-12-11T18:25:39  <achow101> *using it
4742020-12-11T18:25:53  <roconnor> maybe these code signatures are revokable?
4752020-12-11T18:26:01  <luke-jr> roconnor: these days, to *actually* shut macOS up, you have to opt in to Apple privacy violations (it tells them every time the app is opened, and which app it was)
4762020-12-11T18:26:13  <achow101> luke-jr: code signing is required for notarization, and notarization is now the thing to make macOS not give a warning
4772020-12-11T18:26:34  <luke-jr> achow101: but notarization is problematic, and we don't do it
4782020-12-11T18:26:36  *** jonatack <jonatack!~jon@213.152.186.40> has quit IRC (Ping timeout: 240 seconds)
4792020-12-11T18:26:47  <luke-jr> so if the signature-alone is useless, why bother?
4802020-12-11T18:26:59  <achow101> iirc there's 2 levels of warnings, with signature alone being slightly less aggressive?
4812020-12-11T18:27:06  <achow101> and at least it still works on older macOS
4822020-12-11T18:27:17  <luke-jr> older macOS that we dropped support for? :P
4832020-12-11T18:27:55  <dongcarl> luke-jr: Without signing, the warning is much scarier... something like "unknown developer"
4842020-12-11T18:29:28  <achow101> I think there's an older macOS we support that doesn't do the notarization check
4852020-12-11T18:30:11  <luke-jr> anyway, jonas can prob fix this easily
4862020-12-11T18:30:13  *** harrigan <harrigan!~harrigan@ptr-93-89-242-235.ip.airwire.ie> has joined #bitcoin-core-dev
4872020-12-11T18:34:05  *** kexkey <kexkey!~kexkey@static-198-54-132-158.cust.tzulo.com> has joined #bitcoin-core-dev
4882020-12-11T18:45:08  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has quit IRC (Remote host closed the connection)
4892020-12-11T18:46:01  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has joined #bitcoin-core-dev
4902020-12-11T18:46:43  *** troygiorshev <troygiorshev!~troygiors@d67-193-140-136.home3.cgocable.net> has quit IRC (Ping timeout: 260 seconds)
4912020-12-11T18:50:21  *** jonatack <jonatack!~jon@37.170.61.94> has joined #bitcoin-core-dev
4922020-12-11T18:58:24  *** troygiorshev <troygiorshev!~troygiors@d67-193-140-136.home3.cgocable.net> has joined #bitcoin-core-dev
4932020-12-11T19:03:09  *** jonatack <jonatack!~jon@37.170.61.94> has quit IRC (Read error: Connection reset by peer)
4942020-12-11T19:08:03  <jonasschnelli> sorry.. was afk.
4952020-12-11T19:08:06  <jonasschnelli> Reading backlog
4962020-12-11T19:09:08  <jonasschnelli> I codesigned 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
4972020-12-11T19:10:10  *** chri_eb <chri_eb!~chris@gateway/tor-sasl/chrieb/x-28824719> has quit IRC (Quit: chri_eb)
4982020-12-11T19:13:21  <jonasschnelli> achow101, luke-jr: I just checked the DMG is built initially and it works, signature is correct
4992020-12-11T19:13:36  <jonasschnelli> I got 998dddf3c0f9b568fc0c39e61e3d61d2843dfb968016b7ceaf23aca94ace2542  bitcoin-osx-signed.dmg as the only one
5002020-12-11T19:15:23  *** yanmaani <yanmaani!~yanmaani@gateway/tor-sasl/yanmaani> has quit IRC (Ping timeout: 240 seconds)
5012020-12-11T19:15:45  <jonasschnelli> But yes,... 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599  bitcoin-osx-signed.dmg has an invalid code signature
5022020-12-11T19:15:47  *** yanmaani <yanmaani!~yanmaani@gateway/tor-sasl/yanmaani> has joined #bitcoin-core-dev
5032020-12-11T19:16:06  <jonasschnelli> but I signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
5042020-12-11T19:16:08  <jonasschnelli> let me try agai
5052020-12-11T19:17:48  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has joined #bitcoin-core-dev
5062020-12-11T19:20:26  <jonasschnelli> I double checked and I can confirm that I have signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
5072020-12-11T19:20:44  <achow101> is signing deterministic?
5082020-12-11T19:20:46  <jonasschnelli> The signature is not deterministic,... doing it again gives me a different file/hash
5092020-12-11T19:20:58  <jonasschnelli> I can't verify what went wrong
5102020-12-11T19:22:06  <jonasschnelli> I have verified the signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz and it works
5112020-12-11T19:22:46  <jonasschnelli> I don't know what to do,... I can sign and push again. But I rather know why it happend
5122020-12-11T19:23:29  <achow101> How can you check what was signed?
5132020-12-11T19:23:55  <jonasschnelli> I keep the files
5142020-12-11T19:24:44  <achow101> ah
5152020-12-11T19:25:20  <achow101> I guess for an rc it's fine to leave it, but in the future we should test the signed binary first
5162020-12-11T19:26:30  *** jonatack <jonatack!~jon@213.152.161.244> has joined #bitcoin-core-dev
5172020-12-11T19:26:46  <wumpus> we'll correct it for the next rc (or final) I guess, it's too bad the sig verification can't run in linux
5182020-12-11T19:26:58  <achow101> we can't verify the signature after signing?
5192020-12-11T19:27:49  <sipa> we have a linux based signing tool, right?
5202020-12-11T19:28:13  <achow101> only for windows afaik
5212020-12-11T19:28:14  <sipa> i'd be surprised if it doesn't have most of the code needed for verification too
5222020-12-11T19:28:20  <sipa> oh
5232020-12-11T19:28:55  <achow101> for windows we can (and do I think) verify the sig after signing
5242020-12-11T19:30:01  <sipa> so the osx signature is created on osx, but stapled onto the binary in gitian/linux ?
5252020-12-11T19:30:07  <sipa> *macos
5262020-12-11T19:31:27  <achow101> yes
5272020-12-11T19:33:02  <luke-jr> jonasschnelli: can you upload or compare 998dddf3c0f9b568fc0c39e61e3d61d2843dfb968016b7ceaf23aca94ace2542 with 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599 ?
5282020-12-11T19:33:15  <luke-jr> I guess our combiner is malfunctioning?
5292020-12-11T19:33:34  *** luke <luke!~luke@bitnomial/staff/luke> has joined #bitcoin-core-dev
5302020-12-11T19:34:41  <jonasschnelli> luke-jr: 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599  is here -> https://bitcoin.jonasschnelli.ch/gitian/build/360
5312020-12-11T19:39:19  <jonasschnelli> luke-jr: I don't have 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599
5322020-12-11T19:39:43  <jonasschnelli> my first gitian build went somehow wrong: https://bitcoin.jonasschnelli.ch/gitian/build/358
5332020-12-11T19:40:18  <jonasschnelli> the build log from the signer was actually a full macOS unsigned build: https://bitcoin.jonasschnelli.ch/gitian/builds/358/build_osx_signer.log (for whatever reason)
5342020-12-11T19:41:11  <jonasschnelli> I think the dmg hash mismatch has nothing to do with the invalid signature
5352020-12-11T19:46:25  <jonasschnelli> hmm...
5362020-12-11T19:47:04  <jonasschnelli> I think what I need to do before pushing the macOS signatures is doing a gbuild with gitian-descriptors/gitian-osx-signer.yml with the detached signature
5372020-12-11T19:47:13  <jonasschnelli> as sort of a dbl-check
5382020-12-11T19:51:55  <luke-jr> would still be nice to figure out why it didn't work this time :x
5392020-12-11T19:53:51  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
5402020-12-11T19:53:51  <bitcoin-git> [bitcoin] vova557 opened pull request #20628: Владелец (master...patch-10) https://github.com/bitcoin/bitcoin/pull/20628
5412020-12-11T19:53:52  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
5422020-12-11T19:54:25  *** greypw <greypw!~greypw@unaffiliated/greypw> has quit IRC (Quit: I'll be back!)
5432020-12-11T19:54:35  <jonasschnelli> luke-jr: Indeed. I'm still trying to figure that out.
5442020-12-11T19:54:49  *** greypw <greypw!~greypw@unaffiliated/greypw> has joined #bitcoin-core-dev
5452020-12-11T20:02:19  *** davec <davec!~davec@072-183-054-196.res.spectrum.com> has quit IRC (Quit: leaving)
5462020-12-11T20:03:45  *** miketwen_ <miketwen_!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Ping timeout: 240 seconds)
5472020-12-11T20:11:34  <jonasschnelli> one of the main problem is, that I currently can't test the detached signature before pushing it to the git repository
5482020-12-11T20:11:52  <jonasschnelli> I try now to modify the gitian descriptor to allow that
5492020-12-11T20:12:55  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
5502020-12-11T20:12:55  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #20628: Владелец (master...patch-10) https://github.com/bitcoin/bitcoin/pull/20628
5512020-12-11T20:12:56  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
5522020-12-11T20:13:07  <achow101> jonasschnelli: you can put it in gitian-builder/inputs/signature
5532020-12-11T20:13:11  <achow101> and make a local tag
5542020-12-11T20:13:51  <jonasschnelli> and set --commit signature=<tag> to that custom tag?
5552020-12-11T20:14:17  <achow101> i just set it to the actual version name
5562020-12-11T20:14:30  <achow101> and delete it later for the actual build
5572020-12-11T20:15:03  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
5582020-12-11T20:15:04  <jonasschnelli> but AFAIK the signer always pulls https://github.com/bitcoin-core/bitcoin-detached-sigs.git?
5592020-12-11T20:15:27  <achow101> it pulls but it shouldn't do anything
5602020-12-11T20:15:29  <jonasschnelli> and overwrites the git?
5612020-12-11T20:15:50  <achow101> it doesn't overwrite
5622020-12-11T20:15:54  <jonasschnelli> ah.
5632020-12-11T20:16:05  <jonasschnelli> I added now a part to the descriptor that allows putting a signatures.tar.gz in inputs
5642020-12-11T20:16:11  <jonasschnelli> and if there,... it'll take it
5652020-12-11T20:17:50  <luke-jr> jonasschnelli: -u bitcoin=/path/to/local/bitcoin
5662020-12-11T20:18:03  <luke-jr> tells gbuild where to pull from
5672020-12-11T20:18:11  <luke-jr> I always have ti pull from local
5682020-12-11T20:18:41  <jonasschnelli> luke-jr: https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-descriptors/gitian-osx-signer.yml#L11
5692020-12-11T20:18:58  <jonasschnelli> I guess it always pulls it from there.
5702020-12-11T20:19:02  <midnight> Oh nice, no changes necessary in the gitian signing env and I'm getting matches.
5712020-12-11T20:19:04  <luke-jr> not if you specify -u
5722020-12-11T20:19:21  <luke-jr>     -u, --url PAIRS                  comma separated list of DIRECTORY=URL pairs
5732020-12-11T20:19:32  <luke-jr> not very useful --help lol
5742020-12-11T20:19:38  <jonasschnelli> I do that for bitcoin=
5752020-12-11T20:19:53  <jonasschnelli> but wasn't aware you can also do it then for signature=
5762020-12-11T20:20:00  <luke-jr> it's no different ;)
5772020-12-11T20:20:16  <jonasschnelli> However,.. I'm going for the additional input file
5782020-12-11T20:20:20  <midnight> just me and wumpus I think with the windows sigs for the rc but
5792020-12-11T20:20:20  <jonasschnelli> Seems dummysave
5802020-12-11T20:20:21  * midnight shrug
5812020-12-11T20:21:05  <luke-jr> midnight: ?
5822020-12-11T20:21:29  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
5832020-12-11T20:21:32  <midnight> luke-jr: working nicely still, in spite of my semi-custom build environment.
5842020-12-11T20:21:47  <luke-jr> midnight: I see lots of windows sigs for 0.21.0rc3
5852020-12-11T20:21:59  <midnight> hrm
5862020-12-11T20:24:24  <jonasschnelli> midnight: I guess semi-custom is good. Better than if everyone uses the same host/env with the same script
5872020-12-11T20:24:44  <luke-jr> my attempt to semi-custom was met with mismatching
5882020-12-11T20:24:58  <luke-jr> but my idea of semi-custom was a ppc64le VM image :P
5892020-12-11T20:24:58  <jonasschnelli> have you figured out why?
5902020-12-11T20:25:02  <midnight> jonasschnelli: The maintenance of the custom(ish) build env is entirely on me, and I figure it's better to arrive at similar results this way.
5912020-12-11T20:25:19  <luke-jr> jonasschnelli: apparently the ppc64le compilers do not output the same objects as the x86_64 compilers, even cross compiling
5922020-12-11T20:25:32  <luke-jr> no idea why :/
5932020-12-11T20:25:59  <midnight> triangulation and all that.
5942020-12-11T20:26:24  <luke-jr> you would *think* i686-w64-mingw64-gcc (or whatever) would produce the same stuff on x86_64 and ppc64le, but apparently not
5952020-12-11T20:28:40  *** belcher <belcher!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 272 seconds)
5962020-12-11T20:30:03  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
5972020-12-11T20:30:03  <bitcoin-git> [bitcoin] dongcarl opened pull request #20629:  depends: Improve id string robustness  (master...2020-12-improve-depends-id-string) https://github.com/bitcoin/bitcoin/pull/20629
5982020-12-11T20:30:14  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
5992020-12-11T20:32:45  <jonasschnelli> i'm doing the 0.20.2 mac signatures asap (need to track-down the previous issue first)
6002020-12-11T20:44:10  *** belcher <belcher!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
6012020-12-11T20:46:44  <jonasschnelli> is there a way to speeup gitians "Upgrading system, may take a while (log in var/install.log)"?
6022020-12-11T20:46:50  <jonasschnelli> Re-create the base system?
6032020-12-11T20:46:57  <jonasschnelli> s/system/image
6042020-12-11T20:47:32  <sipa> iirc, yes
6052020-12-11T20:47:42  <sipa> creating an image after the update speeds things up
6062020-12-11T20:48:18  <jonasschnelli> the install part takes >10mins here.. annoying for testing
6072020-12-11T20:48:47  <wumpus> yes, the "Upgrading image" takes longer than the actual builds here
6082020-12-11T20:48:59  <jonasschnelli> ideed
6092020-12-11T20:49:04  <jonasschnelli> +n
6102020-12-11T20:49:42  <wumpus> I have no idea how to speed it up, I hacked it once to upgrade the base image but then somehow ended up with two versions of every package and non-deterministic results, so yea, if regenerating the base image solves it I'd definitely recommend that path
6112020-12-11T20:50:18  <jonasschnelli> Okay... I'll try that
6122020-12-11T20:50:37  <wumpus> fairly sure that works for one of {KVM, LXC}, don't know which one, the other always gets the ancient image
6132020-12-11T20:51:28  <wumpus> (they both generate the base image in completely different ways)
6142020-12-11T20:53:09  <wumpus> it would make sense to look into it some day but also guix builds are around the corner right
6152020-12-11T20:54:13  <sipa> Any Day Now(tm)
6162020-12-11T20:55:02  <wumpus> releases are sufficiently rare anyway, but the extra delays are mostly annoying when testing/iterating something
6172020-12-11T21:04:25  *** tylerchambers <tylerchambers!uid477594@gateway/web/irccloud.com/x-raafweykxwvatycu> has joined #bitcoin-core-dev
6182020-12-11T21:10:43  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has quit IRC (Ping timeout: 240 seconds)
6192020-12-11T21:11:59  <jonasschnelli> I'm really confused
6202020-12-11T21:12:21  <jonasschnelli> I have downloaded again 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-osx-unsigned.tar.gz ...
6212020-12-11T21:12:28  <jonasschnelli> ./detached-sig-create.sh -s "Bitcoin"
6222020-12-11T21:12:49  *** bitdex <bitdex!~bitdex@gateway/tor-sasl/bitdex> has joined #bitcoin-core-dev
6232020-12-11T21:12:49  <jonasschnelli> verified code-signature  codesign -v dist/Bitcoin-Qt.app
6242020-12-11T21:13:00  <jonasschnelli> uploaded signature.tar.gz
6252020-12-11T21:13:06  <jonasschnelli> did a gitian signer build
6262020-12-11T21:13:11  <jonasschnelli> downloaded the dmg
6272020-12-11T21:13:14  *** luke <luke!~luke@bitnomial/staff/luke> has quit IRC (Quit: sleep)
6282020-12-11T21:13:17  <jonasschnelli> verified the signature and I get:
6292020-12-11T21:13:27  <jonasschnelli> * /Volumes/Bitcoin-Core 4/Bitcoin-Qt.app: invalid signature (code or signature have been modified)
6302020-12-11T21:13:44  <achow101> huh. did apple change something recently?
6312020-12-11T21:14:31  <sipa> jonasschnelli: this is an old version?
6322020-12-11T21:14:37  <sipa> of bitcoin core?
6332020-12-11T21:14:58  <jonasschnelli> sipa: no.. still trying to figure out what happend for 0.21.0rc3
6342020-12-11T21:15:22  <jonasschnelli> AFAIK 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11 was the unsigned tarball (bitcoin-osx-unsigned.tar.gz)
6352020-12-11T21:16:03  <achow101> yep
6362020-12-11T21:16:22  <jonasschnelli> I redid everything and still getting invalid signatures
6372020-12-11T21:16:49  <jonasschnelli> Might it be possible that there is an edge-case with the signature size or the attaching of the signature?
6382020-12-11T21:20:11  <achow101> No issues on 10.15.7, so maybe just a big sur problem
6392020-12-11T21:21:24  <achow101> nvm, didn't click far enough
6402020-12-11T21:22:51  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
6412020-12-11T21:22:51  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #20630: Allow providing local signatures in gitian osx signer (master...2020/12/macos_gitian_signer) https://github.com/bitcoin/bitcoin/pull/20630
6422020-12-11T21:22:52  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
6432020-12-11T21:25:29  <jonasschnelli> Here is the gist (how I did it): https://gist.github.com/jonasschnelli/39f0b03981e1bbffff929f538cf139a9
6442020-12-11T21:25:44  <jonasschnelli> I have absolutely no clue why the signatures are wrong
6452020-12-11T21:28:03  <jonasschnelli> if anyone like to go down the rabbit hole:
6462020-12-11T21:28:28  <jonasschnelli> my local signed and detached signature directory: https://bitcoin.jonasschnelli.ch/bitcoin-osx-unsigned-nowsigned.zip
6472020-12-11T21:28:46  <jonasschnelli> ended up in build: https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed3.dmg
6482020-12-11T21:28:51  <jonasschnelli> (both files are in the gist above)
6492020-12-11T21:29:18  <jonasschnelli> maybe cfields has some insights?
6502020-12-11T21:30:02  <jonasschnelli> I need some sleep and try it again with the 0.20.2rc1 release (and see what happens there)
6512020-12-11T21:30:46  * jonasschnelli afk
6522020-12-11T21:34:37  *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has quit IRC (Quit: Going offline, see ya! (www.adiirc.com))
6532020-12-11T21:36:37  *** vincenzopalazzo <vincenzopalazzo!~vincent@2001:b07:6474:9d49:849d:db24:7f93:fb8a> has quit IRC (Quit: Leaving)
6542020-12-11T21:48:31  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:759a:a9e9:8a19:69d2> has joined #bitcoin-core-dev
6552020-12-11T22:00:07  <midnight> gah, xcode is a 7+ GB download.
6562020-12-11T22:00:20  <sipa> yes :(
6572020-12-11T22:03:33  <sipa> and we only need 47 MB from it
6582020-12-11T22:11:57  *** Talkless <Talkless!~Talkless@mail.dargis.net> has joined #bitcoin-core-dev
6592020-12-11T22:29:50  <achow101> interesting. there's a single byte difference between jonasschnelli signing result and his gitian result
6602020-12-11T22:34:49  <sipa> what byte?
6612020-12-11T22:35:43  <achow101> https://0bin.net/paste/NuOSAQ26#QxBJq7yNS7ynp6oMsXzvKQWT0vXTLYboU2QRfyBqu2d
6622020-12-11T22:36:39  <sipa> if you fix it, does it verify?
6632020-12-11T22:36:53  <achow101> yes, the original verifies
6642020-12-11T22:38:55  <sipa> what do you mean by signing result and gitian result?
6652020-12-11T22:40:13  <sipa> the output of the macos signing tool, and the result of gitian applying the extracted signature?
6662020-12-11T22:40:14  <achow101> the results from https://bitcoin.jonasschnelli.ch/bitcoin-osx-unsigned-nowsigned.zip which come from the signing process. it contains a fully signed binary
6672020-12-11T22:40:42  <sipa> got it
6682020-12-11T22:40:43  <achow101> the gitian result is https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed3.dmg which comes from gitian builder combining the unsigned with the signature.tar.gz from the above zip
6692020-12-11T22:40:55  <sipa> so the extraction/reattaching process doesn't result in the exact same binary
6702020-12-11T22:41:18  <achow101> yeah
6712020-12-11T22:43:09  <sipa> can you run "pagestuff -p" on both?
6722020-12-11T22:45:32  <achow101> identical
6732020-12-11T22:47:13  <sipa> so
6742020-12-11T22:48:38  <sipa> the detached-sig-create.sh tool uses pagestuff -p | tail -2 | grep offset | sed 's/[^0-9]*//g' to figure out the offset
6752020-12-11T22:49:17  <sipa> and the size of the sign file for the size
6762020-12-11T22:49:25  <achow101> the sizes are off by one byte
6772020-12-11T22:49:29  <sipa> aha
6782020-12-11T22:49:34  <sipa> that's what i was going to ask next
6792020-12-11T22:49:41  <sipa> i think this is the cause
6802020-12-11T22:50:14  <achow101> x86_64-apple-darwin-otool -l Bitcoin-Qt for segment "__LINKEDIT" says "vmsize 0x000000000007b000" but on the other it's "vmsize 0x000000000007c000"
6812020-12-11T22:50:14  <sipa> which one is bigger?
6822020-12-11T22:50:26  <achow101> gitian one is bigger
6832020-12-11T22:50:33  <achow101> wait no, that's backwards
6842020-12-11T22:50:38  <achow101> signed one is bigger, gitian is smaller
6852020-12-11T22:52:34  <sipa> that's a 4 kB difference though
6862020-12-11T22:52:36  <sipa> not 1 byte
6872020-12-11T22:53:18  <achow101> The files are the same size
6882020-12-11T22:53:30  <achow101> and diffoscope only sees the difference on that single byte
6892020-12-11T22:54:05  <sipa> what if you run the same on the unsigned gitian output (= the input to the macos signing tool)
6902020-12-11T22:54:57  <achow101> not sure what you mean
6912020-12-11T22:55:07  <sipa> x86_64-apple-darwin-otool -l Bitcoin-Qt
6922020-12-11T22:55:15  <sipa> oh
6932020-12-11T22:55:16  <sipa> nvm
6942020-12-11T22:56:32  <sipa> yeah, what if you run that on the unsigned Bitcoin-Qt binary?
6952020-12-11T22:57:53  <achow101> __LINKEDIT segment is the same size as the gitian result
6962020-12-11T22:58:48  <sipa> so
6972020-12-11T22:59:12  <sipa> the codesigning tool modified the __LINKEDIT segment, but applying the detached signature didn't do the same?
6982020-12-11T22:59:50  <achow101> seems so
6992020-12-11T23:01:56  <sipa> if you run "pagestuff -p Bitcoin-Qt | tail -2 | grep size | sed 's/[^0-9]*//g'", what do you get?
7002020-12-11T23:02:03  <sipa> on the various version
7012020-12-11T23:06:05  <achow101> ah wait, I was looking at the wrong file earlier
7022020-12-11T23:06:54  <sipa> where do the linux binaries pagestuff and codesign_allocate come from?
7032020-12-11T23:07:07  <achow101> the otool on the unsigned binary gives a way smaller __LINKEDIT, presumably because the code signature is involved in there
7042020-12-11T23:07:43  <sipa> i guess that's expected
7052020-12-11T23:08:09  <achow101> they come from one of the depends packages, not sure which
7062020-12-11T23:08:35  <sipa> ah i see
7072020-12-11T23:11:15  <achow101> the sizes from that pagestuff command are the same for both gitian and signed
7082020-12-11T23:11:28  <sipa> and the offsets too?
7092020-12-11T23:11:41  <achow101> not the same for unsigned, but that's expected because the section size it looks for doesn't exist as it's the code sig
7102020-12-11T23:11:54  <sipa> yeah
7112020-12-11T23:11:56  <achow101> same offests
7122020-12-11T23:14:45  <sipa> ok, so the byte that changed is in the MP_MACH_HEADERS section, in the very beginning of the file
7132020-12-11T23:14:50  <sipa> while the codesig is at the very end
7142020-12-11T23:15:16  <achow101> presumably that section is a table
7152020-12-11T23:21:56  <achow101> so the problem must be with codesign_allocate because that's what sets that size value
7162020-12-11T23:22:49  *** davterra <davterra!~davterra@107.182.237.18> has quit IRC (Quit: Leaving)
7172020-12-11T23:25:40  <sipa> the codesign tool has a --detached option to construct detached signatures directly
7182020-12-11T23:26:00  <sipa> why is the tooling doing signing directly, and then extracting the signatures from the result?
7192020-12-11T23:26:16  <sipa> this may be a reason for a difference
7202020-12-11T23:26:31  <achow101> this workflow probably was setup before that existed
7212020-12-11T23:27:27  <sipa> is it possible that the codesign_allocate tool is out of date?
7222020-12-11T23:28:08  <achow101> possibly
7232020-12-11T23:29:05  <sipa> what else does otool or anything else say about __LINKEDIT?
7242020-12-11T23:32:22  <achow101> nothing else afaict
7252020-12-11T23:35:30  *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 272 seconds)
7262020-12-11T23:38:17  <sipa> achow101: does macos come with a codesign_allocate tool too?
7272020-12-11T23:39:38  *** AmberJ_ <AmberJ_!~AmberJ_@178.239.168.171> has quit IRC (Remote host closed the connection)
7282020-12-11T23:40:26  <achow101> yes
7292020-12-11T23:40:30  <sipa> the native_cctools depends package is only a few months old, and most changes since have been for apple ARM stuff it seems
7302020-12-11T23:43:43  <sipa> achow101: if you run "codesign_allocate -i <unsigned Bitcoin-Qt filename> -a x86_64 225312 -o <some temp file>
7312020-12-11T23:43:49  <sipa> on macos
7322020-12-11T23:44:12  <sipa> and then inspect that temp file, does it have the correct __LINKEDIT vmsize?
7332020-12-11T23:44:23  <achow101> will try
7342020-12-11T23:46:16  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo_)
7352020-12-11T23:48:38  *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
7362020-12-11T23:48:44  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has quit IRC (Ping timeout: 258 seconds)
7372020-12-11T23:52:30  <achow101> It's the correct size
7382020-12-11T23:53:39  <sipa> hmm, we should try updating to the latest native_cctools package i guess and see if that fixes it
7392020-12-11T23:57:47  <sipa> an alternative is making jonasschnelli downgrade his codesign_allocate tool to one that's compatible with the native_ccttols we use, and then tell his codesign tool to use that
7402020-12-11T23:59:03  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
7412020-12-11T23:59:52  <achow101> I wonder if we're just running into some weird edge case now