12018-09-21T00:01:11 *** AaronvanW has quit IRC
22018-09-21T00:01:48 *** AaronvanW has joined #bitcoin-core-dev
32018-09-21T00:02:31 *** pipp8 has quit IRC
42018-09-21T00:03:56 *** Murch has quit IRC
52018-09-21T00:06:10 *** AaronvanW has quit IRC
62018-09-21T00:11:51 *** AaronvanW has joined #bitcoin-core-dev
72018-09-21T00:16:29 *** VanCo has joined #bitcoin-core-dev
82018-09-21T00:16:38 *** AaronvanW has quit IRC
92018-09-21T00:16:41 *** lnostdal has joined #bitcoin-core-dev
102018-09-21T00:23:28 <gmaxwell> https://bitcoincore.org/en/2018/09/20/notice/
112018-09-21T00:25:12 <luke-jr> this seems way too premature; only 2% of the network's upgraded :/
122018-09-21T00:27:19 <gmaxwell> unfortunately, the issue was made public.
132018-09-21T00:27:30 <luke-jr> :/
142018-09-21T00:28:06 <gmaxwell> it wasn't clear if it would propagate or die out, but since it was moderatly easy to discover on your own, even just the rumors of it risked someone exploiting it.
152018-09-21T00:30:12 *** gribble has quit IRC
162018-09-21T00:31:29 <achow101> gmaxwell: luke-jr: there's an r/btc thread about someone finding the bug
172018-09-21T00:32:12 <gmaxwell> achow101: it's referenced in the message. 'circulating'
182018-09-21T00:34:08 <jamesob> Wow, that was fast
192018-09-21T00:36:35 *** jarthur has joined #bitcoin-core-dev
202018-09-21T00:40:52 <promag> jnewbery: fyi #14283
212018-09-21T00:41:05 *** gribble has joined #bitcoin-core-dev
222018-09-21T00:43:22 *** VanCo has quit IRC
232018-09-21T00:45:16 *** tryphe has quit IRC
242018-09-21T00:45:41 *** tryphe has joined #bitcoin-core-dev
252018-09-21T00:46:34 *** intcat has quit IRC
262018-09-21T00:46:34 *** VanCo has joined #bitcoin-core-dev
272018-09-21T00:48:16 *** intcat has joined #bitcoin-core-dev
282018-09-21T01:05:00 *** promag has quit IRC
292018-09-21T01:05:13 *** drexl has quit IRC
302018-09-21T01:10:43 *** Murch has joined #bitcoin-core-dev
312018-09-21T01:11:16 <gmaxwell> theymos has a new announcement up https://www.reddit.com/r/Bitcoin/comments/9hkoo6/new_info_escalates_importance_upgrading_to_0163/
322018-09-21T01:12:31 *** AaronvanW has joined #bitcoin-core-dev
332018-09-21T01:17:38 *** AaronvanW has quit IRC
342018-09-21T01:20:13 <gmaxwell> should I PR the extended test case?
352018-09-21T01:35:35 *** promag has joined #bitcoin-core-dev
362018-09-21T01:38:05 <luke-jr> gmaxwell: I'd wait
372018-09-21T01:39:59 <gmaxwell> OK.
382018-09-21T01:41:06 *** promag has quit IRC
392018-09-21T01:44:04 *** Chris_Stewart_5 has joined #bitcoin-core-dev
402018-09-21T01:45:18 *** tryphe has quit IRC
412018-09-21T01:45:40 *** tryphe has joined #bitcoin-core-dev
422018-09-21T01:50:09 *** VanCo has quit IRC
432018-09-21T01:56:05 <luke-jr> did anyone mail the announcement ML about 0.16.3? I don't think I saw one..
442018-09-21T01:56:54 *** Victorsueca has quit IRC
452018-09-21T01:56:57 *** Krellan has quit IRC
462018-09-21T01:58:03 <achow101> luke-jr: there was an announcement for it
472018-09-21T02:00:38 <aj> is bitcoin-core-dev the announcement list? https://lists.linuxfoundation.org/pipermail/bitcoin-core-dev/2018-September/000060.html
482018-09-21T02:00:51 <luke-jr> no
492018-09-21T02:00:57 <luke-jr> this one https://bitcoincore.org/en/list/announcements/join/
502018-09-21T02:03:34 <nanotube> would it make sense to propose a 'contact' page on bitcoin.org similar to the one on bitcoincore.org? it appears it is non-trivial to find where to privately report security issues unless one knows to go to bitcoincore.org, since earlz had to come asking on #bitcoin-dev and -core-dev for a way to report.
512018-09-21T02:03:54 *** Krellan has joined #bitcoin-core-dev
522018-09-21T02:04:21 *** Chris_Stewart_5 has quit IRC
532018-09-21T02:05:48 <gmaxwell> nanotube: :( I feel really uncomfortable with people going to bitcoin.org for that kind of information.
542018-09-21T02:05:49 <harding> nanotube: any page on Bitcoin.org, top menu, Participate, Development, "to report an issue, please see the Bug Reporting page", Responsible Disclosure.
552018-09-21T02:05:56 <gmaxwell> but its there
562018-09-21T02:05:57 <gmaxwell> yea
572018-09-21T02:08:18 *** Krellan has quit IRC
582018-09-21T02:11:26 <nanotube> yes not ideal, but people probably still go there unless they know to check bitcoincore.org... so, good that it has the bug reporting page in there somewhere.
592018-09-21T02:35:57 *** RubenSomsen has joined #bitcoin-core-dev
602018-09-21T02:45:16 *** tryphe has quit IRC
612018-09-21T02:45:40 *** tryphe has joined #bitcoin-core-dev
622018-09-21T02:55:07 *** promag has joined #bitcoin-core-dev
632018-09-21T02:58:39 <echeveria> there used to be bitcoin-security, but that was handled sort of poorly.
642018-09-21T02:59:07 <echeveria> the contents of it ended up being published when someone stole the old satoshi email address.
652018-09-21T02:59:27 *** promag has quit IRC
662018-09-21T03:11:30 *** promag has joined #bitcoin-core-dev
672018-09-21T03:13:20 *** AaronvanW has joined #bitcoin-core-dev
682018-09-21T03:15:47 *** promag has quit IRC
692018-09-21T03:18:41 *** AaronvanW has quit IRC
702018-09-21T03:37:29 *** promag has joined #bitcoin-core-dev
712018-09-21T03:41:43 *** promag has quit IRC
722018-09-21T03:45:16 *** tryphe has quit IRC
732018-09-21T03:45:41 *** tryphe has joined #bitcoin-core-dev
742018-09-21T03:45:51 *** shinohai has quit IRC
752018-09-21T04:00:08 *** gribble has quit IRC
762018-09-21T04:08:45 *** BlueMatt has quit IRC
772018-09-21T04:09:16 *** jnewbery has quit IRC
782018-09-21T04:13:45 *** promag has joined #bitcoin-core-dev
792018-09-21T04:14:08 *** BlueMatt has joined #bitcoin-core-dev
802018-09-21T04:17:04 <kanzure> hi. mailing list admin has hit a bug and is unusable at the moment.
812018-09-21T04:18:32 <kanzure> We're sorry, we hit a bug!
822018-09-21T04:18:33 <kanzure> Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs.
832018-09-21T04:21:31 *** promag has quit IRC
842018-09-21T04:22:27 <sipa> great.
852018-09-21T04:22:38 *** gribble has joined #bitcoin-core-dev
862018-09-21T04:32:35 *** paracyst_ has joined #bitcoin-core-dev
872018-09-21T04:32:58 *** paracyst has quit IRC
882018-09-21T04:33:41 <echeveria> I was looking at one of the public internet mapping tools for bitcoin core versions. there's a pretty disturbing number of hosts that have 8332 open.
892018-09-21T04:35:27 *** baldur has quit IRC
902018-09-21T04:36:56 <echeveria> is there some tool or setup guide that is telling people to open this port? I thought it was pretty difficult (not a single switch) to get Bitcoin Core to bind the RPC interface to 0.0.0.0.
912018-09-21T04:45:16 *** tryphe has quit IRC
922018-09-21T04:45:41 *** tryphe has joined #bitcoin-core-dev
932018-09-21T04:46:26 <echeveria> of 8000 IPv4 nodes, 1142 have a RPC port 8332 that respond to a SYN.
942018-09-21T04:48:00 *** bsm117532 has joined #bitcoin-core-dev
952018-09-21T04:48:16 *** baldur has joined #bitcoin-core-dev
962018-09-21T04:49:44 *** promag has joined #bitcoin-core-dev
972018-09-21T04:54:23 *** promag has quit IRC
982018-09-21T04:54:52 *** CubicEarth has quit IRC
992018-09-21T04:56:05 *** CubicEarth has joined #bitcoin-core-dev
1002018-09-21T04:59:56 *** jarthur has quit IRC
1012018-09-21T05:09:24 *** promag has joined #bitcoin-core-dev
1022018-09-21T05:10:17 <gmaxwell> maybe honeypots?
1032018-09-21T05:10:34 <gmaxwell> as you note, you must take extra steps to bind..
1042018-09-21T05:13:49 *** promag has quit IRC
1052018-09-21T05:45:16 *** tryphe has quit IRC
1062018-09-21T05:45:42 *** tryphe has joined #bitcoin-core-dev
1072018-09-21T05:52:03 <echeveria> I don't think so. they're over a huge number of different hosts, old and new.
1082018-09-21T05:53:45 *** promag has joined #bitcoin-core-dev
1092018-09-21T05:58:21 *** promag has quit IRC
1102018-09-21T06:03:46 *** promag has joined #bitcoin-core-dev
1112018-09-21T06:08:04 *** promag has quit IRC
1122018-09-21T06:14:40 *** promag has joined #bitcoin-core-dev
1132018-09-21T06:17:42 *** promag has quit IRC
1142018-09-21T06:20:22 *** StopAndDecrypt has quit IRC
1152018-09-21T06:22:11 *** StopAndDecrypt has joined #bitcoin-core-dev
1162018-09-21T06:25:15 *** Urgo has joined #bitcoin-core-dev
1172018-09-21T06:45:16 *** tryphe has quit IRC
1182018-09-21T06:45:41 *** tryphe has joined #bitcoin-core-dev
1192018-09-21T06:52:16 *** hebasto has joined #bitcoin-core-dev
1202018-09-21T07:01:34 *** Murch has quit IRC
1212018-09-21T07:04:45 *** ken2812221_ has joined #bitcoin-core-dev
1222018-09-21T07:06:25 <ken2812221_> MarcoFalke: Your gpg signing key has expired
1232018-09-21T07:07:33 *** nullptr| has quit IRC
1242018-09-21T07:15:02 *** nullptr| has joined #bitcoin-core-dev
1252018-09-21T07:16:02 *** AaronvanW has joined #bitcoin-core-dev
1262018-09-21T07:18:29 *** harrymm_ has quit IRC
1272018-09-21T07:20:36 *** AaronvanW has quit IRC
1282018-09-21T07:21:33 *** go1111111 has joined #bitcoin-core-dev
1292018-09-21T07:45:16 *** tryphe has quit IRC
1302018-09-21T07:45:41 *** tryphe has joined #bitcoin-core-dev
1312018-09-21T07:46:29 *** hebasto has quit IRC
1322018-09-21T07:48:21 *** ishvarts has joined #bitcoin-core-dev
1332018-09-21T07:55:13 *** Guyver2 has joined #bitcoin-core-dev
1342018-09-21T08:02:45 *** setpill has joined #bitcoin-core-dev
1352018-09-21T08:07:40 *** games_ has quit IRC
1362018-09-21T08:13:57 *** Victorsueca has joined #bitcoin-core-dev
1372018-09-21T08:33:05 *** promag has joined #bitcoin-core-dev
1382018-09-21T08:45:16 *** tryphe has quit IRC
1392018-09-21T08:45:29 *** promag has quit IRC
1402018-09-21T08:45:41 *** tryphe has joined #bitcoin-core-dev
1412018-09-21T08:46:27 *** AaronvanW has joined #bitcoin-core-dev
1422018-09-21T08:49:25 *** bralyclow has joined #bitcoin-core-dev
1432018-09-21T08:49:40 *** Guyver2 has quit IRC
1442018-09-21T08:51:34 *** justan0theruser has quit IRC
1452018-09-21T08:52:40 *** promag has joined #bitcoin-core-dev
1462018-09-21T09:22:08 *** rex4539 has quit IRC
1472018-09-21T09:26:04 *** rex4539 has joined #bitcoin-core-dev
1482018-09-21T09:42:11 *** timothy has joined #bitcoin-core-dev
1492018-09-21T09:45:16 *** tryphe has quit IRC
1502018-09-21T09:45:41 *** tryphe has joined #bitcoin-core-dev
1512018-09-21T09:56:21 *** promag has quit IRC
1522018-09-21T10:14:11 *** harrymm has joined #bitcoin-core-dev
1532018-09-21T10:45:16 *** tryphe has quit IRC
1542018-09-21T10:45:41 *** tryphe has joined #bitcoin-core-dev
1552018-09-21T11:01:41 *** drexl has joined #bitcoin-core-dev
1562018-09-21T11:05:24 *** rex4539 has quit IRC
1572018-09-21T11:15:03 *** ishvarts has quit IRC
1582018-09-21T11:22:02 *** rh0nj has quit IRC
1592018-09-21T11:23:07 *** rh0nj has joined #bitcoin-core-dev
1602018-09-21T11:45:16 *** tryphe has quit IRC
1612018-09-21T11:45:41 *** tryphe has joined #bitcoin-core-dev
1622018-09-21T11:50:54 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1632018-09-21T12:01:01 *** SopaXorzTaker has joined #bitcoin-core-dev
1642018-09-21T12:04:47 *** Jmabsd has joined #bitcoin-core-dev
1652018-09-21T12:05:19 <Jmabsd> the code that qualifies and validates a segwit transaction starts on what code locations?
1662018-09-21T12:10:26 <Jmabsd> mostly validation.cpp's AcceptToMemoryPoolWorker
1672018-09-21T12:21:43 <Jmabsd> Where is the code that checks the witness merkle root in the coinbase transaction?
1682018-09-21T12:23:51 *** SopaXorzTaker has quit IRC
1692018-09-21T12:34:03 *** SopaXorzTaker has joined #bitcoin-core-dev
1702018-09-21T12:36:34 *** setpill has quit IRC
1712018-09-21T12:36:43 *** rex4539 has joined #bitcoin-core-dev
1722018-09-21T12:45:16 *** tryphe has quit IRC
1732018-09-21T12:45:28 *** owowo has quit IRC
1742018-09-21T12:45:40 *** tryphe has joined #bitcoin-core-dev
1752018-09-21T12:48:44 *** rex4539 has quit IRC
1762018-09-21T12:50:40 *** owowo has joined #bitcoin-core-dev
1772018-09-21T13:04:27 *** promag has joined #bitcoin-core-dev
1782018-09-21T13:07:44 *** csknk has joined #bitcoin-core-dev
1792018-09-21T13:12:07 *** games_ has joined #bitcoin-core-dev
1802018-09-21T13:13:24 *** Sentineo has joined #bitcoin-core-dev
1812018-09-21T13:24:47 *** grubles_ has joined #bitcoin-core-dev
1822018-09-21T13:24:58 *** promag has quit IRC
1832018-09-21T13:27:13 *** grubles has quit IRC
1842018-09-21T13:45:19 *** tryphe has quit IRC
1852018-09-21T13:45:53 *** tryphe has joined #bitcoin-core-dev
1862018-09-21T13:48:52 *** voltron has joined #bitcoin-core-dev
1872018-09-21T13:54:04 *** hebasto has joined #bitcoin-core-dev
1882018-09-21T13:56:43 *** owowo has quit IRC
1892018-09-21T14:00:34 *** Krellan has joined #bitcoin-core-dev
1902018-09-21T14:01:48 *** owowo has joined #bitcoin-core-dev
1912018-09-21T14:02:27 *** bralyclow has quit IRC
1922018-09-21T14:03:06 *** bralyclow has joined #bitcoin-core-dev
1932018-09-21T14:05:04 *** Krellan has quit IRC
1942018-09-21T14:06:44 *** promag has joined #bitcoin-core-dev
1952018-09-21T14:07:27 *** rafalcpp has quit IRC
1962018-09-21T14:13:09 *** qinfengling has joined #bitcoin-core-dev
1972018-09-21T14:19:46 *** jarthur has joined #bitcoin-core-dev
1982018-09-21T14:21:47 *** rafalcpp has joined #bitcoin-core-dev
1992018-09-21T14:25:01 *** rh0nj has quit IRC
2002018-09-21T14:26:07 *** rh0nj has joined #bitcoin-core-dev
2012018-09-21T14:38:00 *** grubles_ has quit IRC
2022018-09-21T14:45:17 *** tryphe has quit IRC
2032018-09-21T14:45:43 *** tryphe has joined #bitcoin-core-dev
2042018-09-21T14:49:12 *** grubles has joined #bitcoin-core-dev
2052018-09-21T14:56:46 *** owowo has quit IRC
2062018-09-21T15:00:53 *** michaelsdunn1 has joined #bitcoin-core-dev
2072018-09-21T15:05:35 *** owowo has joined #bitcoin-core-dev
2082018-09-21T15:09:25 *** rex4539 has joined #bitcoin-core-dev
2092018-09-21T15:18:32 *** grubles has quit IRC
2102018-09-21T15:18:52 *** grubles has joined #bitcoin-core-dev
2112018-09-21T15:29:27 *** owowo has quit IRC
2122018-09-21T15:31:45 *** emilengler has joined #bitcoin-core-dev
2132018-09-21T15:33:13 <emilengler> If I download the linux tarball, will I be able to select a download path for the blockchain ?
2142018-09-21T15:33:59 <luke-jr> yes, if you know how or use the GUI
2152018-09-21T15:34:09 <luke-jr> better topic for #Bitcoin
2162018-09-21T15:34:31 <emilengler> Ok I will keep this in mind excuse me
2172018-09-21T15:34:57 *** owowo has joined #bitcoin-core-dev
2182018-09-21T15:40:12 *** emilengler has quit IRC
2192018-09-21T15:40:45 *** ExtraCrispy has joined #bitcoin-core-dev
2202018-09-21T15:45:17 *** tryphe has quit IRC
2212018-09-21T15:45:41 *** tryphe has joined #bitcoin-core-dev
2222018-09-21T16:04:07 *** Jmabsd has quit IRC
2232018-09-21T16:11:14 *** Murch has joined #bitcoin-core-dev
2242018-09-21T16:14:19 *** rex4539 has quit IRC
2252018-09-21T16:16:41 *** BashCo has joined #bitcoin-core-dev
2262018-09-21T16:18:27 *** mr_paz has joined #bitcoin-core-dev
2272018-09-21T16:23:15 *** jarthur has quit IRC
2282018-09-21T16:27:44 *** mr_paz_ has joined #bitcoin-core-dev
2292018-09-21T16:28:22 *** mr_paz has quit IRC
2302018-09-21T16:29:25 *** emilengler has joined #bitcoin-core-dev
2312018-09-21T16:35:15 *** Victorsueca has quit IRC
2322018-09-21T16:36:26 *** Victorsueca has joined #bitcoin-core-dev
2332018-09-21T16:45:17 *** tryphe has quit IRC
2342018-09-21T16:45:41 *** tryphe has joined #bitcoin-core-dev
2352018-09-21T17:18:04 *** voltron has quit IRC
2362018-09-21T17:29:59 *** timothy has quit IRC
2372018-09-21T17:31:52 <provoostenator> I used invalidateblock on a remote node to go back to ~ 475000, but lost the connection after a few hours. The last debug message is from an hour ago, an updatetip down to 508954. It's in a weird state.
2382018-09-21T17:32:32 <provoostenator> Memory usage was swinging between 5GB and 8GB. I was able to shut it down via rpc, though the last message was "net thread exit" which sounds like an unclean exit.
2392018-09-21T17:32:36 *** treebeardd has joined #bitcoin-core-dev
2402018-09-21T17:34:37 <provoostenator> Restarting the node, now it's "Replaying blocks", "Rolling back ... 542229" and down from there.
2412018-09-21T17:34:57 *** Sinclair_ has joined #bitcoin-core-dev
2422018-09-21T17:34:59 *** Sinclair6 has quit IRC
2432018-09-21T17:38:24 *** jnewbery has joined #bitcoin-core-dev
2442018-09-21T17:45:17 *** tryphe has quit IRC
2452018-09-21T17:45:41 *** tryphe has joined #bitcoin-core-dev
2462018-09-21T17:47:46 *** hebasto has quit IRC
2472018-09-21T17:59:13 <sdaftuar> provoostenator: what version bitcoind was it?
2482018-09-21T18:00:05 <provoostenator> sdaftuar: v0.17.0rc2 (I was actually dumb enough to not upgrade it before doing this)
2492018-09-21T18:01:38 <provoostenator> I also don't know if invalidateblock is supposed to work for such a huge rollback. Though if not, then perhaps the documentation should warn against that.
2502018-09-21T18:02:00 <sdaftuar> well, i think we do want it to work
2512018-09-21T18:02:22 <provoostenator> Also, the RPC call is blocking. Does getting disconnected have any bearing on that?
2522018-09-21T18:02:51 <sdaftuar> no, the invalidateblock function should continue even after the rpc client disconnects, i think
2532018-09-21T18:03:01 <sdaftuar> i believe if you had waited long enough, it probably would have finished?
2542018-09-21T18:03:07 <sdaftuar> but it might be several hours
2552018-09-21T18:03:10 <provoostenator> The logs also suggest it continued for about 30 minutes after disconnecting.
2562018-09-21T18:04:12 <provoostenator> It got about halfway in 2-3 hours, so indeed it looked like it would have made it.
2572018-09-21T18:04:12 <sdaftuar> disconnecting blocks is heavily disk-bound. when i last looked at it (on different hardware than i use today), i think i noticed i could disconnect on the order of 3-5 blocks/second, on average
2582018-09-21T18:04:47 <provoostenator> This is an iMac with SSD and plenty of memory.
2592018-09-21T18:05:00 <sdaftuar> we used to have an issue where the memory usage could grow sort of unbounded, as disconnected blocks would have their transactions added to the mempool
2602018-09-21T18:05:08 <sdaftuar> but that was fixed
2612018-09-21T18:05:23 <provoostenator> Yeah the weird thing I noticed is how dbcache kept growing as it was disconnecting.
2622018-09-21T18:05:25 <sdaftuar> but your comment about 5-8GB of memory has me slightly concerned
2632018-09-21T18:05:42 <provoostenator> The machine has 64 GB so it didn't run out.
2642018-09-21T18:05:51 <sdaftuar> what is -dbcache set to?
2652018-09-21T18:06:16 <provoostenator> 5000, so that's bad
2662018-09-21T18:06:49 *** itaseski has joined #bitcoin-core-dev
2672018-09-21T18:07:02 <provoostenator> The mempool is just the default, so that shouldn't have grown so much, right?
2682018-09-21T18:07:29 <sdaftuar> yeah assuming the code works correctly, the mempool's memory usage would have been bounded pretty well
2692018-09-21T18:08:56 <provoostenator> Last log entry had cache=4650.7MiB
2702018-09-21T18:09:18 <sdaftuar> oh so that seems good then
2712018-09-21T18:10:13 <provoostenator> The 5-8 GB RAM usage was an hour after the last log entry, when I reconnected, found through "top".
2722018-09-21T18:14:11 <sdaftuar> alright well maybe this is all expected (crappy) behavior. i don't know of any clever ideas to speed up block disconnection, unfortunately.
2732018-09-21T18:14:37 <sdaftuar> maybe someone could implement https://github.com/bitcoin/bitcoin/issues/8037
2742018-09-21T18:17:35 <provoostenator> Maybe invalidateblock could have a "don't bother adding to the mempool" option?
2752018-09-21T18:19:09 <provoostenator> I just noticed I have txindex=1, so that could be another issue.
2762018-09-21T18:19:34 <sdaftuar> provoostenator: yeah that's fair but i suspect it would still take hours
2772018-09-21T18:19:43 <gmaxwell> provoostenator: it's not clear to me what you're saying you saw
2782018-09-21T18:19:54 <gmaxwell> provoostenator: was it still rolling back when you stopped it?
2792018-09-21T18:20:34 <gmaxwell> if it was then it just sounds like expected behavior.
2802018-09-21T18:20:46 <provoostenator> gmaxwell: rolling back was after I restarted (it's still doing that now).
2812018-09-21T18:20:48 <gmaxwell> I've rolled back all the way to block 0 many times, though not recently.
2822018-09-21T18:21:02 <gmaxwell> provoostenator: yea, it'll keep going until it finishes.
2832018-09-21T18:21:14 <kanzure> mailing list bug has been reoslved; can someone send the post mortem link to the mailing list subscribers plzkthx? like https://bitcoincore.org/en/2018/09/20/notice/
2842018-09-21T18:21:30 <sipa> kanzure: didn't BlueMatt send one?
2852018-09-21T18:21:45 <provoostenator> Before I disconnected (a few hours ago) it was doing "UpdateTip: new best ..." in reverse order, as expcted from doing invalidateblock
2862018-09-21T18:21:55 <kanzure> i don't see one in the mod queue
2872018-09-21T18:22:03 <kanzure> and i don't see it on https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-September/thread.html
2882018-09-21T18:22:15 <provoostenator> The logs shows it kept doing that for 30 mins after I disconnected from the machine.
2892018-09-21T18:23:01 <provoostenator> When I logged back into the machine, bitcoind was still running, using 5-8 GB of RAM (it was actually going up and down in the space of minutes), but log wasn't updating. I then stopped it via rpc and restarted.
2902018-09-21T18:23:25 <provoostenator> So it seems it was still doing _something_, despite not logging.
2912018-09-21T18:23:57 <luke-jr> FYI, I still didn't get anything for 0.6.3/CVE from https://bitcoincore.org/en/list/announcements/join/ yet
2922018-09-21T18:24:02 <gmaxwell> provoostenator: what you're seeing without the logs is the atomic flush roll forward probably.
2932018-09-21T18:24:07 <luke-jr> sipa: ^ since you are one of the 3 who can apparently send those
2942018-09-21T18:24:16 <sdaftuar> provoostenator: gmaxwell: it does seem surprising i guess that an unclean shutdown happened?
2952018-09-21T18:24:27 <sdaftuar> how would that be possible if you just use rpc to stop the node?
2962018-09-21T18:25:25 <provoostenator> gmaxwell: what is a "atomic flush roll forward"?
2972018-09-21T18:25:55 <sdaftuar> provoostenator: on startup, we detect if the utxo state wasn't finished being written as of what we think our tip is.
2982018-09-21T18:26:09 <sdaftuar> in that situation, we have a rollback / rollforward mechanism to fix the utxo
2992018-09-21T18:26:39 <sdaftuar> by disconnecting blocks that are no longer on our chain, and replaying the blocks that might need applying to the utxo state
3002018-09-21T18:27:18 <sdaftuar> that should only happen after an unclean shutdown though
3012018-09-21T18:27:45 <gmaxwell> not to change subject but anyone know what this is? https://www.reddit.com/r/Bitcoin/comments/9hrusk/orhpan_blocks/e6e4zhk/?context=3
3022018-09-21T18:28:14 <gmaxwell> sdaftuar: indeed. I missed that the shutdown was supposted to be clean.
3032018-09-21T18:33:57 *** Guyver2 has joined #bitcoin-core-dev
3042018-09-21T18:38:23 <provoostenator> gmaxwell: that warning is triggered here: https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/AbstractBlockChain.java#L584
3052018-09-21T18:45:17 *** tryphe has quit IRC
3062018-09-21T18:45:42 *** tryphe has joined #bitcoin-core-dev
3072018-09-21T18:45:42 *** Victorsueca has quit IRC
3082018-09-21T18:45:50 <provoostenator> "duplicate block" seems to mean that it already processed it, nothing to do with orphans.
3092018-09-21T18:46:26 *** rafalcpp has quit IRC
3102018-09-21T18:46:56 *** Victorsueca has joined #bitcoin-core-dev
3112018-09-21T18:48:44 *** rafalcpp has joined #bitcoin-core-dev
3122018-09-21T18:53:35 <kanzure> *poke* postmortem email plzkthx
3132018-09-21T19:01:24 *** BashCo has quit IRC
3142018-09-21T19:06:24 *** rex4539 has joined #bitcoin-core-dev
3152018-09-21T19:09:06 *** BashCo has joined #bitcoin-core-dev
3162018-09-21T19:12:24 *** jarthur has joined #bitcoin-core-dev
3172018-09-21T19:12:42 <provoostenator> Ok, so roll back made it to 509650 and all seems well. Except the node seems to have forgotten I invalidated block 485000, because it jumped right into IBD and is moving forward again.
3182018-09-21T19:16:09 <provoostenator> I guess that's because the node doesn't check the full block index at launch.
3192018-09-21T19:19:05 <provoostenator> Restarted, now using v0.17.0rc4, doing nanother invalidateblock. Memory usage is almost 2GB higher than cache= shown in the logs, and seems to outpace it.
3202018-09-21T19:19:18 <provoostenator> I've also turned off the index.
3212018-09-21T19:21:20 <gmaxwell> what are the actual dirty page counts?
3222018-09-21T19:21:51 <gmaxwell> we recently realized that OS cached pages in mmaped files show up in res.
3232018-09-21T19:22:15 <provoostenator> Also, it's still going even though bitcoin-cli stop said it would stop. I'll look at the dirty page counts...
3242018-09-21T19:22:46 <provoostenator> (note to self: do not google "top dirty pages")
3252018-09-21T19:23:09 <sipa> lol
3262018-09-21T19:23:47 <gmaxwell> oh sorry, pmap -x $(pidof bitcoind) | tail -n 1 | tr -s ' ' | cut -d' ' -f 4
3272018-09-21T19:26:48 <provoostenator> macOS doesn't have pmap, but vmmap gives me this summary: https://gist.github.com/Sjors/6b01711ccd0f96128c7db5230c85ae8f
3282018-09-21T19:27:10 *** SopaXorzTaker has quit IRC
3292018-09-21T19:28:37 <provoostenator> Also a long list of "mapped file", e.g. many "locks/index/*.ldb"
3302018-09-21T19:29:03 <gmaxwell> k, so ~2GB of your resident size is mapped files.
3312018-09-21T19:29:31 <gmaxwell> whats your dbcache setting?
3322018-09-21T19:30:00 <provoostenator> dbcache=5000 MB, the log currently says cache=2300 MiB, so that part makes sense?
3332018-09-21T19:30:55 <provoostenator> It's the just the other 6 GB that needs explaining. Memory usage is now 10 GB. 38 more and the machine is going to OOM, which I'm not going to allow.
3342018-09-21T19:32:31 <provoostenator> MALLOC_TINY is now at 8.7, so that seems to be the thing that's mooning.
3352018-09-21T19:36:00 <provoostenator> (actually this is still v0.17.0rc2, sorry, though hopefully that doesn't matter here)
3362018-09-21T19:36:11 *** owowo has quit IRC
3372018-09-21T19:36:53 <provoostenator> (no, it is v0.17.0rc4)
3382018-09-21T19:37:58 <provoostenator> kill has no effect, kill -9 did
3392018-09-21T19:45:17 *** tryphe has quit IRC
3402018-09-21T19:45:44 *** tryphe has joined #bitcoin-core-dev
3412018-09-21T19:46:50 <provoostenator> Getting fairly consistent behavior now, even with disablewallet=1. bitcoin-cli stop seems to stop the RPC server, but not the invalidation process. Curious if anyone can reproduce. I'll let it sync to the tip before trying again.
3422018-09-21T19:47:48 <sipa> if invalidateblock does not succeed, its state isn't writte
3432018-09-21T19:48:05 <sipa> it first disconnects the blocks, and then marks them as invalid
3442018-09-21T19:48:21 <sipa> so at startup they will be connected again if bitcoind was killed in the middle
3452018-09-21T19:50:13 <provoostenator> That makes sense. I wonder if it matter that I was essentially interrupting IBD with that invalidateblock call. Memory usage seemed way worse than what I saw earlier today.
3462018-09-21T19:50:49 *** owowo has joined #bitcoin-core-dev
3472018-09-21T19:50:59 <sipa> invalidateblock also keeps a list of transactions to re-add to the mempool after the invalidation completes
3482018-09-21T19:51:09 <sipa> i assume that's the memory usage you see
3492018-09-21T19:51:56 <provoostenator> Is it also not abortable once in progress?
3502018-09-21T19:52:02 <sipa> no
3512018-09-21T19:53:21 <provoostenator> Ok, so in that case the way to roll back a long way would be to do it in smaller increments.
3522018-09-21T19:54:36 <sipa> right
3532018-09-21T19:54:39 <sipa> that should work
3542018-09-21T19:55:09 <gmaxwell> sipa: the mempool usage is limited.
3552018-09-21T19:55:49 <sipa> gmaxwell: how?
3562018-09-21T19:58:49 <gmaxwell> https://github.com/bitcoin/bitcoin/pull/9208
3572018-09-21T20:01:11 <sipa> gmaxwell: it doesn't look like DisconnectedBlockTransactions enforces any memory limits
3582018-09-21T20:01:41 <gmaxwell> MAX_DISCONNECTED_TX_POOL_SIZE
3592018-09-21T20:02:04 <sipa> oh
3602018-09-21T20:02:14 <provoostenator> There's also this open issue: #9027
3612018-09-21T20:02:16 <gribble> https://github.com/bitcoin/bitcoin/issues/9027 | Unbounded reorg memory usage · Issue #9027 · bitcoin/bitcoin · GitHub
3622018-09-21T20:02:51 <sipa> yup
3632018-09-21T20:03:01 <sipa> i was expecting the code to be elsewhere, my bad
3642018-09-21T20:05:46 <provoostenator> I take great pride in doing stupid things that lead to a new release candidate, so hopefully you'll find something :-)
3652018-09-21T20:08:06 <gmaxwell> provoostenator: are you running txindex?
3662018-09-21T20:08:36 <provoostenator> No, I did the first time today, but turned that off in more recent attempts.
3672018-09-21T20:10:11 *** rex4539 has quit IRC
3682018-09-21T20:11:57 *** rex4539 has joined #bitcoin-core-dev
3692018-09-21T20:13:49 <provoostenator> Re incremental approach: I rolled back ~10,000 blocks using about 13GB of RAM, cache=2200 at the peak. Sounds like it's holding all transactions in memory.
3702018-09-21T20:16:47 <provoostenator> But then it gets weird. ERROR: AcceptToMemoryPoolWorker: Consensus::CheckTxInputs: ... bad-txns-premature-spend-of-coinbase, tried to spend coinbase at depth 92
3712018-09-21T20:17:11 <provoostenator> InvalidChainFound: invalid block [the block I invalidated]
3722018-09-21T20:17:15 <gmaxwell> thats normal.
3732018-09-21T20:17:31 <provoostenator> Yeah, but then it starts syncing again.
3742018-09-21T20:19:03 *** rex4539 has quit IRC
3752018-09-21T20:19:43 *** rex4539 has joined #bitcoin-core-dev
3762018-09-21T20:22:30 <provoostenator> Ok, now I think I destroyed my chain :-) At boot: "assertion failed: (!setBlockIndexCandidates.empty()), function PruneBlockIndexCandidates, file validation.cpp, line 2547"
3772018-09-21T20:32:59 <sipa> provoostenator: i found the issue
3782018-09-21T20:33:05 <sipa> it's specific to InvalidateBlock
3792018-09-21T20:33:50 <provoostenator> sipa: nice!
3802018-09-21T20:38:25 *** emilengler has quit IRC
3812018-09-21T20:41:21 <provoostenator> sipa: is it because disconnectpool holds on to transactions which reference a shared_ptr<CBlock> pblock, so those don't get deallocated?
3822018-09-21T20:41:29 <sipa> provoostenator: no
3832018-09-21T20:42:51 *** JackH has joined #bitcoin-core-dev
3842018-09-21T20:44:34 <sipa> the event queue holds on to the shared_ptr<CBlock> objects in callbacks to DisconnectedBlock
3852018-09-21T20:44:52 <sipa> and InvalidateBlock doesn't limit the size of the queue
3862018-09-21T20:46:39 <sipa> provoostenator: could you check whether this issue also occurs when Rewinding?
3872018-09-21T20:47:00 <sipa> create a 0.13.0 node, sync it to tip, and then upgrade to 0.17+
3882018-09-21T20:47:15 <sipa> i suspect it is, and if that's the case, i would consider it a release blocker
3892018-09-21T20:48:10 <provoostenator> That's the rewind that happens if you upgrade a non-segwit node to a segwit node?
3902018-09-21T20:48:15 <sipa> yup
3912018-09-21T20:49:10 <provoostenator> I'll give it a try this weekend or early next week. Getting a bit late here. Maybe someone else gets to it first.
3922018-09-21T20:49:21 <sipa> thanks!
3932018-09-21T20:50:14 <provoostenator> I'm not looking forward to doing another release notes for 0.14 and 0.15 backports :-)
3942018-09-21T21:03:49 *** treebeardd has quit IRC
3952018-09-21T21:23:36 *** rafaeltokyo has joined #bitcoin-core-dev
3962018-09-21T21:24:29 *** rafaeltokyo has left #bitcoin-core-dev
3972018-09-21T21:28:00 *** rafaeltokyo has joined #bitcoin-core-dev
3982018-09-21T21:28:08 *** rafaeltokyo has left #bitcoin-core-dev
3992018-09-21T21:40:10 *** justan0theruser has joined #bitcoin-core-dev
4002018-09-21T21:42:30 <MarcoFalke> About #14289, was it ever supported to call invalidateblock on a block very far back?
4012018-09-21T21:42:31 <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
4022018-09-21T21:43:02 <sipa> MarcoFalke: i would say no, but it'd be a nice-to-have if it worked
4032018-09-21T21:43:21 <sipa> having invalidateblock 100000 blocks deep use a massive amount of memory is not a blocker, i think
4042018-09-21T21:43:54 <MarcoFalke> Ok, that was my impression because every time I tried that it would deadlock the node a bit until I got impatient and CTRL+C out
4052018-09-21T21:44:22 <MarcoFalke> If that is supported with reasonable memory gurantees, we should add a test/benchmark so it doesn't randomly regress
4062018-09-21T21:45:06 <MarcoFalke> Also my key is de-expired, but I am having issues uploading it to keyservers.
4072018-09-21T21:45:17 *** tryphe has quit IRC
4082018-09-21T21:45:37 <MarcoFalke> All of them return some obscure proxy error or timeout or ...
4092018-09-21T21:45:43 *** tryphe has joined #bitcoin-core-dev
4102018-09-21T21:46:30 <gmaxwell> MarcoFalke: yes worked fine since 9208.
4112018-09-21T21:46:52 <gmaxwell> the rpc will disconnect, because the rpc timeout isn't long enough for it to finish, but a node will happly work its way back to block 0.
4122018-09-21T21:48:55 *** Victorsueca has quit IRC
4132018-09-21T21:50:11 *** Victorsueca has joined #bitcoin-core-dev
4142018-09-21T22:02:32 <Murch> MarcoFalke: Luckily Keyservers may soonish be a thing of the past: https://wiki.gnupg.org/WKD
4152018-09-21T22:27:07 *** michaelsdunn1 has quit IRC
4162018-09-21T22:37:49 *** Guyver2 has quit IRC
4172018-09-21T22:45:17 *** tryphe has quit IRC
4182018-09-21T22:45:43 *** tryphe has joined #bitcoin-core-dev
4192018-09-21T22:47:18 *** lnostdal has quit IRC
4202018-09-21T22:49:08 *** JackH has quit IRC
4212018-09-21T22:55:23 *** csknk has quit IRC
4222018-09-21T23:14:51 *** jarthur has quit IRC
4232018-09-21T23:32:19 *** ken2812221__ has joined #bitcoin-core-dev
4242018-09-21T23:33:28 *** gribble has quit IRC
4252018-09-21T23:33:39 *** adam3us has quit IRC
4262018-09-21T23:35:18 *** rafalcpp has quit IRC
4272018-09-21T23:35:18 *** promag has quit IRC
4282018-09-21T23:35:18 *** ken2812221_ has quit IRC
4292018-09-21T23:35:39 *** rafalcpp has joined #bitcoin-core-dev
4302018-09-21T23:36:33 *** adam3us has joined #bitcoin-core-dev
4312018-09-21T23:39:31 *** gribble has joined #bitcoin-core-dev
4322018-09-21T23:45:17 *** tryphe has quit IRC
4332018-09-21T23:45:41 *** tryphe has joined #bitcoin-core-dev
4342018-09-21T23:45:41 *** r8921039 has joined #bitcoin-core-dev
4352018-09-21T23:49:39 *** r8921039 has quit IRC