12018-06-10T00:33:51  *** AaronvanW has joined #bitcoin-core-dev
  22018-06-10T00:38:56  *** AaronvanW has quit IRC
  32018-06-10T00:39:55  *** unholymachine has joined #bitcoin-core-dev
  42018-06-10T00:40:46  *** AaronvanW has joined #bitcoin-core-dev
  52018-06-10T00:42:49  *** promag has joined #bitcoin-core-dev
  62018-06-10T00:45:12  *** AaronvanW has quit IRC
  72018-06-10T01:01:20  *** promag has quit IRC
  82018-06-10T01:03:02  *** d9b4bef9 has quit IRC
  92018-06-10T01:04:08  *** d9b4bef9 has joined #bitcoin-core-dev
 102018-06-10T01:23:47  *** snickerfritz has quit IRC
 112018-06-10T01:24:10  *** snickerfritz has joined #bitcoin-core-dev
 122018-06-10T01:24:10  *** snickerfritz has quit IRC
 132018-06-10T01:24:10  *** snickerfritz has joined #bitcoin-core-dev
 142018-06-10T01:40:27  *** games_ has joined #bitcoin-core-dev
 152018-06-10T01:46:00  *** nmnkgl has joined #bitcoin-core-dev
 162018-06-10T01:50:25  *** nmnkgl has quit IRC
 172018-06-10T01:53:38  *** promag has joined #bitcoin-core-dev
 182018-06-10T01:58:45  *** promag has quit IRC
 192018-06-10T02:05:28  *** drexl has quit IRC
 202018-06-10T02:29:35  *** ohnx has quit IRC
 212018-06-10T02:37:48  *** Randolf has joined #bitcoin-core-dev
 222018-06-10T02:41:34  *** AaronvanW has joined #bitcoin-core-dev
 232018-06-10T02:46:41  *** AaronvanW has quit IRC
 242018-06-10T02:48:08  <gmaxwell> sipa: did you look at their implementation? https://github.com/armfazh/flo-shani-aesni/blob/master/sha256/flo-shani.c
 252018-06-10T02:52:48  <sipa> gmaxwell: yes, just interleaving
 262018-06-10T02:54:25  *** promag has joined #bitcoin-core-dev
 272018-06-10T02:58:27  *** promag has quit IRC
 282018-06-10T03:30:20  *** zivl_ has quit IRC
 292018-06-10T03:50:10  *** Chris_Stewart_5 has quit IRC
 302018-06-10T03:55:07  *** promag has joined #bitcoin-core-dev
 312018-06-10T03:59:47  *** promag has quit IRC
 322018-06-10T04:03:15  *** snickerfritz has quit IRC
 332018-06-10T04:11:29  *** unholymachine has quit IRC
 342018-06-10T04:40:52  *** Krellan has joined #bitcoin-core-dev
 352018-06-10T04:42:19  *** AaronvanW has joined #bitcoin-core-dev
 362018-06-10T04:46:57  *** AaronvanW has quit IRC
 372018-06-10T04:51:02  *** d9b4bef9 has quit IRC
 382018-06-10T04:52:16  *** d9b4bef9 has joined #bitcoin-core-dev
 392018-06-10T04:53:01  *** d9b4bef9 has quit IRC
 402018-06-10T04:53:37  *** dongcarl has joined #bitcoin-core-dev
 412018-06-10T04:54:16  *** d9b4bef9 has joined #bitcoin-core-dev
 422018-06-10T04:54:53  *** promag has joined #bitcoin-core-dev
 432018-06-10T04:55:02  *** d9b4bef9 has quit IRC
 442018-06-10T04:56:08  *** d9b4bef9 has joined #bitcoin-core-dev
 452018-06-10T04:57:01  *** d9b4bef9 has quit IRC
 462018-06-10T04:59:47  *** promag has quit IRC
 472018-06-10T05:00:08  *** d9b4bef9 has joined #bitcoin-core-dev
 482018-06-10T05:01:02  *** d9b4bef9 has quit IRC
 492018-06-10T05:02:07  *** d9b4bef9 has joined #bitcoin-core-dev
 502018-06-10T05:07:10  *** grafcaps has quit IRC
 512018-06-10T05:46:00  *** ilufang has joined #bitcoin-core-dev
 522018-06-10T06:08:29  *** bitconner has joined #bitcoin-core-dev
 532018-06-10T06:26:09  *** ctrlbreak has quit IRC
 542018-06-10T06:42:58  *** AaronvanW has joined #bitcoin-core-dev
 552018-06-10T06:44:34  *** ctrlbreak has joined #bitcoin-core-dev
 562018-06-10T06:47:21  *** AaronvanW has quit IRC
 572018-06-10T06:50:47  *** Krellan has quit IRC
 582018-06-10T06:51:53  *** Krellan has joined #bitcoin-core-dev
 592018-06-10T06:55:39  *** promag has joined #bitcoin-core-dev
 602018-06-10T06:59:47  *** promag has quit IRC
 612018-06-10T07:10:05  *** ilufang has quit IRC
 622018-06-10T07:13:10  *** vicenteH has quit IRC
 632018-06-10T07:17:18  *** grafcaps has joined #bitcoin-core-dev
 642018-06-10T07:18:18  *** ilufang has joined #bitcoin-core-dev
 652018-06-10T07:21:58  *** grafcaps has quit IRC
 662018-06-10T07:37:25  <ilufang> quit
 672018-06-10T07:37:27  *** ilufang has quit IRC
 682018-06-10T07:51:45  *** schnerch- has quit IRC
 692018-06-10T07:53:35  *** schnerchi has joined #bitcoin-core-dev
 702018-06-10T07:55:59  *** promag has joined #bitcoin-core-dev
 712018-06-10T08:00:28  *** promag has quit IRC
 722018-06-10T08:56:52  *** promag has joined #bitcoin-core-dev
 732018-06-10T09:01:49  *** promag has quit IRC
 742018-06-10T09:08:21  <provoostenator> sipa: thanks for the extra context. Maintaining a large dbcache is mainly useful during IBD, so can't the problem of reorgs be avoided by only doing the optimization for very deep blocks?
 752018-06-10T09:10:43  <provoostenator> And just in case, if during IBD an alternative set of headers is found that would trigger a deep reorg, you'd flush the cache and turn off the optimization, before switching to that new branch.
 762018-06-10T09:14:31  <provoostenator> Right now it seems that 500 MB < dbcache < 7000 MB is a performance dead zone. Though I can try tweaking #11658 to see where the diminishing returns are.
 772018-06-10T09:14:33  <gribble> https://github.com/bitcoin/bitcoin/issues/11658 | During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after by luke-jr · Pull Request #11658 · bitcoin/bitcoin · GitHub
 782018-06-10T09:25:49  *** Victorsueca has quit IRC
 792018-06-10T09:27:03  *** Victorsueca has joined #bitcoin-core-dev
 802018-06-10T09:30:30  *** promag has joined #bitcoin-core-dev
 812018-06-10T09:43:09  *** zivl has joined #bitcoin-core-dev
 822018-06-10T09:43:44  *** promag has quit IRC
 832018-06-10T09:49:34  *** promag has joined #bitcoin-core-dev
 842018-06-10T09:58:41  *** promag has quit IRC
 852018-06-10T09:58:57  *** promag has joined #bitcoin-core-dev
 862018-06-10T10:03:49  *** promag has quit IRC
 872018-06-10T10:05:07  *** Guyver2 has joined #bitcoin-core-dev
 882018-06-10T10:10:30  *** promag has joined #bitcoin-core-dev
 892018-06-10T10:14:16  *** promag has quit IRC
 902018-06-10T10:18:40  *** Empact has quit IRC
 912018-06-10T10:29:43  *** zeta has joined #bitcoin-core-dev
 922018-06-10T10:43:50  *** promag has joined #bitcoin-core-dev
 932018-06-10T10:51:51  *** cryptapus has quit IRC
 942018-06-10T11:01:50  *** promag has quit IRC
 952018-06-10T11:03:13  *** zeta has quit IRC
 962018-06-10T11:09:50  *** ExtraCrispy has joined #bitcoin-core-dev
 972018-06-10T11:16:07  *** AaronvanW has joined #bitcoin-core-dev
 982018-06-10T11:18:57  *** nullptr| has quit IRC
 992018-06-10T11:22:10  *** promag has joined #bitcoin-core-dev
1002018-06-10T11:25:21  *** nullptr| has joined #bitcoin-core-dev
1012018-06-10T11:28:01  *** drexl has joined #bitcoin-core-dev
1022018-06-10T11:31:22  *** promag has quit IRC
1032018-06-10T11:32:09  *** AaronvanW has quit IRC
1042018-06-10T11:39:26  *** grafcaps has joined #bitcoin-core-dev
1052018-06-10T11:42:36  *** AaronvanW has joined #bitcoin-core-dev
1062018-06-10T11:44:14  *** grafcaps has quit IRC
1072018-06-10T11:44:46  *** Sinclair6 has quit IRC
1082018-06-10T12:07:46  *** promag has joined #bitcoin-core-dev
1092018-06-10T12:14:31  *** promag has quit IRC
1102018-06-10T12:22:07  *** Victorsueca has quit IRC
1112018-06-10T12:22:38  *** AaronvanW has quit IRC
1122018-06-10T12:23:33  *** Victorsueca has joined #bitcoin-core-dev
1132018-06-10T12:29:12  *** gloata has joined #bitcoin-core-dev
1142018-06-10T12:31:51  *** ula has joined #bitcoin-core-dev
1152018-06-10T13:02:54  *** SopaXorzTaker has joined #bitcoin-core-dev
1162018-06-10T13:07:18  *** zivl has quit IRC
1172018-06-10T13:31:40  *** SopaXorzTaker has quit IRC
1182018-06-10T13:35:31  *** bitconner has quit IRC
1192018-06-10T13:37:19  *** promag has joined #bitcoin-core-dev
1202018-06-10T13:49:09  *** promag has quit IRC
1212018-06-10T14:27:05  *** Randolf has quit IRC
1222018-06-10T15:01:38  *** jcorgan has joined #bitcoin-core-dev
1232018-06-10T15:02:23  *** GoldenBear has joined #bitcoin-core-dev
1242018-06-10T15:06:20  *** promag has joined #bitcoin-core-dev
1252018-06-10T15:07:37  *** promag has quit IRC
1262018-06-10T15:15:42  *** grafcaps has joined #bitcoin-core-dev
1272018-06-10T15:20:01  *** grafcaps has quit IRC
1282018-06-10T15:34:27  *** grafcaps has joined #bitcoin-core-dev
1292018-06-10T15:37:12  *** cryptapus has joined #bitcoin-core-dev
1302018-06-10T15:37:13  *** cryptapus has quit IRC
1312018-06-10T15:37:13  *** cryptapus has joined #bitcoin-core-dev
1322018-06-10T15:47:32  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1332018-06-10T15:52:01  *** d9b4bef9 has quit IRC
1342018-06-10T15:53:16  *** d9b4bef9 has joined #bitcoin-core-dev
1352018-06-10T15:58:17  <provoostenator> Do I understand correctly that the only way for a coin cache entry to be dirty, is if the UTXO existed before the last flush and was spent since then? Would it be worth trying to bypass the cache in those cases and update the disk when spending a UTXO that's not in the cache?
1362018-06-10T16:01:45  <provoostenator> I wonder if OS's make any effort to optimize a write to the same physical place on disk that you just read from.
1372018-06-10T16:02:26  <sipa> provoostenator: it can be dirty because it's created after the last flush, or spent after the last flush while ot was created before
1382018-06-10T16:03:27  <sipa> and of course we can bypass the cache... if we don't care about the performance it offers
1392018-06-10T16:04:38  <sipa> provoostenator: i guess we could only do the background flushing during IBD, but that's still very scary
1402018-06-10T16:07:24  <provoostenator> "dirty because it's created after the last flush" - how does that work? I thought they always get the FRESH flag in that case.
1412018-06-10T16:08:03  <provoostenator> (I meant DIRTY flag, not dirty in general db terminology)
1422018-06-10T16:09:19  <provoostenator> Of course I do care about the performance impact of such a change. My working theory is that too many DITRY entries slows things down to a state that's worse than a smaller cache. So perhaps preventing accumulation of DIRTY entries would prevent that.
1432018-06-10T16:10:22  <provoostenator> (my "aggresive" pruning branch is much slower than master, despite the cache growing much bigger)
1442018-06-10T16:11:33  <provoostenator> I'm currently running IBD from block 320,000 - 480,000 on my iMac several times with decreasing dbcache (and once from genesis without interrupting) to see what happens.
1452018-06-10T16:18:29  <provoostenator> My hypothesis, based on what I've seen so far, is that when running from genesis to with "infinite" cache, going from 320K to 480K will be fastest. Followed by starting at 320K with infinite cache. A 3 GB cache will be slower, but a 500 MB cache will _faster_ than a 3 GB cache. Possibly regardless of pruning.
1462018-06-10T16:19:34  <gmaxwell> I think that would be very surprising.
1472018-06-10T16:20:00  <provoostenator> Indeed
1482018-06-10T16:24:12  <bitcoin-git> [bitcoin] ken2812221 opened pull request #13426: [WIP, bugfix] Add u8path and u8string to boost to fix #13103 (master...u8path_u8string) https://github.com/bitcoin/bitcoin/pull/13426
1492018-06-10T16:24:48  *** SopaXorzTaker has joined #bitcoin-core-dev
1502018-06-10T16:31:01  <sipa> provoostenator: FRESH implies DIRTY
1512018-06-10T16:31:46  <sipa> provoostenator: too many dirty entries slows things down... there may be a memory locality effect from just having many entries, but i don't see any way how dirtyness can impact that
1522018-06-10T16:32:02  <provoostenator> sipa: ah I see, so I should have said "DIRTY but not FRESH"
1532018-06-10T16:33:20  <provoostenator> Is there any sorting going on when entries are added?
1542018-06-10T16:33:30  *** Victorsueca has quit IRC
1552018-06-10T16:34:10  <sipa> no
1562018-06-10T16:34:14  <sipa> it's a hash table
1572018-06-10T16:34:41  <sipa> provoostenator: i meant to say "about too many dirty entries slowing things doen"
1582018-06-10T16:34:49  <sipa> i don't believe that can be the case
1592018-06-10T16:35:18  *** Victorsueca has joined #bitcoin-core-dev
1602018-06-10T16:39:45  <sipa> provoostenator: the time to flush itself may be proportional or worse to the number of dirty entries, though
1612018-06-10T16:42:17  <provoostenator> From what I saw on my AWS nodes, the pruning (which usually coincided with a cache flash) took just minutes and happened just a dozen or so times, on a IBD measured in days.
1622018-06-10T16:43:18  <sipa> right
1632018-06-10T16:43:25  <sipa> that seems expected
1642018-06-10T16:43:44  <provoostenator> So if an entry is not found int he cache, it starts walking through the disk looking for it? But there's no reason to assume that would be slower than without cache.
1652018-06-10T16:44:20  <sipa> of course disk will be slower than cache
1662018-06-10T16:44:57  <sipa> is it possible you're running into swap space?
1672018-06-10T16:45:27  <provoostenator> Amazon Ubuntu images don't have swap on by default, so I don't think so, but I already deleted those machine.
1682018-06-10T16:45:39  *** AaronvanW has joined #bitcoin-core-dev
1692018-06-10T16:49:14  <provoostenator> At least I can rule that out in this current experiment, since I have 48 GB RAM
1702018-06-10T16:50:52  *** promag has joined #bitcoin-core-dev
1712018-06-10T16:52:25  <provoostenator> When there's a cache, every time it calls CCoinsViewCache::FetchCoin it walks through the memory cache and if nothing is found walks through the disk cache. So there's potentially some duplicate effort, maybe that becomes a problem?
1722018-06-10T16:53:02  <provoostenator> Oh no, because it's a hash table, it's not walking, it just fetches it.
1732018-06-10T16:53:51  <provoostenator> The term "iterator" confused me there.
1742018-06-10T16:54:05  <sipa> yes
1752018-06-10T16:54:22  *** Chris_Stewart_5 has quit IRC
1762018-06-10T16:54:32  <sipa> and on disk, it just fetches from leveldb, which has indexes and other structure to guide the search - it's isn't really iterating either
1772018-06-10T16:56:19  <provoostenator> If a big cache causes a slowdown compared to a small cache, it has to be the in-memory stuff I would guess.
1782018-06-10T16:58:25  *** promag has quit IRC
1792018-06-10T17:02:24  *** promag has joined #bitcoin-core-dev
1802018-06-10T17:04:38  <sipa> how long does flushing take?
1812018-06-10T17:04:52  <sipa> it can be minutes even on high end systems for multi-gb caches
1822018-06-10T17:05:46  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1832018-06-10T17:07:29  <provoostenator> Minutes as far as I know, let me upload the logs...
1842018-06-10T17:10:51  *** Sinclair6 has joined #bitcoin-core-dev
1852018-06-10T17:11:22  *** Guyver2 has quit IRC
1862018-06-10T17:17:48  <provoostenator> https://ufile.io/tlvv3 (prune3000_sjors.log was the slowest, I gave up after 5 days)
1872018-06-10T17:20:14  *** promag has quit IRC
1882018-06-10T17:20:44  <provoostenator> TIL about OnionShare, so here you go: http://4nzykwc37ncqcwhp.onion/recall-shiftless
1892018-06-10T17:29:26  *** Chris_Stewart_5 has quit IRC
1902018-06-10T17:32:05  *** Krellan has quit IRC
1912018-06-10T17:33:01  *** Krellan has joined #bitcoin-core-dev
1922018-06-10T17:34:25  *** ossifrage has quit IRC
1932018-06-10T17:40:06  *** AaronvanW has quit IRC
1942018-06-10T17:42:58  *** ossifrage has joined #bitcoin-core-dev
1952018-06-10T17:44:07  *** promag has joined #bitcoin-core-dev
1962018-06-10T17:59:36  *** bitconner has joined #bitcoin-core-dev
1972018-06-10T18:02:09  *** CubicEarths has joined #bitcoin-core-dev
1982018-06-10T18:05:09  *** bitconner has quit IRC
1992018-06-10T18:07:32  *** Victorsueca has quit IRC
2002018-06-10T18:09:02  *** Victorsueca has joined #bitcoin-core-dev
2012018-06-10T18:20:09  *** SopaXorzTaker has quit IRC
2022018-06-10T18:21:29  *** justanotheruser has quit IRC
2032018-06-10T18:26:13  *** justanotheruser has joined #bitcoin-core-dev
2042018-06-10T18:30:05  *** AaronvanW has joined #bitcoin-core-dev
2052018-06-10T18:34:15  *** bitconner has joined #bitcoin-core-dev
2062018-06-10T19:02:30  *** bitconner has quit IRC
2072018-06-10T19:05:04  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2082018-06-10T19:05:58  *** Aaronvan_ has joined #bitcoin-core-dev
2092018-06-10T19:09:22  *** AaronvanW has quit IRC
2102018-06-10T19:27:34  *** zivl has joined #bitcoin-core-dev
2112018-06-10T19:41:40  *** Aaronvan_ has quit IRC
2122018-06-10T20:10:13  *** AaronvanW has joined #bitcoin-core-dev
2132018-06-10T20:18:35  *** AaronvanW has quit IRC
2142018-06-10T20:36:27  *** promag has quit IRC
2152018-06-10T20:53:32  *** bitconner has joined #bitcoin-core-dev
2162018-06-10T20:57:57  *** bitconner has quit IRC
2172018-06-10T21:04:02  *** promag has joined #bitcoin-core-dev
2182018-06-10T21:12:39  *** bitconner has joined #bitcoin-core-dev
2192018-06-10T21:13:11  *** opdenkamp has joined #bitcoin-core-dev
2202018-06-10T21:21:54  *** bitconner has quit IRC
2212018-06-10T21:22:08  *** bitconner has joined #bitcoin-core-dev
2222018-06-10T21:30:27  *** bitconner has quit IRC
2232018-06-10T21:31:26  *** bitconner has joined #bitcoin-core-dev
2242018-06-10T21:31:50  *** justanotheruser has quit IRC
2252018-06-10T21:44:46  *** bitconner has quit IRC
2262018-06-10T21:52:46  *** Empact has joined #bitcoin-core-dev
2272018-06-10T22:14:55  *** promag has quit IRC
2282018-06-10T22:23:01  *** d9b4bef9 has quit IRC
2292018-06-10T22:24:08  *** d9b4bef9 has joined #bitcoin-core-dev
2302018-06-10T22:24:21  *** drexl has quit IRC
2312018-06-10T22:26:55  *** bitconner has joined #bitcoin-core-dev
2322018-06-10T22:27:40  *** Kvaciral has quit IRC
2332018-06-10T22:31:27  *** bitconner has quit IRC
2342018-06-10T22:41:19  *** Kvaciral has joined #bitcoin-core-dev
2352018-06-10T22:43:11  *** AaronvanW has joined #bitcoin-core-dev
2362018-06-10T22:44:13  *** nuke_bloodaxe has quit IRC
2372018-06-10T22:44:55  *** bitconner has joined #bitcoin-core-dev
2382018-06-10T22:44:55  *** jimpo has quit IRC
2392018-06-10T22:46:30  *** nullptr| has quit IRC
2402018-06-10T22:48:12  *** nuke_bloodaxe has joined #bitcoin-core-dev
2412018-06-10T22:48:50  *** AaronvanW has quit IRC
2422018-06-10T22:49:02  *** promag has joined #bitcoin-core-dev
2432018-06-10T22:53:20  *** globalchain has joined #bitcoin-core-dev
2442018-06-10T22:53:28  *** jimpo has joined #bitcoin-core-dev
2452018-06-10T22:57:41  *** nullptr| has joined #bitcoin-core-dev
2462018-06-10T22:58:10  *** bitconner has quit IRC
2472018-06-10T22:58:28  *** bitconner has joined #bitcoin-core-dev
2482018-06-10T23:03:06  *** bitconner has quit IRC
2492018-06-10T23:05:20  *** drexl has joined #bitcoin-core-dev
2502018-06-10T23:08:35  *** Guest47913 is now known as eenoch
2512018-06-10T23:09:04  *** eenoch is now known as Guest99396
2522018-06-10T23:10:21  *** nullptr| has quit IRC
2532018-06-10T23:19:37  *** justanotheruser has joined #bitcoin-core-dev
2542018-06-10T23:21:41  *** bitconner has joined #bitcoin-core-dev
2552018-06-10T23:27:22  *** bitconner has quit IRC
2562018-06-10T23:28:24  *** AaronvanW has joined #bitcoin-core-dev
2572018-06-10T23:30:00  *** bitconner has joined #bitcoin-core-dev
2582018-06-10T23:34:06  *** OS-11936 has quit IRC
2592018-06-10T23:34:27  *** bitconner has quit IRC
2602018-06-10T23:35:09  <sipa> gmaxwell: i win
2612018-06-10T23:35:48  <sipa> intel's SSE4 sha256 code, transliterated to sse4 intrinsics... is 8% faster than the asm version
2622018-06-10T23:35:57  <sipa> (on a Ryzen system)
2632018-06-10T23:38:45  *** bitconner has joined #bitcoin-core-dev
2642018-06-10T23:43:52  *** promag has quit IRC
2652018-06-10T23:45:07  *** nullptr| has joined #bitcoin-core-dev
2662018-06-10T23:45:14  *** globalchain has quit IRC
2672018-06-10T23:49:30  <sipa> on i7 the intrinsics version is slightly slower (0.7% slower for long hashes, 1.5% slower for double-SHA256, 4$ slower for 32-byte hashes)
2682018-06-10T23:49:35  *** nullptr| has quit IRC
2692018-06-10T23:54:05  *** AaronvanW has quit IRC
2702018-06-10T23:55:09  *** AaronvanW has joined #bitcoin-core-dev