12018-08-26T00:03:12  *** meshcollider_ has quit IRC
  22018-08-26T00:09:21  *** plankers has quit IRC
  32018-08-26T00:15:30  <gmaxwell> sipa: ah that might have been why I didn't bother trying to make it interpolate-- the seeks are in an index.
  42018-08-26T00:22:58  *** Giszmo has quit IRC
  52018-08-26T00:25:23  *** plankers has joined #bitcoin-core-dev
  62018-08-26T00:29:05  *** Giszmo has joined #bitcoin-core-dev
  72018-08-26T00:30:22  *** Krellan has quit IRC
  82018-08-26T00:46:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  92018-08-26T00:53:46  *** Chris_Stewart_5 has quit IRC
 102018-08-26T01:31:27  *** plankers has quit IRC
 112018-08-26T02:34:15  *** Jmabsd has joined #bitcoin-core-dev
 122018-08-26T02:38:00  <jamesob> sipa gmaxwell: mem usage increase is for sure the leveldb change (https://i.imgur.com/yMHJI37.png) and sipa's change didn't change the elevated usage
 132018-08-26T02:39:06  <sipa> jamesob: so it's still elevated with my patch?
 142018-08-26T02:39:41  <sipa> or it's not elevated anymore?
 152018-08-26T02:40:07  <jamesob> still elevated :(
 162018-08-26T02:41:41  <gmaxwell> might want to make a microbenchmar to figure out if its mmap behavior or something  else in leveldb.
 172018-08-26T02:42:10  <gmaxwell> also because it'll be easier to expirement (e.g. see if MAP_NORESERVE does anything) when we don't have to wait 5 hours.
 182018-08-26T02:42:43  <gmaxwell> just a loop to mmap all the files in a directory then sleep 30 seconds should test the earlier hypothesis.
 192018-08-26T02:42:43  *** Krellan has joined #bitcoin-core-dev
 202018-08-26T02:43:13  <jamesob> gmaxwell: yup, agree
 212018-08-26T02:43:22  <jamesob> will write if I have time tomorrow
 222018-08-26T02:50:39  *** murrayn has joined #bitcoin-core-dev
 232018-08-26T03:12:08  *** BashCo has quit IRC
 242018-08-26T03:13:04  *** BashCo has joined #bitcoin-core-dev
 252018-08-26T03:16:04  *** phwalkr has joined #bitcoin-core-dev
 262018-08-26T03:20:04  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 272018-08-26T03:22:01  *** d9b4bef9 has quit IRC
 282018-08-26T03:28:07  *** d9b4bef9 has joined #bitcoin-core-dev
 292018-08-26T03:28:17  *** Rootsudo has joined #bitcoin-core-dev
 302018-08-26T03:30:54  *** plankers has joined #bitcoin-core-dev
 312018-08-26T03:38:20  *** plankers has quit IRC
 322018-08-26T03:46:35  *** vexbuy_ has quit IRC
 332018-08-26T03:47:13  *** vexbuy has joined #bitcoin-core-dev
 342018-08-26T03:51:33  *** vexbuy has quit IRC
 352018-08-26T03:53:57  <ossifrage> It seems like MADV_RANDOM is useful if there is less locality to the reads then the size of the prefetch window, but I was under the impression that the prefetch is triggered by the pagefault and not by the mmap() so you actually have to do reads to get the extra pages
 362018-08-26T04:06:24  *** Giszmo has quit IRC
 372018-08-26T04:17:57  *** Giszmo has joined #bitcoin-core-dev
 382018-08-26T04:29:09  *** murrayn has quit IRC
 392018-08-26T04:29:59  *** sipa has quit IRC
 402018-08-26T04:30:12  *** sipa has joined #bitcoin-core-dev
 412018-08-26T04:31:53  *** Krellan has quit IRC
 422018-08-26T04:37:40  *** plankers has joined #bitcoin-core-dev
 432018-08-26T04:42:15  *** vexbuy has joined #bitcoin-core-dev
 442018-08-26T04:45:14  *** Jmabsd has quit IRC
 452018-08-26T04:46:38  *** murrayn has joined #bitcoin-core-dev
 462018-08-26T04:48:25  *** BashCo has quit IRC
 472018-08-26T04:49:45  *** BashCo has joined #bitcoin-core-dev
 482018-08-26T04:53:55  *** phwalkr has quit IRC
 492018-08-26T04:54:42  *** vexbuy has quit IRC
 502018-08-26T05:14:47  <luke-jr> was there a point to https://github.com/bitcoin-core/leveldb/pull/19 considering we limit leveldb to 1000 open files anyway?
 512018-08-26T05:17:15  *** vexbuy has joined #bitcoin-core-dev
 522018-08-26T05:19:18  <sipa> luke-jr: mmap'ed files aren't kept open after creating the mapping
 532018-08-26T05:19:31  <sipa> so they don't count towards the 1000 files limit iirc
 542018-08-26T05:24:10  <luke-jr> sipa: but we increased the file limit to 1000 *because* mmaps didn't count toward the fd limit
 552018-08-26T05:24:28  <luke-jr> the limit I'm talking about is max_open_files, not ulimit
 562018-08-26T05:32:01  *** Taifta has quit IRC
 572018-08-26T05:35:50  <sipa> luke-jr: yes, but there is a separate mmap limit it seems
 582018-08-26T05:51:01  *** TheHoliestRoger has quit IRC
 592018-08-26T05:53:49  *** TheHoliestRoger has joined #bitcoin-core-dev
 602018-08-26T05:59:22  *** TheHoliestRoger has quit IRC
 612018-08-26T06:02:10  *** TheHoliestRoger has joined #bitcoin-core-dev
 622018-08-26T06:02:14  *** Chris_Stewart_5 has quit IRC
 632018-08-26T06:02:58  *** ken2812221 has joined #bitcoin-core-dev
 642018-08-26T06:03:28  *** Krellan has joined #bitcoin-core-dev
 652018-08-26T06:04:27  *** Krellan_ has joined #bitcoin-core-dev
 662018-08-26T06:05:22  *** nOgAnOo_ has quit IRC
 672018-08-26T06:06:39  *** d_t has joined #bitcoin-core-dev
 682018-08-26T06:07:38  *** Giszmo has quit IRC
 692018-08-26T06:08:18  *** Krellan has quit IRC
 702018-08-26T06:13:02  <luke-jr> sipa: max_open_files effectively limits the number of mmaps too
 712018-08-26T06:13:29  <luke-jr> so it seems like the LevelDB mmap limit change is effectively a no-op since we keep the 1000 max_open_files limit :x
 722018-08-26T06:14:59  <gmaxwell> luke-jr: it's very much not a no op change.
 732018-08-26T06:15:16  <luke-jr> gmaxwell: how does it have an effect?
 742018-08-26T06:15:43  <gmaxwell> it causes more mmaps to be open, and less files. you can see this easily with lsof.
 752018-08-26T06:16:06  <luke-jr> the mmap limit is enforced by NewRandomAccessFile, which is only ever called by TableCache which enforces the max_open_files limit :/
 762018-08-26T06:16:41  <gmaxwell> Well I'm not telling you what the source code says, I'm telling you what it does. :P
 772018-08-26T06:16:45  <luke-jr> is there a bug in the LRU cache stuff? :/
 782018-08-26T06:18:25  <luke-jr> gmaxwell: is there an easy way to reproduce what you observe?
 792018-08-26T06:19:45  <gmaxwell> yea, start a node up no connect, with and without the change and lsof it and look at the number of mmaped files. you might need to crank checkblocks up a bit.
 802018-08-26T06:21:20  *** Giszmo has joined #bitcoin-core-dev
 812018-08-26T06:32:09  *** plankers has quit IRC
 822018-08-26T06:33:32  *** Giszmo has quit IRC
 832018-08-26T06:35:52  *** BashCo has quit IRC
 842018-08-26T06:36:50  *** BashCo has joined #bitcoin-core-dev
 852018-08-26T06:46:04  *** plankers has joined #bitcoin-core-dev
 862018-08-26T06:53:42  *** SopaXorzTaker has joined #bitcoin-core-dev
 872018-08-26T06:57:30  *** luke-jr has quit IRC
 882018-08-26T06:57:45  *** Victorsueca has quit IRC
 892018-08-26T06:58:55  *** Victorsueca has joined #bitcoin-core-dev
 902018-08-26T07:03:31  *** vexbuy has quit IRC
 912018-08-26T07:05:02  *** d9b4bef9 has quit IRC
 922018-08-26T07:05:34  *** BashCo has quit IRC
 932018-08-26T07:06:08  *** d9b4bef9 has joined #bitcoin-core-dev
 942018-08-26T07:06:19  *** BashCo has joined #bitcoin-core-dev
 952018-08-26T07:09:30  *** plankers has quit IRC
 962018-08-26T07:10:43  *** Taifta has joined #bitcoin-core-dev
 972018-08-26T07:11:33  *** BashCo has quit IRC
 982018-08-26T07:14:12  *** BashCo has joined #bitcoin-core-dev
 992018-08-26T07:15:14  *** luke-jr has joined #bitcoin-core-dev
1002018-08-26T07:15:22  <wumpus> 0.17.0rc2 binaries up https://bitcoincore.org/bin/bitcoin-core-0.17.0/test.rc2/
1012018-08-26T07:19:56  *** vexbuy has joined #bitcoin-core-dev
1022018-08-26T07:23:38  *** d_t has quit IRC
1032018-08-26T07:37:25  *** vexbuy has quit IRC
1042018-08-26T07:48:33  *** vexbuy has joined #bitcoin-core-dev
1052018-08-26T07:50:03  *** BashCo has quit IRC
1062018-08-26T07:50:16  <wumpus> luke-jr: the only reason that the change to num files was acceptable is because leveldb, at least the version in our tree, doesn't actually keep more files open (on platforms with mmap) see also https://github.com/bitcoin/bitcoin/pull/12495#issuecomment-377228329
1072018-08-26T07:50:45  *** BashCo has joined #bitcoin-core-dev
1082018-08-26T07:53:03  <luke-jr> gmaxwell: cannot reproduce with -checkblocks=10000 :/
1092018-08-26T07:53:44  <luke-jr> wumpus: I understand that. I just don't understand why increase the mmap-specific limit, without increasing the num files limit that is required for any mmap
1102018-08-26T07:53:58  <luke-jr> we will never have more than 1001 mmaps because the files limit is 1000
1112018-08-26T08:04:27  <wumpus> --I don't think mappings into closed files count against the open files ulimit
1122018-08-26T08:06:39  *** vexbuy has quit IRC
1132018-08-26T08:07:17  *** vexbuy has joined #bitcoin-core-dev
1142018-08-26T08:07:51  *** Rootsudo has quit IRC
1152018-08-26T08:09:58  <sipa> wumpus: luke's point is that internally in leveldb, the mmapped files count towards the open file limit
1162018-08-26T08:10:17  <wumpus> but we increased leveldb's limit
1172018-08-26T08:11:11  <wumpus> though to be honest I'm completely confused now with all the different limites, how they interact, and how they're supposed to interact
1182018-08-26T08:11:34  <sipa> wumpus: yeah...
1192018-08-26T08:12:09  <wumpus> given all the confusion about this, I'm starting to think it might be better to revert these changes, I like understanding things even if it means not getting every last possible % of performance
1202018-08-26T08:12:19  <sipa> so there is a configurable leveldb max open files limit (which is 1000), and a mmap limit (which is hardcoded to 1000, but we raised it to 4096)
1212018-08-26T08:12:47  <wumpus> and seemingly it now relies on all kinds of undocumented behavior
1222018-08-26T08:12:59  <sipa> luke says (i haven't verified) that mmapped files count towards that limit... which is inconsistent with our observations
1232018-08-26T08:13:17  <sipa> i'm just clarifying my understanding as there seems to be some confusion
1242018-08-26T08:13:18  *** Krellan_ has quit IRC
1252018-08-26T08:13:27  <wumpus> thanks
1262018-08-26T08:13:45  <gmaxwell> right if we revert though we should revert both changes.
1272018-08-26T08:13:51  <wumpus> yes
1282018-08-26T08:13:57  <gmaxwell> I wouldn't oppose doing that for 0.17.
1292018-08-26T08:13:59  <sipa> both? what other chanfe?
1302018-08-26T08:14:01  *** Krellan has joined #bitcoin-core-dev
1312018-08-26T08:14:10  <sipa> i saw the mmap increaee
1322018-08-26T08:14:11  <gmaxwell> sipa: the fd count change and the mmap limit change.
1332018-08-26T08:15:30  <gmaxwell> first FD count was increased with an argument that basically it wouldn't use more FDs due to using mmap.  But then we found out that actually when it hit the mmap limit, it started actually using more FDs.
1342018-08-26T08:15:47  <wumpus> #12495 and https://github.com/bitcoin-core/leveldb/pull/19
1352018-08-26T08:15:50  <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke · Pull Request #12495 · bitcoin/bitcoin · GitHub
1362018-08-26T08:16:05  <gmaxwell> Since the lack of mmaps is probably also bad for performance (though I don't think anyone benchmarked specifically for that), it seemed like it made sense to increase mmaps rather than revert the fd count change.
1372018-08-26T08:16:19  <wumpus> yes
1382018-08-26T08:16:47  <wumpus> "By default LevelDB will only mmap() up to 1000 ldb files for reading and then fall back
1392018-08-26T08:16:50  <wumpus> to using file desciptors."
1402018-08-26T08:17:00  <wumpus> that pull 19 is *supposed* to change that
1412018-08-26T08:17:29  <wumpus> (from 1000 to 4096, which is still an arbitrary value) -- maybe luke-jr is using a system leveldb?
1422018-08-26T08:17:45  <sipa> where did we increase the FDs?
1432018-08-26T08:18:02  <wumpus> didn't that happen in #12495?
1442018-08-26T08:18:04  <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke · Pull Request #12495 · bitcoin/bitcoin · GitHub
1452018-08-26T08:18:14  <wumpus> I don't know anymore, sorry, should probably get coffee
1462018-08-26T08:18:53  <gmaxwell> wumpus: 12495, as you say.
1472018-08-26T08:19:05  <sipa> ah yes
1482018-08-26T08:19:09  <sipa> hmm
1492018-08-26T08:19:20  <sipa> we were using 64 files only before
1502018-08-26T08:19:36  <sipa> but using 1000 there seems pretty standard practice; it's leveldb's own default
1512018-08-26T08:19:51  <luke-jr> wumpus: not for this testing
1522018-08-26T08:19:55  <wumpus> sipa: correct
1532018-08-26T08:20:21  <gmaxwell> sipa: after increasing it, we were using a lot more file descriptors.
1542018-08-26T08:20:23  <luke-jr> wumpus: but in any case, for testing this, I'm changing the 4096 back to 1000, and can't get LDB to use a fd
1552018-08-26T08:20:39  <gmaxwell> luke-jr: are you using some system leveldb?
1562018-08-26T08:20:59  <luke-jr> gmaxwell: like I said, not for this; but in any case, it would be the same 1000 mmaps
1572018-08-26T08:21:47  <gmaxwell> luke-jr: dunno why it didn't reproduce for you, I saw leveldb using FDs, after using 1000 maps... and not anymore after increasing the map limit.
1582018-08-26T08:21:55  <luke-jr> hmm
1592018-08-26T08:22:02  <gmaxwell> sipa also saw it using fd's on his node.
1602018-08-26T08:22:09  <luke-jr> testnet or mainnet?
1612018-08-26T08:22:13  <gmaxwell> mainnet
1622018-08-26T08:22:14  <sipa> mainnet
1632018-08-26T08:22:19  <luke-jr> I wonder if that's why. :x
1642018-08-26T08:22:21  <gmaxwell> testnet probably doesn't have enough ldb files.
1652018-08-26T08:22:26  <luke-jr> ah
1662018-08-26T08:22:49  <sipa> how large is testnet's utxo set?
1672018-08-26T08:24:53  *** PatBoy has quit IRC
1682018-08-26T08:26:12  *** PatBoy has joined #bitcoin-core-dev
1692018-08-26T08:35:52  <wumpus> "disk_size": 1025042445, as of height 1384188 (it is at 1392776 now, don't know how much it changed since then)
1702018-08-26T08:48:45  <luke-jr> hopefully I won't "damage" my mainnet .bitcoin trying this.. :x
1712018-08-26T08:49:05  <luke-jr> (I still run 0.13 wallets from time to time)
1722018-08-26T08:50:34  *** plankers has joined #bitcoin-core-dev
1732018-08-26T08:50:41  <luke-jr> looks like the UTXO db upgrade isn't triggering it, at least
1742018-08-26T08:55:24  *** plankers has quit IRC
1752018-08-26T09:04:44  * luke-jr facepalms. After UTXO upgrade, it says I need to reindex >_<
1762018-08-26T09:09:22  *** plankers has joined #bitcoin-core-dev
1772018-08-26T09:10:00  <wumpus> does it say why / give an error message? the whole point of the utxo upgrade would be to not reindex, unless it fails
1782018-08-26T09:11:54  <provoostenator> If you're traveling and stuck with a slow laptop, the following gist lets you compile remotely and then connect via SSH tunnel: https://gist.github.com/Sjors/fda9b601e9464f61332ac6490f487c0a (improvements welcome)
1792018-08-26T09:26:16  <wumpus> provoostenator: nit: you could pass NO_UPNP to the depends build instead of --with-miniupnpc=no
1802018-08-26T09:26:32  <wumpus> not that libminiupnp takes long to compile
1812018-08-26T09:27:10  <provoostenator> wumpus: thanks, in this case it's probably easier to keep the configure line flexible.
1822018-08-26T09:27:17  *** murrayn has quit IRC
1832018-08-26T09:28:07  <provoostenator> I first tried copying bitcoind over and just running it locally but that resulted in a world of pain, probably because the two systems aren't perfectly identical.
1842018-08-26T09:28:12  *** plankers has quit IRC
1852018-08-26T09:28:46  <wumpus> yes copying executables between linux systems is fraught with problems
1862018-08-26T09:29:02  <provoostenator> It's works fine with c-lightning though.
1872018-08-26T09:29:16  <wumpus> though it should work with depends and the glibc compat enabled
1882018-08-26T09:29:53  <provoostenator> I tried with depends as well, haven't tried glibc compat, might try that next time. But this also prevents me from accidentally syncing the chain over 4G.
1892018-08-26T09:30:03  <wumpus> -static-libstdc++ helps as well
1902018-08-26T09:30:24  <wumpus> (but only if all c++ dependencies are static, which is the case with a depends build but not otherwise)
1912018-08-26T09:32:17  <wumpus> then you end up with an executable that is as-good-as-static and it should be portable to non-ancient linux versions with the same CPU architecture
1922018-08-26T09:33:14  <wumpus> of course, that's the theory, there are still some things that can interfere for example if glibc is built with different options (I vaguely remember redhat and some security/hardening flag), or the target platform has a different libc
1932018-08-26T09:35:17  <wumpus> this is one reason they came up with "snap"s which are executables that essentially carry around part of the system dependencies with them in a container
1942018-08-26T09:35:39  <wumpus> (along with some security/isolation features)
1952018-08-26T09:35:45  <provoostenator> Being able to run locally is especially useful for QT, and with the multiprocess stuff it would be nice to run bitcoind remote and QT locally.
1962018-08-26T09:36:31  <wumpus> aanyhow if your two systems have the same release of the same distro, and are at the same update level, it should work too, that's probably easiest for most...
1972018-08-26T09:50:05  <luke-jr> wumpus: I reflink'd the blocks/ and chainstate/ dirs to a new datadir. I'm guessing that simply wasn't sufficient somehow
1982018-08-26T09:50:30  *** plankers has joined #bitcoin-core-dev
1992018-08-26T09:52:52  *** murrayn has joined #bitcoin-core-dev
2002018-08-26T09:54:34  *** BashCo has quit IRC
2012018-08-26T09:54:57  <luke-jr> oh well, I'll let it sync overnight and see if it hits the breakpoint
2022018-08-26T09:55:12  <luke-jr> env_posix.cc:379 if anyone else who can reproduce it wants to try
2032018-08-26T09:55:36  <luke-jr> (if you hit it, maybe see if you can work out why it got there..)
2042018-08-26T09:56:03  *** BashCo has joined #bitcoin-core-dev
2052018-08-26T10:18:13  *** SopaXorzTaker has quit IRC
2062018-08-26T10:20:57  *** Krellan has quit IRC
2072018-08-26T10:21:43  *** Krellan has joined #bitcoin-core-dev
2082018-08-26T10:33:31  *** as1nc has quit IRC
2092018-08-26T10:46:30  *** JackH has joined #bitcoin-core-dev
2102018-08-26T11:01:42  *** vexbuy has quit IRC
2112018-08-26T11:14:37  *** vexbuy has joined #bitcoin-core-dev
2122018-08-26T11:23:18  <provoostenator> Any volunteer for making a PR based on this commit? https://github.com/bitcoin/bitcoin/commit/a9db3dada0119c183d16627805e90c4dbca05c6a
2132018-08-26T11:23:46  <provoostenator> I left it out of my rebase #13937 because I'd rather not touch thread stuff.
2142018-08-26T11:23:47  <gribble> https://github.com/bitcoin/bitcoin/issues/13937 | Track best-possible-headers (TheBlueMatt) by Sjors · Pull Request #13937 · bitcoin/bitcoin · GitHub
2152018-08-26T11:32:53  *** belcher_ has joined #bitcoin-core-dev
2162018-08-26T11:32:57  *** SopaXorzTaker has joined #bitcoin-core-dev
2172018-08-26T11:37:37  *** str4d has joined #bitcoin-core-dev
2182018-08-26T11:41:02  *** cdecker has joined #bitcoin-core-dev
2192018-08-26T11:51:14  *** str4d has quit IRC
2202018-08-26T11:51:32  *** str4d has joined #bitcoin-core-dev
2212018-08-26T12:06:26  <provoostenator> Never mind, that's already been done in #13023
2222018-08-26T12:06:29  <gribble> https://github.com/bitcoin/bitcoin/issues/13023 | Fix some concurrency issues in ActivateBestChain() by skeees · Pull Request #13023 · bitcoin/bitcoin · GitHub
2232018-08-26T12:13:33  *** Emcy has quit IRC
2242018-08-26T12:19:12  *** sipa has quit IRC
2252018-08-26T12:19:32  *** sipa has joined #bitcoin-core-dev
2262018-08-26T12:20:43  *** sipa has quit IRC
2272018-08-26T12:28:30  *** Krellan has quit IRC
2282018-08-26T12:29:20  *** Krellan has joined #bitcoin-core-dev
2292018-08-26T12:31:23  *** BashCo has quit IRC
2302018-08-26T12:33:49  *** BashCo has joined #bitcoin-core-dev
2312018-08-26T12:34:35  *** sipa has joined #bitcoin-core-dev
2322018-08-26T12:36:19  *** farmerwampum has joined #bitcoin-core-dev
2332018-08-26T12:36:57  *** farmerwampum has quit IRC
2342018-08-26T12:37:19  *** farmerwampum has joined #bitcoin-core-dev
2352018-08-26T12:39:29  *** wxss has quit IRC
2362018-08-26T12:47:22  *** csknk has joined #bitcoin-core-dev
2372018-08-26T12:58:43  *** Taifta has quit IRC
2382018-08-26T13:02:07  *** Guyver2 has joined #bitcoin-core-dev
2392018-08-26T13:15:00  *** Taifta has joined #bitcoin-core-dev
2402018-08-26T13:15:01  *** Victorsueca has quit IRC
2412018-08-26T13:16:14  *** Victorsueca has joined #bitcoin-core-dev
2422018-08-26T13:28:22  *** Taifta has quit IRC
2432018-08-26T13:28:23  *** BashCo has quit IRC
2442018-08-26T13:28:35  *** booyah has quit IRC
2452018-08-26T13:29:46  *** booyah has joined #bitcoin-core-dev
2462018-08-26T13:31:06  *** booyah has quit IRC
2472018-08-26T13:31:09  *** BashCo has joined #bitcoin-core-dev
2482018-08-26T13:32:46  *** vexbuy has quit IRC
2492018-08-26T13:36:54  *** vexbuy has joined #bitcoin-core-dev
2502018-08-26T13:44:48  *** booyah has joined #bitcoin-core-dev
2512018-08-26T13:50:37  *** Emcy has joined #bitcoin-core-dev
2522018-08-26T13:55:30  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2532018-08-26T13:56:38  *** Taifta has joined #bitcoin-core-dev
2542018-08-26T14:06:19  <luke-jr> wumpus: heh, I was tired last night; the reason it needed a reindex (or I guess re-IBD) seems to be that I named the blocks directory "newblocks" XD
2552018-08-26T14:06:29  <luke-jr> err, "tmpblocks"
2562018-08-26T14:25:00  *** Taifta has quit IRC
2572018-08-26T14:34:04  *** Emcy has quit IRC
2582018-08-26T14:36:25  *** Krellan has quit IRC
2592018-08-26T14:37:27  *** Krellan has joined #bitcoin-core-dev
2602018-08-26T14:39:02  *** elichai2 has joined #bitcoin-core-dev
2612018-08-26T14:51:35  *** vexbuy has quit IRC
2622018-08-26T14:54:00  *** vexbuy has joined #bitcoin-core-dev
2632018-08-26T15:02:51  *** vexbuy has quit IRC
2642018-08-26T15:03:29  *** vexbuy has joined #bitcoin-core-dev
2652018-08-26T15:06:51  *** spinza has quit IRC
2662018-08-26T15:06:57  *** Chris_Stewart_5 has quit IRC
2672018-08-26T15:20:30  *** spinza has joined #bitcoin-core-dev
2682018-08-26T15:29:30  *** BashCo has quit IRC
2692018-08-26T15:30:12  *** BashCo has joined #bitcoin-core-dev
2702018-08-26T15:34:46  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2712018-08-26T15:46:47  *** plankers has quit IRC
2722018-08-26T15:48:38  *** vexbuy_ has joined #bitcoin-core-dev
2732018-08-26T15:48:59  *** vexbuy has quit IRC
2742018-08-26T15:49:39  *** Krellan has quit IRC
2752018-08-26T15:50:12  *** Krellan has joined #bitcoin-core-dev
2762018-08-26T15:55:09  *** Krellan has quit IRC
2772018-08-26T15:55:50  *** Emcy has joined #bitcoin-core-dev
2782018-08-26T15:56:01  *** vexbuy_ has quit IRC
2792018-08-26T15:56:37  *** vexbuy has joined #bitcoin-core-dev
2802018-08-26T16:07:48  *** csknk has quit IRC
2812018-08-26T16:12:15  *** Krellan has joined #bitcoin-core-dev
2822018-08-26T16:13:23  *** sipa_ has joined #bitcoin-core-dev
2832018-08-26T16:14:09  *** sipa has quit IRC
2842018-08-26T16:14:39  <Chris_Stewart_5> BIP157 doesn't currently have an open PR does it?
2852018-08-26T16:15:27  *** intcat has quit IRC
2862018-08-26T16:17:14  *** Krellan has quit IRC
2872018-08-26T16:18:38  <instagibbs> Chris_Stewart_5, https://github.com/bitcoin/bitcoin/pull/13144
2882018-08-26T16:18:54  *** intcat has joined #bitcoin-core-dev
2892018-08-26T16:26:05  <Chris_Stewart_5> instagibbs: So that is just a refactor PR to make it easier to implement BIP157 right?
2902018-08-26T16:29:19  *** BashCo has quit IRC
2912018-08-26T16:30:15  *** BashCo has joined #bitcoin-core-dev
2922018-08-26T16:36:03  <instagibbs> doh wrong one
2932018-08-26T16:36:04  <instagibbs> one sec
2942018-08-26T16:36:47  <instagibbs> Chris_Stewart_5, merged two hours ago, didnt see that https://github.com/bitcoin/bitcoin/pull/12254
2952018-08-26T16:37:02  <instagibbs> nothing open after that
2962018-08-26T16:37:16  <Chris_Stewart_5> Hmm, is that only BIP158 though? I don't think that BIP158 includes any p2p networking stuff
2972018-08-26T16:37:26  <Chris_Stewart_5> BIP157 seems to have the semantics for that
2982018-08-26T16:48:12  *** Krellan has joined #bitcoin-core-dev
2992018-08-26T16:50:51  *** itaseski has joined #bitcoin-core-dev
3002018-08-26T16:55:44  *** BashCo has quit IRC
3012018-08-26T16:57:55  *** Taifta has joined #bitcoin-core-dev
3022018-08-26T16:58:09  *** BashCo has joined #bitcoin-core-dev
3032018-08-26T17:18:50  *** reallll has joined #bitcoin-core-dev
3042018-08-26T17:21:11  *** Victorsueca has quit IRC
3052018-08-26T17:21:11  *** BashCo has quit IRC
3062018-08-26T17:21:58  *** BashCo has joined #bitcoin-core-dev
3072018-08-26T17:22:25  *** Victorsueca has joined #bitcoin-core-dev
3082018-08-26T17:22:43  *** belcher_ has quit IRC
3092018-08-26T17:34:41  *** promag has joined #bitcoin-core-dev
3102018-08-26T17:36:23  *** reallll has quit IRC
3112018-08-26T17:41:35  *** belcher_ has joined #bitcoin-core-dev
3122018-08-26T17:42:30  *** promag has quit IRC
3132018-08-26T18:03:44  *** vexbuy_ has joined #bitcoin-core-dev
3142018-08-26T18:05:25  *** vexbuy has quit IRC
3152018-08-26T18:10:34  *** sipa_ is now known as sipa
3162018-08-26T18:15:02  *** BashCo has quit IRC
3172018-08-26T18:17:18  *** BashCo has joined #bitcoin-core-dev
3182018-08-26T18:24:45  *** promag has joined #bitcoin-core-dev
3192018-08-26T18:26:17  *** promag has quit IRC
3202018-08-26T18:28:29  <jimpo> Chris_Stewart_5: No, I'm working on the next PR towards BIP 157 support
3212018-08-26T18:28:39  <jimpo> Which is building filter indexes
3222018-08-26T18:28:45  *** promag has joined #bitcoin-core-dev
3232018-08-26T18:33:00  *** promag has quit IRC
3242018-08-26T18:34:16  *** Krellan has quit IRC
3252018-08-26T18:34:36  *** Krellan has joined #bitcoin-core-dev
3262018-08-26T18:37:02  *** jb55 has joined #bitcoin-core-dev
3272018-08-26T18:39:33  <Chris_Stewart_5> gmaxwell: PIR? Without a PIR method of fetching blocks, use of this is still toxic to user privacy
3282018-08-26T18:43:32  <sipa> Chris_Stewart_5: anything that doesn't fetch all blocks will leak some information, inevitably
3292018-08-26T18:45:58  <Chris_Stewart_5> What is the acronym? PIR -- i'm guessing it is not Passive Infrared Sensor hah
3302018-08-26T18:47:03  <sipa> private information retrieval
3312018-08-26T18:53:00  *** Taifta has quit IRC
3322018-08-26T19:01:42  *** BashCo has quit IRC
3332018-08-26T19:03:17  *** BashCo has joined #bitcoin-core-dev
3342018-08-26T19:03:17  <Chris_Stewart_5> jimpo: These test vectors seem to be a dead link https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki#appendix-c-test-vectors
3352018-08-26T19:03:38  <Chris_Stewart_5> at least the first link
3362018-08-26T19:07:51  *** Salve has joined #bitcoin-core-dev
3372018-08-26T19:18:44  *** elichai2 has quit IRC
3382018-08-26T19:25:26  *** BashCo has quit IRC
3392018-08-26T19:26:21  *** BashCo has joined #bitcoin-core-dev
3402018-08-26T19:33:42  *** BashCo has quit IRC
3412018-08-26T19:41:15  *** Salve has quit IRC
3422018-08-26T19:53:44  *** rex4539 has quit IRC
3432018-08-26T19:54:21  *** Scrat has joined #bitcoin-core-dev
3442018-08-26T19:56:50  *** Scrat has quit IRC
3452018-08-26T20:04:15  *** rex4539 has joined #bitcoin-core-dev
3462018-08-26T20:05:37  *** rex4539 has joined #bitcoin-core-dev
3472018-08-26T20:06:31  *** JackH has quit IRC
3482018-08-26T20:08:54  *** kallisteiros has joined #bitcoin-core-dev
3492018-08-26T20:09:11  *** hebasto has quit IRC
3502018-08-26T20:10:08  *** owowo has quit IRC
3512018-08-26T20:10:40  *** SopaXorzTaker has quit IRC
3522018-08-26T20:16:36  <jimpo> Oh, thanks, will open a PR to fix that. Need to add a test block anyway.
3532018-08-26T20:17:33  <Chris_Stewart_5> Thanks!
3542018-08-26T20:43:44  *** Krellan has quit IRC
3552018-08-26T20:44:46  *** Krellan has joined #bitcoin-core-dev
3562018-08-26T21:05:01  *** kallisteiros has quit IRC
3572018-08-26T21:14:54  *** itaseski has quit IRC
3582018-08-26T21:15:34  *** chainhead has joined #bitcoin-core-dev
3592018-08-26T21:41:16  *** promag has joined #bitcoin-core-dev
3602018-08-26T21:44:26  *** promag has quit IRC
3612018-08-26T21:44:45  *** promag has joined #bitcoin-core-dev
3622018-08-26T21:51:02  *** d9b4bef9 has quit IRC
3632018-08-26T21:52:08  *** d9b4bef9 has joined #bitcoin-core-dev
3642018-08-26T21:53:07  *** belcher_ has quit IRC
3652018-08-26T22:03:27  *** plankers has joined #bitcoin-core-dev
3662018-08-26T22:06:55  *** promag has quit IRC
3672018-08-26T22:07:30  *** promag has joined #bitcoin-core-dev
3682018-08-26T22:12:04  *** promag has quit IRC
3692018-08-26T22:18:05  *** promag has joined #bitcoin-core-dev
3702018-08-26T22:28:12  <luke-jr> gmaxwell: sipa: wumpus: sync'd up to 341411 and still no non-mmap opens from LDB..
3712018-08-26T22:29:10  <sipa> luke-jr: ?
3722018-08-26T22:29:15  <sipa> what does that mean
3732018-08-26T22:29:50  <gmaxwell> 341411 is a long time ago.
3742018-08-26T22:30:29  <gmaxwell> sipa: he's trying to reproduce the behavior ossifrage, you, and I saw where leveldb would hit the mmap limit and start using more FDs.
3752018-08-26T22:30:36  <sipa> you'll only exceed 1000 files/mmaps once you're above ~2GB UTXO set size
3762018-08-26T22:30:43  <sipa> as each file is 2MB ish
3772018-08-26T22:31:55  <luke-jr> hmm
3782018-08-26T22:32:25  <ossifrage> pmap 24199 | grep \.ldb | wc -l    == 1894    (for a node up ~26 days)
3792018-08-26T22:34:52  <sipa> luke-jr: so you'll need to get to somewhere in 2017 at least
3802018-08-26T22:35:23  * luke-jr retries the UTXO upgrade, this time with the blocks dir named correctly
3812018-08-26T22:35:25  <ossifrage> That includes the txindex and the chainstate
3822018-08-26T22:36:46  <sipa> ossifrage: per database; txindex and chainstate are separate
3832018-08-26T22:37:28  <gmaxwell> has anyone just tried making a dummy program that mmaps a couple thousand files to see what the RES shows up as?
3842018-08-26T22:38:14  *** Victorsueca has quit IRC
3852018-08-26T22:39:25  *** Victorsueca has joined #bitcoin-core-dev
3862018-08-26T22:40:01  <ossifrage> is having a larger resident size really an issue, you only get those pages if memory is available, the os should evict them right quick under memory pressure
3872018-08-26T22:40:45  <sipa> presumably, indeed
3882018-08-26T22:41:23  <sipa> it may be just an methodology problem, what rss doesn't exactly corresponds to what we care about
3892018-08-26T22:41:45  <gmaxwell> ossifrage: if thats actually the case, thing things are fine.
3902018-08-26T22:43:52  <gmaxwell> and if so we probably need to figure out how to start tracking "total dirty pages" or something.
3912018-08-26T22:44:37  <ossifrage> leveldb only uses mmap for reading, so the dirty pages from all the maps should be 0
3922018-08-26T22:44:45  <gmaxwell> Right.
3932018-08-26T22:44:55  <luke-jr> personally, I only look at ps's "size" metric
3942018-08-26T22:45:25  <luke-jr>        size        SIZE      approximate amount of swap space that would be required if the process were to dirty all
3952018-08-26T22:45:26  <luke-jr>                              writable pages and then be swapped out.  This number is very rough!
3962018-08-26T22:45:38  <gmaxwell> ossifrage: so to catch you up. We have infrastructure that benchmarks the software regularly and charts performance... so we find out if some commit accidentally hurts performance.
3972018-08-26T22:46:03  <gmaxwell> ossifrage: we noticed around the time of the 0.17 feature freeze that the claimed memory usage suddenly went up 20%!
3982018-08-26T22:46:20  <gmaxwell> during the week where a lot of last minute stuff got merged.
3992018-08-26T22:46:37  <gmaxwell> Bisection turned up your change to increase the leveldb maps count.
4002018-08-26T22:46:48  <ossifrage> gmaxwell, where does the claimed memory measurement come from?
4012018-08-26T22:47:44  <gmaxwell> Thats what we're trying to determine. after finding out that mmap automatically pages in some of the file tried madvising DONTNEED all the maps, but that didn't seem to fix it.
4022018-08-26T22:48:08  <gmaxwell> It might well be that the increased usage is totally fictional... that it's just counting paged-in non-dirty pages.... and they're not real memory pressure and we don't need to worry about them.
4032018-08-26T22:48:17  <ossifrage> I'm writing a mmap() test case out of morbid curiosity
4042018-08-26T22:49:05  <gmaxwell> Or maybe, though we haven't spotted it yet, leveldb is using 200kb of god knows what per open map. :P
4052018-08-26T22:49:40  <gmaxwell> Meanwhile, luke-jr went and read more of the leveldb code and believes that it can't have more mmaps open than the FD limit, and so he thinks the mmap increase should be a noop.
4062018-08-26T22:50:05  <gmaxwell> (which it clearly isn't, but it would be good if everyone was on the same page about how leveldb works)
4072018-08-26T22:50:08  *** Chris_Stewart_5 has quit IRC
4082018-08-26T22:53:50  <ossifrage> I currently have 992 chainstate ldb files mapped and 859 txindex and 43 blocks/indexes ldb files, but it took a fairly long uptime for the original problem to show up
4092018-08-26T22:53:57  *** Krellan has quit IRC
4102018-08-26T22:54:42  *** Krellan has joined #bitcoin-core-dev
4112018-08-26T22:58:22  *** Guyver2 has quit IRC
4122018-08-26T23:00:35  <ossifrage> I don't see much of a change between the htop reported RES size between mapping 1 txindex ldb file and mmaping all 2041 of them
4132018-08-26T23:00:47  <luke-jr> gmaxwell: well, I can reproduce it now, but I haven't figure out why yet
4142018-08-26T23:00:53  <ossifrage> (but I never read from the map)
4152018-08-26T23:02:04  <gmaxwell> ossifrage: k? try reading like, the last byte of them or something?
4162018-08-26T23:02:24  <ossifrage> The VIRT line goes up as expected but RES stays fairly small
4172018-08-26T23:03:24  <luke-jr> hmm, is everyone reproducing this using txindex?
4182018-08-26T23:04:25  <ossifrage> gmaxwell, yeah that caused the RES size to approach the VIRT size (read the last byte)
4192018-08-26T23:05:09  <sipa> gmaxwell: "it can't have more mmaps open than the FD limit" -> s/FD limit/max_open_files setting/
4202018-08-26T23:05:27  <gmaxwell> now, try MADV_RANDOM ?
4212018-08-26T23:05:34  <gmaxwell> ossifrage: ^
4222018-08-26T23:05:51  <ossifrage> I read the last byte, so I'm not sure what it would prefetch :-)
4232018-08-26T23:06:09  *** andytoshi has joined #bitcoin-core-dev
4242018-08-26T23:06:14  <gmaxwell> I'm guessing that it's prefetching the whole thing.
4252018-08-26T23:06:25  <gmaxwell> andytoshi: HI!
4262018-08-26T23:07:23  <andytoshi> gmaxwell: hiya, sorry, my server shit out this morning .. i have no idea why, the logs show everything clear, but it was apparently completely locked up even to a local terminal user
4272018-08-26T23:07:28  <andytoshi> so i called my dad to hard-boot it and it seems ok now
4282018-08-26T23:07:46  <ossifrage> MADV_RANDOM didn't seem to change the bump in RES caused by reading last byte
4292018-08-26T23:08:00  *** promag has quit IRC
4302018-08-26T23:09:02  <gmaxwell> ossifrage: saddness.  ... so run pmap on the process, and see how many dirty pages it says you have, presumably like none.
4312018-08-26T23:11:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
4322018-08-26T23:12:38  <sipa> gmaxwell: mmap is only used in leveldb for read-only files; none should ever be dirty
4332018-08-26T23:12:54  <gmaxwell> Interesting, there are only 41.3 BTC stored in bare multisig outputs, but there are 410k of them in total.
4342018-08-26T23:13:22  <gmaxwell> sipa: I know but I also would have expected that increasing the mmap limit wouldn't have increased RES. :)
4352018-08-26T23:13:51  <ossifrage> One sec, I might have screwed up...
4362018-08-26T23:14:12  <gmaxwell> I'm going to suggest we start charting the total dirty pages from pmap... and not worry about RES...
4372018-08-26T23:14:13  <luke-jr> it looks like the mmap limit is global
4382018-08-26T23:18:32  <ossifrage> Okay, it looks like MADV_RANDOM isn't helping, but just reading the last byte does cause the RES size to jump quite a bit (but I'm not sure how much I expect it to go up)
4392018-08-26T23:18:43  <ossifrage> at least 1 page per file?
4402018-08-26T23:18:53  <sipa> yes, 1 page per file is expected
4412018-08-26T23:21:41  <gmaxwell> I expect that 1 page per file is what you should get, which is a modest amount of memory.
4422018-08-26T23:21:49  <gmaxwell> in bitcoin we're seeing an increase of hundreds of meg.
4432018-08-26T23:21:56  <gmaxwell> way more than one page per file.
4442018-08-26T23:23:55  <ossifrage> When reading the first byte, MAD_RANDOM does decrease the RES usage
4452018-08-26T23:28:33  <ossifrage> Even with MADV_RANDOM, it would depend entirely on how many pages are faulted in by ldb
4462018-08-26T23:34:20  *** promag has joined #bitcoin-core-dev
4472018-08-26T23:37:11  <ossifrage> Err, it looks like for me it is prefetching ~64K
4482018-08-26T23:43:34  <gmaxwell> 64K with MADV_RANDOM?
4492018-08-26T23:44:05  <ossifrage> I think I might be suffering from previous state problems
4502018-08-26T23:45:57  <ossifrage> Okay, dropping the cache and madv_random does seem to only read 4K for most of the files mapped (some have more for some odd reason)
4512018-08-26T23:47:08  <gmaxwell> oh interesting, so, if you cause the files to be paged in, then start a program with madv_random it will reflect higher res becuase you mmaped files that were already in cache?
4522018-08-26T23:47:10  <ossifrage> Yeah, without MADV_RANDOM all the files have 64K RES when I read the 1st byte
4532018-08-26T23:47:27  <ossifrage> echo 3 > /proc/sys/vm/drop_caches     between runs
4542018-08-26T23:47:51  <ossifrage> Yeah I was getting confusing results until I realized that
4552018-08-26T23:47:54  <gmaxwell> Cool so also we might not have seen the effects of MADV_RANDOM when testing with actual leveldb due to that.
4562018-08-26T23:48:15  <ossifrage> Depends heavily on the specific hardware yeah
4572018-08-26T23:48:43  <ossifrage> But drop_caches is a mighty big hammer
4582018-08-26T23:49:14  <gmaxwell> sure, just in terms of "huh why did MADV_RANDOM seem to do nothing"... now we know.
4592018-08-26T23:50:19  <ossifrage> I suspect the minimum amount read without MADV_RANDOM depends on things like the filesystem and the underlying block device
4602018-08-26T23:51:26  <ossifrage> Like reading 1 page with a SSD is no big deal vs 1 page with a RAID5 md array
4612018-08-26T23:55:28  <ossifrage> Any ideas how much leveldb actually reads at a time?
4622018-08-26T23:55:43  *** Victorsueca has quit IRC
4632018-08-26T23:56:35  <ossifrage> It would have to touch at least some of the index and then the actual data it wants to read
4642018-08-26T23:56:55  *** Victorsueca has joined #bitcoin-core-dev
4652018-08-26T23:57:48  <gmaxwell> it does a bisection search in the index and then I think it only reads the record of interest.