12015-11-10T00:15:12  *** zooko` has joined #bitcoin-core-dev
  22015-11-10T00:16:29  *** zooko has quit IRC
  32015-11-10T00:17:15  *** zooko` is now known as zooko
  42015-11-10T00:28:57  *** zooko has quit IRC
  52015-11-10T00:30:02  *** zooko` has joined #bitcoin-core-dev
  62015-11-10T00:31:34  *** GAit has joined #bitcoin-core-dev
  72015-11-10T00:33:01  *** zooko` is now known as zokoo
  82015-11-10T00:33:02  *** zokoo is now known as zooko
  92015-11-10T00:53:49  *** Guest51134 is now known as pigeons
 102015-11-10T01:08:14  *** GAit has quit IRC
 112015-11-10T01:16:09  *** GAit has joined #bitcoin-core-dev
 122015-11-10T01:34:24  *** Ylbam has quit IRC
 132015-11-10T01:42:39  *** randy-waterhouse has joined #bitcoin-core-dev
 142015-11-10T01:56:25  <PRab> no surprise, but running 0.11.2rc1 uneventfully.
 152015-11-10T01:56:36  <gmaxwell> Likewise.
 162015-11-10T01:58:10  <PRab> If you guys want, I can crash my computer. I have been hit by the DB corruption bug several times and it looks like this release might fix that.
 172015-11-10T02:00:37  <btcdrak> PRab, great.
 182015-11-10T02:00:57  *** CodeShark_ has joined #bitcoin-core-dev
 192015-11-10T02:01:06  <PRab> btcdrak: Great that its running, or great that I can test crashing?
 202015-11-10T02:01:19  <btcdrak> that it is running great
 212015-11-10T02:01:27  <gmaxwell> PRab: yes, it should fix most of the corruption reports on windows. All except the anti-virus related ones, as far as we know.
 222015-11-10T02:01:55  *** CodeShark is now known as Guest30488
 232015-11-10T02:02:06  *** CodeShark_ has quit IRC
 242015-11-10T02:02:28  *** CodeShark_ has joined #bitcoin-core-dev
 252015-11-10T02:02:53  *** CodeShark_ has quit IRC
 262015-11-10T02:08:49  *** Guest30488 has quit IRC
 272015-11-10T02:09:10  *** CodeShark_ has joined #bitcoin-core-dev
 282015-11-10T02:09:46  *** CodeShark_ has quit IRC
 292015-11-10T02:09:56  *** CodeShark has joined #bitcoin-core-dev
 302015-11-10T02:15:53  *** zooko has quit IRC
 312015-11-10T02:21:08  *** zooko has joined #bitcoin-core-dev
 322015-11-10T02:25:59  <GitHub106> [bitcoin] theuni opened pull request #6978: Alternative fix for #6248 (qt+fPIE) (master...qt-pie) https://github.com/bitcoin/bitcoin/pull/6978
 332015-11-10T02:56:24  *** CodeShark has quit IRC
 342015-11-10T03:01:31  *** CodeShark has joined #bitcoin-core-dev
 352015-11-10T03:07:12  *** guest234234 has joined #bitcoin-core-dev
 362015-11-10T03:09:25  *** zooko` has joined #bitcoin-core-dev
 372015-11-10T03:12:45  *** zooko`` has joined #bitcoin-core-dev
 382015-11-10T03:13:17  *** zooko has quit IRC
 392015-11-10T03:16:18  *** zooko` has quit IRC
 402015-11-10T03:21:42  *** zooko`` has quit IRC
 412015-11-10T03:50:12  *** zooko has joined #bitcoin-core-dev
 422015-11-10T03:56:35  *** zooko has quit IRC
 432015-11-10T04:06:27  *** zooko` has joined #bitcoin-core-dev
 442015-11-10T04:15:18  *** CodeShark has quit IRC
 452015-11-10T04:26:39  *** tulip has quit IRC
 462015-11-10T04:31:12  *** guest234234 has quit IRC
 472015-11-10T05:02:17  *** skyraider has quit IRC
 482015-11-10T05:08:09  *** zooko` has quit IRC
 492015-11-10T05:12:41  *** tulip has joined #bitcoin-core-dev
 502015-11-10T05:39:26  *** guest234234 has joined #bitcoin-core-dev
 512015-11-10T06:23:03  *** Ylbam has joined #bitcoin-core-dev
 522015-11-10T06:46:07  *** fkhan has quit IRC
 532015-11-10T06:50:42  *** Thireus has quit IRC
 542015-11-10T06:51:02  *** Thireus has joined #bitcoin-core-dev
 552015-11-10T06:53:31  *** CodeShark has joined #bitcoin-core-dev
 562015-11-10T07:05:35  *** guest234234 has quit IRC
 572015-11-10T07:09:25  *** tulip has quit IRC
 582015-11-10T07:13:12  *** fkhan has joined #bitcoin-core-dev
 592015-11-10T07:13:21  *** CodeShark_ has joined #bitcoin-core-dev
 602015-11-10T07:14:59  *** CodeShark has quit IRC
 612015-11-10T07:15:33  *** CodeShark has joined #bitcoin-core-dev
 622015-11-10T07:17:26  *** CodeShark_ has quit IRC
 632015-11-10T07:32:15  *** Thireus has quit IRC
 642015-11-10T07:33:20  *** CodeShark has quit IRC
 652015-11-10T08:11:19  <jonasschnelli> PRab: would be nice if you could test the v0.11.2rc1 on real-windows (non VM).
 662015-11-10T08:12:11  <jonasschnelli> I did some VMWare power-off simulations... it did mess up the db even with the fix. But this particular fix is better tested in a non-vm environment.
 672015-11-10T08:17:07  *** GAit has joined #bitcoin-core-dev
 682015-11-10T08:18:03  *** Thireus has joined #bitcoin-core-dev
 692015-11-10T08:28:46  *** fanquake has joined #bitcoin-core-dev
 702015-11-10T08:31:21  *** BashCo has quit IRC
 712015-11-10T08:44:32  *** fanquake has quit IRC
 722015-11-10T08:48:38  *** randy-waterhouse has quit IRC
 732015-11-10T08:53:51  *** trippysalmon has joined #bitcoin-core-dev
 742015-11-10T08:57:35  *** Guyver2 has joined #bitcoin-core-dev
 752015-11-10T09:17:16  *** CodeShark has joined #bitcoin-core-dev
 762015-11-10T09:22:00  <wumpus> I tested it on a real windows laptop (with NotMyFault to inject kernel faults) and wasn't able to get any corruption with the new syncing behavior, while the old behavior was to corrupt every single time
 772015-11-10T09:22:15  <wumpus> so even if not perfect it's a lot better
 782015-11-10T09:58:13  *** rubensayshi has joined #bitcoin-core-dev
 792015-11-10T10:03:02  <wumpus> gitian is broken for me :( "Could not download some packages, please run gbuild --upgrade" when trying to sign the mac package
 802015-11-10T10:03:39  <dcousens> jonasschnelli: I've had similar nodes get 'stuck' before, so unless its repeatable, I'm not sure that is related to that PR
 812015-11-10T10:03:58  <wumpus> nothing wrong in install.log
 822015-11-10T10:04:08  <jonasschnelli> dcousens: right. I wrote that in a comment. Probably unrelated to the secp256k1 switch PR
 832015-11-10T10:04:28  <jonasschnelli> But a serious bug,...
 842015-11-10T10:04:30  <wumpus> faketime is already the newest version. libc6:i386 is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
 852015-11-10T10:04:35  <dcousens> jonasschnelli: I know, just providing an anecdote that might further that
 862015-11-10T10:04:45  <dcousens> further support*
 872015-11-10T10:05:04  <wumpus> jonasschnelli: I had a similar issue a while back of a node that just stopped catching up with the chain - never been able to repeat it
 882015-11-10T10:05:11  <dcousens> also, you wrote F.I.Y, for interested yours? :P
 892015-11-10T10:05:34  <jonasschnelli> yeah... it seams to be hard to create clear steps to reproduce...
 902015-11-10T10:05:40  <wumpus> (and I didn't build with debug symbols at the time, so was unable to do anything useful to troubleshoot internal state)
 912015-11-10T10:05:54  <jonasschnelli> s/F.I.Y/F.Y.I... :)
 922015-11-10T10:06:06  <wumpus> if it happens again I'm prepared. But it never happened again
 932015-11-10T10:06:16  <dcousens> wumpus: I had it a week ago
 942015-11-10T10:06:47  <jonasschnelli> wumpus: while(1) { IBD() } ...
 952015-11-10T10:06:56  <wumpus> here: https://github.com/bitcoin/bitcoin/issues/6188
 962015-11-10T10:07:28  <wumpus> well it didn't happen during IBD in my case, just with a node that was synced but suddenly stopped without rejecting the chain or anything like that
 972015-11-10T10:07:43  <wumpus> (and after four days it magically started again)
 982015-11-10T10:07:49  <dcousens> That said, in restarting the node, I found the most prominent factor was the peers it was connected to
 992015-11-10T10:08:02  <jonasschnelli> true.
1002015-11-10T10:08:39  *** CodeShark has quit IRC
1012015-11-10T10:09:00  <jonasschnelli> the strange thing is, as you can see here (https://bitcoin.jonasschnelli.ch/secp/stuck_debug.log), no block was rejected.
1022015-11-10T10:09:02  <wumpus> that wasn't the problem in my case. it was a node with many incoming connections, and I did try dropping the network to get new connections
1032015-11-10T10:09:44  *** BashCo has joined #bitcoin-core-dev
1042015-11-10T10:09:59  <wumpus> it is likely some race condition, where it forgets about being insistent about requesting the next block it needs
1052015-11-10T10:10:24  <wumpus> likely caused by one or more uncooprorative peers
1062015-11-10T10:15:24  *** pmienk has joined #bitcoin-core-dev
1072015-11-10T10:20:30  *** GAit has quit IRC
1082015-11-10T10:21:00  *** GAit has joined #bitcoin-core-dev
1092015-11-10T10:23:55  <wumpus> re: the gitian issue, adding --upgrade to my gbuild line seems to have solved it. I don't understand why this is suddenly needed, but ok, great :)
1102015-11-10T11:07:45  *** ParadoxSpiral has joined #bitcoin-core-dev
1112015-11-10T11:48:15  *** MarcoFalke has joined #bitcoin-core-dev
1122015-11-10T12:43:30  *** jonasschnelli has quit IRC
1132015-11-10T12:44:19  *** jonasschnelli has joined #bitcoin-core-dev
1142015-11-10T12:50:54  *** Naphex has joined #bitcoin-core-dev
1152015-11-10T12:50:57  <GitHub175> [bitcoin] jonasschnelli opened pull request #6979: [Qt] simple mempool info in debug window (master...2015/11/qt_mempool_easyinfo) https://github.com/bitcoin/bitcoin/pull/6979
1162015-11-10T12:58:42  *** jonasschnelli has quit IRC
1172015-11-10T13:00:20  *** jonasschnelli has joined #bitcoin-core-dev
1182015-11-10T13:04:09  *** MarcoFalke has quit IRC
1192015-11-10T13:14:02  *** GAit has quit IRC
1202015-11-10T13:14:37  *** GAit has joined #bitcoin-core-dev
1212015-11-10T13:26:06  *** GAit has quit IRC
1222015-11-10T13:26:55  *** GAit has joined #bitcoin-core-dev
1232015-11-10T13:44:41  <dcousens> jonasschnelli: what block did you pause on OOI?
1242015-11-10T13:45:07  <jonasschnelli> dcousens: OOI?
1252015-11-10T13:45:14  <dcousens> out of interest
1262015-11-10T13:45:41  <jonasschnelli> dcousens: 00000000000000001043ada919c3851e17876deca67acc19f365fab4a79bd59d, 382748
1272015-11-10T13:45:55  <jonasschnelli> dcousens: check: https://bitcoin.jonasschnelli.ch/secp/stuck_debug.log, search after last UpdateTip
1282015-11-10T13:46:44  <GitHub50> [bitcoin] laanwj pushed 2 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/3dcb390fe9e2...7e278929df53
1292015-11-10T13:46:44  <GitHub50> bitcoin/0.11 ab6ff12 David A. Harding: [doc] 0.11.2 release notes: use original pull numbers...
1302015-11-10T13:46:44  <GitHub175> [bitcoin] laanwj closed pull request #6975: [doc] 0.11.2 release notes: use original pull numbers (0.11...note-0.11.2-orig-prs) https://github.com/bitcoin/bitcoin/pull/6975
1312015-11-10T13:46:45  <GitHub50> bitcoin/0.11 7e27892 Wladimir J. van der Laan: Merge pull request #6975...
1322015-11-10T13:47:36  <dcousens> jonasschnelli: interesting
1332015-11-10T13:47:52  <dcousens> I stopped around 362250
1342015-11-10T13:48:08  <dcousens> But wumpus' post was around 370k
1352015-11-10T13:48:45  <jonasschnelli> dcousens: do you see a relation between 362250 and 382748?
1362015-11-10T13:49:21  <dcousens> jonasschnelli: i haven't checked, the only relation I have so far is that my block parser has exploded around those blocks recently
1372015-11-10T13:49:31  <dcousens> and I haven't yet debugged it, could be completely unrelated
1382015-11-10T13:49:57  <jonasschnelli> hmm... yes. Would be nice if we could track down that problem.
1392015-11-10T13:50:27  <dcousens> anyway, late here, will have a closer look in the morrow :)
1402015-11-10T13:50:29  <dcousens> night
1412015-11-10T13:52:52  <wumpus> 0.10.4rc1 executables live https://bitcoin.org/bin/bitcoin-core-0.10.4/test/
1422015-11-10T13:55:11  *** dcousens has quit IRC
1432015-11-10T14:05:14  *** zooko has joined #bitcoin-core-dev
1442015-11-10T14:12:41  <jonasschnelli> wumpus : thanks for the reminder on #6900, just started a gitian build with the new descriptor...
1452015-11-10T14:12:55  <jonasschnelli> hmm... i need the base images first i assume
1462015-11-10T14:25:30  *** fanquake has joined #bitcoin-core-dev
1472015-11-10T14:25:51  <fanquake> wumpus building the gitian pull now
1482015-11-10T14:32:15  <GitHub179> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/503ff6e1ae69...77beab70deae
1492015-11-10T14:32:15  <GitHub179> bitcoin/master 87cbdb8 Jorge Timón: Globals: Explicit Consensus::Params arg for main:...
1502015-11-10T14:32:16  <GitHub179> bitcoin/master 77beab7 Wladimir J. van der Laan: Merge pull request #6163...
1512015-11-10T14:32:20  <GitHub104> [bitcoin] laanwj closed pull request #6163: Chainparams: Explicit Consensus::Params arg for almost all remaining functions (master...chainparams-remaining-consensus) https://github.com/bitcoin/bitcoin/pull/6163
1522015-11-10T14:32:30  <GitHub170> [bitcoin] laanwj closed pull request #6248: Fix Qt build on arch by setting -fPIC (master...archbuild) https://github.com/bitcoin/bitcoin/pull/6248
1532015-11-10T14:35:41  *** ParadoxSpiral_ has joined #bitcoin-core-dev
1542015-11-10T14:38:19  *** ParadoxSpiral has quit IRC
1552015-11-10T14:38:35  *** zooko has quit IRC
1562015-11-10T14:48:45  <GitHub35> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/77beab70deae...755b4ba848bc
1572015-11-10T14:48:45  <GitHub35> bitcoin/master fd55571 Luke Dashjr: wallet: Expose GUI labels in RPC
1582015-11-10T14:48:46  <GitHub35> bitcoin/master 755b4ba Wladimir J. van der Laan: Merge pull request #5574...
1592015-11-10T14:48:46  <GitHub43> [bitcoin] laanwj closed pull request #5574: Expose GUI labels in RPC as comments (master...rpc_labels) https://github.com/bitcoin/bitcoin/pull/5574
1602015-11-10T14:51:48  <GitHub120> [bitcoin] laanwj closed pull request #6693: Set Windows TCP buffers to 64KB to match OSX and Unix (master...issue_6554) https://github.com/bitcoin/bitcoin/pull/6693
1612015-11-10T14:58:09  <GitHub78> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/755b4ba848bc...9fa54a1b0c1a
1622015-11-10T14:58:10  <GitHub78> bitcoin/master 5f46a7d MarcoFalke: transaction_tests: Be more strict checking dust...
1632015-11-10T14:58:10  <GitHub78> bitcoin/master 536766c MarcoFalke: [trivial] New DEFAULT_MIN_RELAY_TX_FEE = 1000
1642015-11-10T14:58:11  <GitHub78> bitcoin/master e20d924 MarcoFalke: [trivial] init: Use defaults MIN_RELAY_TX_FEE & TRANSACTION_MAXFEE
1652015-11-10T14:58:18  <GitHub65> [bitcoin] laanwj closed pull request #6822: [tests] Be more strict checking dust (master...MarcoFalke-2015-minRelayTxFeeCleanup) https://github.com/bitcoin/bitcoin/pull/6822
1662015-11-10T14:59:01  <wumpus> jonasschnelli: you need one new base image, trusty amd64
1672015-11-10T15:01:38  *** zooko has joined #bitcoin-core-dev
1682015-11-10T15:08:53  <fanquake> I'll put up a pull to bump boost, as well as ccache & miniupnpc
1692015-11-10T15:09:27  <fanquake> ccache because of an osx compile regression & miniupnpc so we get wumpus's string handling/overflow fixes.
1702015-11-10T15:10:08  *** jgarzik has joined #bitcoin-core-dev
1712015-11-10T15:11:07  <wumpus> awesome
1722015-11-10T15:13:05  <jonasschnelli> wumpus: did you face similar issues when building the trusy base image:...
1732015-11-10T15:13:05  <jonasschnelli> Process (['mkfs.ext4', '-F', '/dev/mapper/loop0p1']) returned 1. stdout: , stderr: mke2fs
1742015-11-10T15:13:12  <jonasschnelli> The file /dev/mapper/loop0p1 does not exist and no size was specified.
1752015-11-10T15:13:22  <fanquake> jonasschnelli yes
1762015-11-10T15:13:34  <jonasschnelli> i have updates gitian to the master git tip
1772015-11-10T15:13:42  <jonasschnelli> fanquake: ah. How did you solve that?
1782015-11-10T15:13:51  <wumpus> don't remember that, I did succeed in making the image
1792015-11-10T15:14:27  <fanquake> jonasschnelli I haven't yet. Got sidetracked looking at depends.
1802015-11-10T15:15:03  <jonasschnelli> fanquake: okay. Thanks.. then i go down that rabbit hole...
1812015-11-10T15:16:09  <wumpus> just did `bin/make-base-vm --arch amd64 --suite trusty`
1822015-11-10T15:18:50  *** gribble has quit IRC
1832015-11-10T15:26:04  <jonasschnelli> wumpus: i guess your on ubuntu? On debian i always had issues with gitian...
1842015-11-10T15:26:43  <wumpus> right,ubuntu
1852015-11-10T15:31:46  *** gribble has joined #bitcoin-core-dev
1862015-11-10T15:32:18  <GitHub111> [bitcoin] fanquake opened pull request #6980: [Depends] Bump Boost, miniupnpc & ccache (master...depends-bump-boost) https://github.com/bitcoin/bitcoin/pull/6980
1872015-11-10T15:35:21  <GitHub99> [bitcoin] laanwj closed pull request #6937: Fix Boost 1.58.0 build for mips arch (master...mips-options-fix) https://github.com/bitcoin/bitcoin/pull/6937
1882015-11-10T15:37:42  *** jtimon has joined #bitcoin-core-dev
1892015-11-10T15:39:25  <GitHub85> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9fa54a1b0c1a...32d8b1570cb0
1902015-11-10T15:39:25  <GitHub85> bitcoin/master 7ca73dc Jonathan Cross: Improving labels for Sent / Received "Bytes"...
1912015-11-10T15:39:26  <GitHub85> bitcoin/master 32d8b15 Wladimir J. van der Laan: Merge pull request #6940...
1922015-11-10T15:39:35  <GitHub102> [bitcoin] laanwj closed pull request #6940: Improving labels for Sent / Received "Bytes" (master...patch-1) https://github.com/bitcoin/bitcoin/pull/6940
1932015-11-10T15:45:13  <GitHub4> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/b56953e9bb5a32bc35365d1f0c5de5528c0650dd
1942015-11-10T15:45:14  <GitHub4> bitcoin/master b56953e Wladimir J. van der Laan: qt: Periodic translations update
1952015-11-10T15:53:10  *** bsm1175321 has quit IRC
1962015-11-10T16:00:24  *** zooko has quit IRC
1972015-11-10T16:03:48  *** fanquake has left #bitcoin-core-dev
1982015-11-10T16:14:03  *** Thireus has quit IRC
1992015-11-10T16:26:22  <wumpus> jonasschnelli: any luck building the image? maybe try updating vmbuilder, if you're building it from source
2002015-11-10T16:30:27  *** GAit has quit IRC
2012015-11-10T16:34:07  *** PaulCapestany has quit IRC
2022015-11-10T16:34:17  *** PaulCape_ has joined #bitcoin-core-dev
2032015-11-10T16:35:10  *** trippysalmon has quit IRC
2042015-11-10T16:43:04  *** randy-waterhouse has joined #bitcoin-core-dev
2052015-11-10T16:43:26  *** Thireus has joined #bitcoin-core-dev
2062015-11-10T16:45:56  <sipa> wumpus, jonasschnelli: where in the codebase are we using powers-of-1024 based units?
2072015-11-10T16:46:52  <sipa> bitcoin traditionally uses 1000-based units (feerate is per 1000 bytes, block limit is 1000000 bytes), but I can't say I've payed much attention to it myself
2082015-11-10T16:46:57  <jgarzik> sipa, it's all over Qt
2092015-11-10T16:47:42  <sipa> if anything, be consistent (don't use MB for 1048576 bytes, and certainly don't use KB - that would mean kelvin bytes)
2102015-11-10T16:47:47  <jgarzik> several core uses in default values, usually related to file size
2112015-11-10T16:52:55  <wumpus> sipa: I'm not sure. But in this case jonasschnelli tried to introduce a use of 1024*1024
2122015-11-10T16:53:12  <sipa> I know
2132015-11-10T16:53:13  <wumpus> which was sensibly commented on
2142015-11-10T16:54:08  <sipa> I think using MB = 1024^2 is a no-go. My question is whether or not we should aim for only MB = 1000000 or only MiB = 1024^2
2152015-11-10T16:54:10  <wumpus> there's probably a few other cases but in general the idea is to use 1000000/MB
2162015-11-10T16:55:00  <wumpus> only 1000 *unless* there is a convincing reason to use 1048576/MiB, which I'm not sure exists
2172015-11-10T16:55:30  *** rubensayshi has quit IRC
2182015-11-10T16:55:35  <sipa> We clearly need a --si commandline flag
2192015-11-10T16:55:37  <sipa> *ducks*
2202015-11-10T16:55:52  <wumpus> e.g. I can understand using MiB when you're selling memory chips which for hardware reasons, only in powers of two
2212015-11-10T16:56:10  <wumpus> but for bitcoin core which is a high-level application there shoulod be no reason to not just use SI units
2222015-11-10T16:56:17  <sipa> agree
2232015-11-10T16:56:50  <wumpus> and it's always been that way, there's not really a reason to discuss or revise this :)
2242015-11-10T16:57:45  * wumpus likes Kelvin*Bytes though
2252015-11-10T16:57:45  <sipa> I was just asking whether or not we currently have cases where 1024-based units are exposed to users. Consistency with already existing uses would be a reason.
2262015-11-10T16:58:13  <sipa> "Our cache is very hot. Many KB."
2272015-11-10T16:58:20  <wumpus> hehe
2282015-11-10T16:59:15  * wumpus going to add a note about using SI units to the developer notes
2292015-11-10T16:59:35  <jgarzik> as noted, we have many cases exposed to users
2302015-11-10T17:00:53  <sipa> i think -dbcache may be 1024-based
2312015-11-10T17:01:17  <wumpus> should be changed
2322015-11-10T17:01:19  <jgarzik> -dblogsize, several Qt UI attributes
2332015-11-10T17:01:37  <jgarzik> several defaults are pow2
2342015-11-10T17:02:00  *** GAit has joined #bitcoin-core-dev
2352015-11-10T17:02:39  <sipa> i'm in favor of making user-exposed amounts consistently 1000-based, if there aren't too many
2362015-11-10T17:03:25  <wumpus> yes
2372015-11-10T17:07:42  <jgarzik> +1
2382015-11-10T17:08:06  <wangchun> Dear core devs, how do you plan to response to the latest actions regarding BIP101?
2392015-11-10T17:09:40  <wumpus> not at all
2402015-11-10T17:10:41  <sipa> Bitcoin Core will implement any uncontroversial hard fork.
2412015-11-10T17:11:16  <jgarzik> wangchun, Scaling Bitcoin Pt 2 is the next step in figuring out uncontroversial hard fork
2422015-11-10T17:11:49  *** skyraider has joined #bitcoin-core-dev
2432015-11-10T17:14:30  *** bsm1175321 has joined #bitcoin-core-dev
2442015-11-10T17:14:47  *** paveljanik has joined #bitcoin-core-dev
2452015-11-10T17:15:56  *** PaulCape_ has quit IRC
2462015-11-10T17:16:21  *** PaulCapestany has joined #bitcoin-core-dev
2472015-11-10T17:21:26  <btcdrak> sipa: I just added some commits which I think address your concerns https://github.com/bitcoin/bitcoin/pull/6312#issuecomment-154652895
2482015-11-10T17:26:15  *** GAit has quit IRC
2492015-11-10T17:26:51  *** GAit has joined #bitcoin-core-dev
2502015-11-10T17:34:04  <GitHub159> [bitcoin] laanwj opened pull request #6981: doc: Add note about SI units to developer notes (master...2015_11_si_units) https://github.com/bitcoin/bitcoin/pull/6981
2512015-11-10T17:41:08  *** molly has quit IRC
2522015-11-10T17:53:49  *** BashCo has quit IRC
2532015-11-10T17:57:17  *** molly has joined #bitcoin-core-dev
2542015-11-10T18:19:10  <GitHub108> [bitcoin] laanwj closed pull request #6965: Benchmark sanity checks and fork checks in ConnectBlock (master...bench) https://github.com/bitcoin/bitcoin/pull/6965
2552015-11-10T18:19:24  <GitHub111> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b56953e9bb5a...de7d4591a7ce
2562015-11-10T18:19:25  <GitHub111> bitcoin/master 77f1f59 Matt Corallo: Benchmark sanity checks and fork checks in ConnectBlock
2572015-11-10T18:19:25  <GitHub111> bitcoin/master de7d459 Wladimir J. van der Laan: Merge pull request #6965...
2582015-11-10T18:21:39  *** gribble has quit IRC
2592015-11-10T18:31:37  *** gribble has joined #bitcoin-core-dev
2602015-11-10T18:47:36  *** GAit has quit IRC
2612015-11-10T19:01:36  <GitHub176> [bitcoin] jtimon opened pull request #6982: Fix #6163 (AcceptBlockHeader) (master...fix-6163) https://github.com/bitcoin/bitcoin/pull/6982
2622015-11-10T19:06:09  <jonasschnelli> consensus on MB and /1000^2?
2632015-11-10T19:06:32  <jgarzik> IMO the consensus was already "move to MiB" before today...
2642015-11-10T19:06:57  * jonasschnelli is confused... :/
2652015-11-10T19:09:13  * Luke-Jr dislikes "_iB" notation and would avoid other software enforcing it.
2662015-11-10T19:09:27  <Luke-Jr> I don't suppose Qt has a nice localisation method for these?
2672015-11-10T19:09:27  <sipa> jgarzik: now i am confused too
2682015-11-10T19:09:49  <jgarzik> ok, I meant SI units
2692015-11-10T19:10:05  <jgarzik> whatever the abbrevation is
2702015-11-10T19:10:13  <Luke-Jr> SI units are 1000-based kB/MB/GB
2712015-11-10T19:10:15  <jgarzik> Luke-Jr, agree!
2722015-11-10T19:10:29  <jgarzik> AFAIK the consensus is SI units </correction>
2732015-11-10T19:10:59  <Luke-Jr> I'd prefer configurable at some level (DE level would be nice), but not worth the time if Qt doesn't make it easy.
2742015-11-10T19:11:12  <sipa> jgarzik: there are 2 questions? "should we use 1000 or 1024-based units" and "should we use Mi-style prefixes for 1024-based units?"
2752015-11-10T19:11:42  <Luke-Jr> Right now, the traffic thing uses 1024-based KB/MB/GB
2762015-11-10T19:11:51  <Luke-Jr> (which is what I prefer)
2772015-11-10T19:12:03  <sipa> that's the only unacceptable thing for me :)
2782015-11-10T19:12:21  <sipa> either 1000000/MB, or 1049576/MiB
2792015-11-10T19:12:34  <sipa> with a preference for the first
2802015-11-10T19:12:44  <sipa> 1048576 i mean
2812015-11-10T19:12:51  <gmaxwell> sipa: really you have a prefered for the first?
2822015-11-10T19:13:18  <sipa> yes, though i have a tendency for the aecond
2832015-11-10T19:13:32  <jgarzik> 1) 1000-based units,    2) I hate Mi style prefixes -- grew up thinking 1024==KB, 1024*1024==MB, etc.  However I think that's the direction of the world...
2842015-11-10T19:14:06  <jgarzik> easy enough to ensure we eliminate as many 1024-based units as possible, to minimize Mi prefixes
2852015-11-10T19:14:07  <Luke-Jr> jgarzik: the direction of the world is defined by people in the world, such as us to a small degree ;)
2862015-11-10T19:14:41  <gmaxwell> Normally _bandwidth_ is measured in megabits which is a 1000 based unit. Transfer regretably is often in 1024 based units. "MB" should never be used for 1024 based because it's unnecessarily confusing; MiB is unambigious. The purpose of the messages is to communicate, not to be pretty.
2872015-11-10T19:15:12  <sipa> gmaxwell: i think the world would be a (marginally) better place if everything used 1000-based prefixes, it would make reasoning so much simpler
2882015-11-10T19:15:17  <jgarzik> problem - so few know WTF MiB is, out in the world
2892015-11-10T19:15:55  * Luke-Jr did not realise megabits were typically 1000-based
2902015-11-10T19:16:30  <gmaxwell> Luke-Jr: not just typically but always except when handled by drooling idiots. :)
2912015-11-10T19:17:12  <gmaxwell> jgarzik: far more common than you think, I think. But even still, if so then their ignorance is apparent to them and a 10 second search will resolve it.
2922015-11-10T19:17:22  <sipa> Luke-Jr: a 100 Mb/s link can do 11.9 MiB/s :)
2932015-11-10T19:17:41  *** zooko has joined #bitcoin-core-dev
2942015-11-10T19:18:44  <Luke-Jr> Well, in that case, /me suggests 1000-based with an unchecked-by-default checkbox that uses 1024 in GUI without the "i" silliness. <.<
2952015-11-10T19:19:09  <gmaxwell> Luke-Jr: please, no freeing options.
2962015-11-10T19:19:14  <gmaxwell> er freeking.
2972015-11-10T19:19:26  <gmaxwell> It'll pepper the code with conditional logic that will never get tested.
2982015-11-10T19:19:37  <jgarzik> nod
2992015-11-10T19:19:41  <sipa> agree
3002015-11-10T19:19:56  <Luke-Jr> gmaxwell: this is an option that different people want to use. so obviously it would get tested.
3012015-11-10T19:20:06  <sipa> nobody will bother
3022015-11-10T19:20:20  <sipa> no actual user will bother, rather
3032015-11-10T19:20:28  <Luke-Jr> at least I will
3042015-11-10T19:20:45  <gmaxwell> Do you even actually use the GUI except for testing? :)
3052015-11-10T19:20:48  <Luke-Jr> yes
3062015-11-10T19:21:05  <Luke-Jr> the only time I use the RPC is for testing.
3072015-11-10T19:21:37  <jgarzik> Similar to -logtimemicros option -- it's basically an option 1-2 people will use, and the rest of the world will be unaware.
3082015-11-10T19:21:48  <jgarzik> Should just pick a useful behavior and not make it an option.
3092015-11-10T19:22:03  <Luke-Jr> let's just drop all non-English languages too :P
3102015-11-10T19:22:11  <sipa> it doesn't hurt to add more digits
3112015-11-10T19:22:27  <sipa> and the cases when they're useful is not when you know it in advance
3122015-11-10T19:22:55  <jgarzik> I would rather remove -logtimemicros and make it unconditionally default-on.
3132015-11-10T19:22:59  <sipa> agree
3142015-11-10T19:23:01  <sipa> me too
3152015-11-10T19:23:01  <wumpus> I disagree
3162015-11-10T19:23:10  <jgarzik> From OS perspective there is no added cost
3172015-11-10T19:23:12  <wumpus> seconds are precise enough for logging
3182015-11-10T19:23:39  <jgarzik> 1) other userland servers log microseconds,  2) it's a pointless option that will be used by no one - yet we must maintain
3192015-11-10T19:23:43  <sipa> i would love to be able to get bug reports "when did X take 10ms?! it should just be changing a variable!"
3202015-11-10T19:23:49  <Luke-Jr> let's go with systemd binary logs! /s
3212015-11-10T19:23:51  <sipa> nobody will even notice now
3222015-11-10T19:24:08  <Luke-Jr> sipa: lol
3232015-11-10T19:24:17  <jgarzik> default-off options should be terminated with extreme prejudice.
3242015-11-10T19:24:23  <zooko> jgarzik: +1
3252015-11-10T19:24:34  <kanzure> replaced with what?
3262015-11-10T19:24:38  <jgarzik> (translation: there should be a good argument for keeping and maintaining them, above "it's nice")
3272015-11-10T19:24:45  <wumpus> bla, this is useless
3282015-11-10T19:25:03  <sipa> wumpus: what is your reason against microsecond logs?
3292015-11-10T19:25:04  <kanzure> ah okay. "it's nice" as insufficient. ok.
3302015-11-10T19:25:30  <kanzure> you could send all logs through zeromq inside the process, then people can subscribe to logs in different ways using zeromq.
3312015-11-10T19:25:38  <wumpus> sipa: too precise timestamps are a possible security issue
3322015-11-10T19:25:43  <wumpus> e.g. timing attacks
3332015-11-10T19:25:48  <sipa> hmm
3342015-11-10T19:26:01  <wumpus> also it makes correlation easier, to breach privacy
3352015-11-10T19:26:03  <sipa> i thought we dropped that concern when switching to -logtimestamps default on
3362015-11-10T19:26:08  <jgarzik> sipa, yep
3372015-11-10T19:26:09  <wumpus> yes, for seconds
3382015-11-10T19:26:13  <gmaxwell> I would rather stop logging things that breach privacy.
3392015-11-10T19:26:33  <Luke-Jr> as long as sipa hates 1024 KB, and I prefer 1024 KB, there's no way to satisfy both without an option. If you all decide to make it 1000-based only, I can just add an option to Bitcoin LJR if I ever care enough (unlikely)
3402015-11-10T19:26:35  <wumpus> microsconds just goes too far, I see no point in that kind of precision
3412015-11-10T19:26:44  <sipa> Luke-Jr: kB
3422015-11-10T19:26:46  <sipa> :p
3432015-11-10T19:26:50  <Luke-Jr> sipa: kB is 1000-based always
3442015-11-10T19:26:50  <gmaxwell> We generate a LOT of excess IO via logging. Lets reduce the chattyness, that is a much more robust privacy protection than decreasing timestamp precision.
3452015-11-10T19:27:02  <jgarzik> There is no useful privacy argument for seconds, which does not also apply to microseconds.  The matter of degrees is tiny.
3462015-11-10T19:27:04  <jgarzik> +1 gmaxwell
3472015-11-10T19:27:09  <wumpus> god damnit
3482015-11-10T19:27:18  <kanzure> timing attacks tho
3492015-11-10T19:27:40  <kanzure> oh
3502015-11-10T19:27:47  <wumpus> I agree with reducing chattyness as well, but that's a different concern
3512015-11-10T19:28:00  <kanzure> right, i think you mean to say "microseconds does not have any useful privacy advantage over seconds".
3522015-11-10T19:28:54  <Luke-Jr> it does have a screen-space advantage, but I can't believe we're spending time discussing this.
3532015-11-10T19:29:15  <jgarzik> kanzure, no.  the argument being made is "microseconds goes too far" and "it makes correlation easier, to breach privacy"
3542015-11-10T19:29:41  <wumpus> Luke-Jr: and it's already possible to enable microseconds if you want to use them
3552015-11-10T19:29:43  <sipa> Luke-Jr: KB is kelvin byte. I don't understand where your capital K even comes from; IEC uses Ki for 1024 because ki would make it look like a unit rather than a prefix
3562015-11-10T19:30:03  <jgarzik> kanzure, no privacy advantage in microseconds is being claimed - just the opposite - though the matter of degree is tiny
3572015-11-10T19:30:10  <wumpus> I don't understand this insistence on enabling it by default
3582015-11-10T19:30:11  <jgarzik> versus the diagnostic utility and common practice elsewhere
3592015-11-10T19:30:14  <kanzure> jgarzik: understood
3602015-11-10T19:30:15  <Luke-Jr> KB is the traditional standard notation for 1024 bytes, defined in at least JEDEC standards
3612015-11-10T19:31:15  <Luke-Jr> sipa: ^
3622015-11-10T19:31:17  <kanzure> i believe the BFOH approach was "KiBi" instead of KB or kB or KiB. (not serious here) (but i did see this proposed once by a BOFH).
3632015-11-10T19:31:53  <sipa> Luke-Jr: interesting :)
3642015-11-10T19:32:04  <jgarzik> wumpus, Because merging options that nearly-nobody will use is dumb
3652015-11-10T19:32:14  <sipa> wumpus: being able to benchmark things after the fact
3662015-11-10T19:32:23  <wumpus> jgarzik: people that want to benchmark things with precision will enable it
3672015-11-10T19:32:36  <wumpus> if no one wanted it it wouldn't have been merged at all
3682015-11-10T19:32:49  <kanzure> was this cli option or was this build option?
3692015-11-10T19:32:55  <sipa> cli
3702015-11-10T19:32:56  <jgarzik> c.f. "1-2 users"    Most people will not even know about it.
3712015-11-10T19:32:57  <jgarzik> cli
3722015-11-10T19:33:04  <kanzure> was build option considered?
3732015-11-10T19:33:09  <wumpus> why?
3742015-11-10T19:33:28  <kanzure> because cli options are disagreeable for good reasons
3752015-11-10T19:33:32  <wumpus> what?
3762015-11-10T19:33:53  <wumpus> disagreeable for what reason?
3772015-11-10T19:34:18  <kanzure> 11:24 < jgarzik> default-off options should be terminated with extreme prejudice.
3782015-11-10T19:34:29  <wumpus> and that doesn't apply to build options?
3792015-11-10T19:34:32  <jgarzik> build option is even worse - conditional compilation creates code not even built always, but still must be maintained.
3802015-11-10T19:34:33  *** BashCo has joined #bitcoin-core-dev
3812015-11-10T19:34:42  <gmaxwell> wumpus: it's more conditional code which will be inadequately tested in one form or another, especially all the combinations will not be tested.
3822015-11-10T19:34:44  <kanzure> hmm okay.
3832015-11-10T19:34:45  <wumpus> yeah this is absolutely going the wrong way
3842015-11-10T19:35:03  <wumpus> gmaxwell: I agree if it's a complex conditional, but come on, a logging option
3852015-11-10T19:35:24  <gmaxwell> E.g. all of us will turn microseconds on;  and then no-microseconds + disable wallet will end up crashing; and we won't notice until after a release.  ... but yes; in this particular case this argument is not strong. I agree that it's mostly safe.
3862015-11-10T19:35:37  <wumpus> but anyhow feel free to remove the option, I'm done discussing this
3872015-11-10T19:36:04  <kanzure> what about zeromq as logging transport, then just use whatever temporal resolution at receive time of each message?
3882015-11-10T19:36:07  <kanzure> ok nevermind then
3892015-11-10T19:36:18  <Luke-Jr> kanzure: ZeroMQ is stupidly unreliable.
3902015-11-10T19:36:27  <wumpus> debug logging is *not* meant as application interface
3912015-11-10T19:36:36  <kanzure> Luke-Jr: fedpeg.py is just misconfigured :P
3922015-11-10T19:36:37  <wumpus> it's just for debugging and troubleshooting
3932015-11-10T19:36:48  <wumpus> not for processing by other applications
3942015-11-10T19:36:49  <Luke-Jr> kanzure: fedpeg.py is not my only experience with ZeroMQ>
3952015-11-10T19:37:21  <wumpus> if you insist you can already use -logtoconsole and pipe it to something
3962015-11-10T19:38:20  <gmaxwell> kanzure: I have other expirence with ZeroMQ too. It does not correctly handle non-lossless networks.
3972015-11-10T19:40:18  <jgarzik> Makes sense.  zmq was built for LANs, with similar use cases to RDMA.
3982015-11-10T19:40:41  <Luke-Jr> LANs aren't always lossless (hi wifi)
3992015-11-10T19:40:42  *** berndj has quit IRC
4002015-11-10T19:40:43  <morcos> just to be clear, i love and will always use logtimemicros, but i have another annoying question
4012015-11-10T19:40:46  <kanzure> so complaint is lack of application-level retries? or tcp delivery guarantees too strongly stated in zeromq docs?
4022015-11-10T19:40:48  <sipa> zmq is explicitly lossy
4032015-11-10T19:40:51  <sipa> like udp
4042015-11-10T19:40:54  <Luke-Jr> also, ZeroMQ 4.0 is incompatible with 4.1
4052015-11-10T19:41:01  <sipa> i thought?
4062015-11-10T19:41:07  <morcos> wumpus: i have created two new functions EstimateApproxFee and EstimateApproxPriority in 6134
4072015-11-10T19:41:17  <morcos> my plan was not to expose them via RPC
4082015-11-10T19:41:29  <morcos> because i think over time the fee estimator still has some more significant evolution
4092015-11-10T19:41:49  <morcos> and then in the end it might be nice to expose what we think the final interface should be
4102015-11-10T19:42:12  <morcos> however sdaftuar suggested i might be able to expose them now and comment they might change or keep them hidden or something
4112015-11-10T19:42:25  <morcos> they would primarily be useful in developing tests for the new functionality i think?
4122015-11-10T19:43:28  <morcos> in any case, would you like me to keep them unexposed for now?
4132015-11-10T19:43:49  *** Guest8443 has joined #bitcoin-core-dev
4142015-11-10T19:44:42  <jgarzik> morcos, Respectfully submitted, wumpus is the release manager not The DecisionMaker - ask the crowd not the one
4152015-11-10T19:45:12  <jgarzik> There's a reason why I pushed Gavin hard for multiple committers when Satoshi disappeared - Gavin was release manager, not Bitcoin Leader
4162015-11-10T19:45:36  <morcos> jgarzik: sure, s/wumpus/everyone/ .  although i thought i recalled him expressing an opinion about rpc api churn before
4172015-11-10T19:50:37  <jgarzik> "release early, release often"  :)     IMO Expose them.   Maybe name the methods "beta.XXXX" to emphasize they are not final.   The goal is the communicate to users that churn will be forthcoming, but also expose them for testing and further field development.
4182015-11-10T19:52:10  <wumpus> morcos: I don't mind, if it's useful to have them then add them
4192015-11-10T19:52:30  <wumpus> morcos: if it is an unstable interface mention it in the help
4202015-11-10T19:53:38  <morcos> ok thanks
4212015-11-10T19:53:40  <wumpus> morcos: there's less an issue with adding new commands than changing old ones, especially adding arguments to existing calls is messy
4222015-11-10T19:53:54  <jgarzik> +1
4232015-11-10T19:53:55  *** zooko has quit IRC
4242015-11-10T19:54:16  <morcos> wumpus: yes that was my concern with the fact that this one might change or disappear later, but i'll clearly mention it
4252015-11-10T19:58:20  *** d4de has joined #bitcoin-core-dev
4262015-11-10T19:59:33  <wumpus> morcos: you could decide to not list them in the overview, like invalidateblock/reconsiderblock
4272015-11-10T20:00:51  <wumpus> (the "hidden" category)
4282015-11-10T20:01:27  *** skyraider has quit IRC
4292015-11-10T20:01:38  *** skyraider has joined #bitcoin-core-dev
4302015-11-10T20:08:28  <GitHub60> [bitcoin] laanwj closed pull request #6981: doc: Add note about SI units to developer notes (master...2015_11_si_units) https://github.com/bitcoin/bitcoin/pull/6981
4312015-11-10T20:11:03  <morcos> sipa: question re: in memory sizes..   i was thinking of making a quick pull that defaults the dbcache to 2 * maxmempool and the maxsigcachesize to 1/5 * maxmempool.  or do you think there is a better way to control total mem usage?
4322015-11-10T20:11:53  <morcos> i'd like to test out a couple of different configurations just to make sure there aren't any regressions with a particularly large or particularly small mempool, and i was hoping to have an idea of what a typical large configuration and typical small configuration might look like
4332015-11-10T20:13:08  <morcos> if those are the default ratios, then i'd just try out 50M mempool , 300M mempool and 2G mempool  (hmm  maybe 4G is too much for dbcache in that case?)  seems like it woudl be nice to just control 1 number
4342015-11-10T20:14:09  <jgarzik> nod - in general users should not need to fiddle N settings just to get a usable configuration
4352015-11-10T20:14:39  *** fkhan has quit IRC
4362015-11-10T20:14:43  <sipa> morcos: where do you 2G and 4G numbers from from?
4372015-11-10T20:16:09  <morcos> i didn't get it from anywhere, i was just goign to try out somethign thats pretty big.  i think 2G is about as big as mempools got in the recent spam attack, maybe between 2-3...  also easily enough for a weeks worth of backlog in txs which we had previously discussed aiming for
4382015-11-10T20:17:21  <morcos> mostly i want to see if any of the code that traverses the whole mempool gets a bit slow then or if the resorts from multindex changing get slow.
4392015-11-10T20:18:16  <sipa> I think 2G is too much..
4402015-11-10T20:18:49  <morcos> maybe i explained wrong
4412015-11-10T20:18:58  <morcos> i was going to leave maxmempool default at 300
4422015-11-10T20:19:17  <morcos> and change dbache default to 2x maxmempool and maxsigcachesize to 1/5
4432015-11-10T20:19:47  <morcos> and then i was going to test what happens if someone runs a particularly small or big node.
4442015-11-10T20:20:08  <morcos> but i think it makes sense as jgarzik says for their other defaults to scale with setting one value (unless they explicitly set them otherwise)
4452015-11-10T20:20:54  <morcos> so yeah 2G is huge, but thats sure what i'd do if i was a miner
4462015-11-10T20:22:24  <jgarzik> We're well away from miners caring about maximizing fees to an Nth degree...   Reliability, orphaning and other factors drive miner conservatism in settings like this...
4472015-11-10T20:23:35  <gmaxwell> The benchmarks w/ the libsecp256k1 pull really highlight the impact of increasing the size of the cache on IBD time.
4482015-11-10T20:24:09  <morcos> gmaxwell, if you're in IBD, we should steal maxmempool size for dbacache . :)
4492015-11-10T20:24:34  <sdaftuar> +1
4502015-11-10T20:25:29  <morcos> so the question is what should the defaults for each of these 3 things be.  and if you want to turn 1 knob to change to big mem footprint or small, how should that work?
4512015-11-10T20:25:50  <jgarzik> Related tech note - if one option sets another option, it becomes order-dependent
4522015-11-10T20:26:07  <jgarzik> i.e. set dbcache before mempoolsize, and dbcache gets stomped.
4532015-11-10T20:26:25  <jgarzik> Still, I think the base argument holds - users do not want to set N settings to achieve a useful config.
4542015-11-10T20:26:41  <morcos> jgarzik, sure if you change the order of lines of code things get messed up
4552015-11-10T20:26:56  <jgarzik> morcos, no, order of configuration file defines
4562015-11-10T20:27:04  <morcos> no i dont' think so
4572015-11-10T20:27:10  <sipa> ideally there is just a single setting for "amount of cache memory" which is sum of dbcache and mempool, and the flush criteron becomes "too large percentage of the memory is dirty cache entries"
4582015-11-10T20:27:32  <sipa> jgarzik: no, the logic for options settings other options is earlier in init.cpp
4592015-11-10T20:27:42  <jgarzik> ok, good
4602015-11-10T20:27:57  <wumpus> jgarzik: in case of bitcoin core that's noto the case, only still-unset settings get overridden
4612015-11-10T20:28:54  <morcos> sipa: so you still need a limit for the mempool right, which makes sense to be a fraction of your total
4622015-11-10T20:29:13  <morcos> you cant just go and trim your mempool a bunch when a whole new set of txins get cached
4632015-11-10T20:29:22  *** GAit has joined #bitcoin-core-dev
4642015-11-10T20:29:39  <wumpus> I don't think utxo cache and mempool are very related
4652015-11-10T20:30:01  <morcos> yeah i mostly agree with wumpus there
4662015-11-10T20:30:14  <sipa> wumpus: well they're both essentially caches of information that we'll except to need for validating the next block
4672015-11-10T20:30:21  <wumpus> enough reasons to increase one and not the other. I'm fine with setting some sane defaults, but they shouldn't be linked in general
4682015-11-10T20:30:21  <sipa> *expect
4692015-11-10T20:30:41  <sipa> except the utxo cache is *also* a buffer of unwritten changes
4702015-11-10T20:30:53  <wumpus> well a smaller mempool doens't make your node slower, a smaller dbcache does
4712015-11-10T20:30:55  <morcos> sipa: the mempool doesn't serve as a cache does it
4722015-11-10T20:30:58  <morcos> right
4732015-11-10T20:31:16  *** GAit has quit IRC
4742015-11-10T20:31:17  <wumpus> so even though they're 'caches' they have a widely different reason
4752015-11-10T20:31:19  *** fkhan has joined #bitcoin-core-dev
4762015-11-10T20:31:26  <wumpus> mempool is necessary for correct functioning, dbcache is just for performance
4772015-11-10T20:31:57  <morcos> wumpus, the idea of linking though is that why make the user figure out how to divide up his available N gigs of memory
4782015-11-10T20:32:03  <jgarzik> User experience.  What does the user need to do to achieve a useful config on both large & small boxes?
4792015-11-10T20:32:07  <morcos> we should just do soemthing vaguely smart by default
4802015-11-10T20:32:09  *** GAit has joined #bitcoin-core-dev
4812015-11-10T20:32:09  <jgarzik> nod
4822015-11-10T20:32:43  <wumpus> morcos: yes as said by default you could do that, make a -memoryquota parameter or such that automatically allocates them, like automatic partitioning in an OS instlal
4832015-11-10T20:32:55  <wumpus> morcos: but it should certainly be possible to set them separately
4842015-11-10T20:32:59  <dhill> getrlimit and base initial memory sizes on that?
4852015-11-10T20:33:02  <morcos> ok agreed
4862015-11-10T20:33:17  <wumpus> dhill: that assumes everyone wants to give all their memory to bitcoind
4872015-11-10T20:33:58  <dhill> naw
4882015-11-10T20:34:01  <wumpus> and no, just because I have 32GB of memory now doesn't mean I want to allocate it all to my node, and have compiles crash again...
4892015-11-10T20:34:02  *** jgarzik has quit IRC
4902015-11-10T20:34:10  <dhill> i didn't suggest that
4912015-11-10T20:34:19  <morcos> so -maxmempool default is some fraction of memoryquota which also has a default.  , but then if you individually set your maxmempool,sigcahe,or dbcache, you are not necessarily going to respect your memoryquota
4922015-11-10T20:34:51  <sipa> i think it makes sense for bitcoin-qt to do something smartish to guess how much memory to use for caches
4932015-11-10T20:35:15  *** jgarzik has joined #bitcoin-core-dev
4942015-11-10T20:35:31  <morcos> well if we do this memory quota idea, then we can always have smart code that sets that as a later addition if we want
4952015-11-10T20:35:45  <wumpus> agreed, doing something smart is good, but basing it (trivially) on the available memory does't make too much sense imo
4962015-11-10T20:36:08  <wumpus> people generally want to run their node inthe background
4972015-11-10T20:36:12  <morcos> so back to my original quesiton  if i set memoryquota=N.   what should the 3 components each be
4982015-11-10T20:36:30  <wumpus> having applications use more memory just because it is available, and no othe reason is...weird
4992015-11-10T20:36:42  *** jgarzik has quit IRC
5002015-11-10T20:37:00  <morcos> .4 N for maxmempol, .5N for dbacache , .1N for maxsigcachesize ?
5012015-11-10T20:37:04  <wumpus> I would be angry if firefox used more memory per tab just because I have more, that's so self defeating
5022015-11-10T20:37:15  <morcos> sdaftuar was trying to convince me 2 * is too much for dbcache
5032015-11-10T20:38:42  <morcos> and default memory quota to 800M which should be around the 300M mempool we had previously been planning for, and keep total usage to under a gig (ugh, i hope, what else uses memory)
5042015-11-10T20:38:44  <wumpus> dbcache is nice but apart from during IBD ,not too important
5052015-11-10T20:39:32  <morcos> wumpus: i like the idea that block relay over p2p is very fast.  i think that depending on the relay network for efficient block relay is a bad idea, even if p2p will never be quite as fast
5062015-11-10T20:39:34  <wumpus> I sort of like the idea to use the mempool quota for UTXO cache during IBD, after all the mempool will be almost empty at that time
5072015-11-10T20:39:56  <wumpus> though making them linked is less nice for seperation of concerns
5082015-11-10T20:40:04  <morcos> for block relay to be fast, you need decent dbcache size
5092015-11-10T20:43:09  *** paveljanik has quit IRC
5102015-11-10T20:43:18  <wumpus> a better eviction policy would help too there :)
5112015-11-10T20:44:44  *** jgarzik has joined #bitcoin-core-dev
5122015-11-10T20:44:44  *** jgarzik has joined #bitcoin-core-dev
5132015-11-10T20:44:59  <morcos> i have a PR for that.  :)  but it requires still a decent size dbcache.  a 1MB block might need over 50MB of dbcache just for itself, if you knew exactly what was goign to be confirmed in that block
5142015-11-10T20:46:49  <wumpus> ok cool
5152015-11-10T20:47:01  *** zooko has joined #bitcoin-core-dev
5162015-11-10T20:48:04  <jgarzik> "having applications use more memory just because it is available, and no othe reason is...weird"   <--  That's pretty much exactly how OS page cache works
5172015-11-10T20:48:28  <jgarzik> And I'm looking at how memory usage changes once mempool is on disk, and nearly everything goes through OS page cache
5182015-11-10T20:49:37  <wumpus> yes at the OS level you can get away wit hit
5192015-11-10T20:50:21  <wumpus> e.g. page cache isn't considered 'used' memory in most computations, it can be evicted when an application really needs it
5202015-11-10T20:52:23  <jgarzik> quite true if you s/application/OS/   -- though "working set" is part of the app.  If working set size gets below a threshold, performance falls due to increased I/O.   As such, it is part of the app, and one that scales when more memory is available.
5212015-11-10T20:52:27  <wumpus> you really can't do that in a well-behaving application - yes I know it's possible to use some madvise trick to map application pages as volatile, but supporting that cross platform will be a nightmare, and it will probably still look to the user as if it uses a lot of memory unconditionally
5222015-11-10T20:55:27  *** Guest8443 is now known as berndj
5232015-11-10T20:57:14  *** zooko has quit IRC
5242015-11-10T20:58:18  *** CodeShark has joined #bitcoin-core-dev
5252015-11-10T20:58:36  <wumpus> yes I wasn't talking about the page cache, but a hypothetical application that would ask the OS how much memory is available and claim a fixed part of that. It would make the egotistical assumption that a user buys more memory just to give your application more space :)
5262015-11-10T20:59:10  <jgarzik> hehe
5272015-11-10T21:00:19  *** zooko has joined #bitcoin-core-dev
5282015-11-10T21:05:23  <wumpus> it would be very nice if we could rely on the page cache though for caching the db, this whole utxo cache is mostly a workaround because there is a lot of retrieve overhead even if the data is cached in RAM (do we even know why? deserialization/allocation overhead?)
5292015-11-10T21:06:41  <gmaxwell> has nothing to do with storage vs ram, you get basically the same speedups from big utxo cache even if the datadir is in tmpfs.
5302015-11-10T21:07:49  <morcos> gmaxwell: oh really?? why is that
5312015-11-10T21:08:27  <wumpus> that's what I mean with overhead even if the data is already in RAM
5322015-11-10T21:09:19  <wumpus> storing the datadir in tmpfs is an extreme example of that, but normally the page cache would make sure at least a part of the files are still cached
5332015-11-10T21:10:10  <wumpus> morcos: that's what I wonder too, probably deserialization/allocation overhead, going from storage representation to internal data structure
5342015-11-10T21:11:29  *** belcher has joined #bitcoin-core-dev
5352015-11-10T21:12:21  <gmaxwell> I had a prior belief but pieter tells me my understanding of leveldb's behavior was not correct. (I thought leveldb did a sequential scan of the log to find any records there)
5362015-11-10T21:12:22  <wumpus> another possibility is that leveldb queries are really slow even if there is no seek/io overhead - maybe because of the all-pervasive checksum verification
5372015-11-10T21:13:10  <sipa> gmaxwell: the log is sequential, but is never read
5382015-11-10T21:13:31  <wumpus> anecdotally when I broke off block verification a few times in gdb it always ended up in leveldb checksum verification. That's no serious way to do profiling though :-)
5392015-11-10T21:13:35  <sipa> gmaxwell: it's compacted into the database at startup (which is why startup after shutdown with high dbcache is slow, it has large log to process)
5402015-11-10T21:14:14  <sipa> the database however has several levels, and several (non-overlapping) files for every level
5412015-11-10T21:14:26  <sipa> which means it may need to scan through multiple levels to find data
5422015-11-10T21:14:50  <sipa> every file has a bloom filter, so it will quickly skip the ones that don't have the requested amount
5432015-11-10T21:14:58  <wumpus> right
5442015-11-10T21:15:05  <sipa> within every file, it uses a bisection search to find the data, i think
5452015-11-10T21:15:21  <wumpus> yes IIRC too
5462015-11-10T21:15:24  <sipa> or a bisection search in its index
5472015-11-10T21:15:27  <sipa> which points to the data
5482015-11-10T21:18:14  *** GAit has quit IRC
5492015-11-10T21:18:21  *** GAit has joined #bitcoin-core-dev
5502015-11-10T21:26:45  *** GAit has quit IRC
5512015-11-10T21:28:39  *** GAit has joined #bitcoin-core-dev
5522015-11-10T21:32:03  *** dcousens has joined #bitcoin-core-dev
5532015-11-10T21:34:17  *** molly has quit IRC
5542015-11-10T21:40:21  *** ParadoxSpiral_ has quit IRC
5552015-11-10T21:56:34  *** CodeShark_ has joined #bitcoin-core-dev
5562015-11-10T21:56:45  *** CodeShark_ has quit IRC
5572015-11-10T21:57:54  *** CodeShark_ has joined #bitcoin-core-dev
5582015-11-10T21:58:19  *** CodeShark has quit IRC
5592015-11-10T22:04:54  *** Guyver2 has quit IRC
5602015-11-10T22:32:30  *** GAit has quit IRC
5612015-11-10T22:33:17  *** GAit has joined #bitcoin-core-dev
5622015-11-10T22:44:58  *** CodeShark_ has quit IRC
5632015-11-10T22:45:11  *** CodeShark has joined #bitcoin-core-dev
5642015-11-10T22:52:01  *** GAit has quit IRC
5652015-11-10T22:53:16  *** GAit has joined #bitcoin-core-dev
5662015-11-10T22:57:02  *** GAit has quit IRC
5672015-11-10T23:15:38  *** molly has joined #bitcoin-core-dev
5682015-11-10T23:58:31  <gmaxwell> Would having a protocol message that says "please don't INV loose transactions to me, I'm mostly only interested in transactions in blocks"  be a bad idea?  I want to have bandwidth limited nodes that don't care about unconfirmed txn not recieve anything but confirmed transactions.