12015-10-28T00:01:19  *** zooko has quit IRC
  22015-10-28T00:07:08  *** Thireus1 has quit IRC
  32015-10-28T00:07:23  *** Thireus has joined #bitcoin-core-dev
  42015-10-28T00:09:49  *** zooko has joined #bitcoin-core-dev
  52015-10-28T00:12:39  *** Thireus1 has joined #bitcoin-core-dev
  62015-10-28T00:13:59  *** Thireus2 has joined #bitcoin-core-dev
  72015-10-28T00:15:10  *** Thireus has quit IRC
  82015-10-28T00:17:57  *** Thireus1 has quit IRC
  92015-10-28T00:21:01  <GitHub30> [bitcoin] sipa opened pull request #6895: Update to my new key (master...newkey) https://github.com/bitcoin/bitcoin/pull/6895
 102015-10-28T00:36:07  *** Apocalyptic has quit IRC
 112015-10-28T00:42:41  *** Apocalyptic has joined #bitcoin-core-dev
 122015-10-28T00:55:23  *** zooko has quit IRC
 132015-10-28T01:07:31  <GitHub19> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8f3b3cdee497...d0badb916e51
 142015-10-28T01:07:31  <GitHub19> bitcoin/master 298e040 Pieter Wuille: Fix chainstate serialized_size computation
 152015-10-28T01:07:32  <GitHub19> bitcoin/master d0badb9 Gregory Maxwell: Merge pull request #6865...
 162015-10-28T01:07:37  <GitHub22> [bitcoin] gmaxwell closed pull request #6865: Fix chainstate serialized_size computation (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6865
 172015-10-28T01:20:28  *** treehug88 has quit IRC
 182015-10-28T01:22:14  <GitHub43> [bitcoin] sipa pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/d0badb916e51...93521a4f56ce
 192015-10-28T01:22:15  <GitHub43> bitcoin/master 27252b7 Matt Corallo: Fix pre-push-hook regexes
 202015-10-28T01:22:15  <GitHub43> bitcoin/master 1d94b72 Matt Corallo: Whitelist commits signed with Pieter's now-revoked key
 212015-10-28T01:22:16  <GitHub43> bitcoin/master 6e800c2 Matt Corallo: Add Pieter's new PGP key to verify-commits/trusted-keys
 222015-10-28T01:22:24  <GitHub113> [bitcoin] sipa closed pull request #6875: Fix pre-push-hook regexes (master...verify-commits-fixes) https://github.com/bitcoin/bitcoin/pull/6875
 232015-10-28T01:24:03  <GitHub157> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/93521a4f56ce...8756c986420c
 242015-10-28T01:24:03  <GitHub157> bitcoin/master 4252cd0 Pieter Wuille: Update to my new key
 252015-10-28T01:24:04  <GitHub157> bitcoin/master 8756c98 Pieter Wuille: Merge pull request #6895...
 262015-10-28T01:24:13  <GitHub179> [bitcoin] sipa closed pull request #6895: Update to my new key (master...newkey) https://github.com/bitcoin/bitcoin/pull/6895
 272015-10-28T01:25:32  <GitHub93> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8756c986420c...e06c14fb59ee
 282015-10-28T01:25:32  <GitHub93> bitcoin/master ab1f560 Pieter Wuille: Support -checkmempool=N, which runs checks on average once every N transactions
 292015-10-28T01:25:33  <GitHub93> bitcoin/master e06c14f Pieter Wuille: Merge pull request #6776...
 302015-10-28T01:25:37  <GitHub73> [bitcoin] sipa closed pull request #6776: Support -checkmempool=N, which runs checks once every N transactions (master...fraccheck) https://github.com/bitcoin/bitcoin/pull/6776
 312015-10-28T01:31:04  <GitHub67> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e06c14fb59ee...4764f5db9d2c
 322015-10-28T01:31:04  <GitHub67> bitcoin/master 214de7e Philip Kaufmann: [Trivial] ensure minimal header conventions...
 332015-10-28T01:31:05  <GitHub67> bitcoin/master 4764f5d Pieter Wuille: Merge pull request #6892...
 342015-10-28T01:31:12  <GitHub110> [bitcoin] sipa closed pull request #6892: [Trivial] ensure minimal header conventions (master...headers-new) https://github.com/bitcoin/bitcoin/pull/6892
 352015-10-28T01:34:35  <GitHub1> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/4764f5db9d2c...8daffe227bc6
 362015-10-28T01:34:35  <GitHub1> bitcoin/master ad5aae1 Philip Kaufmann: constify missing catch cases...
 372015-10-28T01:34:36  <GitHub1> bitcoin/master 8daffe2 Pieter Wuille: Merge pull request #6891...
 382015-10-28T01:34:44  <GitHub130> [bitcoin] sipa closed pull request #6891: constify missing catch cases (master...const-ex) https://github.com/bitcoin/bitcoin/pull/6891
 392015-10-28T01:46:53  *** d_t has quit IRC
 402015-10-28T01:53:39  *** jtimon has quit IRC
 412015-10-28T02:01:51  <gmaxwell> Oh I think I know one reason we see corruption reports on windows.
 422015-10-28T02:02:34  <gmaxwell> Stopnode often takes an awful long time, for example on my laptop running defaults it just took 12 seconds.  In windows, on shutdown, tasks that don't stop pretty much immediately get killed.
 432015-10-28T02:02:54  <gmaxwell> thats probably a maximally bad case for leveldb as it'll be in the middle of flushing out a bunch of cached updates.
 442015-10-28T02:10:09  <GitHub4> [bitcoin] sipa opened pull request #6896: Make -checkmempool=1 not fail through int32 overflow (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6896
 452015-10-28T02:32:51  *** belcher has quit IRC
 462015-10-28T02:35:12  <tripleslash> gmaxwell: its specifically HungAppTimeout and is defaulted to 5 seconds.
 472015-10-28T02:38:02  <morcos> test_bitcoin hangs for me a decent fraction of the time.  It looks like it gets stuck at line 109 in scheduler_tests.cpp
 482015-10-28T02:56:22  <morcos> ah, i see it is an issue, #6540
 492015-10-28T03:12:09  *** fkhan has quit IRC
 502015-10-28T03:44:21  *** Ylbam has quit IRC
 512015-10-28T04:02:12  *** baldur has quit IRC
 522015-10-28T04:02:28  *** baldur has joined #bitcoin-core-dev
 532015-10-28T04:03:44  *** blur3d has joined #bitcoin-core-dev
 542015-10-28T05:12:10  *** blur3d has quit IRC
 552015-10-28T05:24:42  *** gribble has quit IRC
 562015-10-28T05:27:20  *** gribble has joined #bitcoin-core-dev
 572015-10-28T05:45:39  *** zooko has joined #bitcoin-core-dev
 582015-10-28T06:27:19  *** ParadoxSpiral has joined #bitcoin-core-dev
 592015-10-28T06:36:29  *** gavinandresen has quit IRC
 602015-10-28T06:47:26  *** ParadoxSpiral has quit IRC
 612015-10-28T07:08:59  *** deepcore has joined #bitcoin-core-dev
 622015-10-28T07:11:14  *** fkhan has joined #bitcoin-core-dev
 632015-10-28T07:25:03  *** crescendo has quit IRC
 642015-10-28T07:29:23  *** deepcore has quit IRC
 652015-10-28T07:40:31  *** Ylbam has joined #bitcoin-core-dev
 662015-10-28T08:13:08  <wumpus> gmaxwell: that will likely contribute to it, although most of the time corruption seems to happen on crashes / power failures, when there is no time to flush at all
 672015-10-28T08:27:46  <wumpus> but if that is the case too then the flush+sync on windows is not essentially not working at all
 682015-10-28T08:27:59  <wumpus> -first not
 692015-10-28T08:40:38  <gmaxwell> someone was commenting that we were writing via mmap on windows and that the sync we were using there didn't work on maps; which sounds like the mac problem. I didn't verify these claims at all.
 702015-10-28T08:58:35  *** rubensayshi has joined #bitcoin-core-dev
 712015-10-28T09:01:31  *** BashCo has quit IRC
 722015-10-28T09:11:19  *** aaaaok has joined #bitcoin-core-dev
 732015-10-28T09:11:36  <wumpus> didn't check that either
 742015-10-28T09:21:23  *** BashCo has joined #bitcoin-core-dev
 752015-10-28T09:26:11  *** BashCo has quit IRC
 762015-10-28T09:53:00  *** jtimon has joined #bitcoin-core-dev
 772015-10-28T10:03:48  <dcousens> wumpus: aye, 1 OOM and my chain was broke
 782015-10-28T10:22:48  *** aaaaok has quit IRC
 792015-10-28T10:47:03  *** evoskuil has quit IRC
 802015-10-28T11:57:57  *** fanquake has joined #bitcoin-core-dev
 812015-10-28T12:05:05  *** fanquake has quit IRC
 822015-10-28T12:05:28  *** fanquake has joined #bitcoin-core-dev
 832015-10-28T12:20:22  *** fanquake has left #bitcoin-core-dev
 842015-10-28T12:48:53  *** paveljanik has joined #bitcoin-core-dev
 852015-10-28T12:48:53  *** paveljanik has joined #bitcoin-core-dev
 862015-10-28T12:57:57  *** treehug88 has joined #bitcoin-core-dev
 872015-10-28T13:02:39  *** randy-waterhouse has joined #bitcoin-core-dev
 882015-10-28T13:03:44  *** randy-waterhouse has quit IRC
 892015-10-28T13:09:19  *** molly has joined #bitcoin-core-dev
 902015-10-28T13:12:19  *** moli has quit IRC
 912015-10-28T13:14:02  *** dcousens has quit IRC
 922015-10-28T13:56:06  *** zooko has quit IRC
 932015-10-28T13:56:59  *** gavinandresen has joined #bitcoin-core-dev
 942015-10-28T13:57:31  *** zooko has joined #bitcoin-core-dev
 952015-10-28T14:02:59  *** davec has quit IRC
 962015-10-28T14:19:04  *** davec has joined #bitcoin-core-dev
 972015-10-28T14:22:00  *** moli has joined #bitcoin-core-dev
 982015-10-28T14:25:19  *** molly has quit IRC
 992015-10-28T14:55:27  <jcorgan> cfields: did you ever make progress on #6681?
1002015-10-28T14:58:40  <jcorgan> i guess that would be #6819 now
1012015-10-28T15:10:14  *** MarcoFalke has joined #bitcoin-core-dev
1022015-10-28T15:16:32  <cfields> jcorgan: no, i stopped there. I just wanted to get it building so that someone who knows zmq could make it actually work
1032015-10-28T15:22:27  <jcorgan> got it
1042015-10-28T15:32:18  *** ParadoxSpiral has joined #bitcoin-core-dev
1052015-10-28T15:38:58  *** bsm1175321 has quit IRC
1062015-10-28T15:47:06  *** MarcoFalke has quit IRC
1072015-10-28T16:00:06  *** MarcoFalke has joined #bitcoin-core-dev
1082015-10-28T16:02:57  *** bsm1175321 has joined #bitcoin-core-dev
1092015-10-28T16:03:05  *** jl2012 has quit IRC
1102015-10-28T16:03:17  *** jl2012 has joined #bitcoin-core-dev
1112015-10-28T16:46:49  *** jl2012_ has joined #bitcoin-core-dev
1122015-10-28T16:47:39  *** jl2012 has quit IRC
1132015-10-28T16:57:43  *** deepcore has joined #bitcoin-core-dev
1142015-10-28T17:07:40  *** d_t has joined #bitcoin-core-dev
1152015-10-28T17:28:16  *** molly has joined #bitcoin-core-dev
1162015-10-28T17:31:19  *** moli has quit IRC
1172015-10-28T17:47:48  <morcos> gmaxwell: For this fast template generation on new block code.  Are you envisioning you switch to a new empty template after a new most work header?  Even if you haven't validated the block you're connecting yet?  And then set some sort of timeout with which you'll switch back to not being willing to build off a headers only chain?
1182015-10-28T17:48:51  <morcos> Once you've connected a new block, I'd say there is no reason not to wait the extra few ms to generate a block template with txs in it.  Which can then be validated after its already been served to you.
1192015-10-28T17:49:02  <gmaxwell> No I was not. This is less safe than many people think it is, and I think not needed if the other details are handled correctly.
1202015-10-28T17:49:47  <morcos> Well thats where all the delay is right?  Receiving and connecting the new best block.
1212015-10-28T17:49:49  <sipa> how about building a new template, switching to working on it immediately, and then starting a validation for it
1222015-10-28T17:50:32  <morcos> sipa: yes thats what i'm suggesting.  but thats after you've connected the best block.  if we're still requiring to wait for that, don't we think people will choose to short circuit it less safely on their own
1232015-10-28T17:50:34  <gmaxwell> morcos: No, CNB latency is tens of times slower than validating normally.
1242015-10-28T17:51:45  <morcos> gmaxwell: ok, i'll give you multiples, but probably less than 10x unless your mempool is really really big.  and thats just compared to validating, what about waiting to receive?
1252015-10-28T17:52:16  <morcos> i haven't looked at the time delay from receiving most work header to finishing connecting the block, any idea what that is typically?
1262015-10-28T17:52:19  <morcos> i bet its long
1272015-10-28T17:52:20  <gmaxwell> When the relay network is working normally about 80% of blocks are transmitted in a single packet.
1282015-10-28T17:52:30  <morcos> ah yes, forgot about relay network
1292015-10-28T17:52:42  <morcos> thats why i ask questions
1302015-10-28T17:53:30  <sipa> it still makes sense to have numbers for the time between receive inv and CNB building a template on top
1312015-10-28T17:53:39  <gmaxwell> Most of the delays in mining right now appear to be from outside of bitcoin core, actually.
1322015-10-28T17:54:00  <morcos> sipa in a relay node connected case or regular node or both
1332015-10-28T17:54:20  <sipa> morcos: in "reality"
1342015-10-28T17:54:22  <sipa> :)
1352015-10-28T17:55:08  <morcos> ok, well i'm almost ready to push a WIP branch.  it still doesn't do it in a thread, but the gain is really rather limited at this point, and i'll save that i think for a second pull
1362015-10-28T17:55:21  <morcos> but the question i want to resolve is what to do when TestBlockValidity fails
1372015-10-28T17:55:32  <morcos> right now it throws an error
1382015-10-28T17:55:34  <gmaxwell> hm. we've done something recently that slowed down connectblock (or the network behavior changes have)
1392015-10-28T17:55:51  <gmaxwell> oh dear
1402015-10-28T17:55:53  <morcos> connectblock has always been slow since i've measured it
1412015-10-28T17:56:05  <gmaxwell> debug.log:2015-10-28 16:11:38 - Connect block: 7256.55ms [7475.14s]
1422015-10-28T17:56:39  <sipa> how much is fetching inputs vs verifying inputs?
1432015-10-28T17:56:47  <gmaxwell> wtf.
1442015-10-28T17:57:07  <gmaxwell> since my node here's last update to master connect block's time has increased monotonically for every block.
1452015-10-28T17:58:02  <sipa> gmaxwell: coincache being thrown out my mempool bloat?
1462015-10-28T17:59:48  <morcos> gmaxwell, what block hash was that?
1472015-10-28T18:01:41  <morcos> gmaxwell: are you still running mempool.check?  it runs inside that timer
1482015-10-28T18:01:43  <gmaxwell> morcos: before this latest rounds of attack this node was taking <100ms to connect block.
1492015-10-28T18:01:58  <gmaxwell> morcos: must be the mempool checks then.
1502015-10-28T18:02:29  <gmaxwell> morcos: it's all of them.
1512015-10-28T18:02:34  <sipa> gmaxwell: turning on mempool checks are a certain way to blow away your cache every block
1522015-10-28T18:02:48  <morcos> the blocks around that time for me took 500ms and then 1ms (only coinbase)
1532015-10-28T18:04:41  <gmaxwell> in any case, debug logs ran this thing out of space a couple hours ago, so I've restarted it. I'll run without mempool checks to get some good timings.
1542015-10-28T18:05:02  <gmaxwell> Numers I posted from shortly before the MTL event were about 80ms.
1552015-10-28T18:06:52  <morcos> 80ms?? hmm...
1562015-10-28T18:07:47  <sipa> so with a 200 MB mempool i seem to need a 700-1200 MB coincache
1572015-10-28T18:07:52  <sipa> with matt's latest patch
1582015-10-28T18:09:22  <sipa> i wonder if we should (as a short term hack) treat some factor of the size-of-pulled-in-memory of a tx as its txsize
1592015-10-28T18:11:38  <morcos> gmaxwell:  i looked at some old numbers of mine (couple of months ago) and they were like 500ms on average (during fairly busy time, mid July)
1602015-10-28T18:11:48  <gmaxwell> morcos: Interesting!
1612015-10-28T18:12:31  <morcos> sipa: so what do you mean "need"?
1622015-10-28T18:13:02  <morcos> a 200MB mempool isn't really the right measure right
1632015-10-28T18:13:06  *** d_t has quit IRC
1642015-10-28T18:13:52  <sipa> morcos: how do you mean?
1652015-10-28T18:13:53  <morcos> because as you go from 0 - 200MB you'll pull in a certain amount of txin's.  but as you keep running, youre mempool stays at 200MB, but you'll i think tend to pull in more txin's?   or does matt's patch actually remove txin's no longer requested when a tx gets mined?
1662015-10-28T18:14:51  <sipa> when a tx gets mined it goes into the cache with dirty bit on
1672015-10-28T18:15:03  <sipa> so it can't be removed from the cache anymore
1682015-10-28T18:15:06  <sipa> until a flush
1692015-10-28T18:15:23  <sipa> but just a 700 MB coincache sounds pretty painful already
1702015-10-28T18:16:48  <morcos> yes i think 700MB is pretty painful, but we need to think a bit about how to be smarter about it
1712015-10-28T18:17:49  <sipa> per-txout cache...
1722015-10-28T18:18:41  <morcos> yeah if it was pertxout, then you'd solve that problem
1732015-10-28T18:18:55  <morcos> every so often you don't flush your cache, you just write out the dirty entries
1742015-10-28T18:19:20  <morcos> we could still do that now, except we don't know if that tx should still be in cache b/c of other mempool txs
1752015-10-28T18:19:46  <sipa> well an lru or random eviction of the utxo set could work too
1762015-10-28T18:19:55  <sipa> the cache would just become less effective
1772015-10-28T18:23:30  *** rubensayshi has quit IRC
1782015-10-28T18:25:30  <morcos> so when we flush the cache, we have to write everything correct?  so its consistent.  i think if we could jsut get smarter about what we wiped from the cache at that point, then maybe we could jsut flush a lot more often, or do you think that would be bad.
1792015-10-28T18:26:09  <sipa> yes, a flush shouldn't wipe everything
1802015-10-28T18:26:44  <morcos> tell me if this would be too cumbersome.  i think it might be pretty fast, how long does a flush take?
1812015-10-28T18:27:50  <morcos> after you write everything, you quickly scan the top 10MB of txs in the mempool, and insert all of their txin.prevout.hash's into a set, and then you iterate throught the cachemap erasing anything thats not one of those
1822015-10-28T18:30:57  *** MarcoFalke has quit IRC
1832015-10-28T18:33:35  <morcos> so actually flushing takes over a second right?  i think you might be able to do something like i suggested on the order of 10's of ms, but not sure
1842015-10-28T18:33:52  *** MarcoFalke has joined #bitcoin-core-dev
1852015-10-28T18:35:31  *** MarcoFalke has quit IRC
1862015-10-28T18:36:11  <morcos> i don't know if it's bad (or maybe its even good) to flush more regularly.  but if we did something like that, we wouldn't even need to worry about matt's patch.  we could just "flush" every time the cache was getting too big.
1872015-10-28T18:37:40  *** MarcoFalke has joined #bitcoin-core-dev
1882015-10-28T18:38:41  *** MarcoFalke has quit IRC
1892015-10-28T18:38:57  *** MarcoFalke has joined #bitcoin-core-dev
1902015-10-28T18:39:01  *** treehug88 has quit IRC
1912015-10-28T18:41:06  *** d_t has joined #bitcoin-core-dev
1922015-10-28T18:43:56  *** PaulCapestany has quit IRC
1932015-10-28T18:45:42  *** PaulCapestany has joined #bitcoin-core-dev
1942015-10-28T18:51:58  <morcos> re: TestBlockValidity failing.  I think I'm going to log an error and return NULL.  Seems better than throwing an error.  I'd like to reuse FormatStateMessage (in main.cpp), should I move it to a different file, or just declare it in main.h?
1952015-10-28T18:54:32  *** MarcoFalke has quit IRC
1962015-10-28T19:34:29  *** hustler has joined #bitcoin-core-dev
1972015-10-28T19:38:35  *** hustler has left #bitcoin-core-dev
1982015-10-28T19:56:19  *** zxzzt has quit IRC
1992015-10-28T19:56:56  *** sdaftuar has quit IRC
2002015-10-28T19:57:01  *** morcos has quit IRC
2012015-10-28T19:58:01  *** zxzzt has joined #bitcoin-core-dev
2022015-10-28T19:58:26  *** sdaftuar has joined #bitcoin-core-dev
2032015-10-28T19:58:58  *** morcos has joined #bitcoin-core-dev
2042015-10-28T20:09:28  *** zooko has quit IRC
2052015-10-28T20:11:31  *** zooko has joined #bitcoin-core-dev
2062015-10-28T20:11:39  *** belcher has joined #bitcoin-core-dev
2072015-10-28T20:45:54  *** CodeShark has joined #bitcoin-core-dev
2082015-10-28T20:47:41  *** d_t has quit IRC
2092015-10-28T20:54:57  <Luke-Jr> morcos: ⁇? if the block is invalid you do NOT want to use the template ever
2102015-10-28T20:55:21  <Luke-Jr> morcos: pretty sure you would entirely break proposals too
2112015-10-28T20:55:42  <morcos> Luke-Jr: I know you don't want to use it, the question is how to handle that case.
2122015-10-28T20:56:01  <morcos> It means code is broken somewhere, so human intervention is going to be required at some point
2132015-10-28T20:56:53  <morcos> The existing code would have thrown an error.  I chose to return NULL and log the error (which will cause getblocktemplate to throw a now misnamed JSONRPC error)
2142015-10-28T20:57:23  <morcos> Another option would be to try to return a template with no tx's instead (since the likely bug is mempool consistency was broken)
2152015-10-28T20:57:53  <morcos> I'm about to PR as a WIP... , would be great if you want to take a look
2162015-10-28T20:58:22  <gmaxwell> morcos: I vaguly recall something that if createnewblock can fail there is a crash elsewher.e
2172015-10-28T20:58:25  <GitHub181> [bitcoin] morcos opened pull request #6898: [WIP] Rewrite CreateNewBlock (master...fasterCNB) https://github.com/bitcoin/bitcoin/pull/6898
2182015-10-28T20:58:37  <gmaxwell> or maybe I also fixed that in anticipation of people making it possible to fail again.
2192015-10-28T20:59:30  <morcos> It took a long time to make it produce the exact same blocks as the old code, but helped work out a couple of bugs.
2202015-10-28T20:59:41  <Luke-Jr> IMO human intervention is preferable to silently mining empty blocks
2212015-10-28T21:00:00  <Luke-Jr> and debug.log is not non-silent
2222015-10-28T21:00:13  <gmaxwell> Most miners will never see anything in debug.log.
2232015-10-28T21:00:16  <morcos> I really hate the ugliness around hacking in the ability to still do priority space in the blocks.
2242015-10-28T21:00:54  <Luke-Jr> morcos: if that requires ugliness, the old code was better ?/
2252015-10-28T21:00:59  <Luke-Jr> :\
2262015-10-28T21:01:17  <morcos> well, I'd call the old code ugly as well.
2272015-10-28T21:01:22  <gmaxwell> Luke-Jr: the old code is very ugly.
2282015-10-28T21:01:47  <Luke-Jr> slow != ugly
2292015-10-28T21:01:47  <morcos> Also I didn't clean up the whole thing, I just am putting this out there for proof of concept.
2302015-10-28T21:02:32  <Luke-Jr> but I cant see the new code to compare yet..
2312015-10-28T21:03:45  <morcos> Luke-Jr: the big problem is priority is very difficult to calculate
2322015-10-28T21:04:27  <Luke-Jr> ? no
2332015-10-28T21:04:33  <morcos> If there is a consensus that its an important metric to keep, then I think we should #6357 which would make it much faster to calculate the correct priority in the mining code
2342015-10-28T21:04:49  <morcos> however it'll still be impossible to keep a sort based on it (i think)
2352015-10-28T21:05:04  <morcos> it changes!
2362015-10-28T21:05:09  <Luke-Jr> you need to lookup the inputs anyway
2372015-10-28T21:05:15  <morcos> not anymore
2382015-10-28T21:06:18  <morcos> but even if you do, the biggest problem i see with the old code, is that you have to look up the inputs for ALL the txs in your mempool
2392015-10-28T21:06:24  <morcos> not just the ones you're putting in a block
2402015-10-28T21:06:40  <Luke-Jr> hmm
2412015-10-28T21:06:45  <morcos> and if you're going to do a priority portion of the block, you have to keep doing that
2422015-10-28T21:07:06  <Luke-Jr> resorting once per block seems reasonable imo?
2432015-10-28T21:07:34  <morcos> although maybe the dynamic priority calculation would fix that, i haven't looked at it in a while..   and maybe it could be made even easier now with the concept of mempool children
2442015-10-28T21:08:31  <morcos> Luke-Jr: you're right, that would be the way to improve this code if priority isn't going away
2452015-10-28T21:09:16  <Luke-Jr> morcos: I still plan to redo all this btw :p
2462015-10-28T21:09:27  <morcos> keep an index (either part of the multi-index or separate) of all the priorities sorted, and only update it once per block
2472015-10-28T21:09:31  <morcos> yeah me too! :)
2482015-10-28T21:09:41  <Luke-Jr> so the mempool is a list of block templates
2492015-10-28T21:10:05  <morcos> but i was hoping we might be able to get something simple'ish done for 0.12, which would make GBT run a lot faster.
2502015-10-28T21:10:11  <gmaxwell> morcos: the input look up problem exists for fees too. :(
2512015-10-28T21:10:24  <morcos> gmaxwell: they are already stored in CTxMemPoolEntries
2522015-10-28T21:10:32  <morcos> and now i addded sigops to that too
2532015-10-28T21:10:41  <gmaxwell> Oh I see right fees can be cached but priority cannot.
2542015-10-28T21:10:57  <Luke-Jr> right
2552015-10-28T21:11:49  <Luke-Jr> priority is our best metric right now I think, so I wouldnt want to lose ikt even temporarily
2562015-10-28T21:12:39  <morcos> best metric for what?  why do you think its better than fees?
2572015-10-28T21:12:54  <gmaxwell> Luke-Jr: I don't really agree there. Priority works fine for you and I, I don't think he serves most users all that well.
2582015-10-28T21:13:11  <Luke-Jr> morcos: spammers are happy to pay fees
2592015-10-28T21:13:34  <morcos> Luke-Jr: yeah i actually agree its a pretty good anti-spam mechanism
2602015-10-28T21:13:38  <morcos> but thats not how we use it now!
2612015-10-28T21:14:01  <Luke-Jr> we do both now
2622015-10-28T21:14:37  <Luke-Jr> gmaxwell: imo thats why it isnt *exclusively* priority
2632015-10-28T21:14:55  <gmaxwell> Luke-Jr: point was spam goes through currently.
2642015-10-28T21:14:58  <morcos> Luke-Jr: there is also the problem of incentives
2652015-10-28T21:15:21  <Luke-Jr> gmaxwell: not via priority…?
2662015-10-28T21:15:29  <morcos> how do you make miners prefer priority vs fee
2672015-10-28T21:15:51  <Luke-Jr> morcos: you dont
2682015-10-28T21:16:23  <phantomcircuit> wumpus, i haven't forgotten about getting you a copy of a corrupted datadir btw
2692015-10-28T21:16:24  <Luke-Jr> longterm fees are the only realistic metric
2702015-10-28T21:16:28  <morcos> Luke-Jr: so you view priority as like an HOV lane.  at least some txs will sneak past even if the spam is causing congestion on most of the block
2712015-10-28T21:16:39  <Luke-Jr> but for now we want to try to keep fees low
2722015-10-28T21:16:40  <gmaxwell> morcos: and it does have that effect.
2732015-10-28T21:17:00  <Luke-Jr> morcos: basically
2742015-10-28T21:17:21  <Luke-Jr> also a nice fallback
2752015-10-28T21:17:24  *** paveljanik has quit IRC
2762015-10-28T21:17:26  <morcos> gmaxwell: if we care about preserving that, why don't we just redefine priority, to be your priority at the time the tx was accepted
2772015-10-28T21:17:50  <morcos> then it can be cached and its easy to reason about and who cares if different nodes/miners calculate it differently
2782015-10-28T21:17:50  <Luke-Jr> as long as we have priority, every tx can get confirmed *eventually*
2792015-10-28T21:17:56  <gmaxwell> morcos: we could expect then it'll turn into dead weight in the mempool.
2802015-10-28T21:18:16  <morcos> oh sorry, thats not what i mean
2812015-10-28T21:18:17  <morcos> t
2822015-10-28T21:18:35  <morcos> i meant the priority only depends on your inputs that were confirmed at the time you were accpeted
2832015-10-28T21:18:51  <morcos> so its still a bit complicated, but way less than currently
2842015-10-28T21:19:12  <phantomcircuit> a better question is, why would miners use priority?
2852015-10-28T21:19:13  <gmaxwell> I think that would be fine.  Then it only needs to update by some ratio of the size, I guess.
2862015-10-28T21:19:16  <Luke-Jr> that may work good enough as a temporary thing
2872015-10-28T21:19:53  <Luke-Jr> phantomcircuit: already had that minidiscussion scroll up
2882015-10-28T21:21:39  <morcos> gmaxwell: but yeah, i mean even easier, lets just make it not age once its in your mempool.  if you guessed wrong and it doesn't get confirmed soonish, then you resubmit after it expires (currently 72 hours)
2892015-10-28T21:22:47  <Luke-Jr> morcos: maybe have a post-block resort for new confirmed inputs optionally enabled in conf file
2902015-10-28T21:22:49  <Luke-Jr> just in case it turns out bad
2912015-10-28T21:22:53  <Luke-Jr> re solrt I mean
2922015-10-28T21:23:15  <Luke-Jr> re sort sorry
2932015-10-28T21:23:34  <gmaxwell> morcos: thats a little obnoxious, in that if it doesn't make it in the first block then immediately anything new added has an advantage. Maybe it okay?
2942015-10-28T21:25:03  <morcos> gmaxwell: yeah i think its a tradeoff.  the annoying thing however is i'm not sure you can combine it into one score very easily and still have it serve the purpose you want it to serve
2952015-10-28T21:25:59  <Luke-Jr> afk
2962015-10-28T21:26:07  <gmaxwell> I supposed if we cared more we could have a background task that just goes and recosts them from time to time.,
2972015-10-28T21:26:50  <gmaxwell> presumably we're doing some kind of linear scan for the expiration? (I haven't kept up with the latest changes)
2982015-10-28T21:33:50  <morcos> gmaxwell: expiration?  oh no, there is an entry time index as well.
2992015-10-28T21:45:16  *** evoskuil has joined #bitcoin-core-dev
3002015-10-28T21:48:20  *** zooko` has joined #bitcoin-core-dev
3012015-10-28T21:49:25  *** zooko has quit IRC
3022015-10-28T21:51:42  *** zooko` is now known as zooko
3032015-10-28T21:55:00  <phantomcircuit> gmaxwell, im seeing ~200ms on average for Connect total on my server for the last two months
3042015-10-28T22:00:06  <gmaxwell> morcos: so... perhaps another way to handle priority is to maintain a seperate very small mempool for it.
3052015-10-28T22:00:35  <gmaxwell> so then the cost of having to update and resort the 'whole mempool' is not very lage.
3062015-10-28T22:00:38  <gmaxwell> er large*
3072015-10-28T22:04:35  *** ParadoxSpiral has quit IRC
3082015-10-28T22:08:19  *** zooko has quit IRC
3092015-10-28T22:12:14  *** belcher has quit IRC
3102015-10-28T22:14:18  *** d_t has joined #bitcoin-core-dev
3112015-10-28T22:16:23  <phantomcircuit> gmaxwell, why not just mark transaction sin the mempool as dirty when there's a new block and update the priority in a background thread?
3122015-10-28T22:17:42  *** belcher has joined #bitcoin-core-dev
3132015-10-28T22:40:42  <sipa> I'm probably wasting my time, but I'm writing a more (space) efficient std::vector drop-in replacement
3142015-10-28T22:41:02  <sipa> a vector has an average overhead of 40 bytes currently...
3152015-10-28T22:42:08  <gmaxwell> sipa: probably; ... maybe a more useful thing to do is to find things using vectors for small amounts of data and make them not use vectors. :P
3162015-10-28T22:42:55  <gmaxwell> sipa: question;  I can't figure out why we would need to cache failing signatures at all:  rationale: constructing a new novel failing signature is free.
3172015-10-28T22:42:56  <sipa> gmaxwell: the idea is to use a union to either store elements directly, or store the size + pointer of a dynamic array
3182015-10-28T22:43:17  <sipa> gmaxwell: we cache failing transactions, not signatures
3192015-10-28T22:43:32  <sipa> to prevent downloading and verifying them over and over again
3202015-10-28T22:44:32  <gmaxwell> that was just a brainfart, for some reason I thought the sigcache type included a validity flag.
3212015-10-28T22:44:43  <gmaxwell> of course it doesn't.
3222015-10-28T22:45:41  <gmaxwell> sipa: I've wondered before why the normal std::vector didn't special case small numbers of objects by making one of the pointers null and storing the objects internally.
3232015-10-28T22:47:28  <gmaxwell> sipa: even if you superoptimize the vector, there is still malloc overhead and fragmentation and the pointer to the vector object in its parent object which can all be eliminated in cases where the vector can be avoided.
3242015-10-28T22:48:33  <sipa> gmaxwell: well for example almost every CScript (which is a std::vector<unsigned char>) in the utxo cache and mempool stores at least 25 bytes
3252015-10-28T22:49:14  <gmaxwell> Every vector<char> is probably suspect to begin with. :P
3262015-10-28T22:49:23  <sipa> i can make a vector that is exactly 32 bytes (4 bytes size + 28 bytes data, or 4 bytes size + 4 bytes alloced size + pointer to actual data)
3272015-10-28T22:50:06  <gmaxwell> It might be interesting to instrument std::vector so we can get a report of vectors in the codebase that have a constant number of objects in them.
3282015-10-28T22:50:23  <sipa> so it's 32B for up to 28 bytes of data, or 32B + N for N > 28 bytes of data
3292015-10-28T22:51:17  <sipa> instead of vector's 40B + N for everything
3302015-10-28T22:52:50  <gmaxwell> sipa: or you could make the parent object able to store up to 32 bytes internally, and only create a vector if there is more than that. (I suppose it could use a union with the char[32] to store the pointer to the vector when there is one).
3312015-10-28T22:53:00  <sipa> gmaxwell: that's what i'm doing
3322015-10-28T22:53:50  <sipa> when i say "vector" i mean parent type; i think you're interpreting that word as "malloced data"
3332015-10-28T22:54:05  <gmaxwell> Got it.
3342015-10-28T22:55:00  <gmaxwell> why 28? why not 31?  you don't need a 4 byte length for only 31 bytes of range.
3352015-10-28T22:56:13  <sipa> it complicates things a bit if you can't share the size for one with the other
3362015-10-28T22:56:35  <sipa> but yes, indeed
3372015-10-28T22:58:30  <gmaxwell> ah I see, actually that also lets you avoid a flag.. since you can switch the behavior on size.
3382015-10-28T22:59:23  <sipa> i thought about that first, but that's unfortunately not the case
3392015-10-28T22:59:39  <sipa> as vector guarantees that a resize that shrinks doesn't invalidate pointers
3402015-10-28T23:00:01  <sipa> so dropping from over the threshold to below the threshold cannot switch to in-object storage
3412015-10-28T23:01:15  <sipa> so i use 1 bit of the 4-byte size to determine what type of storage to use
3422015-10-28T23:02:08  <gmaxwell> what I was thinking about when I said changing the datastructure was just flattening. e.g. one dynamically sized txout object that contains everything in it directly, and no pointers at all. It's not like we ever modify any of this except setting flags.
3432015-10-28T23:02:26  <gmaxwell> so having any pointers in it under any condition, is just waste and overhead.
3442015-10-28T23:02:56  <gmaxwell> but it means that there needs to be smart accessors for it.
3452015-10-28T23:49:46  <GitHub107> [bitcoin] mcelrath opened pull request #6899: Warnings clean with C++11 (master...cpp11) https://github.com/bitcoin/bitcoin/pull/6899