12016-07-29T00:04:16  *** nets1n has joined #bitcoin-core-dev
  22016-07-29T00:04:38  *** nets1n has joined #bitcoin-core-dev
  32016-07-29T00:17:32  <cfields> NicolasDorier: hmm, I hadn't seen your hash cache stuff yet
  42016-07-29T00:18:01  <cfields> I had been working on something very similar for the purpose of making the sigcache lock free
  52016-07-29T00:18:09  <cfields> (after seeing morcos's numbers and discussing with jeremyrubin)
  62016-07-29T00:19:07  <NicolasDorier> cfields: why is it so important to be lock free ?
  72016-07-29T00:19:10  <cfields> I was going to wait to discuss with Jeremy since it was his idea, but I suppose I should push it up now for discussion. It'd be a shame if we collided with eachother and missed out on the possible speedup
  82016-07-29T00:19:44  <cfields> NicolasDorier: it's not, but morcos reported very significant lock contention with 16 cores. and it's not actually necessary to lock it
  92016-07-29T00:20:34  <NicolasDorier> for the cached hash make, the lock is tacken very briefly for a lookup in a map
 102016-07-29T00:20:40  <cfields> NicolasDorier: sec, I'll push up my (not-fully-vetted) work
 112016-07-29T00:20:41  <NicolasDorier> cached hash map
 122016-07-29T00:20:47  <NicolasDorier> ok
 132016-07-29T00:25:49  <cfields> NicolasDorier: https://github.com/theuni/bitcoin/tree/sigcache-speedup
 142016-07-29T00:26:13  <cfields> NicolasDorier: much of it is quite similar to your work :)
 152016-07-29T00:27:38  <cfields> NicolasDorier: if i can convince morcos to bench that and there's no real improvement, I'll drop it. Otherwise, we might want to merge approaches a bit
 162016-07-29T00:28:16  <sipa> alternatively, for just this, we can just comoute the 3 cached hashes before doing any validation
 172016-07-29T00:28:21  <NicolasDorier> oh cool I'll take a look, in my PR I have a benchmark code if you need
 182016-07-29T00:28:24  <sipa> and then not have any locking at all
 192016-07-29T00:28:26  <NicolasDorier> we can compare easily
 202016-07-29T00:29:06  *** wumpus has quit IRC
 212016-07-29T00:30:55  <cfields> that also has the advantage of making the sigcache usable by libbitcoinconsensus in a way that's thread-friendly, but I'm not sure how necessary/desirable that is
 222016-07-29T00:31:41  <NicolasDorier> cfields: are we talkign about the same thing ? I was talking about midhash caches of segwit, not sig cache in fact
 232016-07-29T00:31:48  <cfields> NicolasDorier: that's great. Sorry for not poking at it further before chiming in, making dinner atm. I'll review yours more carefully tomorrow
 242016-07-29T00:31:50  <NicolasDorier> looking at your PR you are on the sig cache
 252016-07-29T00:32:35  <cfields> NicolasDorier: yes, we're talking about different things. But I think we may want to approach them the same way.
 262016-07-29T00:33:02  <NicolasDorier> ah yes indeed I'll check that.
 272016-07-29T00:34:26  <cfields> like I said, I just took a quick glimpse and wanted to push it up for discussion before it was too late. I'll have a good look at the hash cache tomorrow :)
 282016-07-29T00:35:10  <cfields> bbl
 292016-07-29T00:35:49  *** Ylbam has quit IRC
 302016-07-29T00:36:25  <NicolasDorier> cool thanks !
 312016-07-29T00:39:30  *** wumpus has joined #bitcoin-core-dev
 322016-07-29T00:44:14  *** nets1n has quit IRC
 332016-07-29T00:45:12  *** Giszmo has quit IRC
 342016-07-29T00:47:06  <morcos> NicolasDorier: cfields: jeremyrubin: I'm happy to bench anything.
 352016-07-29T00:47:59  <morcos> What I've been working with so far is just a very small change to sigcache that does the erases at the end of block validation (since thats the only thing we need the lock for) and i'm using a boost lock free queue for that
 362016-07-29T00:48:15  <morcos> of course it would be easy'ish to make a custom version
 372016-07-29T00:49:09  <morcos> jeremy has been hard at work at removing the locking from checkqueue.  i know at least that the time to finish connecting all the transactions (and populating the checkqueue) is much slower due to lock contention..  so its a guess that possibly there is a lot of improvement in the actual validation too
 382016-07-29T00:49:24  <morcos> cfields, if your code is ready i'm happy to try that out tomorrow
 392016-07-29T00:50:51  <morcos> i've combined this with an approach to keep an always hot CCoinsViewCache on top of pcoinstip instead of creating a new view in ConnectBlock.  This both saves a lot of copying if things are already in pcoinstip and serves to keep things loaded from disk.
 402016-07-29T00:51:39  <morcos> I do this by running CreateNewBlock once every 30 seconds.  and then pulling into that hot cache the inputs that would be needed to validate that block
 412016-07-29T00:53:34  <morcos> I think this makes it unnecessary to ever have a dbcache bigger than about 300M (although you need up to about 150M additional for the hot cache, but some of that memory you would have been using anyway in block validation)
 422016-07-29T00:55:10  <CodeShark> there seems to be a phishing attack going on where you get an email that seems to come from xapo and you can get 10 bitcoins immediately if you download and install a wallet
 432016-07-29T00:55:21  <CodeShark> where by wallet, I mean malware
 442016-07-29T00:55:40  <morcos> cfields: here is what i have been using: https://github.com/morcos/bitcoin/commit/3e8a10aa78a8a84a44eca9350178ecf7c308c21c
 452016-07-29T00:57:56  <NicolasDorier> morcos: is there some kind of free lock map so I can get rid of lock in my hash cache map ?
 462016-07-29T00:58:22  <NicolasDorier> ah also, those cache objects might find their way in libconsensus, we'll be able to reference boost ?
 472016-07-29T00:58:48  <morcos> NicolasDorier: i don't think boost is the right end decision, and i think lock free coding is beyond the level i feel very comfortable doing myself
 482016-07-29T00:58:59  <morcos> so thats why i'm offering to test other peoples implementations
 492016-07-29T00:59:11  <morcos> but my goal here is to optimize current bitcoin core performance
 502016-07-29T00:59:20  <morcos> how we make that work with libconsensus is a bit of a separate issue
 512016-07-29T01:00:12  <morcos> without seeing a design document or us all agreeing on a roadmap for libconsensus, i don't feel able to help with that.  i've been saying that for a long time now.  i think its a worthy goal, but approaching piecemeal like this is frustrating.
 522016-07-29T01:04:07  *** Chris_Stewart_5 has quit IRC
 532016-07-29T01:04:19  <cfields> morcos: mine's a little different....
 542016-07-29T01:05:05  <cfields> morcos: jeremy made the observation that we only ever work with the sigcache from one thread at a time. ignoring the fact that _that_ thread spawns threads that trample on eachother
 552016-07-29T01:05:48  <cfields> so the idea is for each of the workers to return what they would have cached, then batch cache/uncache after each block
 562016-07-29T01:06:35  <morcos> cfields: thats what that commit i showed you does
 572016-07-29T01:06:37  <cfields> rather, the workers return a list of cache hits/misses. the caller can then add the misses and optionally purge the hits (as in the case of connecting a real block)
 582016-07-29T01:06:45  <morcos> you dont' actually cache anything in validation
 592016-07-29T01:06:48  <morcos> its only uncaching
 602016-07-29T01:07:00  <morcos> you dont' add misses in validation
 612016-07-29T01:07:08  <morcos> only in ATMP, which isn't multithreaded
 622016-07-29T01:07:19  <morcos> although good to make that work so we can make ATMP multi threaded
 632016-07-29T01:07:28  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 642016-07-29T01:07:30  <morcos> anyway, is that branch you have something thats ready to be tested
 652016-07-29T01:07:34  <morcos> thats easy enough for me to do
 662016-07-29T01:08:47  <morcos> got to go, will be online tomorrow
 672016-07-29T01:09:09  <cfields> morcos: hmm, ok, I'll reread when I'm not preoccupied. I just saw it setting from get()
 682016-07-29T01:09:37  <cfields> morcos: that'd be great, thanks. I'll read yours more carefully along with NicolasDorier's
 692016-07-29T01:10:30  <NicolasDorier> cfields: wait tomorrow for reviewing my PR, I'm changing some stuff I got from sipa's feedback
 702016-07-29T01:15:30  <cfields> NicolasDorier: roger
 712016-07-29T01:16:08  <cfields> morcos: ah, heh, the diff threw me off
 722016-07-29T01:27:27  *** Alopex has quit IRC
 732016-07-29T01:28:32  *** Alopex has joined #bitcoin-core-dev
 742016-07-29T01:42:18  *** TomMc has quit IRC
 752016-07-29T01:56:20  *** nets1n has joined #bitcoin-core-dev
 762016-07-29T01:57:41  <GitHub165> [bitcoin] NicolasDorier opened pull request #8422: Cache hashes (master...cachedhashes2) https://github.com/bitcoin/bitcoin/pull/8422
 772016-07-29T01:58:36  <GitHub146> [bitcoin] NicolasDorier closed pull request #8259: Hash Cache (master...cachedhashes) https://github.com/bitcoin/bitcoin/pull/8259
 782016-07-29T01:58:47  <NicolasDorier> cfields: when you feel like reviewing cached hashes : https://github.com/bitcoin/bitcoin/pull/8422
 792016-07-29T02:00:56  *** a87ry5 has joined #bitcoin-core-dev
 802016-07-29T02:01:04  *** nets1n has quit IRC
 812016-07-29T02:02:46  *** fengling has quit IRC
 822016-07-29T02:08:12  *** nets1n has joined #bitcoin-core-dev
 832016-07-29T02:09:01  *** Alopex has quit IRC
 842016-07-29T02:10:06  *** Alopex has joined #bitcoin-core-dev
 852016-07-29T02:12:27  *** a87ry5 has quit IRC
 862016-07-29T02:13:30  *** fengling has joined #bitcoin-core-dev
 872016-07-29T02:26:01  *** pmienk has quit IRC
 882016-07-29T02:29:14  *** nets1n has quit IRC
 892016-07-29T02:34:33  *** Chris_Stewart_5 has quit IRC
 902016-07-29T02:38:06  *** pmienk has joined #bitcoin-core-dev
 912016-07-29T02:42:01  *** Alopex has quit IRC
 922016-07-29T02:43:06  *** Alopex has joined #bitcoin-core-dev
 932016-07-29T02:52:04  *** supasonic has quit IRC
 942016-07-29T03:00:38  <GitHub111> [bitcoin] fanquake opened pull request #8423: [depends] expat 2.2.0, ccache 3.2.7 (master...expat-ccache-jul) https://github.com/bitcoin/bitcoin/pull/8423
 952016-07-29T03:01:15  *** fanquake has joined #bitcoin-core-dev
 962016-07-29T03:01:40  <fanquake> morning
 972016-07-29T03:18:01  *** Alopex has quit IRC
 982016-07-29T03:19:07  *** Alopex has joined #bitcoin-core-dev
 992016-07-29T03:21:29  *** molz has joined #bitcoin-core-dev
1002016-07-29T03:23:14  *** moli has quit IRC
1012016-07-29T03:24:02  *** supasonic has joined #bitcoin-core-dev
1022016-07-29T03:38:01  *** Alopex has quit IRC
1032016-07-29T03:38:42  *** fanquake has left #bitcoin-core-dev
1042016-07-29T03:39:06  *** Alopex has joined #bitcoin-core-dev
1052016-07-29T03:44:41  *** goatpig has quit IRC
1062016-07-29T03:49:06  *** Alopex has quit IRC
1072016-07-29T03:50:12  *** Alopex has joined #bitcoin-core-dev
1082016-07-29T03:59:59  *** d_t has joined #bitcoin-core-dev
1092016-07-29T04:00:36  *** d_t has joined #bitcoin-core-dev
1102016-07-29T04:02:57  *** supasonic has quit IRC
1112016-07-29T04:07:07  *** Alopex has quit IRC
1122016-07-29T04:08:12  *** Alopex has joined #bitcoin-core-dev
1132016-07-29T04:23:21  *** Alopex has quit IRC
1142016-07-29T04:24:26  *** Alopex has joined #bitcoin-core-dev
1152016-07-29T04:24:31  *** anu0 has joined #bitcoin-core-dev
1162016-07-29T04:28:05  *** anu1 has quit IRC
1172016-07-29T04:41:43  *** justanotheruser has quit IRC
1182016-07-29T04:42:11  *** justanotheruser has joined #bitcoin-core-dev
1192016-07-29T04:43:06  *** fengling has quit IRC
1202016-07-29T04:47:02  *** Alopex has quit IRC
1212016-07-29T04:48:07  *** Alopex has joined #bitcoin-core-dev
1222016-07-29T04:55:53  *** d_t has quit IRC
1232016-07-29T05:09:46  *** fengling has joined #bitcoin-core-dev
1242016-07-29T05:13:07  <luke-jr> jtimon: the issue is entirely giving bad advice to miners. regardless of what they might or might not configure in the end, we should not give bad advice
1252016-07-29T05:13:37  <luke-jr> (btw, I see no reason the performance "hit" of using maxsize would be greater than the performance hit of using 0.13 over 0.12..)
1262016-07-29T05:14:26  *** fengling has quit IRC
1272016-07-29T05:22:24  *** kadoban has quit IRC
1282016-07-29T05:25:21  *** Alopex has quit IRC
1292016-07-29T05:26:26  *** Alopex has joined #bitcoin-core-dev
1302016-07-29T05:40:06  *** fengling has joined #bitcoin-core-dev
1312016-07-29T05:51:40  *** LeMiner2 has quit IRC
1322016-07-29T05:52:20  *** LeMiner2 has joined #bitcoin-core-dev
1332016-07-29T06:06:27  <GitHub151> [bitcoin] PrinceofOrange opened pull request #8424: merge (0.8...master) https://github.com/bitcoin/bitcoin/pull/8424
1342016-07-29T06:08:33  <GitHub87> [bitcoin] laanwj closed pull request #8424: merge (0.8...master) https://github.com/bitcoin/bitcoin/pull/8424
1352016-07-29T06:09:59  <luke-jr> O.o
1362016-07-29T06:12:53  <wumpus> yes that's the only possible response - this seems deliberate spammyness, not merely an accident
1372016-07-29T06:21:37  <GitHub182> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/ad087638ee48...842bf8d2c5af
1382016-07-29T06:21:38  <GitHub182> bitcoin/master 1de2a46 Suhas Daftuar: Ignore GETBLOCKTXN requests for unknown blocks...
1392016-07-29T06:21:38  <GitHub182> bitcoin/master 1d06e49 Suhas Daftuar: Ignore CMPCTBLOCK messages for pruned blocks...
1402016-07-29T06:21:39  <GitHub182> bitcoin/master 842bf8d Wladimir J. van der Laan: Merge #8408: Prevent fingerprinting, disk-DoS with compact blocks...
1412016-07-29T06:21:47  <GitHub16> [bitcoin] laanwj closed pull request #8408: Prevent fingerprinting, disk-DoS with compact blocks (master...cb-misbehaving) https://github.com/bitcoin/bitcoin/pull/8408
1422016-07-29T06:28:39  <GitHub2> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/b7e201181bcc0f6328e0a499803f1dbb2c2dbd28
1432016-07-29T06:28:39  <GitHub2> bitcoin/0.13 b7e2011 Suhas Daftuar: Prevent fingerprinting, disk-DoS with compact blocks...
1442016-07-29T06:39:07  *** Alopex has quit IRC
1452016-07-29T06:40:12  *** Alopex has joined #bitcoin-core-dev
1462016-07-29T06:41:22  *** molly has joined #bitcoin-core-dev
1472016-07-29T06:44:10  *** molz has quit IRC
1482016-07-29T07:02:29  *** BashCo has quit IRC
1492016-07-29T07:23:08  *** BashCo has joined #bitcoin-core-dev
1502016-07-29T07:24:33  *** cdecker has joined #bitcoin-core-dev
1512016-07-29T07:39:00  *** cdecker has quit IRC
1522016-07-29T07:39:06  *** Alopex has quit IRC
1532016-07-29T07:40:12  *** Alopex has joined #bitcoin-core-dev
1542016-07-29T07:40:24  *** cdecker has joined #bitcoin-core-dev
1552016-07-29T07:40:41  *** cdecker has quit IRC
1562016-07-29T07:44:40  *** molly has quit IRC
1572016-07-29T07:50:47  <GitHub65> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/b06808c58eb7a997c42b55cba63688aec448a0ea
1582016-07-29T07:50:47  <GitHub65> bitcoin/0.13 b06808c Wladimir J. van der Laan: doc: Release notes update for rc2
1592016-07-29T07:56:48  <GitHub72> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/ced6c940da35fcf33160d1c7f2f54a99dc7eedb1
1602016-07-29T07:56:48  <GitHub72> bitcoin/0.13 ced6c94 Wladimir J. van der Laan: qt: Translations update pre-rc2
1612016-07-29T07:59:32  *** Guyver2 has joined #bitcoin-core-dev
1622016-07-29T08:24:46  *** Giszmo has joined #bitcoin-core-dev
1632016-07-29T08:35:36  <wumpus>  * [new tag]         v0.13.0rc2 -> v0.13.0rc2
1642016-07-29T08:54:13  *** Ylbam has joined #bitcoin-core-dev
1652016-07-29T09:07:06  *** fanquake has joined #bitcoin-core-dev
1662016-07-29T09:17:43  *** Guyver2_ has joined #bitcoin-core-dev
1672016-07-29T09:18:04  *** Guyver2 has quit IRC
1682016-07-29T09:18:37  *** Guyver2_ is now known as Guyver2
1692016-07-29T09:20:54  *** Guyver2 has quit IRC
1702016-07-29T09:23:19  *** arubi has quit IRC
1712016-07-29T09:24:01  *** arubi has joined #bitcoin-core-dev
1722016-07-29T09:29:25  <GitHub55> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/842bf8d2c5af...b77bb95b3cb4
1732016-07-29T09:29:26  <GitHub55> bitcoin/master 755aa05 Cory Fields: httpserver: use a future rather than relying on boost's try_join_for
1742016-07-29T09:29:26  <GitHub55> bitcoin/master d3773ca Cory Fields: httpserver: explicitly detach worker threads...
1752016-07-29T09:29:27  <GitHub55> bitcoin/master 7e87033 Cory Fields: httpserver: replace boost threads with std...
1762016-07-29T09:29:37  <GitHub56> [bitcoin] laanwj closed pull request #8421: httpserver: drop boost (#8023 dependency) (master...http-thread) https://github.com/bitcoin/bitcoin/pull/8421
1772016-07-29T09:49:16  *** arubi_ has joined #bitcoin-core-dev
1782016-07-29T09:53:08  *** arubi has quit IRC
1792016-07-29T10:03:39  *** laurentmt has joined #bitcoin-core-dev
1802016-07-29T10:04:25  *** laurentmt has quit IRC
1812016-07-29T10:11:52  *** anu0 has quit IRC
1822016-07-29T10:12:29  *** mkarrer has quit IRC
1832016-07-29T10:31:03  *** anu0 has joined #bitcoin-core-dev
1842016-07-29T10:33:10  <GitHub12> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/b77bb95b3cb4...7a2d40272717
1852016-07-29T10:33:11  <GitHub12> bitcoin/master 695041e Wladimir J. van der Laan: util: Update tinyformat...
1862016-07-29T10:33:12  <GitHub12> bitcoin/master a5072a7 Wladimir J. van der Laan: util: Remove zero-argument versions of LogPrint and error...
1872016-07-29T10:33:12  <GitHub12> bitcoin/master 7a2d402 Wladimir J. van der Laan: Merge #8274: util: Update tinyformat...
1882016-07-29T10:33:16  <GitHub181> [bitcoin] laanwj closed pull request #8274: util: Update tinyformat (master...2016_06_update_tinyformat) https://github.com/bitcoin/bitcoin/pull/8274
1892016-07-29T10:35:12  *** owowo has quit IRC
1902016-07-29T10:42:58  *** molly has joined #bitcoin-core-dev
1912016-07-29T10:51:06  *** fengling has quit IRC
1922016-07-29T10:52:46  <fanquake> Looks like sigs match @wumpus
1932016-07-29T10:53:38  <fanquake> Have you noticed quite a lot of output during the Windows build now?
1942016-07-29T10:53:46  <fanquake> Along the lines of warning: conflicts with previous declaration ‘void boost::signals2
1952016-07-29T11:03:50  <wumpus> good that sigs match!
1962016-07-29T11:04:07  <wumpus> no, haven't noticed
1972016-07-29T11:04:35  <fanquake> I'll get the logs and open an issue.
1982016-07-29T11:04:40  <wumpus> thanks
1992016-07-29T11:13:19  *** owowo has joined #bitcoin-core-dev
2002016-07-29T11:23:28  *** BashCo_ has joined #bitcoin-core-dev
2012016-07-29T11:26:01  *** fanquake has quit IRC
2022016-07-29T11:26:26  *** BashCo has quit IRC
2032016-07-29T11:53:32  *** slackircbridge has quit IRC
2042016-07-29T11:54:07  *** slackircbridge has joined #bitcoin-core-dev
2052016-07-29T11:55:18  *** mkarrer has joined #bitcoin-core-dev
2062016-07-29T12:14:19  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2072016-07-29T12:24:51  *** Chris_Stewart_5 has quit IRC
2082016-07-29T12:41:04  <morcos> luke-jr: i was just discussing with sdaftuar yesterday, i'm not sure there is much of a perf hit, it just hasn't been benched yet.  i may try to do that today.  why does 0.13 have worse performance than 0.12, i don't think thats true.
2092016-07-29T12:50:56  <michagogo> The "native_biplist" has a confusing name
2102016-07-29T12:51:14  <morcos> luke-jr: ugh, sorry. i keep getting confused.  i guess there are two ways it could get slower, and my anecdotal evidence was only that the extra serialization wasn't that bad.  but the repeated failing to find a final tx is blockmaxsize is your constraining factor seems like it could be slow, especially on a big mempool
2112016-07-29T12:51:26  <michagogo> My first thought was, wtf? Why is a list of BIPs its own package?
2122016-07-29T12:51:52  <michagogo> Then I clicked it and saw it was downloading from PyPI, and that was even more confusing... until I looked up the package description
2132016-07-29T12:52:48  <morcos> anyway, too much annoyance to go test all these posibilities..  i think its a clearly correct recommendation that until segwit activates using only blockmaxweight is the desired configuration.  miners will be upgrading anyway for segwit, so there is a chance to argue something different for that
2142016-07-29T12:52:56  <morcos> lets concentrate on getting 0.13 right.
2152016-07-29T12:56:52  *** TomMc has joined #bitcoin-core-dev
2162016-07-29T12:58:39  *** TomMc has quit IRC
2172016-07-29T13:00:33  *** cryptapus_afk is now known as cryptapus
2182016-07-29T13:04:49  <sdaftuar> cfields: i'm looking at your crash from yesterday. seems odd! can you give some more color on what the environment was: mainnet/testnet/regtest? fresh node or already synced?  what commit were you running?
2192016-07-29T13:08:34  *** [b__b] has quit IRC
2202016-07-29T13:08:59  *** [b__b] has joined #bitcoin-core-dev
2212016-07-29T13:44:23  <instagibbs> is it expected behavior for a reindex to not show blocks being processed for while a while?
2222016-07-29T13:44:38  <instagibbs> in getinfo at least
2232016-07-29T13:44:54  <instagibbs> I seem to recall different behavior
2242016-07-29T13:57:06  *** TomMc has joined #bitcoin-core-dev
2252016-07-29T13:59:12  *** instagibbs has quit IRC
2262016-07-29T14:14:05  <cfields> sdaftuar: uhmm, from my http-thread branch, which was merged today
2272016-07-29T14:14:19  <cfields> sdaftuar: so current master should be basically the same thing
2282016-07-29T14:15:26  <cfields> checking my backlog to see how i was running
2292016-07-29T14:17:08  <cfields> sdaftuar: ah, a funky config, forgot I was running that way. probably significant...
2302016-07-29T14:17:15  <cfields> gdb --args ./bitcoind -testnet -printtoconsole -reindex-chainstate -debug=http
2312016-07-29T14:17:51  <cfields> the -reindex-chainstate was accidental, left in there from testing something before.
2322016-07-29T14:24:21  <morcos> cfields: i have some results.  here are the average times per block for  Connect transactions/Verify txins/Connect block   (meausres are total time so far as reported in bench)
2332016-07-29T14:24:31  <morcos> in ms
2342016-07-29T14:24:46  <morcos> master: 26 / 80 / 111
2352016-07-29T14:25:11  <morcos> your sigcache: 39 / 46 / 82
2362016-07-29T14:25:45  <morcos> batched deletes using boost lock free (commit i showed you): 22 / 32 / 63
2372016-07-29T14:26:53  <cfields> morcos: hmm. that's surprising. that's 16 cores?
2382016-07-29T14:27:11  <morcos> yeah, 16 cores with 8GB dbcache
2392016-07-29T14:27:40  <morcos> for yours and the batched deletes test i also included your copy-move improvements and my hot tipcache, but those don't make much difference on master
2402016-07-29T14:28:04  <cfields> ok
2412016-07-29T14:28:30  <TomMc> In regards to signature hashtypes, if the hashtype is SIGHASH_ALL, is the last byte given a value of 0x01?
2422016-07-29T14:28:31  <cfields> i'm still working on the prevector btw, i've almost got that rewritten and specialized for the unsigned char case
2432016-07-29T14:29:00  <morcos> i'll dig into it more, so see if i can understand why your code slowed down Connect transactions
2442016-07-29T14:29:00  <cfields> i wonder what throws the connect time off...
2452016-07-29T14:30:01  <cfields> morcos: thanks a bunch for testing. that's very interesting. I'll look into the connect as well.
2462016-07-29T14:30:06  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2472016-07-29T14:31:37  <cfields> morcos: one more interesting test, if you don't mind
2482016-07-29T14:31:59  <morcos> cfields: the connect time can be reduced to as low as 10ms if you eliminate the locking contention in checkqueue
2492016-07-29T14:32:14  <cfields> morcos: run with -par=1 as a baseline. I was surprised to see that it's actually faster on some of my machines that way
2502016-07-29T14:33:06  <morcos> :) i know!  how i found this stuff in the first place was running with blocks only mode was actually faster than running with tx, implying using the sigcache was slower than verifying the sigs!
2512016-07-29T14:33:16  <cfields> morcos: right. I'm looking forward to pairing those two, I should think they would improve eachother
2522016-07-29T14:33:33  <cfields> heh, right
2532016-07-29T14:33:52  <morcos> will try par=1 as well
2542016-07-29T14:34:29  <cfields> morcos: wait. current master actually _is_ faster with -par=1 on 16 cores?!
2552016-07-29T14:34:56  <morcos> oh, no i haven't tried that, i was just saying i know in that its not surprising to me that it could be in some configs
2562016-07-29T14:35:22  <cfields> oh, whew :)
2572016-07-29T14:35:38  <cfields> looks to me like it breaks even around ~3 as-is
2582016-07-29T14:36:31  <cfields> but I've tested so many configs now that I can't keep the numbers straight, so that could be wrong
2592016-07-29T14:41:16  <cfields> morcos: ah, looks like it's the overhead of creating a copy of the to-test CScriptCheck that slows down the connect. that could be much improved
2602016-07-29T14:41:44  <morcos> oh, that was my guess at first but then i didn't see where that happened b/c i thought it was just shared pointers
2612016-07-29T14:42:15  <sipa> cfields: before libsecp256k1-based verification, i benchmarked where the break even point was on a large-number-of-cores machine, and set the max value for -par based on that
2622016-07-29T14:42:25  <morcos> the method of batching deletes is extremely effective
2632016-07-29T14:42:45  <cfields> morcos: yea, those should be cheap, but I'm assuming it's the constant inserts() that suck
2642016-07-29T14:42:46  <sipa> when we switched to libsecp validation, the break even point probably dropped a lot
2652016-07-29T14:43:02  <sipa> cfields: copy? it should be using swap
2662016-07-29T14:43:03  <cfields> sipa: ah yes, that makes sense then.
2672016-07-29T14:43:13  <morcos> sipa: depends on whether you are taking advantage of the sigcache a lot or not though
2682016-07-29T14:43:14  <cfields> sipa: they're shared_ptrs
2692016-07-29T14:43:51  <sipa> cfields: really? when was that changed?
2702016-07-29T14:44:01  <morcos> :)
2712016-07-29T14:44:03  <morcos> its not in master
2722016-07-29T14:44:06  <cfields> sipa: in the branch we're discussing
2732016-07-29T14:44:40  <cfields> sipa: https://github.com/bitcoin/bitcoin/compare/master...theuni:sigcache-speedup
2742016-07-29T14:44:52  <morcos> back in a few
2752016-07-29T14:45:36  <sipa> ah, ok
2762016-07-29T14:45:45  <cfields> morcos: that whole vector business (including checkqueue) could be much improved with a list i think. Then we could splice all over the place instead.
2772016-07-29T14:45:47  <sipa> i thought you were comparing with master
2782016-07-29T14:46:19  <cfields> sipa: no. discussing different ways to reduce the sigcache lock contention
2792016-07-29T14:46:26  <sipa> ok ok
2802016-07-29T14:46:34  <sipa> looking forward to the results
2812016-07-29T14:48:28  *** d_t has joined #bitcoin-core-dev
2822016-07-29T14:48:33  <michagogo> cfields: are you by any chance in the process of building rc2?
2832016-07-29T14:48:36  <morcos> i can't remember if i mentioned this on channel before, but i think we made a mistake in the design of BIP 68
2842016-07-29T14:48:39  <michagogo> Looks like we have 4 sets of sigs already
2852016-07-29T14:48:47  <morcos> we should have designed that as a per input check
2862016-07-29T14:49:17  <cfields> morcos: another route would be to maintain a single list rather than a vector of per-txin hits/misses. But I was afraid that would slow down one of the checks. I'll whip that up for another data point
2872016-07-29T14:49:22  <morcos> that would have done away with that whole issue of calculating prevheights
2882016-07-29T14:49:32  <cfields> michagogo: yes, builds in progress
2892016-07-29T14:49:47  <morcos> cfields: keep in mind you never need both hits and misses
2902016-07-29T14:49:49  <michagogo> (i.e. are detached sigs imminent? If not, I'll just wait until Saturday night, but if they are I can leave the VM running)
2912016-07-29T14:49:57  <michagogo> Cool, thanks
2922016-07-29T14:49:57  <morcos> i made a couple comments on your commit
2932016-07-29T14:51:22  <morcos> and i actually haven't looked at how this interacts with the segwit style cachehashes that NicolasDorier is working on
2942016-07-29T14:51:28  <cfields> morcos: well you at least need each txin's misses shared with eachother. for the 2dup..checksig..2drop..checksig case, no?
2952016-07-29T14:53:01  <morcos> cfields: hmm..  explain a bit more.  you mean if you are checking the same sig multiple times you dont' want to have to redo the actual verfication?
2962016-07-29T14:53:22  <morcos> we don't solve that problem now do we?
2972016-07-29T14:53:29  <cfields> morcos: right
2982016-07-29T14:53:40  <cfields> morcos: i believe so, as they're all cached on-the-fly
2992016-07-29T14:53:51  <morcos> they aren't cached in ConnectBlock
3002016-07-29T14:53:59  <morcos> they are only cached in ATMP
3012016-07-29T14:54:16  <sipa> morcos: sorry, what is 'they' ?
3022016-07-29T14:54:30  <morcos> signature verification results
3032016-07-29T14:54:40  <morcos> whats in the existing sigcache
3042016-07-29T14:54:52  <sipa> ah, yes
3052016-07-29T14:56:50  <cfields> morcos: hmm, yes, you're right
3062016-07-29T14:58:00  <sipa> we store results in atmp, delete results in connectblock
3072016-07-29T14:58:33  <sipa> if you duplicate sigs i guess that indeed will hurt
3082016-07-29T14:58:42  *** instagibbs has joined #bitcoin-core-dev
3092016-07-29T15:02:28  <morcos> cfields: but you were talking about sharing a txin's misses with other txin checks()'s ?  thats not necessary right?  it's only within a given txin that you'll have a duplicated signature?
3102016-07-29T15:03:17  <morcos> seems like if thats a worthwhile performance improvement thats easy enough to add separately from the sigcache..  kind of a separate question
3112016-07-29T15:04:01  <sipa> cfields, morcos: with nicolasdorier's cache, we could add some sort of accumulator to the cache, which lists all the signatures that were valudated (or even sigcache iterators to them), and then delete them all at once after the whole block is validated
3122016-07-29T15:04:25  <cfields> morcos: right. atm, they're per-txin. But since there's the overhead of creating a vector/list/whatever for each, I was (thinking outloud) that it may end up quicker to do per-block instead
3132016-07-29T15:04:44  <cfields> sipa: that's exactly what this branch is doing :)
3142016-07-29T15:04:55  <cfields> sipa: and morcos's as well
3152016-07-29T15:04:55  <sipa> great
3162016-07-29T15:05:01  * sipa shuts up
3172016-07-29T15:05:31  <morcos> sipa: but right that it should probably be combined with NicolasDorier's cache too, b/c you'll want to batch the deleting of the hashes as well
3182016-07-29T15:05:36  <cfields> sipa: no, it's helpful. The issue is that that means returning a list of hits/misses for each txin, which has a considerable overhead
3192016-07-29T15:06:12  <morcos> cfields: yes, but a lock free list is relatively straight forward right... we don't have to use the boost version
3202016-07-29T15:06:58  <morcos> but can accomplish the same thing ourselves..   so instead of returning the list, we just append to the shared lists of hits and misses
3212016-07-29T15:07:53  *** aalex has joined #bitcoin-core-dev
3222016-07-29T15:08:16  <cfields> morcos: sure, but i _really_ like the idea of moving the cache out so that it can be passed in without worry of threading semantics. that lets others (libbitcoinconsensus) use it as well
3232016-07-29T15:08:41  <cfields> but that's not the strongest argument, so I'd be fine with yours as well
3242016-07-29T15:10:36  <cfields> morcos: passing it in also means that ATMP can be threaded as well (as you mentioned yesterday). I suppose you could do the same with your approach by adding a similar batch-to-add structure
3252016-07-29T15:10:46  <morcos> right
3262016-07-29T15:12:09  <sipa> cfields: one way would be to have a simple preallocated vector in each validation thread
3272016-07-29T15:12:20  <sipa> and afterwards, all results are combined
3282016-07-29T15:13:44  <cfields> sipa: i started that way, but was hesitant to re-intruduce contention in the re-combining. Did you have something in mind to avoid that?
3292016-07-29T15:14:25  <sipa> cfields: i meant recombining only after all txn are validated
3302016-07-29T15:15:41  <morcos> sipa: yep, that was another approach jeremy and i discussed
3312016-07-29T15:15:52  <cfields> sipa: ah, so the threads would set a pointer to their thread_local vector at creation, then master could iterate when finished
3322016-07-29T15:16:07  <morcos> cfields: i'm not sure i understand why your branch is safe to remove all the locks...
3332016-07-29T15:16:14  <cfields> yes, that would be much better
3342016-07-29T15:16:54  <cfields> morcos: a const cache is passed in, it's not changing during validation
3352016-07-29T15:16:59  <morcos> ah, i guess its protected by cs_main?
3362016-07-29T15:17:16  <morcos> yes, but how did you know another thread isn't modifying it while you are reading from it
3372016-07-29T15:17:36  <cfields> morcos: right, it needs to be globally protected. atm nothing else touches it. so i suppose it's cs_main :)
3382016-07-29T15:17:56  <morcos> right a rpc call to submit a tx or something
3392016-07-29T15:19:22  <cfields> yes
3402016-07-29T15:19:54  <cfields> ok, I'll try the per-thread suggestion. should be easy to whip up i think
3412016-07-29T15:32:13  *** I_DID_LSD_ON_A_P has quit IRC
3422016-07-29T15:32:29  *** gribble has quit IRC
3432016-07-29T15:39:58  <GitHub38> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7a2d40272717...bbcb8fd88433
3442016-07-29T15:39:58  <GitHub38> bitcoin/master 54af51d Jonas Schnelli: [QA] Add walletdump RPC test (including HD- & encryption-tests)
3452016-07-29T15:39:59  <GitHub38> bitcoin/master bbcb8fd Wladimir J. van der Laan: Merge #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests)...
3462016-07-29T15:40:10  <GitHub139> [bitcoin] laanwj closed pull request #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests) (master...2016/07/dump_test) https://github.com/bitcoin/bitcoin/pull/8417
3472016-07-29T15:40:10  *** gribble has joined #bitcoin-core-dev
3482016-07-29T15:42:05  <luke-jr> morcos: CPFP requires updating stuff as parents get included in the block, which probably isnt *that* expensive, but a heck of a lot more than merely adding numbers (the overhead of maxsize); AFAIK it doesn't do any extra serialization, just adding sizes up
3492016-07-29T15:42:22  *** zooko has joined #bitcoin-core-dev
3502016-07-29T15:42:53  <morcos> luke-jr: i believe suhas's bench marks for CPFP actually showed it neglibly faster than the pre CPFP code.  this is b/c you can add a whole package of txs at a time i think.
3512016-07-29T15:43:52  <GitHub141> [bitcoin] laanwj opened pull request #8427: net: Ignore `notfound` P2P messages (master...2016_07_notfound) https://github.com/bitcoin/bitcoin/pull/8427
3522016-07-29T15:44:42  <GitHub72> [bitcoin] laanwj closed pull request #8403: Process "notfound" messages, and safeguard against unreasonably long … (master...ProcessNotfound) https://github.com/bitcoin/bitcoin/pull/8403
3532016-07-29T15:47:21  <luke-jr> morcos: benchmarks with what tx sets? I suspect it would vary significantly depending on the input
3542016-07-29T15:47:50  <morcos> historical simulation over a default mempool with default policy i think
3552016-07-29T15:52:38  <luke-jr> what is a default mempool? anyhow, I guess the slower path is unlikely to be used historically
3562016-07-29T15:52:51  <morcos> 300M
3572016-07-29T15:53:07  <luke-jr> probably not too likely to be used more with CPFP either
3582016-07-29T15:53:29  <luke-jr> morcos: oh, but we don't have historical mempool contents
3592016-07-29T15:54:03  *** Chris_Stewart_5 has quit IRC
3602016-07-29T15:57:28  <morcos> luke-jr: i'm not sure what you mean.  sdaftuar wrote code to save p2p messages to disk and be able to replay them back off disk to a node.  we've been using that for years to test and bench different changes.
3612016-07-29T15:58:25  <luke-jr> morcos: oh, I was not aware; so you have years of data like that? :o
3622016-07-29T15:58:40  <morcos> yep
3632016-07-29T16:00:28  *** zooko has quit IRC
3642016-07-29T16:03:10  <morcos> its not a perfect system by any means, but its good for benching and it was very useful for evaluating different fee estimation algorithms
3652016-07-29T16:07:24  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3662016-07-29T16:08:58  <Chris_Stewart_5> Where do I add a new unit test file to have it included in test_bitcoin? I looked in test_bitcoin.cpp and didn't find an obvious spot, unless i'm blind
3672016-07-29T16:09:23  *** zooko has joined #bitcoin-core-dev
3682016-07-29T16:09:46  <sipa> Chris_Stewart_5: just add it to the makefile
3692016-07-29T16:10:00  <sipa> there is no explicit list of all tests
3702016-07-29T16:11:08  <Chris_Stewart_5> Hmm, the README seems a litle misleading then
3712016-07-29T16:14:49  <luke-jr> then clarify it too while you're at it :p
3722016-07-29T16:18:35  <Chris_Stewart_5> but it is so much more fun to complain! :-)
3732016-07-29T16:31:04  *** Chris_Stewart_5 has quit IRC
3742016-07-29T16:34:21  *** Sosumi has quit IRC
3752016-07-29T16:36:14  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3762016-07-29T16:43:05  <cfields> gitian signers: v0.13.0rc2 sigs are pushed
3772016-07-29T16:43:07  <cfields> michagogo: ping ^^
3782016-07-29T16:47:53  <sdaftuar> cfields: i can reproduce your segfault.  kind of an interesting case really, though i believe unrelated to #8096.
3792016-07-29T16:48:29  <sdaftuar> cfields: looks like the issue is that if you ctrl-c on startup, while running with -reindex-chainstate, chainActive will not be initialized
3802016-07-29T16:48:43  <sdaftuar> because ActivateBestChain returns immediately on ShutdownRequested()
3812016-07-29T16:48:52  <sdaftuar> that violates an assumption in RewindBlockIndex
3822016-07-29T16:49:40  *** netzin has joined #bitcoin-core-dev
3832016-07-29T16:53:37  <GitHub108> [bitcoin] Christewart opened pull request #8428: Update README.md (master...imporve_test_readme) https://github.com/bitcoin/bitcoin/pull/8428
3842016-07-29T16:53:47  <Chris_Stewart_5> luke-jr: ^ :-)
3852016-07-29T17:03:11  *** d_t has quit IRC
3862016-07-29T17:14:17  <cfields> sdaftuar: damn. Yea, completely forgot I was running with reindex-chainstate.
3872016-07-29T17:14:27  <cfields> sdaftuar: thanks for reproducing
3882016-07-29T17:24:02  *** YOU-JI has joined #bitcoin-core-dev
3892016-07-29T17:28:39  *** Chris_Stewart_5 has quit IRC
3902016-07-29T17:28:53  *** BashCo_ has quit IRC
3912016-07-29T17:34:13  *** arubi_ is now known as arubi
3922016-07-29T17:44:35  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3932016-07-29T17:52:44  *** jgarzik has joined #bitcoin-core-dev
3942016-07-29T17:59:30  *** adiabat has joined #bitcoin-core-dev
3952016-07-29T18:02:26  *** BashCo has joined #bitcoin-core-dev
3962016-07-29T18:03:34  *** netzin has quit IRC
3972016-07-29T18:11:43  *** netzin has joined #bitcoin-core-dev
3982016-07-29T18:12:31  *** netzin has joined #bitcoin-core-dev
3992016-07-29T18:28:53  *** anu1 has joined #bitcoin-core-dev
4002016-07-29T18:32:07  *** anu0 has quit IRC
4012016-07-29T18:59:12  *** d_t has joined #bitcoin-core-dev
4022016-07-29T19:36:35  *** Guyver2 has joined #bitcoin-core-dev
4032016-07-29T19:42:09  *** jtimon has quit IRC
4042016-07-29T19:54:33  *** TomMc has quit IRC
4052016-07-29T19:58:04  *** TomMc has joined #bitcoin-core-dev
4062016-07-29T20:01:22  *** laurentmt has joined #bitcoin-core-dev
4072016-07-29T20:08:21  *** zooko has quit IRC
4082016-07-29T20:15:29  *** ovovo has joined #bitcoin-core-dev
4092016-07-29T20:16:32  *** owowo has quit IRC
4102016-07-29T20:17:01  *** jannes has quit IRC
4112016-07-29T20:19:36  *** netzin has quit IRC
4122016-07-29T20:34:39  *** belcher has joined #bitcoin-core-dev
4132016-07-29T20:35:49  *** laurentmt has quit IRC
4142016-07-29T20:54:34  *** dasaj has joined #bitcoin-core-dev
4152016-07-29T21:18:23  *** YOU-JI has quit IRC
4162016-07-29T21:37:19  *** arubi has quit IRC
4172016-07-29T21:38:25  *** arubi has joined #bitcoin-core-dev
4182016-07-29T21:43:13  *** Chris_Stewart_5 has quit IRC
4192016-07-29T21:57:10  *** spudowiar has joined #bitcoin-core-dev
4202016-07-29T22:00:29  *** whphhg_ has joined #bitcoin-core-dev
4212016-07-29T22:01:33  *** whphhg has quit IRC
4222016-07-29T22:02:17  *** whphhg_ is now known as whphhg
4232016-07-29T22:39:11  *** cryptapus is now known as cryptapus_afk
4242016-07-29T23:11:01  *** Alopex has quit IRC
4252016-07-29T23:12:06  *** Alopex has joined #bitcoin-core-dev
4262016-07-29T23:20:36  *** d_t has quit IRC
4272016-07-29T23:23:35  *** Guyver2 has quit IRC
4282016-07-29T23:25:43  *** TomMc has quit IRC
4292016-07-29T23:43:33  *** netzin has joined #bitcoin-core-dev