12016-07-13T00:06:48  *** Cory has joined #bitcoin-core-dev
  22016-07-13T00:15:45  *** belcher has quit IRC
  32016-07-13T00:23:28  *** Chris_Stewart_5 has quit IRC
  42016-07-13T00:30:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  52016-07-13T00:36:47  *** Cory has quit IRC
  62016-07-13T00:39:49  *** fengling has joined #bitcoin-core-dev
  72016-07-13T00:40:54  *** luke-jr has quit IRC
  82016-07-13T00:44:33  *** Chris_Stewart_5 has quit IRC
  92016-07-13T00:53:01  <phantomcircuit> sipa, i want to (optinally) save the mempool and sigcache
 102016-07-13T00:53:04  <phantomcircuit> on shutdown
 112016-07-13T00:53:08  <phantomcircuit> where's the right place to put that?
 122016-07-13T00:54:19  <gmaxwell> don't save the sigcache. the reload of the mempool from disk will repopulate it.
 132016-07-13T00:54:29  <gmaxwell> the stuff that doesn't get repopulated shouldn't be there in any case.
 142016-07-13T00:58:23  *** justanotheruser has quit IRC
 152016-07-13T00:59:09  *** justanotheruser has joined #bitcoin-core-dev
 162016-07-13T01:03:13  *** spudowiar has quit IRC
 172016-07-13T01:09:46  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 182016-07-13T01:12:32  <phantomcircuit> gmaxwell, yes but repopulating the mempool will take forever without saving the sigcache
 192016-07-13T01:15:53  <gmaxwell> phantomcircuit: 300mb of validation isn't ~that~ big a deal.
 202016-07-13T01:16:08  <gmaxwell> at least so long as it reloads the mempool in the background.
 212016-07-13T01:16:55  <gmaxwell> okay, perhaps worse than I'm thinking due to the fact that the signature validation isn't parallel
 222016-07-13T01:18:17  <gmaxwell> but even if the 300mbytes is alll txins, thats 145 bytes per signature, or about 2 million signatures, which even on a single moderately fast core should only be about 121 cpu seconds.
 232016-07-13T01:19:10  <gmaxwell> well whatever, 30% slower since I'm assuming libsecp256k1 with performance features we're not using yet.
 242016-07-13T01:24:34  <phantomcircuit> gmaxwell, it's quite a bit more than 300MiB on my system :)
 252016-07-13T01:25:24  <gmaxwell> phantomcircuit: having a mempool much larger than typical shouldn't increase your hitrate much and if you're mining may cause your to mine poorly propagated crap.
 262016-07-13T01:27:09  <phantomcircuit> it wont improve the hitrate for sigcache or compact blocks very much
 272016-07-13T01:27:22  <phantomcircuit> i dont see how it can result in you mining poorly propagated things though?
 282016-07-13T01:30:03  <gmaxwell> say someone advertises a txn with a very low feerate, it doesn't enter or quickly falls out of most nodes mempools.
 292016-07-13T01:30:05  <Chris_Stewart_5> If I add a different version of boost to my /usr/include I end up getting errors with core not being able to find it, is there some where I need to reference a newer version?
 302016-07-13T01:30:08  <Chris_Stewart_5> cfields:
 312016-07-13T01:30:58  <gmaxwell> phantomcircuit: but not yours... later, the network runs low on transactions in the pool or CPFP ranks that straggler up. now you mine it. And it's a surprise to everyone.
 322016-07-13T01:35:46  *** Ylbam has quit IRC
 332016-07-13T01:47:58  *** TomMc has quit IRC
 342016-07-13T01:57:55  <petertodd> bsm117532: for consensus critical applications, you're going to need to use the raw iterator, and even worse, follow the byte-level tests in the segwit codebase exactly
 352016-07-13T02:05:24  *** Chris_Stewart_5 has quit IRC
 362016-07-13T02:22:27  *** TomMc has joined #bitcoin-core-dev
 372016-07-13T02:23:39  *** go1111111 has quit IRC
 382016-07-13T02:25:21  *** DigiByteDev has joined #bitcoin-core-dev
 392016-07-13T02:33:18  *** DigiByteDev has joined #bitcoin-core-dev
 402016-07-13T02:34:32  *** DigiByteDev has joined #bitcoin-core-dev
 412016-07-13T02:39:41  *** Cory has joined #bitcoin-core-dev
 422016-07-13T03:04:12  <phantomcircuit> gmaxwell, true
 432016-07-13T03:04:30  <phantomcircuit> i think that is generally not easily solved though
 442016-07-13T03:04:45  <phantomcircuit> indeed unless you have the exact same limit as the entire rest of the network
 452016-07-13T03:04:48  <phantomcircuit> you're screwed there
 462016-07-13T03:04:56  *** TomMc has quit IRC
 472016-07-13T03:05:03  <phantomcircuit> otoh my brain is on fire so maybe im wrong
 482016-07-13T03:09:41  *** hsmiths has quit IRC
 492016-07-13T03:18:20  *** hsmiths has joined #bitcoin-core-dev
 502016-07-13T03:22:52  *** cryptapus_ has joined #bitcoin-core-dev
 512016-07-13T03:31:40  *** cryptapus_ has quit IRC
 522016-07-13T03:42:22  *** GreenIsMyPepper has quit IRC
 532016-07-13T03:51:31  *** cryptapus_ has joined #bitcoin-core-dev
 542016-07-13T03:51:31  *** cryptapus_ has joined #bitcoin-core-dev
 552016-07-13T03:55:46  *** GreenIsMyPepper has joined #bitcoin-core-dev
 562016-07-13T04:06:38  *** GreenIsMyPepper has quit IRC
 572016-07-13T04:11:47  *** GreenIsMyPepper has joined #bitcoin-core-dev
 582016-07-13T04:17:33  *** cryptapus_ has quit IRC
 592016-07-13T04:20:47  *** cryptapus_ has joined #bitcoin-core-dev
 602016-07-13T04:34:48  *** GreenIsMyPepper has quit IRC
 612016-07-13T04:40:18  *** GreenIsMyPepper has joined #bitcoin-core-dev
 622016-07-13T05:11:34  *** cryptapus_ has quit IRC
 632016-07-13T05:12:03  *** jtimon has joined #bitcoin-core-dev
 642016-07-13T05:25:51  *** frankenmint has joined #bitcoin-core-dev
 652016-07-13T05:30:16  *** GreenIsMyPepper has quit IRC
 662016-07-13T05:43:22  *** GreenIsMyPepper has joined #bitcoin-core-dev
 672016-07-13T05:44:11  *** frankenmint has quit IRC
 682016-07-13T05:49:51  *** fengling_ has joined #bitcoin-core-dev
 692016-07-13T05:50:06  *** fengling has quit IRC
 702016-07-13T05:56:16  *** GreenIsMyPepper has quit IRC
 712016-07-13T06:07:56  *** GreenIsMyPepper has joined #bitcoin-core-dev
 722016-07-13T06:08:15  *** morcos has quit IRC
 732016-07-13T06:08:16  *** zxzzt has quit IRC
 742016-07-13T06:09:02  *** zxzzt has joined #bitcoin-core-dev
 752016-07-13T06:09:07  *** morcos has joined #bitcoin-core-dev
 762016-07-13T06:10:10  *** DigiByteDev has quit IRC
 772016-07-13T06:11:48  *** go1111111 has joined #bitcoin-core-dev
 782016-07-13T06:12:18  *** GreenIsMyPepper has quit IRC
 792016-07-13T06:13:55  *** GreenIsMyPepper has joined #bitcoin-core-dev
 802016-07-13T06:39:23  *** Michail1 has quit IRC
 812016-07-13T06:40:37  *** davidlj95 has joined #bitcoin-core-dev
 822016-07-13T06:47:13  *** assder has joined #bitcoin-core-dev
 832016-07-13T06:56:10  *** bustd_soket has quit IRC
 842016-07-13T07:12:04  *** chris2000 has joined #bitcoin-core-dev
 852016-07-13T07:15:08  *** Ylbam has joined #bitcoin-core-dev
 862016-07-13T07:19:50  *** chris2000 has quit IRC
 872016-07-13T07:19:52  <GitHub99> [bitcoin] maiiz opened pull request #8336: Issues #8334 (master...issues-8334) https://github.com/bitcoin/bitcoin/pull/8336
 882016-07-13T07:20:37  *** bustd_soket has joined #bitcoin-core-dev
 892016-07-13T07:25:02  *** murch has joined #bitcoin-core-dev
 902016-07-13T07:25:38  <murch> gmaxwell: https://github.com/bitcoin/bitcoin/issues/7664#issuecomment-232230899 <-- I think you meant "is a large multiple" instead of "isn't"
 912016-07-13T07:28:57  *** Cheeseo has quit IRC
 922016-07-13T07:29:49  <jtimon> so NicolasDorier and I are discussing libconsensus
 932016-07-13T07:29:52  <gmaxwell> murch: ya, thanks.
 942016-07-13T07:30:57  <jtimon> do we want locks inside libconsensus? my understanding was no, due to complexity reasons. NicolasDorier thinks it's fine with std C++11 but it wasn't fine with boost
 952016-07-13T07:31:41  <jtimon> well, personally I think libconsensus should move towards plain C, but that's another discussion it seems I lost already
 962016-07-13T07:32:36  <sipa> jtimon: i was talking about this with NicolasDorier earlier
 972016-07-13T07:32:54  *** paveljanik has quit IRC
 982016-07-13T07:33:03  <sipa> jtimon: i expected that the point of contention would be the idea of it managing its own state
 992016-07-13T07:33:26  <sipa> (which would imply using locks for that state)
1002016-07-13T07:33:36  <NicolasDorier> ah yes you wanted to keep the libconsensus "stateless"
1012016-07-13T07:34:18  <sipa> i don't know whether it should be stateless - having state, especially for caches, would hugely simplify things
1022016-07-13T07:34:44  <sipa> but i think that should be the first thing to discuss
1032016-07-13T07:34:55  <sipa> as it using locks or not follows naturally
1042016-07-13T07:36:36  <phantomcircuit> sipa: that pr doesn't seem to have improved removeForBlock times by much
1052016-07-13T07:36:58  <gmaxwell> phantomcircuit: that means it improved them at all?
1062016-07-13T07:37:33  <phantomcircuit> im gonna need to do the mempool save/restor thing to know for sure unfortunately
1072016-07-13T07:37:54  <phantomcircuit> if it did it certainly wasn't spectacular
1082016-07-13T07:38:21  <NicolasDorier> I'm working right now on a verify_block for consensus lib. (like jtimon doing) I think the question of state won't come immediately to me as I first need to finish that. Once I done my verify_blocks, I'll just make several proposal for handling the hash cache of segwit
1092016-07-13T07:38:52  <NicolasDorier> I have several idea, will just try to code them see if they make sense at all :p
1102016-07-13T07:40:09  <sipa> NicolasDorier: how does your verifyblock get access to the chainstate and the block index?
1112016-07-13T07:40:23  *** KwukDuck has joined #bitcoin-core-dev
1122016-07-13T07:40:50  <KwukDuck> My Bitcoin Core client keeps crashing since a few days... http://pastebin.com/fiPXrzZY
1132016-07-13T07:40:53  <KwukDuck> any ideas?
1142016-07-13T07:41:18  <jtimon> sipa: well, I think the interface for the caches can be really simple, and we can provide the implementations we have for those who don't want to implement their own, so I disagree on "hugely simplify things"
1152016-07-13T07:41:35  <NicolasDorier> sipa: By cheating, I delegate to the client the calculation of the flags and contextual informations (height, bestblock, mtp)
1162016-07-13T07:41:56  <NicolasDorier> such that all the consensus code does not depends on CBlockIndex at all
1172016-07-13T07:42:03  <NicolasDorier> but just to 2 types
1182016-07-13T07:42:04  <phantomcircuit> KwukDuck, hardware error
1192016-07-13T07:42:13  <NicolasDorier> CConsensusContextInfo and CConsensusFlags
1202016-07-13T07:42:35  <jtimon> but yeah, my plan has always been to have a C++ version of verifyBlock first, well, I mean, first VerifyHeader, verifyTx...but never got past the point of simply moving the consensus functions out of main...
1212016-07-13T07:42:37  <NicolasDorier> both can be calculated by "main" with pIndex trivially. Then passed to consensus methods
1222016-07-13T07:42:39  <sipa> jtimon: yes, but that means the api changes whenever a cache is added/changed
1232016-07-13T07:43:13  <jtimon> yes, if you add a cache the api changes, not sure what you mean by changing the cache
1242016-07-13T07:43:30  <sipa> if it ends up working differently somehow
1252016-07-13T07:43:34  <jtimon> just as if the storage changes, the api for the storage would change as well
1262016-07-13T07:44:12  <NicolasDorier> I started working on https://github.com/NicolasDorier/bitcoin/commits/blkconsensus but still not done
1272016-07-13T07:44:13  <jtimon> assuming people want libconsensus to be storageless
1282016-07-13T07:44:40  <sipa> jtimon: yeah, i understand the use case
1292016-07-13T07:45:29  <sipa> it's just a lot harder to introduce abstractions for block indexes and chainstate, especially without degrading performance
1302016-07-13T07:45:48  <jtimon> I was working on https://github.com/jtimon/bitcoin/commits/jt but I broke the branch when backported bip9 and never cared to fix it (but review is still very welcomed since I plan to rewrite/cherry pick most of the stuff from there)
1312016-07-13T07:46:34  <jtimon> my interfaces: https://github.com/jtimon/bitcoin/commit/c97fa96c2f9e8989b9f1a9dd41688e07bcc163b9#diff-c2a099d775bac1dccc5f146a3cda81b8R82 https://github.com/jtimon/bitcoin/commit/34780bd9af20ddf60dd309eff90b9caf01b63f64
1322016-07-13T07:47:53  <NicolasDorier> my main problem with interface is that the callback will probably cross language boundary (C++ calling C#) and it is a performance nightmare (but maybe it has gone better now). I'm not against callback but this has to be coarse grained.
1332016-07-13T07:48:28  <NicolasDorier> ho I finish my PR I'll see when I hit there
1342016-07-13T07:49:13  <KwukDuck> phantomcircuit: Hardware error? Like what..? I think my hardware is fine? Experiencing no other issues with the system, even tried running it on a different disk.
1352016-07-13T07:49:56  <jtimon> at least for bitcoin core as a caller (assuming one day bitcoin core directly calls libconsensus instead of its internals) I don't see how performance would be affected very negatively, I worry more about serializing and deserialzing the parameters (like the tx in verify) for example, since people have complained (that's the reason why libbitcoin copies the code instead of using libconsensus' API directly)
1362016-07-13T07:50:32  <gmaxwell> KwukDuck: that kind of issue is usually caused by hardware error, lose or bad disk cables, or bad memory (run memtest86?)-- sometimes antivirus software can screw with the files bitcoin is using.
1372016-07-13T07:50:57  <jtimon> NicolasDorier: as said privately, I think people should use the function pointer to call their DB directly, not calling their external language there, but you said that may be complicated too
1382016-07-13T07:51:00  <gmaxwell> or running out of disk space potentially.
1392016-07-13T07:51:54  <NicolasDorier> jtimon: yes, it really depends on the backend technology. Some might have poor support of C lib. :(
1402016-07-13T07:52:00  <KwukDuck> gmaxwell: I considered a bad disk that's why i ran it on another one. Not sure about the memory though, but i'm not experiencing any other issues so... I guess i'll be running memtest later. Thanks :)
1412016-07-13T07:52:25  <NicolasDorier> also I know I'm not pro in C so it would take me 10 times  more than you to make use of a C lib even if it existed
1422016-07-13T07:53:04  <jtimon> on another topic, should utilmoneystr be part of libconsensus only for this line or should we just write the message in satoshis? https://github.com/bitcoin/bitcoin/pull/8329/files#diff-ca81084f62961a188f5c1e86a5ff1d7cR238
1432016-07-13T07:53:25  <jtimon> maaku things yes it should be part of libconsensus, I'm not very convinced
1442016-07-13T07:53:51  *** kadoban has quit IRC
1452016-07-13T07:55:00  <jtimon> if we're sure about it belonging in libconsensus, I should squash https://github.com/bitcoin/bitcoin/pull/8329/commits/8af7c64f6908ea4526efde9896d046c431016e00 in #8328
1462016-07-13T07:55:53  <jtimon> NicolasDorier: thoughts ?
1472016-07-13T07:57:00  <NicolasDorier> checking that a sec
1482016-07-13T07:57:43  *** laurentmt has joined #bitcoin-core-dev
1492016-07-13T07:58:34  <NicolasDorier> jtimon: well, I think the smaller the diff the better, so I am more inclined to keep it for now, but not really strongly neither
1502016-07-13T08:01:47  <jtimon> well, if I keep it and move it to the consensus dir, that actually makes the diff bigger
1512016-07-13T08:02:30  *** laurentmt has quit IRC
1522016-07-13T08:16:22  *** DigiByteDev has joined #bitcoin-core-dev
1532016-07-13T08:20:59  *** DigiByteDev has quit IRC
1542016-07-13T08:34:42  *** AaronvanW has quit IRC
1552016-07-13T08:37:21  *** AaronvanW has joined #bitcoin-core-dev
1562016-07-13T08:37:21  *** AaronvanW has joined #bitcoin-core-dev
1572016-07-13T08:41:34  *** Guyver2 has joined #bitcoin-core-dev
1582016-07-13T08:53:07  *** chris200_ has joined #bitcoin-core-dev
1592016-07-13T09:43:01  *** laurentmt has joined #bitcoin-core-dev
1602016-07-13T09:55:46  *** fengling_ has quit IRC
1612016-07-13T10:04:38  *** ArthurNumbanumba has joined #bitcoin-core-dev
1622016-07-13T10:22:49  *** YOU-JI has joined #bitcoin-core-dev
1632016-07-13T10:33:40  <jonasschnelli> sipa, gmaxwell: mind doing a review of https://github.com/bitcoin/bitcoin/pull/8323? I'd like and I think its important to have this in 0.13
1642016-07-13T10:41:12  *** fengling_ has joined #bitcoin-core-dev
1652016-07-13T10:45:46  *** fengling_ has quit IRC
1662016-07-13T10:55:01  *** xinxi_ has joined #bitcoin-core-dev
1672016-07-13T10:57:06  *** xinxi has quit IRC
1682016-07-13T11:17:58  <jonasschnelli> sipa, gmaxwell: maybe you find time to quickly review the BIP151 switch from HMAC_SHA512 "KDF"  to HKDF: https://github.com/bitcoin/bips/compare/master...jonasschnelli:2016/07/bip151_hkdf?expand=1
1692016-07-13T11:30:48  *** assder has quit IRC
1702016-07-13T11:31:39  *** harrymm has quit IRC
1712016-07-13T11:34:08  *** davidlj95 has quit IRC
1722016-07-13T11:39:00  *** arubi_ has joined #bitcoin-core-dev
1732016-07-13T11:42:28  *** fengling_ has joined #bitcoin-core-dev
1742016-07-13T11:43:04  *** arubi has quit IRC
1752016-07-13T11:43:17  *** afk11 has quit IRC
1762016-07-13T11:46:40  *** harrymm has joined #bitcoin-core-dev
1772016-07-13T11:47:26  *** fengling_ has quit IRC
1782016-07-13T11:57:10  *** cryptapus_ has joined #bitcoin-core-dev
1792016-07-13T11:57:10  *** cryptapus_ has joined #bitcoin-core-dev
1802016-07-13T11:57:11  *** cryptapus_ is now known as cryptapus_afk
1812016-07-13T11:57:28  *** cryptapus has quit IRC
1822016-07-13T12:06:37  *** arubi__ has joined #bitcoin-core-dev
1832016-07-13T12:10:30  *** arubi_ has quit IRC
1842016-07-13T12:20:10  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1852016-07-13T12:26:51  *** Lysander1 has quit IRC
1862016-07-13T12:40:41  *** YOU-JI has quit IRC
1872016-07-13T12:43:59  *** fengling_ has joined #bitcoin-core-dev
1882016-07-13T12:48:46  *** fengling_ has quit IRC
1892016-07-13T12:48:54  *** Chris_Stewart_5 has quit IRC
1902016-07-13T13:07:35  *** PaulCapestany has quit IRC
1912016-07-13T13:09:02  *** PaulCapestany has joined #bitcoin-core-dev
1922016-07-13T13:09:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1932016-07-13T13:13:00  *** laurentmt has quit IRC
1942016-07-13T13:15:34  *** go1111111 has quit IRC
1952016-07-13T13:21:21  <GitHub12> [bitcoin] jtimon closed pull request #8328: Consensus: Rename: Move consensus code to the consensus directory (master...0.12.99-consensus-rename) https://github.com/bitcoin/bitcoin/pull/8328
1962016-07-13T13:21:56  *** arubi__ is now known as arubi
1972016-07-13T13:26:10  <GitHub147> [bitcoin] jtimon reopened pull request #8328: Consensus: Rename: Move consensus code to the consensus directory (master...0.12.99-consensus-rename) https://github.com/bitcoin/bitcoin/pull/8328
1982016-07-13T13:27:07  *** Cheeseo has joined #bitcoin-core-dev
1992016-07-13T13:28:38  *** go1111111 has joined #bitcoin-core-dev
2002016-07-13T13:28:49  <jtimon> kanzure: NicolasDorier: https://github.com/bitcoin/bitcoin/pull/8337
2012016-07-13T13:32:46  <jtimon> and I will continue in https://github.com/jtimon/bitcoin/tree/0.12.99-consensus but I won't open more PRs for now
2022016-07-13T13:39:06  *** laurentmt has joined #bitcoin-core-dev
2032016-07-13T13:41:04  *** laurentmt has quit IRC
2042016-07-13T13:42:04  *** Chris_Stewart_5 has quit IRC
2052016-07-13T13:44:23  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2062016-07-13T13:46:47  *** fengling_ has joined #bitcoin-core-dev
2072016-07-13T13:49:21  *** Chris_Stewart_5 has quit IRC
2082016-07-13T13:50:26  *** fengling_ has quit IRC
2092016-07-13T13:54:12  *** TomMc has joined #bitcoin-core-dev
2102016-07-13T14:04:35  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2112016-07-13T14:20:57  *** arubi_ has joined #bitcoin-core-dev
2122016-07-13T14:23:52  *** kinlo has quit IRC
2132016-07-13T14:24:43  *** kinlo has joined #bitcoin-core-dev
2142016-07-13T14:25:04  *** arubi has quit IRC
2152016-07-13T14:27:06  *** bsm117532 is now known as bsm2319171311753
2162016-07-13T14:27:27  *** bsm2319171311753 is now known as bsm117532
2172016-07-13T14:30:27  *** bsm117532 is now known as bsm2357
2182016-07-13T14:47:01  *** fengling_ has joined #bitcoin-core-dev
2192016-07-13T14:50:46  *** laurentmt has joined #bitcoin-core-dev
2202016-07-13T14:51:02  *** laurentmt has quit IRC
2212016-07-13T14:51:46  *** fengling_ has quit IRC
2222016-07-13T15:09:07  *** arubi__ has joined #bitcoin-core-dev
2232016-07-13T15:13:51  *** arubi_ has quit IRC
2242016-07-13T15:21:28  *** molz has joined #bitcoin-core-dev
2252016-07-13T15:23:06  *** moli has quit IRC
2262016-07-13T15:28:50  *** arubi__ is now known as arubi
2272016-07-13T15:38:51  *** Chris_Stewart_5 has quit IRC
2282016-07-13T15:48:32  *** fengling_ has joined #bitcoin-core-dev
2292016-07-13T15:53:26  *** fengling_ has quit IRC
2302016-07-13T16:24:17  *** LeMiner2 has joined #bitcoin-core-dev
2312016-07-13T16:25:52  *** LeMiner has quit IRC
2322016-07-13T16:25:52  *** LeMiner2 is now known as LeMiner
2332016-07-13T16:38:09  <jtimon> NicolasDorier: it just occurred to me that you can always implement my function pointers interface by pre-filling memory with whatever data you were counting on passing as a parameter. Therefore it will be trivial to offer the all-data-upfront interface you want while using my more generic API inside. The costs will be to reading the pointer to a function, the function and executed it, but the function should be merely a map from
2342016-07-13T16:38:09  <jtimon>  your parameters to a pre-filled position in memory. Does that sound accetable to you? no C# inside
2352016-07-13T16:38:58  <jtimon> we can offer both interfaces with trivial development costs I think, let's see
2362016-07-13T16:39:51  *** owowo has quit IRC
2372016-07-13T16:42:01  *** spudowiar has joined #bitcoin-core-dev
2382016-07-13T16:55:05  *** owowo has joined #bitcoin-core-dev
2392016-07-13T17:08:41  *** laurentmt has joined #bitcoin-core-dev
2402016-07-13T17:08:58  *** laurentmt has quit IRC
2412016-07-13T17:12:33  *** spudowiar has quit IRC
2422016-07-13T17:32:36  <jtimon> just force pushed 0.12.99-consensus
2432016-07-13T17:33:09  <jtimon> but I will force push again soon
2442016-07-13T17:43:08  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2452016-07-13T17:44:11  *** spudowiar has joined #bitcoin-core-dev
2462016-07-13T17:51:44  *** fifth_ has joined #bitcoin-core-dev
2472016-07-13T18:07:26  *** fifth_ has quit IRC
2482016-07-13T18:08:10  *** kadoban has joined #bitcoin-core-dev
2492016-07-13T18:15:31  *** jgarzik has left #bitcoin-core-dev
2502016-07-13T18:15:31  *** jgarzik has quit IRC
2512016-07-13T18:30:13  <bsm2357> I expect a common error for people using python-bitcoinlib will be to call SignatureHash with the scriptPubKey of the corresponding segwit input, rather than an actual script.  This is easy to detect, and insert the corresponding script for calculating the sighash, but causes the SignatureHash function to depart from the behavior of the one in core.
2522016-07-13T18:30:34  <bsm2357> What are your opinions?  Is detecting this and inserting the correct script desirable or not?
2532016-07-13T18:33:49  <bsm2357> Hmmm.  Taking this question to #bitcoin-dev.
2542016-07-13T18:34:39  <sipa> bsm2357: perhaps also good to discuss on the repository itself in an issue
2552016-07-13T18:35:45  *** Chris_Stewart_5 has quit IRC
2562016-07-13T18:35:58  <bsm2357> Of course.  I'll push my latest there soon with a working SignatureHash.  Just looking for some quick feedback from people who might use it, before I go write code.
2572016-07-13T18:41:49  *** frankenmint has joined #bitcoin-core-dev
2582016-07-13T18:49:00  *** Chris_Stewart_5 has joined #bitcoin-core-dev
2592016-07-13T18:53:55  *** fengling_ has joined #bitcoin-core-dev
2602016-07-13T18:58:06  *** fengling_ has quit IRC
2612016-07-13T19:03:36  <Chris_Stewart_5> Is it unusual for nodes to give back one ip address back for a getaddr message?
2622016-07-13T19:05:13  <sipa> we only respond to one getaddr per connection
2632016-07-13T19:05:14  <sipa> after that, only normal addr relay
2642016-07-13T19:06:58  <Chris_Stewart_5> Ok, but it it unusual to respond with only one ip address? The behavior I am seeing is connecting to a node, I send a getaddr message, then I only get ONE ip address back in the addr message
2652016-07-13T19:07:20  <Chris_Stewart_5> and that ip address is the node's ip address. I guess I was under the impression that it would send back the nodes that it is connected to
2662016-07-13T19:07:31  <sipa> it's not responding at all
2672016-07-13T19:07:42  <sipa> the addr you see is likely just an independently relayed addr
2682016-07-13T19:08:04  <sipa> hell no, nodes try to hide who they are connected to
2692016-07-13T19:08:05  <xinxi_> jtimon: are you still there?
2702016-07-13T19:09:41  <Chris_Stewart_5> so how do you connect with more than one node on the network then? It seems that you could only connect to dns seeds then.
2712016-07-13T19:10:02  <sipa> i don't understand the question
2722016-07-13T19:10:16  <sipa> normally, you get 1000 IP addresses back in response to getaddr
2732016-07-13T19:10:26  <sipa> unless the peer does not know about many, of course
2742016-07-13T19:10:42  <Chris_Stewart_5> I should preface this with it is testnet, perhaps the behavior is different?
2752016-07-13T19:10:58  <sipa> it's very likely that on testnet the peer just does not know about many peers
2762016-07-13T19:11:21  <jtimon> xinxi_: yes
2772016-07-13T19:11:24  <sipa> also, if you send getaddr multiple times, the second and later times it is just ignored
2782016-07-13T19:11:47  <xinxi_> jtimon: great. I want to discuss with you about the libconsensus that you are working on.
2792016-07-13T19:11:58  <xinxi_> it's a great project.
2802016-07-13T19:12:14  <jtimon> xinxi_: absolutely, always interested in discussing that
2812016-07-13T19:12:46  <Chris_Stewart_5> sipa: I found that out the hard way :-). But it seems unlikely that a response to a getaddr message would have just ONE ip address even on testnet, right?
2822016-07-13T19:13:22  <xinxi_> my I know when this will be merged into the master branch?
2832016-07-13T19:14:01  <jtimon> xinxi_: right now the longest branch I have is https://github.com/jtimon/bitcoin/tree/0.12.99-consensus but I still have things to cherry pick from https://github.com/jtimon/bitcoin/commits/jt (plus some things that will just have to be new)
2842016-07-13T19:15:41  <jtimon> xinxi_: uff, no idea myself, I wish I knew, I believe the idea is focusing on refactors like this and the ones going on in net.o and wallet right after forking 0.13's branch, which should be soon
2852016-07-13T19:16:07  <jtimon> whether the PRs will be merged or not, I cannot know, depends on review
2862016-07-13T19:16:11  *** sipa has quit IRC
2872016-07-13T19:16:22  <xinxi_> does libconsensus include the wallet and the network parts?
2882016-07-13T19:16:28  <jtimon> no
2892016-07-13T19:16:48  <jtimon> but there's other people doing refactors both in the net and wallet parts
2902016-07-13T19:17:18  <jtimon> ie cfields and jonasschnelli respectively
2912016-07-13T19:17:19  <morcos> sipa: i think on machines with a lot of cores, the sig cache actually slows down block validation.   my guess is its the contention on the lock since we erase every time
2922016-07-13T19:17:46  <morcos> sipa: i think it might be better to batch erase, or randomly erase or something else
2932016-07-13T19:18:06  <cfields> morcos: i've scratched my head at that one several times. seems we'd be better off nuking a bucket at a time or so
2942016-07-13T19:18:38  <xinxi_> jtimon: cool. so all the code in libconsensus is in this directory https://github.com/jtimon/bitcoin/tree/0.12.99-consensus/src/consensus, right?
2952016-07-13T19:18:43  *** spudowiar has quit IRC
2962016-07-13T19:19:03  <morcos> cfields: it defaults to nuking a bucket at a time if it gets full anyway (although i'm not sure thats super efficient either, but that tends to happen in ATMP, so who cares)
2972016-07-13T19:19:14  <jtimon> morcos: I once tried with a huge sigcache just to see if the swap partition was working. didn't benchmarked but it certainly didn't seem to accelerate things
2982016-07-13T19:19:31  <cfields> morcos: eh? i thought it just killed a single entry each time
2992016-07-13T19:20:50  <jtimon> xinxi_: no, there's still some in main.cpp, but the idea it's that it all should end up there (without counting the code in src/crypto and libsecp256k1)
3002016-07-13T19:20:51  <cfields> morcos: looks to me like it just picks a random bucket and removes the first entry in it. am i misreading?
3012016-07-13T19:20:57  <morcos> cfields: the problem i'm referring to is that on validating a block, you erase each stored signature as you read it
3022016-07-13T19:20:58  *** sipa has joined #bitcoin-core-dev
3032016-07-13T19:21:29  <morcos> but yes, if it's gotten to big anyway, then it'll pick a random bucket and remove entires until its below the limit, but isn't that what you said?
3042016-07-13T19:21:38  <jtimon> I will keep updating jtimon/0.12.99-consensus moving more things there though
3052016-07-13T19:21:42  <xinxi_> jtimon: I guess most of the work has already been done?
3062016-07-13T19:21:51  <Chris_Stewart_5> sipa: also you said it can relay 1000 addresses at a time, doesn't that effectively give a history of the nodes they are connected to?
3072016-07-13T19:22:01  <morcos> the second thing happens when trying to set a new entry, which is in ATMP, so not important to shave micros... as it is in block validation
3082016-07-13T19:22:05  <sipa> Chris_Stewart_5: nodes typically know way more 1000 addresses
3092016-07-13T19:22:20  <sipa> Chris_Stewart_5: and typically know of many addresses they are not currently, or even never have been, connected to
3102016-07-13T19:22:39  <Chris_Stewart_5> sipa: So basically caching addresses from addr messages?
3112016-07-13T19:22:56  <sipa> Chris_Stewart_5: yes, and from dns seed responses
3122016-07-13T19:22:56  <Chris_Stewart_5> unsolicited ones
3132016-07-13T19:23:01  <sipa> Chris_Stewart_5: that's addrman
3142016-07-13T19:23:17  <cfields> morcos: ah yes, i was talking about the full case
3152016-07-13T19:23:54  <xinxi_> jtimon: we plan to prove the correctness of the consensus part.
3162016-07-13T19:24:20  <sipa> xinxi_: again, correctness compared to which specification?
3172016-07-13T19:24:37  <morcos> cfields: yeah the problem is lock contention on validation since each of your script validating cores is contending on the sig cache lock since they each have to erase from it.  if you eliminate or batch those erases, it speeds up validation significantly
3182016-07-13T19:24:47  <jtimon> xinxi_: well, the part of making a complete verifyBlock, I would say it has been done several times. exposing it as a C API is another thing, specially since probably the APIs will be discussed extensively (plus I never decoupled Consensus::Params from uint256)
3192016-07-13T19:24:48  <sipa> morcos: interesting
3202016-07-13T19:24:49  <xinxi_> sipa: as discussed, we are going to derive a specification from the code.
3212016-07-13T19:25:04  <jtimon> xinxi_: interesting
3222016-07-13T19:25:13  <sipa> xinxi_: a specification derived from the code will by definition match the code, no? :)
3232016-07-13T19:25:32  <Chris_Stewart_5> sipa: Is there documentation for addrman any where? Doesn't seem to be in the p2p networking part: https://bitcoin.org/en/developer-reference#p2p-network
3242016-07-13T19:25:41  <sipa> Chris_Stewart_5: in addrman.h :)
3252016-07-13T19:25:45  <jtimon> sipa: I guess the goal is that from then on you can't change the code without appropriately changing the specifications or the tests will fail
3262016-07-13T19:25:58  <sipa> jtimon, xinxi_: yes, that would be awesome
3272016-07-13T19:26:00  <cfields> morcos: hmm. ignoring larger fixes, if the contention is that high, i wonder if the shared lock is doing more harm than good
3282016-07-13T19:26:00  <xinxi_> sipa: i would say it's going to match the code as much as possible unless we find bugs/inconsistencies in the specification.
3292016-07-13T19:26:12  <sipa> xinxi_: then the specification is wrong
3302016-07-13T19:26:17  <jtimon> hehe
3312016-07-13T19:26:41  <sipa> (but it would still be very interesting to know)
3322016-07-13T19:26:46  <jtimon> the specification is clearly my branch
3332016-07-13T19:26:56  <xinxi_> sipa: yeah, as discussed, we don't want to have a specification with flaws or bugs. if we find that, we should upgrade.
3342016-07-13T19:27:11  <sipa> xinxi_: i think you're dreaming if you think that's going to happen
3352016-07-13T19:27:18  <morcos> sipa: on my 16 core machine, i think it speeds up the "Connect total" part by about 40% to just not erase..   But I didnt' run that over a long enough period to model the lower hit rate that'll eventually result from not having smart erasing.  If we can conveniently batch the erases from the block that'll be best
3362016-07-13T19:27:27  <xinxi_> sipa: why is that?
3372016-07-13T19:27:49  <xinxi_> a hard fork is not possible?
3382016-07-13T19:27:56  <sipa> xinxi_: because i expect a hard fork to switch to something with only theoretical advantages to be very controversial
3392016-07-13T19:28:41  <xinxi_> sipa: well, we will definitely test that first.
3402016-07-13T19:28:53  <cfields> morcos: whoa
3412016-07-13T19:28:56  <jtimon> xinxi_: yes, but a hardfork doesn't magically get deployed in all computers when you update the specification, whatever is deployed is the specification in practice
3422016-07-13T19:29:16  <sipa> xinxi_: not trying to discourage you... i would be very exciting if you can contribute to the correctness of the system using formal proofs, at whatever level
3432016-07-13T19:29:27  <sipa> xinxi_: but please... keep it about analysing the actual code
3442016-07-13T19:29:35  <sipa> *excited
3452016-07-13T19:29:47  <xinxi_> sipa: the actual code is in C++, which is almost impossible to prove.
3462016-07-13T19:30:01  <xinxi_> if you rewrite it in C, we may be able to help.
3472016-07-13T19:30:12  <sipa> even that has been controversial in the past
3482016-07-13T19:30:20  <jtimon> xinxi_: in the end the new tests are going to fail when you change either the specification or the implementation, right?
3492016-07-13T19:30:52  <xinxi_> jtimon: what do you mean by failure?
3502016-07-13T19:31:09  <sipa> xinxi_: you could of course do a rewrite in C, and analyse that... we could learn very interesting things from that
3512016-07-13T19:31:20  <sipa> xinxi_: but that still doesn't mean we need to actually switch to that C code
3522016-07-13T19:31:30  <jtimon> if you changed the specification, sipa was right, the bug was there, if you change the implementation but you were't expecting change in funtionality, then your changes are wrong
3532016-07-13T19:31:55  <xinxi_> jtimon: how did you guys fix bugs in the past I am wondering.
3542016-07-13T19:32:01  <sipa> xinxi_: soft forks
3552016-07-13T19:32:05  <xinxi_> so you just leave the bugs there?
3562016-07-13T19:32:06  <jtimon> let me search for a ccc talk about what I'm thinking you wanted to do
3572016-07-13T19:32:22  <sipa> xinxi_: if they're not dangerous and merely annoying.. yes, we have no other choice
3582016-07-13T19:32:23  <jtimon> xinxi_: no, sipa answered: softforks
3592016-07-13T19:32:29  <sipa> xinxi_: we don't decide the rules of the network
3602016-07-13T19:32:59  <xinxi_> OK. so can we just deploy the slightly different bug free version using a soft fork?
3612016-07-13T19:33:05  <sipa> maybe
3622016-07-13T19:33:16  <sipa> that depends on the kind of changes you want to make
3632016-07-13T19:33:18  <jtimon> for example, the code in bip99 fixes a known bug which is not a priority
3642016-07-13T19:33:27  <jtimon> but that fix is a hardfork
3652016-07-13T19:33:33  *** d_t has joined #bitcoin-core-dev
3662016-07-13T19:34:09  <xinxi_> so can you explain to me what kind of changes require a soft fork and what kind of changes require a hard fork?
3672016-07-13T19:34:23  <xinxi_> and what's the difference between the hard and soft forks?
3682016-07-13T19:34:24  <sipa> xinxi_: everything that leaves things that were previously invalid invalid is a softfork
3692016-07-13T19:35:05  <xinxi_> how do you define invalid?
3702016-07-13T19:35:20  <jtimon> meh, can't find the ccc video about proving correctness, I though you were coming from there
3712016-07-13T19:35:27  <sipa> xinxi_: abstractly speaking, the consensus rules are black box which you feed a sequence of blocks
3722016-07-13T19:35:47  <sipa> xinxi_: and it returns whether that sequence of blocks is valid or not
3732016-07-13T19:35:51  <jtimon> their "specification" was basically another program for an abstract machine
3742016-07-13T19:36:08  <sipa> it cannot depend on the order in which those blocks were received, or whatever earlier states the node went through
3752016-07-13T19:36:46  <sipa> xinxi_: a softfork leaves all previously invalid inputs invalid
3762016-07-13T19:37:44  <xinxi_> so a soft fork can only invalid a previously valid input.
3772016-07-13T19:37:48  <sipa> correct
3782016-07-13T19:37:51  *** frankenmint has quit IRC
3792016-07-13T19:37:55  <jtimon> or in other words, it only add restrictions, doesn't remove them: everything that was invalid remains invalid
3802016-07-13T19:37:55  <Chris_Stewart_5> xinxi_: You can think of the set of all consensus rules R with cardinality n, a soft fork increments the cardinality of R one.
3812016-07-13T19:38:03  <xinxi_> and a hard fork has no such restriction.
3822016-07-13T19:38:07  <Chris_Stewart_5> xinxi_: Any removal from the set R is a hard fork though
3832016-07-13T19:38:08  <gmaxwell> xinxi_: the whole purpose of the system it to come to a consistent worldwide agreement on the history of the ledger. This is the first and foremost definition of correctness in the system.
3842016-07-13T19:38:21  <sipa> xinxi_: and it's very surprising how many things are actually possible with softforks
3852016-07-13T19:38:46  <xinxi_> sipa: sure, i can imaging that.
3862016-07-13T19:38:52  <sipa> xinxi_: one analogy is seeing the consensus rules a block of marble from which pieces are cut away that give the complexity
3872016-07-13T19:39:32  <xinxi_> so how do hard forks and soft forks differ in terms of deployment?
3882016-07-13T19:39:39  <jtimon> some people theorize that everything that is possible via hardfork is also possible via softfork in some other way, but I'm not convinced that means we should always prefer softforks
3892016-07-13T19:39:47  <sipa> a hard fork: convince the whole world to switch to different code; period
3902016-07-13T19:39:55  <sipa> a soft fork: read BIP34 and BIP9
3912016-07-13T19:40:05  <sipa> (BIP9 is newer)
3922016-07-13T19:40:25  <xinxi_> has any hard fork happened?
3932016-07-13T19:40:37  <sipa> very early on in the history of the system, yes
3942016-07-13T19:40:43  <sipa> in the first 1.5 years
3952016-07-13T19:40:52  <jtimon> xinxi_: for softforks miners need to coordinate for deployment, then other nodes can upgrade when they want/can. for hardforks everybody needs to upgrade before deployment, which makes them slower to deploy
3962016-07-13T19:41:35  <sipa> xinxi_: there is also BIP50 which is a post-mortem of a consensus failure that occurred between old and new versions of the system
3972016-07-13T19:41:42  <xinxi_> jtimon: how can you upgrade without deployment?
3982016-07-13T19:42:05  <xinxi_> basically, you can automatically activate when a certain height is reached?
3992016-07-13T19:42:09  <sipa> xinxi_: softforks only apply new rules to blocks after some changeover point
4002016-07-13T19:42:18  <sipa> they use old rules for everything before
4012016-07-13T19:42:32  <xinxi_> sipa: yeah, you've explained that well.
4022016-07-13T19:42:38  <sipa> BIP30 used a fixed timestamp for the changeover point
4032016-07-13T19:42:51  <jtimon> xinxi_: for softforks bip9 describes it, for hardforks there's different opinions on what would be the best way to signal activation
4042016-07-13T19:43:22  <xinxi_> jtimon: got it.
4052016-07-13T19:43:30  *** d_t has quit IRC
4062016-07-13T19:43:46  *** d_t has joined #bitcoin-core-dev
4072016-07-13T19:43:56  <sipa> BIP34 used block versions above a certain number to let miners indicate they are enforcing the new rules
4082016-07-13T19:44:01  <xinxi_> so who makes the final decision on whether a fork should be deployed or not?
4092016-07-13T19:44:10  <sipa> xinxi_: for a soft fork: miners
4102016-07-13T19:44:34  <sipa> as it's really just miners deciding to start enforcing some additional rules in addition to what the network requires them to
4112016-07-13T19:44:43  <sipa> for a hard fork: politics
4122016-07-13T19:44:52  <sipa> (let's not go there)
4132016-07-13T19:45:01  <xinxi_> the politics between whom?
4142016-07-13T19:45:05  <jtimon> xinxi_: here's a few different ways to coordinate activation, the last 3 in there use bip 9 : https://github.com/jtimon/bitcoin/blob/0.12.99-consensus/src/consensus/versionbits.cpp#L172
4152016-07-13T19:45:11  <xinxi_> I think it's better to clarify things.
4162016-07-13T19:45:12  <petertodd> xinxi_: a hard fork is arguably the creation of a new currency for starters...
4172016-07-13T19:45:20  <sturles> xinxi_: All bitcoin users.
4182016-07-13T19:45:38  <sipa> xinxi_: some people believe developers or some committee should decide on hard forks, and then assume everyone will follow them
4192016-07-13T19:46:30  <sipa> xinxi_: others think hard forks should be reserved for clearly uncontroversial things
4202016-07-13T19:46:36  <xinxi_> is there any democratic decision making process?
4212016-07-13T19:46:56  <sipa> democracy is pretty hard in a system designed to avoid fixed identities
4222016-07-13T19:47:00  <petertodd> xinxi_: there's no agreement on what a "democratic decision making process" even looks like
4232016-07-13T19:47:13  <jtimon> bip2 tries to clarify things, but it can only decided on whether something was "accepted" or not observing adoption a posteriori. BIP99 also tries to clarify things, but it's based on a vaguely defined concept of "uncontroversial"
4242016-07-13T19:47:20  <xinxi_> sipa: sybil attack.
4252016-07-13T19:47:24  <petertodd> xinxi_: whose votes do you count? miner-triggered soft-forks at least have a natural metric: hashing power
4262016-07-13T19:47:46  <sdaftuar> 15:43 < sipa> xinxi_: for a soft fork: miners
4272016-07-13T19:47:49  <xinxi_> then, can we just let miners vote?
4282016-07-13T19:47:56  <sdaftuar> ^ this is not correct, users have to adopt the rules as well
4292016-07-13T19:48:07  <sdaftuar> but please, this discussion should be elsewhere
4302016-07-13T19:48:09  <jtimon> I wouldn't call that a vote, but just confirmation that they have upgraded
4312016-07-13T19:48:16  <jtimon> for coordination
4322016-07-13T19:48:20  <sipa> xinxi_: miners should absolutely not be allowed to decide on hardforks (they could for example decide to increase their own reward that way)
4332016-07-13T19:48:29  <jtimon> the softfork is supposed to be uncontroversial in the first place
4342016-07-13T19:48:29  <xinxi_> sipa: why not?
4352016-07-13T19:48:52  <petertodd> xinxi_: "why not?" is a political question...
4362016-07-13T19:48:55  <sipa> xinxi_: "who watches the watchmen?"
4372016-07-13T19:49:20  <jeremyrubin> xinxi_: probably discuss this on #bitcoin
4382016-07-13T19:49:20  <jtimon> sipa: not only they shouldn't not be allowed but they have no power over hardforks in practice, "should" or "shouldn't" is irrelevant here
4392016-07-13T19:49:24  <sipa> xinxi_: they're called consensus rules because every user in the system choose to accept them
4402016-07-13T19:49:37  <xinxi_> sipa: sure. that's a good reason.
4412016-07-13T19:49:44  <sipa> agree, this is getting off topic
4422016-07-13T19:49:45  <sipa> sorry
4432016-07-13T19:50:01  <jtimon> yep, sorry, #bitcoin I guess
4442016-07-13T19:50:02  <xinxi_> well, it's not off topic actually.
4452016-07-13T19:50:15  <jtimon> well, it's not really development
4462016-07-13T19:50:15  <sipa> it is off topic for the developer of bitcoin core, which is what this channel is about
4472016-07-13T19:50:28  <xinxi_> i need to make sure that tremendous amount of effort that could make Bitcoin even greater will be deployed.
4482016-07-13T19:50:49  <petertodd> xinxi_: I'd suggest you take that kind of discussion to #bitcoin-wizards at least
4492016-07-13T19:51:17  <sipa> xinxi_: if your tremendous amount of effort requires a hard fork, the answer is that nobody can promise you it will happen
4502016-07-13T19:51:32  <xinxi_> if there is no certainty but extremely high risk, we will just have to give up.
4512016-07-13T19:52:05  <sipa> xinxi_: i think there are very interesting things you can do if you aim at something a bit less ambitious than proofs about the entire consensus system
4522016-07-13T19:52:20  <petertodd> xinxi_: if you want certainty, then yes, I'd suggest you give up. but I'd prefer you took sipa's advice
4532016-07-13T19:52:54  <xinxi_> petertodd: so you mean i should at least try no matter what the result is?
4542016-07-13T19:53:38  <sipa> xinxi_: you can focus on for example just a scripting language... rewriting that part in C would not be very hard, and the result could be very interesting
4552016-07-13T19:54:04  <petertodd> xinxi_: I'm just saying, that anything that for any work you do that may require a hard-fork to deploy, you should accept that you may need to create a new currency, economically distint from bitcoin, to deploy your work in production
4562016-07-13T19:54:07  <sipa> proofs about its complexity, memory usage, equivalence after certain changes
4572016-07-13T19:54:35  *** fengling_ has joined #bitcoin-core-dev
4582016-07-13T19:54:36  <petertodd> xinxi_: whereas if you want high certainty of being able to deploy your work in production on bitcoin, you'll need to set your sights lower to thinks that can be done in a soft-fork
4592016-07-13T19:55:02  <xinxi_> sipa: complexity, memory usage, etc are certainly important but not the most critical thing.
4602016-07-13T19:55:10  <sipa> xinxi_: also, we can always propose a softfork that effectively delegates validation under certain conditions to a completely new scripting language
4612016-07-13T19:55:15  <petertodd> xinxi_: for example basically all my scalability research is stuff that I do knowing full well that it may never be able to be deployed on bitcoin
4622016-07-13T19:55:18  <Chris_Stewart_5> sipa: Does it make any sense at all that a dns seed on testnet would only relay one addr from a getaddr message? Sorry if answered already but I'm still unclear on this
4632016-07-13T19:55:22  <sipa> xinxi_: which can be implemented in a different language
4642016-07-13T19:55:43  <sipa> xinxi_: BIP143 (which we're in the process of finished up) makes that very easy
4652016-07-13T19:56:01  <sipa> xinxi_: it's effectively designed to allow new scripting language to be plugged in in a softfork compatible way
4662016-07-13T19:56:25  <xinxi_> petertodd: do you have a fulltime job?
4672016-07-13T19:56:48  <sipa> xinxi_: there are people working on thinking about improvements that could be made in a new scripting language
4682016-07-13T19:57:07  <sipa> xinxi_: and for that, all options are open
4692016-07-13T19:57:12  <xinxi_> sipa: why does a scripting language make a difference?
4702016-07-13T19:57:25  <petertodd> xinxi_: I'm a consultant, who works pretty much full time on contracts in this space (including Bitcoin Core development contracts)
4712016-07-13T19:57:32  *** d_t has quit IRC
4722016-07-13T19:57:44  <sipa> xinxi_: by 'scripting' we mean bitcoin Script here, which is the language used inside transaction outputs to describe the condition under which they can be spent
4732016-07-13T19:57:51  <sipa> xinxi_: it's not scripting as in bash or python
4742016-07-13T19:57:57  <xinxi_> petertodd: So Bitcoin Core developers get paid?
4752016-07-13T19:58:15  <sipa> it's a bytecode language inspired by Forth
4762016-07-13T19:58:41  <sipa> xinxi_: and arguable, the implementation of that language interpreter is the most complicated piece of the consensus rules
4772016-07-13T19:58:43  <petertodd> xinxi_: many do - there's no "bitcoin foundation" that pays core developers, but there's lots of ways to get paid
4782016-07-13T19:58:50  <xinxi_> sipa: yeah, i know what you are talking about. but doesn't that require a hard fork?
4792016-07-13T19:58:56  <sipa> xinxi_: nope!
4802016-07-13T19:59:04  <petertodd> xinxi_: (there is a "bitcoin foundation", but they don't pay anyone, and are essentially bankrupt last I checked)
4812016-07-13T19:59:17  <sipa> xinxi_: the basic idea is that we can redefine some of the remaining NOP opcodes in the language
4822016-07-13T19:59:26  *** fengling_ has quit IRC
4832016-07-13T19:59:32  <sipa> xinxi_: it goes further than that, but it would lead me to far to explain it all here
4842016-07-13T19:59:49  <xinxi_> sipa: i know what you mean.
4852016-07-13T20:00:03  <sipa> and redefining NOPs can be done as a softfork
4862016-07-13T20:00:17  <sipa> (as can the new script versions introduced by BIP143)
4872016-07-13T20:00:56  <xinxi_> sipa: you invalid more blocks syntatically, but you add more semantics.
4882016-07-13T20:01:27  <sipa> xinxi_: we basically take a particular script whose meaning was "anyone can spend this", and replace it with a new meaning
4892016-07-13T20:01:51  <sipa> aka moving to a new piece of marable
4902016-07-13T20:01:53  <sipa> marble
4912016-07-13T20:02:09  <xinxi_> now I also have a feeling that softforks can solve all problems.
4922016-07-13T20:02:59  <sipa> to give an extreme example... we can switch to a different proof of work function using softforks
4932016-07-13T20:03:16  <petertodd> sipa: explain :)
4942016-07-13T20:03:29  <sipa> (in a hacky way that likely wouldn't actually work, as it would go against the interest of miners who decide on softforks...)
4952016-07-13T20:04:02  <sipa> petertodd: define a height-dependent function that maps difficulty of the old PoW to difficulty of the new one
4962016-07-13T20:04:12  <sipa> petertodd: now demand that every block satisfies both PoWs
4972016-07-13T20:04:31  <sipa> and choose the function so that the difficulty of the old one becomes almost irrelevant over time
4982016-07-13T20:04:38  <petertodd> sipa: eh... that's not "switching" to a different PoW, that's additing an additional PoW
4992016-07-13T20:04:46  <Chris_Stewart_5> interesting
5002016-07-13T20:04:57  <sipa> of course... in a softfork we can by definition only 'add' rules
5012016-07-13T20:05:08  <sipa> but in practice it would mean switching
5022016-07-13T20:05:36  <sipa> (this is a contrived example, and not something i'd ever actually propose)
5032016-07-13T20:05:36  <petertodd> sipa: also, I'm not sure I'd want to call that a soft-fork, given that the cost for an attacker to attack old clients goes down to zero - I'd limit the use of the term "soft-fork" to things that retain lite-client level security for non-adopting clients indefinitely
5042016-07-13T20:05:48  *** d_t has joined #bitcoin-core-dev
5052016-07-13T20:06:00  <petertodd> sipa: example: the switch to most-work-chain from bitcoin 0.1's longest-chain rule is something I'd definitely call a hard-fork
5062016-07-13T20:06:02  <sipa> petertodd: fair point
5072016-07-13T20:06:35  <sipa> interesting... i'm not sure i would see the chain selection rule as a part of consensus
5082016-07-13T20:06:42  *** moli has joined #bitcoin-core-dev
5092016-07-13T20:06:44  <petertodd> I certainly would!
5102016-07-13T20:06:54  <sipa> it's necessary for convergence... but many things are necessary for convergence, like a working p2p system
5112016-07-13T20:07:38  <xinxi_> about soft fork, can you explain to me how it gets deployed and why we can only be more restrictive?
5122016-07-13T20:07:56  <sipa> xinxi_: read BIP9 about how to deploy it
5132016-07-13T20:08:10  <petertodd> I could make the chain selection rule be "only headers whose merkle-root end with the byte string 'peter todd'", and we'd quickly e in a situation where far less than 50% of hashing power is sufficient to attack non-adopting clients
5142016-07-13T20:08:19  <sipa> xinxi_: why we can only be more restrictive... so that all rules demanded by old nodes remain satisfied by the majority of the hash rate
5152016-07-13T20:08:37  <petertodd> sipa: _exactly_: your example fails the "majority of the hash rate" condition
5162016-07-13T20:08:52  <sipa> petertodd: interesting
5172016-07-13T20:08:55  <sipa> i agree
5182016-07-13T20:09:04  *** molz has quit IRC
5192016-07-13T20:09:22  <sipa> it is a validation-rule-more-restrictive change then, but not a soft forking change
5202016-07-13T20:09:42  <petertodd> sipa: having said that, I think we need a new term for the case where a hard-fork is introduced gradually; maybe staged hard-fork?
5212016-07-13T20:10:05  <sipa> we should name the different types of consensus rule changes after pokemon
5222016-07-13T20:10:26  * sipa hides
5232016-07-13T20:10:40  <xinxi_> Ha, I see the key is backward compatibility.
5242016-07-13T20:10:47  * petertodd glares at sipa
5252016-07-13T20:10:53  <petertodd> xinxi_: yup
5262016-07-13T20:10:55  <xinxi_> nodes accepted by new nodes should be accepted by old nodes too.
5272016-07-13T20:10:59  <sipa> xinxi_: otherwise, any old node will end up not accepting the majority chain
5282016-07-13T20:11:06  <xinxi_> that's the easiest explanation to me.
5292016-07-13T20:11:08  <petertodd> xinxi_: you mean, blocks accepted by...
5302016-07-13T20:11:19  <xinxi_> petertodd: yep
5312016-07-13T20:11:25  <xinxi_> that's a typo.
5322016-07-13T20:11:26  <sipa> xinxi_: and the ledger will split in half, where each pre-existing coin becomes spendable on both sides of the fork
5332016-07-13T20:11:56  <xinxi_> that's interesting.
5342016-07-13T20:11:57  <petertodd> xinxi_: spendable on both sides of the fork, because the hard-fork _is_ the creation of a new currency
5352016-07-13T20:12:19  <xinxi_> do you know COM+ invented by Microsoft? It's also backward compatible.
5362016-07-13T20:12:43  <xinxi_> But they can keep releasing new versions of DirectX, which is based on COM+.
5372016-07-13T20:12:44  <sipa> yup, just a currency that implicitly assigns the new coins exactly according to how they were distributed at the forking point in the old currency
5382016-07-13T20:12:49  <petertodd> xinxi_: heck, I've been asked two or three times now by exchanges and the like to vet their plans for separating UTXO's cleanly onto both sides post-hard-fork, so they can trade/withdraw them separately (likely to be a legal requirement)
5392016-07-13T20:13:28  <xinxi_> petertodd: that's funny.
5402016-07-13T20:13:30  *** Chris_Stewart_5 has quit IRC
5412016-07-13T20:18:56  <xinxi_> so how to nodes check whether a block is valid or not?
5422016-07-13T20:19:17  <xinxi_> do they just check the syntactics?
5432016-07-13T20:19:44  *** afk11 has joined #bitcoin-core-dev
5442016-07-13T20:19:45  *** afk11 has quit IRC
5452016-07-13T20:19:45  *** afk11 has joined #bitcoin-core-dev
5462016-07-13T20:19:56  <petertodd> xinxi_: what do you mean by "the syntactics"?
5472016-07-13T20:20:09  <xinxi_> i mean the format of the block data.
5482016-07-13T20:20:23  <petertodd> xinxi_: you mean, without the context of previous blocks?
5492016-07-13T20:20:55  <xinxi_> petertodd: that's unlikely to be true. they need to verify transactions, which are dependent on history.
5502016-07-13T20:21:17  <petertodd> xinxi_: indeed
5512016-07-13T20:21:32  <xinxi_> petertodd: yeah, so how do they check?
5522016-07-13T20:21:36  <sipa> xinxi_: 1) we receive headers and verify them syntactically, and check whether their PoW matches, then check whether they connect to previous headers, and their expected difficulty is correct, then store them
5532016-07-13T20:22:08  <xinxi_> how about the block size? the op codes used?
5542016-07-13T20:22:10  <petertodd> xinxi_: I'm not clear what you mean by "how" - are you thinking that full nodes check blocks differently than miners?
5552016-07-13T20:22:30  <sipa> xinxi_: 2) we download blocks along the best headers from our peers, and when a block arrives, check it syntactically and that its transactions' hash matches what is in the claimed header we already know; and if so, then store them
5562016-07-13T20:23:06  <xinxi_> petertodd: should miners and full nodes use the same set of rules to check blocks?
5572016-07-13T20:23:10  <sipa> yes
5582016-07-13T20:23:21  <petertodd> xinxi_: if they didn't, how would we ever get miners to enforce rules that we wanted enforced?
5592016-07-13T20:23:44  <xinxi_> yeah, that's what I expected.
5602016-07-13T20:23:44  <xinxi_> what if a node see a newer version of blocks?
5612016-07-13T20:24:07  <sipa> xinxi_: 3) once we have all blocks along a chain whose total work exceeds that of our previous best chain, we look up its input in a database of unspent transaction outputs, validate the scripts, check resource limits, and various other things, ... and if succesful, remove the outputs it spent from the database, and add the outputs it created
5622016-07-13T20:24:12  <sipa> xinxi_: what does that mean?
5632016-07-13T20:24:33  <petertodd> xinxi_: then a soft-fork may have happened, adding rules to the existing set of rules, so the node should warn the user that the consensus rules enforced by it are no longer complete
5642016-07-13T20:25:00  <petertodd> xinxi_: of course, changing the version number for soft-forks is just a nicity - we can't force miners to do that
5652016-07-13T20:25:15  <petertodd> xinxi_: (modulo radical redesigns of how bitcoin works, like my client-side validation concepts)
5662016-07-13T20:25:18  <xinxi_> i mean, if a Bitcoin client of a very low version sees a block generated by the newest version of Bitcoin client, what will happen?
5672016-07-13T20:25:31  <sipa> xinxi_: it will just accept it
5682016-07-13T20:25:44  <sipa> xinxi_: all software down to 0.2.10 should still work
5692016-07-13T20:25:53  <sipa> (and that was because of a change in the p2p protocol)
5702016-07-13T20:26:06  <xinxi_> so 0.2.10 is a hard fork.
5712016-07-13T20:26:09  <petertodd> xinxi_: all that can happen is the node can warn the user that a soft-fork may have happened
5722016-07-13T20:26:14  <sipa> xinxi_: nope, it was a p2p change
5732016-07-13T20:26:29  <petertodd> xinxi_: not quite, that was a p2p change that can be easily worked around w/o changing the core consensus rules
5742016-07-13T20:26:30  <sipa> xinxi_: you could create a bridge between 0.2.9 and the current network
5752016-07-13T20:26:36  <xinxi_> based on your definition, soft forks should be backward compatible.
5762016-07-13T20:26:48  <sipa> xinxi_: validation of the blocks should be backward compatible
5772016-07-13T20:26:50  <sipa> and it is
5782016-07-13T20:27:09  <sipa> the 0.2.9 software just doesn't know how to talk to newer clients anymore
5792016-07-13T20:27:54  <xinxi_> OK. although the consensus protocol did not change, it's not backward compatible.
5802016-07-13T20:28:12  <xinxi_> how did you guys manage to upgrade from 0.2.9 to 0.2.10?
5812016-07-13T20:28:36  <petertodd> sipa: note that the change from longest-chain to most-work was in 0.3.3 https://github.com/bitcoin/bitcoin/commit/40cd0369419323f8d7385950e20342e998c994e1#diff-623e3fd6da1a45222eeec71496747b31R420
5822016-07-13T20:28:50  <sipa> xinxi_: long before my time
5832016-07-13T20:28:59  *** chris200_ has quit IRC
5842016-07-13T20:29:04  <morcos> is there really no better channel for this discussion?
5852016-07-13T20:29:07  <petertodd> xinxi_: it'd be very easy for people to continue to run nodes that had backwards compatbility with 0.2.9
5862016-07-13T20:29:20  <petertodd> morcos makes a good point...
5872016-07-13T20:29:23  <sipa> ack, let's move to #bitcoin
5882016-07-13T20:29:49  <xinxi_> so this is not even dev related?
5892016-07-13T20:29:56  <petertodd> xinxi_: it's covering really basic material
5902016-07-13T20:30:17  <xinxi_> but that's still dev. anyway, let's move.
5912016-07-13T20:30:20  *** shatoshi has joined #bitcoin-core-dev
5922016-07-13T20:30:22  <shatoshi> INCREDIBLE! Send me some bitcoin and I can turn it into MUCH more, using special blockchain accelerating technology. Your bitcoin wallet will explode! Guaranteed to work & vouched by the OPS. PM me to begin!
5932016-07-13T20:30:29  *** ChanServ sets mode: +o sipa
5942016-07-13T20:30:33  *** sipa sets mode: +b *!*legitimo@*.dhcp.kgpt.tn.charter.com
5952016-07-13T20:30:33  *** shatoshi was kicked by sipa (shatoshi)
5962016-07-13T20:30:37  <morcos> ha ha, you've been replaced by somethign even worse
5972016-07-13T20:30:39  <sdaftuar> thank you x 2
5982016-07-13T20:30:56  *** jtimon has quit IRC
5992016-07-13T20:37:58  *** Chris_Stewart_5 has joined #bitcoin-core-dev
6002016-07-13T20:38:55  *** frankenmint has joined #bitcoin-core-dev
6012016-07-13T20:44:56  *** frankenmint has quit IRC
6022016-07-13T20:48:27  *** spudowiar has joined #bitcoin-core-dev
6032016-07-13T20:51:18  *** harrymm has quit IRC
6042016-07-13T20:54:32  *** d_t has quit IRC
6052016-07-13T20:56:04  *** fengling_ has joined #bitcoin-core-dev
6062016-07-13T20:59:00  *** Chris_Stewart_5 has quit IRC
6072016-07-13T21:00:46  *** fengling_ has quit IRC
6082016-07-13T21:00:51  *** Sosumi has quit IRC
6092016-07-13T21:14:10  *** Chris_Stewart_5 has joined #bitcoin-core-dev
6102016-07-13T21:16:12  *** MiraclePerson has joined #bitcoin-core-dev
6112016-07-13T21:16:14  <MiraclePerson> SUPER!!!! Want more bitcoin? Send me some Bitcoin and I'll instantly send you MORE back. I use special block-chain exploding skills. Totally safe & secure. Vouched by all the OPS! Pm me to begin!
6122016-07-13T21:16:43  *** ChanServ sets mode: +o sipa
6132016-07-13T21:16:46  *** sipa sets mode: +b *!*parson@*.range86-171.btcentralplus.com
6142016-07-13T21:16:46  *** MiraclePerson was kicked by sipa (MiraclePerson)
6152016-07-13T21:37:34  *** droark has quit IRC
6162016-07-13T21:39:37  *** xinxi_ has quit IRC
6172016-07-13T21:40:10  *** xinxi has joined #bitcoin-core-dev
6182016-07-13T21:41:13  *** xinxi has quit IRC
6192016-07-13T21:46:30  *** veleiro has joined #bitcoin-core-dev
6202016-07-13T21:53:45  *** BlueMatt has joined #bitcoin-core-dev
6212016-07-13T21:57:08  *** fengling_ has joined #bitcoin-core-dev
6222016-07-13T22:01:39  <morcos> sipa: how do you feel about boost::lockfree::queue?
6232016-07-13T22:02:00  <morcos> as a proof of concept that solved the lock contention problem really well
6242016-07-13T22:02:06  *** fengling_ has quit IRC
6252016-07-13T22:02:42  <morcos> jeremyrubin pointed me in this direction, but we weren't sure if this is somethign we wanted in the code
6262016-07-13T22:02:48  <morcos> https://github.com/morcos/bitcoin/commit/3e8a10aa78a8a84a44eca9350178ecf7c308c21c
6272016-07-13T22:07:13  <sipa> morcos: interesting
6282016-07-13T22:07:58  *** belcher has joined #bitcoin-core-dev
6292016-07-13T22:07:58  <sipa> morcos: alternatively, there could be thread-local lists of entries to erase, that get applied once there are enough
6302016-07-13T22:08:19  <sipa> morcos: though i believe performance of thread locals may be platform dependent
6312016-07-13T22:08:48  <jeremyrubin> sipa: lockfree is also platform dependent unfortunately
6322016-07-13T22:08:56  <morcos> sipa: yeah i thought about that, but found the boost thing the easiest to code
6332016-07-13T22:09:06  <sipa> morcos: did you benchmark?
6342016-07-13T22:09:34  *** belcher is now known as beIcher
6352016-07-13T22:09:51  <morcos> yes, for validation its equally as fast as not erasing at all
6362016-07-13T22:09:56  <sipa> cool!
6372016-07-13T22:09:59  <morcos> so maybe on the order of a 40% speedup
6382016-07-13T22:10:04  <morcos> there is the added erase time
6392016-07-13T22:10:16  <sipa> that's much more than i would have expected
6402016-07-13T22:10:18  <morcos> but the way i coded it it happens the next time ATMP gets called
6412016-07-13T22:10:19  *** sipa sets mode: -o sipa
6422016-07-13T22:10:27  <morcos> well 16 cores, so thats a lot of lock contention
6432016-07-13T22:10:28  *** beIcher is now known as belcher
6442016-07-13T22:10:41  <sipa> ah, yes
6452016-07-13T22:10:57  <sipa> we should probably also look for ways to avoid sha256 there
6462016-07-13T22:11:09  <morcos> the added erase time is small though, maybe 1ms per block, but again happening at a much better time, and a small fraction of the savings
6472016-07-13T22:12:30  <jeremyrubin> sipa: we compared to using the iterator, but I think it was negligible
6482016-07-13T22:12:51  <jeremyrubin> but with the benefit of not having to check for iterator validity before erase
6492016-07-13T22:13:26  <jeremyrubin> (otherwise you would need to construct a set of iterators from the GC-list because concurrent Gets could add the same iter)
6502016-07-13T22:14:37  <jeremyrubin> sipa: wait could you clarify which you are referring to re: sha256? the uint256_t on the eviction list?
6512016-07-13T22:18:58  <sipa> jeremyrubin: the sigcache stores sha256's of the cached data
6522016-07-13T22:19:08  <sipa> sha256 is relatively slow
6532016-07-13T22:19:11  *** TomMc has quit IRC
6542016-07-13T22:20:39  <sipa> jeremyrubin: no, not talking about erasing... just the cache in general
6552016-07-13T22:20:57  <jeremyrubin> sipa: gotcha, should be fine to use siphash right?
6562016-07-13T22:21:33  <sipa> jeremyrubin: i'm not sure
6572016-07-13T22:25:39  <sipa> jeremyrubin: it's not unreasonable to have a cache size of a billion with certain hardware today
6582016-07-13T22:25:54  <jeremyrubin> sipa: if it is salted then it wouldn't be possible to predictably create a block with bad cache lookup performance?
6592016-07-13T22:26:37  <sipa> if siphash's security claims are correct, salting should make it unpredictablr
6602016-07-13T22:26:48  <sipa> but the 64-bit hash may not be enough
6612016-07-13T22:26:58  <sipa> perhaps a 128-bit hash works
6622016-07-13T22:31:47  <jeremyrubin> sipa: ttyl
6632016-07-13T22:43:28  *** murch has quit IRC
6642016-07-13T22:47:50  *** xinxi has joined #bitcoin-core-dev
6652016-07-13T22:48:10  <gmaxwell> it's quite easy to just create a lockfree queue using atomics. I think bluematt has one in his fibre codebase.
6662016-07-13T22:48:38  <gmaxwell> so we don't need to take the additional boost dep if its irritating, or if we do it shouldn't be too hard to get rid of.
6672016-07-13T22:49:18  <BlueMatt> https://github.com/bitcoinfibre/bitcoinfibre/blob/master/src/udpnet.cpp#L1175
6682016-07-13T22:49:24  <BlueMatt> actually lockless ring buffer, but close enough
6692016-07-13T22:50:50  <gmaxwell> sipa: we could have a faster wide cryptographic hash, but last I looked low hitrate seemed to be the bigger performance ipediment than the time spent hashing.
6702016-07-13T22:53:04  *** xinxi has quit IRC
6712016-07-13T22:54:54  <sipa> gmaxwell: a sparse hash set would also be much more space efficient
6722016-07-13T22:55:05  <sipa> (no malloc for each entry)
6732016-07-13T22:56:41  *** Guyver2 has quit IRC
6742016-07-13T22:58:37  *** fengling_ has joined #bitcoin-core-dev
6752016-07-13T23:02:19  *** shangzhou has joined #bitcoin-core-dev
6762016-07-13T23:02:29  <sipa> BlueMatt: interesting!
6772016-07-13T23:03:04  <sipa> a ring buffer would suffice here, i guess... if it overflows you can always grab the lock and apply the erases regardless
6782016-07-13T23:03:26  *** fengling_ has quit IRC
6792016-07-13T23:05:00  <shangzhou> sipa: the bitcoin.sipa.be data looks like not up to date
6802016-07-13T23:05:06  *** BonyM has quit IRC
6812016-07-13T23:07:12  *** belcher has quit IRC
6822016-07-13T23:07:27  <sipa> shangzhou: oh?
6832016-07-13T23:08:04  <sipa> oh, yds
6842016-07-13T23:08:06  <sipa> yes
6852016-07-13T23:08:30  *** Guest64966 has joined #bitcoin-core-dev
6862016-07-13T23:12:51  *** belcher has joined #bitcoin-core-dev
6872016-07-13T23:17:12  *** laurentmt has joined #bitcoin-core-dev
6882016-07-13T23:17:12  *** xinxi has joined #bitcoin-core-dev
6892016-07-13T23:17:14  *** laurentmt has quit IRC
6902016-07-13T23:20:01  *** BonyM has joined #bitcoin-core-dev
6912016-07-13T23:23:20  <BlueMatt> thoughts on decreasing transaction-serialization time?
6922016-07-13T23:23:28  <BlueMatt> its currently slow as fuck, even when serializing into a static buffer
6932016-07-13T23:23:41  <BlueMatt> caching it helps a lot, but eats a decent chunk of memory
6942016-07-13T23:24:16  *** xinxi has quit IRC
6952016-07-13T23:25:14  <BlueMatt> (and, mostly, caching tx serialization diverges fibre a decent chunk from core, which is annoying)
6962016-07-13T23:27:18  <cfields> BlueMatt: working on a few optims to tx right now. not sure how applicable though...
6972016-07-13T23:27:38  <cfields> BlueMatt: easy quick gain is lazy hashing
6982016-07-13T23:28:31  <BlueMatt> cfields: I said serialization, not deserialization :p
6992016-07-13T23:29:23  <sipa> BlueMatt: i think transaction data shoild be stored in a songle malloced buffer, with internal pointers
7002016-07-13T23:29:24  <cfields> BlueMatt: oh right, heh. i just assumed you were working on the other direction. nevermind then :)
7012016-07-13T23:29:33  <cfields> lazy hashing would be a quick way to slow you down, then :p
7022016-07-13T23:29:38  <BlueMatt> what we really should do is calculate the hash at the same time as we serialize/deserialize, since thats easy
7032016-07-13T23:29:40  <sipa> BlueMatt: that does require encapsulating all fields though
7042016-07-13T23:29:49  <BlueMatt> sipa: yes, that is a bit longer-term, though
7052016-07-13T23:29:53  <sipa> agree
7062016-07-13T23:29:54  <BlueMatt> but, yea, that would help a ton of things
7072016-07-13T23:30:46  <cfields> sipa: speaking of which, after profiling noticing how much time was spent in prevector, i've fixed up copying/moving to be much quicker. you opposed to dinking around in there?
7082016-07-13T23:34:57  <sipa> cfields: please do!
7092016-07-13T23:35:27  <sipa> cfields: actually, we should make it class safe...
7102016-07-13T23:35:41  <cfields> sipa: better, i added static_asserts :p
7112016-07-13T23:35:46  <sipa> and use std::move etc instead of realloc etx
7122016-07-13T23:35:53  <BlueMatt> cfields: ironically, no, FIBRE is mostly tx copy/serialize - it copies most tx from mempool into a CBlock, and serializes those txn to build the data-FEC chunks from which to calculate the missing chunks
7132016-07-13T23:36:07  <BlueMatt> it only ever has to deserialize txn that it missed from mempool (which is relatively few)
7142016-07-13T23:36:37  <sipa> BlueMatt: we could also add a means to serialize a block from a list of tx shared_pointers
7152016-07-13T23:36:38  <cfields> sipa: i started by special-casing it with enable_if<> and SFINAE, but I think we're better off just specializing for ~std::is_trivial<>
7162016-07-13T23:37:00  <gmaxwell> What fiber FECs is not the block directly, but the block with padding to make transactions line up on packet boundaries.
7172016-07-13T23:37:04  <sipa> BlueMatt: instead first needing a full deep copy of the tx into the block
7182016-07-13T23:37:04  <gmaxwell> er fibre
7192016-07-13T23:37:06  <BlueMatt> sipa: who said anything about serializing?
7202016-07-13T23:37:18  <BlueMatt> oh, what gmaxwell said
7212016-07-13T23:37:42  <BlueMatt> sipa: no, it just deep-copies into the CBlock that it hands to ProcessNewBlock
7222016-07-13T23:38:02  <sipa> "FIBRE is mostly tx copy/serializr" -- BlueMatt
7232016-07-13T23:38:28  <sipa> BlueMatt: ok, maybe we should just make CBlock have shared_ptr to transactions...
7242016-07-13T23:38:44  <BlueMatt> that would help a lot, but it doesnt solve my serialize-time problem
7252016-07-13T23:38:47  <BlueMatt> (which is like...all my time)
7262016-07-13T23:38:49  <sipa> ok
7272016-07-13T23:47:34  *** adamg has quit IRC
7282016-07-13T23:50:28  *** adamg has joined #bitcoin-core-dev
7292016-07-13T23:51:34  *** xinxi has joined #bitcoin-core-dev
7302016-07-13T23:54:48  *** kadoban has quit IRC
7312016-07-13T23:56:04  *** xinxi has quit IRC