12016-12-04T00:22:01 *** moli has joined #bitcoin-core-dev
22016-12-04T01:13:40 *** jtimon has joined #bitcoin-core-dev
32016-12-04T01:22:13 *** Atomicat has quit IRC
42016-12-04T01:30:28 *** jeremyrubin has quit IRC
52016-12-04T01:30:32 *** Madars has quit IRC
62016-12-04T01:30:36 *** jeremyrubin has joined #bitcoin-core-dev
72016-12-04T01:31:50 *** Madars has joined #bitcoin-core-dev
82016-12-04T01:41:17 *** Atomicat has joined #bitcoin-core-dev
92016-12-04T01:52:55 *** echonaut has quit IRC
102016-12-04T01:53:02 *** echonaut has joined #bitcoin-core-dev
112016-12-04T01:56:09 *** moli has quit IRC
122016-12-04T02:02:55 *** moli has joined #bitcoin-core-dev
132016-12-04T02:29:59 *** MrHodl has quit IRC
142016-12-04T02:30:44 *** fuc has joined #bitcoin-core-dev
152016-12-04T02:53:32 *** Giszmo has quit IRC
162016-12-04T03:22:41 *** Ylbam has quit IRC
172016-12-04T03:33:13 *** Cheeseo has quit IRC
182016-12-04T03:41:51 *** cryptapus has joined #bitcoin-core-dev
192016-12-04T03:41:51 *** cryptapus has joined #bitcoin-core-dev
202016-12-04T03:42:07 *** gribble has quit IRC
212016-12-04T03:42:40 *** blkdb has quit IRC
222016-12-04T03:42:42 *** cryptapus_afk has quit IRC
232016-12-04T03:42:49 *** blkdb has joined #bitcoin-core-dev
242016-12-04T03:47:14 *** wasi has quit IRC
252016-12-04T03:49:35 *** gribble has joined #bitcoin-core-dev
262016-12-04T04:07:01 *** Alopex has quit IRC
272016-12-04T04:08:06 *** Alopex has joined #bitcoin-core-dev
282016-12-04T04:09:12 *** Atomicat has quit IRC
292016-12-04T04:35:37 *** wasi has joined #bitcoin-core-dev
302016-12-04T04:44:21 *** Alopex has quit IRC
312016-12-04T04:45:26 *** Alopex has joined #bitcoin-core-dev
322016-12-04T04:51:24 *** Chris_Stewart_5 has quit IRC
332016-12-04T05:03:44 *** alpalp has quit IRC
342016-12-04T05:07:52 *** Chris_Stewart_5 has joined #bitcoin-core-dev
352016-12-04T05:09:22 *** ThomasV has joined #bitcoin-core-dev
362016-12-04T05:22:30 *** Chris_Stewart_5 has quit IRC
372016-12-04T05:29:05 *** fuc has quit IRC
382016-12-04T05:38:16 *** kadoban has quit IRC
392016-12-04T05:52:21 *** Alopex has quit IRC
402016-12-04T05:53:26 *** Alopex has joined #bitcoin-core-dev
412016-12-04T07:08:41 *** ThomasV has quit IRC
422016-12-04T07:38:06 *** Alopex has quit IRC
432016-12-04T07:39:11 *** Alopex has joined #bitcoin-core-dev
442016-12-04T07:43:51 *** ThomasV has joined #bitcoin-core-dev
452016-12-04T07:59:29 *** To7 has quit IRC
462016-12-04T08:08:04 *** jtimon has quit IRC
472016-12-04T08:08:26 *** Alopex has quit IRC
482016-12-04T08:09:32 *** Alopex has joined #bitcoin-core-dev
492016-12-04T08:25:00 <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #9273: Remove unused CDiskBlockPos* argument from ProcessNewBlock (master...2016-12-remove-unused-pnb-param) https://github.com/bitcoin/bitcoin/pull/9273
502016-12-04T08:35:02 *** Alopex has quit IRC
512016-12-04T08:36:07 *** Alopex has joined #bitcoin-core-dev
522016-12-04T08:37:16 *** Ylbam has joined #bitcoin-core-dev
532016-12-04T08:37:42 *** gabridome has joined #bitcoin-core-dev
542016-12-04T08:56:58 *** ThomasV has quit IRC
552016-12-04T08:59:44 *** gabridome has quit IRC
562016-12-04T09:00:38 *** gabridome has joined #bitcoin-core-dev
572016-12-04T09:00:46 *** gabridome has quit IRC
582016-12-04T09:05:48 *** ThomasV has joined #bitcoin-core-dev
592016-12-04T09:12:55 *** sanada has quit IRC
602016-12-04T09:31:06 *** Alopex has quit IRC
612016-12-04T09:32:11 *** Alopex has joined #bitcoin-core-dev
622016-12-04T10:15:47 *** Lightsword has quit IRC
632016-12-04T10:16:12 *** Lightsword has joined #bitcoin-core-dev
642016-12-04T10:31:42 *** moli has quit IRC
652016-12-04T10:32:06 *** Alopex has quit IRC
662016-12-04T10:33:12 *** Alopex has joined #bitcoin-core-dev
672016-12-04T10:36:47 *** laurentmt has joined #bitcoin-core-dev
682016-12-04T10:37:57 *** laurentmt has quit IRC
692016-12-04T10:45:01 *** Alopex has quit IRC
702016-12-04T10:46:06 *** Alopex has joined #bitcoin-core-dev
712016-12-04T11:01:49 *** AaronvanW has joined #bitcoin-core-dev
722016-12-04T11:33:40 *** ThomasV has quit IRC
732016-12-04T12:03:50 *** ThomasV has joined #bitcoin-core-dev
742016-12-04T12:41:17 *** alpalp has joined #bitcoin-core-dev
752016-12-04T12:41:17 *** alpalp has joined #bitcoin-core-dev
762016-12-04T13:12:44 *** Atomicat has joined #bitcoin-core-dev
772016-12-04T13:26:37 *** moli has joined #bitcoin-core-dev
782016-12-04T13:36:06 *** CubicEarth has quit IRC
792016-12-04T13:36:41 *** CubicEarth has joined #bitcoin-core-dev
802016-12-04T13:41:12 *** CubicEarth has quit IRC
812016-12-04T13:47:17 *** moli has quit IRC
822016-12-04T13:48:01 *** ThomasV has quit IRC
832016-12-04T13:48:13 *** Guyver2 has joined #bitcoin-core-dev
842016-12-04T13:55:32 *** moli has joined #bitcoin-core-dev
852016-12-04T14:14:33 *** moli has quit IRC
862016-12-04T14:17:17 *** afk11 has quit IRC
872016-12-04T14:25:15 *** alpalp has quit IRC
882016-12-04T14:26:09 *** afk11 has joined #bitcoin-core-dev
892016-12-04T14:26:09 *** afk11 has quit IRC
902016-12-04T14:26:09 *** afk11 has joined #bitcoin-core-dev
912016-12-04T14:30:23 *** alpalp has joined #bitcoin-core-dev
922016-12-04T14:31:00 *** murchandamus has quit IRC
932016-12-04T14:31:48 *** murchandamus has joined #bitcoin-core-dev
942016-12-04T14:41:35 *** murchandamus has quit IRC
952016-12-04T14:42:24 *** murchandamus has joined #bitcoin-core-dev
962016-12-04T14:45:33 *** JackH has quit IRC
972016-12-04T14:46:24 *** JackH has joined #bitcoin-core-dev
982016-12-04T14:48:36 *** moli has joined #bitcoin-core-dev
992016-12-04T14:54:46 *** alpalp has quit IRC
1002016-12-04T14:57:32 *** alpalp has joined #bitcoin-core-dev
1012016-12-04T15:05:36 *** murchandamus has quit IRC
1022016-12-04T15:06:08 *** murchandamus has joined #bitcoin-core-dev
1032016-12-04T15:08:11 *** murchandamus has quit IRC
1042016-12-04T15:09:17 *** moli has quit IRC
1052016-12-04T15:12:48 *** moli has joined #bitcoin-core-dev
1062016-12-04T15:18:21 *** murchandamus has joined #bitcoin-core-dev
1072016-12-04T15:47:57 *** kadoban has joined #bitcoin-core-dev
1082016-12-04T15:52:58 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1092016-12-04T16:09:26 *** Giszmo has joined #bitcoin-core-dev
1102016-12-04T16:09:42 *** Cheeseo has joined #bitcoin-core-dev
1112016-12-04T16:09:42 *** Cheeseo has joined #bitcoin-core-dev
1122016-12-04T16:09:43 *** Chris_Stewart_5 has quit IRC
1132016-12-04T16:15:02 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1142016-12-04T16:16:58 *** Chris_Stewart_5 has quit IRC
1152016-12-04T16:17:18 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1162016-12-04T16:39:47 *** Sosumi has joined #bitcoin-core-dev
1172016-12-04T16:48:57 *** wvr has joined #bitcoin-core-dev
1182016-12-04T16:49:03 *** protomar has joined #bitcoin-core-dev
1192016-12-04T16:59:30 *** wvr has quit IRC
1202016-12-04T17:16:37 *** RoyceX has joined #bitcoin-core-dev
1212016-12-04T17:17:26 *** Giszmo has quit IRC
1222016-12-04T17:19:33 *** Cheeseo has quit IRC
1232016-12-04T17:45:28 *** Chris_Stewart_5 has quit IRC
1242016-12-04T17:47:57 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1252016-12-04T17:56:13 *** jtimon has joined #bitcoin-core-dev
1262016-12-04T17:57:26 *** laurentmt has joined #bitcoin-core-dev
1272016-12-04T17:57:42 *** laurentmt has quit IRC
1282016-12-04T18:21:13 *** RoyceX has quit IRC
1292016-12-04T18:24:44 *** MarcoFalke has joined #bitcoin-core-dev
1302016-12-04T18:24:55 <bitcoin-git> [bitcoin] MarcoFalke opened pull request #9274: [qa] pruning: Use cached utxo set to run faster (master...Mf1612-qaPruningCacheUtxos) https://github.com/bitcoin/bitcoin/pull/9274
1312016-12-04T18:25:51 *** JackH has quit IRC
1322016-12-04T18:26:56 *** alpalp has quit IRC
1332016-12-04T18:40:50 *** RoyceX has joined #bitcoin-core-dev
1342016-12-04T18:40:50 *** RoyceX has joined #bitcoin-core-dev
1352016-12-04T18:52:23 *** LeMiner2 has joined #bitcoin-core-dev
1362016-12-04T18:54:51 *** LeMiner has quit IRC
1372016-12-04T18:54:52 *** LeMiner2 is now known as LeMiner
1382016-12-04T19:02:57 *** Guyver2__ has joined #bitcoin-core-dev
1392016-12-04T19:05:26 *** Guyver2 has quit IRC
1402016-12-04T19:05:31 *** Guyver2__ is now known as Guyver2
1412016-12-04T19:07:50 *** Guyver2__ has joined #bitcoin-core-dev
1422016-12-04T19:11:26 *** Guyver2 has quit IRC
1432016-12-04T19:11:33 *** Guyver2__ is now known as Guyver2
1442016-12-04T19:11:42 *** Sosumi has quit IRC
1452016-12-04T19:13:58 *** Sosumi has joined #bitcoin-core-dev
1462016-12-04T19:27:55 *** JackH has joined #bitcoin-core-dev
1472016-12-04T19:32:02 *** wvr has joined #bitcoin-core-dev
1482016-12-04T19:32:19 *** alpalp has joined #bitcoin-core-dev
1492016-12-04T19:32:19 *** alpalp has joined #bitcoin-core-dev
1502016-12-04T19:38:15 *** CubicEarth has joined #bitcoin-core-dev
1512016-12-04T19:58:22 <bitcoin-git> [bitcoin] jonasschnelli pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/2efcfa5acfac...4d955fc5824b
1522016-12-04T19:58:23 <bitcoin-git> bitcoin/master 827d9a3 Wladimir J. van der Laan: qt: Replace NetworkToggleStatusBarControl with generic ClickableLabel...
1532016-12-04T19:58:23 <bitcoin-git> bitcoin/master 042f9fa Wladimir J. van der Laan: qt: Show progress overlay when clicking spinner icon...
1542016-12-04T19:58:24 <bitcoin-git> bitcoin/master 4d955fc Jonas Schnelli: Merge #9218: qt: Show progress overlay when clicking spinner icon...
1552016-12-04T19:58:37 <bitcoin-git> [bitcoin] jonasschnelli closed pull request #9218: qt: Show progress overlay when clicking spinner icon (master...2016_11_overlay_when_clicking_sync_icon) https://github.com/bitcoin/bitcoin/pull/9218
1562016-12-04T20:18:38 <BlueMatt> jtimon: re: libconsensus: have you thought more about it after the discussions in milan? I'm still not a huge fan of exposing a ton of functions and prefer a simple refactor that a) makes it so that everything ProcessNewBlock calls isnt aware of disk positioning/etc and b) move all that stuff into a class which owns mapBlockIndex, chainActive, etc
1572016-12-04T20:18:42 <BlueMatt> jtimon: see https://github.com/TheBlueMatt/bitcoin/commits/2016-12-cconsensus
1582016-12-04T20:20:20 <jtimon> I keep thinking the same thing more or less
1592016-12-04T20:21:25 <jtimon> if you don't want to expose verifyHeader or verifyTx...I know some people want verifyHeader for SPV wallets, but not so sure about verifyTx
1602016-12-04T20:21:27 *** bobbytux__ has joined #bitcoin-core-dev
1612016-12-04T20:21:48 <jtimon> regarding process block, as discussed, I think we should expose verifyBlock instead
1622016-12-04T20:22:33 *** bobbytux_ has quit IRC
1632016-12-04T20:22:42 <BlueMatt> jtimon: why, though? if you expose ProcessNewBlock{,Headers} someone can implement a full spv client and full client all in the same codepaths, without exposing low-level partial-verification functions
1642016-12-04T20:23:38 <BlueMatt> (or, excuse me, without requiring users do their own utxo-management shit, while still giving them access to query the utxo state)
1652016-12-04T20:23:39 <jtimon> how can an SPV node calle ProcessNewBlock?
1662016-12-04T20:23:48 <BlueMatt> ProcessNewBlockHeaders
1672016-12-04T20:23:51 <jtimon> why would they want to depend on levelDB ?
1682016-12-04T20:23:59 <BlueMatt> nono, that part should be abstracted out
1692016-12-04T20:24:01 <BlueMatt> as discussed
1702016-12-04T20:24:33 <BlueMatt> eg see the blockstores in bitcoinj: https://github.com/bitcoinj/bitcoinj/tree/master/core/src/main/java/org/bitcoinj/store
1712016-12-04T20:25:18 <jtimon> oh, right, you were ok with abstracting storage, just wanted libconsensus to manage reorgs and all the rest in an abstracted way
1722016-12-04T20:25:35 <BlueMatt> yea, best to handle /all/ the complexity imo
1732016-12-04T20:25:39 <jtimon> isn't ProcessNewBlockHeaders an intermediate function?
1742016-12-04T20:25:54 <BlueMatt> no? it takes headers and adds them and selects the best headers chain it has seen
1752016-12-04T20:27:06 <jtimon> then I'm not sure what you mean by "intermediate" in this context
1762016-12-04T20:27:57 <BlueMatt> ehh, I apologize, I forgot that your definition of verifyBlock also means checking all the txes connect, ie essentially TestBlockValidity
1772016-12-04T20:28:02 <BlueMatt> instead of existing CheckBlock-type functions
1782016-12-04T20:28:22 <jtimon> well, no really testblockvelidity since that call connectBlock too
1792016-12-04T20:28:42 <BlueMatt> hmm?
1802016-12-04T20:28:44 <jtimon> not really
1812016-12-04T20:28:58 <BlueMatt> huh
1822016-12-04T20:29:36 <jtimon> my proposed verifyBlock just tells you whether a block is valid or not, but it doesn't do anything with it
1832016-12-04T20:29:47 *** MykelSIlver has joined #bitcoin-core-dev
1842016-12-04T20:29:59 <BlueMatt> indeed, like TestBlockValidity?
1852016-12-04T20:31:55 <jtimon> mhmm, connect block updates transactions, updates undo info and updates the view of the best block, no?
1862016-12-04T20:32:12 <BlueMatt> but TestBlockValidity throws away all that info afterwards
1872016-12-04T20:32:21 <jtimon> oh, I see
1882016-12-04T20:32:25 <BlueMatt> and you have to keep an updated-utxo-state as you connect the block to test validity
1892016-12-04T20:32:42 <jtimon> well, then yes, but without generating info than then throws away
1902016-12-04T20:33:18 <BlueMatt> you cand check a block without doing that
1912016-12-04T20:33:24 <jtimon> yeah, you need to update at least a view of the utxo
1922016-12-04T20:33:50 <jtimon> right, that was my idea for verifyBlock
1932016-12-04T20:39:52 <BlueMatt> jtimon: ok, so back to my original question...what do you think of not exposing that and actually having a utxo/disk-storage-abstraced full consensus layer
1942016-12-04T20:39:52 <BlueMatt> ?
1952016-12-04T20:40:56 <jtimon> well, where I have te strongest opinion is in abstracting from storage
1962016-12-04T20:41:10 <jtimon> and agree on that
1972016-12-04T20:41:12 <BlueMatt> ok...?
1982016-12-04T20:41:25 <BlueMatt> oh, well sure, I dont think anyone wants libconsensus to require a certain storage layer
1992016-12-04T20:41:55 <jtimon> what would be the API? precessnewblock and processnewheaders?
2002016-12-04T20:43:12 <BlueMatt> jtimon: essentially, yes, PNB, PNBH, some initialization api (see https://github.com/TheBlueMatt/bitcoin/blob/2016-12-cconsensus/src/validation.cpp#L80) and a utxo-state-accessor/loadblock-accessor api
2012016-12-04T20:43:17 <jtimon> well, I do think some people want to depend on a concrete storage , but since it's neither of us, let's leave that aside
2022016-12-04T20:44:01 <jtimon> what about caches? any API for those?
2032016-12-04T20:44:03 <BlueMatt> well once the bitcoin core codebase "eats its own dogfood", as it were, we can take the storage layer we have and expose that optionally
2042016-12-04T20:44:24 <jtimon> right
2052016-12-04T20:44:35 <BlueMatt> no, caches are up to the client application, ie the client application gives libbitcoinconsensus pcoinsTip
2062016-12-04T20:44:52 <BlueMatt> similarly, in the future, maybe we can expose CCoinsViewCache or whatever
2072016-12-04T20:44:59 <BlueMatt> but for v1, its uneccessary
2082016-12-04T20:45:18 <jtimon> mhmm, I mean for example the versionbits cache
2092016-12-04T20:45:40 *** droark has quit IRC
2102016-12-04T20:45:45 <jtimon> is that hidden to the caller or exposed somehow?
2112016-12-04T20:46:07 <BlueMatt> since its currently not exposed, hidden and present
2122016-12-04T20:46:27 <BlueMatt> dont expose anything unless its required, and for v1, dont worry too much about performance or memory usage
2132016-12-04T20:47:04 <sipa> the reason for not exposing would exactly be performance and memory usage, i think
2142016-12-04T20:47:23 <sipa> it's pretty hard to design a stable api that can be used efficiently for those things
2152016-12-04T20:47:45 <sipa> so i'd say make all state (caches, indexes, ...) by default part of the abstracted state
2162016-12-04T20:48:01 <sipa> and perhaps later introduce a way for the caller to install callbacks to override it, and provide their own
2172016-12-04T20:48:05 <jtimon> well, the api may change if consensus rules change
2182016-12-04T20:52:48 <jtimon> BlueMatt: I highly doubt some implementors (say libbitcoin) will use processNewBlock, then again I also doubted they were going to use verifyBlock directly either, I was hoping maybe verifyHeader, verifyTx + GetConsensusFlags
2192016-12-04T20:53:39 <BlueMatt> sipa: yea, except for v1 it doesnt matter, really - just dont expose anything and later, if we remove caches to make the library lighter, expose hooks to replace the caches
2202016-12-04T20:53:55 <bitcoin-git> [bitcoin] morcos opened pull request #9276: Some minor testing cleanups (master...rpccleanup) https://github.com/bitcoin/bitcoin/pull/9276
2212016-12-04T20:53:57 <jtimon> leaving that aside, I would like libconsensus to be the specification of the consensus rules for a given block to be valid, it seems that with your approach it would be more than that
2222016-12-04T20:54:33 <jtimon> but bitcoin core is currently way more than that, so...
2232016-12-04T20:54:53 <BlueMatt> jtimon: I mean the way any sane full node is structured there is an abstracted utxo/blocks-on-disk storage and something which checks everything and itnerfaces with that state
2242016-12-04T20:55:07 <jtimon> I also hope you don't intend to put half of the server package in the consensus package
2252016-12-04T20:55:27 <BlueMatt> jtimon: I see no reason why libbitcoinconsensus shouldnt repalce everything in between because if the goal is to reduce the risk of consensus incompatibility, we should replace everything we can
2262016-12-04T20:55:52 <BlueMatt> jtimon: I dont care what it weighs, only what it does....cutting down the weight is for later versions
2272016-12-04T20:55:53 <BlueMatt> :p
2282016-12-04T20:56:40 * BlueMatt -> brunch
2292016-12-04T20:57:55 <jtimon> because some implementors (like libbitcoin) value more the ability to have their own concurrency model (ie without locks) than reducing risks of consensus incompatibility beyond verifyScript (which they also use only optionally)
2302016-12-04T20:58:22 * jtimon should have brinner soon too
2312016-12-04T21:00:34 <jtimon> I agree, that we can leave "cutting weight" for later (like we're leaving moving CFeeRate for later for very long), my point is that with your approach, even after you're done, it will be much bigger than "the specification for whether a block is valid or not"
2322016-12-04T21:01:17 <jtimon> it would be more like the specification of "how to follow and store the longest valid chain"
2332016-12-04T21:01:54 <jtimon> then someone will want prunning too...
2342016-12-04T21:02:22 *** droark has joined #bitcoin-core-dev
2352016-12-04T21:04:02 <sipa> jtimon: yes, it's the specification for whether a blockchain is valid or not
2362016-12-04T21:04:22 <jtimon> sipa: processNewBlock does much more than that
2372016-12-04T21:04:40 <sipa> well, yes, due to necessity
2382016-12-04T21:04:50 <sipa> we can't track the validity of every single chain
2392016-12-04T21:04:59 <sipa> so we only have a utxo set at a tip
2402016-12-04T21:05:47 <jtimon> that's we, maybe other callers want to do things differently
2412016-12-04T21:07:09 <jtimon> then maybe someone else asks for "sharded prunning" or something else, and soon you end up again with a lot of things that don't have anything to do with consensus rules
2422016-12-04T21:07:38 <sipa> yes, so we can later provide ways to plug in your own state storage (utxo/caches/indexes/...)
2432016-12-04T21:08:13 <jtimon> sipa: oh, BlueMatt's approach already gives you an abstraction for the utxo and chain index storage
2442016-12-04T21:08:49 <sipa> oj
2452016-12-04T21:08:51 <sipa> ok
2462016-12-04T21:09:36 *** MarcoFalke has left #bitcoin-core-dev
2472016-12-04T21:09:57 <jtimon> but my example is that if it supports prunning, that needs to be exposed somehow
2482016-12-04T21:10:05 *** Sosumi has quit IRC
2492016-12-04T21:10:09 <sipa> heh?
2502016-12-04T21:10:20 <sipa> how does block storage have anything to do with this
2512016-12-04T21:10:36 <jtimon> isn't prunning deleting blocks you are storing?
2522016-12-04T21:10:49 <sipa> yes, but blocks wouldn't be stored by libconsensus
2532016-12-04T21:11:03 <jtimon> they would with BlueMatt's approach
2542016-12-04T21:11:15 <sipa> hmm, that seems strange
2552016-12-04T21:11:32 <jtimon> if I undesrtood him correctly
2562016-12-04T21:11:47 <sipa> i would have an api where you plug in a way libconsensus to ask you for a block
2572016-12-04T21:12:06 <sipa> if yiu're abstracting utxo storage, i don't see why you wouldn't abstract nlock storage
2582016-12-04T21:12:16 <jtimon> that's why he needs something like https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/store/BlockStore.java#L39
2592016-12-04T21:12:18 <sipa> which is even less consensus critical
2602016-12-04T21:12:41 <sipa> but that's the user that provides the blockstore
2612016-12-04T21:12:44 <sipa> not the library
2622016-12-04T21:14:15 <jtimon> right, the library provides an API and callbacks the implementor for stored data in both cases, but in bluematt's case, it also updates the storage, it doesn't use it in a read only way
2632016-12-04T21:14:40 <jtimon> maybe prunning was a bad example, I agree you don't need to put it in libconsensus even in that case
2642016-12-04T21:15:59 <gmaxwell> I think abstracting things like all this storage maybe getting into the realm of imaginary users... is there really many users that want to make no change (not even instrumentation) to consensus but is very interested at implementing their own UTXO database?
2652016-12-04T21:16:00 <jtimon> in my case, it takes a block and says valid or validation error x and does nothing more than that
2662016-12-04T21:16:31 <sipa> jtimon: but you need to reimplement a lot of complicated logic in your application for it
2672016-12-04T21:16:36 <sipa> utxo caching is hard
2682016-12-04T21:16:48 <sipa> block index tracking with skiplists is not trivial
2692016-12-04T21:16:54 <jtimon> gmaxwell: I agree that part of the problem here is not having a clear idea of the users or their preferences
2702016-12-04T21:16:59 <gmaxwell> I thought the goal for libconsensus is that so that when someone wants to make their own custom wallet thing they wouldn't need to reimplment any of the consensus logic, just to create their own application logic.
2712016-12-04T21:17:20 <jtimon> gmaxwell: but it seems that your objection would apply to both my approach and matt's
2722016-12-04T21:17:25 <sipa> i really don't care about users at this point - we don't have any, and have no easy way to build somwthing others could use
2732016-12-04T21:17:39 <sipa> i care about separating consensus from nonconsensus logic
2742016-12-04T21:18:00 <gmaxwell> if so that would really mean all the chain management would really be inside the library.
2752016-12-04T21:18:19 <gmaxwell> sipa: okay so for the purpose of consensus safty of changes, the caches and utxo database are all consensus critical--
2762016-12-04T21:18:35 <sipa> gmaxwell: fair enough, but so is the c++ standard library
2772016-12-04T21:18:42 <jtimon> gmaxwell: right, matt puts the chain management inside, but also provides an abstraction for storage
2782016-12-04T21:19:19 <sipa> i think having block storage outside the library is perfectly fine, as consensus logic really does not need blocks except for reorgs
2792016-12-04T21:19:30 <gmaxwell> jtimon: Providing an abstraction for IO though is different than something that expects the user to implement a complex database.
2802016-12-04T21:20:01 <sipa> even utxo storage i can be convinced of, i think
2812016-12-04T21:20:21 <sipa> but block chain tracking (cblockindex/mapblockindex/...) should really be inside
2822016-12-04T21:20:38 <sipa> it's highly dependent on how we manage chains
2832016-12-04T21:20:55 <jtimon> providing a wrapper that includes the same storage as bitcoin core, as an extended library should be easy
2842016-12-04T21:21:20 <gmaxwell> so long as the requirements are very clear and simple... where no bitcoin specific understanding is required... sure. though doing that without breaking efficiency might be hard.
2852016-12-04T21:21:29 <sipa> i don't want every cblockindex access to go through wrapper calls
2862016-12-04T21:22:00 <gmaxwell> e.g. there are ordering requirements for fixation of block data vs utxo data to disk.
2872016-12-04T21:22:02 <jtimon> yep, efficiency is the main concern with my approach I think
2882016-12-04T21:23:38 <jtimon> btw, as always comments on https://github.com/bitcoin/bitcoin/pull/8493 welcomed
2892016-12-04T21:25:10 <jtimon> this kind of thing (for the abstracted storage) is disruptive though https://github.com/bitcoin/bitcoin/pull/8493/commits/0a1ff996c5db5265742e3e90f3690a0b8d9cf045
2902016-12-04T21:26:51 <jtimon> and the way I implemented it there, is also inneficient for bitcoin core
2912016-12-04T21:36:35 <jtimon> anyway, there's 2 separated topics here
2922016-12-04T21:36:47 <jtimon> whether to abstract and expose storage
2932016-12-04T21:37:22 <jtimon> and whether or not to manage reorgs and other things beyond "check whether a given block is valid or not"
2942016-12-04T21:39:41 <jtimon> my answers are yes and no, matt's is yes and yes. If gmaxwell's is no and yes, and sipa's are yes and yes, we have all the options represented :p
2952016-12-04T21:43:02 <jtimon> s/sipa's are no and no/
2962016-12-04T21:45:35 <sipa> the first thing we should probably do is separate CBlockIndex into block storage data (file, offset, undo, ...), and chain data (block index, work, validation)
2972016-12-04T21:46:01 <sipa> that would be required if we'd want to abstract block storage out from a library
2982016-12-04T21:46:08 <sipa> but i thibk it shoukd hapoen regardless
2992016-12-04T21:46:31 <jtimon> I don't need to do that in my approach
3002016-12-04T21:46:34 <sipa> the block storage data doesn't even need to be in memory permnantly
3012016-12-04T21:46:50 <sipa> i understand
3022016-12-04T21:47:08 <sipa> but my point is that we should do that, even if there never is a library at all
3032016-12-04T21:48:10 *** Guyver2__ has joined #bitcoin-core-dev
3042016-12-04T21:48:32 <jtimon> chain.o and coins.o are already separated, I'm not entirely sure I understand what you mean
3052016-12-04T21:50:33 <sipa> CBlockIndex stores things about where blocks are on disk
3062016-12-04T21:50:44 <jtimon> oh, I see
3072016-12-04T21:50:44 <sipa> (which has nothing to do with consensus)
3082016-12-04T21:50:56 <sipa> and things like validation flags and chainwork
3092016-12-04T21:51:00 <sipa> and the block header
3102016-12-04T21:51:02 *** Guyver2 has quit IRC
3112016-12-04T21:51:03 <sipa> which do
3122016-12-04T21:51:04 *** Guyver2__ is now known as Guyver2
3132016-12-04T21:53:45 <jtimon> regarding the validation flags, I replaced #7779 with https://github.com/bitcoin/bitcoin/pull/9271 (although got errors in unittests because we're already relying on the consensus flags being coupled for testing it seems)
3142016-12-04T21:53:47 <gribble> https://github.com/bitcoin/bitcoin/issues/7779 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
3152016-12-04T21:56:24 <jtimon> see https://github.com/bitcoin/bitcoin/pull/9271/commits/7a90173b839e682932426f7d6ac91d2df88e2cfb
3162016-12-04T22:02:36 *** RoyceX has quit IRC
3172016-12-04T22:04:52 <bitcoin-git> [bitcoin] s-matthew-english opened pull request #9277: clear typo "skipepd" changed to "skipped" (master...patch-12) https://github.com/bitcoin/bitcoin/pull/9277
3182016-12-04T22:11:50 <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9277: clear typo "skipepd" changed to "skipped" (master...patch-12) https://github.com/bitcoin/bitcoin/pull/9277
3192016-12-04T22:19:32 *** RoyceX has joined #bitcoin-core-dev
3202016-12-04T22:20:49 *** protomar has quit IRC
3212016-12-04T22:26:01 *** Guyver2 has quit IRC
3222016-12-04T22:38:10 *** bitdev01 has joined #bitcoin-core-dev
3232016-12-04T22:45:43 *** bitdev01 has quit IRC
3242016-12-04T22:56:35 <sdaftuar> gmaxwell: regarding bumpfee... i'd been advising mrbandrews to try to simplify what the rpc command is doing as much as possible, with the hope of getting a helpful utility in place tht we could layer more advanced behavior on top of
3252016-12-04T22:57:05 <sdaftuar> gmaxwell: so in particular, i wasn't aware of a robust way to identify the change output, so i suggested for this rpc to have the user supply it
3262016-12-04T22:57:14 <sdaftuar> is there a robust way to identify the change output?
3272016-12-04T22:58:10 <sipa> yes, IsMine() and not in address book
3282016-12-04T23:03:20 <sdaftuar> sorry i'm so unfamiliar with this... i guess there's an IsChange() function, with a TODO saying that this might not work with multisig behavior in the future. is that not a concern now?
3292016-12-04T23:09:11 <gmaxwell> sdaftuar: We really should be storing the data in the wallet, but just using IsMine would be better than asking for it.
3302016-12-04T23:09:41 <gmaxwell> (I think specifying it is basically something that would only be useful to the same people who could use createrawtransaction)
3312016-12-04T23:10:25 <gmaxwell> Simplifying it is good, but we can't make it a hazard to users-- which is why I was saying it needed to somehow assure that the outputs will not get spent while unconfirmed.
3322016-12-04T23:11:14 <gmaxwell> Simiarly, just diminishing the change and never adding inputs will produce transactions which won't relay (dust). though I suppose instead of being able to add inputs it could just limit itself to places where that wouldn't be the case.
3332016-12-04T23:11:43 <sdaftuar> gmaxwell: i haven't looked at the PR lately but i thought it was going to do the right thing with dust.
3342016-12-04T23:11:54 <gmaxwell> Because of the need to restrict dependant transaction chains I thought a 'bump everything' might be simpler.
3352016-12-04T23:12:20 <gmaxwell> sdaftuar: it didn't look like it could add inputs to me but equally possible that I was confused.
3362016-12-04T23:13:02 <sdaftuar> gmaxwell: oh it won't currently add inputs, as i recall, but i think if it would reduce the change output to less than dust, then it would get dropped altogether
3372016-12-04T23:13:13 <sdaftuar> (that's what i recall some discussion of, anyway)
3382016-12-04T23:13:33 <sdaftuar> are you saying though that you don't think it should allow feebumping a tx that has descendants?
3392016-12-04T23:13:54 <sdaftuar> i thought that was part of the point of this RPC, to get that right.
3402016-12-04T23:17:01 *** Chris_Stewart_5 has quit IRC
3412016-12-04T23:18:24 <gmaxwell> imagine you have txn A, and B that spends A. Then you bump A creating a replaceme A' that replaces A/B. Now you make nother spend that spends the output of A' ... you really need to draft two versions of it one that spends B and one that spends A'... and switch to relaying the other depending on what confirms.
3422016-12-04T23:18:38 <gmaxwell> I thought that was too advanced, which is why I was suggesting prohibiting descendants.
3432016-12-04T23:18:48 <sdaftuar> ahh right
3442016-12-04T23:18:58 <sdaftuar> wow, i totally missed that
3452016-12-04T23:19:50 <gmaxwell> GAit has implemented bumping in the green address wallet and I think it deals with all this stuff.
3462016-12-04T23:20:00 <sdaftuar> PRs welcome? :)
3472016-12-04T23:20:39 <gmaxwell> even my suggestion of making unspendable for one confirm is a bit risky, since it couple be reorged out. :(
3482016-12-04T23:20:43 <sdaftuar> so i think bumpunconfirmed is a good idea, but i worry about old transactions that have long since been double spent
3492016-12-04T23:20:52 <gmaxwell> esp since there is non-trivial amounts of hashpower that won't take replacements.
3502016-12-04T23:20:54 <sdaftuar> or rather, distinguishing those from other unconfirmed transactions
3512016-12-04T23:21:06 <gmaxwell> well we track conflicted transactions.
3522016-12-04T23:21:11 <sdaftuar> but not perfectly
3532016-12-04T23:21:34 <sdaftuar> we can't track conflicted transactions where the conflict chain is broken with tx's not in our wallet
3542016-12-04T23:22:33 <gmaxwell> Point. Though it would be sufficient for this functionality to only work on transactions where the closure of unconfirmed parents are all in the wallet, I guess.
3552016-12-04T23:24:20 *** droark has quit IRC
3562016-12-04T23:25:38 <gmaxwell> I wonder how often someone has multiple unconfirmeds pending and doesn't want to just bump them all. Usually it will save them total fees if they do so.
3572016-12-04T23:25:54 <gmaxwell> ignoring crazy corner cases like some being doublespent.
3582016-12-04T23:26:57 <sdaftuar> yeah i think it does seem like a useful feature
3592016-12-04T23:27:39 <gmaxwell> it just seemed simpler to me than getting all the corner cases right.
3602016-12-04T23:28:14 <sdaftuar> well i think that has a bunch of corner cases too. you probably want to make sure you conflict with anything that might possibly confirm, so you have to include even abandoned transactions right?
3612016-12-04T23:28:26 <sdaftuar> (maybe even 1-confirm tx's!)
3622016-12-04T23:28:41 <sdaftuar> er 1-conflicted
3632016-12-04T23:29:11 <gmaxwell> You need to conflict with anything whos outputs you include, so that you don't potentially double-pay.
3642016-12-04T23:30:07 <sdaftuar> i guess that raises a good UI point
3652016-12-04T23:30:31 <sdaftuar> probably the user would want to see all the outputs and transactions being bumped, and ack that?
3662016-12-04T23:30:47 <sdaftuar> in case of the really old wallet tx that the user forgot about problem
3672016-12-04T23:31:59 <sdaftuar> i dunno, i guess i just assume that heavy wallet users that have been running for a while must have lots of those, maybe that's not a good assumption
3682016-12-04T23:32:06 <sdaftuar> ?
3692016-12-04T23:32:57 <gmaxwell> I think it would work like this: take the set of unconfirmed transactions which are AllFromMe (whatever the test that excludes coinjoins is called). Then eliminate all transactions whos parents aren't either all AllFromMe or 6 confirmed. Then gather up all their outputs, eliminate all the IsMine outputs, and generate a new transaction which conflicts each transaction in the set, and pays suffi
3702016-12-04T23:33:03 <gmaxwell> ciently more fees. Then its change output (if any) should be marked so that it will not be treated as IsFromMe for spending purposes. (UI wise we should encourage somewhat overpaying due to this last point)
3712016-12-04T23:33:07 *** Chris_Stewart_5 has joined #bitcoin-core-dev
3722016-12-04T23:34:00 <gmaxwell> I don't think most users have a lot of long lost transactions. I hope not. :) since they'll pretty likely to lose coins if they ditch the wallet thinking the balance was zero.
3732016-12-04T23:34:06 <sdaftuar> that seems like it ought to work.
3742016-12-04T23:34:18 <sdaftuar> do we have a way right now to do the mark-change-as-unspendable thing?
3752016-12-04T23:34:37 <gmaxwell> We have something that doesn't do what we want here.
3762016-12-04T23:34:47 <gmaxwell> (lockunspent)
3772016-12-04T23:35:35 <gmaxwell> For this, I think we need to add a boolen to the wallet txn and have isfromme check that.
3782016-12-04T23:36:07 <gmaxwell> Later we could have more advanced bumping logic that does the make-multiple-transaction-versions.
3792016-12-04T23:37:40 *** MarcoFalke has joined #bitcoin-core-dev
3802016-12-04T23:37:56 <gmaxwell> sdaftuar: if you really think that there are many users with txn stuck never confirmed... then we really need to add a 'Pending Payments' -- sum of confirmed outputs which are spent by unconfirmed transactions -- in the UI/rpc. So that people don't discard wallets as empty when they're really not.
3812016-12-04T23:38:26 <sdaftuar> gmaxwell: actually, maybe i am missing something. with your proposed algorithm, what would happen if you bumpunconfirmed twice?
3822016-12-04T23:39:23 <gmaxwell> You would ignore the existing bump (it would fail the AllFrom me in step 1) and make a new bump of all the other unconfirmed transactions in your wallet.
3832016-12-04T23:40:15 <sdaftuar> i didn't follow why it would fail the AllFromMe step, but if you managed to construct a new tx that didn't conflict with the older bumped fee one, you're screwed right?
3842016-12-04T23:41:25 <gmaxwell> It would fail the all from me, because I propose the change output of the bumb be flagged specifically so that it does. ... so that under no condition wall the wallet spend it until it is 6 confirmed. Specfically to avoid having to deal with things spending it then having the originals confirm.
3852016-12-04T23:42:05 <sdaftuar> isn't IsAllFromMe purely a function of the tx inputs?
3862016-12-04T23:42:27 <gmaxwell> The new tx doesn't need to conflict with it, so long as it doesn't pay to any of its outputs, which it won't because it wasn't included in the set of available txn.
3872016-12-04T23:43:02 <sdaftuar> i think i am pretty confused right now, why don't i spend another day thinking about the wallet and then rejoin this conversation
3882016-12-04T23:43:33 <gmaxwell> It's a function of the input's outputs. Is every input an output of mine.
3892016-12-04T23:44:22 <gmaxwell> sdaftuar: this stuff makes my head hurt too. I really want to get some kind of bumping in ASAP, it's hard to come up with a minimal feature...
3902016-12-04T23:45:49 <sdaftuar> one last try before i call it a night -- let's say i have tx1, tx2, tx3 -- each one is IsAllFromMe. i bump and create tx4, which conflicts with each, and pays all their outputs, and the change output of tx4 is specially flagged or whatever to be unspendable until it confirms.
3912016-12-04T23:45:59 <sdaftuar> wont't tx4 still be IsAllFromMe?
3922016-12-04T23:46:59 <sdaftuar> (actually i have to run, back later)
3932016-12-04T23:47:01 <gmaxwell> Right, okay -- I should have said IsMine and IsAllFromMe -- the flagging should make it not IsMine.
3942016-12-04T23:47:05 <gmaxwell> sdaftuar: ttyl
3952016-12-04T23:50:30 *** Chris_Stewart_5 has quit IRC
3962016-12-04T23:50:51 *** droark has joined #bitcoin-core-dev
3972016-12-04T23:58:44 <BlueMatt> gmaxwell: ok, I missed a bunch of scrollback, but hell yes...everyone wants the utxo db in their own format so that they can do other queries against it
3982016-12-04T23:58:58 <BlueMatt> jtimon: no, I absolutely think we should abstract out block storage
3992016-12-04T23:59:16 <BlueMatt> jtimon: hell, the branch I sent you did that for ConnectBlock/DisconnectBlock in the first commit
4002016-12-04T23:59:45 <jtimon> right, I said "yes and yes" for you