12016-12-05T00:00:04  <jtimon> but let me check that commit
  22016-12-05T00:00:04  <BlueMatt> oh, what? you told sipa that the "it updates the storage"
  32016-12-05T00:00:53  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  42016-12-05T00:02:11  <jtimon> I mean it updates what you store, through the api (ie the library handles reorgs and updates things)
  52016-12-05T00:02:46  <BlueMatt> well of course, you tell the library about a block and it connects what it can, possibly asking the library owner for blocks that it previously saw but (ofc) didnt store
  62016-12-05T00:03:00  <BlueMatt> I dont see how else you'd possibly do it
  72016-12-05T00:03:33  <BlueMatt> gmaxwell: see, eg, the bitcoinj stuff, where all the contributions to "full mode" since it was added have been to add it to database X, so that people can put it in their own db
  82016-12-05T00:03:35  <jtimon> see the backlog for when I say for people answer the 4 possible combinations to 2 yes/no questions
  92016-12-05T00:05:38  <jtimon> you and me agree on abstracting storage
 102016-12-05T00:05:57  <jtimon> I mean, if I understood everyone's position correctly
 112016-12-05T00:06:38  <gmaxwell> BlueMatt: without using a highly efficient format like ours, I'm dubious that the system can stay in sync without insane hardware.
 122016-12-05T00:06:46  <jtimon> the other question is only check that a block is valid or also handle reorgs, update tables etc
 132016-12-05T00:06:52  <BlueMatt> gmaxwell: thats not our problem, thats theirs
 142016-12-05T00:06:58  <BlueMatt> gmaxwell: hell, they can use leveldb if they want
 152016-12-05T00:07:02  <jtimon> ie verifyBlock vs processBlock
 162016-12-05T00:07:20  <BlueMatt> gmaxwell: and, if we prefer, its also easy to add eg a single flag like "use X MB of utxo cache, because I dont want to implement that myself"
 172016-12-05T00:07:23  <gmaxwell> BlueMatt: not just about leveldb or not, but the compressed ccoins representation is worthless for queries.
 182016-12-05T00:08:07  *** Chris_Stewart_5 has quit IRC
 192016-12-05T00:08:27  <BlueMatt> gmaxwell: the compressed ccoins thing doesnt matter all that much if you're talking about an actually-high-performance db on high-performance hardware....if folks need their shit in their own db and are willing to pay $$$$$ for it to run, more power to them
 202016-12-05T00:08:33  <gmaxwell> As far as 'their problem' goes, we shouldn't waste our resources (or code base clarity, or performance) supporting functionality that won't be useful to anyone (or to anyone beyond a couple centeralized API services).
 212016-12-05T00:08:45  <jtimon> we can add a wrapper with our own implementation of the interfaces, beyond that, right, it is their problem
 222016-12-05T00:08:55  <BlueMatt> gmaxwell: ok, then what is the point of libconsensus at all?
 232016-12-05T00:09:19  <sipa> BlueMatt: i don't care about libconsensus. i care about abstracting consensus logic out
 242016-12-05T00:09:20  <BlueMatt> and whats your answer to folks like btcd/the new javascript one, who do have dbs that are performant enough to stay in sync
 252016-12-05T00:09:20  <gmaxwell> The way it was pitched to me is so that people could make wallets and other similar applications without having to reimplement consensus logic.
 262016-12-05T00:09:34  <jtimon> right
 272016-12-05T00:09:38  *** RoyceX has quit IRC
 282016-12-05T00:09:53  <BlueMatt> gmaxwell: sure, but that doesnt mean we have to handle db logic ourselves
 292016-12-05T00:09:58  <sipa> well, the hardest part of that is already available: we expose script verification
 302016-12-05T00:10:02  <jtimon> how is that incompatible with allowing them to chose their database implementation?
 312016-12-05T00:10:14  <BlueMatt> sipa: I dont think thats the only hard part, really
 322016-12-05T00:10:16  <sipa> but i also don't think it's very hard to abstract utxo storage out... so if there is a use case, sure
 332016-12-05T00:10:44  <BlueMatt> sipa: indeed, abstracting out utxo/block storage is also abstracting consensus logic out of other crap
 342016-12-05T00:10:45  <sipa> maybe it is.
 352016-12-05T00:11:11  <sipa> something like changing to per-output rather than per-tx utxo model would be impossible with a stable utxo storage abstraction
 362016-12-05T00:11:38  <jtimon> I assume the use cases come from the fact that they want to reuse that database for some of their logic somehow
 372016-12-05T00:11:44  <sipa> s/impossible/inefficient or complicated/
 382016-12-05T00:11:47  <gmaxwell> I'm not opposed to it being abstractable-- but I don't see how this is related to that goal-- it's the opposite of it, the blockchain storage and utxo set is consensus and may even be quite normative in its behavior (e.g. if we have a commited utxo set of some kind), and if it trashes performance or code clarity then it's not a good move.
 392016-12-05T00:12:15  *** MarcoFalke has left #bitcoin-core-dev
 402016-12-05T00:12:52  <BlueMatt> sipa: oh? if you query per-utxo then per-tx can be hidden on the backend and could still be pretty performant....indeed, the other way around doesnt really work
 412016-12-05T00:12:53  <gmaxwell> and for an example of the kind of complexity it creates: if you think you can just query a utxo database it means we can't have writeback caching internally.
 422016-12-05T00:13:03  <BlueMatt> gmaxwell: sure, no one wants to do anything that ends up introducing performance regressions
 432016-12-05T00:13:41  <sipa> abstracting storage is more to avoid dependencies rather than it being reusable
 442016-12-05T00:13:55  <gmaxwell> I think callers that want a database probably don't want to replace the database used for consensus-- what they want is a node that maintains an external database for their application.
 452016-12-05T00:14:09  <gmaxwell> Which is probably not the same thing due to consistency requirements.
 462016-12-05T00:15:19  <jtimon> gmaxwell: it shouldn't trash performance or code clarity, I agree
 472016-12-05T00:15:32  <BlueMatt> gmaxwell: it might be worse performance, but syncing after every block after ibd (ie having a flag to sync) isnt all that hard, either....
 482016-12-05T00:15:42  <BlueMatt> eg ProcessNewBlock(block); SyncToDB();
 492016-12-05T00:16:10  <sipa> BlueMatt: but for a full node, you probably don't want to sync after every block
 502016-12-05T00:16:11  <BlueMatt> gmaxwell: and its easy to do that without introducing our own performance regressions....just dont call Sync after every block....
 512016-12-05T00:16:21  <BlueMatt> sipa: depending on your application, maybe you do
 522016-12-05T00:16:30  <jtimon> sipa: once abstracted out, changing the interface in certain ways may be painful, correct
 532016-12-05T00:16:33  <BlueMatt> at least after IBD
 542016-12-05T00:16:37  <gmaxwell> Not writing out the utxo set constantly is critical for performance. Leveldb is slow.
 552016-12-05T00:16:47  <sipa> BlueMatt: but it'd be using our block validation code, which has known performance characteristics
 562016-12-05T00:17:04  <sipa> BlueMatt: and if you don't care about that, you wouldn't be using libconsensus at all
 572016-12-05T00:17:14  <BlueMatt> sipa: as long as we dont drop the cache when we flush (like we do now, which we already need to fix), I dont see a performance issue there?
 582016-12-05T00:17:30  <sipa> BlueMatt: fair enough
 592016-12-05T00:17:42  <sipa> BlueMatt: that's a good point - but the current code doesn't do that
 602016-12-05T00:17:52  <gmaxwell> I can't imagine an application which needs to muck around storing the utxo database in an application format which wouldn't be equally or better served by block processing callback that maintains an external database that isn't used for validation.
 612016-12-05T00:17:57  <BlueMatt> suresure, but these are all minor issues that are trivial to fix
 622016-12-05T00:17:57  <jtimon> avoiding dependencies is also a great gain, although I think we should have a version depending on levelDB and our own implementation too
 632016-12-05T00:18:00  <sipa> again, i'm not against abstracting things out
 642016-12-05T00:18:17  <gmaxwell> Widespread application visiblity into the actual utxo database would be pretty toxic for commited utxo or stxo improvements.
 652016-12-05T00:18:22  <BlueMatt> gmaxwell: duplicate databases? people who run shit on modern services where you literally have no local persistent storage?
 662016-12-05T00:18:25  <sipa> i very much feel that utxo storage is one of the things that is abstractable
 672016-12-05T00:18:45  <sipa> but that doesn't mean it's necessarily useful for sharing that information with other purposes
 682016-12-05T00:18:49  <sipa> it also doesn't mean it's not
 692016-12-05T00:19:15  <jtimon> yeah, I assume one use case could be having everything in memory
 702016-12-05T00:19:25  <gmaxwell> BlueMatt: basically any standard database approach would have horrible performance to the point that it would only be useable on very high end hardware.  Having two copies of the utxo set would hardly be a consideration there, our copy is only about 2GB data.
 712016-12-05T00:20:00  <BlueMatt> gmaxwell: I seriously dont believe that...maybe it takes 10 seconds or 30 seconds to sync a block to the db, but so what? you just dont flush all the time during IBD and then wait it out
 722016-12-05T00:20:19  <BlueMatt> (and, yes, I get that in most dbs it actually will take 30 seconds, but it wont take much longer than that)
 732016-12-05T00:20:25  <BlueMatt> or a minute with segwit blocks
 742016-12-05T00:20:29  * BlueMatt -> out
 752016-12-05T00:20:46  <gmaxwell> BlueMatt: see also the electrum channel with people complaing about their servers falling multiple blocks behind.
 762016-12-05T00:20:57  <gmaxwell> and two minute updates.
 772016-12-05T00:21:36  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 782016-12-05T00:22:05  <BlueMatt> gmaxwell: ok...? that doesnt mean its impossible to build a db that can store the utxo set with reasonable performance, even if not blazing fast performance?
 792016-12-05T00:22:22  <gmaxwell> I just feel that the purpose of these changes is no longer clear. Expecting the user to implement complex interfaces with bitcoin specific and consensus critical behavior is at odds with what I understood to be the stated goal.
 802016-12-05T00:22:31  * BlueMatt doesnt see the problem with it taking a minute for your db to sync the utxo set....if you're just a merchant and its slow, so what?
 812016-12-05T00:22:38  <jtimon> it is also possible to build something faster than our solution, isn't it?
 822016-12-05T00:23:02  <BlueMatt> gmaxwell: I highly disagree that selecting a sane DB (ie not using some external SQL thing) is "implementing complex interfaces"
 832016-12-05T00:23:08  <gmaxwell> And if maintaining a database for a block explorer is a goal-- then it can be done in a better way then also trying to use that database for consensus... just run it in parallel, the resource overhead will be moderate.
 842016-12-05T00:23:27  <BlueMatt> and I dont expect every exchange to do so, I'd expect folks like btcd/bitcoinj/javascript thinggy to do it
 852016-12-05T00:23:29  <BlueMatt> and people to use it
 862016-12-05T00:23:44  <BlueMatt> gmaxwell: go look at the bitcoinj users...people actually use its full validation shitshow so that they can do exactly this
 872016-12-05T00:23:47  <jtimon> gmaxwell: we don't need to expect the user to reimplement it, we should provide our own implementation to the interfaces
 882016-12-05T00:23:47  <BlueMatt> and its slow, but they dont care
 892016-12-05T00:23:54  <BlueMatt> (and the interface is trivial)
 902016-12-05T00:24:11  <gmaxwell> BlueMatt: Complex interfaces is that you need to actually pass the data as fields and not opaque blobs.   And what happens when the set of data needed for consensus changes (as sequence locktiming did).
 912016-12-05T00:24:15  <BlueMatt> its like 4 functions and only a single non-trivial requirement for which there are unit tests (the utxo-replacement-thing)
 922016-12-05T00:24:39  <jtimon> again, you keep discussing the slow case, what if I'm faster than our implementation?
 932016-12-05T00:24:54  <gmaxwell> BlueMatt: that usage doesn't even need full node support at all, run your bitcoinj behind a full node, have it log blocks to a database. Hurray.
 942016-12-05T00:25:09  <BlueMatt> gmaxwell: yes, except people dont do that
 952016-12-05T00:25:20  <sipa> maybe we should write a simple daemon that maintains the utxo set in SQL
 962016-12-05T00:25:22  <BlueMatt> people prefer to run full validation shit in btcd or bitcoinj, despite knowingly putting themselves at risk
 972016-12-05T00:25:27  <sipa> (without validation)
 982016-12-05T00:25:28  <jtimon> or are we discarding that as a possibility?
 992016-12-05T00:25:31  <gmaxwell> sipa: that is what I was saying.
1002016-12-05T00:25:38  <gmaxwell> BlueMatt: they don't know they're putting themselves at risk.
1012016-12-05T00:26:12  <BlueMatt> gmaxwell: ok, well either way I dont see an alternative great solution here? most developers do want a library that handles all the background validation shit for them
1022016-12-05T00:26:19  <BlueMatt> and as long as such things exist, they will use them
1032016-12-05T00:26:20  <gmaxwell> jtimon: because there exists nothing faster currently, or else we'd be using it.
1042016-12-05T00:26:24  <BlueMatt> proxy-nodes be damned
1052016-12-05T00:26:42  <gmaxwell> BlueMatt: the alternative is to have support for maintaining a database synced with the blockchain-- that doesn't mean inserting things into the consensus logic.
1062016-12-05T00:27:15  <gmaxwell> It just means having a simple set of hooks that run post tip change and update the database.
1072016-12-05T00:27:39  <BlueMatt> gmaxwell: you missed my previous point about how lots of "modern" developers run shit on services with ~no persistent storage
1082016-12-05T00:27:48  <BlueMatt> anyway, I actually need to run, have an apt to keep
1092016-12-05T00:27:58  <sipa> run, BlueMatt, run
1102016-12-05T00:28:10  <jtimon> there's no storage solution optimized for having everything in memory that outperforms leveldb or offers some other advantage?
1112016-12-05T00:28:11  <sipa> apt-keep BlueMatt
1122016-12-05T00:28:24  <sipa> jtimon: -dbcache=8000
1132016-12-05T00:28:31  <gmaxwell> We already have that built in.
1142016-12-05T00:28:31  <sipa> jtimon: leveldb won't be used at all
1152016-12-05T00:29:11  <jtimon> sipa: right, I know, but I doubt there's nothing better than levle db with unbounded cache
1162016-12-05T00:29:23  <jtimon> sipa: levledb is what we're using, no?
1172016-12-05T00:29:25  *** e4xit has quit IRC
1182016-12-05T00:29:38  <gmaxwell> jtimon: that doesn't use leveldb (except to persist across restarts)
1192016-12-05T00:29:54  <jtimon> oh
1202016-12-05T00:30:09  <jtimon> I see, I didn't know that, thanks
1212016-12-05T00:32:09  <jtimon> even if we only expose the version with our implementation, I think it would be good to abstract consensus storage even for bitcoin core rewardless of libconsensus users
1222016-12-05T00:32:48  <gmaxwell> our storage is already abstracted.
1232016-12-05T00:34:16  <jtimon> I really think not trashing preformance for our own implementation should be the priority at first, if the interface needs to go through a few iterations not to trash other implementations too, I think that's fine since we will be offering the "don't use storage abstractions" version too anyway
1242016-12-05T00:35:30  *** alpalp has quit IRC
1252016-12-05T00:35:35  <jtimon> yes, our storage is abstracted in more ways than we would want to expose in an storage-independent libconsensus C API
1262016-12-05T00:37:15  <jtimon> I mean, since I'm in favor of exposing both, one storage independent and one that is not, I'm fine with starting with the one that is not, I'm just more interested technically in the other one
1272016-12-05T00:39:27  <jtimon> can we talk a little bit about the other question?
1282016-12-05T00:42:55  <jtimon> ie whether libconsensus should fully validate a given block, or also accept it and update the state, manage reorgs, etc
1292016-12-05T00:44:11  <jtimon> again I'm ok with offering both but I'm more interested in the smaller one
1302016-12-05T00:44:46  *** droark has quit IRC
1312016-12-05T00:47:42  <jtimon> or at least that's what I have been imagining all this time, I wasn't counting reorgs or updating the tip as part of the validation
1322016-12-05T00:50:52  *** e4xit has joined #bitcoin-core-dev
1332016-12-05T01:01:06  *** alpalp has joined #bitcoin-core-dev
1342016-12-05T01:01:06  *** alpalp has joined #bitcoin-core-dev
1352016-12-05T01:05:30  <morcos> gmaxwell: sdaftuar: to return to the fee bumping, in suhas' example, where you first try to bump tx1,2,3 with tx4, and then you try again.
1362016-12-05T01:05:54  <morcos> i can see how you could invent a way to prevent tx4 from getting bumped, but how are you stopping 1,2,3 from being bumped again?
1372016-12-05T01:06:30  <morcos> or more generally, lets say you manually had tried to bump and tx1 and tx1a both are trying to pay the same guy (maybe you were smart enought to conflict, maybe not)
1382016-12-05T01:06:33  <gmaxwell> They should be marked abandoned when the bump is created.
1392016-12-05T01:06:40  <morcos> how do you stop a bumpunconfirmed from bumping both
1402016-12-05T01:06:47  <gmaxwell> (but even if they weren't they're conflicted at that point)
1412016-12-05T01:06:54  <morcos> they aren't conflicted
1422016-12-05T01:07:03  <gmaxwell> by tx4, which is in your mempool.
1432016-12-05T01:07:06  <morcos> conflicted means the conflicting tx is in the block
1442016-12-05T01:07:23  <morcos> a conflicting tx in the mempool is something you are not even aware of
1452016-12-05T01:08:03  <morcos> which brings me to the second point i wanted to make, aobut what you said about people abandoning wallets they think are empty
1462016-12-05T01:08:14  <morcos> thats a fantastic point, that i wish had been made a while ago
1472016-12-05T01:08:31  <gmaxwell> you're right, I'd been assuming it would be conflicted without thinking carefully what that test actually means right now.
1482016-12-05T01:08:53  <gmaxwell> I know for sure people do abandon wallets that they think are empty (or even 'almost empty')
1492016-12-05T01:09:09  <morcos> before we made the change to the confliction logic (for 0.12 ?)  then if your spend was not in your mempool, it was considered conflicted (regardless of whether it was by an in-block, in-mempool tx, or nothing)
1502016-12-05T01:09:23  <morcos> so it would be kind of rare for you to think you were out of money but not
1512016-12-05T01:09:31  <morcos> but now, for sure that might happen
1522016-12-05T01:10:01  <morcos> now if you issue a tx that never makes it into a block or for some reason can't ever make it into your own mempool
1532016-12-05T01:10:13  <gmaxwell> I seem to vaguely recall that something else would still prevent us from doublespending the inputs on txn that weren't in the mempool, even then.
1542016-12-05T01:10:15  <morcos> regardless of any conflicting txs, its uses up your balance until you abandon it
1552016-12-05T01:10:25  <morcos> i don't think so
1562016-12-05T01:11:04  <morcos> it seems like we need another notion of balance (or maybe 2 more)
1572016-12-05T01:11:53  <morcos> potential balance (which is not reduced by non-in chain (6 deep?) spends) and maybe even a pending receive balance (although i guess that hasn't been a problem historically, that hasnt' changed)
1582016-12-05T01:15:16  <gmaxwell> Yes, agreed. I worry about the use 'balance' for a number which will go down without user interaction. :)
1592016-12-05T01:15:20  *** CubicEarth has quit IRC
1602016-12-05T01:15:56  *** CubicEarth has joined #bitcoin-core-dev
1612016-12-05T01:16:13  <gmaxwell> More like "pending outgoing payments: outbound payments which are not N confirms deep yet" and "pending incoming payments".
1622016-12-05T01:16:26  <Taek> Would it be enough to have a [confirmed balance] and an [unconfirmed diff]?
1632016-12-05T01:16:44  <morcos> if this is really the only use case, it would be easy enough to make a rpc call that just give you a report on how "empty" your wallet is
1642016-12-05T01:17:14  <morcos> Taek: we already have that..  , oh yeah, nm, thats the second thing i was talking abou tthen, the pending received balance
1652016-12-05T01:17:22  <morcos> getunconfirmedbalance
1662016-12-05T01:17:49  <Taek> if I'm following correctly, the worry is about coins that become unconfirmed due to e.g. change outputs?
1672016-12-05T01:18:15  *** droark has joined #bitcoin-core-dev
1682016-12-05T01:18:16  <morcos> i think the primary worry is that if you spend some coins , but your spend never makes it into a block
1692016-12-05T01:18:27  <morcos> your wallet still deducts that spend from your balance
1702016-12-05T01:18:29  <morcos> forever
1712016-12-05T01:19:04  <morcos> until you manually mark it as abandoned (which is sort of an advanced feature, that we don't often recommend)
1722016-12-05T01:19:38  <Taek> technically some adversary could un-abandon any transaction that hasn't been double-spent
1732016-12-05T01:19:57  *** CubicEarth has quit IRC
1742016-12-05T01:20:04  <morcos> exactly why its an advanced manual feature
1752016-12-05T01:20:05  <Taek> imo your confirmed balance should not change until the tx is in the blockchain
1762016-12-05T01:20:54  <Taek> and the confirmed balance should be what is presented to the user as the primary balance
1772016-12-05T01:21:03  <morcos> perhaps confirmed balance is the wrong word for what getbalance returns...  it returns your spendable balance..  which certainly should be decremented for spends that haven't yet made it into the blockchain
1782016-12-05T01:21:16  <morcos> and i think thats what people expect to see when they ask their balance
1792016-12-05T01:22:53  <Taek> I don't think it's safe enough to show the user just one number.
1802016-12-05T01:23:14  <Taek> simply because the whole unconfirmed uncertainty is unseparatable from the blockchain way of doing thigns
1812016-12-05T01:23:30  <Taek> (well, lightning doesn't really have this issue)
1822016-12-05T01:25:40  <BlueMatt> gmaxwell: thinking about it more, the way we'd probably do it is, initially (ie v1) you make the libconsensus consumer provide a k-v store api, and we use that the same way we use leveldb, and then add functionality to parse the blobs we provide the k-v store into things like scriptPubKeys later
1832016-12-05T01:25:41  <jtimon> perhaphs both confirmed and spendable balances should be shown?
1842016-12-05T01:25:52  <BlueMatt> this provides functionality, without breaking our ability to change the format to add new things
1852016-12-05T01:26:56  <jtimon> I understand k-v is key-value kind of storage
1862016-12-05T01:27:30  <jtimon> BlueMatt: if so, what would the values be? C structs ?
1872016-12-05T01:27:47  <BlueMatt> the values are binary blobs that libconsensus provides which the user does not have any visibility into
1882016-12-05T01:28:02  <BlueMatt> (in our case its the serialization of CCoins or whatever with our compression stuff)
1892016-12-05T01:28:19  <sipa> BlueMatt: yup, that's what i imagined
1902016-12-05T01:28:32  <BlueMatt> if the user wants to know whats inside, there is some api which can parse it into a c struct or whatever
1912016-12-05T01:28:49  <sipa> a batch key-value write operation, and a key read operation
1922016-12-05T01:29:05  <jtimon> kind of like https://github.com/bitcoin/bitcoin/pull/8493/commits/ad3ac37387b231378573f4996c59467247fccd44 ?
1932016-12-05T01:30:15  <jtimon> ^ for the "binary blobs the caller doesn't know about"
1942016-12-05T01:30:59  <sipa> jtimon: i don't see such a thing at all
1952016-12-05T01:31:22  <sipa> that commit is about bitcoinconsensus_create_consensus_parameters
1962016-12-05T01:32:06  <jtimon> well, yeah, sorry, this are void pointers the other it's just data like in the tx for current verifyScript
1972016-12-05T01:33:06  <jtimon> in this case we would use the serialize lib to interpret and produce the "blobs", correct?
1982016-12-05T01:33:55  <sipa> if there needs to be a way to view the utxo set, i'd just provide a separate api for that
1992016-12-05T01:34:00  <sipa> not a parser for the database
2002016-12-05T01:34:09  <sipa> and not in the first stage
2012016-12-05T01:36:01  <jtimon> I am extremely interested in hearing in what other people's next steps, or more feedback on my own proposed next step
2022016-12-05T01:38:30  <jtimon> sipa: I'm still not sure if you prefer to expose verifyBlock or processBlock
2032016-12-05T01:39:41  <sipa> jtimon: i don't think we can do so right now anyway, without having a way to abstract state out
2042016-12-05T01:40:12  <sipa> imho the first step is just continuing refactoring so that consensus logic and other things become better separated internally
2052016-12-05T01:40:24  <sipa> and not focus on exposing things
2062016-12-05T01:40:49  <sipa> but others may disagree - i think wumpus prefers first having a clear idea of what will be exposed
2072016-12-05T01:41:29  <sipa> even a verifyBlock will need a way to pass in the utxo set and the block index
2082016-12-05T01:42:27  <sipa> the only difference is that a processBlock doesn't need a way to update set utxo set, and doesn't need to be able to request other blocks in case of a reorg
2092016-12-05T01:42:41  *** Ylbam has quit IRC
2102016-12-05T01:44:07  <jtimon> althought I tend to agree, I feel that's very vague and doesn't help on clarifying priorities, thinking of the next thing to expose, I think, helps clarify what the goal of the refactors should be and where are we supposed to be moving towards to
2112016-12-05T01:44:36  <sipa> you know my opinion - i don't care about exposing anything at all at this point, so i'm the wrong person to ask
2122016-12-05T01:44:47  <jtimon> yes verifyBlock would need an interface to access data from the utxo
2132016-12-05T01:44:51  <sipa> i think we have harder problems to solve before exposing even comes into question
2142016-12-05T01:45:22  *** Giszmo has joined #bitcoin-core-dev
2152016-12-05T01:45:52  <jtimon> dcousens: proposal was to pass all required data explicitly for the block you were validating
2162016-12-05T01:46:15  <sipa> essentially, i think we should first introduce clean abstractions between certain modules inside bitcoin core, in such a way that it's effectively bitcoind using a consensus library already, without it being exposed
2172016-12-05T01:46:31  <sipa> when it's good enough for us to use, we can think about exposing it to others
2182016-12-05T01:47:06  <sipa> (but again, others may see things differently)
2192016-12-05T01:48:01  <jtimon> right, and I think the module that should be a priority to cleanly separate is the part of the code that is required to fully validate whether a block is valid or not from everything else
2202016-12-05T01:48:54  <sipa> but that's very tightly coupled with validation of the whole chain, through CBlockIndex
2212016-12-05T01:49:26  <jtimon> right, it basically depends on chain.o and coins.o
2222016-12-05T01:50:09  <sipa> you can't validate a block without knowing its CBlockIndex
2232016-12-05T01:50:24  <jtimon> more specifically on two existing classes on them, an API for that is not hard
2242016-12-05T01:50:26  <sipa> so i'm not sure whether "single block validation" is a useful abstraction on its own
2252016-12-05T01:50:47  <sipa> transaction validation may be useful
2262016-12-05T01:51:36  <jtimon> CBlockIndex is the storage interface I abstract (or abstract from its own exsiting abstraction) in 8493
2272016-12-05T01:51:48  <sipa> i don't want an abstraction for CBlockIndex
2282016-12-05T01:52:24  <jtimon> my proposed next steps are single header validation or single tx validation
2292016-12-05T01:52:33  <jtimon> but without policy checks
2302016-12-05T01:53:34  <sipa> as i said, i don't think we should focus on exposing interfaces now, but on separating modules
2312016-12-05T01:53:49  <sipa> and i think separating block validation from chain validation is hard
2322016-12-05T01:54:03  <BlueMatt> jtimon: I'm with sipa here - The Main Split was the first of many steps that make sense on their own to abstract out consensus and non-consensus code
2332016-12-05T01:54:19  <BlueMatt> jtimon: the few commits I sent you earlier form the very tiny beginning of what I think are the next steps
2342016-12-05T01:54:48  <jtimon> separating network things was absolutely brilliant, thanks again
2352016-12-05T01:54:50  <BlueMatt> jtimon: ie having a state object internally which keeps chainstate in it and calls out to things for disk access and has ProcessNewBlock as a member function
2362016-12-05T01:55:09  <sipa> BlueMatt: chainstate include mapBlockIndex?
2372016-12-05T01:55:25  <BlueMatt> and its ~no code changes, just some function splitting and putting ClassName:: in front of them
2382016-12-05T01:55:34  <BlueMatt> sipa: yes, mapBlockIndex and chainActive and related variables
2392016-12-05T01:55:51  <BlueMatt> sipa: but calling out for ReadBlockFromDisk, and pcoinsTip is just a pointer that is passed to it
2402016-12-05T01:56:03  <sipa> BlueMatt: got it
2412016-12-05T01:56:08  <sipa> seems like an easy first step
2422016-12-05T01:56:34  <jtimon> sipa: if you don't care about exposing, that's fine, let's talk about dependencies, I want the consensus module to fully verify a single tx and a single header and a single block without depending on coins.o or chain.o
2432016-12-05T01:56:34  <BlueMatt> yea, should be pretty clean...I dont have time to do it for the next week or two...do you want to take it up jtimon?
2442016-12-05T01:56:50  <BlueMatt> I also want to work on splitting up net_processing more so that we can multithread ProcessMessages
2452016-12-05T01:57:07  <BlueMatt> jtimon: I dont see how thats possibel?
2462016-12-05T01:57:19  <sipa> yes, i think those are possible
2472016-12-05T01:57:22  <jtimon> I don't know what you want to do, how can I pick it up?
2482016-12-05T01:57:23  <sipa> (don't
2492016-12-05T01:57:36  <BlueMatt> jtimon: literally the point of "fully validating" a tx is to validate it against a CCoins-holding UTXO db
2502016-12-05T01:57:51  <BlueMatt> jtimon: did you look at the top commit on the branch I sent you?
2512016-12-05T01:57:53  <sipa> efficiency of validation is highly dependent on low-level access to coins and chain
2522016-12-05T01:57:57  <jtimon> sigh, I thought I had proved it was possible repeated times...
2532016-12-05T01:58:05  <BlueMatt> jtimon: https://github.com/TheBlueMatt/bitcoin/commit/54c967e73a1d1abba8f426fce0f14c5eeaf277e6
2542016-12-05T01:58:08  <sipa> it's possible if you introduce abstractions everywhere
2552016-12-05T01:58:23  <jtimon> is it possible to fully validate a header without depending on chain.o?
2562016-12-05T01:58:33  <sipa> no
2572016-12-05T01:58:59  <sipa> (unless you abstract it out, of course)
2582016-12-05T01:59:02  <jtimon> then what's happening in 8493
2592016-12-05T01:59:06  <jtimon> right
2602016-12-05T01:59:32  <sipa> but i think such abstraction are both a performance issue and an unnecessary code complication
2612016-12-05T02:00:22  <jtimon> well, more than half of 8493 is purely for demonstrating the exposed api and without benchmarking of any kind
2622016-12-05T02:03:50  <jtimon> my goal was to separate the code the verify a full block, depending on chain.o and coins.o (but only on those related to storage) [maybe put it all in the consensus folder? or wait for later?] but not putting it in the consensus module until you want to expose more and abstract it from chain and coins
2632016-12-05T02:05:42  <jtimon> anyway, I'm happy reviewing any related refactors, please ping me
2642016-12-05T02:07:01  <jtimon> sipa: does the GetConsensusFlag make any sense to you at a first glance? at least more than the previous version?
2652016-12-05T02:08:22  <jtimon> without exposing anything, just as a refactor (note that calling GetConsensusFlag inside ContextualCheckBlock is painful performance-wise)
2662016-12-05T02:27:28  *** netzin has joined #bitcoin-core-dev
2672016-12-05T02:32:08  *** netzin has quit IRC
2682016-12-05T02:38:54  *** wasi has quit IRC
2692016-12-05T02:43:22  *** To7 has joined #bitcoin-core-dev
2702016-12-05T02:46:57  *** wasi has joined #bitcoin-core-dev
2712016-12-05T03:24:20  *** Guest13131 has quit IRC
2722016-12-05T03:30:12  *** Guest13131 has joined #bitcoin-core-dev
2732016-12-05T03:36:46  *** molz has joined #bitcoin-core-dev
2742016-12-05T03:38:06  *** Alopex has quit IRC
2752016-12-05T03:39:11  *** Alopex has joined #bitcoin-core-dev
2762016-12-05T03:39:57  *** moli has quit IRC
2772016-12-05T03:49:02  *** moli has joined #bitcoin-core-dev
2782016-12-05T03:50:12  *** molz has quit IRC
2792016-12-05T03:52:37  *** alpalp has quit IRC
2802016-12-05T03:57:54  *** wasi has quit IRC
2812016-12-05T04:03:53  *** justan0theruser has joined #bitcoin-core-dev
2822016-12-05T04:04:00  *** justanotheruser has quit IRC
2832016-12-05T04:16:07  *** Alopex has quit IRC
2842016-12-05T04:17:12  *** Alopex has joined #bitcoin-core-dev
2852016-12-05T04:17:24  *** molz has joined #bitcoin-core-dev
2862016-12-05T04:20:09  *** moli has quit IRC
2872016-12-05T04:42:41  *** PaulCapestany has quit IRC
2882016-12-05T04:45:27  *** PaulCapestany has joined #bitcoin-core-dev
2892016-12-05T05:02:45  *** MykelSIlver has quit IRC
2902016-12-05T05:04:41  *** Giszmo has quit IRC
2912016-12-05T05:07:19  *** arubi_ has joined #bitcoin-core-dev
2922016-12-05T05:09:44  *** arubi has quit IRC
2932016-12-05T05:45:51  <bitcoin-git> [bitcoin] jtimon opened pull request #9279: Consensus: Move CFeeRate out of libconsensus (master...0.13-consensus-dust-out-minimal) https://github.com/bitcoin/bitcoin/pull/9279
2942016-12-05T05:46:14  <bitcoin-git> [bitcoin] jtimon closed pull request #7820: Consensus: Policy: Move CFeeRate out of consensus module and create CPolicy interface (master...0.12.99-consensus-dust-out) https://github.com/bitcoin/bitcoin/pull/7820
2952016-12-05T05:52:04  *** Atomicat_ has joined #bitcoin-core-dev
2962016-12-05T05:53:09  *** _mn3monic has quit IRC
2972016-12-05T05:53:17  *** _mn3monic has joined #bitcoin-core-dev
2982016-12-05T05:53:41  *** To7 has quit IRC
2992016-12-05T05:54:13  *** Atomicat has quit IRC
3002016-12-05T05:54:16  *** Atomicat_ is now known as Atomicat
3012016-12-05T06:10:03  *** NielsvG has quit IRC
3022016-12-05T06:15:28  *** NielsvG has joined #bitcoin-core-dev
3032016-12-05T06:15:28  *** NielsvG has joined #bitcoin-core-dev
3042016-12-05T06:40:49  *** Ylbam has joined #bitcoin-core-dev
3052016-12-05T06:47:20  <gmaxwell> FWIW, I'm noticing connection slots full on my nodes.
3062016-12-05T06:47:37  *** BCBot_ has joined #bitcoin-core-dev
3072016-12-05T06:48:20  *** bobbytux_ has joined #bitcoin-core-dev
3082016-12-05T06:48:59  <gmaxwell> including some clown at 138.68.10.138 who looks like he's connected three times to everyone; while pretending to be android wallet (he's not).
3092016-12-05T06:50:03  *** Alina-malina_ has joined #bitcoin-core-dev
3102016-12-05T06:50:16  *** ville- has joined #bitcoin-core-dev
3112016-12-05T06:51:45  *** Cory has quit IRC
3122016-12-05T06:52:30  *** bobbytux__ has quit IRC
3132016-12-05T06:52:41  *** CubicEarth has joined #bitcoin-core-dev
3142016-12-05T06:53:50  *** owowo has quit IRC
3152016-12-05T06:53:50  *** Alina-malina has quit IRC
3162016-12-05T06:53:51  *** ville-- has quit IRC
3172016-12-05T06:53:51  *** BCBot has quit IRC
3182016-12-05T06:54:56  *** Cory has joined #bitcoin-core-dev
3192016-12-05T06:55:52  *** owowo has joined #bitcoin-core-dev
3202016-12-05T06:57:01  *** paveljanik has quit IRC
3212016-12-05T07:00:13  *** dermoth has quit IRC
3222016-12-05T07:01:07  *** dermoth has joined #bitcoin-core-dev
3232016-12-05T07:06:15  <bitcoin-git> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/4d955fc5824b...46904ee5d2ce
3242016-12-05T07:06:16  <bitcoin-git> bitcoin/master a188353 Pieter Wuille: Switch GetTransaction to returning a CTransactionRef
3252016-12-05T07:06:16  <bitcoin-git> bitcoin/master c3f5673 Pieter Wuille: Make CWalletTx store a CTransactionRef instead of inheriting
3262016-12-05T07:06:17  <bitcoin-git> bitcoin/master 42fd8de Pieter Wuille: Make DecodeHexTx return a CMutableTransaction
3272016-12-05T07:08:08  <bitcoin-git> [bitcoin] laanwj pushed 7 new commits to master: https://github.com/bitcoin/bitcoin/compare/46904ee5d2ce...d04aebaec7bb
3282016-12-05T07:08:09  <bitcoin-git> bitcoin/master 6fdd43b Matt Corallo: Add struct to track block-connect-time-generated info for callbacks
3292016-12-05T07:08:10  <bitcoin-git> bitcoin/master fd9d890 Matt Corallo: Keep blocks as shared_ptrs, instead of copying txn in ConnectTip
3302016-12-05T07:08:10  <bitcoin-git> bitcoin/master ae4db44 Matt Corallo: Create a shared_ptr for the block we're connecting in ActivateBCS
3312016-12-05T07:08:20  <bitcoin-git> [bitcoin] laanwj closed pull request #9014: Fix block-connection performance regression (master...2016-10-fix-tx-regression) https://github.com/bitcoin/bitcoin/pull/9014
3322016-12-05T07:15:25  *** luke-jr has quit IRC
3332016-12-05T07:27:07  *** fanquake has joined #bitcoin-core-dev
3342016-12-05T07:36:05  *** Alina-malina_ has quit IRC
3352016-12-05T07:36:06  *** Alina-malina_ has joined #bitcoin-core-dev
3362016-12-05T07:36:31  *** AaronvanW has quit IRC
3372016-12-05T07:36:37  *** fanquake has quit IRC
3382016-12-05T07:42:44  *** kadoban has quit IRC
3392016-12-05T07:44:06  <sipa> BlueMatt: i can't build flto with -O3, but i can with -O2
3402016-12-05T08:00:51  <BlueMatt> sipa: strange...I was not doing that, iirc
3412016-12-05T08:01:05  <BlueMatt> I think I tried -march=native originally, but turned it off when it didnt work
3422016-12-05T08:01:08  <BlueMatt> (and ccache and all that shit)
3432016-12-05T08:01:22  *** murchandamus has quit IRC
3442016-12-05T08:02:03  <BlueMatt> gmaxwell: this is why I auto-ban on strange-ass nodes like that (and had already banned that ip)
3452016-12-05T08:02:34  <BlueMatt> and why my public node is behind stupid-ridiculous ddos protection :)
3462016-12-05T08:03:00  *** murchandamus has joined #bitcoin-core-dev
3472016-12-05T08:04:51  <gmaxwell> sipa: why does O3 break it?!
3482016-12-05T08:05:09  <sipa> gmaxwell: sense it makes none
3492016-12-05T08:05:29  <sipa> a linker error with ~boost::filesystem::path
3502016-12-05T08:07:17  <wumpus> never tried flto with -O3, no issues with -O2
3512016-12-05T08:09:18  <sipa> (which afaik was what BlueMatt was seeing as well)
3522016-12-05T08:10:02  <BlueMatt> sipa: yes, it was, though I thought I went back and disabled all the strange optimizations for testing when it failed
3532016-12-05T08:10:06  <BlueMatt> actually, pretty sure I did
3542016-12-05T08:17:13  <gmaxwell> I have a newly updated mass-connector/spy banlist which added 13 new entries:
3552016-12-05T08:17:16  <gmaxwell> https://0bin.net/paste/iSGeBTgtmA8cSHGX#VBd97-ME4hUiIqJIOsd24oLfM57UXP3F3QfgzuoVHAD  -- with commandline syntax
3562016-12-05T08:17:19  <gmaxwell> https://0bin.net/paste/zplzkwGmn6oeP6Ec#oTFWsZFpGQ1zR10Ofv3mJbIOGCnoshxQBQBJoWhSaip -- with debug console syntax
3572016-12-05T08:19:18  *** LeMiner2 has joined #bitcoin-core-dev
3582016-12-05T08:21:18  *** molz has quit IRC
3592016-12-05T08:21:38  <wumpus> thanks
3602016-12-05T08:21:43  *** LeMiner has quit IRC
3612016-12-05T08:21:44  *** LeMiner2 is now known as LeMiner
3622016-12-05T08:24:19  *** LeMiner2 has joined #bitcoin-core-dev
3632016-12-05T08:25:27  *** afk11 has quit IRC
3642016-12-05T08:26:15  *** afk11 has joined #bitcoin-core-dev
3652016-12-05T08:26:15  *** afk11 has quit IRC
3662016-12-05T08:26:15  *** afk11 has joined #bitcoin-core-dev
3672016-12-05T08:26:51  *** LeMiner has quit IRC
3682016-12-05T08:26:52  *** LeMiner2 is now known as LeMiner
3692016-12-05T08:30:57  <Lightsword> gmaxwell, maybe 138.197.197.164 as well
3702016-12-05T08:31:45  <Lightsword> and 138.197.197.132 and 180.173.203.229 and 138.197.197.108
3712016-12-05T08:31:48  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #9280: [Qt] Show ModalOverlay by pressing the progress bar, allow hiding (master...2016/12/qt_modal) https://github.com/bitcoin/bitcoin/pull/9280
3722016-12-05T08:32:37  <gmaxwell> Lightsword: I've only been including ones that show up on all my input hosts, unfortunately since everyone is at limits, that conceals a few.
3732016-12-05T08:35:06  <BlueMatt> gmaxwell: just take the unlimited-connection-slots patch?
3742016-12-05T08:37:25  <sipa> you can set -maxconnections=1000 without any patches
3752016-12-05T08:41:41  <BlueMatt> sipa: huh? I thought that sets you at 125?
3762016-12-05T08:42:09  <sipa> no
3772016-12-05T08:42:16  <sipa> 125 is just the default
3782016-12-05T08:42:29  <BlueMatt> sipa: its limited by available sockets
3792016-12-05T08:42:31  <wumpus> depends on what the fd lmiit is
3802016-12-05T08:42:35  <BlueMatt> which can be super low, because select()
3812016-12-05T08:42:57  <BlueMatt> or this used to be the case
3822016-12-05T08:43:00  <sipa> Warning: Reducing -maxconnections from 1000 to 873, because of system limitations.
3832016-12-05T08:43:04  <sipa> ok, 873.
3842016-12-05T08:43:10  <BlueMatt> oh, 873, hum, I thought it was lower
3852016-12-05T08:43:12  <wumpus> select() can handle 1024 on most systems, that's pretty much enough for most cases
3862016-12-05T08:43:22  <BlueMatt> whatever, I carry a patch to use poll() to make it actually higher....
3872016-12-05T08:44:24  <wumpus> I guess we've held up switching to poll because we expected to switch to libevent any day, that's taking a bit longer than expected :)
3882016-12-05T08:44:42  <sipa> Soon! (tm)
3892016-12-05T08:44:47  <wumpus> yea :-)
3902016-12-05T08:44:52  <gmaxwell> IIRC matt's patch is darn near trivial.
3912016-12-05T08:45:14  *** JackH has quit IRC
3922016-12-05T08:45:36  *** JackH has joined #bitcoin-core-dev
3932016-12-05T08:46:26  <BlueMatt> gmaxwell: yes, but given that its 873 not 1XX as I'd thought, probably not worth it
3942016-12-05T08:46:39  <BlueMatt> and libevent is actually sooner now
3952016-12-05T08:49:00  *** BashCo has quit IRC
3962016-12-05T09:01:13  *** jl2012 has quit IRC
3972016-12-05T09:01:34  *** jl2012 has joined #bitcoin-core-dev
3982016-12-05T09:03:29  *** Guest13131 has quit IRC
3992016-12-05T09:07:05  *** ChillazZ has joined #bitcoin-core-dev
4002016-12-05T09:07:28  *** ChillazZ is now known as Guest15721
4012016-12-05T09:13:43  <bitcoin-git> [bitcoin] kallewoof opened pull request #9281: Refactor: Removed using namespace <xxx> from bench/ & test/ sources (master...no-using-namespace-bench-test) https://github.com/bitcoin/bitcoin/pull/9281
4022016-12-05T09:18:44  *** wvr has quit IRC
4032016-12-05T09:27:38  *** BashCo has joined #bitcoin-core-dev
4042016-12-05T09:28:51  *** Guest15721 has quit IRC
4052016-12-05T09:28:51  *** ChillazZ has joined #bitcoin-core-dev
4062016-12-05T09:29:20  *** ChillazZ is now known as Guest62446
4072016-12-05T09:33:54  *** Victorsueca has joined #bitcoin-core-dev
4082016-12-05T09:36:12  *** Victor_sueca has quit IRC
4092016-12-05T09:55:36  *** jl2012 has quit IRC
4102016-12-05T09:55:57  *** jl2012 has joined #bitcoin-core-dev
4112016-12-05T09:56:53  *** murchandamus has quit IRC
4122016-12-05T09:57:16  *** murchandamus has joined #bitcoin-core-dev
4132016-12-05T09:57:42  *** murchandamus has joined #bitcoin-core-dev
4142016-12-05T09:57:46  *** wvr has joined #bitcoin-core-dev
4152016-12-05T09:58:58  *** jl2012 has quit IRC
4162016-12-05T09:59:03  <bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/d04aebaec7bb...613bda418f83
4172016-12-05T09:59:04  <bitcoin-git> bitcoin/master 634ad51 Pieter Wuille: Squashed 'src/leveldb/' changes from 20ca81f..a31c8aa...
4182016-12-05T09:59:05  <bitcoin-git> bitcoin/master 605d701 Pieter Wuille: Merge in LevelDB 1.19 changes
4192016-12-05T09:59:05  <bitcoin-git> bitcoin/master 613bda4 Wladimir J. van der Laan: Merge #8613: LevelDB 1.19...
4202016-12-05T10:00:40  *** jl2012 has joined #bitcoin-core-dev
4212016-12-05T10:02:38  *** AaronvanW has joined #bitcoin-core-dev
4222016-12-05T10:02:38  *** AaronvanW has quit IRC
4232016-12-05T10:02:38  *** AaronvanW has joined #bitcoin-core-dev
4242016-12-05T10:03:07  <sipa> wumpus: yeah, i didn't mean to say we should use reuse_logs, just that the feature over time may be interesting to us
4252016-12-05T10:03:13  <sipa> *roght now
4262016-12-05T10:05:15  <wumpus> sipa: right
4272016-12-05T10:09:00  *** ClockCat has joined #bitcoin-core-dev
4282016-12-05T10:09:51  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/613bda418f83...43e8150ef690
4292016-12-05T10:09:52  <bitcoin-git> bitcoin/master 2efc438 Pieter Wuille: Align struct COrphan definition
4302016-12-05T10:09:52  <bitcoin-git> bitcoin/master 43e8150 Wladimir J. van der Laan: Merge #9269: Align struct COrphan definition...
4312016-12-05T10:10:05  <bitcoin-git> [bitcoin] laanwj closed pull request #9269: Align struct COrphan definition (master...oneorphan) https://github.com/bitcoin/bitcoin/pull/9269
4322016-12-05T10:16:08  <bitcoin-git> [bitcoin] paveljanik opened pull request #9282: CMutableTransaction is defined as struct (master...20161205_CMutableTransaction_is_struct) https://github.com/bitcoin/bitcoin/pull/9282
4332016-12-05T10:29:30  *** Victor_sueca has joined #bitcoin-core-dev
4342016-12-05T10:31:55  *** Victorsueca has quit IRC
4352016-12-05T10:34:51  *** MarcoFalke has joined #bitcoin-core-dev
4362016-12-05T10:35:21  *** luke-jr has joined #bitcoin-core-dev
4372016-12-05T10:36:47  <bitcoin-git> [bitcoin] sipa opened pull request #9283: A few more CTransactionRef optimizations (master...sharedblock2) https://github.com/bitcoin/bitcoin/pull/9283
4382016-12-05T10:55:57  *** NielsvG has quit IRC
4392016-12-05T11:19:55  *** MarcoFalke has left #bitcoin-core-dev
4402016-12-05T11:47:54  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/43e8150ef690...c01f16aaa0ed
4412016-12-05T11:47:55  <bitcoin-git> bitcoin/master ea83d00 instagibbs: SendMoney: use already-calculated balance
4422016-12-05T11:47:55  <bitcoin-git> bitcoin/master c01f16a Wladimir J. van der Laan: Merge #9165: SendMoney: use already-calculated balance...
4432016-12-05T11:48:04  <bitcoin-git> [bitcoin] laanwj closed pull request #9165: SendMoney: use already-calculated balance (master...triv-curbal) https://github.com/bitcoin/bitcoin/pull/9165
4442016-12-05T11:53:09  *** dcousens has quit IRC
4452016-12-05T11:55:06  *** dcousens has joined #bitcoin-core-dev
4462016-12-05T11:55:25  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c01f16aaa0ed...7d5d44969b4a
4472016-12-05T11:55:25  <bitcoin-git> bitcoin/master c4b6fa8 Pavel Janík: CMutableTransaction is defined as struct.
4482016-12-05T11:55:26  <bitcoin-git> bitcoin/master 7d5d449 Wladimir J. van der Laan: Merge #9282: CMutableTransaction is defined as struct...
4492016-12-05T11:55:41  <bitcoin-git> [bitcoin] laanwj closed pull request #9282: CMutableTransaction is defined as struct (master...20161205_CMutableTransaction_is_struct) https://github.com/bitcoin/bitcoin/pull/9282
4502016-12-05T11:59:27  *** _biO_ has joined #bitcoin-core-dev
4512016-12-05T12:28:42  *** Victor_sueca has quit IRC
4522016-12-05T12:29:15  *** Alina-malina_ is now known as Alina-malina
4532016-12-05T12:29:51  *** Victor_sueca has joined #bitcoin-core-dev
4542016-12-05T12:30:55  *** CubicEarth has quit IRC
4552016-12-05T12:32:45  *** alpalp has joined #bitcoin-core-dev
4562016-12-05T13:02:44  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #9284: Suppress some annoying deprecation warnings (OSX) (master...2016/12/osx_warnings) https://github.com/bitcoin/bitcoin/pull/9284
4572016-12-05T13:08:09  *** moli has joined #bitcoin-core-dev
4582016-12-05T13:27:19  *** PaulCapestany has quit IRC
4592016-12-05T13:28:07  *** BashCo_ has joined #bitcoin-core-dev
4602016-12-05T13:30:03  *** Chris_Stewart_5 has quit IRC
4612016-12-05T13:31:17  *** BashCo has quit IRC
4622016-12-05T13:31:21  *** CubicEarth has joined #bitcoin-core-dev
4632016-12-05T13:36:11  *** CubicEarth has quit IRC
4642016-12-05T13:37:08  *** alpalp has quit IRC
4652016-12-05T13:38:56  <jonasschnelli> And comments on our keypoolrefil RPC call behavior?
4662016-12-05T13:38:56  <jonasschnelli> https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/keypool.py#L45
4672016-12-05T13:39:10  <jonasschnelli> The tests proof, that nodes[0].keypoolrefill(3) result in 4 available keys.
4682016-12-05T13:39:22  <jonasschnelli> But reading the API docs, it should be 3.
4692016-12-05T13:39:34  <jonasschnelli> IMO the +1 is wrong here
4702016-12-05T13:39:54  <jonasschnelli> (I'd ask because I'd like to fix this with the HD split in ext/int chain)
4712016-12-05T13:46:48  *** Chris_Stewart_5 has joined #bitcoin-core-dev
4722016-12-05T13:56:30  <dcousens> hmm, CTransaction assignment was totally removed aye
4732016-12-05T14:00:05  *** Cory has quit IRC
4742016-12-05T14:00:51  <dcousens> sipa: was removing CTransaction& operator=(const CTransaction& tx); necessary, or just a safety precaution?
4752016-12-05T14:02:43  *** Cory has joined #bitcoin-core-dev
4762016-12-05T14:08:47  <dcousens> meh, I guess I can just CTransactionRef anyway
4772016-12-05T14:20:24  *** jtimon has quit IRC
4782016-12-05T14:21:19  <dcousens> eh, nvm, rebased all my local code, tl;dr was juts changing CTransaction to CTransactionRef, .vout to ->vout ... and thats it.
4792016-12-05T14:21:25  <dcousens> LGTM :)
4802016-12-05T14:25:28  <instagibbs> dcousens, I love happy endings
4812016-12-05T14:32:41  *** Guyver2 has joined #bitcoin-core-dev
4822016-12-05T14:32:45  *** CubicEarth has joined #bitcoin-core-dev
4832016-12-05T14:37:42  *** CubicEarth has quit IRC
4842016-12-05T14:49:45  <dcousens> instagibbs: not so happy yet ha
4852016-12-05T14:50:49  *** murchandamus has quit IRC
4862016-12-05T14:51:18  *** murchandamus has joined #bitcoin-core-dev
4872016-12-05T14:52:40  <dcousens> trying a fresh-recompile,  but master seems to just lock up for me atm
4882016-12-05T14:54:42  *** Giszmo has joined #bitcoin-core-dev
4892016-12-05T15:03:28  *** thib has joined #bitcoin-core-dev
4902016-12-05T15:08:59  *** murchandamus has quit IRC
4912016-12-05T15:09:18  *** murchandamus has joined #bitcoin-core-dev
4922016-12-05T15:10:10  *** bsm1175321 has joined #bitcoin-core-dev
4932016-12-05T15:11:32  *** Chris_Stewart_5 has quit IRC
4942016-12-05T15:13:03  *** bsm1175321 has quit IRC
4952016-12-05T15:18:13  *** bsm1175321 has joined #bitcoin-core-dev
4962016-12-05T15:20:42  *** laurentmt has joined #bitcoin-core-dev
4972016-12-05T15:21:59  *** bsm1175321 has quit IRC
4982016-12-05T15:27:14  *** Chris_Stewart_5 has joined #bitcoin-core-dev
4992016-12-05T15:28:12  *** bsm1175321 has joined #bitcoin-core-dev
5002016-12-05T15:30:34  *** bsm1175321 has quit IRC
5012016-12-05T15:30:48  *** Chris_Stewart_5 has quit IRC
5022016-12-05T15:31:08  *** Chris_Stewart_5 has joined #bitcoin-core-dev
5032016-12-05T15:32:22  *** bsm1175321 has joined #bitcoin-core-dev
5042016-12-05T15:33:29  *** CubicEarth has joined #bitcoin-core-dev
5052016-12-05T15:36:41  *** laurentmt has quit IRC
5062016-12-05T15:37:57  *** CubicEarth has quit IRC
5072016-12-05T15:46:48  *** murchandamus has quit IRC
5082016-12-05T15:48:09  *** murchandamus has joined #bitcoin-core-dev
5092016-12-05T15:49:02  *** morcos has quit IRC
5102016-12-05T15:49:43  *** zxzzt has quit IRC
5112016-12-05T15:51:06  *** morcos has joined #bitcoin-core-dev
5122016-12-05T15:51:14  *** zxzzt has joined #bitcoin-core-dev
5132016-12-05T15:56:33  *** squidicuz has quit IRC
5142016-12-05T15:58:06  *** paveljanik has joined #bitcoin-core-dev
5152016-12-05T15:58:06  *** paveljanik has joined #bitcoin-core-dev
5162016-12-05T16:30:00  *** TomMc has joined #bitcoin-core-dev
5172016-12-05T16:34:12  *** CubicEarth has joined #bitcoin-core-dev
5182016-12-05T16:38:37  *** CubicEarth has quit IRC
5192016-12-05T16:39:01  *** laurentmt has joined #bitcoin-core-dev
5202016-12-05T16:47:40  *** paveljanik has quit IRC
5212016-12-05T16:48:12  *** abpa has joined #bitcoin-core-dev
5222016-12-05T16:48:31  *** paveljanik has joined #bitcoin-core-dev
5232016-12-05T16:48:32  *** paveljanik has joined #bitcoin-core-dev
5242016-12-05T16:55:31  *** laurentmt has quit IRC
5252016-12-05T17:00:40  *** laurentmt has joined #bitcoin-core-dev
5262016-12-05T17:04:46  *** wvr has quit IRC
5272016-12-05T17:10:19  *** BCBot_ has quit IRC
5282016-12-05T17:10:41  *** BCBot has joined #bitcoin-core-dev
5292016-12-05T17:29:56  <jl2012> in what situation, the "Warning: We do not appear to fully agree with our peers! You may need to upgrade, or other nodes may need to upgrade" will be shown?
5302016-12-05T17:34:58  *** CubicEarth has joined #bitcoin-core-dev
5312016-12-05T17:39:49  *** CubicEarth has quit IRC
5322016-12-05T17:40:09  <morcos> jonasschnelli: you around?
5332016-12-05T17:41:33  <morcos> re: #8501, I agree with not fixing the frequency..  But i'm unsure about the no duplicating the same value over and over again..
5342016-12-05T17:41:35  <gribble> https://github.com/bitcoin/bitcoin/issues/8501 | Add mempool statistics collector by jonasschnelli · Pull Request #8501 · bitcoin/bitcoin · GitHub
5352016-12-05T17:42:14  <morcos> It might depend on the use case, but for instance it might be valuable to know that it stayed the same for a while and then incremented all at once, as opposed to not being able to tell that it just hadn't been simpled in between
5362016-12-05T17:43:04  <morcos> my thought was if we saved having to record the time stamp, we might be able to put up with lots of duplicate values..  especially if we're saving for instance only 1000 data points at second frequency, it just won't use all that much memory
5372016-12-05T17:43:23  *** kadoban has joined #bitcoin-core-dev
5382016-12-05T17:43:27  <morcos> anywya, sorry, dont mean to redesign your whole PR months after you opened it
5392016-12-05T17:44:08  *** Victor_sueca has quit IRC
5402016-12-05T17:49:56  *** davec has quit IRC
5412016-12-05T18:15:34  *** BashCo_ has quit IRC
5422016-12-05T18:16:11  *** BashCo has joined #bitcoin-core-dev
5432016-12-05T18:19:36  *** laurentmt has quit IRC
5442016-12-05T18:20:28  *** BashCo has quit IRC
5452016-12-05T18:35:42  *** CubicEarth has joined #bitcoin-core-dev
5462016-12-05T18:37:06  *** BashCo has joined #bitcoin-core-dev
5472016-12-05T18:40:23  *** CubicEarth has quit IRC
5482016-12-05T18:46:54  <Chris_Stewart_5> Does -txindex significantly impact performance of IBD? I tried to sync last night and only synced ~10K blocks, which seems slow. Is that reasonable?
5492016-12-05T18:48:56  <sipa> that's totally unreasonable
5502016-12-05T18:49:04  <sipa> is it stuck?
5512016-12-05T18:49:08  <sipa> or just slow?
5522016-12-05T18:52:28  <Chris_Stewart_5> extremely slow it seems. I'm using out of box settings on 0.13.1 with -txindex.
5532016-12-05T18:52:43  *** CubicEarth has joined #bitcoin-core-dev
5542016-12-05T18:52:53  <sipa> does increasing dbcache help?
5552016-12-05T18:53:13  *** CubicEarth has quit IRC
5562016-12-05T18:53:43  <Chris_Stewart_5> I'll try it later and report back, default is 2GB?
5572016-12-05T18:54:32  <sipa> default is 300MB
5582016-12-05T18:55:20  <Chris_Stewart_5> mmm that is probably why. I thought it was significantly higher. How long does IBD take other people with that setting as default?
5592016-12-05T18:59:03  <sipa> what height are you at now?
5602016-12-05T19:00:03  *** pavel_ has joined #bitcoin-core-dev
5612016-12-05T19:01:06  *** pavel_ has quit IRC
5622016-12-05T19:01:36  *** paveljanik has quit IRC
5632016-12-05T19:03:13  *** aalex__ has quit IRC
5642016-12-05T19:03:33  *** aalex__ has joined #bitcoin-core-dev
5652016-12-05T19:04:28  *** aalex has joined #bitcoin-core-dev
5662016-12-05T19:05:45  *** davec has joined #bitcoin-core-dev
5672016-12-05T19:07:53  <Chris_Stewart_5> 403817
5682016-12-05T19:08:03  *** aalex__ has quit IRC
5692016-12-05T19:08:36  *** harrymm has quit IRC
5702016-12-05T19:11:14  *** moli has quit IRC
5712016-12-05T19:12:07  *** MarcoFalke has joined #bitcoin-core-dev
5722016-12-05T19:12:37  *** dcousens has quit IRC
5732016-12-05T19:22:14  *** lightningbot has joined #bitcoin-core-dev
5742016-12-05T19:22:32  *** timothy has quit IRC
5752016-12-05T19:22:35  *** drizztbsd has joined #bitcoin-core-dev
5762016-12-05T19:23:03  *** drizztbsd is now known as timothy
5772016-12-05T19:23:28  *** sdaftuar_ has joined #bitcoin-core-dev
5782016-12-05T19:23:46  *** kanzure_ has joined #bitcoin-core-dev
5792016-12-05T19:24:35  *** laurentmt has joined #bitcoin-core-dev
5802016-12-05T19:24:54  *** harrymm has joined #bitcoin-core-dev
5812016-12-05T19:25:14  *** bobbytux_ has quit IRC
5822016-12-05T19:26:16  *** jyap_ has joined #bitcoin-core-dev
5832016-12-05T19:27:15  *** adam3us_ has joined #bitcoin-core-dev
5842016-12-05T19:27:20  *** BlueMatt_ has joined #bitcoin-core-dev
5852016-12-05T19:28:23  *** BlueMatt has quit IRC
5862016-12-05T19:28:26  *** BlueMatt_ is now known as BlueMatt
5872016-12-05T19:28:26  *** BlueMatt has joined #bitcoin-core-dev
5882016-12-05T19:28:32  *** ville- has quit IRC
5892016-12-05T19:28:32  *** Atomicat has quit IRC
5902016-12-05T19:28:33  *** sdaftuar has quit IRC
5912016-12-05T19:28:33  *** adam3us has quit IRC
5922016-12-05T19:28:33  *** kanzure has quit IRC
5932016-12-05T19:28:33  *** jyap has quit IRC
5942016-12-05T19:28:33  *** Bootvis has quit IRC
5952016-12-05T19:28:33  *** jyap_ is now known as jyap
5962016-12-05T19:28:34  *** jyap has joined #bitcoin-core-dev
5972016-12-05T19:28:39  *** Atomicat_ is now known as Atomicat
5982016-12-05T19:28:41  *** ville-- has joined #bitcoin-core-dev
5992016-12-05T19:29:53  *** Bootvis has joined #bitcoin-core-dev
6002016-12-05T19:29:58  *** harrymm has quit IRC
6012016-12-05T19:37:42  *** aalex_ has joined #bitcoin-core-dev
6022016-12-05T19:41:16  *** aalex has quit IRC
6032016-12-05T19:49:26  *** kanzure_ is now known as kanzure
6042016-12-05T20:00:43  *** Madars has quit IRC
6052016-12-05T20:03:18  *** Madars has joined #bitcoin-core-dev
6062016-12-05T20:08:42  *** arubi_ is now known as arubi
6072016-12-05T20:33:43  <Chris_Stewart_5> sipa: I should have been more clear, I have been trying to do IBD over the course of a few nights, with results like I said ~10k blocks a night.
6082016-12-05T20:34:46  <Chris_Stewart_5> the first ~250k blocks went relatively fast (a couple hour period) but I think some might have already been on disk? Perhaps i'm using the term IBD a little too loosely
6092016-12-05T20:34:58  <Chris_Stewart_5> but it is a major sync
6102016-12-05T20:40:11  <instagibbs> sdaftuar_, why would you want to spend a coin from you wallet that has descendants(already spent)? I'm surely thinking of this wrong
6112016-12-05T20:41:48  <instagibbs> or are descendants calculated from a tx point of view, ie other output has been spent in a chain, therefore that adds to that count
6122016-12-05T20:42:23  <sdaftuar_> instagibbs: oh, yeah i meant in-mempool descendants
6132016-12-05T20:42:35  <sdaftuar_> say you have a tx that has 2 outputs, you send me money and give yourself change.
6142016-12-05T20:42:39  <sdaftuar_> then i chain 24 transactions off it
6152016-12-05T20:42:48  <sdaftuar_> you try to spend your change, but that'll fail
6162016-12-05T20:42:54  <instagibbs> ok, didnt think of the fact that outputs are linked re:descendants
6172016-12-05T20:43:21  <sdaftuar_> that does seem to be a confusing property of the mempool limiting :)
6182016-12-05T20:43:31  *** sdaftuar_ is now known as sdaftuar
6192016-12-05T20:43:31  <instagibbs> but that is obv in hindsight. Ok, well one issue is you might have asymmetrical limits.
6202016-12-05T20:43:39  <sdaftuar> yeah, i suggested using min()
6212016-12-05T20:43:50  <gmaxwell> it might have been better if that limit was split across outputs.
6222016-12-05T20:44:06  <sdaftuar> gmaxwell: that would then fail to capture the issue at hand, i think
6232016-12-05T20:44:09  <instagibbs> sdaftuar, hmm says max on my screen
6242016-12-05T20:44:21  <gmaxwell> e.g. A can have up to 24 decendants, it has two outputs, each can have 12 under it.
6252016-12-05T20:44:35  <sdaftuar> sorry, max(tx->ancestor, tx->descendants()) should be less than min(ancestorlimit, descendantlimit)
6262016-12-05T20:44:39  <instagibbs> oh i see nvm
6272016-12-05T20:45:02  <sdaftuar> gmaxwell: oh, hm.
6282016-12-05T20:47:04  <gmaxwell> in the worst case though it reduces your maximum to a log_outputs(depth), which isn't awesome.
6292016-12-05T20:47:17  <sdaftuar> maybe doable, but kind of yuck to implement i think
6302016-12-05T20:48:00  <gmaxwell> but it would prevent other people from chewing up your limit. I think this hasn't actually been a problem, though I could imagine it being one in certian kinds of transaction protocols.
6312016-12-05T20:48:42  <sdaftuar> what kinds of protocols do you have in mind?
6322016-12-05T20:49:39  <gmaxwell> In the abstract, protocols where someone delaying your transaction can allow the party to cheat like atomic swaps. Not that big of a concern since unless the head transaction is confirmed those protocols are not secure against miners.
6332016-12-05T20:52:28  <sdaftuar> right, if someone comes up with a use case that does rely on the parent not necessarily being confirmed, then that should alter our thinking
6342016-12-05T20:53:38  *** wasi has joined #bitcoin-core-dev
6352016-12-05T20:59:40  *** windsok has quit IRC
6362016-12-05T21:19:41  *** aalex__ has joined #bitcoin-core-dev
6372016-12-05T21:23:24  *** aalex_ has quit IRC
6382016-12-05T21:53:30  *** windsok has joined #bitcoin-core-dev
6392016-12-05T21:58:07  *** jtimon has joined #bitcoin-core-dev
6402016-12-05T22:13:41  *** jl2012 has quit IRC
6412016-12-05T22:15:54  *** laurentmt has quit IRC
6422016-12-05T22:23:33  *** Chris_Stewart_5 has quit IRC
6432016-12-05T22:28:30  *** aalex has joined #bitcoin-core-dev
6442016-12-05T22:31:32  *** aalex__ has quit IRC
6452016-12-05T22:44:11  *** Guyver2 has quit IRC
6462016-12-05T22:58:48  *** grubles has quit IRC
6472016-12-05T22:59:30  *** JackH has quit IRC
6482016-12-05T23:01:03  *** tunafizz has joined #bitcoin-core-dev
6492016-12-05T23:12:42  *** laurentmt has joined #bitcoin-core-dev
6502016-12-05T23:18:24  *** laurentmt has quit IRC
6512016-12-05T23:31:07  *** cryptapus is now known as cryptapus_afk
6522016-12-05T23:33:08  *** dcousens has joined #bitcoin-core-dev
6532016-12-05T23:39:13  *** MarcoFalke has quit IRC
6542016-12-05T23:50:22  <dcousens> hmph
6552016-12-05T23:50:29  <dcousens> So I'm running master, no changes at all
6562016-12-05T23:50:54  <dcousens> And my bitcoind finishes up to verify, then just sits there on 100% CPU usage (probably forever, but who knows)
6572016-12-05T23:51:13  <gmaxwell> 'up to verify'?
6582016-12-05T23:51:15  <dcousens> It fails to open up the RPC, or start synchronizing
6592016-12-05T23:51:20  <dcousens> checkblocks
6602016-12-05T23:51:27  <gmaxwell> what is the last log entry?
6612016-12-05T23:51:34  <gmaxwell> can you attach GDB?
6622016-12-05T23:51:47  <dcousens> It still keeps logging, but solely the tor control messages
6632016-12-05T23:52:25  <gmaxwell> that sounds like a deadlock then.
6642016-12-05T23:52:29  <dcousens> What do I need to do to attach GDB? Happy to do it
6652016-12-05T23:52:38  <gmaxwell> dcousens: what OS are you on?
6662016-12-05T23:53:47  <dcousens> just collecting info, sec
6672016-12-05T23:54:27  <gmaxwell> on *nix:   ps aux | grep bitcoin   to get the bitcoind pid    then gdb -p <pid> to attach.     then run thread apply all bt full   to get backtraces from every thread, and 0bin that to me,  and then you can type q<enter> to quit
6682016-12-05T23:58:04  *** moli has joined #bitcoin-core-dev
6692016-12-05T23:58:37  <dcousens> http://pastie.org/private/a40ppfohw2kxxxundn1iw - for the info, doing gdb now