12015-10-19T00:10:39  <Luke-Jr> gmaxwell: perhaps. optimizing the sorted list would help in multiple places though
  22015-10-19T00:13:49  <gmaxwell> as far as optimizing, how about adding one of the satoshi dice addreses to a wallet and rescanning?
  32015-10-19T00:14:05  <Luke-Jr> (just the last item will actually change the algorithm btw)
  42015-10-19T00:14:30  <gmaxwell> Luke-Jr: sure other optimizations are good too, but AFAIR all that needs to do is track a the extrema.
  52015-10-19T00:16:03  <Luke-Jr> gmaxwell: it ignores any transactions timestamped >5m in the future from the local time, which means such txns would have to be reconsidered over time
  62015-10-19T00:18:54  <gmaxwell> Luke-Jr: uh. how are they ever going to get learned then?
  72015-10-19T00:19:08  <Luke-Jr> gmaxwell: ?
  82015-10-19T00:19:24  <Luke-Jr> ignores in the context of looking at other txns timestamps to decide the timestamp for this one
  92015-10-19T00:20:09  <phantomcircuit> what's the relative cost of checking an address vs pattern matching the output to start with?
 102015-10-19T00:20:11  <gmaxwell> remind me precisely what it does?
 112015-10-19T00:20:23  <gmaxwell> phantomcircuit: go read the pr
 122015-10-19T00:20:28  <gmaxwell> (the text on the opening)
 132015-10-19T00:21:23  <Luke-Jr> gmaxwell: it's complex enough that I'd just be rewriting the code in pseudo-code to explain it :/
 142015-10-19T00:21:45  <gmaxwell> Luke-Jr: good well that means it can change behavior and not violate anyone's expectations because apparently no one can understand it. :P
 152015-10-19T00:21:59  <Luke-Jr> it's not hard to understand, it's just complex :p
 162015-10-19T00:22:10  <phantomcircuit> gmaxwell, i never understood why we even bothered to guess when an unconfirmed transaction happened
 172015-10-19T00:22:19  <Luke-Jr> the point is that it meets expectations
 182015-10-19T00:22:27  <gmaxwell> phantomcircuit: because we show them in listtransactions
 192015-10-19T00:22:56  <phantomcircuit> gmaxwell, "this transaction happened now" seems reasonable for all unconfirmed transactions
 202015-10-19T00:23:13  <gmaxwell> phantomcircuit: but then its inconsistent with what you get when it confirms.
 212015-10-19T00:23:21  <Luke-Jr> phantomcircuit: no, definitely not when you're syncing a node that was off
 222015-10-19T00:23:48  <phantomcircuit> Luke-Jr, how about null
 232015-10-19T00:23:50  <gmaxwell> (the listtransaction output gives several different times when you care about details... and a smart time)
 242015-10-19T00:24:03  <gmaxwell> phantomcircuit: bet peoples software will handle that super awesomely well.
 252015-10-19T00:24:06  <Luke-Jr> phantomcircuit: … that is not useful
 262015-10-19T00:24:24  <phantomcircuit> the timestamp we're returning now isn't very useful either
 272015-10-19T00:24:31  <gmaxwell> phantomcircuit: It's usually pretty good.
 282015-10-19T00:24:58  <gmaxwell> Luke-Jr: I think you might need to write out the algorithim again, we cannot continue with this being slow like this.
 292015-10-19T00:25:33  <phantomcircuit> i guess...
 302015-10-19T00:26:06  <Luke-Jr> gmaxwell: that's why I'm working on optimizing it.. if we want accounting to keep working, I can deal with that, just would have saved time to omit it
 312015-10-19T00:26:55  <Luke-Jr> (this code actually seems to be somewhat broken with accounts, fwiw)
 322015-10-19T00:27:37  <gmaxwell> Luke-Jr: I thought smarttime just returned blocktime for confirmed transactions that you didn't observe first?
 332015-10-19T00:29:08  <Luke-Jr> not necessarily
 342015-10-19T00:29:12  *** Thireus1 has quit IRC
 352015-10-19T00:29:50  <gmaxwell> unless the blocktime is in the future, in which case?
 362015-10-19T00:30:04  *** Thireus has joined #bitcoin-core-dev
 372015-10-19T00:32:27  <phantomcircuit> this seems much smarter than it needs to be...
 382015-10-19T00:32:33  <Luke-Jr> it's approximately max(last_txn_time, min(block_time, max(last_txn_time, now)))
 392015-10-19T00:32:38  <phantomcircuit> also it seems like all of these values should be cachable
 402015-10-19T00:33:17  <gmaxwell> Luke-Jr: than can be simplified to refer only to blocktimes in the context of a rescan.
 412015-10-19T00:33:37  <Luke-Jr> gmaxwell: half the goal here is to avoid transactions ever having timestamps out of order
 422015-10-19T00:34:03  <gmaxwell> which they cannot, in a rescan.
 432015-10-19T00:34:08  <Luke-Jr> uh, why not?
 442015-10-19T00:34:19  <Luke-Jr> block times are out of order all the time
 452015-10-19T00:34:40  <phantomcircuit> what's the rational behind not setting the transaction timestamp to the block timestamp?
 462015-10-19T00:34:52  <gmaxwell> Luke-Jr: as I just said, you can apply that filter to blocktimes only!
 472015-10-19T00:35:14  <phantomcircuit> oh you want to avoid the timestamps disagreeing with the nOrdPos
 482015-10-19T00:35:15  <gmaxwell> phantomcircuit: becuase they end up out of order which really screws with people.
 492015-10-19T00:35:48  <Luke-Jr> gmaxwell: that sounds a lot more complex, and doesn't consider that we can ignore it when blocks don't have any txns of ours
 502015-10-19T00:36:16  <gmaxwell> Luke-Jr: this code takes _9.5 hours of processing_ in the requesting users example.
 512015-10-19T00:36:23  <gmaxwell> And you are going to complain about complex?
 522015-10-19T00:38:14  <gmaxwell> Luke-Jr: having it use some really old time because your last transaction was 50 blocks ago isn't good.
 532015-10-19T00:38:22  <Luke-Jr> I find it simpler to just optimise the ordered tx list
 542015-10-19T00:38:51  <gmaxwell> e.g. if the latest block is in the future, okay you don't use the time. Then it should use the time of the prior block, even if there was no transaction there, not the time of your last transaction.
 552015-10-19T00:39:09  <gmaxwell> at least in the context of a rescan where you do not learn times anywhere except blocks.
 562015-10-19T00:39:17  <gmaxwell> I think the current behavior is just busted there.
 572015-10-19T00:39:38  <Luke-Jr> gmaxwell: it would use `now` in that case
 582015-10-19T00:40:07  <Luke-Jr> in a rescan, block times will basically never be in the future.
 592015-10-19T00:40:16  <gmaxwell> yea, unlikely, indeed.
 602015-10-19T00:42:05  <Luke-Jr> gmaxwell: will you NACK an optimization that preserves existing behaviour?
 612015-10-19T00:42:49  <gmaxwell> if it's measurably slower, yes.  This should not take any time at all. as I said, I would have NAKED smarttimes if I knew it caused a measurable slowdown.
 622015-10-19T00:44:18  *** Ylbam has quit IRC
 632015-10-19T00:54:33  *** equil_ has joined #bitcoin-core-dev
 642015-10-19T01:16:13  *** CodeShark has joined #bitcoin-core-dev
 652015-10-19T01:16:31  *** belcher has quit IRC
 662015-10-19T01:32:49  *** maaku has quit IRC
 672015-10-19T01:33:19  <Luke-Jr> gmaxwell: I *think* it is possible to optimise so it's reasonably close, but in case not, there is a possibility of rescan just skipping smart-time assignment (which currently causes "time" to fall back to received time = rescan time, but maybe that should change?)
 682015-10-19T01:34:23  <Luke-Jr> side note: hmm, sync of my node confused BFGMiner as it got notified of old blocks O.o
 692015-10-19T01:34:41  <gmaxwell> Luke-Jr: of course, if this function is slow during rescan its also slow during runtime... so I do support speeding it up generally too.
 702015-10-19T01:35:00  <gmaxwell> Luke-Jr: would rather fall back to monotoneized blocktime than rescan time.
 712015-10-19T01:35:08  <Luke-Jr> well, part of why it was unoptimised is probably because runtime is not time sensitive
 722015-10-19T01:36:06  <gmaxwell> Luke-Jr: hope you're not mining from a node with a wallet...
 732015-10-19T01:36:34  <gmaxwell> but really having a very big wallet (Even bigger than the ones there) would eventually be disruptive.
 742015-10-19T01:36:52  *** maaku has joined #bitcoin-core-dev
 752015-10-19T01:37:16  *** maaku is now known as Guest57960
 762015-10-19T01:37:30  <Luke-Jr> eh? it's just BFGMiner (running for testing) automatically picking up my hot wallet
 772015-10-19T01:38:06  <Luke-Jr> it's using Eligius mainly, but the local node's block disagreements somewhat confused it
 782015-10-19T01:38:43  <Luke-Jr> no big deal if mining is negatively impacted by it; maybe a slightly bigger deal if it were to compromise my hot wallet
 792015-10-19T01:39:46  <Luke-Jr> but if that latter case is a risk, we should fix it. I'm not aware of such a problem.
 802015-10-19T01:43:23  <gmaxwell> I wasn't commenting on your bfgminer comment.
 812015-10-19T01:44:28  <Luke-Jr> oh
 822015-10-19T01:44:37  <gmaxwell> I'm saying that you shouldn't dismiss that the performance of sorting the whole wallet on an add as being important. As an example, someone who is mining with a wallet cares about a 100ms delay, (which is the ballpark of what that user was reporting).. or very big wallets.
 832015-10-19T01:45:34  *** Guest57960 is now known as maaku
 842015-10-19T01:48:46  <phantomcircuit> Luke-Jr, cant you just cache the oldest record timestamp?
 852015-10-19T01:49:24  <Luke-Jr> phantomcircuit: newest*, but no, because of the 5 minute future rule
 862015-10-19T01:50:06  <Luke-Jr> optimising OrderedTxList looks like an easy big win. let me finish that and we can see where we are and if we need more improvements?
 872015-10-19T01:59:56  <gmaxwell> Luke-Jr: sounds like something to try.
 882015-10-19T02:08:25  <phantomcircuit> bleh the real problem is the rpc api is wrong
 892015-10-19T02:08:36  <phantomcircuit> probably cant reasonably fix that though
 902015-10-19T02:11:36  <gmaxwell> on my not very huge wallet, the PR on github is a ~1% speedup.
 912015-10-19T02:12:14  <gmaxwell> really we're doing something dumb if every wallet isn't approximately the same speed for rescan.
 922015-10-19T02:27:51  *** d_t has quit IRC
 932015-10-19T02:29:52  *** d_t has joined #bitcoin-core-dev
 942015-10-19T02:56:34  <kanzure> rescan was killing all my rpcthreads a few weeks ago and i didn't have any insight into what was going on :-( maybe i'll submit an rpcthread status info command.
 952015-10-19T02:56:42  <kanzure> s/killed/pegged? what's the right term
 962015-10-19T02:57:30  *** jgarzik has joined #bitcoin-core-dev
 972015-10-19T02:57:30  *** jgarzik has joined #bitcoin-core-dev
 982015-10-19T03:01:19  *** maaku has quit IRC
 992015-10-19T03:02:34  *** maaku has joined #bitcoin-core-dev
1002015-10-19T03:02:58  *** maaku is now known as Guest67646
1012015-10-19T03:11:23  *** d_t has quit IRC
1022015-10-19T03:13:36  *** Guest67646 is now known as maaku
1032015-10-19T03:13:43  *** d_t has joined #bitcoin-core-dev
1042015-10-19T04:13:57  *** Thireus has quit IRC
1052015-10-19T04:16:48  *** Thireus has joined #bitcoin-core-dev
1062015-10-19T04:30:15  *** Thireus has quit IRC
1072015-10-19T04:30:48  *** Thireus has joined #bitcoin-core-dev
1082015-10-19T05:10:23  *** Thireus has quit IRC
1092015-10-19T05:12:29  *** equil_ has quit IRC
1102015-10-19T05:35:34  *** Thireus has joined #bitcoin-core-dev
1112015-10-19T05:38:24  *** dcousens has joined #bitcoin-core-dev
1122015-10-19T05:38:37  <dcousens> jgarzik: not sure what users would use it tbh
1132015-10-19T05:38:42  <dcousens> I know I would for an API perspective
1142015-10-19T05:38:46  <dcousens> So, in my case, automated?
1152015-10-19T05:39:07  <dcousens> And I guess for sites like webbtc, they lightly dress it up in some readable HTML
1162015-10-19T05:40:02  <dcousens> hell, if an addrindex existed, a block explorer would solely be a HTML dress up of the bitcoind REST API ha
1172015-10-19T05:40:27  <jgarzik> dcousens, yeah the 'GET /foo' case is more likely to be used manually, versus PUT/POST
1182015-10-19T05:40:52  <dcousens> jgarzik: probably a more important question is,  what is the default
1192015-10-19T05:40:58  <dcousens> that will dictate how easy it is for curl users
1202015-10-19T05:41:17  <dcousens> And even then, content type isn't that hard for curl users?
1212015-10-19T05:47:27  <dcousens> I feel like most users might do something along the lines of     'bitcoin-tx ... | curl 'http://somepopularnode.com/tx' --data-raw -H "Content-Type:application/octet-stream"
1222015-10-19T05:47:52  *** challisto has quit IRC
1232015-10-19T05:47:54  <dcousens> or application/json if `-json` was used
1242015-10-19T05:48:24  <dcousens> Probably easiest to just specify the format to whatever the default is, and have `-json` or whatever on bitcoin-tx though
1252015-10-19T05:52:29  *** paveljanik has joined #bitcoin-core-dev
1262015-10-19T06:30:37  *** Thireus has quit IRC
1272015-10-19T06:35:35  *** baldur has quit IRC
1282015-10-19T06:48:31  *** baldur has joined #bitcoin-core-dev
1292015-10-19T06:49:28  *** Ylbam has joined #bitcoin-core-dev
1302015-10-19T06:52:01  <wumpus> kanzure: rescan blocks all your RPC threads, on aquiring cs_main
1312015-10-19T06:53:10  <wumpus> kanzure: long-running RPC commands tend to block the rest, apart from gettxoutsetinfo which I changed at some point to not hold locks (as it runs on a snapshot of the database)
1322015-10-19T06:54:34  <wumpus> probably in rescan the lock could be moved down too, and only aquired when block database structures are accessed
1332015-10-19T06:56:56  *** d_t has quit IRC
1342015-10-19T07:03:21  *** Thireus has joined #bitcoin-core-dev
1352015-10-19T07:35:55  *** equil_ has joined #bitcoin-core-dev
1362015-10-19T08:02:20  *** paveljanik has quit IRC
1372015-10-19T08:15:51  <Luke-Jr> gmaxwell: caching ordered tx list is faster than removing smart time. good enough? :x
1382015-10-19T08:16:52  <gmaxwell> Luke-Jr: so what happens when the wallet is huge?
1392015-10-19T08:17:20  <gmaxwell> then we end up with this gigantic tx list cache in ram. :(
1402015-10-19T08:17:41  <Luke-Jr> eh, right now we store all the tx anyway..
1412015-10-19T08:17:53  <Luke-Jr> is storing a list of pointers all that much more?
1422015-10-19T08:19:48  <gmaxwell> I guess not, probably only another three gigabytes of ram with the whole blockchain in your wallet.
1432015-10-19T08:20:09  <gmaxwell> I'm not super keen on this whole issue existing though. :-/
1442015-10-19T08:25:57  <gmaxwell> Re: 6844 I am twitching a bit about more things being added to the rest api, while little effort has been performed to gain strong confidence that it's not a remote code execution vector. IIRC, we were able to crash the node via it when it went in and we addressed that by disabling it by default. I absolutely don't want to crap on 6844 itself, as it looks like a great feature. But we are making
1452015-10-19T08:26:03  <gmaxwell> an error if we continue to offer a potentially insecure external interface and make it more and more attractive to turn it on and expose it.
1462015-10-19T08:29:21  <gmaxwell> (keep in mind that even if it is localhost only; a local webbrowser has almost free reign over it.)
1472015-10-19T08:32:44  <phantomcircuit> whoa what?
1482015-10-19T08:32:49  <phantomcircuit> you can remotely crash the rest api?
1492015-10-19T08:35:06  <gmaxwell> phantomcircuit: Could initially. I'm unsure if it's fixed, it's a huge attack surface in any case-- we don't have any fuzzing harness for it either.
1502015-10-19T08:35:34  <gmaxwell> (the prior crash I was aware of was just send it an infinityish sized request...)
1512015-10-19T08:55:25  <wumpus> it's relatively safe as long as the JSON parser is not used for input data, and if there is an input argument it is just a simple linear string directly part of the URI, with straightforward validation
1522015-10-19T08:56:08  <wumpus> the crash was a case where JSON spirit was used to parse input data query, which had a known out of memory crash if you make structures too deep
1532015-10-19T08:56:51  <gmaxwell> heh. I found a (local) buffer overflow bug in wget while trying to crash it.
1542015-10-19T08:56:54  <wumpus> IMO REST is for 'trivial, quickly answered, traightforward queries', not where you need to pass in JSON encoded monstriosities
1552015-10-19T08:57:30  <jonasschnelli> wumpus, yes. Probably a good idea. What about maximum length of URLs in case of submit tx?
1562015-10-19T08:57:32  <wumpus> gmaxwell: that could still be interesting; some scripts are bound to pass remote input to wget's parameters
1572015-10-19T08:57:48  <wumpus> jonasschnelli: there definitely should be a limit!
1582015-10-19T08:58:23  <jonasschnelli> wumpus: right. But it might be, that http clients or proxys are using limits lower then the user might expect.
1592015-10-19T08:58:35  <jonasschnelli> I think for large tx /GET is not suitable.
1602015-10-19T08:58:43  * wumpus wishes he had more time to hunt for vulnerabilities in bitcoind's codebase
1612015-10-19T08:58:49  <jonasschnelli> POST with hex/bin should be supported IMO
1622015-10-19T08:59:38  <wumpus> using GET to provide input for tx submission is very un-restful
1632015-10-19T08:59:39  <jonasschnelli> most webservers (and probably proxys) have a limit of 8KB for URL (HTTP GET) length
1642015-10-19T08:59:54  <wumpus> should be PUT?
1652015-10-19T09:00:09  <wumpus> or yes, POST
1662015-10-19T09:00:14  <jonasschnelli> wumpus: agree. PUT or POST would be better. But there comes in the question of supporting JSON... which is a security risk.
1672015-10-19T09:00:19  <wumpus> no, no JSON
1682015-10-19T09:00:19  <jonasschnelli> Maybe just support BIN/HEX
1692015-10-19T09:00:22  <wumpus> just raw POST data
1702015-10-19T09:00:23  <jonasschnelli> (or only BIN)
1712015-10-19T09:00:32  <jonasschnelli> wumpus: agree.
1722015-10-19T09:01:51  <wumpus> JSON  is ok for returning data in some cases, but should not be *parsed* in REST
1732015-10-19T09:02:06  <Luke-Jr> can we parse plain English language instead?
1742015-10-19T09:02:07  * Luke-Jr hides
1752015-10-19T09:02:36  <wumpus> Luke-Jr: http://inform7.com/ ? :p
1762015-10-19T09:05:16  <gmaxwell> please don't do the rest submittx at all if its only bin, if it won't take hex it will be massively less used, and we'll just have more code to hang around and bloat the codebase; while people will continue to eschew bitcoin core and use some web api that actually gives an interface people expect. :-/
1772015-10-19T09:05:50  <gmaxwell> okay, I feel better now-- I have a whole bunchof fuzzers running against it at the moment and it's not immediately dying.
1782015-10-19T09:05:55  <gmaxwell> \O/
1792015-10-19T09:05:59  <jonasschnelli> gmaxwell: agree. Hex should be supported in the first place.
1802015-10-19T09:06:52  <wumpus> hex is fine, just don't add a zillion kinds of input formats...
1812015-10-19T09:07:04  <wumpus> each one adds more attack surface
1822015-10-19T09:07:05  <Luke-Jr> tonal pls
1832015-10-19T09:07:13  <wumpus> hah...
1842015-10-19T09:07:22  <Luke-Jr> (no, tonal isn't used for binary data..)
1852015-10-19T09:08:29  <Luke-Jr> is -connect=0 the best way to avoid any external interactions with p2p?
1862015-10-19T09:08:38  <wumpus> -connect=0.0.0.0
1872015-10-19T09:08:46  <Luke-Jr> k
1882015-10-19T09:09:05  <wumpus> -connect=0 will work too, but will spam your debug.log with errors
1892015-10-19T09:10:05  <wumpus> but yes I feel bad about adding more and more to the REST interface as well. Initially it was meant to just request data, making it possible to submit transactions unauthenticated is a different thing.
1902015-10-19T09:10:52  <wumpus> I understand why it's useful of course...
1912015-10-19T09:12:01  <Luke-Jr> IMO it will get horridly abused
1922015-10-19T09:12:14  <Luke-Jr> when Eligius offered such a thing, it got hard-coded into spam scripts
1932015-10-19T09:13:29  <wumpus> you're not *supposed* to offer REST interface to the open internet, but lacking auth it may be easy to do accidentally (same as zillions of other critical systems are accidentally on the internet)
1942015-10-19T09:13:49  <gmaxwell> I dunno if there is any real reason to do anything but hex.
1952015-10-19T09:13:49  <gmaxwell> I suppose if block submission was also supported then the hex overhead would be pretty unwelcome.
1962015-10-19T09:13:49  <gmaxwell> and for that we'd want binary data. so there might be a uniformity argument.
1972015-10-19T09:13:49  *** gmaxwell has quit IRC
1982015-10-19T09:13:57  *** gmaxwell has joined #bitcoin-core-dev
1992015-10-19T09:14:18  *** Guest52664 has quit IRC
2002015-10-19T09:14:21  *** gmaxwell is now known as Guest89849
2012015-10-19T09:14:34  *** petertodd has joined #bitcoin-core-dev
2022015-10-19T09:14:57  *** petertodd is now known as Guest82934
2032015-10-19T09:15:13  <wumpus> binary is the least risky to accept, I don't think there's anything to be won by not accepting that
2042015-10-19T09:15:49  <wumpus> then again I don't really care, if you want to make it hex-only, be my guest, just don't add JSON parsing
2052015-10-19T09:16:18  *** BlueMatt has quit IRC
2062015-10-19T09:16:47  <wumpus> (or any encoding more involved than straightforward encode-two-characters-to-one like hex)
2072015-10-19T09:20:51  *** BlueMatt has joined #bitcoin-core-dev
2082015-10-19T09:24:17  <GitHub21> [bitcoin] luke-jr opened pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851
2092015-10-19T09:26:43  <wumpus> Luke-Jr: why is -zapwallettxes performance a goal?
2102015-10-19T09:27:20  <wumpus> isn't that a troubleshooting option that should be quite rarely used?
2112015-10-19T09:28:43  <Luke-Jr> wumpus: I don't know, gmaxwell was annoyed it was taking some guy with a big wallet 9 hours. Apparently it's more common because we don't handle malleability right still.
2122015-10-19T09:29:15  <Luke-Jr> wumpus: but right now, it sorts the entire wallet's transactions every time it adds a new transaction to the wallet. so it's pretty bad.
2132015-10-19T09:29:47  <GitHub48> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d78a880900c1...5888046323dd
2142015-10-19T09:29:47  <GitHub48> bitcoin/master d2e3fbc Thomas Kerin: Mention PHP bindings to libbitcoinconsensus
2152015-10-19T09:30:19  <GitHub48> bitcoin/master 5888046 Wladimir J. van der Laan: Merge pull request #6849...
2162015-10-19T09:30:19  <GitHub130> [bitcoin] laanwj closed pull request #6849: Mention PHP bindings to libbitcoinconsensus (master...bitcoinconsensus-php-bindings) https://github.com/bitcoin/bitcoin/pull/6849
2172015-10-19T09:30:53  <wumpus> we had a RPC pull request at some point where it was possible to remove conflicted transactions one-by-one, I greatly preferred that to the hammer that is zapwallettxes
2182015-10-19T09:31:55  <wumpus> it's pretty bad but does it warrant using more memory even in the cases where people don't use zapwallettxes?
2192015-10-19T09:32:23  <wumpus> I don't think so - memory usage of bitcoind is already famously bad
2202015-10-19T09:32:49  <Luke-Jr> for the wallet? I've only heard complaints when people fail to configure their mempool policy
2212015-10-19T09:33:09  <wumpus> sorry to be grumpy about this but there's so many people complaining about that
2222015-10-19T09:33:15  <wumpus> well the entire wallet is kept in memory at alll times
2232015-10-19T09:34:33  <Luke-Jr> yes, that's why I didn't think the overhead from this would be significant ;)
2242015-10-19T09:35:32  <Luke-Jr> (my hot wallet has been in regularish use since 2009 and hasn't reached 40 MB yet..)
2252015-10-19T09:35:39  <wumpus> i don't like "it's already bad, so making it any worse doesn't matter" arguments, but ok...
2262015-10-19T09:35:39  <Luke-Jr> 2010*
2272015-10-19T09:36:33  <dcousens> IMHO, I'd argue the REST API shouldn't exist at all
2282015-10-19T09:36:39  <dcousens> Its trivial to expose it via the RPC if you want to do that
2292015-10-19T09:37:56  <wumpus> that was my argument too; it could just as well have been an external script that wraps RPC and proides a new API
2302015-10-19T09:37:56  <dcousens> For the same reasons Guest89849 (gmaxwell) mentioned before, its just a huge surface vector for limited gain
2312015-10-19T09:38:07  <wumpus> but who listens to me...
2322015-10-19T09:38:12  <Luke-Jr> wumpus: the benefits IMO significantly outweigh the costs. someone would need to be running some major service provider to use a lot of memory for the wallet, and that use case is just out of reach for bitcoind's wallet without any kind of major rewrite
2332015-10-19T09:38:36  <wumpus> not everything has to be in the monolithic bitcoind executable
2342015-10-19T09:38:36  <Luke-Jr> even aside from zap, it improves performance for listtransactions RPC etc
2352015-10-19T09:42:00  <wumpus> Luke-Jr: oh if it helps perfomance in other places too you should definitely mention that!
2362015-10-19T09:42:17  <dcousens> wumpus: Luke-Jr has mentioned a few times now people writing their own mempool policy, which is fine, and maybe including stricter default policies might be slightly controversial (too restrictive etc),  but,  maybe some pre-written policies that users can easily enable that have some sane defaults would be beneficial?
2372015-10-19T09:42:18  <wumpus> Luke-Jr: I wasn't aware, and just 'it improves zapwallettx' is not a big selling point to me
2382015-10-19T09:43:36  <Luke-Jr> k, added it to the summary. looks like it's *just* listtransactions
2392015-10-19T09:43:44  <wumpus> dcousens: if someone wants to write and test them, I don't see why not, there are some pulls that factor out the mempool policy stuff which makes it easier to create custom ones, at least in theory
2402015-10-19T09:44:02  <dcousens> wumpus: I mean, I guess that would be the ideal case right?
2412015-10-19T09:44:13  <dcousens> Everyone configuring their policy to what they are happy with
2422015-10-19T09:44:29  <wumpus> dcousens: the ideal case - from the network view - is that everyone runs the same, predictable policy
2432015-10-19T09:44:41  *** paveljanik has joined #bitcoin-core-dev
2442015-10-19T09:44:57  <dcousens> wumpus: true, but varying hardware requirements mean that is difficult to cater for
2452015-10-19T09:45:13  <dcousens> So, I guess, you could go the highly configurable policy route
2462015-10-19T09:45:30  <wumpus> dcousens: if people insist on using their own mempool policy on their miner there's no way to prevent that, and arguably it shouldn't be made more difficult than necessary to prevent stupid issues from patching
2472015-10-19T09:45:42  <dcousens> Or what seems to be the current direction, scalable policy which matches your hardware (if configured)
2482015-10-19T09:45:52  <wumpus> dcousens: but I'm not sure it makes sense to talk about the ideal case there
2492015-10-19T09:46:04  <wumpus> how does hardware determine the policy? just how much memory available?
2502015-10-19T09:46:14  <dcousens> wumpus: yes
2512015-10-19T09:46:27  <wumpus> if so, adding a parameter to set the amount of memory used (with as little impact otherwise) would be preferable
2522015-10-19T09:46:43  <Luke-Jr> wumpus: everyone running the same predictable policy necessarily implies centralisation and censorship risk
2532015-10-19T09:46:46  <wumpus> that's the whole point of the mempool limiting
2542015-10-19T09:47:05  <wumpus> add limiting but don't mess up the network too much or introduce DDoSes
2552015-10-19T09:47:38  <wumpus> I don't see the point of 'highly configurable policy' just to work around hardware limitations
2562015-10-19T09:47:43  <dcousens> Luke-Jr: only if the default policy is restrictive right?
2572015-10-19T09:48:30  <Luke-Jr> dcousens: which it must be
2582015-10-19T09:48:43  <dcousens> Luke-Jr: or do you mean, encouraging people just to use the bundled policy is the centralizatoin risk?
2592015-10-19T09:48:53  <dcousens> (as it seemed you implied to me last night)
2602015-10-19T09:49:00  <wumpus> Luke-Jr: censorship risk? that's a strawman, I'm talking a about a *sane* default policy, not some blacklist hellscape
2612015-10-19T09:49:01  <Luke-Jr> dcousens: I was responding to wumpus saying ideal is everyone on the same policy.
2622015-10-19T09:49:52  <wumpus> if bitcoin core starts to include code that censors specific transactions/senders, start running very fast
2632015-10-19T09:49:53  <Luke-Jr> wumpus: a unified policy means there is someone making the decisions
2642015-10-19T09:50:18  <dcousens> Luke-Jr: you could say the same thing about a unified protocol
2652015-10-19T09:51:34  *** zveda has joined #bitcoin-core-dev
2662015-10-19T09:51:49  *** zveda has left #bitcoin-core-dev
2672015-10-19T09:56:35  <wumpus> right, it is about encouraging people to use the same or similar policy for relaying, there is no mechanism to force nodes to use a certain policy. And if someone powerful wanted to censor it'd be much more effective to approach the (heavily centralized) miners than trying to get it merged upstream.
2682015-10-19T10:01:27  <GitHub166> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5888046323dd...fc41278e9689
2692015-10-19T10:01:27  <GitHub166> bitcoin/master 99963b9 Mitchell Cash: Correct spelling mistakes in doc folder...
2702015-10-19T10:01:28  <GitHub166> bitcoin/master fc41278 Wladimir J. van der Laan: Merge pull request #6843...
2712015-10-19T10:01:37  <GitHub177> [bitcoin] laanwj closed pull request #6843: Trivial: Correct spelling mistakes in doc folder (master...spelling_fixes) https://github.com/bitcoin/bitcoin/pull/6843
2722015-10-19T10:13:26  <GitHub163> [bitcoin] laanwj opened pull request #6852: build: make sure OpenSSL heeds noexecstack (master...2015_10_openssl_noexecstack) https://github.com/bitcoin/bitcoin/pull/6852
2732015-10-19T10:15:50  <GitHub98> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/fc41278e9689...5f5ee0f3ce6b
2742015-10-19T10:15:51  <GitHub98> bitcoin/master 0658d0b Alex Morcos: temporarily comment out rpcbind.py...
2752015-10-19T10:15:51  <GitHub98> bitcoin/master 5f5ee0f Wladimir J. van der Laan: Merge pull request #6837...
2762015-10-19T10:16:00  <GitHub134> [bitcoin] laanwj closed pull request #6837: Cleanup rpc-tests.py (master...fixupRPCtests) https://github.com/bitcoin/bitcoin/pull/6837
2772015-10-19T10:44:08  <GitHub199> [bitcoin] laanwj closed pull request #6162: Use 512-bit arithmetic for difficulty retarget calculation. (master...regtest_consensus_fix) https://github.com/bitcoin/bitcoin/pull/6162
2782015-10-19T10:48:02  *** murr4y has joined #bitcoin-core-dev
2792015-10-19T10:54:11  *** equil_ has quit IRC
2802015-10-19T10:56:42  *** Thireus has quit IRC
2812015-10-19T10:56:56  *** Thireus has joined #bitcoin-core-dev
2822015-10-19T11:20:30  *** randy-waterhouse has joined #bitcoin-core-dev
2832015-10-19T11:58:56  *** ParadoxSpiral has joined #bitcoin-core-dev
2842015-10-19T12:13:53  *** randy-waterhouse has quit IRC
2852015-10-19T12:26:36  <GitHub124> [bitcoin] CodeShark opened pull request #6853: Added fNoRetargeting field to Consensus::Params (master...fNoRetargeting) https://github.com/bitcoin/bitcoin/pull/6853
2862015-10-19T12:39:20  <CodeShark> why are the travis tests failing on windows? https://travis-ci.org/bitcoin/bitcoin/jobs/86161511
2872015-10-19T12:40:31  <btcdrak> Trolling for reviews/ACKs please https://github.com/bitcoin/bitcoin/pull/6566 (Median past locktime)
2882015-10-19T12:44:53  *** malte has quit IRC
2892015-10-19T12:47:09  *** malte has joined #bitcoin-core-dev
2902015-10-19T13:01:56  <GitHub52> [bitcoin] laanwj opened pull request #6854: devtools: Add security-check.py (master...2015_10_security_checks) https://github.com/bitcoin/bitcoin/pull/6854
2912015-10-19T13:03:21  <GitHub166> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5f5ee0f3ce6b...da7d57fb9501
2922015-10-19T13:03:21  <GitHub166> bitcoin/master bfcdc21 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
2932015-10-19T13:03:22  <GitHub166> bitcoin/master da7d57f Wladimir J. van der Laan: Merge pull request #6852...
2942015-10-19T13:03:27  <GitHub39> [bitcoin] laanwj closed pull request #6852: build: make sure OpenSSL heeds noexecstack (master...2015_10_openssl_noexecstack) https://github.com/bitcoin/bitcoin/pull/6852
2952015-10-19T13:05:11  <GitHub136> [bitcoin] laanwj pushed 1 new commit to 0.11: https://github.com/bitcoin/bitcoin/commit/21e58b8e3544987db0bdaa91c23bb398e655bd21
2962015-10-19T13:05:12  <GitHub136> bitcoin/0.11 21e58b8 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
2972015-10-19T13:05:49  <GitHub70> [bitcoin] laanwj pushed 1 new commit to 0.10: https://github.com/bitcoin/bitcoin/commit/0b3fd07fd24a96b205d4f85d00157d5f115fad9e
2982015-10-19T13:05:50  <GitHub70> bitcoin/0.10 0b3fd07 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
2992015-10-19T13:09:46  <kanzure> wumpus: for rescan locking, would be nice to show lock status in another rpc command.
3002015-10-19T13:09:53  <kanzure> lock statuses per thread or something
3012015-10-19T13:20:24  <wumpus> meh, I don't think debugging information belongs on RPC
3022015-10-19T13:20:54  <wumpus> at least not something that low level - can't gdb show locking debug information?
3032015-10-19T13:22:24  <wumpus> I guess it could be something explicitly enabled with --enable-debug like DEBUG_LOGCONTENTION
3042015-10-19T13:22:32  <wumpus> LOCK*
3052015-10-19T13:27:41  <GitHub47> [bitcoin] laanwj closed pull request #6836: Bring historical release notes up to date (0.10...0.10-add-release-notes) https://github.com/bitcoin/bitcoin/pull/6836
3062015-10-19T13:27:41  <GitHub169> [bitcoin] laanwj pushed 2 new commits to 0.10: https://github.com/bitcoin/bitcoin/compare/0b3fd07fd24a...7e9a9874f3cc
3072015-10-19T13:27:41  <GitHub169> bitcoin/0.10 fb818b6 Micha: Bring historical release notes up to date...
3082015-10-19T13:27:42  <GitHub169> bitcoin/0.10 7e9a987 Wladimir J. van der Laan: Merge pull request #6836...
3092015-10-19T13:29:35  *** dcousens has quit IRC
3102015-10-19T14:00:23  *** jgarzik has quit IRC
3112015-10-19T14:00:38  *** jgarzik has joined #bitcoin-core-dev
3122015-10-19T14:00:38  *** jgarzik has joined #bitcoin-core-dev
3132015-10-19T14:10:08  *** Guest82934 is now known as petertodd
3142015-10-19T14:20:45  *** randy-waterhouse has joined #bitcoin-core-dev
3152015-10-19T14:23:27  <wumpus> the more I think about it the more I think we should have implemented REST as a separate script that wraps the RPC interface... I really don't like how another interface to the same things is converging more and more on RPC's functionality
3162015-10-19T14:24:51  <btcdrak> ^
3172015-10-19T14:25:25  <btcdrak> wumpus: could still be changed that way now
3182015-10-19T14:26:05  <wumpus> yes
3192015-10-19T14:28:58  <wumpus> with libevent's http, and univalue-as-as-library it shouldn't be that difficult at all
3202015-10-19T14:31:04  <wumpus> (even to do it in C wouldn't, I mean, in python it's obviously easy)
3212015-10-19T14:32:53  *** treehug88 has joined #bitcoin-core-dev
3222015-10-19T14:37:01  <btcdrak> wumpus: so really, if you want this to be done you'll have to refuse to merge new duplicate REST methods until the wrapper is done.
3232015-10-19T14:37:32  <wumpus> if I want it to be done I'll have to do it myself...
3242015-10-19T15:05:06  *** Thireus has quit IRC
3252015-10-19T15:14:02  *** paveljanik has quit IRC
3262015-10-19T15:14:19  *** d_t has joined #bitcoin-core-dev
3272015-10-19T15:15:35  *** malte has quit IRC
3282015-10-19T15:21:41  *** malte has joined #bitcoin-core-dev
3292015-10-19T15:22:18  *** Thireus has joined #bitcoin-core-dev
3302015-10-19T16:00:01  *** d_t has quit IRC
3312015-10-19T16:06:14  *** d_t has joined #bitcoin-core-dev
3322015-10-19T16:18:32  *** Guest89849 is now known as gmaxwell
3332015-10-19T16:18:33  *** gmaxwell has joined #bitcoin-core-dev
3342015-10-19T16:23:26  <GitHub73> [bitcoin] lclc opened pull request #6855: [REST] API versioning (master...restVersioning) https://github.com/bitcoin/bitcoin/pull/6855
3352015-10-19T16:28:24  *** sdaftuar has quit IRC
3362015-10-19T16:29:01  *** jcorgan has quit IRC
3372015-10-19T16:29:02  *** isis has quit IRC
3382015-10-19T16:29:37  *** petertodd has quit IRC
3392015-10-19T16:29:37  *** kanzure has quit IRC
3402015-10-19T16:30:08  *** sdaftuar has joined #bitcoin-core-dev
3412015-10-19T16:30:53  *** petertodd has joined #bitcoin-core-dev
3422015-10-19T16:31:03  *** kanzure has joined #bitcoin-core-dev
3432015-10-19T16:31:17  *** petertodd is now known as Guest98736
3442015-10-19T16:35:01  *** jcorgan has joined #bitcoin-core-dev
3452015-10-19T16:35:02  *** jcorgan has joined #bitcoin-core-dev
3462015-10-19T16:37:52  *** isis has joined #bitcoin-core-dev
3472015-10-19T16:55:19  *** maaku has quit IRC
3482015-10-19T16:56:01  <gmaxwell> wumpus: I like that too.
3492015-10-19T16:56:44  *** maaku has joined #bitcoin-core-dev
3502015-10-19T16:57:07  *** maaku is now known as Guest17159
3512015-10-19T17:33:31  *** paveljanik has joined #bitcoin-core-dev
3522015-10-19T17:33:33  *** paveljanik has joined #bitcoin-core-dev
3532015-10-19T17:53:58  *** Guest17159 is now known as maaku
3542015-10-19T18:29:09  *** maaku has quit IRC
3552015-10-19T18:30:32  *** maaku has joined #bitcoin-core-dev
3562015-10-19T18:30:52  *** maaku is now known as Guest13301
3572015-10-19T18:31:21  *** Guest13301 is now known as maaku
3582015-10-19T19:28:59  *** CodeShark has quit IRC
3592015-10-19T19:32:09  *** belcher has joined #bitcoin-core-dev
3602015-10-19T19:38:40  <GitHub130> [bitcoin] morcos opened pull request #6856: Do not allow block file pruning during reindex. (master...noPruneDuringReindex) https://github.com/bitcoin/bitcoin/pull/6856
3612015-10-19T19:41:10  *** maaku__ has joined #bitcoin-core-dev
3622015-10-19T19:44:23  <GitHub123> [bitcoin] morcos opened pull request #6857: Require nLastBlockFile to be the highest numbered file. (master...nLastBlockFile) https://github.com/bitcoin/bitcoin/pull/6857
3632015-10-19T20:08:24  *** randy-waterhouse has quit IRC
3642015-10-19T20:16:12  *** ParadoxSpiral has quit IRC
3652015-10-19T20:49:00  *** paveljanik has quit IRC
3662015-10-19T21:07:05  *** midnightmagic has quit IRC
3672015-10-19T21:10:48  *** Thireus has quit IRC
3682015-10-19T21:12:58  *** midnightmagic has joined #bitcoin-core-dev
3692015-10-19T21:31:07  *** treehug88 has quit IRC
3702015-10-19T21:40:46  *** midnightmagic_ has joined #bitcoin-core-dev
3712015-10-19T21:43:48  *** midnightmagic has quit IRC
3722015-10-19T21:44:06  *** midnightmagic_ is now known as midnightmagic
3732015-10-19T23:54:18  *** Ylbam has quit IRC