12015-11-14T00:06:51  *** squidicuz has quit IRC
  22015-11-14T00:24:22  <gmaxwell> wumpus: jonasschnelli:  Care to give me a quick design sniff test?  I'm thinking of having someone implement functionality for external signers in Bitcoin core along these lines, http://0bin.net/paste/7HCkLWNJBDqJB7Up#hPgdwfs06m7FJw-WQZUCBNTNfCZnrix2K0Mo++6jpbH
  32015-11-14T00:25:05  <gmaxwell> I realize that some might want a more extensive wallet redesign, but I think this would be good incremental progress and would enable a _lot_ of flexibility for minimal code impact.
  42015-11-14T00:29:14  *** Squidicuz has joined #bitcoin-core-dev
  52015-11-14T01:13:20  *** JackH has quit IRC
  62015-11-14T01:45:32  *** d_t has quit IRC
  72015-11-14T01:53:33  *** ParadoxSpiral has quit IRC
  82015-11-14T01:54:27  *** Ylbam has quit IRC
  92015-11-14T02:16:52  *** dgenr8 has quit IRC
 102015-11-14T02:21:39  *** randy-waterhouse has joined #bitcoin-core-dev
 112015-11-14T03:52:53  *** PaulCape_ has joined #bitcoin-core-dev
 122015-11-14T03:55:51  *** PaulCapestany has quit IRC
 132015-11-14T04:00:26  *** randy-waterhouse has quit IRC
 142015-11-14T04:00:26  *** dcousens has joined #bitcoin-core-dev
 152015-11-14T04:33:29  *** PaulCape_ has quit IRC
 162015-11-14T04:33:50  *** PaulCapestany has joined #bitcoin-core-dev
 172015-11-14T04:37:57  *** d_t has joined #bitcoin-core-dev
 182015-11-14T05:00:14  *** tulip has joined #bitcoin-core-dev
 192015-11-14T05:09:43  *** kanzure_ has joined #bitcoin-core-dev
 202015-11-14T05:09:45  *** nkuttler_ has joined #bitcoin-core-dev
 212015-11-14T05:10:26  *** petertod1 has joined #bitcoin-core-dev
 222015-11-14T05:10:52  *** tripleslash_l has joined #bitcoin-core-dev
 232015-11-14T05:14:25  *** kanzure has quit IRC
 242015-11-14T05:14:25  *** tripleslash has quit IRC
 252015-11-14T05:14:26  *** nkuttler has quit IRC
 262015-11-14T05:14:26  *** bsm117532 has quit IRC
 272015-11-14T05:14:27  *** petertodd has quit IRC
 282015-11-14T05:14:30  *** nkuttler_ is now known as nkuttler
 292015-11-14T05:15:37  *** bsm117532 has joined #bitcoin-core-dev
 302015-11-14T05:17:50  <GitHub147> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/dbd2c135ddb9...e0a5ef84272b
 312015-11-14T05:17:51  <GitHub147> bitcoin/master 61e1eb2 Peter Todd: Actually use includeWatching value in fundrawtransaction...
 322015-11-14T05:17:51  <GitHub147> bitcoin/master 10953a7 Peter Todd: Better error message for fundrawtransaction w/ empty vout...
 332015-11-14T05:17:52  <GitHub147> bitcoin/master e0a5ef8 Gregory Maxwell: Merge pull request #7010...
 342015-11-14T05:18:00  <GitHub67> [bitcoin] gmaxwell closed pull request #7010: Fix fundrawtransaction handling of includeWatching (master...2015-11-fix-fundrawtransaction-bugs) https://github.com/bitcoin/bitcoin/pull/7010
 352015-11-14T05:31:59  *** jtimon has quit IRC
 362015-11-14T05:35:21  *** dcousens has quit IRC
 372015-11-14T05:48:08  *** luke-jr_ has joined #bitcoin-core-dev
 382015-11-14T05:49:15  *** pmienk_ has joined #bitcoin-core-dev
 392015-11-14T05:50:51  *** michagogo_ has joined #bitcoin-core-dev
 402015-11-14T05:51:02  *** nkuttler_ has joined #bitcoin-core-dev
 412015-11-14T05:51:03  *** michagogo has quit IRC
 422015-11-14T05:51:04  *** lclc has quit IRC
 432015-11-14T05:51:05  *** teward has quit IRC
 442015-11-14T05:51:06  *** berndj has quit IRC
 452015-11-14T05:51:06  *** baldur has quit IRC
 462015-11-14T05:51:08  *** phantomcircuit has quit IRC
 472015-11-14T05:51:08  *** Luke-Jr has quit IRC
 482015-11-14T05:51:09  *** pmienk has quit IRC
 492015-11-14T05:51:11  *** zmanian_ has quit IRC
 502015-11-14T05:51:11  *** warren has quit IRC
 512015-11-14T05:51:14  *** midnightmagic has quit IRC
 522015-11-14T05:51:15  *** nkuttler has quit IRC
 532015-11-14T05:51:16  *** nkuttler_ is now known as nkuttler
 542015-11-14T05:51:43  *** berndj has joined #bitcoin-core-dev
 552015-11-14T05:52:14  *** zmanian_ has joined #bitcoin-core-dev
 562015-11-14T05:52:45  *** midnightmagic has joined #bitcoin-core-dev
 572015-11-14T05:52:47  *** warren has joined #bitcoin-core-dev
 582015-11-14T05:52:58  *** michagogo_ is now known as michagogo
 592015-11-14T05:53:08  *** lclc has joined #bitcoin-core-dev
 602015-11-14T05:53:08  *** baldur has joined #bitcoin-core-dev
 612015-11-14T05:54:39  *** phantomcircuit has joined #bitcoin-core-dev
 622015-11-14T05:57:17  *** teward has joined #bitcoin-core-dev
 632015-11-14T05:59:38  <phantomcircuit> gmaxwell, thoughts on a "-miner" mode that does things like reduce (or even remove) the sleep calls in some of the networking threads?
 642015-11-14T05:59:56  <phantomcircuit> (ie please do everything possible to reduce latency even if you spin my cpu at 100% constantly)
 652015-11-14T06:07:53  <gmaxwell> phantomcircuit: what sleep calls are there at all in networking? (other than in e.g. connect loops, which are needed to avoid dos attacking peers)
 662015-11-14T06:08:41  *** luke-jr_ is now known as Luke-Jr
 672015-11-14T06:09:22  <Luke-Jr> we probably should have a miner mode regardless. relay nodes have no real need for the mempool.
 682015-11-14T06:09:38  <Luke-Jr> (they just need to relay and forget, plus some txid record so they can't be DoS'd)
 692015-11-14T06:10:42  <phantomcircuit> Luke-Jr, please define relay node more specifically
 702015-11-14T06:12:00  *** d_t has quit IRC
 712015-11-14T06:12:06  <Luke-Jr> phantomcircuit: a node that wishes to participate (non-leech) in the p2p network, but has no miners
 722015-11-14T06:12:37  <phantomcircuit> Luke-Jr, they require a mempool if they are going to forward transactions otherwise they become a DoS amplifier
 732015-11-14T06:12:42  <phantomcircuit> gmaxwell, there's a sleep on select() error (im not even sure why that's there actually)
 742015-11-14T06:12:58  <Luke-Jr> phantomcircuit: why can't they just remember txids?
 752015-11-14T06:13:36  <phantomcircuit> and there's messageHandlerCondition which can be replaced by a spin lock
 762015-11-14T06:14:09  <phantomcircuit> there's also a bunch of other things like changing the defaults for various things to be much much larger
 772015-11-14T06:14:39  <gmaxwell> Luke-Jr: forgetting things is not so useful for avoiding the 2x++ redundant transmission when the same data shows up again in a block.
 782015-11-14T06:15:55  <phantomcircuit> gmaxwell, i'd also add a background thread that just calls CreateNewBlock in a loop trying to improve on the current cached block template
 792015-11-14T06:16:16  <Luke-Jr> gmaxwell: hm, that's a point
 802015-11-14T06:16:39  <Luke-Jr> phantomcircuit: merely calling CNB won't use/update the cache
 812015-11-14T06:16:53  <phantomcircuit> Luke-Jr, yes i know
 822015-11-14T06:17:09  * phantomcircuit hand waves details
 832015-11-14T06:24:41  <phantomcircuit> Luke-Jr, it'll update every 5 seconds if the memory pool was modified, correct?
 842015-11-14T06:24:52  <phantomcircuit> (or the tip changes obviously)
 852015-11-14T06:25:27  <Luke-Jr> phantomcircuit: the cache is in the RPC code
 862015-11-14T06:25:35  <phantomcircuit> yes i know
 872015-11-14T06:25:55  <phantomcircuit> that's the logic though right? (just verifying i read it correctly)
 882015-11-14T06:27:01  <gmaxwell> phantomcircuit: I was recommending before that we change the gbt behavior to _never_ run createnewblock at the time the rpc is called.
 892015-11-14T06:27:30  <phantomcircuit> gmaxwell, yes that's what i would be implementing, a background thread only
 902015-11-14T06:29:07  <phantomcircuit> there's some interesting optimizations that can be done if we're willing to spend a bunch of cpu time on it
 912015-11-14T06:30:02  <phantomcircuit> the obvious one is optimizing for the block that pays the highest fee
 922015-11-14T06:30:10  <phantomcircuit> (or whatever)
 932015-11-14T06:44:01  <wumpus> gmaxwell: concept ACK
 942015-11-14T06:44:35  <wumpus> gmaxwell:  I'd personally prefer not adding more 'invoke external programs', I know we already have walletnotify etc but it makes sandboxing the thing harder because we can't disable fork/exec
 952015-11-14T06:45:24  <wumpus> gmaxwell: I know there is walletnotify, blocknotify etc but at least we have an alternative notification mechanism for that now, so they may be deprecated in some future version
 962015-11-14T06:51:17  <wumpus> gmaxwell: anyhow so I'm not strongly against that, just wary of it, also with regard to argument shell injection. Although at least that an be easily avoided by executing a program directly instead of using ::system
 972015-11-14T06:58:29  <wumpus> an alternative would be to use a little protocol intead for communicating with the signer
 982015-11-14T07:00:09  <gmaxwell> wumpus: hm. we can limit to what we can exec (via seccomp) .. interesting I was thinking the seperate process there was a sandboxing benefit, e.g. because it can have totally distinct selinux / seccomp settings.
 992015-11-14T07:00:26  <wumpus> sure, just be really careful
1002015-11-14T07:00:46  <gmaxwell> But the biggest reason for the seperate process is to tear down development barriers. E.g. so someone can build a module without the scarryness in bitcoin core. :)
1012015-11-14T07:00:46  <wumpus> walletnotify/blocknotify use ::system that has always bugged me
1022015-11-14T07:01:12  <wumpus> sure that's why I dont give "integrate it into bitcoin core" as an alternative, but say communicate through a pipe or socket
1032015-11-14T07:01:50  <gmaxwell> wumpus: still need a way to start the thing that speaks the protocol.
1042015-11-14T07:02:05  <wumpus> could be similar to e.g. torcontrol.cpp which speaks a simple protocol with Tor
1052015-11-14T07:02:10  <gmaxwell> But okay, reasonable.
1062015-11-14T07:02:12  <wumpus> true
1072015-11-14T07:02:18  <wumpus> ok, let's leave that isue for later
1082015-11-14T07:02:47  *** CodeShark_ has joined #bitcoin-core-dev
1092015-11-14T07:02:55  <wumpus> for initial version invoking a process is fine
1102015-11-14T07:03:11  <wumpus> just don't use ::system :D
1112015-11-14T07:03:48  *** pigeons has joined #bitcoin-core-dev
1122015-11-14T07:03:55  *** Guest78100 has quit IRC
1132015-11-14T07:03:59  *** zxzzt has quit IRC
1142015-11-14T07:04:01  *** da2ce7 has quit IRC
1152015-11-14T07:04:12  *** pigeons is now known as Guest25458
1162015-11-14T07:04:39  *** zxzzt has joined #bitcoin-core-dev
1172015-11-14T07:05:44  *** da2ce7 has joined #bitcoin-core-dev
1182015-11-14T07:24:50  <GitHub73> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e0a5ef84272b...36baa9f47587
1192015-11-14T07:24:51  <GitHub73> bitcoin/master b3ae384 Peter Todd: Remove LOCK(cs_main) from decodescript...
1202015-11-14T07:24:51  <GitHub73> bitcoin/master 36baa9f Wladimir J. van der Laan: Merge pull request #7013...
1212015-11-14T07:24:55  <GitHub81> [bitcoin] laanwj closed pull request #7013: Remove LOCK(cs_main) from decodescript (master...2015-11-remove-cs-main-from-decodescript) https://github.com/bitcoin/bitcoin/pull/7013
1222015-11-14T07:26:15  <GitHub151> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/44ac42e50d567b08d3cb3f3c5766588468ce5bbf
1232015-11-14T07:26:16  <GitHub151> bitcoin/master 44ac42e Wladimir J. van der Laan: Merge pull request #7004...
1242015-11-14T07:26:20  <GitHub79> [bitcoin] laanwj closed pull request #7004: update jonasschnellis gpg key (master...2015/11/signing_key_jonasschnelli) https://github.com/bitcoin/bitcoin/pull/7004
1252015-11-14T07:47:52  *** PaulCape_ has joined #bitcoin-core-dev
1262015-11-14T07:50:42  *** PaulCapestany has quit IRC
1272015-11-14T07:55:04  <GitHub18> [bitcoin] gmaxwell pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/44ac42e50d56...9ffc687288dd
1282015-11-14T07:55:05  <GitHub18> bitcoin/master d61fcff Jonas Schnelli: don't enforce maxuploadtargets disconnect for whitelisted peers
1292015-11-14T07:55:05  <GitHub18> bitcoin/master 5760749 Jonas Schnelli: [docs] rename reducetraffic.md to reduce-traffic.md
1302015-11-14T07:55:06  <GitHub18> bitcoin/master e495ed5 Jonas Schnelli: add documentation for exluding whitelistes peer from maxuploadtarget
1312015-11-14T07:55:09  <GitHub44> [bitcoin] gmaxwell closed pull request #6984: don't enforce maxuploadtarget's disconnect for whitelisted peers (master...2015/11/maxupload_whitebind) https://github.com/bitcoin/bitcoin/pull/6984
1322015-11-14T08:01:37  *** Ylbam has joined #bitcoin-core-dev
1332015-11-14T08:05:56  *** paveljanik has joined #bitcoin-core-dev
1342015-11-14T08:05:56  *** paveljanik has joined #bitcoin-core-dev
1352015-11-14T08:07:45  <gmaxwell> wumpus: do you have any advice for phantomcircuit on my DEFAULT_BLOCKSONLY complaint?  My response to him was not terribly constructive... in part because I don't have any good advice.
1362015-11-14T08:08:22  <gmaxwell> I'd have merged it already but for that issue.
1372015-11-14T08:10:30  <wumpus> what is the issue exactly? can't find it in the comments
1382015-11-14T08:11:01  <wumpus> I do see DEFAULT_BLOCKSONLY is not actually used (besides documentation)
1392015-11-14T08:11:14  <wumpus> but what prevents him from doing that?
1402015-11-14T08:11:17  <gmaxwell> oh he adds a DEFAULT_BLOCKSONLY, but then doesn't use it in the getargs because some of them are in net.cpp.
1412015-11-14T08:11:38  <wumpus> move the constant to net.h?
1422015-11-14T08:11:46  <gmaxwell> And he would pass it as an argument but apparently pushversion is called in a blinking constructor.
1432015-11-14T08:12:00  * gmaxwell checks if he can do that easily
1442015-11-14T08:12:21  <wumpus> in general the constants should be defined in the header of the cpp they are used in
1452015-11-14T08:12:52  <wumpus> if they are used in multiple places that could be annoying, but at least init.cpp already inclused net.h so that's not a problem
1462015-11-14T08:12:58  <gmaxwell> :) I'd assumed there was a reason he couldn't do that, but looking now.
1472015-11-14T08:13:51  <gmaxwell> yup he can. one is in main.cpp but it also pulls in net.h.
1482015-11-14T08:13:52  <wumpus> otherwise define a global bool and assign that in init.cpp, instead of calling GetBoolArg every time, we do that for more settings (especially those used in inner loops where the overhead of argument-parsing-every-time hurts)
1492015-11-14T08:13:57  <wumpus> ok!
1502015-11-14T08:14:09  <gmaxwell> okay thanks for the cluestick. I'd assumed that he used them someplace that didn't have net.h but I didn't look.
1512015-11-14T08:25:35  <gmaxwell> sipa: someone was seeking feedback from you here: https://github.com/bitcoin/bitcoin/pull/6844
1522015-11-14T08:35:39  *** dcousens has joined #bitcoin-core-dev
1532015-11-14T08:46:04  *** PaulCape_ has quit IRC
1542015-11-14T08:46:30  *** PaulCapestany has joined #bitcoin-core-dev
1552015-11-14T09:22:46  *** moli has joined #bitcoin-core-dev
1562015-11-14T09:26:41  *** molly has quit IRC
1572015-11-14T09:39:57  *** molly has joined #bitcoin-core-dev
1582015-11-14T09:44:14  *** moli has quit IRC
1592015-11-14T09:49:01  <wumpus> interesting, so this seccomp filter is a little program (in a restricted instruction set) that is uploaded to the kernel that checks system calls and arguments and returns whether allowed or not
1602015-11-14T09:50:01  <wumpus> never expected it to be so dynamic
1612015-11-14T09:51:48  <wumpus> and it should be possible to set it as non-privileged user, given that PR_SET_NO_NEW_PRIVS is set first to avoid tricking set-uid executables with it
1622015-11-14T09:54:07  <wumpus> sounds ActuallyUseful, should do some experiments with it some time...
1632015-11-14T09:55:45  <gmaxwell> Yes. It is. I tried to do it for bitcoin core eons ago, but the utility library was GPLed... they since changed the licensing.
1642015-11-14T09:55:47  <wumpus> (somehow always avoided use of seccomp because I assumed you'd need root to set them, like chroot etc)
1652015-11-14T09:56:10  <wumpus> cjdns uses it without utility library
1662015-11-14T09:56:55  <gmaxwell> nope. The thing that makes me most sad right now is walletbackup and the import thing, since they require arbritary file access. But I'd like to replace them with little utility programs, which we'd let ourselves exec and talk to over pipes.
1672015-11-14T09:57:01  <wumpus> (but cjdns does recommend starting it as root, which was one of the things that put the idea in my head, but they need it for tun setup)
1682015-11-14T09:57:29  <wumpus> (as well as other jail functionality, which, unfortunately, does need root)
1692015-11-14T09:58:26  <wumpus> well my idea at least at first would be to have a restricted, secure mode, that disabled calls like that (and also walletnotify/blocknotify and everything that calls external processes)
1702015-11-14T09:58:45  <gmaxwell> ACK
1712015-11-14T10:00:25  <wumpus> but you're right about walletbackup/import, I would prefer if they just dumped/retrieved their data over HTTP instead of making files
1722015-11-14T10:01:21  <wumpus> within the JSON RPC framework this is difficult to solve, but outside that w/ using the http server directly it'd be pretty easy to stream instead
1732015-11-14T10:05:25  *** Madars has joined #bitcoin-core-dev
1742015-11-14T10:06:59  <gmaxwell> I think there is some utility for triggering local backups, at least. But instead of arbritary file access it could simply be making it write out dated files; without giving the rpc caller huge freedom.
1752015-11-14T10:17:11  *** btcdrak_ has joined #bitcoin-core-dev
1762015-11-14T10:17:15  *** Arnavion has quit IRC
1772015-11-14T10:17:19  *** Arnavion3 has joined #bitcoin-core-dev
1782015-11-14T10:17:23  *** Arnavion3 is now known as Arnavion
1792015-11-14T10:17:50  *** jgarzik_ has joined #bitcoin-core-dev
1802015-11-14T10:19:10  *** tripleslash has joined #bitcoin-core-dev
1812015-11-14T10:20:08  *** Thireus1 has joined #bitcoin-core-dev
1822015-11-14T10:20:11  *** midnightmagic_ has joined #bitcoin-core-dev
1832015-11-14T10:20:12  *** sipa_ has joined #bitcoin-core-dev
1842015-11-14T10:20:29  *** tripleslash_l has quit IRC
1852015-11-14T10:20:29  *** btcdrak has quit IRC
1862015-11-14T10:20:32  *** midnightmagic has quit IRC
1872015-11-14T10:20:35  *** zarathustra has quit IRC
1882015-11-14T10:20:38  *** Thireus has quit IRC
1892015-11-14T10:20:38  *** jgarzik has quit IRC
1902015-11-14T10:20:39  *** sipa has quit IRC
1912015-11-14T10:20:47  *** btcdrak_ is now known as btcdrak
1922015-11-14T10:30:12  <wumpus> one thing that would make sense is to write to the datadir only
1932015-11-14T10:30:29  <wumpus> it also proves that the user using the calls has access to it
1942015-11-14T10:31:01  <wumpus> on the other hand, of course people want to backup to mounted filesystems etc
1952015-11-14T10:31:44  *** ParadoxSpiral has joined #bitcoin-core-dev
1962015-11-14T10:32:08  <wumpus> hence I'd really prefer it to stream from/to http so that the client can decide where to put it or take it from, anywhere *they* have access to
1972015-11-14T10:33:20  <wumpus> this avoids using bitcoind in a confused deputy attack (e.g have it write /etc/passwd :-) )
1982015-11-14T10:34:18  <wumpus> (or read, for that matter, although you'd be hard pressed to find a file that it likes to read and interpret)
1992015-11-14T10:34:33  <wumpus> (except maybe *other* backups it happens to have access to)
2002015-11-14T10:43:32  *** midnightmagic_ is now known as midnightmagic
2012015-11-14T10:50:30  *** tulip has quit IRC
2022015-11-14T10:53:07  *** zarathustra has joined #bitcoin-core-dev
2032015-11-14T10:58:01  <gmaxwell> IMO if you want to backup to a mounted FS you can either symlink it into the datadir, or have something external copy there. but ::shrugs::
2042015-11-14T11:08:06  *** tulip has joined #bitcoin-core-dev
2052015-11-14T11:26:09  *** jtimon has joined #bitcoin-core-dev
2062015-11-14T11:32:53  <morcos> phantomcircuit: I wrote a prototype separate thread for running CreateNewBlock, but the naive approach turned out much worse than the existing implementation.
2072015-11-14T11:33:37  <morcos> The issue was that the template creation code kept needing to hold cs_main, so you ended up holding that lock all the time and holding up things like block relay.
2082015-11-14T11:34:14  <morcos> Actually it makes a lot of sense to be sure you are not holding that lock at all when a new block comes in so you can validate it as quickly as possilbe and decide whether you should now be building off that
2092015-11-14T11:34:58  <morcos> Of course redesigning the template generation code to not need all the locking is probably what we need to do, but it turned out to be a much easier win just to speed up how long it takes
2102015-11-14T11:35:39  <morcos> It's really a mempool lock it needs but thats unfortunately cs_main partly.
2112015-11-14T11:36:32  <morcos> sipa and I discussed some ideas on having a transaction storage object, which the mempool and the mining code could separately refer to.
2122015-11-14T11:36:41  <gmaxwell> a while back matt was also just suggesting that CNB could terminate early if there was contention for the lock from a new block coming in.
2132015-11-14T11:37:03  <gmaxwell> which sounds a little hackish, but probably the kind of prudent hack that would work pretty well.
2142015-11-14T11:37:12  <gmaxwell> "Look, we've invented cooperative multitasking!"
2152015-11-14T11:37:14  <morcos> unfortunately everything uses cs_main though
2162015-11-14T11:37:31  <morcos> the hold createnewblock would have constantly been holding up ATMP by 3 secs or more
2172015-11-14T11:37:42  <morcos> old
2182015-11-14T11:37:57  <gmaxwell> yea, not at all a replacement for making it not embarassingly slow.
2192015-11-14T11:38:05  <morcos> i'm not saying its not fixable, but its a bit of work
2202015-11-14T11:38:19  *** CodeShark_ has quit IRC
2212015-11-14T11:38:27  <gmaxwell> ::nods::
2222015-11-14T11:38:34  <morcos> also the other issue is all the package stucture that sdaftuar put into the mempool is potentially really important for mining
2232015-11-14T11:38:47  <morcos> and so we have to decide whether it makes sense to duplicate all that code
2242015-11-14T11:38:54  <morcos> should their be two mempools?
2252015-11-14T11:39:03  <morcos> that are each dynamically updated
2262015-11-14T11:39:17  <morcos> should the mining code just copy all it cares about from the real mempool and then do its work
2272015-11-14T11:39:46  <morcos> or perhaps my favorite approach is to fine grain the locks a little better, and then have a mode where txs can queue up to enter the mempool
2282015-11-14T11:40:04  <morcos> so you can manipulate the actual mempool in the mining code (i think it'll be relatively fast)
2292015-11-14T11:40:27  <morcos> but anyway, properly sorting out the locks is a precursor to any of it
2302015-11-14T11:40:55  <morcos> btw, i'm a bit confused by your 30 secs of the same work for mining hardware
2312015-11-14T11:41:14  <morcos> i didn't analyze too closely, but the antminer seemed to be sending new work much more often than that, or at least considering doing so
2322015-11-14T11:43:52  <gmaxwell> All modern devices have internal queing and pipelines so they don't stall, some are unfortunately pretty deep.
2332015-11-14T11:45:43  <morcos> will the pool software that phantomcircuit suggested use the longpool feature of gbt, or is that not what most people do
2342015-11-14T11:46:14  <morcos> constantly polling getblockcount and getblockhash seems pretty silly (what cgminer does)
2352015-11-14T11:46:21  <gmaxwell> the 30 second rule of thumb is an amalgmation of total delays, in pratice;-- including most pool server software being fairly slow. as far as hardware goes I'm not sure what the antminer s7 is like; S1 was very good (I beat on them pre-release); S2 was pretty terrible (and completely unusable with p2pool which has 30 second blocks); I think I heard S5 was better than S2, I haven't heard about S7
2362015-11-14T11:46:27  <gmaxwell> .  KNC devices have quite high latency (seconds to respond to new work in my expirence),  SP10/SP20 have quite low latency.
2372015-11-14T11:47:03  <gmaxwell> ckpool uses blocknotify, many things (including p2pool) talk to bitcoind on the p2p port.  Eloipool will use GBT longpool I think, as well as p2p port.
2382015-11-14T11:47:48  <morcos> shocking that there are so many different mining software implementations, when its essentially consensus critical too
2392015-11-14T11:47:53  <gmaxwell> Even when things long poll they can take a really long time to get it out to the devices due to 'reasons'. (such as almost no one ever actually measuring)
2402015-11-14T11:48:09  <morcos> and do the big miners use the existing implementations, or have custom code.   and who maintains all those things
2412015-11-14T11:48:14  <morcos> sorry for the barrage of ?'s
2422015-11-14T11:48:16  <gmaxwell> morcos: well it's armored by bitcoin nodes in and out; not that there haven't been problems.
2432015-11-14T11:49:39  <gmaxwell> morcos: so historically mining pools have mostly been custom code; sometimes hacked on top of parts openly available.  Historically, public pools spend a lot of their development brainpower on dealing with DOS attacks; then data management for payout computation..
2442015-11-14T11:50:32  <morcos> are "pools" still what dominate mining or is it mostly single entities using the pool infrastructure b/c thats what exists
2452015-11-14T11:51:34  <gmaxwell> So there are still big public pools but many of them have their own hashpower (either directly or in commonly owned partner companies). The exact breakdown of things is unclear.
2462015-11-14T11:53:03  <gmaxwell> There are, of course, big entities now and they have simpler mining challenges (e.g. not so much having to worry about DOS).  They sometimes use existing public pools. (I mean some large entities are always on public pools; some only use them sometimes for various reasons)
2472015-11-14T11:54:02  <gmaxwell> Mining ecosystem is weird; as you observe there are a bazillion seperate implementations of this or that... and then no one goes and makes createnewblock fast.
2482015-11-14T11:54:17  <morcos> yeah, thats what i was wondering about
2492015-11-14T11:55:00  <morcos> but i suppose for our purposes (not that it would lead to a different design goal) what we should envision is i guess a p2pool implementation
2502015-11-14T11:55:09  <gmaxwell> Only 'mining pool' people who've really been active in core are luke-jr and phantomcircuit (who used to run the gear for a big-mining entitiy before working at BS).  And mostly, even for these guys the ideal strategy is to work around limitations; because doing so eas easier/safer.
2512015-11-14T11:55:14  <morcos> i mean the whole point is to make the small miner able to compete?
2522015-11-14T11:56:20  <gmaxwell> E.g. eloipool (the mining pool software luke-jr wrote that runs eligius) compensates for CNB latency by constructing empty blocks on its own, until CNB responds. And most mining operations just run multiple bitcoinds: one for GBT and several others that they submit through.
2532015-11-14T11:56:28  <tulip> morcos: many of the larger pools appear to run completely custom software, for the most part their quirks are completely unique. many of the smaller ones now use ckpool, which seems to be for the most part significantly faster than everybody else.
2542015-11-14T11:57:37  <morcos> tulip: thanks.  i'm trying to narrow the universe of pool/mining software to explore
2552015-11-14T11:59:03  <tulip> morcos: ckpool, eloipool, p2pool, NOMP
2562015-11-14T11:59:18  <gmaxwell> P2pool is nice idea with cute features, but has always struggled with the high startup cost of having to run a bitcoin node with it; and then high latency asics showed up (in particular, some of the early BFL products had 10%+ hashrate loss from 10 second retasking)
2572015-11-14T12:00:14  <gmaxwell> and that pretty much killed it there, and when it wasn't yet dead enough; somewhat later antpool announced that they'd be converting to p2pool based (but with a friendly front end where they run it for you) and lots of p2pool users moved over, then they didn't actually do it.
2582015-11-14T12:00:52  <gmaxwell> so now p2pool is too low of a hashrate for anyone who is variance twitchy to use. :(
2592015-11-14T12:01:03  <morcos> oh no, thats sad
2602015-11-14T12:01:51  <gmaxwell> Also through this time the developer burned out on Bitcoin, and only does life support maintaince on p2pool now.
2612015-11-14T12:02:16  <gmaxwell> at the moment P2Pool's hashrate is 918 TH, says my local p2pool node.
2622015-11-14T12:02:31  <gmaxwell> (it's pretty neat, has a built in webserver and graphs and such)
2632015-11-14T12:03:58  <gmaxwell> (Forrestv seems to have flamed out due to a mixture of bitcoin drama, and then donations to support p2pool being relatively low (compared to life changing amounts of income centeralized pool operators were getting), and then he got ripped off ... twice, I think, by mining hardware makers.)
2642015-11-14T12:07:40  <tulip> morcos: there's also stratum proxies like bfgminer and stratum-mining-proxy which present work to downstream miners, based on either an upstream work server or a Bitcoin node. the latter was meant for use with early hardware miners which supported getwork but not stratum.
2652015-11-14T12:10:09  <gmaxwell> then of course there is all these mining devices with embedded mips and arm linux systems that run usually very old versions of cgminer (or sometimes bfgminer)
2662015-11-14T12:24:50  <gmaxwell> morcos: while mentioning p2pool I was looking at my stats, and the 1 year view on GBT latency is amusing: https://people.xiph.org/~greg/p2pool-gbt.png
2672015-11-14T12:43:11  *** tulip has quit IRC
2682015-11-14T13:06:06  <midnightmagic> sigh
2692015-11-14T13:23:27  <GitHub86> [bitcoin] gmaxwell pushed 9 new commits to master: https://github.com/bitcoin/bitcoin/compare/9ffc687288dd...b632145edeb3
2702015-11-14T13:23:27  <GitHub86> bitcoin/master 4044f07 Patick Strateman: Add blocksonly mode
2712015-11-14T13:23:28  <GitHub86> bitcoin/master 420fa81 Patick Strateman: Do not process tx inv's in blocksonly mode
2722015-11-14T13:23:29  <GitHub86> bitcoin/master 3a96497 Patick Strateman: Add whitelistalwaysrelay option
2732015-11-14T13:23:37  <GitHub168> [bitcoin] gmaxwell closed pull request #6993: Add -blocksonly option (master...blocksonly) https://github.com/bitcoin/bitcoin/pull/6993
2742015-11-14T13:28:27  <phantomcircuit> morcos, generally speaking.. yes i dont care that cnb is slow because it's mostly irrelevant
2752015-11-14T13:28:52  <phantomcircuit> and i really dont care that it holds a lock for ages since blocks are submitted to nodes dedicated for that
2762015-11-14T13:29:01  <gmaxwell> it's much less irrelevant if you're not assuming a big miner farm.
2772015-11-14T13:29:14  <phantomcircuit> gmaxwell, true
2782015-11-14T13:30:21  <morcos> phantomcircuit: i thinking having getblocktemplate and createnewblock quickly return a valid block after a new tip is important to everyone
2792015-11-14T13:30:37  <morcos> even better if it is with txs, but not critical
2802015-11-14T13:31:00  <phantomcircuit> morcos, well making it return *something* immediately afterwards would certainly be nice
2812015-11-14T13:31:14  <phantomcircuit> that would reduce the need for some of the comical hacks
2822015-11-14T13:31:41  <phantomcircuit> but as gmaxwell said most of the hardware has queues that are very long and dont flush
2832015-11-14T13:31:54  <phantomcircuit> (there's good reason for this but i wont get into that)
2842015-11-14T13:32:07  <morcos> right so the way i look at it is assuming we don't want to code up validationless mining (which i am not yet convinced of)  then we need 100ish ms to validate the new tip and about 100ish ms to generate a new block with txs (with the new code)
2852015-11-14T13:32:48  <morcos> so at most we could carve out the second 100ms, but at this point we've gotten 95% of the improvement, so maybe not worth doing that unless we are going all the way
2862015-11-14T13:32:52  <phantomcircuit> morcos, 100ms to return an empty but validated block template would be better :P
2872015-11-14T13:33:38  <morcos> and before gmaxwell reaches through the tubes to strangle me, my primary argument for considering doing validationless mining is that if people are going to implement it anyway, we might as well make it as safe as possible
2882015-11-14T13:34:37  <morcos> but i agree that if there are other bottlenecks to switching work, then its not something we can do...   (validationless block header then 100-200ms later the validated version, if you can't switch after 200ms, then its not safe to start on the unvalidated version)
2892015-11-14T13:35:44  <morcos> but that same argument applies much less importantly to empty blocks.  if you're not going to switch after 100ms to the block with txs...  then you probably dont' want to start with the empty block...   depends on the amount of fees and the minimum switching delay i guess
2902015-11-14T13:35:52  <gmaxwell> If we want to implement validationless mining, then we need to do it right; which would mean doing something like signaling in the block header what we haven't verified prev, and so SPV clients should ignore the block for conf count purposes.
2912015-11-14T13:36:04  <gmaxwell> But I'd rather get validation lower first. :)
2922015-11-14T13:36:30  <phantomcircuit> morcos, there's basically no bottleneck in the validation
2932015-11-14T13:37:11  <morcos> gmaxwell: sure agreed.   phantomcircuit: no bottleneck? i'm talking about latency in sending new work to hardware outside of bitcoind
2942015-11-14T13:37:43  <morcos> i'd guess that a significant chunk of the remaining 100ms is allocation (in the non degenerate block case)
2952015-11-14T13:38:15  <phantomcircuit> morcos, yes... the issue is almost entirely not in validation but in selecting transactions
2962015-11-14T13:38:20  <gmaxwell> it's an interesting point that the extra 100ms doesn't matter. But rather, I think getting the 100ms upfront matters, and then it doesn't really matter if CNB is slow (except for lock holding reasons)
2972015-11-14T13:38:30  <phantomcircuit> the mempool indexing + limiting mostly fixes that
2982015-11-14T13:38:37  <morcos> phantomcircuit: huh? no thats backwards.  selecting txs is blazingly fast.  (well until cpfp)
2992015-11-14T13:39:01  <morcos> oh you're talking about in master
3002015-11-14T13:39:59  <morcos> yeah sure.. i'm talking about the remaining 100ms...  which is 10ms tx selection and 90 ms validation.   tx selection is still only 20ms if you scan an entire 300mb mempool to also sort by priority (with a fast priority calc)
3012015-11-14T13:40:46  <phantomcircuit> morcos, im not (and i dont think anybody paying attention) is worried about validation times of 100ms
3022015-11-14T13:41:14  <morcos> i see 100 ms, and i think the units still need to change.  :)
3032015-11-14T13:41:19  <phantomcircuit> that's like 0.01% orphan rate
3042015-11-14T13:41:42  <phantomcircuit> yes well faster is always better
3052015-11-14T13:42:03  <gmaxwell> hah. I think 100ms is really slow, but people clearly don't mind that much. But they don't mind in part because they perform hacks; some of which are harmful and we'd like them to stop. :)
3062015-11-14T13:42:17  <morcos> actually we are forgetting to measure some things there
3072015-11-14T13:42:36  <morcos> i'm measuring from receive block to new tip = 100ms  and new tip to new template = 100ms
3082015-11-14T13:42:36  <gmaxwell> when I say they don't mind that much; a year ago there were two places in the network handling code that just interted 100ms _sleeps_.
3092015-11-14T13:42:48  <morcos> but we also need to worry about receive most work header to receive block
3102015-11-14T13:43:05  <morcos> that's probably what we need to optimize now
3112015-11-14T13:43:42  <phantomcircuit> yes the push head instead of inv patch is a huge win for that
3122015-11-14T13:43:50  <morcos> or even someone else genarates new tip to receiving most work header.  direct headers will help with that
3132015-11-14T13:44:22  <gmaxwell> yes, well we can remove a one way delay by nominating your favorite peer to relay uninvited; beyond that needs something like matt's relay protocol. (or better)
3142015-11-14T13:44:40  <morcos> yeah i need to look into how the relay client works...  if you generate a new tip... how do you communicate that to the relay network, same as p2p
3152015-11-14T13:44:48  * phantomcircuit looks at brain clock *wow so late* goes to sleep
3162015-11-14T13:45:13  <gmaxwell> morcos: the relay network has its own protocol, and a local proxy you run that speaks bitcoin p2p.
3172015-11-14T13:45:32  <morcos> oh i suppose if you submitted to a node thats not mining, then it doesn't have the problem of that immediate cs_main lock from gbt after you generate a new tim
3182015-11-14T13:45:42  <morcos> tip
3192015-11-14T13:45:54  <gmaxwell> yea, thats why the earlier mentioned architecture of multiple nodes.
3202015-11-14T13:46:57  <gmaxwell> The relay protocol is super simple. It can send lose transactions, and keeps a history of the X sent. When you want to send a block it sends the header and a list of two byte indexes into that transaction history, as well as any history-miss transactions.
3212015-11-14T13:47:22  <gmaxwell> other end reconstructs the reblock and passes it on to bitcond unsolicited.
3222015-11-14T13:48:28  <gmaxwell> his server side, passes all transactions through a local bitcoind, and only relays what it spits out.. but for blocks it checks work and relays them on without more then minimal verification.
3232015-11-14T13:50:07  <gmaxwell> This really simple approach frequently managed to turn 1MB blocks into 3.6KB. http://bitcoinrelaynetwork.org/stats.html  and leaves matt either fighting sha256 speed ... or mempool spam that breaks his hitrates.
3242015-11-14T13:57:49  <GitHub101> [bitcoin] gmaxwell opened pull request #7016: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN. (master...without_EVENT_LOG_WARN) https://github.com/bitcoin/bitcoin/pull/7016
3252015-11-14T14:19:27  *** zarathustra has quit IRC
3262015-11-14T15:13:33  *** zarathustra has joined #bitcoin-core-dev
3272015-11-14T15:25:16  *** Guyver2 has joined #bitcoin-core-dev
3282015-11-14T15:26:21  *** kanzure_ is now known as kanzure
3292015-11-14T16:23:27  *** bsm117532 has quit IRC
3302015-11-14T16:27:11  *** jgarzik_ has quit IRC
3312015-11-14T16:27:11  *** jgarzik_ has joined #bitcoin-core-dev
3322015-11-14T16:27:14  *** jgarzik_ is now known as jgarzik
3332015-11-14T16:30:34  *** sipa_ is now known as sipa
3342015-11-14T16:34:10  *** vev__ has joined #bitcoin-core-dev
3352015-11-14T16:34:13  <vev__> we need people to support us #libreidea
3362015-11-14T16:39:17  *** vev__ has left #bitcoin-core-dev
3372015-11-14T16:52:06  *** jl2012 has quit IRC
3382015-11-14T16:52:28  *** jl2012 has joined #bitcoin-core-dev
3392015-11-14T17:31:17  *** zooko has joined #bitcoin-core-dev
3402015-11-14T18:25:40  *** lightningbot has joined #bitcoin-core-dev
3412015-11-14T18:28:55  *** Thireus1 has quit IRC
3422015-11-14T18:31:13  *** Thireus has joined #bitcoin-core-dev
3432015-11-14T18:34:29  *** d_t has joined #bitcoin-core-dev
3442015-11-14T18:40:31  *** jl2012 has quit IRC
3452015-11-14T18:58:19  *** zooko has quit IRC
3462015-11-14T19:03:22  <GitHub50> [bitcoin] morcos closed pull request #6292: Rename and comment priority calculation in TxMemPoolEntry (master...PriorityComment) https://github.com/bitcoin/bitcoin/pull/6292
3472015-11-14T19:06:04  <morcos> Can someone tag #6134 with 0.12 milestone.
3482015-11-14T19:13:14  <morcos> I also think #6915 should go in for 0.12.
3492015-11-14T19:17:43  *** jl2012 has joined #bitcoin-core-dev
3502015-11-14T19:21:40  *** zooko has joined #bitcoin-core-dev
3512015-11-14T19:42:26  *** dgenr8 has joined #bitcoin-core-dev
3522015-11-14T19:44:38  *** zooko has quit IRC
3532015-11-14T19:50:36  *** zooko has joined #bitcoin-core-dev
3542015-11-14T20:26:17  *** d_t has quit IRC
3552015-11-14T20:27:42  *** zarathustra has quit IRC
3562015-11-14T20:27:42  *** zarathustra has joined #bitcoin-core-dev
3572015-11-14T21:39:22  *** PRab has quit IRC
3582015-11-14T21:42:59  *** andytoshi has quit IRC
3592015-11-14T21:42:59  *** andytoshi has joined #bitcoin-core-dev
3602015-11-14T21:43:46  <GitHub136> [bitcoin] MarcoFalke opened pull request #7019: [trivial] travis: cover *receivedby* rpcs (master...MarcoFalke-2015-receivedby) https://github.com/bitcoin/bitcoin/pull/7019
3612015-11-14T21:46:51  *** PaulCape_ has joined #bitcoin-core-dev
3622015-11-14T21:49:26  *** PaulCapestany has quit IRC
3632015-11-14T21:57:30  *** zooko has quit IRC
3642015-11-14T22:03:22  *** Guyver2 has quit IRC
3652015-11-14T22:05:25  *** petertod1 is now known as petertodd
3662015-11-14T22:05:55  *** petertodd is now known as Guest19775
3672015-11-14T22:14:35  *** MarcoFalke has joined #bitcoin-core-dev
3682015-11-14T22:32:08  *** PaulCape_ has quit IRC
3692015-11-14T22:32:32  *** PaulCapestany has joined #bitcoin-core-dev
3702015-11-14T22:35:05  *** zooko has joined #bitcoin-core-dev
3712015-11-14T23:10:16  *** CodeShark_ has joined #bitcoin-core-dev
3722015-11-14T23:11:16  *** MarcoFalke has quit IRC
3732015-11-14T23:17:46  *** Guest19775 is now known as petertodd
3742015-11-14T23:25:08  *** alpalp has joined #bitcoin-core-dev
3752015-11-14T23:25:49  *** alpalp has quit IRC
3762015-11-14T23:35:33  *** MarcoFalke has joined #bitcoin-core-dev
3772015-11-14T23:44:44  *** moli has joined #bitcoin-core-dev
3782015-11-14T23:48:35  *** molly has quit IRC
3792015-11-14T23:55:52  *** molly has joined #bitcoin-core-dev
3802015-11-14T23:59:38  *** moli has quit IRC