12015-10-27T00:05:06  <phantomcircuit> morcos, my huge sigcache is mostly fine
  22015-10-27T00:05:30  <morcos> phantomcircuit: i'm running with 1M now and its still noticeably slower (than infinite)
  32015-10-27T00:05:38  <phantomcircuit> changing the max arena 100% fixed the problem without rewritting a bunch of stuff too :)
  42015-10-27T00:06:05  <phantomcircuit> hmm it really shouldn't be
  52015-10-27T00:06:13  <morcos> well i'm chaning the other stuff anyway  :)
  62015-10-27T00:06:58  <morcos> well noticeably slower i mean just that...  say instead of taking 30us to verify a txin on average it takes 40us
  72015-10-27T00:07:19  <morcos> but without a cache at all of course its 10x higher
  82015-10-27T00:09:51  <morcos> its the process of filling up your 300M mempool for the first time that makes it slow...  i guess it should improve once its full and the mempool gets a minfee
  92015-10-27T00:10:48  <morcos> scarily fast filling up to 300M from starting from scratch.  like 45mins
 102015-10-27T00:11:39  <phantomcircuit> morcos, without the limiting
 112015-10-27T00:11:42  <phantomcircuit> im at about 2GB now
 122015-10-27T00:11:50  <phantomcircuit> 3.168GB
 132015-10-27T00:12:30  <morcos> i'm still disturbed by how this same backlog of txs is so readily rerelayed
 142015-10-27T00:13:19  <gmaxwell> esp since the network behavior is supposted to not relay transactions once they've already been relayed. :(
 152015-10-27T00:13:39  <gmaxwell> unfortunately there seem to be some abusive nodes out there that behave strangely, constantly resending transactions and such.
 162015-10-27T00:14:40  <morcos> yeah i noticed that, i was wondering if they were nodes who considerd these wallet tx's or if was specifically designed to retransmit them
 172015-10-27T00:15:44  <morcos> ok modulo the final TestBlockValidity check, i brought the time of CreateNewBlock from a bit over 1 second to under 10 ms.
 182015-10-27T00:15:44  <gmaxwell> the ones I've caught doing it with an instrumented node appeared to regurgitate their whole mempool every block.
 192015-10-27T00:16:01  <morcos> unfortunately i wouldn't feel good about dumping that check.
 202015-10-27T00:16:22  <morcos> that also doesn't include adding any priority transactions.  i can add that back in, but i think it'll be slow
 212015-10-27T00:16:40  <morcos> gmaxwell: oh really?? thats bad
 222015-10-27T00:18:05  <midnightmagic> fwiw, gmaxwell recall you wanted me to check and see whether any of those nodes were resending me tx dupes: across the entire time I was monitoring, none of them did that I could detect while parsing the debug.log.
 232015-10-27T00:18:26  <midnightmagic> I don't think I actually got back to you about that.
 242015-10-27T00:18:38  <midnightmagic> gmaxwell: What's the periodicity of the resends?
 252015-10-27T00:19:35  <gmaxwell> midnightmagic: at least some were doing this after every block.
 262015-10-27T00:20:04  <gmaxwell> e.g. a bogus assumption that any transaction known would make it in a block, and if it didn't you must have just 'lost' it.
 272015-10-27T00:20:09  <gmaxwell> or the like.
 282015-10-27T00:20:33  <midnightmagic> I wonder if some services are attempting to "help" the network by propagating tx in order to guarantee reach for nodes that have rebooted or something.
 292015-10-27T00:29:04  <CodeShark> sending as in sending the actual transaction? or just an inv?
 302015-10-27T00:32:47  <phantomcircuit> CodeShark, same thing for this purpose
 312015-10-27T00:32:52  <phantomcircuit> zombie transactions
 322015-10-27T00:33:49  <CodeShark> for what purpose? I would think sending the mempool (as in the actual txs, not just the inv) is MUCH worse for bandwidth
 332015-10-27T00:34:09  <morcos> would anybody else find useful a feature which split the minimum fee for acceptance to your mempool and the min fee for relay?
 342015-10-27T00:34:37  <morcos> often during this testing, i either hack up my node to not relay, or i feel bad about the fact that i'm helping propogate these low fee txs all over again
 352015-10-27T00:34:43  <CodeShark> I would find really useful a feature that would allow you to get paid for running a validator/relay node :p
 362015-10-27T00:35:01  <morcos> you get paid in food for your soul
 372015-10-27T00:35:04  <CodeShark> lol
 382015-10-27T00:35:19  <sipa> CodeShark: i think that screws incentives completely
 392015-10-27T00:35:37  <CodeShark> sipa: howso?
 402015-10-27T00:35:48  <sipa> CodeShark: if you can get paid to run a node, you can equally get paid to run a compromised node
 412015-10-27T00:36:15  <CodeShark> presumably you'd have to actually perform the function sufficiently well
 422015-10-27T00:36:35  <sipa> you can't prove that you are
 432015-10-27T00:37:09  <CodeShark> well, relay might be more easily amenable to micropayments - validation is a little trickier
 442015-10-27T00:37:13  *** deepcore has quit IRC
 452015-10-27T00:38:37  <CodeShark> but I think ultimately doable...not sure how, but my gut tells me at the very least some efficient probabilistic checks are possible
 462015-10-27T00:39:07  <CodeShark> without having to go to SNARKs
 472015-10-27T00:39:18  <sipa> their validation is unobservable... it's a feature they provide to their users, not to you
 482015-10-27T00:42:42  <CodeShark> it's a feature they provide to the network
 492015-10-27T00:43:01  <sipa> no
 502015-10-27T00:43:07  <sipa> relay is what they provide to the network
 512015-10-27T00:43:16  <sipa> and you can't observe that they validate before relay
 522015-10-27T00:43:40  <sipa> 1) they may just be connected to honest nodes them selves, so they'll only relay valid blocks through that
 532015-10-27T00:44:23  <sipa> 2) past performance is not an indicator for future success. One awesome way to sybil attack the network is to spin up many totally honest nodes, and then suddenly turn evil once you notice a sufficient portion of the network depends on you.
 542015-10-27T00:45:14  <sipa> relay/ibd is a dumb service they offer to the network - which hopefully verifies what you tell them enough to not need to trust you
 552015-10-27T00:46:10  <sipa> the power of validation is only through knowing that the person running it has economic activity depending on it
 562015-10-27T00:48:16  <CodeShark> so I guess if the network is to subsidize it it would have to either require miners to be doing validation somehow...or to incentivize invalidation
 572015-10-27T00:50:16  <CodeShark> in any case, if we could just take care of incentives for relay for now I'd be happy :)
 582015-10-27T00:50:29  <CodeShark> that would go a long ways towards solving the mempool issues
 592015-10-27T00:57:30  <CodeShark> providing SPV proofs could be incentivized...and perhaps nodes that can provide SPV proofs would also have to perform full validation
 602015-10-27T01:00:16  <CodeShark> or it would be in their interest to, I suppose
 612015-10-27T01:01:10  *** fkhan has quit IRC
 622015-10-27T01:15:29  *** fkhan has joined #bitcoin-core-dev
 632015-10-27T01:15:41  <phantomcircuit> morcos, i always return early in relaytransaction when messing with mempool stuff
 642015-10-27T01:22:21  <phantomcircuit> ha i can detect which peers are master by the order of their responses to the mempool command
 652015-10-27T01:23:52  <phantomcircuit> 2015-10-27 01:23:33 CSignatureCache.setValid.size() 230249
 662015-10-27T01:23:53  <phantomcircuit> 2015-10-27 01:23:33 AcceptToMemoryPool: peer=15: accepted 3f5cfddb91105914ddbe9dc8dd3f32542bd9b1c5fe6b73873b4b6a5c87b62f51 (poolsz 3360)
 672015-10-27T01:23:55  <phantomcircuit> interesting
 682015-10-27T01:25:41  <gmaxwell> whats interesting?
 692015-10-27T01:26:13  <morcos> phantomcircuit: is this a new node?  not the 3GB one?  its not surprising.  most of these spam txs have 100txins, so you'd assume about 100 to 1 ration of txs recieved to setValid size
 702015-10-27T01:27:35  <phantomcircuit> gmaxwell, sigcache vs mempool size
 712015-10-27T01:27:44  <phantomcircuit> morcos, it's currently talking upto being full
 722015-10-27T01:27:46  <phantomcircuit> takes ages
 732015-10-27T01:34:20  *** Ylbam has quit IRC
 742015-10-27T01:53:58  <phantomcircuit> ok so 1,000,000 sigcache ~= 300MiB mempool
 752015-10-27T01:54:34  <phantomcircuit> we should definitely try to switch to a single parameter "-memlimit"
 762015-10-27T01:54:40  <phantomcircuit> and then split it out as percentages
 772015-10-27T02:30:20  *** d_t has quit IRC
 782015-10-27T02:43:50  *** belcher has quit IRC
 792015-10-27T02:46:32  *** jl2012_ has quit IRC
 802015-10-27T02:46:45  *** jl2012 has joined #bitcoin-core-dev
 812015-10-27T04:03:40  *** dcousens has quit IRC
 822015-10-27T04:28:16  *** CodeShark has quit IRC
 832015-10-27T04:41:18  *** dcousens has joined #bitcoin-core-dev
 842015-10-27T04:41:49  <dcousens> hmph.    CXX      libbitcoin_server_a-init.o
 852015-10-27T04:41:50  <dcousens> g++: internal compiler error: Killed (program cc1plus)
 862015-10-27T04:41:58  <dcousens> Awesome :S
 872015-10-27T04:42:43  *** fkhan has quit IRC
 882015-10-27T04:43:18  <jcorgan> in my experience that is almost certainly an out-of-memory problem
 892015-10-27T04:43:47  <dcousens> jcorgan: interesting, I am trying to compile it on a device with only 1GB
 902015-10-27T04:45:15  <jcorgan> look in dmesg for linux OOM killer action
 912015-10-27T04:45:24  <dcousens> yeah I did, you were right
 922015-10-27T04:46:02  <gmaxwell> add swap. There are some options you can give g++ which will drastically reduce memory usage... but I have no idea what they are. :)
 932015-10-27T04:46:16  <dcousens> gmaxwell: easier to just put in another stick atm
 942015-10-27T04:52:53  *** blur3d has joined #bitcoin-core-dev
 952015-10-27T04:56:36  *** fkhan has joined #bitcoin-core-dev
 962015-10-27T04:59:22  *** blur3d has quit IRC
 972015-10-27T06:08:02  *** Guest20816 has quit IRC
 982015-10-27T06:08:51  *** pigeons has joined #bitcoin-core-dev
 992015-10-27T06:09:13  *** pigeons is now known as Guest72716
1002015-10-27T06:12:47  *** d_t has joined #bitcoin-core-dev
1012015-10-27T06:18:06  *** Thireus has joined #bitcoin-core-dev
1022015-10-27T06:20:25  *** d_t has quit IRC
1032015-10-27T06:23:38  *** ParadoxSpiral has joined #bitcoin-core-dev
1042015-10-27T06:31:16  <dcousens> poop
1052015-10-27T06:31:20  <dcousens> 2GB still dies lol
1062015-10-27T06:31:26  <dcousens> time to look up those g++ options...
1072015-10-27T06:31:34  <dcousens> or swap... fk it I'll add swap
1082015-10-27T06:32:33  <dcousens> hmmm, dmesg not showing OOM this time
1092015-10-27T06:32:35  <dcousens> but same error
1102015-10-27T06:45:17  *** deepcore has joined #bitcoin-core-dev
1112015-10-27T06:46:39  *** ParadoxSpiral has quit IRC
1122015-10-27T06:56:09  <wumpus> 2GB should certainly be enough. See e.g. https://github.com/bitcoin/bitcoin/issues/6658 , worst contender main.cpp uses 1.2MB while compiling (on gcc 4.8). Adding swap may help though, linux' memory management works better if swap is enabled even if you don't need the extra memory
1132015-10-27T06:56:38  <wumpus> if you want to build bitcoind for a smaller device I'd recommend cross compiling though, it's easy with the depends system
1142015-10-27T06:56:45  <dcousens> I ended up `git clean -xdf`, re-configuring/autconf and it got past that second hurdle
1152015-10-27T06:56:54  <dcousens> Guessing it configured something different when I only had 1GB?
1162015-10-27T06:57:05  <wumpus> it doesn't
1172015-10-27T06:57:11  <gmaxwell> wumpus: some people are managing to run with no overcommit _and_ no swap; ... dunno how anything at all works for them.
1182015-10-27T06:57:31  <dcousens> Then I have no idea, except that it worked this time,  but again, no OOM from dmesg so not sure
1192015-10-27T06:57:34  <wumpus> but it may be  possible for gcc, when it crashes, to generate a poop file that will crash it later
1202015-10-27T06:59:06  <dcousens> gmaxwell: eh, on my own home nodes I always ran without swap
1212015-10-27T06:59:19  <dcousens> but, they always had a bit more hardware to play with
1222015-10-27T06:59:45  <gmaxwell> dcousens: note that I said both no swap and no overcommit.
1232015-10-27T07:02:59  <wumpus> gmaxwell: I guess if the workload is very predictable
1242015-10-27T07:03:24  <wumpus> it sounds like the way to go for e.g. embedded devices
1252015-10-27T07:06:15  <wumpus> but certainly not if you run desktop applications like browsers :-)
1262015-10-27T07:07:02  <Luke-Jr> dcousens: note that Linux likes to OOM-kill processes long before it runs out of RAM, if you don't have swap :P
1272015-10-27T07:07:16  <dcousens> gmaxwell: I know.  Eh, I've always kept that at the default, though, I'd be half tempted to turn it off if it meant I'd actually get std:error in my logs haha
1282015-10-27T07:08:57  <dcousens> Luke-Jr: hasn't been my experience haha, I don't run swap on my most of my work machines and if I OOM, it just grinds to a complete halt usually
1292015-10-27T07:15:26  <wumpus> I used to have no swap enabled when 8GB still seemed like a lot of memory, but I can confirm what Luke-Jr says, even compile with -j3 would invoke the killer penguin
1302015-10-27T07:18:29  <wumpus> the problem was that any amount of swapping somehow instantly brought the entire system to a halt (even mouse cursor would no longer move) until it was done.  Managed to improve this somewhat by setting high swappiness.
1312015-10-27T07:18:51  <wumpus> (which makes it swap little used pages in advance instead of as last desperate measure)
1322015-10-27T07:19:26  <Luke-Jr> nice thing about a desktop PC is I have like 4-6 hard drives I can RAID0 the swap across..
1332015-10-27T07:23:40  <wumpus> good idea
1342015-10-27T07:27:28  *** Ylbam has joined #bitcoin-core-dev
1352015-10-27T07:29:43  <Luke-Jr> (fwiw, that's builtin to Linux by using swapon -o pri=N where N is identical for everything)
1362015-10-27T07:37:17  <wumpus> oh!
1372015-10-27T07:48:39  <GitHub175> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2b625510d374...38369dda325d
1382015-10-27T07:48:40  <GitHub175> bitcoin/master 0d699fc Jonas Schnelli: fix locking issue with new mempool limiting...
1392015-10-27T07:48:40  <GitHub175> bitcoin/master 38369dd Wladimir J. van der Laan: Merge pull request #6889...
1402015-10-27T07:48:48  <GitHub109> [bitcoin] laanwj closed pull request #6889: fix locking issue with new mempool limiting (master...2015/10/fix_mempool_lock) https://github.com/bitcoin/bitcoin/pull/6889
1412015-10-27T08:04:13  <wumpus> sipa: what is the idea with #6865? I think we should either remove or change the meaningless test
1422015-10-27T08:59:05  *** BashCo has quit IRC
1432015-10-27T09:21:00  *** BashCo has joined #bitcoin-core-dev
1442015-10-27T09:47:28  *** deepcore has quit IRC
1452015-10-27T09:50:30  *** moli has joined #bitcoin-core-dev
1462015-10-27T09:53:19  *** molly has quit IRC
1472015-10-27T10:09:14  *** isis has quit IRC
1482015-10-27T10:09:21  <GitHub182> [bitcoin] paveljanik closed pull request #5525: Add removeaddress RPC call (remove watch-only address) (master...rpc_removeaddress) https://github.com/bitcoin/bitcoin/pull/5525
1492015-10-27T10:19:33  *** Arnavion has quit IRC
1502015-10-27T10:20:54  *** AtashiCon has quit IRC
1512015-10-27T10:24:41  *** AtashiCon has joined #bitcoin-core-dev
1522015-10-27T10:24:46  *** Arnavion has joined #bitcoin-core-dev
1532015-10-27T10:59:00  <GitHub78> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/38369dda325d...8f3b3cdee497
1542015-10-27T10:59:00  <GitHub78> bitcoin/master 2d8c49d Casey Rodarmor: Clean up tx prioritization when conflict mined
1552015-10-27T10:59:01  <GitHub78> bitcoin/master 8f3b3cd Wladimir J. van der Laan: Merge pull request #6464...
1562015-10-27T10:59:06  <GitHub144> [bitcoin] laanwj closed pull request #6464: Always clean up manual transaction prioritization (master...manual-tx-prioritization-cleanup) https://github.com/bitcoin/bitcoin/pull/6464
1572015-10-27T12:04:40  *** dcousens has quit IRC
1582015-10-27T12:35:30  *** droark has quit IRC
1592015-10-27T12:48:03  *** ParadoxSpiral has joined #bitcoin-core-dev
1602015-10-27T13:02:09  *** isis has joined #bitcoin-core-dev
1612015-10-27T14:05:17  *** danielsocials has joined #bitcoin-core-dev
1622015-10-27T14:07:55  <morcos> I have some questions about changes I'd like to make to manual transaction prioritization and CreateNewBlock and block priority space.
1632015-10-27T14:09:32  <morcos> I added a fee rate sort to the mempool.  Using this and examining 600 calls to CreateNewBlock over the last 12 hours, the time to create a new block has been reduced from 2.337 seconds + 222 ms to validate to 11 ms + 345 ms to validate
1642015-10-27T14:09:57  <morcos> I think the increased validation time is because the originaly version made the cache warm first
1652015-10-27T14:10:26  <morcos> However, this does not create a priority space and does not apply manual transaction prioritization
1662015-10-27T14:11:02  <morcos> Fixing the manual transaction prioritzation shouldn't be a problem as long as we're willing to store that data in the CTxMemPoolEntry.  I know it was previously discussed not to do that because it might only apply to a few txs
1672015-10-27T14:11:37  <morcos> However I don't think a sort based on the modified fee will work very well otherwise.  So question 1, can I store the fee delta in the CTxMemPoolEntry.
1682015-10-27T14:12:03  <morcos> Question 2:  I assume I'm not allowed to get rid of block priority space?
1692015-10-27T14:13:21  <morcos> I was thinking I could just basically repeat the priority sort of the old alogrithm first, but I believe we might lose some advantages of the speedup.  However I think we could just skip that if blockprioritysize=0.  The code will be a bit uglier that way, but sort of optimized for the case where blockprioritysize=0?
1702015-10-27T14:13:26  <morcos> Any thoughts?
1712015-10-27T14:19:05  <morcos> Question 3: BlueMatt: In your issue about tie-breaking you seem to suggest its random.  But isn't the current tiebreaker (for the fee sort) priority?  What would you like the tiebreaker to be?
1722015-10-27T14:20:43  <morcos> BTW, since the current tie-breaker is priority, and its expensive to calculate priority, I'm not going to try to make my code generate the exact same blocks as the old code.
1732015-10-27T14:21:17  <morcos> sipa: gmaxwell: wumpus: see above questions please
1742015-10-27T14:28:16  <sipa> morcos: BlueMatt's problem occurs when there is an abundance of transactions with the exact same priority, i think
1752015-10-27T14:29:23  <sipa> morcos: i too believe we should get rid of priority, and replace it with a custom function that alters size or fee of a transaction
1762015-10-27T14:40:50  <gavinandresen> sipa: I agree, but a simple function with a miner-tweakable multiplier (maybe zero by default to mean "no priority adjustment") would be fine, I think.
1772015-10-27T14:41:21  *** bsm1175321 has quit IRC
1782015-10-27T14:42:40  <sipa> gavinandresen: right, i was taking code-wise... the policy code can implement an overridable function, and we know that everything keeps.workimg whatever that function is
1792015-10-27T14:42:58  <sipa> the default implementation can be a simple multiplier
1802015-10-27T14:43:14  <sipa> or a constant that get added to both fee and size
1812015-10-27T14:43:30  <gavinandresen> spiffy, I was worried somebody would go and implement a javascript interpreter so miners could tweak the function at runtime... :-)
1822015-10-27T14:43:59  <sipa> ha
1832015-10-27T14:44:04  <wumpus> should definitely use lua for that *ducks*
1842015-10-27T14:44:28  <sipa> we should use bitcoin script with the op_x86 extension
1852015-10-27T14:45:00  <wumpus> morcos: what is the main drawback of storing the fee delta in CTxMemPoolEntry? increasing the size of the structure?
1862015-10-27T14:47:11  <wumpus> bitcoin script, hah, even better
1872015-10-27T14:47:37  *** danielsocials has quit IRC
1882015-10-27T14:47:46  *** bsm1175321 has joined #bitcoin-core-dev
1892015-10-27T14:50:24  <wumpus> morcos: optimizing for blockprioritysize=0 sounds good to me, removing it is somewhat controversial - but as it seems to be where things are going anyway, and simplifies the algorithm a lot it makes sense to optimize for the case
1902015-10-27T14:52:03  <morcos> ok thanks everyone.  yes only downside is more bloat to the mempool, but its relatively small.
1912015-10-27T14:53:01  <morcos> sipa: so perhaps i'll make a CTxMemPoolEntry function GetMiningPriority (ugh I hate overuse of the word priority) which for now = GetFee() + feeDelta
1922015-10-27T14:56:41  <morcos> or should I store a miningPriority variable, and use an outside of CTxMempool function to calculate it and set it? I guess it could set it at creation and further calls to prioritizeTransaction would adjust it?
1932015-10-27T14:59:18  *** danielsocials has joined #bitcoin-core-dev
1942015-10-27T15:13:16  *** treehug88 has joined #bitcoin-core-dev
1952015-10-27T15:36:44  <jonasschnelli> cfields: any idea why i get "clang: error: unknown argument: '-framework QtCore'" on my osx machine with qt5.5 [homebrew]?
1962015-10-27T15:39:20  <cfields> jonasschnelli: mm, that's an odd error. is anything quoted funny?
1972015-10-27T15:39:54  <jonasschnelli> cfields: i just switched back to Qt 5.4.1 .. there i don't have a such problem..
1982015-10-27T15:40:10  <jonasschnelli> cfields: no additional details... just the clang error output during linking
1992015-10-27T15:40:35  <cfields> jonasschnelli: can you paste the exact link-line? (make V=1)
2002015-10-27T15:40:56  <cfields> if there's a problem with the framework, i'd expect "framework not found" or so
2012015-10-27T15:41:03  <jonasschnelli> cfields: will do soon... currently switched back to qt5.4.1
2022015-10-27T15:42:54  <bsm1175321> Can discuss c++11 here if you prefer.  But then what's #bitcoin-dev for?
2032015-10-27T15:44:00  *** zooko has joined #bitcoin-core-dev
2042015-10-27T15:44:02  <wumpus> for more abstract things, nitty gritty implementation details for bitcoin core belong here
2052015-10-27T15:46:21  *** danielsocials has quit IRC
2062015-10-27T15:46:42  <bsm1175321> wumpus: indeed an explicit shared_ptr gets rid of the compiler error.  Let's see if it runs...
2072015-10-27T15:47:13  <wumpus> welcome zooko
2082015-10-27T15:47:15  <wumpus> bsm1175321: great!
2092015-10-27T15:49:58  <cfields> jonasschnelli: https://github.com/Homebrew/homebrew/commit/fdbb338f2274e093b96209e7ca3ff9bf5460714e
2102015-10-27T15:50:13  <cfields> patch is here: https://gist.githubusercontent.com/UniqMartin/a54542d666be1983dc83/raw/f235dfb418c3d0d086c3baae520d538bae0b1c70/qtbug-47162.patch
2112015-10-27T15:50:19  <cfields> it seems to leave out the closing quote
2122015-10-27T15:51:14  <cfields> er, nm, misread
2132015-10-27T15:51:23  <jonasschnelli> cfields : you mean a syntax error for the two ends?
2142015-10-27T15:54:46  <cfields> jonasschnelli: oh, according to the bug report, that patchshould actually fix your problem
2152015-10-27T15:55:10  <cfields> possible you hadn't updated brew lately before building 5.5?
2162015-10-27T15:55:58  *** malte has left #bitcoin-core-dev
2172015-10-27T15:57:14  <cfields> (remember that we use static libs for depends builds, so we wouldn't hit this there for our 5.5.0 build)
2182015-10-27T15:58:50  *** yammq has joined #bitcoin-core-dev
2192015-10-27T16:14:20  *** d_t has joined #bitcoin-core-dev
2202015-10-27T16:17:48  *** yammq has quit IRC
2212015-10-27T16:30:06  *** BashCo has quit IRC
2222015-10-27T16:32:04  *** rubensayshi has joined #bitcoin-core-dev
2232015-10-27T16:40:19  <jonasschnelli> cfields: thanks for looking at it! I think i can solve it now... but need to finish stuff with qt 5.4.1 now until i go back to 5.5.
2242015-10-27T16:41:02  <GitHub34> [bitcoin] Diapolo opened pull request #6891: constify missing catch cases (master...const-ex) https://github.com/bitcoin/bitcoin/pull/6891
2252015-10-27T16:46:09  <GitHub81> [bitcoin] Diapolo opened pull request #6892: [Trivial] ensure minimal header conventions (master...headers-new) https://github.com/bitcoin/bitcoin/pull/6892
2262015-10-27T16:54:59  <GitHub176> [bitcoin] Diapolo opened pull request #6893: forward-declare univalue in rpcclient + remove include in rpcserver.cpp (master...univalue) https://github.com/bitcoin/bitcoin/pull/6893
2272015-10-27T16:56:48  *** treehug88 has quit IRC
2282015-10-27T16:59:40  *** lclc has joined #bitcoin-core-dev
2292015-10-27T17:04:12  <wumpus> sigh...
2302015-10-27T17:09:58  <jonasschnelli> ;-)
2312015-10-27T17:19:52  *** treehug88 has joined #bitcoin-core-dev
2322015-10-27T17:27:26  *** MarcoFalke has joined #bitcoin-core-dev
2332015-10-27T17:28:09  <btcdrak> Trolling for review: https://github.com/bitcoin/bitcoin/pull/6312
2342015-10-27T17:33:25  *** BashCo has joined #bitcoin-core-dev
2352015-10-27T17:38:50  *** treehug88 has quit IRC
2362015-10-27T17:40:39  *** treehug88 has joined #bitcoin-core-dev
2372015-10-27T17:42:57  *** MarcoFalke has quit IRC
2382015-10-27T17:55:41  *** treehug88 has quit IRC
2392015-10-27T18:00:02  <gmaxwell> btcdrak: is no one backporting mediantimepast to 0.11?
2402015-10-27T18:00:16  <btcdrak> gmaxwell: I backported it...
2412015-10-27T18:00:42  <btcdrak> gmaxwell: https://github.com/bitcoin/bitcoin/issues/6884
2422015-10-27T18:01:35  <btcdrak> gmaxwell: Not sure if we need to backport to 0.10, didnt have time yet anyway.
2432015-10-27T18:01:59  <gmaxwell> ah okay I missed it.
2442015-10-27T18:04:01  <btcdrak> gmaxwell: I also made a branch with BIP68+OP_CSV in one as some people asked for it. I'm debating if I should make a PR or just leave the two separate PRs as is.
2452015-10-27T18:06:28  *** rubensayshi has quit IRC
2462015-10-27T18:10:54  <gmaxwell> btcdrak: this patch does not appear to enforce (as standardness) the new behavior.
2472015-10-27T18:13:51  <morcos> there are some unit tests in miner_tests.cpp which verify that CreateNewBlock won't mine non-final txs that are in the mempool
2482015-10-27T18:13:59  <morcos> they were added by petertodd
2492015-10-27T18:14:26  <morcos> i was under the impression that we don't want to allow non-final txs into the mempool
2502015-10-27T18:15:18  <morcos> i suppose i should have checked that my code passed the unit tests earlier on, but the whole idea is to not recheck everything while building the block
2512015-10-27T18:16:08  <gmaxwell> morcos: the finality test is against the mined block itself, while the mempool is running one block behind.
2522015-10-27T18:16:47  <morcos> i think that was fixed so the mempool checks against +1, but that would have been a problem in the opposite direction
2532015-10-27T18:16:51  <gmaxwell> So the mepool need a one block lookahead or there will be an extra delay in mining transactions. (which would presumably mean a competative advantage for miners that change that code)
2542015-10-27T18:17:30  <morcos> the unit test specifically calls addUnchecked with a non-final tx, and then sees if CreateNewBlock would mine it
2552015-10-27T18:17:48  <morcos> non-final for 1 block ahead (and a time based version as well)
2562015-10-27T18:18:24  <morcos> what i'm saying is I'd like to be able ot rely on there being no non-final txs in the mempool, i think this was the idea sipa had in mind too?
2572015-10-27T18:19:08  <morcos> the one thing I can think of is that its possible we need a function analogous to removeCoinbaseSpends for locktime based txs in the event of a reorg if thats not happening already
2582015-10-27T18:19:16  <gmaxwell> The hight based can be precise, e.g. so it's not possible to add something that won't be final; and I think that would be good enough for what you're doing. But time based can't be precise; but MPL is fixing that.  I'm fine with there beinging nothing that won't be final.
2592015-10-27T18:20:17  <gmaxwell> I'm not sure if we're on the same page, so I'm saying that I think there should be currently non-final tx in the mempool which will be final from the perspective of createnewblock,  but none that won't be final from the perspective of createnewblock.
2602015-10-27T18:21:28  <gmaxwell> Before MTP this was hard to achieve without adding additional delays, but MTP will effectively make that additional delay universal.
2612015-10-27T18:21:30  *** treehug88 has joined #bitcoin-core-dev
2622015-10-27T18:21:51  <morcos> I guess thats the same page.  Mempool adds 1 to chainActive.height before checking finality, so i would consider it to be final from the perspective of both the mempool and CNB
2632015-10-27T18:22:10  <morcos> and yes i agree that the current state is no non-final txs will end up in the mempool
2642015-10-27T18:22:35  <morcos> what i'm saying is the unit tests require that even if they do, CNB will not put them in the template.  (that breaks for me, and i think is unnecessary computation)
2652015-10-27T18:22:41  <gmaxwell> okay, fair enough, I was defining current as chainActive.Tip().
2662015-10-27T18:22:53  <morcos> also what i'm saying is that a reorg could probably break that assumption
2672015-10-27T18:23:19  <gmaxwell> I think we can drop that test, but we need to make sure that reorgs are removing things correctly at the same time. (I don't know if we have a test for that).
2682015-10-27T18:23:20  <sdaftuar> morcos: bluematt had a pr recently to fix that
2692015-10-27T18:24:09  <sdaftuar> #6595
2702015-10-27T18:24:32  <sdaftuar> we need to pick that up again...
2712015-10-27T18:24:54  <sdaftuar> oh ha. i remember now, we were going to revamp the way reorgs are handled
2722015-10-27T18:24:58  <morcos> ah, ok good.  yes we need that.
2732015-10-27T18:25:12  <morcos> and yeah thats what i was thinking, that you only want to run this after the reorg
2742015-10-27T18:25:17  <morcos> but why is that difficult
2752015-10-27T18:25:28  <sdaftuar> i'll dust off that code.  after packages went in, bluematt's open pulls needed some rebasing
2762015-10-27T18:25:53  <sdaftuar> see also #6656
2772015-10-27T18:25:53  <morcos> you odn't have to redo the whole way reorgs are handled, you just have to call the removeForReorg after ActivateBestChainStep returns
2782015-10-27T18:26:02  <sdaftuar> yeah i think that's what 6656 does
2792015-10-27T18:26:04  <sdaftuar> er, did
2802015-10-27T18:26:11  <morcos> perhaps you need to flag that the height or mediantimepast might have changed
2812015-10-27T18:26:14  <morcos> down
2822015-10-27T18:26:17  <gmaxwell> Perhaps that unit test could be changed to addunchecked but then trigger the logic that handles a reorg.
2832015-10-27T18:27:37  <morcos> heh, i think thats in a different unit test
2842015-10-27T18:28:02  <morcos> sdaftuar: so 6656 really supercedes 6595 then.  jsut for some reason 6595 was left open
2852015-10-27T18:29:12  <morcos> gmaxwell: this actually does lead me to a concern.  i was leaving in the final TestBlockValidity test, but what should the code do in the event it fails?
2862015-10-27T18:29:39  <morcos> I'm thinking it should switch into mining empty block mode, because otherwise there is no way to figure out what part of your block assembly broke
2872015-10-27T18:30:08  <morcos> the down side of this, is if there is some error in mempool code that lets non valid txs in, all miners will just start switching to mining empty blocks
2882015-10-27T18:30:12  <morcos> ugh that sounds terrible
2892015-10-27T18:30:39  <gmaxwell> We assert now, no? .. yea, but I was just about to suggest it could start doing that. OTOH, that doesn't exactly encourage people to fix things, and empty blocks may mean that you don't failover to another daemon.
2902015-10-27T18:31:14  <morcos> its much less likely to happen now, because we CheckInputs on every tx as we add it anyway, and if that fails we skip that txs
2912015-10-27T18:31:22  <gmaxwell> morcos: well empty blocks are safe(er). Assuming that _prior_ chain is not invalid, they're no worse than not mining and likely better.
2922015-10-27T18:31:27  <morcos> perhaps i should put that code back in
2932015-10-27T18:31:40  <morcos> and just leave it off, unless you hit an invalid block, and then switch it on
2942015-10-27T18:31:48  <morcos> because that way you'll just skip over the invalid txs
2952015-10-27T18:31:52  <morcos> at least for that failure mode
2962015-10-27T18:32:07  <gmaxwell> Running checkinputs again sounds expensive.
2972015-10-27T18:32:12  <morcos> but then it won't be a very tested code path, unless we have a mode to test it
2982015-10-27T18:32:23  <morcos> yeah well thats why the existing code is slow
2992015-10-27T18:33:34  <morcos> its a nice property of the existing code, that if somebody injects an invalid tx into mempools, the mining code will just keep skipping it and assembling valid blocks from the rest of the txs
3002015-10-27T18:34:15  <gmaxwell> In any case, thinking in terms of what the user of this code wants;  they either want empty blocks (if they have no fail over) or they want the node to stop responding (if they have fail-over).
3012015-10-27T18:34:30  <gmaxwell> morcos: right, it keeps the mempool behavior seperate from consensus.
3022015-10-27T18:39:29  *** zooko has quit IRC
3032015-10-27T18:44:20  <sipa> morcos: i think the mempool should avoid but not outlaw non-final transactions; CNB should skip the nonfinal ones though
3042015-10-27T18:48:47  <gmaxwell> what about checkinputs redundancy?  at least doing another IsFinal check is cheap.
3052015-10-27T18:52:15  <sipa> i think all mempool entries should be valid regarding 1) not spending nonexisting inputs 2) script validity
3062015-10-27T18:53:12  <sipa> if we have that, i think CNB could just not validate tye result
3072015-10-27T18:53:22  <sipa> but that seems controversial
3082015-10-27T18:56:31  <morcos> sipa: so you'd like me to put IsFinal check back in?, you might be right, it might be cheap...
3092015-10-27T18:57:05  <morcos> i suppose it would be nice to not have to worry about removing txs in the event of a reorg, you know they'll be valid again very soon
3102015-10-27T18:57:51  <morcos> what do you think about BIP68 and sequence numbers though
3112015-10-27T18:58:06  <morcos> thats not cheap to check b/c you have to look up the inputs
3122015-10-27T18:58:37  <gmaxwell> sipa: really if we make CNB no longer latency critical (via mechenisms discussed before) then I think validate after remains a cheap protection.
3132015-10-27T18:59:07  <gmaxwell> but I'd rather remove some of the internal double checking where it is expensive and really shouldn't be needed but continue to validate after.
3142015-10-27T19:00:37  <sipa> morcos: i think enforcing nonfinalness in the mempool.is expensive, and the worst a reorg can cheaply do is put a bunch of things in that will be check anyway
3152015-10-27T19:01:10  <sipa> morcos: fair enough, bip68 does make this significantly more expensive
3162015-10-27T19:01:50  *** Thireus has quit IRC
3172015-10-27T19:02:11  *** Thireus has joined #bitcoin-core-dev
3182015-10-27T19:03:31  <morcos> sipa: i feel like i'm really overloading CTxMempoolEntry, but seems like it could store a block height and time at which the tx is final
3192015-10-27T19:04:08  <morcos> that way the locktime and bip68 checks that you do at ATMP don't have to be repeated (ugh... i guess unless there is a reorg that affects inputs)
3202015-10-27T19:04:23  *** randy-waterhouse has joined #bitcoin-core-dev
3212015-10-27T19:07:57  *** zooko has joined #bitcoin-core-dev
3222015-10-27T19:14:18  <sipa> morcos: sgtm, if that can guarantee not needing to verify after creation
3232015-10-27T19:16:29  *** zooko has quit IRC
3242015-10-27T19:18:21  *** zooko has joined #bitcoin-core-dev
3252015-10-27T19:19:42  *** d_t has quit IRC
3262015-10-27T19:23:19  *** d_t has joined #bitcoin-core-dev
3272015-10-27T19:27:31  <morcos> sipa: well then we're back to the reorg problem right?  but maybe that's solvable more quickly by only having to do work on smaller subset of the txs.  for instance if you know the lowest height and mediantPast time you disconnected down to, you only need to look at txs with CTxMemPoolEntry valid time and heights > that
3282015-10-27T19:32:07  <GitHub163> [bitcoin] giacecco closed pull request #6885: Instructions on how to make the Homebrew OpenSSL headers visible... (master...master) https://github.com/bitcoin/bitcoin/pull/6885
3292015-10-27T19:33:36  *** belcher has joined #bitcoin-core-dev
3302015-10-27T19:36:21  *** PaulCapestany has quit IRC
3312015-10-27T19:37:59  *** PaulCapestany has joined #bitcoin-core-dev
3322015-10-27T19:39:41  *** zooko` has joined #bitcoin-core-dev
3332015-10-27T19:40:33  *** CodeShark has joined #bitcoin-core-dev
3342015-10-27T19:41:35  *** zooko has quit IRC
3352015-10-27T19:41:55  *** zooko has joined #bitcoin-core-dev
3362015-10-27T19:43:20  *** zooko` has quit IRC
3372015-10-27T19:43:25  *** zooko has quit IRC
3382015-10-27T19:43:46  *** zooko has joined #bitcoin-core-dev
3392015-10-27T19:45:01  *** CodeShark has quit IRC
3402015-10-27T19:46:36  *** randy-waterhouse has quit IRC
3412015-10-27T19:58:43  <GitHub51> [bitcoin] sdaftuar opened pull request #6894: [Tests] Fix BIP65 p2p test (master...fix-bip65-p2p-test) https://github.com/bitcoin/bitcoin/pull/6894
3422015-10-27T19:59:03  <morcos> sipa: BlueMatt: sdaftuar and i were just talking.  the ccoinsviewcache growing out of control.  the biggest problem is a reorg!  a reorg will cause removeCoinbaseSpends to run, which will pull every single txin into the cache
3432015-10-27T19:59:19  <morcos> i assume it'll flush right after that, but at that point it'll be too late if you OOM'ed
3442015-10-27T19:59:41  <sipa> hmmm
3452015-10-27T20:00:16  <morcos> i think storing this valid height/time will be very useful for coinbase spends, and moderately useful for checklocktime and csv
3462015-10-27T20:01:04  <morcos> arguably we should actually store 3 numbers, the height/time its valid at, and the height which if you reorg below you need to recalculate validity
3472015-10-27T20:01:24  <morcos> then on a reorg we just scan the whole mempool checking that last number, and recalculate validity
3482015-10-27T20:01:48  <morcos> unfortunatley anti-fee sniping will cause any txs which have arrived in the last couple blocks to be triggered for recalc
3492015-10-27T20:02:17  <morcos> and wihtout storing more info, you'd have to recalc for coinbase, checklocktime and bip68 sequence, bc you don't know which was invalidated
3502015-10-27T20:02:25  <morcos> so its a bit of an expensive recalc
3512015-10-27T20:06:37  *** Thireus1 has joined #bitcoin-core-dev
3522015-10-27T20:10:18  *** Thireus has quit IRC
3532015-10-27T20:31:34  *** zooko has quit IRC
3542015-10-27T20:35:08  <BlueMatt> morcos: no, there is no current tie breaker in CreateNewBlock
3552015-10-27T20:35:37  <morcos> BlueMatt: sure why not, it uses TxPriorityCompare right?  which sorts by fee then priority
3562015-10-27T20:36:51  <BlueMatt> morcos: the way I read it, it first creates a heap according to priority, then resorts to create a heap using fee
3572015-10-27T20:36:55  <BlueMatt> it never uses fee then priority
3582015-10-27T20:37:31  <morcos> yes correct, but if you look at that compartor, in either case it uses the other one as the tie break
3592015-10-27T20:37:51  <morcos> comparatator
3602015-10-27T20:38:33  <BlueMatt> hmmmm, indeed, I hadnt seen that before....how is it that ordering seems randomish, then?
3612015-10-27T20:39:08  <BlueMatt> lemme go look at mempool
3622015-10-27T20:40:25  <morcos> yeah i was wondering that, you would think that would be fairly consistent across nodes
3632015-10-27T20:41:25  <morcos> i wonder if in this recent attack there are many things with also the same priority...  seems a bit unlikely..
3642015-10-27T20:41:50  <sipa> that was my assumption
3652015-10-27T20:42:04  <sipa> if all outputs are created in the same block, and the spends are the same size
3662015-10-27T20:42:10  <sipa> that's what you would get
3672015-10-27T20:57:38  *** zooko has joined #bitcoin-core-dev
3682015-10-27T21:03:54  *** Thireus has joined #bitcoin-core-dev
3692015-10-27T21:04:46  *** ParadoxSpiral has quit IRC
3702015-10-27T21:07:50  *** Thireus1 has quit IRC
3712015-10-27T21:27:08  *** Thireus1 has joined #bitcoin-core-dev
3722015-10-27T21:27:47  *** Thireus2 has joined #bitcoin-core-dev
3732015-10-27T21:28:17  *** Thireus1 has quit IRC
3742015-10-27T21:28:30  *** Thireus1 has joined #bitcoin-core-dev
3752015-10-27T21:29:51  *** Thireus3 has joined #bitcoin-core-dev
3762015-10-27T21:30:28  *** Thireus has quit IRC
3772015-10-27T21:30:30  *** Thireus2 has quit IRC
3782015-10-27T21:33:51  *** Thireus1 has quit IRC
3792015-10-27T21:51:12  *** deepcore has joined #bitcoin-core-dev
3802015-10-27T21:55:21  <BlueMatt> morcos: hmmmm, now I'm very confused
3812015-10-27T21:55:32  <BlueMatt> morcos: there are almost no txn in my mempool which have the same fee and prio
3822015-10-27T21:56:41  <morcos> BlueMatt: I didn't look into the root issue, its just that you are not seeing the blocks you expect to see?  It's not a result of malleated txs?  Where some nodes have one version and others have another
3832015-10-27T21:59:43  <BlueMatt> it could be related to malleated txn, but ive definitely seen it every time mempools get large, not necessarily when malleation is happening
3842015-10-27T22:01:43  <sipa> might it just be different miners using different policies?
3852015-10-27T22:05:50  <BlueMatt> afaiu there are effectively no different policies in use by miners today
3862015-10-27T22:05:54  <BlueMatt> (aside from eligius)
3872015-10-27T22:06:30  *** droark has joined #bitcoin-core-dev
3882015-10-27T22:06:59  *** deepcore has quit IRC
3892015-10-27T22:12:24  *** deepcore has joined #bitcoin-core-dev
3902015-10-27T22:14:36  *** dcousens has joined #bitcoin-core-dev
3912015-10-27T22:50:35  *** deepcore has quit IRC
3922015-10-27T23:11:17  <dcousens> when are the IRC meetings planned?
3932015-10-27T23:11:22  <dcousens> ooi
3942015-10-27T23:15:21  <sipa> thursday 7pm UTC
3952015-10-27T23:15:28  <dcousens> ta
3962015-10-27T23:27:08  *** Thireus has joined #bitcoin-core-dev
3972015-10-27T23:29:57  *** Thireus3 has quit IRC
3982015-10-27T23:33:52  *** Thireus has quit IRC
3992015-10-27T23:34:05  *** Thireus has joined #bitcoin-core-dev
4002015-10-27T23:39:53  *** Thireus has quit IRC
4012015-10-27T23:40:13  *** Thireus has joined #bitcoin-core-dev
4022015-10-27T23:51:34  *** Thireus1 has joined #bitcoin-core-dev
4032015-10-27T23:54:50  *** Thireus has quit IRC