12015-10-26T00:07:18 *** moli has joined #bitcoin-core-dev
22015-10-26T00:10:15 *** molly has quit IRC
32015-10-26T00:25:31 *** CodeShark has quit IRC
42015-10-26T00:25:35 *** CodeShark_ has quit IRC
52015-10-26T00:25:45 *** CodeShark has joined #bitcoin-core-dev
62015-10-26T00:38:45 *** molly has joined #bitcoin-core-dev
72015-10-26T00:40:07 *** evoskuil has quit IRC
82015-10-26T00:41:56 *** moli has quit IRC
92015-10-26T01:10:19 *** evoskuil has joined #bitcoin-core-dev
102015-10-26T01:13:46 *** CodeShark has quit IRC
112015-10-26T01:14:20 *** Ylbam has quit IRC
122015-10-26T01:23:10 *** jgarzik has quit IRC
132015-10-26T02:01:01 *** dgenr8 has quit IRC
142015-10-26T02:02:29 *** dgenr8 has joined #bitcoin-core-dev
152015-10-26T02:40:28 *** d_t has quit IRC
162015-10-26T02:47:45 *** belcher has quit IRC
172015-10-26T02:54:26 <BlueMatt> someone wanna kick travis on #6875
182015-10-26T02:57:49 <sipa> done
192015-10-26T02:58:42 *** dgenr8 has quit IRC
202015-10-26T02:59:37 *** dgenr8 has joined #bitcoin-core-dev
212015-10-26T03:05:37 *** dgenr8 has quit IRC
222015-10-26T03:29:42 *** d_t has joined #bitcoin-core-dev
232015-10-26T04:16:51 *** PaulCape_ has joined #bitcoin-core-dev
242015-10-26T04:19:10 *** PaulCapestany has quit IRC
252015-10-26T04:19:21 *** d_t has quit IRC
262015-10-26T04:20:24 *** JoeLiu has joined #bitcoin-core-dev
272015-10-26T04:24:31 *** d_t has joined #bitcoin-core-dev
282015-10-26T04:30:36 *** dgenr8 has joined #bitcoin-core-dev
292015-10-26T05:08:28 *** go1111111 has joined #bitcoin-core-dev
302015-10-26T05:13:45 *** d_t has quit IRC
312015-10-26T05:32:54 *** d_t has joined #bitcoin-core-dev
322015-10-26T06:25:48 *** JoeLiu has quit IRC
332015-10-26T07:06:33 <GitHub133> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/46f74379b86b...450893769f7a
342015-10-26T07:06:33 <GitHub133> bitcoin/master ceb2a9c Wladimir J. van der Laan: doc: mention BIP65 softfork in bips.md
352015-10-26T07:06:34 <GitHub133> bitcoin/master 4508937 Wladimir J. van der Laan: Merge pull request #6879...
362015-10-26T07:06:43 <GitHub137> [bitcoin] laanwj closed pull request #6879: doc: mention BIP65 softfork in bips.md (master...2015_10_bip65) https://github.com/bitcoin/bitcoin/pull/6879
372015-10-26T07:21:23 <GitHub163> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/450893769f7a...867d6c90b850
382015-10-26T07:21:23 <GitHub163> bitcoin/master dca7bd3 Wladimir J. van der Laan: doc: Add developer notes about gitignore...
392015-10-26T07:21:24 <GitHub163> bitcoin/master 867d6c9 Wladimir J. van der Laan: Merge pull request #6878...
402015-10-26T07:21:28 <GitHub17> [bitcoin] laanwj closed pull request #6878: doc: Add developer notes about gitignore (master...2015_10_ignore_files) https://github.com/bitcoin/bitcoin/pull/6878
412015-10-26T07:35:06 <jonasschnelli> gmaxwell: re: -maxuploadtarget. Will try to complete it today. Need to adapt sdaftuar tests and rebase/squash.
422015-10-26T07:45:17 *** paveljanik has quit IRC
432015-10-26T07:45:36 *** paveljanik has joined #bitcoin-core-dev
442015-10-26T07:45:47 *** paveljanik has quit IRC
452015-10-26T08:06:13 *** Arnavion has quit IRC
462015-10-26T08:07:04 *** Arnavion has joined #bitcoin-core-dev
472015-10-26T08:07:26 *** Ylbam has joined #bitcoin-core-dev
482015-10-26T08:10:04 <GitHub137> [bitcoin] laanwj pushed 7 new commits to master: https://github.com/bitcoin/bitcoin/compare/867d6c90b850...5242bb32c723
492015-10-26T08:10:05 <GitHub137> bitcoin/master 4d2a926 dexX7: Ignore coverage data related and temporary test files
502015-10-26T08:10:05 <GitHub137> bitcoin/master d425877 dexX7: Remove coverage and test related files, when cleaning up...
512015-10-26T08:10:06 <GitHub137> bitcoin/master 8e3a27b dexX7: Require Python for RPC tests, when using lcov...
522015-10-26T08:10:15 <GitHub173> [bitcoin] laanwj closed pull request #6813: Support gathering code coverage data for RPC tests with lcov (master...btc-test-lcov-rpc) https://github.com/bitcoin/bitcoin/pull/6813
532015-10-26T08:34:54 <jonasschnelli> anyone getting the same build errors (osx) for current master: boost: mutex lock failed in pthread_mutex_lock: Invalid argument?
542015-10-26T08:35:06 * jonasschnelli is doing a fresh checkout / build
552015-10-26T08:43:30 <jonasschnelli> hmm.. same issue when doing a fresh checkout / build...
562015-10-26T08:44:03 <wumpus> is that a *build* error?
572015-10-26T08:44:20 <jonasschnelli> libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument
582015-10-26T08:44:32 <wumpus> looks like a runtime errror to me, something abusing boost?
592015-10-26T08:45:00 <jonasschnelli> i try now to upgarde from 1.57 to 1.58...
602015-10-26T08:45:09 <wumpus> let's first try to debug it
612015-10-26T08:45:15 <wumpus> when does it happen? can you get a traceback?
622015-10-26T08:45:26 <jonasschnelli> stacktrace stops when program tries to call the first LOCK
632015-10-26T08:46:02 <wumpus> ok
642015-10-26T08:46:17 <jonasschnelli> EnterCritical() / push_lock()
652015-10-26T08:46:20 <wumpus> when was the last time it worked? maybe try a git bisect
662015-10-26T08:46:42 <jonasschnelli> that is strange: *pthread_mutex_lock: Invalid argument*
672015-10-26T08:47:10 <jonasschnelli> wumpus: okay... i'll try to find out if it's related to a source change or something on my local machine.
682015-10-26T08:47:12 <wumpus> can be a use-after-free of a lock
692015-10-26T08:48:21 <wumpus> if it happens at startup it may be a unexpected error path that doesn't wind down properly
702015-10-26T09:01:27 *** BashCo has quit IRC
712015-10-26T09:02:03 *** BashCo has joined #bitcoin-core-dev
722015-10-26T09:06:39 *** BashCo has quit IRC
732015-10-26T09:07:22 <GitHub136> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5242bb32c723...26f5b34e8838
742015-10-26T09:07:23 <GitHub136> bitcoin/master 10e2eae Wladimir J. van der Laan: rpc: Add maxmempool and effective min fee to getmempoolinfo
752015-10-26T09:07:23 <GitHub136> bitcoin/master 26f5b34 Wladimir J. van der Laan: Merge pull request #6877...
762015-10-26T09:07:32 <GitHub170> [bitcoin] laanwj closed pull request #6877: rpc: Add maxmempool and effective min fee to getmempoolinfo (master...2015_10_mempool_effective_fee) https://github.com/bitcoin/bitcoin/pull/6877
772015-10-26T09:14:19 *** dcousens has joined #bitcoin-core-dev
782015-10-26T09:19:18 *** BashCo has joined #bitcoin-core-dev
792015-10-26T09:24:43 *** AtashiCon has quit IRC
802015-10-26T09:27:56 *** rubensayshi has joined #bitcoin-core-dev
812015-10-26T09:28:11 *** AtashiCon has joined #bitcoin-core-dev
822015-10-26T09:33:32 <dcousens> gmaxwell: with the address index comment, you mention it might drive away users, but, this is an opt-in patch
832015-10-26T09:34:38 <dcousens> I'd much rather use this than my own self-maintained version of the patch, hell, several block explorers I know have written their own implementations eating up much more than 50GB of space etc
842015-10-26T09:38:46 <btcdrak> dcousens: FYI, my .bitcoin folder is 67GB with addrindex turned on. What is it without?
852015-10-26T09:39:16 <dcousens> btcdrak: moment, just checking
862015-10-26T09:40:03 <dcousens> 56G
872015-10-26T09:40:14 <dcousens> with -txindex
882015-10-26T09:40:27 <btcdrak> dcousens: but as a maintainer of an addrindex fork, I think this particular implementation is absolutely not the right way to go about it. It's ugly. I'd much prefer to see something like #5048 which maintains a separate index entirely.
892015-10-26T09:40:35 <dcousens> Oh no doubt
902015-10-26T09:40:38 <dcousens> I meant in concept only
912015-10-26T09:40:53 <dcousens> Hence, only my concept ACK
922015-10-26T09:41:53 <dcousens> There might be a way to do this as gmaxwell pointed out, via fast re-scan or something
932015-10-26T09:42:17 <dcousens> It would just have to be super fast in the case of being feasible for block explorers etc, but, I guess you could just throw some caches/LBs in front of it
942015-10-26T09:43:53 <btcdrak> dcousens: yeah, I'm not sure how addrindex would scale in the long term.
952015-10-26T09:45:09 <dcousens> yeah, but I mean, we have that same concern for the blockchain itself haha
962015-10-26T09:45:28 <dcousens> I just figured, in terms of worrying about users losing resources, in the end, their opting-in, so they'd be aware no?
972015-10-26T09:46:39 <btcdrak> dcousens: I'm no longer as passionate because I provide a fork for those that want it. It's easy enough to maintain.
982015-10-26T09:46:46 <wumpus> indexing the whole block chain is generally the wrong thing to do
992015-10-26T09:47:57 <wumpus> and I understand there are forensic or research reasons to do so, but if you end up at that it's generally an indication you're doing something wrong in your design
1002015-10-26T09:48:35 <dcousens> wumpus: that really depends on your trust model though?
1012015-10-26T09:48:49 <wumpus> no, it depends on your scaling model
1022015-10-26T09:49:44 <wumpus> it's best to regard the block chain as transitory data used to update the utxo state
1032015-10-26T09:50:02 <wumpus> (as well as other things, such as wallets)
1042015-10-26T09:51:40 <dcousens> agreed, but the UTXO may not be all you care about. But yes your right, in that sense, its your scaling model
1052015-10-26T09:52:05 <wumpus> that's why I added (as well as other things) - it's data you want to use to apply changes to your running state, then throw away
1062015-10-26T09:52:43 <dcousens> wumpus: but that lacks the information in the case of a user wanting to see the 'history' of a certain address
1072015-10-26T09:52:51 <wumpus> bitcoind's wallet, as one exaple, does this correctly. It only requires the whole beast for rescans. It would be nice to have an external example though that isn't embedded into bitcoind.
1082015-10-26T09:52:51 <dcousens> the only way to generate that history would be to view all hte transitory data
1092015-10-26T09:53:10 <wumpus> the way to generate that history would be to store it as it comes along, as metadata
1102015-10-26T09:53:33 <dcousens> that assumes you know the address from the start
1112015-10-26T09:53:38 <wumpus> the same way bitcoind's wallet does - store the transactions, or whatever you need to list it later
1122015-10-26T09:54:29 <dcousens> for example for a wallet I maintain, we very often have users that come along with BIP39 mnemonics with 100's of used addresses, forcing a rescan for each time they introduce a new mnemonic/wallet would be a huge rescan burden for us
1132015-10-26T09:54:36 <wumpus> history (or the part of history that you want to store for accounting purposes) is part of your running state
1142015-10-26T09:54:45 <dcousens> Which again, relates to our scaling model, but, that too relates directly to this patch (as a concept)
1152015-10-26T09:57:03 <wumpus> yes, if you essentially offer rescanning-as-a-service it makes sense to keep an index. I still think it should be something external though, not part of bitcoind. There are tons of differents kinds of indexes one could want, for specific purposes
1162015-10-26T09:57:48 <wumpus> ideally some external indexing tool would source block data from bitcoind then keep an index, as running state
1172015-10-26T09:59:27 <wumpus> so my points are: do not encourage people to keep that index, and do not clutter the codebase with all kinds of extra indexes not necessary for node functionaity. It is not "never create an index". sure there may be reasons to do so but it's very custom
1182015-10-26T10:00:24 <dcousens> wumpus: hmmm, but, maintaing this indexing is not a simple task
1192015-10-26T10:00:37 <btcdrak> wumpus: I think I finally see your point
1202015-10-26T10:00:40 <wumpus> I haven't used the word 'simple'
1212015-10-26T10:00:45 <dcousens> It requires a lot of complex logic in regards to re-orgs etc
1222015-10-26T10:01:00 <dcousens> wumpus: I know, just putting it out there as to why this is probably a persistent feature request
1232015-10-26T10:01:08 <wumpus> yes, it requires logic with regard to re-orgs. To be precise, you need to be able to roll back updates from a block.
1242015-10-26T10:01:33 <wumpus> bitcoind does this by keeping undo files, but your database may have features to do so as well
1252015-10-26T10:02:08 <wumpus> and once this code is written people can just use it, it's no more complex than using your patch
1262015-10-26T10:02:38 <dcousens> wumpus: in my mind, it seems more like its just a way to offload the maintenance burden haha
1272015-10-26T10:03:44 <btcdrak> why dont we use zmq to notify external apps?
1282015-10-26T10:03:59 <wumpus> there's --enable-zmq?
1292015-10-26T10:04:41 <dcousens> wumpus: is there a re-org notification?
1302015-10-26T10:04:48 <dcousens> That would make the roll backs super simple
1312015-10-26T10:04:53 <btcdrak> it could store whatever it wants, bitcoind would notify about reorgs etc. that would allow to build a sort of indexd. The indexd could always query bitcoind for things like blocks as a passthrough
1322015-10-26T10:05:46 <wumpus> not sure - there is a notification when there is a new block, you can use that to determine if there has been a reorg. But a more direct notification of what blocks have to be un-done/applies might be useful too. Though be careful, zmq notifications can be lossy, so you shouldn't rely on their content.
1332015-10-26T10:06:08 <wumpus> btcdrak: right
1342015-10-26T10:06:18 <wumpus> indexd would pull the data it requires from bitcoind
1352015-10-26T10:06:20 <btcdrak> wumpus: if we're missing notifications I'm sure they are easy to add.
1362015-10-26T10:06:36 <dcousens> wumpus: really? Are they consistently lossy?
1372015-10-26T10:07:02 <wumpus> dcousens: yes, if the queue is full new messages are discarded
1382015-10-26T10:07:54 <dcousens> wumpus: right, so its just a matter of consumption
1392015-10-26T10:08:35 <wumpus> not sure if are any guarantees for delivery - that's why I have always been kind of careful there, it's easy to use it in a wrong way. Notifications are "hey, wake up, there is new information". If those get lost there's no big issue, it will pick them up next time.
1402015-10-26T10:10:21 <dcousens> wumpus: mmm, I guess, in the end, its as btcdrak as suggested. The underyling problem is that the maintenance of the address index becomes difficult due to re-org detection. If we can have bitcoind provide us with that information in a clear way, most of this becomes trivial
1412015-10-26T10:11:49 <wumpus> I'm not sure reorg detection is so difficult? keep the list of hashes up to the current block, then if a new best block comes in, look at its parents, where it intersects with the list of your blocks is the fork/reorg point
1422015-10-26T10:12:05 <dcousens> wumpus: requires maintaing another list, another source of data
1432015-10-26T10:12:10 <wumpus> yes it does
1442015-10-26T10:12:58 <wumpus> that's life, you need to maintain the data structures that you need to do your work. only be keeping this state yourself you can be sure your state is consistent.
1452015-10-26T10:13:30 <btcdrak> indexd could even use bitcoind for SPV lookups
1462015-10-26T10:14:04 <wumpus> would be great to have some library or existing software to handle this for you, I'm just talking concepts here not implementation
1472015-10-26T10:14:33 <dcousens> btcdrak: I was thinking about that
1482015-10-26T10:14:55 <dcousens> you don't even need to verify since you'd already trust your own node
1492015-10-26T10:15:10 <wumpus> right - you outsource trust to your node, verification, that's what it's for
1502015-10-26T10:15:24 <dcousens> just as wumpus said, probably still need to maintain the hash-chain if theres no notification for re-orgs
1512015-10-26T10:15:43 <wumpus> well you need to know what hash-chain your state is based on
1522015-10-26T10:16:54 <dcousens> well, not really, all you actually care about is what blocks got reverted, such that you can roll back the relevant transactions in the index
1532015-10-26T10:17:08 <wumpus> otherwise the couling is dangerously tight, say, your indexd was disconnected from bitcoind for a while and you reconnect and have to catch up
1542015-10-26T10:17:17 <wumpus> coupling*
1552015-10-26T10:17:42 <wumpus> yes, but you need to detect that yourself. You can't rely on bitcoind to keep state for you
1562015-10-26T10:17:59 <dcousens> wumpus: it would know what blocks get reverted in a re-org, no?
1572015-10-26T10:18:31 <wumpus> but only in realtime - ideally you don't want that kind of tight coupling. What if indexd wasn't connected at the time the reorg happens. Do you want to restart all the way from the beginning?
1582015-10-26T10:18:58 <dcousens> true
1592015-10-26T10:19:05 <wumpus> you have a) your own hash chain b) bitcoind's hash chain , you always want to go from a to b
1602015-10-26T10:19:42 <wumpus> (given that you trust bitcoind to give you the best chain)
1612015-10-26T10:20:28 <jonasschnelli> wumpus: i can successfully build up to https://github.com/bitcoin/bitcoin/commit/579b863cd7586b98974484ad55e19be2a54d241d
1622015-10-26T10:20:48 <jonasschnelli> wumpus: https://github.com/bitcoin/bitcoin/commit/a09297010e171af28a7a3fcad65a4e0aefea53ba seams to break things.. how can this even be possible...
1632015-10-26T10:20:59 <jonasschnelli> no code changes at all
1642015-10-26T10:21:02 <dcousens> btcdrak: happy to work with you on indexd concept
1652015-10-26T10:21:12 <wumpus> would be awesome if someone implemented that
1662015-10-26T10:21:40 <wumpus> jonasschnelli: huh that just changes my silly dev tools :)
1672015-10-26T10:22:01 <jonasschnelli> yeah. I know... let me investigate more
1682015-10-26T10:28:02 <wumpus> so 0fbfc51 works?
1692015-10-26T10:30:35 <wumpus> (that's the last commit before merging #6854)
1702015-10-26T10:33:06 <GitHub168> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/26f5b34e8838...ff057f41aa14
1712015-10-26T10:33:07 <GitHub168> bitcoin/master 9d55050 Mark Friedenbach: Add rules--presently disabled--for using GetMedianTimePast as endpoint for lock-time calculations...
1722015-10-26T10:33:07 <GitHub168> bitcoin/master dea8d21 Mark Friedenbach: Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints...
1732015-10-26T10:33:08 <GitHub168> bitcoin/master ff057f4 Wladimir J. van der Laan: Merge pull request #6566...
1742015-10-26T10:33:11 <GitHub161> [bitcoin] laanwj closed pull request #6566: BIP-113: Mempool-only median time-past as endpoint for lock-time calculations (master...medianpasttimelock) https://github.com/bitcoin/bitcoin/pull/6566
1752015-10-26T10:36:38 <btcdrak> dcousens: sure.
1762015-10-26T10:38:13 <dcousens> wumpus: would it better to maintain indexd in repo or as a separate application?
1772015-10-26T10:39:48 <wumpus> well - better is always to have it completely detached, but I suppose developing it inside the repository is easier for now if you need serialization and certain data structures
1782015-10-26T10:40:04 <wumpus> we don't have a way to export those as a library yet
1792015-10-26T10:40:22 <jonasschnelli> wumpus : found the problem: https://github.com/bitcoin/bitcoin/pull/6722/files#diff-ca74c4b28865382863b8fe7633a85cd6R312
1802015-10-26T10:40:40 <dcousens> wumpus: aye
1812015-10-26T10:40:50 <jonasschnelli> clear() calls LOCK() in cxx_global_var_init()
1822015-10-26T10:41:15 <jonasschnelli> i think this should be changed.
1832015-10-26T10:42:41 *** BashCo_ has joined #bitcoin-core-dev
1842015-10-26T10:43:19 *** BashCo has quit IRC
1852015-10-26T10:43:50 *** BashCo has joined #bitcoin-core-dev
1862015-10-26T10:44:06 <wumpus> common, straightforward to do this is to separate it out into a private function, say clear_(), and a public function clear() that gets the lock and calls clear_()
1872015-10-26T10:44:35 <wumpus> this way, functions that already have the lock, or don't need it otherwise, can call clear_ instead
1882015-10-26T10:45:06 <wumpus> in this case it is the constructor so obviously it doesn't need the lock
1892015-10-26T10:45:28 <jonasschnelli> okay... can try a patch.
1902015-10-26T10:47:32 *** BashCo_ has quit IRC
1912015-10-26T10:49:14 <wumpus> not that it should normally hurt to lock a lock that is part of your object inthe constructor... but this looks like some global obscure initialization order thing because the mempool object is global :(
1922015-10-26T10:51:57 <wumpus> (probably it would be better to use an explicit lifetime for objects such as these, but fixing that is more than you bargained for)
1932015-10-26T10:54:59 *** dcousens has quit IRC
1942015-10-26T11:03:50 *** dcousens has joined #bitcoin-core-dev
1952015-10-26T11:20:35 *** jtimon has quit IRC
1962015-10-26T11:32:25 <Luke-Jr> huh, the wallet rescan % shown in the splash screen is much higher than the debug.log Progress - block count vs tx count? :/
1972015-10-26T11:36:13 <wumpus> GUIstd::max(1, std::min(99, (int)((Checkpoints::GuessVerificationProgress(chainParams.Checkpoints(), pindex, false) - dProgressStart) / (dProgressTip - dProgressStart) * 100)))
1982015-10-26T11:36:23 <wumpus> log: Checkpoints::GuessVerificationProgress(chainParams.Checkpoints(), pindex)
1992015-10-26T11:37:51 <wumpus> the GUI takes into account the starting point, which you'd expect would result in it showing a *lower* progress % when it makes a difference
2002015-10-26T11:38:01 <wumpus> but the GUI also has fSigchecks=false for GuessVerificationprogress, maybe tha'ts it
2012015-10-26T12:42:47 <wumpus> this is curious, anyone else having problems with address parsing on windows 10? https://github.com/bitcoin/bitcoin/issues/6886
2022015-10-26T12:44:13 <wumpus> looks like the IP parsing (using getaddrinfo) always returns 0:0:0:0:0:0:0:0/128, IPv4 parsing as well as IPv6 parsing fails
2032015-10-26T12:49:35 *** jtimon has joined #bitcoin-core-dev
2042015-10-26T12:51:13 *** ParadoxSpiral has joined #bitcoin-core-dev
2052015-10-26T13:57:56 <GitHub175> [bitcoin] jonasschnelli opened pull request #6889: fix locking issue with new mempool limiting (master...2015/10/fix_mempool_lock) https://github.com/bitcoin/bitcoin/pull/6889
2062015-10-26T14:07:30 *** Guest25608 has quit IRC
2072015-10-26T14:07:36 *** treehug88 has joined #bitcoin-core-dev
2082015-10-26T14:14:00 *** pigeons has joined #bitcoin-core-dev
2092015-10-26T14:14:24 *** pigeons is now known as Guest20816
2102015-10-26T14:18:07 *** danielsocials has joined #bitcoin-core-dev
2112015-10-26T14:23:37 *** danielsocials has quit IRC
2122015-10-26T14:26:45 *** danielsocials has joined #bitcoin-core-dev
2132015-10-26T14:46:36 <morcos> Does anybody have any thoughts on storing the number of sigOps in a tx in the CTxMempoolEntry?
2142015-10-26T14:47:06 <morcos> We calculate this on ATMP, and we need it in CreateNewBlock, but it is on the expensive'ish side to calculate.
2152015-10-26T14:48:24 <sipa> sounds good
2162015-10-26T14:48:40 <morcos> Wow, I was prepared to have to fight for that... :)
2172015-10-26T14:50:29 <morcos> I was naively hoping you could just assume every scriptSig was satisfying a P2SH and you wouldn't over count by too much, but turns out that is NOT the case..
2182015-10-26T14:55:39 <dgenr8> morcos: any plans to cache ancestor pkg sums the way descendants are?
2192015-10-26T14:56:59 <morcos> dgenr8: sdaftuar has written something which he might publish soon. but i don't think there is any point in merging it until we have mining code that can take advantage of it.
2202015-10-26T14:57:15 <morcos> i'm starting by just trying to make the existing mining code much faster
2212015-10-26T14:59:03 <gmaxwell> 02:58 < gmaxwell> Is there a reason that for fee purposes we are not using max(size, sigops*BLOCK_MAX_BYTES/BLOCK_MAX_SIGOPS) as the size of a transaction?
2222015-10-26T14:59:26 <gmaxwell> as in "the size" that we use for mempool/etc. purposes.
2232015-10-26T15:02:07 <gmaxwell> dcousens: Opt-in isn't sufficient to prevent it from driving users out, I do explain this...
2242015-10-26T15:02:33 <morcos> gmaxwell: i saw that. on my list somewhere is to research algorithms for satisfying multiple constraints. presumably if we know that actual size is usually the limiting factor, then it doesn't seem right to sort something lower b/c it has a lot of sigops even though you're not going to hit that limit
2252015-10-26T15:02:56 <morcos> i was thnking about it in the terms of some future consensus requirement on utxo deltas as well
2262015-10-26T15:03:11 *** bsm1175321 has quit IRC
2272015-10-26T15:05:05 <morcos> the discussion we were having the other day about a separate thread for template generation seems very difficult to me. its very hard to extract code from needing to lock at least the mempool.
2282015-10-26T15:05:45 <morcos> i think to do that, we might need 2 mempools, or to just allow the mempool to queue tx's while the mining code is running
2292015-10-26T15:06:43 <morcos> but its an easy win to make existing createnewblock significantly faster just by adding an index on individual tx feerate, so i figured i'd start with that.
2302015-10-26T15:12:04 *** bsm1175321 has joined #bitcoin-core-dev
2312015-10-26T15:13:58 *** BashCo_ has joined #bitcoin-core-dev
2322015-10-26T15:17:35 *** BashCo has quit IRC
2332015-10-26T15:19:27 <gmaxwell> morcos: I've generally assumed the multivariable optimization would be much harder and not really worth it-- the sigops limit is high enough that it doesn't normally have an effect. What I suggested at least prevents the case where some silly dos attack transaction with a high per byte fee kills a feerate greedy selection.
2342015-10-26T15:20:41 <gmaxwell> morcos: hm. So I didn't expect the seperate template thread would avoid locking the mempool. I assumed it would. But that having it would allow the createnewblock to always run without taking any long-held locks.
2352015-10-26T15:21:33 <gmaxwell> E.g. block creation thread would lock the mempool and build a block. Then take a blocktemplate lock for just an instant to replace the last template.
2362015-10-26T15:21:45 <morcos> gmaxwell: ah, interesting, so trick the mining code into producing small blocks unecessarily.. maybe that is a good idea.
2372015-10-26T15:23:33 <morcos> gmaxwell: i see. i guess i need to understand how getblocktemplate is used in practice. i was assuming the threading was so we could be optimizing all the time. i didn't realize it was because you wanted to return a previously generated template to someone calling gbt. i assumed they had already gotten the previous template
2382015-10-26T15:24:07 <gmaxwell> and if the cached template is out of date with the chain when gbt runs, you just generate an empty template, which you can do without touching he mempool.
2392015-10-26T15:25:32 <morcos> yeah, so i did write a threaded version that does what you suggested, but it was bad, because it was basically just busy grabbing cs_main and mempool, and i thought well ok we'll have to make it run only every so often, but then i didn't see why that was better than the existing usage
2402015-10-26T15:26:02 <morcos> but you want to optimize for the case where the caller currently has nothing valid to work on?
2412015-10-26T15:26:05 <gmaxwell> When a new block is accepted by the node the miner process learns this (either via p2p inspection or via longpolling) and calls GBT. You want the lowest latency response possible; because until it returns the mining gear is working on a fork.
2422015-10-26T15:26:57 <gmaxwell> Then the mining clients will periodically poll to update the transaction list. In this polling you don't care if the transaction data returned is seconds stale.
2432015-10-26T15:27:04 <morcos> yes, understood that the new block case needs to be optimized, that was my next step, and that seems pretty easy
2442015-10-26T15:27:21 <gmaxwell> since all that does is deny you the little incremental fee rate from new transactions that came in.
2452015-10-26T15:28:12 <gmaxwell> We already do createnewblock caching. but it only helps with the second and later requests.
2462015-10-26T15:28:18 <morcos> so what i was assuming is that when you periodically poll, which is the common use case, you already have valid work, so the existing longpoll functionality is fine, and there is no need for background process
2472015-10-26T15:28:28 <morcos> yes right
2482015-10-26T15:29:30 *** jl2012 has quit IRC
2492015-10-26T15:29:54 *** jl2012 has joined #bitcoin-core-dev
2502015-10-26T15:30:10 <morcos> so i think what you're saying is that for instance a new block could come in while you happen to not be longpolling, so then it won't be until you call getblocktemplate that it even tries to do anything, and it woudl be better if it started immediately?
2512015-10-26T15:30:10 <gmaxwell> Ultimately it would be good to avoid the case where if you poll just before a new block is accepted, you don't delay getting work on the new block. Though, indeed, then I think you run into locking issues with the mempool.
2522015-10-26T15:31:23 <gmaxwell> morcos: yes, when a new block comes in and you have been mining it should immediately compute a new block template, and until thats done, return empty ones so you'll at least be working on extending the longest valid chain.
2532015-10-26T15:32:24 <gmaxwell> (a failure to make this optimization is part of what contributes to miners mining on other miners work without validating-- they go and 'optimize' at a layer higher than Bitcoin, and as a resut complex details like validation go out the window. :) )
2542015-10-26T15:32:51 <morcos> it seems like you'd want to be notified of 2 things, when there is a new block template (empty) and when there is the first template with txs based on the new block. how do you get that notification if you're not longpolling constantly?
2552015-10-26T15:36:22 <morcos> i was just going to change it so the longpool returns immediately with an empty block if a new block comes in, via a flag to create new block or something. and then your next long polling call to getblocktemplate also calls CNB immediately which will return with txs.
2562015-10-26T15:36:42 <gmaxwell> We could trigger the GBT LP's every N seconds or when the fees change by X, I don't actually know what (if anything) triggers the GBT LP now except a new block. I also don't know how widely GBT LP is used at the moment: RPC concurrency has been kinda broken for a long time (fixed in git master), and GBT LP was a late addition, so I think most things decide when to poll GBT based on monitoring
2572015-10-26T15:36:48 <gmaxwell> P2P and then otherwise run on a timer.
2582015-10-26T15:37:34 <morcos> ok... well doing it the threaded way shouldn't be hard either...
2592015-10-26T15:38:03 <gmaxwell> morcos: we do want to get people mining transactions quickly too-- as otherwise you're missing out on fees (and the public tends to get really angry when they're waiting for transactions and an empty block shows up. :) )
2602015-10-26T15:38:36 <morcos> gmaxwell: yes, will be much much faster, so as long as you poll again immediately, shouldn't be more than 100ms
2612015-10-26T15:39:37 <morcos> i'll probably have a lot more questions when our mining hardware shows up.. :)
2622015-10-26T15:43:07 <gmaxwell> There is some ... potential for "can't win" here, so there is some hardware that really dislikes being longpolled often. E.g. for some dumb reason (usually a complete lack of understanding about all the reasons you might LP) and they flush the pipeline completely. Now, GBT LPing doesn't necessarily mean the hardware gets disrupted, as some higher layer thing can ignore an early long poll and pi
2632015-10-26T15:43:12 <gmaxwell> ck up the non-empty template on a later timed run.
2642015-10-26T15:44:37 <gmaxwell> So it might be the case that we expose breakage if we start LPing 100ms after the last time we LPed, just because we now have transactions. But if so, we can probably add a flag to delay that second LP event.
2652015-10-26T15:44:52 <gmaxwell> (that same hardware is the stuff that cannot be used with P2Pool)
2662015-10-26T15:52:37 *** danielsocials has quit IRC
2672015-10-26T16:25:33 <GitHub52> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/ff057f41aa14...c8322ff7f754
2682015-10-26T16:25:33 <GitHub52> bitcoin/master 143d173 Eric Lombrozo: Use BOOST_CHECK_MESSAGE() rather than BOOST_CHECK() in alerts_tests.cpp and initialize strMiscWarning before calling PartitionCheck()."
2692015-10-26T16:25:34 <GitHub52> bitcoin/master c8322ff Wladimir J. van der Laan: Merge pull request #6888...
2702015-10-26T16:25:43 <GitHub96> [bitcoin] laanwj closed pull request #6888: Clear strMiscWarning before running PartitionAlert (master...alert_tests) https://github.com/bitcoin/bitcoin/pull/6888
2712015-10-26T16:34:07 *** challisto has quit IRC
2722015-10-26T16:42:19 <GitHub16> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/c8322ff7f754...dbc5ee821ecd
2732015-10-26T16:42:20 <GitHub16> bitcoin/master d4aa54c Kevin Cooper: added org.bitcoin.bitcoind.plist for launchd (OS X)
2742015-10-26T16:42:20 <GitHub16> bitcoin/master e04b0b6 Kevin Cooper: added OS X documentation to doc/init.md
2752015-10-26T16:42:21 <GitHub16> bitcoin/master dbc5ee8 Wladimir J. van der Laan: Merge pull request #6621...
2762015-10-26T16:42:21 <GitHub118> [bitcoin] laanwj closed pull request #6621: added org.bitcoin.bitcoind.plist for launchd (OS X) (master...master) https://github.com/bitcoin/bitcoin/pull/6621
2772015-10-26T16:48:13 *** CodeShark has joined #bitcoin-core-dev
2782015-10-26T16:54:20 <GitHub11> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/dbc5ee821ecd...7939164d8985
2792015-10-26T16:54:21 <GitHub69> [bitcoin] laanwj closed pull request #6622: Introduce -maxuploadtarget (master...2015/09/maxuploadtarget) https://github.com/bitcoin/bitcoin/pull/6622
2802015-10-26T16:54:21 <GitHub11> bitcoin/master 872fee3 Jonas Schnelli: Introduce -maxuploadtarget...
2812015-10-26T16:54:22 <GitHub11> bitcoin/master 17a073a Suhas Daftuar: Add RPC test for -maxuploadtarget
2822015-10-26T16:54:22 <GitHub11> bitcoin/master 7939164 Wladimir J. van der Laan: Merge pull request #6622...
2832015-10-26T16:54:36 *** MarcoFalke has joined #bitcoin-core-dev
2842015-10-26T17:06:19 *** d_t has joined #bitcoin-core-dev
2852015-10-26T17:13:12 *** BashCo_ has quit IRC
2862015-10-26T17:14:16 *** instagibbs_ is now known as instagibbs
2872015-10-26T17:21:23 *** PaulCape_ has quit IRC
2882015-10-26T17:22:52 *** PaulCapestany has joined #bitcoin-core-dev
2892015-10-26T17:32:16 *** BashCo has joined #bitcoin-core-dev
2902015-10-26T17:47:53 *** rubensayshi has quit IRC
2912015-10-26T18:33:55 <sipa> morcos, sdaftuar, BlueMatt: perhaps we need to talk a bit about what the expectation of the coincache + mempool size limiting is
2922015-10-26T18:34:44 <sipa> do we want a "all of the mempool's dependencies are guaranteed to always fit in memory" ?
2932015-10-26T18:36:29 <morcos> sipa: hmm, i don't think i was trying to argue for that. in fact i don't think it would be at all reasonable to do that.
2942015-10-26T18:37:04 <morcos> i think we should set the default sizes so that is true in general though
2952015-10-26T18:37:22 <dgenr8> Was reversing the feerate sort just an expressiveness change, or did it fix something? 78b82f4
2962015-10-26T18:37:55 <sdaftuar> dgenr8: expressiveness mostly, avoids an issue with using project
2972015-10-26T18:38:00 <sipa> morcos: one way to accomplish that is to just have a combined coincache+mempool memory limit size, and depending on whether the dependencies are large or small, effectively have a smaller resp. larger mempool limit as a result
2982015-10-26T18:38:36 <sipa> morcos: but if we don't have that, we _must_ enforce the coincache limit
2992015-10-26T18:38:38 <dgenr8> sdaftuar: thx!
3002015-10-26T18:38:41 <sdaftuar> i think that might be a little dangerous though because that has an effect on relay policy
3012015-10-26T18:39:01 <sipa> morcos: if the coincache exceeds its size, it will be wiped anyway when the next block comes
3022015-10-26T18:39:33 <sipa> calling flush in the tx code just makes it happen a bit earlier to avoid an OOM
3032015-10-26T18:39:36 <sdaftuar> sipa: i think we should revisit that behavior. ideally the coinscache would have in it the set of things likely to be needed when the next block arrives
3042015-10-26T18:39:40 <morcos> sipa: you realize that we can't always enforce the coincache size? when we're connecting a block it could temporarily exceed for example
3052015-10-26T18:40:15 <sipa> morcos: yeah...
3062015-10-26T18:40:18 <morcos> i agree that we should _eventually_ put in something that enforces limitation of the coincache size between blocks
3072015-10-26T18:40:26 <morcos> but that might as well wait for something smarter than just flushing
3082015-10-26T18:40:34 <sipa> i think the only real solution is to switching to a per-txout cache instead of a per-tx cache
3092015-10-26T18:40:45 <sipa> so the ratio is more or less fixed
3102015-10-26T18:40:45 <sdaftuar> ah i was wondering how feasible that would be
3112015-10-26T18:40:54 <morcos> and in the meantime we set the ratio of the defaults to be reasonable so that we don't usually blow through the cache limit
3122015-10-26T18:41:26 <morcos> given that you have to pay for relay to increase the cache size there is some cost to the attack after all
3132015-10-26T18:41:47 <morcos> and how big are we worried the cache size could get?
3142015-10-26T18:41:48 <sipa> define usually
3152015-10-26T18:42:57 <morcos> i guess i'm thinking that if we set mempool to 300M then it'd be rare than the coinsview cache was over 600M or something
3162015-10-26T18:43:13 <morcos> thats just a lot bigger than the current default
3172015-10-26T18:45:26 <morcos> i just think flushing after txs should be rare, if we want to set some higher limit to protect against OOM and flush after txs there, then thats fine, lets just set all these defaults so that what happens usually is the cache is big enough to contain a good chunk of the mempool. it doesn't really have to contain the whole thing, because
3182015-10-26T18:45:33 <morcos> you wont' get to 100% mempool size in between blocks
3192015-10-26T19:08:18 *** jl2012_ has joined #bitcoin-core-dev
3202015-10-26T19:08:24 *** jl2012 has quit IRC
3212015-10-26T19:33:22 <phantomcircuit> indeed why are we keeping the already accepted mempool dependencies in the cache? they're already validated isn't that going to reduce the cache hit % ?
3222015-10-26T19:34:19 <sipa> phantomcircuit: they're validated again when a block is received
3232015-10-26T19:34:25 <sipa> which we want to make fast
3242015-10-26T19:35:01 <phantomcircuit> sipa, ah that's right
3252015-10-26T19:35:05 <sipa> an approach to avoid that is caching the success, but if you store that cache value in the mempool, you're making the mempool code consensus critical
3262015-10-26T19:35:21 <phantomcircuit> yeah lets not do that :)
3272015-10-26T19:35:26 <sipa> eh, and that doesn't even help
3282015-10-26T19:35:39 <sipa> signature validation is already cached anyway
3292015-10-26T19:35:49 <phantomcircuit> oh that reminds me
3302015-10-26T19:35:59 <sipa> and you need to be able to update the spends in the chainstate anyway after the block is validated
3312015-10-26T19:36:08 <phantomcircuit> the size of the sig cache isn't in the help menu
3322015-10-26T19:36:24 <sipa> good point
3332015-10-26T19:36:25 <phantomcircuit> it significantly reduces worst case AcceptNewBlock() latency to increase that
3342015-10-26T19:36:40 <phantomcircuit> miners should basically all have that be 10-100x the default
3352015-10-26T19:36:51 <gmaxwell> what?
3362015-10-26T19:37:21 <gmaxwell> the entries are _tiny_, we should be able to have a "big enough for ~100% hitrate" cache for everyone.
3372015-10-26T19:37:47 <sipa> i actually have no idea what size the sigcache is
3382015-10-26T19:39:04 <phantomcircuit> huh it is in the help
3392015-10-26T19:39:13 <gmaxwell> phantomcircuit: for block acceptance speed it may make sense to seperately cache success and failure.
3402015-10-26T19:39:16 <phantomcircuit> 50000 entries
3412015-10-26T19:40:48 <phantomcircuit> gmaxwell, yes map<prevblockhash, set<txid> >
3422015-10-26T19:41:01 <phantomcircuit> should work nicely under non adversarial conditions
3432015-10-26T19:41:09 <sipa> that means that an entry "3000 ago" (which is around 1000 tx) only has a 96% hitrate
3442015-10-26T19:41:16 <sipa> eh, 94%
3452015-10-26T19:41:17 <gmaxwell> phantomcircuit: what, no.
3462015-10-26T19:41:26 <sipa> it uses random replacement
3472015-10-26T19:41:46 <phantomcircuit> sipa, yup 10-100x makes a big difference
3482015-10-26T19:41:52 <gmaxwell> phantomcircuit: transactions can spend unconfirmed outputs my friend.
3492015-10-26T19:42:14 <phantomcircuit> gmaxwell, yeah you still have to check that the dependencies
3502015-10-26T19:43:00 <phantomcircuit> that *does* make the mempool view consensus critical though
3512015-10-26T19:43:01 <gmaxwell> phantomcircuit: if you only mean to check that the ECDSA passes, than <txid> is enough.
3522015-10-26T19:43:18 <sipa> the sigcache is per signature, not per transaction
3532015-10-26T19:43:34 <sipa> per transaction is pretty complicated, as you need to take active softforks into account etc
3542015-10-26T19:44:23 <phantomcircuit> sipa, hadn't considered that
3552015-10-26T19:44:47 <sipa> there have been patching before that cached full tx validation
3562015-10-26T19:44:58 <sipa> but actually just script validation caching is cheaper
3572015-10-26T19:45:02 <sipa> s/cheaper/easier
3582015-10-26T19:45:54 <phantomcircuit> hmm
3592015-10-26T19:46:09 <sipa> i can implement that easily
3602015-10-26T19:46:13 <phantomcircuit> have we considered that upload limiting disconnecting peers will have an impact on the network topology?
3612015-10-26T19:47:02 <gmaxwell> phantomcircuit: it only disconnects peers that fetch historic blocks currently; largely avoiding the concern.
3622015-10-26T19:47:48 <phantomcircuit> that seems safe
3632015-10-26T19:48:23 <gmaxwell> well safer at least. it doesn't completely escape topology impact.
3642015-10-26T19:48:41 <gmaxwell> Because a new node will, of course, not remain connected to anything over its limit.
3652015-10-26T19:49:01 <gmaxwell> so even once its synced up it will end up with a different position in the topology than it would otherwise.
3662015-10-26T19:49:34 <phantomcircuit> export MALLOC_ARENA_MAX=1
3672015-10-26T19:49:39 <phantomcircuit> that completely fixes the problem
3682015-10-26T19:49:42 <phantomcircuit> neat
3692015-10-26T19:50:08 <sipa> IBD's stall detection already causes significant movement in the topology
3702015-10-26T19:50:12 <sipa> phantomcircuit: ??
3712015-10-26T19:50:19 <sipa> to reduce fragmentation?
3722015-10-26T19:50:23 <phantomcircuit> gmaxwell, right which means new nodes will tend to end up connected to nodes that have no upload limit
3732015-10-26T19:50:38 <phantomcircuit> sipa, yeah, calling getblocktemplate in a loop and memory usage <1GB
3742015-10-26T19:50:41 <gmaxwell> phantomcircuit: at least during their first runtime, after restart their topo will be more uniform.
3752015-10-26T19:51:04 <gmaxwell> phantomcircuit: we really need outbound rotation to flatten topo hotspots.
3762015-10-26T19:51:36 <phantomcircuit> gmaxwell, for that we need "outbound slots" basically
3772015-10-26T19:52:11 <phantomcircuit> the easiest way to do that is to just assign the second half of the outbound connections (based on vNodes position) as rotating
3782015-10-26T19:52:33 <phantomcircuit> definitely dont want to rotate all the outbounds :)
3792015-10-26T19:52:48 <gmaxwell> right, it's a good way to wander into a sybil attack.
3802015-10-26T19:53:46 <CodeShark> are you guys submitting proposals for hong kong?
3812015-10-26T19:53:55 <CodeShark> sorry if off-topic
3822015-10-26T19:54:53 <phantomcircuit> hmm this is actually kind of annoying to test
3832015-10-26T19:55:03 <phantomcircuit> have to get the mempool full again
3842015-10-26T19:55:21 <phantomcircuit> sending the "mempool" command to peers when they connect results in comical performance issues
3852015-10-26T19:55:38 <sipa> phantomcircuit: i run with that for benchmarking mempool behaviour
3862015-10-26T19:55:57 *** molly has quit IRC
3872015-10-26T19:56:14 <phantomcircuit> sipa, have you noticed that you just sit there spinning at 100% cpu processing something (i haven't bothered to figur eout what yet)
3882015-10-26T19:56:48 <sipa> phantomcircuit: turn on debug=mempool and debug=mempoolrej
3892015-10-26T19:57:15 <phantomcircuit> i run this node with debug=
3902015-10-26T20:04:54 <GitHub138> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7939164d8985...2b625510d374
3912015-10-26T20:04:55 <GitHub138> bitcoin/master 7bbc7c3 Suhas Daftuar: Add option for microsecond precision in debug.log
3922015-10-26T20:04:55 <GitHub138> bitcoin/master 2b62551 Wladimir J. van der Laan: Merge pull request #6881...
3932015-10-26T20:05:03 <GitHub108> [bitcoin] laanwj closed pull request #6881: Debug: Add option for microsecond precision in debug.log (master...add-microsecond-timestamps) https://github.com/bitcoin/bitcoin/pull/6881
3942015-10-26T20:08:14 <phantomcircuit> BlueMatt, plz2rebase seed
3952015-10-26T20:14:08 <phantomcircuit> sipa, non-scientific gdb interrupt+ bt says it's spinning on ECDSA_verify
3962015-10-26T20:14:14 <phantomcircuit> which i guess is to be expected :)
3972015-10-26T20:16:28 *** Thireus has quit IRC
3982015-10-26T20:16:39 <wumpus> is it catching up or in initial sync? if so, that's expected
3992015-10-26T20:16:51 <wumpus> if in steady state it's not normal
4002015-10-26T20:17:22 <sipa> wumpus: this is with sending out a BIP35 "mempool" command, so at startup it receives zillions of transactions
4012015-10-26T20:18:00 <wumpus> ok...
4022015-10-26T20:18:29 <wumpus> yes, then it makes sense too
4032015-10-26T20:18:38 <sipa> so i do expect you'd be accepting mempool txn at the speed you can handle
4042015-10-26T20:26:33 <phantomcircuit> wumpus, http://0bin.net/paste/8349G1adfXBgLthd#l6rCtj99ayGFnNdx6-C+bbP1fYFF0EJIvg4qvV4JnD9
4052015-10-26T20:32:28 <wumpus> you do have commit 5ce43da03dff3ee655949cd53d952dace555e268?
4062015-10-26T20:33:39 <phantomcircuit> wumpus, yes
4072015-10-26T20:33:44 <wumpus> if so, this shouldn't happen - unless gdb happens to reenable the signal
4082015-10-26T20:34:42 <phantomcircuit> it shouldn't
4092015-10-26T20:34:46 <wumpus> yeah it does, it reports all signals by default. Try: "handle SIGPIPE nostop noprint pass"
4102015-10-26T20:34:49 <phantomcircuit> but who knows... gmaxwell do you know :)
4112015-10-26T20:35:12 <phantomcircuit> wumpus, sigh
4122015-10-26T20:35:17 <phantomcircuit> ok yes that's good
4132015-10-26T20:36:14 <phantomcircuit> bleh why does it do that?
4142015-10-26T20:57:29 <phantomcircuit> wumpus, can verify that MALLOC_ARENA_MAX=1 works btw
4152015-10-26T20:57:52 <sipa> phantomcircuit: what about 2 instead of 1?
4162015-10-26T20:58:19 <phantomcircuit> sipa, not sure it takes ages to load enough transactions into the mempool for the result to be valid
4172015-10-26T20:58:21 <wumpus> phantomcircuit: awesome
4182015-10-26T20:59:20 <sipa> too bad we can't set MALLOC_ARENA_MAX from the program itself, i suppose?
4192015-10-26T20:59:53 <phantomcircuit> sipa, well we can but...
4202015-10-26T21:00:00 <sipa> how so?
4212015-10-26T21:00:18 <wumpus> I think you can
4222015-10-26T21:00:32 <sipa> we can change the env variable, but i expect that the arenas are created at library load time
4232015-10-26T21:00:36 <sipa> before we can change it
4242015-10-26T21:00:54 <phantomcircuit> sipa, the easiest would be to set the env and execve
4252015-10-26T21:01:03 <phantomcircuit> ie h4x
4262015-10-26T21:01:04 <wumpus> let me see glibc source...
4272015-10-26T21:03:11 <wumpus> it indeed makes no sense to set the env var, as it parses it at startup as expected, however you should be able to do mallopt(M_ARENA_MAX, 1)
4282015-10-26T21:05:50 <wumpus> /* mallopt options that actually do something */ in /usr/include/malloc.h :-)
4292015-10-26T21:06:39 <sipa> man mallopt does not list M_ARENA_MAX here
4302015-10-26T21:07:16 <wumpus> it's an undocumented option
4312015-10-26T21:08:59 <wumpus> probably should call it before any call to malloc to be effective
4322015-10-26T21:09:07 <wumpus> or at least before calling it in a thread
4332015-10-26T21:09:20 <sipa> the execve approach works...
4342015-10-26T21:09:49 <sipa> it's ugly because it needs to know the name of the binary you just started
4352015-10-26T21:09:51 <wumpus> that's really ugly, and makes debugging harder
4362015-10-26T21:10:18 <sipa> agree
4372015-10-26T21:10:22 <phantomcircuit> ha it's not even in the libc manual
4382015-10-26T21:10:33 <wumpus> and that (though you should always have that in argv[0] on unix)
4392015-10-26T21:10:56 <phantomcircuit> it looks like we could set M_MMAP_THRESHOLD which would be slower but seems to guarantee memory is actually free'd
4402015-10-26T21:11:09 *** ParadoxSpiral has quit IRC
4412015-10-26T21:12:28 <wumpus> but gdb will show something like "process 28686 is executing new program: /bin/dash" when a program execve's, and it loses all its state
4422015-10-26T21:13:31 <phantomcircuit> wumpus, yeah i know thus "h4x"
4432015-10-26T21:13:40 <phantomcircuit> it would probably also break anybody using start-stop-daemon
4442015-10-26T21:13:52 <phantomcircuit> gtg
4452015-10-26T21:14:20 <wumpus> but the mallopt should work too (can't check it right now though)
4462015-10-26T21:14:29 <wumpus> ok later
4472015-10-26T21:16:18 *** randy-waterhouse has joined #bitcoin-core-dev
4482015-10-26T21:28:44 *** treehug88 has quit IRC
4492015-10-26T21:48:13 *** deepcore has joined #bitcoin-core-dev
4502015-10-26T22:01:54 *** molly has joined #bitcoin-core-dev
4512015-10-26T22:12:55 *** MarcoFalke has quit IRC
4522015-10-26T22:20:48 *** d_t has quit IRC
4532015-10-26T22:21:33 *** d_t has joined #bitcoin-core-dev
4542015-10-26T22:31:08 *** belcher has joined #bitcoin-core-dev
4552015-10-26T23:06:07 <phantomcircuit> 41k transaction in the mempool and counting
4562015-10-26T23:07:06 <sipa> yeah, 50k sigcache entries won't be good in that case...
4572015-10-26T23:07:53 <phantomcircuit> yeah i changed mine to maxsigcachesize=1000000
4582015-10-26T23:13:15 <gmaxwell> sipa: sigcache is currently uint256,vector<char>,cpubkey.... bloat bloat bloat.
4592015-10-26T23:14:02 <gmaxwell> Could be compressed with H(the above).
4602015-10-26T23:14:31 <gmaxwell> H() need not even be very collission resistant if there is a per node secret nonce.
4612015-10-26T23:21:44 <phantomcircuit> gmaxwell, yeah that's what i was thinking
4622015-10-26T23:21:58 <sipa> it's around 220 bytes per entry now
4632015-10-26T23:23:15 <phantomcircuit> sipa, depends on the size of the pubkey script right?
4642015-10-26T23:23:28 <phantomcircuit> it would be nice if it didn't
4652015-10-26T23:23:59 <phantomcircuit> upto 44.5k now
4662015-10-26T23:24:10 <sipa> phantomcircuit: it's a pubkey; not a script
4672015-10-26T23:24:15 <sipa> pubkeys are 65 bytes
4682015-10-26T23:24:17 <sipa> in memory
4692015-10-26T23:24:20 <sipa> + alignment
4702015-10-26T23:24:27 <phantomcircuit> oh
4712015-10-26T23:26:03 <gmaxwell> sipa: plus whatever overhead there is from the std::set.
4722015-10-26T23:26:31 <sipa> gmaxwell: around 40 bytes, i included that
4732015-10-26T23:38:49 *** randy-waterhouse has quit IRC
4742015-10-26T23:39:10 <gmaxwell> sipa: would it be really ugly if accepting the block carried a flag to the checker that told it to remove the entry? with that alone the hitrate would no longer be low absent attacks because the cache would not be full all the time.
4752015-10-26T23:52:15 <morcos> sipa: phantomcircuit: i don't think we need to worry about M_ARENA_MAX. we can drastically reduce the size of the problem. first we rewrite CreateNewBlock to only pull into the cache 1 blocks worth of txs. second we make it run in only 1 thread.
4762015-10-26T23:53:17 <morcos> but yeah that maxsigcachesize will bite you. i couldn't figure out why my new code was slower and it was that its sigcache was getting blown away