12017-04-20T00:01:14  *** abpa has quit IRC
  22017-04-20T00:06:40  *** owowo has quit IRC
  32017-04-20T00:16:05  *** Victorsueca has quit IRC
  42017-04-20T00:17:25  *** laurentmt has quit IRC
  52017-04-20T00:22:33  *** Victorsueca has joined #bitcoin-core-dev
  62017-04-20T00:30:58  *** Ylbam has quit IRC
  72017-04-20T00:33:26  *** goksinen has joined #bitcoin-core-dev
  82017-04-20T00:53:37  *** belcher has quit IRC
  92017-04-20T00:54:07  *** owowo has joined #bitcoin-core-dev
 102017-04-20T01:00:38  *** owowo has quit IRC
 112017-04-20T01:05:39  *** owowo has joined #bitcoin-core-dev
 122017-04-20T01:07:04  *** CubicEarth has joined #bitcoin-core-dev
 132017-04-20T01:11:54  *** justanotheruser has joined #bitcoin-core-dev
 142017-04-20T01:12:15  *** jouke has quit IRC
 152017-04-20T01:13:59  *** jouke has joined #bitcoin-core-dev
 162017-04-20T01:13:59  *** jouke has quit IRC
 172017-04-20T01:13:59  *** jouke has joined #bitcoin-core-dev
 182017-04-20T01:21:04  *** NewLiberty has quit IRC
 192017-04-20T01:33:43  *** PaulCape_ has quit IRC
 202017-04-20T01:37:22  *** tw2006 has joined #bitcoin-core-dev
 212017-04-20T01:42:14  *** tw2006 has quit IRC
 222017-04-20T02:13:17  <luke-jr> Does this look like a malicious peer? https://drive.google.com/file/d/0ByBPjWWp3nRqTHd2T2JqeVlGLUk/view
 232017-04-20T02:17:58  *** wasi has quit IRC
 242017-04-20T02:18:22  *** wasi has joined #bitcoin-core-dev
 252017-04-20T02:23:24  *** luxor has joined #bitcoin-core-dev
 262017-04-20T02:37:56  *** luxor has quit IRC
 272017-04-20T02:45:42  *** wangchun has quit IRC
 282017-04-20T02:45:56  *** d_t has quit IRC
 292017-04-20T02:46:08  *** Dyaheon has quit IRC
 302017-04-20T02:46:41  *** jtimon has quit IRC
 312017-04-20T02:46:46  *** Giszmo has quit IRC
 322017-04-20T02:48:18  *** wangchun has joined #bitcoin-core-dev
 332017-04-20T02:48:43  *** Dyaheon has joined #bitcoin-core-dev
 342017-04-20T03:26:17  *** tw2006 has joined #bitcoin-core-dev
 352017-04-20T03:30:25  *** justan0theruser has joined #bitcoin-core-dev
 362017-04-20T03:30:44  *** tw2006 has quit IRC
 372017-04-20T03:33:26  *** justanotheruser has quit IRC
 382017-04-20T03:58:10  <bitcoin-git> [bitcoin] NicolasDorier closed pull request #10184: [Wallet] Worst case performance improvement on KeyPool filtering (master...hdinternalperf) https://github.com/bitcoin/bitcoin/pull/10184
 392017-04-20T04:26:55  *** PaulCapestany has joined #bitcoin-core-dev
 402017-04-20T04:29:05  *** dodomojo has joined #bitcoin-core-dev
 412017-04-20T04:31:10  *** talmai has joined #bitcoin-core-dev
 422017-04-20T04:31:57  *** goksinen has quit IRC
 432017-04-20T04:35:32  *** dodomojo has quit IRC
 442017-04-20T04:35:48  *** goksinen has joined #bitcoin-core-dev
 452017-04-20T04:40:30  *** goksinen has quit IRC
 462017-04-20T04:48:52  <gmaxwell> morcos: that writeup is pretty much just what I hoped. I thougth there was also an element of using the current mempool queue too?
 472017-04-20T05:07:06  *** d_t has joined #bitcoin-core-dev
 482017-04-20T05:15:18  *** tw2006 has joined #bitcoin-core-dev
 492017-04-20T05:19:38  *** tw2006 has quit IRC
 502017-04-20T05:26:03  *** talmai has quit IRC
 512017-04-20T05:43:05  *** annanay25 has quit IRC
 522017-04-20T05:44:11  *** annanay25 has joined #bitcoin-core-dev
 532017-04-20T05:48:42  *** RubenSomsen has joined #bitcoin-core-dev
 542017-04-20T05:50:51  *** luke-jr has quit IRC
 552017-04-20T05:50:59  *** luke-jr has joined #bitcoin-core-dev
 562017-04-20T06:02:44  *** jnewshoes has quit IRC
 572017-04-20T06:03:23  *** goksinen has joined #bitcoin-core-dev
 582017-04-20T06:06:05  <wumpus> luke-jr: I don't think we usually backport help texts, butit's fine w/ fe
 592017-04-20T06:06:56  <wumpus> indeed, I've enabled the auto-cancel feature in travis where superceded jobs are automatically cancelled
 602017-04-20T06:07:26  <wumpus> if this doesn't work properly I could disable it again, but in general speeding up tests by avoiding unncessary work is a good thing
 612017-04-20T06:09:21  <wumpus> however travis admits it's still in beta stage
 622017-04-20T06:15:04  *** goksinen has quit IRC
 632017-04-20T06:19:17  <bitcoin-git> [bitcoin] laanwj pushed 1 new commit to 0.14: https://github.com/bitcoin/bitcoin/commit/51c787dfb4963d2a59dc8944f45e065be0a06613
 642017-04-20T06:19:17  <bitcoin-git> bitcoin/0.14 51c787d Warren Togami: Clarify importprivkey help text with example of blank label without rescan...
 652017-04-20T06:19:33  *** jnewshoes has joined #bitcoin-core-dev
 662017-04-20T06:26:55  <jonasschnelli> [08:26:20]  <luke-jr>	[21:56:40] Travis jobs are randomly cancelling?
 672017-04-20T06:27:13  <jonasschnelli> I sometime manually cancel (if I force push a couple of times)
 682017-04-20T06:27:26  <jonasschnelli> no need to build obsolete branches
 692017-04-20T06:29:36  *** harrymm has joined #bitcoin-core-dev
 702017-04-20T06:31:20  *** dermoth has joined #bitcoin-core-dev
 712017-04-20T06:33:39  *** Ylbam has joined #bitcoin-core-dev
 722017-04-20T06:46:08  <NicolasDorier> sipa: Have you considered https://github.com/bitcoin/bitcoin/pull/10148#issuecomment-295599772 ? I think it would make the whole thing a lot easier, with less risk.
 732017-04-20T06:54:55  *** BashCo has quit IRC
 742017-04-20T06:55:32  *** Dyaheon has quit IRC
 752017-04-20T06:56:46  <NicolasDorier> edited it
 762017-04-20T06:57:03  *** Dyaheon has joined #bitcoin-core-dev
 772017-04-20T06:58:21  *** goksinen has joined #bitcoin-core-dev
 782017-04-20T07:03:39  *** goksinen has quit IRC
 792017-04-20T07:04:12  *** tw2006 has joined #bitcoin-core-dev
 802017-04-20T07:08:54  *** tw2006 has quit IRC
 812017-04-20T07:15:47  *** BashCo has joined #bitcoin-core-dev
 822017-04-20T07:17:23  *** d_t has quit IRC
 832017-04-20T07:20:10  *** BashCo has quit IRC
 842017-04-20T07:22:13  *** BashCo has joined #bitcoin-core-dev
 852017-04-20T07:30:40  <sipa> NicolasDorier: we already have a maximum size on the cache
 862017-04-20T07:30:49  <sipa> it's what -dbcache controls
 872017-04-20T07:30:56  <NicolasDorier> I know
 882017-04-20T07:31:09  <sipa> then i don't understand your suggestion
 892017-04-20T07:32:38  <NicolasDorier> sipa: basically following the same thing as done here
 902017-04-20T07:32:39  <NicolasDorier> https://github.com/bitcoin/bitcoin/blob/2584925077f9658b3953ad931b74779006e59807/src/validation.cpp#L1982
 912017-04-20T07:32:59  <NicolasDorier> with a fCacheCriticalBatchSize
 922017-04-20T07:33:11  <NicolasDorier> the problem that I found out by trying that
 932017-04-20T07:33:23  <NicolasDorier> is that I was not expecting a flush to wipeout the cache clean
 942017-04-20T07:33:48  <NicolasDorier> I thought only dirty entries in the cache would be commited into a batch
 952017-04-20T07:33:48  <sipa> i think you misunderstand what this does
 962017-04-20T07:34:14  <sipa> it's not about limiting the number of dirty entries
 972017-04-20T07:34:22  <NicolasDorier> This flush the CoinViewCache when certain condition arise
 982017-04-20T07:34:25  <NicolasDorier> right ?
 992017-04-20T07:34:33  <sipa> it's splitting them up over different batches
1002017-04-20T07:34:52  <sipa> we want to limit the size of the batches, as they cause a memory blowup
1012017-04-20T07:34:59  <NicolasDorier> I know
1022017-04-20T07:35:20  <NicolasDorier> My point is that we can prevent a batch to be too big at the CoinViewCache level
1032017-04-20T07:35:27  <sipa> no
1042017-04-20T07:36:12  <NicolasDorier> I was assuming CoinViewCache was not deleting all cached coin during a flush
1052017-04-20T07:36:22  <NicolasDorier> I was wrong on that
1062017-04-20T07:36:24  <sipa> yeah, that's weird
1072017-04-20T07:36:43  <sipa> but we've tried mny things to change that, and they all make things slower
1082017-04-20T07:36:50  <NicolasDorier> if it was not deleting all cached coin during a flush, then we could flush when the CoinViewCache know that a batch size would be too big
1092017-04-20T07:37:00  <NicolasDorier> ha I see
1102017-04-20T07:37:06  <sipa> that's an eventual goal here
1112017-04-20T07:37:35  <sipa> but this is doing something much more basic: not requiring that flushes are consistent with blocks
1122017-04-20T07:38:11  <sipa> let me take a step back
1132017-04-20T07:38:56  <NicolasDorier> I understood your PR and the goal of it
1142017-04-20T07:39:22  <NicolasDorier> I thought there was a simpler way of doing, but I relied on the assumption that a flush did not throw away all the cachedCoins
1152017-04-20T07:39:52  <NicolasDorier> if my assumption was right, fixing the problem you attempt to solve would have been way more easy.
1162017-04-20T07:40:08  <sipa> the reason why we have the cache, and go through all this effort (as opposed to just using leveldb's caching), is that we can do an awesome optimization: if a utxo is created in the cache, and spent before it is ever flushed, we can just delete it from the cache, preventing it from ever being serialized or written to disk
1172017-04-20T07:40:23  <NicolasDorier> yes I am aware of it
1182017-04-20T07:40:39  <sipa> that means that we must maximize the time a utxo is in the cache before being flushed
1192017-04-20T07:41:10  <sipa> what you're suggesting is limiting the amount of dirty entries in the cache
1202017-04-20T07:41:24  <NicolasDorier> right
1212017-04-20T07:41:27  <sipa> that would radically reduce our ability to use that optimization above
1222017-04-20T07:41:36  <NicolasDorier> yes because at every flush
1232017-04-20T07:41:41  <NicolasDorier> you throw away all the cachedCoins
1242017-04-20T07:41:45  <sipa> no
1252017-04-20T07:41:57  <sipa> that has nothing to do with it
1262017-04-20T07:42:02  <sipa> it's more fundamental
1272017-04-20T07:42:37  <NicolasDorier> ah yes let me think
1282017-04-20T07:42:39  <sipa> you're suggesting more frequent flushes (which don't wipe the cache, just mark the written entries as non-dirty, roght?)
1292017-04-20T07:42:45  <NicolasDorier> yes
1302017-04-20T07:43:13  <sipa> more frequent flushes would reduce the time between a utxo is created and the time they hit disk
1312017-04-20T07:43:21  <NicolasDorier> aaaaah
1322017-04-20T07:43:23  <NicolasDorier> got it
1332017-04-20T07:43:31  <sipa> at that point, it's too late
1342017-04-20T07:43:46  <NicolasDorier> yes, I understand now
1352017-04-20T07:43:52  <NicolasDorier> I remove my comment
1362017-04-20T07:44:06  <NicolasDorier> removed
1372017-04-20T07:44:10  <sipa> so my eventual goal is to have continuous flushing that indeed only writes small amounts of entries
1382017-04-20T07:44:26  <sipa> but not by limiting the amount of dirty entries
1392017-04-20T07:44:44  *** JackH has joined #bitcoin-core-dev
1402017-04-20T07:44:49  <NicolasDorier> mmh so how ?
1412017-04-20T07:45:00  <sipa> just have sort of a rolling window... once an entry has been in the cache without being written, flush it
1422017-04-20T07:45:25  <sipa> however, that requires that the disk cache is allowed to be in an inconsistent state
1432017-04-20T07:45:40  <sipa> and needs replay to correct at startup
1442017-04-20T07:46:00  <sipa> which this PR is the first step towards (and a great memory usage improvement on its own)
1452017-04-20T07:46:06  <NicolasDorier> I see
1462017-04-20T07:47:06  <sipa> it's a very usual cache structure, which only works because we mostly create and delete recently created entries
1472017-04-20T07:47:20  <sipa> and only write them once, read them once, delete them once
1482017-04-20T07:47:32  <sipa> most caches are optimized for many reads
1492017-04-20T07:48:36  <NicolasDorier> ok this is clear thanks. So I am fine with 10148, would just like to see a simple python test though
1502017-04-20T07:49:40  <sipa> agree, if you have ideas for what it should do, let me know
1512017-04-20T07:51:11  <NicolasDorier> sipa: A dbbatchsize of 1 bytes, dbcrashratio of 10. Check if the node can eventually sync 200 blocks and have the same utxoset hash than the other node.
1522017-04-20T07:51:25  <NicolasDorier> does not test the forking logic though
1532017-04-20T07:51:32  <sipa> ah, yes
1542017-04-20T07:52:13  <sipa> well we could construct forks as well, i guess
1552017-04-20T07:52:41  <NicolasDorier> yes, I don't think this is very difficult to test. Just the dbbatchsize being in MB make it inconvenient
1562017-04-20T07:52:56  <NicolasDorier> maybe have a magic number just for the tests be enough
1572017-04-20T07:52:58  <sipa> agree
1582017-04-20T07:53:00  *** goksinen has joined #bitcoin-core-dev
1592017-04-20T07:53:09  <sipa> it can be in bytes
1602017-04-20T07:53:22  <sipa> it's a test-only option really
1612017-04-20T07:53:30  <NicolasDorier> yes indeed
1622017-04-20T07:54:05  *** cysm_ has quit IRC
1632017-04-20T07:54:47  *** xinxi has joined #bitcoin-core-dev
1642017-04-20T07:56:21  *** cysm_ has joined #bitcoin-core-dev
1652017-04-20T07:57:54  *** goksinen has quit IRC
1662017-04-20T08:06:53  <xinxi> gmaxwell: Please check my PM.
1672017-04-20T08:20:32  *** RubenSomsen has quit IRC
1682017-04-20T08:21:43  *** timothy has joined #bitcoin-core-dev
1692017-04-20T08:24:24  *** xinxi has quit IRC
1702017-04-20T08:24:51  *** xinxi has joined #bitcoin-core-dev
1712017-04-20T08:29:05  *** xinxi has quit IRC
1722017-04-20T08:35:54  *** jtimon has joined #bitcoin-core-dev
1732017-04-20T08:46:50  *** xinxi has joined #bitcoin-core-dev
1742017-04-20T08:51:27  *** CubicEarth has quit IRC
1752017-04-20T08:53:08  *** tw2006 has joined #bitcoin-core-dev
1762017-04-20T08:57:48  *** tw2006 has quit IRC
1772017-04-20T09:03:47  <wumpus> indeed, no specific reason that abortrescan should not allowed in safe mode, though should anything that triggers rescan be allowed?
1782017-04-20T09:04:00  <wumpus> safe mode = the block chain is in uncertain state
1792017-04-20T09:05:01  <wumpus> I guess it cannot really hurt
1802017-04-20T09:06:05  *** xinxi_ has joined #bitcoin-core-dev
1812017-04-20T09:06:21  *** paveljanik has quit IRC
1822017-04-20T09:07:43  *** paveljanik has joined #bitcoin-core-dev
1832017-04-20T09:07:43  *** paveljanik has joined #bitcoin-core-dev
1842017-04-20T09:08:38  *** xinxi_ has quit IRC
1852017-04-20T09:09:05  *** xinxi_ has joined #bitcoin-core-dev
1862017-04-20T09:09:28  *** xinxi has quit IRC
1872017-04-20T09:13:34  *** xinxi_ has quit IRC
1882017-04-20T09:15:24  *** laurentmt has joined #bitcoin-core-dev
1892017-04-20T09:25:58  *** harrymm has quit IRC
1902017-04-20T09:28:58  <bitcoin-git> [bitcoin] laanwj closed pull request #10232: [0.14] release-notes: Accurately explain getblocktemplate improvements (0.14...0.14_relnotes_mining) https://github.com/bitcoin/bitcoin/pull/10232
1912017-04-20T09:41:54  <gmaxwell> sipa: I think for testing that atomic flushing what we really need is a few bugs to insert in the code, and see that the tests catch them... at least that would give some feel for how adequate the tests are.
1922017-04-20T09:44:07  *** AaronvanW has joined #bitcoin-core-dev
1932017-04-20T09:46:59  <jonasschnelli> gmaxwell: You once mentioned that there are better filters for "block filters" then bloom. I think what mostly matters is the compactness. What filter type would you recommend?
1942017-04-20T09:47:47  <bitcoin-git> [bitcoin] laanwj pushed 9 new commits to master: https://github.com/bitcoin/bitcoin/compare/c91ca0ace9bd...a987def4f629
1952017-04-20T09:47:47  <bitcoin-git> bitcoin/master 23e6e64 John Newbery: Allow disconnectnode() to be called with node id...
1962017-04-20T09:47:48  <bitcoin-git> bitcoin/master d6564a2 John Newbery: [tests] fix nodehandling.py flake8 warnings
1972017-04-20T09:47:48  <bitcoin-git> bitcoin/master e367ad5 John Newbery: [tests] rename nodehandling to disconnectban
1982017-04-20T09:48:06  <bitcoin-git> [bitcoin] laanwj closed pull request #10143: [net] Allow disconnectnode RPC to be called with node id (master...disconnect_node_by_id) https://github.com/bitcoin/bitcoin/pull/10143
1992017-04-20T09:48:26  <sipa> jonasschnelli: the optimal datastructure is something like a bloom filter, but with only 1 hash function, and then instead of storing the bits directly, store the distances between the 1s
2002017-04-20T09:48:53  <jonasschnelli> sipa: okay.. I try to parse your answer.. give me some days. :)
2012017-04-20T09:50:07  <sipa> it's 44% smaller than bloom filters, afaik
2022017-04-20T09:50:19  <sipa> but much much slower to look things up
2032017-04-20T09:50:36  <sipa> as you need to decompress the data
2042017-04-20T09:52:07  <jonasschnelli> So just use a single MurmurHash? I need to understand what you mean with "store the distances between the 1s". Let me think a bit about it.
2052017-04-20T09:53:16  <sipa> if you only have 1 hash function, there will be a few 1s and many 0s in your filter
2062017-04-20T09:54:08  <sipa> which can be compressed using run length encoding
2072017-04-20T09:54:55  <jonasschnelli> Okay. Got that part.
2082017-04-20T09:54:58  <gmaxwell> jonasschnelli: I think this stuff is a waste of time. even with commited filters the users privacy is significantly harmed. Saving 14kb/s hardly seems worth it.
2092017-04-20T09:55:28  <jonasschnelli> gmaxwell: You propose to just scan all blocks?
2102017-04-20T09:55:40  <gmaxwell> sipa: as far as slower, I doubt it, even a pretty huge filter the decompression time would be insignificant compared to transfer time.
2112017-04-20T09:55:42  *** elkalamar has joined #bitcoin-core-dev
2122017-04-20T09:55:54  <gmaxwell> jonasschnelli: scan blocks since the creation of the keys.
2132017-04-20T09:56:15  <sipa> gmaxwell: i mean cpu time
2142017-04-20T09:56:31  <jonasschnelli> gmaxwell: Yes. But catching up two weeks on a cellphone would result in 144*14MB = 2GB of data...
2152017-04-20T09:56:44  <jonasschnelli> Resulting in users stick to the current BF SPV model
2162017-04-20T09:57:33  <gmaxwell> users don't use that in any large number now.
2172017-04-20T09:57:42  <gmaxwell> jonasschnelli: already almost no one uses multibit/android wallet due to poor performance, even though it completely destroys the user's privacy. You cannot compete with server based lookup which is what almost everyone uses.
2182017-04-20T09:58:29  <gmaxwell> also, why would the cellphone be two weeks out of date? should be catching up in the background...
2192017-04-20T09:58:34  <jonasschnelli> Hmm... well, .. SPV BF is pretty fast and I think it's used by a large usergroup... though, I don't have reliable numbers.
2202017-04-20T09:59:01  <sipa> what is BF?
2212017-04-20T09:59:02  <jonasschnelli> gmaxwell: Catching up data in the background puts your app in a different app-group
2222017-04-20T09:59:06  <jonasschnelli> BF = bloom filter
2232017-04-20T09:59:28  <jonasschnelli> gmaxwell: the group that has the significant warning about battery consumption. :)
2242017-04-20T09:59:51  <sipa> what is an app group?
2252017-04-20T10:00:08  <gmaxwell> jonasschnelli: http://luke.dashjr.org/programs/bitcoin/files/charts/software.html  BF using wallets are relatively rare I seldom see more than one connected.
2262017-04-20T10:00:52  <jonasschnelli> sipa: It may be different on Android. But on apples iOS, if you use backgroup activity, you'll end up in a different app-group resulting to different reviews,... I think you need to add warnings to your app description, etc.
2272017-04-20T10:01:05  <gmaxwell> sipa: I fail to see how cpu time alone is interesting for what is being discussed. :)
2282017-04-20T10:01:12  <sipa> gmaxwell: that chart is just reachable nodes
2292017-04-20T10:01:15  <sipa> no?
2302017-04-20T10:01:16  <gmaxwell> sipa: no.
2312017-04-20T10:01:25  <gmaxwell> sipa: the opposite of that, in fact.
2322017-04-20T10:01:35  <sipa> gmaxwell: i was discussing the data structure, not the use case
2332017-04-20T10:02:00  <gmaxwell> ah well, you wouldn't use it for any of the things you normally use a bloomfilter for.
2342017-04-20T10:02:50  <gmaxwell> Though FWIW, the cuckoo-like filter also is close to capacity achieving (at least if you high enough N to allow a high fill rate) and works incrementally.
2352017-04-20T10:29:08  *** jannes has joined #bitcoin-core-dev
2362017-04-20T10:36:53  *** NewLiberty has joined #bitcoin-core-dev
2372017-04-20T10:37:47  *** NewLiberty_ has joined #bitcoin-core-dev
2382017-04-20T10:38:39  *** NewLiberty_ has quit IRC
2392017-04-20T10:38:49  *** Joseph__ has joined #bitcoin-core-dev
2402017-04-20T10:41:18  *** NewLiberty has quit IRC
2412017-04-20T10:42:00  *** tw2006 has joined #bitcoin-core-dev
2422017-04-20T10:46:29  *** tw2006 has quit IRC
2432017-04-20T11:08:03  <jonasschnelli> Hmm... is it entirely stupid to only extend and check the kepool-keyrange (HD restore) if the wallet's bestblock lacks some blocks behind the chaintip? I think a check during init could give an indication if the wallet is in restore mode or not.
2442017-04-20T11:08:28  <jonasschnelli> Alternatively we could add an option hdalwayscheckkeypool=1
2452017-04-20T11:11:48  <wumpus> what would be the rationale behind doing that, then only?
2462017-04-20T11:21:38  <jonasschnelli> wumpus: a) performance for unencrypted wallets, b) [more important] encrypted wallets would require a unlock during the time of the scan
2472017-04-20T11:22:32  <wumpus> "encrypted wallets would require a unlock during the time of the scan" but only if the scan notices that new keys are needed, right?
2482017-04-20T11:22:34  <jonasschnelli> for bitcoind, there is the problem how to want the user if the gap limit is reached, because, at this point, he would need to unlock the wallet in order to continue scanning
2492017-04-20T11:22:44  <jonasschnelli> wumpus: Right.
2502017-04-20T11:23:09  <wumpus> yes I understand there's a notificatino problem there. The GUI could just pop up a dialog, not so much for bitcoind
2512017-04-20T11:23:16  <jonasschnelli> And... to do a precaution scan, you probably should use a gap limit of 100 (configurable).
2522017-04-20T11:24:01  <jonasschnelli> Yes. The GUI way is much simpler.. but even there. Do we always want to extend up to a default gap limit (even in normal operations)?
2532017-04-20T11:24:05  <wumpus> so yes it may make sense to have a seprate 'wallet is reconstructing' mode
2542017-04-20T11:24:29  <jonasschnelli> Because someone may have handed out 100+ addresses and want to make sure he catches all of them in a HD rescan
2552017-04-20T11:24:44  <jonasschnelli> Though, most people probably dont want to auto-extend their keypool over 100+
2562017-04-20T11:24:44  <wumpus> this flag could also be stored in the wallet (like the reindex flag in the utxo db) instead of determining this based on the wallet's bestblock
2572017-04-20T11:25:32  <jonasschnelli> but if you load an initial HD wallet backup, you probably can only identify the possible backup by comparing the bestblock against the chaintip
2582017-04-20T11:25:46  <wumpus> one argument against treating reconstruction specially would be simultaneous use of the wallet on different machines. I know, we don't support this, but then it could be detected.
2592017-04-20T11:26:02  <wumpus> jonasschnelli: that's true
2602017-04-20T11:26:03  <jonasschnelli> Yes. That.
2612017-04-20T11:26:25  <jonasschnelli> Asking the user make sense... (GUI).
2622017-04-20T11:26:36  <wumpus> yes
2632017-04-20T11:26:54  <jonasschnelli> "Wallets is out of sync, do you want to restore a backup?"
2642017-04-20T11:27:04  <jonasschnelli> Then extend the keypool +1000 or ask about the previous usage
2652017-04-20T11:27:31  <wumpus> #10231 gives me a compile error : bitcoin/src/qt/clientmodel.h:85:30: error: implicit instantiation of undefined template
2662017-04-20T11:27:31  <wumpus>  'std::atomic<int>'
2672017-04-20T11:27:33  <gribble> https://github.com/bitcoin/bitcoin/issues/10231 | [Qt] Reduce a significant cs_main lock freeze by jonasschnelli · Pull Request #10231 · bitcoin/bitcoin · GitHub
2682017-04-20T11:27:45  <wumpus> needs a header probably
2692017-04-20T11:27:55  <jonasschnelli> Oh... missed <atomic> include...
2702017-04-20T11:28:09  <jonasschnelli> Thanks. Let me fix this
2712017-04-20T11:29:51  <jonasschnelli> Added a commit. But why did travis not complain?!
2722017-04-20T11:32:41  <wumpus> my report is on ubuntu 16.04 at least, maybe it's different for 14.04
2732017-04-20T11:33:04  <wumpus> most likely cause: different boost versions
2742017-04-20T11:33:22  <wumpus> or qt
2752017-04-20T11:38:39  <wumpus> anyhow with your change it all compiles
2762017-04-20T11:41:35  *** xinxi has joined #bitcoin-core-dev
2772017-04-20T11:41:42  <bitcoin-git> [bitcoin] laanwj pushed 6 new commits to master: https://github.com/bitcoin/bitcoin/compare/a987def4f629...987a6c09562e
2782017-04-20T11:41:43  <bitcoin-git> bitcoin/master 7148f5e Jonas Schnelli: Reduce cs_main locks during modal overlay by adding an atomic cache
2792017-04-20T11:41:44  <bitcoin-git> bitcoin/master cf92bce Jonas Schnelli: Update the remaining blocks left in modaloverlay at init.
2802017-04-20T11:41:44  <bitcoin-git> bitcoin/master 610a917 Jonas Schnelli: Declare headers height/time cache mutable, re-set the methods const
2812017-04-20T11:42:08  <bitcoin-git> [bitcoin] laanwj closed pull request #10231: [Qt] Reduce a significant cs_main lock freeze (master...2017/04/qt_freeze) https://github.com/bitcoin/bitcoin/pull/10231
2822017-04-20T11:44:33  *** laurentmt has quit IRC
2832017-04-20T11:46:11  *** xinxi has quit IRC
2842017-04-20T11:48:01  *** laurentmt has joined #bitcoin-core-dev
2852017-04-20T11:48:05  *** laurentmt has quit IRC
2862017-04-20T12:14:26  *** Guyver2 has joined #bitcoin-core-dev
2872017-04-20T12:23:06  *** xinxi has joined #bitcoin-core-dev
2882017-04-20T12:25:32  *** SopaXorzTaker has joined #bitcoin-core-dev
2892017-04-20T12:30:59  *** tw2006 has joined #bitcoin-core-dev
2902017-04-20T12:31:46  *** mol has joined #bitcoin-core-dev
2912017-04-20T12:34:50  *** moli_ has quit IRC
2922017-04-20T12:36:09  *** tw2006 has quit IRC
2932017-04-20T12:38:41  *** laurentmt has joined #bitcoin-core-dev
2942017-04-20T12:41:59  *** RubenSomsen has joined #bitcoin-core-dev
2952017-04-20T12:59:52  *** n1ce has joined #bitcoin-core-dev
2962017-04-20T13:03:16  *** gielbier has quit IRC
2972017-04-20T13:03:24  *** tw2006 has joined #bitcoin-core-dev
2982017-04-20T13:03:29  *** talmai has joined #bitcoin-core-dev
2992017-04-20T13:14:49  *** gielbier has joined #bitcoin-core-dev
3002017-04-20T13:22:45  *** talmai has quit IRC
3012017-04-20T13:24:21  *** talmai has joined #bitcoin-core-dev
3022017-04-20T13:28:52  *** gielbier has quit IRC
3032017-04-20T13:28:52  *** gielbier has joined #bitcoin-core-dev
3042017-04-20T13:29:50  *** talmai has quit IRC
3052017-04-20T13:29:50  *** RubenSomsen has quit IRC
3062017-04-20T13:34:40  *** Guest12838 has quit IRC
3072017-04-20T13:35:01  *** Guest12838 has joined #bitcoin-core-dev
3082017-04-20T13:36:25  <morcos> gmaxwell: fee estimation currently does not use mempool queue (nor in the improvements for 0.15)  it's an idea that i've been contemplating since the beginning, but i never settled on a design that i thought met all the criteria
3092017-04-20T13:36:55  <morcos> balancing performance, usefulness, and security is hard.
3102017-04-20T13:41:25  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #10238: Change setKeyPool to hold flexible entries (master...2017/04/keypool_fix_a) https://github.com/bitcoin/bitcoin/pull/10238
3112017-04-20T13:45:29  *** JackH has quit IRC
3122017-04-20T13:47:10  <bitcoin-git> [bitcoin] sipa opened pull request #10239: Make Boost use std::atomic internally (master...boost_std_atomic) https://github.com/bitcoin/bitcoin/pull/10239
3132017-04-20T13:48:49  <BlueMatt> sipa: hmm...why does boost prefer its own impls? just for compat?
3142017-04-20T13:49:17  <sipa> BlueMatt: only bug report i found about it was "gcc's atomics were not so good in the past... it's probably better now... discussion dies"
3152017-04-20T13:49:46  <BlueMatt> lol, yay boost
3162017-04-20T13:49:58  <sipa> these macros were only added in 1.54 though... i don't know what versions of boost we're using everywhere
3172017-04-20T13:50:35  *** d_t has joined #bitcoin-core-dev
3182017-04-20T13:51:07  <sipa> http://boost.2283326.n4.nabble.com/compiling-boost-using-C-11-atomics-td4687878.html
3192017-04-20T13:51:51  <BlueMatt> well i suppose thats good...if you have a new boost you'll probably have a new gcc which has good atomics as well :)
3202017-04-20T13:52:54  <sipa> well we already rely on std::atomics anywhere
3212017-04-20T13:53:44  *** laurentmt has quit IRC
3222017-04-20T13:53:53  <BlueMatt> indeed
3232017-04-20T13:55:05  *** d_t has quit IRC
3242017-04-20T14:04:42  *** felco has quit IRC
3252017-04-20T14:07:35  *** xinxi has quit IRC
3262017-04-20T14:08:19  *** felco has joined #bitcoin-core-dev
3272017-04-20T14:11:21  *** goksinen has joined #bitcoin-core-dev
3282017-04-20T14:16:27  *** goksinen has quit IRC
3292017-04-20T14:19:11  *** vicenteH` is now known as vicenteH
3302017-04-20T14:32:03  *** Giszmo has joined #bitcoin-core-dev
3312017-04-20T14:44:40  *** jtimon has quit IRC
3322017-04-20T15:02:14  *** talmai has joined #bitcoin-core-dev
3332017-04-20T15:05:35  *** goksinen has joined #bitcoin-core-dev
3342017-04-20T15:10:09  *** goksinen has quit IRC
3352017-04-20T15:19:01  <luke-jr> wumpus: FYI I get a different result of update-translations than rc2 has: specifically, bitcoin_af is not created
3362017-04-20T15:20:14  <bitcoin-git> [bitcoin] jnewbery reopened pull request #10198: [tests] Remove is_network_split from functional test framework (master...remove_is_network_split) https://github.com/bitcoin/bitcoin/pull/10198
3372017-04-20T15:22:54  *** harrymm has joined #bitcoin-core-dev
3382017-04-20T15:23:21  *** harrymm has joined #bitcoin-core-dev
3392017-04-20T15:26:31  *** talmai has quit IRC
3402017-04-20T15:29:39  *** talmai has joined #bitcoin-core-dev
3412017-04-20T15:30:40  *** gm2051 has joined #bitcoin-core-dev
3422017-04-20T15:30:53  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #10240: [WIP] Add basic HD wallet restore functionality (master...2017/04/hd_rescan) https://github.com/bitcoin/bitcoin/pull/10240
3432017-04-20T15:34:21  *** mol has quit IRC
3442017-04-20T15:34:46  *** Giszmo has quit IRC
3452017-04-20T15:34:55  *** moli_ has joined #bitcoin-core-dev
3462017-04-20T15:47:58  *** talmai has quit IRC
3472017-04-20T15:52:18  *** abpa has joined #bitcoin-core-dev
3482017-04-20T16:11:26  *** gm2052 has joined #bitcoin-core-dev
3492017-04-20T16:15:14  *** gm2051 has quit IRC
3502017-04-20T16:16:19  *** Giszmo has joined #bitcoin-core-dev
3512017-04-20T16:17:18  *** jtimon has joined #bitcoin-core-dev
3522017-04-20T16:23:19  *** n1ce has quit IRC
3532017-04-20T16:25:42  *** n1ce has joined #bitcoin-core-dev
3542017-04-20T16:28:35  *** talmai has joined #bitcoin-core-dev
3552017-04-20T16:42:17  *** BashCo has quit IRC
3562017-04-20T16:50:09  *** Sosumi has joined #bitcoin-core-dev
3572017-04-20T17:02:39  *** BashCo has joined #bitcoin-core-dev
3582017-04-20T17:12:26  *** talmai has quit IRC
3592017-04-20T17:16:39  *** talmai has joined #bitcoin-core-dev
3602017-04-20T17:25:43  <wumpus> luke-jr: did I add bitcoin_af for rc2?
3612017-04-20T17:26:12  <wumpus> adding languages is something I do manually, when I think there's enough in a .ts file to warrant it
3622017-04-20T17:26:26  *** gm2053 has joined #bitcoin-core-dev
3632017-04-20T17:26:35  <luke-jr> wumpus: I don't know about added, but it's not created by the translation scripts anymore
3642017-04-20T17:26:40  <wumpus> but I think _af was a while ago
3652017-04-20T17:26:49  <wumpus> interesting, maybe it was deleted on transifex
3662017-04-20T17:26:57  <luke-jr> when updating translations, I delete *.ts (except en) first
3672017-04-20T17:27:00  <wumpus> I don't track lanugage deletions, only adding
3682017-04-20T17:27:32  <wumpus> unless someone notified me that it was removed for a good reason (e.g. the fake austrian(?) translation that was there at some point)
3692017-04-20T17:28:04  <wumpus> but I'll check that for next language update, thanks
3702017-04-20T17:29:11  *** talmai has quit IRC
3712017-04-20T17:29:29  <luke-jr> np
3722017-04-20T17:29:38  *** gm2052 has quit IRC
3732017-04-20T17:31:43  *** mol has joined #bitcoin-core-dev
3742017-04-20T17:34:38  *** moli_ has quit IRC
3752017-04-20T17:36:57  *** d_t has joined #bitcoin-core-dev
3762017-04-20T17:44:35  *** tw2006 has quit IRC
3772017-04-20T17:45:10  *** tw2006 has joined #bitcoin-core-dev
3782017-04-20T17:45:42  *** Guyver2 has quit IRC
3792017-04-20T17:48:05  *** goksinen has joined #bitcoin-core-dev
3802017-04-20T17:53:08  *** goksinen has quit IRC
3812017-04-20T17:55:00  *** timothy has quit IRC
3822017-04-20T18:35:18  *** Joseph__ has quit IRC
3832017-04-20T18:45:46  <jonasschnelli> Why do test fail when they are successful:  :/ https://travis-ci.org/bitcoin/bitcoin/jobs/224004681#L5249
3842017-04-20T18:46:01  *** SopaXorzTaker has quit IRC
3852017-04-20T18:46:13  <jonasschnelli> A stderr warning leads always to a test failure
3862017-04-20T18:48:24  <wumpus> stderr output in itself results in test failure?
3872017-04-20T18:48:32  <wumpus> I don't think so, it's based o nthe return code
3882017-04-20T18:48:34  <jonasschnelli> wumpus: Yes. It looks like.
3892017-04-20T18:48:41  <jonasschnelli> They pass locally...
3902017-04-20T18:48:56  <jonasschnelli> and they pass on travis... but test runner markes them as failed
3912017-04-20T18:49:02  <jonasschnelli> wumpus: https://travis-ci.org/bitcoin/bitcoin/jobs/224004681#L5249
3922017-04-20T18:49:47  <jonasschnelli> IMO at least we should flag them as passed if stderr contains only warnings...
3932017-04-20T18:49:53  <jonasschnelli> otherwise we can't test warnings
3942017-04-20T18:50:16  <wumpus> test fail/pass shouldn't be based on stderr output at all
3952017-04-20T18:50:31  <wumpus> if it is, that's kind of weird
3962017-04-20T18:50:32  <luke-jr> jonasschnelli: if an early step fails, travis keeps running the rest and still marks it as failed
3972017-04-20T18:51:18  <jonasschnelli> luke-jr: I think its not that: check the signmessage test: https://travis-ci.org/bitcoin/bitcoin/jobs/224004681#L5243
3982017-04-20T18:51:29  <wumpus> where does it check stderr output?
3992017-04-20T18:51:41  <luke-jr> looks like a test_runner.py thing
4002017-04-20T18:51:41  <jonasschnelli> (INFO): Tests successful... but signmessages.py failed, Duration: 3 s
4012017-04-20T18:52:54  <jonasschnelli> if proc.returncode == TEST_EXIT_PASSED and stderr == "":
4022017-04-20T18:53:03  <jonasschnelli> the later if statement
4032017-04-20T18:53:30  <jonasschnelli> or at least split by newline and pass if all lines start with /Warning/
4042017-04-20T18:53:42  <jonasschnelli> (or a clever regex)
4052017-04-20T18:54:06  <wumpus> stderr == "" should go
4062017-04-20T18:54:20  <wumpus> return code should be what determines whether a test passed
4072017-04-20T18:54:31  <wumpus> anything else is insane
4082017-04-20T18:54:43  <jonasschnelli> I think so. Tests may by successful is there is something in stderr
4092017-04-20T18:55:06  <jonasschnelli> Okay. I'll PR
4102017-04-20T18:58:07  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #10241: Allow tests to pass even when stderr got populated (master...2017/04/test_stderr) https://github.com/bitcoin/bitcoin/pull/10241
4112017-04-20T18:59:07  <gmaxwell> I think the sanitizer stuff is only useful in our current test harnesses because we fail on stderr output.
4122017-04-20T18:59:20  <luke-jr> ah
4132017-04-20T19:00:03  <wumpus> sanitizer stuff?
4142017-04-20T19:00:23  <gmaxwell> TSAN/ASAN/UBSAN.
4152017-04-20T19:00:59  <wumpus> do we use that in travis?
4162017-04-20T19:01:41  <jonasschnelli> Well, we could add a test_runner argument (fail_on_stderr) if someone wants to use that with sanitizer
4172017-04-20T19:02:04  <sdaftuar> meeting time?
4182017-04-20T19:02:06  <wumpus> but ok, at least I understand why the stderr check is there now, it's for private test runs with sanitizer?
4192017-04-20T19:02:07  <jonasschnelli> however.. meeting
4202017-04-20T19:02:12  <wumpus> #startmeeting
4212017-04-20T19:02:12  <lightningbot> Meeting started Thu Apr 20 19:02:12 2017 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
4222017-04-20T19:02:12  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
4232017-04-20T19:02:14  <luke-jr> IIRC one of the sanitisers used to require a special env var to cause an exit
4242017-04-20T19:02:20  <gmaxwell> Not yet, only with some not yet merged PRs are we finally TSAN clean, but many of us run it locally and it has found real bugs.   I'm not protesting, but just bringing up the one thing I remember that interacts with that assumption.
4252017-04-20T19:02:23  <luke-jr> but I can't find that now (some do need an extra build option tho)
4262017-04-20T19:02:34  <gmaxwell> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
4272017-04-20T19:02:40  <instagibbs> present
4282017-04-20T19:02:44  <wumpus> gmaxwell: yes I think it's a good point, not trying to disparage it, but we should document things like this
4292017-04-20T19:02:45  <kanzure> hi.
4302017-04-20T19:02:50  <cfields> hi
4312017-04-20T19:02:56  <jtimon> hola
4322017-04-20T19:03:06  <wumpus> topics?
4332017-04-20T19:03:08  <gmaxwell> wumpus: my thinking on seeing the comments above was "oh oh ... that interacts with something ... what was it? what was it?"
4342017-04-20T19:03:20  <gmaxwell> wumpus: 0.14.x release?
4352017-04-20T19:03:29  <wumpus> #topic 0.14.1 release
4362017-04-20T19:03:34  <wumpus> let's push the button?
4372017-04-20T19:03:37  <luke-jr> k
4382017-04-20T19:04:10  <jonasschnelli> Okay for me... to bad #10231 missed 0.14.1
4392017-04-20T19:04:12  <gribble> https://github.com/bitcoin/bitcoin/issues/10231 | [Qt] Reduce a significant cs_main lock freeze by jonasschnelli · Pull Request #10231 · bitcoin/bitcoin · GitHub
4402017-04-20T19:04:23  <gmaxwell> https://www.youtube.com/watch?v=rLMCjuge6oE "Turn your key SIR"
4412017-04-20T19:04:32  <cfields> hooray!
4422017-04-20T19:04:34  <luke-jr> jonasschnelli: I can put it in Knots 0.14.1
4432017-04-20T19:04:53  <jonasschnelli> luke-jr: Yes. Do that.
4442017-04-20T19:04:53  <wumpus> jonasschnelli: is that even tagged for backport? anyhow, tag it for 0.14.2 I'd say
4452017-04-20T19:05:13  <jonasschnelli> wumpus: Yeah. I tagged (not the project though).. 0.14.2 is good IMO.
4462017-04-20T19:05:42  <wumpus> jonasschnelli: ok!
4472017-04-20T19:06:14  <wumpus> next topic?
4482017-04-20T19:06:57  * jonasschnelli damns cs_main
4492017-04-20T19:07:23  <wumpus> jonasschnelli: if it's any consolation, many projects had a similar issue with a central lock
4502017-04-20T19:07:35  * luke-jr coughs at Python
4512017-04-20T19:07:52  <wumpus> I was thinkinkg about the Big Kernel Lock, but yes, python is guilty too
4522017-04-20T19:07:55  <jonasschnelli> wumpus: Yes. I guess there is much room for optimisation.
4532017-04-20T19:08:36  <gmaxwell> There has been some interesting discussion in github related to the wallets handling of address reuse and dust and what not. anyone interested in that subject might want to check out the discussion on #10233  and PRs linked from there.
4542017-04-20T19:08:38  <gribble> https://github.com/bitcoin/bitcoin/issues/10233 | Wallet: Support not reusing addresses by jet0 · Pull Request #10233 · bitcoin/bitcoin · GitHub
4552017-04-20T19:08:49  <wumpus> #topic blocker PRs for review
4562017-04-20T19:09:20  <gmaxwell> jonasschnelli: on locking we need some better lock profiling. If we have some instrumention that yelled anytime lock contention caused >100ms delays, we'd probably find a number of things to fix.
4572017-04-20T19:09:35  <jonasschnelli> gmaxwell: Yes. That!
4582017-04-20T19:09:47  <gmaxwell> I don't think cs_main is itself really the issue there... just not carefully avoiding it via things like caches.
4592017-04-20T19:09:56  <jonasschnelli> gmaxwell: I was printf profiling yesterday
4602017-04-20T19:09:59  <luke-jr> I looked into disabling address reuse and it looks harder than I'd like :/
4612017-04-20T19:10:09  <morcos> i would like to briefly discuss fee estimation (maybe as separate topic)
4622017-04-20T19:10:29  <gmaxwell> on the blocker PRs-- I'm kinda lost where we are with non-atomic writes.
4632017-04-20T19:10:30  <jonasschnelli> blocker PR: https://github.com/bitcoin/bitcoin/projects/8
4642017-04-20T19:10:35  * BlueMatt #10179
4652017-04-20T19:10:37  <gribble> https://github.com/bitcoin/bitcoin/issues/10179 | Give CValidationInterface Support for calling notifications on the CScheduler Thread by TheBlueMatt · Pull Request #10179 · bitcoin/bitcoin · GitHub
4662017-04-20T19:10:50  <BlueMatt> gm2053: i think its ready for review now?
4672017-04-20T19:10:53  <morcos> gmaxwell: #10148 in its current form without multi head just needs more review i think
4682017-04-20T19:10:56  <gribble> https://github.com/bitcoin/bitcoin/issues/10148 | [WIP] Use non-atomic flushing with block replay by sipa · Pull Request #10148 · bitcoin/bitcoin · GitHub
4692017-04-20T19:11:26  <morcos> and maybe more tests?
4702017-04-20T19:11:35  <jonasschnelli> sipa still has the chacha20 rnd as blocker #9792 ...
4712017-04-20T19:11:37  * BlueMatt utacked this morning
4722017-04-20T19:11:37  <gribble> https://github.com/bitcoin/bitcoin/issues/9792 | FastRandomContext improvements and switch to ChaCha20 by sipa · Pull Request #9792 · bitcoin/bitcoin · GitHub
4732017-04-20T19:11:56  <wumpus> #10179 is already on the list BlueMatt
4742017-04-20T19:11:57  <gribble> https://github.com/bitcoin/bitcoin/issues/10179 | Give CValidationInterface Support for calling notifications on the CScheduler Thread by TheBlueMatt · Pull Request #10179 · bitcoin/bitcoin · GitHub
4752017-04-20T19:12:30  <BlueMatt> wumpus: oh, wasnt sure if it got switched after the last merge, sorry
4762017-04-20T19:12:32  <wumpus> adding 10148
4772017-04-20T19:12:33  <gmaxwell> 9792 isn't hard to review, FWIW, in my expirence.
4782017-04-20T19:12:52  <sdaftuar> gmaxwell: i've become more comfortable conceptually with the non-atomic writes (it did take me a while to come around to it being worth the effort).  i'd like to review and test more.
4792017-04-20T19:13:01  <morcos> ditto
4802017-04-20T19:13:25  <wumpus> 9792 is also already on the list
4812017-04-20T19:13:43  <BlueMatt> #9942 can get merged, I think...
4822017-04-20T19:13:45  <gribble> https://github.com/bitcoin/bitcoin/issues/9942 | Refactor CBlockPolicyEstimator by morcos · Pull Request #9942 · bitcoin/bitcoin · GitHub
4832017-04-20T19:13:55  <jtimon> #8855 isn't hard to review either...
4842017-04-20T19:13:57  <gribble> https://github.com/bitcoin/bitcoin/issues/8855 | Use a proper factory for creating chainparams by jtimon · Pull Request #8855 · bitcoin/bitcoin · GitHub
4852017-04-20T19:14:04  <wumpus> if there's something that an be merged you should tell me, preferably outside the meeting :)
4862017-04-20T19:14:14  <morcos> yeah wumpus i think that is now just wasting review cycles, it has more than enouch ACK's (we have told you a couple times.. :) )
4872017-04-20T19:14:15  <luke-jr> multiwallet is rebased and nits fixed btw
4882017-04-20T19:14:25  <gmaxwell> sdaftuar: will let us effectively double the dbcache size being a first benefit... plus it should allow some really nince improvements later post per-txo. Sorry if it wasn't communicated well, the value was more obvious to pieter and I perhaps because we've been hammering on caching policy changes based on per-txo for a while.
4892017-04-20T19:14:33  <luke-jr> CWalletDB still needs some serious refactoring, but IMO that's something to do outside multiwallet's PR
4902017-04-20T19:15:06  <jonasschnelli> luke-jr: agree
4912017-04-20T19:15:18  <wumpus> morcos: I don't remember
4922017-04-20T19:15:49  <morcos> wumpus: no problem, i just wasnt telling you because i didn't want to tell you too many times..  in any case i think its ready (9942 that is)
4932017-04-20T19:16:03  <bitcoin-git> [bitcoin] luke-jr closed pull request #7289: [WIP] Make arguments reconfigurable at runtime via RPC (master...rpc_setarg) https://github.com/bitcoin/bitcoin/pull/7289
4942017-04-20T19:16:30  <sdaftuar> gmaxwell: yeah, makes sense to me now -- there are a lot of steps of "why don't we do X simpler thing instead" that i know you guys have tried/thought through already, that i needed to think through myself
4952017-04-20T19:17:49  <morcos> fee estimation?
4962017-04-20T19:17:51  <jonasschnelli> ack
4972017-04-20T19:18:04  <bitcoin-git> [bitcoin] laanwj pushed 8 new commits to master: https://github.com/bitcoin/bitcoin/compare/987a6c09562e...14c948987f0b
4982017-04-20T19:18:04  <bitcoin-git> bitcoin/master ae7327b Alex Morcos: Make feeEstimator its own global instance of CBlockPolicyEstimator
4992017-04-20T19:18:05  <bitcoin-git> bitcoin/master f6187d6 Alex Morcos: Make processBlockTx private.
5002017-04-20T19:18:05  <bitcoin-git> bitcoin/master dbb9e36 Alex Morcos: Give CBlockPolicyEstimator it's own lock
5012017-04-20T19:18:15  <morcos> Thanks!
5022017-04-20T19:18:23  <bitcoin-git> [bitcoin] laanwj closed pull request #9942: Refactor CBlockPolicyEstimator (master...moveTxConfirmStats) https://github.com/bitcoin/bitcoin/pull/9942
5032017-04-20T19:18:40  <BlueMatt> morcos: yes, fee estimation
5042017-04-20T19:18:46  <morcos> I wrote this to describe the existing algorithm https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41 , which I'm happy to discuss if anyone has any questions on it.
5052017-04-20T19:18:55  <gmaxwell> morcos: thanks for that fee estimation writeup, I guess I understood it better than I thought I did, I think I thought more of the discussed things were actually implemented.
5062017-04-20T19:18:58  <bitcoin-git> [bitcoin] ryanofsky opened pull request #10242: [qt] Don't call method on null WalletModel object (master...pr/rbfnull) https://github.com/bitcoin/bitcoin/pull/10242
5072017-04-20T19:19:33  <gmaxwell> morcos: I think that writeup is good and should go into the codebase.
5082017-04-20T19:19:36  <morcos> And then I wrote #10199 with a bunch of improvements.  I suppose it makes sense to add another section the gist that provides a high level overview of the improvements?
5092017-04-20T19:19:38  <gribble> https://github.com/bitcoin/bitcoin/issues/10199 | Better fee estimates by morcos · Pull Request #10199 · bitcoin/bitcoin · GitHub
5102017-04-20T19:19:46  <gmaxwell> morcos: that would be good.
5112017-04-20T19:20:32  <gmaxwell> I think the estimatior is a complex enough machine that we should maintain a seperate description of it, if not an actual spec.   Just like we do for many major protocol features.
5122017-04-20T19:20:49  <morcos> But what I would like to do is err on the side of merging 10199 early and then if there are small bugs or fixes, we can fix them in master
5132017-04-20T19:20:51  * jtimon remembers that he also wants to decouple the estimator from the mempool
5142017-04-20T19:20:53  <sipa> oops, forgot about meeting
5152017-04-20T19:21:01  <gmaxwell> morcos: the writeup could use some more details about the reliablity estimates and how it merges bins.
5162017-04-20T19:21:04  <morcos> it takes 2 weeks of continuous up time to even explore all the code paths
5172017-04-20T19:21:11  <gmaxwell> sipa: the meeting did not forget about you.
5182017-04-20T19:21:33  <morcos> jtimon: yes, i have a plan to do that that builds off BlueMatt's CValidationInterface.  The groundwork is laid in 9942 that was just merged
5192017-04-20T19:21:51  <gmaxwell> morcos: are we not saving enough data between restarts that we really do need two weeks continious uptime to hit it all?
5202017-04-20T19:21:57  <morcos> reliability estimates?  reliability OF estimates?
5212017-04-20T19:22:14  <gmaxwell> morcos: I know that if there aren't many samples in a bin it doesn't use the bin.
5222017-04-20T19:22:37  <morcos> gmaxwell: well if you want to know how much fee it'll take to be confirmed in a week, you sure as hell better wait at least a week  (but yes once you've done that once, you may not need to do it again on a restart)
5232017-04-20T19:23:02  <morcos> gmaxwell: some of that stuff is changed in 10199 (for the better, obviously i guess)
5242017-04-20T19:23:10  <luke-jr> if anyone else acks #10242, maybe mention the meeting going on in a P.S. :p
5252017-04-20T19:23:12  <gribble> https://github.com/bitcoin/bitcoin/issues/10242 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
5262017-04-20T19:23:12  <jtimon> morcos: I know, I reveiwed it yesterday and linked to similar PRs of my own, at the time you only wanted to decouple the mempool from the estimator (9942 just did it), but not the estimator from the mempool, happy if you changed your mind and want both like me now
5272017-04-20T19:23:37  <gmaxwell> I guess in general a thing to keep in mind for this sort of description is that we should try to make it detailed enough that if an academic showed up and wrote a paper on it based on only the description (which they will) would their results be likely useful to us or not. :)
5282017-04-20T19:23:57  <morcos> but there are some open questions in 10199 that would be helpful to get feedback on
5292017-04-20T19:24:06  <morcos> such as starting with not being able to upgrade from the old estimates
5302017-04-20T19:24:14  <wumpus> imo the most important about the description is that we understand it
5312017-04-20T19:24:20  <gmaxwell> morcos: we should think about saving more of its state in the future.  I have nodes that don't spend more than a few minutes down per month, but don't often make it to two weeks up.
5322017-04-20T19:25:12  <luke-jr> morcos: how complicated is the upgrading? we only would need it for one version at most IMO
5332017-04-20T19:25:15  <morcos> gmaxwell: i think thats problematic for being able to predict really long time horizons...
5342017-04-20T19:25:44  <wumpus> upgrading isn't much of an issue, if the estimation algorithm changes, feel free to throw away the data from the previous one
5352017-04-20T19:25:55  <wumpus> just make sure it doesn't crash on upgrading/downgrading
5362017-04-20T19:26:23  <gmaxwell> well, to really advance I think what we would probably want is a simulator (perhaps based on historical data) and a metric of success.
5372017-04-20T19:26:38  <wumpus> yes
5382017-04-20T19:26:41  <morcos> luke-jr: the complicated part is deciding what we want to do, implementing it probably isn't that bad...  but for instance the new estimates are smart about whether your estimates file is stale...  but should it just dumbly use your old estimates until it has new estimates...  what if the new estimates for 5 blocks which you do have is lower than the old estimate for 25 (which you dont' have a new estimate for)
5392017-04-20T19:26:47  <morcos> etc.
5402017-04-20T19:26:48  <gmaxwell> I think it's more or less fine to toss out data on algo changes. we could worry about doing better when the algo is stable for a long time.
5412017-04-20T19:26:52  *** Giszmo has quit IRC
5422017-04-20T19:27:13  <wumpus> the difficult part, as with coin selection, is evaluationg algorithms
5432017-04-20T19:27:31  <morcos> the historical data is useless... the question is whether you use the old estimates until your new estimates are warmed up (by calculating them before you throw away the data)
5442017-04-20T19:27:58  <luke-jr> huh, someone's playing malleability games on testnet.
5452017-04-20T19:27:59  *** belcher has joined #bitcoin-core-dev
5462017-04-20T19:28:10  <gmaxwell> Electrum does some things with using static estimates when it doesn't have data to estimate on its own.  I think there are a lot of interesting tradeoffs we could make to hotstart. But I don't think starting speed is at all our biggest concern.
5472017-04-20T19:28:17  <gmaxwell> luke-jr: they have been for months.
5482017-04-20T19:28:47  <gmaxwell> The purely retrospective algorithim is really slow to update to changing network conditions, in particular, it doesn't track the weekly load cycle well.
5492017-04-20T19:28:49  <morcos> gmaxwell: right so that is the question..  my preference would be to merge as is.. and then if we get around to it before 0.15 we add a smarter hotstart
5502017-04-20T19:29:06  <luke-jr> morcos: well, if it's  not already implemented, I'd say it's not important enough to spend time implementing
5512017-04-20T19:29:27  *** arubi_ has joined #bitcoin-core-dev
5522017-04-20T19:29:43  <luke-jr> (upgrading, that is)
5532017-04-20T19:29:56  <morcos> gmaxwell: yes... one of the main problems the new design was meant to address...  still using only a purely retrospective algorithm, so the problem fundamentally remains, but in practice its much more responsive  (b/c it looks at different time horizons simultaneously)
5542017-04-20T19:30:59  <jcorgan> clearly this calls for Deep Fee Estimation
5552017-04-20T19:31:08  <gmaxwell> die
5562017-04-20T19:31:36  <jcorgan> tell me what you *really* think
5572017-04-20T19:31:40  <wumpus> hehe, deep fee estimation
5582017-04-20T19:31:55  <luke-jr> no no, Xtreme Deep Fee Estimation!
5592017-04-20T19:32:07  <morcos> anyway, ok for now i'll update the gist with a high level description of the algorithm
5602017-04-20T19:32:11  <gmaxwell> I have a lovely algorithim for an efficient limited memory 2D exponentially weighed moving average somewhere....
5612017-04-20T19:32:17  <gmaxwell> morcos: great.
5622017-04-20T19:32:32  <sipa> Xthin fees
5632017-04-20T19:32:33  *** arubi has quit IRC
5642017-04-20T19:32:36  <luke-jr> XD
5652017-04-20T19:32:46  <morcos> but my basic point here is that ideally we need weeks/months of testing in master to uncover possible edge cases
5662017-04-20T19:33:14  <morcos> i'm relatively confident that overall this is better, but thats not the same thing as saying it doesn' thave problems that need fixing...
5672017-04-20T19:33:16  <wumpus> yes, it shouldn't be merged last minute
5682017-04-20T19:33:34  <gmaxwell> morcos: well get your description up soon, and I'll review shortly after.  I think fee estimation is self contained enough we could merge something and back it out if we don't like it... but we need to have more than you understanding what we're doing at least. :)
5692017-04-20T19:33:58  <morcos> gmaxwell: yes basically my point..  ok sounds good
5702017-04-20T19:34:08  <gmaxwell> (if for no other reason than we need to understand it better to spot failures with it.)
5712017-04-20T19:34:12  <jonasschnelli> morcos: maybe it was asked already, how fast are the estimations available after startup? Does it work with prune=550?
5722017-04-20T19:34:26  <morcos> prune is irrelevant
5732017-04-20T19:34:50  <morcos> it can give you an estimate for a target of N once it has been caught up for 2*N blocks...
5742017-04-20T19:35:01  <gmaxwell> jonasschnelli: Estimations for depth X need to at least see some multiple of X blocks.
5752017-04-20T19:35:22  <morcos> but then it saves that
5762017-04-20T19:35:24  <gmaxwell> Becuase you have a moving window of analysis, and no data for longer windows.
5772017-04-20T19:35:54  <jonasschnelli> So for a conf target of 2 you need to wait ~40min after startup?
5782017-04-20T19:35:57  <morcos> so if you stop and restart you're starting over again for increasing your max possible target, but you still have access for up to that max possible target
5792017-04-20T19:36:34  <morcos> jonasschnelli: correct, but again, only the first time  (or if you have been down for more than 6 weeks i think)
5802017-04-20T19:36:37  <gmaxwell> jonasschnelli: well not really, because you save the results. so the first time, yes. But that goes back to the hot start question.. and there are lots of ways we could hot start these things, if we really had something that was working well otherwise.
5812017-04-20T19:36:50  <jtimon> jcorgan: after alphago took go away from me I was looking for other problem to solve with https://github.com/jtimon/preann as an excuse to run it again </offtopic spam>
5822017-04-20T19:37:42  <jonasschnelli> Okay. Thanks... I'll test 10199 with the mainnet GUI then a bit (before of after merging)
5832017-04-20T19:37:51  <bitcoin-git> [bitcoin] ryanofsky opened pull request #10244: [qt] Add abstraction layer for accessing node and wallet functionality from gui (master...pr/ipc-local) https://github.com/bitcoin/bitcoin/pull/10244
5842017-04-20T19:37:52  *** Dyaheon has quit IRC
5852017-04-20T19:37:53  <jonasschnelli> *or
5862017-04-20T19:38:44  <morcos> gmaxwell: in the meantime the PR description in 10199 covers the changes pretty close to what i will write up on the gist
5872017-04-20T19:39:27  *** Dyaheon has joined #bitcoin-core-dev
5882017-04-20T19:40:09  <morcos> i guess i need to do a quick rebase now that 9942 is done
5892017-04-20T19:44:25  * luke-jr crickets
5902017-04-20T19:44:32  <wumpus> any other topics?
5912017-04-20T19:44:54  <gmaxwell> I want to talk to luke some about the address reuse thing, but it can be post meeting.
5922017-04-20T19:45:06  <wumpus> time to tag 0.14.1 final and start gitian building
5932017-04-20T19:45:25  * luke-jr spins out a Knots branch :p
5942017-04-20T19:45:54  <wumpus> gmaxwell: well there's time
5952017-04-20T19:45:58  <wumpus> #topic address reuse thing
5962017-04-20T19:46:47  <gmaxwell> so a serious privacy problem which has been actively exploited for a long time is that people make near-dust payments to addresses once they've been spent from, so that you spend from them again in new txn, creating snowballing that links all your transactions togeather.
5972017-04-20T19:46:54  <wumpus>  * [new tag]         v0.14.1 -> v0.14.1
5982017-04-20T19:47:32  <instagibbs> https://www.reddit.com/r/BitcoinBeginners/comments/66az3b/why_do_i_keep_getting_000000001_deposits/ for example
5992017-04-20T19:47:35  <gmaxwell> Latest discussions seem to be driven by a user that runs a gambling site and whom cares about this because his customers get running into issues transactions that link back to him.
6002017-04-20T19:47:42  <wumpus> do we need a 'block transacton' functionality against such transaction abuse?
6012017-04-20T19:47:45  <gmaxwell> but it's a general concern for everyone.
6022017-04-20T19:48:16  <gmaxwell> So there have been discussion about some very heavy handed manual methods, but I think I have a suggestion that could potentially be a default behavior:
6032017-04-20T19:48:28  <gmaxwell> but I'm interested in hearing if other people think I'm crazy.
6042017-04-20T19:48:31  <wumpus> wouldn't be the first time I have an UTXO I just want to ignore
6052017-04-20T19:48:37  <morcos> stop the suspense!
6062017-04-20T19:48:52  <BlueMatt> morcos: ack
6072017-04-20T19:48:57  <gmaxwell> First create a seperate quarenteened balance. Any address or specfic txo could be manually quarenteened or unquarenteed at any itme.
6082017-04-20T19:49:29  <gmaxwell> Then adjust coin selection to always spend all payments to a particular address at once (+/- some filtering with dust that might be needed to prevent dust attacks).
6092017-04-20T19:49:56  <gmaxwell> Then once an address has been spent from, it's automatically added to tue quarenteen list (with any outputs that weren't spent, e.g. whatever failed the dust filtering).
6102017-04-20T19:50:21  <wumpus> I think the quarantaine is a good idea, not so sure about adding things automatically though
6112017-04-20T19:50:26  <gmaxwell> If users want to intentionally reuse an address, I suppose they'd need a way to prevent them from being reblocked.
6122017-04-20T19:50:35  <morcos> i like the general idea
6132017-04-20T19:50:42  <luke-jr> gmaxwell: and quarantined funds are excluded from balance or tx list somehow?
6142017-04-20T19:51:05  <gmaxwell> Well I think the attacks will continue unless we could come up with something that was close to automatic... Could be something that gives a GUI user a choice the first time it happens or if the Q balance becomes non-negligble.
6152017-04-20T19:51:05  <BlueMatt> morcos: I might prefer if we were Quarantine ing things and not quarantaine or quarenteened :p
6162017-04-20T19:51:06  <morcos> but what if you have 10 10btc utxos at the same address and you need to pay someone 1 btc
6172017-04-20T19:51:09  <morcos> you spend them all?
6182017-04-20T19:51:28  <gmaxwell> luke-jr: they'd be shown in the tx list, but skipped for spending, and shown as a seperate balances. Like confirmed vs unconfirmed balance.
6192017-04-20T19:51:59  <BlueMatt> gmaxwell: I'm somewhat unsold as default policy
6202017-04-20T19:52:03  <gmaxwell> morcos: yep. and create a big change. Which I think is fine.  I think a seperate issue is that we should auto-split very large change. But thats 90% independant.
6212017-04-20T19:52:04  <BlueMatt> it seems to be a somewhat-surprising break
6222017-04-20T19:52:18  <jcorgan> and the name 'quarantined' might be a bit heavy handed
6232017-04-20T19:52:27  <gmaxwell> BlueMatt: the current privacy trashing is itself a very surprising break.
6242017-04-20T19:52:32  <BlueMatt> fair
6252017-04-20T19:52:36  <luke-jr> it might be less confusing if only the first receive ever was displayed/accepted, and all subsequent ones got quarantined
6262017-04-20T19:52:48  <gmaxwell> jcorgan: well I came up with that on the spot, on github they're calling it frozen, which I think is super misleading (bank froze my funds!). :P
6272017-04-20T19:52:55  <jcorgan> reserved?
6282017-04-20T19:53:11  <luke-jr> suspicious? :P
6292017-04-20T19:53:13  <wumpus> trash can
6302017-04-20T19:53:16  <Chris_Stewart_5> extraneous?
6312017-04-20T19:53:19  <gmaxwell> luke-jr: first recieved leads to an immediate attack: dust spammer races the payment then you get the dust and not the payment. :)
6322017-04-20T19:53:22  <morcos> gmaxwell: hmm...   i do like the idea of auto-quarantining spent address or dust left over in mostly spent addresses, but not sure i like default spending all the inputs and possibly giving you large change
6332017-04-20T19:53:35  <luke-jr> gmaxwell: first confirmed with a larger value?
6342017-04-20T19:53:47  <gmaxwell> morcos: really I'm surprised at that. That change alone is something I've wanted to do for a while (and was carrying patches for it for a bit)
6352017-04-20T19:53:49  <morcos> quarantine is a good name, but lets not bikeshed that
6362017-04-20T19:54:14  <luke-jr> gmaxwell: maybe auto-quarantine dust too
6372017-04-20T19:54:26  <instagibbs> morcos, assuming spending the dust is worthwhile, what's the concern?
6382017-04-20T19:54:26  <BlueMatt> gmaxwell: I'm not sold on non-end-user wallets here. it seems like it woul dbreak many merchant workflows that use bitcoind
6392017-04-20T19:54:29  <sdaftuar> auto-spending all the funds with a given address makes sense to me as well
6402017-04-20T19:54:42  <wumpus> morcos: let's quarantine the bikeshed
6412017-04-20T19:54:49  <luke-jr> BlueMatt: merchant workflows that reuse addresses are broken anyway
6422017-04-20T19:54:52  <BlueMatt> (eg you receive half a payment, your coin selection spends from that addr, then you receive the other half, and now you dont realize you got paid?)
6432017-04-20T19:54:53  <luke-jr> wumpus: lol
6442017-04-20T19:54:55  <gmaxwell> BlueMatt: why? (thats why I'm asking.) -- obviously it would be configurable.
6452017-04-20T19:55:25  <jtimon> re bikeshedding: the class managing this obviously needs to be called quarantiner
6462017-04-20T19:55:33  <gmaxwell> BlueMatt: how often are merchants doing that?  I mean you can get into advanced things like biasing selection to chose SPK that have least recently recieved funds to avoid that.
6472017-04-20T19:55:56  <BlueMatt> gmaxwell: I assume most merchants at least support multiple txn to complete your payment?
6482017-04-20T19:56:02  <BlueMatt> most guis ive read seem to imply that
6492017-04-20T19:56:36  <morcos> yeah i suppose if its configurable, an option that autospends everythign from any address that gets spent makes sense
6502017-04-20T19:56:40  <gmaxwell> BlueMatt: kinda. but they also require them to be recieved at effectively the same time. I think it's managable.
6512017-04-20T19:56:45  <instagibbs> I'm not sure they accept multiple txn as policy
6522017-04-20T19:56:50  <instagibbs> err automatically
6532017-04-20T19:57:01  <gmaxwell> Just the autospend alone would radically improve privacy, and would almost be enough except for the malicious dust creation.
6542017-04-20T19:57:03  <BlueMatt> gmaxwell: to make it compatible you'd have to never spend outputs newer than X, where X is merchant time frame
6552017-04-20T19:57:32  <gmaxwell> BlueMatt: ya, which would be a trivial 'first try without x' in the current framework.
6562017-04-20T19:57:35  <BlueMatt> i agree in principal, but it sounds like you'd break some folks' workflow in subtle ways. adding the option and defaulting off for non-gui users, maybe?
6572017-04-20T19:58:05  <sipa> i don't see how autospending would break anything?
6582017-04-20T19:58:11  <luke-jr> or just tweak how RPC shows quarantine
6592017-04-20T19:58:13  <gmaxwell> well it could evolve over time, too-- I do think it's not worth our time to do things here that we don't think could be on for a majority of users eventually in some form.
6602017-04-20T19:58:21  <instagibbs> At a minimum you could make near-dust be quarantined
6612017-04-20T19:58:25  <morcos> i would think there could be some threshold for auto-un-quarantining too right?  like if your quarantine address receives 1 BTC?  or maybe not, maybe it just becomes common sense to check that
6622017-04-20T19:58:35  <wumpus> in the first version this is introduced it should be disabled by default in any case, I think, let's present it as a security feature first. Could always be enabled by default later but that should not be the initial goal.
6632017-04-20T19:58:43  <gmaxwell> Because already people who are super aware of privacy can and will already manually do coin selection to achieve ends like this.
6642017-04-20T19:58:50  <BlueMatt> gmaxwell: agreed in principal, but there are also easier fixes we can do initially. eg bias coin selection towards this with fallbacks?
6652017-04-20T19:58:51  <luke-jr> wumpus: true
6662017-04-20T19:58:55  <gmaxwell> wumpus: absolutely.
6672017-04-20T19:59:02  *** d9b4bef9 has quit IRC
6682017-04-20T19:59:08  <wumpus> anything that potentially 'disappears' funds shouldn't be enabled lightly
6692017-04-20T19:59:09  <luke-jr> it's harmless to add if it's disabled by default initially
6702017-04-20T19:59:23  <wumpus> luke-jr: exactly
6712017-04-20T19:59:23  <morcos> ok, so this sounds like general agreement that this is a good idea and has degenerated into arguing about defaults.   all development discussion in a nutshell!
6722017-04-20T19:59:24  <BlueMatt> "disabled by default" can also mean "if something fails, fall back to the current behavior"
6732017-04-20T19:59:31  <gmaxwell> Just in principle I don't think the resource investment is worth if it we don't think that the end goal couldn't be default-ish (e.g. GUI) use.
6742017-04-20T19:59:32  <BlueMatt> morcos: yup
6752017-04-20T19:59:34  <wumpus> I'd enable it personally
6762017-04-20T19:59:54  <wumpus> it's worth the resource investment if we think it's useful to have
6772017-04-20T20:00:07  *** d9b4bef9 has joined #bitcoin-core-dev
6782017-04-20T20:00:12  <sdaftuar> so perhaps a first simple step would be to enable auto-spending of all funds from a given address in the coin selection logic?
6792017-04-20T20:00:19  <gmaxwell> I think users should be okay with 'multiple balances' we already have confirmed vs unconfirmed, and normal bank accounts have multiple balances.
6802017-04-20T20:00:20  <luke-jr> IMO the end goal should be to treat address reuse as something that just doesn't work, and have a quarantine people can dig out lost funds if necessary
6812017-04-20T20:00:22  <BlueMatt> sdaftuar: ack
6822017-04-20T20:00:26  <BlueMatt> DONG
6832017-04-20T20:00:35  <jtimon> I would start with the quarantine thing as only usable manually, which we all seem to like, and then propose automatic things
6842017-04-20T20:00:36  <wumpus> #endmeeting
6852017-04-20T20:00:36  <lightningbot> Meeting ended Thu Apr 20 20:00:36 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
6862017-04-20T20:00:36  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-04-20-19.02.html
6872017-04-20T20:00:36  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-04-20-19.02.txt
6882017-04-20T20:00:36  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-04-20-19.02.log.html
6892017-04-20T20:00:36  <gmaxwell> sdaftuar: yea, that would be a good first step with minimal impact, we might have to add some extra features with it, like automatic change splitting.
6902017-04-20T20:00:42  <morcos> i do think it could be default on..  i just think its a matter of avoiding surprised users who are wondering why money appears to be lost, but that can be solved with informing them
6912017-04-20T20:00:50  <wumpus> luke-jr: yeah, for the long long term I agree on that
6922017-04-20T20:00:52  <BlueMatt> morcos: or fallbacks
6932017-04-20T20:00:53  <luke-jr> jtimon: +1
6942017-04-20T20:00:55  *** Sosumi has quit IRC
6952017-04-20T20:01:07  <BlueMatt> "wait, i hit spend and it says not enough money" can be prevented while still using this
6962017-04-20T20:01:35  <gmaxwell> sdaftuar: I used to have a patch that basically post processes coin selection to add all other inputs that were in the same "address group" (listaddressgroupings) as anything selected.
6972017-04-20T20:02:02  <gmaxwell> sdaftuar: but it was kind of suboptimal, and with the existance of coinjoin it's less important to use a whole group rather than a whole address.
6982017-04-20T20:02:08  <wumpus> no one would send 1 BTC to an address to breach someones privacy
6992017-04-20T20:02:16  <wumpus> it's always small amounts
7002017-04-20T20:02:43  <luke-jr> wumpus: if they did, many people would be happy to give up their privacy XD
7012017-04-20T20:02:52  <gmaxwell> wumpus: well never say never, but "champaign problems"
7022017-04-20T20:02:54  <wumpus> luke-jr: xD
7032017-04-20T20:03:19  <sdaftuar> gmaxwell: one thing just occurred to me, if you receive lots of payments to the same address, and issue lots of payments yourself, then this will tie up lots of your utxo's, which could be operationally annoying?
7042017-04-20T20:03:28  <gmaxwell> unfortunately it can be realistic to send several dollars to do so, no problem, but there is only so much we can do.
7052017-04-20T20:03:35  <sdaftuar> ie you'll be generating lots of big unconfirmed chains, and run out of utxos
7062017-04-20T20:03:44  <sdaftuar> could*
7072017-04-20T20:03:45  <gmaxwell> sdaftuar: thus the comment about change splitting.
7082017-04-20T20:03:51  <sdaftuar> that doesn't help?
7092017-04-20T20:03:56  <sdaftuar> oh, some
7102017-04-20T20:04:00  <gmaxwell> yes it does, they'll go to different addresses.
7112017-04-20T20:04:06  <jcorgan> jtimon: i'll take a look at it
7122017-04-20T20:04:20  <sdaftuar> well i was thinking that you're still spending an unconfirmed output, which will be under the descendant limit for the parent... eh, not sure how it would work out.
7132017-04-20T20:04:20  <morcos> also the threshold as to what is too small to include in the spend and leave quarantined is tricky...
7142017-04-20T20:04:38  <wumpus> in any case I agree on the long run, address reuse should be seen as something suspicious unless the user opted in to it (e.g. a publically published address)
7152017-04-20T20:04:54  <gmaxwell> morcos: well I think there is a simple objective measure: at the current target feerate, what is the break even level?
7162017-04-20T20:05:35  <gmaxwell> monero seems to get along okay with protocol prohibited address reuse, we're maybe too conservative on some of these things. :)
7172017-04-20T20:05:44  <morcos> lets say you have 1mBTC that would cost 0.5mBTC to add as an input at the fee rate you are proposing for this tx...  maybe you prefer to leave that quarantined and send it separately at a lower fee rate  ..   certainly if you have multiple ones liek that that could be combined
7182017-04-20T20:05:51  <jtimon> jcorgan: unfortunately the documentation for the university had to be in spanish and I never bothered translating it https://github.com/jtimon/preann/blob/master/doc/pfc-jorge-timon.org (there's a latex generated from that and can give you a pdf as well) </sorry offtopic again>
7192017-04-20T20:06:07  <wumpus> so isn't such a bad idea to 'forget' old addresses after they've been used too long ago, or auto-quarantaine at least
7202017-04-20T20:06:09  <gmaxwell> morcos: if only we had a fee estimator that could give us a reasonable floor feerate! :)
7212017-04-20T20:06:20  <morcos> we do!
7222017-04-20T20:06:30  <morcos> 2 sat/byte will get you confirmed within 500 blocks right now
7232017-04-20T20:06:31  <wumpus> but shouldn't be enabled by default at this point
7242017-04-20T20:07:04  <morcos> ahh, it went up to 3.3 since i last checked.. :)
7252017-04-20T20:07:27  <wumpus> I'd like my transactions to be confirmed within 100 years at least :)
7262017-04-20T20:08:21  <gmaxwell> morcos: right so you could use a floor feerate. But also I think it would be reasonable for us to have behavior that cleans up the UTXO set some at the users (small) expense, I think most users would support that, especially when it has privacy benefits.
7272017-04-20T20:08:30  <wumpus> otherwise I'd have to restart my node first to prevent 64 bit node ids from overflowing
7282017-04-20T20:08:55  <instagibbs> wumpus, no patience eh?
7292017-04-20T20:09:11  <gmaxwell> There is a whole layer of extra features we could think about what to do with the quarantined funds... but I think that should be future work.
7302017-04-20T20:09:17  <jcorgan> jtimon: eso no es problema
7312017-04-20T20:09:28  <jtimon> hahaha, estupendo!
7322017-04-20T20:10:29  <gmaxwell> e.g. if later we have some kind of coinjoin intergration, they could be preferentially sent into that.
7332017-04-20T20:11:33  <gmaxwell> too many things going on, we've responded to this too slowly. :( oh well, in any case, I think that this will prevent a lot of dust from getting created.
7342017-04-20T20:12:28  <gmaxwell> so it's kind of counter intutive, you might worry that the quarantine would result in UTXO bloat, but I suspect the opposite, at least if we're able to make this default-ish: with the incentive to make the tracing payments gone, they'll stop being created.
7352017-04-20T20:12:49  <wumpus> add an 'empty trash can' that sends the quarantained funds into devnull
7362017-04-20T20:13:25  <morcos> participate in network spam attack using quarantined funds button
7372017-04-20T20:13:31  <gmaxwell> I've mused about the idea about having some shred wallet feature that creates some long timelocked spend of any remaining coins and gives them over to fees... then sends them off somewhere.
7382017-04-20T20:14:06  <wumpus> 'send wallet to wumpus'
7392017-04-20T20:14:16  <sipa> ACK
7402017-04-20T20:14:24  <gmaxwell> because I am somewhat pained by all the utxo bloat created by people who end up with 0.00001 BTC in a wallet, in 100 inputs, and then they just delete the file because its effectively worthless.
7412017-04-20T20:14:51  <gmaxwell> yea, wumpus is fine too. tricky part is timelocking it so that they have some time to reconsider their decision. :P
7422017-04-20T20:15:07  <gmaxwell> personally I wouldn't want it, you'll get clowns using it as a backup service and demanding their funds back. :P
7432017-04-20T20:15:35  <wumpus> gmaxwell: yes, there are certainly some practical issues :p
7442017-04-20T20:15:36  <morcos> that reminds me, we should revisit before 0.15 the dust level the wallet will create
7452017-04-20T20:16:32  <wumpus> sending to fees would be a better idea
7462017-04-20T20:16:43  <wumpus> especially those small amounts...
7472017-04-20T20:16:55  <gmaxwell> morcos: if we had a really good lower bound fee estimat it would be sensible to use that. e.g. don't create any change output where more than half its value would be lost in fees.
7482017-04-20T20:17:29  <morcos> but it depends on what you mean by that
7492017-04-20T20:17:49  <wumpus> morcos: what would you like to revisit it to?
7502017-04-20T20:18:23  <morcos> wumpus: not change the network definition, but make the wallet smarter about not creating (ever) outputs just about the network standard limit
7512017-04-20T20:18:32  <morcos> for instance like #9343 does
7522017-04-20T20:18:34  <gribble> https://github.com/bitcoin/bitcoin/issues/9343 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
7532017-04-20T20:18:52  <morcos> gmaxwell: the problem is historically the lower bound fee estimate is 1 sat/byte
7542017-04-20T20:19:17  <morcos> i think any transaction ever created which paid more than that could have been mined by now, some probably weren't because they were collectively forgotten about
7552017-04-20T20:19:40  <morcos> but the lowest feerate mined on the weekend often drops that low
7562017-04-20T20:19:50  <morcos> people are just in a hurry to be confirmed quickly
7572017-04-20T20:20:10  <wumpus> morcos: tagging #9343 for 0.15
7582017-04-20T20:20:12  <gribble> https://github.com/bitcoin/bitcoin/issues/9343 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
7592017-04-20T20:20:34  <morcos> tag 10199 too please
7602017-04-20T20:20:59  <wumpus> done
7612017-04-20T20:29:21  *** Squidicuz has quit IRC
7622017-04-20T20:32:03  *** Giszmo has joined #bitcoin-core-dev
7632017-04-20T20:45:38  *** tw2006 has quit IRC
7642017-04-20T20:47:13  *** molz_ has joined #bitcoin-core-dev
7652017-04-20T20:49:58  *** mol has quit IRC
7662017-04-20T21:01:23  <sipa> nanotube: can we have the bot not produce output instead of "An error has occurred" ?
7672017-04-20T21:08:08  <BlueMatt> why does it fial so much in the first place?
7682017-04-20T21:24:21  <luke-jr> is 0.14.1 supposed to be building a new Qt?
7692017-04-20T21:24:41  *** mol has joined #bitcoin-core-dev
7702017-04-20T21:24:46  <wumpus> relative to 0.14.1rcX? no
7712017-04-20T21:24:57  <bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/14c948987f0b...86ea3c2ff247
7722017-04-20T21:24:58  <bitcoin-git> bitcoin/master a1fd450 Jorge Timón: Trivial: Remove unneeded includes from .h:...
7732017-04-20T21:24:59  <bitcoin-git> bitcoin/master 1c897fc Jorge Timón: Missing includes
7742017-04-20T21:24:59  <bitcoin-git> bitcoin/master 86ea3c2 Wladimir J. van der Laan: Merge #10181: Include cleanup...
7752017-04-20T21:25:15  <bitcoin-git> [bitcoin] laanwj closed pull request #10181: Include cleanup  (master...2017-04-10-includes) https://github.com/bitcoin/bitcoin/pull/10181
7762017-04-20T21:25:47  <wumpus> doesn't seem to be doing that here, win already finished buildling
7772017-04-20T21:26:14  <luke-jr> wumpus: relative to 0.14.0
7782017-04-20T21:26:19  <luke-jr> I missed the RCs
7792017-04-20T21:27:06  <wumpus> I'm not sure of that
7802017-04-20T21:27:48  <cfields> yes
7812017-04-20T21:27:48  *** molz_ has quit IRC
7822017-04-20T21:27:55  <cfields> due to zlib bump
7832017-04-20T21:28:01  <luke-jr> ah, since zlib is a dep?
7842017-04-20T21:28:10  <cfields> yep
7852017-04-20T21:28:34  * luke-jr goes to figure out food then
7862017-04-20T21:36:19  *** dgenr8 has joined #bitcoin-core-dev
7872017-04-20T21:43:53  <jtimon> regarding https://github.com/bitcoin/bitcoin/pull/10193 I'm still failing at replacing BOOST_REVERSE_FOREACH and I don't understand why, maybe I should reduce the scope of the PR (I moved from working stuff to trying to fully remove boost/foreach.hpp by popular demand)
7882017-04-20T21:44:07  <jtimon> ?
7892017-04-20T21:46:16  <wumpus> zlib is at the bottom of the food chain, dependency-wise, not surprised it triggers rebuild of everything
7902017-04-20T21:46:30  *** tw2006 has joined #bitcoin-core-dev
7912017-04-20T21:46:40  *** tripleslash has quit IRC
7922017-04-20T21:46:51  <wumpus> jtimon: what's the problem with BOOST_REVERSE_FOREACH?
7932017-04-20T21:47:45  *** molz_ has joined #bitcoin-core-dev
7942017-04-20T21:49:11  <jtimon> wumpus: I'm trying to replace it with https://github.com/bitcoin/bitcoin/pull/10193/commits/1e59de57eecc1a61b036d7f89ff6bf918c2c7f67, which I found in the interwebs but although I thought I fully understood, it seems I don't
7952017-04-20T21:49:32  <nanotube> sipa: yes, though it's easier to just fix the problem :) which is that github decided to add fancy middle-dots to its title, which make the existing code barf with unicode errors >_<
7962017-04-20T21:50:03  <luke-jr> lol
7972017-04-20T21:50:07  <luke-jr> nanotube: you're alive!
7982017-04-20T21:51:04  *** mol has quit IRC
7992017-04-20T21:51:32  *** tw2006 has quit IRC
8002017-04-20T21:51:36  <nanotube> luke-jr: o/ :)
8012017-04-20T21:52:38  <jtimon> wumpus: it seems to interfere with prevector templates in prevector_tests.cpp, see https://github.com/bitcoin/bitcoin/pull/10193/commits/cfef34884684e71c6f43ef3e4f2e87590fc87c9e for the trick to make the PR pass travis. probably I should remove that commit already, but I wanted to make sure the commits after removing BOOST_REVERSE_FOREACH weren't breaking something else, plus that commit is kind of the link that maybe answers your
8022017-04-20T21:52:38  <jtimon>  question
8032017-04-20T21:53:47  <jtimon> I would really prefer to just solve the problem but I'm kind of lost
8042017-04-20T21:54:10  <wumpus> jtimon: it's sad that c++11 doesn't provide an as-is replacement
8052017-04-20T21:54:44  <jtimon> wumpus: yep, it seems things get slightly better on c
8062017-04-20T21:54:48  <cfields> wumpus: does reverse_iterate not end up being a dangling reference? I'm not sure what lifetime is expected for the container in a range-based-for
8072017-04-20T21:55:01  <jtimon> c++0.14, I mean...c++14
8082017-04-20T21:55:38  <cfields> (rather, I don't know if the loop extends the lifetime of the container)
8092017-04-20T21:55:53  <cfields> er, that was for jtimon, sorry
8102017-04-20T21:56:19  <wumpus> jtimon: ok, so it's just a matter of waiting a few years (hey at least not 100 years :p)
8112017-04-20T21:56:41  <wumpus> c++17 is pretty nice too, esp std::optional
8122017-04-20T21:57:08  <nanotube> testing gribble title fix #9343
8132017-04-20T21:57:10  <gribble> https://github.com/bitcoin/bitcoin/issues/9343 | Dont create change at dust limit by morcos · Pull Request #9343 · bitcoin/bitcoin · GitHub
8142017-04-20T21:57:17  <nanotube> \o/
8152017-04-20T21:57:25  <wumpus> woohoo
8162017-04-20T21:57:32  <jtimon> no, this can be certainly solved in c++11, but maybe not in a very clean way, perhaps we want our own macro to replace it
8172017-04-20T21:57:35  *** tripleslash has joined #bitcoin-core-dev
8182017-04-20T21:58:40  <wumpus> I'd say replacing it with another macro would not be worth it
8192017-04-20T21:59:39  <wumpus> there are still significant impediments to ditching boost wholesale, and until we have replacements for those, there's no use in rolling our own for the simpler things. Certainly not if they're not clean and simple.
8202017-04-20T21:59:59  <jtimon> or just get rid of BOOST_FOREACH, ¿Q_FOREACH? and PAIRTYPE for now, but not BOOST_REVERSE_FOREACH or #include <boost/foreach.hpp> for now (although the include would only be used for BOOST_REVERSE_FOREACH now)
8212017-04-20T22:00:27  <wumpus> but maybe we can get your reverse_iterator to work
8222017-04-20T22:00:43  *** Giszmo has quit IRC
8232017-04-20T22:00:45  <wumpus> I would be surprised if it's not possible
8242017-04-20T22:02:12  <jtimon> I am already surprised that this is not working, it's working for all the other cases using BOOST_REVERSE_FOREACH, just not compiling prevector_tests for reasons beyond me. it seems the templating is colliding somehow
8252017-04-20T22:03:06  <jtimon> but the option of reducing the scope and leave fully removing boost/foreach.hpp for a later PR is also there, that's why I ask
8262017-04-20T22:03:31  <jtimon> btw, sorry for asking again, but any blockers for  #10189 ?
8272017-04-20T22:03:33  <gribble> https://github.com/bitcoin/bitcoin/issues/10189 | devtools/net: add a verifier for scriptable changes. Use it to make CNode::id private. by theuni · Pull Request #10189 · bitcoin/bitcoin · GitHub
8282017-04-20T22:05:12  *** mol has joined #bitcoin-core-dev
8292017-04-20T22:05:22  <TD-Linux> gmaxwell, you could have a checkbox when generating an address that indicates it's supposed to be public
8302017-04-20T22:05:30  <jtimon> nevermind, just remembered cfields needs to enforce a commit tittle prefix
8312017-04-20T22:05:54  <cfields> jtimon: that's done
8322017-04-20T22:05:57  <TD-Linux> it seems more useful than the "reuse an existing address" checkbox there now
8332017-04-20T22:06:07  <cfields> I suppose I should comment
8342017-04-20T22:06:33  <gmaxwell> TD-Linux: there is a reuse checkbox? I don't recall that.
8352017-04-20T22:07:08  <TD-Linux> gmaxwell, maybe it's gone in the latest version, let me check. regardless I don't understand the use case
8362017-04-20T22:07:19  <luke-jr> gmaxwell: I have a PR open to remove it..
8372017-04-20T22:08:10  *** molz_ has quit IRC
8382017-04-20T22:08:14  <jtimon> cfields: oops, great! thanks
8392017-04-20T22:08:30  <gmaxwell> is that just the thing that brings up the dislog that shows you existing addresses.
8402017-04-20T22:12:09  <jtimon> let me rebase and remove the travis-cheating commit for everyone to see the error without having to build locally, maybe it's obvious to someone else
8412017-04-20T22:15:15  <TD-Linux> https://github.com/bitcoin/bitcoin/pull/3716
8422017-04-20T22:17:32  <gmaxwell> oh well generating a QR code for an older address still seems useful.
8432017-04-20T22:17:42  <gmaxwell> should probably be elsewhere though.
8442017-04-20T22:18:34  <TD-Linux> I'd rather have it accessible via a right click in the list below but I guess that list is only so long
8452017-04-20T22:28:35  *** Giszmo has joined #bitcoin-core-dev
8462017-04-20T22:38:41  *** laurentmt has joined #bitcoin-core-dev
8472017-04-20T22:39:11  *** laurentmt has quit IRC
8482017-04-20T22:41:41  <bitcoin-git> [bitcoin] shigeya opened pull request #10245: Minor fix in build documentation for FreeBSD 11 (master...freebsd-11-build-doc-fix) https://github.com/bitcoin/bitcoin/pull/10245
8492017-04-20T22:44:48  *** d_t has quit IRC
8502017-04-20T22:54:37  *** d_t has joined #bitcoin-core-dev
8512017-04-20T22:58:44  *** Squidicuz has joined #bitcoin-core-dev
8522017-04-20T23:02:42  <bitcoin-git> [bitcoin] practicalswift opened pull request #10246: Silence "warning: "MSG_DONTWAIT" redefined" when compiling under Linux (master...silence-msg_dontwait-warning) https://github.com/bitcoin/bitcoin/pull/10246
8532017-04-20T23:06:15  <jtimon> also I insist if you don't find #8855 interesting to review, maybe you do find #8994 interesting
8542017-04-20T23:06:18  <gribble> https://github.com/bitcoin/bitcoin/issues/8855 | Use a proper factory for creating chainparams by jtimon · Pull Request #8855 · bitcoin/bitcoin · GitHub
8552017-04-20T23:06:19  <gribble> https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
8562017-04-20T23:12:04  *** AaronvanW has quit IRC
8572017-04-20T23:12:39  *** AaronvanW has joined #bitcoin-core-dev
8582017-04-20T23:18:48  *** jouke has quit IRC
8592017-04-20T23:20:40  *** AaronvanW has quit IRC
8602017-04-20T23:21:51  <bitcoin-git> [bitcoin] practicalswift closed pull request #10246: Silence "warning: "MSG_DONTWAIT" redefined" when compiling under Linux (master...silence-msg_dontwait-warning) https://github.com/bitcoin/bitcoin/pull/10246
8612017-04-20T23:22:21  *** jouke has joined #bitcoin-core-dev
8622017-04-20T23:22:21  *** jouke has joined #bitcoin-core-dev
8632017-04-20T23:32:36  *** AaronvanW has joined #bitcoin-core-dev
8642017-04-20T23:33:17  *** Giszmo has quit IRC
8652017-04-20T23:35:25  *** tw2006 has joined #bitcoin-core-dev
8662017-04-20T23:36:27  <luke-jr> y'all slow today; I had to build the new deps and still beat everyone but achow101 :P
8672017-04-20T23:37:11  *** AaronvanW has quit IRC
8682017-04-20T23:39:58  *** tw2006 has quit IRC
8692017-04-20T23:46:27  *** AaronvanW has joined #bitcoin-core-dev
8702017-04-20T23:49:56  *** Giszmo has joined #bitcoin-core-dev
8712017-04-20T23:52:42  *** tripleslash has quit IRC
8722017-04-20T23:54:51  *** tripleslash has joined #bitcoin-core-dev