12017-10-05T00:01:41  *** Alina-malina has joined #bitcoin-core-dev
  22017-10-05T00:12:21  *** Alina-malina has quit IRC
  32017-10-05T00:13:16  *** Alina-malina has joined #bitcoin-core-dev
  42017-10-05T00:20:59  *** Ylbam has quit IRC
  52017-10-05T00:21:33  *** Alina-malina has quit IRC
  62017-10-05T00:23:16  *** Alina-malina has joined #bitcoin-core-dev
  72017-10-05T00:25:25  *** AaronvanW has quit IRC
  82017-10-05T00:32:12  *** arubi has quit IRC
  92017-10-05T00:37:03  *** arubi has joined #bitcoin-core-dev
 102017-10-05T00:43:24  *** Alina-malina has quit IRC
 112017-10-05T00:45:48  *** Alina-malina has joined #bitcoin-core-dev
 122017-10-05T00:53:52  *** Alina-malina has quit IRC
 132017-10-05T00:56:19  *** Alina-malina has joined #bitcoin-core-dev
 142017-10-05T01:02:05  *** Alina-malina has quit IRC
 152017-10-05T01:05:16  *** Alina-malina has joined #bitcoin-core-dev
 162017-10-05T01:07:06  *** jb55 has joined #bitcoin-core-dev
 172017-10-05T01:11:23  *** Alina-malina has quit IRC
 182017-10-05T01:12:43  *** Alina-malina has joined #bitcoin-core-dev
 192017-10-05T01:31:02  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 202017-10-05T01:39:22  *** Murch has quit IRC
 212017-10-05T01:48:01  *** d9b4bef9 has quit IRC
 222017-10-05T01:49:08  *** d9b4bef9 has joined #bitcoin-core-dev
 232017-10-05T02:00:47  *** berndj has quit IRC
 242017-10-05T02:01:34  *** ybit has quit IRC
 252017-10-05T02:02:03  *** berndj has joined #bitcoin-core-dev
 262017-10-05T02:03:35  *** ybit has joined #bitcoin-core-dev
 272017-10-05T02:13:52  *** Alina-malina has quit IRC
 282017-10-05T02:15:51  *** Alina-malina has joined #bitcoin-core-dev
 292017-10-05T02:21:24  *** ula has quit IRC
 302017-10-05T02:22:32  *** Chris_Stewart_5 has quit IRC
 312017-10-05T02:27:46  *** wxxs has quit IRC
 322017-10-05T02:35:43  *** Alina-malina has quit IRC
 332017-10-05T02:38:52  *** Alina-malina has joined #bitcoin-core-dev
 342017-10-05T02:51:48  *** atroxes has quit IRC
 352017-10-05T02:52:32  *** atroxes has joined #bitcoin-core-dev
 362017-10-05T03:05:57  *** [Author] has quit IRC
 372017-10-05T03:24:45  *** [Author] has joined #bitcoin-core-dev
 382017-10-05T03:29:14  *** wraithm has joined #bitcoin-core-dev
 392017-10-05T03:29:20  *** meshcollider has quit IRC
 402017-10-05T03:33:33  *** wraithm has quit IRC
 412017-10-05T03:36:36  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10517: Factor out CCoinsView based AreInputsStandard/IsWitnessStandard (master...2017/06/policy_compile) https://github.com/bitcoin/bitcoin/pull/10517
 422017-10-05T03:37:42  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10238: Change setKeyPool to hold flexible entries (master...2017/04/keypool_fix_a) https://github.com/bitcoin/bitcoin/pull/10238
 432017-10-05T04:05:30  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10251: Add balances cache / GUI: use a signal instead of a poll thread (master...2017/04/gui_rm_locks) https://github.com/bitcoin/bitcoin/pull/10251
 442017-10-05T04:09:48  *** arubi has quit IRC
 452017-10-05T05:29:34  *** Ylbam has joined #bitcoin-core-dev
 462017-10-05T05:41:33  *** jb55 has quit IRC
 472017-10-05T06:03:01  *** d9b4bef9 has quit IRC
 482017-10-05T06:04:07  *** d9b4bef9 has joined #bitcoin-core-dev
 492017-10-05T06:28:19  *** intcat has quit IRC
 502017-10-05T06:38:45  *** Emcy has quit IRC
 512017-10-05T07:06:15  *** protomar has joined #bitcoin-core-dev
 522017-10-05T07:06:18  *** BashCo has quit IRC
 532017-10-05T07:21:24  *** SopaXorzTaker has joined #bitcoin-core-dev
 542017-10-05T07:25:38  *** BashCo has joined #bitcoin-core-dev
 552017-10-05T07:44:19  *** meshcollider has joined #bitcoin-core-dev
 562017-10-05T07:46:09  *** JackH has joined #bitcoin-core-dev
 572017-10-05T07:54:59  *** AaronvanW has joined #bitcoin-core-dev
 582017-10-05T08:01:45  *** DrOlmer has joined #bitcoin-core-dev
 592017-10-05T08:17:18  *** Guyver2 has joined #bitcoin-core-dev
 602017-10-05T08:31:30  *** timothy has joined #bitcoin-core-dev
 612017-10-05T08:39:46  *** AaronvanW has quit IRC
 622017-10-05T08:41:05  *** AaronvanW has joined #bitcoin-core-dev
 632017-10-05T08:45:07  *** Ylbam has quit IRC
 642017-10-05T08:46:07  *** arubi has joined #bitcoin-core-dev
 652017-10-05T08:57:01  *** promag has joined #bitcoin-core-dev
 662017-10-05T09:07:33  *** ghost43 has quit IRC
 672017-10-05T09:08:36  *** promag has quit IRC
 682017-10-05T09:08:53  *** promag has joined #bitcoin-core-dev
 692017-10-05T09:15:15  *** pigeons has quit IRC
 702017-10-05T09:16:08  *** rabidus has quit IRC
 712017-10-05T09:18:40  *** promag has quit IRC
 722017-10-05T09:21:40  *** promag has joined #bitcoin-core-dev
 732017-10-05T09:43:42  *** AaronvanW has quit IRC
 742017-10-05T09:44:27  *** AaronvanW has joined #bitcoin-core-dev
 752017-10-05T09:44:54  *** promag has quit IRC
 762017-10-05T09:46:17  *** promag has joined #bitcoin-core-dev
 772017-10-05T09:48:12  *** AaronvanW has quit IRC
 782017-10-05T09:49:55  *** AaronvanW has joined #bitcoin-core-dev
 792017-10-05T09:51:42  *** AaronvanW has joined #bitcoin-core-dev
 802017-10-05T09:53:00  *** wxxs has joined #bitcoin-core-dev
 812017-10-05T09:54:44  *** promag has quit IRC
 822017-10-05T09:55:32  *** Lesley has joined #bitcoin-core-dev
 832017-10-05T10:21:17  *** promag has joined #bitcoin-core-dev
 842017-10-05T10:22:56  *** promag has quit IRC
 852017-10-05T10:26:11  *** W4RL0RD has joined #bitcoin-core-dev
 862017-10-05T10:27:53  *** Aaronvan_ has joined #bitcoin-core-dev
 872017-10-05T10:31:03  *** AaronvanW has quit IRC
 882017-10-05T11:16:38  *** Aaronvan_ has quit IRC
 892017-10-05T11:17:14  *** AaronvanW has joined #bitcoin-core-dev
 902017-10-05T11:18:36  *** AaronvanW has joined #bitcoin-core-dev
 912017-10-05T11:22:39  <sturles> I have a problem with boost after upgrading to Debian 9.  I did an apt-get --reinstall install libboost-all-dev to be sure I have everything, but it still fails: https://0bin.net/paste/NjYBdILEZBChNmbI#rsDCuSOm7iKfR-UZj7ytJdxoAYMLRdB3szd43qbZmvH
 922017-10-05T11:23:24  *** promag has joined #bitcoin-core-dev
 932017-10-05T11:24:02  <sturles> Any ideas?
 942017-10-05T11:28:33  *** promag has quit IRC
 952017-10-05T11:40:14  *** protomar has quit IRC
 962017-10-05T11:52:33  *** Alina-malina has quit IRC
 972017-10-05T11:53:45  *** Alina-malina has joined #bitcoin-core-dev
 982017-10-05T12:00:00  *** wxxs has quit IRC
 992017-10-05T12:00:01  *** Alina-malina has quit IRC
1002017-10-05T12:01:27  *** wxxs has joined #bitcoin-core-dev
1012017-10-05T12:03:45  *** Alina-malina has joined #bitcoin-core-dev
1022017-10-05T12:08:40  *** rabidus has joined #bitcoin-core-dev
1032017-10-05T12:10:20  *** SopaXorzTaker has quit IRC
1042017-10-05T12:15:40  *** afk11 has quit IRC
1052017-10-05T12:16:01  *** d9b4bef9 has quit IRC
1062017-10-05T12:17:09  *** d9b4bef9 has joined #bitcoin-core-dev
1072017-10-05T12:22:53  *** SopaXorzTaker has joined #bitcoin-core-dev
1082017-10-05T12:34:08  *** meshcollider has quit IRC
1092017-10-05T12:40:38  *** SopaXorzTaker has quit IRC
1102017-10-05T12:42:09  *** SopaXorzTaker has joined #bitcoin-core-dev
1112017-10-05T13:09:15  <bitcoin-git> [bitcoin] MarcoFalke pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/167cef8082e2...e93fff1463ae
1122017-10-05T13:09:16  <bitcoin-git> bitcoin/master 58d91af MeshCollider: Fix race for mapBlockIndex in AppInitMain
1132017-10-05T13:09:16  <bitcoin-git> bitcoin/master 35aeabe MeshCollider: Make fReindex atomic to avoid race
1142017-10-05T13:09:17  <bitcoin-git> bitcoin/master 731065b MeshCollider: Consistent parameter names in txdb.h
1152017-10-05T13:09:49  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #11107: Fix races in AppInitMain and others with lock and atomic bools (master...fix_mapBlockIndex_race) https://github.com/bitcoin/bitcoin/pull/11107
1162017-10-05T13:36:28  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1172017-10-05T13:48:40  *** pigeons has joined #bitcoin-core-dev
1182017-10-05T13:51:03  *** Alina-malina has quit IRC
1192017-10-05T13:52:51  *** Alina-malina has joined #bitcoin-core-dev
1202017-10-05T14:03:26  *** Alina-malina has quit IRC
1212017-10-05T14:05:53  *** Alina-malina has joined #bitcoin-core-dev
1222017-10-05T14:05:53  *** Emcy has joined #bitcoin-core-dev
1232017-10-05T14:30:33  *** Chris_Stewart_5 has quit IRC
1242017-10-05T14:38:56  *** rafalcpp has quit IRC
1252017-10-05T14:39:58  *** rafalcpp has joined #bitcoin-core-dev
1262017-10-05T14:44:12  *** wraithm has joined #bitcoin-core-dev
1272017-10-05T14:49:25  *** arubi has quit IRC
1282017-10-05T14:50:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1292017-10-05T15:00:20  *** deltaT has joined #bitcoin-core-dev
1302017-10-05T15:01:19  <deltaT> simple question i hope, when i created my bitcoin core wallet i didnt write down the private key, how do i find it now?
1312017-10-05T15:02:27  *** Chris_Stewart_5 has quit IRC
1322017-10-05T15:06:43  <jonasschnelli> deltaT: which Bitcoin Core version?
1332017-10-05T15:07:16  <jonasschnelli> deltaT: did you lost you wallet.dat? file.... also, this question should be asked in #bitcoin (this here is the development channel)
1342017-10-05T15:07:26  <deltaT> 0.14.2
1352017-10-05T15:07:27  <BlueMatt> wumpus: #9572 looks relatively merge-able...it is consensus so if you want to ack first that'd also be nice, but its simple and has 4 already
1362017-10-05T15:07:29  <gribble> https://github.com/bitcoin/bitcoin/issues/9572 | Skip witness sighash cache for non-segwit transactions by jl2012 · Pull Request #9572 · bitcoin/bitcoin · GitHub
1372017-10-05T15:08:56  *** deltaT has quit IRC
1382017-10-05T15:11:57  *** jb55 has joined #bitcoin-core-dev
1392017-10-05T15:13:39  <bitcoin-git> [bitcoin] mess110 opened pull request #11455: CTxMemPool::GetMinFee should not return CFeeRate(0) (master...fix_mempool_GetMinFee_bug_returning_below_minRelayTxFee) https://github.com/bitcoin/bitcoin/pull/11455
1402017-10-05T15:16:06  *** rafalcpp_ has joined #bitcoin-core-dev
1412017-10-05T15:17:01  *** rafalcpp has quit IRC
1422017-10-05T15:18:38  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #11456: Replace relevant services logic with a function suite. (master...2017-09-service-flags-cleanups) https://github.com/bitcoin/bitcoin/pull/11456
1432017-10-05T15:20:19  <bitcoin-git> [bitcoin] mess110 closed pull request #11410: [rpc] [tests] mempoolminfee should not drop below minRelayTxFee (master...add_minrelaytxfee_to_getmempoolinfo) https://github.com/bitcoin/bitcoin/pull/11410
1442017-10-05T15:26:13  *** jb55 has quit IRC
1452017-10-05T15:28:28  *** tloriato has joined #bitcoin-core-dev
1462017-10-05T15:31:17  <tloriato> Good afternoon. I was looking into BIP45, that tries to standardize the Structure for Deterministic P2SH Multisignature Wallets, but I've read the emails discussing it and seems to have a little or disagreement between the need for this particular BIP. I'm wondering if there is another standard way to generate a Deterministic Multisig Wallet? I was inclined to generate a standard Mnemonic (BIP39) and split it using Shamir
1472017-10-05T15:33:10  *** arubi has joined #bitcoin-core-dev
1482017-10-05T15:36:28  <BlueMatt> sipa: did you find time to write up the segwit wallet tradeoffs between the various ways of doing it and making an argument for why your pr is your preferred version?
1492017-10-05T15:36:36  <BlueMatt> sipa: I believe you said you were gonna do that last meeting
1502017-10-05T15:48:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1512017-10-05T15:50:19  <jonasschnelli> tloriato: AFAIK BIP45 is the only "standard" to create multisig addresses with a set of given pubkeys.
1522017-10-05T15:50:27  <jonasschnelli> *extended pubkeys
1532017-10-05T15:53:50  <tloriato> jonasschnelli: thanks
1542017-10-05T15:55:55  *** rafalcpp_ has quit IRC
1552017-10-05T15:56:15  *** rafalcpp has joined #bitcoin-core-dev
1562017-10-05T16:00:30  *** Murch has joined #bitcoin-core-dev
1572017-10-05T16:00:45  *** tloriato has quit IRC
1582017-10-05T16:00:47  * BlueMatt has a super interesting C++ performance puzzle if someone wants it: https://github.com/bitcoinfibre/bitcoinfibre/blob/matts-servers/src/udprelay.cpp#L501 sometimes (about once every 2nd or 3rd day's worth of blocks) takes on the rorder of 30ms on dedicated hardware (!!!), but all it does is allocate
1592017-10-05T16:00:51  <BlueMatt> nothing else does anything close
1602017-10-05T16:01:41  *** DrOlmer has quit IRC
1612017-10-05T16:02:38  *** DrOlmer has joined #bitcoin-core-dev
1622017-10-05T16:05:08  *** Emcy has quit IRC
1632017-10-05T16:05:18  *** BashCo has quit IRC
1642017-10-05T16:05:35  *** Emcy has joined #bitcoin-core-dev
1652017-10-05T16:05:52  *** BashCo has joined #bitcoin-core-dev
1662017-10-05T16:06:40  <bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/e93fff1463ae...becbd71b0c16
1672017-10-05T16:06:41  <bitcoin-git> bitcoin/master 4f890ba Donal OConnor: Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system
1682017-10-05T16:06:41  <bitcoin-git> bitcoin/master 696ce46 fanquake: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04
1692017-10-05T16:06:42  <bitcoin-git> bitcoin/master becbd71 Wladimir J. van der Laan: Merge #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04...
1702017-10-05T16:07:25  <bitcoin-git> [bitcoin] laanwj closed pull request #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (master...windows-build-1704) https://github.com/bitcoin/bitcoin/pull/11437
1712017-10-05T16:08:15  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/becbd71b0c16...9e8ef9d99179
1722017-10-05T16:08:16  <bitcoin-git> bitcoin/master f3ba869 practicalswift: [tests] Add libFuzzer support....
1732017-10-05T16:08:16  <bitcoin-git> bitcoin/master 9e8ef9d Wladimir J. van der Laan: Merge #10440: [tests] Add libFuzzer support...
1742017-10-05T16:08:30  <bitcoin-git> [bitcoin] laanwj closed pull request #10440: [tests] Add libFuzzer support (master...libfuzzer) https://github.com/bitcoin/bitcoin/pull/10440
1752017-10-05T16:10:05  *** BashCo has quit IRC
1762017-10-05T16:17:12  <sipa> BlueMatt: i've been distracted by something else
1772017-10-05T16:18:05  *** jb55 has joined #bitcoin-core-dev
1782017-10-05T16:35:53  *** Aaronvan_ has joined #bitcoin-core-dev
1792017-10-05T16:39:22  *** AaronvanW has quit IRC
1802017-10-05T16:40:12  *** Aaronvan_ has quit IRC
1812017-10-05T16:42:50  *** BashCo has joined #bitcoin-core-dev
1822017-10-05T16:45:38  *** abpa has joined #bitcoin-core-dev
1832017-10-05T16:53:56  *** AaronvanW has joined #bitcoin-core-dev
1842017-10-05T17:02:54  <bitcoin-git> [bitcoin] sdaftuar closed pull request #10200: Mining: Skip recent transactions if fee difference is small (master...2017-04-dont-mine-recent-tx) https://github.com/bitcoin/bitcoin/pull/10200
1852017-10-05T17:06:03  *** Emcy has quit IRC
1862017-10-05T17:06:47  *** Emcy has joined #bitcoin-core-dev
1872017-10-05T17:12:30  *** Aaronvan_ has joined #bitcoin-core-dev
1882017-10-05T17:12:37  *** promag has joined #bitcoin-core-dev
1892017-10-05T17:13:37  *** timothy has quit IRC
1902017-10-05T17:14:31  *** Emcy has quit IRC
1912017-10-05T17:14:44  *** Emcy has joined #bitcoin-core-dev
1922017-10-05T17:14:59  *** kingjockey has joined #bitcoin-core-dev
1932017-10-05T17:15:28  *** Aaronvan_ has quit IRC
1942017-10-05T17:16:07  *** Aaronvan_ has joined #bitcoin-core-dev
1952017-10-05T17:16:29  *** AaronvanW has quit IRC
1962017-10-05T17:19:48  *** promag has quit IRC
1972017-10-05T17:25:01  *** Ylbam has joined #bitcoin-core-dev
1982017-10-05T17:26:27  *** Emcy has quit IRC
1992017-10-05T17:27:50  *** Emcy has joined #bitcoin-core-dev
2002017-10-05T17:30:12  *** Chris_Stewart_5 has quit IRC
2012017-10-05T17:44:12  *** Guyver2 has quit IRC
2022017-10-05T17:48:01  *** RealM9 has joined #bitcoin-core-dev
2032017-10-05T17:48:40  <RealM9> Hello, how long until meeting starts? At what time it starts?
2042017-10-05T17:49:30  <sipa> 11 minutes from now
2052017-10-05T17:49:44  <sipa> sorry, 71 minutes
2062017-10-05T17:50:11  <sipa> it's at 7pm UTC
2072017-10-05T17:50:16  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9e8ef9d99179...17f2acedbe07
2082017-10-05T17:50:16  <bitcoin-git> bitcoin/master 0da49b5 Johnson Lau: Skip precompute sighash for transactions without witness
2092017-10-05T17:50:17  <bitcoin-git> bitcoin/master 17f2ace Wladimir J. van der Laan: Merge #9572: Skip witness sighash cache for non-segwit transactions...
2102017-10-05T17:50:23  <bitcoin-git> [bitcoin] laanwj closed pull request #9572: Skip witness sighash cache for non-segwit transactions (master...nocache) https://github.com/bitcoin/bitcoin/pull/9572
2112017-10-05T17:51:09  *** promag has joined #bitcoin-core-dev
2122017-10-05T17:51:16  <RealM9> Ok, thnx sipa
2132017-10-05T17:57:05  *** Aaronvan_ has quit IRC
2142017-10-05T17:57:16  *** AaronvanW has joined #bitcoin-core-dev
2152017-10-05T17:58:35  *** kingjockey has quit IRC
2162017-10-05T17:58:51  *** promag has quit IRC
2172017-10-05T17:59:31  <jl2012> why do we need a vector of PrecomputedTransactionData here? https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1755
2182017-10-05T17:59:36  *** laurentmt has joined #bitcoin-core-dev
2192017-10-05T18:00:28  <sipa> jl2012: what alternative do you suggest?
2202017-10-05T18:00:29  *** AaronvanW has quit IRC
2212017-10-05T18:00:38  <jl2012> why couldn't we just have a PrecomputedTransactionData for each transaction?
2222017-10-05T18:00:48  <sipa> we do?
2232017-10-05T18:00:57  <sipa> that's just the vector that holds them
2242017-10-05T18:01:06  *** AaronvanW has joined #bitcoin-core-dev
2252017-10-05T18:01:34  <jl2012> yes, but why need to hold them all?
2262017-10-05T18:01:59  <sipa> variables need to exist somewhere...
2272017-10-05T18:03:23  <jl2012> yes, I mean, why not just have a line PrecomputedTransactionData txdata(tx); inside the loop "for (unsigned int i = 0; i < block.vtx.size(); i++)"?
2282017-10-05T18:03:59  <sipa> jl2012: oh, i read "why need to them *at* all", sorry
2292017-10-05T18:04:18  <sipa> the reason is that they're used from other threads during parallel validation
2302017-10-05T18:05:41  <jl2012> ok...so without the vector, each thread will re-compute the hashes once?
2312017-10-05T18:06:07  <sipa> depends how you do it
2322017-10-05T18:06:26  <sipa> if you do what you suggest, you'd get undefined behaviour
2332017-10-05T18:06:49  <jl2012> why?
2342017-10-05T18:06:53  <sipa> as the precomputation data would be out of scope before it's accessed from the validation threads
2352017-10-05T18:07:39  *** Emcy has quit IRC
2362017-10-05T18:08:09  *** W4RL0RD has quit IRC
2372017-10-05T18:08:15  <sipa> most of the validation happens after that loop exits, and before the queue's wait call returns
2382017-10-05T18:08:18  *** Emcy has joined #bitcoin-core-dev
2392017-10-05T18:10:35  <JeremyRubin> jl2012: I have a patch that makes them shared_ptrs
2402017-10-05T18:10:45  <BlueMatt> sipa: not anymore (tx script caching) :p
2412017-10-05T18:10:46  <JeremyRubin> conceptually, that's what you want
2422017-10-05T18:10:55  <JeremyRubin> performance wise, a vector is better
2432017-10-05T18:12:49  <jonasschnelli> gmaxwell: for the notification system, you had concerns about the reliability.. would a long poll queue that only removes elements from the queue on an ack from the client be something that would defeat your concerns?
2442017-10-05T18:13:00  <jonasschnelli> With that concept, loosing notifications would be very unlikely
2452017-10-05T18:13:04  <sipa> BlueMatt: ok, i reformulate - most of the validation, if it happens, happens after that loop exits
2462017-10-05T18:15:34  <jl2012> oh, I thought all script validation is done with CheckInputs?
2472017-10-05T18:15:55  <sipa> jl2012: yes and no
2482017-10-05T18:16:17  <sipa> CheckInputs returns a vector of CScriptCheck objects, which represent validation that will happen on another thread
2492017-10-05T18:16:33  <sipa> those get pushed to the checkqueue, where validation threads pick them up
2502017-10-05T18:16:35  *** Emcy has quit IRC
2512017-10-05T18:16:51  <jl2012> ontrol.Add(vChecks); ?
2522017-10-05T18:16:58  <jl2012> control.Add(vChecks); ?
2532017-10-05T18:16:59  <JeremyRubin> yes
2542017-10-05T18:17:20  <gmaxwell> jonasschnelli: it would but wumpus pointed out that that queue would potentially grow without bound if the client stops acking entirely.
2552017-10-05T18:17:58  <sipa> jl2012: and the CScriptCheck objects have a pointer to the precomputation data
2562017-10-05T18:18:05  <JeremyRubin> control.Add returns immediately
2572017-10-05T18:18:12  <wumpus> yes, 100% reliable notification given any behavior of the receiver is not possible given finite space
2582017-10-05T18:18:24  <jonasschnelli> gmaxwell: a queue limit is unavoidable... if you poll to lazy and the queue limit is to little, the only way to detect would be via the sequence number
2592017-10-05T18:18:31  <sipa> jl2012: which means we must guarantee that that precomputation data remains alive as long as other threads may dereference the pointer
2602017-10-05T18:18:40  <wumpus> some notification queue middleware logs events to disk in that case
2612017-10-05T18:18:43  *** abpa has quit IRC
2622017-10-05T18:18:58  <jonasschnelli> wumpus: in long polling, the queue must be finite because we don't know when the client polls next,...
2632017-10-05T18:19:14  <wumpus> e.g. rabbitmq etc, not by any means suggesting we take that up
2642017-10-05T18:19:21  *** AaronvanW has quit IRC
2652017-10-05T18:19:53  <wumpus> jonasschnelli: I agree there needs to be a limit realistically, it's enough if a client can detect missed events to resync
2662017-10-05T18:19:58  *** AaronvanW has joined #bitcoin-core-dev
2672017-10-05T18:20:08  <jonasschnelli> I personally think acking notifications is unnecessary,.. making it configurable seems also an overkill... so unsure.
2682017-10-05T18:20:32  <JeremyRubin> jl2012: this is why I say a shared_ptr is what you really want, because the lifetime is automatically handled for you, whereas the vector is only by careful programming. However, that careful programming is currently correct :)
2692017-10-05T18:21:18  <sipa> jonasschnelli: if you don't ask notifications, then your clients must manually deal with restarts... which implies having a way to ask for all events since some time... which, if you have it, removes the need for an event log entirely, as you can just use that all the time
2702017-10-05T18:21:28  <jl2012> JeremyRubin, sipa: thanks. I'm thinking how I could do this: https://github.com/bitcoin/bitcoin/pull/9572#issuecomment-334282408
2712017-10-05T18:21:47  <wumpus> what's important is to document the limitations of the notification system in that regard
2722017-10-05T18:22:10  <sipa> jl2012: do what?
2732017-10-05T18:22:41  <jl2012> "I'd be a little more comfortable with this if PrecomputedTransactionData were passed around as const, so nobody's tempted to mess with "ready" for some crazy reason. But that can always be done later."
2742017-10-05T18:22:54  <gmaxwell> shared_ptrs are a way to produce a software engineering disaster, because they let you be sloppy with ownership. There are cases of frestanding objects that don't have organized ownership but they are relatively rare. They also have non-trivial overhad.
2752017-10-05T18:23:25  <sipa> jl2012: seems trivial?
2762017-10-05T18:23:54  *** Emcy has joined #bitcoin-core-dev
2772017-10-05T18:23:55  <sipa> i think you're overthinking it
2782017-10-05T18:24:01  *** AaronvanW has quit IRC
2792017-10-05T18:24:28  <jl2012> I'm thinking something like "const PrecomputedTransactionData txdata(tx);"
2802017-10-05T18:25:30  <sipa> jl2012: cfields is only asking to pass it around as const; not to make the entire object const
2812017-10-05T18:26:35  <cfields> ^^ yep
2822017-10-05T18:27:17  <cfields> though i think making the members const would be trivial too
2832017-10-05T18:27:32  <cfields> either way, it's not really important for that PR. just an observation.
2842017-10-05T18:27:44  *** Emcy_ has joined #bitcoin-core-dev
2852017-10-05T18:27:48  <jl2012> it's even better if we could make the object const? There is no reason to make any modification
2862017-10-05T18:28:20  <JeremyRubin> gmaxwell: am aware, but strictly speaking that is the point here: they are designed to track the actual needed lifetime, and are a 'tighter fit' to the problem in this situation. That they have problems is why I didn't PR my patch.
2872017-10-05T18:28:32  <cfields> jl2012: just make the members const, then. use initializers rather than setting them in the function body
2882017-10-05T18:29:03  *** chjj has quit IRC
2892017-10-05T18:29:04  <JeremyRubin> jl2012: I think that we shouldn't make it const, because there will be in the future per-tx state that we'll modify
2902017-10-05T18:29:38  <JeremyRubin> e.g., if at some point in the future signature agg ends up needing a per-tx mutable struct that each input modifies
2912017-10-05T18:29:52  <gmaxwell> JeremyRubin: in the case of the validation the ownership is just a straght line though, the creating code owns it, then the ownership is passed to the queue, then the ownership passes to the verifier thread..
2922017-10-05T18:30:37  <JeremyRubin> Maybe it should be a different struct, but I think (to me), changing PreComputedTxData to PerTxData is straightforward.
2932017-10-05T18:30:56  <sipa> JeremyRubin: i think that should be separate
2942017-10-05T18:31:07  <sipa> if you want to avoid locking on the precomputed data
2952017-10-05T18:31:29  *** Emcy has quit IRC
2962017-10-05T18:31:32  <JeremyRubin> gmaxwell: yes, and then they are kept alive for longer than strictly needed. After the last scriptcheck executes for that tx, they can be freed.
2972017-10-05T18:31:33  <cfields> agree. It'd be nice to keep the factual data separate from what's aggregated/stateful
2982017-10-05T18:31:58  <sipa> JeremyRubin: yes, so? it doesn't change the worst case resource consumption
2992017-10-05T18:32:04  <JeremyRubin> cfields: all the data should be factual in consensus?
3002017-10-05T18:33:12  <jonasschnelli> sipa: persist the notification queue (with it's sequence numbers) to make it restart-safe?
3012017-10-05T18:33:12  <JeremyRubin> sipa: I'm really only trying to talk about lifetimes here, which is the core of what jl2012 is discussing
3022017-10-05T18:33:24  <JeremyRubin> sipa: asking
3032017-10-05T18:35:21  <sipa> JeremyRubin: i'm just trying to point out that minimizing the time an object lives is not always wanted
3042017-10-05T18:35:26  <cfields> JeremyRubin: i just meant that it can be safely computed ahead of time and seen as immutible after that.
3052017-10-05T18:35:45  <cfields> factual was the wrong word, i suppose.
3062017-10-05T18:36:09  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3072017-10-05T18:37:02  <JeremyRubin> sipa: of course
3082017-10-05T18:38:31  <sipa> jonasschnelli: that's one way (but it has pretty bad costs, if you want to make it durable - now you need synchronization across all files/databases)
3092017-10-05T18:39:10  <sipa> jonasschnelli: another model is that you just have RPCs like listsinceblock, where the client tells the server effectively what it already knows
3102017-10-05T18:39:31  <sipa> jonasschnelli: and then all you need is a notification like "there's something for you to look at"
3112017-10-05T18:40:06  <jonasschnelli> sipa: by a rolling hash?
3122017-10-05T18:40:12  <sipa> jonasschnelli: wut?
3132017-10-05T18:40:29  <sipa> no, just like listsinceblock
3142017-10-05T18:40:30  <jonasschnelli> how would the client tell the server what notfications it has... just the sequence number?
3152017-10-05T18:40:36  <jonasschnelli> (need to check that)
3162017-10-05T18:40:43  <jonasschnelli> thanks
3172017-10-05T18:40:43  <sipa> no, i'm saying there are no notification
3182017-10-05T18:40:53  <sipa> there are just state changes
3192017-10-05T18:41:23  <sipa> and the client can ask "i've synced up to block X" or "i've seen transactions up to timestamp Y", or "the last balance update i saw was Z"
3202017-10-05T18:42:00  <sipa> because most data has inherent sequencing anyway already
3212017-10-05T18:42:22  <jonasschnelli> could it also do long polling? ... because IMO that is what users want (not const. polling)
3222017-10-05T18:42:41  <sipa> yes, but the long poll just returns "there is something you should look at", not what
3232017-10-05T18:43:01  <jonasschnelli> ah. And then the state update call.
3242017-10-05T18:43:05  *** promag has joined #bitcoin-core-dev
3252017-10-05T18:43:08  <sipa> ... no
3262017-10-05T18:43:28  <sipa> i mean *exactly* like listsinceblock
3272017-10-05T18:43:41  <jonasschnelli> heh.. okay let me dive into there first
3282017-10-05T18:43:50  <sipa> it has no sequence numbers, or notifications
3292017-10-05T18:44:15  <sipa> it's a call where the client tells the server "hey, i've seen all transactions up to block X. what new transactions are there"
3302017-10-05T18:44:38  <sipa> the next time the client calls, it uses the block hash at the time of its previous call
3312017-10-05T18:44:54  <sipa> the server doesn't need to keep track of anything
3322017-10-05T18:45:50  <jonasschnelli> would that also work with non block/tx data? Like new wallet txns (locally injected) or bandwith watermarks?
3332017-10-05T18:46:08  <jonasschnelli> although not sure if there is a need for that
3342017-10-05T18:46:10  <sipa> perhaps - maybe it's not possible for everything
3352017-10-05T18:46:41  *** jcorgan has joined #bitcoin-core-dev
3362017-10-05T18:46:45  <sipa> bandwidth watermarks are easy... the client doesn't care if he misses one
3372017-10-05T18:47:06  *** AaronvanW has joined #bitcoin-core-dev
3382017-10-05T18:47:29  <sipa> for wallet txn you can use the txid of the last seen tx
3392017-10-05T18:47:37  *** promag has quit IRC
3402017-10-05T18:48:20  <sipa> jonasschnelli: my point is that you probably need something like that anyway, at least for when a new client starts up or went offline for a while
3412017-10-05T18:49:01  <jonasschnelli> sipa: Yes. Your idea make sense.
3422017-10-05T18:49:12  <sipa> and when you do, why bother with a separate event queue - all events could just be "hey something happened, you should check what"
3432017-10-05T18:49:37  <sipa> that's a big concern i had with ZMQ... as it passes the actual new tx and blocks, but without guarantees that it delivers all
3442017-10-05T18:49:49  <sipa> it was fixed with sequence numbers
3452017-10-05T18:49:50  <jonasschnelli> maybe you should be able to not get such notifications on new mempool txns
3462017-10-05T18:50:10  <sipa> but it could also have been fixed by not having it at all, and making clients query for the data they didn't have after every notifications"
3472017-10-05T18:50:57  <sipa> anyway, just an idea
3482017-10-05T18:51:10  <jonasschnelli> Thanks for sharing... need to think about it a bit more
3492017-10-05T18:51:14  <sipa> having events you can subscribe to individually that persist etc... is certainly more convenient for clients
3502017-10-05T18:51:50  <sipa> but it makes bitcoin core responsible for tracking who knows what, instead of leaving that up to clients (who are arguably in a better position to know what they already know)
3512017-10-05T18:52:08  <jonasschnelli> Indeed. And also the ressources.
3522017-10-05T18:52:53  *** promag has joined #bitcoin-core-dev
3532017-10-05T18:55:34  <sipa> it would be pretty nice if there was a way to subscribe to "hey, let me know when txid X gets Y confirmations or gets reorged", but perhaps that could be done in a python script that just uses a simpler interface with bitcoidn
3542017-10-05T19:00:07  <achow101> meeting?
3552017-10-05T19:00:39  *** meshcollider has joined #bitcoin-core-dev
3562017-10-05T19:00:44  <instagibbs> yes
3572017-10-05T19:01:12  <sipa> yes
3582017-10-05T19:02:18  <jonasschnelli> #startmeeting
3592017-10-05T19:02:18  <lightningbot> Meeting started Thu Oct  5 19:02:18 2017 UTC.  The chair is jonasschnelli. Information about MeetBot at http://wiki.debian.org/MeetBot.
3602017-10-05T19:02:18  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
3612017-10-05T19:02:22  <wumpus> yes
3622017-10-05T19:02:26  <jonasschnelli> Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
3632017-10-05T19:02:32  <kanzure> hi.
3642017-10-05T19:02:34  <jonasschnelli> wumpus: sry: though your where OL
3652017-10-05T19:02:42  <cfields> hi
3662017-10-05T19:02:43  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101
3672017-10-05T19:02:43  <meshcollider> Hello
3682017-10-05T19:02:45  <achow101> hi
3692017-10-05T19:02:49  <luke-jr> hi
3702017-10-05T19:03:10  <wumpus> jonasschnelli: yes I was a bit late, sorry
3712017-10-05T19:03:21  <wumpus> #topic high-priority for review
3722017-10-05T19:03:43  <michagogo> Huh?
3732017-10-05T19:03:43  <jonasschnelli> #chair wumpus
3742017-10-05T19:03:43  <lightningbot> Current chairs: jonasschnelli wumpus
3752017-10-05T19:03:44  <achow101> #10637 please?
3762017-10-05T19:03:47  <gribble> https://github.com/bitcoin/bitcoin/issues/10637 | Coin Selection with Murchs algorithm by achow101 · Pull Request #10637 · bitcoin/bitcoin · GitHub
3772017-10-05T19:04:15  <michagogo> Oh, right
3782017-10-05T19:04:19  <michagogo> It's actually Thursday
3792017-10-05T19:04:34  *** jb55 has quit IRC
3802017-10-05T19:04:34  <wumpus> achow101: ok
3812017-10-05T19:05:16  <wumpus> I also added #11389 today as it's blocking segwit wallet support
3822017-10-05T19:05:16  * jtimon locks at #8498 and hides for the rest of the meeting
3832017-10-05T19:05:17  <gribble> https://github.com/bitcoin/bitcoin/issues/11389 | Support having SegWit always active in regtest by sipa · Pull Request #11389 · bitcoin/bitcoin · GitHub
3842017-10-05T19:05:20  <gribble> https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
3852017-10-05T19:05:47  <meshcollider> #11403 itself should be in there too probably?
3862017-10-05T19:05:50  <gribble> https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
3872017-10-05T19:06:07  <sipa> wumpus: i haven't had the time to work further on 11403, though concept review is certainly welcome
3882017-10-05T19:06:16  <jnewbery> I've been reviewing 11389 this afternoon. It looks generally good, but breaks assumevalid.py, which I'm trying to fix now
3892017-10-05T19:06:28  <jtimon> s/locks/looks/
3902017-10-05T19:06:47  <BlueMatt> sipa: I think we need a document on the various possible approaches, tbh
3912017-10-05T19:06:58  <sipa> BlueMatt: yes, i'll work on that soon
3922017-10-05T19:07:00  <BlueMatt> there are a few and talking through all of them is going to need something more formal
3932017-10-05T19:07:02  <BlueMatt> thanks
3942017-10-05T19:07:21  <morcos> achow101: does 10637 implement all the coin selection logic we discussed in SF or only BnB?  Is there a high level description somewhere of what the PR is purporting to accomplish and what else will need to be done before 0.16?
3952017-10-05T19:07:32  <achow101> morcos: only BnB
3962017-10-05T19:07:48  <achow101> morcos: IIRC Murch is working on all of the coin selection stuff that we discussed
3972017-10-05T19:07:52  <wumpus> btw I posted a proposed release schedule for 0.16.0 yesterday
3982017-10-05T19:07:55  <wumpus> #link https://github.com/bitcoin/bitcoin/issues/11449
3992017-10-05T19:08:28  <morcos> achow101: ok.. i've already forgotten what that is, so might be nice to have that written up in an issue or something so we remember the goal and can think about how this BnB implementation is going to fit into the big picture
4002017-10-05T19:09:16  <achow101> morcos: the description of what 10637 does is in the first comment.
4012017-10-05T19:09:35  <achow101> I can make an issue for coin selection changes in general
4022017-10-05T19:09:42  <achow101> *to keep track of
4032017-10-05T19:10:48  *** jtimon has quit IRC
4042017-10-05T19:10:53  <wumpus> ok, I think that concludes high priority for review proposals
4052017-10-05T19:10:57  <wumpus> any other topics?
4062017-10-05T19:11:20  <achow101> topic suggestion: bad block interrogation/invalid block peer banning
4072017-10-05T19:11:25  <wumpus> #action achow101 make an issue for coin selection changes in general
4082017-10-05T19:11:38  <wumpus> #topic bad block interrogation/invalid block peer banning
4092017-10-05T19:11:53  <achow101> relevant PR is #11446 (I did this in class so it kinda sucks)
4102017-10-05T19:11:54  <gribble> https://github.com/bitcoin/bitcoin/issues/11446 | [WIP] Bad block interrogation by achow101 · Pull Request #11446 · bitcoin/bitcoin · GitHub
4112017-10-05T19:12:06  <Murch> hey, sorry, was still in a meeting
4122017-10-05T19:12:16  *** abpa has joined #bitcoin-core-dev
4132017-10-05T19:12:34  <achow101> basically the idea is gmaxwell's. when we receive an invalid block, we want to make sure that all of our peers would also reject that block as invalid. If they don't ban them
4142017-10-05T19:12:37  <Murch> I've been working on it, but since I do that in my free time in the evenings, it's been rather slow.
4152017-10-05T19:12:53  <luke-jr> wumpus: this feels delayed?
4162017-10-05T19:13:07  <gmaxwell> The general idea is that we aren't sufficiently agressive about punting peers on different consensus rules, so they can DOS attack us by sucking up slots, potentially hours per peer leaving us isolated... So there are number of things we can to do seek and destroy to speed up up.
4172017-10-05T19:13:09  <wumpus> luke-jr: what feels delayed?
4182017-10-05T19:13:14  <luke-jr> wumpus: 0.16
4192017-10-05T19:13:17  <Murch> @achow101: If you want to collaborate on a write-up, I'd make myself available for that.
4202017-10-05T19:13:20  <gmaxwell> luke-jr: release schedule is delayed because of 0.15.1
4212017-10-05T19:13:23  <achow101> Murch: ok
4222017-10-05T19:13:24  <luke-jr> i c
4232017-10-05T19:13:31  <wumpus> luke-jr: yes, two months extra added, I mention that in the issue
4242017-10-05T19:13:53  <achow101> what I wanted to discuss was the way to actually go about determining who to ban
4252017-10-05T19:14:20  <Murch> @achow101: Gonna be traveling the next three weeks, so I might actually have more time. ;)
4262017-10-05T19:14:22  <sipa> what is the issue with just looking at headers?
4272017-10-05T19:14:23  <gmaxwell> achow101: I was kinda hoping we could implement something just from the messages we already get, it's my belief (could be wrong) that effectively we always learn the peers best header chain-- so we can begin kicking off peers based on that, as a first pass.
4282017-10-05T19:14:34  <luke-jr> achow101: this contradicts the fixes in #10593
4292017-10-05T19:14:36  <gribble> https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
4302017-10-05T19:14:58  <gmaxwell> achow101: I think we should be also drawing a distinction between inbound and outbound: the issue is what if we have a peer that accepts a broader set of blocks but would switch to our chain after learning of it.
4312017-10-05T19:15:15  <achow101> gmaxwell: that's what I am not sure about. I don't think we necessarily know our peer's best header chain. suppose both us and them are fully synced, how do we know their best header chain until a new block appears?
4322017-10-05T19:15:47  <wumpus> luke-jr: maybe two months is too much, but we'll see...
4332017-10-05T19:16:03  <luke-jr> wumpus: nah, that sounds reasonable
4342017-10-05T19:16:06  <sipa> achow101: when a new block appears, assuming it's PoW-valid to us, we'll learn about it through inv/headers/cb/...
4352017-10-05T19:16:14  <jonasschnelli> Yes. +2 M seems okay to me
4362017-10-05T19:16:36  <gmaxwell> sipa: but I believe he's right, we would have to wait for a new block, which is among the situations we're trying to resolve.
4372017-10-05T19:17:07  <achow101> sipa: right, but I'm concerned about before a new block appears. we just connected to them or they just connected to us. we want to know then if we should ban them or not
4382017-10-05T19:17:20  <luke-jr> IMO the desirable logic would be: for outbound connections, disconnect (don't ban) peers that aren't on the same chain; for inbound, tolerate it unless they reject a known-valid block
4392017-10-05T19:17:22  <sdaftuar> we send getheaders messages on connect, typically
4402017-10-05T19:17:25  <gmaxwell> For example say we are surrounded by ForkCoin peers, they are rejecting all bitcoin blocks.  There are few forkcoin miners so they only get blocks once per day.
4412017-10-05T19:18:01  <gmaxwell> We don't want to wait for them to get a new block just to figure out our current batch of peers are already on a chain we reject.
4422017-10-05T19:18:02  <achow101> sdaftuar: are you sure? all I could find is that we sometimes send getheaders, not all the time
4432017-10-05T19:18:29  <sdaftuar> achow101: we send getheaders messages to all our peers at some point after startup, but they might ignore them
4442017-10-05T19:18:34  <cfields> sdaftuar: not to incoming light clients, i think?
4452017-10-05T19:18:35  <sdaftuar> eg if they are doing ibd themselves or something
4462017-10-05T19:18:44  <sdaftuar> not to light clients, correct
4472017-10-05T19:18:52  <sipa> light clients don't matter here
4482017-10-05T19:18:53  <sdaftuar> but to inbound node_network ndoes we do
4492017-10-05T19:19:07  <gmaxwell> If we _always_ sent getheaders and then kicked outbound peers whos chain has a block we've rejected, then I think that is the best we can do per that concern (still not a perfect fix, since you're isolated until forkcoin finds at least one block)
4502017-10-05T19:19:18  <achow101> sdaftuar: if we are sending getheaders, if they are on a different chain, we still wouldn't necessarily know because our start block may not be on their best chain
4512017-10-05T19:19:23  <gmaxwell> oh hm. then perhaps we already do where it matters.
4522017-10-05T19:19:42  <sdaftuar> gmaxwell: the difficult part might be that you don't know the chain they're on is invalid
4532017-10-05T19:19:47  <sdaftuar> if it's got less work than yours
4542017-10-05T19:19:53  <RealM9> Topic suggestion: s2x
4552017-10-05T19:20:04  <jonasschnelli> RealM9: no
4562017-10-05T19:20:06  <luke-jr> sdaftuar: do you care?
4572017-10-05T19:20:28  <luke-jr> if they're rejecting your better chain, you want to disconnect them anyway
4582017-10-05T19:20:33  <gmaxwell> sdaftuar: seems like a seperate concern, we should also be kicking outbound peers that have less work than us, I think.
4592017-10-05T19:20:38  <RealM9> Ok, but community is pretty interested. Are you going to change POW?
4602017-10-05T19:20:43  <sdaftuar> gmaxwell: i think that would be a good idea, yeah
4612017-10-05T19:20:43  <gmaxwell> But it would be silly to be overly agressive.
4622017-10-05T19:20:48  <sipa> RealM9: us?
4632017-10-05T19:20:58  <achow101> sdaftuar: gmaxwell what I propose is that we send a getheaders for our earliest known invalid block (within a certain time frame) and see if they respond with invalid blocks
4642017-10-05T19:21:18  <luke-jr> RealM9: that's a decision for the community, not for developers. anyhow, ask on #bitcoin if you really want to discuss it
4652017-10-05T19:21:34  <gmaxwell> achow101: I don't think we need to do that: for sync purposes any outbound peer we should be makign sure we learn their headers chain period (they may have a better chain than us and we should sync up ASAP)
4662017-10-05T19:21:36  <sdaftuar> achow101: i'm not sure that's necessary?
4672017-10-05T19:21:45  <morcos> RealM9: we're in a meeting , but please see: https://bitcoincore.org/en/2017/08/18/btc1-misleading-statements/
4682017-10-05T19:21:49  <gmaxwell> achow101: if we're already doing that we'll notice known invalid block in their header chaip (well we will once we have code for that)
4692017-10-05T19:21:52  <sdaftuar> i think if we do gmaxwell's suggestion of booting inbound peers who are on less work chains, then we'd be in good shape
4702017-10-05T19:21:58  <sdaftuar> s/inbound/outbound/
4712017-10-05T19:22:14  <luke-jr> I think we may actually want to track the headers of invalid chains..
4722017-10-05T19:22:21  <achow101> what about inbound peers?
4732017-10-05T19:22:31  <gmaxwell> For _inbound_ I think we should be setting a flag that they're consensus inconsistent which excludes them from the inbound peer management connection reservation.
4742017-10-05T19:22:35  <sdaftuar> achow101: i think we should more aggerssively evict inbound peers if they appear to be on invalid chains
4752017-10-05T19:22:43  <gmaxwell> so they'll get kicked off in favor of other inbound peers.
4762017-10-05T19:22:52  <meshcollider> Agreed
4772017-10-05T19:23:02  <gmaxwell> so we don't need to be agressive: they'll just get pushed out by other inbound peers.
4782017-10-05T19:23:02  <luke-jr> consider: if an invalid chain has higher hashrate than the real chain, and then suddenly the invalid chain's hashrate drops off, without an equivalent increase on the main chain, we should consider that a possible attack and hold back on confirming transactions until it is resolved
4792017-10-05T19:23:16  <sdaftuar> gmaxwell: yes i agree with you
4802017-10-05T19:23:49  <achow101> my point is how do we know that an inbound peer is on an invalid chain?
4812017-10-05T19:24:03  <gmaxwell> luke-jr: I think there is some need for smarter wallet confirmation logic but I think thats a seperate matter. (there was a paper 6-ish months ago that also points out the the reorg probablity math in the whitepaper is somewhat incomplete)
4822017-10-05T19:24:19  <sdaftuar> achow101: set a flag if they relay an invalid block/blockheader
4832017-10-05T19:24:32  <luke-jr> gmaxwell: right, but this is relevant because we can't assume "relays invalid headers" means the other node *accepts* the invalid block
4842017-10-05T19:24:44  <gmaxwell> and we still interogate their headers if they're NODE_NETWORK/NODE_LIMITED
4852017-10-05T19:24:46  *** RoyceX has joined #bitcoin-core-dev
4862017-10-05T19:24:58  <luke-jr> sdaftuar: we intentionally relay blocks before checking validity now
4872017-10-05T19:25:03  <achow101> sdaftuar: that requires them to have a block to relay to us, which could take hours or days
4882017-10-05T19:25:18  <gmaxwell> luke-jr: the protocol does not have you realying a header of a block you haven't accepted. If you do that you risk dos attacking peers already.
4892017-10-05T19:25:19  <sdaftuar> achow101: i don't think we need to worry as much about inbound peers
4902017-10-05T19:25:34  <sdaftuar> achow101: for instance an attacker can already try to use all your inbound slots and not send you anything
4912017-10-05T19:25:40  <gmaxwell> luke-jr: the only place that happens in the protocol is HB BIP152 messages.
4922017-10-05T19:25:51  <achow101> sdaftuar: right, ok
4932017-10-05T19:25:58  <luke-jr> gmaxwell: which may be all you see from CB peers
4942017-10-05T19:26:18  <gmaxwell> sdaftuar: yes, for inbound we can just deprive them of reservations.
4952017-10-05T19:26:49  <sdaftuar> luke-jr: even with bip152 the headers need to be valid
4962017-10-05T19:27:04  <luke-jr> sdaftuar: yes, the header itself; but it can be a valid header for an invalid block
4972017-10-05T19:27:16  <gmaxwell> yes, though we'd catch it on the _next_ block.
4982017-10-05T19:27:19  <sdaftuar> luke-jr: if it builds on an invalid chain, i believe the header would be invalid
4992017-10-05T19:27:21  *** laurentmt has quit IRC
5002017-10-05T19:27:26  <achow101> so when we connect to an outbound peer, we will send them a getheaders so we know their best headers chain and ban accordingly
5012017-10-05T19:27:29  <luke-jr> (note I tried to keep track of peer bestblocks in #10512 and basically gave up)
5022017-10-05T19:27:31  <gribble> https://github.com/bitcoin/bitcoin/issues/10512 | Rework same-chain from abusing DoS banning, to explicit checks by luke-jr · Pull Request #10512 · bitcoin/bitcoin · GitHub
5032017-10-05T19:27:41  <gmaxwell> when they relay a CB message for a child of an invalid block.
5042017-10-05T19:27:51  *** Cheeseo has quit IRC
5052017-10-05T19:28:04  <gmaxwell> achow101: yes, but based on the above I believe we already always send it.
5062017-10-05T19:28:11  <achow101> the other part of 11446 is to ban other peers for relaying us an invalid block for which we already know is invalid
5072017-10-05T19:28:20  <gmaxwell> achow101: because we send it to nodenetwork peers and outbound always are (or or disconnected)
5082017-10-05T19:28:20  <achow101> but I'm not sure how that interacts with compact blocks
5092017-10-05T19:28:33  <gmaxwell> achow101: FWIW, I think we should probably be just disconnecting and not banning.
5102017-10-05T19:28:37  <sdaftuar> achow101: oh that interaction might be tricky
5112017-10-05T19:28:54  <achow101> gmaxwell: why not ban?
5122017-10-05T19:29:07  <gmaxwell> I think the interaction isn't too bad, for this purpose a BIP152 CB HB block is relaying you the header of its parent.
5132017-10-05T19:29:20  *** chjj has joined #bitcoin-core-dev
5142017-10-05T19:29:22  <luke-jr> achow101: in a softfork, old nodes will send invalid blocks
5152017-10-05T19:29:27  <luke-jr> potentially
5162017-10-05T19:29:42  <gmaxwell> achow101: because it's hardly any better and it means that when some dimbulb tries running forkcoin it results in him being unable to run Bitcoin (perhaps concurrently) on the same host.
5172017-10-05T19:30:03  <achow101> gmaxwell: ok
5182017-10-05T19:30:04  <gmaxwell> it also blocks inbound from that peer, which we'd be find allowing.
5192017-10-05T19:30:09  <gmaxwell> s/find/fine/
5202017-10-05T19:30:34  <gmaxwell> In general we should be moving away from bans except when the thing we banned for was expensive for us.
5212017-10-05T19:31:03  <achow101> so 11446 can really just be reduced to an ~1 line change to disconnect on a header for a block we already know is invalid
5222017-10-05T19:31:11  <BlueMatt> yea, that
5232017-10-05T19:31:18  <sdaftuar> achow101: agree, though we have to be careful about compact blocks i think
5242017-10-05T19:31:20  <luke-jr> achow101: aka #10593 …
5252017-10-05T19:31:22  <gribble> https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
5262017-10-05T19:31:48  <gmaxwell> achow101: yes, but for compact block interactions (HB mode will relay us blocks that are invalid).
5272017-10-05T19:32:27  <achow101> gmaxwell: so we would have to check the specific type of invalidness and whether it was a CB?
5282017-10-05T19:32:31  <gmaxwell> luke-jr: IIRC when you proposed that before you got squaked at that it would undermine our protection against isolation...
5292017-10-05T19:33:04  <gmaxwell> achow101: or just don't do it for the peers maked for HB CBs for now
5302017-10-05T19:33:17  <achow101> gmaxwell: isn't that likely to be most peers though
5312017-10-05T19:33:19  <achow101> ?
5322017-10-05T19:33:21  *** SopaXorzTaker has quit IRC
5332017-10-05T19:33:23  <gmaxwell> No, it's at most three.
5342017-10-05T19:33:33  <luke-jr> gmaxwell: I don't see how. It's literally what achow101 was just describing.
5352017-10-05T19:34:32  <luke-jr> gmaxwell: maybe you're thinking of the predecessor 10512?
5362017-10-05T19:34:35  <achow101> luke-jr: it doesn't look like you are handling invalid duplicates?
5372017-10-05T19:34:40  <gmaxwell> probably.
5382017-10-05T19:34:57  *** promag has quit IRC
5392017-10-05T19:34:58  <gmaxwell> achow101: in any case, we can pick this up on a PR and later discussion.
5402017-10-05T19:35:09  <achow101> gmaxwell: ok
5412017-10-05T19:36:01  *** wraithm has quit IRC
5422017-10-05T19:36:08  <achow101> next topic then?
5432017-10-05T19:36:09  <wumpus> any other topics?
5442017-10-05T19:36:10  <luke-jr> achow101: IIRC it does, we can go over it later if you like
5452017-10-05T19:36:14  <achow101> luke-jr: ok
5462017-10-05T19:37:09  <jnewbery> luke-jr: any progress on multiwallet GUI without the rpcauth parts?
5472017-10-05T19:37:12  *** promag has joined #bitcoin-core-dev
5482017-10-05T19:37:21  <wumpus> #topic multiwallet ui
5492017-10-05T19:37:32  <luke-jr> jnewbery: not yet, I'll plan to push the update later today
5502017-10-05T19:38:04  <jnewbery> great! I had a look myself, and I think it's just a one-line change to the debug console commit
5512017-10-05T19:38:05  <jonasschnelli> https://github.com/bitcoin/bitcoin/pull/11383
5522017-10-05T19:38:12  <sipa> topic suggestion: dealing with platform-specific code
5532017-10-05T19:38:24  <jonasschnelli> luke-jr: I can continue to work on 11383 if you want?
5542017-10-05T19:38:32  <jonasschnelli> (remove the auth stuff :P)
5552017-10-05T19:38:50  <luke-jr> jnewbery: certainly not that simple.. still need to resolve wallet name to CWallet earlier
5562017-10-05T19:39:13  <jnewbery> ok, well I've got a branch that works with just that change. Happy to share with you
5572017-10-05T19:39:47  <gmaxwell> Sounds good.
5582017-10-05T19:40:00  <luke-jr> jnewbery: push it and I'll take a look
5592017-10-05T19:40:17  <jnewbery> thanks
5602017-10-05T19:40:48  *** AaronvanW has quit IRC
5612017-10-05T19:41:03  <wumpus> #topic dealing with platform-specific code (sipa)
5622017-10-05T19:41:19  <sipa> i've recently been looking into faster parallel hashing code
5632017-10-05T19:41:24  *** AaronvanW has joined #bitcoin-core-dev
5642017-10-05T19:41:37  <wumpus> hashing as in sha256?
5652017-10-05T19:41:50  <sipa> in particular, for 8-way parallel SHA256 (which would be useful in merkle root computation and block deserialization), a 5x speedup is doable with AVX2
5662017-10-05T19:42:04  <sipa> and maybe 2.5x with SSE2
5672017-10-05T19:42:22  <wumpus> and parallel in this case means computing multiple hashes of multiple pieces of data at once?
5682017-10-05T19:42:27  <sipa> correct
5692017-10-05T19:42:28  <luke-jr> how much speedup is this for the entire IBD?
5702017-10-05T19:42:57  <gmaxwell> luke-jr: It saves something like 10 minutes on IBD.  But the greater impact is in block relay.
5712017-10-05T19:43:01  <wumpus> (I guess there are constraints there, do all the inputs need to be the same size?)
5722017-10-05T19:43:05  <luke-jr> I imagine merkle root is a tiny fraction of the overall process, but otoh it's also possibly a blocker on parallelization
5732017-10-05T19:43:05  <gmaxwell> Where hash tree computation is most of the time.
5742017-10-05T19:43:25  <luke-jr> gmaxwell: it is? :O
5752017-10-05T19:43:32  <sipa> wumpus: yes and no; for now, it's just a primitive that you give a pointer to N 64-byte inputs, and produces 32-byte outputs
5762017-10-05T19:43:33  <luke-jr> oh, because the signature checks are cached?
5772017-10-05T19:43:37  <BlueMatt> in terms of compact block relay, merkle root calculation and deserialize are about the only big timesinks before you can relay
5782017-10-05T19:43:46  <sipa> wumpus: which is specific for merkle root computation
5792017-10-05T19:43:48  <gmaxwell> luke-jr: yes for HB BIP152 we don't to validation except hashing!
5802017-10-05T19:43:50  <sipa> but it can certainly be adapted
5812017-10-05T19:44:37  <wumpus> sipa: ok
5822017-10-05T19:44:38  <cfields> sipa: i had a scare when reviewing some new leveldb crc32 changes that i thought (at first glance) could be a consensus issue. I was very angry at myself at that point for not adding a fallback un-optimized verification of the optimized path.
5832017-10-05T19:44:43  <sipa> anyway, there are multiple ways to integrate this: separate asm code, inline asm blocks, or code using intrinsics (my preference, it's much more easy to review, and has no OS-specific warts like the L label prefix...)
5842017-10-05T19:44:56  <gmaxwell> sipa has actually implemented the 8-way AVX2 sha2 and a hash tree computation that uses it... along with specialized implementation of 64-byte input double sha2.. which affords an addition 20%-ish speedup over generic sha2.
5852017-10-05T19:45:09  <cfields> very cool :)
5862017-10-05T19:45:12  <wumpus> I prefer intrinsics
5872017-10-05T19:45:21  <wumpus> (except for arm32 whre they suck)
5882017-10-05T19:45:48  <sipa> so, for intrinsics... do we want to have a separate LIBCRYPTO_AVX2 LIBCRYPTO_SSE2 LIBCRYPTO_... with different compile flags each?
5892017-10-05T19:46:05  <sipa> or could we rely on __attribute__((target("avx2")))
5902017-10-05T19:46:05  <gmaxwell> Historically, For some code you cannot achieve equivilent performance w/ intrinsics because you must manage register allocation precisely for things to work, but that isn't the case here....
5912017-10-05T19:46:08  <wumpus> but 64 bit platforms the SIMD instructions have been specially tweaked to work well with compilers and intrinsics
5922017-10-05T19:46:12  <sipa> (which works on both clang and gcc)
5932017-10-05T19:46:23  *** EricCartman has quit IRC
5942017-10-05T19:46:36  <cfields> sipa: i think we should test for the target attribute and use it if possible, but not completely rely on it
5952017-10-05T19:46:44  <cfields> iirc that improves dispatching time as well?
5962017-10-05T19:46:49  <sipa> cfields: no
5972017-10-05T19:46:57  <wumpus> different compile flags for different compile units, that's the only portable way
5982017-10-05T19:47:01  <luke-jr> sipa: I think we can't assume intrinsics for all platforms, so we want the separate lib route
5992017-10-05T19:47:07  <gmaxwell> dispatching is via a function pointer ultimately in all those cases.
6002017-10-05T19:47:14  <wumpus> luke-jr: you're confusing, that's not about intrinsics
6012017-10-05T19:47:30  <sipa> the only difference is avoiding the need for build system complication
6022017-10-05T19:47:35  <wumpus> intrinsics inthis case are headers like xmmintr.h which provides functions that work on vector types
6032017-10-05T19:47:42  <sipa> exactly
6042017-10-05T19:47:52  <luke-jr> wumpus: __attribute__((target("avx2"))) isn't an option for separate asm code, though, right?
6052017-10-05T19:48:00  <sipa> luke-jr: it also isn't needed for asm code
6062017-10-05T19:48:02  <cfields> gmaxwell: isn't there elf data that allows them to be setup at load time?
6072017-10-05T19:48:14  <wumpus> oh no no ELF magic please
6082017-10-05T19:48:24  <luke-jr> hmm
6092017-10-05T19:48:36  <cfields> not by hand, i thought __attribute__(target) did that behind the scenes
6102017-10-05T19:48:41  <sipa> target("avx2") just means "this function is compiled as if -mavx2 was passed on the command line
6112017-10-05T19:49:02  <sipa> cfields: GCC also has target("default"), where you can have multiple versions of the same function... which causes automatic dispatch to be added
6122017-10-05T19:49:08  <sipa> that's non-portable and has other issues
6132017-10-05T19:49:09  <gmaxwell> cfields: they're setup at load time, yes-- but they're still just a function pointer, which we could also have setup at load time.  Though it is nice that the function override trick can make them run before main.
6142017-10-05T19:49:11  <wumpus> I'm normally not scared of low level ELF hacking, but for bitcoin, let's try to keep it safe and portable
6152017-10-05T19:49:12  <luke-jr> sipa: how does it behave if I have an explicit -mno-avx2?
6162017-10-05T19:49:15  <sipa> (in particular clang doesn't have that)
6172017-10-05T19:49:17  <cfields> sipa: ah yes, that's what i was thinking of.
6182017-10-05T19:49:33  <sipa> cfields: so i'm not suggesting using that
6192017-10-05T19:49:39  <sipa> luke-jr: i assume it overrides it
6202017-10-05T19:49:43  <cfields> ok
6212017-10-05T19:50:02  <luke-jr> I suppose I can test it
6222017-10-05T19:50:05  <wumpus> yes, gcc can do it automatically on some platforms, but I'm afraid the only portable way is to make our own dispatch logic
6232017-10-05T19:50:15  <sipa> yes, we'll want our own dispatch logic anyway
6242017-10-05T19:50:18  <wumpus> we already have some CPUID bits checking
6252017-10-05T19:50:20  <sipa> so we can test things
6262017-10-05T19:50:21  <wumpus> so it's nothing new erally
6272017-10-05T19:50:23  *** Chris_Stewart_5 has quit IRC
6282017-10-05T19:50:25  <sipa> and report which version is being chosen
6292017-10-05T19:50:29  <cfields> np, i wasn't suggesting. just trying to understand the advantages of one vs the other.
6302017-10-05T19:50:30  <wumpus> yes, exactly
6312017-10-05T19:51:01  <sipa> but if possible i'd like to avoid the overhead of needing half a dozen libcrypto_XXX.a things that need to be linked in everywhere
6322017-10-05T19:51:08  <sipa> though that's really the only advantage
6332017-10-05T19:51:38  <wumpus> so I'd say: yes, use intrinsics instead of inline/offline asm,  and use our own dispatching, and compile units compiled with appropriate compiler flags
6342017-10-05T19:51:55  <sipa> okay.
6352017-10-05T19:52:28  <gmaxwell> can we say prefer intrinsics instead of use? :) I don't think we'd eschew inline asm if we thought it was better in a particular case.
6362017-10-05T19:52:29  <wumpus> yes regarding build system it's just verbose, not really complex
6372017-10-05T19:52:41  <cfields> sipa: see my point above about a fallback, though. In the case of mismatch hashes, i think it's worthwhile to re-check with a generic implementation before deciding it's failed.
6382017-10-05T19:53:12  <gmaxwell> cfields: we should be testing these things in startup tests.
6392017-10-05T19:53:12  <luke-jr> (yes, it seems to override -mno-*
6402017-10-05T19:53:17  <wumpus> gmaxwell: well if there's a case you can do much better than the compiler, sure...
6412017-10-05T19:53:25  *** promag has quit IRC
6422017-10-05T19:53:37  * BlueMatt has a weak preference for compile units, but only cause I'd use them in FIBRE for my FEC stuff, too, but thats not much of a reasoning
6432017-10-05T19:54:26  * luke-jr hopes we can have POWER9 asm in 0.16 <.<
6442017-10-05T19:54:32  * BlueMatt agrees
6452017-10-05T19:54:38  <cfields> gmaxwell: an implementation bug in some branch of one optimized path is scary...
6462017-10-05T19:54:56  <gmaxwell> cfields: try differently if it fails is just not reasonable in a lot of cases; and often would add a lot of complexity (now you have to not cache hashes, but instead only use hash-verify methods) ... and we don't have expected values for thigns like single transaction hashes, just hash roots.
6472017-10-05T19:54:57  <cfields> gmaxwell: in particular, the crc issue had to do with incoming data alignment on x86_64
6482017-10-05T19:55:24  <wumpus> cfields: I agree
6492017-10-05T19:55:51  <wumpus> cfields: I think we should only do asm optimization in cases where it really makes a lot of difference, for that reason, ther risk has to be worth it
6502017-10-05T19:56:01  <gmaxwell> cfields: yes, thats something that always needs careful review and we should have unit tests that also stress alignment.
6512017-10-05T19:56:33  <wumpus> special-casing everything makes things a lot harder to review, and test, especially when it starts to need different kinds of hardware
6522017-10-05T19:56:54  <wumpus> but for testable low-level primitives like SHA256 I'd say it's ok
6532017-10-05T19:57:11  <gmaxwell> good thing no one is talking about special casing everything. :)
6542017-10-05T19:57:29  <gmaxwell> yea, sha2 etc have simple testable interfaces.
6552017-10-05T19:57:37  <wumpus> no, that's just one extreme, I've seen soome graphics drivers which are scary in that regard :)
6562017-10-05T19:58:09  <gmaxwell> But benchmarks!
6572017-10-05T19:58:11  <wumpus> oh let's special case 4x4 tiles, 4x5 tiles, 4x6 tiles, ... for 3 different architectures
6582017-10-05T19:58:20  <wumpus> right :)
6592017-10-05T19:58:27  <cfields> mmm. I don't see the harm in doing a quick re-check in a few certain cases (merkle mismatch is a good example)
6602017-10-05T19:58:51  <wumpus> special-casing benchmarks is a curious form of over-learning
6612017-10-05T19:58:55  <cfields> anyway, i've made my case
6622017-10-05T19:59:01  <gmaxwell> cfields: because it requires restructing the code to not return hashes but instead only have uncachable hash_Verify methods.
6632017-10-05T19:59:06  <wumpus> cfields: re-check in what case?
6642017-10-05T19:59:24  <luke-jr> although someone did manage to screw up xpub serialisation at one point IIRC
6652017-10-05T19:59:25  <wumpus> cfields: you mean re-run the validation w/ different implementations if an  incoming block fails?
6662017-10-05T19:59:55  <wumpus> (what about false positives?)
6672017-10-05T20:00:00  <cfields> wumpus: that's a big hammer, but yes-ish
6682017-10-05T20:00:11  <gmaxwell> cfields: and for small functions like a hash a check in an innerloop will measurably lower performance. ... and you also create the opposite problem, what if the alternative function is the wrong one?
6692017-10-05T20:00:26  <gmaxwell> (I'd actually consider whole block level more reasonable)
6702017-10-05T20:00:29  <wumpus> gmaxwell: I think he means on a high level
6712017-10-05T20:01:02  <wumpus> on the inner level it's just NASA-level crazy, let's run three implementations and see which ones agree
6722017-10-05T20:01:06  <sipa> i think re-checking a block if it fails is reasonable... but why switch hash functions? it's massively more likely your CPU is fried than that the hash function implementation is wrong all along and you never noticed
6732017-10-05T20:01:13  <gmaxwell> but then the dispatch is mutable not just set once at init. :(
6742017-10-05T20:01:28  <wumpus> yeah ... I think we're overdesigning this
6752017-10-05T20:01:30  <gmaxwell> right we have a constant slow stream of complaints from users whos hosts have falsely rejected the blockchain.
6762017-10-05T20:01:37  <wumpus> just continue with what you were doing sipa :)
6772017-10-05T20:01:40  <cfields> gmaxwell: just have a generic non-dispatchable one
6782017-10-05T20:01:41  <gmaxwell> I would like to see that improved somehow.
6792017-10-05T20:01:43  <wumpus> any other topics?
6802017-10-05T20:01:46  <wumpus> oh wait, it's time
6812017-10-05T20:01:57  <wumpus> #endmeeting
6822017-10-05T20:01:57  <lightningbot> Meeting ended Thu Oct  5 20:01:57 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
6832017-10-05T20:01:57  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.html
6842017-10-05T20:01:57  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.txt
6852017-10-05T20:01:57  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.log.html
6862017-10-05T20:02:02  <sipa> #lunch
6872017-10-05T20:02:51  <meshcollider> And for anyone who doesn't know yet, hacktoberfest = free t-shirt for 4 PRs in october
6882017-10-05T20:02:56  <meshcollider> https://hacktoberfest.digitalocean.com
6892017-10-05T20:03:12  <sipa> if someone is interested: https://github.com/sipa/bitcoin/commits/201709_dsha256_64 <- 64-byte specialzed SHA256 and AVX2 code
6902017-10-05T20:03:14  <gmaxwell> sneaky way to learn developers mailing addresses.
6912017-10-05T20:03:20  <sipa> (way too WIP to PR)
6922017-10-05T20:03:28  <achow101> ooh free t-shirts
6932017-10-05T20:03:48  * sipa found the student
6942017-10-05T20:03:52  <meshcollider> gmaxwell: Heh true
6952017-10-05T20:03:54  <cfields> sipa: very cool. didn't mean to rain on your parade.
6962017-10-05T20:03:56  <luke-jr> gmaxwell: never got any spam that I can tell came from last year's
6972017-10-05T20:04:08  <luke-jr> just don't use your home address for mailing address ;)
6982017-10-05T20:04:23  <wumpus> sipa: nice
6992017-10-05T20:04:55  *** jb55 has joined #bitcoin-core-dev
7002017-10-05T20:05:08  <gmaxwell> cfields: so imagine that sha2 implementation isn't alignment safe. you get a block and miscompute one of the hashes due to alignment. ...  I don't see any way of efficiently accomplishing your 'try another function' approach that would stop the false rejection.
7012017-10-05T20:05:56  <sipa> wumpus: too bad the 64-byte specialized generic-x86 code is slower than the generic-data-size SSE4-specialized version
7022017-10-05T20:06:07  <sipa> wumpus: otherwise i'd straight up PR the 64-byte specialized code
7032017-10-05T20:06:15  <wumpus> the avx code looks very recognizable
7042017-10-05T20:06:26  <sipa> wumpus: it's pretty much search replace on SSE4 code
7052017-10-05T20:06:35  <wumpus> heh
7062017-10-05T20:06:47  <gmaxwell> it's only detectable at the hash root check at the end a long computation pipeline... when that fails do we just go back and re-deseralize the entire block with different code to compute new hashes in the CTransactions?
7072017-10-05T20:07:43  <gmaxwell> (and if we do, we magnify a DOS vector for someone sending us invalid blocks, though perhaps not enough to worry about)
7082017-10-05T20:07:58  <wumpus> that's another advantage of intrinsics, it's usually easier to review than straight up asm
7092017-10-05T20:08:06  <sipa> wumpus: absolutely
7102017-10-05T20:08:16  <wumpus> no need to keep track in your head where all the registers go
7112017-10-05T20:08:30  <cfields> gmaxwell: addmittedly that's ugly, but yes, i think that's worth considering
7122017-10-05T20:08:36  <sipa> which may be a good thing or a bad thing
7132017-10-05T20:08:47  <sipa> 1) no need to keep in your head where the registers go!
7142017-10-05T20:08:57  <sipa> 2) no way to tell the compiler to keep a certainly value in a register!
7152017-10-05T20:09:39  <wumpus> well combined with the pipelining/interleaving that optimized code needs, and the large number of registers that SIMD architectures have, that can be quite difficult
7162017-10-05T20:09:40  <gmaxwell> I wish you could assign variables to registers, and have the compiler yell at you if you tried to assign two live variables to the same register.
7172017-10-05T20:10:39  <wumpus> usually there's so many registers that it would be good to be able to tell that something is *not* worth storing in a register
7182017-10-05T20:10:49  *** goatpig has joined #bitcoin-core-dev
7192017-10-05T20:10:53  <sipa> i believe there is actually a way (in GCC) to force a particular variable into a particular register
7202017-10-05T20:11:04  <sipa> int x asm("%edx");
7212017-10-05T20:11:25  <gmaxwell> cfields: and still doesn't help with accepting something we shouldn't, which is usually a more serious issue.
7222017-10-05T20:12:32  <gmaxwell> cfields: if we had some generic infrastructure to retry a failed validation (e.g. to cope with lossy hardware) then perhaps what you're thinking about could be dropped into it.
7232017-10-05T20:12:47  <cfields> gmaxwell: i'm not saying it's something we have to do, or something that wouldn't be ugly. I'm moreso coming from a place where I was in full-out panic for a few hours because I thought newer x86_64 machines were about to start diverging...
7242017-10-05T20:12:53  <sipa> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
7252017-10-05T20:13:11  <cfields> so you're right, (good!) testing should negate those worries.
7262017-10-05T20:13:37  <gmaxwell> Well, in particular runtime tests... since they'll catch failures on the actual hardware the user has.
7272017-10-05T20:13:55  <cfields> gmaxwell: for reference: https://github.com/google/leveldb/commit/2964b803b857932ff7499d7bebb61dc5514dab7c
7282017-10-05T20:13:58  <cfields> yes
7292017-10-05T20:14:19  *** promag has joined #bitcoin-core-dev
7302017-10-05T20:15:01  *** Emcy_ has quit IRC
7312017-10-05T20:15:17  *** Emcy_ has joined #bitcoin-core-dev
7322017-10-05T20:16:00  *** promag has quit IRC
7332017-10-05T20:17:41  *** wumpus has quit IRC
7342017-10-05T20:19:22  <morcos> I also don't want to rain on anyone's parade, and this is OSS so people work on what they find interesting, but I think we shoudl be careful to think about what are priorities for the project
7352017-10-05T20:19:53  <morcos> More importantly however, we shoudl be careful about making changes that are not easily reviewable by more than 1 or 2 people unless they are really warranted
7362017-10-05T20:20:59  <morcos> I've been thinking more about this since bech32.  I think bech32 is completely awesome, I'm super happy we're doing it and imo its a good priority project. But it essentially got no review.  Did anyone other than sdaftuar review it?
7372017-10-05T20:21:45  <morcos> Sometimes things will have to be like that, but it shoudl be a tradeoff we consider carefully...  how tricky are we trying to be vs how much is it warranted
7382017-10-05T20:22:03  * BlueMatt tends to agree, though noting that a part of my agreement is my different priorities from some others - performance is maybe much more of a concern for many others in the project more than I
7392017-10-05T20:22:14  *** wumpus has joined #bitcoin-core-dev
7402017-10-05T20:22:14  <morcos> I haven't evaluated that in the context of the parallelized hashing, but its something I think we should
7412017-10-05T20:25:12  <gmaxwell> morcos: bech32 had more review then it appeared because we solicited extensive review before publishing the bip.. what didn't get review was the checksum itself other than myself and pieter, until sdaftuar.  ... but who reviewed the prior address checksum?
7422017-10-05T20:25:30  <gmaxwell> Clearly no one, because it needlessly sucked. :P
7432017-10-05T20:25:55  <sipa> well it was too late to review it by the time any of us were around
7442017-10-05T20:26:27  <luke-jr> I didn't review Bech32 because I figured it was over my head (especially magic checksum stuff).
7452017-10-05T20:26:41  <gmaxwell> There were many design changes to the earliest Bech32 proposal that arose out of review, e.g. the delimiter character.
7462017-10-05T20:27:27  <morcos> Yes and of course the 2 people that can't be blamed are you and pieter since you did the work.  But I don't want us to fall into a trap of just assuming if something is too difficult or outside of our field to review properly that someone else must be doign a good job with it
7472017-10-05T20:28:04  <morcos> gmaxwell: and yes i was referring to the checksum
7482017-10-05T20:28:15  <morcos> but thats a good point...
7492017-10-05T20:29:38  <gmaxwell> We deal with this for libsecp256k1 too, that fact that it's in a different repo is just enabling you to ignore it. :)
7502017-10-05T20:29:54  <gmaxwell> Though we do have effective review there too.
7512017-10-05T20:30:13  <gmaxwell> Though not as much as I'd like.
7522017-10-05T20:30:41  <sipa> but at least for secp256k1 it's clear what the goal is (implement secp256k1 EC correctly), so someone could review e.g. just the test and judge that they're sufficient for that goal
7532017-10-05T20:30:54  <cfields> morcos: yes, well said. I think that's why I find the asm changes scary. I spent a full day trying to learn and understand the sse42 optimized sha2, and only because it was failing some tests. At best, I agree that it looks right, but I could never say it with any degree of certainty. I deferred to "it passes all the tests".
7542017-10-05T20:31:10  <sipa> with bech32, the effort is in the design, not the implementation, and there is relatively little proof that the design actually has the properties it claims to have
7552017-10-05T20:31:28  <gmaxwell> But thats the same thing as reviewing the bech32 checksum. and fwiw, if feedback on bech32 we got was we needed more reviewers for the checksum, I would have gone and asked a libsecp256k1 contributor to do it.
7562017-10-05T20:32:07  <gmaxwell> because I know e.g. andrew (or peter dettman) aren't frightened off by math.
7572017-10-05T20:32:14  <morcos> I think my feedback is one meta level up.  There should have been more people that questioned how much reivew it got
7582017-10-05T20:32:21  <gmaxwell> yes, agreed.
7592017-10-05T20:32:38  <gmaxwell> well I was thinking that before you commented. Started thinking it as soon as I saw other bitcoin software had merged it.
7602017-10-05T20:32:51  <morcos> so i know i'm not going to review the hashing stuff, just want to make sure other peopel are going to ensure it is properly reviewed
7612017-10-05T20:33:13  <morcos> i'm frightened off by code. :)
7622017-10-05T20:34:19  <gmaxwell> had someone commented on that earlier, I would have agreed.  I reviewed the checksum work too, but pieter and I worked a lot togeather on it, and if he screwed up he probably would have tained me.
7632017-10-05T20:34:24  <sipa> the cool thing about hash functions is that they have essentially no branches... so it's very hard (though not impossible, see the alignment issue) to make it be incorrect only for a hard-to-detect small subset of inputs
7642017-10-05T20:35:19  <gmaxwell> yes on hash function correctness, but that doesn't help with BECH32 design, which I guess as your point above.  It's easy to be confident a new implementation of it is conformant... not so much that the design is good.
7652017-10-05T20:35:50  <sipa> right
7662017-10-05T20:36:24  <gmaxwell> and still no one has basically reviewed our decision to use a cyclic code -- perhaps if we were call coding theorists someone would have known an even better tool... but there is a limit to how far down a rabbit hole we can go.
7672017-10-05T20:36:31  <morcos> gmaxwell: but part of my point was the tradeoff on how important the feature is.  i am definitely +1 on bech32.  and not negative on parallel hashing just raising points for us to think about
7682017-10-05T20:36:35  <morcos> anyway, got to run
7692017-10-05T20:52:29  *** promag has joined #bitcoin-core-dev
7702017-10-05T21:04:58  *** promag has quit IRC
7712017-10-05T21:07:13  *** ybit has quit IRC
7722017-10-05T21:07:50  *** ybit has joined #bitcoin-core-dev
7732017-10-05T21:26:11  *** wraithm has joined #bitcoin-core-dev
7742017-10-05T21:26:21  *** jtimon has joined #bitcoin-core-dev
7752017-10-05T21:29:15  *** RoyceX has quit IRC
7762017-10-05T21:33:37  <BlueMatt> someone wanna close 11454?
7772017-10-05T21:52:53  *** Cheeseo has joined #bitcoin-core-dev
7782017-10-05T21:56:03  <gmaxwell> morcos: I think you can decompose your concerns into two folks-- people will work on whatever they like, but if it's going to get merged it needs to deserve the required review attention, since that isn't just an indivigual decision.
7792017-10-05T21:56:25  <gmaxwell> forks*
7802017-10-05T21:56:59  <gmaxwell> morcos: and then unrelated, we shouldn't be in a state where people don't review or even provide meta review of things which are two mathy or too low level and so they assume that they won't be of use.
7812017-10-05T21:57:15  *** Chris_Stewart_5 has joined #bitcoin-core-dev
7822017-10-05T21:57:32  <gmaxwell> And the thing to encourage there is that even if its over your head you can still ask some of the right meta questions.
7832017-10-05T22:12:57  *** Cheeseo has quit IRC
7842017-10-05T22:13:22  *** Cheeseo has joined #bitcoin-core-dev
7852017-10-05T22:14:36  *** promag has joined #bitcoin-core-dev
7862017-10-05T22:19:01  *** wraithm has quit IRC
7872017-10-05T22:29:12  *** Cheeseo has quit IRC
7882017-10-05T22:29:37  *** Cheeseo has joined #bitcoin-core-dev
7892017-10-05T22:30:34  *** echonaut1 has joined #bitcoin-core-dev
7902017-10-05T22:31:07  *** echonaut has quit IRC
7912017-10-05T22:36:32  *** Chris_Stewart_5 has quit IRC
7922017-10-05T23:08:31  *** Cheeseo has quit IRC
7932017-10-05T23:15:00  <bitcoin-git> [bitcoin] theuni opened pull request #11457: Introduce BanMan (master...move-bandb) https://github.com/bitcoin/bitcoin/pull/11457
7942017-10-05T23:20:47  *** vicenteH has quit IRC
7952017-10-05T23:29:29  <luke-jr> jnewbery: your version seems to pass the wallet by name instead of CWalletRef
7962017-10-05T23:36:08  *** stevenroose has quit IRC
7972017-10-05T23:38:21  *** stevenroose has joined #bitcoin-core-dev
7982017-10-05T23:49:27  *** Emcy_ has quit IRC
7992017-10-05T23:49:54  *** Emcy_ has joined #bitcoin-core-dev
8002017-10-05T23:58:26  *** abpa has quit IRC