12016-09-29T00:05:28  *** jnewbery has joined #bitcoin-core-dev
   22016-09-29T00:05:28  *** aureianimus has quit IRC
   32016-09-29T00:05:37  *** aureianimus has joined #bitcoin-core-dev
   42016-09-29T00:27:00  *** Ylbam has quit IRC
   52016-09-29T00:29:39  *** shesek has joined #bitcoin-core-dev
   62016-09-29T00:38:13  *** aureianimus_ has joined #bitcoin-core-dev
   72016-09-29T00:38:14  *** aureianimus has quit IRC
   82016-09-29T00:49:07  *** aureianimus has joined #bitcoin-core-dev
   92016-09-29T00:49:08  *** aureianimus_ has quit IRC
  102016-09-29T00:59:03  *** jnewbery has quit IRC
  112016-09-29T01:01:50  *** aureianimus_ has joined #bitcoin-core-dev
  122016-09-29T01:02:00  *** aureianimus has quit IRC
  132016-09-29T01:18:46  *** aureianimus has joined #bitcoin-core-dev
  142016-09-29T01:19:13  *** aureianimus_ has quit IRC
  152016-09-29T01:19:23  *** aureianimus has joined #bitcoin-core-dev
  162016-09-29T01:56:04  *** Chris_Stewart_5 has quit IRC
  172016-09-29T02:00:09  *** aureianimus has quit IRC
  182016-09-29T02:00:20  *** aureianimus has joined #bitcoin-core-dev
  192016-09-29T02:00:37  *** fengling has joined #bitcoin-core-dev
  202016-09-29T02:12:07  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  212016-09-29T02:21:27  *** Alopex has quit IRC
  222016-09-29T02:22:32  *** Alopex has joined #bitcoin-core-dev
  232016-09-29T02:29:38  *** Chris_Stewart_5 has quit IRC
  242016-09-29T02:31:22  *** aureianimus_ has joined #bitcoin-core-dev
  252016-09-29T02:31:33  *** aureianimus has quit IRC
  262016-09-29T02:32:50  *** moli has joined #bitcoin-core-dev
  272016-09-29T02:40:06  *** Alopex has quit IRC
  282016-09-29T02:41:12  *** Alopex has joined #bitcoin-core-dev
  292016-09-29T02:42:15  *** aureianimus_ has quit IRC
  302016-09-29T02:42:18  *** aureianimus has joined #bitcoin-core-dev
  312016-09-29T02:49:24  *** tunafizz has joined #bitcoin-core-dev
  322016-09-29T03:03:24  *** aureianimus has quit IRC
  332016-09-29T03:03:35  *** aureianimus has joined #bitcoin-core-dev
  342016-09-29T03:13:10  *** droark has quit IRC
  352016-09-29T03:14:54  *** aureianimus_ has joined #bitcoin-core-dev
  362016-09-29T03:15:04  *** aureianimus has quit IRC
  372016-09-29T03:26:04  *** aureianimus has joined #bitcoin-core-dev
  382016-09-29T03:26:16  *** aureianimus_ has quit IRC
  392016-09-29T03:36:50  *** aureianimus has quit IRC
  402016-09-29T03:37:02  *** aureianimus has joined #bitcoin-core-dev
  412016-09-29T03:46:00  *** cryptapus has joined #bitcoin-core-dev
  422016-09-29T03:46:34  *** Alina-malina has quit IRC
  432016-09-29T03:47:11  *** Alopex has quit IRC
  442016-09-29T03:47:12  *** cryptapus_afk has quit IRC
  452016-09-29T03:48:16  *** Alopex has joined #bitcoin-core-dev
  462016-09-29T03:49:11  *** Alina-malina has joined #bitcoin-core-dev
  472016-09-29T03:49:31  *** Bootvis has quit IRC
  482016-09-29T03:50:21  *** davec has quit IRC
  492016-09-29T03:50:39  *** davec has joined #bitcoin-core-dev
  502016-09-29T03:50:48  *** jouke has quit IRC
  512016-09-29T03:54:57  *** Bootvis has joined #bitcoin-core-dev
  522016-09-29T03:57:36  *** jouke has joined #bitcoin-core-dev
  532016-09-29T03:57:57  *** aureianimus has quit IRC
  542016-09-29T03:58:01  *** aureianimus_ has joined #bitcoin-core-dev
  552016-09-29T04:24:50  *** aureianimus_ has quit IRC
  562016-09-29T04:25:01  *** aureianimus has joined #bitcoin-core-dev
  572016-09-29T04:34:57  *** Giszmo has quit IRC
  582016-09-29T04:36:01  *** Alopex has quit IRC
  592016-09-29T04:36:12  *** aureianimus_ has joined #bitcoin-core-dev
  602016-09-29T04:36:19  *** aureianimus has quit IRC
  612016-09-29T04:37:06  *** Alopex has joined #bitcoin-core-dev
  622016-09-29T04:37:10  *** veleiro has joined #bitcoin-core-dev
  632016-09-29T04:47:18  *** aureianimus_ has quit IRC
  642016-09-29T04:47:26  *** aureianimus has joined #bitcoin-core-dev
  652016-09-29T05:08:40  *** aureianimus has quit IRC
  662016-09-29T05:08:51  *** aureianimus has joined #bitcoin-core-dev
  672016-09-29T05:16:31  *** Alopex has quit IRC
  682016-09-29T05:17:37  *** Alopex has joined #bitcoin-core-dev
  692016-09-29T05:21:37  *** aureianimus has quit IRC
  702016-09-29T05:21:48  *** aureianimus has joined #bitcoin-core-dev
  712016-09-29T05:27:33  *** fengling has quit IRC
  722016-09-29T05:32:51  *** aureianimus has quit IRC
  732016-09-29T05:32:52  *** aureianimus_ has joined #bitcoin-core-dev
  742016-09-29T05:40:50  *** Ylbam has joined #bitcoin-core-dev
  752016-09-29T05:44:27  *** aureianimus_ has quit IRC
  762016-09-29T05:44:38  *** aureianimus has joined #bitcoin-core-dev
  772016-09-29T05:58:38  <GitHub190> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/dc641415e75e...d675984fdfa4
  782016-09-29T05:58:39  <GitHub190> bitcoin/master f4dffdd Luke Dashjr: Add MIT license to Makefiles
  792016-09-29T05:58:39  <GitHub190> bitcoin/master 3b4b6dc Luke Dashjr: Add MIT license to autogen.sh and share/genbuild.sh
  802016-09-29T05:58:40  <GitHub190> bitcoin/master 3f8a5d8 Luke Dashjr: Trivial: build-aux/m4/l_atomic: Fix typo
  812016-09-29T05:58:53  <GitHub162> [bitcoin] laanwj closed pull request #8784: Copyright headers for build scripts (master...license_build) https://github.com/bitcoin/bitcoin/pull/8784
  822016-09-29T06:01:25  <wumpus> this is strange, https://github.com/bitcoin/bitcoin/pull/8832   "waxmihs2902 approved these changes 10 hours ago"   clicking on that users name gives a 404 page
  832016-09-29T06:03:20  <paveljanik> User deleted himself?
  842016-09-29T06:03:52  <wumpus> I guess
  852016-09-29T06:04:02  <wumpus> or github deleted him
  862016-09-29T06:05:22  *** aureianimus has quit IRC
  872016-09-29T06:05:31  *** aureianimus has joined #bitcoin-core-dev
  882016-09-29T06:06:58  <wumpus> as it is impossible to remove approvals, random-approval-spamming trolls wouldn't be unthinkable. Note that for normal posts, the user gets changed to 'ghost' if they disappear, so the link doesn't break.
  892016-09-29T06:08:24  <paveljanik> I think the approval stuff is half-baked...
  902016-09-29T06:09:20  <paveljanik> So maybe someone is trying DoS on us...
  912016-09-29T06:09:49  <wumpus> it's only on one issue AFAIK, so I doubt it'd be targetted on us :)
  922016-09-29T06:10:26  <paveljanik> ./DoS ... and then while :; do ./DoS; done ;-)
  932016-09-29T06:10:40  <wumpus> or they got caught *really* early; more likely such a troll/bot would just randomly approve pulls all over the site
  942016-09-29T06:11:29  <paveljanik> wumpus, search for the username on the github...
  952016-09-29T06:11:33  <paveljanik> complete github...
  962016-09-29T06:12:08  <wumpus> luckily they didn't add a message, the message added with review approval is also un-removable/un-editable IIRC
  972016-09-29T06:12:34  <wumpus> yes its half-baked :)
  982016-09-29T06:12:45  <wumpus> I'm sure they'll fix this eventually
  992016-09-29T06:13:13  <GitHub131> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d675984fdfa4...7d563cc16d64
 1002016-09-29T06:13:13  <GitHub131> bitcoin/master fa05cfd MarcoFalke: [rpc] throw JSONRPCError when utxo set can not be read
 1012016-09-29T06:13:14  <GitHub131> bitcoin/master 7d563cc Wladimir J. van der Laan: Merge #8832: [rpc] throw JSONRPCError when utxo set can not be read...
 1022016-09-29T06:13:26  <GitHub93> [bitcoin] laanwj closed pull request #8832: [rpc] throw JSONRPCError when utxo set can not be read (master...Mf1610-rpcUtxoFail) https://github.com/bitcoin/bitcoin/pull/8832
 1032016-09-29T06:20:43  *** fengling has joined #bitcoin-core-dev
 1042016-09-29T06:33:29  *** AaronvanW has quit IRC
 1052016-09-29T06:54:12  *** aureianimus has quit IRC
 1062016-09-29T06:54:21  *** aureianimus has joined #bitcoin-core-dev
 1072016-09-29T06:57:20  *** rubensayshi has joined #bitcoin-core-dev
 1082016-09-29T07:05:08  *** aureianimus has quit IRC
 1092016-09-29T07:05:16  *** aureianimus has joined #bitcoin-core-dev
 1102016-09-29T07:19:48  <GitHub3> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7d563cc16d64...489a6ab5073c
 1112016-09-29T07:19:48  <GitHub3> bitcoin/master 64047f8 Wladimir J. van der Laan: depends: Add libevent compatibility patch for windows...
 1122016-09-29T07:19:49  <GitHub3> bitcoin/master 489a6ab Wladimir J. van der Laan: Merge #8730: depends: Add libevent compatibility patch for windows...
 1132016-09-29T07:19:58  <GitHub165> [bitcoin] laanwj closed pull request #8730: depends: Add libevent compatibility patch for windows (master...2016_09_libevent_windows_gcc_531) https://github.com/bitcoin/bitcoin/pull/8730
 1142016-09-29T07:20:46  *** aureianimus has quit IRC
 1152016-09-29T07:20:52  *** aureianimus_ has joined #bitcoin-core-dev
 1162016-09-29T07:21:13  <GitHub162> [bitcoin] laanwj closed pull request #7522: Bugfix: Only use git for build info if the repository is actually the right one (master...bugfix_gitdir) https://github.com/bitcoin/bitcoin/pull/7522
 1172016-09-29T07:24:47  <wumpus> cfields_: I've assigned some build-system issues to you, hope you don't mind
 1182016-09-29T07:31:39  *** aureianimus_ has quit IRC
 1192016-09-29T07:31:51  *** aureianimus has joined #bitcoin-core-dev
 1202016-09-29T07:33:06  *** DigiByteDev has joined #bitcoin-core-dev
 1212016-09-29T07:34:56  *** Alina-malina has quit IRC
 1222016-09-29T07:34:56  *** Alina-malina has joined #bitcoin-core-dev
 1232016-09-29T07:37:43  *** DigiByteDev has quit IRC
 1242016-09-29T07:46:00  *** aureianimus has quit IRC
 1252016-09-29T07:46:12  *** aureianimus has joined #bitcoin-core-dev
 1262016-09-29T07:58:00  *** DigiByteDev has joined #bitcoin-core-dev
 1272016-09-29T07:58:15  *** ghtdak has quit IRC
 1282016-09-29T08:02:40  *** ghtdak has joined #bitcoin-core-dev
 1292016-09-29T08:34:04  *** DigiByteDev_ has joined #bitcoin-core-dev
 1302016-09-29T08:34:09  *** Guyver2 has joined #bitcoin-core-dev
 1312016-09-29T08:34:45  *** DigiByteDev has quit IRC
 1322016-09-29T08:34:48  *** DigiByteDev_ is now known as DigiByteDev
 1332016-09-29T08:49:07  *** aureianimus has quit IRC
 1342016-09-29T08:49:18  *** aureianimus has joined #bitcoin-core-dev
 1352016-09-29T08:49:42  *** DigiByteDev has quit IRC
 1362016-09-29T08:50:33  *** MarcoFalke has joined #bitcoin-core-dev
 1372016-09-29T08:50:57  <GitHub87> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/489a6ab5073c...8ca69a2a88a7
 1382016-09-29T08:50:58  <GitHub87> bitcoin/master 54e5d7c jnewbery: Add bitcoin-tx JSON tests
 1392016-09-29T08:50:58  <GitHub87> bitcoin/master 8ca69a2 MarcoFalke: Merge #8829: Add bitcoin-tx JSON tests...
 1402016-09-29T08:51:07  <GitHub21> [bitcoin] MarcoFalke closed pull request #8829: Add bitcoin-tx JSON tests (master...test-bitcoin-tx-json) https://github.com/bitcoin/bitcoin/pull/8829
 1412016-09-29T08:55:45  *** pedrobranco has joined #bitcoin-core-dev
 1422016-09-29T08:59:08  *** pedrobranco has quit IRC
 1432016-09-29T08:59:12  *** pedrobra_ has joined #bitcoin-core-dev
 1442016-09-29T09:00:41  <wumpus> weird, I'm on testnet and trying to rebroadcast a transaction using "sendrawtransaction" but it's not working, I don't see anything appear in my wireshark session
 1452016-09-29T09:01:48  <wumpus> maybe all the current nodes already know of it, otoh the "broadcast through 1 node(s)" in status doesn't increase either
 1462016-09-29T09:03:27  <sipa> could it be that your peers knew about it, but evicted it?
 1472016-09-29T09:03:49  <wumpus> but for they they'd have to request it first
 1482016-09-29T09:04:26  <MarcoFalke> What about resendwallettransactions
 1492016-09-29T09:04:28  <sipa> not necessarily from you
 1502016-09-29T09:05:03  <wumpus> MarcoFalke: this should work with sendrawtransaction
 1512016-09-29T09:05:38  <wumpus> could be that the net refactoring work changed the assumptions, will add some debugging...
 1522016-09-29T09:06:31  <sipa> i don't think so
 1532016-09-29T09:06:42  <MarcoFalke> its a noop when fHaveMempool?
 1542016-09-29T09:07:00  <sipa> no
 1552016-09-29T09:07:05  <wumpus> I don't hope so
 1562016-09-29T09:07:24  <MarcoFalke> sendrawtransaction will only put in in your mempool
 1572016-09-29T09:07:36  <sipa> and then loop over all peers
 1582016-09-29T09:07:41  <sipa> and call PushInventory
 1592016-09-29T09:07:48  <MarcoFalke> oh
 1602016-09-29T09:08:30  <sipa> but PushInventory is a noop if filterInventoryKnown for that peer already contains the tx
 1612016-09-29T09:09:08  <wumpus> but it can't be there unless the peer requested it from *us*right?
 1622016-09-29T09:09:40  <wumpus> hm or if they inved it for relay to us, I guess
 1632016-09-29T09:10:46  <wumpus> I don't have a capture of the whole session so we'll never know for sure
 1642016-09-29T09:10:48  <sipa> well you can enable -debug=net and see if any messages go out in response to sendrawtransaction
 1652016-09-29T09:10:56  <sipa> oh
 1662016-09-29T09:11:00  <sipa> you're already doing that
 1672016-09-29T09:11:14  <wumpus> no, no messages went out
 1682016-09-29T09:11:30  <wumpus> and according to the tx metadata apparently one peer ever requested the transaction
 1692016-09-29T09:11:38  <wumpus> but that was before I started logging
 1702016-09-29T09:13:33  <wumpus> it could also be that that counter is broken, of course, I don't think that functionality is tested anywhere it's UI only
 1712016-09-29T09:16:01  *** aureianimus has quit IRC
 1722016-09-29T09:16:10  *** aureianimus has joined #bitcoin-core-dev
 1732016-09-29T09:33:17  <wumpus> ok, seems the behaviour was correct. After restart the transaction is no longer in the filter, so I do sendrawtransaction again, it sends out an inv to every node.
 1742016-09-29T09:33:17  <sipa> i wonder if we should add some random memory/cpu intensive task occasionally during block validation (so it doesn't consume more than 1% cpu rate or so)
 1752016-09-29T09:33:29  <sipa> and if that fails, tell the user their hardware is too unreliable
 1762016-09-29T09:33:47  <wumpus> yes, some games have that, it's not a bad idea
 1772016-09-29T09:33:56  <wumpus> it allows distinguishing bugs from hw failures
 1782016-09-29T09:34:57  <wumpus> we have the same problem, 'support' is overflowed with issues that are probably hw failures but we can't be sure so it wastes a lot of time
 1792016-09-29T09:38:42  <paveljanik> We could ask users for the result of bitcoind -sanitychecks or something if we suspect HW issue...
 1802016-09-29T09:39:05  *** jannes has joined #bitcoin-core-dev
 1812016-09-29T09:39:13  <sipa> paveljanik: the problem is that such a sanitycheck would need to run for several hours or so to be reliable
 1822016-09-29T09:40:02  <paveljanik> sure, but 90% rule...
 1832016-09-29T09:40:46  <sipa> if a failure is detectable within minutes, it also means their block validation like fails within minutes
 1842016-09-29T09:40:51  <sipa> and many other things
 1852016-09-29T09:41:03  <sipa> most random failure i hear about happen after several hours of sync
 1862016-09-29T09:42:16  <wumpus> the sanity check definitely needs to run automatically and periodically for it to be useful
 1872016-09-29T09:42:26  <wumpus> because otherwise it won't run in the right conditinos
 1882016-09-29T09:43:19  <wumpus> we should start selling a hardware quality certificates: has synced the bitcoin block chain succesfully
 1892016-09-29T09:43:44  *** Guyver2_ has joined #bitcoin-core-dev
 1902016-09-29T09:43:55  <wumpus> if it can do that, it won't crash on games either. Well. Maybe after GPU secp256k1 verification is implemented ;)
 1912016-09-29T09:44:18  <sipa> maybe we should instead just market secp256k1 asics
 1922016-09-29T09:45:12  <wumpus> would be a very interesting project if there's a market for that
 1932016-09-29T09:46:11  <wumpus> hm, scrap that. THe market for that is key-cracking :(
 1942016-09-29T09:47:00  *** aureianimus has quit IRC
 1952016-09-29T09:47:01  *** Guyver2 has quit IRC
 1962016-09-29T09:47:05  *** aureianimus_ has joined #bitcoin-core-dev
 1972016-09-29T09:47:08  *** Guyver2_ is now known as Guyver2
 1982016-09-29T09:50:45  <wumpus> searching around a bit, apparently, many people *started* on a verilog/vhdl FPGA implementation of secp256k1, but there's no code to be found anywhere
 1992016-09-29T09:57:49  *** aureianimus_ has quit IRC
 2002016-09-29T09:58:01  *** aureianimus has joined #bitcoin-core-dev
 2012016-09-29T09:58:40  <wumpus> I'm pleasantly surprised how well the wireshark dissector for the bitcoin protocol works, can just do "tcp port 18333" then use a display filter of "bitcoin" and gets a running list of bitcoin packets easy to inspect using the tree structure
 2022016-09-29T10:01:27  <wumpus> compared to trying to mentally parse debug.log output this is a breeze
 2032016-09-29T10:02:16  <waxwing> wumpus: they've had that dissector for years, i remember being pleasantly surprised in like 2013/14
 2042016-09-29T10:02:29  <waxwing> not that i need it or anything, but it's cool :)
 2052016-09-29T10:02:49  <wumpus> yeah yeah it's probably not new
 2062016-09-29T10:03:14  <wumpus> but just in case someone didn't discover it 20 years ago yet, there you go...
 2072016-09-29T10:03:30  <waxwing> sorry wasn't trying to be a hipster :)
 2082016-09-29T10:03:44  <wumpus> :-)
 2092016-09-29T10:03:47  <sipa> damn. i believe this is a blocker for bip151.
 2102016-09-29T10:04:26  <wumpus> agree sipa
 2112016-09-29T10:08:25  <wumpus> though it is possible to have dissectors with parameters, such as a key, but it's so much less user friendly!
 2122016-09-29T10:08:48  *** aureianimus has quit IRC
 2132016-09-29T10:08:58  *** aureianimus has joined #bitcoin-core-dev
 2142016-09-29T10:12:28  <wumpus> a more practical way to go at this, post-bip151, would be add functionality to dump packets to disk after decryption on receive and before encryption on send, it's for debugging anyhow not snooping
 2152016-09-29T10:14:04  <waxwing> iirc there are per-dissector settings, for ssl you can enter the session key there etc.
 2162016-09-29T10:14:26  <wumpus> yes
 2172016-09-29T10:14:30  <waxwing> there's an env variable you can set when running firefox that dumps the keys for you, and you can import it in
 2182016-09-29T10:14:40  <sipa> implementing bip151 in wireshark will be fu
 2192016-09-29T10:14:40  <waxwing> and chrome i think
 2202016-09-29T10:14:48  <wumpus> waxwing: but if you deal with lots of different sessions, or sessions that are created on the fly
 2212016-09-29T10:15:21  <waxwing> right, which i guess might be of particular import in your case (/me hasn't read bip151 tho)
 2222016-09-29T10:15:44  <wumpus> it's certainly possible, the only remark I was making was in regard to what was the most practical way :)
 2232016-09-29T10:16:45  <wumpus> one advantage of doing the decryption in the dissector would be that it could detect crypto problems
 2242016-09-29T10:18:31  <sipa> i don't know what language wireshark uses... but reimplementing sha256, chacha20 and poly1305 doesn't sound trivial
 2252016-09-29T10:18:48  <sipa> (unless those are already available as primitives)
 2262016-09-29T10:18:49  <wumpus> C
 2272016-09-29T10:19:09  <sipa> oh, the filters too?
 2282016-09-29T10:19:20  <wumpus> it's also possible to write dissectors in lua, but that's only recommended for one-off projects, as performance is abysmal
 2292016-09-29T10:19:23  <sipa> i thought those would be in a plugin language like lua or so
 2302016-09-29T10:19:24  <wumpus> yes, most of them
 2312016-09-29T10:19:31  <sipa> ah.
 2322016-09-29T10:20:27  <sipa> the bitcoin dissector is in c currently?
 2332016-09-29T10:20:43  <wumpus> let me see
 2342016-09-29T10:21:31  <sipa> i vaguely remember seeing the code for it, years ago
 2352016-09-29T10:22:52  <wumpus> https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-bitcoin.c
 2362016-09-29T10:24:23  <GitHub156> [bitcoin] MarcoFalke opened pull request #8834: [qa] blockstore: Switch to dumb dbm (master...Mf1610-qaBlockstoreDumb) https://github.com/bitcoin/bitcoin/pull/8834
 2372016-09-29T10:24:52  <wumpus> apparently it's not yet updated for 0.12.0+, no handler for sendheaders, feefilter etc
 2382016-09-29T10:27:21  *** mmeijeri has joined #bitcoin-core-dev
 2392016-09-29T10:27:29  <mmeijeri> The same environment variable (SSLKEYLOGFILE) also works for Chrome.
 2402016-09-29T10:29:17  <wumpus> oh it's a dynamic file? that's a good idea
 2412016-09-29T10:29:19  <waxwing> mmeijeri: it's coming back to me, i have a feeling that it handles multiple sessions transparently, by just checking which key material works for which session. not that this matters, sorry for OT :)
 2422016-09-29T10:30:31  <waxwing> ah yes that was it, i think it stores the premaster secrets for all sessions, then in each handshake it tries them out until it finds what works. something like that.
 2432016-09-29T10:34:21  *** wumpus has quit IRC
 2442016-09-29T10:42:16  *** droark has joined #bitcoin-core-dev
 2452016-09-29T10:57:38  *** aureianimus has quit IRC
 2462016-09-29T10:57:47  *** aureianimus has joined #bitcoin-core-dev
 2472016-09-29T11:08:29  *** aureianimus has quit IRC
 2482016-09-29T11:08:42  *** aureianimus has joined #bitcoin-core-dev
 2492016-09-29T11:10:28  <GitHub26> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8ca69a2a88a7...cc9e8aca5f95
 2502016-09-29T11:10:28  <GitHub26> bitcoin/master a0f8482 Suhas Daftuar: [qa] Split up slow RPC calls to avoid pruning test timeouts
 2512016-09-29T11:10:29  <GitHub26> bitcoin/master cc9e8ac MarcoFalke: Merge #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts...
 2522016-09-29T11:10:48  <GitHub122> [bitcoin] MarcoFalke closed pull request #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts (master...fix-pruning-timeout) https://github.com/bitcoin/bitcoin/pull/8827
 2532016-09-29T11:15:22  *** molz has joined #bitcoin-core-dev
 2542016-09-29T11:16:06  *** moli has quit IRC
 2552016-09-29T11:19:43  <luke-jr> any reason for me to NOT switch to 64-bit after I sleep?
 2562016-09-29T11:22:21  *** aureianimus has quit IRC
 2572016-09-29T11:22:33  *** aureianimus has joined #bitcoin-core-dev
 2582016-09-29T11:22:46  <phantomcircuit> luke-jr: something about  #8828 results in a null pointer dereference
 2592016-09-29T11:22:58  <phantomcircuit> afaict i just moved code around basically
 2602016-09-29T11:23:03  <phantomcircuit> you wrote the original method
 2612016-09-29T11:23:07  <phantomcircuit> can you take a look?
 2622016-09-29T11:23:17  *** benma has joined #bitcoin-core-dev
 2632016-09-29T11:23:53  <luke-jr> phantomcircuit: too sleepy now, can you ask me after I finish switching into 64-bit (probably tomorrow)? :x
 2642016-09-29T11:24:09  <phantomcircuit> luke-jr: whoa now
 2652016-09-29T11:24:13  <phantomcircuit> you're moving to 64bit?
 2662016-09-29T11:24:23  <luke-jr> phantomcircuit: unless someone stops me before I wake up
 2672016-09-29T11:24:56  <phantomcircuit> also the line gdb thinks it's segfaulting on makes no sense
 2682016-09-29T11:25:01  <luke-jr> ?
 2692016-09-29T11:25:01  <phantomcircuit> nOrderPosNext = 0;
 2702016-09-29T11:25:03  *** cryptapus_ has joined #bitcoin-core-dev
 2712016-09-29T11:25:09  <luke-jr> so this is NULL? O.o
 2722016-09-29T11:25:46  <luke-jr> +    int64_t& nOrderPosNext = nOrderPosNext;
 2732016-09-29T11:25:48  <luke-jr> wtf is this
 2742016-09-29T11:26:13  <luke-jr> bet that's your problem, it makes no sense
 2752016-09-29T11:26:16  <luke-jr> prob undefined behaviour
 2762016-09-29T11:26:35  <luke-jr> but I'm half asleep, so who knows
 2772016-09-29T11:28:09  *** aureianimus has quit IRC
 2782016-09-29T11:28:21  *** aureianimus has joined #bitcoin-core-dev
 2792016-09-29T11:35:12  *** harrymm has left #bitcoin-core-dev
 2802016-09-29T11:40:01  *** fengling has quit IRC
 2812016-09-29T11:40:04  *** aureianimus has quit IRC
 2822016-09-29T11:40:17  *** aureianimus has joined #bitcoin-core-dev
 2832016-09-29T11:45:18  *** harrymm has joined #bitcoin-core-dev
 2842016-09-29T11:52:55  *** mol has joined #bitcoin-core-dev
 2852016-09-29T11:55:04  <luke-jr> oh well, hope that helped, otherwise ping me tomorrow I guess :x
 2862016-09-29T11:55:06  <luke-jr> night
 2872016-09-29T11:55:41  *** molz has quit IRC
 2882016-09-29T11:57:15  *** fengling has joined #bitcoin-core-dev
 2892016-09-29T12:01:36  <phantomcircuit> luke-jr: that's weird but apparently correct
 2902016-09-29T12:05:13  *** Soligor has quit IRC
 2912016-09-29T12:10:16  <midnightmagic> that's a weird scoping problem
 2922016-09-29T12:10:26  <midnightmagic> lol
 2932016-09-29T12:13:37  *** Soligor has joined #bitcoin-core-dev
 2942016-09-29T12:15:19  *** fengling has quit IRC
 2952016-09-29T12:16:29  *** Squidicc has joined #bitcoin-core-dev
 2962016-09-29T12:19:01  *** Squidicuz has quit IRC
 2972016-09-29T12:31:15  *** aureianimus has quit IRC
 2982016-09-29T12:31:23  *** aureianimus has joined #bitcoin-core-dev
 2992016-09-29T12:40:12  *** jnewbery has joined #bitcoin-core-dev
 3002016-09-29T12:50:51  *** droark has quit IRC
 3012016-09-29T13:00:43  <GitHub52> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/cc9e8aca5f95...9b94cca41f3e
 3022016-09-29T13:00:43  <GitHub52> bitcoin/master 64d9507 Pavel Janík: [WIP] Remove unused statement in serialization
 3032016-09-29T13:00:44  <GitHub52> bitcoin/master 9b94cca Wladimir J. van der Laan: Merge #8658: Remove unused statements in serialization...
 3042016-09-29T13:00:53  <GitHub41> [bitcoin] laanwj closed pull request #8658: Remove unused statements in serialization (master...20160902_nVersion_serialization_cleanup) https://github.com/bitcoin/bitcoin/pull/8658
 3052016-09-29T13:02:24  <kanzure> wumpus: tcpdump if you want to generate .pcap files without wireshark's gui
 3062016-09-29T13:05:09  *** paveljanik has quit IRC
 3072016-09-29T13:05:37  *** limpkin_ is now known as limpkin
 3082016-09-29T13:05:43  <GitHub166> [bitcoin] jgarzik closed pull request #6451: BIP 102: Increase block size limit to 2MB (master...2015_2mb_blocksize) https://github.com/bitcoin/bitcoin/pull/6451
 3092016-09-29T13:06:21  * midnightmagic stabs wireshark's gui
 3102016-09-29T13:07:26  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 3112016-09-29T13:09:13  *** veleiro has quit IRC
 3122016-09-29T13:12:19  *** aureianimus has quit IRC
 3132016-09-29T13:12:23  *** aureianimus_ has joined #bitcoin-core-dev
 3142016-09-29T13:24:06  *** aureianimus_ has quit IRC
 3152016-09-29T13:24:18  *** aureianimus has joined #bitcoin-core-dev
 3162016-09-29T13:27:46  *** benma has quit IRC
 3172016-09-29T13:33:18  *** AaronvanW has joined #bitcoin-core-dev
 3182016-09-29T13:35:34  *** To7 has joined #bitcoin-core-dev
 3192016-09-29T13:36:13  <GitHub180> [bitcoin] MarcoFalke opened pull request #8835: [qa] nulldummy.py: Don't run unused code (master...Mf1610-qaNulldummyUnused) https://github.com/bitcoin/bitcoin/pull/8835
 3202016-09-29T13:45:11  *** aureianimus has quit IRC
 3212016-09-29T13:45:12  *** aureianimus_ has joined #bitcoin-core-dev
 3222016-09-29T13:51:23  *** Chris_Stewart_5 has quit IRC
 3232016-09-29T14:01:04  *** aureianimus_ has quit IRC
 3242016-09-29T14:01:15  *** aureianimus has joined #bitcoin-core-dev
 3252016-09-29T14:02:29  *** pedrobra_ has quit IRC
 3262016-09-29T14:05:09  *** MarcoFalke has left #bitcoin-core-dev
 3272016-09-29T14:22:14  *** aureianimus_ has joined #bitcoin-core-dev
 3282016-09-29T14:22:24  <GitHub8> [bitcoin] jnewbery opened pull request #8836: bitcoin-util-test.py should fail if the output file is empty (master...bitcoin-tx-no-empty-outputs) https://github.com/bitcoin/bitcoin/pull/8836
 3292016-09-29T14:22:25  *** aureianimus has quit IRC
 3302016-09-29T14:25:26  *** rubensayshi has quit IRC
 3312016-09-29T14:26:55  *** wumpus has joined #bitcoin-core-dev
 3322016-09-29T14:27:04  <GitHub100> [bitcoin] jnewbery opened pull request #8837: allow bitcoin-tx to parse partial transactions (master...bitcoin-tx-partial-transactions) https://github.com/bitcoin/bitcoin/pull/8837
 3332016-09-29T14:31:12  *** Giszmo has joined #bitcoin-core-dev
 3342016-09-29T14:34:23  <GitHub16> [bitcoin] jnewbery opened pull request #8838: Only log block size if block size is being accounted (master...dont_log_size) https://github.com/bitcoin/bitcoin/pull/8838
 3352016-09-29T14:35:39  *** mmeijeri has quit IRC
 3362016-09-29T14:41:02  *** aalex_ has quit IRC
 3372016-09-29T14:41:03  *** paveljanik has joined #bitcoin-core-dev
 3382016-09-29T14:46:14  <GitHub177> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9b94cca41f3e...2dd57e4f9f58
 3392016-09-29T14:46:15  <GitHub177> bitcoin/master fa156c6 MarcoFalke: [qa] nulldummy: Don't run unused code
 3402016-09-29T14:46:15  <GitHub177> bitcoin/master 2dd57e4 Wladimir J. van der Laan: Merge #8835: [qa] nulldummy.py: Don't run unused code...
 3412016-09-29T14:46:29  <GitHub194> [bitcoin] laanwj closed pull request #8835: [qa] nulldummy.py: Don't run unused code (master...Mf1610-qaNulldummyUnused) https://github.com/bitcoin/bitcoin/pull/8835
 3422016-09-29T15:03:16  *** aureianimus has joined #bitcoin-core-dev
 3432016-09-29T15:03:29  *** aureianimus_ has quit IRC
 3442016-09-29T15:04:33  <GitHub44> [bitcoin] laanwj opened pull request #8839: test: Avoid ConnectionResetErrors during RPC tests (master...2016_09_freebsd_rpctest_fix) https://github.com/bitcoin/bitcoin/pull/8839
 3452016-09-29T15:05:56  *** aalex has joined #bitcoin-core-dev
 3462016-09-29T15:07:38  *** laurentmt has joined #bitcoin-core-dev
 3472016-09-29T15:08:53  <GitHub42> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2dd57e4f9f58...c84181665f34
 3482016-09-29T15:08:54  <GitHub42> bitcoin/master 16f8823 fanquake: [depends] Boost 1.61.0
 3492016-09-29T15:08:54  <GitHub42> bitcoin/master c841816 Wladimir J. van der Laan: Merge #8819: [depends] Boost 1.61.0...
 3502016-09-29T15:09:08  <GitHub98> [bitcoin] laanwj closed pull request #8819: [depends] Boost 1.61.0 (master...depends-boost-1-61-0) https://github.com/bitcoin/bitcoin/pull/8819
 3512016-09-29T15:10:26  *** laurentmt has quit IRC
 3522016-09-29T15:18:31  *** moli has joined #bitcoin-core-dev
 3532016-09-29T15:21:57  *** mol has quit IRC
 3542016-09-29T15:23:48  <GitHub53> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c84181665f34...c9d7b0de2fc9
 3552016-09-29T15:23:48  <GitHub53> bitcoin/master fa9cd25 MarcoFalke: [qa] blockstore: Switch to dumb dbm
 3562016-09-29T15:23:49  <GitHub53> bitcoin/master c9d7b0d Wladimir J. van der Laan: Merge #8834: [qa] blockstore: Switch to dumb dbm...
 3572016-09-29T15:23:58  <GitHub36> [bitcoin] laanwj closed pull request #8834: [qa] blockstore: Switch to dumb dbm (master...Mf1610-qaBlockstoreDumb) https://github.com/bitcoin/bitcoin/pull/8834
 3582016-09-29T15:27:49  <GitHub17> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c9d7b0de2fc9...f560d9564f74
 3592016-09-29T15:27:49  <GitHub17> bitcoin/master 7e5fd71 Pavel Janík: Do not include env_win.cc on non-Windows systems
 3602016-09-29T15:27:49  <GitHub17> bitcoin/master f560d95 Wladimir J. van der Laan: Merge #8826: Do not include env_win.cc on non-Windows systems...
 3612016-09-29T15:27:59  <GitHub75> [bitcoin] laanwj closed pull request #8826: Do not include env_win.cc on non-Windows systems (master...20160928_leveldb_no_win) https://github.com/bitcoin/bitcoin/pull/8826
 3622016-09-29T15:44:16  *** aureianimus has quit IRC
 3632016-09-29T15:44:27  *** aureianimus has joined #bitcoin-core-dev
 3642016-09-29T15:51:57  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 3652016-09-29T15:55:00  *** cryptapus__ has joined #bitcoin-core-dev
 3662016-09-29T15:58:01  *** laurentmt has joined #bitcoin-core-dev
 3672016-09-29T15:58:41  *** cryptapus_ has quit IRC
 3682016-09-29T16:03:49  *** aureianimus has quit IRC
 3692016-09-29T16:04:01  *** aureianimus has joined #bitcoin-core-dev
 3702016-09-29T16:05:24  <GitHub51> [bitcoin] laanwj opened pull request #8840: test: Explicitly set encoding to utf8 when opening text files (master...2016_09_textfiles_locale) https://github.com/bitcoin/bitcoin/pull/8840
 3712016-09-29T16:08:03  *** laurentmt has quit IRC
 3722016-09-29T16:23:13  <GitHub57> [bitcoin] jl2012 opened pull request #8841: [qa] fix nulldummy test (master...nulldummytest) https://github.com/bitcoin/bitcoin/pull/8841
 3732016-09-29T16:32:53  *** aalex has quit IRC
 3742016-09-29T16:33:46  *** aalex has joined #bitcoin-core-dev
 3752016-09-29T16:35:24  *** aureianimus has quit IRC
 3762016-09-29T16:35:28  *** Giszmo has quit IRC
 3772016-09-29T16:35:35  *** aureianimus has joined #bitcoin-core-dev
 3782016-09-29T16:46:25  *** jnewbery has quit IRC
 3792016-09-29T16:47:05  *** jnewbery has joined #bitcoin-core-dev
 3802016-09-29T16:51:53  *** jnewbery has quit IRC
 3812016-09-29T16:57:32  *** molz has joined #bitcoin-core-dev
 3822016-09-29T16:57:39  *** moli has quit IRC
 3832016-09-29T17:03:49  *** jnewbery has joined #bitcoin-core-dev
 3842016-09-29T17:16:21  <cfields_> wumpus: no problem
 3852016-09-29T17:17:04  <cfields_> jonasschnelli: i had a look at your circular dep issue. No luck here. The (hack) solution is to use grouping libs, but I'd really rather not go down that road. I've started untangling dependencies instead.
 3862016-09-29T17:20:18  *** Arnavion has quit IRC
 3872016-09-29T17:23:27  *** Arnavion has joined #bitcoin-core-dev
 3882016-09-29T17:25:56  <wumpus> which dependencies are the problem there? the circular dependency between libbitcoin_server and libbitcoin_wallet?
 3892016-09-29T17:26:45  <wumpus> that would "just" require solving https://github.com/bitcoin/bitcoin/issues/7965
 3902016-09-29T17:26:52  <arubi> jnewbery, :)  I was just about to whine about 'if (!DecodeHexTx(txDecodeTmp, strHexTx)'
 3912016-09-29T17:26:57  <arubi> thanks!
 3922016-09-29T17:36:22  *** aureianimus has quit IRC
 3932016-09-29T17:36:31  *** aureianimus has joined #bitcoin-core-dev
 3942016-09-29T17:38:07  *** mrkent has joined #bitcoin-core-dev
 3952016-09-29T17:38:10  *** jnewbery has quit IRC
 3962016-09-29T17:47:57  *** cryptapus__ has quit IRC
 3972016-09-29T17:48:11  *** cryptapus__ has joined #bitcoin-core-dev
 3982016-09-29T17:48:11  *** cryptapus__ has joined #bitcoin-core-dev
 3992016-09-29T17:48:17  *** cryptapus__ is now known as cryptapus_
 4002016-09-29T17:54:08  *** laurentmt has joined #bitcoin-core-dev
 4012016-09-29T17:54:26  *** laurentmt has quit IRC
 4022016-09-29T17:57:32  <arubi> too bad, I wanted to ask him about it here
 4032016-09-29T17:57:35  *** jtimon has joined #bitcoin-core-dev
 4042016-09-29T18:01:21  *** MarcoFalke has joined #bitcoin-core-dev
 4052016-09-29T18:12:17  *** aureianimus has quit IRC
 4062016-09-29T18:12:19  *** aureianimus_ has joined #bitcoin-core-dev
 4072016-09-29T18:20:36  <jonasschnelli> wumpus, cfields_: Yes. I think we need to complete 7965 first
 4082016-09-29T18:23:11  *** aureianimus_ has quit IRC
 4092016-09-29T18:23:22  *** aureianimus has joined #bitcoin-core-dev
 4102016-09-29T18:32:20  *** gabridome has joined #bitcoin-core-dev
 4112016-09-29T18:36:43  *** Squidicc is now known as squidicuz
 4122016-09-29T18:37:36  <jl2012> Travis test failed for #8841 with an unrelated test
 4132016-09-29T18:37:45  <jl2012> https://www.irccloud.com/pastebin/pWEgvccJ/
 4142016-09-29T18:38:14  <jl2012> https://travis-ci.org/bitcoin/bitcoin/jobs/163788152
 4152016-09-29T18:39:24  *** jnewbery has joined #bitcoin-core-dev
 4162016-09-29T18:40:22  <luke-jr> phantomcircuit: correct for what?
 4172016-09-29T18:40:38  <MarcoFalke> jl2012: We should probably create an issue for those failures
 4182016-09-29T18:43:18  *** Soligor_ has joined #bitcoin-core-dev
 4192016-09-29T18:48:11  *** lightningbot has joined #bitcoin-core-dev
 4202016-09-29T19:00:04  *** BakSAj has joined #bitcoin-core-dev
 4212016-09-29T19:00:15  <jonasschnelli> bingbong
 4222016-09-29T19:00:35  <luke-jr> hi
 4232016-09-29T19:00:41  <sipa> ovatobat
 4242016-09-29T19:00:45  <CodeShark> yo
 4252016-09-29T19:01:09  *** cdecker has quit IRC
 4262016-09-29T19:01:19  <wumpus> #startmeeting
 4272016-09-29T19:01:19  <lightningbot> Meeting started Thu Sep 29 19:01:19 2016 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
 4282016-09-29T19:01:19  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
 4292016-09-29T19:01:26  <BakSAj> hi
 4302016-09-29T19:01:36  <CodeShark> meeting time!
 4312016-09-29T19:01:46  <MarcoFalke> meeting!
 4322016-09-29T19:01:56  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
 4332016-09-29T19:02:04  <MarcoFalke> (oh already started)
 4342016-09-29T19:02:07  <btcdrak> here
 4352016-09-29T19:02:07  <cfields_> hi
 4362016-09-29T19:02:12  <kanzure> hi
 4372016-09-29T19:02:20  <wumpus> any proposed topics?
 4382016-09-29T19:02:24  <jonasschnelli> topic proposal: pruning and blockrelay
 4392016-09-29T19:02:56  <petertodd> hi
 4402016-09-29T19:03:11  <sipa> policy against uncompressed keys or not
 4412016-09-29T19:03:11  *** Chris_Stewart_5 has quit IRC
 4422016-09-29T19:03:11  <wumpus> #topic pruning and blockrelay
 4432016-09-29T19:03:38  <jonasschnelli> I think we should add a service flag for block relay with a min-height
 4442016-09-29T19:03:52  <jonasschnelli> NODE_PRUNENETWORK or something
 4452016-09-29T19:03:57  <sipa> there have been multiple ideas around that
 4462016-09-29T19:04:35  <petertodd> IMO whatever we do, we should recognise that w/ segwit's larger blocks we can expect a lot of full nodes to run out of disk space quite soon
 4472016-09-29T19:04:35  <sipa> the easiest is just to add a flag that says you relay valid blocks and transactions, but not historical blocks more than a few deep
 4482016-09-29T19:04:35  <jonasschnelli> I guess people slowly start to prune the blockchain to a max of 80GB or similar... but I guess not everyone is aware of the fact that you don't relay then
 4492016-09-29T19:04:36  <wumpus> it would be nice to support more than one range, e.g. also archive nodes that host part of the old blocks
 4502016-09-29T19:04:49  <sipa> it becomes harder when you want multiple ranger
 4512016-09-29T19:04:57  <petertodd> do we have a reason for more than one range?
 4522016-09-29T19:05:00  <jonasschnelli> We could introduce another message type... blockrange  or so
 4532016-09-29T19:05:01  <sipa> it becomes even harder when you want to support sharding in an efficient way
 4542016-09-29T19:05:08  <wumpus> I'm not sure why itb ecomes hard, just add a query message that returns what ranges are supported
 4552016-09-29T19:05:11  <petertodd> sipa: what do you mean by sharding exactly?
 4562016-09-29T19:05:25  <sipa> petertodd: you'd configure your node to maintain a certain % of blocks
 4572016-09-29T19:05:45  <jonasschnelli> wumpus: query, yes, why not, or just inform like we do with sendheaders
 4582016-09-29T19:05:47  <petertodd> sipa: see, given that the bitcoin protocol can't be safely sharded right now, I think we can safely say that we don't need to support sharding in block relay yet
 4592016-09-29T19:05:50  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 4602016-09-29T19:05:55  <petertodd> sipa: doing so might even be dangerous if people start using it
 4612016-09-29T19:05:59  *** cdecker has joined #bitcoin-core-dev
 4622016-09-29T19:06:04  <sipa> petertodd: not in block relay
 4632016-09-29T19:06:08  <sipa> petertodd: for block archival
 4642016-09-29T19:06:16  <petertodd> sipa: but why shard vs. keep ranges?
 4652016-09-29T19:06:23  <petertodd> sipa: (ranges of full blocks)
 4662016-09-29T19:06:25  <luke-jr> BitTorrent already does this. Surely we can learn from that?
 4672016-09-29T19:06:38  <petertodd> luke-jr: I don't think so - bittorrent is a very different problem than bitcoin
 4682016-09-29T19:06:38  <wumpus> this is for letting other peers know what ranges of blocks are hosted, I don't think this should affect releay
 4692016-09-29T19:06:41  <sipa> so, i've been running statistics on what block depths are being requested from nodes
 4702016-09-29T19:06:47  <luke-jr> petertodd: learn from it, not use it directly
 4712016-09-29T19:07:01  <luke-jr> petertodd: BitTorrent's problem isn't very different from IBD
 4722016-09-29T19:07:08  <jonasschnelli> sipa: interesting.. do you have the stats public available somewhere
 4732016-09-29T19:07:16  <jonasschnelli> I wanted to do this a long time
 4742016-09-29T19:07:20  <petertodd> luke-jr: so, the thing is bittorrent has the problem of a diverse set of files, we just don't have that problem and can optimise differently because everyone needs access t othe same set of data
 4752016-09-29T19:08:03  <sipa> there are something like 4 meaningful 'ranges' 1) the top 2 blocks (just relay) 2) up to ~2500 blocks deep... requested very often 3) up to ~10000 deep... requested a few times more than the next range 4) the rest
 4762016-09-29T19:08:05  *** veleiro has joined #bitcoin-core-dev
 4772016-09-29T19:08:15  <wumpus> otoh bittorrent has a fixed block size :)
 4782016-09-29T19:08:22  <sipa> wumpus: so do we *ducks*
 4792016-09-29T19:08:38  *** jnewbery has joined #bitcoin-core-dev
 4802016-09-29T19:08:38  <petertodd> sipa: that probably corresponds to how long people leave their nodes offline :)
 4812016-09-29T19:08:51  <btcdrak> inb4 Bittorrent XT
 4822016-09-29T19:08:51  *** dgenr8 has joined #bitcoin-core-dev
 4832016-09-29T19:09:04  <sipa> jonasschnelli: they're not available, and the ranges i gave above are just from me quickly glancing over the result
 4842016-09-29T19:09:04  <petertodd> btcdrak: I use Bittorrent Unlimited myself
 4852016-09-29T19:09:05  <jonasschnelli> What about fingerprinting issued in conjunction with available ranges?
 4862016-09-29T19:09:16  <jonasschnelli> *issues
 4872016-09-29T19:09:20  <petertodd> jonasschnelli: make them powers of two?
 4882016-09-29T19:09:33  <sipa> well 4 ranges can be done with 2 service bit flags
 4892016-09-29T19:09:45  <sipa> gmaxwell: you've worked on these ideas before, comments?
 4902016-09-29T19:09:47  <jonasschnelli> But would that work with the flexible pruning option based on MB?
 4912016-09-29T19:10:05  <petertodd> jonasschnelli: sure, just find the biggest range less than the pruning amount
 4922016-09-29T19:10:12  <sipa> jonasschnelli: you'd change your service bits on the fly
 4932016-09-29T19:10:20  <wumpus> why would the ranges need to be in the flags?
 4942016-09-29T19:10:30  <gmaxwell> sorry, I missed that the meeting started.
 4952016-09-29T19:10:32  <jonasschnelli> Yes. Why? Better add an explicit message for the range
 4962016-09-29T19:10:34  <sipa> how would you otherwise discover what nodes to connect to?
 4972016-09-29T19:10:45  <sipa> just randomly try?
 4982016-09-29T19:10:45  <petertodd> jonasschnelli: oh right, you mean if MB != blocks... sorry.
 4992016-09-29T19:10:49  <wumpus> I think you'll need a service flag to show support for the protocol, but not what ranges you have
 5002016-09-29T19:10:52  <jonasschnelli> query or inform the other node if proto-ver > NODE_PRUNENETWORK
 5012016-09-29T19:11:00  <wumpus> well that can be negotiated later, like bittorrent does I guess
 5022016-09-29T19:11:08  <sipa> wumpus: well you do want addr messages to contain this information
 5032016-09-29T19:11:18  <wumpus> I doubt bitcoin has 'service flags' in its tracker what blocks nodes have
 5042016-09-29T19:11:18  <petertodd> sipa: so the nice thing about bitcoin, is just randomly try will probably work fairly often due to the low number of ranges out there
 5052016-09-29T19:11:26  <wumpus> as that changes all the time anyhow
 5062016-09-29T19:11:29  <gmaxwell> I was strongly of the view that we needed to signal at least two ranges. Sipa's latest measurements make me think at least three are needed.
 5072016-09-29T19:11:31  <wumpus> s/bitcoin/bittorrent/
 5082016-09-29T19:11:37  <jonasschnelli> I think informing other nodes ranges over addr is another thing...
 5092016-09-29T19:11:46  <jonasschnelli> A first step would be a information after connect
 5102016-09-29T19:11:55  <wumpus> yes, addr is another thing
 5112016-09-29T19:12:22  <gmaxwell> I think ranges in service bits are no big deal, the harder question is what to do about the history. having nodes with 150GB of history in order to serve the last range is not very viable.
 5122016-09-29T19:12:23  <wumpus> could be done later if an efficient way is needed to *locate* peers with certain ranges
 5132016-09-29T19:12:31  <wumpus> but that seems premature optimization
 5142016-09-29T19:12:36  <gmaxwell> We will need to redo addr sometime relatively soon in any case, as our messages are not compatible with HS-NG.
 5152016-09-29T19:12:52  <petertodd> gmaxwell: oh, you mean Tor's new hidden services standard right?
 5162016-09-29T19:12:56  <gmaxwell> petertodd: yes.
 5172016-09-29T19:13:02  <gmaxwell> (also I2P though thats not new)
 5182016-09-29T19:13:18  <wumpus> I think the number of ranges should be variable
 5192016-09-29T19:13:24  <wumpus> redesigning addr is a different topic
 5202016-09-29T19:13:35  <wumpus> also necessary, but again, doesn't need to be on one heap
 5212016-09-29T19:13:45  <gmaxwell> wumpus: when I'm saying ranges I am specifically referring to the top-N zomes.
 5222016-09-29T19:14:14  <petertodd> well, so if we add service bits for recent history ranges, that should be possible to implement as a separate feature to archival history ranges, and it'd be a big first step
 5232016-09-29T19:14:22  <wumpus> I think it should be possible to, say, only host the first 20GB of blocks
 5242016-09-29T19:14:37  <jonasschnelli> historic only nodes
 5252016-09-29T19:14:37  <wumpus> I don't see why it should be restricted to only recent history
 5262016-09-29T19:14:38  <petertodd> I don't think it's likely we'll see the two different features collide, so maybe implement recent history ranges first
 5272016-09-29T19:14:55  <wumpus> or I mean first 20GB + last 144 blocks
 5282016-09-29T19:15:07  <gmaxwell> For history storage, I was previously working on a proposal where nodes could signal a small (32 bit) seed and a size and from that everyone would know what parts of the history they would store.  I was so far unable to unify two different schemes, one which was computationally efficient to figure out who had what, and one which never required a peer to fetch a block it had previously deleted.
 5292016-09-29T19:15:15  <sipa> so very quick breakdown: out of 7M requested blocks, 100k were for the tip, range 2-2500 has around 200-2000 requests per block, and from 10000 to genesis deep there are around 20 per block
 5302016-09-29T19:16:19  <gmaxwell> I think for now we should not worry about the old history part and only worry about Top-n vs everything, as that fits into the pruning we already have and can be accomplished purely with service bits.
 5312016-09-29T19:16:22  <wumpus> the bittorrent problem is different in that there the goal of each node is to have everything
 5322016-09-29T19:16:25  <petertodd> so a social consideration here, is we can think in terms of recent history as "if there's a flaw, how much would we ever reorg w/o just saying bitcoin has failed?"
 5332016-09-29T19:16:44  <gmaxwell> petertodd: thats partly why we have the 288 block maximum amount of pruning.
 5342016-09-29T19:17:13  <petertodd> gmaxwell: indeed, and that's only two days...
 5352016-09-29T19:17:36  <jonasschnelli> Using multiple service bits for 4 ranges seems to be a hackish-design IMO
 5362016-09-29T19:17:49  <gmaxwell> at 100 blocks any reorg will _necessarily_ cause unrecoverable losses. So 288 basically gives a day plus an extra day for overhead.
 5372016-09-29T19:17:50  <petertodd> there's also a natural time criteria from how the difficulty adjustments reduce your resistance to 51% attack - if your node is offline longer, the minimum attacker size to fool you goes down
 5382016-09-29T19:17:52  <sipa> strangely enough: i see much more requests around 1000 deep than around 100 deep
 5392016-09-29T19:18:04  <gmaxwell> jonasschnelli: I don't see anything hackish.
 5402016-09-29T19:18:05  <wumpus> jonasschnelli: I also think it's a strange use of service bits
 5412016-09-29T19:18:07  <jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
 5422016-09-29T19:18:24  <petertodd> sipa: probably because people don't turn their nodes on and off every day
 5432016-09-29T19:18:25  <gmaxwell> sipa: you probably want to filter out the bitnodes spider, as I believe it requests a block to check the node is working.
 5442016-09-29T19:18:35  <sipa> gmaxwell: ah.
 5452016-09-29T19:18:53  <gmaxwell> petertodd: someone who hasn't turned their node on will request all of 0 to -1000. so it will not make 1000 greater.
 5462016-09-29T19:19:07  <gmaxwell> jonasschnelli: NAK.
 5472016-09-29T19:19:07  <petertodd> gmaxwell: oh! I didn't know we did that
 5482016-09-29T19:19:09  <sipa> i'm a bit surprised people think there is no need to have the available block ranges indicated in addr messages
 5492016-09-29T19:19:30  <sipa> (whether through service bits, or some extension)
 5502016-09-29T19:19:30  <jonasschnelli> I think there is a need... but it could be a second step
 5512016-09-29T19:19:38  <wumpus> jonasschnelli: appending to version should be unnecessary, that's also a hack :)
 5522016-09-29T19:19:50  <sipa> jonasschnelli: if it's a second step, we need to extend addr, and the whole management of addresses
 5532016-09-29T19:19:51  <jonasschnelli> Okay. Agree. What about a new message type?
 5542016-09-29T19:19:55  <jonasschnelli> blockrange
 5552016-09-29T19:19:58  <sipa> jonasschnelli: you don't understand.
 5562016-09-29T19:20:18  <gmaxwell> jonasschnelli: look at pieter's request figures, if nodes are effectively forced to go to peers that have everything whenever they connect becuase if they don't know they'll be able to fetch any blocks at all, then it will put lots more load on them.. causing people to stop offering blocks... causing more pressure on what remains.
 5572016-09-29T19:20:56  <sipa> jonasschnelli: the point of having it in service bits is so nodes can find peers that have the range they need
 5582016-09-29T19:20:58  <wumpus> but addr information gets old really fast
 5592016-09-29T19:21:17  <sipa> wumpus: much less so with feeler connections
 5602016-09-29T19:21:26  <wumpus> nodes may dynamically change what blocks they have, so there will always be cases of nodes connecting and realizing they have nothing to offer each other
 5612016-09-29T19:21:27  <jonasschnelli> Okay. I see the point.
 5622016-09-29T19:21:29  <sipa> (presumably, i don't have numbers)
 5632016-09-29T19:21:54  <wumpus> just like currently nodes will try to connect into black holes that no longer host a node
 5642016-09-29T19:22:24  <petertodd> so another interesting thing here is that ranges are queried linearly - you download blocks in a roughly linear fashion - so we could take advantage of that by making sure that nodes with one range keep track of nodes with adjacent ranges
 5652016-09-29T19:22:29  <wumpus> sipa: sure, feeler connections make it somewhat better
 5662016-09-29T19:22:35  <gmaxwell> wumpus: yes, sometimes the data is wrong. But there is a big difference between having 80% of the nodes on the network giving you no idea if they'll be useful at all until after you connect, vs a suggestion that might sometimes be wrong.
 5672016-09-29T19:22:40  *** Giszmo has joined #bitcoin-core-dev
 5682016-09-29T19:22:41  <wumpus> but I don't think addr is a very up-to-date information source
 5692016-09-29T19:22:46  <petertodd> thus, as you sync the first time, ask nodes with the range you're syncing at this moment for the next range you need
 5702016-09-29T19:23:08  <luke-jr> wumpus: if ranges are deterministic, they don't need to be up to date
 5712016-09-29T19:23:15  <sipa> petertodd: yes, any sharding plan wouldn't randomly distribute the kept blocks, but keep randomly distributed ranges
 5722016-09-29T19:23:32  <gmaxwell> wumpus: I don't know if you realize that sipa and I are not thinking in terms of absolute ranges here. but nodes saying "I keep the last 288" or "I keep the last 2016" or "I have all of history".
 5732016-09-29T19:23:38  <wumpus> gmaxwell: but indeed this is a different problem from the bittorrent problem where everyone's goal is to have everything
 5742016-09-29T19:23:56  <sipa> gmaxwell: well that's sharding... maybe that is something to postpone for later
 5752016-09-29T19:24:01  <petertodd> sipa: sure, I'm more talking about how the linearity affects the network p2p design - prefentially peering with peers with the adjacent range may even be a reasonable design
 5762016-09-29T19:24:03  <luke-jr> wumpus: eh, everyone needs to get everything
 5772016-09-29T19:24:03  <wumpus> there, nodes can just connect randomly and have a high change the other nodes has something to offer them
 5782016-09-29T19:24:04  <gmaxwell> wumpus: and I wouldn't expect that data to go out of date fast.. pretty much only when nodes go up and down.
 5792016-09-29T19:24:04  <sipa> oh, nvm, i'm misreading
 5802016-09-29T19:24:20  <wumpus> luke-jr: only initially
 5812016-09-29T19:24:28  <luke-jr> oh, I see the distinction
 5822016-09-29T19:24:41  <wumpus> luke-jr: bittorrent nodes don't throw away blocks, generally
 5832016-09-29T19:24:53  <luke-jr> f(best-height, seed-in-addr) -> ranges
 5842016-09-29T19:25:12  <gmaxwell> for the spreading the history around, as mentioned I came up with concrete schemes (based on consistent hashes) that have nice properties.
 5852016-09-29T19:25:30  <sipa> i wonder whether we need to have that in the first go at this
 5862016-09-29T19:26:04  <jonasschnelli> I think a first simple solution that allow to extend it further would be appriciated.
 5872016-09-29T19:26:10  <sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
 5882016-09-29T19:26:15  <wumpus> making the ranges deterministic makes some sense, on the other hand, it does restrict the flexibilty of nodes to choose what ranges they host, it means everything has to be got right in first try
 5892016-09-29T19:26:23  <gmaxwell> sipa: thats what I am saying.
 5902016-09-29T19:26:27  <jonasschnelli> sipa: agree
 5912016-09-29T19:26:30  <gmaxwell> I do not think we can do better immediately anyways.
 5922016-09-29T19:26:45  <sipa> 21:18:07 < jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
 5932016-09-29T19:26:47  <gmaxwell> sipa: though your latest figures suggest that the 2016 depth is important too.
 5942016-09-29T19:26:48  <sipa> 21:19:07 < gmaxwell> jonasschnelli: NAK.
 5952016-09-29T19:27:10  <petertodd> if nodes attempt to maintain a few connections to peers that have the next range after they have, maybe it doesn't matter exactly what the ranges actually are? any given node would have a few connections to the next range, and anyone syncing from them could ask for those connections
 5962016-09-29T19:27:15  <gmaxwell> sipa: my understanding of jonasschnelli comment was there should be a bit that says "I relay blocks but don't have history" I am NAK on that.
 5972016-09-29T19:27:36  <wumpus> as there is no scope for later optimization, because all nodes have to agree what ranges are implied
 5982016-09-29T19:27:40  <jonasschnelli> We could add a service bit that says "I relay only the last 288 blocks"
 5992016-09-29T19:27:50  <wumpus> jannes: yes that would be the initial idea
 6002016-09-29T19:27:54  <wumpus> jonasschnelli*
 6012016-09-29T19:28:08  <sipa> gmaxwell: how is that different from what i suggested?
 6022016-09-29T19:28:11  <sipa> 21:26:10 < sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
 6032016-09-29T19:28:21  <jonasschnelli> I think my initial idea with the general pruning sevice bit and a new message type is to complex and inflexible
 6042016-09-29T19:28:28  <gmaxwell> jonasschnelli: yes, that would be better, though pieter's data suggests that there are a LOT of requests at 1000. I think if I had that data I would have been suggesting the maximum pruning should be 2016, and then had the bit at that dep.
 6052016-09-29T19:28:50  <gmaxwell> sipa: the ability to relay blocks at depth -10.
 6062016-09-29T19:29:04  <sipa> gmaxwell: less than 2% of blocks requested from my node are at the tip
 6072016-09-29T19:29:22  <sipa> (but the tip is still 100x more frequent than any other individual depth)
 6082016-09-29T19:29:50  <sipa> gmaxwell: "a service bit to indicate pruned blockchain" implies you can serve 288 deep :)
 6092016-09-29T19:30:08  <petertodd> gmaxwell: re: maximum pruning depth, it's reasonable for that to be a similar % of the total data that storing the UTXO set takes - if you have 10GB of UTXO, 2GB of block data isn't a big change
 6102016-09-29T19:30:11  <wumpus> yes, you could define it as that
 6112016-09-29T19:30:15  <gmaxwell> I don't think there is any remaining disagreement on using bit(s) to signal I have a top-n.  But I have some doubt on N. it needs to capture the largest amount of the block realy bandwidth without being unduely pruning incompatible.
 6122016-09-29T19:30:37  <wumpus> 288 is the minimum pruning amount in bitcoin core already so it'd be a valid choice
 6132016-09-29T19:30:44  <morcos> as a first pass, i wonder if you preferentially downloaded from pruned peers whenever you were behind by less than 288 blocks, that would take enough load of peers serving full history?
 6142016-09-29T19:30:50  <gmaxwell> morcos: absolutely.
 6152016-09-29T19:30:57  *** aureianimus has quit IRC
 6162016-09-29T19:30:57  *** aureianimus_ has joined #bitcoin-core-dev
 6172016-09-29T19:31:01  <jonasschnelli> Good idea
 6182016-09-29T19:31:09  <wumpus> yes, that would make sense
 6192016-09-29T19:31:23  <gmaxwell> unfortunately, sipa's data suggests that 288 sheds less traffic than measurements years ago suggested.
 6202016-09-29T19:31:43  <sipa> maybe i should compute statistics in bytes rather than blocks
 6212016-09-29T19:31:45  <morcos> gmaxwell: it wasn't clear to me what the integral from 1 to 288 was compared to 288 to inf
 6222016-09-29T19:31:46  <wumpus> well it is a compromise
 6232016-09-29T19:31:53  <wumpus> putting the threshold higher makes some peers completely useless
 6242016-09-29T19:31:57  <sipa> to see what percentage of bandwidth is needed in 1-288
 6252016-09-29T19:31:58  <wumpus> which reduces morcos 's argument
 6262016-09-29T19:32:04  <jonasschnelli> Yes. I guess you convinced me to use two service bits then. -288 and -2016
 6272016-09-29T19:32:16  <gmaxwell> which is why it might be useful to use two bits and be able to signal 1-288, 1-2016... and perhaps start encouraging people to not prune shorter than 2016.
 6282016-09-29T19:32:37  <sipa> i think we're getting into a design discussion here
 6292016-09-29T19:32:47  <sipa> my number are very premature and not well analysed
 6302016-09-29T19:32:50  <wumpus> it'd also be possible to add a 288-flag now, and then consider a 2016 flag later
 6312016-09-29T19:32:54  <gmaxwell> sipa: indeed, thought that was the input you requested from me.
 6322016-09-29T19:32:57  <morcos> wumpus: yes, thats what i'm saying
 6332016-09-29T19:32:59  <gmaxwell> wumpus: yes! indeed.
 6342016-09-29T19:33:03  <jonasschnelli> Agree with wumpus
 6352016-09-29T19:33:04  <wumpus> if it turns out to be necessary
 6362016-09-29T19:33:10  <petertodd> wumpus: ACK
 6372016-09-29T19:33:15  <sipa> yes, i think just a 1-288 one seems useful
 6382016-09-29T19:33:16  <wumpus> good :)
 6392016-09-29T19:33:16  <jonasschnelli> Start with a simple tip-288 relay, and get some experience
 6402016-09-29T19:33:23  <gmaxwell> wumpus: it looks pretty clearly necessary but no need to do everything at once.
 6412016-09-29T19:33:30  <petertodd> wumpus: basically advice is, turn your node on at least once every two days
 6422016-09-29T19:33:54  <wumpus> petertodd: yes
 6432016-09-29T19:33:55  <gmaxwell> petertodd: we really should have cron mode for the daemon where it just syncs up and shuts off. :P
 6442016-09-29T19:34:06  <gmaxwell> bitcoind -oneshot
 6452016-09-29T19:34:08  <gmaxwell> :P
 6462016-09-29T19:34:17  <petertodd> gmaxwell: heh, that's not a crazy idea - I'd use it on my laptop
 6472016-09-29T19:34:18  <jonasschnelli> didn't we once had a proposal for the pause option?
 6482016-09-29T19:34:24  <wumpus> right, there's a flag that quits after reindex, but none that exits after sync
 6492016-09-29T19:34:31  <wumpus> would be easy to add tho
 6502016-09-29T19:34:41  <morcos> we could just ask for the utxo set, shoudl we discuss ideas how to do that
 6512016-09-29T19:34:51  <CodeShark> ^ yes :)
 6522016-09-29T19:34:58  <petertodd> make -oneshot run in the foreground with a progress bar :)
 6532016-09-29T19:34:59  <wumpus> without utxo commitment that's a no-go
 6542016-09-29T19:35:05  <morcos> thanks codeshark
 6552016-09-29T19:35:20  <petertodd> wumpus: +1
 6562016-09-29T19:35:42  <gmaxwell> morcos: pointless when we were unable to get past the discussion for the security model change to not validate the past history based on proof of work.
 6572016-09-29T19:35:48  <petertodd> and lets not underestimate how dangerous UTXO commitments can be - I'm very dubious about committing to the (U)TXO set more recently than maybe a month or two
 6582016-09-29T19:35:51  <CodeShark> would be great to query utxo for quick sync, then go backwards in time fetching blocks to increase security...but yes, this is a design discussion
 6592016-09-29T19:35:53  <morcos> i was making a joke, sorry
 6602016-09-29T19:36:18  <CodeShark> alas, quick sync doesn't look feasible in the nearterm
 6612016-09-29T19:36:20  <wumpus> ok, next topic?
 6622016-09-29T19:36:36  <gmaxwell> but since that was brought up... Can we talk about removing checkpoints?
 6632016-09-29T19:36:55  <wumpus> #topic removing checkpoints
 6642016-09-29T19:36:59  <sipa> what % of transactions are before the last checkpoint
 6652016-09-29T19:37:01  <sipa> does anyone know?
 6662016-09-29T19:37:03  <morcos> someone should write up a design proposal for that to be evaluated
 6672016-09-29T19:37:18  <gmaxwell> Right now they're used for two things, preventing header flooding with low difficulty headers; and skipping signatures in earlier blocks.
 6682016-09-29T19:37:21  <petertodd> gmaxwell: just removing checkpoints, or assuming sigs are valid if buried deep enough?
 6692016-09-29T19:37:27  <sipa> gmaxwell: and 3) estimating progress
 6702016-09-29T19:37:45  <wumpus> keeping something for estimating progress would make sense
 6712016-09-29T19:37:47  <sipa> i think 1) remains needed and 3) remains useful
 6722016-09-29T19:37:50  <wumpus> that doesn't need to be checkpoints
 6732016-09-29T19:38:06  <gmaxwell> because very few percentage of the transactions are below the checkpoint .. since libsecp256k1 (and I expect the checkqueue)-- my point two is basically pointless, and I think it could just be removed
 6742016-09-29T19:38:29  <gmaxwell> I think on a desktop it only adds 15-20 minutes to the sync.
 6752016-09-29T19:38:29  <petertodd> gmaxwell: I'd ACK simply removing checkpoints entirely; I'm not happy to see them replaced with another scheme to skip sig checking
 6762016-09-29T19:38:33  <wumpus> a block-height-to-relative-difficulty map would have much less of a stigma
 6772016-09-29T19:38:46  <wumpus> eh, verification difficulty that is
 6782016-09-29T19:38:52  <sipa> gmaxwell: really?
 6792016-09-29T19:38:54  <gmaxwell> petertodd: I think we could remove CP from reason two without implementing the replcement.
 6802016-09-29T19:39:06  <gmaxwell> petertodd: morcos is right that needs a design proposal outside of the meeting.
 6812016-09-29T19:39:12  <sdaftuar> i'm a bit confused about how to think about checkpoints for signature skipping
 6822016-09-29T19:39:22  <gmaxwell> sipa: I benchmarked before but I'm going off of memory, I could be wildly wrong. I will test again if there is interest.
 6832016-09-29T19:39:52  *** jl2012 has quit IRC
 6842016-09-29T19:39:52  <jonasschnelli> Removing checkpoints would slow down (maybe insignificant) a scan in a possible SPV hybrid mode?
 6852016-09-29T19:39:54  <gmaxwell> For reason (1) the only answer I have is that I think we should proposal a bit to perpetually increase the minimum difficulty from 1 to something else.
 6862016-09-29T19:40:00  <sdaftuar> for instance the recent ISM change caused us to do less validation for certain blocks in our history (blocks in a softfork between the 75% and 95% thresholds)
 6872016-09-29T19:40:10  <sipa> jonasschnelli: SPV mode won't validate *anything* at all
 6882016-09-29T19:40:24  *** mturquette has quit IRC
 6892016-09-29T19:40:28  <gmaxwell> (with a checkpoint like bypass of that new rule, for existing blocks that break it) As little as 100,000 would eliminate the header flooding vulenrablity.
 6902016-09-29T19:40:33  <jonasschnelli> Yes. But assume we would add an SPV hibrid mode in oder to received payment during IBD
 6912016-09-29T19:40:50  <jonasschnelli> One would need to download 400k headers without a checkpoint at h400k
 6922016-09-29T19:40:53  <luke-jr> maybe checkpoints should just be disabled by default before complete removal?
 6932016-09-29T19:41:00  <sipa> jonasschnelli: i think you're confused
 6942016-09-29T19:41:07  <gmaxwell> for Sipa's (3) reason for 'checkpoints' I don't give a darn, use chicken bones for progress estimation for all I care. :P it's historical accident that checkpoints and progress use the same data structure.
 6952016-09-29T19:41:11  *** jl2012 has joined #bitcoin-core-dev
 6962016-09-29T19:41:24  <morcos> gmaxwell: :) +1
 6972016-09-29T19:41:28  <wumpus> gmaxwell: yes, my point too
 6982016-09-29T19:41:33  *** jl2012_ has joined #bitcoin-core-dev
 6992016-09-29T19:41:34  <sipa> gmaxwell: agree, those could be completely separated
 7002016-09-29T19:41:36  <petertodd> gmaxwell: ACK checken bones
 7012016-09-29T19:41:36  <gmaxwell> Might as well fit a cubic spline to the height vs txn count... and store the parameters.
 7022016-09-29T19:41:53  <wumpus> right
 7032016-09-29T19:42:02  *** jl2012 has left #bitcoin-core-dev
 7042016-09-29T19:42:09  <petertodd> gmaxwell: heh, if we do that with floating point math that has the advantage that it _can't_ be used for consensus :)
 7052016-09-29T19:42:18  * sipa now remembers a song our student organization wrote to the melody of staying alive, called 'cubic spline'
 7062016-09-29T19:42:21  <gmaxwell> so my proposal, if there is interest, is that I'll measure the performance impact of removing the signature skippingentirely (esp post checkqueue). And if it's not awful, we'll remove.
 7072016-09-29T19:42:37  <wumpus> +1
 7082016-09-29T19:42:44  <sipa> gmaxwell: i'm unconvinced
 7092016-09-29T19:42:52  <wumpus> it doesn't hurt to benchmark
 7102016-09-29T19:42:55  *** mturquette has joined #bitcoin-core-dev
 7112016-09-29T19:43:00  <gmaxwell> and maybe I'll tender a proposal to up the minimum difficulty, but I'd like to know what people think about that.
 7122016-09-29T19:43:04  <wumpus> measuring is always better than making assumptions
 7132016-09-29T19:43:11  <sipa> with a replacement for sig skipping that isn't based on checkpoints we could significantly improve things
 7142016-09-29T19:43:22  *** zmanian___ has quit IRC
 7152016-09-29T19:43:39  <petertodd> sipa: I don't think such a replacement can exist without changing the security assumptions; I'd *rather* have checkpoints than trusting hashing power for that
 7162016-09-29T19:43:44  <sipa> the last checkpoint currently is very old for the very reason that we've been planning to replace it
 7172016-09-29T19:43:47  <gmaxwell> sipa: would you like to help work on a proposal for that? it has been controversial in the past. I'd like to do something good, because otherwise imprudent attempts will be adopted instead.
 7182016-09-29T19:44:15  <sipa> so it's unfair to use the "the last checkpoint is old" as a given; it's something we've affected indirectly
 7192016-09-29T19:44:19  <petertodd> sipa: though what checkpoints should do is say "Something big has changed; you can disable checkpoints with --no-checkpoints, but you should find out what this means before doing so."
 7202016-09-29T19:44:29  <gmaxwell> (for example Bitcoin Classic's current behavior simply looks at block header timestamps and ignores signatures when they're more than 24 hours (*par) old by the local clock. It's easily exploited and makes me sad.
 7212016-09-29T19:44:40  <sipa> petertodd: it's my opinion that on a timescale of months, it doesn't matter
 7222016-09-29T19:44:53  <sipa> IF you can guarantee it's actually a timescale of months
 7232016-09-29T19:44:55  <wumpus> yes that makes me sad too
 7242016-09-29T19:44:58  <petertodd> sipa: on a timescale of months, checkpoints shouldn't matter either...
 7252016-09-29T19:45:06  <wumpus> anything based on time seems very brittle
 7262016-09-29T19:45:27  <sipa> petertodd: look at the current hashrate; what's 3 months worth of chain work at that hashrate
 7272016-09-29T19:45:31  <petertodd> wumpus: and anything based on work isn't much better if you're running an old client, and mining has advanced significantly
 7282016-09-29T19:45:37  <jonasschnelli> sipa: I (hope) I'm not confused. If we would add a SPV hybrid mode directly fetch blocks at the tip (in order to received payments), no available checkpoint would result in downloading all headers *losing* maybe 3-4mins before you can start using SPV... minor issue though, I agree
 7292016-09-29T19:45:38  <petertodd> sipa: that assumes you know what the current hashrate is
 7302016-09-29T19:45:41  <gmaxwell> wumpus: the prior proposals were based on work,  e.g. skip if the best chain you see dominates the next conflicted chain at that hight by N months of work.
 7312016-09-29T19:45:44  <Chris_Stewart_5> gmaxwell: How have we solved the problem that checkpoints were originally created for? You have an excerpt in here: https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_5):_Initial_Block_Download#Checkpoints
 7322016-09-29T19:45:45  <petertodd> sipa: your node might be surrounded by sybils
 7332016-09-29T19:45:52  <gmaxwell> wumpus: with a 'minimum total work' coded in as part of the release proces.
 7342016-09-29T19:46:00  <sipa> Chris_Stewart_5: headers first sync
 7352016-09-29T19:46:08  <sipa> Chris_Stewart_5: 0.10
 7362016-09-29T19:46:08  <gmaxwell> Chris_Stewart_5: headers first sync.
 7372016-09-29T19:46:09  <wumpus> gmaxwell: right. Well, at the least it should be measured whether such a change is really worth it.
 7382016-09-29T19:46:16  <sipa> petertodd: yes, i know...
 7392016-09-29T19:46:21  <sipa> so, let's measure.
 7402016-09-29T19:46:25  <sipa> and discuss later
 7412016-09-29T19:46:37  <gmaxwell> Chris_Stewart_5: and the signature skipping behavior in checkpoints was actually a result of a bug fixed years ago.. mlock being used on all allocations making script validation INSANELY slow.
 7422016-09-29T19:46:41  <wumpus> so much of the verification overhead is looking up UTXOs
 7432016-09-29T19:46:44  *** zmanian___ has joined #bitcoin-core-dev
 7442016-09-29T19:46:44  <gmaxwell> sipa: okay.
 7452016-09-29T19:46:47  <wumpus> something you'll not avoid
 7462016-09-29T19:46:53  *** Lauda_ has joined #bitcoin-core-dev
 7472016-09-29T19:47:20  <gmaxwell> Chris_Stewart_5: but then with chain growth we became dependant on it to keep sync times reasonable. but libsecp256k1 made signature validation >5x faster.
 7482016-09-29T19:47:21  <wumpus> especially for recent blocks
 7492016-09-29T19:47:30  *** Lauda has quit IRC
 7502016-09-29T19:47:38  *** Lauda_ is now known as Lauda
 7512016-09-29T19:47:38  <wumpus> if you do any benchmarking please look at the recent blocks, not the first N
 7522016-09-29T19:47:38  *** Lauda has quit IRC
 7532016-09-29T19:47:39  *** Lauda has joined #bitcoin-core-dev
 7542016-09-29T19:48:04  <gmaxwell> wumpus: it's still a major speed up on existing blocks.
 7552016-09-29T19:48:07  <sipa> in a side node: i've already updated my logging to measure bandwidth vs blockdepth instead of just count.
 7562016-09-29T19:48:11  <Chris_Stewart_5> So header sync solves the attack of flooding disk space, but not having your entire network hijacked, correct?
 7572016-09-29T19:48:29  *** harambea has joined #bitcoin-core-dev
 7582016-09-29T19:48:32  <wumpus> Chris_Stewart_5: huh?
 7592016-09-29T19:48:41  <wumpus> gmaxwell: sure, could be
 7602016-09-29T19:48:42  <gmaxwell> Chris_Stewart_5: isolation can be resolved by simply knowing what the total work of the best chain was at release.
 7612016-09-29T19:48:52  *** lesderid_ has joined #bitcoin-core-dev
 7622016-09-29T19:49:00  <gmaxwell> Chris_Stewart_5: sorry, this was discussed prior times removing checkpoints had come up, I haven't completely described the background.
 7632016-09-29T19:49:19  <Chris_Stewart_5> gmaxwell: Thanks for the explanation, i'll keep digging.
 7642016-09-29T19:49:45  <wumpus> Chris_Stewart_5: ah, you mean being isolated and being fed a wrong chain, sorry I was imaginging some wacky things at having your network hijacked :)
 7652016-09-29T19:51:01  *** lesderid has quit IRC
 7662016-09-29T19:51:01  <wumpus> ok, next topic?
 7672016-09-29T19:51:01  <gmaxwell> wumpus: just the "you got a faithful bitcoin core download but the attacker controls your network"... but that doesn't need a checkpoint to fix, a simple partitioning detction that knows the total work of the best chain at releast time is sufficient.
 7682016-09-29T19:51:01  <gmaxwell> Thanks for the discussion.
 7692016-09-29T19:51:13  <wumpus> #topic segwit against uncompressed keys or not
 7702016-09-29T19:51:17  <wumpus> (10 minutes to go)
 7712016-09-29T19:51:24  <wumpus> (9 minutes to go)
 7722016-09-29T19:51:27  <petertodd> so to be clear, *just* segwit right?
 7732016-09-29T19:51:30  <CodeShark> does anyone still use uncompressed keys?
 7742016-09-29T19:51:33  <wumpus> yes, only segwit
 7752016-09-29T19:51:39  <achow101> CodeShark: armory does
 7762016-09-29T19:51:42  <luke-jr> seems uncontroversial
 7772016-09-29T19:51:49  <petertodd> I'm happy to ACK that given just segwit
 7782016-09-29T19:51:55  *** aureianimus_ has quit IRC
 7792016-09-29T19:51:57  <achow101> having segwit enforce uncompressed keys would delay segwit adoption for armory users
 7802016-09-29T19:52:01  *** aureianimus has joined #bitcoin-core-dev
 7812016-09-29T19:52:03  <achow101> *compressed
 7822016-09-29T19:52:05  <jl2012_> it's in #8499
 7832016-09-29T19:52:09  <luke-jr> achow101: why? just compress them
 7842016-09-29T19:52:13  <wumpus> gmaxwell: yes, though we had a lot of trouble with partitioning detection, I remember some code being stripped out and such. But anyhow, yes that's the better approach if it can be gotten to work.
 7852016-09-29T19:52:22  <sipa> achow101: sigh, does armory still not do that?
 7862016-09-29T19:52:30  <achow101> luke-jr: we have to change the whole wallet structure (it's still going to happen anyways)
 7872016-09-29T19:53:34  <wumpus> gmaxwell: without too much false positives
 7882016-09-29T19:53:34  <luke-jr> achow101: why?
 7892016-09-29T19:53:34  <sipa> achow101: alan said somewhere in 2013 he was implementing it...
 7902016-09-29T19:53:34  <achow101> alan's gone now..
 7912016-09-29T19:53:34  <luke-jr> afaik the only downside to using compressed keys is it changes the address, which segwit is changing anyway
 7922016-09-29T19:53:34  <CodeShark> it's not a very complicated change
 7932016-09-29T19:53:34  <wumpus> armory still uses uncompressed keys?!
 7942016-09-29T19:53:34  <luke-jr> there's no reason you'd need to change the wallet structure I can see
 7952016-09-29T19:53:34  <wumpus> in any case this only applies to segwit, not to old transactions
 7962016-09-29T19:53:34  <achow101> the plan is to have a new wallet structure with bip32 that supports segwit and compressed keys
 7972016-09-29T19:53:41  <gmaxwell> wumpus: "you're partitioned until you see a header chain with at least work X" is a pretty simple critera. :P
 7982016-09-29T19:53:44  <sipa> luke-jr: it had fixed size records in its wallet format for pubkeys
 7992016-09-29T19:54:05  <sipa> achow101: well if a new wallet format is needed for segwit anyway, it doesn't matter right?
 8002016-09-29T19:54:10  <gmaxwell> achow101: oh god please do not use uncompressed keys with segwit. why would you do that?
 8012016-09-29T19:54:13  <luke-jr> sipa: zero-pad it?
 8022016-09-29T19:54:35  <achow101> sipa: well no, we don't need a new wallet for segwit as it could still work with the old one with a little bit of hacking
 8032016-09-29T19:54:48  <achow101> that was the original plan
 8042016-09-29T19:54:48  <luke-jr> achow101: no less than compressed could
 8052016-09-29T19:55:15  <luke-jr> sipa: or store the uncompressed key, and compress it at address-generation/signing
 8062016-09-29T19:55:26  <gmaxwell> achow101: why cant the same hack that indicates segwit is in use indicate compressed.. you just chop off some bytes of the key pretty much.
 8072016-09-29T19:55:43  <sipa> btw, uncompressed keys account for 0.7% of used keys in succesful sigs on the network (in the past 2 hours)
 8082016-09-29T19:55:44  <gmaxwell> it could be done entirely inside the process that seralizes the segwit scriptpubkey.
 8092016-09-29T19:55:54  *** Chris_Stewart_5 has quit IRC
 8102016-09-29T19:56:06  <achow101> gmaxwell: idk. ask goatpig
 8112016-09-29T19:56:06  <gmaxwell> achow101: okay
 8122016-09-29T19:56:09  * michagogo pokes his head in belatedly
 8132016-09-29T19:56:10  <CodeShark> I think we should encourage all wallets to use compressed keys - achow101, if you need help with this I'd be willing to help
 8142016-09-29T19:56:25  <sipa> agree - we should help
 8152016-09-29T19:56:27  <gmaxwell> yes, lots of people would be glad to help.
 8162016-09-29T19:56:32  <sipa> instead of just yell
 8172016-09-29T19:56:50  <gmaxwell> well I offered to help armory move off uncompressed keys to alan several times, including offering to pay to do it.
 8182016-09-29T19:56:56  <gmaxwell> so please don't say anyone just yelled.
 8192016-09-29T19:58:39  <CodeShark> I initially designed my account structures to only use compressed keys - but later added a compressed bit to support legacy stuff
 8202016-09-29T19:59:06  <petertodd> CodeShark: what legacy stuff specifically? legacy armory users?
 8212016-09-29T19:59:08  <wumpus> CodeShark: bah,it's kind of sad that to hear some things seem to be going back instead of forward :)
 8222016-09-29T19:59:18  <CodeShark> yes, to support other wallets
 8232016-09-29T19:59:27  *** anchow101 has joined #bitcoin-core-dev
 8242016-09-29T19:59:27  <wumpus> it's time
 8252016-09-29T19:59:44  *** lesderid_ has quit IRC
 8262016-09-29T19:59:50  <CodeShark> but I think we really do need to prod all wallets to move to compressed keys
 8272016-09-29T19:59:52  *** lesderid has joined #bitcoin-core-dev
 8282016-09-29T19:59:53  *** anchow101 has quit IRC
 8292016-09-29T20:00:07  <CodeShark> there's really no reason to continue to support uncompressed keys - other than perhaps some migration tools
 8302016-09-29T20:00:15  <wumpus> #endmeeting
 8312016-09-29T20:00:15  *** anchow101 has joined #bitcoin-core-dev
 8322016-09-29T20:00:15  <lightningbot> Meeting ended Thu Sep 29 20:00:15 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
 8332016-09-29T20:00:15  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.html
 8342016-09-29T20:00:15  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.txt
 8352016-09-29T20:00:15  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.log.html
 8362016-09-29T20:00:15  <gmaxwell> CodeShark: as pieter notes, virutally nothing is already.
 8372016-09-29T20:00:29  <gmaxwell> 0.7% percent
 8382016-09-29T20:00:38  <anchow101> Yes, help would be appreciated
 8392016-09-29T20:01:05  <wumpus> well supporting it in consensus for the normal network keeps making sense, but segwit is just such a great oppertunity to get rid of it
 8402016-09-29T20:01:31  <petertodd> wumpus: yeah, I don't see how we can remove backwards compatibility for it w/o confiscating funds, but no reason to not remove support in new addresses
 8412016-09-29T20:01:40  <wumpus> petertodd: indeed
 8422016-09-29T20:01:56  <gmaxwell> yes, thats why its important to get rid of now. otherwise I wouldn't care if action were taken n months later.
 8432016-09-29T20:01:56  <luke-jr> if anything, we should be discussing whether to make it a consensus rule rather than a policy ;)
 8442016-09-29T20:02:10  *** achow101_ has joined #bitcoin-core-dev
 8452016-09-29T20:02:20  <gmaxwell> luke-jr: I like to but many people feel that addining an additional consensus rule for segwit now wouldn't be prudent.
 8462016-09-29T20:02:37  <gmaxwell> making it non-standard is sufficient in my view, such that we'd be able to make it a consensus rule later.
 8472016-09-29T20:02:51  <btcdrak> achow101 seems to be having connection problems
 8482016-09-29T20:02:51  <luke-jr> sure
 8492016-09-29T20:02:55  <achow101_> btcdrak: just a little bit. switching computers
 8502016-09-29T20:02:57  <CodeShark> gmaxwell: having to deal with the additional case complicates implementations
 8512016-09-29T20:02:59  <luke-jr> just saying the rule shouldn't be controversial itself really
 8522016-09-29T20:03:02  <CodeShark> not very much, but still
 8532016-09-29T20:03:07  <petertodd> gmaxwell: well, so long as we loudly warn that this is intended to become unspendable later if you bypass the standardness
 8542016-09-29T20:03:27  <morcos> gmaxwell: if so, we should be as clear about it being not allowed now as if we were to make it a consensus rule now.
 8552016-09-29T20:04:25  <gmaxwell> for those thinking that we have to verify all the old stuff for all time, that might be true for bitcoin core, but in the future I could imagine some implementations just not bothering to verify old stuff.
 8562016-09-29T20:04:25  <gmaxwell> morcos: petertodd: agreed for sure.
 8572016-09-29T20:04:34  <sdaftuar> petertodd: ntoe that it's standard to create an output with an uncompressed pubkey hash, as we can't detect the issue until a spend attempt (right?)
 8582016-09-29T20:04:34  <jtimon> I agree with luke-jr on not seeing the controversy in it being consensus rule with the rest of segwit
 8592016-09-29T20:04:43  <petertodd> gmaxwell: I don't mean verifying old stuff, I mean verifying new txs spending old coins
 8602016-09-29T20:04:55  <petertodd> sdaftuar: yes, nothing we can do about that though
 8612016-09-29T20:05:16  <gmaxwell> jtimon: well in the case of low-S which we also wanted to make a consensus rule, jl2012 discovered that there were corner conditions we wanted to think about more carefully before making it a consensus rule.
 8622016-09-29T20:05:23  <sdaftuar> petertodd: right; just want to make sure we're all on the same page as i think communicating this widely/loudly is important
 8632016-09-29T20:05:23  <petertodd> for example, we've made hybrid pubkeys non-standard, and given basically no-one's ever used them in production for anything I'd have no issues with making them unspendable in a soft-fork
 8642016-09-29T20:05:43  <jtimon> gmaxwell: thanks
 8652016-09-29T20:06:30  <gmaxwell> jtimon: I don't think the same applies to uncompressed keys, because the criteria there is even simpler. but the lowS reason is part of why we punted this collection of improvements to policy for now.
 8662016-09-29T20:06:49  <jtimon> mhmm
 8672016-09-29T20:06:57  <michagogo> I saw a movie that depicts a form of distributed/decentralized system, to avoid it getting shut down. Or in the words of the character that explains it, "everyone that logs on is a server". It's said to be "open source", but then that's explained as "anyone can edit the code, like Wikipedia.
 8682016-09-29T20:07:18  <achow101_> so if anyone wants to help armory with segwit support, bip32, compressed keys, we accept PRs. All our work happens in the dev branch, not master
 8692016-09-29T20:07:19  <michagogo> And the code is deployed when a majority of users approve it
 8702016-09-29T20:07:42  *** veleiro has quit IRC
 8712016-09-29T20:08:01  <wumpus> michagogo: heh, open source in some weird twisted mirror world
 8722016-09-29T20:08:05  <gmaxwell> achow101: is there a IRC channel where things are discussed? E.g. where should I ask goatpig about compressed pubkeys in segwit.
 8732016-09-29T20:08:11  <sipa> michagogo: i believe they're mistakingly not describing a computer network, but politics.
 8742016-09-29T20:08:12  *** BakSAj has quit IRC
 8752016-09-29T20:08:12  <achow101_> #bitcoin-armory
 8762016-09-29T20:08:24  <jtimon> michagogo: that is, when sybil decides so...
 8772016-09-29T20:08:50  <luke-jr> achow101_: meh, should just collapse that into #bitcoin-dev :p
 8782016-09-29T20:09:27  <michagogo> And it's completely vulnerable to Sybil attacks…
 8792016-09-29T20:09:30  *** cryptapus_ has quit IRC
 8802016-09-29T20:09:43  <michagogo> Gah, lagging
 8812016-09-29T20:09:57  <michagogo> Yeah
 8822016-09-29T20:10:17  <michagogo> And of course, when the last user logs off, it doesn't just stop working
 8832016-09-29T20:10:44  <michagogo> The sybil attackers are able to watch it dramatically implode with special effects, "graphical corruption" type stuff
 8842016-09-29T20:11:50  <michagogo> And there's the obligatory "they're blocking all our foreign IPs" and that kind of stuff, with no explanation of who "they" are
 8852016-09-29T20:13:45  <jtimon> so what was there a conclusion for the range service bits? nothing/top-288/everything?
 8862016-09-29T20:13:45  <jtimon> what about the getrange message and "sharding"
 8872016-09-29T20:13:45  <GitHub159> [bitcoin] laanwj opened pull request #8843: rpc: Handle `getinfo` client-side in bitcoin-cli w/ `-getinfo` (master...2016_09_getinfo_clientside) https://github.com/bitcoin/bitcoin/pull/8843
 8882016-09-29T20:13:45  <wumpus> conclusion was to add one service bit: last-288-served
 8892016-09-29T20:13:45  <wumpus> and maybe later one for last-1000-served
 8902016-09-29T20:13:45  <jtimon> wumpus: I see, and leave the rest for later, thanks
 8912016-09-29T20:13:45  <luke-jr> 1024 would be rounder. ☺
 8922016-09-29T20:13:45  <wumpus> and a jackpot for whoever enabled both at once
 8932016-09-29T20:13:51  <luke-jr> if you set both, does it mean last 288000? :P
 8942016-09-29T20:14:16  *** aureianimus has quit IRC
 8952016-09-29T20:14:22  <jtimon> would it be crazy to just have last-1024 without last-288 and just change prunning's default?
 8962016-09-29T20:14:29  *** aureianimus has joined #bitcoin-core-dev
 8972016-09-29T20:14:48  <wumpus> 288 is not just the default, it's the minimum
 8982016-09-29T20:15:05  <wumpus> I'd be okay with changing the default not the minimum, but that'd keep some nodes completely useless
 8992016-09-29T20:15:18  <wumpus> whereas by far most requests are in the last 288
 9002016-09-29T20:15:25  <luke-jr> wumpus: useless for syncing*
 9012016-09-29T20:15:57  <luke-jr> frankly, there are enough full-archive nodes out there that we really don't *need* to do anything right now, so meh :p
 9022016-09-29T20:16:06  <sipa> wumpus: actually, not true.
 9032016-09-29T20:16:13  <jtimon> well, the users know what to do to stop being useless...
 9042016-09-29T20:16:16  <wumpus> which, as morcos remarked, preferentially downloading the last blocks from would take a lot of load of nodes that do keep more blocks
 9052016-09-29T20:16:22  <sipa> there are more requests in 101-1000 deep then 2-100 deep
 9062016-09-29T20:16:29  <wumpus> ok...
 9072016-09-29T20:16:33  <sipa> *than
 9082016-09-29T20:16:37  <wumpus> I misremembered apparently, never mind
 9092016-09-29T20:16:39  *** laurentmt has joined #bitcoin-core-dev
 9102016-09-29T20:16:45  <luke-jr> an unsyncable-from node is still more useful than a syncable node that isn't used for a wallet
 9112016-09-29T20:16:46  *** laurentmt has quit IRC
 9122016-09-29T20:17:08  <luke-jr> syncable-from*
 9132016-09-29T20:17:26  <jtimon> maybe we can changethe prunning minimum if that simplifies things?
 9142016-09-29T20:17:30  <sipa> wumpus: well, sample of 1 long-term-running node over the course of a few weeks of data
 9152016-09-29T20:17:35  <sipa> wumpus: more samples welcome
 9162016-09-29T20:17:57  <wumpus> sipa: do you have a special patch for statistics collection?
 9172016-09-29T20:18:00  <gmaxwell> sipa: need to filter out bitnotes.
 9182016-09-29T20:18:13  <sipa> gmaxwell: right; how do you suggest to do that?
 9192016-09-29T20:18:26  <wumpus> sipa: or a script for parsing logs?
 9202016-09-29T20:18:39  <sipa> wumpus: both; i'll publish them after a little cleanup
 9212016-09-29T20:18:46  <wumpus> I could put it up on a few nodes, no problem
 9222016-09-29T20:19:05  <sipa> it just logs an extra line with depth and block size for each requested block
 9232016-09-29T20:19:13  <wumpus> nice
 9242016-09-29T20:19:16  <jtimon> I guess it's not completely crazy, but nobody seem to specially like it
 9252016-09-29T20:19:35  <sipa> en then
 9262016-09-29T20:19:36  <sipa> S=0; fgrep DEEP ~/.bitcoin/debug.log | cut -d ' ' -f 4 | sort -g | uniq -c | tac | while read C D; do S=$(($S+$C)); echo "$D $C $S"; done | tac | less
 9272016-09-29T20:19:39  <sipa> to inspect :)
 9282016-09-29T20:20:14  *** laurentmt has joined #bitcoin-core-dev
 9292016-09-29T20:20:14  <wumpus> jtimon: no, it's not completely crazy, using only one service bit is kind of elegant
 9302016-09-29T20:21:11  <jtimon> :)
 9312016-09-29T20:22:20  <wumpus> jtimon: if 1000 is really one-size-fits-all, and <1000-keeping nodes may as well be ignored. It's just hard to say without better statistics.
 9322016-09-29T20:22:34  <wumpus> also statistics about what pruning sizes people prefer
 9332016-09-29T20:23:00  <wumpus> I mean if everyone prefers the minimum and no one sets 1000 in practice
 9342016-09-29T20:23:17  *** anchow101 has quit IRC
 9352016-09-29T20:23:18  <gmaxwell> sipa: just do Satoshi:(recent) useragents.
 9362016-09-29T20:23:35  <jtimon> well, independenlty of the statistics we will eventually need a more generic solution for flexible sharding, right?
 9372016-09-29T20:23:45  <sipa> jtimon: maybe
 9382016-09-29T20:24:05  <sipa> "need" is a big word imho
 9392016-09-29T20:24:13  <sipa> but i agree it would be nice
 9402016-09-29T20:24:14  <gmaxwell> jtimon: I think we do, would you like to finish the solution for that I started on?
 9412016-09-29T20:24:16  <wumpus> jtimon: well there needs to be a different solution for historical block hosting IMO, but that's a different thing
 9422016-09-29T20:25:26  *** aureianimus has quit IRC
 9432016-09-29T20:25:26  <gmaxwell> sipa: I think excepting participants to keep around hundreds of gigs of blockchain is not conducive to the surival of the network, the alternative I see is a hardfork that drops off the history past some point. (e.g. just restarts the chain from a utxo commitment made a year before)
 9442016-09-29T20:25:33  *** aureianimus has joined #bitcoin-core-dev
 9452016-09-29T20:26:26  <sipa> gmaxwell: well, or just stop supporting historical block fetching more than 1 year or whatever number back on the p2p protocol, and use http
 9462016-09-29T20:26:40  <wumpus> or bittorrent *ducks*
 9472016-09-29T20:26:45  *** brainwave has joined #bitcoin-core-dev
 9482016-09-29T20:26:50  <jtimon> wumpus: yeah, historical hosting is what I mean
 9492016-09-29T20:27:15  <jtimon> gmaxwell: maybe, but it sounded deterministic like luke-jr proposed instead of flexible like wumpus wanted
 9502016-09-29T20:27:22  <wumpus> it could be anything that supports downloading ranges of data...
 9512016-09-29T20:27:46  <brainwave> Under overview, balances, on the right side of  available, pending, total, add ~ exchange rate for dollars, pounds, euro
 9522016-09-29T20:28:00  <sipa> brainwave: bitcoin core does not and cannot know exchange rates
 9532016-09-29T20:28:37  <sipa> (because it would require contacting a centralized service, which we don't do by design)
 9542016-09-29T20:29:09  <wumpus> yes or someone would need to commit them to the chain, but that'd still be trusting a central issuer/signer of the information
 9552016-09-29T20:29:12  <wumpus> it's just a no-go
 9562016-09-29T20:30:27  *** laurentmt has quit IRC
 9572016-09-29T20:32:12  *** brainwave has quit IRC
 9582016-09-29T20:35:52  <gmaxwell> well if the users of bitcoin accepted that kind of security model change, what I would suggest is something like every 26280 blocks the block is required to have a commitment to the utxo set (could be a linear hash) as of 2016 blocks prior. and then six months of work after that, that commitment becomes usable for initial sync. and so then no one need process more than a year of blocks at sync.
 9592016-09-29T20:35:58  <gmaxwell> .. though you would have to store three copies of the utxo set (though perhaps deduplicated)
 9602016-09-29T20:36:46  <gmaxwell> jtimon: I don't know why anyone would find determinstic less desirable.
 9612016-09-29T20:37:21  <sipa> gmaxwell: well i expect the controversy to not be about the change in security model, but about the perpetual requirement of having a utxo set
 9622016-09-29T20:37:42  *** bad_duck has quit IRC
 9632016-09-29T20:37:55  *** aureianimus has quit IRC
 9642016-09-29T20:38:55  *** aureianimus has joined #bitcoin-core-dev
 9652016-09-29T20:38:55  *** bad_duck has joined #bitcoin-core-dev
 9662016-09-29T20:40:06  <wumpus> gmaxwell: I explained that: if you make it deterministic you have to be sure of the parameters in advance, there is no room for tweaking or optimizing later on
 9672016-09-29T20:42:12  <gmaxwell> wumpus: well you simply extend the protocol to have a new signaling mechenism for the tweaked thing.
 9682016-09-29T20:42:16  <wumpus> sipa: yes the bigger problem is the ever-growing UTXO set
 9692016-09-29T20:42:24  <wumpus> gmaxwell: but then it loses backwards compatibility every time
 9702016-09-29T20:42:29  *** jnewbery has quit IRC
 9712016-09-29T20:43:03  <gmaxwell> something that just signals absolute heights has the problem that the communicated information will always be out of date. .. or if nodes don't change the ranges they host, we will end up with highly irregular distributions of information.
 9722016-09-29T20:43:42  <sipa> the type of tweaking needed, and the potentially aging problem depend on the specific proposal
 9732016-09-29T20:44:09  <sipa> i'm sure we can come up with something that seems reasonable to all
 9742016-09-29T20:44:28  <wumpus> agree, there may be a compromise that is somewhat flexible and still deterministic
 9752016-09-29T20:44:50  <gmaxwell> well what I suggested might not be viable after all too. I'm not sure, I wasn't successful in achieving all my goals at once.
 9762016-09-29T20:45:26  <wumpus> I just don't think setting it all in stone in advance is a good idea, for the whole reason that it's so hard to achieve all your goals all at once
 9772016-09-29T20:45:40  <wumpus> especially if you don't know some of those goals yet
 9782016-09-29T20:45:47  <gmaxwell> I wanted a scheme that would result in a uniform distribution of blocks, that didn't depend on peers to look to see what other peers had (because that could be spoofed), required minimal communication (not a long list of blocks in an addr message).. and retainined uniformity as the chain grew, without causing peers to redownload blocks they already forgot.
 9792016-09-29T20:47:13  <gmaxwell> So I had found two schemes, one where peers had a ID and the amount of blocks they would store, and from that they could determine which they would store, and as new blocks came in they might store then and drop some group of old ones.  The problem with it was that to figure out if a particular peer had block X you had to do computation linear in the number of blocks in the chain.
 9802016-09-29T20:47:37  <wumpus> darn, also fingerprinting will be hard to avoid
 9812016-09-29T20:47:45  <gmaxwell> Then I had another scheme that was sublinerar work, BUT a peer might drop a block but later have to go fetch it again.
 9822016-09-29T20:47:54  <gmaxwell> wumpus: thats unavoidable with any split up scheme.
 9832016-09-29T20:48:14  <wumpus> yes
 9842016-09-29T20:48:15  <sipa> make the IP address part of the seed
 9852016-09-29T20:48:25  <sipa> if your dhcp changes, you have to resync, sorry.
 9862016-09-29T20:48:30  <wumpus> unless a substantial part of nodes are the same
 9872016-09-29T20:48:31  <gmaxwell> sipa: then when you change IP, you have to go download a different set of blocks.. :P hah
 9882016-09-29T20:48:47  <wumpus> e.g. there are only 8 IDs, pick one
 9892016-09-29T20:48:54  <gmaxwell> wumpus: well I was thinking 32 bits, but perhaps a smaller collection would be fine.
 9902016-09-29T20:49:18  <gmaxwell> but that gives you at best only 1/8th spitting storage. :( maybe fine now, but not in the long term.
 9912016-09-29T20:49:47  <wumpus> maybe the number of groups can grow over time, a doubling every so many blocks :)
 9922016-09-29T20:50:02  <sipa> hah: if you get a request through an IP that doesn't correspond to your local storage, just proxy all requests through to another node which does, and use that to gradually resync for the new seed.
 9932016-09-29T20:50:05  <gmaxwell> Part of why I haven't given this that much more thought is because I think bitcoin will need to move to the commit state and forget history model; the ever growing sync time is too big a tide to stand against.
 9942016-09-29T20:50:15  <gmaxwell> sipa: lol!
 9952016-09-29T20:50:32  <gmaxwell> sipa: I think thats actually how the freenet location swapping works, funny enough.
 9962016-09-29T20:50:43  <wumpus> hehe
 9972016-09-29T20:50:46  <sipa> downside: if you want this to be fingerprint resistant, you have no way to determine how many proxies your blocks actually went through
 9982016-09-29T20:50:49  <sipa> => instant mixnet
 9992016-09-29T20:51:37  <gmaxwell> sipa: freenet nodes change position over time, and they do it by swapping their location with a direct neighbor, when that location swap makes them both closer to where they want to be, ... when requests come in for the new location, they don't have the data, but it's only one hop away..
10002016-09-29T20:52:55  <wumpus> gmaxwell: I've always thought that, it's hard to imagine this continuing for 10's of years, but where to put the anchor...
10012016-09-29T20:52:59  <gmaxwell> in any case. if there were only 8 flavors of nodes, then it all becomes simple, block_height//1000 % 8 = flavor.
10022016-09-29T20:53:32  * gmaxwell lunch
10032016-09-29T20:54:18  <wumpus> that seems kind of elegant and straightforward, there must be a catch
10042016-09-29T20:58:57  *** aureianimus has quit IRC
10052016-09-29T20:59:06  *** aureianimus has joined #bitcoin-core-dev
10062016-09-29T20:59:47  <jtimon> gmaxwell: sorry, well the deterministic seems to come at the cost of less flexibility
10072016-09-29T21:01:37  <sipa> wumpus: i'm trying to think about why 8 isn't enough
10082016-09-29T21:01:37  <wumpus> if you want to automatically scale the number of flavors up with height you could divide height 0..N into X flavors, the N..3N into 2*N flavors, and so on, where each flavor gets flavor (x<<1)+randbit()
10092016-09-29T21:01:37  <jtimon> only 8 flavors requires you to store 1/8 of the blockchain
10102016-09-29T21:01:37  <sipa> and we could have names for the first 8 top-level flavours or so... so your wallet could report "Looking for a bittersweet node..."
10112016-09-29T21:01:40  <wumpus> (well those numbers are arbitrary but the idea is that if a doubling of the # is needed, the new flavor, a member of a twice as big set, would contain the previous one)
10122016-09-29T21:02:21  <wumpus> hehe, yes assigning names would be nice
10132016-09-29T21:13:53  *** jasonv76 has quit IRC
10142016-09-29T21:14:10  *** jasonv76 has joined #bitcoin-core-dev
10152016-09-29T21:18:39  *** jnewbery has joined #bitcoin-core-dev
10162016-09-29T21:18:42  *** cryptapus has quit IRC
10172016-09-29T21:19:47  *** achow101_ has quit IRC
10182016-09-29T21:19:56  *** aureianimus has quit IRC
10192016-09-29T21:20:16  *** aureianimus has joined #bitcoin-core-dev
10202016-09-29T21:21:59  *** achow101_ has joined #bitcoin-core-dev
10212016-09-29T21:22:12  *** achow101_ has quit IRC
10222016-09-29T21:31:14  *** cryptapus has joined #bitcoin-core-dev
10232016-09-29T21:40:06  *** sipa has quit IRC
10242016-09-29T21:40:06  *** sipa has joined #bitcoin-core-dev
10252016-09-29T21:40:51  *** aureianimus has quit IRC
10262016-09-29T21:40:59  *** aureianimus has joined #bitcoin-core-dev
10272016-09-29T21:41:13  *** MarcoFalke has left #bitcoin-core-dev
10282016-09-29T21:46:23  *** cdecker has quit IRC
10292016-09-29T21:46:24  <GitHub131> [bitcoin] jnewbery opened pull request #8844: change sigops cost to sigops weight (master...sigops_weight) https://github.com/bitcoin/bitcoin/pull/8844
10302016-09-29T21:48:11  *** gabridome has quit IRC
10312016-09-29T21:49:09  <GitHub72> [bitcoin] jnewbery opened pull request #8845: Don't return the address of a P2SH of a P2SH (master...trivial-P2SH-P2SH) https://github.com/bitcoin/bitcoin/pull/8845
10322016-09-29T21:51:37  *** luke-jr has quit IRC
10332016-09-29T21:51:55  <gmaxwell> jtimon: "less flexible" -- everything is less flexible short of sending someone arbritary x86 bytecode that they run.
10342016-09-29T21:52:02  *** aureianimus_ has joined #bitcoin-core-dev
10352016-09-29T21:52:02  *** aureianimus has quit IRC
10362016-09-29T21:54:00  <jtimon> less flexible in the amount of data you store, but maybe 8 flavors can be subidivided in 16 flavors half the size as wumpus was suggeting, then 16 to 32, etc. That may be flexible enough
10372016-09-29T21:54:30  <gmaxwell> jtimon: I was recommending 2^32 'flavors' but wumpus was concerned about reducing fingerprinting.
10382016-09-29T21:55:10  <gmaxwell> the whole reason to reduce the amount was to make it more difficult to follow a node around as it changes network identity.
10392016-09-29T21:55:30  <gmaxwell> sipa: 8 isn't enough if the chain is perpetually growing.
10402016-09-29T21:56:05  <jtimon> I see
10412016-09-29T21:56:15  <sipa> yeah, increasing number, the further back you go, may make sense
10422016-09-29T21:56:34  <gmaxwell> a year from now the chain will be 200 gb, a year after 300 gb-- at that size it is larger than the most common ssd size currently. a year after that 400gb.... and at that point an 8 way split is again running common hosts out of disk even if the common ssd size has moved up to 500gb by then.
10432016-09-29T21:56:59  <jtimon> well, maybe archive nodes that don't want to store everything have to get a privacy hit
10442016-09-29T21:57:32  <gmaxwell> who will bother running one if it takes speical effort above and beyond running a node, and draws more resources?
10452016-09-29T21:57:39  <sipa> well if only we'd have a separate network for archivsl
10462016-09-29T21:57:53  <sipa> there are no privacy issues at all then
10472016-09-29T22:01:17  *** jnewbery has quit IRC
10482016-09-29T22:06:25  <gmaxwell> and no one run them.
10492016-09-29T22:06:57  <gmaxwell> s/run/running/
10502016-09-29T22:07:27  <sipa> i was about to say that separate network doesn't need to imply separate nodes
10512016-09-29T22:07:42  <sipa> but of course, that doesn't work because you'd get a privacy leak from correlating
10522016-09-29T22:08:46  <sipa> however, you can reconcile those by only having nodes with a long-term IP provide archival further back than some threshold
10532016-09-29T22:09:37  *** jannes has quit IRC
10542016-09-29T22:10:02  <gmaxwell> sipa: not just that, but if it's a special very resource intensive mode.. few will do it, pliling more resources onto it... causing fewer to do it...
10552016-09-29T22:10:37  <sipa> it's true that it's resource intensive, but it's a different kind of resources than most of the rest of running a node
10562016-09-29T22:10:43  <sipa> it needs disk space and bandwidth
10572016-09-29T22:10:45  <gmaxwell> I might think it's not over the threshold of that, except already people don't run regular nodes due to costs.
10582016-09-29T22:11:12  <sipa> rather than memory and cpu
10592016-09-29T22:11:13  <gmaxwell> which are what people usually complain about.
10602016-09-29T22:11:31  <sipa> then why aren't we seeing more pruned nodes?
10612016-09-29T22:11:57  <sipa> one reason may be that pruned nodes don't advertize, so we just don't know about them
10622016-09-29T22:12:33  <gmaxwell> because you have to edit a config file or change an obscure setting, we don't advertise it, and it breaks rescan and reindex. (which is part of why we don't really advertise it)
10632016-09-29T22:12:46  *** aureianimus_ has quit IRC
10642016-09-29T22:12:55  *** aureianimus has joined #bitcoin-core-dev
10652016-09-29T22:14:43  <sipa> well people mostly complain about the sync time for a node
10662016-09-29T22:15:14  <gmaxwell> yes, though I think thats most because so many stop there and give up before they get a chance to complain about the rest.
10672016-09-29T22:15:20  <sipa> perhaps
10682016-09-29T22:18:16  <TD-Linux> a first-run dialog box with a slider for disk usage and an estimated sync time would be very nice
10692016-09-29T22:18:49  <sipa> except the sync time does not depend on the value of the slider
10702016-09-29T22:19:57  <TD-Linux> yes, I meant it there so it'd appear at start. I guess having it in the status bar is sufficient
10712016-09-29T22:20:36  <sipa> ah
10722016-09-29T22:20:48  <sipa> well there will be an overlay with sync time indication in 0.14
10732016-09-29T22:21:16  <gmaxwell> doesn't it still incorrectly say you can't transact while syncing?
10742016-09-29T22:21:30  <sipa> we still have a lot time until 0.14
10752016-09-29T22:23:08  <gmaxwell> :)
10762016-09-29T22:23:14  <wumpus> well you can, but most people probably shouldn't do so
10772016-09-29T22:23:34  <gmaxwell> yes they should
10782016-09-29T22:23:52  <wumpus> during the initial sync they won't have any coins to send anyway, and receiving them is a bad idea as they'll only see them when the entire thing is done
10792016-09-29T22:23:55  <wumpus> oh?
10802016-09-29T22:24:02  <wumpus> why?
10812016-09-29T22:24:18  <gmaxwell> initial sync isn't my concern there:
10822016-09-29T22:24:18  <gmaxwell> probably one of the most common usage patterns for a wallet user is that you start your wallet up in order to pay someone, and it's three weeks behind. You can go ahead and pay, no problems.. why wouldn't you?
10832016-09-29T22:24:36  <gmaxwell> during initial sync you just won't have any coins, indeed. :)
10842016-09-29T22:24:48  <wumpus> the biggest problem is people giving out addresses during initial sync
10852016-09-29T22:24:52  <wumpus> then realizing how long it takes
10862016-09-29T22:25:10  <wumpus> this is what the overlay is designed to prevent
10872016-09-29T22:25:26  <wumpus> sure, you can send coins if you're three weeks behind, no problem, although fee computation could be off
10882016-09-29T22:25:39  <gmaxwell> yes, that is a large source of complaints, but we shouldn't tell people that they cant send funds already in their wallet when they start up and they're a bit behind, it's already a common mistaken belief that they cant (and then they complain about how long it takes to catch up a month of blcks)
10892016-09-29T22:25:40  <TD-Linux> the warning could be conditional on having zero funds
10902016-09-29T22:26:13  <gmaxwell> TD-Linux: the earlier warning text was fine-- saying that you won't see payments to you yet, but for some reason it was changed to say that you cannot send funds.
10912016-09-29T22:26:17  <wumpus> yeah fix one thing and they'll start complaining about another, it's a never ending source of fun...
10922016-09-29T22:26:33  <sipa> i don't think anyone will read the text anyway
10932016-09-29T22:26:47  <gmaxwell> I also complained that the text is now too long and won't get read.
10942016-09-29T22:26:50  <wumpus> of course people will read it
10952016-09-29T22:26:55  <sipa> the important thing is that it's in the way, and gives accurate (by then, hopefully) information
10962016-09-29T22:26:57  <wumpus> heck, users aren't stupid
10972016-09-29T22:26:59  <gmaxwell> The first text was better.
10982016-09-29T22:27:10  <sipa> gmaxwell: PR welcome
10992016-09-29T22:27:14  <wumpus> maybe some are, but not all of them, some will actually read and understand
11002016-09-29T22:27:18  <gmaxwell> Well I'm stupid, and looked at the notice in its updated state and didn't read the list line.
11012016-09-29T22:27:32  <gmaxwell> first*
11022016-09-29T22:27:48  <gmaxwell> because when there is too much text many people go a bit banner blind and skim past headings and such.
11032016-09-29T22:27:51  <wumpus> if we don't believe peopel actually pay attention then why do anything at all
11042016-09-29T22:28:14  <gmaxwell> saying that a wall of text is too much is not saying that people don't pay attention.
11052016-09-29T22:28:20  <wumpus> I think it's an improvement to what was there, indeed, if you want to imrpvoe further then pulls are welcome
11062016-09-29T22:28:49  <sipa> right, that's what i'm saying - having there being an overlay at all is more important than what the text says
11072016-09-29T22:29:00  <gmaxwell> and re: being able to send, people already complain that they have to wait a long time after starting to send because they already frequently mistakingly believe they can't.
11082016-09-29T22:29:00  <sipa> and we have time to improve the latter
11092016-09-29T22:29:05  <wumpus> but I'm a bit tired of people always saying "users won't read anyway" to everything that adds documentation , help or warnings
11102016-09-29T22:29:29  <wumpus> a lot of users are definitely looking for more help and guidance when they first open the program, and a bit of text helps there
11112016-09-29T22:29:40  <gmaxwell> wumpus: why should I waste my time when I point out that THE TEXT IS OUTRIGHT UNTRUE and your response is to accuse me of thinking users are stupid?  my comment was that the earlier version of the text which was simple and NOT UNTRUE was better.
11122016-09-29T22:29:54  <sipa> please guys
11132016-09-29T22:30:00  <sipa> gmaxwell: go propose something
11142016-09-29T22:30:05  <gmaxwell> I did!
11152016-09-29T22:30:15  <wumpus> gmaxwell: well if the text is wrong then it should be fixed obviously, change it to a better text
11162016-09-29T22:30:49  <wumpus> I don't know what the previous version of the text was
11172016-09-29T22:31:56  <sipa> it's been changed a dozen times in the lifetime of the pull
11182016-09-29T22:33:03  <sipa> also, it says "Spending bitcoins may not be possible until synchronization has finished."
11192016-09-29T22:33:08  <sipa> which is not untrue.
11202016-09-29T22:34:44  <gmaxwell> okay, it was changed after I last saw it.
11212016-09-29T22:35:13  <wumpus> ok that was useless :)
11222016-09-29T22:35:18  <gmaxwell> by saying 'may' which is still misleading, but worse, that text is the bold.
11232016-09-29T22:35:24  <gmaxwell> er is the only bold part.
11242016-09-29T22:35:42  <sipa> well, improvements welcome
11252016-09-29T22:35:54  <gmaxwell> So now it says "mumble mumble mumble  Spending bitcoins may not be possible during that phase!" :-/
11262016-09-29T22:36:14  <gmaxwell> it's a waste of my time, I already raised these issues and it was then merged.
11272016-09-29T22:36:45  <wumpus> it had to be merged at some point, with the idea it could be improved later
11282016-09-29T22:36:58  <gmaxwell> well to be fair the last change did improve it, its true.
11292016-09-29T22:37:38  <gmaxwell> but created the problem that if you skim it is that all you extract is that you can't spend, .. which misses the really critical thing: which is that you wallet may look empty when it isn't.
11302016-09-29T22:37:57  *** aureianimus has quit IRC
11312016-09-29T22:37:58  <wumpus> that doesn't mean it's final, most will only see the message when it is merged, and can improve it then, there are already some pulls open to improve that overlay
11322016-09-29T22:38:00  <gmaxwell> but okay, I can open a PR.
11332016-09-29T22:38:17  <wumpus> (but I don't think they change that message)
11342016-09-29T22:38:20  *** aureianimus has joined #bitcoin-core-dev
11352016-09-29T22:38:53  *** Guyver2 has quit IRC
11362016-09-29T22:39:08  <sipa> gmaxwell: i think people didn't really understand the point of your concern (i didn't): if you're looking at it from a point of view that this would be be mostly seen (and intended to convey information) during IBD, it's perfectly reasonable to warn users they won't be able to spend the money they're still to receive... and a simplification to reduce the length of the text may be warranted
11372016-09-29T22:39:31  <sipa> it's a good point that it's also seen during non-IBD
11382016-09-29T22:39:34  <gmaxwell> It will mostly not be seen during IBD.
11392016-09-29T22:39:54  <gmaxwell> during IBD sure someone will see it then, say of course I knew that (even if they didn't) minimize and go on with life. :P
11402016-09-29T22:40:13  <gmaxwell> but then users will see it every single time they start.
11412016-09-29T22:40:22  <sipa> i'm aware, you don't need to argue about this
11422016-09-29T22:40:30  <sipa> i'm just explaining why maybe you felt misunderstood
11432016-09-29T22:40:30  <gmaxwell> sorry, not arguing-- clarifying.
11442016-09-29T22:40:43  <gmaxwell> Yes, I see that and I didn't before.
11452016-09-29T22:41:19  <gmaxwell> when I first saw this PR I even took the time to go through the code carefully to check to see if there was anything that made it IBD only.
11462016-09-29T22:42:02  <gmaxwell> because I couldn't understand why people wanted the text that it had.
11472016-09-29T22:42:18  <gmaxwell> it did not occure to me that other people might be only thinking about IBD.
11482016-09-29T22:42:26  <gmaxwell> sorry for being thoughtless there.
11492016-09-29T22:42:46  <wumpus> #8805 fixed a few minor grammar nits, #8821 fixes a blocking problem with the overlay, there are no pulls yet that improve the message
11502016-09-29T22:43:17  <sipa> sorry, we (including me) aren't being careful with terminology here... IBD is also used for syncup when you were previously synced to a month ago
11512016-09-29T22:43:22  *** To7 has quit IRC
11522016-09-29T22:43:47  <wumpus> it's very easy to forget about catching up nodes
11532016-09-29T22:44:18  <wumpus> but yes we shouldn't
11542016-09-29T22:44:21  <sipa> well it's mostly designed to help with that first sync
11552016-09-29T22:45:07  <gmaxwell> more obvious to me just by chance of hering more people complain about it, also I've stopped running a node 24/7 on my laptop because I've been watching the battlestar galactica series in evenings and bitcoin interupts video playback. :)
11562016-09-29T22:45:15  <wumpus> during catch-up it's reasonably useful too, people may not know they won't see transactions newer than their sync point and worry, but yes it's mostly important for the initial IBD (lol)
11572016-09-29T22:45:21  <gmaxwell> so every time I go to use bitcoin I'm stuck waiting for it to catch up.
11582016-09-29T22:45:42  <gmaxwell> yes, we should have this message during catch up. But it's important to not make people think they can't spend funds that they can see.
11592016-09-29T22:46:03  <gmaxwell> The important message is that you may not see all payments to you yet (and you can't spend what you can't see).
11602016-09-29T22:46:31  <wumpus> bitcoin interrupts video playback? even in steady state mode?
11612016-09-29T22:46:34  <TD-Linux> one option would be to put it on the payment request generation page instead. but even in its current state it's far better than what was there before (nothing)
11622016-09-29T22:47:29  <gmaxwell> wumpus: yes. on my laptop... playback from a local file. The issue is IO or cpu related, probably the former but I haven't tested extensively to know for sure.
11632016-09-29T22:48:03  <wumpus> that's very strange. I'd expect that during intial sync when it maxes out CPU and I/O usage, but not when it's up to date
11642016-09-29T22:48:13  <gmaxwell> TD-Linux: the big problem we should be solving here is that people see a balance of zero then delete the wallet. I think thats the priority because any other issue doesn't cause irrecoverable loss.
11652016-09-29T22:48:24  *** cryptapus is now known as cryptapus_afk
11662016-09-29T22:48:43  <gmaxwell> wumpus: I notice it during ordinary computer use.. causes IO hangs, but its not irritating except when watching video.
11672016-09-29T22:49:28  <wumpus> do you have a lot of mlocked memory? is it swapping?
11682016-09-29T22:50:01  <gmaxwell> no, not swapping 8gb ram. I think that when a bunch of random writes happen it causes long delays for garbage collection in the SSD.
11692016-09-29T22:50:02  <wumpus> swapping seems to be the foremost cause of I/O related hangs here, as essentially the memory subsystem has to wait for I/O to complete
11702016-09-29T22:50:40  <wumpus> heh as if 8gb ram means no swapping these days :)
11712016-09-29T22:50:59  <gmaxwell> well on my laptop its enough most of the time.
11722016-09-29T22:51:29  <gmaxwell> The stalls seemed to get better for a while after I freed up a bunch of space and trimmed the drive, but got worse after which is why I think SSD GC plays a roll.
11732016-09-29T22:51:50  <gmaxwell> but in any case, while watching the show every block arrival causes a second-long pause in playback.
11742016-09-29T22:51:51  <wumpus> maybe someone is requesting a lot of blocks from you with a bloom filter? :-) it would be interesting to find out what your node is actually doing at those times
11752016-09-29T22:52:02  <gmaxwell> nah, outbound only.
11762016-09-29T22:52:09  <gmaxwell> I know its at the same time as blocks showing up.
11772016-09-29T22:52:19  <TD-Linux> gmaxwell, easy way to verify that would be to increase your video player's lookahead cache
11782016-09-29T22:52:21  <wumpus> ok so it's block verification, leveldb seeks
11792016-09-29T22:52:57  <gmaxwell> TD-Linux: think mpv uses non-blocking reads of the disk?
11802016-09-29T22:53:21  <TD-Linux> gmaxwell, yup it does. I've increase the setting to 10s when using sshfs and it works fine
11812016-09-29T22:54:04  <gmaxwell> in any case, performance distraction aside, when this happens I shut down bitcoind then it may stay off for a week before I need to do something with it, then waiting for it to catch up is irritating.
11822016-09-29T22:55:23  <gmaxwell> (and of course my systems performance is seriously impacted while it catches up)
11832016-09-29T22:55:27  <wumpus> yes, nothing to do about that, I guess if hybrid SPV mode is implemented it could also work during catch-up
11842016-09-29T22:55:55  <wumpus> indeed, it's either slow down the catch up or tolerate it hogging the whole system
11852016-09-29T22:58:09  <gmaxwell> I have wondered if it might be useful to split the chainstate into two parts, one with txouts created in the most recent N blocks, and one with the rest. Then on start we could just load the whole first one into the cache.
11862016-09-29T22:58:10  <wumpus> the default setting of hogging all cores during IBD/catch-up is a bit rude, certainly if it is a background process
11872016-09-29T22:58:59  *** aureianimus has quit IRC
11882016-09-29T22:59:02  *** aureianimus_ has joined #bitcoin-core-dev
11892016-09-29T22:59:04  <gmaxwell> if we did that much of the cost would then be signature valdation instead of random IO, and signature validation could run in the background, following behind the blocks.. and at lower priority.
11902016-09-29T22:59:51  <wumpus> so that would be like a 'prefer keeping recent UTXOs' cache policy?
11912016-09-29T23:00:14  <gmaxwell> I guess thats a stat that I still haven't collected.  "what is the average age-in-blocks of inputs that are consumed" (/what is the distribution of that age)
11922016-09-29T23:00:28  <gmaxwell> we know recent ones are spent more often but I don't have good numbers on it.
11932016-09-29T23:00:31  <gmaxwell> wumpus: yes.
11942016-09-29T23:02:56  <gmaxwell> probably not the highest priority improvement in any case.
11952016-09-29T23:02:58  <wumpus> well, currently the whole cache is emptied at a write, I think there are many eviction policies that would do better
11962016-09-29T23:03:32  <gmaxwell> right, also the in memory representation of the cache entries is quite inefficient.
11972016-09-29T23:04:03  <gmaxwell> so its effective size could potentially be doubled if its entries were flat allocated.
11982016-09-29T23:04:08  <wumpus> though that helps it actually being an efficient cache, a more efficient representation shouldn't come at a higher access cost
11992016-09-29T23:06:48  <wumpus> I did an experiment once with storing the UTXOs in serialized form in memory: https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db
12002016-09-29T23:09:31  <gmaxwell> interesting!
12012016-09-29T23:09:33  <wumpus> although that's for the entire UTXO set, not just a limited cache
12022016-09-29T23:10:52  <gmaxwell> yea, the on disk seralization is most efficient, but not fast.
12032016-09-29T23:11:11  *** aureianimus_ has quit IRC
12042016-09-29T23:11:19  *** aureianimus has joined #bitcoin-core-dev
12052016-09-29T23:11:25  <gmaxwell> I wouldn't be surprised though for the current in memory representation if there wasn't more bytes spend in malloc/container overhead than actual transaction data though.
12062016-09-29T23:11:29  <wumpus> but given that most time is wasted on disk seeks anyway, it may not make too much difference in practice
12072016-09-29T23:11:43  <wumpus> depends on the system...
12082016-09-29T23:12:00  <wumpus> yes, the malloc overhead is somewhat bad
12092016-09-29T23:12:13  *** aureianimus has quit IRC
12102016-09-29T23:12:25  *** aureianimus has joined #bitcoin-core-dev
12112016-09-29T23:14:08  <wumpus> but not more than the actual data size, from what I remember
12122016-09-29T23:14:54  *** aalex has quit IRC
12132016-09-29T23:15:16  <wumpus> in any case improvements are certainly possible there, without any rocket science, it's just that it's such risky code to change
12142016-09-29T23:15:53  <wumpus> if it was any other project people would have optimized the shit out of it by now
12152016-09-29T23:18:52  *** aalex has joined #bitcoin-core-dev
12162016-09-29T23:21:06  <wumpus> unfortunately the damages of a bug there are unfathomable, not just skipping a few video frames
12172016-09-29T23:22:24  <gmaxwell> well the hope I had there was that with making the cache more efficient, it could be increased in size and avoid more disk IO. :)
12182016-09-29T23:24:30  <gmaxwell> on the earlier subject of 'alt' implementations doing inadvisable things, some of them have a genius performance optimization which they're crowing about, -- where they only validate transactions that weren't already in the mempool; something we explicity decided not to do because of the long history of subtle mempool corruption issues.
12192016-09-29T23:25:36  <wumpus> exactly, there are tons of ways to optimize and get things just slightly wrong
12202016-09-29T23:25:45  <gmaxwell> I worry that there will be a race to the bottom, where by making risky / security reducing optimizations implementations will gain significant performance advantages, and suffer no cost until the inevitable spectacular failure that results.
12212016-09-29T23:26:06  <wumpus> which doesn't matter if no one runs your code anyway, but we have to be really careful
12222016-09-29T23:26:33  <gmaxwell> and being safe doesn't matter if peopel don't run it in favor of things that are faster.
12232016-09-29T23:26:43  <gmaxwell> people*
12242016-09-29T23:26:52  <wumpus> we also shouldn't overestimate how important the performance is to most users, many just run it on a server or otherwise unused computer
12252016-09-29T23:27:31  *** aureianimus_ has joined #bitcoin-core-dev
12262016-09-29T23:27:41  *** aureianimus has quit IRC
12272016-09-29T23:27:51  <wumpus> well you'd say safeness is really important, if the inevitable spectacular failure happens you don't want to be at the center of it
12282016-09-29T23:28:32  <gmaxwell> well sure. indeed.
12292016-09-29T23:28:39  <wumpus> better slow than dead :)
12302016-09-29T23:29:44  <gmaxwell> but for things like the security model change to use a 6 month old utxo commitment instead of syncing the history... the potential for a spectacular failure there which a more conservative approach could have stopped is negligible.
12312016-09-29T23:29:52  <TD-Linux> gmaxwell, still think you should verify that it's actually IO that's the problem before going too deep :)
12322016-09-29T23:29:56  <gmaxwell> And if we don't investgate things like that, someone will do something dumber.
12332016-09-29T23:30:18  <wumpus> TD-Linux: yes, measuring is better than assumptions :)
12342016-09-29T23:30:28  <gmaxwell> TD-Linux: Oh IO is an issue for sure regardless of whats causing my mpv stalls. I'll find out tonight (I'm not going to sit here and watch video for an hour just to check)
12352016-09-29T23:31:54  <gmaxwell> TD-Linux: SSD vs a fast spinning disk with small dbcache here is a <4 hour sync (from a local peer) vs a >9hour sync.
12362016-09-29T23:32:26  <wumpus> and sure, it's good to investigate things like that
12372016-09-29T23:33:00  <wumpus> I/O is absolutely a problem sometimes, leveldb generates *tons* of seeks and small reads, better caching would help avoid some of that
12382016-09-29T23:34:27  <wumpus> I had no luck with other databases, I remember trying with lmdb at some point, which was faster in reading, but instead... does tons of seeks and small writes at write, so it just moves the problem
12392016-09-29T23:35:21  <TD-Linux> yeah, on a SSD at least, the former is much less detrimental to system latency
12402016-09-29T23:35:27  <wumpus> indeed!
12412016-09-29T23:35:52  <gmaxwell> for operation at the tip, using the mempool instead of validating would be a big aid... but the safty of that remains dubious. :)
12422016-09-29T23:36:07  <sipa> gmaxwell: i believe for the mempool it's approximately a factor 2 overhead
12432016-09-29T23:36:24  <sipa> gmaxwell: that that is also indexes, orderings, accounting, ...
12442016-09-29T23:38:07  <wumpus> yes that remains dubious, especially with regard to isFinal and such
12452016-09-29T23:38:27  <wumpus> I mean there is a part of transaction validation that can be obviously cached, and a part that may change in time
12462016-09-29T23:39:03  <gmaxwell> fortunately the MTP change made that much safer.
12472016-09-29T23:41:48  <gmaxwell> e.g. before a block could come in with a time before your local time, and contain txn which are isfinal invalid according to the block but okay with respect to the local time, and you'd accept it. I wonder if the alt implementations had that bug.
12482016-09-29T23:48:12  <wumpus> I remember that one, tricky... and there may be more problems of that kind not yet found
12492016-09-29T23:49:43  <gmaxwell> looks like their change dodged it, because the finality test is in the ContextualCheckBlock, and the bypass patch only bypasses checkinputs...
12502016-09-29T23:49:55  <gmaxwell> (which also means that it doesn't manage to avoid accessing the utxo cache entries)
12512016-09-29T23:51:16  <wumpus> I just realized, if the problem is that the block validation hiccups other things happening on your PC, the solution may be actually to slow it down :)
12522016-09-29T23:52:00  <wumpus> put a small sleep between each UTXO lookup, limit the validation to one thread
12532016-09-29T23:52:38  <wumpus> not something you'd want to do during initial sync if you're waiting for it, but if you don't care and it runs in the background...
12542016-09-29T23:54:14  <wumpus> after all you run it to keep up, you don't need to outrace it
12552016-09-29T23:56:05  <gmaxwell> I've thought before that if we have bandwidth limiting enabled we should delay announcement of new blocks to reduce the number of peers that request them from us... but slowing down the validation would work as well.
12562016-09-29T23:56:18  <gmaxwell> small sleeps perhaps aren't so good because it may busy spin. :P
12572016-09-29T23:57:44  <wumpus> heh, not that small
12582016-09-29T23:58:57  <wumpus> or use some OS-dependent way to reduce the I/O priority
12592016-09-29T23:59:35  <wumpus> as long as it's done by the time the next block comes in, so taking 10 minutes would take it too far :)