19:01:56 <wumpus> #startmeeting
19:01:56 <lightningbot> Meeting started Thu Mar  7 19:01:56 2019 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:56 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:01:59 <provoostenator> hi
19:02:00 <jonasschnelli> Hi
19:02:00 <gmaxwell> I'd love to be using epoll/kqueue where available, -- but  compared to many other possible improvements, I think these things would just be nice to have changes that make the system prettier and more technically perfect.
19:02:13 <achow101> hi
19:02:25 <meshcollider> hi
19:02:40 <jonasschnelli> topic proposal: txindex and prune
19:02:45 <jonasschnelli> (for later)
19:02:50 <wumpus> this is true for, say, simple HTTP servers that can handle connections at network-bound speed, we're definitely CPU bound in many cases
19:02:52 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb
19:02:56 <cfields_> At this point I obviously agree with all of the above.
19:02:57 <sipa> gmaxwell: i think people looking into integrating these (through libevent based network code or elsewhere) is useful; we just shouldn't let it stand in the way of protocol improvements
19:03:33 <gmaxwell> cfields_: it's not that I was right, hell I didn't know it would go out the way it did. :) I just don't want to repeat what happened before, and I can't see how (right now) that sort of change should be a priority worth delaying anything.
19:03:45 <wumpus> #topic P2P networking
19:03:59 <gmaxwell> sipa: sure, though I'm sure most people would prefer to work on suff that is going to get integrated soon! :P
19:04:07 <sipa> of course
19:04:46 <jonasschnelli> In case anyone wants to review the new v2 protocol (BIP151), including the new special ChaCha20Poly1305@bitcoin AEAD,... its here -> https://gist.github.com/jonasschnelli/c530ea8421b8d0e80c51486325587c52
19:04:59 <gmaxwell> oh interesting, cool. Will do.
19:05:31 <sipa> jonasschnelli: interesting; this is the born-encrypted design, rather than upgrade-existing-connection one?
19:05:51 <jonasschnelli> yes. The born encrypted
19:05:59 <phantomcircuit> hi
19:06:06 <jonasschnelli> with NODE_ENCRYPTED and the 32byte key exchange before everything else
19:06:15 <jonasschnelli> but fallback to a version message
19:06:19 <wumpus> #topic BIP151
19:06:44 <echeveria> jonasschnelli: (minor, why is the command still possible to be null padded ASCII?)
19:06:44 <sipa> at this point it may be worth having it as a separate bip
19:07:13 <jonasschnelli> echeveria: in v2 its a varstring or a short varint
19:07:16 <jonasschnelli> no padding
19:07:32 <echeveria> ok, same question.
19:07:34 <jonasschnelli> sipa: what as a seperate BIP? the handshake, or the NODE_ENCRYPTED?
19:08:11 <sipa> jonasschnelli: it looks like you plan to overwrite BIP151... given that it already has a bip number, and you're substantially changing the design, maybe it should be a separate one
19:08:32 <sipa> (and abandon bip151)
19:08:35 <jonasschnelli> Yes. I thought the same...
19:08:38 <gmaxwell> +1
19:08:48 <jonasschnelli> Though we must discourage to use BIP151
19:09:03 <gmaxwell> Sure.
19:09:05 <phantomcircuit> jonasschnelli, why is the command ascii and not just a integer?
19:09:16 <jonasschnelli> Also, there is a BIP150 weakness if used with plain (old) BIP151
19:09:21 <jonasschnelli> phantomcircuit: it is
19:09:28 <jonasschnelli> read the BIP linked above
19:09:48 <echeveria> from the BIP. "1-13 	encrypted command 	variable 	ASCII command (or one byte short command ID)"
19:09:50 <wumpus> phantomcircuit: extensibility I guess, names are much easier to work on on parallel instead of having to reserve small integers
19:10:12 <jonasschnelli> ASCII commands are still possible
19:10:32 <gmaxwell> Basically for extensibility, the existing normal commands get short IDs but then its possible to have longer ones for new/infrequent commands.
19:10:33 <wumpus> right
19:10:36 <sipa> this is probably not the place to discuss the details of the bip
19:10:46 <jonasschnelli> yes. Will move it to the ML
19:10:46 <gmaxwell> if you only have a small integer you end up with an allocation problem.
19:11:05 <jonasschnelli> I will rebase and overhaul the implementation #14032 asap
19:11:08 <gribble> https://github.com/bitcoin/bitcoin/issues/14032 | Add p2p layer encryption with ECDH/ChaCha20Poly1305 by jonasschnelli · Pull Request #14032 · bitcoin/bitcoin · GitHub
19:11:35 <jonasschnelli> In the meantime, reviews welcome for the puzzles required for the p2p enc. -> #15512, #15519, #14047, #14049
19:11:36 <gribble> https://github.com/bitcoin/bitcoin/issues/15512 | Add ChaCha20 encryption option (XOR) by jonasschnelli · Pull Request #15512 · bitcoin/bitcoin · GitHub
19:11:38 <gribble> https://github.com/bitcoin/bitcoin/issues/15519 | Add Poly1305 implementation by jonasschnelli · Pull Request #15519 · bitcoin/bitcoin · GitHub
19:11:40 <gribble> https://github.com/bitcoin/bitcoin/issues/14047 | Add HKDF_HMAC256_L32 and method to negate a private key by jonasschnelli · Pull Request #14047 · bitcoin/bitcoin · GitHub
19:11:42 <gribble> https://github.com/bitcoin/bitcoin/issues/14049 | Enable libsecp256k1 ecdh module, add ECDH function to CKey by jonasschnelli · Pull Request #14049 · bitcoin/bitcoin · GitHub
19:11:46 <jonasschnelli> (sry for the wall)
19:12:24 <gmaxwell> We should talk about a meta issue here. Whats our position on e.g. voskuil opposing any encryption?  My view is that he's free to not implement it, but we shouldn't let generalized opposition stand in the way of doing something like that.
19:12:44 <jonasschnelli> voskuil seems to be okay with the new BIP since its now fully backward comp.
19:12:52 <instagibbs> Is it authentication or encryption he has issues with?
19:13:05 <jonasschnelli> Auth. is BIP150 which is still in discussion
19:13:09 <gmaxwell> It was always backwards compatible... but okay...
19:13:23 <sipa> instagibbs: his position is that authentication without encryption is pointless (i agree, it mostly is), and that authentication of bitcoin connections is a slippery slope
19:13:24 <jonasschnelli> BIP151 (or the new #) is opportunistic encryption
19:13:29 <gmaxwell> instagibbs: he argued that encryption was pointless without authentication and authentication was bad.
19:13:37 <gmaxwell> what sipa says.
19:13:44 <gmaxwell> but if thats changed. fine!
19:14:01 <sipa> eh, encryption without authentication
19:14:07 <gmaxwell> I'd just ask that if we move discussion to the ML we not let stupid debates mire it down.
19:14:09 <instagibbs> sipa, yeah i figured :)
19:14:18 <wumpus> I... don't understand why such a high-level discussion of the desirability of those things comes now, while BIP150/151 have existed for ages
19:14:19 <jonasschnelli> I think there are basic benefits of encryption without authentication... but of course, with autrhentication there are more possibilities,... but we need to create puzzle piece by puzzle piece
19:15:07 <provoostenator> I like being able to detect when my ISP starts messing with my Bitcoin P2P traffic :-)
19:15:20 <jonasschnelli> indeed
19:15:23 <wumpus> would need to be a really good reason it's bad to stop now
19:15:42 <jonasschnelli> I like that my ISP(s) know that I can detect in case they want mess my p2p traffic
19:15:48 <sipa> jonasschnelli: fwiw, i do have a writeup on countersign (which allows authentication without a MitM knowing it was even attempted); it turns out to be fairly hard for multiple keys simultaneously, but for one key it's pretty simple; https://gist.github.com/sipa/d7dcaae0419f10e5be0270fada84c20b
19:15:49 <gmaxwell> yes yes, we're all in agreement here.
19:16:07 <sipa> jonasschnelli: also, hasn't had thorough review yet
19:16:15 <jonasschnelli> sipa: thanks... i'll look at it in a free minute
19:16:28 <gmaxwell> I was assumping sipa and I would advance countersign once the oppturnistic encryption was in.
19:16:50 <jonasschnelli> Yes. There is the possibility of multiple authentication shemes
19:16:53 <jonasschnelli> *schemes
19:17:26 <wumpus> I could see how a "connect only to authenticated, known peers" becoming general could be problematic, though, but I don't think it was ever to be the only option?
19:17:55 <sipa> jonasschnelli: the idea behind countersign is that you'd always use it; even when no authentication is desired (in which case you'd run it with a random pubkey)
19:18:11 <sipa> but a MitM can't tell whether it's for a real key or not
19:18:40 <gmaxwell> So that a MitM can't only selectively tamper with unauthenticated links.
19:18:45 <jonasschnelli> We currently authenticate peers by ips IPv4/IPv6... so..
19:18:52 <sipa> haha yes
19:18:53 <jonasschnelli> (addnode / connect)
19:19:09 <gmaxwell> So anyone using authentication creates a halo of protection against MitM for everyone.
19:19:11 <jonasschnelli> We don't change the uses cases
19:19:56 <jonasschnelli> Authentication just makes things more secure and reliable that are already possible
19:20:07 <gmaxwell> Indeed, we're clear on this. I only raised the concern in the context of the mailing list because of bad expirences before discussing this.
19:20:08 <sipa> preaching to the choir :)
19:21:17 <gmaxwell> I think if the industry (or other implementers) opposes the idea in general: we don't care, obviously we'll continue to support the old procol so long as it's needed, and obviously we'd hear out any concerns. But fundimentally P2P is non-consensus, we can implement any additional protocol we want and still interpo wih anyone who doesn't want to implement it.
19:21:55 <wumpus> right
19:22:01 <gmaxwell> So someone showing up demanding we don't implement crypto at all or use TLS can just be told after we hear their case, sorry, this project isn't interested in that.
19:22:56 <gmaxwell> jonasschnelli: the particular thing about countersign is that it makes it less attractive to MitM anyone, even people not actually using it. Though to get that benefit it has to be 'on' all the time, even when not used.
19:25:08 <gmaxwell> I think that covers it? jonasschnelli's new writup needs review.
19:25:12 <wumpus> next topic I guess
19:25:19 <jonasschnelli> yes
19:25:21 <wumpus> #topic txindex and prune (jonasschnelli)
19:25:37 <gmaxwell> (and people might want to look at sipa's countersign writeup)
19:25:54 <jonasschnelli> I think there is demand to rund the txindex on pruned peers....
19:26:01 <jonasschnelli> *run
19:26:14 <jonasschnelli> I know its for development purposed only..
19:26:17 <gmaxwell> I don't understand how that makes logical sense?
19:26:42 <gmaxwell> txindex works by accessing the blocks, so what exactly would txindex+prune mean?
19:26:53 <jonasschnelli> Users running low cost devices (with pruning) may want to look up txids on their own system rather then use a blockexplorer
19:26:53 <gmaxwell> just returning errors on inaccessible blocks?
19:27:05 <achow101> jonasschnelli: I feel like people who want to do that have a fundamental misunderstanding of what pruning and txindex do
19:27:08 <jonasschnelli> The idea is that the index points to a blockhash, position (in the end)
19:27:12 <sipa> in a manual pruning it may make sense; some client applications knows how far it needs blocks, prunes them, but still expects to be able to lookup transactions in the unpruned one
19:27:14 <instagibbs> with manual pruning it can make sense
19:27:15 <jonasschnelli> So you can fetch it via p2p on pruned peers
19:27:18 <instagibbs> jynx sipa
19:27:30 <jonasschnelli> Its not efficient, its not fast... but it works for pesonal debug uses cases
19:27:44 <gmaxwell> ugh, fetching blocks in response to queries sounds kind of abusive on he network.
19:27:57 <gmaxwell> s/he/the/
19:27:59 <sipa> jonasschnelli: i don't understand the p2p side of things here
19:28:09 <provoostenator> That used to be useful for Lightning, but now there's a proposal to add SPV proofs to gossip, it's probably no longer useful for that.
19:28:15 <gmaxwell> sipa: he is suggesting you resolve missing blocks by fetching the block.
19:28:29 <sipa> provoostenator: oh that's great to hear
19:28:57 <jonasschnelli> sipa: well,.. if the txindex resolves to blockhash/pos, one may want to fetch the tx, and since blocks are not available, fetch a single block in order to decompose and display the tx
19:29:03 <provoostenator> It's mentioned in the Lightning 1.1 meeting docs, not sure if anyone implemented it, but that's really useful for running Lightning on pruned nodes.
19:29:09 <gmaxwell> jonasschnelli: I suspect that if people start doing this, it would probably hasten the future where there are few to no non-limited peers on the future.
19:29:43 <sipa> jonasschnelli: so you expect the txindex to cover even the pruned blocks?
19:29:47 <gmaxwell> provoostenator: if you have a pruned node, why doesn't it just check the channel against the utxo set?
19:30:27 <jonasschnelli> sipa: initially, allow to move the index from disk pos to blockhash/pos to allow to resolve to something that is useful if you have the blocks _not_ on disl
19:30:40 <gmaxwell> Txindex being able to return txes or a "pruned error" when it hits a pruned block seems like an obvious +1 to me if it would actually be useful to anyone.. (though I do have a little doubt since we didn't really like 'probablistic' RPCs in the past)
19:30:40 <sipa> jonasschnelli: meh
19:30:59 <sipa> jonasschnelli: i think a txindex that just covers the blocks you have may make sense if there's a use case
19:31:05 <instagibbs> gmaxwell, gettxout is a bit weird in some uses.
19:31:10 <provoostenator> gmaxwell: I believe Lightning gossip messages don't contain the full transaction id, they use a short notation block height + part of tx id
19:31:19 <instagibbs> sipa, liquid could have used it(we rolled our own instead)
19:31:21 <gmaxwell> provoostenator: oy
19:31:43 <sipa> jonasschnelli: but if the txindex must cover all of history, it breaks the advantage that pruning was supposed to give
19:31:47 <jonasschnelli> sipa: I think the diskpos approach is great for non pruned peers,.. I don't propose to change this,.. just allow an alternative index value of hash/pos
19:31:56 <provoostenator> gmaxwell: https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md#requirements
19:32:03 <gmaxwell> provoostenator: that sounds like a protocol flaw in lightning. (and an example of the kind of design damage txindex does.. :P )
19:32:22 <jonasschnelli> Right now, pruned peers have no possibility to securily fetch a transaction by its ID,... they need to switch to a centralized API
19:32:23 <instagibbs> merkle proofs should fix that eh?
19:32:29 <jonasschnelli> and they did verify the blocks
19:33:02 <jonasschnelli> allowing the alternative blockhash/pos instead of diskpos would give the an opportunity to lookup a txid
19:34:33 <jonasschnelli> And its just so that users still want to fetch transactions not indexed by the wallet for semi-experimental purposes like lightning, etc.
19:35:07 <gmaxwell> that seems like it would just prolong eventually fatal but avoidable design flaws... like assuming you have a txindex.
19:35:30 <jonasschnelli> I agree for productive systems,...
19:35:54 <gmaxwell> txindex = eventually rely on a centeralized service, -- we've always known this.  By twiddling this or that you can change when you have to give up on not using a centeralized service...
19:36:28 <gmaxwell> Expecting random peers to serve up random blocks to you just to fetch a transaction is a pretty costly way to forestall the inevitable.
19:36:44 <jonasschnelli> IMO its already done on the p2p network...
19:36:55 <jonasschnelli> Also, compact block filters are pretty much the same
19:37:11 <gmaxwell> Yes, they make the data available but if people start using it, it'll eventually have to go away.
19:37:35 <gmaxwell> jonasschnelli: yes, which have been problematic, though disuse of them has limited the amount of problems they cause.
19:37:58 <provoostenator> (by the way, we skipped high prioirity issues and 0.18 release topics)
19:38:41 <wumpus> provoostenator: yea because everyone was already discussing things when the meeting started, if you have anything to discuss re: 0.18 please let us know
19:38:47 <jonasschnelli> I see all implications and somehow I think adding the p2p part could be done in a second step. Allowing the txindex to be hash/pos based would give pruned peer alternatives (not only the p2p fetch method). But I agree about the "meh",... but I personally see the usefulness
19:38:55 <gmaxwell> jonasschnelli: just your figures the other day were showing that >90% of your bandwidth is already expended serviging historic blocks.
19:38:56 <provoostenator> wumpus: I don't have anything
19:39:06 <jonasschnelli> gmaxwell: fair point
19:39:09 <wumpus> PSA: 0.18.0rc1 binaries were uploaded today and release announcement posted
19:39:16 <gmaxwell> jonasschnelli: it doesn't need to be hash based, fwiw, it could just be a reference into the blockindex I think, that would avoid bloating it.
19:39:16 <midnightmagic> \o/
19:39:49 <jonasschnelli> gmaxwell: yes. I think it has to to keep it compact. Was simplifying it for discussion purposes
19:40:23 <jonasschnelli> It would be more or less the same diskspace (for the index),.. also same database structure (reuse of existing fields)
19:40:42 <jonasschnelli> but I accept the "meh" but won't stop bother about it. :)
19:41:32 <gmaxwell> So there are two possibilties for pruned + txindex = whole tx index but returns "in pruned block X" when pruned, or just a txindex of the unpruned portion.  The former will radically increase disk usage, since it undermines pruning.  The latter seems easily viable, I'm not sure is actually useful to anyone (has anyone asked for something like that?)
19:42:01 <wumpus> re: "high priority for review" we haven't discussed that in the meeting for a while because around the 0.18.0 release, the obvious high-priority PRs are the ones tagged with that, but maybe now after the branch-off and rc1 is the time to start again (or next week)
19:42:20 <jonasschnelli> The main driver behind this are plug-n-play nodes (CasaHODL like consumer devices)
19:42:29 <gmaxwell> Either are better than the current "you just cant' combine these options", but dunno how important the improvement would be.
19:42:58 <gmaxwell> jonasschnelli: that _is_ production...
19:43:01 <provoostenator> jonasschnelli: I'm not convinced these plug-n-play nodes needs indexes.
19:43:10 <jonasschnelli> gmaxwell: its a debug option in a production device
19:43:28 <jonasschnelli> gmaxwell: thanks for the "in pruned block X" thought,... let me follow this a bit..
19:43:36 <jonasschnelli> I think there are other topics
19:43:46 <gmaxwell> (aside, CasaHODL has a 1TB drive.)
19:43:54 <gmaxwell> jonasschnelli: k
19:44:08 <jonasschnelli> gmaxwell: yes. But its a lame device,... I think we will see a bunch of fast devices with <64GB SSDs
19:44:20 <wumpus> are there other topics?
19:45:06 <gmaxwell> I would like to leave for lunch. :P
19:45:14 <jonasschnelli> ack
19:45:23 * sipa is generally in favor of food
19:45:25 <wumpus> ok :D
19:45:40 <wumpus> #endmeeting