19:01:14 <wumpus> #startmeeting
19:01:14 <lightningbot> Meeting started Thu Sep  5 19:01:14 2019 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:14 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:01:15 <sipa> meeting?
19:01:18 <jonasschnelli> hi
19:01:21 <sipa> hi
19:01:29 <achow101> hi
19:01:31 <aj> hola
19:01:32 <MarcoFalke> hoy
19:01:48 <meshcollider> hi
19:02:03 <moneyball> hi
19:02:11 <instagibbs> hi
19:02:16 <wumpus> there are three proposed topics in https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a
19:02:32 <wumpus> - proposed by jnewbery:  review/merge #16704 or #16713 to remove worrying "unknown new rules activated (versionbit 1)" warning
19:02:34 <wumpus> - proposed by fjahr: Rolling UTXO set hash
19:02:34 <gribble> https://github.com/bitcoin/bitcoin/issues/16704 | Dont warn about activated buried BIP 9 deployments by achow101 · Pull Request #16704 · bitcoin/bitcoin · GitHub
19:02:36 <gribble> https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub
19:02:36 <wumpus> - proposed by achow101: avoid loading every wallet transaction into memory
19:02:43 <jeremyrubin> hi
19:02:48 <fjahr> hi
19:02:51 <jonatack> hi
19:03:03 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral
19:03:42 <wumpus> but let's start with the usual
19:03:47 <wumpus> #topic High priority for review
19:04:04 <wumpus> 7 blockers, 7 things chasing concept ACK https://github.com/bitcoin/bitcoin/projects/8
19:04:09 <gleb> hi
19:04:18 <BlueMatt> I think its already there, but https://github.com/bitcoin/bitcoin/pull/16421 is close to landing and I still really want it for 19
19:04:20 <wumpus> note that the feature freeze for 0.19 is in 10 days
19:04:23 <BlueMatt> (and its a small diff!)
19:04:32 <wumpus> so we likely want to prioritize features that are close to ready now
19:04:54 <wumpus> right
19:05:08 <wumpus> would be nice if that makes it in
19:05:32 <gleb> #16702 is done code-wise (I think),  but perhaps it's suboptimal to add it to high prio at this point when we're close to the feature freeze.
19:05:35 <gribble> https://github.com/bitcoin/bitcoin/issues/16702 | p2p: supplying and using asmap to improve IP bucketing in addrman by naumenkogs · Pull Request #16702 · bitcoin/bitcoin · GitHub
19:05:47 <wumpus> also #15759
19:05:51 <gribble> https://github.com/bitcoin/bitcoin/issues/15759 | p2p: Add 2 outbound block-relay-only connections by sdaftuar · Pull Request #15759 · bitcoin/bitcoin · GitHub
19:06:22 <wumpus> gleb: yes, we might want to keep that for 0.20
19:07:31 <wumpus> also requires too much review to still make it to 0.19 anyway
19:07:38 <wumpus> still, great to hear you're making progress!
19:08:18 <sipa> yeah, that sounds like too close to make it
19:08:35 <jeremyrubin> Not super critical, and it's relatively new so not much time for review, but would help me to get  #16766 as my work on OP_SECURETHEBAG wallet support depends on it
19:08:37 <gribble> https://github.com/bitcoin/bitcoin/issues/16766 | wallet: Make IsTrusted scan parents recursively by JeremyRubin · Pull Request #16766 · bitcoin/bitcoin · GitHub
19:08:38 <sipa> i'll review 15759 again
19:08:46 <BlueMatt> 16702 is probably a bit far review-wise, though it would be nice, but 15759 is also really close and should god
19:08:47 <jeremyrubin> And I think it is a bug
19:09:57 <sipa> bug fixes can go after the feature freeze
19:10:11 <wumpus> yes
19:10:32 <jeremyrubin> wasn't sure as it's a substantial behavior change for wallet, but fine :)
19:10:45 <wumpus> they can go any time ("scan parents recursively" sounds scary to me though, performance wise :)
19:11:04 <sipa> it's cached to avoid exponential blowup
19:11:10 <sipa> but otherwise, yeah
19:11:15 <instagibbs> also depends on how new the bug is. I *think* it's ancient behavior that "normally" never hits
19:11:16 <wumpus> phew
19:11:17 <instagibbs> anyways
19:11:46 <sipa> instagibbs: agree
19:12:16 <jeremyrubin> instagibbs: I think that is correct
19:12:29 <instagibbs> with fancier wallet setups we may actually hit it :)
19:12:46 <wumpus> anyhow I've added it to high priority for review as requested, people can choose for themselves what to review
19:13:26 <wumpus> #topic remove worrying "unknown new rules activated (versionbit 1)" warning (jnewbery)
19:13:30 <wumpus> yes please
19:13:54 <wumpus> so I guess the disussion is: #16704 or #16713
19:13:55 <gribble> https://github.com/bitcoin/bitcoin/issues/16704 | Dont warn about activated buried BIP 9 deployments by achow101 · Pull Request #16704 · bitcoin/bitcoin · GitHub
19:13:56 <gribble> https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub
19:14:24 <achow101> 16713 is the simpler, easier to review fix
19:14:52 <achow101> But I think it would be better to eventually get rid of these deployment parameters and have a more permanent solution that lets us reuse bits in the future
19:15:19 <wumpus> what disadvantages does it have compared to the other one?
19:15:25 <MarcoFalke> bits can be reused as long as they don't overlap in time
19:15:27 <aj> 16713 is updated as of a few hours to be even simpler; i think it lets us reuse bits fine?
19:15:42 <wumpus> e.g. why are there two open at all?
19:16:00 <wumpus> MarcoFalke: yup
19:16:10 <MarcoFalke> I already closed the one by achow101 *hides*
19:16:26 <aj> MarcoFalke: smooth
19:17:24 <wumpus> ok, that concludes the discussion then I guess :)
19:17:26 <achow101> my understanding of how consensus.vDeployments worked was that you couldn't define two forks with the same bit since they'd have to occupy the same index position and that's not possible
19:17:40 <wumpus> oh
19:17:42 <achow101> but 16713 has changed since I last reviewed it and it looks very different
19:17:43 <wumpus> sorry
19:18:06 <MarcoFalke> oh, maybe you are right when it comes to the implementation. Though, BIP9 does allow it
19:18:21 <wumpus> BIP9 definitely allows it, that was an important part of the design
19:18:25 <aj> achow101: vDeployments just matches the enum, the actual bits used are independant, and just have to not overlap per their corresponding timestamps
19:18:51 <sipa> i haven't looked at the code in a while, but it was certainly intended to permit reuse of bits
19:19:26 <sipa> the deployments array index is independent from the bip9 bit position
19:19:38 <instagibbs> yep
19:19:57 <aj> achow101: see VersionBitsConditionChecker::Mask in versionbits.cpp, it pulls out the .bit field from BIP9Deployment struct
19:21:08 <achow101> i'm probably wrong
19:21:24 <instagibbs> It was confusing the first time I read it, I came to same wrong conclusion
19:22:10 <aj> achow101: there's also a unit test for overlapping bit usage on mainnet in src/test/versionbits_tests.cpp, "Verify that the deployment windows of [...]"
19:22:26 <wumpus> if even experienced developers get confused by it, some documentation/comments might help in that case
19:24:04 <wumpus> #action please review #16713 so that it can be merged asap
19:24:04 <MarcoFalke> I think it shouldn't matter in practice, hopefully there are less than 27 softforks in the future
19:24:06 <gribble> https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub
19:24:27 <wumpus> given that 'the future' is unbounded, that's a difficult statement
19:25:07 <wumpus> #topic Rolling UTXO set hash (fjahr)
19:25:40 <fjahr> Did anyone have time to look at the proposal? Any questions?
19:25:54 <achow101> link?
19:26:01 <instagibbs> fjahr, pitch it to us in a few sentences :)
19:26:23 <fjahr> https://gist.github.com/fjahr/fa4892874b090d3a4f4fccc5bafa0210
19:26:24 <sipa> fjahr: it's not clear to me exactly what you're proposal :)
19:26:32 <fjahr> I have picked up Pieter Wuille's proposal from 2017 to use a rolling hash for the UTXO set hash. It deals with the problem of a long computation time of the UTXO set hash which results in a slow RPC call gettxoutsetinfo (can take several minutes depending on hardware). I investigated three hash functions: MuHash, ECMH and LtHash and started implementing them in Bitcoin Core for comparison. However only MuHash
19:26:32 <fjahr> has a rolling hash implementation so far and my LtHash code is not as optimized as MuHash and ECMH. I am looking for feedback on concept, which choice to make for the hash function and implementation details before filing a PR to Bitcoin Core.
19:27:04 <sipa> it's a complicated question, as the right design may depend on how we intend to use the construction
19:27:12 <wumpus> fjahr: thanks for picking it up!
19:27:33 <MarcoFalke> I think one use case is assumeutxo
19:28:01 <fjahr> sipa: by construction you mean the hash, right?
19:28:08 <sipa> fjahr: right
19:28:24 <sipa> assumeutxo (at least with a from-network-sync approach) will probably need more than just a single hash, as you want to be able to verify chunks etc
19:28:58 <sipa> fjahr: if i recall correctly, the biggest question i had was how to prioritize computation time vs use-from-cache time
19:29:44 <sipa> ECMH is slower to compute, but very fast to use from cached values (e.g. if you have precomputed the "diff" ECMH hash a transaction has, applying to a global sum is super fast)
19:29:51 <sipa> but MuHash is faster is overall computation time
19:31:21 <fjahr> From my benchmarks ECMH was faster overall but I am not sure why MuHash did not perform as you expect in you Mail
19:32:07 <sipa> where do you see a discrepancy? in the hash-to-group-element operation, or in the multity?
19:32:11 <sipa> *multiply?
19:32:24 <sipa> actually i think the discussion of what hash to pick is less important for this meeting
19:32:34 <sipa> we should probably focus on ways to integrate
19:33:16 <fjahr> ok, and also if there is enough interest for this
19:33:25 <aj> it's super cool
19:33:48 <jeremyrubin> Are both constructions one-way?
19:33:55 <jeremyrubin> How does that interplay with reorgs
19:33:58 <fjahr> In terms of integration I chose to implement this as an index, any feedback on this?
19:34:11 <jeremyrubin> Maybe I should read more out of band of here...
19:34:12 <sipa> fjahr: that really depends what for
19:34:56 <fjahr> jeremyrubin: hashes can also be removed again so no problem with reorgs
19:35:00 <sipa> i think as an index it's hard to use precomputation
19:35:15 <sipa> which you really want if you want the rolling part
19:37:27 <fjahr> sipa: what would you suggest in terms of integration?
19:38:04 <sipa> that really depends on what we want to use it for
19:39:17 <aj> sipa: could have it rolling, but in it's own slightly delayed thread like tx indexes, without worrying about precomputation, at least if you don't want to enforce it in consensus
19:39:39 <wumpus> there's 20 minutes to go and still a topic left, let's move on?
19:39:52 <sipa> fjahr: let's talk more after the meeting
19:40:00 <fjahr> sipa: sure
19:40:13 <wumpus> #topic avoid loading every wallet transaction into memory (achow101)
19:40:18 <wumpus> thanks
19:40:18 <sipa> btw, the secp ECMH code you have looks great
19:40:37 <achow101> This is a wallet topic, and probably better for the wallet meeting, but that's next week..
19:40:50 <wumpus> is there any hurry? :-)
19:41:00 <achow101> I was thinking about ways to reduce the memory footprint of loaded wallet files
19:41:17 <wumpus> concept ACK anyhow
19:41:24 <jonasschnelli> jup. also ack
19:41:36 <sipa> achow101: seems hard
19:41:42 <achow101> I was wondering if anyone who was more familiar with the wallet tracking part of the wallet knew if this was attempted before or would majorly break something?
19:42:00 <wumpus> it always seemed unnecessary to me to keep all transactions, even historical ones with all outputs spent, in memory
19:42:15 <jonasschnelli> There is a PR where all wallettxns where kept in mem
19:42:31 <jonasschnelli> +different DB formst
19:42:34 <wumpus> but yes, it's such a part of the current wallet design, it's definitely not going to be easy
19:42:49 <jonasschnelli> #5686 (very old)
19:42:51 <gribble> https://github.com/bitcoin/bitcoin/issues/5686 | [Wallet] replace BDB with internal append only (logdb) backend by jonasschnelli · Pull Request #5686 · bitcoin/bitcoin · GitHub
19:42:53 <sipa> achow101: you mean not _show_ them anymore
19:42:59 <sipa> or just not load them in memory?
19:43:15 <achow101> not load them into memory, so it would read from disk when the full tx is needed
19:43:21 <wumpus> right
19:43:26 <achow101> I plan on just keeping UTXOs and txids in memory
19:43:40 <sipa> achow101: i wish you good luck :)
19:43:44 <jonasschnelli> heh
19:43:46 <wumpus> outward behavior probably shouldn't change
19:44:05 <jonasschnelli> Why not just loading everything into memory?
19:44:15 <jonasschnelli> I think 1000+ wallets are OOS
19:44:26 * jeremyrubin wonders how big the largest wallet is
19:44:37 <sipa> OOS?
19:44:41 <jonasschnelli> out of scope
19:44:42 <jeremyrubin> out of scope
19:44:48 <wumpus> there are some heavy users of bitcoin core which have to re-cycle their wallet once in a while because it becomes to big
19:45:00 <sipa> i don't think memory usage is the problem there
19:45:10 <achow101> big wallets also take a while to load, although I don't expect this to effect that
19:45:12 <sipa> just the size of the maps that's being traversed for a multitude of operations
19:45:18 <wumpus> loading time is likely the problem, yes
19:45:31 <wumpus> then again that's directly related
19:45:37 <achow101> ideally this will also reduce the time it takes to calculate things like balances since not every single transaction will be iterated over
19:45:54 <wumpus> yes, exactly
19:45:56 <sipa> achow101: i'm very very scared of things like that
19:45:59 <jarthur> Are exchanges and other large-scale wallet-holders recommended to use Core purely for networking, and maintain their own non-Core wallets?
19:46:01 <sipa> it needs a completely new design i'm afraid
19:46:16 <achow101> sipa: how so?
19:46:27 <sipa> everything is transaction oriented in the current wallet
19:46:34 <wumpus> jarthur: some use core, but can't mention any names
19:46:40 <sipa> new transactions that can others to become conflicted etc
19:47:04 <sipa> changing that to a UTXO model and keeping it in sync with the list of transactions... sounds very hard
19:47:19 <achow101> right, but I think those can still be done by just a list of txids, spent prevouts, and utxos
19:47:24 <jeremyrubin> is it a more important goal to reduce the number of txns or the amount of data each one is storing?
19:47:45 <sipa> but you still need all the dependency tracking between transactions to compute the utxos
19:48:04 <sipa> you can cache the utxo list; that would probably be a worthwhile performance improvement
19:48:18 <wumpus> that would likely be a better first step
19:48:28 <sipa> but getting rid of the transaction entirely... i don't see how
19:48:36 <achow101> also, maybe just loading a neutered transaction without input scripts, because we don't need those
19:48:49 <sipa> achow101: yeah that can work
19:48:53 <jeremyrubin> achow101: that's what I was getting at :)
19:49:18 <sipa> achow101: also, don't let me discourage you if you see a good way to implement it :)
19:49:32 <sipa> i'm happy to be convinced otherwise
19:49:35 <wumpus> there's definitely some cut-off possible, I mean, transactions from years ago can't really become conflicted anymore
19:49:35 <jeremyrubin> achow101: a good first step might be to cache the scriptPubKey of inputs in a WtX
19:50:36 <sipa> i suspect thay in native descriptor wallets the IsMine check will be a lot faster than it currently is
19:50:38 <achow101> anyways, that's all. just wanted to get some opinions before diving in
19:50:43 <wumpus> at some point spent transactions deep enough in the chain can be permantly archived
19:51:24 <wumpus> might be non-trivial to come up with criteria but I don't think every transaction needs to be potentially active forever
19:51:37 <wumpus> achow101: good luck !
19:53:03 <wumpus> #endmeeting