19:00:20 <wumpus> #startmeeting
19:00:20 <lightningbot> Meeting started Thu Dec 13 19:00:20 2018 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:20 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:00:37 <moneyball> hi
19:00:56 <wumpus> proposed topic: drop Windows Vista support, make minimum supported Windows 7
19:01:03 <moneyball> just one topic proposed during the week - jamesob
19:01:06 <gmaxwell> wumpus: ping people.
19:01:35 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb
19:01:42 <provoostenator> (ah well, good way to get more people to review that RNG PR post merge)
19:01:43 <achow101> hi
19:01:44 <meshcollider> hi
19:01:45 <jonasschnelli> hi
19:02:04 <luke-jr> hi
19:02:09 <chenpo> hi
19:02:41 <wumpus> #topic High priority for review
19:02:45 <provoostenator> Let's call the topic Asta la la Vista :-)
19:03:19 <wumpus> https://github.com/bitcoin/bitcoin/projects/8 three PRs left on the list right now: #14336 #14565 #14573
19:03:22 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
19:03:24 <gribble> https://github.com/bitcoin/bitcoin/issues/14565 | Overhaul importmulti logic by sipa · Pull Request #14565 · bitcoin/bitcoin · GitHub
19:03:26 <gribble> https://github.com/bitcoin/bitcoin/issues/14573 | qt: Add Window menu by promag · Pull Request #14573 · bitcoin/bitcoin · GitHub
19:03:27 <wumpus> provoostenator: good idea
19:03:58 <sipa> hi
19:03:59 <wumpus> the poll one should be (almost?) ready to merge
19:04:09 <provoostenator> Nominating #11082
19:04:11 <gribble> https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
19:04:41 <provoostenator> Needs rebase, but really could use more review before that.
19:04:48 <wumpus> provoostenator: already needs rebase now
19:04:58 <wumpus> (and has, for 24 days)
19:05:12 <jonasschnelli> provoostenator: Maybe take that over from luke-jr
19:05:23 <sipa> does it need concept discussion?
19:05:34 <jonasschnelli> probably...
19:05:38 <provoostenator> I will eventually, but I have 15 other thing on my list. I also think it needs concept discussion.
19:06:02 <luke-jr> it doesn't need taking over, although if someone wants to rebase it sooner, I can push that
19:06:13 <provoostenator> Though we've had IRC chats about it before and everyone seemed to think it was at least a reasonable approach, compared to alterantives.
19:06:18 <wumpus> if it needs concept discussion, it definitely doesn't belong on the high priority for review list
19:06:26 <wumpus> yes, I think it's a reasonable approach
19:06:54 <provoostenator> Alright, maybe add it after rebase?
19:06:55 <wumpus> I dread making init option parsing even more involved, but it's better than the alternatives
19:07:16 <provoostenator> Dynamic wallet loading also needs this to make it good UX.
19:07:17 <sipa> yeah, fair
19:07:25 <wumpus> this needs *very* good tests
19:07:27 <provoostenator> Otherwise you'd have to manually load each time you start.
19:07:44 <sipa> provoostenator: i was expecting that to go in the qt settings
19:07:50 <wumpus> especially for the qt case ast hat's even crazier
19:08:14 <wumpus> how many levels of overlapping options can you have :)
19:08:21 <provoostenator> It has a bunch of Boost tests, but can probably use some QT tests and functional tests.
19:08:38 <wumpus> though this is supposed to get rid of most of the qt settings I guess?
19:08:46 <gmaxwell> obviously we should store what wallets get loaded in wallet.dat...
19:08:50 <jonasschnelli> Overlapping options is already problematic on the QT layer...
19:08:54 <provoostenator> Well, fewer levels thanks to my followup #12833
19:08:54 <luke-jr> wumpus: in a subsequent PR
19:08:55 <sipa> gmaxwell: wahaha
19:08:57 <gribble> https://github.com/bitcoin/bitcoin/issues/12833 | [qt] move QSettings to bitcoin_rw.conf where possible by Sjors · Pull Request #12833 · bitcoin/bitcoin · GitHub
19:09:01 <wumpus> gmaxwell: yesss
19:09:17 <wumpus> luke-jr: right
19:09:35 <provoostenator> But I'm reluctant to add more settings, as that makes rebasing 12833 a pain.
19:09:54 <wumpus> luke-jr: so eventually it'll replace non-pure-user-interface settings in the qt settings, I mean?
19:10:01 <provoostenator> (and that one also needs tests, I haven't delved into how to write QT tests yet, can someone make me a Python framework?)
19:10:04 <luke-jr> wumpus: ideally
19:10:28 <provoostenator> Yeah the idea is to get rid of QTsettings, with the exception of the datadir.
19:10:34 <jonasschnelli> I guess the QT settings files will always be there for window sizes and such... but non critical settings for sure
19:10:39 <wumpus> qtsettings is fine for *ui* settings
19:10:45 <wumpus> such as the last position of the window
19:10:46 <provoostenator> And what jonasschnelli says.
19:10:48 <wumpus> and things like that
19:10:48 <jonasschnelli> indeed
19:10:57 <wumpus> but not for settings shared with the core
19:11:02 <provoostenator> But not stuff that's shared with bitcoind like prune=
19:11:03 <jonasschnelli> yes
19:11:07 <wumpus> such as dbcache, etc
19:11:10 <wumpus> right.
19:11:25 <jonasschnelli> That was just a bypass of not having a way to write to a config section
19:11:45 <luke-jr> ?
19:12:09 <jonasschnelli> We wanted dbcache, proxys in Qt but didn't had a way to write to the config, so we added another layer
19:12:15 <wumpus> yep
19:12:37 <jonasschnelli> Which is no longer a clever approach once we have rw_config
19:12:46 <sipa> i'd be more comfortable if the set of modifiable settings (the list of wallets?) were distinct from those that aren't
19:12:53 <wumpus> we definitely don't want to write to bitcoin.conf directly, but having a separate configuration file that's writable is fine
19:13:02 <sipa> and have the GUI (where you really want everything to be modifable) directly modify bitcoin.conf
19:13:07 <wumpus> noooooo
19:13:14 <wumpus> don't write to bitcoin.conf, never
19:13:18 <sipa> if you use the GUI, the GUI manages the settings
19:13:23 <wumpus> conf should be read only
19:13:24 <sipa> if you don't, the config file does
19:13:26 <wumpus> we've been over this, please
19:13:35 <sipa> right, or have a separate config entirely
19:13:48 <wumpus> let's not change the idea now
19:13:56 <sipa> i really dislike having a UI edit things at the same level as the user is supposed to
19:13:59 <sipa> okay.
19:14:00 * sipa shuts up
19:14:40 <wumpus> we can have this discussion for a few years then never decide to do anything
19:14:50 <provoostenator> There's also a bunch of settings that can go straight into the wallet payload, like RBF and address type (though not their defaults).
19:14:58 <provoostenator> (or maybe even their defaults)
19:15:25 <provoostenator> #13044
19:15:26 <gribble> https://github.com/bitcoin/bitcoin/issues/13044 | [RFC] Long term plan for wallet command-line args · Issue #13044 · bitcoin/bitcoin · GitHub
19:15:39 <jonasschnelli> maybe this also raises the question how to distinct global settings between wallets
19:15:45 <provoostenator> That also simplifies QT, because wallet settings can be updated without any of the QTSettings stuff.
19:16:01 <wumpus> I'm unconfortable with writing to bitcoin.conf directly, as it is specified with -conf, which might point to a read-only configuration file, it should not be assumed it is writable
19:16:32 <wumpus> putting settings in the wallet again?
19:16:37 <gmaxwell> ugh
19:16:41 <sipa> ugh
19:16:42 <wumpus> yea deja vu...
19:16:50 <gmaxwell> putting settings in the wallet I think is even worse now that we have multiwallet.
19:16:51 <jonasschnelli> wumpus: there are the wallet flags now...
19:16:55 <provoostenator> wumpus: yes, that was the idea. Wallets have meta data entries. We already put binary flags in the wallet.
19:16:57 <jonasschnelli> but yeah...not settings
19:17:07 <sipa> provoostenator: originally all settings were in the wallet, it was pretty bad :)
19:17:12 <luke-jr> it might make sense for some subset of settings
19:17:14 <wumpus> sure, wallet-specific metadata should be stored in the wallet
19:17:18 <wumpus> but not settings
19:17:19 <gmaxwell> so you load another wallet and then your software starts behaving differently, madness!
19:17:23 <provoostenator> Yes, obviously only wallet settings.
19:17:27 <jamesob> write to wallet.conf.qt in datadir, have that override -conf settings?
19:17:30 <gmaxwell> yea sure wallet specific metadata sure fine whatever.
19:17:43 <jamesob> *bitcoin.conf.qt
19:17:44 <wumpus> jamesob: that's exactly the idea behind the bitcoin_rw.conf
19:17:44 <jonasschnelli> I just though about address type per wallet,... how do we handle different address types with different wallets?
19:17:46 <luke-jr> eg, you might have a non-segwit wallet, and a segwit wallet
19:17:49 <provoostenator> See the list in the issue I linked to above.
19:17:50 <jonasschnelli> *thought
19:18:07 <jamesob> wumpus: my bad, getting to the meetin glate
19:18:07 <wumpus> jamesob: it contains writable settings which override what is in bitcoin.conf
19:18:34 <wumpus> jamesob: this means being able to edit settings through RPC as well as the GUI
19:18:47 <jamesob> oh cool
19:18:49 <provoostenator> Some of these make sense int he wallet: addresstype, changetype, discardfee, fallbackfee, keypool, mintxfee, paytxfee, txconfirmtarget, etc. For others it doesn't make sense.
19:19:07 <gmaxwell> I don't see how fee settings make much sense in a wallet.
19:19:10 <wumpus> I'm not sure
19:19:17 <gmaxwell> Addresstype I could buy. maybe.
19:19:23 <provoostenator> gmaxwell: to make them behave consistently.
19:19:30 <jonasschnelli> Addresstype certenly,... fees, probably not
19:19:38 <sipa> addresstype will go away in the brave new world future anyway
19:19:43 <wumpus> yes
19:19:43 <gmaxwell> keypool no, it almost could go away.
19:19:44 <jonasschnelli> But I know a lot of people running a SW and a non-SW wallet in parallel
19:19:45 <provoostenator> But maybe fee preferences are more mempool weather dependent than wallet specific.
19:19:59 <gmaxwell> jonasschnelli: sounds brain damaged.
19:20:15 <gmaxwell> jonasschnelli: do you mean they just want to get keys with different address types?
19:20:18 <jonasschnelli> It may be,... but it's just how transition phases happens
19:20:42 <provoostenator> "keypool" might be replaced with a descriptor specific keypool, but I don't think that changes anything. Unless we stop using hardened derivation at the address level.
19:20:43 <wumpus> jamesob: please review #11082 :)
19:20:46 <gribble> https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
19:20:53 <gmaxwell> in any case, changing wallets doesn't change what network you're on, so changing txfee settings doesn't really follow logically.
19:20:55 <sipa> provoostenator: yes keypool will go away as well
19:21:11 <wumpus> gmaxwell: agree
19:21:18 <luke-jr> gmaxwell: maybe if you're worried the wallets will get linked by behaviour?
19:21:21 <gmaxwell> provoostenator: keypool was configurable in the first place because it interacted with backup durability.
19:21:23 <jonasschnelli> gmaxwell: I guess they want a SW wallet that derives P2SH(P2WPKH) and and their old wallet that derives P2PKH... (and not mix them)
19:21:28 <jamesob> wumpus: roger that
19:21:33 <provoostenator> I do like the idea of giving config / rw_config wallet specific sections.
19:21:52 <wumpus> scoping settings on the wallet will be confusing for users, should imo be avoided if possible
19:22:16 <gmaxwell> provoostenator: so it's not clear to me that forward caching of keys would be configurable in the future, or at least that it would be anything but an advanced thing that users usually shouldn't mess with.
19:22:26 <wumpus> if anything it's very difficult to come up with a good user interface for that
19:22:33 <sipa> i think all of those can either be reasonably done at the node level, or turn into descriptor-specific things in the wallet (and thus not need config)
19:22:43 <provoostenator> gmaxwell: ah I see, making it "just work" could make sense
19:22:45 * luke-jr suggests ignoring wallet-specific settings for initial rwconf purposes <.<
19:22:52 <wumpus> luke-jr: yes, I agree
19:22:59 <gmaxwell> yea agreed.
19:23:01 <wumpus> luke-jr: one step at a time
19:23:06 <wumpus> we've been on this step for years
19:23:45 <wumpus> every time it's the same discussion, though I haven't heard the idea of settings in the wallet seriously proposed for a while :)
19:23:49 <gmaxwell> to the extent that there are wallet specific things they probably should be in the wallet so they move with the wallet.  Regardless, they don't need to be part of the rwconf discussion.
19:24:27 <provoostenator> Agree regarding getting rwconf out the door without adding anything to it.
19:24:43 <jonasschnelli> wumpus: since its only addresstype that may be relevant and will go away over time,... I take back the argument that settings per wallet are relevant
19:24:53 <wumpus> jonasschnelli: thank you
19:25:01 <gmaxwell> (I'm just doubtful that there actually are more than a couple wallet specific things, addresstypes seem the most realistic of the ones mentioned here)
19:25:34 <provoostenator> Indeed, that seems the most important thing to keep consistent per wallet for privacy reasons.
19:25:50 <jonasschnelli> boolean things like disable-privatekeys can be handles with the 64bit wallet flags
19:25:54 <jonasschnelli> *handled
19:26:00 <provoostenator> One day everyone will send to bech32 addresses, and then we'll do v1 SegWit to start the problem over again :-)
19:26:11 <sipa> jonasschnelli: even that we don't need post descriptors
19:26:14 <wumpus> at the least, settings that are part of the wallet *should not* be part of the global options sytem
19:26:23 <wumpus> otherwise it just becomes too many levels
19:26:26 <sipa> you'd just not a have a descriptor with private keys if you don't want private keys
19:26:37 <gmaxwell> provoostenator: no, because v1 segwit doesnt' change the address type, bech32 already supports it.
19:26:49 <gmaxwell> (shocking, we already anticipated that problem... :P )
19:27:05 <luke-jr> gmaxwell: but p2sh^2 could (independnetly of segwitv1)
19:27:22 <gmaxwell> provoostenator: I'm sure there will be some broken sites, but it should be much less of an issue.
19:27:26 <jonasschnelli> sipa: is that similar of saying "you don't need to import private keys if you don't want to have private keys"?
19:28:08 <jonasschnelli> disable-private key seems to be another layer,... a footgun protection. But seems like we are going OT
19:28:12 <sipa> jonasschnelli: it would turn no-private-keys into a flag for wallet creation perhaps, to determine what descriptors a new wallet is born with; but it wouldn't affect anything later
19:28:17 <provoostenator> gmaxwell: but you still don't want to e.g. consistently use or not use schnorr, so you still need to maintain a preference. Even if it's all bech32.
19:28:24 <provoostenator> *do want
19:28:33 <luke-jr> so back to rwconf.. <.<
19:28:34 <sipa> anyway, sure - we could keep it as a footgun protection, but it seems kind of pointless
19:28:40 <gmaxwell> provoostenator: yes but thats just a property of which keys/descriptors are in the wallet.
19:28:42 <sipa> yeah, getting offtopic, sorry
19:28:51 <provoostenator> gmaxwell: good point
19:28:51 <wumpus> I think it's time to move to the next topic
19:29:06 <provoostenator> Descriptors are very useful.
19:29:10 <meshcollider> sipa: I assume you'd still need a way to fail if you try and import private keys though
19:29:43 <provoostenator> That's the foodgun protection.
19:29:46 <provoostenator> footgun
19:30:21 <provoostenator> I see two topics proposed here https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a
19:30:29 <provoostenator> jamesob and moneyball
19:30:32 <wumpus> #topic Asta la la Vista
19:30:51 <jamesob> Terminator movies?
19:31:03 <provoostenator> Dropping Vista support
19:31:07 <wumpus> so the idea to move the minimum supported windows to W7 recently came up in #14922
19:31:09 <gribble> https://github.com/bitcoin/bitcoin/issues/14922 | [WIP] windows: Set _WIN32_WINNT to 0x0601 (Windows 7) by ken2812221 · Pull Request #14922 · bitcoin/bitcoin · GitHub
19:31:29 <sipa> vista extended support ended on april 11, 2017
19:31:36 <wumpus> apparently this was already changed in the release notes #12546
19:31:38 <gribble> https://github.com/bitcoin/bitcoin/issues/12546 | [docs] Minor improvements to Compatibility Notes by randolf · Pull Request #12546 · bitcoin/bitcoin · GitHub
19:31:46 <sipa> end of mainstream support was in 2009
19:31:51 <luke-jr> for Linux, we just support latest stable distros, so if we mirror that for Windows, we can move to Win10 :P
19:32:05 <sipa> it was also one of the least popular windows releases...
19:32:14 <wumpus> so I think dropping support for Vista is pretty non-controversial
19:32:15 <wumpus> yes
19:32:22 <wumpus> it wasn't even popular at the time
19:32:29 <achow101> +1
19:32:32 <jonasschnelli> ack
19:32:34 <meshcollider> Yep
19:32:41 <sipa> so i think there is no real to keep it
19:32:51 <provoostenator> I'm a Mac fanboy, so conflict of interest :-)
19:32:54 <sipa> i actually expect less opposition than the opposition we got from dropping XP support
19:33:08 <gmaxwell> People are still using XP, I think less so than vista.
19:33:13 <luke-jr> sipa: this may be the first time we actually break XP, note
19:33:15 <wumpus> luke-jr: W7 is still used quite a lot by people unwilling to move to W10
19:33:16 <gmaxwell> er more so than vista.
19:33:38 <gmaxwell> the bigger issue will be W10 which is kinda spyware land windows from what I'm told.
19:33:50 <wumpus> yes
19:33:51 <meshcollider> Most people consider W7 as a strict improvement over vista so there should be minimal resistance :)
19:33:52 <sipa> is there anything between w7 and w10?
19:33:59 <luke-jr> 8.1
19:34:08 <wumpus> there's a windows 8 but I don't think it ever caught on much
19:34:40 <sipa> ah yes
19:34:47 <sipa> anyway, ack dropping vista support
19:34:52 <wumpus> ok
19:35:02 <sipa> based on what i read; not a windows user myself
19:35:04 <gmaxwell> that PR will it make it actually not work when it does otherwise?
19:35:24 <bitcoin-git> [13bitcoin] 15MarcoFalke opened pull request #14953: test: Make g_insecure_rand_ctx thread_local (06master...06Mf1812-testThreadLocal) 02https://github.com/bitcoin/bitcoin/pull/14953
19:35:25 <MarcoFalke> Should fix the thread sanitizer issue ^
19:35:30 <gmaxwell> That isn't what we mean by supported on other platforms, on linux if you want to try running on really old stuff, you can try but you're own your own.
19:35:34 <wumpus> yes it sets the minimum API level so that it won't start in <W7, and makes it possible to use newer APIs
19:35:52 * luke-jr pokes MarcoFalke
19:36:13 <gmaxwell> ah it gates newer APIs.  Is there a reason to not announce we don't support vista, but only bump that flag when we actually go to use one of those apis?
19:36:34 <gmaxwell> (just blocking software where it otherwise works doesn't really feel in the spirit of open source)
19:36:47 <wumpus> you're pedaling back now?
19:36:58 <luke-jr> gmaxwell: IIRC some PR wants to use Win7 API
19:37:05 <gmaxwell> luke-jr: okay.
19:37:05 <wumpus> FWIW qt is also going to break vista support
19:37:16 <luke-jr> I did suggest making the change in *that* PR..;
19:37:24 <wumpus> this is for 0.18
19:37:31 <wumpus> which is still a few months away
19:37:38 <gmaxwell> wumpus: no wumpus, I'm not "pedaling back"   but the logic given above that we don't support older linux doesn't actually apply to actively breaking older windows.
19:37:50 <gmaxwell> If it's going to be broken by other changes in any case, thats fine then.
19:37:53 <luke-jr> eg, ionice_win bumps the Windows API version too, but wasn't merged yet
19:37:58 <wumpus> gmaxwell: fair enough but luke-jr was talking about supporting *only* windows 10
19:38:05 <wumpus> which is kind of extreme
19:38:18 <wumpus> W7 as bottom should be in line with all modern software
19:38:29 <luke-jr> wumpus: just mentioning it as a comparison to how we handle Linux distros ;)
19:38:35 <gmaxwell> from all I've heard running bitcoin on windows 10 sounds like it's probably a bad idea in general.
19:38:38 <meshcollider> From the PR, "#14881 is using inet_pton and it's only for Vista or later. So I create this PR just for that."
19:38:40 <gribble> https://github.com/bitcoin/bitcoin/issues/14881 | Tests: Contract testing for the procedure AddTimeData by mmachicao · Pull Request #14881 · bitcoin/bitcoin · GitHub
19:39:05 <gmaxwell> wumpus: okay, concern withdrawn. I just thought it would be sad to gratitiously break it, but since we have a reason to to do, good to go.
19:39:07 <luke-jr> gmaxwell: but only Windows 10 is actually supports anymore IIRC
19:39:16 <wumpus> luke-jr: yes, I know it was not seriously, but would be in line with what we do for wsay, MacOSX, but mac users like upgrading a lot more than windows users
19:39:41 <luke-jr> supported*
19:39:57 <wumpus> luke-jr: what does microsoft still officially support? only W10? are you sure?
19:40:03 * luke-jr double checks
19:40:16 <luke-jr> Win 8.1: Mainstream support ended on January 9, 2018
19:40:23 <luke-jr> apparently you can pay for support until 2023
19:40:42 <achow101> windows 7 will have security updates until jan 2020
19:40:56 <wumpus> great
19:41:06 <luke-jr> maybe I misunderstand extended support
19:41:18 <luke-jr> Win 7: Mainstream support ended on January 13, 2015. Extended support until January 14, 2020 (January 2023 for Professional and Enterprise, users will need to pay for security updates).[5][6]
19:41:39 <wumpus> so w7 is still more or less relevant
19:41:49 <achow101> yes
19:42:04 <achow101> and i'm pretty sure a lot of people still use it too
19:42:09 <wumpus> right
19:42:26 <wumpus> let's move to next topic
19:42:51 <wumpus> #topic "add address index" PRs (jamesob)
19:43:19 <jamesob> I've talked to a number of companies who're clamoring for an address index and we've got four attempts buuuut
19:43:49 <jamesob> sipa has potentially disabused me of the notion that an addr index is a legitimate approach to just about anything that isn't debugging or analysis
19:43:49 <jonasschnelli> jamesob: you mean unspent address index?
19:44:03 <jamesob> sure, or a more general script descriptor index
19:44:06 <jonasschnelli> I agree with sipa
19:44:22 <achow101> jamesob: for what are the companies trying to use an address index for?
19:44:32 <wumpus> an address index over the full block chain never was a good idea, one over the utxo set was considered but never made itin yet
19:44:41 <jamesob> in any case, I think we should have some sort of recommendation (and ideally a maintained piece of software) to recommend to the folks who want to do addr-indexy thing
19:44:44 <jamesob> s
19:45:04 <luke-jr> jamesob: "don't do that" :p
19:45:08 <gmaxwell> I think unspents indexes are fine, generally, but shouldn't be a priority for the project over other concerns.. if someone wants to come and do the work for them, and to do them well, great.
19:45:15 <jamesob> achow101: afaict mostly watching for activity on various addresses
19:45:23 <jonasschnelli> jamesob: you could do the address index externaly via p2p..... look at https://github.com/jonasschnelli/bitcoincore-indexd (experiment)
19:45:24 <provoostenator> Always happy to test and review those address index PRs. I think that on top of the new index scheme it's not too bad to maintain.
19:45:40 <luke-jr> what if we have a full TXO/script index, and only expose it in the GUI as a block explorer? (ie, no RPC to be abused)
19:45:42 <gmaxwell> jamesob: our general recommendation for watching is to import them into a watching wallet.
19:45:42 <wumpus> *watcing activiity* shouldn't require an index of everything over the whole block chain
19:45:44 <provoostenator> Or we could come up with a plugin system for indexes, though that's also a can of worms.
19:45:47 <jamesob> jonasschnelli: indeed, as well as projects like electrs
19:46:10 <wumpus> wasn't there some external project addressing this?
19:46:11 <jonasschnelli> I think we should not load a complete address index on the shoulders of Core
19:46:14 <gmaxwell> jamesob: and I think if there are limitations on importing for watching, we'd like to improve those.
19:46:18 <luke-jr> provoostenator: I thought we had that now
19:46:25 <sipa> gmaxwell: yup!
19:46:31 <wumpus> it's a *huge* database in any case
19:46:33 <jonasschnelli> But I agree, externally makes it a bit more complex
19:46:41 <jamesob> in the next few weeks I'm going to be getting in touch with companies to get a more concrete idea of the usecases, so I'll report back with what I find
19:46:59 <provoostenator> luke-jr: have what? People can compile their own client of course, but that's pretty high bar and can easily lead to rebase nightmares.
19:47:01 <wumpus> bitcoin core is not meant as a chain analysis platform
19:47:06 <luke-jr> wumpus: I did it in ~2 GB a year or so ago, IIRC
19:47:07 <wumpus> you can do forensics using your own tools
19:47:13 <jamesob> wumpus: very much agree
19:47:20 <gmaxwell> jonasschnelli: right the problem for externally, is that implementing blockchain consistency is non-trivial and in my expirence most people who want to build an index don't want to deal with that.
19:47:44 <gmaxwell> jamesob: more information on the actual usecases would be very helpful.
19:47:49 <wumpus> it's non trivial but not rocket science either
19:47:54 <jonasschnelli> Sadly people expect fast BIP44 recovery (incl. history). This seems to be the most prominent real-world usecase for an address index
19:48:07 <wumpus> it shouldn't be that anything non-trivial needs to end up[ in bitcoin core
19:48:12 <wumpus> we're not the world of non-trivial solutions
19:48:26 <gmaxwell> jonasschnelli: most (I believe most, many at least) electrum servers don't do history, so I'm not entirely clear on the including history part of your comment.
19:48:35 <provoostenator> Let's also not forget to update #14053 with concept ACK / NACK so people don't waste time on trying it if it's undesirable.
19:48:38 <gmaxwell> wumpus: heh.
19:48:39 <gribble> https://github.com/bitcoin/bitcoin/issues/14053 | Add address-based index (attempt 4?) by marcinja · Pull Request #14053 · bitcoin/bitcoin · GitHub
19:48:43 <jonasschnelli> gmaxwell: IMO they do (index everything)
19:49:04 <jonasschnelli> (not electrum personal server though)
19:49:04 <wumpus> I mean this is another thing that's been *years*
19:49:09 <gmaxwell> jonasschnelli: no, I know that for a fact-- many electrum servers don't index history.
19:49:11 <wumpus> really, no one has been able to build a solution to this?
19:49:14 <gmaxwell> (I just don't know if its most of them)
19:49:18 <wumpus> not one of all those companies that want it?
19:49:24 <jamesob> provoostenator: and its sibling #14035
19:49:25 <luke-jr> wumpus: people capable of it don't want it :p
19:49:26 <gribble> https://github.com/bitcoin/bitcoin/issues/14035 | Utxoscriptindex by mgrychow · Pull Request #14035 · bitcoin/bitcoin · GitHub
19:49:37 <sipa> wumpus: blockstream.info uses electrs, which seems to work very well
19:49:43 <gmaxwell> wumpus: I think the process of learning enough to do it well does a pretty good job of convincing you that you don't want it.
19:49:51 <wumpus> gmaxwell: heh
19:49:59 <provoostenator> jonasschnelli: BIP44 recovery can be handled once we have descriptor support for importmulti and slightly more sane behavior (or a replacement for) the keypool.
19:50:02 <wumpus> sipa: that's a good suggestion then!
19:50:31 <gmaxwell> provoostenator: history recovery requires a scan of the chain, or a phenominally expensive index.
19:50:31 <jonasschnelli> provoostenator: you can't recover the history without an address index or a complete rescan (which is not possible for pruned peers)
19:50:43 <wumpus> https://github.com/romanz/electrs
19:50:51 <provoostenator> Ok, rescan for pruned nodes isn't there yet.
19:50:53 <gmaxwell> jonasschnelli: note that an index is also not compatible with pruning. :P
19:50:56 <wumpus> alrady had it bookmarked apparently :)
19:50:59 <jonasschnelli> The only light here is the scantxoutset where you can recover within seconds but not the spent-history
19:51:19 <provoostenator> Though you could use the neutrino filters for that and then refetch the relevant blocks.
19:51:24 <jonasschnelli> gmaxwell: it could be,... if we just keep the pointers and load the data via p2p once requested
19:51:32 <gmaxwell> jonasschnelli: I think we'd get a lot more bang for our buck working on making 'wallet without history before x' (pruned wallet) a first class supported thing.
19:51:48 <jonasschnelli> gmaxwell: completely agree on this
19:51:54 <gmaxwell> jonasschnelli: transfering 200GB of data to do the input is not really reasonable...
19:52:24 <gmaxwell> (and then not saving it... this would be pretty harmful to the p2p network if people were actually patient enough to use it. :) )
19:52:28 <jonasschnelli> gmaxwell: I mean there is a PR that enabled txindex with pruning... and fetches the tx over p2p once requested outside the kept block area
19:52:39 <jonasschnelli> which is slow.... but for debugging proposes okay
19:52:43 <luke-jr> (this makes me think again of the concept of prune-to-slow-storage where you can plug in a USB drive when/if you need old data..)
19:52:50 <gmaxwell> jonasschnelli: oh, I didn't recall that PR.
19:53:06 <wumpus> luke-jr: prune-to-tape *hides*
19:53:11 <provoostenator> luke-jr: slow storage being the internet? :-)
19:53:17 <jonasschnelli> #13014
19:53:19 <gribble> https://github.com/bitcoin/bitcoin/issues/13014 | Allow txindex in prune mode by jonasschnelli · Pull Request #13014 · bitcoin/bitcoin · GitHub
19:53:30 <luke-jr> provoostenator: could be local
19:53:53 <gmaxwell> jonasschnelli: bip 157 filtering could be used similarly.
19:54:15 <gmaxwell> jonasschnelli: e.g. saved the bip157 filters locally, and scan against them.
19:54:15 <jonasschnelli> yes... less disk space, more time spent on filtering... but same idea
19:54:37 <luke-jr> hmm
19:54:55 <luke-jr> what data do they want returned from the index? maybe we don't need to retain/fetch the block
19:55:08 <luke-jr> eg, if the client would be happy with a list of block numbers or txids..
19:55:08 <jamesob> are any of the BIP157/158 PRs on the high prio list? if not, they should be
19:55:11 <gmaxwell> wumpus: sadly, freeking tape costs about as much as HDDs today...  LTO7 tapes (6TB) cost about $70.
19:55:28 <wumpus> gmaxwell: ouch
19:55:38 <provoostenator> So the wallet recovery flow would be: add descriptors, download BIP-157 filters (if you didn't keep them), process filters, fetch a few blocks. User can immediately use their wallet to receive and send.
19:55:48 <wumpus> gmaxwell: I guess they're more reliable though
19:55:49 <luke-jr> jamesob: making light wallets stronger has a negative impact on Bitcoin :/
19:56:23 <gmaxwell> provoostenator: I don't really think thats a viable long term workflow either...  rather if you couldn't afford the space to keep them you probably don't want the time/bandwidth to download them.
19:56:27 <sipa> luke-jr: BIP158 helps our local wallet too
19:56:28 <provoostenator> luke-jr: not making them possible sends most people to web wallets, not to full nodes.
19:56:35 <jamesob> luke-jr: better light wallets might help adoption
19:56:39 <gmaxwell> luke-jr: I'm only interested in it for the local wallets.
19:57:00 <luke-jr> jamesob: better to not have that adoption..
19:57:05 <luke-jr> sipa / gmaxwell: yes, that's a point
19:57:22 <provoostenator> gmaxwell: recovery should be a rare thing anyway, I assume it only happens after a disaster. In which case you'd probably download the whole chain again anyway.
19:57:32 <sipa> provoostenator: it should be.
19:57:34 <gmaxwell> jamesob: history has shown otherwise, bip158 doesn't make lite wallets fundimentally more usable than they are now.  They're still massively worse than server driven wallets like electrum or web wallets.
19:58:05 <gmaxwell> provoostenator: right but thats also a reason that fetching them when you don't have them isn't that interesting, IMO.
19:58:22 <jamesob> good point - but I guess if existing light wallets switched to bip157 it'd at least ease load on existing full nodes
19:58:41 <gmaxwell> jamesob: what light wallets?
19:58:56 <provoostenator> gmawell is that _because_ of bip158 or just because there aren't that many developers working on light (non web, non electrum) wallets? That could change over time.
19:58:59 <wumpus> 1 minute to go
19:59:04 <jamesob> anything using bloom-filter-based SPV
19:59:12 <gmaxwell> jamesob: at least connections I see there is virtually no acutal usage of p2p lite wallets anymore.
19:59:15 <wumpus> please wrap up the meeting
19:59:44 <jamesob> I'll report on any compelling usecases I find for addr index, but I suspect sipa et al are right that that's usually just the Wrong way
19:59:48 <gmaxwell> provoostenator: P2p lite wallets that scan the chain just end up with a very poor user expirence.
20:00:06 <jamesob> in the meantime I recommend we give concept ACK/NACK to outstanding PRs which are related
20:00:11 <gmaxwell> and that doesn't really have anything to do with bip158 vs bip37.
20:00:36 <gmaxwell> (in fact BIP158 is somewhat worse, but slightly less of a privacy disaster)
20:00:37 <wumpus> #endmeeting