12017-03-10T00:01:07  *** CubicEarth has joined #bitcoin-core-dev
  22017-03-10T00:02:29  <cbits_> Yeah, and it automatically sets rbf checked when you have it set to visible
  32017-03-10T00:08:51  *** CubicEarth has quit IRC
  42017-03-10T00:19:37  *** CubicEarth has joined #bitcoin-core-dev
  52017-03-10T00:21:36  *** GriTBalL has joined #bitcoin-core-dev
  62017-03-10T00:27:13  *** CubicEarth has joined #bitcoin-core-dev
  72017-03-10T00:32:37  *** CubicEarth has quit IRC
  82017-03-10T00:35:14  *** cbits_ has quit IRC
  92017-03-10T00:35:38  *** jannes has quit IRC
 102017-03-10T00:39:12  *** wudayoda has quit IRC
 112017-03-10T00:49:03  *** GriTBalL has quit IRC
 122017-03-10T00:58:34  *** abpa has quit IRC
 132017-03-10T00:59:07  *** cbits_ has joined #bitcoin-core-dev
 142017-03-10T01:00:31  *** arubi has quit IRC
 152017-03-10T01:03:20  *** randy-waterhouse has quit IRC
 162017-03-10T01:04:24  *** gribble has joined #bitcoin-core-dev
 172017-03-10T01:05:55  *** wasi has quit IRC
 182017-03-10T01:18:30  *** wasi has joined #bitcoin-core-dev
 192017-03-10T01:21:14  *** wudayoda has joined #bitcoin-core-dev
 202017-03-10T01:30:10  *** AaronvanW has quit IRC
 212017-03-10T01:30:40  *** Victor_sueca has joined #bitcoin-core-dev
 222017-03-10T01:30:57  *** Victorsueca has quit IRC
 232017-03-10T01:31:03  *** Victor_sueca is now known as Victorsueca
 242017-03-10T01:33:50  *** bityogi has quit IRC
 252017-03-10T01:38:42  *** cbits_ has quit IRC
 262017-03-10T01:56:37  *** gribble has quit IRC
 272017-03-10T02:04:34  *** wudayoda has quit IRC
 282017-03-10T02:26:24  *** Giszmo has quit IRC
 292017-03-10T02:34:34  *** gribble has joined #bitcoin-core-dev
 302017-03-10T02:44:05  *** Ylbam has quit IRC
 312017-03-10T02:44:56  *** wudayoda has joined #bitcoin-core-dev
 322017-03-10T02:50:27  *** wudayoda has quit IRC
 332017-03-10T02:56:46  *** dodomojo_ has joined #bitcoin-core-dev
 342017-03-10T02:58:39  <gmaxwell> wumpus: would we perhaps want to consider removing zap? we added it because we had no way to abandon transactions, but we do now. I've seen it used in a way that created a lot of damage. (user ran into the unconfirmed depth limit and couldn't make transactions. Then zapped their wallet, then stared paying again, doublspending the @#$@# out of themselves.  .. then were stuck trying to reassemble
 352017-03-10T02:58:45  <gmaxwell> the pieces... figure out who they still owed, etc.
 362017-03-10T03:02:39  *** CubicEarth has joined #bitcoin-core-dev
 372017-03-10T03:02:40  *** PRab has quit IRC
 382017-03-10T03:05:15  *** CubicEarth has quit IRC
 392017-03-10T03:13:29  *** harrymm1 has joined #bitcoin-core-dev
 402017-03-10T03:15:48  *** harrymm has quit IRC
 412017-03-10T03:20:53  *** Victor_sueca has joined #bitcoin-core-dev
 422017-03-10T03:23:48  *** Victorsueca has quit IRC
 432017-03-10T03:38:09  *** wudayoda has joined #bitcoin-core-dev
 442017-03-10T03:47:27  *** wudayoda has quit IRC
 452017-03-10T03:52:01  *** alpalp has quit IRC
 462017-03-10T03:52:19  *** CubicEarth has joined #bitcoin-core-dev
 472017-03-10T03:53:22  *** PatBoy has quit IRC
 482017-03-10T03:58:21  *** grubles has quit IRC
 492017-03-10T03:58:21  *** PatBoy has joined #bitcoin-core-dev
 502017-03-10T04:00:18  *** gribble has quit IRC
 512017-03-10T04:34:29  *** chris200_ has joined #bitcoin-core-dev
 522017-03-10T04:37:27  *** chris2000 has quit IRC
 532017-03-10T04:57:38  *** dodomojo_ has quit IRC
 542017-03-10T05:01:09  *** Sosumi has quit IRC
 552017-03-10T05:01:54  *** juscamarena has quit IRC
 562017-03-10T05:09:37  *** tonebox has joined #bitcoin-core-dev
 572017-03-10T05:10:40  *** scan_email_bot has joined #bitcoin-core-dev
 582017-03-10T05:11:50  <tonebox> It seems like segwit is a temporary fix...  4M is going to be overwhelmed, just like 1mb now...  Wouldn't a better solution be to change the time-base, so now it's 10 minutes per block... Soon, 5 min, then 2.5...  It could revert to 10, and be automatically adjusted just like the difficulty.
 592017-03-10T05:13:23  <Lightsword> tonebox, no, that results in higher orphan rates due to latency
 602017-03-10T05:14:55  <tonebox> Ok...  Thanks...   Would there be any way to make segwit not fixed at 4mb so this won't be a problem that needs to be solved again in a few years?
 612017-03-10T05:17:09  <tonebox> Also, it would seem like a dynamic timebase and fixing the issue with orphans would be a better solution long term.
 622017-03-10T05:20:10  <CubicEarth> Lightsword: I always thought moving to a 5 minute block time would be perfectly fine
 632017-03-10T05:20:25  <gwillen> This isn't really a good channel for this kind of discussion -- better in #bitcoin probably.
 642017-03-10T05:20:36  <CubicEarth> a slightly higher orphan rate wouldn't hurt anything
 652017-03-10T05:23:06  *** gribble has joined #bitcoin-core-dev
 662017-03-10T05:28:09  *** CubicEarth has quit IRC
 672017-03-10T05:43:54  *** wudayoda has joined #bitcoin-core-dev
 682017-03-10T05:47:57  *** wudayoda has quit IRC
 692017-03-10T05:52:05  *** juscamarena has joined #bitcoin-core-dev
 702017-03-10T05:52:15  *** juscamarena_ has joined #bitcoin-core-dev
 712017-03-10T05:56:09  *** juscamarena_ has quit IRC
 722017-03-10T05:57:29  *** wudayoda has joined #bitcoin-core-dev
 732017-03-10T05:59:17  <wumpus> gmaxwell: I'm fine with that...
 742017-03-10T06:01:02  <wumpus> gmaxwell: what I mostly don't like about it is that it requires a rescan, and is very non-selective (zap all unconfirmed)
 752017-03-10T06:02:32  <wumpus> gmaxwell: mostly it's useful to troubleshoot issues with wallet bugs and corruption, when a certain transaction behavres strangely.  But a tool to remove a single transaction from the database would work much better for that and be less impactful. Back in the day,though,that was hard to do for some reason
 762017-03-10T06:02:46  *** wudayoda has quit IRC
 772017-03-10T06:03:29  <wumpus> I agree abandontransaction replaces all end-use servicable reasons to use zap
 782017-03-10T06:11:42  <gmaxwell> It might just make sense to rename it and hide it for that reason. If there is a reason to not do that, we should probably enhance abandon further.
 792017-03-10T06:12:51  <wumpus> me preference would be to hide it in some dangerous-sounding wallet salvage or editing tool, not have it in the main executable at least or maybe not even in the main distribution
 802017-03-10T06:13:15  <wumpus> I mean there's a use for low-level-ish wallet editing, but it certainly shouldn't be easily available
 812017-03-10T06:14:51  <gmaxwell> Salvage is also pretty raw... I've said before it should be called "savage (verb) wallet"
 822017-03-10T06:15:07  <wumpus> hehe
 832017-03-10T06:15:52  <wumpus> salvage has been actually useful to a lot of people though, it tends to be the only thing available if the rest if something is corrupted
 842017-03-10T06:16:35  <luke-jr> doesn't zap recover from corrupt bdbs we can't open?
 852017-03-10T06:16:46  <wumpus> no, zap doesn't do that
 862017-03-10T06:17:01  <wumpus> it assumes that records are simply readable, what it does is remove all transactions
 872017-03-10T06:17:15  <wumpus> with a mode to keep metadata (by default) and another to trash it
 882017-03-10T06:17:23  <gmaxwell> salvage does. though at least historically it would also miss data in perfectly fine wallet.dats (though I think some of that was due to bugs which have been fixed)
 892017-03-10T06:17:37  <luke-jr> ah, mixed them up I guess
 902017-03-10T06:17:58  <wumpus> I think those issues have been fixed
 912017-03-10T06:18:32  <wumpus> thoug there are still some weird issues with salvagewallet, for example berkeleydb can return an error when salvaging an otherwise ok wallet (but it doesn't lose records anymore IIRC)
 922017-03-10T06:18:56  <wumpus> then again this is the kind of thing we're asking for with not updating the backend library for years. BDB should die.
 932017-03-10T06:20:44  <gmaxwell> so I did some testing and later BDB now appear to be bidirectionally compatible?!
 942017-03-10T06:20:55  <gmaxwell> but I couldn't find any announcement of it.
 952017-03-10T06:20:59  <gmaxwell> it just worked.
 962017-03-10T06:21:05  <wumpus> it works in some cases
 972017-03-10T06:21:31  <wumpus> that's been the case when I tried too. But I don't trust it.
 982017-03-10T06:21:54  <gmaxwell> ah, I wasn't aware that it ever worked before.
 992017-03-10T06:22:41  <wumpus> I think the backward incompatiblity thing is just a matter of no one ever seriously researching this, and what are the edge cases of it, than a sure thing
1002017-03-10T06:22:55  <wumpus> it doesn't work in *all* cases that is clear
1012017-03-10T06:26:32  <wumpus> one thing that is not backwards compatible is the log files; so if there are still log files behind, the old version will error out
1022017-03-10T06:33:47  *** aalex__ has joined #bitcoin-core-dev
1032017-03-10T06:36:14  <wumpus> it may well be that a "clean" .dat file, like produced by backupwallet, is always backwards compatible. Though that doesn't help a user that crashes on first run with the new version then tries to go back, the wallet being in intermediate state.
1042017-03-10T06:37:27  <wumpus> of course it's fairly simple to work around this with a conversion tools and/or making and automatic backup at first run. But it was just never deemed worth the trouble
1052017-03-10T06:37:38  <wumpus> especially as newer BDBs have license issues
1062017-03-10T06:37:58  <wumpus> the plan was ,and should still be, to move away from it
1072017-03-10T06:39:28  *** Victor_sueca has quit IRC
1082017-03-10T06:40:35  *** Victor_sueca has joined #bitcoin-core-dev
1092017-03-10T07:06:15  *** gribble has quit IRC
1102017-03-10T07:07:35  *** aalex__ has quit IRC
1112017-03-10T07:15:51  *** arubi has joined #bitcoin-core-dev
1122017-03-10T07:33:05  *** Ylbam has joined #bitcoin-core-dev
1132017-03-10T07:44:40  *** tonebox has quit IRC
1142017-03-10T07:54:48  *** BashCo has quit IRC
1152017-03-10T07:58:32  *** wudayoda has joined #bitcoin-core-dev
1162017-03-10T08:11:17  *** BashCo has joined #bitcoin-core-dev
1172017-03-10T08:23:05  *** gribble has joined #bitcoin-core-dev
1182017-03-10T08:27:23  *** Guyver2 has joined #bitcoin-core-dev
1192017-03-10T08:28:59  *** JackH has joined #bitcoin-core-dev
1202017-03-10T08:46:34  *** Guyver2 has quit IRC
1212017-03-10T08:51:35  *** Alina-malina has quit IRC
1222017-03-10T08:58:10  <wumpus> baahh.. this is the second time I have trouble with the tests due to a stale qa/cache directory
1232017-03-10T08:58:49  <wumpus> we should probably nix it when a change in bitcoind is detected
1242017-03-10T09:00:20  <wumpus> e.g. write that path and sha256sum of the bitcoind used into the cache directory, and if that changes, delete it
1252017-03-10T09:02:11  <sipa> unsure that's worth it... breaks of the cache direction are very infrequent, i think
1262017-03-10T09:02:25  <sipa> at least in my experience
1272017-03-10T09:02:40  <wumpus> any change to the wallet, at least
1282017-03-10T09:03:39  <wumpus> node0 keeps a wallet from mining the initial blocks, which is usually what causes the problems, if the test expects a certain newly introduced property of transactions
1292017-03-10T09:04:15  <wumpus> maybe it's infrequent but it is really frustrating and can lead to hours of misdirected search for bugs if it happens
1302017-03-10T09:04:59  <wumpus> conceptually it's also not *valid* to use an older cache, there is no guarantee that your tests passing is worth anything
1312017-03-10T09:05:18  <wumpus> the new bitcoind may completely mess up the initial steps and it'd still pass because it is cached
1322017-03-10T09:06:24  <wumpus> but ok I'll just add a message "Using cached node state in %s" to the test output, maybe that's enough to remind people to delete it if they run into weird issues
1332017-03-10T09:06:56  <sipa> no, i think you're right
1342017-03-10T09:07:10  <sipa> we shouldn't be using outdated caches
1352017-03-10T09:08:57  *** gribble has quit IRC
1362017-03-10T09:14:09  *** Chris_Stewart_5 has quit IRC
1372017-03-10T09:18:31  *** Victor_sueca is now known as Victorsueca
1382017-03-10T09:19:59  <jonasschnelli> wumpus: https://github.com/bitcoin/bitcoin/pull/9294#issuecomment-285612826
1392017-03-10T09:20:08  <jonasschnelli> This is strange... can it be a caching issue?
1402017-03-10T09:20:33  <jonasschnelli> I can't see a reason why the dump is different on your machine than on mine / travis.
1412017-03-10T09:21:25  <jonasschnelli> Maybe you have a chance and check the file "wallet.unencrypted.dump" (maybe pastebin it) when running with --nocleanup
1422017-03-10T09:23:43  * jonasschnelli starting Ubuntu 16.04
1432017-03-10T09:28:00  <jonasschnelli> sipa: maybe you have a chance to review #9965. It seems that you ate most familiar with that code part. It also touches the segwit.py test (where I'm not sure if I did the right thing).
1442017-03-10T09:29:33  <wumpus> jonasschnelli: yes it was a caching issue
1452017-03-10T09:30:06  <jonasschnelli> Great! *relief*
1462017-03-10T09:30:10  <wumpus> jonasschnelli: that why I wrote the posts above ^^
1472017-03-10T09:33:06  <wumpus> I think we should store a hash of the bitcoind executable in the cache directory and delete it when it mismatches
1482017-03-10T09:37:46  *** riemann has joined #bitcoin-core-dev
1492017-03-10T09:39:25  <jonasschnelli> wumpus: Ah. Thanks. I haven't read the scrollback. Yes. Good idea with the binary-hash mismatch detection.
1502017-03-10T10:06:18  *** afk11 has quit IRC
1512017-03-10T10:14:00  *** kadoban has quit IRC
1522017-03-10T10:26:16  *** gribble has joined #bitcoin-core-dev
1532017-03-10T10:26:30  *** kadoban has joined #bitcoin-core-dev
1542017-03-10T10:30:56  *** MarcoFalke has joined #bitcoin-core-dev
1552017-03-10T10:31:02  <MarcoFalke> 1
1562017-03-10T10:41:45  *** AaronvanW has joined #bitcoin-core-dev
1572017-03-10T10:43:06  *** BirneGetreide_ has joined #bitcoin-core-dev
1582017-03-10T11:22:12  *** MarcoFalke_ has joined #bitcoin-core-dev
1592017-03-10T11:22:12  *** MarcoFalke has quit IRC
1602017-03-10T11:22:12  *** MarcoFalke_ is now known as MarcoFalke
1612017-03-10T11:24:44  *** Victorsueca has quit IRC
1622017-03-10T11:25:27  *** Victorsueca has joined #bitcoin-core-dev
1632017-03-10T11:58:57  *** jtimon has quit IRC
1642017-03-10T12:10:29  *** BashCo_ has joined #bitcoin-core-dev
1652017-03-10T12:11:07  *** cryptapus_afk is now known as cryptapus
1662017-03-10T12:12:31  *** wasi has quit IRC
1672017-03-10T12:13:56  *** BashCo has quit IRC
1682017-03-10T12:20:08  <MarcoFalke> no, the salvagewallet issue is not yet fixed
1692017-03-10T12:21:20  <MarcoFalke> At least I am not aware that anyone fixed it and the tests are still disabled
1702017-03-10T12:22:01  *** jtimon has joined #bitcoin-core-dev
1712017-03-10T12:22:55  *** jannes has joined #bitcoin-core-dev
1722017-03-10T12:28:52  <wumpus> MarcoFalke: from what I remember what causes the test to fail is that salvagewallet can return false when the answer should be true. I don't think any keys are still being lost
1732017-03-10T12:39:05  *** jtimon has quit IRC
1742017-03-10T12:39:28  <wumpus> at least that was my experience from last time I tried to reproduce the issue
1752017-03-10T12:40:19  *** cryptapus has joined #bitcoin-core-dev
1762017-03-10T12:42:58  <MarcoFalke> ok, going to enable the test on the nightly builds and see what happens
1772017-03-10T12:58:02  <wumpus> good idea.
1782017-03-10T12:58:57  <wumpus> what I also found back then is that if you have a wallet that it fails on, it's fully reproducible with that. So it's something in the specific database that triggers it
1792017-03-10T12:59:48  <Victorsueca> be careful, maybe the code becomes self-aware and starts gathering private keys all around the world and then spends all UTXOs to 1Yoink.... :P
1802017-03-10T13:18:27  <wumpus> :p
1812017-03-10T13:27:21  *** paveljanik has quit IRC
1822017-03-10T13:27:23  *** cryptapus is now known as cryptapus_afk
1832017-03-10T13:31:18  *** Guyver2 has joined #bitcoin-core-dev
1842017-03-10T13:32:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
1852017-03-10T13:45:00  *** alpalp has joined #bitcoin-core-dev
1862017-03-10T13:45:00  *** alpalp has joined #bitcoin-core-dev
1872017-03-10T13:54:01  *** alpalp has quit IRC
1882017-03-10T13:56:31  <morcos> Re: fee estimation and RBF.  My plan for Core wallet is as follows:
1892017-03-10T13:56:54  <morcos> Fee estimation allowed for targets of: 2, 4, 6, 12, 24, 48, 144, 1008
1902017-03-10T13:57:41  <morcos> There will be 2 types of estimate for each target, a conservative estimate (probably not too different from todays estimates, but still a bit less conservative) and an actual estimate
1912017-03-10T13:58:19  <morcos> I was imagining some kind of interaction between those and RBF, such that if you don't have RBF enabled then it prompts you to use the conservative estimate or something
1922017-03-10T13:58:58  <morcos> Via RPC, you'll be able to get all kinds of more specific information if you choose
1932017-03-10T14:00:24  <morcos> Turns out its a bit tricky to do the longer time horizon estimates, b/c if your node hasn't been up for that long, you can't really know..  And if you shut your node down, it doesn't currently record all the txs stuck in your mempool as failing.
1942017-03-10T14:00:50  <morcos> Fixing these issues is what's holding me up now...   And then I think I may have a performance problem to fix.
1952017-03-10T14:01:44  <petertodd> morcos: whatever you do, I'd suggest you plan for far more opt-in usage in the future; with ppl spamming the network to apparently raise fees the next round may make whatever estimating thing we come up with unreliable; tx replacement otoh is much harder to game
1962017-03-10T14:02:25  *** dcousens_ has quit IRC
1972017-03-10T14:02:52  <morcos> Yeah I think a separate but related project is to have an auto-replace mode...
1982017-03-10T14:02:56  <petertodd> (e.g. something that ignored opt-in txs entirely my fail if the % of them becomes much higher)
1992017-03-10T14:03:15  <petertodd> yeah. auto-replace mode is nice - gmaxwell has a neat way to do it where nlocktime is used to ensure replacements can be done prematurely
2002017-03-10T14:03:30  <morcos> As far as fee estimates dealing with opt-in txs.. I don't think thats much of a problem.. The only issue with that was to be cautious when they were knew that some miners might nto accept them
2012017-03-10T14:03:49  <morcos> prematurely?  oh you mean sign in advance?
2022017-03-10T14:04:11  <petertodd> morcos: exactly! he suggested signing n contradictory txs, each with a higher nlocktime and higher fee
2032017-03-10T14:04:25  *** dcousens has joined #bitcoin-core-dev
2042017-03-10T14:04:30  <petertodd> morcos: particulkarly relevant for hw wallets like trezor
2052017-03-10T14:04:39  <morcos> interesting...  and maybe you could have a differential relay too... :)
2062017-03-10T14:05:05  <petertodd> morcos: yeah, differential relay would be nice, although at least replacement BW just increases the average; peak BW is our actual bottleneck
2072017-03-10T14:16:20  *** mol has joined #bitcoin-core-dev
2082017-03-10T14:16:44  *** molz_ has quit IRC
2092017-03-10T14:20:45  *** Giszmo has joined #bitcoin-core-dev
2102017-03-10T14:35:40  *** voyager_ has quit IRC
2112017-03-10T14:44:30  *** aalex__ has joined #bitcoin-core-dev
2122017-03-10T14:46:49  *** BashCo has joined #bitcoin-core-dev
2132017-03-10T14:47:38  *** BashCo__ has joined #bitcoin-core-dev
2142017-03-10T14:48:06  *** voyager_ has joined #bitcoin-core-dev
2152017-03-10T14:49:28  *** BashCo_ has quit IRC
2162017-03-10T14:51:04  *** BashCo has quit IRC
2172017-03-10T15:06:23  *** Magma has quit IRC
2182017-03-10T15:07:14  *** Magma has joined #bitcoin-core-dev
2192017-03-10T15:07:35  *** dcousens has quit IRC
2202017-03-10T15:17:46  *** roidster has joined #bitcoin-core-dev
2212017-03-10T15:17:47  *** roidster is now known as Guest65095
2222017-03-10T15:19:58  *** schmidty has joined #bitcoin-core-dev
2232017-03-10T15:24:13  *** whphhg has quit IRC
2242017-03-10T15:37:10  *** schmidty has quit IRC
2252017-03-10T15:38:49  *** schmidty has joined #bitcoin-core-dev
2262017-03-10T15:47:47  *** whphhg has joined #bitcoin-core-dev
2272017-03-10T15:59:32  <bsm1175322> I have a need to deliver witness data to SPV clients, which necessitates a one-line change: https://github.com/VidaID/bitcoin/commit/2a3052622596db9b1fe29cd357cfc58a831b050c
2282017-03-10T15:59:55  *** riemann has quit IRC
2292017-03-10T16:00:08  <bsm1175322> Would we make this an option or something?
2302017-03-10T16:00:11  <bsm1175322> *could
2312017-03-10T16:06:03  *** bityogi has joined #bitcoin-core-dev
2322017-03-10T16:09:07  <BlueMatt> bsm1175322: you need to do this over p2p?
2332017-03-10T16:09:14  <bsm1175322> yes
2342017-03-10T16:09:57  <BlueMatt> bsm1175322: if you can do it over rpc via the gettxoutproof/verifytxoutproof stuff we could probably tweak the format to include proofs of witnesses as well, but p2p....ugh, i think everyone wants to completely remove that code sooner or later, its not good
2352017-03-10T16:10:11  <BlueMatt> bsm1175322: (need to replace it with bloom filter commitments in blocks or so)
2362017-03-10T16:10:12  <bsm1175322> BlueMatt: agreed on that.
2372017-03-10T16:10:26  <bsm1175322> Well giving every random client direct access to RPC is not a good idea for a number of reasons
2382017-03-10T16:10:44  <bsm1175322> Stage 2 of this project will be to redesign BIP37.  I'm well aware of its flaws.
2392017-03-10T16:11:06  <bsm1175322> But, for the moment we're just using it, warts and all.
2402017-03-10T16:11:10  <BlueMatt> bsm1175322: well the other thing we can do is extend the rpc to support it and then your patch will be simpler :)
2412017-03-10T16:11:31  <BlueMatt> (right now your patch isnt providing any proof of the witnesses, only the tx data, and providing the witness itself just as extra)
2422017-03-10T16:12:33  <bsm1175322> Oh I hadn't thought of that...there's a Merkle proof that's possible for the witness data too...
2432017-03-10T16:12:56  <bsm1175322> But...who would care about that?  The client can verify that the witness signature is correct...
2442017-03-10T16:13:08  <BlueMatt> bsm1175322: yea, would have to provide the merkle path to coinbase + merkle path to the witness in question
2452017-03-10T16:13:24  <BlueMatt> well assuming they have the transactions being spent, sure
2462017-03-10T16:14:14  <bsm1175322> Hmmm since you're here in NYC...we should get together soon and have a brainstorming session about a BIP37 replacement...
2472017-03-10T16:16:22  <BlueMatt> sure, iirc there was some ml post not too long ago on it
2482017-03-10T16:16:40  <BlueMatt> something about committed bloom filters, I dont recall if they concluded that the filters were too big to be practical or if they were excited though
2492017-03-10T16:16:44  <BlueMatt> maybe it was a few months ago.....
2502017-03-10T16:17:13  <bsm1175322> Oh that one...and I calculated the size was unreasonable...
2512017-03-10T16:17:21  <BlueMatt> awww, damn
2522017-03-10T16:17:31  <BlueMatt> well, ok, brainstorming it is
2532017-03-10T16:17:47  <bsm1175322> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012674.html
2542017-03-10T16:17:58  <BlueMatt> oh wow that was months ago
2552017-03-10T16:18:06  <bsm1175322> original: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012636.html
2562017-03-10T16:18:19  *** CubicEarth has joined #bitcoin-core-dev
2572017-03-10T16:20:56  <bsm1175322> UTXO set commitment in some form are #1 on my wishlist for improving light client security.
2582017-03-10T16:20:58  <gribble> https://github.com/bitcoin/bitcoin/issues/1 | JSON-RPC support for mobile devices ("ultra-lightweight" clients) · Issue #1 · bitcoin/bitcoin · GitHub
2592017-03-10T16:22:59  <bitcoin-git> [bitcoin] sdaftuar opened pull request #9970: Improve readability of segwit.py (master...2017-03-segwit-test-improvements) https://github.com/bitcoin/bitcoin/pull/9970
2602017-03-10T16:23:44  <BlueMatt> bsm1175322: hmm, yea, 12 GB total isnt trivial, though I dont think its insane...I mean its not like you download that whole dataset unless you dont know when your keys were created
2612017-03-10T16:23:53  <BlueMatt> bsm1175322: does have lots of challenges, though :(
2622017-03-10T16:24:04  <bsm1175322> There's probably a workable mutation of that idea...
2632017-03-10T16:24:18  <BlueMatt> bsm1175322: utxo commitments dont help you sync, though, but, yea, are a huge win
2642017-03-10T16:24:49  <BlueMatt> its unclear what the "right" solution is, I mean scanning the chain for your transactions makes less and less sense every day, especially given that folks are moving towards off-chain txn
2652017-03-10T16:24:53  <BlueMatt> cant scan for those.....
2662017-03-10T16:25:41  <bsm1175322> That's a different problem entirely ;-)  SPV-lightning...
2672017-03-10T16:26:38  <bsm1175322> Another idea I'm a fan of on this topic is andytoshi's PoW skiplists...
2682017-03-10T16:29:03  *** Alina-malina has joined #bitcoin-core-dev
2692017-03-10T16:29:09  <BlueMatt> yea, PoW skiplists are cool, though you have to be careful with them depending on your use-case
2702017-03-10T16:29:24  <BlueMatt> what use-case do you have for them? I mean 80 bytes * 400k blocks isnt that much, still, today?
2712017-03-10T16:29:43  <bsm1175322> Linear algorithms suck. ;-)
2722017-03-10T16:30:02  <bsm1175322> Initial sync in SPV mode still takes a non-trivial amount of time on phones.
2732017-03-10T16:30:19  <bsm1175322> It's just hashing those 400k blocks...
2742017-03-10T16:32:16  <BlueMatt> i mean we can improve that....we need a "give me all headers without the previous block hash in binary form" thing
2752017-03-10T16:32:26  <BlueMatt> mabye not p2p, just connect to blockchainheaders.com and get it
2762017-03-10T16:32:28  <BlueMatt> its 18MB
2772017-03-10T16:32:36  <BlueMatt> and hashing that cant take that long, no?
2782017-03-10T16:34:01  <bsm1175322> It does take that long, for whatever reason.  Bcoin only verifies headers at a rate of ~20/s on the phone.
2792017-03-10T16:35:09  <dgenr8> bsm1175322: did you mean committed bloom filters?
2802017-03-10T16:35:11  <bsm1175322> @chjj is there any other reason you can think of besides sha256 speed which might be causing spv sync to be slower on the phone?  Obviously it's quite fast in nodejs.
2812017-03-10T16:35:37  <bsm1175322> dgenr8: yes that's what we were discussing
2822017-03-10T16:36:25  <BlueMatt> bsm1175322: I mean I assume its also p2p latency, which isnt fun
2832017-03-10T16:36:28  <dgenr8> BlueMatt: size unreasonable with what fp rate
2842017-03-10T16:36:31  <BlueMatt> 20/s seems supperrr slow
2852017-03-10T16:37:22  <bsm1175322> dgenr8: see the referenced post with my calculation, I used fp rate= 1/height ~ 10^-6
2862017-03-10T16:37:37  <bsm1175322> came out to about 12GB of committed filters.
2872017-03-10T16:37:46  <bsm1175322> obviously this can be tuned...
2882017-03-10T16:37:53  <bsm1175322> At the cost of holding blocks you don't need.
2892017-03-10T16:37:56  <BlueMatt> (though we may want a similar command to remove checkpoints entirely for ibd - much easier to connect to a few peers and ask them for 4MB of headers (~100k blocks) at a time and then get a header chain and ban anyone who tried to spam you
2902017-03-10T16:38:02  <BlueMatt> instead of the current getheaders stuff
2912017-03-10T16:38:38  <BlueMatt> bsm1175322: to be fair, you probably want something much higher than 10^-6
2922017-03-10T16:38:47  <bsm1175322> yes
2932017-03-10T16:38:50  <BlueMatt> bsm1175322: you definitely want to download some extra blocks
2942017-03-10T16:39:14  *** kadoban has quit IRC
2952017-03-10T16:40:12  <bsm1175322> The desirable false positive rate is still (constant)/(height) though, or it's still a linear algorithm...
2962017-03-10T16:41:14  <BlueMatt> bsm1175322: welcome to the real world, low-cost linear is perfectly ok :P
2972017-03-10T16:42:11  <BlueMatt> as long as batteries and lte improve faster than your linear increases, at least for as long as you're not using 2nd layer stuff, you should be fine :)
2982017-03-10T16:42:42  <bsm1175322> Only in the bitcoin world...it makes for a horrible user experience. :-P
2992017-03-10T16:42:45  <BlueMatt> (well, and data caps...fuck data caps)
3002017-03-10T16:43:02  <bsm1175322> I'll keep looking for logarithmic solutions :-P
3012017-03-10T16:43:23  <BlueMatt> 10 seconds in a linear scan isnt all that much different from 10 seconds in a magical logarithmic scan, at least for users :P
3022017-03-10T16:43:35  <BlueMatt> I see your point, but I'm less worried
3032017-03-10T16:43:49  <BlueMatt> superlinear, well...lets not do that
3042017-03-10T16:44:16  <bsm1175322> Well...having been doing dev work on testnet for a few months...where it takes 30 minutes for bcoin to do an initial spv sync...
3052017-03-10T16:45:00  <BlueMatt> lol, ok, fair point
3062017-03-10T16:45:14  <BlueMatt> see previous comment about chunking header requests with new p2p messages
3072017-03-10T16:45:18  <BlueMatt> :)
3082017-03-10T16:45:41  <BlueMatt> its slow because we've been busy optimizing other things, should be easy to optimize, though
3092017-03-10T16:47:25  <bsm1175322> Yeah I'm going to have to look into that on bcoin.  For now we're beta testing on regtest so are avoiding the problem.
3102017-03-10T16:49:21  *** abpa has joined #bitcoin-core-dev
3112017-03-10T16:53:04  *** aalex__ has quit IRC
3122017-03-10T16:54:53  *** riemann has joined #bitcoin-core-dev
3132017-03-10T17:00:12  *** neha has quit IRC
3142017-03-10T17:01:37  *** neha has joined #bitcoin-core-dev
3152017-03-10T17:02:17  *** aalex__ has joined #bitcoin-core-dev
3162017-03-10T17:03:38  *** Alina-malina has quit IRC
3172017-03-10T17:03:39  *** Alina-malina has joined #bitcoin-core-dev
3182017-03-10T17:05:46  *** arubi has quit IRC
3192017-03-10T17:07:15  *** arubi has joined #bitcoin-core-dev
3202017-03-10T17:11:51  *** nsh has quit IRC
3212017-03-10T17:11:52  *** ensign- has quit IRC
3222017-03-10T17:12:42  *** musalbas has quit IRC
3232017-03-10T17:13:49  *** musalbas has joined #bitcoin-core-dev
3242017-03-10T17:14:02  *** nsh has joined #bitcoin-core-dev
3252017-03-10T17:17:03  *** ensign_ has joined #bitcoin-core-dev
3262017-03-10T17:21:59  *** voyager_ has quit IRC
3272017-03-10T17:22:49  *** voyager_ has joined #bitcoin-core-dev
3282017-03-10T17:29:57  *** Alina-malina has quit IRC
3292017-03-10T17:30:13  *** Alina-malina has joined #bitcoin-core-dev
3302017-03-10T17:32:15  *** Alina-malina has quit IRC
3312017-03-10T17:32:16  *** Alina-malina has joined #bitcoin-core-dev
3322017-03-10T17:40:01  <bsm1175322> BlueMatt: FYI another idea that's floating in my head for improving SPV is whether we could use some form of https://en.wikipedia.org/wiki/Oblivious_transfer
3332017-03-10T17:44:38  <BlueMatt> bsm1175322: hmm, possibly useful to receive blocks after a high-fp-rate filter commitment or something? Maybe too high overhead, though, gmaxwell might have more to say
3342017-03-10T17:45:09  <bsm1175322> To be clear, I haven't figured out any algorithm that works and I'm not making a proposal...but I want to find a way...
3352017-03-10T17:45:52  <BlueMatt> heh, ok
3362017-03-10T17:46:01  *** CubicEarth has quit IRC
3372017-03-10T17:46:09  <BlueMatt> the old "that sounds cool, we should use it somewhere" approach :)
3382017-03-10T17:46:52  <bsm1175322> exactly
3392017-03-10T18:02:27  *** tripleslash has quit IRC
3402017-03-10T18:03:44  *** tripleslash has joined #bitcoin-core-dev
3412017-03-10T18:24:38  <gmaxwell> .... there have been many concrete proposals before. the performance is bad though.
3422017-03-10T18:32:33  <bsm1175322> Yes...in order to be oblivous about which of N bytes are sent, you have to read/process all N bytes, for *each* request...
3432017-03-10T18:32:52  <bsm1175322> I'm still hoping there's a way around that observation...
3442017-03-10T18:35:44  <BlueMatt> uhh, that would be kinda obvious if you didnt....
3452017-03-10T18:35:58  <BlueMatt> "hey, i dont know what I'm sending you, but its one of A or B, and I never read B from my hdd......"
3462017-03-10T18:36:36  <bsm1175322> Insert some preprocessing/Merkle tree magic...
3472017-03-10T18:38:22  <BlueMatt> heh
3482017-03-10T18:42:45  *** BirneGetreide_ has quit IRC
3492017-03-10T18:43:25  <gmaxwell> bsm1175322: https://bitcointalk.org/index.php?topic=1762589.0
3502017-03-10T18:44:43  *** BirneGetreide_ has joined #bitcoin-core-dev
3512017-03-10T18:45:58  *** BashCo__ has quit IRC
3522017-03-10T18:46:34  *** BashCo has joined #bitcoin-core-dev
3532017-03-10T18:46:55  *** adiabat has joined #bitcoin-core-dev
3542017-03-10T18:51:04  *** BashCo has quit IRC
3552017-03-10T18:52:57  *** Guest65095 has quit IRC
3562017-03-10T19:09:44  *** BashCo has joined #bitcoin-core-dev
3572017-03-10T19:56:47  <bitcoin-git> [bitcoin] MarcoFalke opened pull request #9971: qa: Initialize log in TestManager (master...Mf1703-logFixup) https://github.com/bitcoin/bitcoin/pull/9971
3582017-03-10T19:57:46  *** jtimon has joined #bitcoin-core-dev
3592017-03-10T20:14:28  *** shesek has quit IRC
3602017-03-10T20:18:26  *** BirneGetreide_ has quit IRC
3612017-03-10T20:19:41  *** str4d has joined #bitcoin-core-dev
3622017-03-10T20:24:03  <bitcoin-git> [bitcoin] jnewbery opened pull request #9972: Fix extended rpc tests broken by #9768 (master...test_logging_fixups) https://github.com/bitcoin/bitcoin/pull/9972
3632017-03-10T20:24:56  *** chjj has quit IRC
3642017-03-10T20:27:48  *** shesek has joined #bitcoin-core-dev
3652017-03-10T20:30:27  *** Chris_Stewart_5 has quit IRC
3662017-03-10T20:38:20  *** chjj has joined #bitcoin-core-dev
3672017-03-10T20:40:12  <gmaxwell> https://www.reddit.com/r/Bitcoin/comments/5yojyp/on_the_recent_bout_of_malleated_transactions/
3682017-03-10T20:40:41  <midnightmagic> \o/
3692017-03-10T20:52:48  *** Lauda has quit IRC
3702017-03-10T20:54:19  *** Lauda has joined #bitcoin-core-dev
3712017-03-10T21:04:54  *** jtimon has quit IRC
3722017-03-10T21:05:12  *** CubicEarth has joined #bitcoin-core-dev
3732017-03-10T21:08:22  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3742017-03-10T21:08:42  *** roidster has joined #bitcoin-core-dev
3752017-03-10T21:08:45  *** roidster is now known as Guest95471
3762017-03-10T21:15:24  *** jnewbery has quit IRC
3772017-03-10T21:21:49  *** Guyver2 has quit IRC
3782017-03-10T21:53:04  <bitcoin-git> [bitcoin] theuni opened pull request #9973: depends: fix zlib build on osx (master...fix-zlib-osx) https://github.com/bitcoin/bitcoin/pull/9973
3792017-03-10T21:53:27  *** tripleslash has quit IRC
3802017-03-10T21:55:50  *** tripleslash has joined #bitcoin-core-dev
3812017-03-10T21:57:00  <bitcoin-git> [bitcoin] ryanofsky opened pull request #9974: Add basic Qt wallet test (master...pr/qt-test) https://github.com/bitcoin/bitcoin/pull/9974
3822017-03-10T22:00:30  *** tripleslash has quit IRC
3832017-03-10T22:01:28  *** str4d has quit IRC
3842017-03-10T22:01:44  *** fanquake has joined #bitcoin-core-dev
3852017-03-10T22:08:28  <bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8910b4717e5b...21833f9456f6
3862017-03-10T22:08:28  <bitcoin-git> bitcoin/master d055bd6 John Newbery: Fix extended rpc tests broken by 8910b4717e5bb946ee6988f7fe9fd461f53a5935
3872017-03-10T22:08:29  <bitcoin-git> bitcoin/master 21833f9 MarcoFalke: Merge #9972: Fix extended rpc tests broken by #9768...
3882017-03-10T22:08:56  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9972: Fix extended rpc tests broken by #9768 (master...test_logging_fixups) https://github.com/bitcoin/bitcoin/pull/9972
3892017-03-10T22:10:59  *** fanquake has quit IRC
3902017-03-10T22:28:32  *** harrymm1 has quit IRC
3912017-03-10T22:35:25  *** tripleslash has joined #bitcoin-core-dev
3922017-03-10T22:39:26  *** tripleslash has quit IRC
3932017-03-10T22:43:50  *** harrymm has joined #bitcoin-core-dev
3942017-03-10T22:49:53  *** tripleslash has joined #bitcoin-core-dev
3952017-03-10T22:52:04  *** str4d has joined #bitcoin-core-dev
3962017-03-10T22:54:22  *** tripleslash has quit IRC
3972017-03-10T23:01:29  *** CubicEarth has quit IRC
3982017-03-10T23:04:56  *** riemann has quit IRC
3992017-03-10T23:09:46  *** tripleslash has joined #bitcoin-core-dev
4002017-03-10T23:23:34  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9971: qa: Initialize log in TestManager (master...Mf1703-logFixup) https://github.com/bitcoin/bitcoin/pull/9971
4012017-03-10T23:27:35  *** Guest95471 has quit IRC
4022017-03-10T23:30:09  *** Telmo has joined #bitcoin-core-dev
4032017-03-10T23:30:31  *** Telmo has quit IRC
4042017-03-10T23:32:09  *** Telmo has joined #bitcoin-core-dev
4052017-03-10T23:32:23  <Telmo> Ola
4062017-03-10T23:33:10  *** Telmo has quit IRC
4072017-03-10T23:33:57  *** str4d has quit IRC
4082017-03-10T23:45:10  *** CubicEarth has joined #bitcoin-core-dev
4092017-03-10T23:52:24  *** aalex__ has quit IRC