12018-09-27T00:07:01 *** Murch has joined #bitcoin-core-dev
22018-09-27T00:11:36 *** Krellan has quit IRC
32018-09-27T00:12:42 *** Krellan has joined #bitcoin-core-dev
42018-09-27T00:16:01 *** irc_viewer_test has quit IRC
52018-09-27T00:26:39 *** murrayn has quit IRC
62018-09-27T00:28:42 *** murrayn has joined #bitcoin-core-dev
72018-09-27T00:36:46 *** promag has quit IRC
82018-09-27T00:56:19 *** Randolf has joined #bitcoin-core-dev
92018-09-27T01:05:56 *** jarthur has joined #bitcoin-core-dev
102018-09-27T01:06:25 <jarthur> Any of you pretty familiar with the functional tests?
112018-09-27T01:07:02 <jarthur> In the ones that spin up a couple peers, I'm curious how deterministic the peer numbers are when asserting log text.
122018-09-27T01:14:26 <sipa> the peer numbers are sequential
132018-09-27T01:14:31 <sipa> time of connectio
142018-09-27T01:19:26 <jarthur> Makes sense. Do the multi-peer test usually connect the nodes in sequence to avoid non-deterministic output?
152018-09-27T01:19:31 <jarthur> s/test/tests/
162018-09-27T01:25:46 *** Emcy has quit IRC
172018-09-27T01:28:36 *** Emcy has joined #bitcoin-core-dev
182018-09-27T02:01:06 <phantomcircuit> anybody know why the linter is failing on #14336
192018-09-27T02:01:08 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
202018-09-27T02:06:22 <phantomcircuit> ah i see nvm
212018-09-27T02:06:39 <gmaxwell> jarthur: I think they are in practice, but as a general rule a test should try to avoid being sensitive to things other than what they're testing, otherwise it makes them brittle.
222018-09-27T02:06:57 <phantomcircuit> the white space linter is confusing when run against multiple commits the corresponding message are nonsensical
232018-09-27T02:21:21 *** Randolf has quit IRC
242018-09-27T02:26:54 *** Murch has quit IRC
252018-09-27T02:32:03 <jarthur> gmaxwell: thanks
262018-09-27T02:45:03 *** charley has joined #bitcoin-core-dev
272018-09-27T03:02:27 <phantomcircuit> aight i have no idea why it's failing that test and cannot reproduce locally
282018-09-27T03:03:00 <gmaxwell> phantomcircuit: looking
292018-09-27T03:03:17 <phantomcircuit> https://travis-ci.org/bitcoin/bitcoin/jobs/433861290
302018-09-27T03:03:27 <phantomcircuit> bitcoind exiting with code -6 during initialization
312018-09-27T03:03:39 <gmaxwell> phantomcircuit: so thats an enable debug build.
322018-09-27T03:03:46 <phantomcircuit> yes
332018-09-27T03:04:44 <gmaxwell> so with enable debug, rpc bind functional test passes for you?
342018-09-27T03:05:28 <phantomcircuit> yes
352018-09-27T03:05:32 <phantomcircuit> all tests pass actually
362018-09-27T03:06:00 <phantomcircuit> ./configure --with-debug --with-incompatible-bdb --enable-zmq --with-gui=qt5 --enable-glibc-back-compat --enable-reduce-exports --enable-debug
372018-09-27T03:08:39 <phantomcircuit> the only -6 is see as a constant is the addrman check but that should just write to the log
382018-09-27T03:08:51 <gmaxwell> I dunno what -6 means there... like if it died on due to an unhandled signal it would one hundred and something.
392018-09-27T03:11:19 <gmaxwell> perhaps, $ errno 6
402018-09-27T03:11:19 <gmaxwell> ENXIO 6 No such device or address
412018-09-27T03:11:41 <gmaxwell> there might be a case where the test triggers an error that the old code ignores and the new code propagates.
422018-09-27T03:12:00 <gmaxwell> like what happens if you try to bind to 127.0.0.1 but there isn't a 127.0.0.1?
432018-09-27T03:13:15 *** prometheus_falli has quit IRC
442018-09-27T03:16:42 *** prometheus_falli has joined #bitcoin-core-dev
452018-09-27T03:18:18 <phantomcircuit> gmaxwell, should fail in the same ways im pretty sure
462018-09-27T03:18:51 *** irc_viewer_test has joined #bitcoin-core-dev
472018-09-27T03:21:10 <gmaxwell> in any case a reason that test may fail on travis and work for you, is that the travis enviroment probably has pretty different networking.
482018-09-27T03:21:25 <phantomcircuit> yeah im sure it does
492018-09-27T03:22:52 <gmaxwell> And the reason it would matter is if under that case, you handle errors differently, e.g. the actual bug may be elsewhere in bitcoin or in the test but were hidden by the old behavior. The fact that you can't reproduce it locally is kind of annoying. you could try to figure out which test case is failing, by adding commits to change the test.
502018-09-27T03:25:18 *** asoltys has joined #bitcoin-core-dev
512018-09-27T03:30:38 *** irc_viewer_test has quit IRC
522018-09-27T03:45:30 <phantomcircuit> gmaxwell, indeed
532018-09-27T04:06:38 *** jarthur has quit IRC
542018-09-27T04:07:16 *** jarthur has joined #bitcoin-core-dev
552018-09-27T04:09:49 <phantomcircuit> gmaxwell, i guess travis doesn't have ipv4?
562018-09-27T04:09:57 <phantomcircuit> it's specifically the ipv4 rpc bind that's failing
572018-09-27T04:11:35 <phantomcircuit> hmm it doesn't , not specifically 127.0.0.1
582018-09-27T04:11:50 <phantomcircuit> which is what the test tries to bind to
592018-09-27T04:22:40 *** jarthur has quit IRC
602018-09-27T04:29:06 <phantomcircuit> gmaxwell, blargh there's still the select() calls in netbase
612018-09-27T04:33:21 *** molz has joined #bitcoin-core-dev
622018-09-27T04:34:32 *** mol has quit IRC
632018-09-27T04:55:36 *** escrivner has joined #bitcoin-core-dev
642018-09-27T05:05:04 *** qrestlove has quit IRC
652018-09-27T05:13:53 <kallewoof> wumpus: noticed that NicolasDorier is not in the Credits list despite him being listed for 9991 at https://github.com/bitcoin-core/bitcoin-devwiki/wiki/0.17.0-Release-notes
662018-09-27T05:14:07 <kallewoof> we just edit? I thought credits were automagic
672018-09-27T05:19:56 *** itaseski has joined #bitcoin-core-dev
682018-09-27T05:39:10 *** qrestlove has joined #bitcoin-core-dev
692018-09-27T05:44:38 *** Victorsueca has quit IRC
702018-09-27T05:45:46 *** Victorsueca has joined #bitcoin-core-dev
712018-09-27T06:08:16 *** itaseski has quit IRC
722018-09-27T06:20:25 *** go1111111 has quit IRC
732018-09-27T06:33:15 *** emzy has quit IRC
742018-09-27T06:36:26 *** go1111111 has joined #bitcoin-core-dev
752018-09-27T07:02:36 *** setpill has joined #bitcoin-core-dev
762018-09-27T07:34:36 <jonasschnelli> Bitcoin Qt looks a bit strange in OSX 10.14 (Mojaves) dark mode. :)
772018-09-27T07:35:07 *** t0adst00l has joined #bitcoin-core-dev
782018-09-27T07:35:29 *** nullptr| has quit IRC
792018-09-27T07:36:25 *** prometheus_falli has quit IRC
802018-09-27T07:45:06 *** nullptr| has joined #bitcoin-core-dev
812018-09-27T08:12:16 *** Krellan has quit IRC
822018-09-27T08:12:38 *** Krellan has joined #bitcoin-core-dev
832018-09-27T08:13:30 *** emzy has joined #bitcoin-core-dev
842018-09-27T08:16:55 *** timothy has joined #bitcoin-core-dev
852018-09-27T08:25:41 *** jb55 has joined #bitcoin-core-dev
862018-09-27T08:37:02 <wumpus> kallewoof: the lists are generated automatically but always need human editing
872018-09-27T08:37:44 <wumpus> kallewoof: the list of PRs is generated from github API data, the authors list from git commit authorsâthis might be what causes the divergence in this case
882018-09-27T08:38:17 <wumpus> jonasschnelli: that's curious, I always run it with dark gtk themes and that seems to go well
892018-09-27T08:38:55 <jonasschnelli> wumpus: Could be because of the different platformstyles... investigating now
902018-09-27T08:39:19 <jonasschnelli> black icons on black background are not ideal.. but at least you can see them
912018-09-27T08:39:33 <jonasschnelli> correction: black icons on dark-gray background
922018-09-27T08:40:46 <wumpus> jonasschnelli: yes that's it, I think on UNIX it does an attempt to grab the icon color from the theme, on MacOS is defaults to black
932018-09-27T08:41:06 <jonasschnelli> jup...
942018-09-27T08:42:40 <wumpus> kallewoof: that's it; the commits in 9991 are by JeremyRubin and apparently my script lists authors only, not committers
952018-09-27T08:43:11 <wumpus> kallewoof: I'll try to fix the script, otherwise I'll just edit him in manually
962018-09-27T08:57:02 *** arubi has quit IRC
972018-09-27T08:59:27 *** promag has joined #bitcoin-core-dev
982018-09-27T09:01:01 <wumpus> kallewoof: added in committers to my list-authors script; looks like Dorier was the only one missed to this
992018-09-27T09:04:23 <wumpus> phantomcircuit: travis does have IPv4, but no IPv6 IIRC
1002018-09-27T09:04:29 <wumpus> (*not even localhost*)
1012018-09-27T09:10:25 <echeveria> I fixed testnet, for what it's worth. the most work is now in a valid chain.
1022018-09-27T09:24:49 *** hebasto has joined #bitcoin-core-dev
1032018-09-27T09:28:11 <jonasschnelli> wumpus: using the dark arc theme in Ubuntu Bionic, the background of Bitcoin Qt is still light gray/whiteish? Is that expected?
1042018-09-27T09:30:27 *** e4xit has quit IRC
1052018-09-27T09:32:44 <wumpus> jonasschnelli: I don't think so; the bitcoin-qt background should be the same as other gtk applications, say "charmap"
1062018-09-27T09:33:21 <wumpus> jonasschnelli: though that might only work if you build from source, theme integration is not available with the shipped binaries
1072018-09-27T09:33:54 <jonasschnelli> aha... I see.
1082018-09-27T09:34:41 *** e4xit has joined #bitcoin-core-dev
1092018-09-27T09:34:48 <wumpus> (the latter because it's based on plugins, which are not available when building qt statically)
1102018-09-27T09:35:47 *** SopaXorzTaker has joined #bitcoin-core-dev
1112018-09-27T09:36:12 <jonasschnelli> wumpus: I would have expected that the OS provides some sort of color scheme for background, etc. (macOS Mojave) handles it that way). Qt could use OsBackgroundColor(), etc. for certain things.
1122018-09-27T09:36:20 <wumpus> huhâlooks broken on ubuntu 18.04, I get a light grey background now too w/ Adwaita-dark theme
1132018-09-27T09:36:33 <wumpus> built against system qt
1142018-09-27T09:36:42 <wumpus> AHH when did this happen
1152018-09-27T09:36:42 <jonasschnelli> hmm...
1162018-09-27T09:36:54 <jonasschnelli> Must be a Qt upstream issue
1172018-09-27T09:36:59 <jonasschnelli> (or a flag)
1182018-09-27T09:37:14 <wumpus> yes, indeed
1192018-09-27T09:37:25 <jonasschnelli> https://askubuntu.com/questions/910012/how-can-i-get-qt5-applications-to-use-the-gtk-theme-in-ubuntu-17-04
1202018-09-27T09:37:31 <wumpus> I see the same in Quasselclient
1212018-09-27T09:37:37 <wumpus> which is also a qt (5 AFAIK) app
1222018-09-27T09:37:54 <jonasschnelli> «The problem has occurred since Qt5.7. In this release, the GTK2 platform theme and style was removed and replaced with the GTK3 platform theme»
1232018-09-27T09:38:29 <wumpus> but how! I'm sure my theme also covers gtk3
1242018-09-27T09:38:59 <wumpus> 'charmap' is gtk3 and has a dark grey background
1252018-09-27T09:39:53 * jonasschnelli loving QT
1262018-09-27T09:42:47 *** t0adst00l has quit IRC
1272018-09-27T09:44:55 *** belcher has joined #bitcoin-core-dev
1282018-09-27T09:52:45 <wumpus> I guess... it is the least of evils, with regard to cross-platform GUIs. Though losing theme integration on by far most linux distributions (except KDE-based ones?) is a pity.
1292018-09-27T09:56:23 <wumpus> installing qt5-style-plugins and then doing 'export QT_QPA_PLATFORMTHEME=gtk2' works here, but that's helluva awkward
1302018-09-27T09:57:20 <hebasto> wumpus: what is the way to build bitcoin-qt against system qt?
1312018-09-27T10:06:24 <wumpus> hebasto: it does that by default if you don't do a depends build
1322018-09-27T10:06:33 <wumpus> just follow the steps in build-unix.md
1332018-09-27T10:09:16 <wumpus> that way, bitcoin won't magically install its own qt; so it will build against system qt if available, and not build against qt at all if not available
1342018-09-27T10:12:00 <wumpus> this is the preferred way, we even used to ship that way for qt4, but unfortunately the range of portability of the executables is much smaller then so it's not an option anymore (then again, nothing of this matters anymore, if qt upstream broke desktop integration...)
1352018-09-27T10:17:29 *** t0adst00l has joined #bitcoin-core-dev
1362018-09-27T10:23:30 <hebasto> wumpus: thank you
1372018-09-27T10:28:24 *** Krellan has quit IRC
1382018-09-27T10:30:44 *** Krellan has joined #bitcoin-core-dev
1392018-09-27T10:43:06 *** SopaXorzTaker has quit IRC
1402018-09-27T10:43:37 *** SopaXorzTaker has joined #bitcoin-core-dev
1412018-09-27T10:57:32 *** Victor_sueca has joined #bitcoin-core-dev
1422018-09-27T10:57:54 *** Victorsueca has quit IRC
1432018-09-27T11:00:47 *** Victor_sueca has quit IRC
1442018-09-27T11:01:38 *** Victor_sueca has joined #bitcoin-core-dev
1452018-09-27T11:10:07 <provoostenator> wumpus: "great" to hear macOS isn't the only platform having QT headaches now.
1462018-09-27T11:17:54 *** setpill has quit IRC
1472018-09-27T11:20:01 *** Kvaciral has quit IRC
1482018-09-27T11:25:06 *** Krellan has quit IRC
1492018-09-27T11:26:05 *** Krellan has joined #bitcoin-core-dev
1502018-09-27T11:56:51 *** promag has quit IRC
1512018-09-27T12:25:03 *** adiabat has quit IRC
1522018-09-27T12:26:06 <wumpus> GUIs were a mistake
1532018-09-27T12:30:19 <Sentineo> an audio interface would have been betterperhaps :)
1542018-09-27T12:33:26 *** schnerchi has joined #bitcoin-core-dev
1552018-09-27T12:41:43 <wumpus> direct brain interface ftw :)
1562018-09-27T12:42:22 <Sentineo> yeah :)
1572018-09-27T12:56:11 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1582018-09-27T13:10:41 *** elichai2 has joined #bitcoin-core-dev
1592018-09-27T13:22:03 *** scroll2 has joined #bitcoin-core-dev
1602018-09-27T13:24:38 *** csknk has joined #bitcoin-core-dev
1612018-09-27T13:30:24 *** irc_viewer_test has joined #bitcoin-core-dev
1622018-09-27T13:37:29 *** promag has joined #bitcoin-core-dev
1632018-09-27T13:48:50 *** irc_viewer_test1 has joined #bitcoin-core-dev
1642018-09-27T13:50:47 <promag> hebasto: hi
1652018-09-27T13:51:14 <hebasto> promag: hi
1662018-09-27T13:51:23 <promag> hebasto: I don't have space to install fedora :/ but I hope to get around that today/tomorrow
1672018-09-27T13:51:30 *** irc_viewer_test has quit IRC
1682018-09-27T13:51:45 <hebasto> promag: I see
1692018-09-27T13:51:56 <promag> hebasto: can only test by then
1702018-09-27T13:53:10 <hebasto> promag: don't be hurry, I'm working on your suggestions right now
1712018-09-27T13:53:39 <promag> so in fact it's necessary to not map?
1722018-09-27T13:54:17 <hebasto> yes, it is. I've tested on fedora.
1732018-09-27T14:00:08 <hebasto> promag: would you mind to look into #14222?
1742018-09-27T14:00:09 <gribble> https://github.com/bitcoin/bitcoin/issues/14222 | Qt: Fix restoration of minimized to tray window by hebasto · Pull Request #14222 · bitcoin/bitcoin · GitHub
1752018-09-27T14:01:37 *** Emcy has quit IRC
1762018-09-27T14:04:53 *** Emcy has joined #bitcoin-core-dev
1772018-09-27T14:16:03 <promag> hebasto: will fo
1782018-09-27T14:16:08 <promag> *do
1792018-09-27T14:28:11 *** profmac has joined #bitcoin-core-dev
1802018-09-27T14:34:57 *** AaronvanW has quit IRC
1812018-09-27T14:37:32 *** csknk has quit IRC
1822018-09-27T14:39:15 *** promag has quit IRC
1832018-09-27T14:39:50 *** AaronvanW has joined #bitcoin-core-dev
1842018-09-27T14:50:14 *** michaelsdunn1 has joined #bitcoin-core-dev
1852018-09-27T14:55:29 *** michaelsdunn1 has quit IRC
1862018-09-27T15:01:46 *** michaelsdunn1 has joined #bitcoin-core-dev
1872018-09-27T15:02:23 *** irc_viewer_test1 has quit IRC
1882018-09-27T15:16:43 *** jarthur has joined #bitcoin-core-dev
1892018-09-27T15:24:28 *** Murch has joined #bitcoin-core-dev
1902018-09-27T15:28:10 *** TheCharlatan has joined #bitcoin-core-dev
1912018-09-27T15:34:00 *** irc_viewer_test has joined #bitcoin-core-dev
1922018-09-27T15:39:54 *** Victor_sueca is now known as Victorsueca
1932018-09-27T15:40:24 *** emilengler has joined #bitcoin-core-dev
1942018-09-27T15:43:01 *** irc_viewer_test has quit IRC
1952018-09-27T16:09:54 *** Krellan has quit IRC
1962018-09-27T16:10:46 *** emilengler has quit IRC
1972018-09-27T16:10:59 *** Krellan has joined #bitcoin-core-dev
1982018-09-27T16:12:01 *** emilengler has joined #bitcoin-core-dev
1992018-09-27T16:13:18 *** emilengler has quit IRC
2002018-09-27T16:19:46 *** adiabat has joined #bitcoin-core-dev
2012018-09-27T16:28:06 <luke-jr> 57b59260952742aa51dca79a37849429a456496292d3e4f28cdf7de3eef516f3
2022018-09-27T16:29:12 *** Krellan has quit IRC
2032018-09-27T16:32:06 *** jarthur has quit IRC
2042018-09-27T16:32:16 *** promag has joined #bitcoin-core-dev
2052018-09-27T16:34:26 *** Krellan has joined #bitcoin-core-dev
2062018-09-27T16:36:36 *** jarthur has joined #bitcoin-core-dev
2072018-09-27T16:40:37 <promag> luke-jr: heh
2082018-09-27T16:54:04 *** jungly has quit IRC
2092018-09-27T16:55:51 *** rex4539 has quit IRC
2102018-09-27T16:58:47 *** tryphe has joined #bitcoin-core-dev
2112018-09-27T17:06:18 *** tryphe has quit IRC
2122018-09-27T17:14:11 *** promag has quit IRC
2132018-09-27T17:42:45 *** t0adst00l has quit IRC
2142018-09-27T17:44:21 *** Victorsueca has quit IRC
2152018-09-27T17:45:39 *** Victorsueca has joined #bitcoin-core-dev
2162018-09-27T17:54:03 *** escrivner has left #bitcoin-core-dev
2172018-09-27T18:15:52 *** tryphe has joined #bitcoin-core-dev
2182018-09-27T18:19:42 *** timothy has quit IRC
2192018-09-27T18:24:09 *** irc_viewer_test has joined #bitcoin-core-dev
2202018-09-27T18:28:14 *** Krellan has quit IRC
2212018-09-27T18:48:16 *** irc_viewer_test has quit IRC
2222018-09-27T18:49:31 *** promag has joined #bitcoin-core-dev
2232018-09-27T18:55:39 *** clarkmoody has joined #bitcoin-core-dev
2242018-09-27T19:00:33 <wumpus> meeting time?
2252018-09-27T19:00:36 <jonasschnelli> hiyes
2262018-09-27T19:00:52 <wumpus> #startmeeting
2272018-09-27T19:00:52 <lightningbot> Meeting started Thu Sep 27 19:00:52 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
2282018-09-27T19:00:52 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
2292018-09-27T19:01:02 <promag> hi
2302018-09-27T19:01:19 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircu
2312018-09-27T19:01:21 <provoostenator> hi
2322018-09-27T19:01:22 <wumpus> it codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
2332018-09-27T19:01:26 <cfields> hi
2342018-09-27T19:01:29 <achow101> hi
2352018-09-27T19:02:01 <meshcollider> Hi
2362018-09-27T19:02:24 <jonasschnelli> Status of 0.15.2 and 0.14.3?
2372018-09-27T19:02:45 <phantomcircuit> hi
2382018-09-27T19:03:27 <wumpus> jonasschnelli: good question; are there enough gitian sigs to upload binaries?
2392018-09-27T19:03:42 <jonasschnelli> I think so... 5 or 6
2402018-09-27T19:03:47 <luke-jr> yeah
2412018-09-27T19:04:15 <jamesob> hi
2422018-09-27T19:04:19 <achow101> 0.14.3 has 6, 0.15.2 has 5
2432018-09-27T19:04:19 <jonasschnelli> 6 for 0.14.3 and 5 for 0.15.2 AFAIK
2442018-09-27T19:04:34 <wumpus> ok, will do that then, I'm back from Riga so I'm able to sign/upload binaries again
2452018-09-27T19:04:36 <jonasschnelli> due to my shitty setup, I can't compile trusty stuff on Gitian anymore
2462018-09-27T19:04:46 <jonasschnelli> thanks wumpus
2472018-09-27T19:05:05 <wumpus> 0.14/0.15 build still seems to work here
2482018-09-27T19:05:31 <provoostenator> It took some pain for me as well, but I still keep an old Gitian VM for backports.
2492018-09-27T19:05:32 <wumpus> #topic 0.17.0 release
2502018-09-27T19:06:24 <jonasschnelli> Not sure if #14339 is something we want to address for 0.17... probably not
2512018-09-27T19:06:24 <gribble> https://github.com/bitcoin/bitcoin/issues/14339 | Qt 0.17.0rc4 (and master) not running on Ubuntu 14.04.5 LTS · Issue #14339 · bitcoin/bitcoin · GitHub
2522018-09-27T19:06:43 <provoostenator> So macOS GUI compilation seems completely broken: #14327, but that wouldn't stop cross compling a release I suppose.
2532018-09-27T19:06:44 <gribble> https://github.com/bitcoin/bitcoin/issues/14327 | macOS Mojave QT 5.11 compilation fails · Issue #14327 · bitcoin/bitcoin · GitHub
2542018-09-27T19:06:49 <wumpus> so the blocker for 0.17 is #14289
2552018-09-27T19:06:50 <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
2562018-09-27T19:07:25 <wumpus> the GUI problems are annoying and need to be fixed but are not blocking the release at this stage, IMO
2572018-09-27T19:07:25 <instagibbs> hi
2582018-09-27T19:07:32 <jonasschnelli> provoostenator: hmm.. compiled master yesterday on a fresh Mojave installation
2592018-09-27T19:07:36 <jonasschnelli> (no problems)
2602018-09-27T19:07:50 <jonasschnelli> wumpus: agree
2612018-09-27T19:08:18 <provoostenator> jonasschnelli: ok, maybe it's machine specific? Can you and others comment on that issue?
2622018-09-27T19:08:26 <jonasschnelli> (will do later)
2632018-09-27T19:09:15 *** jimmysong has joined #bitcoin-core-dev
2642018-09-27T19:09:23 <provoostenator> (I'm trying now on my Macbook as well, maybe it's just my iMac being a pain)
2652018-09-27T19:09:27 *** SopaXorzTaker has quit IRC
2662018-09-27T19:09:42 <jonasschnelli> What about #14289 and #14104?
2672018-09-27T19:09:43 <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
2682018-09-27T19:09:44 <gribble> https://github.com/bitcoin/bitcoin/issues/14104 | 0.17.2rc issue (standardness change for bare multisig) · Issue #14104 · bitcoin/bitcoin · GitHub
2692018-09-27T19:09:50 <wumpus> although, it's likely that #14289 is not a regression for 0.17 it's still something that needs to be addressed in some way
2702018-09-27T19:09:51 <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
2712018-09-27T19:10:05 <wumpus> jonasschnelli: I don't think #14104 is a blocker, but the other one is nasty
2722018-09-27T19:10:06 <gribble> https://github.com/bitcoin/bitcoin/issues/14104 | 0.17.2rc issue (standardness change for bare multisig) · Issue #14104 · bitcoin/bitcoin · GitHub
2732018-09-27T19:10:23 <provoostenator> 14289: one "solution" could be to recommend people to resync if they're still on v0.13, since it's impractically slow anyway even without the memory problem.
2742018-09-27T19:10:37 <jonasschnelli> We just need to make sure to communicate the standardness change in 0.17.0
2752018-09-27T19:10:55 <provoostenator> Or they can install 0.15 first, wait for that process to finish and then install 0.17
2762018-09-27T19:11:05 <jonasschnelli> meh
2772018-09-27T19:11:10 <sipa> hi, i'm sortof here - ping me if needed
2782018-09-27T19:11:17 <wumpus> provoostenator: agree; though putting a message in the code itself if people are upgrading from 0.13.0 would make sense, for those not carefully reading the release notes, but anyhow
2792018-09-27T19:11:57 <gmaxwell> I think even though 0.16 appears to have added the replay bloat, 0.17 makes bloat worse because it adds an additional place where they're queued. (this doesn't change that I think notices are probably the best move for now)
2802018-09-27T19:12:00 <wumpus> but yes for the most common case (0.13.0 upgrade rollback), adding a message to the release notes would be acceptable solution
2812018-09-27T19:12:02 <luke-jr> couldn't we detect the reorg needed, and just prompt for user action instead of trying to upgrade it?
2822018-09-27T19:12:12 <provoostenator> If it can be done safely, having the upgrade simply refuse and throw an error message seems safer than just a release note.
2832018-09-27T19:12:24 <sipa> gmaxwell: i'm not sure anything was added in 0.17
2842018-09-27T19:12:40 <sipa> i blamed the txindex change, but the asynchronous processing was added earlier
2852018-09-27T19:12:48 <wumpus> so I guess there isn't really a hurry to release 0.17.0 at this point
2862018-09-27T19:12:51 <gmaxwell> sipa: txindex also schedulers queues block connections and disconnection, no?
2872018-09-27T19:13:28 <gmaxwell> regardless... We don't yet have a proper fix for the issue, and I don't think we're still learning much about 0.17rc.
2882018-09-27T19:13:52 <sipa> gmaxwell: i think (sorry no time to look now) is that 0.17 just added the txindex as a listener for those blockconnected events; the issue is not that, but the events in the first place
2892018-09-27T19:14:05 <gmaxwell> ah.
2902018-09-27T19:14:35 <sipa> my suggestion is to just point out in release notes that upgrading from 0.13.0 or before is a known memory bloat issue, which can be worked around with -reindex
2912018-09-27T19:14:42 <gmaxwell> I didn't walk through the patches so it wasn't clear to me that the events existed before. Got it.
2922018-09-27T19:14:45 <luke-jr> (ActivateBestChain actually checks for the queue length and blocks on it getting caught up)
2932018-09-27T19:14:59 <sipa> luke-jr: it does; but RewindBlock and InvalidateBlock don't
2942018-09-27T19:15:13 <luke-jr> sipa: would it be so bad if they did?
2952018-09-27T19:15:31 <sipa> luke-jr: they need to release cs_main for that, which would be a major refactor for those functions
2962018-09-27T19:15:33 <gmaxwell> luke-jr: that can be tricky to do safely as car has to be taken to make sure they don't wait while holding any locks.
2972018-09-27T19:15:39 <gmaxwell> care*
2982018-09-27T19:15:42 <luke-jr> hmm
2992018-09-27T19:15:57 <sipa> but we can probably remove the upgrading logic from pre-segwit blocks anyway - as provoostenator says, it's already pretty unwieldy even without the memory bloat issue
3002018-09-27T19:16:12 <gmaxwell> yea, reindex might already actually be faster.
3012018-09-27T19:16:19 <sipa> i'm pretty sure it is
3022018-09-27T19:16:25 <gmaxwell> but I assumed we'd want to use the rewind for future softforks.
3032018-09-27T19:16:30 <provoostenator> Does reindex just toss out block files it doesn't need?
3042018-09-27T19:16:37 <sipa> i don't care so much that invalidateblock takes a lot of memory - it would be a nice to have if we could actually make it revert to genesis, but it's not a priority
3052018-09-27T19:16:42 <sipa> provoostenator: yes
3062018-09-27T19:16:58 <gmaxwell> sipa: uh it's a little worse than that.
3072018-09-27T19:17:25 <gmaxwell> I mean it hits 64+GB usage rewinding only a couple months of blocks.
3082018-09-27T19:17:45 <sipa> yeah, ok
3092018-09-27T19:17:46 <provoostenator> And it doesn't stop once it's going.
3102018-09-27T19:18:03 <gmaxwell> indeed, and it doesn't return the memory, ever.
3112018-09-27T19:18:11 <sipa> we'll need to refactor InvalidateBlock to deal with that; doesn't sound impossible, but probably for 0.17.1?
3122018-09-27T19:18:30 <gmaxwell> Not a reason to hold 0.17, but it's not an irrelevant inefficiency.
3132018-09-27T19:18:38 <sipa> fair
3142018-09-27T19:18:42 <gmaxwell> sipa: sounds fine to me.
3152018-09-27T19:18:56 <luke-jr> <2% of the network (<2000 nodes) run non-segwit software at this point; throwing an error that we can't upgrade them anymore seems reasonable
3162018-09-27T19:19:27 <wumpus> yes
3172018-09-27T19:19:50 <sipa> luke-jr: that's a useful statistics
3182018-09-27T19:19:52 <gmaxwell> I still think we shouldn't just can the rewinding code though.
3192018-09-27T19:19:55 <provoostenator> Maybe also throw an error if the user tries to invalidate more than 10K blocks? They can always do it in increments.
3202018-09-27T19:20:06 <gmaxwell> ugh.
3212018-09-27T19:20:33 <gmaxwell> just release note it, then we'll fix it later, please don't add additional falure cases to the function.
3222018-09-27T19:20:40 <wumpus> agree with gmaxwell
3232018-09-27T19:20:45 <wumpus> please don't overdesign temporary code
3242018-09-27T19:20:58 <sipa> the refactor will effectively just do that - split it up in batches of X blocks to disconnect at once
3252018-09-27T19:21:00 *** michaelsdunn1 has quit IRC
3262018-09-27T19:21:12 <wumpus> this needs to be fixed properly, in master, and in the future we should be careful to review anything that adds a queue or global data structure for this possiblity
3272018-09-27T19:21:20 <sipa> agree
3282018-09-27T19:21:22 <wumpus> but don't spend too much time on the workaround
3292018-09-27T19:21:49 <provoostenator> I guess anyone upgrading all the way from 0.13 will probably pay more than average attention to the Upgrade Notice section in bold at the top...
3302018-09-27T19:22:32 <wumpus> I think most people still running 0.13.x do so intentionally, and won't be upgrading to 0.17.x any time soon
3312018-09-27T19:22:43 <wumpus> (not those nodes, at least!)
3322018-09-27T19:22:44 *** michaelsdunn1 has joined #bitcoin-core-dev
3332018-09-27T19:22:56 <luke-jr> most 0.13 nodes are 0.13.2 anyway
3342018-09-27T19:23:04 *** elichai2 has quit IRC
3352018-09-27T19:23:07 <sipa> yes; 0.13 is not the issue; 0.13.0 is
3362018-09-27T19:23:12 <luke-jr> sipa: https://luke.dashjr.org/programs/bitcoin/files/charts/services.html fwiw
3372018-09-27T19:23:23 <wumpus> e.g. some users want to run old node software to cross-validate
3382018-09-27T19:25:01 <wumpus> so: for 0.17.0, we should add a mention to the release notes for users upgrading from 0.13.0. This would require no new rc, so the way to final can continue as normal.
3392018-09-27T19:25:42 <achow101> there's a pr for backports, will that be fore 0.17.1, or are those going in for .0?
3402018-09-27T19:26:04 <wumpus> I haven't seen it, but unless they solve critical problems, they are not going in .0
3412018-09-27T19:26:18 <achow101> #14328
3422018-09-27T19:26:20 <gribble> https://github.com/bitcoin/bitcoin/issues/14328 | [0.17] Backports by MarcoFalke · Pull Request #14328 · bitcoin/bitcoin · GitHub
3432018-09-27T19:27:35 <wumpus> I don't think any of those are very serious
3442018-09-27T19:27:59 <wumpus> potential unititialized value sounds dangerous, but looking at the PR, it's impossible to trigger in practice
3452018-09-27T19:28:08 <wumpus> I hate that kind of PR naming
3462018-09-27T19:28:39 <provoostenator> PR bait? :-)
3472018-09-27T19:28:42 <gmaxwell> I've complained about that a number of times in the past.
3482018-09-27T19:28:57 <wumpus> me too, so many times, the guy doesn't listen to me
3492018-09-27T19:29:02 <achow101> so does that mean 0.17.0 final after the meeting?
3502018-09-27T19:29:11 <wumpus> (or gal, I don't really know)
3512018-09-27T19:29:19 <provoostenator> Works for me.
3522018-09-27T19:29:40 <wumpus> I guess so? would be nice to have the release notes change in
3532018-09-27T19:29:47 <wumpus> before tagging
3542018-09-27T19:31:05 <gmaxwell> just needs a one liner, no? "If upgrading from 0.13 or prior, you must start with -reindex"
3552018-09-27T19:31:33 <sipa> yah
3562018-09-27T19:31:58 <wumpus> I've noted it here https://github.com/bitcoin/bitcoin/issues/12391#issuecomment-425211080
3572018-09-27T19:32:01 <gmaxwell> k
3582018-09-27T19:32:08 <achow101> we also need to add a known issues section
3592018-09-27T19:33:32 <wumpus> yepp
3602018-09-27T19:33:35 <wumpus> any other topics?
3612018-09-27T19:33:56 <phantomcircuit> can anybody reproduce the travis error on #14336
3622018-09-27T19:33:58 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
3632018-09-27T19:34:02 <phantomcircuit> i cannot reproduce it locally
3642018-09-27T19:34:25 <wumpus> #topic Travis error on poll() PR
3652018-09-27T19:34:32 *** jarthur has quit IRC
3662018-09-27T19:34:49 <jonasschnelli> IMO after the meeting
3672018-09-27T19:34:58 <wumpus> I guess this is a action item, that people shoudl try the PR locally?
3682018-09-27T19:34:58 <gmaxwell> instagibbs had a related question, where are the bitcoind exit codes coming from. phantomcircuit's travis failure bitcoind has a return value of -6
3692018-09-27T19:35:12 <wumpus> after the meeting, yes, doesn't make sense to do a real-time debugging session I think :)
3702018-09-27T19:35:32 <instagibbs> I shall wait
3712018-09-27T19:35:38 <jonasschnelli> would be fun... but yes. Better later.
3722018-09-27T19:35:42 <promag> wumpus: topic suggestion: multiwallet
3732018-09-27T19:35:43 <wumpus> #action try to run tests on #14336 on different environments to see if it reproduces travis error
3742018-09-27T19:35:45 <jonasschnelli> High-Prio list?
3752018-09-27T19:35:45 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
3762018-09-27T19:36:02 <wumpus> #topic multiwallet (promag)
3772018-09-27T19:36:12 <promag> cc jnewbery
3782018-09-27T19:36:25 <promag> just want some feedback regarding https://github.com/bitcoin/bitcoin/pull/13100#issuecomment-424342813
3792018-09-27T19:36:34 <promag> also, regarding listwalletdir
3802018-09-27T19:36:49 <wumpus> jonasschnelli: I haven't paid attention to the high-prio list at all this week, with the CVE issue and Riga travel so I'm not sure there's much to do there, but sure we can discuss it
3812018-09-27T19:37:12 <jonasschnelli> I think Concept ack on both (promag)! Will test more soon.
3822018-09-27T19:37:20 <promag> and I'm going to submit a refactor PR introducing WalletsModel
3832018-09-27T19:37:37 <promag> that combines loaded wallets and existing wallets
3842018-09-27T19:37:53 <jonasschnelli> wumpus: Yeah. I only wanted to ask for a review on #14046 from gmaxwell and sipa since they both commented already on it (fixed stuff)
3852018-09-27T19:37:55 <gribble> https://github.com/bitcoin/bitcoin/issues/14046 | net: Refactor message parsing (CNetMessage), adds flexibility by jonasschnelli · Pull Request #14046 · bitcoin/bitcoin · GitHub
3862018-09-27T19:38:01 <promag> otherwise #13100 looks like junk code..
3872018-09-27T19:38:03 <gribble> https://github.com/bitcoin/bitcoin/issues/13100 | gui: Add dynamic wallets support by promag · Pull Request #13100 · bitcoin/bitcoin · GitHub
3882018-09-27T19:39:53 <promag> any objections to WalletsModel? IIRC it was previously suggested
3892018-09-27T19:40:56 <gmaxwell> I'll look at 14046 again, thanks for the ping.
3902018-09-27T19:42:54 <jonasschnelli> topic proposal: factor out berkeley-db
3912018-09-27T19:43:06 <wumpus> #topic factor out berekey-db
3922018-09-27T19:43:11 <wumpus> (jonasschnelli)
3932018-09-27T19:43:12 *** Krellan has joined #bitcoin-core-dev
3942018-09-27T19:43:15 <jonasschnelli> I tried this serval times but seems more complex then initially though..
3952018-09-27T19:43:32 <jonasschnelli> But I think we should slowly consider alternative database systems (internal) for wallet files
3962018-09-27T19:43:46 <jonasschnelli> And factroing out BDB should be done sooner then later
3972018-09-27T19:43:58 <jamesob> where did the complexity come from?
3982018-09-27T19:44:06 <provoostenator> Could be combined with jnewbery's standalone wallet tool, which can hold on to the direct dependnecy a bit longer.
3992018-09-27T19:44:20 <jonasschnelli> jamesob: I think mostly due to the complex code layering...
4002018-09-27T19:44:47 <jonasschnelli> I think switching the database backend on runtime should be possible....
4012018-09-27T19:45:08 <promag> switching the database backend on runtime should be possible <- why?
4022018-09-27T19:45:11 <jonasschnelli> I hope someone working on the wallet can initiate that refactor: jamesob, jnewbery, ryanofsky
4032018-09-27T19:45:23 <provoostenator> I assume it makes most sense to move it to leveldb since we're using that for most other things?
4042018-09-27T19:45:33 <sipa> leveldb is very annoying
4052018-09-27T19:45:34 <gmaxwell> what gah no
4062018-09-27T19:45:42 <jamesob> I don't think so; leveldb can't use a single .dat-ish file
4072018-09-27T19:45:43 <jonasschnelli> promag: we must assume there will be a pretty long "transition" phase from the old to the new database layer
4082018-09-27T19:45:45 <jonasschnelli> Not leveldb...
4092018-09-27T19:45:48 <sipa> it needs whole directories, and provides way too many features
4102018-09-27T19:45:50 <gmaxwell> and not a particularly good fit for what its used for here.
4112018-09-27T19:45:56 <jonasschnelli> sipa one wrote a simple append only database...
4122018-09-27T19:46:00 <jamesob> sqlite IMO seems like a good bet
4132018-09-27T19:46:13 <gmaxwell> jonasschnelli: do we need to assume there is a long transistion instead of a standalone conversion util?
4142018-09-27T19:46:16 <wumpus> yeah, though FWIW for the berkekeydb we also suggest a whole directory per wallet at the moment
4152018-09-27T19:46:21 <jonasschnelli> (logdb),... there is a 2-4 year old PR somewhere (search after logdb)
4162018-09-27T19:46:25 <sipa> sqlite is fine, though we also don't need any of its features, apart from safe updating
4172018-09-27T19:46:29 <gmaxwell> wumpus: yes, but we know we don't like to do that. :)
4182018-09-27T19:46:35 <jamesob> or honestly just a raw format of our own creation
4192018-09-27T19:46:47 <wumpus> but the dangerous thing here is that anything you choose for the wallet will need to be supported more or less forever, so it's not an easy choice
4202018-09-27T19:46:49 <jonasschnelli> gmaxwell: could also be a conversion utility,.. but at least the utility must be capable to run both database systems,... so won't change that much for the refactroing)
4212018-09-27T19:46:52 <provoostenator> If we add another dependency, maybe pick one that's potentially useful for block and index storage?
4222018-09-27T19:46:54 <gmaxwell> esp since we just end up loading the whole thing into memory, a fancy database is kinda overkill.
4232018-09-27T19:47:13 <jonasschnelli> gmaxwell: agree
4242018-09-27T19:47:24 <sipa> provoostenator: those things actually need efficient querying, atomic batch updates, ...
4252018-09-27T19:47:27 <wumpus> we don't *need* to load the whole thing in memroy, that's a current design choice
4262018-09-27T19:47:36 <sipa> provoostenator: for the wallet we just need a key-value store with some append only records :)
4272018-09-27T19:47:37 <gmaxwell> (and also makes the files much more fragile and harder to work with than they would be otherwise)
4282018-09-27T19:47:38 <cfields> sqlite is also nice in that they provide a monolithic source file and encourage direct integration.
4292018-09-27T19:47:47 <wumpus> if there's proper indexing, loading every single transaction into memory could be avoided
4302018-09-27T19:47:50 <sipa> yeah, i'm not opposed to sqlite
4312018-09-27T19:47:54 <jonasschnelli> logdb (#5686) would be a simple append only hard to corrupt "database"... everything is hold in memory
4322018-09-27T19:47:57 <gribble> https://github.com/bitcoin/bitcoin/issues/5686 | [Wallet] replace BDB with internal append only (logdb) backend by jonasschnelli · Pull Request #5686 · bitcoin/bitcoin · GitHub
4332018-09-27T19:47:57 <sipa> it has very thorough tests
4342018-09-27T19:48:00 <luke-jr> cfields: that's not a good thing. -.-
4352018-09-27T19:48:00 <provoostenator> Someone once complained that the wallet didn't have atomicity guarantees.
4362018-09-27T19:48:02 <gmaxwell> wumpus: indeed. but that decision should be made jointly.
4372018-09-27T19:48:03 <jonasschnelli> Or sqlite... yeah
4382018-09-27T19:48:11 <wumpus> I like sqlite, especially with deterministic wallets it wouldn't need to store all the keys
4392018-09-27T19:48:20 <jamesob> sqlite seems like a pretty safe bet
4402018-09-27T19:48:22 <cfields> luke-jr: the alternative is the reason we're switching away...
4412018-09-27T19:48:24 <wumpus> so it's pretty much a metadata database, and sqlite is great for metadata and querying metadata
4422018-09-27T19:48:32 <sipa> wumpus: with descriptor based wallets we don't need that anyway :)
4432018-09-27T19:48:35 <luke-jr> cfields: what?
4442018-09-27T19:48:41 <wumpus> sipa: even beter
4452018-09-27T19:48:42 <gmaxwell> I don't think sqlite makes much sense unless the intent is also to move away from pulling everything into memory.
4462018-09-27T19:48:50 <jonasschnelli> Is it guaranteed that sqlite databases are interoperatable between platforms and versions of sqlite?
4472018-09-27T19:48:59 <jamesob> gmaxwell: what would you propose in lieu?
4482018-09-27T19:49:01 <wumpus> please move away from loading everyting into memory in the long run
4492018-09-27T19:49:02 <gmaxwell> And if we're going to do that, the scheme in the database matters a lot, so that change should probably be made at the same time.
4502018-09-27T19:49:11 <wumpus> on the short term it's not a priority
4512018-09-27T19:49:14 <wumpus> but don't make it impossible
4522018-09-27T19:49:27 <wumpus> (in a new format)
4532018-09-27T19:49:35 <gmaxwell> jamesob: if we're loading the whole thing into memory, a simple serialized format like logdb is I think vastly superior.
4542018-09-27T19:49:37 <jonasschnelli> I agree with gmaxwell: sqlite makes most sense if we want to one active handling of merchant size wallets
4552018-09-27T19:49:46 <jonasschnelli> and not sure if we want that
4562018-09-27T19:49:56 <promag> does it have to be an embedded database?
4572018-09-27T19:49:58 <wumpus> some large users of the wallet run into memory issues, and have to remake a new wallet perioidically because of this limitation
4582018-09-27T19:50:06 <sipa> promag: no
4592018-09-27T19:50:19 <gmaxwell> if we just use sqllite but then just treat it like a blob holder, then the whole schema will need to change to avoid memory loading it in any case.
4602018-09-27T19:50:20 <phantomcircuit> jonasschnelli, think it makes most sense to have a tool which is a separate binary to convert from bdb to "new" wallet format
4612018-09-27T19:50:21 <jonasschnelli> I don't think it has to be a "database" at all
4622018-09-27T19:50:21 <wumpus> (due to storing all the transactions in memory, and also the time overhead of loading the whopping thing at startup)
4632018-09-27T19:50:22 <instagibbs> wumpus, or abusing rpc calls to whiddle it down
4642018-09-27T19:50:27 <phantomcircuit> and for the new wallet format to simply be a flat file
4652018-09-27T19:50:30 <wumpus> instagibbs: oh :-)
4662018-09-27T19:50:38 <jonasschnelli> phantomcircuit: I agree. But that tools would require the refactoring also
4672018-09-27T19:50:40 <cfields> luke-jr: eh, not worth getting into it and muddying the conversation
4682018-09-27T19:50:43 <wumpus> instagibbs: I mean, :-(
4692018-09-27T19:50:47 <gmaxwell> wumpus: More than memory issues, they run into problems that many of our rpc operations iterate over all txn in the wallet and then become super slow.
4702018-09-27T19:50:55 <instagibbs> gmaxwell, yeah that one
4712018-09-27T19:51:07 <phantomcircuit> jonasschnelli, yes but has the advantage that you can write the conversion tool and then just rip out a ton of the walletdb logic entirely
4722018-09-27T19:51:08 <wumpus> gmaxwell: right - another lmitation of not having indexing, either in memory or on disk
4732018-09-27T19:51:08 <instagibbs> i know people who delete completely spent tx(plus 100 confs or something) to speed it wallets
4742018-09-27T19:51:21 <phantomcircuit> which makes refactoring much easier, cause you dont have to support both simultaneously
4752018-09-27T19:51:31 <jonasschnelli> phantomcircuit: the logic must still be available somewhere,... could be in a tool source only. yeah
4762018-09-27T19:51:49 <phantomcircuit> cfields, sqlite doesn't actually provide a monolithic file in the same way bdb doesn't
4772018-09-27T19:51:50 <wumpus> instagibbs: ah yes, the "wallet only needs a view of current utxos, not all of history" view
4782018-09-27T19:52:05 <gmaxwell> "pruned wallet"
4792018-09-27T19:52:06 <instagibbs> wumpus, either that or listunspent takes forever :(
4802018-09-27T19:52:07 <cfields> phantomcircuit: eh? They for sure used to.
4812018-09-27T19:52:08 <phantomcircuit> to operate in the fast safe mode it needs a separate write ahead log file
4822018-09-27T19:52:09 <wumpus> gmaxwell: right
4832018-09-27T19:52:15 <luke-jr> well, at least we don't need to keep the history in memory
4842018-09-27T19:52:23 <wumpus> luke-jr: indeed
4852018-09-27T19:52:24 <cfields> phantomcircuit: oh, I was talking about source file, not the database format.
4862018-09-27T19:52:24 <phantomcircuit> cfields, you cant have a single file but it's amazingly slow
4872018-09-27T19:52:28 <phantomcircuit> oh
4882018-09-27T19:52:34 <phantomcircuit> yes it does have that but like
4892018-09-27T19:52:35 <phantomcircuit> meh
4902018-09-27T19:52:40 <wumpus> that's where something like sqlite would be, more or less, useful, I like how clightning uses it
4912018-09-27T19:52:50 <gmaxwell> going back to the prior point. ... if the history isn't in memory, then the database storing the wallet needs to be structured in a way that suports that
4922018-09-27T19:53:15 <cfields> phantomcircuit: that makes integration into our build a no-brainer. That's a signifacant feature imo.
4932018-09-27T19:53:49 <phantomcircuit> gmaxwell, and needs to be quite fast actually
4942018-09-27T19:54:05 <wumpus> integrating sqlite into a project is trivial, indeed can be done as a single .cpp file if that's desirable
4952018-09-27T19:54:16 <jamesob> if we're thinking longterm (esp. about not loading everything into memory simultaneously), I think it makes sense to come up with a normalized, relational schema for the wallet and use sqlite. shouldn't be hard to come up with something non-controversial (famous last words)
4962018-09-27T19:54:33 <promag> any reason to not consider postgres for instance?
4972018-09-27T19:54:38 <wumpus> AHHHH
4982018-09-27T19:54:38 <sipa> gmaxwell: i don't think the choice of container format and the choice of database layout need to be made at the same time
4992018-09-27T19:54:41 <sipa> promag: god why
5002018-09-27T19:54:42 <jamesob> wat
5012018-09-27T19:54:43 <luke-jr> promag: uh, lots?
5022018-09-27T19:55:04 <jonasschnelli> Oracle?
5032018-09-27T19:55:09 <cfields> haha
5042018-09-27T19:55:11 <wumpus> jonasschnelli: :-) <3
5052018-09-27T19:55:19 <sipa> Oracle BDB?
5062018-09-27T19:55:22 <promag> luke-jr: name one
5072018-09-27T19:55:28 <jonasschnelli> I think however we proceed (sqlite, logdb, etc.), factoring out BDB in a nice layered way will be require (even helps if we keep BDB forever)
5082018-09-27T19:55:32 <luke-jr> I mean, if we're using sqlite, the queries could be compatible with multiple backends, but expecting regular users to set up Postgres is crazy..
5092018-09-27T19:55:35 <jonasschnelli> I hope someone picks that up
5102018-09-27T19:55:35 <sipa> promag: let's do that outside this meeting
5112018-09-27T19:55:39 <cfields> this might work better in terms of concrete proposals rather than rounds of "how about xyz?"
5122018-09-27T19:55:45 <jonasschnelli> Also BDB is a compile pitfall
5132018-09-27T19:55:48 <promag> sure
5142018-09-27T19:56:03 <wumpus> cfields: good point
5152018-09-27T19:56:12 <wumpus> 'what about mongodb?' :')
5162018-09-27T19:56:28 <wumpus> any other topics? 4 minutes left
5172018-09-27T19:56:29 <cfields> haha
5182018-09-27T19:57:04 <provoostenator> We should just store it on a blockchain.
5192018-09-27T19:57:21 <luke-jr> provoostenator: it would be nice if it was possible to commit to it in such a way
5202018-09-27T19:57:40 <luke-jr> eg, if you could get a historical hash of the wallet state for commitments
5212018-09-27T19:57:45 <wumpus> luke-jr: right, optional support for a large-scale DBM like postgres would be useful for really big users, but that's really a long term goal I suppose, if at all
5222018-09-27T19:57:51 <gmaxwell> I'd prefer it if just ban anyone that ever directly uses the name of any database system from the channel.
5232018-09-27T19:58:09 <jonasschnelli> but leveldb!
5242018-09-27T19:58:11 <wumpus> except oracle, of course *ducks*
5252018-09-27T19:58:17 <gmaxwell> sipa: "container" is basically my point, if we're just using it as a "container" a simple log would be a lot better.
5262018-09-27T19:58:19 <jonasschnelli> heh
5272018-09-27T19:58:52 <wumpus> #endmeering
5282018-09-27T19:58:55 <wumpus> #endmeeting
5292018-09-27T19:58:55 <lightningbot> Meeting ended Thu Sep 27 19:58:55 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
5302018-09-27T19:58:55 <lightningbot> Minutes: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.html
5312018-09-27T19:58:55 <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.txt
5322018-09-27T19:58:55 <lightningbot> Log: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.log.html
5332018-09-27T19:58:57 <sipa> gmaxwell: yes, but sqlite is essentially two layers; a well-tested safe updatable blob storing mechanism, and then a database layer on top
5342018-09-27T19:58:57 <gmaxwell> sipa: if you just dump transactions/keys into a container and then have to load them into memory and index them thats fine.. if you want to avoid having to do that, you can't just use the storage as a container, it needs to be used as a database.
5352018-09-27T19:59:12 <jonasschnelli> The append-only easyness got a bit lost since we added abandontransaction
5362018-09-27T19:59:25 <wumpus> jonasschnelli: yes
5372018-09-27T19:59:26 <jonasschnelli> Its still possible but comes at high disk-mem cost
5382018-09-27T19:59:27 <gmaxwell> jonasschnelli: why?
5392018-09-27T19:59:27 <promag> gmaxwell: how about aws s3? o/
5402018-09-27T19:59:36 <gmaxwell> jonasschnelli: abandon transaction just logs the txn as abandoned.
5412018-09-27T19:59:40 <luke-jr> ^
5422018-09-27T19:59:41 <sipa> gmaxwell: by switching to sqlite as a container and only using the low level format, you get pretty much what we need (apart from the extra code for the database stuff), but you also get the ability to later switch the db schema
5432018-09-27T19:59:52 <gmaxwell> So we could start using sqllite as a container, but then later actually have a schema, thats a thing that could be done, but really its just two incompatible wallet formats.
5442018-09-27T20:00:00 <jamesob> ripping out bdb in favor of sqlite sounds fun... I might take a crack if no one gets around to it in the next few months. sounds like a good christmas break project :)
5452018-09-27T20:00:02 <promag> gmaxwell: and support log compaction?
5462018-09-27T20:00:09 <jonasschnelli> gmaxwell: yeah.. but if you abandon a 10k bytes transaction you have 10k unused bytes (where only 32bytes change)
5472018-09-27T20:00:18 <gmaxwell> promag: why would that be needed?
5482018-09-27T20:00:20 <jonasschnelli> log compactation is of course possible
5492018-09-27T20:00:39 <wumpus> jamesob: it's not too hard to replace the database, I replaced berkeleydb with leveldb once in a day or so
5502018-09-27T20:00:50 <gmaxwell> jonasschnelli: yes so? no one cares about 10k extra data in their wallet, if they did any of these database based solutions would be unacceptable from the getgo as they're all fairly space inefficent.
5512018-09-27T20:00:53 <phantomcircuit> jonasschnelli, nobody is using abandon transaction because of 10kb of disk space being used lol
5522018-09-27T20:00:55 <wumpus> jamesob: (that's in a very naive way, though, proper schemas will be more work)
5532018-09-27T20:00:56 <sipa> i don't see how abandomtransaction changed anything more than a transaction confirming does
5542018-09-27T20:01:02 <jamesob> wumpus: what's the tricky part? providing a migration path?
5552018-09-27T20:01:05 <sipa> it's just an update "this tx is now abandoned"
5562018-09-27T20:01:07 <instagibbs> so, what does exit status "1" mean in the functional test suite, when shutting down a node?
5572018-09-27T20:01:10 <jonasschnelli> wumpus, jamesob: I think the hard part of the refactoring is to make two database work (which is ultimatively required for a conversion tool)
5582018-09-27T20:01:17 <promag> why not? for systems that use bitcoin wallet in a couple of months it can be pretty large
5592018-09-27T20:01:23 <sipa> jonasschnelli: i think that's the easiest part
5602018-09-27T20:01:24 <phantomcircuit> what does -6 mean?
5612018-09-27T20:01:30 <wumpus> jamesob: the "don't load everything into memory and use proper indexes" part, and yes ,migration is also an issue
5622018-09-27T20:01:33 <sipa> jonasschnelli: agreeing on a new database schema will be impossible, however :)
5632018-09-27T20:01:34 <achow101> instagibbs: 1 is EXIT_FAILURE
5642018-09-27T20:01:48 <achow101> phantomcircuit: where do you get -6?
5652018-09-27T20:01:49 <luke-jr> append a "drop the tx" command, and punch a hole where the original data was
5662018-09-27T20:01:49 <jonasschnelli> gmaxwell: Yes. I just meant the easyness (or the effectiviness) got a bit lost with state changes on transactions like abandontx
5672018-09-27T20:01:56 <phantomcircuit> jonasschnelli, that is for sure the principle issue, supporting two formats at the same time is difficult
5682018-09-27T20:02:01 <sipa> jonasschnelli: i really don't see where that comes from
5692018-09-27T20:02:02 <luke-jr> then it can be saved append-only in backups, and also frees the disk space (potentially)
5702018-09-27T20:02:07 <sipa> jonasschnelli: a tx confirming is equally a state change
5712018-09-27T20:02:20 <instagibbs> achow101, ah ok, i shall hunt for this error
5722018-09-27T20:02:30 <gmaxwell> sipa: in any case, my point is that just swapping the backing store doesn't seem to produce any real benefit, but it creates incompatiblity. I might be missing it, but it sounds like replacing a black box with a more trendy black box. Actually making use of it to to keep stuff out of memory would potentially be a big gain, but that isn't the discussion, and if it were done would create
5732018-09-27T20:02:30 <gmaxwell> effectively another new wallet format.
5742018-09-27T20:02:32 <phantomcircuit> achow101, https://travis-ci.org/bitcoin/bitcoin/jobs/433891452
5752018-09-27T20:02:33 <phantomcircuit> test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status -6 during initialization
5762018-09-27T20:02:35 <jamesob> wumpus: as you say earlier, though, we can gradually build in better use of the database; I think it'd be nice just to shed the bdb dependency asap
5772018-09-27T20:02:36 <wumpus> luke-jr: fallocate(FALLOC_FL_PUNCH_HOLE, ...)?
5782018-09-27T20:02:50 <phantomcircuit> have literally no idea what would cause that
5792018-09-27T20:03:03 <sipa> gmaxwell: well the low-level part of sqlite looks like a well-tested implementation of what we'd want actually
5802018-09-27T20:03:08 <gmaxwell> jonasschnelli: I'm still not seeing it, we always made changes to tx.. adding labels, or as sipa notes, confirming them.. spending their outputs, etc.
5812018-09-27T20:03:09 <sipa> gmaxwell: of course, needs investigation
5822018-09-27T20:03:25 <luke-jr> wumpus: I don't have the exact APIs memorised, but probably something like that
5832018-09-27T20:03:43 <luke-jr> wumpus: so long as the loader ignores null bytes, it should be fairly trivial
5842018-09-27T20:04:11 <wumpus> luke-jr: I remembered because it sounds so senselessly aggressive, it replaces a page in a file with a hole (all-zeros) page
5852018-09-27T20:04:33 <luke-jr> and frees the space used by it
5862018-09-27T20:04:34 <jonasschnelli> gmaxwell: I think confirmation is stateless from the wallet perspective (only in conjunction with the blockchain/header state)... but yes. Its non crucial.
5872018-09-27T20:04:38 <luke-jr> aggressive in what sense?
5882018-09-27T20:05:25 <gmaxwell> does windows do sparse files?
5892018-09-27T20:05:34 <wumpus> luke-jr: I mean, as in violent
5902018-09-27T20:05:48 <luke-jr> gmaxwell: yes, but not sure if it has an easy punch-hole
5912018-09-27T20:05:50 <sipa> jonasschnelli: ... no, we add the block hash that contains the tx to the CWalletTx record
5922018-09-27T20:06:04 <wumpus> gmaxwell: modern versions of windows do
5932018-09-27T20:06:12 <achow101> phantomcircuit: I think maybe that's a SIGABRT
5942018-09-27T20:06:30 <jonasschnelli> uh. yes. Right. There is then at least one update from mempool to in-block,... right (and eventually some reorg writes)
5952018-09-27T20:06:51 <luke-jr> we don't support versions that don't I think :p
5962018-09-27T20:08:19 <wumpus> so does MacOS support sparse files?
5972018-09-27T20:08:23 <gmaxwell> achow101: how would sigabrt show up that way? signals are normally very high exit codes
5982018-09-27T20:08:32 *** ott0disk has joined #bitcoin-core-dev
5992018-09-27T20:08:33 <sipa> promag: short answer to why not postgres; we don't need a multi-user networked database, and we don't need sql
6002018-09-27T20:08:45 <sipa> we need a file on disk
6012018-09-27T20:08:55 <sipa> that's safe to interact with and move around
6022018-09-27T20:09:17 <wumpus> requireing postgres would be really evil, though not as evil as requiring oracledb
6032018-09-27T20:10:06 <wumpus> besides being unnecessary it has lots of extra setup steps to get started in the first place
6042018-09-27T20:10:06 <achow101> gmaxwell: exit codes are signed, so anything above 128 would be negative, right?
6052018-09-27T20:10:37 <gmaxwell> exit codes are a char?
6062018-09-27T20:10:44 <sipa> i think so yes
6072018-09-27T20:10:46 <wumpus> on most UNIX, yes
6082018-09-27T20:10:55 <gmaxwell> ha, if I knew that it was long since lost!
6092018-09-27T20:11:01 <provoostenator> There are of course some other cryptocurrency projects dealing with lots of I/O. Anything useful we can use e.g. from libbitcoin, or LMDB that Monero uses?
6102018-09-27T20:11:09 <gmaxwell> okay well SIGABRT would mean he was hitting an assertion most likely.
6112018-09-27T20:11:21 <wumpus> and negative values are generally only used when the process was killed with a signal
6122018-09-27T20:11:22 <provoostenator> (not that the wallet is I/O heavy, so maybe that's not the thing to worry about)
6132018-09-27T20:11:29 <gmaxwell> provoostenator: I have detected you mentioned a database. You are a bad person
6142018-09-27T20:11:30 <jamesob> would it be an acceptable user experience for us to completely strip bdb out in some major release, provide an upgrade tool, and throw an error if users try to start bitcoin with bdb-format wallets?
6152018-09-27T20:11:59 <gmaxwell> jamesob: I had assumed that we would eventually do exactly that.
6162018-09-27T20:12:03 <wumpus> provoostenator: monero doesn't use LMDB for the wallet ,as far as I know, only for the block chain
6172018-09-27T20:12:04 <sipa> provoostenator: we are not talking about a database. we're talking about the wallet
6182018-09-27T20:12:09 <achow101> gmaxwell: according to http://tldp.org/LDP/abs/html/exitcodes.html, the exit codes are 128 + n where n is the signal number
6192018-09-27T20:12:19 <wumpus> monero has some really simple format for the "simplewallet" IIRC
6202018-09-27T20:12:27 <achow101> so I think that means -6 becomes just signal number 6, which is sigabrt
6212018-09-27T20:12:38 <wumpus> achow101: yes
6222018-09-27T20:12:55 <sipa> provoostenator: there are very different trade-offs relevant if we're talking about the UTXO set etc, but that's not the topic now
6232018-09-27T20:12:58 <jamesob> gmaxwell: makes sense - didn't know if anyone was intent on having some kind of runtime migration
6242018-09-27T20:13:05 <promag> sipa: I understand, just playing devil's advocate
6252018-09-27T20:13:58 <provoostenator> I just think it's suboptimal to pull in a whole new dependency _only_ for the wallet.
6262018-09-27T20:14:23 <provoostenator> So might as well think ahead a bit, unless we're sure we never want to move away from leveldb for everything else.
6272018-09-27T20:14:42 <gmaxwell> have you all considered rewritting the software in javascript? it has webscale. it's very popular now, I hear nodecoin uses it.
6282018-09-27T20:14:59 <gmaxwell> provoostenator: the wallets needs are almost but not entirely unlike the needs of the system state.
6292018-09-27T20:15:16 <gmaxwell> It's basically an entirely different task, with very different requirements.
6302018-09-27T20:15:25 <wumpus> yes
6312018-09-27T20:15:58 <wumpus> and sqlite would be a trivial dependency at least
6322018-09-27T20:16:53 <gmaxwell> (as an aside, people have also tested lmdb and sqllite for what we use leveldb fore. lmdb seemed to be about the same or slightly worse, depending on hardware in use... and the report was on sqllite that no one had enough patience to let it finish syncing after the Nth day.. :) )
6332018-09-27T20:17:00 <jamesob> leveldb strikes me as totally fine for the utxo set and uprooting it seems very dangerous (not sure if I'm attacking a strawman here)
6342018-09-27T20:17:15 <gmaxwell> s/fore/for/
6352018-09-27T20:17:30 <provoostenator> "SQLite does not use Git" that's a great start...
6362018-09-27T20:17:40 <provoostenator> (but nothing a checksum can't handle)
6372018-09-27T20:17:46 <sipa> heh
6382018-09-27T20:17:53 <sipa> what does that have to do with anything
6392018-09-27T20:17:53 <luke-jr> SQLite isn't consensus-critical.
6402018-09-27T20:18:19 <luke-jr> just pkgconfig sqlite3, done
6412018-09-27T20:18:23 <wumpus> yes
6422018-09-27T20:18:40 <gmaxwell> phantomcircuit was saying above that the safe mode of sqllite requires a seperate log file. :(
6432018-09-27T20:18:47 <gmaxwell> if so, thats a bummer.
6442018-09-27T20:18:49 <BlueMatt> wait, huh, what happened to append-only wallet?
6452018-09-27T20:18:53 <wumpus> anyhow I don't think we're ever going to agree on this
6462018-09-27T20:18:58 <luke-jr> gmaxwell: do we need the safe mode?
6472018-09-27T20:19:00 <BlueMatt> seems like a really, really bad idea to move from a design like that to some standard db env
6482018-09-27T20:19:02 <sipa> BlueMatt: great idea, just discussing alternaives :)
6492018-09-27T20:19:14 <BlueMatt> heh, we havent had enough fun with bdb?
6502018-09-27T20:19:25 <sipa> sqlite is nothing like bdb :)
6512018-09-27T20:19:41 <wumpus> at least sqlite supports proper indexing and queries...
6522018-09-27T20:19:49 <sipa> (but if what gmaxwell says is true, that makes it mch less attractive)
6532018-09-27T20:19:50 <BlueMatt> true, but still
6542018-09-27T20:19:58 <wumpus> which is great for finding transactions matching a certain criterion, for example
6552018-09-27T20:20:15 <jamesob> wumpus: isn't the sqlite logfile just some optional performance tweak?
6562018-09-27T20:20:22 <gmaxwell> wumpus: unless our way of using it is just throwing in a lot of binary blobs like we pretty much use bdb. :P
6572018-09-27T20:20:30 <wumpus> with keyvalue databses all that indexing has to be implemented manually
6582018-09-27T20:20:44 <wumpus> gmaxwell: as I said, I like how clightning uses it
6592018-09-27T20:21:08 <wumpus> it's really sensible and well-designed
6602018-09-27T20:21:49 <provoostenator> Problem solved then, a SQL lite binary blob inside bdb.
6612018-09-27T20:21:59 <wumpus> somehow I really dislike how we ended up with a wallet embeded in our project and being unable to move away from it
6622018-09-27T20:22:03 <gmaxwell> wumpus: to be clear, I'm +1 on using it with a schema, -1 on using it in a blobby way, neutral on blobby in general, but if we were going to stay blobby a simple thing that just writes out a change log for in memory state would probably be better (more reliable, simpler, etc.).
6632018-09-27T20:22:24 <wumpus> I wish it was just some external thing, but that's too much to wish for at this point
6642018-09-27T20:22:41 <jamesob> wumpus: huge agree
6652018-09-27T20:22:55 <wumpus> everyone is always going to disagree how to move on from here, it's like a really bad local maximum in evolution
6662018-09-27T20:23:11 <gmaxwell> if it were, I think bitcoin would be nearly dead... there would be very little reason for people to bother even trying to start a node.
6672018-09-27T20:23:30 <gmaxwell> Altcoins where node operations are totally distinct from using a wallet basically just don't haver users using nodes.
6682018-09-27T20:23:59 <wumpus> the node could still be shipped with some wallet
6692018-09-27T20:24:02 <luke-jr> on that point, we do need to make it easier to pair nodes with other wallets
6702018-09-27T20:24:29 <luke-jr> the hard part is the port forwarding stuff :/
6712018-09-27T20:24:43 <jamesob> luke-jr: in your opinion is that an issue of making the rpc interface more granular/complete or something else?
6722018-09-27T20:24:48 <wumpus> or like monero simply ship with the most simple wallet possible, with no ambitions of anything more
6732018-09-27T20:24:50 <gmaxwell> luke-jr: because there is so much development surplus that it would be good to spend it on facilitating duplication of efforts that aren't even being duplicated yet. :P
6742018-09-27T20:25:07 <luke-jr> jamesob: RPC interface isn't really used much for other wallets
6752018-09-27T20:25:38 <jamesob> anecdotally, if someone just wants to transact in bitcoin they often just go to electrum/trezor/bread/etc.
6762018-09-27T20:25:40 <luke-jr> gmaxwell: people are using other wallets whether we facilitate it or not
6772018-09-27T20:25:51 <jamesob> I think few people who _just_ want a wallet go to the bother of syncing bitcoind
6782018-09-27T20:26:17 <gmaxwell> luke-jr: I don't see how that connects to your comment.
6792018-09-27T20:26:23 <jamesob> though I am sympathetic to that reasoning... wish there was a way of quantifying it
6802018-09-27T20:26:32 <provoostenator> jamesob: that's probably because they don't understand the privacy tradeoff
6812018-09-27T20:26:39 <gmaxwell> luke-jr: how someing using a hosted web wallet doesn't really have anything to do with bitcoind.
6822018-09-27T20:26:51 <luke-jr> gmaxwell: I mean electrum/trezor/bread/etc
6832018-09-27T20:26:55 <wumpus> in any case, this comes back every time
6842018-09-27T20:27:00 <jamesob> of course - but even a user like me doesn't actually use the core wallet because it doesn't, e.g., support hardware devices
6852018-09-27T20:27:09 <wumpus> the same discussions since 2012
6862018-09-27T20:27:11 <luke-jr> ideally, they could just scan a QR code from Core, and have it use their own node
6872018-09-27T20:27:18 <luke-jr> the trouble is getting past firewalls
6882018-09-27T20:27:21 <gmaxwell> luke-jr: and what does that hae to do with bitcoind?
6892018-09-27T20:27:31 <jamesob> provoostenator: oops ^
6902018-09-27T20:27:33 <gmaxwell> It's a lot more than that.
6912018-09-27T20:27:49 <wumpus> even the "let's use a SQL database, oh why not postgres" I'm fairly sure we alrady had back in 2012
6922018-09-27T20:27:57 <gmaxwell> Trezor, for example, is effectively a hosted web wallet, though with signing using your hardware fob.
6932018-09-27T20:28:26 <jamesob> gmaxwell: right - which I hate but nonetheless use
6942018-09-27T20:28:59 <gmaxwell> I am really frustrated by the fact that any time anyone mentions storing anything on disk there is a tiresome discussion of the trendy database things. Someone above joked mongo, but in the 2013 (IIRC) flavor of these rehashings, it would actually get suggested seriously.
6952018-09-27T20:29:02 <provoostenator> And because people are physically holding it, they're less aware of that problem.
6962018-09-27T20:29:28 <provoostenator> Everyone always saying "don't use web wallets, use a hardware wallet"..
6972018-09-27T20:29:30 <wumpus> the other wallets aren't better in every regard, that's for sure, the core wallet is pretty good, I'm just tired of some kinds of discussions
6982018-09-27T20:29:47 <gmaxwell> jamesob: my point in responding to luke there, that no amount of changed to bitcoind is going to make the 'trezor wallet' work with it.
6992018-09-27T20:29:52 *** Murch has quit IRC
7002018-09-27T20:29:56 <booyah> hashtag nosql
7012018-09-27T20:30:00 <wumpus> gmaxwell: exactly...
7022018-09-27T20:30:02 <provoostenator> jamesob: at least there's hope: https://gist.github.com/achow101/a9cf757d45df56753fae9d65db4d6e1d
7032018-09-27T20:30:05 <gmaxwell> (getting support for hardware wallets, OTOH, would enable people to not use the trezor wallet, and thats another matter)
7042018-09-27T20:30:13 <booyah> gmaxwell: as for trezor, well with electrum it's a signing engine, not webwallet at all
7052018-09-27T20:30:37 *** Murch has joined #bitcoin-core-dev
7062018-09-27T20:30:54 <luke-jr> gmaxwell: Electrum already supports Trezors (altohugh to support Electrum, we need a lot more than just port forwarding)
7072018-09-27T20:31:00 <gmaxwell> booyah: right, and PSBT was a first step in getting thins so we could use devices like that.
7082018-09-27T20:31:01 <wumpus> I'm happy that we managed to depreacte accounts at least, to simplify things, that was also an issue running almost since satoshi left
7092018-09-27T20:31:10 <booyah> gmaxwell: freaking finally ;)
7102018-09-27T20:31:11 <wumpus> yes hardware wallet support is something worth working on!
7112018-09-27T20:31:15 <jamesob> gmaxwell provoostenator: I think even now you can do some bitcoind -> electrum personal server -> electrum -> trezor thing, but I haven't tried it because it sounds like a ton of work
7122018-09-27T20:31:18 <wumpus> that would be actually nice
7132018-09-27T20:31:28 <wumpus> in contrast to the eternal database discussion
7142018-09-27T20:31:41 <gmaxwell> booyah: unfortunately its hard going because the vendors of these things have not been helping, instead mostly just making their own custom stuff for time to market reasons.
7152018-09-27T20:31:47 <jonasschnelli> jamesob: its working and setup is easy
7162018-09-27T20:31:56 <booyah> gmaxwell: satoshi labs was not helpful?
7172018-09-27T20:31:56 <provoostenator> I've tried the electrum personal server approach. Works reasonably well. I've also tried the achow101 approach, which works great once you're used to it, but too manual for now.
7182018-09-27T20:32:07 <jamesob> jonasschnelli: oh cool, I'll try that then.
7192018-09-27T20:32:19 <wumpus> https://github.com/romanz/electrs looks really neat
7202018-09-27T20:32:20 <jonasschnelli> jamesob: Bitbox has a native desktop app that can connect to EPS <-> Core
7212018-09-27T20:32:32 <wumpus> (haven't tried it yet though)
7222018-09-27T20:32:35 <gmaxwell> jamesob: and it involves using electrum... which has plenty of its own downsides.
7232018-09-27T20:32:41 <jamesob> right
7242018-09-27T20:33:15 <gmaxwell> I'm certantly not faulting the HW wallet vendors for focusing on time to market, it's just what it is.
7252018-09-27T20:33:26 <provoostenator> I have some WIP stuff where I try to turn achow101's HWI scripts into an RPC server, which could eventually be a universal "sign something" service, not necessarily restricted to hardware wallets.
7262018-09-27T20:33:59 <wumpus> me neither, though they have been kind of shooting themselves in the foot by not thinking about a common standard, but maybe that's something taht can only arise after experimentation... happy about PSBT
7272018-09-27T20:34:04 <provoostenator> In that case all Core needs to do is make a bunch of RPC calls to a user configured URI (or a file socket maybe).
7282018-09-27T20:34:39 <gmaxwell> provoostenator: I'd suggested previously that we exeute a configured command and shove crap at it over stdin.
7292018-09-27T20:34:42 <provoostenator> And then it's up to hardware wallet makers to listen for those calls.
7302018-09-27T20:35:01 <gmaxwell> (I'm less of a fan of URIs because of the security problems, e.g. when some local JS starts connecting to it...)
7312018-09-27T20:35:08 <sipa> provoostenator: or PSBT and invoke a binary to sign it
7322018-09-27T20:35:08 <provoostenator> Executing commands sounds scary too
7332018-09-27T20:35:20 <gmaxwell> provoostenator: hows it scary? it's configured explicitly.
7342018-09-27T20:35:58 <provoostenator> True, malware gonna malware, no matter how you do it.
7352018-09-27T20:36:06 <gmaxwell> provoostenator: the URI thing in general has a challenge that browsers can make connections to things that listen to the network. There have been a LOT of thefts in eth land, due to 'wallets' that take commands from local sockets.
7362018-09-27T20:36:32 <clarkmoody> Apply UNIX philosophy to wallet: text is the universal interface
7372018-09-27T20:36:36 <gmaxwell> provoostenator: I'm not talking about malware running unconstrained on your host, I'm talking about just stuff in your webbrowser.
7382018-09-27T20:36:38 <wumpus> in any case I think it makes sense to, in meetings, discuss the things each participant plans on working on, not wild far future plans
7392018-09-27T20:36:39 <provoostenator> gmaxwell: c-lightning uses file sockets which at least prevent hat.
7402018-09-27T20:36:56 <luke-jr> provoostenator: on Windows? O.o
7412018-09-27T20:36:57 <wumpus> otherwise we end up discussing the same things from 2012 every time
7422018-09-27T20:37:04 <provoostenator> luke-jr: there's that...
7432018-09-27T20:37:05 <gmaxwell> provoostenator: okay, thats essentially the same security model as what I was talking about. didn't know you could do that on windows.
7442018-09-27T20:37:55 *** michaelsdunn1 has quit IRC
7452018-09-27T20:38:03 <provoostenator> I wish browsers didn't let websites talk to any URL in the universe, but it seems they do.
7462018-09-27T20:38:10 <gmaxwell> wumpus: The way it seems to go though is.. "I plan on working on something something something database" "Excuse me sir, but have you considered Monodb? It has webscale" then goto 2012;
7472018-09-27T20:38:44 *** rex4539 has joined #bitcoin-core-dev
7482018-09-27T20:38:44 <gmaxwell> maybe I just need to write a webpage to link to, with links to every other time we've gone through this loop.
7492018-09-27T20:38:55 <wumpus> windows also has various local RPC methods
7502018-09-27T20:39:49 <wumpus> gmaxwell: agree, there seems to be a lack of memory in these kind of things, although it seems to slightly improve, back then we had the literal same discussion within weeks of each other with new people
7512018-09-27T20:39:51 <provoostenator> gmaxwell: a nice use case for RPC calls would be a web based multisig service.
7522018-09-27T20:40:21 *** michaelsdunn1 has joined #bitcoin-core-dev
7532018-09-27T20:40:23 <gmaxwell> provoostenator: in any case the model of "you tell your wallet to talk to some other local program that has the task of signing things" sounds great to me. It also could be used "talk to some other local program that has the task of collecting other signatures for multisig" easily.
7542018-09-27T20:40:29 <wumpus> (I think recent windows 10 even has UNIX sockets)
7552018-09-27T20:40:47 <gmaxwell> provoostenator: I'd say jinx except this was all pointed out last time I was lobbying in that direction. ;)
7562018-09-27T20:40:58 <provoostenator> Yes, it just means those services have to ship an executable, but that trade-off might be worth it.
7572018-09-27T20:41:18 <wumpus> but in any case, if you want a *secure* RPC method, you will have to determine for every OS what is the best choice
7582018-09-27T20:41:22 <gmaxwell> provoostenator: "Driver", it might well be the case that eventually we ship drivers for common cases ourselves.
7592018-09-27T20:41:34 <jamesob> gmaxwell: you know we're trending upwards because no one *seriously* recommended mongodb this time around
7602018-09-27T20:41:43 <wumpus> just deciding for 'UNIX sockets' is very UNIX-centric by default
7612018-09-27T20:41:56 <gmaxwell> jamesob: I think thats just because the world at large no longer thinks mongodb is the new hotness. :)
7622018-09-27T20:42:04 <jamesob> heh, true
7632018-09-27T20:42:09 <provoostenator> gmaxwell: especially if someone comes up with a proper USB protocol standard for talking with devices, so we wouldn't have to merge device specific stuff?
7642018-09-27T20:42:12 <gmaxwell> wumpus: well thats why I liked stdio, as it does exist everwhere. :P
7652018-09-27T20:42:25 <wumpus> mongodb suggestion was a reductio ad absurdum by me this time
7662018-09-27T20:42:30 *** ott0disk has quit IRC
7672018-09-27T20:42:54 <gmaxwell> provoostenator: right, but it's not just the usb, there are UI considerations. e.g. many hardware wallets have some pin entry on the host computer or similar.
7682018-09-27T20:43:26 <gmaxwell> Basically the need to control enough of the host probably contributes to making your hardware wallet supported by creating a whole web wallet for it.
7692018-09-27T20:43:27 <wumpus> gmaxwell: yes, communicating through a pipe over stdin/stdout seems to be something every OS has
7702018-09-27T20:44:01 <provoostenator> At least Ledger and Trezor these days do pin entry on device. So in that case it's just a matter of waiting.
7712018-09-27T20:44:38 <wumpus> (though on windows it's different; stdio only exists for console applications, not for winmain ones)
7722018-09-27T20:45:06 <jamesob> the Trezor One requires on-host software for pin entry
7732018-09-27T20:45:17 <wumpus> it's such a crazy labyrinth people have created, nothing is straightforward
7742018-09-27T20:45:21 *** michaelsdunn1 has quit IRC
7752018-09-27T20:46:23 <gmaxwell> jamesob: okay that one was the one that I was thinking about.
7762018-09-27T20:46:35 *** irc_viewer_test has joined #bitcoin-core-dev
7772018-09-27T20:48:25 *** clarkmoody has quit IRC
7782018-09-27T20:49:41 <provoostenator> There's a bunch of ports that some browsers ban, but it's quite narrow, and those are already used by the system (that's why they're blocked). I also don't think there's a standard.
7792018-09-27T20:53:00 <provoostenator> You can add authentication, maybe some tofu mechanism. But browers are extremely powerful these days.
7802018-09-27T20:53:14 *** hebasto has quit IRC
7812018-09-27T20:53:45 <provoostenator> Even USB is no longer off limits: https://developers.google.com/web/updates/2016/03/access-usb-devices-on-the-web
7822018-09-27T20:53:54 <phantomcircuit> gmaxwell, sqlite has a safe *but slow* single file mode
7832018-09-27T20:54:03 <phantomcircuit> like really slow
7842018-09-27T20:54:15 <phantomcircuit> cause it's using fsync() multiple times per write
7852018-09-27T20:54:23 *** elichai2 has joined #bitcoin-core-dev
7862018-09-27T20:55:43 <achow101> provoostenator: the whole HWI thing is intended for the "execute a command and shove crap at it over stdin"
7872018-09-27T20:59:23 <wumpus> phantomcircuit: writes to the wallet are more or less rare, though, so it might still be ok
7882018-09-27T20:59:37 <gmaxwell> phantomcircuit: that sounds like it would probably be fine
7892018-09-27T21:01:42 *** jimmysong has quit IRC
7902018-09-27T21:07:22 <wumpus> by far and far, most queries will be reads
7912018-09-27T21:08:15 <wumpus> this is not like the utxo set where every read tends to imply a write/update
7922018-09-27T21:09:07 *** Krellan has quit IRC
7932018-09-27T21:12:07 *** Murch has quit IRC
7942018-09-27T21:12:18 *** Krellan has joined #bitcoin-core-dev
7952018-09-27T21:28:11 *** irc_viewer_test has quit IRC
7962018-09-27T21:38:16 *** irc_viewer_test has joined #bitcoin-core-dev
7972018-09-27T21:38:48 *** Murch has joined #bitcoin-core-dev
7982018-09-27T21:39:38 *** reallll has joined #bitcoin-core-dev
7992018-09-27T21:39:41 *** belcher has quit IRC
8002018-09-27T21:48:08 *** irc_viewer_test has quit IRC
8012018-09-27T21:50:03 *** Chris_Stewart_5 has quit IRC
8022018-09-27T22:02:00 *** jarthur has joined #bitcoin-core-dev
8032018-09-27T22:04:05 *** jimmysong has joined #bitcoin-core-dev
8042018-09-27T22:15:58 *** ghost43 has quit IRC
8052018-09-27T22:16:16 *** ghost43 has joined #bitcoin-core-dev
8062018-09-27T22:31:32 *** jarthur has quit IRC
8072018-09-27T22:41:58 *** spinza has quit IRC
8082018-09-27T22:51:47 *** Chris_Stewart_5 has joined #bitcoin-core-dev
8092018-09-27T22:57:35 <phantomcircuit> wumpus, reads are fast... for a sql database
8102018-09-27T22:57:48 <phantomcircuit> but are significantly slower than the in memory stuff is now
8112018-09-27T22:58:21 <phantomcircuit> gmaxwell, seems the issue is in the commit just before the poll() implementation actually
8122018-09-27T22:59:50 *** promag has quit IRC
8132018-09-27T23:04:06 *** Chris_Stewart_5 has quit IRC
8142018-09-27T23:07:28 *** Chris_Stewart_5 has joined #bitcoin-core-dev
8152018-09-27T23:14:58 *** dqx has joined #bitcoin-core-dev
8162018-09-27T23:22:30 *** paracyst_ has quit IRC
8172018-09-27T23:23:08 *** Chris_Stewart_5 has quit IRC
8182018-09-27T23:26:56 *** grubles has quit IRC
8192018-09-27T23:31:26 *** paracyst has joined #bitcoin-core-dev
8202018-09-27T23:32:08 *** dqx has quit IRC
8212018-09-27T23:36:33 *** Chris_Stewart_5 has joined #bitcoin-core-dev
8222018-09-27T23:36:48 *** gribble has quit IRC
8232018-09-27T23:49:44 *** gribble has joined #bitcoin-core-dev
8242018-09-27T23:50:47 *** promag has joined #bitcoin-core-dev
8252018-09-27T23:58:56 *** Giszmo has quit IRC