12016-08-27T00:00:08  *** kadoban has quit IRC
  22016-08-27T00:05:34  *** spudowiar has quit IRC
  32016-08-27T00:13:40  *** achow101 has quit IRC
  42016-08-27T00:14:31  *** tom3 has joined #bitcoin-core-dev
  52016-08-27T00:22:16  *** Alopex has quit IRC
  62016-08-27T00:23:22  *** Alopex has joined #bitcoin-core-dev
  72016-08-27T00:35:21  *** Alopex has quit IRC
  82016-08-27T00:36:26  *** Alopex has joined #bitcoin-core-dev
  92016-08-27T00:42:12  *** Chris_Stewart_5 has quit IRC
 102016-08-27T00:42:30  *** Kexkey has joined #bitcoin-core-dev
 112016-08-27T00:44:42  <GitHub4> [bitcoin] nomnombtc opened pull request #8608: Install manpages via make install, also add some autogenerated manpages (master...man_automake2) https://github.com/bitcoin/bitcoin/pull/8608
 122016-08-27T00:46:22  *** fengling has joined #bitcoin-core-dev
 132016-08-27T00:51:06  *** fengling has quit IRC
 142016-08-27T00:52:53  *** Samdney has left #bitcoin-core-dev
 152016-08-27T01:05:46  *** achow101 has joined #bitcoin-core-dev
 162016-08-27T01:25:53  *** Ylbam has quit IRC
 172016-08-27T01:27:05  *** mappum has joined #bitcoin-core-dev
 182016-08-27T01:35:07  *** eragmus has joined #bitcoin-core-dev
 192016-08-27T01:35:26  *** Alopex has quit IRC
 202016-08-27T01:36:31  *** Alopex has joined #bitcoin-core-dev
 212016-08-27T01:39:13  *** btcdrak has joined #bitcoin-core-dev
 222016-08-27T01:47:48  *** fengling has joined #bitcoin-core-dev
 232016-08-27T01:49:44  <btcdrak> is there a way to sign each commit during an interactive rebase?
 242016-08-27T01:51:04  *** achow101 has quit IRC
 252016-08-27T01:52:26  *** fengling has quit IRC
 262016-08-27T01:59:52  *** tom3 has quit IRC
 272016-08-27T02:00:21  *** tom3 has joined #bitcoin-core-dev
 282016-08-27T02:00:26  *** Alopex has quit IRC
 292016-08-27T02:01:31  *** Alopex has joined #bitcoin-core-dev
 302016-08-27T02:18:01  *** Alopex has quit IRC
 312016-08-27T02:18:43  *** achow101 has joined #bitcoin-core-dev
 322016-08-27T02:19:06  *** Alopex has joined #bitcoin-core-dev
 332016-08-27T02:35:03  *** Giszmo has quit IRC
 342016-08-27T02:35:22  * luke-jr peers at rebroad.
 352016-08-27T02:47:22  *** tom3 has quit IRC
 362016-08-27T02:48:49  *** fengling has joined #bitcoin-core-dev
 372016-08-27T02:53:46  *** fengling has quit IRC
 382016-08-27T02:56:16  *** Alopex has quit IRC
 392016-08-27T02:57:21  *** Alopex has joined #bitcoin-core-dev
 402016-08-27T03:07:33  *** tom3 has joined #bitcoin-core-dev
 412016-08-27T03:08:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 422016-08-27T03:17:52  *** droark has joined #bitcoin-core-dev
 432016-08-27T03:19:31  *** Kexkey has quit IRC
 442016-08-27T03:26:44  *** achow101 has quit IRC
 452016-08-27T03:28:18  *** kadoban has joined #bitcoin-core-dev
 462016-08-27T03:42:54  <roasbeef> btcdrak: signoff-rebase = "!GIT_SEQUENCE_EDITOR='sed -i -re s/^pick/e/' sh -c 'git rebase -i $1 && while git rebase --continue; do git commit --amend -S --no-edit; done' -"
 472016-08-27T03:43:06  <roasbeef> btcdrak: stuff that into an alias in your .gitconfig
 482016-08-27T03:48:28  *** AtashiCon has quit IRC
 492016-08-27T03:49:48  *** AtashiCon has joined #bitcoin-core-dev
 502016-08-27T03:50:19  *** fengling has joined #bitcoin-core-dev
 512016-08-27T03:53:26  *** Alopex has quit IRC
 522016-08-27T03:54:31  *** Alopex has joined #bitcoin-core-dev
 532016-08-27T03:54:46  *** fengling has quit IRC
 542016-08-27T04:02:50  *** cryptapus_afk is now known as cryptapus
 552016-08-27T04:04:06  *** Alopex has quit IRC
 562016-08-27T04:05:11  *** Alopex has joined #bitcoin-core-dev
 572016-08-27T04:32:14  *** fengling has joined #bitcoin-core-dev
 582016-08-27T04:35:07  *** Alopex has quit IRC
 592016-08-27T04:36:12  *** Alopex has joined #bitcoin-core-dev
 602016-08-27T04:46:24  *** tom3 has quit IRC
 612016-08-27T04:54:27  *** Alopex has quit IRC
 622016-08-27T04:55:32  *** Alopex has joined #bitcoin-core-dev
 632016-08-27T04:57:19  *** btcdrak has quit IRC
 642016-08-27T04:59:02  *** cryptapus is now known as cryptapus_afk
 652016-08-27T05:15:28  <luke-jr> hm, I would have thought verifying blocks was a read-only operation, but it seems to be writing about as much as it's reading?
 662016-08-27T05:15:54  <luke-jr> more actually
 672016-08-27T05:21:33  <gmaxwell> should be much more,
 682016-08-27T05:21:39  <gmaxwell> if it's reading your dbcache is too small.
 692016-08-27T05:43:27  <luke-jr> what's it writing?
 702016-08-27T05:44:33  <gmaxwell> the chainstate.
 712016-08-27T05:51:09  <wumpus> right, it's not the verification that causes writing, but *accepting* the blocks and updating the state
 722016-08-27T05:51:29  <wumpus> rejecting blocks should be a read-only operations
 732016-08-27T05:51:46  <wumpus> as well as verifying transactions outside blocks
 742016-08-27T05:55:35  <wumpus> gmaxwell: re: BU removing outward connection limit, sigh, it was to be expected somehow, you can't stop anti-social people by just telling them not to do a certain behavior, we'll need explicit anti-DoS measures for that
 752016-08-27T05:56:29  <wumpus> such a system could block spy-nodes connecting to everyone in one go
 762016-08-27T05:57:05  <wumpus> that there is no damn reason to connect to connect to more nodes also won't stop anyone
 772016-08-27T05:57:36  <wumpus> 'look mom we're real anarchists, we can misbehave on an open P2P network!'
 782016-08-27T06:02:23  <midnightmagic> any such mechanism if successful would at least in the short term eradicate blockchain.info's spying, as well as the various chainalysis mechs.
 792016-08-27T06:02:26  <wumpus> their name, their whole raison d'etre, is 'remove limits', so they removed another limit. It doesn't matter whether it had a purpose, they've remved a few lines of very limiting-feeling codes, and now they feel happy
 802016-08-27T06:04:30  <wumpus> pre-emtively filling all connection slots would also prevent blockchain.info's spying :-)
 812016-08-27T06:05:01  <midnightmagic> :-)
 822016-08-27T06:05:08  *** kadoban has quit IRC
 832016-08-27T06:07:20  <wumpus> if this escalates and there is no solution, in the longer run, it will turn the P2P network to a ghetto, and I'd expect it to transition it to more of a F2F network w/ authenticated connections
 842016-08-27T06:08:02  <wumpus> but also a few more mass-connectors likely won't break the netwerk, it just depends on the scale
 852016-08-27T06:12:16  <midnightmagic> Bitcoin Classic -- DDoS'ing the Bitcoin network since 2016.
 862016-08-27T06:12:35  <midnightmagic> I wonder if someone could get gavin to scold them for that
 872016-08-27T06:12:46  <midnightmagic> or. Unlimited. Same diff.
 882016-08-27T06:14:07  <wumpus> it reminds me a bit of seeders versus leechers discussion for bittorrent, with the difference is that 'defecting' there is advantageous to the leecher, here it's just about being a jerk just because
 892016-08-27T06:15:53  <wumpus> though connecting to lots of nodes can get you blocks fractially faster, which is useful for miners, you can accomplish the same thing by listening and advertising
 902016-08-27T06:26:46  <wumpus> you can set a virtually unlimited number of incoming connection slots, then by having a node with a high uptime, it will drift up in the DNS seeder rankings, which means its address will be dealt out more often, resulting in more connections to other nodes. It's a slower process but the social thing to do.
 912016-08-27T06:34:33  <luke-jr> shouldn't those blocks (and the chainstate) already be correct? O.o
 922016-08-27T06:35:49  <wumpus> but blocks are a delta to the chainstate
 932016-08-27T06:36:18  <wumpus> if you accept a block, by definition, you have to update the chainstate. And undo files are produced, too, a reverse-delta just in case.
 942016-08-27T06:36:35  <luke-jr> so it's actually rewinding and re-accepting them? somehow I thought it just verified the current state was sane
 952016-08-27T06:37:03  <gmaxwell> wumpus: what you say is true, but its much harder to deal with abusive behavior when you also have many more 'honest' people also being abusive out of ignorance. (in part, because the motivational structure is different; e.g. we can discourage spy nodes by reducing information leaks, moving more users into tor, etc.)
 962016-08-27T06:37:28  <wumpus> luke-jr: no, just applying the block results in writes to the utxo state, to mark outputs as spent, add new outputs. it rewinds only on reorg
 972016-08-27T06:37:38  <gmaxwell> (But ignorance, "I'm gonna help the network out by connecting to EVERYTHING!" ... is harder to resolve by rational measures)
 982016-08-27T06:38:06  <luke-jr> wumpus: I'm talking about the verifying blocks at startup.. checkblocks/checklevel
 992016-08-27T06:38:23  <wumpus> luke-jr: ohhh! that wasn't clear to me.
1002016-08-27T06:38:32  <wumpus> luke-jr: no, that should be read-only
1012016-08-27T06:38:32  <luke-jr> sorry
1022016-08-27T06:38:52  <luke-jr> hmm
1032016-08-27T06:39:34  <wumpus> gmaxwell: sure, at least ignorance can be improved by trying harder to inform people about things
1042016-08-27T06:42:34  <wumpus> people may assume 'moaaarrrr outgoing connections is better' unless it's explained , .e.g in user interfaces, blog posts, that it's bad for yourself as well as others, with alternatives how to get more connections. Sure, some people will ignore it, or be jerks, but hopefully a miniority and most will heed.
1052016-08-27T06:43:39  <wumpus> luke-jr: the rewinding with checklevel 3 completely happens in memory, if it is writing anything to disk that'd be very wrong
1062016-08-27T06:44:01  <wumpus> (there could be a bug of course....)
1072016-08-27T06:56:15  *** btcdrak has joined #bitcoin-core-dev
1082016-08-27T06:57:36  <luke-jr> having trouble reproducing now. flushed Linux's disk caches though, and even with just reading, it's going super-slow :|
1092016-08-27T06:59:58  <luke-jr> (as in 1% every 2 minutes or so, ETA 3 hours at this rate)
1102016-08-27T07:05:29  * luke-jr ponders if iotop would report writing if it was swapping other processes to disk to do caching for us
1112016-08-27T07:09:22  <wumpus> yes it reports swapping as writing, but wouldn't account swapping of *other* processes to disk to bitcoind
1122016-08-27T07:09:46  <wumpus> IIRC there's a kswapd that gets all the blame for swapping
1132016-08-27T07:09:49  <luke-jr> hmm
1142016-08-27T07:10:03  <luke-jr> it did pick up speed and finished in 15 mins (just now) fwiw
1152016-08-27T07:10:18  <luke-jr> didn't see any writing this time either
1162016-08-27T07:11:28  <wumpus> phew
1172016-08-27T07:12:38  <luke-jr> but I guess I did manage to reproduce https://www.reddit.com/r/Bitcoin/comments/4zrxs1/qtcore_client_taking_ages_to_start/ after all
1182016-08-27T07:12:46  <luke-jr> just a matter of dropping caches
1192016-08-27T07:12:47  *** OxADADA_ has joined #bitcoin-core-dev
1202016-08-27T07:12:59  <luke-jr> gmaxwell: can you confirm on your slow laptop? echo 3 > /proc/sys/vm/drop_caches
1212016-08-27T07:14:13  <wumpus> usually if the client takes ages to start there's a backlog of blocks to verify
1222016-08-27T07:14:50  *** asoltys_ has joined #bitcoin-core-dev
1232016-08-27T07:14:59  *** Pasha has joined #bitcoin-core-dev
1242016-08-27T07:15:08  <sipa> question: how many times has the initial verification at startup actually caught corruption
1252016-08-27T07:15:48  <sipa> nobody knows, of course
1262016-08-27T07:15:49  <wumpus> I don't know - but I think it would be just as effective to just check a few blocks
1272016-08-27T07:15:52  <luke-jr> sipa: not sure if it's still a problem, but every time when we had those powerfail-corrupts-db problem? (unless that was caught by something else?)
1282016-08-27T07:16:06  <wumpus> the latest blocks are the most likely to be corrupt and below that it drops off
1292016-08-27T07:16:10  <sipa> luke-jr: those result in a leveldb checksum error
1302016-08-27T07:16:15  * luke-jr didn't realise the slowish startup time because he had checkblocks=4 checklevel=6 in his bitcoin.conf
1312016-08-27T07:16:31  <sipa> there are only 4 checklevels :)
1322016-08-27T07:16:41  <wumpus> luke-jr: same here - I think the default checkblocks should be much lower
1332016-08-27T07:16:42  <luke-jr> well, if there's ever more added, I'm ready! :P
1342016-08-27T07:16:52  *** owowo has quit IRC
1352016-08-27T07:16:52  *** ratoder has quit IRC
1362016-08-27T07:16:53  *** dgenr8 has quit IRC
1372016-08-27T07:16:53  *** OxADADA has quit IRC
1382016-08-27T07:16:53  *** droark has quit IRC
1392016-08-27T07:16:54  *** nobits has quit IRC
1402016-08-27T07:16:54  *** gluytium has quit IRC
1412016-08-27T07:16:54  *** Cory has quit IRC
1422016-08-27T07:16:54  *** asoltys has quit IRC
1432016-08-27T07:16:57  <wumpus> a as-thorough-as-possible check on just a few blocks
1442016-08-27T07:16:58  <sipa> and 4 blocks is not very much
1452016-08-27T07:17:07  <wumpus> well, take 10 then
1462016-08-27T07:17:14  *** nobits has joined #bitcoin-core-dev
1472016-08-27T07:17:14  *** owowo has joined #bitcoin-core-dev
1482016-08-27T07:17:16  <luke-jr> sipa: my PC is on UPS ;)
1492016-08-27T07:17:24  <sipa> wumpus: jonasschnelli has a patch to switch to txcount based limiting
1502016-08-27T07:18:06  <luke-jr> hmm, that's an interesting idea.
1512016-08-27T07:18:07  <wumpus> that's good, but I think the effective default check depth should also be lowered, don't know if it does that
1522016-08-27T07:18:10  *** dgenr8 has joined #bitcoin-core-dev
1532016-08-27T07:18:46  <wumpus> or maybe write a flag on 'clean' shutdown, and do a reduced check in that case?
1542016-08-27T07:18:52  <luke-jr> set it for the equivalent checkblocks back when it was introduced? :p
1552016-08-27T07:18:54  *** gluytium has joined #bitcoin-core-dev
1562016-08-27T07:19:17  <sipa> wumpus: it sets the default to 100000 txn
1572016-08-27T07:19:34  <sipa> i guess it can be lower even
1582016-08-27T07:19:36  <wumpus> sipa: ok
1592016-08-27T07:19:51  <sipa> but it also insists on at least 6 blocks
1602016-08-27T07:19:55  <gmaxwell> we need to do something about the startup check.
1612016-08-27T07:20:23  <gmaxwell> unfortunately just checking a couple blocks is not much of a test, but anything more takes too long for most people.
1622016-08-27T07:20:33  <sipa> wumpus: interesting idea
1632016-08-27T07:20:39  <luke-jr> I wonder how difficult it would be to background it
1642016-08-27T07:20:44  <gmaxwell> wumpus: neat idea!
1652016-08-27T07:21:53  *** Pasha is now known as Cory
1662016-08-27T07:22:10  <gmaxwell> when we were having windows corruption, what was needed to reliably detect that?
1672016-08-27T07:22:56  <wumpus> just opening the leveldb IIRC
1682016-08-27T07:22:58  <btcdrak> roasbeef: thanks!!
1692016-08-27T07:23:19  <wumpus> or maybe the first access. No thorough checking was necessary
1702016-08-27T07:23:22  * sipa suggests: 10000 txn (which corresponds to ~6 blocks currently)
1712016-08-27T07:24:03  <wumpus> sounds good to me
1722016-08-27T07:25:10  <wumpus> it's very helpful that leveldb has its own corruption detection here
1732016-08-27T07:25:22  <gmaxwell> why bother with the txn count?
1742016-08-27T07:25:25  <gmaxwell> just set it to 6 blocks?
1752016-08-27T07:25:30  *** Guyver2 has joined #bitcoin-core-dev
1762016-08-27T07:25:39  <wumpus> because that auto-adapts
1772016-08-27T07:25:55  <gmaxwell> so? other than segwit the txn count of blocks won't change much.
1782016-08-27T07:25:56  <wumpus> if blocks grow again, it won't become  slower
1792016-08-27T07:26:19  <wumpus> heh yes that's a completely different discission
1802016-08-27T07:26:22  <sipa> gmaxwell: if you are early in IBD you probably want more blockss
1812016-08-27T07:26:37  <gmaxwell> sipa: okay, I'll but that.. but kind of a corner case.
1822016-08-27T07:26:39  <wumpus> but the fact is that transaction could is a better measure of run time
1832016-08-27T07:26:43  <wumpus> count*
1842016-08-27T07:26:46  <gmaxwell> I was at the verge of suggesting this just do a single block.
1852016-08-27T07:26:56  <wumpus> and amount of data checked
1862016-08-27T07:27:01  *** Ylbam has joined #bitcoin-core-dev
1872016-08-27T07:27:06  <sipa> heh, fine by me as well
1882016-08-27T07:27:22  <gmaxwell> It's not yielding a high payoff in detected errors, the ones I know it detected (chainstate version corruption, need checking back to your last restat to reliably report)
1892016-08-27T07:27:50  <wumpus> well the first priority is to make the insane wait time go away
1902016-08-27T07:28:01  <wumpus> whether the new number becomes low or ultra-low is less important :)
1912016-08-27T07:28:15  <gmaxwell> One or two block keeps the code live and working, I think this code is useful around refactorings and changes to the code.
1922016-08-27T07:48:01  *** Alopex has quit IRC
1932016-08-27T07:48:11  <btcdrak> i guess the reply to Roger's testnet response is no, BU runs in production and is a menace to the network
1942016-08-27T07:48:44  <btcdrak> BU has no peer review and no chance in hell of being used for real
1952016-08-27T07:49:07  *** Alopex has joined #bitcoin-core-dev
1962016-08-27T08:06:04  <btcdrak> wumpus: rebased #7562
1972016-08-27T08:07:38  <sipa> btcdrak: what was the problem with the tests?
1982016-08-27T08:07:42  <btcdrak> gmaxwell: can you explain a bit more about your suggestion regarding my attempt at extracting the MacOSX sdk on linux?
1992016-08-27T08:09:53  <btcdrak> sipa: changing the defaul tx version affected a couple of tests that were comparing hash values, or sorting txs by hash. obviously the hashes change when tx version is bumped. Those seem innocuous. There is one test however that I dont understand which I commented on. Not sure why it is affected by changing the version number.
2002016-08-27T08:10:16  <btcdrak> I dont know if it's the tests' fault and innocuous, or revealing an issue.
2012016-08-27T08:15:46  *** Megaf has joined #bitcoin-core-dev
2022016-08-27T08:20:26  *** fengling has quit IRC
2032016-08-27T08:22:24  *** Guyver2 has quit IRC
2042016-08-27T08:30:54  <gmaxwell> btcdrak: so, first someone extracts it via OSX.  Then we take the extracted binary and find all the offsets for its data in the decompressed file.
2052016-08-27T08:31:00  <gmaxwell> we can distribute the offset list.
2062016-08-27T08:31:21  <gmaxwell> actually, a took like xdelta or another binary diff tool might just handle it.
2072016-08-27T08:34:18  *** MarcoFalke has joined #bitcoin-core-dev
2082016-08-27T08:40:31  <sipa> but the dmg file is compressed
2092016-08-27T08:40:48  <sipa> can we decompress it?
2102016-08-27T08:41:05  <sipa> without "installing", that is
2112016-08-27T08:41:53  <gmaxwell> it's a 7z file according to drak
2122016-08-27T08:43:10  <gmaxwell> so I was assuming that.
2132016-08-27T08:43:11  <btcdrak> gmaxwell: one thing I am not sure about is if 7z is actually extracting it correctly. It seems to be, and the bug seems more like in the linux implementation of hfsplus; but it is feasible that the compression algo was tweaked and 7z is unaware
2142016-08-27T08:43:24  <gmaxwell> unlikely, 7z has checksums.
2152016-08-27T08:47:05  <sipa> so we'd xdelta the 7z-decompressed dmg with the resulting compressed .tar?
2162016-08-27T08:47:21  <gmaxwell> uncompressed tar.
2172016-08-27T08:47:44  <gmaxwell> oh what we want is a tgz.
2182016-08-27T08:47:45  <gmaxwell> yes.
2192016-08-27T08:47:52  <gmaxwell> so the file we normally get out of it.
2202016-08-27T08:48:05  <gmaxwell> and if the resulting delta is small, we call it done.
2212016-08-27T08:48:28  <luke-jr> btcdrak: what version of 7zip did you use?
2222016-08-27T08:48:49  <sipa> btcdrak: where is hfsplus used?
2232016-08-27T08:49:34  <btcdrak> luke-jr: version 9.20
2242016-08-27T08:49:43  <luke-jr> hmm, it tells me: Error: Can not open file as archive
2252016-08-27T08:51:36  *** fengling has joined #bitcoin-core-dev
2262016-08-27T08:56:44  <btcdrak> sipa: https://www.irccloud.com/pastebin/S3zIzO3D/
2272016-08-27T08:58:29  <luke-jr> oh, my DMG file is truncated
2282016-08-27T08:59:04  <btcdrak> so basically the files we want are in Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk, but using the method above you get a bunch of empty files. On a Mac you run "tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk"
2292016-08-27T08:59:35  <btcdrak> cfields_ documented it here https://github.com/bitcoin/bitcoin/blob/master/doc/README_osx.md
2302016-08-27T09:00:08  <sipa> but we don't need all of the contents of that dmg, right?
2312016-08-27T09:00:27  <luke-jr> not nearly
2322016-08-27T09:00:39  <luke-jr> but it's copyrighted and non-redistributable :<
2332016-08-27T09:00:39  <btcdrak> sipa right, we just need the files from Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/* which we turn into a tar.gz
2342016-08-27T09:00:46  <sipa> i see
2352016-08-27T09:00:55  <btcdrak> That gives us 47MB tar.gz :-p
2362016-08-27T09:02:09  <btcdrak> we drop that in the gitian-builder/inputs/ folder when performing the Gitian rituals.
2372016-08-27T09:02:46  <sipa> got it
2382016-08-27T09:02:53  * luke-jr waits on copy over LAN.. Xcode_7.3.1.dmg   27% 1383MB   6.5MB/s   09:08 ETA
2392016-08-27T09:03:03  <sipa> 14m here
2402016-08-27T09:03:19  <luke-jr> you and your fast internet :x :p
2412016-08-27T09:03:58  <sipa> luke-jr: this is over t-mobile roaming data
2422016-08-27T09:04:26  *** fengling has quit IRC
2432016-08-27T09:04:29  <luke-jr> sadly, T-Mobile data here seems to only be about 5 Mbps
2442016-08-27T09:10:21  <gmaxwell> 2hr 49m remaining here.
2452016-08-27T09:10:33  *** moli has quit IRC
2462016-08-27T09:14:20  <btcdrak> RIP sipa's roaming data charges
2472016-08-27T09:14:25  <sipa> btcdrak: it's free
2482016-08-27T09:14:41  <sipa> it's effectively cheaper than my swiss wired or wireless internet
2492016-08-27T09:15:04  <sipa> s/free/fixed price/
2502016-08-27T09:16:30  <gmaxwell> btcdrak: a while back tmobile ceo put out some open letter about "data abusers" and for some reason it said, 'mining bitcoin' -- for a while we wondered if perhaps he was talking about blockstream. https://newsroom.t-mobile.com/news-and-blogs/stopping-network-abusers.htm
2512016-08-27T09:16:47  <gmaxwell> apparently blockstream was using several TB a month on tmobile.
2522016-08-27T09:18:26  <gmaxwell> doesn't everyone run a full node on their phone?
2532016-08-27T09:18:44  <luke-jr> lol, xdelta is 45 MB
2542016-08-27T09:18:59  <sipa> :(
2552016-08-27T09:19:09  <sipa> luke-jr: how did you decompress the dmg?
2562016-08-27T09:19:12  <luke-jr> 7z
2572016-08-27T09:19:28  <gmaxwell> lol
2582016-08-27T09:19:53  <gmaxwell> might need more options for it to actually find the match
2592016-08-27T09:20:03  <luke-jr> maybe there's some tweaking to the tar we need to put it in a better order?
2602016-08-27T09:20:10  *** fengling has joined #bitcoin-core-dev
2612016-08-27T09:20:17  <gmaxwell> yes, that too.
2622016-08-27T09:20:21  <luke-jr> oh wait
2632016-08-27T09:20:27  <luke-jr> hmm, nm
2642016-08-27T09:20:33  <luke-jr> the uncompressed tar is 306 MB FWIW
2652016-08-27T09:20:38  <luke-jr> but xdelta's patch is compressed.
2662016-08-27T09:20:47  <gmaxwell> were you using the compressed tar as the input?
2672016-08-27T09:20:50  <luke-jr> no
2682016-08-27T09:21:22  <sipa> hpmount, mount -t hfsplus, 7zr... all fail to open the file
2692016-08-27T09:21:36  <gmaxwell> maybe we should just encrypt the damn tar with the whole dmg file as the key, and then call that sufficient for legal purposes. :P
2702016-08-27T09:22:03  <luke-jr> sipa: need 7z, not 7zr
2712016-08-27T09:22:07  <luke-jr> 7zr has formats removed
2722016-08-27T09:22:31  <luke-jr> gmaxwell: hmm, I wonder if that'd be okay
2732016-08-27T09:22:54  <luke-jr> I turned off xdelta's patch compression and it's 261 MB :/
2742016-08-27T09:23:10  <gmaxwell> smaller than 300 so it found something.
2752016-08-27T09:23:48  <gmaxwell> luke-jr: I think doing that would be defensible.
2762016-08-27T09:23:59  <luke-jr> xdelta3: warning: input position 310378496 overflowed instruction buffer, needed 43091 (vs. 32768), consider changing -I
2772016-08-27T09:24:13  <gmaxwell> consider changing -I
2782016-08-27T09:24:20  * luke-jr sets -I to unlimited
2792016-08-27T09:24:29  <luke-jr> not doing much better :<
2802016-08-27T09:24:47  <luke-jr> -rw-r--r-- 1 luke-jr luke-jr 261M Aug 27 09:24 MacOSX.xdelta
2812016-08-27T09:24:52  <gmaxwell> there are some other binary patching tools, xdelta was the first that came to mind.
2822016-08-27T09:25:09  <sipa> xdelta and xdelta3 seem to be different things
2832016-08-27T09:28:03  <gmaxwell> open-vcdiff
2842016-08-27T09:28:23  <luke-jr> I'm writing something custom
2852016-08-27T09:28:52  <luke-jr> hmm
2862016-08-27T09:28:57  <luke-jr> I can't mmap this DMG :<
2872016-08-27T09:29:07  <btcdrak> well surely the better solution would be to fix the hfsplus driver?
2882016-08-27T09:29:09  <gmaxwell> luke-jr: upgrade to a darn 64 bit OS already luke!
2892016-08-27T09:31:51  <luke-jr> oh, if I build with -static I can
2902016-08-27T09:38:52  <sipa> i can't get it below 46 MB
2912016-08-27T09:39:17  <sipa> that's with
2922016-08-27T09:39:19  <sipa> xdelta3 encode -P $((2**30)) -I 0 -B 2147483648 -W 16777216
2932016-08-27T09:39:31  <sipa> (the highest values for each of the limits that work)
2942016-08-27T09:41:45  <sipa> it runs too fast, though
2952016-08-27T09:42:12  <sipa> this is an 8 GB source file, and it only takes a few seconds
2962016-08-27T09:42:26  <gmaxwell> "I demand my np-complete problems solving software be slow"
2972016-08-27T09:42:44  <sipa> i demand it actually uses the whole input
2982016-08-27T09:43:49  <gmaxwell> try open-vcdiff?
2992016-08-27T09:44:28  *** Ginnarr has joined #bitcoin-core-dev
3002016-08-27T09:45:29  <sipa> 42 MB
3012016-08-27T09:45:43  <sipa> gmaxwell: seems that uses the same algorithm as xdelta3
3022016-08-27T09:46:11  <gmaxwell> failure?
3032016-08-27T09:47:27  <luke-jr> http://codepad.org/Yi5FEWML <-- any obvious bugs or possible optimizations?
3042016-08-27T09:49:07  <btcdrak> also, I tried the dmg2img tool, but that doesnt appear to have been updated to handle the altered compression
3052016-08-27T09:50:20  <GitHub180> [bitcoin] ajtowns closed pull request #8575: leveldb: generate lib independent of locale sort (master...leveldb-locale-reproducible) https://github.com/bitcoin/bitcoin/pull/8575
3062016-08-27T09:50:21  <luke-jr> welp, failed to find the first 82 byte file :/
3072016-08-27T09:50:26  <gmaxwell> luke-jr: well the needles may not be in the file in a contigious chunk.
3082016-08-27T09:50:40  <luke-jr> gmaxwell: surely 82 bytes would be :<
3092016-08-27T09:50:58  <gmaxwell> unless the file system stores the first n bytes of files in the inode table.
3102016-08-27T09:51:03  <luke-jr> hmm
3112016-08-27T09:51:04  <gmaxwell> (so that magic is fast)
3122016-08-27T09:51:18  <gmaxwell> or just to prevent the indirection
3132016-08-27T09:51:38  <luke-jr> would it make sense to target the 2nd 4096 bytes of the file then?
3142016-08-27T09:53:55  <gmaxwell> to test your tool grab the second 4096 bytes of the haystack.
3152016-08-27T09:54:02  <gmaxwell> but yes, you could try that.
3162016-08-27T09:54:41  <luke-jr> http://codepad.org/VvfwXbqo isn't getting anything so far either
3172016-08-27T09:54:59  <luke-jr> you mean of the needle, right?
3182016-08-27T09:55:25  <gmaxwell> I mean grab a 4096 byte chunk out of the data you are searching in, just to verify your tool works.
3192016-08-27T09:56:12  <luke-jr> oh
3202016-08-27T09:56:15  <gmaxwell> dd if=5.hfs of=junk bs=4096 skip=1234 count=1
3212016-08-27T09:56:18  <luke-jr> "In Mac OS X Snow Leopard 10.6, HFS+ compression was added. In open source and some other areas this is referred to as AppleFSCompression. Compressed data may be stored in either an extended attribute or the resource fork."
3222016-08-27T09:56:25  <luke-jr> I bet this is what we're dealing with :|
3232016-08-27T09:57:26  *** fengling has quit IRC
3242016-08-27T09:57:26  <sipa> seems likely, indeed
3252016-08-27T09:59:11  <gmaxwell> bleh
3262016-08-27T10:00:19  <luke-jr> http://www.spinics.net/lists/linux-fsdevel/msg55545.html anyone want to implement? :/
3272016-08-27T10:02:38  *** fengling has joined #bitcoin-core-dev
3282016-08-27T10:10:17  <sipa> leveldb 1.19 was released
3292016-08-27T10:11:27  <sipa>  52 files changed, 1976 insertions(+), 429 deletions(-)
3302016-08-27T10:11:33  <sipa> (compared to 1.18)
3312016-08-27T10:11:59  <gmaxwell> doesn't sound too bad to review.
3322016-08-27T10:12:47  <sipa> they added a cache pruning
3332016-08-27T10:13:27  <gmaxwell> doesn't sound especially relevant then.
3342016-08-27T10:13:36  <sipa> it's been 2 years since their previous release
3352016-08-27T10:13:45  <sipa> glad to see activity :)
3362016-08-27T10:15:11  <sipa> ARM64 support for memory barriers seems relevant
3372016-08-27T10:15:16  <sipa> cache size estimation
3382016-08-27T10:15:26  <gmaxwell> oh the arm64 might fix performance on odroid c2
3392016-08-27T10:28:06  <sipa> this is interesting but not included: https://github.com/google/leveldb/pull/309
3402016-08-27T10:29:29  <gmaxwell> wumpus started benchmarking what that might look like.
3412016-08-27T10:29:41  <gmaxwell> odroid c2 has a crc32c accelerator that is sutiable too.
3422016-08-27T10:35:26  *** fengling has quit IRC
3432016-08-27T10:35:52  *** harrymm has quit IRC
3442016-08-27T10:47:08  *** moli has joined #bitcoin-core-dev
3452016-08-27T10:51:10  *** harrymm has joined #bitcoin-core-dev
3462016-08-27T11:03:35  <luke-jr> found implementation of HFS+ compression
3472016-08-27T11:03:44  <sipa> where?
3482016-08-27T11:03:49  <luke-jr> SleuthKit
3492016-08-27T11:04:38  * luke-jr suggests we drop manpages from the tarball
3502016-08-27T11:06:53  <luke-jr> xz -d <inodes.xz | while read inode filename; do filename="y/$filename"; p=$(dirname $filename); mkdir -vp $p; icat  ../5.hfs $inode >"$filename"; done
3512016-08-27T11:07:54  * luke-jr tests result in gitian
3522016-08-27T11:08:18  <luke-jr> the tarball is missing a bunch of stuff that looks like dummy files (but still 66 MB)
3532016-08-27T11:08:35  <sipa> luke-jr: nice!
3542016-08-27T11:09:24  <luke-jr> FWIW, I generated the inode list using mount and stat :p
3552016-08-27T11:10:14  * luke-jr would be surprised if SleuthKit didn't have a way to get that info, but didn't see it
3562016-08-27T11:12:38  <luke-jr> fls seems to be the tool
3572016-08-27T11:15:49  <sipa> sleuthkit added support for HFS+ in version 3.1
3582016-08-27T11:16:14  <luke-jr> I have 4.0.2
3592016-08-27T11:16:57  <sipa> ubuntu 12.04 has sleuthkit 3.2.3
3602016-08-27T11:17:08  <sipa> so i think we're good
3612016-08-27T11:18:03  <luke-jr> oh.
3622016-08-27T11:18:07  <luke-jr> except gitian doesn't like it :<
3632016-08-27T11:18:27  <sipa> due to the missing manpages...?
3642016-08-27T11:18:32  <luke-jr> hmm
3652016-08-27T11:18:34  <luke-jr> I think symlinks
3662016-08-27T11:18:38  <sipa> does it do a checksum check perhaps on the .tgz?
3672016-08-27T11:18:40  <luke-jr> can't find the lib for c++
3682016-08-27T11:18:50  <luke-jr> it can't, everyone's .tgz is different ;p
3692016-08-27T11:19:40  <luke-jr> yeah, I think it's missing symlinks
3702016-08-27T11:27:07  *** belcher has joined #bitcoin-core-dev
3712016-08-27T11:29:41  <luke-jr> fls ../5.hfs -rpF 154283 | perl -nle 'm/^(r|l)\S*\s(\d+)\:\s*(.*$)/ && print "$1 $2 $3"' | while read type inode filename; do filename="MacOSX10.11.sdk/$filename"; mkdir -p "$(dirname "$filename")"; if [ "$type" = "l" ]; then ln -s $(icat ../5.hfs $inode) "$filename"; else icat ../5.hfs $inode >"$filename"; fi; done
3722016-08-27T11:32:18  <luke-jr> tempting to figure out some kind of trick to download data on demand for this :P
3732016-08-27T11:39:28  <luke-jr> hm, surprised there's no single-file curl FUSE fs
3742016-08-27T11:48:28  *** Ginnarr has quit IRC
3752016-08-27T11:54:45  *** MarcoFalke has left #bitcoin-core-dev
3762016-08-27T11:54:48  <luke-jr> gitian build matches
3772016-08-27T12:09:15  <sipa> \o/
3782016-08-27T12:22:54  <btcdrak> wow
3792016-08-27T12:26:19  <luke-jr> ?
3802016-08-27T12:32:40  <CodeShark> I think btcdrak just considers perl code to be inherently aesthetically pleasing :p
3812016-08-27T12:33:34  <luke-jr> lol
3822016-08-27T12:33:40  <luke-jr> latest draft has no perl sadly: http://codepad.org/1wMp5vse
3832016-08-27T12:33:48  <luke-jr> I'll wait until I'm actually awake to turn it into a PR
3842016-08-27T12:34:19  <luke-jr> -rw-r--r-- 1 luke-jr luke-jr  21M Aug 27 12:33 MacOSX10.11.sdk.tar.gz
3852016-08-27T12:35:29  <luke-jr> night
3862016-08-27T12:38:19  *** achow101 has joined #bitcoin-core-dev
3872016-08-27T12:42:04  <phantomcircuit> luke-jr, the client can take ages to start if you closed it after you downloaded a bunch of blocks but before you processed them
3882016-08-27T12:42:14  <phantomcircuit> cause it processes all of them in AppInit2
3892016-08-27T12:42:27  <phantomcircuit> (actually it doesn't anymore so this shouldn't be an issue in 0.13.x)
3902016-08-27T12:43:24  <sipa> leveldb 1.19 should start up significantly faster
3912016-08-27T12:44:18  <phantomcircuit> oh and leveldb reads it's journal which is potentially very slow also
3922016-08-27T12:52:03  *** achow101 has quit IRC
3932016-08-27T12:56:44  *** kadoban has joined #bitcoin-core-dev
3942016-08-27T13:00:12  *** moli has quit IRC
3952016-08-27T13:05:23  *** achow101 has joined #bitcoin-core-dev
3962016-08-27T13:23:18  *** Chris_Stewart_5 has quit IRC
3972016-08-27T13:24:25  <GitHub3> [bitcoin] sipa opened pull request #8610: Share unused mempool memory with coincache (master...sharemem) https://github.com/bitcoin/bitcoin/pull/8610
3982016-08-27T13:26:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
3992016-08-27T13:29:19  <GitHub62> [bitcoin] sipa opened pull request #8611: Reduce default number of blocks to check at startup (master...fastcheck) https://github.com/bitcoin/bitcoin/pull/8611
4002016-08-27T13:32:32  *** moli has joined #bitcoin-core-dev
4012016-08-27T13:34:56  <phantomcircuit> sipa, #8610 instead of doing that can you add a framework for limiting memory globally?
4022016-08-27T13:35:25  <GitHub51> [bitcoin] sipa opened pull request #8612: Check for compatibility with download in FindNextBlocksToDownload (master...fixwitban) https://github.com/bitcoin/bitcoin/pull/8612
4032016-08-27T13:35:31  <sipa> phantomcircuit: i don't have 5 years
4042016-08-27T13:35:36  <phantomcircuit> it can probably even be as simple as a global memory limit goal and percentages for now
4052016-08-27T13:36:06  <sipa> percentages isn't good enough
4062016-08-27T13:36:35  <sipa> you need something that detects "oh, the mempool actually does not need its maximum usage... let's move some of its allocation elsewhere"
4072016-08-27T13:36:48  <sipa> but can change that back when the mempool grows
4082016-08-27T13:37:00  <phantomcircuit> percentages and an atomic of how much everything thinks it's using?
4092016-08-27T13:37:09  <sipa> what does that solve?
4102016-08-27T13:37:17  <phantomcircuit> "usage is below 90% of limit i can exceed my limit"
4112016-08-27T13:37:37  <phantomcircuit> i guess you need callbacks to apply memory pressure then
4122016-08-27T13:37:42  <sipa> yeah
4132016-08-27T13:37:47  <phantomcircuit> but you need that for sharing memory at all
4142016-08-27T13:38:04  <sipa> utxo set and mempool are things that are constantly checked anyway
4152016-08-27T13:38:24  <sipa> what this PR implements is mempool < maxmempool && coincache + mempool < maxtotal
4162016-08-27T13:38:44  <sipa> so arguably it already has something like a global limit (though it's just coincache + mempool)
4172016-08-27T13:39:12  <sipa> the mempool has complicated semantics wrt computing relay fee based on its size
4182016-08-27T13:39:26  <sipa> making its actual limit dynamic is harder, i think
4192016-08-27T13:41:07  <sipa> also, i don't think it's needed to for example give your mempool 10 GB of memory even if you have 64 GB available... that'll just make your memory a sewer for spam that nobody else on the network accepts
4202016-08-27T14:00:38  *** belcher has quit IRC
4212016-08-27T14:01:46  <GitHub42> [bitcoin] sipa opened pull request #8613: [preview] LevelDB 1.19 (master...leveldb119) https://github.com/bitcoin/bitcoin/pull/8613
4222016-08-27T15:00:42  *** spudowiar has joined #bitcoin-core-dev
4232016-08-27T15:08:08  *** fengling has joined #bitcoin-core-dev
4242016-08-27T15:24:26  *** fengling has quit IRC
4252016-08-27T15:39:40  *** spudowiar has quit IRC
4262016-08-27T16:03:18  *** spudowiar has joined #bitcoin-core-dev
4272016-08-27T16:35:05  *** MarcoFalke has joined #bitcoin-core-dev
4282016-08-27T16:39:49  *** harrymm has quit IRC
4292016-08-27T16:53:01  *** harrymm has joined #bitcoin-core-dev
4302016-08-27T17:02:35  *** Megaf has quit IRC
4312016-08-27T17:08:00  *** Megaf has joined #bitcoin-core-dev
4322016-08-27T17:10:09  *** achow101 has quit IRC
4332016-08-27T17:22:34  *** Megaf has quit IRC
4342016-08-27T17:25:53  *** Samdney has joined #bitcoin-core-dev
4352016-08-27T17:36:35  *** JackH has quit IRC
4362016-08-27T17:42:52  *** kadoban has quit IRC
4372016-08-27T17:48:14  *** achow101 has joined #bitcoin-core-dev
4382016-08-27T18:19:15  *** Lysanders has joined #bitcoin-core-dev
4392016-08-27T18:57:52  *** Guyver2 has joined #bitcoin-core-dev
4402016-08-27T19:11:02  *** spudowiar has quit IRC
4412016-08-27T19:15:42  *** JackH has joined #bitcoin-core-dev
4422016-08-27T19:19:51  *** JackH has quit IRC
4432016-08-27T19:22:47  *** justanotheruser has quit IRC
4442016-08-27T19:23:56  *** justanotheruser has joined #bitcoin-core-dev
4452016-08-27T19:27:44  *** justanotheruser has quit IRC
4462016-08-27T19:28:10  *** justanotheruser has joined #bitcoin-core-dev
4472016-08-27T19:45:07  *** dgenr8 has quit IRC
4482016-08-27T19:45:32  *** justanotheruser has quit IRC
4492016-08-27T19:53:36  <gmaxwell> sipa: I think we should backport feeler connections. Beyond the security/robustness improvement, they'll help reduce the harm of network density loss w/ segwit.
4502016-08-27T19:54:55  *** laurentmt has joined #bitcoin-core-dev
4512016-08-27T20:02:54  *** timothy has quit IRC
4522016-08-27T20:07:27  *** justanotheruser has joined #bitcoin-core-dev
4532016-08-27T20:12:10  *** laurentmt has quit IRC
4542016-08-27T20:18:46  <GitHub126> [bitcoin] luke-jr opened pull request #8617: Include instructions to extract Mac OS X SDK on Linux using 7zip and SleuthKit (master...gitian_osx_extractor) https://github.com/bitcoin/bitcoin/pull/8617
4552016-08-27T20:20:27  <luke-jr> sipa: so have you confirmed already that LevelDB 1.19 is reasonably certain to not have forking changes? or are you assuming we'll do collectively that before merging/releasing?
4562016-08-27T20:22:44  <sipa> luke-jr: i'm reasonably certain, yes, but i encourage review
4572016-08-27T20:23:12  <sipa> i'm just pring it to bitcoin already to make testing easier
4582016-08-27T20:53:37  *** kadoban has joined #bitcoin-core-dev
4592016-08-27T21:14:36  *** achow101 has quit IRC
4602016-08-27T21:43:20  *** spudowiar has joined #bitcoin-core-dev
4612016-08-27T21:59:07  *** Giszmo has joined #bitcoin-core-dev
4622016-08-27T22:22:36  *** Guyver2 has quit IRC
4632016-08-27T22:26:21  *** Alopex has quit IRC
4642016-08-27T22:27:26  *** Lysanders has quit IRC
4652016-08-27T22:27:27  *** Alopex has joined #bitcoin-core-dev
4662016-08-27T22:32:02  *** belcher has joined #bitcoin-core-dev
4672016-08-27T22:47:16  *** Yogh has quit IRC
4682016-08-27T22:47:42  *** MarcoFalke has quit IRC
4692016-08-27T22:49:23  *** Yogh has joined #bitcoin-core-dev
4702016-08-27T22:50:07  *** Alopex has quit IRC
4712016-08-27T22:51:12  *** Alopex has joined #bitcoin-core-dev
4722016-08-27T22:51:24  *** harrymm has quit IRC
4732016-08-27T23:14:22  *** nibor has quit IRC
4742016-08-27T23:15:09  *** nibor has joined #bitcoin-core-dev
4752016-08-27T23:16:02  *** Alopex has quit IRC
4762016-08-27T23:17:07  *** Alopex has joined #bitcoin-core-dev
4772016-08-27T23:30:51  *** belcher has quit IRC
4782016-08-27T23:43:45  *** cryptapus_afk is now known as cryptapus