12015-10-30T00:05:34  *** BashCo_ has joined #bitcoin-core-dev
  22015-10-30T00:07:23  <gmaxwell> sipa: ^ are you setup to produce windows binaries right now?
  32015-10-30T00:07:48  <warren> gmaxwell: here?
  42015-10-30T00:08:02  <gmaxwell> right.
  52015-10-30T00:08:09  <warren> gmaxwell: sorry the split wasn't entirely clear to me.
  62015-10-30T00:08:19  *** BashCo has quit IRC
  72015-10-30T00:08:30  <gmaxwell> warren: this channel should be for bitcoin core exclusive discussion.
  82015-10-30T00:08:55  <gmaxwell> no one else gives a hoot at how our rpc auth works. :)
  92015-10-30T00:09:41  <Luke-Jr> Fedora does apparently
 102015-10-30T00:09:42  * Luke-Jr hides
 112015-10-30T00:09:55  <sipa> gmaxwell: do i get 30 minutes to find a bug? :)
 122015-10-30T00:10:23  <gmaxwell> sipa: hahah
 132015-10-30T00:10:55  <gmaxwell> sipa: I don't want it merged; there is a user that manged to get pubkey "" in his wallet on bct, and I want to give him a binary to see if it rejects it.
 142015-10-30T00:11:48  <gmaxwell> warren: as far as selinux policy, I think it's unfortunate that you cannot confine bitcoin core to have no access outside of its own directory.
 152015-10-30T00:12:00  <gmaxwell> though maybe that should be a boolean.
 162015-10-30T00:12:10  <gmaxwell> since its only needed for the walletimport/backup stuff.
 172015-10-30T00:13:22  <warren> Would it be bad if the system-level systemd service were named something like bitcoinsys.service, with a selinux context bitcoinsys_t, and a bitcoin-cli wrapper named something like /usr/bin/bitcoinsys_cli that only adds the --datadir /path/to/system/datadir/
 182015-10-30T00:13:58  <Luke-Jr> and what if the user wants bitcoind run per-user? :P
 192015-10-30T00:16:06  <gmaxwell> warren: what happens if a second datadir is specified to bitcoin-cli?
 202015-10-30T00:16:57  <gmaxwell> warren: it's all ugly, bitcoin core is currently not intended for system level operation.
 212015-10-30T00:17:25  <warren> gmaxwell: hence why i'm against shipping a .service file at all
 222015-10-30T00:20:49  <gmaxwell> in any case that doesn't sound too horrible but the wrapper would need to not prevent you from overriding the datadir, and it needs to be tested with e.g. super long inputs.
 232015-10-30T00:21:30  <gmaxwell> better perhaps would be a compile time setting to adjust the default datadir. Do we have one of those?
 242015-10-30T00:22:45  *** testing-tape has quit IRC
 252015-10-30T00:23:27  <sipa> not afaik
 262015-10-30T00:23:36  <sipa> gmaxwell: bug found, will build
 272015-10-30T00:23:49  <gmaxwell> lol okay.
 282015-10-30T00:24:02  <Luke-Jr> warren: doesn't systemd support per-user services?
 292015-10-30T00:24:31  <sipa> Luke-Jr: systemd would be a great operating system, if it only contained a service manager
 302015-10-30T00:24:47  <sipa> s/if it only/if only it/
 312015-10-30T00:24:55  <Luke-Jr> don't get me wrong, I hate systemd. but I thought this was one of the few benefits to it.
 322015-10-30T00:26:26  *** pavel_ has joined #bitcoin-core-dev
 332015-10-30T00:26:56  <gmaxwell> Seperate subject; https://github.com/bitcoin/bitcoin/pull/6902  I feel that changes like this make the software more opaque.  Now to understand whats being done there you must read two places instead of one.  If you only look at policy.h you will likely erroniously believe it applies to p2sh somehow.
 342015-10-30T00:27:33  <warren> Luke-Jr: yes it's possible
 352015-10-30T00:28:25  <gmaxwell> It's of a class of change where I purposfully try to keep my opinion to myself to avoid amplifying bikeshedding, because it fundimentally does not matter.
 362015-10-30T00:28:34  *** paveljanik has quit IRC
 372015-10-30T00:32:04  <sipa> gmaxwell: 32-bit or 64-bit?
 382015-10-30T00:33:39  <sipa> gmaxwell: will take a while to build dependencies
 392015-10-30T00:34:27  <gmaxwell> user say "or" so 64-bit I guess.
 402015-10-30T00:50:15  <GitHub198> [bitcoin] nathanimpact opened pull request #6909: 0.8 (master...0.8) https://github.com/bitcoin/bitcoin/pull/6909
 412015-10-30T00:50:46  <GitHub180> [bitcoin] gmaxwell closed pull request #6909: 0.8 (master...0.8) https://github.com/bitcoin/bitcoin/pull/6909
 422015-10-30T00:50:57  <GitHub190> [bitcoin] nathanimpact opened pull request #6910: 0.9 (master...0.9) https://github.com/bitcoin/bitcoin/pull/6910
 432015-10-30T00:51:17  <GitHub40> [bitcoin] nathanimpact opened pull request #6911: 0.10 (master...0.10) https://github.com/bitcoin/bitcoin/pull/6911
 442015-10-30T00:51:27  <GitHub112> [bitcoin] gmaxwell closed pull request #6910: 0.9 (master...0.9) https://github.com/bitcoin/bitcoin/pull/6910
 452015-10-30T00:51:42  <GitHub69> [bitcoin] nathanimpact opened pull request #6912: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6912
 462015-10-30T00:51:48  <gmaxwell> I wish github would let you block these.
 472015-10-30T00:51:54  <GitHub93> [bitcoin] gmaxwell closed pull request #6911: 0.10 (master...0.10) https://github.com/bitcoin/bitcoin/pull/6911
 482015-10-30T00:52:12  <GitHub198> [bitcoin] gmaxwell closed pull request #6912: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6912
 492015-10-30T00:52:54  <GitHub45> [bitcoin] nathanimpact opened pull request #6913: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6913
 502015-10-30T00:54:03  <belcher> you can block them(!) on the settings panel somewhere in github you can configure these to happen or not
 512015-10-30T00:54:30  <gmaxwell> belcher: what specifically does it block?
 522015-10-30T00:54:44  <belcher> turns off or on these bots
 532015-10-30T00:55:06  <sipa> belcher: we want to block the PRs, not the reporting here :)
 542015-10-30T00:55:23  <belcher> i see, im not sure thats possible
 552015-10-30T00:55:56  <belcher> if you know they're coming you could set the channel mode +n (i think) which stops them being able to message without joining
 562015-10-30T00:56:59  <petertodd> gmaxwell: ACK re your thoughts on #6902 - this isn't a constant that you can change after all!
 572015-10-30T00:57:01  <gmaxwell> the bots we want, just not the pointless PRs to merge entire (sometimes altcoin) branches into master.
 582015-10-30T00:59:44  <sipa> gmaxwell: building qt failed, sorry
 592015-10-30T00:59:49  <petertodd> gmaxwell: er, scrap that, no policy, not script, dumb of me...
 602015-10-30T00:59:51  <sipa> not going to spend time investigating
 612015-10-30T01:20:43  *** testing-tape has joined #bitcoin-core-dev
 622015-10-30T01:25:50  *** testing-tape has quit IRC
 632015-10-30T01:29:23  <GitHub139> [bitcoin] sipa opened pull request #6914: [WIP] Add pre-allocated vector type and use it for CScript (master...prevector) https://github.com/bitcoin/bitcoin/pull/6914
 642015-10-30T01:31:02  <gmaxwell> sipa: why doesn't it bounds check?
 652015-10-30T01:31:28  <sipa> gmaxwell: because i'm lazy
 662015-10-30T01:32:10  <gmaxwell> sipa: 13% faster with script checking enabled?!
 672015-10-30T01:32:18  <sipa> gmaxwell: disabled
 682015-10-30T01:32:20  <gmaxwell> oh disabled good.
 692015-10-30T01:32:28  <gmaxwell> okay. Yea, missed that on first read.
 702015-10-30T01:33:03  <GitHub7> [bitcoin] sdaftuar opened pull request #6915: [Mempool] Improve removal of invalid transactions after reorgs (master...fix-reorg-handling) https://github.com/bitcoin/bitcoin/pull/6915
 712015-10-30T01:34:26  <sipa> the only place where we actually use ::at() is in CScript::IsPayToScriptHash(), and it's prefixed with a size check there
 722015-10-30T01:35:41  <gmaxwell> maybe just get rid of the at and make it not compile if its used? Rationale: providing something thats almost vector but less safe in some subtle way will be forgotten. :)
 732015-10-30T01:37:12  <gmaxwell> sipa: did you fix the scriptcheck on reindex generally or just patch it out? if the former can you PR the fix?
 742015-10-30T01:37:36  <sipa> i just patched it out
 752015-10-30T01:37:47  <gmaxwell> (that particular misbehavior is a beautiful interaction with the always need to reindex issue in windows I bet. :-/ )
 762015-10-30T01:57:28  <sipa> gmaxwell: building 6906 (+6900, since windows building is broken on modern mingw without 6900)
 772015-10-30T01:58:06  <CodeShark> are you guys trying to reproduce the windows db corruption issues? :)
 782015-10-30T01:58:39  <gmaxwell> not at the moment.
 792015-10-30T01:58:45  <gmaxwell> CodeShark: warren was talking about that earlier.
 802015-10-30T01:59:02  <gmaxwell> I'm trying to debug a wallet that ended up with a corrupted pubkey in it.
 812015-10-30T01:59:19  <gmaxwell> which is in possession of an end user who is helpful but needed a binary.
 822015-10-30T01:59:38  <CodeShark> ah
 832015-10-30T02:01:23  *** daniel___ has quit IRC
 842015-10-30T02:01:48  *** daniel___ has joined #bitcoin-core-dev
 852015-10-30T02:03:06  <gmaxwell> As an aside, same user is running a 0.8GB wallet with hundreds of thousands of transactions, on windows.
 862015-10-30T02:04:18  <CodeShark> hah
 872015-10-30T02:04:21  *** Ylbam has quit IRC
 882015-10-30T02:04:56  <CodeShark> doesn't sound very pleasant on multiple fronts :p
 892015-10-30T02:19:22  <dcousens> sipa: around?
 902015-10-30T02:19:55  <sipa> yes
 912015-10-30T02:20:32  <dcousens> sipa: 23% less memory, sounds like its simply a matter of the allocator being more exact instead of allocating more generously etc
 922015-10-30T02:20:47  <sipa> ?
 932015-10-30T02:21:15  <dcousens> sipa: conceptually, how does https://github.com/bitcoin/bitcoin/pull/6914 save memory, except that in some cases it is allocated on the stack?
 942015-10-30T02:21:33  <sipa> dcousens: well it uses 32-bit instead of 64-bit sizes
 952015-10-30T02:21:37  <dcousens> (and ignoring the 4/8 byte pointer)
 962015-10-30T02:21:40  <sipa> so it always saves 4 to 8 bytes
 972015-10-30T02:21:57  <sipa> apart from that, the only way it saves is by not allocating on the heap
 982015-10-30T02:22:05  <sipa> which has a 16-32 byte overhead
 992015-10-30T02:22:27  <dcousens> and the hunch is that takes up 23% of available memory?
1002015-10-30T02:22:33  <sipa> so the old structure is on average 40 + N bytes storage
1012015-10-30T02:23:21  <sipa> sorry, 48 + N
1022015-10-30T02:23:57  <sipa> the new one is 32 for N<=28, and 56 + N for N > 28
1032015-10-30T02:24:37  <sipa> since almost all CScripts in memory are P2PKH or P2SH, which are less than 28 bytes, this is an improvement
1042015-10-30T02:25:58  <gmaxwell> sipa: aren't the N objects seperately heap allocated?
1052015-10-30T02:27:06  <dcousens> sipa: any reason not to just use union a std::array and std::vector?
1062015-10-30T02:27:55  <sipa> gmaxwell: no
1072015-10-30T02:28:11  <sipa> dcousens: that would be much larger
1082015-10-30T02:28:31  <sipa> the trick here is to share the size variable between the two approaches
1092015-10-30T02:28:44  <sipa> which you can't do if you use the standard std::vector
1102015-10-30T02:30:39  <gmaxwell> sipa: hm for some reason I thought it was, so that the removal of objects could be supported without invalidating pointers.
1112015-10-30T02:30:51  <dcousens> sipa: could the same memory savings be achieved by just abstracting a CScript specialised for smaller sizes?
1122015-10-30T02:31:01  <dcousens> Eh, that'd be as bad nvm
1132015-10-30T02:31:19  <sipa> dcousens: perhaps i failed to highlihght the largest advantage: reducing load on the heap
1142015-10-30T02:31:42  <sipa> which results in worse memory locality and memory fragmentation :)
1152015-10-30T02:31:53  <sipa> gmaxwell: you're thinking about a linked list...
1162015-10-30T02:31:58  <dcousens> indeed, probably where the 13% speedup came from
1172015-10-30T02:32:13  <sipa> dcousens: and simpler copying
1182015-10-30T02:32:23  <sipa> gmaxwell: a vector can't delete elements without invalidating iterators
1192015-10-30T02:33:46  *** belcher has quit IRC
1202015-10-30T02:35:30  <dcousens> sipa: is push_back used at all?
1212015-10-30T02:35:51  <sipa> no clue
1222015-10-30T02:36:05  <dcousens> just wondering, maybe drop the capacity functionality if we don't need it
1232015-10-30T02:36:15  <dcousens> I feel like we' wouldn't be re-allocating these vectors very often
1242015-10-30T02:36:28  <sipa> well i wanted something API compatible :)
1252015-10-30T02:36:44  <sipa> so it's not a huge burden to go review whether it can be used or not
1262015-10-30T02:37:04  <sipa> if you just want a compact vector, there are other changes you can make
1272015-10-30T02:37:08  <dcousens> sipa: I guess if you're looking for public compatibility
1282015-10-30T02:37:42  <dcousens> sipa: IMHO for a change like this, we should start with the smallest subset of changes, optimize on that, then introduce public API compatibility
1292015-10-30T02:38:01  <sipa> the subset of changes is as small as it can be
1302015-10-30T02:38:35  <sipa> the PR contains a few not-strictly necessary changes to just break the assumption that CScript is a vector
1312015-10-30T02:42:27  <dcousens> ooi, why free vs new?
1322015-10-30T02:42:46  <sipa> because i need realloc
1332015-10-30T02:43:08  <sipa> and this works at a lower level: it's allocating memory, not objects
1342015-10-30T02:43:12  <sipa> new is for allocating objects
1352015-10-30T02:43:33  <dcousens> ? you have a clear type though
1362015-10-30T02:43:51  <sipa> which?
1372015-10-30T02:43:57  <dcousens> the realloc might not be necessary if you don't need the push_back/change_capacity etc
1382015-10-30T02:44:00  <dcousens> T
1392015-10-30T02:44:06  <sipa> no, that would call T's constructor
1402015-10-30T02:44:29  <sipa> the reserved but unused memory doesn't contain constructed T objects
1412015-10-30T02:44:36  <dcousens> True, but isn't T in this case always a char*?
1422015-10-30T02:44:47  <sipa> i don't plan to just use this for char
1432015-10-30T02:45:54  <dcousens> hmmm, well, code looks good, tentativee ACK simply cause I'm still not sure its the best way, but, it is a solution, and it exists :) wd
1442015-10-30T02:49:02  <gmaxwell> sipa: did the prior stl instrumentation you did before allow you to easily measure vector overhead in a useful way to go find the biggest offenders?
1452015-10-30T02:49:42  <dcousens> if we broke the API compatibility, I feel like we could just use managed pointers [at the CSCript level] quite easily with a pooled stack allocator for small scripts, but, if this might be easier if you have other future plans for other areas
1462015-10-30T02:50:37  <sipa> dcousens: that would indeed by an improvement; CBlock could have its own pool for example, and CTransaction could couple a pool too
1472015-10-30T02:50:45  <sipa> dcousens: but it's a much more invasive change to do well
1482015-10-30T02:50:50  <dcousens> indeed
1492015-10-30T02:50:51  <sipa> gmaxwell: define offender?
1502015-10-30T02:51:48  <dcousens> sipa: I guess the benefit of this change, with API compatibility, is it doesn't hinder that path in any (except maybe making it look less attractive haha)
1512015-10-30T02:51:55  <gmaxwell> I do not see why we'd want _any_ overhead of having constant allocation indirection for data where 99.999% of the time its a fixed size. even having the extra pointer is overhead we don't need.
1522015-10-30T02:53:01  <dcousens> gmaxwell: as mentioned above, to do that (I assume you mean using fixed size CScripts), it'd be a more invasive change
1532015-10-30T02:53:12  <gmaxwell> sipa: places where many vectors with a total amount of data them that is almost always small with respect to 40 bytes.
1542015-10-30T02:53:17  <sipa> gmaxwell: vectors have 40 byte overhead on average (including the malloc overhead) per vector
1552015-10-30T02:53:57  <dcousens> gmaxwell: or by extra pointer did you mean cap?
1562015-10-30T02:54:29  <gmaxwell> dcousens: what sipa is doing now is almost a fixed size cscript, no-- only when its an exceptional cscript does it fail over to using an allocation.
1572015-10-30T02:55:22  <sipa> i'm using 28 bytes as cutoff; there are many scripts larger (nearly every scriptSig)
1582015-10-30T02:55:40  <sipa> but in places where storage matters, more than 28 is a minority
1592015-10-30T02:56:40  <sipa> i used 36 earlier, and that was still a net benefit, but a smaller one
1602015-10-30T02:56:47  <gmaxwell> ah, I did forget this was used for ScriptSig too.
1612015-10-30T02:58:17  <phantomcircuit> it's kind of amusing how we can optimize for the current set of data and have that not be a huge issue... since it will never change
1622015-10-30T02:58:48  <sipa> if we'd switch to a P3SH with a 256-bit hash, we may need to change it :)
1632015-10-30T02:59:11  <sipa> we could use this for valtype inside the script evaluator too, set to 72 bytes (as we rarely have larger script elements), or to 520 (as we don't support larger at all)
1642015-10-30T02:59:12  <phantomcircuit> heh
1652015-10-30T02:59:41  <phantomcircuit> sipa, sure but like if block_height < 350000 then vectortype = x else vectortype = y
1662015-10-30T03:00:01  <gmaxwell> obviously it needs to be autoadaptive. :P
1672015-10-30T03:00:21  <phantomcircuit> :)
1682015-10-30T03:02:01  <gmaxwell> Another obvious thing to try is to breakpoint at the validation of block X, then breakpoint at malloc/new  and catch every place it hits the heap allocator during verification of that block.
1692015-10-30T03:10:21  <dcousens> ignoring memory, I wonder if you just fixed size it to 520 bytes
1702015-10-30T03:10:47  <gmaxwell> dcousens: it can be larger than 520 bytes.
1712015-10-30T03:11:21  <phantomcircuit> gmaxwell, how?
1722015-10-30T03:11:28  <gmaxwell> 0_o
1732015-10-30T03:11:32  <sipa> CScript can be larger than 520 bytes
1742015-10-30T03:11:38  <sipa> script stack elements can;t
1752015-10-30T03:11:44  <gmaxwell> phantomcircuit: this is the full script pubkey in a transaction or a full scriptsig in a transaction.
1762015-10-30T03:12:06  <gmaxwell> scriptsigs larger than that are probably not even all that uncommon.
1772015-10-30T03:12:51  *** zooko has joined #bitcoin-core-dev
1782015-10-30T03:12:52  <phantomcircuit> ohh
1792015-10-30T03:13:12  <phantomcircuit> i thought you were just optimizing for the push data
1802015-10-30T03:48:29  <warren> gmaxwell: a safe wrapper that passes all parameters to some other program is simple like: http://pkgs.fedoraproject.org/cgit/couchdb.git/tree/couchdb.temporary.sh
1812015-10-30T03:48:46  <warren> https://fedoraproject.org/wiki/BITCOIN
1822015-10-30T03:50:03  <warren> I wrote down what I think would be good for the two different ways in which people want to use bitcoind, as a user and as a system service.  Both would need separate SELinux contexts, and a wrapper like http://pkgs.fedoraproject.org/cgit/couchdb.git/tree/couchdb.temporary.sh  to launch the system service under the different context.
1832015-10-30T04:12:03  *** evoskuil has quit IRC
1842015-10-30T04:14:19  *** zooko has quit IRC
1852015-10-30T04:23:09  *** evoskuil has joined #bitcoin-core-dev
1862015-10-30T04:43:16  *** NLNico has joined #bitcoin-core-dev
1872015-10-30T05:18:14  *** PaulCapestany has quit IRC
1882015-10-30T05:19:59  *** PaulCapestany has joined #bitcoin-core-dev
1892015-10-30T05:35:30  *** PaulCape_ has joined #bitcoin-core-dev
1902015-10-30T05:37:46  *** PaulCapestany has quit IRC
1912015-10-30T06:00:55  *** PaulCapestany has joined #bitcoin-core-dev
1922015-10-30T06:03:01  *** PaulCape_ has quit IRC
1932015-10-30T06:17:11  *** droark has quit IRC
1942015-10-30T06:26:49  *** challisto has joined #bitcoin-core-dev
1952015-10-30T06:26:56  *** PaulCape_ has joined #bitcoin-core-dev
1962015-10-30T06:29:35  *** PaulCapestany has quit IRC
1972015-10-30T07:01:30  *** PaulCape_ has quit IRC
1982015-10-30T07:01:56  *** PaulCapestany has joined #bitcoin-core-dev
1992015-10-30T07:13:13  *** danielsocials has joined #bitcoin-core-dev
2002015-10-30T07:21:39  *** Luke-Jr has quit IRC
2012015-10-30T07:24:55  *** Luke-Jr has joined #bitcoin-core-dev
2022015-10-30T07:27:21  *** PaulCape_ has joined #bitcoin-core-dev
2032015-10-30T07:29:35  *** PaulCapestany has quit IRC
2042015-10-30T07:32:10  *** danielsocials has quit IRC
2052015-10-30T07:33:46  *** danielsocials has joined #bitcoin-core-dev
2062015-10-30T07:38:41  *** danielsocials has quit IRC
2072015-10-30T07:48:43  *** Ylbam has joined #bitcoin-core-dev
2082015-10-30T08:02:27  *** danielsocials has joined #bitcoin-core-dev
2092015-10-30T08:15:16  <jonasschnelli> ping https://github.com/bitcoin/bitcoin/pull/6780
2102015-10-30T08:15:34  <jonasschnelli> should get in before it gets outdated
2112015-10-30T08:16:50  *** danielsocials has quit IRC
2122015-10-30T08:40:54  <GitHub158> [bitcoin] laanwj closed pull request #6913: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6913
2132015-10-30T08:41:56  *** pavel_ has quit IRC
2142015-10-30T08:42:14  *** paveljanik has joined #bitcoin-core-dev
2152015-10-30T08:44:42  *** BashCo_ has quit IRC
2162015-10-30T08:55:57  *** rubensayshi has joined #bitcoin-core-dev
2172015-10-30T09:03:42  *** BashCo has joined #bitcoin-core-dev
2182015-10-30T09:50:20  *** PRab_ has joined #bitcoin-core-dev
2192015-10-30T09:51:01  *** PRab has quit IRC
2202015-10-30T09:51:02  * wumpus has an old crappy laptop with windows now, let the carnage begin...
2212015-10-30T09:51:06  *** PRab_ is now known as PRab
2222015-10-30T09:56:43  <wumpus> I'm sure it comes with rootkits and malware pre-installed to simulate a true, out in the wild windows environment
2232015-10-30T09:57:45  <phantomcircuit> :P
2242015-10-30T09:59:57  *** CodeShark has quit IRC
2252015-10-30T10:15:59  <dcousens> wumpus: what would you say an average time for re-index is?
2262015-10-30T10:16:07  <dcousens> (ballpark, obv. very dependent on hardware)
2272015-10-30T10:19:16  *** dcousens has quit IRC
2282015-10-30T10:31:54  *** NLNico has quit IRC
2292015-10-30T11:27:34  *** NLNico has joined #bitcoin-core-dev
2302015-10-30T11:28:43  *** NLNico has quit IRC
2312015-10-30T11:45:56  *** tulip has joined #bitcoin-core-dev
2322015-10-30T12:04:45  *** evoskuil has quit IRC
2332015-10-30T13:12:31  <morcos> sipa: i've been trying to track down some odd slowness in ConnectBlock and I found something I think interesting
2342015-10-30T13:13:28  <morcos> When we do UpdateCoins and we need to add the outputs for the new coins we are now creating.  We call ModifyCoins(new tx hash) which will end up doing a database read.
2352015-10-30T13:14:00  <morcos> so even if our cache is completely up to date, we have to read the database anyway
2362015-10-30T13:14:27  <morcos> There is already an existince check for BIP30 earlier
2372015-10-30T13:14:44  <morcos> I'm wondering if we should have some sort of negative cache too
2382015-10-30T13:16:39  <morcos> Also do we need to persist the BIP30 check?  Is it still possible to violate that or now that the old duplicate coinbases are sufficiently buried, its not necessary
2392015-10-30T13:16:54  <morcos> Anwyay, I have to go afk.  I'll look into it more later.
2402015-10-30T13:59:07  *** ParadoxSpiral has joined #bitcoin-core-dev
2412015-10-30T14:00:25  *** testing-tape has joined #bitcoin-core-dev
2422015-10-30T14:03:27  *** testing-tape has quit IRC
2432015-10-30T14:08:06  *** Guest75176 has quit IRC
2442015-10-30T14:08:28  *** pigeons has joined #bitcoin-core-dev
2452015-10-30T14:08:52  *** pigeons is now known as Guest35844
2462015-10-30T14:31:20  <morcos> sipa: still afk, but yep, thats a cause of SIGNIFICANT slowness in ConnectBlock.  I think we'd have to change/remove the BIP30 check to really eliminate it, which might be the harder part, bc I think adjusting ModifyCoins to know its a new tx and not lookup seems easy and safe?
2472015-10-30T14:41:49  *** goregrind has quit IRC
2482015-10-30T14:45:10  *** goregrind has joined #bitcoin-core-dev
2492015-10-30T14:55:04  *** [b__b] has quit IRC
2502015-10-30T14:55:25  *** [b__b] has joined #bitcoin-core-dev
2512015-10-30T15:04:17  <wumpus> cfields: what would be the best way to include the windows determinism postprocessing in the gitian build (in https://github.com/bitcoin/bitcoin/pull/6900) - call it from the descriptor or 'make'?
2522015-10-30T15:05:19  <cfields> wumpus: since that's really only useful for gitian (and a specific version of binutils), i'd say in the descriptor is fine
2532015-10-30T15:05:29  <wumpus> yeah, thought as much
2542015-10-30T15:05:39  *** zooko has joined #bitcoin-core-dev
2552015-10-30T15:06:00  <wumpus> do wonder how to make it output the linker map though
2562015-10-30T15:06:04  <cfields> wumpus: i haven't taken a good look yet, but i'm surprised that you can't find the necessary offsets with objdump/nm/etc. I suppose you tried all those first?
2572015-10-30T15:06:27  <wumpus> objdump and friends all report virtual addresses
2582015-10-30T15:06:42  *** zooko` has joined #bitcoin-core-dev
2592015-10-30T15:06:48  *** zooko has quit IRC
2602015-10-30T15:06:52  *** zooko` has quit IRC
2612015-10-30T15:07:10  <cfields> mm, ok
2622015-10-30T15:07:38  *** zooko has joined #bitcoin-core-dev
2632015-10-30T15:07:51  <wumpus> and within a section they're no help at all, for finding the offset, it's not like there's a pointer to this variable (the other option may be to parse DWARF information, but hm...)
2642015-10-30T15:08:32  <wumpus> that may not even work, I don't think gdb has an offset to it, it reports it as GS_ExceptionPointers+8/16
2652015-10-30T15:08:33  <cfields> wumpus: another fleeting thought.. in the mailing-list link you posted, some of my binutils changes are listed in the same changelog...
2662015-10-30T15:08:43  <cfields> you completely sure that this is the only hack necessary?
2672015-10-30T15:09:27  <cfields> iirc it was the one that removed timestamps from windres
2682015-10-30T15:09:42  <wumpus> well I found no other differences between multiple builds
2692015-10-30T15:09:51  <wumpus> remember, we still use faketime
2702015-10-30T15:10:20  <cfields> ah, right
2712015-10-30T15:10:56  <wumpus> the tor folks build binutils themselves
2722015-10-30T15:11:36  <wumpus> but anyhow, the current script works, so I'll just integrate that for now
2732015-10-30T15:11:46  <cfields> sounds good
2742015-10-30T15:13:22  <cfields> wumpus: one last quick thought, another option might be using LD_PRELOAD to wrap malloc (or whatever is leaving unsanitized data)
2752015-10-30T15:17:22  <wumpus> hehe.
2762015-10-30T15:18:01  <wumpus> wonder if  mallopt(M_PERTURB,...) can be set from the environment
2772015-10-30T15:18:45  <wumpus> I think so! it should do exactly this
2782015-10-30T15:19:25  <cfields> hah! neat. I was just doing a quick malloc->calloc wrapper. That looks smarter :)
2792015-10-30T15:22:10  <wumpus>  'export MALLOC_PERTURB_=23'  seems to work
2802015-10-30T15:22:21  <wumpus> (at least outside of gitian, let's try inside)
2812015-10-30T15:22:51  <wumpus> that was an awesome idea
2822015-10-30T15:23:31  * wumpus resents having written a PE parser :-)
2832015-10-30T15:24:03  <cfields> should probably wrap the linker in a script and export it there, rather than globally. It's hard to imagine anything build-side dependent on that, but i'd hate to see something like the debian-openssl thing crop up again :)
2842015-10-30T15:24:52  <wumpus> also reduces the performance hit
2852015-10-30T15:25:28  <cfields> yep
2862015-10-30T15:26:37  <cfields> wumpus: nice find on MALLOC_PERTURB_. that makes this very simple :)
2872015-10-30T15:27:14  <wumpus> but the linker is invoked as 'g++' isn't it?
2882015-10-30T15:27:27  <cfields> g++ invokes ld
2892015-10-30T15:27:31  *** zooko has quit IRC
2902015-10-30T15:28:18  <wumpus> oh yes it's even worse for mingw, it invokes collect2, which is a wrapper for ld. I wonder if it's possible to interpose somehwere there
2912015-10-30T15:28:49  <wumpus> (well obviously, by replacing the executable... but I wonder if it uses path somewhree)
2922015-10-30T15:29:58  <cfields> uhmm, you can specify your own linker i believe
2932015-10-30T15:30:57  * Luke-Jr imagines each of these is used for a reason
2942015-10-30T15:31:27  <wumpus> Luke-Jr: 'collect2' is MS's linker, which for mingw is emulated by ld
2952015-10-30T15:32:27  <Luke-Jr> oh, it's just emulating it?
2962015-10-30T15:33:01  <wumpus> yes - it could really use MS's linker, if you have it available
2972015-10-30T15:33:10  <Luke-Jr> interesting
2982015-10-30T15:37:12  <cfields> wumpus: yup, looks like it checks for 'real-ld'
2992015-10-30T15:38:31  <cfields> and ${prefix}-real-ld
3002015-10-30T15:40:22  <wumpus> I haven't been able to trick it yet into running a different linker outside the $prefix (created all sorts of variants of collect2 and ld in my PATH)
3012015-10-30T15:41:11  <wumpus> one remaning option would be to override gcc's specs, but bleh
3022015-10-30T15:46:52  <wumpus> seems it only looks for 'linker' outside its hardcoded path  /usr/lib/gcc/x86_64-w64-mingw32/4.8 if it cannot find it there
3032015-10-30T15:47:07  *** zooko has joined #bitcoin-core-dev
3042015-10-30T15:48:10  <wumpus> what works is: 'gcc -dumpspecs > specsfile' 'sed -i s/collect2/mycollect2/g specsfile' 'gcc -specs=specsfile'
3052015-10-30T15:48:28  <wumpus> then put mycollect2 somewhere in the path...
3062015-10-30T15:48:36  <wumpus> it's kind of stupid though
3072015-10-30T15:49:06  <wumpus> hmm overrding GCC_EXEC_PREFIX may work too
3082015-10-30T15:49:22  <wumpus> but that means copying the other files
3092015-10-30T15:49:49  <wumpus> oh there is COMPILER_PATH  too, where it will look for subprograms if it cannot find in EXEC_PREFIX
3102015-10-30T15:49:58  <wumpus> great, this may yet be solvable
3112015-10-30T15:50:40  <cfields> wumpus: heh, was just testing that one
3122015-10-30T15:50:42  <cfields> no luck
3132015-10-30T15:51:20  * wumpus considers just defining the malloc perturb globally and be done with it...
3142015-10-30T15:51:21  <cfields> wait, that worked :)
3152015-10-30T15:51:38  <cfields> export COMPILER_PATH=`pwd`
3162015-10-30T15:51:44  <cfields> then wrap ./collect2
3172015-10-30T15:52:06  <wumpus> according to the docs it only looks there if it cannot find it in the EXEC_PREFIX, but will try
3182015-10-30T15:52:38  <wumpus> yep, seems to work!
3192015-10-30T15:58:42  <wumpus> cfields: something like this? https://gist.github.com/laanwj/10b0981d9d75510a124f
3202015-10-30T15:58:50  <cfields> wumpus: http://pastebin.com/W350ZGK5
3212015-10-30T15:58:51  <cfields> hah!
3222015-10-30T15:59:34  <cfields> looks great
3232015-10-30T15:59:48  <wumpus> ah yes, $CC -print-prog-name=collect is nice
3242015-10-30T16:03:02  <wumpus> updated to use that https://gist.github.com/laanwj/10b0981d9d75510a124f  (now request the path of the collect2 tool while creating the wrapper script instead of hardcoding it in the descriptor)
3252015-10-30T16:04:36  <cfields> wumpus: need to unset COMPILER_PATH or you get an infinite loop
3262015-10-30T16:04:53  <wumpus> COMPILER_PATH is not set at that point yet :)
3272015-10-30T16:05:05  <wumpus> it's set in the per-host build loop later on
3282015-10-30T16:06:09  <cfields> oh heh, you're finding the path outside of the script. nm :)
3292015-10-30T16:06:11  *** zooko has quit IRC
3302015-10-30T16:06:30  <wumpus> yes I look up the path upfront
3312015-10-30T16:06:57  <cfields> well that's nice and simple if it works
3322015-10-30T16:09:13  <cfields> wumpus: just to be on the paranoid side, should probably compare before/after to make sure that only the expected bytes changed.
3332015-10-30T16:10:49  <wumpus> well if more bytes change then we should definitely skip trusty, agreed
3342015-10-30T16:12:26  <cfields> (i just mean eyeballing it once, ofc. not suggesting scripting that part)
3352015-10-30T16:16:09  <wumpus> leaking heap data into output is really ugly, can be e.g. a privacy issue, I'm happy it was solved two years ago ;)
3362015-10-30T16:16:37  <wumpus> MS office had some legendary fuckups in this regard
3372015-10-30T16:17:57  <cfields> wumpus: assuming mingw is self-hosted,  we can check the bins for leaks and see if the dev had anything interesting in mem :)
3382015-10-30T16:19:41  <wumpus> hehe
3392015-10-30T16:20:20  <wumpus> well in this case it seems it leaks some internal address; could be used to bypass ASLR, if it was a longer running process
3402015-10-30T16:20:23  *** dixson3 has joined #bitcoin-core-dev
3412015-10-30T16:30:09  <sipa> morcos: we should get rid of the bip30 check; it's superseeded by bip34
3422015-10-30T16:31:34  <cfields> wumpus: for the qt fix, can that ifdef just be c/p to the header as well?
3432015-10-30T16:33:26  <wumpus> cfields: yes I suppose the same principle can be used to use either PIDLIST_ABSOLUTE or ITEMIDLIST
3442015-10-30T16:34:04  *** Guyver2 has joined #bitcoin-core-dev
3452015-10-30T16:34:17  <wumpus> (just copy pasting won't work though because the typedef won't help_
3462015-10-30T16:34:34  *** dgenr8 has quit IRC
3472015-10-30T16:35:28  <cfields> wumpus: ok. mind PRing that separate? or I can? That needs fixing regardless of when we switch gitian
3482015-10-30T16:35:52  <wumpus> no problem if you do :)
3492015-10-30T16:35:53  <cfields> i'm happy to do it, not trying to con you into that one :)
3502015-10-30T16:35:56  <cfields> sure, will do
3512015-10-30T16:36:27  *** dgenr8 has joined #bitcoin-core-dev
3522015-10-30T16:36:41  * wumpus just crashed a windows PC in the middle of initial sync, came back fine after reboot...
3532015-10-30T16:37:20  *** dixson3 has quit IRC
3542015-10-30T16:37:46  *** dixson3 has joined #bitcoin-core-dev
3552015-10-30T16:38:44  <wumpus> "NotMyFault: Use this executable and driver to crash your system in several different ways" lovely
3562015-10-30T16:42:45  *** BashCo has quit IRC
3572015-10-30T16:44:56  *** challisto has quit IRC
3582015-10-30T16:48:52  <wumpus> the only thing hard to reproduce with a laptop would be a direct power failure
3592015-10-30T16:51:27  <sipa> remove the battery :)
3602015-10-30T16:53:05  <wumpus> I'm not sure that's possible with this one without disassembling it
3612015-10-30T16:54:02  <wumpus> and not sure bitcoin core should be resilient to motherboard-bends-while-application-is-running crashes :-)
3622015-10-30T17:04:55  *** goregrind has quit IRC
3632015-10-30T17:05:13  *** BashCo has joined #bitcoin-core-dev
3642015-10-30T17:10:33  *** zooko has joined #bitcoin-core-dev
3652015-10-30T17:16:39  *** Thireus2 has quit IRC
3662015-10-30T17:30:05  <wumpus> cfields: it worked! windows trusty build is now deterministic http://s7.postimg.org/bhrf0eui3/Naamloos.png
3672015-10-30T17:30:48  <cfields> woohoo :)
3682015-10-30T17:30:52  <cfields> but... wtf am i looking at?
3692015-10-30T17:31:23  <mcelrath> What's that usha256?
3702015-10-30T17:31:41  <wumpus> my hex-as-unicode-blockelements hack
3712015-10-30T17:32:34  <wumpus> (somewhat easier to compare patterns than letters/numbers)
3722015-10-30T17:33:25  <cfields> ah, neat
3732015-10-30T17:34:13  <wumpus> "Fout bij openen blokkendatabase" ah, this time I crashed windows and the block database got corrupted
3742015-10-30T17:35:53  <wumpus> usha256: https://gist.github.com/laanwj/3a142c48046ef862bf91
3752015-10-30T17:36:49  <helo> corrupted your error messages too! ;)
3762015-10-30T17:37:44  <zooko> wtf-8
3772015-10-30T17:38:41  <wumpus> lol zooko
3782015-10-30T17:38:48  <zooko> Seriously, what is this for ?
3792015-10-30T17:40:21  <zooko> Wow.
3802015-10-30T17:40:42  <zooko> Intriguing.
3812015-10-30T17:41:13  <mcelrath> I can't decide whether that's easier to differentiate or not. Better let the computer decide: sha256sum a b | awk '{print $1}' | uniq
3822015-10-30T17:43:13  <mcelrath> Also diff -q
3832015-10-30T17:48:43  *** zooko has quit IRC
3842015-10-30T17:52:06  <wumpus> mcelrath: what I don't like about that is that in the succesful case it provides no output, so I can never be sure whether it crashed (and hence produced no output) or succeeded
3852015-10-30T17:52:29  <mcelrath> What provides no output?
3862015-10-30T17:52:36  <wumpus> diff, when files match
3872015-10-30T17:53:06  <sipa> wumpus: how many such block elements are there?
3882015-10-30T17:54:21  <wumpus> sipa: I used the 16 2x2 ones to get 16 bits - but there are also e.g. 1/8th horizontal/vertical full ones, see https://en.wikipedia.org/wiki/Block_Elements
3892015-10-30T17:54:28  <wumpus> to get 4 bits*
3902015-10-30T17:55:24  <wumpus> and if you combine with reverse-video you can get even more, e.g. "right one eights block"
3912015-10-30T17:56:25  <wumpus> most console applications that use thsese, use them for progress bars/gauges etc
3922015-10-30T17:57:31  *** Thireus has joined #bitcoin-core-dev
3932015-10-30T17:59:03  <wumpus> you could also do a kind of Chernof faces visualization using emojis, esp. on mac consoles which render those silly things in color
3942015-10-30T18:07:45  *** PaulCape_ has quit IRC
3952015-10-30T18:08:14  *** PaulCapestany has joined #bitcoin-core-dev
3962015-10-30T18:12:48  <wumpus> "2015-10-30 17:30:05 Corruption: error in middle of record" that's the error with the windows chainstate db after crash
3972015-10-30T18:14:10  *** jl2012_ has quit IRC
3982015-10-30T18:14:25  *** jl2012 has joined #bitcoin-core-dev
3992015-10-30T18:15:41  <wumpus> this happens while reading the log, so it is a partial record error
4002015-10-30T18:15:53  <wumpus> partial write*
4012015-10-30T18:22:06  *** rubensayshi has quit IRC
4022015-10-30T18:37:08  <wumpus> https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/env_win.cc#L587  shouldn't the FlushFuleBuffers and FlushViewOfFile be the other way around?
4032015-10-30T18:37:22  <wumpus> the former syncs changes in the file to disk, the latter changes in the memory map to the file
4042015-10-30T18:37:34  <wumpus> I may be misunderstanding something of course...
4052015-10-30T18:39:32  <sipa> wumpus: that code may well be wrong
4062015-10-30T18:40:08  * wumpus wants to implement a WritableFile with just the win32 base file API and none of this fancy mapping stuff
4072015-10-30T18:40:50  <wumpus> like PosixWritableFile but for windows... oh wait, maybe I can even just copy PosixWritableFile and rely on mingw to do the rest for me
4082015-10-30T18:56:37  *** CodeShark has joined #bitcoin-core-dev
4092015-10-30T19:18:25  *** davec has quit IRC
4102015-10-30T19:19:27  *** davec has joined #bitcoin-core-dev
4112015-10-30T19:24:39  *** zooko has joined #bitcoin-core-dev
4122015-10-30T19:25:34  <GitHub92> [bitcoin] sipa opened pull request #6916: Remove BIP30 enforcement, as it is impossible to trigger since BIP34 (master...nobip30) https://github.com/bitcoin/bitcoin/pull/6916
4132015-10-30T19:26:05  <sipa> wumpus: we should do that, I think
4142015-10-30T19:55:41  <GitHub118> [bitcoin] sipa closed pull request #6916: Remove BIP30 enforcement, as it is impossible to trigger since BIP34 (master...nobip30) https://github.com/bitcoin/bitcoin/pull/6916
4152015-10-30T19:56:31  <gmaxwell> :)
4162015-10-30T19:58:15  <sipa> gmaxwell: i'm trying to think whether enforcing it for all blocks (except those two) that don't have BIP34 is safe
4172015-10-30T19:59:01  <sipa> as BIP34 is only sufficient when there are no actual duplicate coinbases
4182015-10-30T19:59:04  <sipa> but there are
4192015-10-30T19:59:33  <gmaxwell> sipa: if we ship a wad of the 32 least significant bits of the block hashes up to the height bip 34 activates; and then require that, that would be sufficient.
4202015-10-30T20:01:49  <sipa> that's not very different from a checkpoint...
4212015-10-30T20:02:36  <morcos> sipa: ok so i have an even stranger one for you
4222015-10-30T20:03:19  <morcos> when TestBlockValidity ends, between then and when control returns to CreateNewBlock, consistently takes 10-15 ms
4232015-10-30T20:03:38  <morcos> if you recently flushed your cache, i think this number can be 20-25 ms
4242015-10-30T20:04:00  <gmaxwell> sipa: checkpoints don't fix this, since the corruption can slip in between them.
4252015-10-30T20:04:06  <gmaxwell> is it spending 10-15ms in free? :)
4262015-10-30T20:04:10  <morcos> but i have no idea what if anything should be happening then, the CCoinsViewCache destructor runs, but what else
4272015-10-30T20:04:35  <morcos> gmaxwell: is that possible?
4282015-10-30T20:04:58  <sipa> and that destructor doesn't do anything...
4292015-10-30T20:05:19  <sipa> except deleting cacheCoins
4302015-10-30T20:05:37  <sipa> morcos: you could try to do a cacheCoins.clean() in the CCoinsViewCache destructor, and time that?
4312015-10-30T20:05:54  <morcos> ok
4322015-10-30T20:06:13  <sipa> or add a .Clean() method that dispatches to cacheCoins.clear()
4332015-10-30T20:06:14  <gmaxwell> morcos: it's possible e.g. freeing a kazillion and one tiny objects. I'd done some profiling in the past that suggested to me we were losing a lot of time in the allocator.
4342015-10-30T20:06:54  <sipa> #6914 should help there too
4352015-10-30T20:12:09  <morcos> yep, clearing the cache takes 10+ ms
4362015-10-30T20:13:01  <sipa> try with #6914 :)
4372015-10-30T20:13:23  <sipa> actually, it can at most give a 2x speedup
4382015-10-30T20:13:29  <morcos> now the confusing thing is it sometimes takes twice as long if pcoinsTip (the base cache, had recently been semi-flushed)
4392015-10-30T20:13:50  <morcos> i wonder if thats a memory thrashing thing,or it has something to do with the flags on those coins
4402015-10-30T20:24:44  <wumpus> replaced the leveldb win32 writable file, let's see how it goes https://github.com/laanwj/bitcoin/commit/86050deaabe7ca0a19db1c5a65e3174c9cb14511
4412015-10-30T20:25:38  <gmaxwell> \O/ always good to see a fix that removes a lot of code.
4422015-10-30T20:26:48  <wumpus> it is! it's extremely simple now, the only thing I'm not sure about is Sync() versus Flush(), they both do a sync to disk now
4432015-10-30T20:27:05  <wumpus> that's probably overkill but it cannot hurt either
4442015-10-30T20:31:52  <wumpus> going to let it sync a bit and then cause another IRQL_NOT_LESS error and see if it avoids ending up in broken state
4452015-10-30T20:32:17  <sipa> great!
4462015-10-30T20:32:41  <sipa> we don't need particularly good write performance, i guess
4472015-10-30T20:32:51  <sipa> as we mostly do batch writes
4482015-10-30T20:33:24  <morcos> gmaxwell: so the easy simple thing re: BIP 30 would to just be to disable it for blocks after a certain height right?
4492015-10-30T20:34:01  *** evoskuil has joined #bitcoin-core-dev
4502015-10-30T20:34:23  <morcos> eh, never mind
4512015-10-30T20:35:05  <wumpus> agreed
4522015-10-30T20:35:19  <morcos> we could disable it for blocks that have a certain block as their ancestor
4532015-10-30T20:35:19  *** zooko has quit IRC
4542015-10-30T20:35:42  <morcos> such block being after BIP 34 and the pre existing duplicate coinbases diverged and were buried
4552015-10-30T20:35:53  <sipa> morcos: fundamentally, the bip30 check can be disabled for any transaction that depends on a bip34 coinbase
4562015-10-30T20:36:30  <gmaxwell> sipa: the existing BIP30 violations overwrote without proliferation.
4572015-10-30T20:36:34  <sipa> but there have been duplicated coinbases in the past, so there can be transactions that spend such a duplicate, which can become duplicated as well
4582015-10-30T20:36:39  <gmaxwell> meaning there can be no duplicate children.
4592015-10-30T20:36:49  <sipa> gmaxwell: if you know the chain
4602015-10-30T20:36:57  <sipa> you can be fed an alternative chain in which that's not the case
4612015-10-30T20:37:17  <sipa> no?
4622015-10-30T20:37:18  <morcos> right so if we only disable if you have an ancestor block which sufficietnly buries those, then you're good on the main chain
4632015-10-30T20:37:23  <gmaxwell> Right but we could enforce BIP30 for all chains until the point where BIP34 activates.
4642015-10-30T20:37:28  <morcos> andif you're fed an alternate chain you won't disable the check
4652015-10-30T20:37:35  <gmaxwell> oh I see if it proliferates its not enough.
4662015-10-30T20:37:43  <sipa> gmaxwell: BIP34 or not is not enough
4672015-10-30T20:38:07  <sipa> even post BIP34, a transaction that spends a previous duplicate coinbase is possible (in an alternative chain)
4682015-10-30T20:38:10  <sipa> so a checkpoint would work
4692015-10-30T20:38:20  <gmaxwell> yea, so we could disable BIP30 when BIP34 acitivates if and only if the the ancestor chain is the real one.
4702015-10-30T20:38:34  <morcos> yes, thats what i was trying to say
4712015-10-30T20:39:03  <morcos> thats the best kind of checkpoint it doesn't require you to actually be on the real chain, it just saves you some work if you are
4722015-10-30T20:39:22  <sipa> indeed
4732015-10-30T20:41:05  <gmaxwell> the constant ancestor tests are slightly annoying but much better than running bip30 everywhere.
4742015-10-30T20:44:57  <morcos> so erasing these caches is absurdly expensive.  sometimes its 40ms+
4752015-10-30T20:45:13  <morcos> but why is it higher shortly after the underlying cache has been erased, that i don't understand
4762015-10-30T20:45:13  <sipa> how many entries?
4772015-10-30T20:45:30  <morcos> the 10ms case was on the order of 5000 entries
4782015-10-30T20:46:00  <sipa> maybe we need a special throw-away-cache which uses an append-only pooled allocation
4792015-10-30T20:46:29  <sipa> ... though again, a per-txout cache would be better for that
4802015-10-30T20:46:36  <morcos> but the number of entries is irrelevant, its the number of txins in those entries i guess right?
4812015-10-30T20:46:55  <sipa> txouts
4822015-10-30T20:47:08  <morcos> yeah,
4832015-10-30T20:58:03  <gmaxwell> what are the heap allocated objects in it? the entries themselves. scripts inside them (largely fixed by pieters alt-vector)? what else?
4842015-10-30T20:58:58  *** jtimon has quit IRC
4852015-10-30T20:59:40  <sipa> gmaxwell: one allocation in the hashmap per txid; for each txid there is one allocation for a vector of txouts; for each txout there is one allocation for the script (partially fixed by prevector)
4862015-10-30T21:00:30  <gmaxwell> interesting that we always load all the txouts but are still taking per-txout allocation overhead.
4872015-10-30T21:02:26  <gmaxwell> so in any case if transactions are tending to gather from many distinct txids, then you potentially have a huge blowup in allocations; it's e.g. two heap allocations for each txout, ... so if you gather from a 100 txout transaction there is 200 heap allocations for each txin that does that.
4882015-10-30T21:03:21  <gmaxwell> (well, 201) and pieter's vector thingy would reduce that to 101 heap allocations.
4892015-10-30T21:04:37  <gmaxwell> They're probably destroyed in the same order they're created, which is probably a fairly expensive order for malloc too.
4902015-10-30T21:05:57  <gmaxwell> as an aside, after thinking about the heap behavior; I would guess a non-trivial amount of the 'remove on replace' memory savings is ficticious.
4912015-10-30T21:06:05  <gmaxwell> Due to fragmentation.
4922015-10-30T21:08:21  <morcos> "remove on replace"?
4932015-10-30T21:09:30  <gmaxwell> attempts to remove cache entries for transactions evicted from the mempool.  Remove on reject is probably effective, since it's removing things it just added.
4942015-10-30T21:10:02  <morcos> ah yes, well the whole thing i'm trying to accomplish here is making that unnecessary
4952015-10-30T21:10:21  <morcos> if flushing the cache more often is doable (b/c you don't flush out the hot entries you're about to use)
4962015-10-30T21:10:28  <gmaxwell> sipa: so an append only arena that we can dump as a single operation would ... be a thing that could be done, _but_ it leaves us in a world were the only management thing we can do is flush() which would be unfortunate.
4972015-10-30T21:10:47  <morcos> then instead of worrying about what you're adding, just check far more often that you're not too big.  most of the time you wont' have new stuff to write to the DB anyway
4982015-10-30T21:11:03  <sipa> gmaxwell: it would be awesome for blocks, not for the mempool
4992015-10-30T21:11:30  <morcos> but for some reason, the cache clearing of the child cache is slower if you've done one of these controlled flushes on the parent cache (or a real full flush) right before you create the child cache
5002015-10-30T21:11:35  <morcos> that makes no sense to me
5012015-10-30T21:13:03  <gmaxwell> sipa: hm. even for blocks, say you process block 100 and read in transactions X,Y,Z. then you flush. Then block 101 needs transactions Y and Z again. I'm not sure how often that happens, but I would be that its way way more common than chance.
5022015-10-30T21:15:24  *** ParadoxSpiral has quit IRC
5032015-10-30T21:15:59  <sipa> gmaxwell: i mean for the scripts in a block
5042015-10-30T21:16:07  <sipa> gmaxwell: the scriptSigs
5052015-10-30T21:16:20  <sipa> not talking about UTXO set or mempool
5062015-10-30T21:20:12  <gmaxwell> ah, indeed. you even have a nice figure on the size of the arena to allocate: just allocate the size of the block, and then it never needs to be realloced.
5072015-10-30T21:21:18  <sipa> even for a single transaction it's probably worth it
5082015-10-30T21:21:21  <sipa> as they're immutable
5092015-10-30T21:21:48  <sipa> hell, a whole transaction could be a single malloced block
5102015-10-30T21:22:00  <sipa> with some internal offsets/pointers in it
5112015-10-30T21:24:10  <gmaxwell> sipa: thats really what I'd prefer to see. I don't think having one allocation per transaction is unreasonable; but these nested allocations are goofy.
5122015-10-30T21:27:41  <gmaxwell> sipa: similar to that: AFAIK utxo cache entries cannot change in size.
5132015-10-30T21:28:10  <sipa> individual txouts no; the per-tx ones can
5142015-10-30T21:30:09  <gmaxwell> sipa: why? the only operation is that they can have entries spent, but that can be a flag that doesn't require resizing the object.
5152015-10-30T21:31:45  <gmaxwell> (and the operation flushes the dirty part could resize the object when it does so, I suppose, as it will already have exclusive access to it, no?)
5162015-10-30T21:32:45  <sipa> gmaxwell: you want to remove the allocated scripts for spent entries, no?
5172015-10-30T21:33:09  <sipa> gmaxwell: otherwise a full in-memory sync from scratch needs memory for every UTXO every created, rather than a working set
5182015-10-30T21:33:18  *** randy-waterhouse has joined #bitcoin-core-dev
5192015-10-30T21:34:52  <gmaxwell> sipa: not quite, if it's spent completely you just drop the object. So how much partially spent would there be?  Thats also an unusual case. In any case, it could trigger a rebuild at any time you have exclusive acccess. basically batching the malloc overhead.
5202015-10-30T21:35:28  <sipa> you always have exclusive access
5212015-10-30T21:36:02  <gmaxwell> e.g. when you spend an entry you set a dity flag and increment a wasted bytes counter. Then if the wastes bytes are over X when you spend from it, you rebuild in place and realloc.
5222015-10-30T21:36:06  <wumpus> partial spent (of transaction outputs) are quite common
5232015-10-30T21:36:26  <wumpus> there are lots of transactions with many outputs
5242015-10-30T21:36:31  <gmaxwell> And if it's spent completely, you just free it.
5252015-10-30T21:37:44  <gmaxwell> but say a block spends the 100 outputs in order in a block, it makes no sense to resize the object 100 times and then free it.
5262015-10-30T21:38:05  <sipa> vectors never reallocate on shrink
5272015-10-30T21:38:14  <sipa> only on grow
5282015-10-30T21:39:43  *** evoskuil has quit IRC
5292015-10-30T21:39:52  <gmaxwell> sipa: With a vector we instead have 201 mallocs and 201 frees in that case. (and probably a few reallocs along the way)
5302015-10-30T21:40:28  *** zooko has joined #bitcoin-core-dev
5312015-10-30T21:41:02  *** paveljanik has quit IRC
5322015-10-30T21:41:57  <gmaxwell> where as if the cache entry is a single fixed size object, which never grows, and shrinks only in a deferred way when the amount of memory saved by shrinking would be non-trivial, it's 1 malloc, 1 free, and a few reallocs for shrinking along the way... and always in chunks large enough where its meaningful (e.g. where fragmentation doesn't prevent the freed memory from being used anyways).
5332015-10-30T21:43:29  <GitHub21> [bitcoin] laanwj opened pull request #6917: leveldb: Win32WritableFile without memory mapping (master...2015_10_leveldb_win_nomap) https://github.com/bitcoin/bitcoin/pull/6917
5342015-10-30T21:44:19  <wumpus> seems to have worked...
5352015-10-30T21:44:42  <gmaxwell> \O/
5362015-10-30T21:46:17  <gmaxwell> too bad the chainstate obfscuation is a bit invasive; it would be nice to have a 0.11.2 that fixes all known corruption issues for windows users.
5372015-10-30T21:47:41  <sipa> i think it's backportable
5382015-10-30T21:48:05  <wumpus> it's not too bad
5392015-10-30T21:49:18  <gmaxwell> (it wasn't that interesting to consider when the leveldb ones remained)
5402015-10-30T21:52:41  <gmaxwell> warren: if you want to do a bounty, instead of fixing windows (which wumpus just did, and we should send him flowers)-- perhaps the bounty we should ask for is this:  A NBD server that sakes an initial disk image file (compatible with loopback mount) and logs all writes to it, instead of actually writing them, and then applies the writes as a patch in memory (doesn't matter if uses lots of ram).
5412015-10-30T21:52:48  <gmaxwell>   Then it lets you restart it with a image + old log + N and it will truncate the last N writes off the old log.  This would let you construct a test harness where we start a node against an existing chain, sync one block, and shut down. Then attempt to restart with one write less. and so on, for every single write back to the first one, and make sure the node can restart at every point.
5422015-10-30T21:54:57  *** Guyver2 has quit IRC
5432015-10-30T21:58:00  <wumpus> yes, that could be useful
5442015-10-30T22:03:26  <gmaxwell> without that kind of test harness I dunno how _anyone_ manages to write database software that actually works (well: answer, according to that usenix paper, they mostly just fail. :) )
5452015-10-30T22:04:24  <warren> gmaxwell: you might have convinced me that I shouldn't be spending personal income on bounties anymore, I'd be happy to specify it though =)
5462015-10-30T22:04:52  <warren> kanzure: ping
5472015-10-30T22:05:04  <kanzure> pong
5482015-10-30T22:05:34  <kanzure> for which thing have i been summoned?
5492015-10-30T22:07:12  <wumpus> there are a few debugging and reverse engineering tools that can rewind execution in memory, but I know of no file systems that can rewind/replay operations explicitly in fine granularity. You'd say that with journalling file systems it should be possible by truncating/rewinding the journal, but never been able to find reports of anyone doing that...
5502015-10-30T22:07:23  <gmaxwell> kanzure: ever seen people who own both a parrot and a dog, ... and the parrot will sometimes call the dog just to see it come?
5512015-10-30T22:07:39  <kanzure> there was a linux tool for replay but requires kernel module, does replay and rewind of execution state, very bizarre alien thing
5522015-10-30T22:07:59  <gmaxwell> wumpus: I wasted a bunch of time searching and could find nothing.  But it seems to me that this is probably only a few days work on top of a simplistic NBD server.
5532015-10-30T22:08:00  <kanzure> http://velvetpulse.com/2012/11/27/scribe-the-deterministic-transparent-record-replay-engine/
5542015-10-30T22:08:04  <kanzure> and maybe http://rr-project.org/ but more the last link
5552015-10-30T22:08:20  <wumpus> yes, for execution state for example pandare can do that
5562015-10-30T22:08:22  <kanzure> this could possibly be extended to windows vm things but w/e
5572015-10-30T22:08:37  <gmaxwell> RR is awesome, and can do this for execution state, but we need it for disk state seperate from execution.
5582015-10-30T22:08:41  <kanzure> can't find pandare, please link?
5592015-10-30T22:09:08  <wumpus> but that's not really what we want here - we want to restart the application at every point of disk writing
5602015-10-30T22:09:25  <wumpus> kanzure: https://github.com/moyix/panda
5612015-10-30T22:09:34  <gmaxwell> I expect most of the work related to my NBD suggestion will be discovering that all filesystems are broken, and being unable to get results against bitcoin until the file systems are fixed. :(
5622015-10-30T22:09:50  <kanzure> yeah probably just write your own fusefs plugin or something
5632015-10-30T22:10:06  <kanzure> and ramfs with snapshotting or something
5642015-10-30T22:10:10  * kanzure handwaves
5652015-10-30T22:10:27  <gmaxwell> NBD is bascially the perfect interface for this. It is a remote block device whos primitives are "read this block" and "write this block"
5662015-10-30T22:11:40  <kanzure> "network block device"
5672015-10-30T22:11:46  <kanzure> i hope this isn't samba
5682015-10-30T22:11:52  <gmaxwell> so the code becomes  if read  check the log replay hashtable, no hits? go to the read only backing device. if hit return that.  if write, append log, and add to the hashtable.   On restart, replay old log into hashtable up to the cutoff point.
5692015-10-30T22:12:35  <wumpus> samba is not a network block device, but a network filesystem :)
5702015-10-30T22:12:45  <kanzure> (samba is many things)
5712015-10-30T22:14:51  <gmaxwell> the test harness is, mount the thing, sync a block. shut down. unmount. restart daemon with -1, mount, start daemon sync next block and record success, unmount, go back to the original oldlog, start with -2 ... and so on. for each offset. If there are a million writes made while syncing that first block, you restart a million times.   Once that starts failing, you change the behavior so that it
5722015-10-30T22:14:57  <gmaxwell> turns the last write to line-noise... and try it again.
5732015-10-30T22:16:02  <gmaxwell> and then you know that, absent reordering at lower levels, there is no point where unclean termination can leave the state irrecoverable in the log-under-test.
5742015-10-30T22:16:52  <gmaxwell> obviously this could model reordering too, if barriers get communicated across NBD (I dunno if they do)... but the state space becomes huge fast, so I think you could only try random cases, and not an exhaustive test.
5752015-10-30T22:18:25  <kanzure> warren: random pings in the future are much appreciated, thank you
5762015-10-30T22:19:57  <gmaxwell> wumpus: please post test binaries with the leveldb fix. 99.9% of windows users won't test from source.
5772015-10-30T22:20:01  <wumpus> the randomized brute force approach would be to crash, restart, crash, restart a VM running bitcoind in a loop, then a zillion times, trying to get it into a state where it won't recover
5782015-10-30T22:20:34  <GitHub91> [bitcoin] sipa opened pull request #6918: Make sigcache faster, more efficient, larger (master...smallsigcache) https://github.com/bitcoin/bitcoin/pull/6918
5792015-10-30T22:20:43  <wumpus> gmaxwell: maybe jonasschnelli would like to post executables; I don't have the infrastrcuture for that at the moment
5802015-10-30T22:20:48  <gmaxwell> sipa: "faster, stronger, better!"
5812015-10-30T22:21:26  <kanzure> you could selectively break parts of the vm maybe
5822015-10-30T22:21:34  <kanzure> e.g. increase unreliability of different system calls
5832015-10-30T22:23:36  <gmaxwell> sipa: hitrate could be improved further if you store the height at the time an entry is added the random eviction picks N entries and drops the one with the lowest height.
5842015-10-30T22:24:27  <gmaxwell> sipa: even with it 10x larger, hitrate for a transactions in the current mempool will not be 100% and thats nuts.
5852015-10-30T22:25:53  *** randy-waterhouse has quit IRC
5862015-10-30T22:26:13  <sipa> gmaxwell: around 99.3% hitrate on average for the last 8000 entries
5872015-10-30T22:28:19  <gmaxwell> so lets say the block has 2000 signatures drawn from the last 8000, and a 99.3% hitrate. So 14 misses.  If a verify takes 420 microseconds (as it does for openssl), then thats 6 ms added to block processing.
5882015-10-30T22:28:49  <sipa> the last 1000 have 99.9% hitrate
5892015-10-30T22:29:10  <sipa> one way to do it would be to use a unordered_map to int64_t, which just increases sequentially
5902015-10-30T22:29:23  <sipa> and when removing, randomly find 4 entries and sort them by that number
5912015-10-30T22:29:33  <gmaxwell> or 1%-6% slowdown (depending on how much we speed up the rest of block processing).
5922015-10-30T22:29:52  <sipa> maintaining a fully ordered index adds a lot of memory
5932015-10-30T22:29:58  <gmaxwell> I was basically suggesting that but using height, because I think we actually want the oldest of the ones at the current height usually.
5942015-10-30T22:30:46  <sipa> otherwise we need to pass down height through several layers of code that are currently height-oblivious
5952015-10-30T22:30:48  <gmaxwell> so map to int32_t, with the height at insertion time. and then pull four entries and reject the lowest height.
5962015-10-30T22:31:11  <gmaxwell> oh well meh, true. though block height advance is the magical event that should cause entries to become useless. Not age.
5972015-10-30T22:31:39  <sipa> we could make it wipe entries on block verify
5982015-10-30T22:31:56  <sipa> that's trivial
5992015-10-30T22:33:09  <gmaxwell> sounds even better. I think with that we could avoid increasing its memory usage, and stay at 10mb.
6002015-10-30T22:34:15  <gmaxwell> sipa: if we can do that, then we could also have that same event trigger a counter that a block has been accepted, no?
6012015-10-30T22:34:35  <sipa> gmaxwell: it's a boolean passed down saying whether we want to store or not
6022015-10-30T22:34:43  <sipa> per entry
6032015-10-30T22:34:49  <phantomcircuit> er
6042015-10-30T22:34:51  <phantomcircuit>     bool Encrypt(const CKeyingMaterial& vchPlaintext, std::vector<unsigned char> &vchCiphertext);
6052015-10-30T22:35:03  <phantomcircuit> actually nvm
6062015-10-30T22:35:10  <phantomcircuit> i had them reversed in my brain
6072015-10-30T22:36:23  <gmaxwell> sipa: the eviction is probably enough, though it still has a bias towards the more recent inserts rather than a bias towards preserving the oldest inserts which are since the prior block.
6082015-10-30T22:36:48  <sipa> gmaxwell: meh
6092015-10-30T22:37:41  <gmaxwell> I don't think it matters much, but evict on verify will make that sighash single trick for spending dustspam slow to verify.
6102015-10-30T22:41:48  <gmaxwell> sipa: what is the average hit rate for a transactions 6000 to 8000 ago?
6112015-10-30T22:42:46  <phantomcircuit> gmaxwell, to verify that im reading this right, each time an encrypted wallet is unlocked the first key in mapCryptedKeys is checked against it's pubkey and the first time the wallet is unlocked all of the keys are checked against their pubkey, correct?
6122015-10-30T22:44:48  <gmaxwell> phantomcircuit: thats my recollection of what it does, yes.
6132015-10-30T22:45:21  <gmaxwell> it's a slow and not very good test-- the problem, of course, is you get no verification at all if the wallet is never unlocked.
6142015-10-30T22:45:36  *** [b__b] has quit IRC
6152015-10-30T22:45:57  *** [b__b] has joined #bitcoin-core-dev
6162015-10-30T22:46:16  <phantomcircuit> gmaxwell, right
6172015-10-30T22:46:36  <phantomcircuit> how about make a list for eviction and evict only when a block is completely verified?
6182015-10-30T22:47:16  <gmaxwell> phantomcircuit: complicated and screws up the layering.
6192015-10-30T22:47:40  <gmaxwell> phantomcircuit: I'd rather have a hash basked check that stays with the key, and gets checked on load and any time its used.
6202015-10-30T22:48:06  <phantomcircuit> gmaxwell, no i meant for the sigcache thing
6212015-10-30T22:48:26  <gmaxwell> phantomcircuit: I know, mutiple threads of discussion.
6222015-10-30T22:48:27  <sipa> phantomcircuit: complicated and screws up the layering :)
6232015-10-30T22:48:43  <sipa> phantomcircuit: the sigcache code doesn't know about a 'block' concept at all
6242015-10-30T22:49:43  <gmaxwell> sipa: I don't see why the sigcache couldn't just have a counter in it, and every time you verify a block you take a lock on the sigcache and increment that counter.  And the current counter value rides with the entries in an ordered map, and does the biased eviction I suggested.
6252015-10-30T22:50:03  <sipa> gmaxwell: perfectly doable
6262015-10-30T22:50:05  <gmaxwell> that won't require passing anything height around, it's not hight anymore then, it's just 'sigcache epoch'
6272015-10-30T22:50:38  <sipa> going to benchmark without it to see how large the sigcache actually gets
6282015-10-30T22:51:45  <gmaxwell> well it will get to the maximum size, since there is no eviction except on full. :)
6292015-10-30T22:52:00  <sipa> with the evict-when-seen-in-block rule
6302015-10-30T22:54:50  <phantomcircuit> gmaxwell, heh
6312015-10-30T22:54:56  <gmaxwell> k, that does have that patalogical performance I mentioned. sadly I don't see a nice way to fix that, except perhaps having a one entry FIFO for the last evicted item.
6322015-10-30T22:55:34  <sipa> gmaxwell: hmm, what pathological performance?
6332015-10-30T22:56:59  <gmaxwell> sipa: There is, for example, a block with a 1MB tx of SIGHASH_SINGLE bug using transactions that have all the same signature.   If it's cached its almost instant to verify ... otherwise....
6342015-10-30T22:57:15  <sipa> oh!
6352015-10-30T22:57:22  <gmaxwell> similar OP_DUP2 CHECKSIG OP_DUP2 CHECKSIG ... (or checkmultisig with the same pubkey).
6362015-10-30T22:58:26  *** belcher has joined #bitcoin-core-dev
6372015-10-30T23:03:35  <phantomcircuit> we still end up calling SignatureHash even with the sigcache right?
6382015-10-30T23:04:31  <sipa> yes
6392015-10-30T23:05:54  <phantomcircuit> any reason CryptedKeyMap is in keystore.h instead of src/wallet/crypter.h ? (it's not used anywhere outside of src/wallet/crypter.{h,cpp})
6402015-10-30T23:06:03  <sipa> nope
6412015-10-30T23:06:18  <sipa> CKeyStore itself should move to script/sign.h
6422015-10-30T23:06:28  <sipa> CBasicKeyStore can stay
6432015-10-30T23:06:36  <sipa> and CCryptoKeyStore can indeed move to wallet/crypter
6442015-10-30T23:12:45  <phantomcircuit> gmaxwell, also yes that was my basic plan for adding hashing for ckey entries
6452015-10-30T23:22:06  *** zooko has quit IRC
6462015-10-30T23:22:29  *** danielsocials has joined #bitcoin-core-dev
6472015-10-30T23:22:36  *** zooko has joined #bitcoin-core-dev
6482015-10-30T23:34:47  <GitHub83> [bitcoin] laanwj closed pull request #6893: forward-declare univalue in rpcclient + remove include in rpcserver.cpp (master...univalue) https://github.com/bitcoin/bitcoin/pull/6893
6492015-10-30T23:37:30  *** zooko has quit IRC
6502015-10-30T23:38:58  <GitHub43> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/725539ea0376...d482c0a7b246
6512015-10-30T23:38:58  <GitHub43> bitcoin/master e9e6163 Pieter Wuille: Make -checkmempool=1 not fail through int32 overflow
6522015-10-30T23:38:59  <GitHub43> bitcoin/master d482c0a Wladimir J. van der Laan: Merge pull request #6896...
6532015-10-30T23:39:03  <GitHub192> [bitcoin] laanwj closed pull request #6896: Make -checkmempool=1 not fail through int32 overflow (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6896
6542015-10-30T23:40:05  <GitHub164> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d482c0a7b246...48b5b84ee511
6552015-10-30T23:40:05  <GitHub164> bitcoin/master 30d9662 Gregory Maxwell: Reject invalid pubkeys when reading ckey items from the wallet....
6562015-10-30T23:40:06  <GitHub164> bitcoin/master 48b5b84 Wladimir J. van der Laan: Merge pull request #6906...
6572015-10-30T23:40:15  <GitHub146> [bitcoin] laanwj closed pull request #6906: Reject invalid pubkeys when reading ckey items from the wallet. (master...ckey_pubkey_check) https://github.com/bitcoin/bitcoin/pull/6906
6582015-10-30T23:46:29  *** dcousens has joined #bitcoin-core-dev
6592015-10-30T23:47:11  *** danielsocials has quit IRC
6602015-10-30T23:48:39  *** evoskuil has joined #bitcoin-core-dev
6612015-10-30T23:49:26  <dcousens> sipa: don't mind my nits
6622015-10-30T23:49:40  <sipa> dcousens: you're welcome to ask questions :)
6632015-10-30T23:50:54  <dcousens> PR is super clean, the only semantics that weren't clear to me was the logic around `store`
6642015-10-30T23:52:02  <dcousens> What does it represent? A data 'store', or, whether to allow the sigCache to be used?
6652015-10-30T23:52:28  <sipa> store==true is set for mempool transactions, store==false is set for block validation
6662015-10-30T23:52:59  <sipa> the original intent was that we wouldn't store validations done during block validation, only query the cache
6672015-10-30T23:54:16  <dcousens> So the current usage is whether or not it should use a cache at all
6682015-10-30T23:54:25  <sipa> no
6692015-10-30T23:54:41  <sipa> with store==false, the cache is still queried; the result of validation is just not stored
6702015-10-30T23:54:53  <dcousens> But the cache would be empty no?
6712015-10-30T23:56:17  <dcousens> The set only gets inserted into if `store == true`
6722015-10-30T23:56:24  <sipa> it gets filled by mempool validations
6732015-10-30T23:56:30  <sipa> and queried for block validations
6742015-10-30T23:56:33  *** PaulCape_ has joined #bitcoin-core-dev
6752015-10-30T23:56:37  <dcousens> AH
6762015-10-30T23:56:41  <dcousens> bloody inline statics
6772015-10-30T23:56:44  <dcousens> sorry, missed the static
6782015-10-30T23:56:55  <dcousens> That makes sense now
6792015-10-30T23:57:30  <dcousens> Though signatureCache was a member of CachingTransactionSignatureChecker
6802015-10-30T23:57:37  <dcousens> Thought*
6812015-10-30T23:59:35  *** PaulCapestany has quit IRC
6822015-10-30T23:59:36  <dcousens> I'm too used to just never having globals haha, anyway, cheers for answering questions :)