When we send a directory, we build a temporary zipfile to linearize the
contents for transmission. We store this zipfile inside a
SpooledTemporaryFile, which will hold everything in RAM (for speed) until it
reaches some size threshold, then moves everything over to disk (to avoid
crashing your program as it runs out of memory).
Unfortunately SpooledTemporaryFile doesn't have a default threshold size: if
you don't specify one, it will never switch to the disk-based mode, and
`wormhole send large_directory/` will just use up all your RAM until it
crashes.
I've been using this wrong for five years, since the very beginning of
wormhole's ability to send directories. How embarrassing!
This applies the simple fix: provide a `max_size=` argument, setting the
threshold to 10 MB.
Thanks to @blitz for the report (and my apologies to everyone else who
reported this earlier, to whom I said it was fixed by using
SpooledTemporaryFile, when clearly it was not).
closes#379
This test was incorrectly exercising a member of the endpoint record returned
by `Manager.get_endpoints()`. In the test environment, the `.listen` Endpoint
is actually a Mock, so calling e.g. `listen()` on `endpoints.listen` returns
another Mock instead of a Deferred. Twisted's `assertNoResult` used to
tolerate this silently, but as of Twisted-19.10 it throws an error, causing
the test to fail.
The fix is to assert that the record has attributes with the right names, but
not assume they behave like normal Endpoints, and not call `.listen()` on
them.
closes#366
Specifically, this lets the `wormhole tx` side send the VERSIONS message
before input() happens, allowing the `wormhole rx` side to compute and
display the Verifier. This only matters when the receiver sends both PAKE and
VERSIONS in the same turn, which only happens when tab-completion allowed
them to learn the Nameplate early and thus receive the sender's PAKE early.
In the other cases, the receiver sends PAKE and VERSIONS on separate turns,
so the sender doesn't get into this situation.
The bug this fixes is when both sides use --verify, and the receiver uses
tab-completion, then the sender shows the Verifier and waits for
confirmation, but the receiver doesn't show the Verifier until *after* the
sender confirms. So the two users don't have enough information to compare.
Many thanks to Jacek Politowski (@jpolnetpl) for the catch and initial
investigation.
closes#349
The previous implementation would call the control/receiving Protocol
completely backwards: dataReceived first, then connectionLost, then finally
connectionMade. Which didn't work at all, of course.
When I made it possible to override APPID with a CLI argument (issue #113), I
forgot to also change this w.derive_key() (issue #339). We don't really need
to include APPID in that purpose string at all (the ideal code would be just
`w.derive_key("transit-key", length)`), but we're stuck with it now. Both
sides must use the same derivation process, and it would be pretty
expensive/complicated to negotiate the process ahead of time (and this code
is scheduled to be obsoleted by Dilation anyways).
I added a note to the two sites that use it, and put a local copy of the
APPID there. We should treat that copy as an arbitrary magic string that must
be included for compatibility with existing deployments (potential
file-transfer peers), which is coincidentally similar to the default `APPID`.
closes#339
This should fix the immediate issue of the remote side opening a
subchannel (and sending data on it) before the local side even sees the
Endpoints, so before it can register a listening factory to receive the OPEN.
We were already buffering early OPENs in the SubchannelListenerEndpoint, but
this makes sure that endpoint is available (for the manager's Inbound half to
deliver) them as soon as the dilation connection is established.
The downside to buffering OPENs (and all data written to inbound subchannels)
is that the application has no way to reject or pause them, until it
registers the listening factory. If the application never calls
`listen_ep.listen()`, we'll buffer this data forever (or until the wormhole
is closed). The upside is that we don't lose a roundtrip waiting for an ack
on the OPEN. See ticket #335 for more details.
refs #335
Anyone sending 4GB in a single `transport.write()` is in for a surprise, but
at least we'll surprise them with an assertion *before* spending the time and
memory encrypting that monster.
I think I just managed to forget that inbound_close requires we respond with
a close ourselves. Also outbound open means we must add the subchannel to the
inbound table, so we can receive any data on it at all.
I'm sure I had a good reason for avoiding integers, but it makes logging and
testing more difficult, and both sides are using integers to generate them
anyways (so one side can pick the odd ones, and the other can pick the even
ones).
Without this, the Follower would see data for subchannel 0 before it had a
chance to create the SubChannel object that could accept it. We already have
a mechanism for inbound data to be queued inside the SubChannel until the
endpoint has had a chance to create the Protocol object: we rely on that
mechanism here. We just need to create the SubChannel before telling the
Manager to start, even though we don't reveal the SubChannel to the
caller (via the control endpoint) until the connection is known to succeed.
This helps a manual test get data from one side to the other without throwing
exceptions.
When the follower's connection is accepted, they'll observe a single
dataReceived chunk containing both the leader's KCM and the leader's first
actual data record. The state machine considers the KCM for an eventual-turn
before selecting the connection, so the data record will arrive while the
connection isn't quite ready for it (if consider() were immediate, this
wouldn't be a problem, but Automat doesn't deal with reentrant calls very
well). So we queue any records that arrive before we're selected.
If we had multiple potential connections, the act of cancelling the losing
ones was putting an error into log.err(), which flunked the tests. This
happened to appear on windows because the appveyor environment has different
interfaces than travis hosts.
