When we send a directory, we build a temporary zipfile to linearize the
contents for transmission. We store this zipfile inside a
SpooledTemporaryFile, which will hold everything in RAM (for speed) until it
reaches some size threshold, then moves everything over to disk (to avoid
crashing your program as it runs out of memory).
Unfortunately SpooledTemporaryFile doesn't have a default threshold size: if
you don't specify one, it will never switch to the disk-based mode, and
`wormhole send large_directory/` will just use up all your RAM until it
crashes.
I've been using this wrong for five years, since the very beginning of
wormhole's ability to send directories. How embarrassing!
This applies the simple fix: provide a `max_size=` argument, setting the
threshold to 10 MB.
Thanks to @blitz for the report (and my apologies to everyone else who
reported this earlier, to whom I said it was fixed by using
SpooledTemporaryFile, when clearly it was not).
closes#379
Specifically, this lets the `wormhole tx` side send the VERSIONS message
before input() happens, allowing the `wormhole rx` side to compute and
display the Verifier. This only matters when the receiver sends both PAKE and
VERSIONS in the same turn, which only happens when tab-completion allowed
them to learn the Nameplate early and thus receive the sender's PAKE early.
In the other cases, the receiver sends PAKE and VERSIONS on separate turns,
so the sender doesn't get into this situation.
The bug this fixes is when both sides use --verify, and the receiver uses
tab-completion, then the sender shows the Verifier and waits for
confirmation, but the receiver doesn't show the Verifier until *after* the
sender confirms. So the two users don't have enough information to compare.
Many thanks to Jacek Politowski (@jpolnetpl) for the catch and initial
investigation.
closes#349
When I made it possible to override APPID with a CLI argument (issue #113), I
forgot to also change this w.derive_key() (issue #339). We don't really need
to include APPID in that purpose string at all (the ideal code would be just
`w.derive_key("transit-key", length)`), but we're stuck with it now. Both
sides must use the same derivation process, and it would be pretty
expensive/complicated to negotiate the process ahead of time (and this code
is scheduled to be obsoleted by Dilation anyways).
I added a note to the two sites that use it, and put a local copy of the
APPID there. We should treat that copy as an arbitrary magic string that must
be included for compatibility with existing deployments (potential
file-transfer peers), which is coincidentally similar to the default `APPID`.
closes#339
See https://bugs.python.org/issue26175 . tempfile.SpooledTemporaryFile
doesn't fully implement the IOBase abstract class, which breaks because
py3.7.0's new zipfile module tries to delegate .seekable down to the wrapped
file and causes an AttributeError.
refs #306
We care about how long it takes to import all the wormhole-specific things,
to investigate user-perceived latency from the time the command is launched
to the time they can actually interact with it. So we need to record
`time.time()` before doing the rest of the imports, even though pep8 says all
imports should be done before any non-importing statements.
I've seen intermittent failures in
test_cli.PregeneratedCode.test_text_subprocess where the host was slow (or
overloaded) enough that the "Waiting for sender.." pacifier message was
displayed, which flunks the test because we're looking for a specific output
string. We patch this 1-second timer in the non-subprocess tests, but you
can't patch across a process boundary.
This patch adds an undocumented environment variable that lets you override
the timer values. The test then sets it to something large.
For future consideration: another approach would be to change the test to
tolerate the extra message. This would be trickier to validate, though.
lgtm.com noticed some unreachable code paths, and it turns out that nothing
in the rest of the code base could ever raise WormholeClosedError (I guess it
was leftover from before the big API refactoring). Both sender and receiver
are simpler without the unnecessary checks and state variables.
A tiny update to show the command to execute along with the code required on
the same line, rather than split across two lines. This small change helps
when sending the information to others using copy and paste.
Fixes#266
-=david=-
This ought to help with #251, where bash-on-windows makes it easy to add a
forward-slash, and os.path.normpath() knows how to remove them, but os.sep is
a backslash.
Added the click option to look for relay and transit urls in environment
variables. If you're running your own relay/transit servers (such as
inside a corporate firewall), this will make client's lives easier.
The previous behavior was to throw an Automat exception, when a state machine
was given a LOST event from the initial non-connected state, and it didn't
have a handler for it. This version throws ServerConnectionError instead.
Still needs a test
refs #180
This provides a clear error in case the user doesn't have an internet
connection at all, or something is so broken with their DNS or routing that
they can't reach the server. I think this is better than waiting and
retrying (silently) forever.
If the first connection succeeds, but is then lost, subsequent retries occur
without fanfare.
closes#68
This shifts most reponsibility to the new txtorcon "Controller" object, where
it belongs. We no longer need a list of likely control-port locations, nor do
we need to keep track of the SOCKS port ourselves.
The one downside is that if a control-port is not reachable, then this does
not fall back to using a plain SOCKS port (usually tcp:localhost:9050).
txtorcon no longer uses txsocksx, so it no longer advertises a simple way to
use Tor without the control port. This shouldn't affect users who run the
TorBrowserBundle, or who are running a tor daemon which they can control
directly, but it may break for users who want to use a pre-existing tor
daemon that they don't have permissions to speak control-port to.
This renames all the existing API methods, to use a consistent
"d=get_XYZ()" (for Deferred mode) or "dg.wormhole_got_XYZ()" (for Delegated
mode). It updates cmd_send/cmd_receive/cmd_ssh to use the new API.
Since we now have get_welcome(), apps handle the Welcome message with a
Deferred callback instead of registering a "welcome handler". This lets us
make sure we've finished printing any server message-of-the-day or "you
should update your client" message to stdout before using stdio to ask for
the wormhole code. (Previously, the code-input prompt was overwritten by the
server message, and it was ugly). refs #145. This approach adds an extra
roundtrip to the receiver, but we can fix that (see #145 for details).
Because of that change, the server-is-being-slow message is printed at a
slightly different time, so those tests needed some extra work to exercise it
properly.
I think we're better off without this: the CLI commands propagate the Failure
up to their callers (rather than eating it silently), the callers are using
task.react (which reacts to Failures by exiting with rc!=0), so nothing
should get lost. And doing an extra log.err() just creates more cleanup work
for test cases to flush, and makes the CLI commands double-print the any
errors (maybe task.react also points logging at stderr?).
The Welcome class prints a message if the server recommends a CLI version
that's newer than what the client is currently using, but only if the client
is running a "release" version, not a "local" development one. "local"
versions have a "+" in them (at least when Versioneer creates it), but
Welcome was looking for "-" as an indicator. So it was printing the warning
when it shouldn't be.
re-enable the test, and add an extra one
The comments in cmd_send/cmd_receive now enumerate the four cases where we
might notice that things are taking too long, the three cases where we say
something about it, and the two cases where it might be appropriate to give
up automatically (although we don't do that anywhere yet).
