When I made it possible to override APPID with a CLI argument (issue #113), I
forgot to also change this w.derive_key() (issue #339). We don't really need
to include APPID in that purpose string at all (the ideal code would be just
`w.derive_key("transit-key", length)`), but we're stuck with it now. Both
sides must use the same derivation process, and it would be pretty
expensive/complicated to negotiate the process ahead of time (and this code
is scheduled to be obsoleted by Dilation anyways).
I added a note to the two sites that use it, and put a local copy of the
APPID there. We should treat that copy as an arbitrary magic string that must
be included for compatibility with existing deployments (potential
file-transfer peers), which is coincidentally similar to the default `APPID`.
closes#339
See https://bugs.python.org/issue26175 . tempfile.SpooledTemporaryFile
doesn't fully implement the IOBase abstract class, which breaks because
py3.7.0's new zipfile module tries to delegate .seekable down to the wrapped
file and causes an AttributeError.
refs #306
I've seen intermittent failures in
test_cli.PregeneratedCode.test_text_subprocess where the host was slow (or
overloaded) enough that the "Waiting for sender.." pacifier message was
displayed, which flunks the test because we're looking for a specific output
string. We patch this 1-second timer in the non-subprocess tests, but you
can't patch across a process boundary.
This patch adds an undocumented environment variable that lets you override
the timer values. The test then sets it to something large.
For future consideration: another approach would be to change the test to
tolerate the extra message. This would be trickier to validate, though.
lgtm.com noticed some unreachable code paths, and it turns out that nothing
in the rest of the code base could ever raise WormholeClosedError (I guess it
was leftover from before the big API refactoring). Both sender and receiver
are simpler without the unnecessary checks and state variables.
This shifts most reponsibility to the new txtorcon "Controller" object, where
it belongs. We no longer need a list of likely control-port locations, nor do
we need to keep track of the SOCKS port ourselves.
The one downside is that if a control-port is not reachable, then this does
not fall back to using a plain SOCKS port (usually tcp:localhost:9050).
txtorcon no longer uses txsocksx, so it no longer advertises a simple way to
use Tor without the control port. This shouldn't affect users who run the
TorBrowserBundle, or who are running a tor daemon which they can control
directly, but it may break for users who want to use a pre-existing tor
daemon that they don't have permissions to speak control-port to.
This renames all the existing API methods, to use a consistent
"d=get_XYZ()" (for Deferred mode) or "dg.wormhole_got_XYZ()" (for Delegated
mode). It updates cmd_send/cmd_receive/cmd_ssh to use the new API.
Since we now have get_welcome(), apps handle the Welcome message with a
Deferred callback instead of registering a "welcome handler". This lets us
make sure we've finished printing any server message-of-the-day or "you
should update your client" message to stdout before using stdio to ask for
the wormhole code. (Previously, the code-input prompt was overwritten by the
server message, and it was ugly). refs #145. This approach adds an extra
roundtrip to the receiver, but we can fix that (see #145 for details).
Because of that change, the server-is-being-slow message is printed at a
slightly different time, so those tests needed some extra work to exercise it
properly.
I think we're better off without this: the CLI commands propagate the Failure
up to their callers (rather than eating it silently), the callers are using
task.react (which reacts to Failures by exiting with rc!=0), so nothing
should get lost. And doing an extra log.err() just creates more cleanup work
for test cases to flush, and makes the CLI commands double-print the any
errors (maybe task.react also points logging at stderr?).
re-enable the test, and add an extra one
The comments in cmd_send/cmd_receive now enumerate the four cases where we
might notice that things are taking too long, the three cases where we say
something about it, and the two cases where it might be appropriate to give
up automatically (although we don't do that anywhere yet).
The new TorManager adds --launch-tor and --tor-control-port= arguments
(requiring the user to explicitly request a new Tor process, if that's what
they want). The default (when --tor is enabled) looks for a control port in
the usual places (/var/run/tor/control, localhost:9051, localhost:9151), then
falls back to hoping there's a SOCKS port in the usual
place (localhost:9050). (closes#64)
The ssh utilities should now accept the same tor arguments as ordinary
send/receive commands. There are now full tests for TorManager, and basic
tests for how send/receive use it. (closes#97)
Note that Tor is only supported on python2.7 for now, since txsocksx (and
therefore txtorcon) doesn't work on py3. You need to do "pip install
magic-wormhole[tor]" to get Tor support, and that will get you an inscrutable
error on py3 (referencing vcversioner, "install_requires must be a string or
list of strings", and "int object not iterable").
To run tests, you must install with the [dev] extra (to get "mock" and other
libraries). Our setup.py only includes "txtorcon" in the [dev] extra when on
py2, not on py3. Unit tests tolerate the lack of txtorcon (they mock out
everything txtorcon would provide), so they should provide the same coverage
on both py2 and py3.
This should leave stdout clean for use in `foo | wormhole send --text=-` and
`wormhole rx CODE >foo`, although the forms that want interactive code entry
probably won't work that way.
closes#99
Tools which use `wormhole send` under the hood should use a distinct
--appid= (setting the same URL-shaped value on both sides, starting with a
domain name related to the tool and/or its author), so wormhole codes used by
those tools won't compete for short channelids with other tools, or the
default text/file/directory-sending tool.
Closes#113
closes#91
Also tweaks an error message: don't say "refusing to clobber pre-existing
file FOO" when we don't check that it's actually a file. Just say "..
pre-existing 'FOO'".
there was a function to "abbreviate" sizes, but it was somewhat
unclear and incomplete. reuse the sizeof_fmt_* set of functions from
the borg backup project (MIT licensed) to implement a more complete
and flexible display that will scale up to the Yottabyte and
beyond. it also supports non-IEC units (like "kibibyte", AKA 1024
bytes) if you fancy that stuff.
this is a workaround for #91: it allows users to better see the size
of the file that will be transfered.
*some* places are still kept in bytes, most notably when receive fails
to receive all bytes ("got %d bytes, wanted %d") because we may want
more clarity there.
text transfers also use the "bytes" suffix (instead of "B") because it
will commonly not reach beyond the KiB range.
note that the test suite only covers decimal (non-IEC) prefix, but it
is assumed to be sufficient to be considered correct.
This will allow a future peer to figure out what transit modes we can
and cannot do, and thus avoid spinning up expensive modes that we won't
be able to use (e.g. WebRTC).
This enhances the ACK that wormhole-receive returns when it finishes
receiving all the data to be a dictionary. The dict includes the SHA256
hash of everything it received, and the sender checks this for a match
before declaring the transfer to be a success. This guards against data
being shuffled somehow during transit.
The file-send protocol now sends a "hints-v1" key in the "transit"
message, which contains a list of JSON data structures that describe the
connection hints (a mixture of direct, tor, and relay hints, for now).
Previously the direct/tor and relay hints were sent in different keys,
and all were sent as strings like "tcp:hostname:1234" which had to be
parsed by the recipient.
The new structures include a version string, to make it easier to add
new types in the future. Transit logs+ignores hints it cannot
understand.
In the future, both sides should expect to receive "transit" messages at
any time, and they will add to the list of hints that they should try.
For now, each side only sends a single transit message, before they send
the offer (sender) or answer (receiver).
This moves us slowly towards a file-transfer protocol that exchanges
multiple messages, with a single offer (sender->receiver) and
answer (receiver->sender), and one or more connection hint messages (in
either direction) that appear gradually over time as connection
providers come online.
At present the protocol still expects the whole hint list to be present
in the offer/answer message.
