test_ipaddrs.py had an invalid regexp, caught by the latest version of flake8
The new flake8 complains about both W503 (line break inside a conditional
before a binary operator) and W504 (line break *after* the operator). I think
break-before is the new preferred style, but for now I'm just going to ignore
them both and leave the code alone.
This establishes what our mailbox protocol does to encrypt the
individual (post-SPAKE2) messages, which combines NaCl SecretBox and our own
key-derivation choices. I'd like to move off of NaCl/libsodium and I think
some RFC7539-implementing library might be compatible, and with these test
vectors I can check that. I also want to copy these tests into the Rust port.
See https://bugs.python.org/issue26175 . tempfile.SpooledTemporaryFile
doesn't fully implement the IOBase abstract class, which breaks because
py3.7.0's new zipfile module tries to delegate .seekable down to the wrapped
file and causes an AttributeError.
refs #306
The flake8 config excludes E741, which would complain about using
'l' (lower-case ell) as a variable name. We use this for the Lister object in
one test that uses single-character variable names for all the machines ('b'
for Boss, 'm' for Mailbox, etc). That comment was added before excluding
E741. If we ever restore that warning, we might want to rename the variable.
We care about how long it takes to import all the wormhole-specific things,
to investigate user-perceived latency from the time the command is launched
to the time they can actually interact with it. So we need to record
`time.time()` before doing the rest of the imports, even though pep8 says all
imports should be done before any non-importing statements.
tested with/on:
- ubuntu linux 18.04 amd64
- pyinstaller 3.3.1 (pip install pyinstaller)
- python 3.6.5
There is a good chance it also works on FreeBSD, maybe also on macOS.
The change in __main__.py was required because otherwise it complains about
__main__ not being a package when trying the dot-relative import.
Tests can pass an EventualQueue into wormhole.create(), to override the
default. This lets the tests flush the queue without using a haphazard
real-time delay.
closes#23
(in fact, we added multiple-Deferreds-per-API a while ago, but this does it
in a much cleaner fashion, and with the safety of an eventual-send)
This factors out the various "give me a Deferred for an value that may or may
not eventually be successfully generated" routines in _DeferredWormhole. It
uses an eventual-send to fire the Deferreds to avoid plan-coordination
hazards when the attached callbacks then call back into the Wormhole object
before the rest of the state transition has finished.
We defer starting a new timer until we've completely emptied the queue, since
we know we'll get to any new events added inside one of our callbacks. The
old design in Foolscap (which copied the list, cleared the original, then
fired everything in the copy) didn't look at these new events. OTOH, this
pop(0)-until-empty approach makes it easier to get into an infinite loop (any
callback which queues a new callback will get priority over anything else).
But the code is simpler.
I've seen intermittent failures in
test_cli.PregeneratedCode.test_text_subprocess where the host was slow (or
overloaded) enough that the "Waiting for sender.." pacifier message was
displayed, which flunks the test because we're looking for a specific output
string. We patch this 1-second timer in the non-subprocess tests, but you
can't patch across a process boundary.
This patch adds an undocumented environment variable that lets you override
the timer values. The test then sets it to something large.
For future consideration: another approach would be to change the test to
tolerate the extra message. This would be trickier to validate, though.
This will be necessary to avoid reentrancy hazards, in case the application
code makes other wormhole API calls from the callbacks for
get_unverified_key(), get_verifier(), or get_message().
We were missing two (the calls to choose_nameplate() and choose_words() that
happen after the input() function has finished, but while we're still inside
the thread that makes it safe for input() to block). This almost certainly
caused the crash seen in issue #280.
Update the tests to match: CodeInputter.finish must now be called with
deferToThread from inside tests, or the internal blockingCallFromThread must
be stubbed out.
This causes two threads to use the reactor at the same time, with horrible
results. The _rlcompleter code currently violates this requirement, causing
occasional failures if the messages arrive in just the wrong way (refs #280).
Also log events at the beginning and end of Input.choose_nameplate and
Input.choose_words, since those are the two big locally-driven (UI) triggers
that cause multiple messages to be fired and lots of work to happen.
lgtm.com noticed some unreachable code paths, and it turns out that nothing
in the rest of the code base could ever raise WormholeClosedError (I guess it
was leftover from before the big API refactoring). Both sender and receiver
are simpler without the unnecessary checks and state variables.
This comments out some "if 0: debug()" stuff I keep around to investigate
problems, since lgtm thinks of it as accidentally-unreachable code.
I also deleted a server usage command
entirely (src/wormhole/server/cmd_usage.py show_usage) which was disabled
while I rewrote that schema: the new plan is to move the server into a new
repository altogether, and use a completely different approach to the
usage database.
A slow Travis-CI host caused one the 1.0s KEY_TIMER to fire by accident,
making the test fail because it wasn't expecting to see the "please be
patient" message. Fixed this by increasing the timeout to a very large value
when we aren't explicitly testing it.
A tiny update to show the command to execute along with the code required on
the same line, rather than split across two lines. This small change helps
when sending the information to others using copy and paste.
Fixes#266
-=david=-
This ought to help with #251, where bash-on-windows makes it easy to add a
forward-slash, and os.path.normpath() knows how to remove them, but os.sep is
a backslash.
Added the click option to look for relay and transit urls in environment
variables. If you're running your own relay/transit servers (such as
inside a corporate firewall), this will make client's lives easier.
If you pass --tor-control-port= and we can't use it, throw an error that will
kill the whole process, instead of falling back to the default SOCKS port.
If you omit --tor-control-port=, then if all default control port connections
fail, we'll fall back to the default SOCKS port.
Also, test each combination separately, and improve the status messages.
Linux defaults to a soft limit of 1024, which limits us to 512 simultaneous
non-transit-using connections. The transit relay runs in the same process, so
long-running relayed transfers will compete for those sockets too.
This raises the soft limit to equal the hard limit (if possible), or as much
as we can manage, if the soft limit was less than 10k. If the
resource.setrlimit calls aren't available (e.g. windows), or some other error
happens, this will log a message and continue without changing the limits.
closes#238
The Mailbox object throws CrowdedError, but WebSocketRendezvous wasn't
handling it specifically. The server responded by dropping the connection and
logging an "Unhandled Error", so the client would reconnect and then get the
same error again and again.
This changes WebSocketRendezvous to handle CrowdedError by sending a
"crowded" error response. The client should react to this by giving up on the
connection entirely, and not reconnecting.
We only log the internal (sqlite) ID of the nameplate, not the actual
small-integer name. While investigating misbehavior due to overload, I was
confused into thinking that users were getting nameplates in the 15000+
range, when in fact those were merely the internal database row ids.
This now shares the _compose() decorator with wormhole.cli.cli, and removes
the arguments_to_config() function in favor of just copying all kwargs into
the Config object.
The previous behavior was to throw an Automat exception, when a state machine
was given a LOST event from the initial non-connected state, and it didn't
have a handler for it. This version throws ServerConnectionError instead.
Still needs a test
refs #180
"wormhole-server restart" was broken by the addition of --disallow-list,
because the Click parser wasn't update to include the argument. This test
should exercise that a basic no-argv invocation of both "start" and "restart"
can at least build the Service object successfully.
refs #151
This provides a clear error in case the user doesn't have an internet
connection at all, or something is so broken with their DNS or routing that
they can't reach the server. I think this is better than waiting and
retrying (silently) forever.
If the first connection succeeds, but is then lost, subsequent retries occur
without fanfare.
closes#68
This shifts most reponsibility to the new txtorcon "Controller" object, where
it belongs. We no longer need a list of likely control-port locations, nor do
we need to keep track of the SOCKS port ourselves.
The one downside is that if a control-port is not reachable, then this does
not fall back to using a plain SOCKS port (usually tcp:localhost:9050).
txtorcon no longer uses txsocksx, so it no longer advertises a simple way to
use Tor without the control port. This shouldn't affect users who run the
TorBrowserBundle, or who are running a tor daemon which they can control
directly, but it may break for users who want to use a pre-existing tor
daemon that they don't have permissions to speak control-port to.
This renames all the existing API methods, to use a consistent
"d=get_XYZ()" (for Deferred mode) or "dg.wormhole_got_XYZ()" (for Delegated
mode). It updates cmd_send/cmd_receive/cmd_ssh to use the new API.
Since we now have get_welcome(), apps handle the Welcome message with a
Deferred callback instead of registering a "welcome handler". This lets us
make sure we've finished printing any server message-of-the-day or "you
should update your client" message to stdout before using stdio to ask for
the wormhole code. (Previously, the code-input prompt was overwritten by the
server message, and it was ugly). refs #145. This approach adds an extra
roundtrip to the receiver, but we can fix that (see #145 for details).
Because of that change, the server-is-being-slow message is printed at a
slightly different time, so those tests needed some extra work to exercise it
properly.
The code in util.py can tolerate a missing os.statvfs, but the code which
tests that code's ability to tolerate a missing os.statvfs was itself unable
to tolerate a missing os.statvfs. Sigh.
Apparently there's an intermittent condition in tests that follows this path.
We used to ignore unrecognized states, but to improve test coverage I added a
clause to catch them, and now that clause is happening where I didn't expect
it.
This fixes the case where "wormhole send" would wait forever (upon network
error) instead of terminating with a useful error message.
Testing this will have to wait until we land the branch that abandons the
wormhole if the first connection fails, since that's the easiest way to
provoke a network error before when_code() has fired.
I think we're better off without this: the CLI commands propagate the Failure
up to their callers (rather than eating it silently), the callers are using
task.react (which reacts to Failures by exiting with rc!=0), so nothing
should get lost. And doing an extra log.err() just creates more cleanup work
for test cases to flush, and makes the CLI commands double-print the any
errors (maybe task.react also points logging at stderr?).
The Welcome class prints a message if the server recommends a CLI version
that's newer than what the client is currently using, but only if the client
is running a "release" version, not a "local" development one. "local"
versions have a "+" in them (at least when Versioneer creates it), but
Welcome was looking for "-" as an indicator. So it was printing the warning
when it shouldn't be.
re-enable the test, and add an extra one
The comments in cmd_send/cmd_receive now enumerate the four cases where we
might notice that things are taking too long, the three cases where we say
something about it, and the two cases where it might be appropriate to give
up automatically (although we don't do that anywhere yet).
Previously, w.when_verified() was documented to fire only after a valid
encrypted message was received, but in fact it fired as soon as the shared
key was derived (before any encrypted messages are seen, so no actual
"verification" could occur yet).
This fixes that, and also adds a new w.when_key() API call which fires at the
earlier point. Having something which fires early is useful for the CLI
commands that want to print a pacifier message when the peer is responding
slowly. In particular it helps detect the case where 'wormhole send' has quit
early (after depositing the PAKE message on the server, but before the
receiver has started). In this case, the receiver will compute the shared
key, but then wait forever hoping for a VERSION that will never come. By
starting a timer when w.when_key() fires, and cancelling it when
w.when_verified() fires, we have a good place to tell the user that something
is taking longer than it should have.
This shifts responsibility for notifying Boss.got_verifier, out of Key and
into Receive, since Receive is what notices the first valid encrypted
message. It also shifts the Boss's ordering expectations: it now receives
B.happy() before B.got_verifier(), and consequently got_verifier ought to
arrive in the S2_happy state rather than S1_lonely.
* InputHelper returns full words, not just suffixes. I liked the fact that
suffixes made it impossible to violate the "all matches will start with
your prefix" invariant, but in practice it was fiddly to work with.
* add ih.when_wordlist_is_available(), so the frontend can block (after
claiming the nameplate) until it can return a complete wordlist to
readline. This helps the user experience, because readline wasn't really
built to work with completions that change over time
* make the Wordlist responsible for appending hyphens to all non-final word
completions. InputHelper remains responsible for hyphens on nameplates.
This makes the frontend simpler, but I may change it again in the future if
it helps non-readline GUI frontends.
* CodeInputter: after claiming, wait for the wordlist rather than returning
an empty list
* PGPWordList: change to match
This has the unfortunate side-effect that e.g. typing "3-yucatan-tu TAB"
shows you completions that include the entire phrase: "3-yucatan-tumor
3-yucatan-tunnel", rather than only mentioning the final word. I'd like to
fix this eventually.
I'm still undecided about whether to add this to the mailbox
properties (revealing it to attackers) or continue to require non-default
wordcounts to be provided as a --code-length= argument to the receiver. So
for now the only place that says count=2 is in the default argument on
get_completions().
This updates the unit tests to checks the system (by running 'locale -a' just
like Click does) to use a UTF-8 -safe locale. It prefers C.UTF-8 if
available, then en_US.UTF-8, then will fall back to any UTF-8 it can find.
My macOS box has en_US.UTF-8 (but not C.UTF-8), and my linux box has
C.UTF-8 (but not en_US.UTF-8). This change doesn't help normal runtime, but
ought to allow the unit tests to run on either platform correctly.
This also changes the can-I-run-wormhole check to use C.UTF-8 instead of
en_US.UTF-8, which seems necessary to hush Click on py3. See issue #127 for
more discusson.
