Also clean up test_scripts.PregeneratedCode:
* fetch results from both sides at the same time
* only check rc when using a subprocess, since the direct call doesn't
use rc=0 anymore
* no need to cancel the other side's Deferred when one errors
* provide more information if stderr was non-empty
And provide a close() that can live at the end of a Deferred chain, so
callers can do d.addBoth(w.close).
I like auto-close-on-error in general, but I'm removing it so I can
clean up the error-handling pathways. It will probably come back later.
The constraint is that it must be possible to wait on the return
Deferred that close() gives you (to synchronize tests, or keep the CLI
program running long enough to deallocate the channel) even if something
else (and error handler) called close() earlier. This will require
either a OneShotObserverList, or keeping a "deallocated" Deferred around
in case more callers want to wait on it later.
If we're closing because of an error, we need to sleep through the old
error, to be able to wait for the "deallocated" message. This might want
to be different: maybe clear the error first, or store the errors in a
list and sleep until a second error happens.
These were split out to make the blocking- and twisted- based
implementations share some code, but now that we're down to just
Twisted, it's clearer to merge them back in.
Hitting Control-C (which sends SIGINT) while we're waiting in the
readline-based input_code() function didn't shut down the process
properly: the reactor would wait for the readline thread to exit, which
wouldn't happen until it finished getting a code, which requires the
user to hit Return. I haven't found a good way to force the thread to
exit, or to synthetically inject a newline into stdin. So my compromise
is to tell the user that they need to hit Return to finish interrupting
the command.
See the _warn_readline() function for a list of other potential
approaches.
This should speed up the protocol, since we don't have to wait for
acks (HTTP responses) unless we really want to. It also makes it easier
to have multiple messages in flight at once. The protocol is still
compatible with the old HTTP version (which is still used by the
blocking flavor), but requires an updated Rendezvous server that speaks
websockets.
set_code() no longer touches the network: it just stores the code and
channelid for later. We hold off doing 'claim' and 'watch' until we need
messages, triggered by get_verifier() or get_data() or send_data().
We check for error before sleeping, not just after waking. This makes it
possible to detect a WrongPasswordError in get_data() even if the other
side hasn't done a corresponding send_data(), as long as the other side
finished PAKE (and thus sent a CONFIRM message). The unit test was doing
just this, and was hanging.
This allows the Wormhole setup path to be simpler: consistently doing a
claim() just before watch(), regardless of whether we allocated the
channelid (with get_code), or dictated it (with set_code or
from_serialized).
The websocket lives on a Resource of the main rendezvous web site, and
the websocket URL is derived from the main "relay_url", so there's no
extra port to allocate, and no extra service to shut down.
Deliver not-yet-JSONed objects to listeners (both in broadcast_message
and as the "catch-up" responses to add_listener). Also make the (web)
frontend responsible for adding "sent" timestamps. This all makes
rendezvous.py less web-centric.
This uses a single TCP connection to the relay server for all
requests (although it probably uses a second one for the downstream
EventSource feed). This should squeeze out some of the round-trip times.
This adds an expected= argument to Connection.connectConsumer(), which
then returns a Deferred that fires when enough bytes have been written
to the consumer. It also adds Connection.writeToFile(), a helper method
that writes bytes to a filehandle.
I made the classic dataReceived() mistake, and exited the function after
delivering the first record. Keep at it until there are no complete
records left.
The previous commits improve test failures by dropping relay connections
at shutdown, and flunking a test quickly when one client fails but the
other one hangs.
If that doesn't work (say, some client has a time.sleep(), or other
stall that isn't affected by the relay shutdown), we'll be left with an
active thread holding that hanging client.
This patch adds a check to wormhole.test.common.ServerBase.tearDown that
looks for active threads, waits a second (after stopService), then
checks the threadpool again. If the threadpool is empty, everything is
fine. If not, it prints a message (to stdout) to inform the impatient
user why the test is probably hanging.
When test_scripts ran two clients at the same time, an error in one
could leave the other hanging (in a thread). One Deferred would errback,
the other would hang. Tests wait on one Deferred at a time, so if we're
unlucky and were waiting on the hanging Deferred (instead of the
erroring one), we'll wait forever, or at least until the default test
timeout of 180 seconds.
This adds an errback to notice when either client has errored, and
cancels the other Deferred, so it doesn't matter which one we wait upon
first.
'readline' is part of the python stdlib, so declaring a dependency on it
doesn't help. It doesn't exist on windows, and the pypi 'readline'
module doesn't work on windows. So instead, just attempt to import
readline, and if that fails, fall back to a non-completion flavor.
This ensures that we'll be ready for them. Previously there was a race
between us revealing the direct hints to the peer, and us setting the
transit key (thus allowing us to check inbound handshake requests). The
Transit instance didn't handle the race, causing errors to be thrown
when the other side connected quickly.
This ensures that we'll be ready for them. Previously there was a race
between us revealing the direct hints to the peer, and us setting the
transit key (thus allowing us to check inbound handshake requests). The
Transit instance handles this race (with an interlock on the transit
key), but it's still nicer to do it cleanly.
This exposed a new race in Transit, where the inbound connection would
complete before transit.connect() had been called. The previous commit
adds an interlock to wait for that too. Until this change, the transit
key lock was covering that one up.
Some tests failed to override --transit-helper, which meant they
intermittently talked to the real transit server (briefly, before
deciding the local+direct connection was better).
This limits the buffering to about 10MB (per connection*direction).
Previously, if the sender had more bandwidth than the receiver, the
transit relay would buffer the entire file. With this change, the sender
will be throttled to match the receiver's downstream speed.
The latest Twisted fixes the web.Agent code we need for proper async
support. There's still a daemonization bug that prevents 'wormhole
server start' from succeeding (it hangs).
I'm planning to leave non-EventSource "/get" in until after 0.6.0, then
remove it. I think it's cleaner for the logs to have the two
forms (EventSource and immediate) use different URLs.
In the twisted-style code, the close_on_error() decorator forces the
return value to be a Deferred, which is all wrong for internal uses of
derive_key() (verification string and confirmation message). It might be
useful to have a synchronous form of close_on_error(), but since the
actual close() is async, that's not very straightforward.
So for now, tolerate unclosed Wormhole objects when someone calls
derive_key() too early, or with a non-unicode type string.
This removes the 'allocations' table entirely, and cleans up the way we
prune old messages. This should make it easier to summarize each
connection (for usage stats) when it gets deallocated, as well as making
pruning more reliable.
Without this, old senders will throw a messy 404 traceback when talking
to a modern server.
Unfortunately 0.4.0 receivers don't make API calls in the right order,
so they throw a 404 before seeing our "you need to upgrade" message.
I was really confused about the Server-Sent Events syntax. This new one
is compatible with actual web browsers and the spec:
http://www.w3.org/TR/eventsource/
This requires a DB delete/recreate when upgrading. It changes the server
protocol, and app IDs, so clients cannot interoperate with each other
across this change, nor with the server. Flag day for everyone!
Now apps do not share channel IDs, so a lot of usage of app1 will not
cause the wormhole codes for app2 to get longer.
There are now three ways to invoke send:
* "wormhole send": ask for a text message
* "wormhole send FILENAME": send a file
* "wormhole send --text TEXT": send text message
This removes "side" and "msgnum" from the URLs, and puts them in a JSON
request body instead. The server now maintains a simple set of messages
for each channel-id, and isn't responsible for removing duplicates.
The client now fetches all messages, and just ignores everything it sent
itself. This removes the "reflection attack".
Deallocate now returns JSON, for consistency. DB and API use "phase" and
"body" instead of msgnum/message.
This changes the DB schema, so delete the DB before upgrading the server.
The "bytes % bytes" syntax only appeared on py3.5, so don't use it.
Updated travis to expect py3.4 works.
The twisted side is probably even more broken for py3.4 than it is for
py3.5.
This roughly parallels the way that blocking/eventsource.py and the pypi
"requests" modules work: the server can set the encoding (with
"Content-Type: text/event-stream; charset=utf-8"), and the EventSource
parser will decode accordingly. However eventsource_twisted.py *always*
returns unicode (on both py2/py3), even when the server hasn't set an
encoding. blocking/eventsource.py returns bytes (on py3, and str on py2)
when the server doesn't set an encoding.
In the future, eventsource_twisted.py should return bytes when the
server doesn't set an encoding.
eventsource_twisted.py includes an alternate approach that might be
necessary (a to_unicode() function instead of always using .decode), but
I won't be sure until enough of Twisted has been ported to allow the
EventSourceParser to be tested.
Also fix demo.py for python3.
* declare transit records and handshake keys are bytes, not str
* declare transit connection hints to be str
* use six.moves.socketserver, six.moves.input for Verifier query
* argparse "--version" writes to stderr on py2, stdout on py3
* avoid xrange(), use subprocess.Popen(universal_newlines=True)
* use modern/portable "next(iter)" instead of "iter.next()"
* use six.moves.input() instead of raw_input()
* tell requests' Response.iter_lines that we want str, not bytes
The main wormhole code is str (unicode in py3, bytes in py2). Most
everything else must be passed as bytes in both py2/py3.
Keep the internal "side" string as a str, to make it easier to merge
with other URL pieces.
The twisted.python.logfile in Twisted-15.4.0 is not yet compatible with
py3, but can be bypassed by not daemonizing the server (so it doesn't
write to a logfile). This has been fixed in twisted trunk, so when
15.4.1 or 15.5.0 comes out, this will no longer be needed. But I think
we'll leave it in place, since sometimes it's handy to run a server
without daemonization.
Just make up a code like NUMBER-STUFF, and add --code= to the
send-text/send-file command. Also don't use tab-completion on the
codewords part of the receiving side, unless you stuck to the even/odd
PGP wordlist. (tab still works for the channel-id).
Now the server allocates a channel randomly from set of available ids
with the shortest possible length. So concurrency=1 will always yield a
channel-id between 1 and 9 (inclusive). If we have 9 simultaneous
sessions, we'll start allocating channels from 10 to 99. 100
simultaneous connections kicks us into the 100-999 bucket, etc.
This is a proxy for the other client's version, and encourages both
sides to upgrade to the current version each time the server is
upgraded (which will be once per release).
To be useful, both sides must add -v. If the sender uses -v but the
receiver doesn't, the receiver won't show the verification string, so
the sender can't compare it to anything (and must either abort the
transfer or accept it blindly). Maybe the receiver should show the
verification string unconditionally. Maybe the sender should
indicate (in unprotected plaintext, along with the PAKE message) whether
the receiver should show it or not.
We used to use twisted.python.usage.Options, hence we depended upon
Twisted. Now we depend upon "argparse" instead, which is in the py2.7
stdlib (and on pypi for 2.6). This package will still (eventually)
provide Twisted support, but applications which need it will already
express a dependency on twisted themselves, so by removing the
dependency here, we make life easier for applications that don't use it.
Applications should feel free to pass wormhole.const.RENDEZVOUS_RELAY
here, but I figure it should be clear that you're using a public service
that's hosted *somewhere* external.
This fixes the situation where you start the receiver first, then start
the sender, then you hit TAB on the receiver.
This somewhat improves the situation where you start the receiver first,
hit TAB (getting nothing), then start the sender, then hit TAB on the
receiver again. The second TAB will list the channel-ids, but won't
insert the only one as it's supposed to. You must type something (which
you can erase) and then hit TAB again to get a unique channel-id
inserted. But at least you can tell which one to type.
The first TAB runs the completer with readline.get_completion_type()
equal to 9=TAB=try-to-insert. The second (and subsequent) TABs use
63=?=list-matches, and it won't go back to 9 until you type something.
If all the direct hints resulted in timeouts (e.g. they were to bad IP
addresses where connections just hang), the relay connection would fail.
The establish_connection() function had the same TIMEOUT as the
direct-hint connector, so it would give up just before the relay
connection was initiated.
