I made the classic dataReceived() mistake, and exited the function after
delivering the first record. Keep at it until there are no complete
records left.
The previous commits improve test failures by dropping relay connections
at shutdown, and flunking a test quickly when one client fails but the
other one hangs.
If that doesn't work (say, some client has a time.sleep(), or other
stall that isn't affected by the relay shutdown), we'll be left with an
active thread holding that hanging client.
This patch adds a check to wormhole.test.common.ServerBase.tearDown that
looks for active threads, waits a second (after stopService), then
checks the threadpool again. If the threadpool is empty, everything is
fine. If not, it prints a message (to stdout) to inform the impatient
user why the test is probably hanging.
When test_scripts ran two clients at the same time, an error in one
could leave the other hanging (in a thread). One Deferred would errback,
the other would hang. Tests wait on one Deferred at a time, so if we're
unlucky and were waiting on the hanging Deferred (instead of the
erroring one), we'll wait forever, or at least until the default test
timeout of 180 seconds.
This adds an errback to notice when either client has errored, and
cancels the other Deferred, so it doesn't matter which one we wait upon
first.
'readline' is part of the python stdlib, so declaring a dependency on it
doesn't help. It doesn't exist on windows, and the pypi 'readline'
module doesn't work on windows. So instead, just attempt to import
readline, and if that fails, fall back to a non-completion flavor.
This ensures that we'll be ready for them. Previously there was a race
between us revealing the direct hints to the peer, and us setting the
transit key (thus allowing us to check inbound handshake requests). The
Transit instance didn't handle the race, causing errors to be thrown
when the other side connected quickly.
This ensures that we'll be ready for them. Previously there was a race
between us revealing the direct hints to the peer, and us setting the
transit key (thus allowing us to check inbound handshake requests). The
Transit instance handles this race (with an interlock on the transit
key), but it's still nicer to do it cleanly.
This exposed a new race in Transit, where the inbound connection would
complete before transit.connect() had been called. The previous commit
adds an interlock to wait for that too. Until this change, the transit
key lock was covering that one up.
Some tests failed to override --transit-helper, which meant they
intermittently talked to the real transit server (briefly, before
deciding the local+direct connection was better).
This limits the buffering to about 10MB (per connection*direction).
Previously, if the sender had more bandwidth than the receiver, the
transit relay would buffer the entire file. With this change, the sender
will be throttled to match the receiver's downstream speed.
The latest Twisted fixes the web.Agent code we need for proper async
support. There's still a daemonization bug that prevents 'wormhole
server start' from succeeding (it hangs).
I'm planning to leave non-EventSource "/get" in until after 0.6.0, then
remove it. I think it's cleaner for the logs to have the two
forms (EventSource and immediate) use different URLs.
In the twisted-style code, the close_on_error() decorator forces the
return value to be a Deferred, which is all wrong for internal uses of
derive_key() (verification string and confirmation message). It might be
useful to have a synchronous form of close_on_error(), but since the
actual close() is async, that's not very straightforward.
So for now, tolerate unclosed Wormhole objects when someone calls
derive_key() too early, or with a non-unicode type string.
This removes the 'allocations' table entirely, and cleans up the way we
prune old messages. This should make it easier to summarize each
connection (for usage stats) when it gets deallocated, as well as making
pruning more reliable.
Without this, old senders will throw a messy 404 traceback when talking
to a modern server.
Unfortunately 0.4.0 receivers don't make API calls in the right order,
so they throw a 404 before seeing our "you need to upgrade" message.
I was really confused about the Server-Sent Events syntax. This new one
is compatible with actual web browsers and the spec:
http://www.w3.org/TR/eventsource/
This requires a DB delete/recreate when upgrading. It changes the server
protocol, and app IDs, so clients cannot interoperate with each other
across this change, nor with the server. Flag day for everyone!
Now apps do not share channel IDs, so a lot of usage of app1 will not
cause the wormhole codes for app2 to get longer.
There are now three ways to invoke send:
* "wormhole send": ask for a text message
* "wormhole send FILENAME": send a file
* "wormhole send --text TEXT": send text message
This removes "side" and "msgnum" from the URLs, and puts them in a JSON
request body instead. The server now maintains a simple set of messages
for each channel-id, and isn't responsible for removing duplicates.
The client now fetches all messages, and just ignores everything it sent
itself. This removes the "reflection attack".
Deallocate now returns JSON, for consistency. DB and API use "phase" and
"body" instead of msgnum/message.
This changes the DB schema, so delete the DB before upgrading the server.
The "bytes % bytes" syntax only appeared on py3.5, so don't use it.
Updated travis to expect py3.4 works.
The twisted side is probably even more broken for py3.4 than it is for
py3.5.
This roughly parallels the way that blocking/eventsource.py and the pypi
"requests" modules work: the server can set the encoding (with
"Content-Type: text/event-stream; charset=utf-8"), and the EventSource
parser will decode accordingly. However eventsource_twisted.py *always*
returns unicode (on both py2/py3), even when the server hasn't set an
encoding. blocking/eventsource.py returns bytes (on py3, and str on py2)
when the server doesn't set an encoding.
In the future, eventsource_twisted.py should return bytes when the
server doesn't set an encoding.
eventsource_twisted.py includes an alternate approach that might be
necessary (a to_unicode() function instead of always using .decode), but
I won't be sure until enough of Twisted has been ported to allow the
EventSourceParser to be tested.
Also fix demo.py for python3.
* declare transit records and handshake keys are bytes, not str
* declare transit connection hints to be str
* use six.moves.socketserver, six.moves.input for Verifier query
* argparse "--version" writes to stderr on py2, stdout on py3
* avoid xrange(), use subprocess.Popen(universal_newlines=True)
* use modern/portable "next(iter)" instead of "iter.next()"
* use six.moves.input() instead of raw_input()
* tell requests' Response.iter_lines that we want str, not bytes
The main wormhole code is str (unicode in py3, bytes in py2). Most
everything else must be passed as bytes in both py2/py3.
Keep the internal "side" string as a str, to make it easier to merge
with other URL pieces.
