Commit Graph

283 Commits

Author SHA1 Message Date
Brian Warner
181ef04a91 break out more message components, use SidedMessage 2016-05-20 16:39:59 -07:00
Brian Warner
3b86571de3 fix py3 2016-05-20 12:12:07 -07:00
Brian Warner
390cd08b53 better command/response names: allocate+allocated, claim+claimed 2016-05-20 11:35:30 -07:00
Brian Warner
6c5b517ad1 hush 2016-05-20 11:10:17 -07:00
Brian Warner
ce06d379d9 remove old tests 2016-05-20 11:09:45 -07:00
Brian Warner
0a14901e94 full coverage of websocket 2016-05-20 11:08:10 -07:00
Brian Warner
f044ef0efa tests almost good 2016-05-19 23:50:22 -07:00
Brian Warner
335ed00cb7 build out tests 2016-05-19 19:55:11 -07:00
Brian Warner
2ea5d96290 Channels don't need "welcome" anymore 2016-05-17 17:35:44 -07:00
Brian Warner
5dd91c7311 test too-many-allocate, allocate+claim 2016-05-13 00:46:12 -07:00
Brian Warner
c4c0cf71eb add test 2016-05-13 00:43:59 -07:00
Brian Warner
1198977e06 SCHEMA CHANGE: channelids are now strs, not ints
This will enable the use of large randomly-generated hex or base32
channelids, for post-startup or resumed-connection channels.
2016-05-13 00:37:53 -07:00
Brian Warner
c14e982ae7 rendezvous: allow multiple channels per connection 2016-05-12 18:01:56 -07:00
Brian Warner
85dc0fd41b change server API: "release" instead of "deallocate" 2016-05-12 17:46:15 -07:00
Brian Warner
a34fb2a98b remove plain-HTTP (non-WebSocket) rendezvous server 2016-05-12 16:56:19 -07:00
Brian Warner
104ef44d53 provide wormhole() as a function, rather than a class constructor
You must always provide a reactor= argument. In the future, omitting the
reactor= argument is how you ask for a blocking Wormhole.
2016-05-12 16:45:54 -07:00
Brian Warner
d0ef53fc4d remove phase= from the Wormhole API
Phase are now implicit and numbered.
2016-05-12 16:45:54 -07:00
Brian Warner
501af4b4ec rename send_data/get_data to just send/get 2016-05-12 16:45:54 -07:00
Brian Warner
49785008bb remove blocking implementation: it will return
It will return as a crochet-based wrapper around the Twisted
implementation.
2016-05-12 16:45:54 -07:00
Brian Warner
5530c33185 rdv_ws: send acks for each message
but only if the client is modern enough to include "id" in the message,
which lets us avoid sending acks to an 0.7.5 client (which would cause
them to abort, they don't like unrecognized server messages).

The acks let the client learn the server_rx time of messages that
terminate on the server, like "allocate" and "claim".
2016-05-06 18:51:28 -07:00
Brian Warner
644c7c6840 DB schema change: add/store/return msgid
This enables dump-timing to correlate sender logs with receiver logs.
2016-05-06 18:29:58 -07:00
Brian Warner
8a80242532 rdv_ws: deliver "server_tx", not "sent"
Update twisted/transcribe.py to accept it, update tests.
2016-05-06 18:29:58 -07:00
Brian Warner
d2dfc325d8 deliver stored server_rx to clients
This enables the "dump-timing" tools to display more information.
2016-05-06 18:29:58 -07:00
Brian Warner
fe2dfc1a35 DB schema change: rename 'when' to 'server_rx' 2016-05-06 18:29:58 -07:00
Brian Warner
b70c2f8868 Make get_verifier() wait for _confirm to arrive
This improves the error behavior when --verify is used but there's a
WrongPasswordError: the mismatch is detected before the verifiers are
displayed or confirmation is requested.

It requires that the far end sends a "_confirm" message, which was
introduced in release 0.6.0. Use with older versions (if it doesn't
break for other reasons) will cause a hang.

This patch also deletes test_twisted.Basic.test_verifier_mismatch, since
both sides now detect this on their own. It changes
test_wrong_password() too, since we might now notice the error during
send_data (previously we'd only see it in get_data).
2016-04-25 19:01:55 -07:00
Brian Warner
2f6caa04d3 test_scripts.Cleanup: new test
This asserts that we deallocate the rendezvous channel, even when we
terminate early due to errors.
2016-04-25 17:53:00 -07:00
Brian Warner
34116c7b1f CLI: document and return correct errors
Also clean up test_scripts.PregeneratedCode:

* fetch results from both sides at the same time
* only check rc when using a subprocess, since the direct call doesn't
  use rc=0 anymore
* no need to cancel the other side's Deferred when one errors
* provide more information if stderr was non-empty
2016-04-25 17:52:59 -07:00
Brian Warner
4eaf88d7d2 cmd: remove _twisted suffix from function names
No longer necessary now that all commands are twisted-based.
2016-04-24 22:49:18 -07:00
Brian Warner
86edf96412 switch to tqdm for nicer CLI progress bars 2016-04-24 12:04:05 -07:00
Brian Warner
b2c3f49038 unsplit txwormhole step 2: edit files 2016-04-20 19:18:41 -07:00
Brian Warner
7c45f3d472 unsplit wormhole_server step 2: edit files 2016-04-20 19:15:33 -07:00
Brian Warner
85b6264826 unsplit cli step 2: edit files 2016-04-20 19:15:33 -07:00
Brian Warner
7a5a99a1a8 unsplit tests step 3: modify files 2016-04-20 18:54:10 -07:00
Brian Warner
fc33ea41ff unsplit tests step 2: delete test_load 2016-04-20 18:51:03 -07:00
Brian Warner
c890a850aa unsplit tests step 1: move files 2016-04-20 18:44:33 -07:00
Brian Warner
95706c752c move tests step 1: rename files 2016-04-18 16:45:03 -07:00
Brian Warner
36e63e2347 split wormhole_cli step 2: fix imports, setup.py 2016-04-18 16:43:18 -07:00
Brian Warner
1688da74d9 split txwormhole step 2: fix imports 2016-04-18 16:41:56 -07:00
Brian Warner
87266f1292 split wormhole_server step 3: fix tests 2016-04-18 16:18:51 -07:00
Brian Warner
1a455c05f5 remove wormhole.twisted.util port-allocation functions 2016-04-18 16:16:27 -07:00
Brian Warner
db137c26e5 remove blocking.transit 2016-04-18 15:52:29 -07:00
Brian Warner
4e937c2100 rename cmd_send_twisted to cmd_send, same for cmd_receive 2016-04-18 15:52:29 -07:00
Brian Warner
a1033b06a3 merge send_common into cmd_send_twisted 2016-04-18 15:52:29 -07:00
Brian Warner
8c67a98259 merge test_transit.py into test_server.py 2016-04-18 15:52:29 -07:00
Brian Warner
68b22bec97 remove --twisted, stop using blocking-flavor scripts 2016-04-18 15:52:18 -07:00
Brian Warner
049fac01db tolerate trailing slash on "wormhole send dirname/"
Previously, the trailing slash would cause the receiving side to get an
empty-named directory.
2016-03-24 08:46:29 -07:00
Brian Warner
626732b730 relay_server: include 'sent' timestamp with each message
This will be used by client-side timeline instrumentation, to guide
performance improvements.
2016-03-03 17:52:59 -08:00
Brian Warner
418fe9419e twisted: use persistent connections, slight speedup
This uses a single TCP connection to the relay server for all
requests (although it probably uses a second one for the downstream
EventSource feed). This should squeeze out some of the round-trip times.
2016-03-03 13:39:09 -08:00
Brian Warner
df2384bea2 twisted.transit: move FileConsumer into RecordPipe
This adds an expected= argument to Connection.connectConsumer(), which
then returns a Deferred that fires when enough bytes have been written
to the consumer. It also adds Connection.writeToFile(), a helper method
that writes bytes to a filehandle.
2016-03-02 00:48:43 -08:00
Brian Warner
7234e25897 twisted.transit: handle multiple records in one chunk
I made the classic dataReceived() mistake, and exited the function after
delivering the first record. Keep at it until there are no complete
records left.
2016-03-02 00:48:43 -08:00
Brian Warner
8d82726c51 add DebugTiming object, --dump-timing= option
This writes timeline data to a .json file, which can be examined later
to find likely candidates for optimization.
2016-03-01 18:23:06 -08:00
Brian Warner
84def8a54b add some inlineCallbacks for simplicity
This control flow was getting too hairy.
2016-03-01 18:22:03 -08:00
Brian Warner
fd143caded tests.ServerBase: print message when threadpool is not idle
The previous commits improve test failures by dropping relay connections
at shutdown, and flunking a test quickly when one client fails but the
other one hangs.

If that doesn't work (say, some client has a time.sleep(), or other
stall that isn't affected by the relay shutdown), we'll be left with an
active thread holding that hanging client.

This patch adds a check to wormhole.test.common.ServerBase.tearDown that
looks for active threads, waits a second (after stopService), then
checks the threadpool again. If the threadpool is empty, everything is
fine. If not, it prints a message (to stdout) to inform the impatient
user why the test is probably hanging.
2016-03-01 17:07:48 -08:00
Brian Warner
b1dae14e6d test_scripts: handle hangs in one client
When test_scripts ran two clients at the same time, an error in one
could leave the other hanging (in a thread). One Deferred would errback,
the other would hang. Tests wait on one Deferred at a time, so if we're
unlucky and were waiting on the hanging Deferred (instead of the
erroring one), we'll wait forever, or at least until the default test
timeout of 180 seconds.

This adds an errback to notice when either client has errored, and
cancels the other Deferred, so it doesn't matter which one we wait upon
first.
2016-03-01 17:07:43 -08:00
Brian Warner
c225d57f8c cmd_receive_twisted.py: twisted-based implementation 2016-02-29 07:31:34 -08:00
Brian Warner
01064325a2 cmd_receive: refactor (slight message changes)
This prepares the way for a twisted-based implementation.
2016-02-29 07:30:40 -08:00
Brian Warner
6654efb429 move describe() from Transit to RecordPipe 2016-02-28 01:42:46 -08:00
Brian Warner
ca06e95bbd test_blocking: avoid using real hints 2016-02-27 17:29:04 -08:00
Brian Warner
1ff0792b32 test_scripts: prep for windows compatibility
newlines, os.mkfifo errors, JSON type differences
2016-02-27 14:46:38 -08:00
Brian Warner
e6fba34570 start on twisted receive: move old files into new homes 2016-02-17 19:25:19 -08:00
Brian Warner
aa27bfd32c add twisted form of sender
Currently this is only invokable from tests.
2016-02-17 19:02:36 -08:00
Brian Warner
369854b4cc test build_phase1_data 2016-02-17 18:07:35 -08:00
Brian Warner
79decea9ea rearrange scripts: make room for twisted 2016-02-17 16:26:03 -08:00
Brian Warner
5d572137f4 improve test error message 2016-02-17 14:00:03 -08:00
Brian Warner
fd7d1b97d4 test scripts in a thread, not a subprocess, mostly 2016-02-17 14:00:01 -08:00
Brian Warner
903129f4a2 add --hide-progress, mostly for tests 2016-02-17 13:58:41 -08:00
Brian Warner
e2f3bebe38 allow --relay-helper="" to disable relay
test_scripts now uses this to avoid accidentally using a relay
2016-02-17 13:53:18 -08:00
Brian Warner
5e928ac9f0 rewrite ProgressPrinter as a class, add tests 2016-02-17 12:46:12 -08:00
Brian Warner
00833a4bde test_scripts: factor out common test code 2016-02-17 10:22:31 -08:00
Brian Warner
a235b507c8 twisted.transit: implement producer/consumer flow control 2016-02-15 21:23:20 -08:00
Brian Warner
763d72f582 twisted.transit: implement Deferred-based receive_record() 2016-02-15 11:40:21 -08:00
Brian Warner
fb1461fa8c add twisted.transit, with tests 2016-02-14 17:57:09 -08:00
Brian Warner
36cc0e2bca tests: don't use network
Some tests failed to override --transit-helper, which meant they
intermittently talked to the real transit server (briefly, before
deciding the local+direct connection was better).
2016-02-12 17:11:06 -08:00
Brian Warner
bd912a73ee Transit: blur the file sizes too 2016-01-12 16:32:38 -08:00
Brian Warner
9ed39be346 don't log HTTP requests when blur-usage is on 2015-12-04 17:35:56 -08:00
Brian Warner
1ab66d2fd0 privacy: only store coarse timestamps in the usage table 2015-12-03 21:15:19 -08:00
Brian Warner
909cdfa3dc track transit usage in DB 2015-12-03 19:45:34 -08:00
Brian Warner
6369c17595 test_transit: use better error in Accumulator
apparently this was broken before, but never got triggered
2015-12-03 19:25:40 -08:00
Brian Warner
568903ac48 add test for Transit (blocking only) 2015-12-03 16:24:52 -06:00
Brian Warner
c103441648 test_transit: fix py3.3, no %s in bytestrings 2015-12-02 03:02:50 -06:00
Brian Warner
8240d9e910 add test of transit server, make it work under py3
Also have transit send logs to log.msg, not stdout.
2015-12-02 01:47:52 -06:00
Brian Warner
80603aaa32 finish py3/async support, needs Twisted >= 15.5.0
The latest Twisted fixes the web.Agent code we need for proper async
support. There's still a daemonization bug that prevents 'wormhole
server start' from succeeding (it hangs).
2015-12-01 00:15:24 -06:00
Brian Warner
5b46df133f implement (and test) --output-file for receive-file 2015-11-29 01:40:25 -06:00
Brian Warner
3a343f9895 test send/receive directory 2015-11-29 01:33:15 -06:00
Brian Warner
2318c94169 test_server: error properly when EventSource fails 2015-11-23 16:49:46 -08:00
Brian Warner
c482c248ff server: add "watch" endpoint, deprecate non-ES "get"
I'm planning to leave non-EventSource "/get" in until after 0.6.0, then
remove it. I think it's cleaner for the logs to have the two
forms (EventSource and immediate) use different URLs.
2015-11-23 16:43:25 -08:00
Brian Warner
badf5168ef test_server: rename some functions 2015-11-22 18:06:42 -08:00
Brian Warner
22a1ce2eda add close-with-mood-on-error to twisted style too 2015-11-19 17:08:21 -08:00
Brian Warner
9827a2e50c add twisted/blocking interop test 2015-11-19 16:21:10 -08:00
Brian Warner
1ad001bbc3 WIP: test that we tolerate missing key-confirmation messages 2015-11-16 18:25:28 -08:00
Brian Warner
6b57d7d05d check key-confirmation messages, if present 2015-11-16 18:24:39 -08:00
Brian Warner
ae2a6c6a05 add Channel.get_first_of()
This allows the Wormhole code to wait for multiple messages, which will
be useful for getting Confirmation messages soon.
2015-11-16 16:47:52 -08:00
Brian Warner
b709a45891 get_data/set_data: reserve _ for internal uses 2015-11-16 16:20:00 -08:00
Brian Warner
47d3eee6fe server: treat missing moods (from older clients) as "quiet" 2015-11-15 10:34:40 -08:00
Brian Warner
4f0dde9529 server: summarize transfers, store in DB 2015-11-15 10:34:29 -08:00
Brian Warner
b83062701d server: give old 0.4.0 senders a "you must upgrade" error
Without this, old senders will throw a messy 404 traceback when talking
to a modern server.

Unfortunately 0.4.0 receivers don't make API calls in the right order,
so they throw a 404 before seeing our "you need to upgrade" message.
2015-11-11 17:06:33 -08:00
Brian Warner
595a0e5845 server API: include "mood" when closing the channel
This will be used as a simple unverifiable counter of success/failure.
2015-10-08 18:35:26 -07:00
Brian Warner
0a6ab83bc5 test distinctness of encryption nonces 2015-10-07 16:31:45 -07:00
Brian Warner
df3aee2a86 fix EventSource (server and parsers)
I was really confused about the Server-Sent Events syntax. This new one
is compatible with actual web browsers and the spec:

 http://www.w3.org/TR/eventsource/
2015-10-07 16:31:45 -07:00
Brian Warner
e77b39313a use TypeError for type errors, not UsageError 2015-10-06 20:52:47 -07:00
Brian Warner
ce236ae70c test_blocking: stop aliasing Wormhole 2015-10-06 20:44:32 -07:00
Brian Warner
bf43dae2ad add multiple phases, change key-derivation strings
Because of the key-derivation change, clients will not be compatible
across this commit.
2015-10-06 20:39:20 -07:00
Brian Warner
35768d6738 wormhole/invitation code is now unicode 2015-10-06 19:42:10 -07:00
Brian Warner
574d5f2314 scope channelids to the appid, change API and DB schema
This requires a DB delete/recreate when upgrading. It changes the server
protocol, and app IDs, so clients cannot interoperate with each other
across this change, nor with the server. Flag day for everyone!

Now apps do not share channel IDs, so a lot of usage of app1 will not
cause the wormhole codes for app2 to get longer.
2015-10-06 19:21:53 -07:00
Brian Warner
9e1a00cbd9 appid and derive_key(purpose) are now unicode 2015-10-06 19:21:53 -07:00
Brian Warner
9ba7de6e1e relay-url is now unicode 2015-10-06 19:19:39 -07:00
Brian Warner
fc30fa6cd4 rename channel-id to channelid. changes DB schema. 2015-10-06 19:12:41 -07:00
Brian Warner
0fa0abfbb1 server: split into separate files 2015-10-04 12:40:12 -07:00
Brian Warner
00caa9f6d9 oops, update tests to match 2015-10-03 23:27:28 -07:00
Brian Warner
e7e5c2d079 merge send/receive "file" and "text" into a single command 2015-10-03 22:45:43 -07:00
Brian Warner
f24e05d4cb new Wormhole API: separate send_data()/get_data() calls 2015-10-03 22:03:27 -07:00
Brian Warner
617bb03ad5 rewrite server API
This removes "side" and "msgnum" from the URLs, and puts them in a JSON
request body instead. The server now maintains a simple set of messages
for each channel-id, and isn't responsible for removing duplicates.

The client now fetches all messages, and just ignores everything it sent
itself. This removes the "reflection attack".

Deallocate now returns JSON, for consistency. DB and API use "phase" and
"body" instead of msgnum/message.

This changes the DB schema, so delete the DB before upgrading the server.
2015-10-03 17:38:31 -07:00
Brian Warner
429c5cd962 tests: clean up pairs-of-Deferreds patterns 2015-09-28 16:52:12 -07:00
Brian Warner
8d3ed79ce6 add verifier tests 2015-09-28 16:49:36 -07:00
Brian Warner
5ae1c2d020 tests: switch to defer.gatherResults 2015-09-28 16:44:00 -07:00
Brian Warner
336eea5e78 tests: handle argparse on py3.3
which sends --version to stderr, not stdout. This might make the py3.3
tests pass.
2015-09-28 16:31:35 -07:00
Brian Warner
1522658c9b skip test_twisted on py3 until more of Twisted has been ported 2015-09-28 00:45:33 -07:00
Brian Warner
b5d470fcda make blocking/send-file work on py3
* declare transit records and handshake keys are bytes, not str
* declare transit connection hints to be str
* use six.moves.socketserver, six.moves.input for Verifier query
* argparse "--version" writes to stderr on py2, stdout on py3
* avoid xrange(), use subprocess.Popen(universal_newlines=True)
2015-09-28 00:24:36 -07:00
Brian Warner
a7213d9c9a enforce bytes-vs-str in the API
The main wormhole code is str (unicode in py3, bytes in py2). Most
everything else must be passed as bytes in both py2/py3.

Keep the internal "side" string as a str, to make it easier to merge
with other URL pieces.
2015-09-28 00:24:36 -07:00
Brian Warner
15cc0a1429 test_server: make sure the server is reachable
used to exercise py3 issues with the server
2015-09-28 00:24:00 -07:00
Brian Warner
5d93dccb88 appid and derive_key(purpose=) must be bytes, not unicode 2015-09-28 00:24:00 -07:00
Brian Warner
2e2bd1bb5f tests: split blocking out to its own file
also clean up the output when pip-e -vs- entrypoint -vs- Versioneer
makes the entrypoint script refuse to run from a version mismatch.
2015-09-27 10:43:25 -07:00
Brian Warner
adf55175fb test_scripts: wormhole should live next to python
not necessarily beneath the current working directory
2015-09-26 18:29:46 -07:00
Brian Warner
d3ef3aa29a more verbose, to figure out travis failure 2015-09-26 18:21:56 -07:00
Brian Warner
5ee37cab64 test_scripts: make found-wrong-wormhole error more verbose 2015-09-26 18:17:50 -07:00
Brian Warner
b069e69d35 test scripts too (with spawnToThread) 2015-09-26 18:02:02 -07:00
Brian Warner
1e6fc4601e tests: split common code out 2015-09-26 18:02:02 -07:00
Brian Warner
88dab265de test blocking code, using twisted+deferToThread
Unfortunately Twisted still requires python2, so we can't use this to
test the intended python3-compatibility of the blocking code.
2015-09-21 23:21:40 -07:00
Brian Warner
efd6d27cc6 rename SymmetricWormhole to just "Wormhole"
Update docs too. Now both blocking/ and twisted/ use "Wormhole".
2015-07-24 17:47:46 -07:00
Brian Warner
cebfa71563 minor reformatting, improve test error messages 2015-07-24 16:57:19 -07:00
Brian Warner
dc65b4354d twisted: split allocate_ports() out to util.py
so it can be used by downstream projects
2015-06-21 21:04:33 -07:00
Brian Warner
25472423c6 make twisted work, get serialization into shape, add proper tests 2015-06-20 19:18:21 -07:00
Brian Warner
c14749fae0 add first unit test, 'setup.py test' support 2015-02-10 01:05:15 -08:00