Commit Graph

606 Commits

Author SHA1 Message Date
Brian Warner
f192db5412 track+report mailbox.for_nameplate in usage
This counts the number of "standalone" mailboxes we create, which
happens when a client does open() without first using a nameplate. The
current client doesn't do this, but future clients might.
2016-06-26 11:19:53 -07:00
Brian Warner
db8511c700 schema: add mailbox_usage.for_nameplate too 2016-06-26 11:18:48 -07:00
Brian Warner
bae020d5a7 DB: add mailboxes.for_nameplate
Also add a few indexes to make get_stats() less expensive
2016-06-26 11:01:52 -07:00
Brian Warner
8eb1396f31 server: delete stats file at reboot
This makes it easier to discover an exception in the stats-gathering
timer, since you get a missing stats.json instead of a stale one.
2016-06-26 11:01:52 -07:00
Brian Warner
7b1d4cbcee more stats: count since last reboot 2016-06-26 11:01:52 -07:00
Brian Warner
9e29898207 Rendezvous: add many stats 2016-06-26 11:01:48 -07:00
Brian Warner
62f9a4d702 move stats-file up to RelayServer
This moves responsibility for the periodic prune-everything Timer up to
RelayServer too. That way we can be sure the stats are dumped
immediately after prune, and we can incorporate stats from Transit as
well.
2016-06-25 11:43:13 -07:00
Brian Warner
7f389dc76e server: add --stats-file= scaffolding
no actual stats yet
2016-06-25 11:05:28 -07:00
Brian Warner
cfcd74db4b remove listener on close too
This was failing to clean up after WrongPassword disconnects.
2016-06-24 18:57:49 -07:00
Brian Warner
6a2cbf9014 server: remove listeners on disconnect
This wasn't happening before, so channels were staying alive until
reboot.
2016-06-24 18:48:28 -07:00
Brian Warner
ffb1a9b9c9 change pruning algorithm
The new approach runs every 10 minutes and keeps a
nameplate/mailbox/messages "channel" alive if the mailbox has been
updated within 11 minutes, or if there has been an attached listener
within that time.

Also remove the "nameplates.updated" column. Now we only track "updated"
timestamps on the "mailboxes" table, and a new mailbox will preserve any
attached nameplate.
2016-06-24 17:35:23 -07:00
Brian Warner
6176500cf4 factor out Mailbox._touch 2016-06-24 16:35:30 -07:00
Brian Warner
0158df9b63 stop using is_active(), or pruning inactive apps
Unless/until people start writing new applications (with different
app-ids), this code is unlikely to get used very much, and the code is
simpler without it.
2016-06-24 16:31:00 -07:00
Brian Warner
08443ee288 rename Rendezvous.prune() to prune_all_apps
so grep can distinguish it from Mailbox.prune()
2016-06-24 16:30:55 -07:00
Brian Warner
b8521e81b1 add comment about new pruning plan 2016-06-24 16:26:42 -07:00
Brian Warner
3c0f832bdc test that mailbox timestamp is updated correctly 2016-06-24 16:25:44 -07:00
Brian Warner
550b9e3c94 db: add v2->v3 upgrader 2016-06-24 16:19:39 -07:00
Brian Warner
404925d314 server: mailbox row should always exist
This makes the nameplate's "mailbox_id" into a foreign-key.
2016-06-24 16:02:37 -07:00
Brian Warner
41f229de87 use 'mailbox_sides' instead of cols in 'mailboxes' 2016-06-24 00:01:51 -07:00
Brian Warner
6c725e4a86 db: use 'nameplate_sides' instead of cols in 'nameplates' 2016-06-23 18:27:26 -07:00
Brian Warner
355ece3e47 db: turn on foreign keys 2016-06-23 18:27:26 -07:00
Brian Warner
74a1902fcd schema: add v3, rename nameplates.id to .name 2016-06-23 18:27:26 -07:00
Brian Warner
6d00576ee4 server-stop: ignore missing twistd.pid
I changed my mind, it's actually easier if 'wormhole-server stop' (and
'restart') does *not* throw an error when there wasn't already a server
running in that directory. Specifically that lets me use 'restart' as an
idempotent "make sure a server is running" command.
2016-06-23 18:25:58 -07:00
Brian Warner
fa29789769 test_scripts: re-enable tests on py3
These weren't running because Click complained about an ASCII locale
when running under py3, which triggered an error check that was there to
detect broken virtualenvs, skipping those tests.

The fix appears to be to force the en_US.UTF-8 locale when running the
wormhole program in a subprocess.
2016-06-22 19:28:17 -07:00
Brian Warner
177aac2d84 test upgrader, badly
This adds a test for database upgrades, which I developed on a branch
that added a new DB schema (v3) and an upgrader to match, but then I
changed my mind about the schema and removed that part. The test will be
useful some time in the future when I change the schema in a small
enough way that I bother to write an upgrader for the change. For now,
the test is disabled.

In addition, the upgrader test is kind of lame. I'd really prefer to
assert that the upgraded schema is identical to the schema of a
brand-new (latest-version) database, but ALTER TABLE doesn't quite work
that way (comments are omitted, and the order of the columns is slightly
different).

This also adds database.dump_db() for the tests.
2016-06-22 19:05:52 -07:00
Brian Warner
7250deb94c add schema upgrades 2016-06-22 19:05:48 -07:00
Brian Warner
b613ac1b3b cli.py: add comment 2016-06-22 13:15:07 -07:00
Brian Warner
60b40c2de2 server/cli.py: remove timing instrumentation
I don't anticipate ever caring how long it takes to import code on the
server side. Only the client's performance is interesting to me.
2016-06-22 13:12:45 -07:00
Brian Warner
d8c2c1249e server/cli.py: fix port numbers
There was some vestigal server-cli code (leftover in the client-side
wormhole.cli.cli_args) that used port 3000/3001, and it accidentally got
used for the new Click-based parser, rather than the actual server-cli
code (in wormhole.server.cli_args) that uses port 4000/4001. This
changes the port numbers to match (everything uses 4000/4001 these days,
to avoid confusing interactions with the old 0.7.6 server that might
still be listening on the old ports).
2016-06-22 13:12:34 -07:00
Brian Warner
940aec7570 cli.py: improve parallelism between send+receive 2016-06-22 13:11:43 -07:00
Brian Warner
b1f994e939 transit.py: improve error 2016-06-22 13:10:46 -07:00
meejah
7fab6b3dff Change UsageError -> InternalError, use click.UsageError for human-visible errors 2016-06-22 02:14:34 -06:00
meejah
6b29d62d78 typo 2016-06-22 01:29:13 -06:00
meejah
03594a2d0e consistent return usage 2016-06-22 01:28:53 -06:00
meejah
ad6200c474 resolve XXX comments 2016-06-22 01:27:29 -06:00
meejah
c4cd4db30e Correct exit code when no twistd.pid file 2016-06-22 01:25:47 -06:00
meejah
e16b53817e Refactor to use Click 2016-06-22 01:11:07 -06:00
Brian Warner
d89fbd69dd sketching out a click-based CLI parser 2016-06-22 01:02:09 -06:00
meejah
07b4067727 unit-test for relay request, and fix Data() handling 2016-06-20 16:32:07 -07:00
laharah
2a056059c0 fix for wormhole-server count-* functions
broken when we switched to dictfactory on db
2016-06-04 16:50:56 -07:00
laharah
150caedf73 u"" cleanup on test_scripts.py 2016-06-04 14:09:19 -07:00
laharah
bb970dafc0 u"" cleanup on test_server.py 2016-06-04 14:09:19 -07:00
laharah
2dd76f57c0 u"" cleanup on test_transit.py 2016-06-04 14:09:19 -07:00
laharah
aff13e9121 u"" cleanup on test_util.py 2016-06-04 14:09:19 -07:00
laharah
8fa40d3cd5 u"" cleanup on test_wormhole.py 2016-06-04 14:09:19 -07:00
laharah
255fb12f88 `u"" cleanup on cmd_usage.py 2016-06-04 14:09:19 -07:00
laharah
57d117e4f0 u"" cleanup on rendezvous_websocket.py 2016-06-04 14:09:19 -07:00
laharah
8537d90527 u"" cleanup on rendezvous.py 2016-06-04 14:09:19 -07:00
laharah
2c38795476 u"" cleanup on transit_server.py 2016-06-04 14:09:19 -07:00
laharah
64f7348d05 u"" cleanup on util.py and codes.py 2016-06-04 14:09:19 -07:00
laharah
2c25b4d97d u"" cleanup on wormhole.py 2016-06-04 14:09:19 -07:00
laharah
6a73d50fdd added unicode_literals import to all apropriate modules
bug in twisted serverFromString prevents test.common and transit
from using unicode properly should revisit if twisted gets patched
2016-06-04 12:42:59 -07:00
laharah
33758abd18 added a dict_factory as a new row_factory for the database 2016-06-04 12:28:09 -07:00
Brian Warner
7c15cf7353 code input: don't eat "b" under GNU readline
GNU libreadline, and the libedit-based library shipped on stock OS-X
python, require different key-binding syntaxes to enable tab completion.
The previous commit to fix this (0977ef0) added both binding commands
Unfortunately when GNU libreadline is given the libedit-style
command (i.e. "bind ^I rl_complete"), it binds the letter "b" to a
non-existent command "ind", or something, and as a result the letter "b"
doesn't work anymore.

This patch uses the readline docstring to sense which flavor is
installed, and only runs the one binding command that's appropriate.

refs #37
2016-06-04 00:52:27 -07:00
Brian Warner
996c739b2a oops, fix tests
In some tests, we artificially call _event_connected() without first
initiating a connection.
2016-06-03 23:30:31 -07:00
Brian Warner
a1ce0d8df0 record websocket establishment in timing data 2016-06-03 22:55:52 -07:00
Brian Warner
e13b95660b test_transit: tolerate a localhost-only system
e.g. Appveyor . Refs #16.
2016-06-03 21:46:56 -07:00
Brian Warner
682fe0ae2c transit: allow 127.0.0.1 if that's all we've got
The appveyor tests were failing because their VMs only have 127.0.0.1,
and stripping it out resulted in an empty hint list, which meant Transit
couldn't work at all.
2016-06-03 18:22:53 -07:00
higs4281
55f2fcc3a7 remove commented line 2016-06-03 18:18:34 -07:00
higs4281
eaed2f0a12 Prevent transmit from suggesting 127.0.0.1
The test runs the listener locally, which can turn up 127.0.0.1.
The added code in transmit stops the hint.
2016-06-03 18:18:34 -07:00
Brian Warner
26445c3fc4 test_scripts: fix on windows 2016-06-03 16:43:22 -07:00
Brian Warner
a8c2fb14b1 expire channels faster: 2 hours, not 3 days
With increased usage, I'm seeing a buildup of stale channels. Since the
channels aren't properly ephemeral yet (where they get closed as soon as
the last subscriber disconnects), clients which terminate without
calling close() tend to leave the channel lying around. We don't have
"persistent wormholes" yet, so channels should be much more ephemeral
than they currently are.
2016-06-03 16:23:28 -07:00
Brian Warner
48cc85e88c add file-mode checks to directory test
This new test failed before fixing _extract_file, and now it passes.
2016-06-03 16:05:51 -07:00
Brian Warner
71512809a9 extract _extract_file, add test 2016-06-03 15:38:49 -07:00
Kurt Neufeld
e6f5b9cea4 verify that extracted files are inside abs_destname
also fixed bug where TwistedReceiver.abs_destname was not in fact
absolute.
2016-06-03 14:32:52 -07:00
Kurt Neufeld
b04e434ad4 restore file permissions when extracting zipfile 2016-06-03 12:36:11 -07:00
Kurt Neufeld
608af12b1d verified that ZIP_DEFLATED compresses the files
The docs are a bit misleading but that's how I interpret them.
2016-06-03 11:53:45 -07:00
Brian Warner
0977ef02c4 do tab-completion on OS-X stock python too (libedit)
Apple's stock python doesn't use GNU libreadline, instead it uses BSD
libedit with a readline compatibility interface. The syntax to enable
tab completion is different for libedit. By including both bindings,
autocomplete should work on both flavors.

Closes #37. Thanks to @wsanchez for the catch and the fix.
2016-06-02 20:55:02 -07:00
meejah
5b23669cf3 use ipaddress not ipaddr 2016-06-02 16:20:12 -07:00
Chris Wolfe
0ad8df3750 flush the correct error 2016-06-02 15:21:31 -07:00
Chris Wolfe
65713d8652 make the error message for the cli a bit more specific 2016-06-02 15:21:31 -07:00
Chris Wolfe
bc7dea8bab add simpler exception message, assert that it is being used 2016-06-02 15:21:31 -07:00
Chris Wolfe
5be436b81d raise a specific error when spaces are detected, pass along the docstring to the user 2016-06-02 15:21:31 -07:00
Chris Wolfe
d4d3320277 add failing test, error 2016-06-02 15:21:31 -07:00
Brian Warner
00277c22cf allow MOTD to be displayed multiple times
(one displayed message per received welcome["motd"])

There's not much value in prohibiting the server from sending multiple
MOTD messages, and it would prevent us from using it to display a "your
client is using an old API, please upgrade" message after having already
sent a regular "please donate" MOTD message. (We could send a second
welcome message with ["error"] to kill the client, but ["motd"] is the
most convenient way to deliver a non-fatal warning).
2016-05-28 19:19:22 -07:00
Brian Warner
ea6619bc46 change hostname of public relay
This is an alias for the same host, so it's not really an incompatible
change. The new hostname is my personal domain, and seems a bit more
suitable for this service.
2016-05-28 19:13:44 -07:00
Brian Warner
52e5cbd690 INCOMPATIBILITY: send "current_cli_version", not "current_version"
The reasoning is that this string is only ever likely to refer to the
version of the primary/initial client (the CLI application, written in
Python, that you get with "pip install magic-wormhole"). When there are
other implementations, with unrelated versions, they should obviously
not pay attention to a warning about the other implementation being out
of date.
2016-05-28 19:11:27 -07:00
Brian Warner
0b53094927 INCOMPATIBILITY: send pake message as dict, not raw bytes
This gives us room in the future to put other keys there, like one which
says we want to use Noise for the phase-message encryption instead of
our current HKDF scheme.
2016-05-28 18:30:36 -07:00
Brian Warner
6a108f93e6 switch most everything to use new utility functions 2016-05-28 18:19:45 -07:00
Brian Warner
3850c164f7 move utility functions to util.py, add new ones 2016-05-28 18:13:32 -07:00
Brian Warner
0d30101d7c SCHEMA CHANGE: add 'request_id' to nameplates table
This will be useful for the upcoming "persistent wormhole" mode. A
client might send an allocation request, crash/terminate before
receiving a response, then restart, then re-send the request. If the
server sees a request with the same request_id a previous request, it
can return the same nameplate.

We'll need code changes on both sides to support this (nothing sends or
checks request_id yet), but this lands the schema change early to reduce
future disruption.
2016-05-28 11:46:14 -07:00
Brian Warner
cca2f3cf13 internal rename msg_id, for consistency 2016-05-28 11:19:53 -07:00
Brian Warner
00efdfa4e2 comment out unfinished function
at this point, it's just an idea
2016-05-27 18:47:14 -07:00
Brian Warner
5fe7d320ab oops, pass blur_usage to apps properly, add tests 2016-05-27 18:44:41 -07:00
Brian Warner
eebc9ebd54 rewrite pruning, add full tests
Apparently it was broken: the first time the LoopingCall fired, it would
throw an exception, and never try again. Now it should be fixed.
2016-05-27 18:42:17 -07:00
Brian Warner
0b3863fb52 remove dead code 2016-05-27 18:41:34 -07:00
Brian Warner
5dddeddffd fix "wormhole-server restart" default ports to match plain "start"
now that was confusing.
2016-05-27 18:40:37 -07:00
Brian Warner
1c963170c3 wormhole-server: add count-channels, count-events 2016-05-26 17:45:41 -07:00
Brian Warner
56fec496e2 send transit abilities to far end too, for the future
This will allow a future peer to figure out what transit modes we can
and cannot do, and thus avoid spinning up expensive modes that we won't
be able to use (e.g. WebRTC).
2016-05-26 16:26:00 -07:00
Brian Warner
db31ffe075 minor shuffles 2016-05-26 15:49:45 -07:00
Brian Warner
7acb68d2ee allow WelcomeError to interrupt input_code() 2016-05-26 15:38:19 -07:00
Brian Warner
905ec05696 fix welcome-error signalling
The handler was calling _signal_error() incorrectly.
2016-05-26 15:37:24 -07:00
Brian Warner
80c88387ef improve formatting of error messages 2016-05-26 15:36:44 -07:00
Brian Warner
19702e01ac make send pay attention to receiver errors correctly
Errors like "Refusing to overwrite existing file" would leave the sender
hanging, because it was looking for the "error" key in the wrong place.
2016-05-25 22:44:18 -07:00
Brian Warner
de627ccbef fix --dump-timing= on py3, wants a text-mode file 2016-05-25 22:06:26 -07:00
Brian Warner
708bcf36d4 INCOMPATIBILITY: send+expect hash of data after xfer
This enhances the ACK that wormhole-receive returns when it finishes
receiving all the data to be a dictionary. The dict includes the SHA256
hash of everything it received, and the sender checks this for a match
before declaring the transfer to be a success. This guards against data
being shuffled somehow during transit.
2016-05-25 19:36:56 -07:00
Brian Warner
d8f6126916 transit.writeToFile: add hasher= argument 2016-05-25 19:35:50 -07:00
Brian Warner
01ec5fb2a6 rename test_transit_twisted.py, drop the twisted 2016-05-25 19:21:54 -07:00
Brian Warner
75877effef finish renaming "confirm" to "version" 2016-05-25 19:13:56 -07:00
Brian Warner
dcdddc9d60 INCOMPATIBILITY: rename "confirm" phase to "version"
This better reflects the purpose of the message. Key confirmation is a
side-effect.

This patch only changes the "phase:" name and the key-derivation string.
A subsequent patch will modify the function and variable names to match.
2016-05-25 19:13:56 -07:00