magic-wormhole/NEWS.md

User-visible changes in "magic-wormhole":

Upcoming Release

• Python 3.5 and 3.6 are past their EOL date and support is dropped (#448)

Release 0.12.0 (04-Apr-2020)

• A command like wormhole send /dev/fd0 can send the contents of the named block device (USB stick, SD card, floppy, etc), resulting in a plain file on the other side. (#323)
• Change "accept this file?" default answer from no to yes. (#327 #330 #331)
• Actually use tempfile for large directory transfers. This fixes a five-year old bug which prevents transfers of directories larger than available RAM by finally really building the temporary zipfile on disk. (#379)
• Accept 'wss' for TLS-protected relay connections, which default to port 443 if no other port is accepted. A future release will change the public relay to use TLS. (#144)
• Drop support for python3.4
• Stall --verify long enough to send the verifier. This fixes a bug when both sides use --verify, the receiver uses tab-completion, the sender sees the verifier and waits for the user to confirm, but the receiver cannot show the verifier (enabling that confirmation) until the sender approves the transfer. (#349)

This release also includes an incomplete implementation of the new "Dilation" API (see ticket #312 for details). In the future this will enable restarting interrupted transfers, tolerating changes in network address, bidirectional transfers in a long-running GUI/daemon process, and more. The protocol is not finalized, nor is it backward compatible with the old "Transit" protocol yet, so there is no CLI access so far. The code is present and tested to make sure it doesn't regress and for ease of development, but intrepid folks who want to try it out will need to write a client first (and be aware that the protocol may change out from under them). A future release will add compatibility negotiation with old clients and start using the new protocol.

PRs and tickets addressed in this release: #144 #312 #318 #321 #323 #327 #330 #331 #332 #339 #349 #361 #365 #368 #367 #378 #379.

Thanks to the many contributors of bugs, patches, and other help with this release:

• Adam Spiers aka @aspiers
• Евгений Протозанов aka @WeirdCarrotMonster
• Edward Betts aka @EdwardBetts
• Jacek Politowski aka @jpolnetpl
• Julian Stecklina aka @blitz
• Jürgen Gmach aka @jugmac00
• Louis Wilson aka @louiswins
• Miro Hrončok aka @hroncok
• Moritz Schlichting aka @morrieinmaas
• Shea Polansky aka @Phyxius
• @sneakypete81

Release 0.11.2 (13-Nov-2018)

Rerelease to fix the long description on PyPI. Thanks to Marius Gedminas for tracking down the problem and providing the fix. (#316)

Release 0.11.1 (13-Nov-2018)

• Fix python -m wormhole on py2. (#315)

Thanks to Marius Gedminas, FreddieHo, and Jakub Wilk for patches and bug reports in this release.

Release 0.11.0 (16-Oct-2018)

• Python-3.7 compatibility was fixed. (#306)
• Support for Python-3.4 on Windows has been dropped. py3.4 is still supported on unix-like operating systems.
• The client version is now sent to the mailbox server for each connection. I strive to have the client share as little information as possible, but I think this will help me improve the protocol by giving me a better idea of client-upgrade adoption rates. (#293)

Packaging changes:

• We removed the Rendezvous Server (now named the "Mailbox Server") out to a separate package and repository named magic-wormhole-mailbox-server. We still import it for tests. Use pip install magic-wormhole-mailbox-server to run your own server. (#240)
• The code is now formatted to be PEP8 compliant. (#296)
• The Dockerfile was removed: after the Mailbox Server was moved out, I don't think it was relevant. (#295)

Thanks to Andreas Baeumla Bäuml, Marius mgedmin Gedminas, Ofek ofek Lev, Thomas ThomasWaldmann Waldmann, and Vasudev copyninja Kamath for patches and bug reports in this release.

Release 0.10.5 (14-Feb-2018)

• Upgrade to newer python-spake2, to improve startup speed by not computing blinding factors for unused parameter sets. On a Raspberry Pi 3, this reduces "wormhole --version" time from ~19s to 7s.
• Fix a concurrency bug that could cause a crash if the server responded too quickly. (#280)

Release 0.10.4 (28-Jan-2018)

Minor client changes:

• accept $WORMHOLE_RELAY_URL and $WORMHOLE_TRANSIT_HELPER environment variables, in addition to command-line arguments (#256)
• fix --tor-control-port=, which was completely broken before. If you use --tor but not --tor-control-port=, we'll try the default control ports before falling back to the default SOCKS port (#252)
• fix more directory-separator pathname problems, especially for bash-on-windows (#251)
• change send output format to make copy-paste easier (#266, #267)

We also moved the docs to readthedocs (https://magic-wormhole.readthedocs.io/), rather than pointing folks at the GitHub rendered markdown files. This should encourage us to write more instructional text in the future.

Finally, we removed the Transit Relay server code from the magic-wormhole package and repository. It now lives in a separate repository named magic-wormhole-transit-relay, and we only import it for tests. If you'd like to run a transit relay, you'll want to use pip install magic-wormhole-transit-relay.

Thanks to meejah, Jonathan "jml" Lange, Alex Gaynor, David "dharrigan" Harrigan, and Jaye "jtdoepke" Doepke, for patches and bug reports in this release.

Release 0.10.3 (12-Sep-2017)

Minor client changes:

• wormhole help should behave like wormhole --help (#61)
• accept unicode pathnames (although bugs likely remain) (#223)
• reject invalid codes (with space, or non-numeric prefix) at entry (#212)
• docs improvements (#225, #249)

Server changes:

• wormhole-server start adds --relay-database-path and --stats-json-path (#186)
• accept --websocket-protocol-option= (#196, #197)
• increase RLIMIT_NOFILE to allow more simultaneous client connections (#238)
• "crowded" mailboxes now deliver an error to clients, so they should give up instead of reconnecting (#211)
• construct relay DB more safely (#189)

In addition, the snapcraft packaging was updated (#202), and setup.py now properly marks the dependency on attrs (#248).

Thanks to cclauss, Buckaroo9, JP Calderone, Pablo Oliveira, Leo Arias, Johan Lindskogen, lanzelot1989, CottonEaster, Chandan Rai, Jaakko Luttinen, Alex Gaynor, and Quentin Hibon for patches and bug reports fixed in this release.

Release 0.10.2 (26-Jun-2017)

WebSocket connection errors are now reported properly. Previous versions crashed with an unhelpful automat._core.NoTransition exception when the TCP connection was established but WebSocket negotiation could not complete (e.g. the URL path was incorrect and the server reported a 404, or we connected to an SMTP or other non-HTTP server). (#180)

The unit test suite should now pass: a CLI-version advertisement issue caused the 0.10.1 release tests to fail.

Thanks to Fabien "fdev31" Devaux for bug reports addressed in this release.

Release 0.10.1 (26-Jun-2017)

Server-only: the rendezvous server no longer advertises a CLI version unless specifically requested (by passing --advertise-version= to wormhole-server start). The public server no longer does this, so e.g. 0.10.0 clients will not emit a warning about the server recommending the 0.9.2 release. This feature was useful when the only way to use magic-wormhole was to install the CLI tool with pip, however now that 0.9.1 is in debian Stretch (and we hope to maintain compatibility with it), the nag-you-to-upgrade messages probably do more harm than good. (#179)

No user-visible client-side changes.

Thanks to ilovezfs and JP Calderone for bug reports addressed in this release.

Release 0.10.0 (24-Jun-2017)

The client-side code was completely rewritten, with proper Automat state machines. The only immediately user-visible consequence is that restarting the rendezvous server no longer terminates all waiting clients, so server upgrades won't be quite so traumatic. In the future, this will also support "Journaled Mode" (see docs/journal.md for details). (#42, #68)

The programmatic API has changed (see docs/api.md). Stability is not promised until we reach 1.0, but this should be close, at least for the non-Transit portions.

wormhole send DIRECTORY can now handle larger (>2GB) directories. However the entire zipfile is built in-RAM before transmission, so the maximum size is still limited by available memory (follow #58 for progress on fixing this). (#138)

wormhole rx --output-file= for a pre-existing file will now overwrite the file (noisily), instead of terminating with an error. (#73)

We now test on py3.6. Support for py3.3 was dropped. Magic-wormhole should now work on NetBSD. (#158)

Added a Dockerfile to build a rendezvous/transit-relay server. (#149)

wormhole-server --disallow-list instructs the rendezvous server to not honor "list nameplates" requests, effectively disabling tab-completion of the initial numeric portion of the wormhole code, but also making DoS attacks slightly easier to detect. (#53, #150)

wormhole send --ignore-unsendable-files will skip things that cannot be sent (mostly dangling symlinks and files for which you do not have read permission, but possibly also unix-domain sockets, device nodes, and pipes). (#112, #161)

txtorcon is now required by default, so the magic-wormhole[tor] "extra" was removed, and a simple pip install magic-wormhole should provide tor-based transport as long as Tor itself is available. Also, Tor works on py3 now. (#136, #174)

python -m wormhole is an alternative way to run the CLI tool. (#159)

wormhole send might handle non-ascii (unicode) filenames better now. (#157)

Thanks to Alex Gaynor, Atul Varma, dkg, JP Calderone, Kenneth Reitz, Kurt Rose, maxalbert, meejah, midnightmagic, Robert Foss, Shannon Mulloy, and Shirley Kotian, for patches and bug reports in this release cycle. A special thanks to Glyph, Mark Williams, and the whole #twisted crew at PyCon for help with the transition to Automat.

Release 0.9.2 (16-Jan-2017)

Tor support was rewritten. wormhole send, wormhole receive, wormhole ssh invite, and wormhole ssh accept all now accept three Tor-related arguments:

• --tor: use Tor for all connections, and hide all IP addresses
• --launch-tor: launch a new Tor process instead of using an existing one
• --tor-control-port=: use a specific control port, instead of using the default

If Tor is already running on your system (either as an OS-installed package, or because the TorBrowser application is running), simply adding --tor should be sufficient. If Tor is installed but not running, you may need to use both, e.g. wormhole send --tor --launch-tor. See docs/tor.md for more details. Note that Tor support must be requested at install time (with pip install magic-wormhole[tor]), and only works on python2.7 (not py3). (#64, #97)

The relay and transit URLs were changed to point at the project's official domain name (magic-wormhole.io). The servers themselves are identical (only the domain name changed, not the IP address), so this release is fully compatible with previous releases.

A packaging file for "snapcraft.io" is now included. (#131)

wormhole receive now reminds you that tab-completion is available, if you didn't use the Tab key while entering the code. (#15)

wormhole receive should work on cygwin now (a problem with the readline-completion library caused a failure on previous releases). (#111)

Thanks to Atul Varma, Leo Arias, Daniel Kahn Gillmor, Christopher Wood, Kostin Anagnostopoulos, Martin Falatic, and Joey Hess for patches and bug reports in this cycle.

Release 0.9.1 (01-Jan-2017)

The wormhole client's --transit-helper= argument can now include a "relay priority" via a numerical priority= field, e.g. --transit-helper tcp:example.org:12345:priority=2.5. Clients exchange transit relay suggestions, then try to use the highest-priority relay first, falling back to others after a few seconds if necessary. Direct connections are always preferred to a relay. Clients running 0.9.0 or earlier will ignore priorities, and unmarked relay arguments have an implicit priority of 0. (#103)

Other changes:

• clients now tolerate duplicate peer messages: in the future, this will help clients recover from intermittent rendezvous connections (#121)
• rendezvous server: ensure release() and close() are idempotent (from different connections), also for lost-connection recovery (#118)
• transit server: respect --blur-usage= by not logging connections
• README: note py3.6 compatibility

Thanks to xloem, kneufeld, and meejah for their help this cycle.

Release 0.9.0 (24-Dec-2016)

This release fixes an important "Transit Relay" bug that would have prevented future versions from using non-default relay servers. It is now easier to run wormhole as a subprocess beneath some other program (the long term goal is to provide a nice API, but even with one, there will be programs written in languages without Wormhole bindings that may find it most convenient to use a subprocess).

• fix --transit-helper=: Older versions had a bug that broke file/directory transfers when the two sides offered different transit-relay servers. This was fixed by deduplicating relay hints and adding a new kind of relay handshake. Clients running 0.9.0 or higher now require a transit-relay server running 0.9.0 or higher. (#115)
• wormhole receive: reject transfers when the target does not appear to have enough space (not available on windows) (#91)
• CLI: emit pacifier message when key-verification is slow (#29)
• add --appid= so wrapping scripts can use a distinct value (#113)
• wormhole send: flush output after displaying code, for use in scripts (#108)
• CLI: print progress messages to stderr, not stdout (#99)
• add basic man(1) pages (#69)

Many thanks to patch submitters for this release: Joey Hess, Jared Anderson, Antoine Beaupré, and to everyone testing and filing issues on Github.

Release 0.8.2 (08-Dec-2016)

• CLI: add new "wormhole ssh invite" and "wormhole ssh accept" commands, to facilitate appending your ~/.ssh/id_*.pub key into a suitably-permissioned remote ~/.ssh/authorized_keys file. These commands are experimental: the syntax might be changed in the future, or they might be removed altogether.
• CLI: "wormhole recv" and "wormhole recieve" are now accepted as aliases for "wormhole receive", to help bad spelers :)
• CLI: improve display of abbreviated file sizes
• CLI: don't print traceback upon "normal" errors
• CLI: when target file already exists, don't reveal that fact to the sender, just say "transfer rejected"
• magic-wormhole now depends upon Twisted[tls], which will cause pyOpenSSL and the cryptography package to be installed. This should prevent a warning about the "service_identity" module not being available.
• other smaller internal changes

Thanks to everyone who submitted patches in this release cycle: anarcat, Ofekmeister, Tom Lowenthal, meejah, dreid, and dkg. And thanks to the many bug reporters on Github!

Release 0.8.1 (27-Jul-2016)

This release contains mostly minor changes.

The most noticeable is that long-lived wormholes should be more reliable now. Previously, if you run wormhole send but your peer doesn't run their receive for several hours, a NAT/firewall box on either side could stop forwarding traffic for the idle connection (without sending a FIN or RST to properly close the socket), causing both sides to hang forever and never actually connect. Now both sides send periodic keep-alive messages to prevent this.

In addition, by switching to "Click" for argument parsing, we now have short command aliases: wormhole tx does the same thing as wormhole send, and wormhole rx is an easier-to-spell equivalent of wormhole receive.

Other changes:

• CLI: move most arguments to be attached to the subcommand (new: wormhole send --verify) rather than on the "wormhole" command (old: wormhole --verify send). Four arguments remain on the "wormhole" command: --relay-url=, --transit-helper=, --dump-timing=, and --version.
• docs: add links to PyCon2016 presentation
• reject wormhole-codes with spaces with a better error message
• magic-wormhole ought to work on windows now
• code-input tab-completion should work on stock OS-X python (with libedit)
• sending a directory should restore file permissions correctly
• server changes:
  • expire channels after two hours, not 3 days
  • prune channels more accurately
  • improve munin plugins for server monitoring

Many thanks to the folks who contributed to this release, during the PyCon sprints and afterwards: higs4281, laharah, Chris Wolfe, meejah, wsanchez, Kurt Neufeld, and Francois Marier.

Release 0.8.0 (28-May-2016)

This release is completely incompatible with the previous 0.7.6 release. Clients using 0.7.6 or earlier will not even notice clients using 0.8.0 or later.

• Overhaul client-server websocket protocol, client-client PAKE messages, per-message encryption-key derivation, relay-server database schema, SPAKE2 key-derivation, and public relay URLs. Add version fields and unknown-message tolerance to most protocol steps.
• Hopefully this will provide forward-compatibility with future protocol changes. I have several on my list, and the version fields should make it possible to add these without a flag day (at worst a "flag month").
• User-visible changes are minimal, although some operations should be faster because we no longer need to wait for ACKs before proceeding.
• API changes: .send_data()/.get_data() became .send()/.get(), neither takes a phase= argument (the Wormhole is now a record pipe) .get_verifier() became .verify() (and waits to receive the key-confirmation message before firing its Deferred), wormholes are constructed with a function call instead of a class constructor, close() always waits for server ack of outbound messages. Note that the API remains unstable until 1.0.0 .
• misc/munin/ contains plugins for relay server operators

Release 0.7.6 (08-May-2016)

• Switch to "tqdm" for nicer CLI progress bars.
• Fail better when input-code is interrupted (prompt user to hit Return, rather than hanging forever)
• Close channel upon error more reliably.
• Explain WrongPasswordError better.
• (internal): improve --dump-timing instrumentation and rendering.

Compatibility: this remains compatible with 0.7.x, and 0.8.x is still expected to break compatibility.

Release 0.7.5 (20-Apr-2016)

• The CLI tools now use the Twisted-based library exclusively.
• The blocking-flavor "Transit" library has been removed. Transit is the bulk-transfer protocol used by send-file/send-directory. Upcoming protocol improvements (performance and connectivity) proved too difficult to implement in a blocking fashion, so for now if you want Transit, use Twisted.
• The Twisted-flavor "Wormhole" library now uses WebSockets to connect, rather than HTTP. The blocking-flavor library continues to use HTTP. "Wormhole" is the one-message-at-a-time relay-based protocol, and is used to set up Transit for the send-file and send-directory modes of the CLI tool.
• Twisted-flavor input_code() now does readline-based code entry, with tab completion.
• The package now installs two executables: "wormhole" (for send and receive), and "wormhole-server" (to start and manage the relay servers). These may be re-merged in a future release.

Compatibility:

• This release remains compatible with the previous ones. The next major release (0.8.x) will probably break compatibility.

Packaging:

• magic-wormhole now depends upon "Twisted" and "autobahn" (for WebSockets). Autobahn pulls in txaio, but we don't support it yet (a future version of magic-wormhole might).
• To work around a bug in autobahn, we also (temporarily) depend upon "pytrie". This dependency will be removed when the next autobahn release is available.

Release 0.7.0 (28-Mar-2016)

• wormhole send DIRNAME/ used to deal very badly with the trailing slash (sending a directory with an empty name). This is now fixed.
• Preliminary Tor support was added. Install magic-wormhole[tor], make sure you have a Tor executable on your $PATH, and run wormhole --tor send. This will launch a new Tor process. Do not use this in anger/fear until it has been tested more carefully. This feature is likely to be unstable for a while, and lacks tests.
• The relay now prunes unused channels properly.
• Added --dump-timing= to record timeline of events, for debugging and performance improvements. You can combine timing data from both sides to see where the delays are happening. The server now returns timestamps in its responses, to measure round-trip delays. A web-based visualization tool was added in misc/dump-timing.py.
• twisted.transit was not properly handling multiple records received in a single chunk. Some producer/consumer helper methods were added. You can now run e.g. wormhole --twisted send to force the use of the Twisted implementation.
• The Twisted wormhole now uses a persistent connection for all relay messages, which should be slightly faster.
• Add --no-listen to prevent Transit from listening for inbound connections (or advertising any addresses): this is only useful for testing.
• The tests now collect code coverage information, and upload them to https://codecov.io/github/warner/magic-wormhole?ref=master .

Release 0.6.3 (29-Feb-2016)

Mostly internal changes:

• twisted.transit was added, so Twisted-based applications can use it now. This includes Producer/Consumer -based flow control. The Transit protocol and API are documented in docs/transit.md .
• The transit relay server can blur filesizes, rounding them to some roughly-logarithmic interval.
• Use --relay-helper="" to disable use of the transit relay entirely, limiting the file transfer to direct connections.
• The new --hide-progress option disables the progress bar.
• Made some windows-compatibility fixes, but all tests do not yet pass.

Release 0.6.2 (12-Jan-2016)

• the server can now "blur" usage information: this turns off HTTP logging, and rounds timestamps to coarse intervals
• wormhole server usage now shows Transit usage too, not just Rendezvous

Release 0.6.1 (03-Dec-2015)

• wormhole can now send/receive entire directories. They are zipped before transport.
• Python 3 is now supported for async (Twisted) library use, requiring at least Twisted-15.5.0.
• A bug was fixed which prevented py3-based clients from using the relay transit server (not used if the two sides can reach each other directly).
• The --output-file= argument was finally implemented, which allows the receiver to override the filename that it writes. This may help scripted usage.
• Support for Python-2.6 was removed, since the recent Twisted-15.5.0 removed it too. It might still work, but is no longer automatically tested.
• The transit relay now implements proper flow control (Producer/Consumer), so it won't buffer the entire file when the sender can push data faster than the receiver can accept it. The sender should now throttle down to the receiver's maximum rate.

Release 0.6.0 (23-Nov-2015)

• Add key-confirmation message so "wormhole send" doesn't hang when the receiver mistypes the code.
• Fix wormhole send --text - to read the text message from stdin. wormhole receive >outfile works, but currently appends an extra newline, which may be removed in a future release.
• Arrange for 0.4.0 senders to print an error message when connecting to a current (0.5.0) server, instead of an ugly stack trace. Unfortunately 0.4.0 receivers still display the traceback, since they don't check the welcome message before using a missing API. 0.5.0 and 0.6.0 will do better.
• Improve channel deallocation upon error.
• Inform the server of our "mood" when the connection closes, so it can track the rate of successful/unsuccessful transfers. The server DB now stores a summary of each transfer (waiting time and reported outcome).
• Rename (and deprecate) one server API (the non-EventSource form of "get"), leaving it in place until after the next release. 0.5.0 clients should interoperate with both the 0.6.0 server and 0.6.0 clients, but eventually they'll stop working.

Release 0.5.0 (07-Oct-2015)

• Change the CLI to merge send-file with send-text, and receive-file with receive-text. Add confirmation before accepting a file.
• Change the remote server API significantly, breaking compatibility with 0.4.0 peers. Fix EventSource to match W3C spec and real browser behavior.
• Add py3 (3.3, 3.4, 3.5) compatibility for blocking calls (but not Twisted).
• internals
• Introduce Channel and ChannelManager to factor out the HTTP/EventSource technology in use (making room for WebSocket or Tor in the future).
• Change app-visible API to allow multiple message phases.
• Change most API arguments from bytes to unicode strings (appid, URLs, wormhole code, derive_key purpose string, message phase). Derived keys are bytes, of course.
• Add proper unit tests.

Release 0.4.0 (22-Sep-2015)

This changes the protocol (to a symmetric form), breaking compatibility with 0.3.0 peers. Now both blocking-style and Twisted-style use a symmetric protocol, and the two sides do not need to figure out (ahead of time) which one goes first. The internal layout was rearranged, so applications that import wormhole must be updated.

Release 0.3.0 (24-Jun-2015)

Add preliminary Twisted support, only for symmetric endpoints (no initator/receiver distinction). Lacks code-entry tab-completion. May still leave timers lingering. Add test suite (only for Twisted, so far).

Use a sqlite database for Relay server state, to survive reboots with less data loss. Add "--advertise-version=" to "wormhole relay start", to override the version we recommend to clients.

Release 0.2.0 (10-Apr-2015)

Initial release: supports blocking/synchronous asymmetric endpoints (Initiator on one side, Receiver on the other). Codes can be generated by Initiator, or created externally and passed into both (as long as they start with digits: NNN-anything).