cmd_send: use SpooledTemporaryFile properly, move to disk after 10MB

When we send a directory, we build a temporary zipfile to linearize the
contents for transmission. We store this zipfile inside a
SpooledTemporaryFile, which will hold everything in RAM (for speed) until it
reaches some size threshold, then moves everything over to disk (to avoid
crashing your program as it runs out of memory).

Unfortunately SpooledTemporaryFile doesn't have a default threshold size: if
you don't specify one, it will never switch to the disk-based mode, and
`wormhole send large_directory/` will just use up all your RAM until it
crashes.

I've been using this wrong for five years, since the very beginning of
wormhole's ability to send directories. How embarrassing!

This applies the simple fix: provide a `max_size=` argument, setting the
threshold to 10 MB.

Thanks to @blitz for the report (and my apologies to everyone else who
reported this earlier, to whom I said it was fixed by using
SpooledTemporaryFile, when clearly it was not).

closes #379
This commit is contained in:
Brian Warner 2020-04-04 12:06:00 -07:00
parent 6477f239e8
commit 49354df56e

View File

@ -333,9 +333,12 @@ class Sender:
if os.path.isdir(what):
print(u"Building zipfile..", file=args.stderr)
# We're sending a directory. Create a zipfile in a tempdir and
# send that.
fd_to_send = tempfile.SpooledTemporaryFile()
# We're sending a directory. Create a zipfile and send that
# instead. SpooledTemporaryFile will use RAM until our size
# threshold (10MB) is reached, then moves everything into a
# tempdir (it tries $TMPDIR, $TEMP, $TMP, then platform-specific
# paths like /tmp).
fd_to_send = tempfile.SpooledTemporaryFile(max_size=10*1000*1000)
# workaround for https://bugs.python.org/issue26175 (STF doesn't
# fully implement IOBase abstract class), which breaks the new
# zipfile in py3.7.0 that expects .seekable