cmd_send: use SpooledTemporaryFile properly, move to disk after 10MB
When we send a directory, we build a temporary zipfile to linearize the contents for transmission. We store this zipfile inside a SpooledTemporaryFile, which will hold everything in RAM (for speed) until it reaches some size threshold, then moves everything over to disk (to avoid crashing your program as it runs out of memory). Unfortunately SpooledTemporaryFile doesn't have a default threshold size: if you don't specify one, it will never switch to the disk-based mode, and `wormhole send large_directory/` will just use up all your RAM until it crashes. I've been using this wrong for five years, since the very beginning of wormhole's ability to send directories. How embarrassing! This applies the simple fix: provide a `max_size=` argument, setting the threshold to 10 MB. Thanks to @blitz for the report (and my apologies to everyone else who reported this earlier, to whom I said it was fixed by using SpooledTemporaryFile, when clearly it was not). closes #379
This commit is contained in:
parent
6477f239e8
commit
49354df56e
|
@ -333,9 +333,12 @@ class Sender:
|
|||
|
||||
if os.path.isdir(what):
|
||||
print(u"Building zipfile..", file=args.stderr)
|
||||
# We're sending a directory. Create a zipfile in a tempdir and
|
||||
# send that.
|
||||
fd_to_send = tempfile.SpooledTemporaryFile()
|
||||
# We're sending a directory. Create a zipfile and send that
|
||||
# instead. SpooledTemporaryFile will use RAM until our size
|
||||
# threshold (10MB) is reached, then moves everything into a
|
||||
# tempdir (it tries $TMPDIR, $TEMP, $TMP, then platform-specific
|
||||
# paths like /tmp).
|
||||
fd_to_send = tempfile.SpooledTemporaryFile(max_size=10*1000*1000)
|
||||
# workaround for https://bugs.python.org/issue26175 (STF doesn't
|
||||
# fully implement IOBase abstract class), which breaks the new
|
||||
# zipfile in py3.7.0 that expects .seekable
|
||||
|
|
Loading…
Reference in New Issue
Block a user