add w.when_key(), fix w.when_verified() to fire later

Previously, w.when_verified() was documented to fire only after a valid
encrypted message was received, but in fact it fired as soon as the shared
key was derived (before any encrypted messages are seen, so no actual
"verification" could occur yet).

This fixes that, and also adds a new w.when_key() API call which fires at the
earlier point. Having something which fires early is useful for the CLI
commands that want to print a pacifier message when the peer is responding
slowly. In particular it helps detect the case where 'wormhole send' has quit
early (after depositing the PAKE message on the server, but before the
receiver has started). In this case, the receiver will compute the shared
key, but then wait forever hoping for a VERSION that will never come. By
starting a timer when w.when_key() fires, and cancelling it when
w.when_verified() fires, we have a good place to tell the user that something
is taking longer than it should have.

This shifts responsibility for notifying Boss.got_verifier, out of Key and
into Receive, since Receive is what notices the first valid encrypted
message. It also shifts the Boss's ordering expectations: it now receives
B.happy() before B.got_verifier(), and consequently got_verifier ought to
arrive in the S2_happy state rather than S1_lonely.

docs/api.md

@@ -286,11 +286,17 @@ the rest of the protocol to proceed. If they do not match, then the two
 programs are not talking to each other (they may both be talking to a
 man-in-the-middle attacker), and the protocol should be abandoned.
 
-Once retrieved, you can turn this into hex or Base64 to print it, or render
+Deferred-mode applications can wait for `d=w.when_verified()`: the Deferred
-it as ASCII-art, etc. Once the users are convinced that `verify()` from both
+it returns will fire with the verifier. You can turn this into hex or Base64
-sides are the same, call `send()` to continue the protocol. If you call
+to print it, or render it as ASCII-art, etc.
-`send()` before `verify()`, it will perform the complete protocol without
+
-pausing.
+Asking the wormhole object for the verifier does not affect the flow of the
+protocol. To benefit from verification, applications must refrain from
+sending any data (with `w.send(data)`) until after the verifiers are approved
+by the user. In addition, applications must queue or otherwise ignore
+incoming (received) messages until that point. However once the verifiers are
+confirmed, previously-received messages can be considered valid and processed
+as usual.
 
 ## Welcome Messages
 
@@ -377,6 +383,14 @@ those Deferreds.
   has been told `h.set_words()`, or immediately after `w.set_code(code)` is
   called. This is most useful after calling `w.generate_code()`, to show the
   generated code to the user so they can transcribe it to their peer.
+* key (`yield w.when_key()` / `dg.wormhole_key()`): fired when the
+  key-exchange process has completed and a purported shared key is
+  established. At this point we do not know that anyone else actually shares
+  this key: the peer may have used the wrong code, or may have disappeared
+  altogether. To wait for proof that the key is shared, wait for
+  `when_verified` instead. This event is really only useful for detecting
+  that the initiating peer has disconnected after leaving the initial PAKE
+  message, to display a pacifying message to the user.
 * verified (`verifier = yield w.when_verified()` /
   `dg.wormhole_verified(verifier)`: fired when the key-exchange process has
   completed and a valid VERSION message has arrived. The "verifier" is a byte

docs/state-machines/boss.dot

@@ -73,7 +73,7 @@ digraph {
 
         {rank=same; Other S_closed}
         Other [shape="box" style="dashed"
-               label="rx_welcome -> process (maybe rx_unwelcome)\nsend -> S.send\ngot_message -> got_version or got_phase\ngot_key -> W.got_key\ngot_verifier -> W.got_verifier\nallocate_Code -> C.allocate_code\ninput_code -> C.input_code\nset_code -> C.set_code"
+               label="rx_welcome -> process (maybe rx_unwelcome)\nsend -> S.send\ngot_message -> got_version or got_phase\ngot_key -> W.got_key\ngot_verifier -> W.got_verifier\nallocate_code -> C.allocate_code\ninput_code -> C.input_code\nset_code -> C.set_code"
                ]
 
 

docs/state-machines/key.dot

@@ -55,7 +55,7 @@ digraph {
         S1 -> P1_compute [label="got_pake\npake good"]
         #S1 -> P_mood_lonely [label="close"]
 
-        P1_compute [label="compute_key\nM.add_message(version)\nB.got_key\nB.got_verifier\nR.got_key" shape="box"]
+        P1_compute [label="compute_key\nM.add_message(version)\nB.got_key\nR.got_key" shape="box"]
         P1_compute -> S4
 
         S4 [label="S4: know_key" color="green"]

docs/state-machines/machines.dot

@@ -42,7 +42,7 @@ digraph {
 
         #Boss -> Mailbox [color="blue"]
         Mailbox -> Order [style="dashed" label="got_message (once)"]
-        Key -> Boss [style="dashed" label="got_key\ngot_verifier\nscared"]
+        Key -> Boss [style="dashed" label="got_key\nscared"]
         Order -> Key [style="dashed" label="got_pake"]
         Order -> Receive [style="dashed" label="got_message"]
         #Boss -> Key [color="blue"]
@@ -54,7 +54,7 @@ digraph {
 
         Key -> Receive [style="dashed" label="got_key"]
         Receive -> Boss [style="dashed"
-                         label="happy\nscared\ngot_message"]
+                         label="happy\nscared\ngot_verifier\ngot_message"]
         Nameplate -> Connection [style="dashed"
                                  label="tx_claim\ntx_release"]
         Connection -> Nameplate [style="dashed"

docs/state-machines/receive.dot

@@ -19,7 +19,7 @@ digraph {
         S1 [label="S1:\nunverified key" color="orange"]
         S1 -> P_mood_scary [label="got_message\n(bad)"]
         S1 -> P1_accept_msg [label="got_message\n(good)" color="orange"]
-        P1_accept_msg [shape="box" label="S.got_verified_key\nB.happy\nB.got_message"
+        P1_accept_msg [shape="box" label="S.got_verified_key\nB.happy\nB.got_verifier\nB.got_message"
                        color="orange"]
         P1_accept_msg -> S2 [color="orange"]
 

src/wormhole/_boss.py

@@ -197,8 +197,8 @@ class Boss(object):
     @m.input()
     def got_code(self, code): pass
 
-    # Key sends (got_key, got_verifier, scared)
+    # Key sends (got_key, scared)
-    # Receive sends (got_message, happy, scared)
+    # Receive sends (got_message, happy, got_verifier, scared)
     @m.input()
     def happy(self): pass
     @m.input()
@@ -307,11 +307,11 @@ class Boss(object):
     S1_lonely.upon(close, enter=S3_closing, outputs=[close_lonely])
     S1_lonely.upon(send, enter=S1_lonely, outputs=[S_send])
     S1_lonely.upon(got_key, enter=S1_lonely, outputs=[W_got_key])
-    S1_lonely.upon(got_verifier, enter=S1_lonely, outputs=[W_got_verifier])
     S1_lonely.upon(rx_error, enter=S3_closing, outputs=[close_error])
     S1_lonely.upon(error, enter=S4_closed, outputs=[W_close_with_error])
 
     S2_happy.upon(rx_unwelcome, enter=S3_closing, outputs=[close_unwelcome])
+    S2_happy.upon(got_verifier, enter=S2_happy, outputs=[W_got_verifier])
     S2_happy.upon(_got_phase, enter=S2_happy, outputs=[W_received])
     S2_happy.upon(_got_version, enter=S2_happy, outputs=[process_version])
     S2_happy.upon(scared, enter=S3_closing, outputs=[close_scared])
@@ -322,6 +322,7 @@ class Boss(object):
 
     S3_closing.upon(rx_unwelcome, enter=S3_closing, outputs=[])
     S3_closing.upon(rx_error, enter=S3_closing, outputs=[])
+    S3_closing.upon(got_verifier, enter=S3_closing, outputs=[])
     S3_closing.upon(_got_phase, enter=S3_closing, outputs=[])
     S3_closing.upon(_got_version, enter=S3_closing, outputs=[])
     S3_closing.upon(happy, enter=S3_closing, outputs=[])
@@ -332,6 +333,7 @@ class Boss(object):
     S3_closing.upon(error, enter=S4_closed, outputs=[W_close_with_error])
 
     S4_closed.upon(rx_unwelcome, enter=S4_closed, outputs=[])
+    S4_closed.upon(got_verifier, enter=S4_closed, outputs=[])
     S4_closed.upon(_got_phase, enter=S4_closed, outputs=[])
     S4_closed.upon(_got_version, enter=S4_closed, outputs=[])
     S4_closed.upon(happy, enter=S4_closed, outputs=[])

src/wormhole/_key.py

@@ -166,7 +166,6 @@ class _SortedKey(object):
         with self._timing.add("pake2", waiting="crypto"):
             key = self._sp.finish(msg2)
         self._B.got_key(key)
-        self._B.got_verifier(derive_key(key, b"wormhole:verifier"))
         phase = "version"
         data_key = derive_phase_key(key, self._side, phase)
         plaintext = dict_to_bytes(self._versions)

src/wormhole/_receive.py

@@ -4,7 +4,7 @@ from attr import attrs, attrib
 from attr.validators import provides, instance_of
 from automat import MethodicalMachine
 from . import _interfaces
-from ._key import derive_phase_key, decrypt_data, CryptoError
+from ._key import derive_key, derive_phase_key, decrypt_data, CryptoError
 
 @attrs
 @implementer(_interfaces.IReceive)
@@ -63,6 +63,9 @@ class Receive(object):
     def W_happy(self, phase, plaintext):
         self._B.happy()
     @m.output()
+    def W_got_verifier(self, phase, plaintext):
+        self._B.got_verifier(derive_key(self._key, b"wormhole:verifier"))
+    @m.output()
     def W_got_message(self, phase, plaintext):
         assert isinstance(phase, type("")), type(phase)
         assert isinstance(plaintext, type(b"")), type(plaintext)
@@ -73,7 +76,8 @@ class Receive(object):
 
     S0_unknown_key.upon(got_key, enter=S1_unverified_key, outputs=[record_key])
     S1_unverified_key.upon(got_message_good, enter=S2_verified_key,
-                           outputs=[S_got_verified_key, W_happy, W_got_message])
+                           outputs=[S_got_verified_key,
+                                    W_happy, W_got_verifier, W_got_message])
     S1_unverified_key.upon(got_message_bad, enter=S3_scared,
                            outputs=[W_scared])
     S2_verified_key.upon(got_message_bad, enter=S3_scared,

src/wormhole/test/common.py

@@ -91,3 +91,10 @@ def poll_until(predicate):
         d = defer.Deferred()
         reactor.callLater(0.001, d.callback, None)
         yield d
+
+@defer.inlineCallbacks
+def pause_one_tick():
+    # return a Deferred that won't fire until at least the next reactor tick
+    d = defer.Deferred()
+    reactor.callLater(0.001, d.callback, None)
+    yield d

src/wormhole/test/test_machines.py

@@ -128,7 +128,8 @@ class Receive(unittest.TestCase):
     def build(self):
         events = []
         r = _receive.Receive(u"side", timing.DebugTiming())
-        b = Dummy("b", events, IBoss, "happy", "scared", "got_message")
+        b = Dummy("b", events, IBoss,
+                  "happy", "scared", "got_verifier", "got_message")
         s = Dummy("s", events, ISend, "got_verified_key")
         r.wire(b, s)
         return r, b, s, events
@@ -138,12 +139,14 @@ class Receive(unittest.TestCase):
         key = b"key"
         r.got_key(key)
         self.assertEqual(events, [])
+        verifier = derive_key(key, b"wormhole:verifier")
         phase1_key = derive_phase_key(key, u"side", u"phase1")
         data1 = b"data1"
         good_body = encrypt_data(phase1_key, data1)
         r.got_message(u"side", u"phase1", good_body)
         self.assertEqual(events, [("s.got_verified_key", key),
                                   ("b.happy",),
+                                  ("b.got_verifier", verifier),
                                   ("b.got_message", u"phase1", data1),
                                   ])
 
@@ -153,6 +156,7 @@ class Receive(unittest.TestCase):
         r.got_message(u"side", u"phase2", good_body)
         self.assertEqual(events, [("s.got_verified_key", key),
                                   ("b.happy",),
+                                  ("b.got_verifier", verifier),
                                   ("b.got_message", u"phase1", data1),
                                   ("b.got_message", u"phase2", data2),
                                   ])
@@ -181,12 +185,14 @@ class Receive(unittest.TestCase):
         key = b"key"
         r.got_key(key)
         self.assertEqual(events, [])
+        verifier = derive_key(key, b"wormhole:verifier")
         phase1_key = derive_phase_key(key, u"side", u"phase1")
         data1 = b"data1"
         good_body = encrypt_data(phase1_key, data1)
         r.got_message(u"side", u"phase1", good_body)
         self.assertEqual(events, [("s.got_verified_key", key),
                                   ("b.happy",),
+                                  ("b.got_verifier", verifier),
                                   ("b.got_message", u"phase1", data1),
                                   ])
 
@@ -196,6 +202,7 @@ class Receive(unittest.TestCase):
         r.got_message(u"side", u"phase2", bad_body)
         self.assertEqual(events, [("s.got_verified_key", key),
                                   ("b.happy",),
+                                  ("b.got_verifier", verifier),
                                   ("b.got_message", u"phase1", data1),
                                   ("b.scared",),
                                   ])
@@ -203,6 +210,7 @@ class Receive(unittest.TestCase):
         r.got_message(u"side", u"phase2", bad_body)
         self.assertEqual(events, [("s.got_verified_key", key),
                                   ("b.happy",),
+                                  ("b.got_verifier", verifier),
                                   ("b.got_message", u"phase1", data1),
                                   ("b.scared",),
                                   ])
@@ -216,7 +224,7 @@ class Key(unittest.TestCase):
     def build(self):
         events = []
         k = _key.Key(u"appid", {}, u"side", timing.DebugTiming())
-        b = Dummy("b", events, IBoss, "scared", "got_key", "got_verifier")
+        b = Dummy("b", events, IBoss, "scared", "got_key")
         m = Dummy("m", events, IMailbox, "add_message")
         r = Dummy("r", events, IReceive, "got_key")
         k.wire(b, m, r)
@@ -237,11 +245,10 @@ class Key(unittest.TestCase):
         key2 = sp.finish(msg1_bytes)
         msg2 = dict_to_bytes({"pake_v1": bytes_to_hexstr(msg2_bytes)})
         k.got_pake(msg2)
-        self.assertEqual(len(events), 4, events)
+        self.assertEqual(len(events), 3, events)
         self.assertEqual(events[0], ("b.got_key", key2))
-        self.assertEqual(events[1][0], "b.got_verifier")
+        self.assertEqual(events[1][:2], ("m.add_message", "version"))
-        self.assertEqual(events[2][:2], ("m.add_message", "version"))
+        self.assertEqual(events[2], ("r.got_key", key2))
-        self.assertEqual(events[3], ("r.got_key", key2))
 
     def test_bad(self):
         k, b, m, r, events = self.build()
@@ -274,16 +281,15 @@ class Key(unittest.TestCase):
         self.assertEqual(len(events), 0)
 
         k.got_code(code)
-        self.assertEqual(len(events), 5)
+        self.assertEqual(len(events), 4)
         self.assertEqual(events[0][:2], ("m.add_message", "pake"))
         msg1_json = events[0][2].decode("utf-8")
         msg1 = json.loads(msg1_json)
         msg1_bytes = hexstr_to_bytes(msg1["pake_v1"])
         key2 = sp.finish(msg1_bytes)
         self.assertEqual(events[1], ("b.got_key", key2))
-        self.assertEqual(events[2][0], "b.got_verifier")
+        self.assertEqual(events[2][:2], ("m.add_message", "version"))
-        self.assertEqual(events[3][:2], ("m.add_message", "version"))
+        self.assertEqual(events[3], ("r.got_key", key2))
-        self.assertEqual(events[4], ("r.got_key", key2))
 
 class Code(unittest.TestCase):
     def build(self):
@@ -1178,8 +1184,8 @@ class Boss(unittest.TestCase):
 
         # pretend a peer message was correctly decrypted
         b.got_key(b"key")
-        b.got_verifier(b"verifier")
         b.happy()
+        b.got_verifier(b"verifier")
         b.got_message("version", b"{}")
         b.got_message("0", b"msg1")
         self.assertEqual(events, [("w.got_key", b"key"),

src/wormhole/test/test_wormhole.py

@@ -4,7 +4,7 @@ import mock
 from twisted.trial import unittest
 from twisted.internet import reactor
 from twisted.internet.defer import gatherResults, inlineCallbacks
-from .common import ServerBase, poll_until
+from .common import ServerBase, poll_until, pause_one_tick
 from .. import wormhole, _rendezvous
 from ..errors import (WrongPasswordError,
                       KeyFormatError, WormholeClosed, LonelyError,
@@ -115,10 +115,16 @@ class Wormholes(ServerBase, unittest.TestCase):
         code = yield w1.when_code()
         w2.set_code(code)
 
+        yield w1.when_key()
+        yield w2.when_key()
+
         verifier1 = yield w1.when_verified()
         verifier2 = yield w2.when_verified()
         self.assertEqual(verifier1, verifier2)
 
+        self.successResultOf(w1.when_key())
+        self.successResultOf(w2.when_key())
+
         version1 = yield w1.when_version()
         version2 = yield w2.when_version()
         # TODO: add the ability to set app-versions
@@ -291,12 +297,13 @@ class Wormholes(ServerBase, unittest.TestCase):
         yield w1.close()
         yield w2.close()
 
-        # once closed, all Deferred-yielding API calls get an error
+        # once closed, all Deferred-yielding API calls get an immediate error
-        e = yield self.assertFailure(w1.when_code(), WormholeClosed)
+        f = self.failureResultOf(w1.when_code(), WormholeClosed)
-        self.assertEqual(e.args[0], "happy")
+        self.assertEqual(f.value.args[0], "happy")
-        yield self.assertFailure(w1.when_verified(), WormholeClosed)
+        self.failureResultOf(w1.when_key(), WormholeClosed)
-        yield self.assertFailure(w1.when_version(), WormholeClosed)
+        self.failureResultOf(w1.when_verified(), WormholeClosed)
-        yield self.assertFailure(w1.when_received(), WormholeClosed)
+        self.failureResultOf(w1.when_version(), WormholeClosed)
+        self.failureResultOf(w1.when_received(), WormholeClosed)
 
 
     @inlineCallbacks
@@ -315,16 +322,64 @@ class Wormholes(ServerBase, unittest.TestCase):
         w1.send(b"should still work")
         w2.send(b"should still work")
 
-        # API calls that wait (i.e. get) will errback
+        key2 = yield w2.when_key() # should work
-        yield self.assertFailure(w2.when_received(), WrongPasswordError)
+        # w2 has just received w1.PAKE, and is about to send w2.VERSION
-        yield self.assertFailure(w1.when_received(), WrongPasswordError)
+        key1 = yield w1.when_key() # should work
+        # w1 has just received w2.PAKE, and is about to send w1.VERSION, and
+        # then will receive w2.VERSION. When it sees w2.VERSION, it will
+        # learn about the WrongPasswordError.
+        self.assertNotEqual(key1, key2)
 
+        # API calls that wait (i.e. get) will errback. We collect all these
+        # Deferreds early to exercise the wait-then-fail path
+        d1_verified = w1.when_verified()
+        d1_version = w1.when_version()
+        d1_received = w1.when_received()
+        d2_verified = w2.when_verified()
+        d2_version = w2.when_version()
+        d2_received = w2.when_received()
+
+        # wait for each side to notice the failure
         yield self.assertFailure(w1.when_verified(), WrongPasswordError)
-        yield self.assertFailure(w1.when_version(), WrongPasswordError)
+        yield self.assertFailure(w2.when_verified(), WrongPasswordError)
+        # and then wait for the rest of the loops to fire. if we had+used
+        # eventual-send, this wouldn't be a problem
+        yield pause_one_tick()
+
+        # now all the rest should have fired already
+        self.failureResultOf(d1_verified, WrongPasswordError)
+        self.failureResultOf(d1_version, WrongPasswordError)
+        self.failureResultOf(d1_received, WrongPasswordError)
+        self.failureResultOf(d2_verified, WrongPasswordError)
+        self.failureResultOf(d2_version, WrongPasswordError)
+        self.failureResultOf(d2_received, WrongPasswordError)
+
+        # and at this point, with the failure safely noticed by both sides,
+        # new when_key() calls should signal the failure, even before we
+        # close
+
+        # any new calls in the error state should immediately fail
+        self.failureResultOf(w1.when_key(), WrongPasswordError)
+        self.failureResultOf(w1.when_verified(), WrongPasswordError)
+        self.failureResultOf(w1.when_version(), WrongPasswordError)
+        self.failureResultOf(w1.when_received(), WrongPasswordError)
+        self.failureResultOf(w2.when_key(), WrongPasswordError)
+        self.failureResultOf(w2.when_verified(), WrongPasswordError)
+        self.failureResultOf(w2.when_version(), WrongPasswordError)
+        self.failureResultOf(w2.when_received(), WrongPasswordError)
 
         yield self.assertFailure(w1.close(), WrongPasswordError)
         yield self.assertFailure(w2.close(), WrongPasswordError)
 
+        # API calls should still get the error, not WormholeClosed
+        self.failureResultOf(w1.when_key(), WrongPasswordError)
+        self.failureResultOf(w1.when_verified(), WrongPasswordError)
+        self.failureResultOf(w1.when_version(), WrongPasswordError)
+        self.failureResultOf(w1.when_received(), WrongPasswordError)
+        self.failureResultOf(w2.when_key(), WrongPasswordError)
+        self.failureResultOf(w2.when_verified(), WrongPasswordError)
+        self.failureResultOf(w2.when_version(), WrongPasswordError)
+        self.failureResultOf(w2.when_received(), WrongPasswordError)
 
     @inlineCallbacks
     def test_wrong_password_with_spaces(self):

src/wormhole/wormhole.py

@@ -97,6 +97,7 @@ class _DelegatedWormhole(object):
     def got_code(self, code):
         self._delegate.wormhole_code(code)
     def got_key(self, key):
+        self._delegate.wormhole_key()
         self._key = key # for derive_key()
     def got_verifier(self, verifier):
         self._delegate.wormhole_verified(verifier)
@@ -113,6 +114,7 @@ class _DeferredWormhole(object):
         self._code = None
         self._code_observers = []
         self._key = None
+        self._key_observers = []
         self._verifier = None
         self._verifier_observers = []
         self._versions = None
@@ -138,6 +140,15 @@ class _DeferredWormhole(object):
         self._code_observers.append(d)
         return d
 
+    def when_key(self):
+        if self._observer_result is not None:
+            return defer.fail(self._observer_result)
+        if self._key is not None:
+            return defer.succeed(self._key)
+        d = defer.Deferred()
+        self._key_observers.append(d)
+        return d
+
     def when_verified(self):
         if self._observer_result is not None:
             return defer.fail(self._observer_result)
@@ -180,7 +191,7 @@ class _DeferredWormhole(object):
         """Derive a new key from the established wormhole channel for some
         other purpose. This is a deterministic randomized function of the
         session key and the 'purpose' string (unicode/py3-string). This
-        cannot be called until when_verifier() has fired, nor after close()
+        cannot be called until when_verified() has fired, nor after close()
         was called.
         """
         if not isinstance(purpose, type("")): raise TypeError(type(purpose))
@@ -210,6 +221,9 @@ class _DeferredWormhole(object):
         self._code_observers[:] = []
     def got_key(self, key):
         self._key = key # for derive_key()
+        for d in self._key_observers:
+            d.callback(key)
+        self._key_observers[:] = []
     def got_verifier(self, verifier):
         self._verifier = verifier
         for d in self._verifier_observers:
@@ -232,10 +246,12 @@ class _DeferredWormhole(object):
         if isinstance(result, Exception):
             self._observer_result = self._closed_result = failure.Failure(result)
         else:
-            # pending w.verify()/w.version()/w.read() get an error
+            # pending w.key()/w.verify()/w.version()/w.read() get an error
             self._observer_result = WormholeClosed(result)
             # but w.close() only gets error if we're unhappy
             self._closed_result = result
+        for d in self._key_observers:
+            d.errback(self._observer_result)
         for d in self._verifier_observers:
             d.errback(self._observer_result)
         for d in self._version_observers: