Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames.
This change introduces a new class BufferedFrameDecryptor that is responsible
for decrypting received encrypted frames and passing them on to the
RtpReferenceFinder. This decoupling refactoring was triggered by a new
optimization also introduced in this patch to stash a small number of
undecryptable frames if no frames have ever been decrypted. The goal of this
optimization is to prevent re-fectching of key frames on low bandwidth networks
simply because the key to decrypt them had not arrived yet.
The optimization will stash 24 frames (about 1 second of video) in a ring buffer
and will attempt to re-decrypt previously received frames on the first valid
decryption. This allows the decoder to receive the key frame without having
to request due to short key delivery latencies. In testing this is actually hit
quite often and saves an entire RTT which can be up to 200ms on a bad network.
As the scope of frame encryption increases in WebRTC and has more specialized
optimizations that do not apply to the general flow it makes sense to move it
to a more explicit bump in the stack protocol that is decoupled from the WebRTC
main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect.
One advantage of this approach is the BufferedFrameDecryptor isn't even
constructed if FrameEncryption is not in use.
I have decided against merging the RtpReferenceFinder and EncryptedFrame stash
because it introduced a lot of complexity around the mixed scenario where some
of the frames in the stash are encrypted and others are not. In this case we
would need to mark certain frames as decrypted which appeared to introduce more
complexity than this simple decoupling.
Bug: webrtc:10022
Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c
Reviewed-on: https://webrtc-review.googlesource.com/c/112221
Commit-Queue: Benjamin Wright <benwright@webrtc.org>
Reviewed-by: Philip Eliasson <philipel@webrtc.org>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#25865}diff --git a/video/rtp_video_stream_receiver.h b/video/rtp_video_stream_receiver.h
index a2006d1..ec3f354 100644
--- a/video/rtp_video_stream_receiver.h
+++ b/video/rtp_video_stream_receiver.h
@@ -38,6 +38,9 @@
#include "rtc_base/criticalsection.h"
#include "rtc_base/numerics/sequence_number_util.h"
#include "rtc_base/sequenced_task_checker.h"
+#include "rtc_base/thread_annotations.h"
+#include "rtc_base/thread_checker.h"
+#include "video/buffered_frame_decryptor.h"
namespace webrtc {
@@ -56,7 +59,8 @@
public VCMFrameTypeCallback,
public VCMPacketRequestCallback,
public video_coding::OnReceivedFrameCallback,
- public video_coding::OnCompleteFrameCallback {
+ public video_coding::OnCompleteFrameCallback,
+ public OnDecryptedFrameCallback {
public:
RtpVideoStreamReceiver(
Transport* transport,
@@ -130,6 +134,10 @@
void OnCompleteFrame(
std::unique_ptr<video_coding::EncodedFrame> frame) override;
+ // Implements OnDecryptedFrameCallback.
+ void OnDecryptedFrame(
+ std::unique_ptr<video_coding::RtpFrameObject> frame) override;
+
// Called by VideoReceiveStream when stats are updated.
void UpdateRtt(int64_t max_rtt_ms);
@@ -207,10 +215,13 @@
absl::optional<int64_t> last_received_rtp_system_time_ms_
RTC_GUARDED_BY(rtp_sources_lock_);
- // E2EE Video Frame Decryptor (Optional)
- rtc::scoped_refptr<FrameDecryptorInterface> frame_decryptor_;
- // Set to true on the first successsfully decrypted frame.
- bool has_received_decrypted_frame_ = false;
+ // Used to validate the buffered frame decryptor is always run on the correct
+ // thread.
+ rtc::ThreadChecker network_tc_;
+ // Handles incoming encrypted frames and forwards them to the
+ // rtp_reference_finder if they are decryptable.
+ std::unique_ptr<BufferedFrameDecryptor> buffered_frame_decryptor_
+ RTC_PT_GUARDED_BY(network_tc_);
};
} // namespace webrtc