Calculate the RMS audio level of audio packets being sent before
invoking an encoded frame transform, and pass them with the encode frame
object.
Before this, the audio level was calculated at send time by having rms_levels_ look at all audio samples encoded since the last send. This
is fine without a transform, as this is done synchronously after
encoding, but with an async transform which might take arbitrarily long,
we could end up marking older audio packets with newer audio levels, or
not at all.
This also makes things work correctly if external encoded frames are
injected from elsewhere to be sent, and exposes the AudioLevel on the
TransformableFrame interface.
Bug: chromium:337193823, webrtc:42226202
Change-Id: If55d2c1d30dc03408ca9fb0193d791db44428316
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/349263
Reviewed-by: Jakob Ivarsson <jakobi@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tony Herre <herre@google.com>
Cr-Commit-Position: refs/heads/main@{#42193}
This is the first step in implementing custom codecs in SDP.
Bug: none
Change-Id: I7789478208a769eaefd58b410ae6f488c604594d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/348662
Commit-Queue: Harald Alvestrand <hta@webrtc.org>
Reviewed-by: Tony Herre <herre@google.com>
Cr-Commit-Position: refs/heads/main@{#42171}
Add a StartShortCircuiting() callback to allow clients which have
configured Encoded Transforms when creating a PeerConnection to have
all frames skip the transform. This offers a zero cost path for streams
which don't need transforms.
This is preferable to uninstalling/not installing the transform to allow
implementing the behaviour in
https://w3c.github.io/webrtc-encoded-transform/#stream-creation -
giving web apps a chance to configure transforms within a short window
(before the next JS event loop run, so usually sub-millisecond) after stream creation, without any untransformed frames passing.
Usage in Chromium: crrev.com/c/5040731
Bug: chromium:1502781
Change-Id: I803477db1df51e80bdedf6c84d2d3695b088de83
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/327601
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tony Herre <herre@google.com>
Cr-Commit-Position: refs/heads/main@{#41184}
This is a reland of commit 3ea9fc4cd8
Original change's description:
> Make frame transformer MimeType pure virtual again
>
> after both audio and video have been implemented.
>
> BUG=webrtc:15579
>
> Change-Id: Ib52e8f67292259cbf7497a884672de72f3003282
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/326162
> Commit-Queue: Philipp Hancke <phancke@microsoft.com>
> Reviewed-by: Harald Alvestrand <hta@webrtc.org>
> Reviewed-by: Tony Herre <herre@google.com>
> Cr-Commit-Position: refs/heads/main@{#41114}
BUG=webrtc:15579
Change-Id: Ia020149cba3045022b539f290565d6c1d0e813ce
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/326880
Auto-Submit: Mirko Bonadei <mbonadei@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#41121}
after both audio and video have been implemented.
BUG=webrtc:15579
Change-Id: Ib52e8f67292259cbf7497a884672de72f3003282
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/326162
Commit-Queue: Philipp Hancke <phancke@microsoft.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Reviewed-by: Tony Herre <herre@google.com>
Cr-Commit-Position: refs/heads/main@{#41114}
This change replaces type of absolute_capture_timestamp_ms_ in
TransformableOutgoingAudioFrame from int to optional uint and makes
the function AbsoluteCaptureTimestamp() inside
TransformableAudioFrameInterface pure virtual.
Bug: webrtc:14949
Change-Id: Id3bdbcba63a5f91105ab198208e4f2b11eb3c7db
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/319000
Commit-Queue: Palak Agarwal <agpalak@google.com>
Reviewed-by: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#40814}
Move the SetRTPTimestamp method from TransformableAudioFrameInterface
to the base class, so that RTPTimestamps can also be modified on encoded
video frames.
Bug: webrtc:14709
Change-Id: I355be527c2be201c9201e04c431394c962237140
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/310781
Commit-Queue: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Reviewed-by: Palak Agarwal <agpalak@google.com>
Cr-Commit-Position: refs/heads/main@{#40378}
Instead switch to specific getters, or methods only defined on specific implementations rather than part of the public API.
Once uses are removed from Chromium, I'll mark GetHeader() deprecated
and eventually remove it.
Bug: chromium:1456628
Change-Id: I19b80489b3a0322c201e24994494cfbb742ee13e
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/309780
Commit-Queue: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#40344}
Make outgoing encoded audio frames inherit from the same Audio interface
that incoming frames inherit from, to align them and make it possible to
eg clone frames regardless of their direction.
Also begin removing GetHeader() from the Audio interface, replacing it
with getters for the specific values we actually need to propagate in
the API: sequence number and CSRCs. This makes it much easier to treat
incoming and outgoing frames the same, even if they don't have full
RtpHeaders prepared at the point of the transform.
Bug: chromium:1453226
Change-Id: Ib5b39b30dea8a378b3b26efb1589dfd64741d201
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/308141
Commit-Queue: Tony Herre <herre@google.com>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Reviewed-by: Palak Agarwal <agpalak@google.com>
Cr-Commit-Position: refs/heads/main@{#40309}
This can be done now as the function SetRTPTimestamp is now overriden
in blink MockTransformableAudioFrame.
Change-Id: I4fa4cb81d0282fea864818f0f2d9a5ed881a5d30
Bug: webrtc:14709
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/308361
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Palak Agarwal <agpalak@google.com>
Cr-Commit-Position: refs/heads/main@{#40257}
This change will make it possible to let us modify timestamp in
RTCEncodedAudioFrame.
Change-Id: I97e9571c258fd718d6c211014f1476ca46c78097
Bug: webrtc:14709
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/307501
Reviewed-by: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Palak Agarwal <agpalak@google.com>
Cr-Commit-Position: refs/heads/main@{#40238}
Chromium uses have been migrated to Metadata(), so we should be clear.
Other projects can easily migrate similarly.
Bug: chromium:1420245
Change-Id: I150654812676dabd5c957cff00d40d4c95eaf5d6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295481
Commit-Queue: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#39455}
It was marked deprecated on Feb 9th, ~3 weeks ago.
Bug: chromium:1414370
Change-Id: I251b91984ca9a958e221f6eaf01c63b05c5a7a48
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295506
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tony Herre <herre@google.com>
Cr-Commit-Position: refs/heads/main@{#39422}
Add a method to TransformableVideoFrameInterface which returns a new
instance of VideoFrameMetadata which the caller can move and use as
they like.
This will replace the existing GetMetadata which returns a dangerous const ref to a field which might change if someone calls SetMetadata
etc. That method will be deprecated as soon as we've migrated Chromium
usages.
Bug: webrtc:14708
Change-Id: Id7c15f33d6ec28c4a975ce250cdc791d7a3087bc
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/292940
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tony Herre <herre@google.com>
Reviewed-by: Tove Petersson <tovep@google.com>
Cr-Commit-Position: refs/heads/main@{#39403}
This CL propagates capture_time_identifier introduced in
webrtc::VideoFrame and propagates it to EncodedImage. For use cases
involving EncodedTransforms, this identifier is further propagated to
TransformableVideoSenderFrame.
VideoEncoder::Encode function is overriden by each encoder. Each of
these overriden functions needs to be changed so that they can handle
this new identifier and propagate its value in the created EncodedImage.
Change-Id: I5bea4c5a3fe714f1198e497a4bcb5fd059afe516
Bug: webrtc:14878
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/291800
Reviewed-by: Tony Herre <herre@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Palak Agarwal <agpalak@google.com>
Cr-Commit-Position: refs/heads/main@{#39374}
RTPVideoHeader is changed to non-const to allow modifying it. We want
to do this when implementing setMetadata() in JavaScript or when
refactoring clone() as "construct + set bytes + setMetadata".
Unblocks
https://chromium-review.googlesource.com/c/chromium/src/+/4164979.
Bug: webrtc:14709
Change-Id: I6089df9c03e9aa33feeb0830dd240dd456cb565e
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/290981
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#39113}
RTPHeader is not exported, so the TransformableIncomingAudioFrame can't be mocked in chrome tests, using a getter instead.
Bug: chromium:1247260
Change-Id: I2af4e6a88b3f4772b3bb50ee0ae9d5c80fed3ae4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/278785
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tomas Gunnarsson <tommi@webrtc.org>
Reviewed-by: Tomas Gunnarsson <tommi@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#38352}
Currently the implementation of FrameTransformers uses distinct,
incompatible types for recevied vs about-to-be-sent frames. This adds a
flag in the interface so we can at least check that we are being given
the correct type. crbug.com/1250638 tracks removing the need for this.
Chrome will be updated after this to check the direction flag and provide
a javascript error if the wrong type of frame is written into the
encoded insertable streams writable stream, rather than crashing.
Bug: chromium:1247260
Change-Id: I9cbb66962ea0718ed47c5e5dba19a8ff9635b0b1
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/232301
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Tony Herre <toprice@chromium.org>
Cr-Commit-Position: refs/heads/main@{#35100}
This reverts commit eb89027733.
Reason for revert: We got a successful WebRTC roll into Chromium at last. Relanding, as the issue should be fixed in Chromium by now.
TBR=hta@webrtc.org,philipp.hancke@googlemail.com
Original change's description:
> Revert "frame transformer: make GetPayloadType pure virtual again"
>
> This reverts commit 209ac5fd95.
>
> Reason for revert: Breaks WebRTC autoroll presubmit:
> https://chromium-review.googlesource.com/c/chromium/src/+/3134502
> Example failure https://ci.chromium.org/ui/p/chromium/builders/try/mac-rel/775468/overview
>
> ../../buildtools/third_party/libc++/trunk/include/__memory/unique_ptr.h:725:32: error: allocating an object of abstract class type 'testing::NiceMock<blink::(anonymous namespace)::MockTransformableVideoFrame>'
> return unique_ptr<_Tp>(new _Tp(_VSTD::forward<_Args>(__args)...));
> ^
> ../../third_party/blink/renderer/platform/peerconnection/rtc_encoded_video_stream_transformer_test.cc:69:26: note: in instantiation of function template specialization 'std::make_unique<testing::NiceMock<blink::(anonymous namespace)::MockTransformableVideoFrame>>' requested here
> auto mock_frame = std::make_unique<NiceMock<MockTransformableVideoFrame>>();
> ^
> ../../third_party/webrtc/api/frame_transformer_interface.h:36:19: note: unimplemented pure virtual method 'GetPayloadType' in 'NiceMock'
> virtual uint8_t GetPayloadType() const = 0;
> ^
>
>
> Original change's description:
> > frame transformer: make GetPayloadType pure virtual again
> >
> > after chrome was updated in
> > https://chromium-review.googlesource.com/c/chromium/src/+/3103323
> >
> > BUG=webrtc:13077
> >
> > Change-Id: I7e5ff6aaae81c5dcfbaa41b09ef01bc95bb7251a
> > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/230143
> > Reviewed-by: Harald Alvestrand <hta@webrtc.org>
> > Commit-Queue: Philipp Hancke <philipp.hancke@googlemail.com>
> > Cr-Commit-Position: refs/heads/main@{#34877}
>
> # Not skipping CQ checks because original CL landed > 1 day ago.
>
> Bug: webrtc:13077
> Change-Id: I6b2e4e2804890c857f1f832a6a4faa614ec026c4
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/230920
> Reviewed-by: Olga Sharonova <olka@webrtc.org>
> Reviewed-by: Harald Alvestrand <hta@webrtc.org>
> Commit-Queue: Olga Sharonova <olka@webrtc.org>
> Cr-Commit-Position: refs/heads/main@{#34891}
# Not skipping CQ checks because this is a reland.
Bug: webrtc:13077
Change-Id: I8414f74be87aad62166a95fac0cd400257fd25a4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/231120
Reviewed-by: Olga Sharonova <olka@webrtc.org>
Commit-Queue: Olga Sharonova <olka@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#34901}
This reverts commit 209ac5fd95.
Reason for revert: Breaks WebRTC autoroll presubmit:
https://chromium-review.googlesource.com/c/chromium/src/+/3134502
Example failure https://ci.chromium.org/ui/p/chromium/builders/try/mac-rel/775468/overview
../../buildtools/third_party/libc++/trunk/include/__memory/unique_ptr.h:725:32: error: allocating an object of abstract class type 'testing::NiceMock<blink::(anonymous namespace)::MockTransformableVideoFrame>'
return unique_ptr<_Tp>(new _Tp(_VSTD::forward<_Args>(__args)...));
^
../../third_party/blink/renderer/platform/peerconnection/rtc_encoded_video_stream_transformer_test.cc:69:26: note: in instantiation of function template specialization 'std::make_unique<testing::NiceMock<blink::(anonymous namespace)::MockTransformableVideoFrame>>' requested here
auto mock_frame = std::make_unique<NiceMock<MockTransformableVideoFrame>>();
^
../../third_party/webrtc/api/frame_transformer_interface.h:36:19: note: unimplemented pure virtual method 'GetPayloadType' in 'NiceMock'
virtual uint8_t GetPayloadType() const = 0;
^
Original change's description:
> frame transformer: make GetPayloadType pure virtual again
>
> after chrome was updated in
> https://chromium-review.googlesource.com/c/chromium/src/+/3103323
>
> BUG=webrtc:13077
>
> Change-Id: I7e5ff6aaae81c5dcfbaa41b09ef01bc95bb7251a
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/230143
> Reviewed-by: Harald Alvestrand <hta@webrtc.org>
> Commit-Queue: Philipp Hancke <philipp.hancke@googlemail.com>
> Cr-Commit-Position: refs/heads/main@{#34877}
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: webrtc:13077
Change-Id: I6b2e4e2804890c857f1f832a6a4faa614ec026c4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/230920
Reviewed-by: Olga Sharonova <olka@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Commit-Queue: Olga Sharonova <olka@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#34891}
This reverts commit 576ad5d510.
Reason for revert: Causes compile error in Chrome.
Original change's description:
> Make TransformableVideoFrameInterface::GetMetadata pure virtual.
>
> GetMetadata() has been implemented downstream and can be made pure
> virtual.
>
> Bug: chromium:1069295
> Change-Id: I62a3be6106552d2d82d8c413c6f523d31626b0d8
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175001
> Commit-Queue: Marina Ciocea <marinaciocea@webrtc.org>
> Reviewed-by: Harald Alvestrand <hta@webrtc.org>
> Cr-Commit-Position: refs/heads/master@{#31281}
TBR=hta@webrtc.org,marinaciocea@webrtc.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: chromium:1069295
Change-Id: I5915270d5b8dab9fc30a07f22fddedb29beca01a
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175620
Reviewed-by: Guido Urdaneta <guidou@webrtc.org>
Commit-Queue: Guido Urdaneta <guidou@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31304}
GetMetadata() has been implemented downstream and can be made pure
virtual.
Bug: chromium:1069295
Change-Id: I62a3be6106552d2d82d8c413c6f523d31626b0d8
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175001
Commit-Queue: Marina Ciocea <marinaciocea@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31281}
Define VideoHeaderMetadata, containing a subset of the metadata in RTP
video header, and expose it the TransformableVideoFrameInterface, to
enable web application to compute additional data according to their own
logic, and eventually remove GetAdditionalData() from the interface.
Bug: chromium:1069295
Change-Id: Id85b494a72cfd8bdd4c0614844b9f0ffae98c956
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174822
Commit-Queue: Marina Ciocea <marinaciocea@webrtc.org>
Reviewed-by: Danil Chapovalov <danilchap@webrtc.org>
Reviewed-by: Magnus Flodman <mflodman@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31265}
Add new methods in the FrameTransformerInterfaces, passing the ssrc on
registering the transformed frame callback, to associate separate frame
transformer sinks for each ssrc. Same for unregister.
Bug: chromium:1065838
Change-Id: I8a406815e9d0cce5199f9df06c286d8b10d75b4d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173183
Commit-Queue: Marina Ciocea <marinaciocea@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Reviewed-by: Magnus Flodman <mflodman@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31047}
Add a new frame interface to be used by frame transformers in Insertable
Streams. TransformableFrameInterface will replace
video_coding::EncodedFrame in a follow up CL, once downstream
dependecies are updated to use the new interface.
Until the functions using video_coding::EncodedFrame are removed from
the API, the video sender and receiver frame transformer delegates call
both function versions to avoid breaking tests downstream.
The TransformableFrameInterface will be used for both audio and video
frame transformers in follow-up CLs.
Bug: webrtc:11380
Change-Id: I9389a8549c156e13b1d8c938ff51eaa69c502f33
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/171863
Commit-Queue: Marina Ciocea <marinaciocea@webrtc.org>
Reviewed-by: Magnus Flodman <mflodman@webrtc.org>
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Reviewed-by: Danil Chapovalov <danilchap@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30941}
