Some of the state that's managed in VideoStreamEncoder, is updated
and accessed on the encoder queue, but a method that's used for testing
only (GetAdaptationResources()), represents a race between PostTask
operations that update state and checking for said state on the worker
thread.
This CL removes Wait() operations related to adaptation resources from
the common path and puts one in the test path instead.
Bug: webrtc:13612
Change-Id: Ie3e018e815e24951bc0634ed70de17eaf336a508
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/249220
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Tomas Gunnarsson <tommi@webrtc.org>
Cr-Commit-Position: refs/heads/main@{#35797}
Resource adaptation needs refactoring for async adaptations. For now
the resource adaptation processor can work on the encoder thread, until
it is refactored to support async adaptation.
Bug: webrtc:11867
Change-Id: I9c46da356db19c0fd52748c999ccb216f2ca923b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/182040
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31991}
It was not used by any class and all future uses can use the
VideoSourceRestrictionsListener.
Bug: webrtc:11834
Change-Id: I5c71b93cc503f458dce0ccdd78b91b5a1debc56d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/181062
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31896}
This is one less dependency on the task queue, and will make
things like removing resources and cleanup much easier in the future.
Bug: webrtc:11754
Change-Id: I732f1935d1b58ffe09ca2a2bf59beebc1930214d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178869
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31686}
This is a step needed for multi-stream and new mitigations. It also
cleans up needing to signal adaptation changes in mutiple places
from ResourceAdaptationProcessor.
R=hbos@webrtc.org
Bug: webrtc:11754
Change-Id: Ib185dc9f66fbb4a087eb9e970c68c3f47eafb17f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178874
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31684}
It has no relevance in ResourceAdaptationProcessorInterface.
The logic moves to an interaction directly on the VideoStreamAdapter.
Bug: webrtc:11700
Change-Id: I4c7b3e1e1361722a69b71e8f9bde33f5909c011a
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178380
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31634}
This patch creates a new class which provides the DegradationPreference
thread safe to all classes that need if (BalancedConstraint and
QpScalerResource). It also broadcasts to all listeners when the
preferences are updated, so the ResourceAdaptationProcessor can update
the video if needed.
In future work, this could be used to remove the need for two task
queues for the VideoStreamEncoder resources.
Bug: webrtc:11700
Change-Id: I05480db8b7157b5643f6f86ec9c64850839b3e76
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/177522
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31623}
This moves this responsibility from the ResourceAdaptaitonProcessor
to the VideoStreamAdapter. A new interface for listening to adaptation
changes was added, and the ResourceAdaptationProcessor now listens on
the VideoStreamAdapter for those changes.
This means that going forward,
1. We can do adaptations outside of resource limitations, like setting
prior adaptations on a resource like initial frame dropper is designed
to.
2. Adaptations can be on a different queue than the
ResourceAdaptaitonProcessor's queue since updates are pushed through
the listener.
Bug: webrtc:11700
Change-Id: I6de0dec748dba095e702f0b9893c5afa50b51aa9
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176859
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31615}
When a resource is removed from the ResourceAdaptationProcessor,
and the resource is the most limited resource, we reset the current
adaptation to the next most limited resource level. In the case that
this resource is the only adapted resource then we reset all adaptations.
Bug: webrtc:11636
Change-Id: I29acc5a3934f42f00db79b3bb2171156b1313937
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176406
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31520}
This CL adds AddAdaptationResource to Call and
AddAdaptationResource/GetAdaptationResources method to relevant
VideoSendStream and VideoStreamEncoder interfaces and implementations.
Unittests are added to ensure that resources can be added to the Call
both before and after the creation of a VideoSendStream and that the
resources always gets added to the streams.
In a follow-up CL, we will continue to plumb the resources all the way
to PeerConnectionInterface, and an integration test will then be added
to ensure that injected resources are capable of triggering adaptation.
Bug: webrtc:11525
Change-Id: I499e9c23c3e359df943414d420b2e0ce2e9b2d56
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/177002
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31499}
A more detailed explaination is in the bug, but this changes
the way that adaptation happens when multiple resources are
limited. Only the one that is most limited can trigger an
adaptation up. If multiple resources are most limited both
need to underuse to adapt up.
Some of the changes in this patch to make it all work:
* VideoStreamEncoder unittests that did not reflect this
new behaviour have been changed.
* PeekNextRestrictions returns the adaptation counters as
well as the restrictions.
* Adaptation statstics have changed so that when adapting
up all resources are tagged as triggering the adaptation.
Additionally the statistics for the current adaptation is
now the total number of adaptations per reason, rather then
the number of adaptations due to that reason.
* PreventAdaptUpDueToActiveCounts is removed as most limited
resource is a strong implementation of that.
Bug: webrtc:11553
Change-Id: If1545a201c8e019598edf82657a1befde8b05268
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176128
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31497}
This is a prerequisite to implementing it externally.
Bug: webrtc:11525
Change-Id: I9cb3b4418396485d3eb9f25cafa51cbff6db7817
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176401
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31418}
ResourceListener::OnResourceUsageStateMeasured() now takes
ResourceUsageState as argument, making Resource::UsageState()
superfluous.
With the existing "fire-and-forget" behavior of always clearing usage
state on reacting to a signal, there is no longer a need to call
ClearUsageState() so this too is removed. (We may want to have a
callback in the future to hint to the Resource that it is a good idea
to clear internal measurement samples, i.e. because the load of the
system is about to change, but we can revisit that when we need it.)
Moving the usage state to the callback has the benefit of getting rid
of the assumption that UsageState() has to return the same value every
time it is called in the same task.
This CL is also the final nail in the coffin for Resource needing to
know about the adaptation task queue: ResourceAdaptationProcessor's
ResourceListener now takes care of posting to the adaptation task
queue. To support this, the processor's SequenceChecker is replaced
by a TaskQueueBase pointer.
Bug: webrtc:11525, webrtc:11618
Change-Id: I2277e71cc3759c85b62465020935603f03792c94
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176376
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31416}
IsAdaptationUpAllowed is moved from Resource to AdaptationConstraint.
OnAdaptationApplied is moved from Resource to AdaptationListener.
In a future CL, Resource will be moved to api/, but
AdaptationConstraint and AdaptationListener will stay in call/.
The processor, encode stream and manager are updated to keep track of
both resources, constraints and listeners. Fakes and tests are updated.
After this CL, the manager's inner classes that prevent adaptation
implement AdaptationConstraint instead of Resource.
Bug: webrtc:11525
Change-Id: Ie9cd5b1ba7d8e161951e131ab8f6bd9d5cf765bf
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176368
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31409}
In a future CL, adaptation processing and stream encoder resource
management will happen on different task queues. When this is the case,
asynchronous tasks will be posted in both directions and some resources
will have internal states used on multiple threads.
This CL makes the Resource class reference counted in order to support
posting tasks to a different threads without risk of use-after-free
when a posted task is executed with a delay. This is preferred over
WeakPtr strategies because WeakPtrs are single-threaded and preferred
over raw pointer usage because the reference counted approach enables
more compile-time and run-time assurance. This is also "future proof";
when resources can be injected through public APIs, ownership needs to
be shared between libwebrtc and the application (e.g. Chrome).
To reduce the risk of making mistakes in the future CL, sequence
checkers and task queue DCHECKs are added as well as other DCHECKs to
make sure things have been cleaned up before destruction, e.g:
- Processor gets a sequence checker. It is entirely single-threaded.
- Processor must not have any attached listeners or resources on
destruction.
- Resources must not have any listeners on destruction.
- The Manager, EncodeUsageResource and QualityScalerResource DCHECKs
they are running on the encoder queue.
- TODOs are added illustrating where we want to add PostTasks in the
future CL.
Lastly, upon VideoStreamEncoder::Stop() we delete the
ResourceAdaptationProcessor. Because the Processor is already used in
posted tasks, some if statements are added to ensure the Processor is
not used after destruction.
Bug: webrtc:11542, webrtc:11520
Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31217}
This replaces references to the ResourceAdaptationProcessor with
references to its interface. This would make it possible to have
alternative implementations or inject fake/mock implementations for
testing.
The VideoStreamAdapter is still responsible for constructing the
ResourceAdaptationProcessor, but beyond construction it is agnostic
towards the implementation.
With this CL, I claim https://crbug.com/webrtc/11222 complete.
TBR=ilnik@webrtc.org
Bug: webrtc:11222
Change-Id: I6e7a73bf1d0b5e97bc694f66180a747b27ffb018
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174160
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31148}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This gets to the heart of unblocking call-level adaptation, largely
made possible due to the previous CLs in the chain.
The parts of the code that are responsible for responding to resource
usage signals, obtaining adaptations and applying them are moved to
ResourceAdaptationProcessor in call/adaptation/.
The parts of the code that are responsible for managing
VideoStreamEncoder-specific resources stay inside the
VideoStreamEncoderResourceManager class in video/adaptation/.
After this CL lands it should soon be possible to move the Processor
over to a separate task queue and let the Manager stay on the encoder
queue if PostTasks are added for communication between the two objects.
Bug: webrtc:11172
Change-Id: Ifa212467b4afd16e7ebfb9adfe17d2dca1cb7d67
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173021
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31105}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
By pushing VideoAdaptationCounters updates on VideoSourceRestrictions
changes, alongside the Resource* that triggered the adaptation, we are
able to update |active_counts_| without an explicit dependency on the
VideoStreamAdapter. This allows a future CL to split up "processor"
logic from "video stream encoder resource and active counts" logic,
which will ultimately be necessary in order to do processing on a
"processing queue" and encoder and stats logic on the "encoder queue".
If the restrictions got cleared by an API call
(ResetVideoSourceRestrictions() or SetDegradationPreference()) we pass
null as the "reason_resource". This allows is to clear the
active_counts_, and the code that invokes
OnVideoSourceRestrictionsUpdated() does not have to be aware of
active_counts_ (needed to split the processor module in two).
Bug: webrtc:11172
Change-Id: Icab6d5121c0ebd27d2a00f1bffc8191f8f05f562
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173000
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31103}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The "input state" of a VideoStream, needed for adaptation and
decision-making, are: source resolution and frame rate, codec type and
min pixels per frame (based on encoder scaling settings). These values
are modified on the encoder queue of the VideoStreamEncoder.
But in order to unblock call-level adaptation processing, where
adaptation and decision making happens off the encoder queue, a snapshot
of the input states need to be available at point of processing:
introducing the VideoStreamInputState.
In this CL, the VideoStreamInputStateProvider is added to provide input
state snapshots across threads based on input from VideoStreamEncoder
and VideoStreamEncoderObserver.
The input state's HasInputFrameSizeAndFramesPerSecond() can now be
DCHECKed inside the VideoStreamAdapter in favor of having less
Adaptation::Status codes. Whether input is "sufficient" for adaptation
is now the responsibility of the Processor. (Goal: adapter is purely a
Adaptation generator and apply-er.)
Somewhat tangental, this CL also deletes VideoStreamEncoder-specific
methods from ResourceAdaptationProcessorInterface making them an
implementation detail of ResourceAdaptationProcessor. In a future CL,
the "processor" will be split up into a "processor" part and a "video
stream encoder resource manager" part - more on that later.
Bug: webrtc:11172
Change-Id: Id9b158f569db0140b75360aaf0f7e2e28fb924f4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172928
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31098}
