Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions Documentation/gpu/asahi-usage-stats.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,14 @@ the classic backend uses its hardware configuration's timer frequency.

These are firmware stage envelopes, not hardware shader occupancy measurements.
They can include firmware scheduling, preemption and stalls between the stage's
start and end operations. Intervals are summed without attempting to merge
overlapping commands on the same engine. Missing, zero-start or reversed intervals
are ignored. Failed or cancelled commands do not contribute incomplete intervals;
start and end operations. The shared G17 backend runs several commands of one
queue at once, so their stages overlap. It charges each engine of a queue the
union of that queue's stage intervals (exact, because a queue's stages retire in
the order they started), so a queue never counts more than the wall time on one
engine. Queues of one file still add up. The other backends sum their intervals
without merging overlapping commands on the same engine. Missing, zero-start or
reversed intervals are ignored. Failed or cancelled commands do not contribute
incomplete intervals;
successful stages that retired before a later failure remain counted. An
unexecuted G17 replay contributes only when it eventually retires successfully.
There is no submit-to-completion wall-time fallback in the current backends.
Expand Down Expand Up @@ -74,9 +79,11 @@ scale. Other GPUs print neither key.
Accounting cost
---------------

Each file allocates one three-counter ledger. Queues and pending completions
retain references to it. Retirement adds one relaxed, saturating atomic update
per valid stage, with no new locks or allocations. The native backends reuse
Each file allocates one three-counter ledger, and each G17 queue one three-mark
lane in front of it. Queues and pending completions retain references to them.
Retirement adds one relaxed, saturating atomic update per valid stage, after one
compare-and-swap of the lane's mark on G17, with no new locks or allocations.
The native backends reuse
existing timestamp reads. Classic completion callbacks read two internal cells
per stage; commands without application timestamps also execute two existing
timestamp microsequence operations per stage. Reading fdinfo loads three counters
Expand Down
4 changes: 2 additions & 2 deletions drivers/gpu/drm/asahi/g17/completion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ pub(crate) struct Completion {
/// `compute` alone does not identify the engine.
is_compute: bool,
feed: Option<Arc<super::feed::Feed>>,
usage: Arc<crate::usage::Client>,
usage: Arc<crate::usage::Lane<Arc<crate::usage::Client>>>,
sources: Option<Arc<super::sync_input::Sources>>,
#[pin]
vm_job: Mutex<Option<mmu::VmJobGuard>>,
Expand Down Expand Up @@ -418,7 +418,7 @@ impl Completion {
host_predecessors: bool,
feed: Option<Arc<super::feed::Feed>>,
sources: Option<Arc<super::sync_input::Sources>>,
usage: Arc<crate::usage::Client>,
usage: Arc<crate::usage::Lane<Arc<crate::usage::Client>>>,
) -> Result<Arc<Self>> {
let fence = fence::independent(contexts)?;
let ordered = !compute && (host_predecessors || destinations.iter().any(Option::is_some));
Expand Down
5 changes: 3 additions & 2 deletions drivers/gpu/drm/asahi/g17/queue/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,8 @@ impl Frontend {
pub(crate) struct Queue<B: Backend> {
dev: AsahiDevRef,
context: Arc<Context>,
usage: Arc<crate::usage::Client>,
/// This queue's engine time: the union of its overlapping stages, charged to its file.
usage: Arc<crate::usage::Lane<Arc<crate::usage::Client>>>,
backend: Arc<B>,
render: Option<Scheduler<B>>,
compute: Option<Scheduler<B>>,
Expand Down Expand Up @@ -139,7 +140,7 @@ impl<B: Backend> Queue<B> {
Ok(Self {
dev: dev.into(),
context,
usage,
usage: Arc::new(crate::usage::Lane::new(usage), GFP_KERNEL)?,
backend,
render,
compute: None,
Expand Down
160 changes: 160 additions & 0 deletions drivers/gpu/drm/asahi/usage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
//! Queues retain their file's ledger until their last completion. Only authenticated,
//! successful completion timestamps are charged, before signalling the completion fence.
//! These are command timer ticks, not shader clock cycles. Never export them as drm-cycles.
//!
//! A backend whose firmware overlaps the commands of one queue charges through a [`Lane`]
//! per queue, so each engine of a queue counts the union of its stage intervals.

use core::sync::atomic::{AtomicU64, Ordering};

Expand Down Expand Up @@ -56,6 +59,57 @@ impl Client {
}
}

/// A stage this far behind its lane's mark is not an overlap, since no stage runs this
/// long: the timestamps jumped, so the mark restarts at the stage instead of holding every
/// later charge back.
const DISCONTINUITY_S: u64 = 60;

/// The engine time of one queue, charged to its file's [`Client`].
///
/// Firmware can run several commands of one queue at once. A queue that keeps several
/// commands in flight sees their stage intervals overlap, and their sum then exceeds the
/// time the engine was busy: three seconds of compute per second at depth 4 on the A18 Pro.
/// A lane charges each engine the union of the queue's stage intervals instead. A queue's
/// stages retire in the order they started, so a mark of the latest end charged per engine
/// gives the exact union: a stage adds only its part after the mark. Queues of one file
/// still add up, one union each.
pub(crate) struct Lane<P> {
client: P,
covered: [AtomicU64; 3],
}

impl<P: core::ops::Deref<Target = Client>> Lane<P> {
pub(crate) fn new(client: P) -> Self {
Self {
client,
covered: [const { AtomicU64::new(0) }; 3],
}
}

/// As [`Client::add`], for one retired stage of this lane's queue: only the part of
/// `start..end` after the engine's mark is charged, and the mark moves to `end`.
pub(crate) fn add(&self, engine: Engine, start: u64, end: u64, frequency_hz: u32) {
if start == 0 || end <= start || frequency_hz == 0 {
return;
}
let limit = u64::from(frequency_hz).saturating_mul(DISCONTINUITY_S);
let mark = &self.covered[engine as usize];
let mut covered = mark.load(Ordering::Relaxed);
let from = loop {
let jumped = covered.saturating_sub(start) > limit;
let from = if jumped { start } else { start.max(covered) };
if from >= end {
return;
}
match mark.compare_exchange_weak(covered, end, Ordering::Relaxed, Ordering::Relaxed) {
Ok(_) => break from,
Err(value) => covered = value,
}
};
self.client.add(engine, from, end, frequency_hz);
}
}

fn ticks_to_ns(ticks: u64, frequency_hz: u32) -> u64 {
const NSEC_PER_SEC: u64 = 1_000_000_000;
let hz = u64::from(frequency_hz);
Expand Down Expand Up @@ -140,6 +194,112 @@ mod tests {
assert_eq!(client.snapshot()[2], u64::MAX);
}

/// A queue `depth` commands deep: one command retires every `period` ticks, and each
/// stays in flight for `depth` periods, so neighbouring intervals overlap.
fn pipeline(lane: &Lane<&Client>, base: u64, period: u64, depth: u64, commands: u64) {
for i in 0..commands {
let start = base + i * period;
lane.add(Engine::Compute, start, start + depth * period, HZ);
}
}

#[test]
fn a_deep_queue_counts_its_busy_time_once() {
// 1,181 us per command, as the A18 Pro's ALU load at depth 4.
const PERIOD: u64 = 28_344;
for depth in [1, 4, 8] {
let client = Client::new();
let lane = Lane::new(&client);
pipeline(&lane, 1_000, PERIOD, depth, 1_000);
let wall = ticks_to_ns((1_000 - 1 + depth) * PERIOD, HZ);
assert_eq!(client.snapshot()[2], wall, "depth {depth}");
// The sum of the intervals is `depth` times the busy time.
let summed = Client::new();
for i in 0..1_000 {
let start = 1_000 + i * PERIOD;
summed.add(Engine::Compute, start, start + depth * PERIOD, HZ);
}
assert_eq!(summed.snapshot()[2], 1_000 * ticks_to_ns(depth * PERIOD, HZ));
}
}

#[test]
fn idle_gaps_and_disjoint_stages() {
let client = Client::new();
let lane = Lane::new(&client);
lane.add(Engine::Compute, 24, 48, HZ);
lane.add(Engine::Compute, 96, 120, HZ);
lane.add(Engine::Compute, 100, 110, HZ); // inside the previous stage
lane.add(Engine::Compute, 0, 100, HZ); // no start: nothing
assert_eq!(client.snapshot(), [0, 0, 2_000]);
// A covered stage leaves the mark where it was.
lane.add(Engine::Compute, 104, 144, HZ);
assert_eq!(client.snapshot(), [0, 0, 3_000]);
}

#[test]
fn engines_keep_their_own_marks() {
let client = Client::new();
let lane = Lane::new(&client);
lane.add(Engine::Vertex, 1, 25, HZ);
lane.add(Engine::Fragment, 13, 37, HZ);
lane.add(Engine::Vertex, 13, 37, HZ);
assert_eq!(client.snapshot(), [1_500, 1_000, 0]);
}

#[test]
fn queues_of_one_file_add_up() {
let client = Client::new();
let a = Lane::new(&client);
let b = Lane::new(&client);
a.add(Engine::Compute, 1, 49, HZ);
b.add(Engine::Compute, 1, 49, HZ);
a.add(Engine::Compute, 25, 49, HZ);
assert_eq!(client.snapshot(), [0, 0, 4_000]);
}

#[test]
fn a_timestamp_jump_restarts_the_mark() {
let client = Client::new();
let lane = Lane::new(&client);
let far = 61 * u64::from(HZ);
lane.add(Engine::Compute, 1, far, HZ);
let before = client.snapshot()[2];
// Far behind the mark: charged in full, and the mark follows it.
lane.add(Engine::Compute, 24, 48, HZ);
lane.add(Engine::Compute, 36, 72, HZ);
assert_eq!(client.snapshot()[2] - before, 2_000);
// Within the limit, a stage behind the mark is an overlap.
let near = Client::new();
let lane = Lane::new(&near);
lane.add(Engine::Compute, 1, 59 * u64::from(HZ), HZ);
let before = near.snapshot()[2];
lane.add(Engine::Compute, 24, 48, HZ);
assert_eq!(near.snapshot()[2], before);
}

#[test]
fn concurrent_retirements_never_exceed_the_union() {
let client = std::sync::Arc::new(Client::new());
let lane = std::sync::Arc::new(Lane::new(client.clone()));
let threads: std::vec::Vec<_> = (0..8)
.map(|_| {
let lane = lane.clone();
std::thread::spawn(move || {
for i in 0..10_000u64 {
lane.add(Engine::Compute, 24 + i * 24, 24 + i * 24 + 96, HZ);
}
})
})
.collect();
for thread in threads {
thread.join().unwrap();
}
let union = ticks_to_ns((10_000 - 1) * 24 + 96, HZ);
let charged = client.snapshot()[2];
assert!(charged > 0 && charged <= union, "{charged} > {union}");
}

#[test]
fn concurrent_queues_share_one_file() {
let client = std::sync::Arc::new(Client::new());
Expand Down
Loading