[Embench] New metrics for Embench results - Base/Peak and Speed/Rate

Ofer Shinaar Ofer.Shinaar at wdc.com
Mon Mar 8 09:03:16 CET 2021


Hi Ray,
By raw, I am thinking about the basic compiler flags, something like we currently have. But if we take my example, “save-restore” is something unique to RISCV.
I understand that it is basic usage when doing “size” testing, but we can argue that it should be part of the “none-raw” flags or “special target flags.”
Having the “peek/rate” will give us the option to total separate basic flags from none-basic flags… Plus, we can show how each target can boost performance and decrease size just by compiler “tweaks.”

BR,
Ofer

From: Ray Simar <ray.simar at rice.edu>
Sent: Friday, 5 March 2021 03:21
To: Ofer Shinaar <Ofer.Shinaar at wdc.com>
Cc: David Patterson <pattrsn at cs.berkeley.edu>; embench at lists.librecores.org
Subject: Re: [Embench] New metrics for Embench results - Base/Peak and Speed/Rate

Hi Ofer,

Thanks for the ideas.  I was wondering if we might be able to calculate these measures, at least in part, from the raw measures.  I do like the idea of a single top line number as the team is doing now.  But maybe if we offered the right set of raw measurements people could calculate these additional measures as they needed.

Thoughts?

All the best,
Ray

On Mar 4, 2021, at 3:17 AM, Ofer Shinaar <Ofer.Shinaar at wdc.com<mailto:Ofer.Shinaar at wdc.com>> wrote:

Hi Dave,
Its is more an enhancement and less a “problem-solving.”

For base/peak, we can also allow vendors to publish their best result with compiler tweaks, along with the current “base” typical results.
After all, riscv and arm compiler are not the same; for example, we do provide –msave-restore for RV, but we do not give that ARM because it is just a basic flag that we “must use”.
Peak will allow you to drive more flags and expose where we can get with each compiler per target.

For Rate/Speed:
We only publish Speed results, and some targets want to see IPC/throughput as well. Give the “Rate” option will solve that.

Thanks,
Ofer



From: David PATTERSON <pattrsn at cs.berkeley.edu<mailto:pattrsn at cs.berkeley.edu>>
Sent: Thursday, 4 March 2021 03:44
To: Ofer Shinaar <Ofer.Shinaar at wdc.com<mailto:Ofer.Shinaar at wdc.com>>
Cc: embench at lists.librecores.org<mailto:embench at lists.librecores.org>
Subject: Re: [Embench] New metrics for Embench results - Base/Peak and Speed/Rate

What is the problem you're trying to solve with the new metrics?

Dave

On Wed, Mar 3, 2021 at 3:47 AM Ofer Shinaar <Ofer.Shinaar at wdc.com<mailto:Ofer.Shinaar at wdc.com>> wrote:
Hello all,
I would like to bring to the forum a suggestion on including more results metrics as we have on SPEC


1.       Include metric fo Base and Peak:

-          SPEC url or explanation: https://www.spec.org/cpu2017/Docs/overview.html#Q16

-          Currently, we can say we only have “Base”, on IoT results.

-          Having “Peak” will give us a possibility to expose more compiler flags that can boost performance, and we can publish results + links to tools that can “find the best set of compiler flags”

Example for work done by Craig Blackmore (from Embecosm):

o   https://github.com/craigblackmore/opentuner

o   https://www.groundai.com/project/automatically-tuning-the-gcc-compiler-to-optimize-the-performance-of-applications-running-on-embedded-systems/2

2.        Include metric for Speed and Rate

-          SPEC url for explanation: https://www.spec.org/cpu2017/Docs/overview.html#Q15

-          On the embedded space, sometimes throughput is more important than “time”. Those days we can start seeing IoT cores /MCUs that have a multithreaded design.
Also, we see emerging multithreading designs for RISCV. On those designs rate for throughput will be more important than Time.

Thoughts/comments?

Thanks,
Ofer




Ofer Shinaar
Senior Manager, R&D Engineering – Firmware & Toolchain, CTO Group

Western Digital®
Israel, Migdal Tefen 24959, P.O Box 3
Email: Ofer.shinaar at wdc.com<mailto:Ofer.shinaar at wdc.com>
Office: +972-4-9078783
Mobile: +972-52-2836160


--
Embench mailing list
Embench at lists.librecores.org<mailto:Embench at lists.librecores.org>
https://lists.librecores.org/listinfo/embench
--
Embench mailing list
Embench at lists.librecores.org<mailto:Embench at lists.librecores.org>
https://lists.librecores.org/listinfo/embench

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.librecores.org/pipermail/embench/attachments/20210308/c4cd8f4b/attachment.htm>


More information about the Embench mailing list