[Embench] WD Code size compiler tests

Roger Shepherd roger.shepherd at chipless.eu
Wed Dec 8 12:13:24 CET 2021


I’d like to second Jeremy’s comment:

> Even though the tests are originally created to look at code size, we
> should ensure they can be run, that the result is verified and that we
> can get a performance measure.

I know that the difficulty that these requirements add are annoying and can seem unnecessary but over the years I’ve seen too many cases where the code size has been squeezed down impressively but either it no longer works or it runs very slowly. Remember, it’s pretty easy to generate fast dense code if it doesn’t have to function correctly.

Roger

> On 8 Dec 2021, at 09:37, Jeremy Bennett <jeremy.bennett at embecosm.com> wrote:
> 
> On 16/11/2021 11:40, Nidal Faour wrote:
>> Hi All,
>> A while ago we started this thread and I mistakenly replied to Roger only, so there is the reply:
>> 
>> """
>> I agree with what you suggested, these four bullets must be sorted out.
>> As for the “maturity”, I think it was a misleading word. What I meant is looking at code that the compiler could have made it better in the matter of code size. of course, you need someone to investigate the generated assembly and eliminate the parts where it is an ISA limitation.
>> As for the audience, I think we should target developers at first who want to improve tools/compilers, for the same reason I mentioned about ISA limitation, at least until we can come up with fully automated measurements.
>> How do you think we should proceed from here? 
>> """
>> 
>> Following the meeting yesterday, we will prepare a document to better describe the purpose of the test, how did we come up with these tests, and how to use them.
>> We will also define the way to add more tests to the repo. And what it should contain.
> 
> Hi Nidal,
> 
> As we discussed, I think this is a good proposal. Here are some things I
> would like to see.
> 
> Even though the tests are originally created to look at code size, we
> should ensure they can be run, that the result is verified and that we
> can get a performance measure. We should also generate an overall score,
> even though in practice it is individual benchmark results which matter.
> 
> Best wishes,
> 
> 
> Jeremy
> 
>> 
>> Waiting for your feedback/suggestion on this matter.
>> 
>> Best Regards,
>> 
>> Nidal Faour
>> Staff Engineer, R&D Engineering – Firmware & Toolchain, CTO Group
>> 
>> Western Digital®
>> Migdal Tefen 24959, P.O Box 3
>> Email: nidal.faour at wdc.com
>> Office: +972-4-9078756
>> Mobile: +972-50-8867756
>> 
>>> -----Original Message-----
>>> From: Embench <embench-bounces at lists.librecores.org> On Behalf Of
>>> embench-request at lists.librecores.org
>>> Sent: Friday, 14 May 2021 13:40
>>> To: embench at lists.librecores.org
>>> Subject: Embench Digest, Vol 22, Issue 5
>>> 
>>> CAUTION: This email originated from outside of Western Digital. Do not click
>>> on links or open attachments unless you recognize the sender and know that
>>> the content is safe.
>>> 
>>> 
>>> Send Embench mailing list submissions to
>>>       embench at lists.librecores.org
>>> 
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>       https://lists.librecores.org/listinfo/embench
>>> or, via email, send a message with subject or body 'help' to
>>>       embench-request at lists.librecores.org
>>> 
>>> You can reach the person managing the list at
>>>       embench-owner at lists.librecores.org
>>> 
>>> When replying, please edit your Subject line so it is more specific than "Re:
>>> Contents of Embench digest..."
>>> 
>>> 
>>> Today's Topics:
>>> 
>>>  1. Re: WD Code size compiler tests (Roger Shepherd)
>>> 
>>> 
>>> ----------------------------------------------------------------------
>>> 
>>> Message: 1
>>> Date: Fri, 14 May 2021 11:39:29 +0100
>>> From: Roger Shepherd <roger.shepherd at chipless.eu>
>>> To: embench at lists.librecores.org
>>> Subject: Re: [Embench] WD Code size compiler tests
>>> Message-ID: <56EBE553-056C-4AEE-80D7-269757955E59 at chipless.eu>
>>> Content-Type: text/plain; charset="utf-8"
>>> 
>>> I replied directly to Nidal about this which was probably not the right way to
>>> have done things. Prompted by Jeremy’s response, I’ll copy what I said to
>>> Nidal here. I’ll leave it to Nidal to add his response.
>>> 
>>> Nidal,
>>> 
>>> I’ve had a look carefully at a couple of the benchmarks and I think they give
>>> me a feel for what’s been done. I do like the idea of having a set of Embench
>>> compiler tests and these could be a good starting point. Before we put too
>>> much effort into setting things up, I think we need to try agree a set of
>>> achievable and useful objectives. I think we need to cover
>>> 
>>> 1. Audience/Users
>>> 
>>> 2. Specific aspects of compiler that we’re trying to assess
>>> 
>>> 3. Assessment methodology
>>> 
>>> 4. Anything else
>>> 
>>> To get things going, let me make some suggestions - if only that we can have
>>> something concrete to disagree with!
>>> 
>>> 1. People wanting to assess compiler maturity. This includes compiler
>>> developers and users.
>>> 
>>> 2. A set of particular scenarios where compilers in the past have shown a lack
>>> of specific optimisations
>>> 
>>> 3. Compilation of source source code. An automated measurement of the
>>> quality of the benchmark; perhaps size is good enough.
>>> 
>>> It’s not clear that “size” provides a useful metric for comparing maturity
>>> between architectures or between different abi’s on the same architecture. I
>>> suspect that if undertaking this work personally, I’d probably eyeball the code
>>> to see what was going on and not rely on any automated measurement, but I
>>> think for Embench we should do better than that.
>>> 
>>> 4. There must be a check that the code will execute as expected.
>>> 
>>> I don’t trust code that doesn’t execute. But making code executable has it’s
>>> own problems - in particular, the extra code needed may make it difficult to
>>> see what is happening regarding the particular optimisation that is being
>>> checked for.
>>> 
>>> Before we get too dragged into the details of 3 and 4 I do think we need to
>>> check that we have 1, the audience, sorted out. We have a different task on
>>> our hands if we are looking at providing a set of microbenchmarks for
>>> architects and tool developers, or if we are trying benchmarks to help to
>>> select cores and tools.
>>> 
>>> Roger
>>> 
>>> In the light of this my response to Jeremy’s mail is:
>>> 
>>>> On 02/05/2021 11:57, Nidal Faour wrote:
>>>>> Hi all,
>>>>> I would like to have your opinion about opening a new repo under the
>>>>> Embench to hold the test cases done by WD. (located at the following
>>>>> link)
>>>>> https://github.com/westerndigitalcorporation/riscv32-Code-density-tes
>>>>> t-bench
>>>>> <https://github.com/westerndigitalcorporation/riscv32-Code-density-te
>>>>> st-bench>Hi Nidal,
>>>> 
>>>> I'm up for this. I think we need to get buy in from the Embench community.
>>> What do others think?
>>>> 
>>> It needs buy-in and commitment. That means some people need to commit
>>> to working on it, rather than just agreeing it is a good thing.
>>> 
>>>>> My suggestion is to have a new repo to hold these tests and labeled
>>> “compiler benchmark” or whatever better name you may suggest.
>>>> That would be the thing to do. We'd need to make sure it was consistent
>>> with the 7 principles behind Embench. One that springs to mind is the need to
>>> make sure that the benchmarks are self-verifying.
>>> 
>>> Yes, but… see my (second) point 4 above. I think the tests are probably useful
>>> even if they aren’t benchmarks as such (e.g. “here are some useful code
>>> fragments to use to assess compiler maturity/code size”) but that might
>>> beyond the scope of Embench. I this is also tied in with the questions of who
>>> will use them? and what will they use them for.
>>> 
>>>>> Also, I think it would be better to add a new column labeled “Compiler” to
>>> the Embench Baseline Data slide to emphasize that these tests are made to
>>> test the compiler maturity for the specific target.
>>>> Well they test three things - the processor, the compiler and the libraries.
>>> There is always the issue of how much information to put on the summary.
>>>>> Just to remind you, here is the Baseline Data table I’m referring to
>>>>> as an example that I took from one of Jeremy's presentations. (see
>>>>> the last column I’ve added)
>>>>> 
>>>>> Name
>>>>> Comments
>>>>> Orig Source
>>>>> C LOC
>>>>> code size
>>>>> data size
>>>>> time (ms)
>>>>> branch
>>>>> memory
>>>>> compute
>>>>> Compiler
>>>>> aha-mont64
>>>>> Montgomery multiplication
>>>>> AHA
>>>>> 162
>>>>> 1,072
>>>>> 0
>>>>> 4,004
>>>>> low
>>>>> low
>>>>> high
>>>>> 
>>>>> crc32
>>>>> CRC error checking 32b
>>>>> MiBench
>>>>> 101
>>>>> 284
>>>>> 1,024
>>>>> 4,010
>>>>> high
>>>>> med
>>>>> low
>>>>> 
>>>>> cubic
>>>>> Cubic root solver
>>>>> MiBench
>>>>> 125
>>>>> 1,584
>>>>> 0
>>>>> 3,831
>>>>> low
>>>>> med
>>>>> med
>>>>> 
>>>>> 
>>>> I don't think the compiler alone adds much. What matters is all the
>>> supplementary data (in the JSON file) providing all the details to reproduce
>>> the results.
>>>> 
>>> Agreed that providing the information needed to enable reproducibility is key
>>> for benchmarks.
>>> 
>>> Roger
>>> 
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> <http://lists.librecores.org/pipermail/embench/attachments/20210514/761f7
>>> b7f/attachment.htm>
>>> -------------- next part --------------
>>> A non-text attachment was scrubbed...
>>> Name: signature.asc
>>> Type: application/pgp-signature
>>> Size: 833 bytes
>>> Desc: Message signed with OpenPGP
>>> URL:
>>> <http://lists.librecores.org/pipermail/embench/attachments/20210514/761f7
>>> b7f/attachment.sig>
>>> 
>>> ------------------------------
>>> 
>>> Subject: Digest Footer
>>> 
>>> Embench mailing list
>>> Embench at lists.librecores.org
>>> https://lists.librecores.org/listinfo/embench
>>> 
>>> 
>>> ------------------------------
>>> 
>>> End of Embench Digest, Vol 22, Issue 5
>>> **************************************
> 
> 
> -- 
> Cell: +44 (7970) 676050
> SkypeID: jeremybennett
> Twitter: @jeremypbennett
> Email: jeremy.bennett at embecosm.com <mailto:jeremy.bennett at embecosm.com>
> Web: www.embecosm.com <http://www.embecosm.com/>
> PGP key: 1024D/BEF58172FB4754E1 2009-03-20
> 
> -- 
> Embench mailing list
> Embench at lists.librecores.org <mailto:Embench at lists.librecores.org>
> https://lists.librecores.org/listinfo/embench <https://lists.librecores.org/listinfo/embench>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.librecores.org/pipermail/embench/attachments/20211208/4c84ddad/attachment.htm>


More information about the Embench mailing list