[Embench] Notes from Embench call 20 July 2020

Jeremy Bennett jeremy.bennett at embecosm.com
Mon Jul 20 17:55:34 CEST 2020


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

These are the notes from last week's meeting, summarizing actions.

0. Welcomes
- -----------

Welcome new participants Vangelis Vassalos and Vasileios Pothos from
Irida labs.

1. Actions from previous meetings
- ---------------------------------

ACTION: Paolo Savini to distribute his slide.
- - complete

ACTION: Paolo Savini to explore suitable BEEBS floating point benchmarks.
- - see agenda item 6
- - complete

ACTION: Jeremy Bennett to commit the pull request 63
- - complete

ACTION: Jeremy to add tutorial example to reproduce baseline results
in documentation.
- - captured as GitHub issue 68
- - in progress

ACTION: Roger Shepherd to provide Embench results for ST dev boards,
Apple Macs and Raspberry Pi.
- - see agenda item 4
- - Roger to report

ACTION: Jeremy Bennett/Paolo Savini to attempt to get results for
Atmel AVR using Arduino.
- - see agenda item 4
- - Paolo to report

ACTION: Jeremy Bennett to add link to results page to Wikipedia.
- - added text and link, but link removed - help from Wikipedia expert
welcome!
- - see agenda item 5
- - complete

ACTION: Jeremy Bennett to explore alternatives to Google Docs - possibly
restructured text or hieroglyph, which could be held in a repository.
- - see agenda item 5
- - in progress

ACTION: Nikos Fragoulis to propose benchmark and suitable data from
Irida Labs
- - see agenda item 6
- - Nikos to report

ACTION: Ray Simar to set class project to come up with new benchmarks
based on the TinyML book.
- - see agenda item 6
- - Ray to report

ACTION: David Patterson to convene a meeting with Ray Simar and Jeff
Bier to discuss use of BDTI benchmarks, including how they would be
licensed (anything GPL compatible is fine).
- - see agenda item 6
- - David to report

2. GitHub pull requests
- -----------------------

https://github.com/embench/embench-iot/pull/74
- - fixes issue 69
- - thanks to Brian Grayson of SiFive for the fix

https://github.com/embench/embench-iot/pull/75
- - fixes issue 70
- - thanks to Jeremy Bennett for the fix

3. New GitHub issues
- --------------------

- From embench-iot:

https://github.com/embench/embench-iot/issues/68
- - captures action to document creation of baseline results
- - assigned to Jeremy Bennett

https://github.com/embench/embench-iot/issues/69
- - repeated builds fail noisily
- - bug in build_all.py fixed by pull-request 74
- - thanks to Brian Grayson of SiFive for the issue report and fix

https://github.com/embench/embench-iot/issues/70
- - Update doc to note pyelftools is a prerequisite
- - fixed by pull-request 75
- - thanks to Ed Jones of Embecosm for the issue report

https://github.com/embench/embench-iot/issues/71
- - results directory is not created by run_all.py
- - bug assigned to Jeremy Bennett
- - thanks to Ed Jones of Embecosm for the issue report

https://github.com/embench/embench-iot/issues/72
- - wikisort has a GCC extension in the code
- - assigned to Paolo Savini to fix and check all benchmarks
- - we need to agree what C standard we require (see agenda item 4)
- - thanks to Kjetil Østerås of Silicon Labs for the issue report

https://github.com/embench/embench-iot/issues/73
- - other benchmarks use GCC extensions
- - generalization of issue 72, also assigned to Paolo Savini
- - thanks to Kjetil Østerås of Silicon Labs for the issue report

- From embench-iot-results:

https://github.com/embench/embench-iot-results/issues/4
- - what is the performance of baseline board at full-speed
- - flat out would be affected by Flash memory access times.
- - assigned to Paolo Savini to investigate.
- - Thanks to Charles Papon (Dolu1990) for raising this.

Thanks to all those who raised issues and provided responses.

4. Update on benchmarks
- -----------------------

Issues 72 and 73 highlighted the lack of an agreed standard for C and
C++ langauges to be used. The meeting agreed to use C11 for C programs
and C++11 for C++ programs.

Roger Shepherd is working on new data for ST platforms. Running into
some tool problems

ACTION (ongoing): Roger Shepherd to provide Embench results for ST dev
boards, Apple Macs and Raspberry Pi.

Paolo Savini is still working on Atmel AVR

ACTION (ongoing): Jeremy Bennett/Paolo Savini to attempt to get
results for Atmel AVR using Arduino.

5. Publicity
- ------------

No success on standby at HotChips

The Wikipedia page: https://en.wikipedia.org/wiki/Embench, now talks
about the results repository, but the link was removed. Suggest
putting it in as a footnote.

Jeremy Bennett had an action to explore alternatives to Google for the
standard presentation. There is work in progress on a version in
ReStructuredText using Sphinx/Hieroglyph version here:

https://github.com/jeremybennett/embench-iot/tree/presentations

Some of the tables are stretching the limits of ReStructuredText. Any
suggestions welcome.

An alternative would be Beamer (LaTeX for slides).

ACTION (ongoing): Jeremy Bennett to explore alternatives to Google
Docs - possibly restructured text or hieroglyph, which could be held
in a repository.

David Patterson has submitted a talk to Arm Techcon

Jeremy Bennett will approach ORConf about presentation

David Patterson is blogging. Possibly could write about Embench 0.6.

6. Embench 0.6
- --------------

There is an ongoing shared document to add potential programs for a
new floating point benchmark suite:

https://docs.google.com/document/d/1tC0LXZRYeyoSRTM9aXtapzx3sA00vfFcL9QjOxU3vzQ

Ray Simar reported on BDTI benchmarks and conversation with Jeff Bier.
Definitely interested, perhaps pick just two benchmarks at first.
Looking at two Ray's graduate students helping with this in the Fall.

Jeff Bier is also looking at embedded inference engines, and
interested in helping with this. Ray planning TinyML project in his
class this Fall.

David Patterson suggested the possibility of floating point and
quantized versions of AI engines as described in the TinyML book.

ACTION (ongoing): Ray Simar to set class project to come up with new
benchmarks based on the TinyML book.

ACTION (ongoing): Ray Simar to continue work on BDTI benchmarks,
including how they would be licensed (anything GPL compatible is fine).

Paolo Savini has considered candidate floating point benchmarks from
BEEBS, with matmult-float, ludcmp, qurt, sqrt, fasta, stb_perlin and
frac as being plausible candidates. Suggest matmult-float, ludcmp, and
possibly qurt as the priorities.

Nikos Fragoulis talked about their deep learning technologies. Looking
at the smallest LeNet-5 network, 60kB (integer) or 240kB (floating
point) of data.

Do we need bigger networks in the future? Nikos presented models of
various sizes.

ACTION (ongoing): Nikos Fragoulis to propose benchmark and suitable
data from Irida Labs

Ray Simar to leave FP benchmark initiative.

ACTION: Jeremy to set up repository, with Ray as co-owner. Paolo and
Nikos can contribute their code into that repository.

Timescale: Ray's course starts in late August, and wrap up by
Thanksgiving.

SiFive had some interesting vector benchmarks. Could we do a vector
benchmark suite in the future? Subject for a future meeting.

7. Future meetings
- ------------------

Future meetings will be 1 hour long.

We currently meet regularly on the third Monday of each month 8-10am
California time. Our future meetings will be. Note that 2 dates have
been changed to accommodate upcoming conferences/holidays.

- - Mon 10 Aug 2020 (changed)
- - Mon 21 Sep 2020
- - Mon 19 Oct 2020
- - Mon 16 Nov 2020
- - Mon 14 Dec 2020 (changed)

I've created a calendar entry for our regular attendees, to provide
automated reminders. Please ask if you wish to be added to this.

ACTION: Jeremy to reissue 10 August invitation.

8. AOB
- ------

Best wishes,


Jeremy
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQRASGDWqmhRZUfAaPW+9YFy+0dU4QUCXxW+dQAKCRC+9YFy+0dU
4dPNAJ0cYbYwOn2FnYu0fpSDjbhYHf4RFACeO3ORrOxE2rMoF4gwCVjs4AkIQjo=
=4i9S
-----END PGP SIGNATURE-----



More information about the Embench mailing list