Planet Clang

November 14, 2018

LLVM Blog

30% faster Windows builds with clang-cl and the new /Zc:dllexportInlines- flag

Background

In the course of adding Microsoft Visual C++ (MSVC) compatible Windows support to Clang, we worked hard to make sure the dllexport and dllimport declspecs are handled the same way by Clang as by MSVC.

dllexport and dllimport are used to specify what functions and variables should be externally accessible ("exported") from the currently compiled Dynamic-Link Library (DLL), or should be accessed ("imported") from another DLL. In the class declaration below, S::foo() will be exported when building a DLL:


struct __declspec(dllexport) S {
void foo() {}
};

and code using that DLL would typically see a declaration like this:


struct __declspec(dllimport) S {
void foo() {}
};

to indicate that the function is defined in and should be accessed from another DLL.

Often the same declaration is used along with a preprocessor macro to flip between dllexport and dllimport, depending on whether a DLL is being built or consumed.

The basic idea of dllexport and dllimport is simple, but the semantics get more complicated as they interact with more facets of the C++ language: templates, inheritance, different kinds of instantiation, redeclarations with different declspecs, and so on. Sometimes the semantics are surprising, but by now we think clang-cl gets most of them right. And as the old maxim goes, once you know the rules well, you can start tactfully breaking them.

One issue with dllexport is that for inline functions such as S::foo() above, the compiler must emit the definition even if it's not used in the translation unit. That's because the DLL must export it, and the compiler cannot know if any other translation unit will provide a definition.

This is very inefficient. A dllexported class with inline members in a header file will cause definitions of those members to be emitted in every translation unit that includes the header, directly or indirectly. And as we know, C++ source files often end up including a lot of headers. This behaviour is also different from non-Windows systems, where inline function definitions are not emitted unless they're used, even in shared objects and dynamic libraries.

/Zc:dllexportInlines-

To address this problem, clang-cl recently gained a new command-line flag, /Zc:dllexportInlines- (MSVC uses the /Zc: prefix for language conformance options). The basic idea is simple: since the definition of an inline function is available along with its declaration, it's not necessary to import or export it from a DLL — the inline definition can be used directly. The effect of the flag is to not apply class-level dllexport/dllimport declspecs to inline member functions. In the two examples above, it means S::foo() would not be dllexported or dllimported, even though the S class is declared as such.

This is very similar to the -fvisibility-inlines-hidden Clang and GCC flag used on non-Windows. For C++ projects with many inline functions, it can significantly reduce the set of exported functions, and thereby the symbol table and file size of the shared object or dynamic library, as well as program load time.

On Windows however, the main benefit is not having to emit the unused inline function definitions. This means the compiler has to do much less work, and reduces object file size which in turn reduces the work for the linker. For Chrome, we saw 30 % faster full builds, 30 % shorter link times for blink_core.dll, and 40 % smaller total .obj file size.

The reduction in .obj file size, combined with the enormous reduction in .lib files allowed by previously switching linkers to lld-link which uses thin archives, means that a typical Chrome build directory is now 60 % smaller than it would have been just a year ago.

(Some of the same benefit can be had without this flag if the dllexport inline function comes from a pre-compiled header (PCH) file. In that case, the definition will be emitted in the object file when building the PCH, and so is not emitted elsewhere unless it's used.)

Compatibility

Using /Zc:dllexportInlines- is "half ABI incompatible". If it's used to build a DLL, inline members will no longer be exported, so any code using the DLL must use the same flag to not dllimport those members. However, the reverse scenario generally works: a DLL compiled without the flag (such as a system DLL built with MSVC) can be referenced from code that uses the flag, meaning that the referencing code will use the inline definitions instead of importing them from the DLL.

Like -fvisibility-inlines-hidden, /Zc:dllexportInlines- breaks the C++ language guarantee that (even an inline) function has a unique address within the program. When using these flags, an inline function will have a different address when used inside the library and outside.

Also, these flags can lead to link errors when inline functions, which would normally be dllimported, refer to internal symbols of a DLL:


void internal();

struct __declspec(dllimport) S {
void foo() { internal(); }
}

Normally, references to S::foo() would use the definition in the DLL, which also contains the definition of internal(), but when using /Zc:dllexportInlines-, the inline definition of S::foo() is used directly, resulting in a link error since no definition of internal() can be found.

Even worse, if there is an inline definition of internal() containing a static local variable, the program will now refer to a different instance of that variable than in the DLL:


inline int internal() { static int x; return x++; }

struct __declspec(dllimport) S {
int foo() { return internal(); }
}

This could lead to very subtle bugs. However, since Chrome already uses -fvisibility-inlines-hidden, which has the same potential problem, we believe this is not a common issue.

Summary

/Zc:dllexportInlines- is like -fvisibility-inlines-hidden for DLLs and significantly reduces build times. We're excited that using Clang on Windows allows us to benefit from new features like this.

More information

For more information, see the User's Manual for /Zc:dllexportInlines-.

The flag was added in Clang r346069, which will be part of the Clang 8 release expected in March 2019. It's also available in the Windows Snapshot Build.

Acknowledgements

/Zc:dllexportInlines- was implemented by Takuto Ikuta based on a prototype by Nico Weber.

by Hans Wennborg (noreply@blogger.com) at November 14, 2018 12:49 PM

September 26, 2018

LLVM Blog

Announcing the program for the 2018 LLVM Developers' Meeting Bay Area

The LLVM Foundation is excited to announce the program for the 2018 LLVM Developers' Meeting in San Jose, CA on October 17 & 18.
As a reminder, ticket prices for the event will increase on September 17th. Purchase your tickets today!
Technical Talks
Tutorials
Birds of a Feather
Lightning Talks
Posters

by Tanya Lattner (noreply@blogger.com) at September 26, 2018 02:50 AM

September 25, 2018

LLVM Blog

Integration of libc++ and OpenMP packages into llvm-toolchain

A bit more than a year ago, we gave an update about recent changes in apt.llvm.org. Since then, we noticed an important increase of the usage of the service. Just last month, we saw more than 16.5TB of data being transferred from our CDN.
Thanks to the Google Summer of Code 2018, and after number of requests, we decided to focus our energy to bring new great projects from the LLVM ecosystems into apt.llvm.org.

Starting from version 7, libc++, libc++abi and OpenMP packages are available into the llvm-toolchain packages. This means that, just like clang, lldb or lldb, libc++, libc++abi and OpenMP packages are also built, tested and shipped on https://apt.llvm.org/.

The integration focuses to preserve the current usage of these libraries. The newly merged packages have adopted the llvm-toolchain versioning:

libc++ packages
  • libc++1-7
  • libc++-7-dev
libc++abi packages
  • libc++abi1-7
  • libc++abi-7-dev
OpenMP packages
  • libomp5-7
  • libomp-7-dev
  • libomp-7-doc
This packages are built twice a day for trunk. For version 7, only when new changes happen in the SVN branches.
    Integration of libc++* packages

    Both libc++ and libc++abi packages are built at same time using the clang built during the process. The existing libc++ and libc++abi packages present in Debian and Ubuntu repositories will not be affected (they will be removed at some point). Newly integrated libcxx* packages are not co-installable with them.

    Symlinks have been provided from the original locations to keep the library usage same.

    Example:  /usr/lib/x86_64-linux-gnu/libc++.so.1.0 -> /usr/lib/llvm-7/lib/libc++.so.1.0

    The usage of the libc++ remains super easy:
    Usage:
    $ clang++-7 -std=c++11 -stdlib=libc++ foo.cpp
    $ ldd ./a.out|grep libc++
      libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f62a1a90000)
      libc++abi.so.1 => /usr/lib/x86_64-linux-gnu/libc++abi.so.1 (0x00007f62a1a59000)

    In order to test new developments in libc++, we are also building the experimental features.
    For example, the following command will work out of the box:

    $ clang++-7 -std=c++17 -stdlib=libc++ foo.cpp -lc++experimental -lc++fs

    Integration of OpenMP packages

    While OpenMP packages have been present in the Debian and Ubuntu archives for a while, only a single version of the package was available.

    For now, the newly integrated packages creates a symlink from /usr/lib/libomp.so.5 to /usr/lib/llvm-7/lib/libomp.so.5 keeping the current usage same and making them non co-installable.

    It can be used with clang through -fopenmp flag:
    $ clang -fopenmp foo.c

    The dependency packages providing the default libc++* and OpenMP package are also integrated in llvm-defaults. This means that the following command will install all these new packages at the current version:
    $ apt-get install libc++-dev libc++abi-dev libomp-dev

    LLVM 7 => 8 transition

    In parallel of the libc++ and OpenMP work, https://apt.llvm.org/ has been updated to reflect the branching of 7 from the trunk branches.
    Therefore, we have currently on the platform:

    Stable
    6.0
    Qualification
    7
    Development
    8


    Please note that, from version 7, the packages and libraries are called 7 (and not 7.0).
    For the rational and implementation, see https://reviews.llvm.org/D41869 & https://reviews.llvm.org/D41808.

    Stable packages of LLVM toolchain are already officially available in Debian Buster and in Ubuntu Cosmic.

    Cosmic support

    In order to make sure that the LLVM toolchain does not have too many regressions with this new version, we also support the next Ubuntu version, 18.10, aka Cosmic.

    A Note on coinstallability

    We tried to make them coinstallable, in the resulting packages we had no control over the libraries used during the runtime. This could lead to many unforeseen issues. Keeping these in mind we settled to keep them conflicting with other versions.

    Future work
    • Code coverage build fails for newly integrated packages
    • Move to a 2 phases build to generate clang binary using clang

    Sources of the project are available on the gitlab instance of Debian: https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/tree/7


    Reshabh Sharma & Sylvestre Ledru




    by Reshabh Sharma (noreply@blogger.com) at September 25, 2018 03:29 PM

    September 18, 2018

    LLVM Blog

    Announcing the new LLVM Foundation Board of Directors

    The LLVM Foundation is pleased to announce its new Board of Directors:


    Chandler Carruth
    Mike Edwards (Treasurer)
    Hal Finkel
    Arnaud de Grandmaison
    Anton Korobeynikov
    Tanya Lattner (President)
    Chris Lattner
    John Regehr (Secretary)
    Tom Stellard

    Two new members and seven continuing members were elected to the nine person board.

    We want to thank David Kipping for his 2 terms on the board. David has been actively involved with the LLVM Developer Meetings and was the treasurer for the past 4 years. The treasurer is a time demanding position in that he supports the day to day operation of the foundation, balancing the books, and generates monthly treasurer reports.

    We also want to thank all the applicants to the board. When voting on new board members, we took into consideration all contributions (past and present) and current involvement in the LLVM community. We also tried to create a balanced board of individuals from a wide range of backgrounds and locations to provide a voice to as many groups within the LLVM community. Given this criteria and strong applicants, we increased the board from 8 members to 9.

    About the board of directors (listed alphabetically by last name):


    Chandler Carruth:

    Chandler Carruth has been an active contributor to LLVM since 2007. Over the years, he has has worked on LLVM’s memory model and atomics, Clang’s C++ support, GCC-compatible driver, initial profile-aware code layout optimization pass, pass manager, IPO infrastructure, and much more. He is the current code owner of inlining and SSA formation.

    In addition to his numerous technical contributions, Chandler has led Google’s LLVM efforts since 2010 and shepherded a number of new efforts that have positively and significantly impacted the LLVM project. These new efforts include things such as adding C++ modules to Clang, adding address and other sanitizers to Clang/LLVM, making Clang compatible with MSVC and available to the Windows C++ developer community, and much more.

    Chandler works at Google Inc. as a technical lead for their C++ developer platform and has served on the LLVM Foundation board of directors for the last 4 years.

    Mike Edwards:

    Mike Edwards is a relative newcomer to the LLVM community, beginning his involvement just a few years ago while working for Sony Playstation. Finding the LLVM community to be an incredibly amazing and welcoming group of people, Mike knew he had to find a way to contribute. Mike’s previous work in DevOps led him to get involved in helping to work on the llvm.org infrastructure. Last year, with the help of the Board and several community members, Mike was able to get the llvm.org infrastructure moved onto a modern compute platform at Amazon Web Services. Mike is one of the maintainers of our llvm.org infrastructure.

    Mike is currently working as a Software Engineer at Apple, Inc. working on the Continuous Integration and Quality Engineering efforts for LLVM and Clang development.

    Hal Finkel:

    Hal Finkel has been an active contributor to the LLVM project since 2011. He is the code owner for the PowerPC target, the alias-analysis infrastructure, and other components.

    In addition to his numerous technical contributions, Hal has chaired the LLVM in HPC workshop, which is held in conjunction with Super Computing (SC), for the last five years. This workshop provides a venue for the presentation of peer-reviewed HPC-related researching LLVM from both industry and academia. He has also been involved in organizing an LLVM-themed BoF session at SC and LLVM socials in Austin.

    Hal is Lead for Compiler Technology and Programming Languages at Argonne National Laboratory’s Leadership Computing Facility.

    Arnaud de Grandmaison:

    Arnaud de Grandmaison has been hacking on LLVM projects since 2008. In addition to his open source contributions, he has worked for many years on private out-of-tree LLVM-based projects at Parrot, DiBcom, or Arm. He has also been a leader in the European LLVM community by organizing the EuroLLVM Developers’ meeting, Paris socials, and chaired or participated in numerous program committees for the LLVM Developers’ Meetings and other LLVM related conferences.

    Arnaud has attended numerous LLVM Developers’ meetings and volunteered as moderator or presented as well. He also moderates several LLVM mailing lists. Arnaud is also very involved in community wide discussions and decisions such as re-licensing and code of conduct.

    Arnaud is a Senior Principal Engineer at Arm.

    Anton Korobeynikov:

    Anton Korobeynikov has been an active contributor to the LLVM project since 2006. Over the years, he has numerous technical contributions to areas including Windows support, ELF features, debug info, exception handling, and backends such as ARM and x86. He was the original author of the MSP430 and original System Z backend.

    In addition to his technical contributions, Anton has maintained LLVM’s participation in Google Summer of Code by managing applications, deadlines, and overall organization. He also supports the LLVM infrastructure and has been on numerous program committees for the LLVM Developers’ Meetings (both US and EuroLLVM).

    Anton is currently an associate professor at the Saint Petersburg State University and has served on the LLVM Foundation board of directors for the last 4 years.

    Tanya Lattner:

    Tanya Lattner has been involved in the LLVM project for over 14 years. She began as a graduate student who wrote her master's thesis using LLVM, and continued on using and extending LLVM technologies at various jobs during her career as a compiler engineer.

    Tanya has been organizing the US LLVM Developers’ meeting since 2008 and attended every developer meeting. She was the LLVM release manager for 3 years, moderates the LLVM mailing lists, and helps administer the LLVM infrastructure servers, mailing lists, bugzilla, etc. Tanya has also been on the program committee for the US LLVM Developers’ meeting (4+ years) and the EuroLLVM Developers’ Meeting.

    With the support of the initial board of directors, Tanya created the LLVM Foundation, defined its charitable and education mission, and worked to get 501(c)(3) status.

    Tanya is the Chief Operating Officer and has served as the President of the LLVM Foundation board for the last 4 years.

    Chris Lattner:

    Chris Lattner is well known as the founder for the LLVM project and has a lengthy history of technical contributions to the project over the years. He drove much of the early implementation, architecture, and design of LLVM and Clang.

    Chris has attended every LLVM Developers’ meeting, and presented at many of them. He helped drive the conception and incorporation of the LLVM Foundation, and has served as its secretary. Chris also grants commit access to the LLVM Project, moderates mailing lists, moderates and edits the LLVM blog, and drives important non-technical discussions and policy decisions related to the LLVM project.

    Chris manages a team building machine learning infrastructure at Google and has served on the LLVM Foundation board of directors for the last 4 years.

    John Regehr:

    John Regehr has been involved in LLVM for a number of years. As a professor of computer science at the University of Utah, his research specializes in compiler correctness and undefined behavior. He is well known within the LLVM community for the hundreds of bug reports his group has reported to LLVM/Clang.

    John was a project lead for IOC, a Clang based integer overflow checker that eventually became the basis for the integer parts of UBSan. He was also the primary developer of C-Reduce which utilizes Clang as a library and is often used as a test case reducer for compiler issues.

    In addition to his technical contributions, John has served on several LLVM-related program committees. He also has a widely read blog about LLVM and other compiler-related issues (Embedded in Academia).

    Tom Stellard:

    Tom Stellard has been contributing to the LLVM project since 2012. He was the original author of the AMDGPU backend and was also an active contributor to libclc. He has been the LLVM project’s stable release manager since 2014.

    Tom is currently a Software Engineer at Red Hat and is the technical lead for emerging toolchains including Clang/LLvm. He also maintains the LLVM packages for the Fedora project.

    by Tanya Lattner (noreply@blogger.com) at September 18, 2018 04:00 PM

    August 24, 2018

    LLVM Blog

    2018 LLVM Foundation's Women in Compilers and Tools Workshop


    The LLVM Foundation is excited to announce our first half day Women in Compilers and Tools Workshop held the day before the 2018 LLVM Developers’ Meeting - Bay Area. The workshop will be held at the Fairmont Hotel on October 16th from 1:00-6:30PM and includes a cocktail reception.

    This event aims to connect women in the field of compilers and tools and provide them with ideas and techniques to overcome barriers or enhance their careers. It also is open to anyone (not just women) who are interested in increasing diversity within the LLVM community, their workplace or university.

    Registration for the event will open on Monday, August 27th at 9:00AM PDT. Attendance is limited to 100 attendees and tickets will be priced at $50 (students $25). Please see the EventBrite registration page for details.

    The workshop will consist of 3 topics described below:

    1. Inner Critic: How to Deal with Your Imposter Syndrome
    Presented by Women Catalysts

    You're smart. People really like you. And yet, you can't shake the feeling that maybe you don't really deserve your success. Or that someone else can do what you do better...and what if your boss can see it too? You are not alone: it's called the Imposter Syndrome. Believe it or not, the most confident and successful people often fear that
    they are actually inadequate. The great Maya Angelou once said, "I have written 11 books, but each time I think, 'Uh-oh, they're going to find out now. I've run a game on everybody, and they're going to find me out.’" But it doesn't have to be that way. In this workshop, you'll learn to identify the voice of your Imposter Syndrome and develop with strategies for dealing with your inner critics.

    1. Present! A Techie's Guide to Public Speaking
    Presented by Karen Catlin

    To grow your career, you know what you need to do: improve your public speaking skills.

    Public speaking provides the visibility and professional credibility that helps you score the next big opportunity. But even more important is the fact that it transforms the way you communicate. Improved confidence and the ability to convey messages clearly will impact your relationships with your managers, coworkers, customers, industry peers, and even potential new hires.

    In this presentation, Karen Catlin will cover the importance of speaking at conferences and events, along with strategies to get started. She'll share some favorite tips from the book she co-authored with Poornima Vijayashanker, "Present! A Techie's Guide to Public Speaking." And she'll tell some embarrassing stories that are just too good to keep to herself.

    About Karen: After spending 25 years building software products, Karen Catlin is now an advocate for women in the tech industry. She’s a leadership coach, a keynote and TEDx speaker, and co-author of "Present! A Techie’s Guide to Public Speaking.”

    Formerly, Karen was a vice president of engineering at Macromedia and Adobe.

    Karen holds a computer science degree from Brown University and serves as an advisor to Brown's Computer Science Diversity Initiative. She’s also on the Advisory Boards for The Women’s CLUB of Silicon Valley and WEST (Women Entering & Staying in Technology).

    1. Update on Women in Compilers & Tools Program
    Presented by Tanya Lattner
    Over the past year we have hosted panels and BoFs on women in compilers and tools. We now need to take many of the items discussed during the events and put them into action. We will discuss some key areas and potentially break into smaller groups to determine action plans and steps to move forward.

    FAQ:

    Do I need to attend the LLVM Developers’ Meeting to attend this event?
    This is an independent event which is open to anyone.  

    Is this a women only event?
    Anyone is welcome to attend that values diversity within the field of compiler and tools.  These topics can relate to anyone, not just women, and our mission is to improve inclusion and diversity in general.

    Is there a financial hardship discount?

    We have discounted the tickets for all attendees but please reach out to the organizer and we will decide on a case by case basis.

    by Tanya Lattner (noreply@blogger.com) at August 24, 2018 05:41 AM

    April 10, 2018

    LLVM Blog

    EuroLLVM'18 developers' meeting program

    The LLVM Foundation is excited to announce the program for the EuroLLVM'18 developers' meeting (April 16 - 17 in Bristol/UK) !

    Keynotes

    Tutorials

    Talks

    BoFs

    Student Research Competition

    Lightning Talks

    Posters

    If you are interested in any of this talks, you should register to attend the EuroLLVM'18. Tickets are limited !

    More information about the EuroLLVM'18 is available here

    by Arnaud Allard de Grandmaison (noreply@blogger.com) at April 10, 2018 05:03 PM

    March 13, 2018

    LLVM Blog

    DragonFFI: FFI/JIT for the C language using Clang/LLVM


    Introduction

    A foreign function interface is "a mechanism by which a program written in one programming language can call routines or make use of services written in another".
    In the case of DragonFFI, we expose a library that allows calling C functions and using C structures from any languages. Basically, we want to be able to do this, let's say in Python:
    import pydffi
    CU = pydffi.FFI().cdef("int puts(const char* s);");
    CU.funcs.puts("hello world!")
    or, in a more advanced way, for instance to use libarchive directly from Python:
    import pydffi
    pydffi.dlopen("/path/to/libarchive.so")
    CU = pydffi.FFI().cdef("#include <archive.h>")
    a = funcs.archive_read_new()
    assert a
    ...
    This blog post presents related works, their drawbacks, then how Clang/LLVM is used to circumvent these drawbacks, the inner working of DragonFFI and further ideas.
    The code of the project is available on GitHub: https://github.com/aguinet/dragonffi. Python 2/3 wheels are available for Linux/OSX x86/x64. Python 3.6 wheels are available for Windows x64. On all these architectures, just use:
    $ pip install pydffi
    and play with it :)

    See below for more information.

    Related work

    libffi is the reference library that provides a FFI for the C language. cffi is a Python binding around this library that also uses PyCParserto be able to easily declare interfaces and types. Both these libraries have limitations, among them:
    • libffi does not support the Microsoft x64 ABI under Linux x64. It isn't that trivial to add a new ABI (hand-written ABI, get the ABI right, ...), while a lot of effort have already been put into compilers to get these ABIs right.
    • PyCParser only supports a very limited subset of C (no includes, function attributes, ...).
    Moreover, in 2014, Jordan Rose and John McCall from Apple made a talk at the LLVM developer meeting of San José about how Clang can be used for C interoperability. This talk also shows various ABI issues, and has been a source of inspiration for DragonFFI at the beginning.

    Somehow related, Sean Callanan, who worked on lldb, gave a talk in 2017 at the LLVM developer meeting of San José on how we could use parts of Clang/LLVM to implement some kind of eval() for C++. What can be learned from this talk is that debuggers like lldb must also be able to call an arbitrary C function, and uses debug information among other things to solve it (what we also do, see below :)).

    DragonFFI is based on Clang/LLVM, and thanks to that it is able to get around these issues:
    • it uses Clang to parse header files, allowing direct usage of a C library headers without adaptation;
    • it support as many calling conventions and function attributes as Clang/LLVM do;
    • as a bonus, Clang and LLVM allows on-the-fly compilation of C functions, without relying on the presence of a compiler on the system (you still need the headers of the system's libc thought, or MSVCRT headers under Windows);
    • and this is a good way to have fun with Clang and LLVM! :)
    Let's dive in!

    Creating an FFI library for C

    Supporting C ABIs

    A C function is always compiled for a given C ABI. The C ABI isn't defined per the official C standards, and is system/architecture-dependent. Lots of things are defined by these ABIs, and it can be quite error prone to implement.

    To see how ABIs can become complex, let's compile this C code:

    typedef struct {
    short a;
    int b;
    } A;

    void print_A(A s) {
    printf("%d %d\n", s.a, s.b);
    }

    Compiled for Linux x64, it gives this LLVM IR:

    target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
    target triple = "x86_64-pc-linux-gnu"

    @.str = private unnamed_addr constant [7 x i8] c"%d %d\0A\00", align 1

    define void @print_A(i64) local_unnamed_addr {
    %2 = trunc i64 %0 to i32
    %3 = lshr i64 %0, 32
    %4 = trunc i64 %3 to i32
    %5 = shl i32 %2, 16
    %6 = ashr exact i32 %5, 16
    %7 = tail call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([7 x i8], [7 x i8]* @.str, i64 0, i64 0), i32 %6, i32 %4)
    ret void
    }

    What happens here is what is called structure coercion. To optimize some function calls, some ABIs pass structure values through registers. For instance, an llvm::ArrayRef object, which is basically a structure with a pointer and a size (see https://github.com/llvm-mirror/llvm/blob/release_60/include/llvm/ADT/ArrayRef.h#L51), is passed through registers (though this optimization isn't guaranteed by any standard).

    It is important to understand that ABIs are complex things to implement and we don't want to redo this whole work by ourselves, particularly when LLVM/Clang already know how.

    Finding the right type abstraction

    We want to list every types that is used in a parsed C file. To achieve that goal, various information are needed, among which:
    • the function types, and their calling convention
    • for structures: field offsets and names
    • for union/enums: field names (and values)
    On one hand, we have seen in the previous section that the LLVM IR is too Low Level (as in Low Level Virtual Machine) for this. On the other hand, Clang's AST is too high level. Indeed, let's print the Clang AST of the code above:
    [...]
    |-RecordDecl 0x5561d7f9fc20 <a.c:1:9, line:4:1> line:1:9 struct definition
    | |-FieldDecl 0x5561d7ff4750 <line:2:3, col:9> col:9 referenced a 'short'
    | `-FieldDecl 0x5561d7ff47b0 <line:3:3, col:7> col:7 referenced b 'int'
    We can see that there is no information about the structure layout (padding, ...). There's also no information about the size of standard C types. As all of this depends on the backend used, it is not surprising that these informations are not present in the AST.

    The right abstraction appears to be the LLVM metadata produced by Clang to emit DWARF or PDB structures. They provide structure fields offset/name, various basic type descriptions, and function calling conventions. Exactly what we need! For the example above, this gives (at the LLVM IR level, with some inline comments):

    target triple = "x86_64-pc-linux-gnu"
    %struct.A = type { i16, i32 }
    @.str = private unnamed_addr constant [7 x i8] c"%d %d\0A\00", align 1

    define void @print_A(i64) local_unnamed_addr !dbg !7 {
    %2 = trunc i64 %0 to i32
    %3 = lshr i64 %0, 32
    %4 = trunc i64 %3 to i32
    tail call void @llvm.dbg.value(metadata i32 %4, i64 0, metadata !18, metadata !19), !dbg !20
    tail call void @llvm.dbg.declare(metadata %struct.A* undef, metadata !18, metadata !21), !dbg !20
    %5 = shl i32 %2, 16, !dbg !22
    %6 = ashr exact i32 %5, 16, !dbg !22
    %7 = tail call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([...] @.str, i64 0, i64 0), i32 %6, i32 %4), !dbg !23
    ret void, !dbg !24
    }

    [...]
    ; DISubprogram defines (in our case) a C function, with its full type
    !7 = distinct !DISubprogram(name: "print_A", scope: !1, file: !1, line: 6, type: !8, [...], variables: !17)
    ; This defines the type of our subprogram
    !8 = !DISubroutineType(types: !9)
    ; We have the "original" types used for print_A, with the first one being the
    ; return type (null => void), and the other ones the arguments (in !10)
    !9 = !{null, !10}
    !10 = !DIDerivedType(tag: DW_TAG_typedef, name: "A", file: !1, line: 4, baseType: !11)
    ; This defines our structure, with its various fields
    !11 = distinct !DICompositeType(tag: DW_TAG_structure_type, file: !1, line: 1, size: 64, elements: !12)
    !12 = !{!13, !15}
    ; We have here the size and name of the member "a". Offset is 0 (default value)
    !13 = !DIDerivedType(tag: DW_TAG_member, name: "a", scope: !11, file: !1, line: 2, baseType: !14, size: 16)
    !14 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
    ; We have here the size, offset and name of the member "b"
    !15 = !DIDerivedType(tag: DW_TAG_member, name: "b", scope: !11, file: !1, line: 3, baseType: !16, size: 32, offset: 32)
    !16 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
    [...]

    Internals

    DragonFFI first parses the debug information included by Clang in the LLVM IR it produces, and creates a custom type system to represent the various function types, structures, enumerations and typedefs of the parsed C file. This custom type system has been created for two reasons:
    • create a type system that gathers only the necessary informations from the metadata tree (we don't need the whole debug informations)
    • make the public headers of the DragonFFI library free from any LLVM headers (so that the whole LLVM headers aren't needed to use the library)
    Once we've got this type system, the DragonFFI API for calling C functions is this one:

    DFFI FFI([...]);
    // This will declare puts as a function that returns int and takes a const
    // char* as an argument. We could also create this function type by hand.
    CompilationUnit CU = FFI.cdef("int puts(const char* s);", [...]);
    NativeFunc F = CU.getFunction("puts");
    const char* s = "hello world!";
    void* Args[] = {&s};
    int Ret;
    F.call(&Ret, Args);

    So, basically, a pointer to the returned data and an array of void* is given to DragonFFI. Each void* value is a pointer to the data that must be passed to the underlying function. So the last missing piece of the puzzle is the code that takes this array of void* (and pointer to the returned data) and calls puts, so a function like this:

    void call_puts(void* Ret, void** Args) {
    *((int*)Ret) = puts((const char*) Args[0]);
    }

    We call these "function wrappers" (how original! :)). One advantage of this signature is that it is a generic signature, which can be used in the implementation of DragonFFI. Supposing we manage to compile at run-time this function, we can then call it trivially as in the following:

    typedef void(*puts_call_ty)(void*, void**);
    puts_call_ty Wrapper = /* pointer to the compiled wrapper function */;
    Wrapper(Ret, Args);

    Generating and compiling a function like this is something Clang/LLVM is able to do. For the record, this is also what libffi mainly does, by generating the necessary assembly by hand. We optimize the number of these wrappers in DragonFFI, by generating them for each different function type. So, the actual wrapper that would be generated for puts is actually this one:

    void __dffi_wrapper_0(int32_t( __attribute__((cdecl)) *__FPtr)(char *), int32_t *__Ret, void** __Args) {
    *__Ret = (__FPtr)(*((char **)__Args[0]));
    }

    For now, all the necessary wrappers are generated when the DFFI::cdef or DFFI::compile APIs are used. The only exception where they are generated on-the-fly (when calling CompilationUnit::getFunction) is for variadic arguments. One possible evolution is to let the user chooses whether he wants this to happen on-the-fly or not for every declared function.

    Issues with Clang

    There is one major issue with Clang that we need to hack around in order to have the DFFI::cdef functionality: unused declarations aren't emitted by Clang (even when using -g -femit-all-decls).

    Here is an example, produced from the following C code:

    typedef struct {
    short a;
    int b;
    } A;

    void print_A(A s);
    $ clang -S -emit-llvm -g -femit-all-decls -o - a.c |grep print_A |wc -l
    0

    The produced LLVM IR does not contain a function named print_A! The hack we temporarily use parses the clang AST and generates temporary functions that looks like this:

    void __dffi_force_decl_print_A(A s) { }

    This forces LLVM to generate an empty function named __dffi_force_decl_print_A with the good arguments (and associated debug informations).

    This is why DragonFFI proposes another API, DFFI::compile. This API does not force declared-only functions to be present in the LLVM IR, and will only expose functions that end up naturally in the LLVM IR after optimizations.

    If someone has a better idea to handle this, please let us know!

    Python bindings

    Python bindings were the first ones to have been written, simply because it's the "high level" language I know best.  Python provides its own set of challenges, but we will save that for another blog post.  These Python bindings are built using pybind11, and provides their own set of C types. Lots of example of what can be achieved can be found here and here.

    Project status

    DragonFFI currently supports Linux, OSX and Windows OSes, running on Intel 32 and 64-bits CPUs. Travis is used for continuous integration, and every changes is validated on all these platforms before being integrated.

    The project will go from alpha to beta quality when the 0.3 version will be out (which will bring Travis and Appveyor CI integration and support for variadic functions). The project will be considered stable once these things happen:
    • user and developer documentations exist!
    • another foreign language is supported (JS? Ruby?)
    • the DragonFFI main library API is considered stable
    • a non negligible list of tests have been added
    • all the things in the TODO file have been done :)

    Various ideas for the future

    Here are various interesting ideas we have for the future. We don't know yet when they will be implemented, but we think some of them could be quite nice to have.

    Parse embedded DWARF information

    As the entry point of DragonFFI are DWARF informations, we could imagine parsing these debug informations from shared libraries that embed them (or provide them in a separate file). The main advantage is that all the necessary information for doing the FFI right are in one file, the header files are no longer required. The main drawback is that debug informations tend to take a lot of space (for instance, DWARF informations take 1.8Mb for libarchive 3.32 compiled in release mode, for an original binary code size of 735Kb), and this brings us to the next idea.

    Lightweight debug info?

    The DWARF standard allows to define lots of information, and we don't need all of them in our case. We could imagine embedding only the necessary DWARF objects, that is just the necessary types to call the exported functions of a shared library. One experiment of this is available here: https://github.com/aguinet/llvm-lightdwarf. This is an LLVM optimisation pass that is inserted at the end of the optimisation pipeline, and parse metadata to only keep the relevant one for DragonFFI. More precisely, it only keeps the dwarf metadata related to exported and visible functions, with the associated types. It also keeps debug information of global variables, even thought these ones aren't supported yet in DragonFFI. It also does some unconventional things, like replacing every file and directory by "_", to save space. "Fun" fact, to do this, it borrows some code from the LLVM bitcode "obfuscator" included in recent Apple's clang version, that is used to anonymize some information from the LLVM bitcode that is sent with tvOS/iOS applications (see http://lists.llvm.org/pipermail/llvm-dev/2016-February/095588.html for more information).

    Enough talking, let's see some preliminary results (on Linux x64):
    • on libarchive 3.3.2, DWARF goes from 1.8Mb to 536Kb, for an original binary code size of 735Kb
    • on zlib 1.2.11, DWARF goes from 162Kb to 61Kb, for an original binary code size of 99Kb
    The instructions to reproduce this are available in the README of the LLVM pass repository.
    We can conclude that defining this "light" DWARF format could be a nice idea. One other thing that could be done is defining a new binary format, that would be thus more space-efficient, but there are drawbacks going this way:
    • debug informations are well supported on every platform nowadays: tools exist to parse them, embed/extract them from binary, and so on
    • we already got DWARD and PDB: https://xkcd.com/927/
    Nevertheless, it still could be a nice experiment to try and do this, figuring out the space won and see if this is worth it!

    As a final note, these two ideas would also benefit to libffi, as we could process these formats and create libffi types!

    JIT code from the final language (like Python) to native function code

    One advantage of embedding a full working C compiler is that we could JIT the code from the final language glue to the final C function call, and thus limit the performance impact of this glue code.
    Indeed, when a call is issued from Python, the following things happen:
    • arguments are converted from Python to C according to the function type
    • the function pointer and wrapper and gathered from DragonFFI
    • the final call is made
    All this process involves basically a loop on the types of the arguments of the called function, which contains a big switch case. This loop generates the array of void* values that represents the C arguments, which is then passed to the wrapper. We could JIT a specialised version of this loop for the function type, inline the already-compiled wrapper and apply classical optimisation on top of the resulting IR, and get a straightforward conversion code specialized for the given function type, directly from Python to C.

    One idea we are exploring is combining easy::jit (hello fellow Quarkslab teammates!) with LLPE to achieve this goal.

    Reducing DragonFFI library size

    The DragonFFI shared library embed statically compiled versions of LLVM and Clang. The size of the final shared library is about 55Mb (stripped, under Linux x64). This is really really huge, compared for instance to the 39Kb of libffi (also stripped, Linux x64)!

    Here are some idea to try and reduce this footprint:
    • compile DragonFFI, Clang and LLVM using (Thin) LTO, with visibility hidden for both Clang and LLVM. This could have the effect of removing code from Clang/LLVM that isn't used by DragonFFI.
    • make DragonFFI more modular: - one core module that only have the parts from CodeGen that deals with ABIs. If the types and function prototypes are defined "by hand" (without DFFI::cdef), that's more or less the only part that is needed (with LLVM obviously) - one optional module that includes the full clang compiler (to provide the DFFI::cdef and DFFI::compile APIs)
    Even with all of this, it seems to be really hard to match the 39Kb of libffi, even if we remove the cdef/compile API from DragonFFI. As always, pick the right tool for your needs :)

    Conclusion

    Writing the first working version of DragonFFI has been a fun experiment, that made me discover new parts of Clang/LLVM :) The current goal is to try and achieve a first stable version (see above), and experiment with the various cited ideas.

    It's a really long road, so feel free to come on #dragonffi on FreeNode for any questions/suggestions you might have, (inclusive) or if you want to contribute!

    Acknowledgments

    Thanks to Serge «sans paille» Guelton for the discussions around the Python bindings, and for helping me finding the name of the project :) (one of the most difficult task). Thanks also to him, Fernand Lone-Sang and Kévin Szkudlapski for their review of this blog post!

    by Adrien Guinet (noreply@blogger.com) at March 13, 2018 02:45 PM

    March 08, 2018

    LLVM Blog

    International Women's Day: Celebrating all the women in the LLVM Community!


    Today is International Women's Day! To all the women in the LLVM community, thank you for all your contributions!

    The LLVM Foundation values diversity within the LLVM community and the field of compilers and tools. Our Women in Compilers and Tools program began in 2015 with a birds of a feather discussion during the US LLVM Developers' Meeting and we have been expanding it over the years.

    In 2017, we were a sponsor of the Grace Hopper Conference. With the help of community members Anna Zaks and David Blaikie, the LLVM Foundation had a booth at the career fair to introduce women to LLVM and encourage them to become contributors. It was very exciting to learn that many women knew of LLVM, were using it in their classes or research, using it in their career, or were interested in learning more. We hopefully encouraged more women to get involved with LLVM, compilers, and open source.

    The LLVM Foundation was also a sponsor of the Programming Language Mentoring Workshop at SPLASH 2017. Our sponsorship went towards the travel costs for many women and other minorities to attend this workshop. The workshop focused on encouraging and preparing students to enter research careers in the field of programming languages, compilers, and related fields and to provide first hand perspectives on graduate school.

    We hosted our first Women in Compilers & Tools reception before the 2017 US LLVM Developers' Meeting. Anna Zaks and Alice Chan participated in a panel discussion about the challenges and experiences that they have encountered in their careers and within the open source community. The event was attended by 60 members of the LLVM community.

    In 2018, we look forward to another year of expanding our program. The LLVM Foundation will again sponsor the Grace Hopper Conference and we are looking for LLVM community members to help out at the career booth (more details to come). We will be having two Women in Compilers and Tools events. The first will have a reception and panel discussion before the 2018 EuroLLVM Developers' Meeting. Get your tickets here. The second will be before the 2018 US LLVM Developers' Meeting and details will be announced in the coming months.

    The LLVM Foundation thanks the LLVM community and its sponsors for supporting this work. If you want to participate in the discussion or receive notifications on events, please join the Women in Compilers and Tools mailing list.

    Question for the LLVM Foundation? Email us at llvm-foundation@lists.llvm.org.



    by Tanya Lattner (noreply@blogger.com) at March 08, 2018 07:15 PM

    March 06, 2018

    LLVM Blog

    Clang is now used to build Chrome for Windows

    As of Chrome 64, Chrome for Windows is compiled with Clang. We now use Clang to build Chrome for all platforms it runs on: macOS, iOS, Linux, Chrome OS, Android, and Windows. Windows is the platform with the second most Chrome users after Android according to statcounter, which made this switch particularly exciting.

    Clang is the first-ever open-source C++ compiler that’s ABI-compatible with Microsoft Visual C++ (MSVC) – meaning you can build some parts of your program (for example, system libraries) with the MSVC compiler (“cl.exe”), other parts with Clang, and when linked together (either by MSVC’s linker, “link.exe”, or LLD, the LLVM project’s linker – see below) the parts will form a working program.

    Note that Clang is not a replacement for Visual Studio, but an addition to it. We still use Microsoft’s headers and libraries to build Chrome, we still use some SDK binaries like midl.exe and mc.exe, and many Chrome/Win developers still use the Visual Studio IDE (for both development and for debugging).

    This post discusses numbers, motivation, benefits and drawbacks of using Clang instead of MSVC, how to try out Clang for Windows yourself, project history, and next steps. For more information on the technical side you can look at the slides of our 2015 LLVM conference talk, and the slides linked from there.

    Numbers

    This is what most people ask about first, so let’s talk about it first. We think the other sections are more interesting though.

    Build time

    Building Chrome locally with Clang is about 15% slower than with MSVC. (We’ve heard that Windows Defender can make Clang builds a lot slower on some machines, so if you’re seeing larger slowdowns, make sure to whitelist Clang in Windows Defender.) However, the way Clang emits debug info is more parallelizable and builds with a distributed build service (e.g. Goma) are hence faster.

    Binary size

    Chrome installer size gets smaller for 64-bit builds and slightly larger for 32-bit builds using Clang. The same difference shows in uncompressed code size for regular builds as well (see the tracking bug for Clang binary size for many numbers). However, compared to MSVC builds using link-time code generation (LTCG) and profile-guided optimization (PGO) Clang generates larger code in 64-bit for targets that use /O2 but smaller code for targets that use /Os. The installer size comparison suggests Clang's output compresses better.

    Some raw numbers for versions 64.0.3278.2 (MSVC PGO) and 64.0.3278.0 (Clang). mini_installer.exe is Chrome’s installer that users download, containing the LZMA-compressed code. chrome_child.dll is one of the two main dlls; it contains Blink and V8, and generally has many targets that are built with /O2. chrome.dll is the other main dll, containing the browser process code, mostly built with /Os.



    mini_installer.exe
    chrome.dll
    chrome_child.dll
    chrome.exe
    32-bit win-pgo
    45.46 MB
    36.47 MB
    53.76 MB
    1.38 MB
    32-bit win-clang
    45.65 MB
    (+0.04%)
    42.56 MB (+16.7%)
    62.38 MB
    (+16%)
    1.45 MB
    (+5.1%)
    64-bit win-pgo
    49.4 MB
    53.3 MB
    65.6 MB
    1.6 MB
    64-bit win-clang
    46.27 MB
    (-6.33%)
    50.6 MB
    (-5.1%)
    72.71 MB
    (+10.8%)
    1.57 MB
    (-1.2%)

    Performance

    We conducted extensive A/B testing of performance. Performance telemetry numbers are about the same for MSVC-built and clang-built Chrome – some metrics get better, some get worse, but all of them are within 5% of each other. The official MSVC builds used LTCG and PGO, while the Clang builds currently use neither of these. This is potential for improvement that we look forward to exploring. The PGO builds took a very long time to build due to the need for collecting profiles and then building again, and as a result, the configuration was not enabled on our performance-measurement buildbots. Now that we use Clang, the perf bots again track the configuration that we ship.

    Startup performance was worse in Clang-built Chrome until we started using a link-order file – a form of “PGO light” .

    Stability

    We A/B-tested stability as well and found no difference between the two build configurations.

    Motivation

    There were many motivating reasons for this project, the overarching theme being the benefits of using the same compiler across all of Chrome’s platforms, as well as the ability to change the compiler and deploy those changes to all our developers and buildbots quickly. Here’s a non-exhaustive list of examples.
    • Chrome is heavily using technology that’s based on compiler instrumentation (ASan, CFI, ClusterFuzz—uses ASan). Clang supports this instrumentation already, but we can’t add it to MSVC. We previously used after-the-fact binary instrumentation to mitigate this a bit, but having the toolchain write the right bits in the first place is cleaner and faster.
    • Clang enables us to write compiler plugins that add Chromium-specific warnings and to write tooling for large-scale refactoring. Chromium’s code search can now learn to index Windows code.
    • Chromium is open-source, so it’s nice if it’s built with an open-source toolchain.
    • Chrome runs on 6+ platforms, and most developers are only familiar with 1-3 platforms. If your patch doesn’t compile on a platform you’re unfamiliar with, due to a compiler error that you can’t locally reproduce on your local development machine, it’ll take you a while to fix. On the other hand, if all platforms use the same compiler, if it builds on your machine then it’s probably going to build on all platforms.
    • Using the same compiler also means that compiler-specific micro-optimizations will help on all platforms (assuming that the same -O flags are used on all platforms – not yet the case in Chrome, and only on the same ISAs – x86 vs ARM will stay different).
    • Using the same compiler enables cross-compiling – developers who feel most at home on a Linux box can now work on Windows-specific code, from their Linux box (without needing to run Wine).
    • We can continuously build Chrome trunk with Clang trunk to find compiler regressions quickly. This allows us to update Clang every week or two. Landing a major MSVC update in Chrome usually took a year or more, with several rounds of reporting internal compiler bugs and miscompiles. The issue here isn’t that MSVC is more buggy than Clang – it isn’t, all software is buggy – but that we can continuously improve Clang due to Clang being open-source.
    • C++ receives major new revisions every few years. When C++11 was released, we were still using six different compilers, and enabling C++11 was difficult. With fewer compilers, this gets much easier.
    • We can prioritize compiler features that are important to us. For example:

    Of course, not all – or even most – of these reasons will apply to other projects.

    Benefits and drawbacks of using Clang instead of Visual C++

    Benefits of using Clang, if you want to try for your project:
    • Clang supports 64-bit inline assembly. For example, in Chrome we built libyuv (a video format conversion library) with Clang long before we built all of Chrome with it. libyuv had highly-tuned 64-bit inline assembly with performance not reachable with intrinsics, and we could just use that code on Windows.
    • If your project runs on multiple platforms, you can use one compiler everywhere. Building your project with several compilers is generally considered good for code health, but in Chrome we found that Clang’s diagnostics found most problems and we were mostly battling compiler bugs (and if another compiler has a great new diagnostic, we can add that to Clang).
    • Likewise, if your project is Windows-only, you can get a second compiler’s opinion on your code, and Clang’s warnings might find bugs.
    • You can use Address Sanitizer to find memory bugs.
    • If you don’t use LTCG and PGO, it’s possible that Clang might create faster code.
    • Clang’s diagnostics and fix-it hints.
    There are also drawbacks:
    • Clang doesn’t support C++/CX or #import “foo.dll”.
    • MSVC offers paid support, Clang only gives you the code and the ability to write patches yourself (although the community is very active and helpful!).
    • MSVC has better documentation.
    • Advanced debugging features such as Edit & Continue don’t work when using Clang.

    How to use

    If you want to give Clang for Windows a try, there are two approaches:
    1. You could use clang-cl, a compiler driver that tries to be command-line flag compatible with cl.exe (just like Clang tries to be command-line flag compatible with gcc). The Clang user manual describes how you can tell popular Windows build systems how to call clang-cl instead of cl.exe. We used this approach in Chrome to keep the Clang/Win build working alongside the MSVC build for years, with minimal maintenance cost. You can keep using link.exe, all your current compile flags, the MSVC debugger or windbg, ETW, etc. clang-cl even writes warning messages in a format that’s compatible with cl.exe so that you can click on build error messages in Visual Studio to jump to the right file and line. Everything should just work.
    2. Alternatively, if you have a cross-platform project and want to use gcc-style flags for your Windows build, you can pass a Windows triple (e.g. --target=x86_64-windows-msvc) to regular Clang, and it will produce MSVC-ABI-compatible output. Starting in Clang 7.0.0, due Fall 2018, Clang will also default to CodeView debug info with this triple.
    Since Clang’s output is ABI-compatible with MSVC, you can build parts of your project with clang and other parts with MSVC. You can also pass /fallback to clang-cl to make it call cl.exe on files it can’t yet compile (this should be rare; it never happens in the Chrome build).

    clang-cl accepts Microsoft language extensions needed to parse system headers but tries to emit -Wmicrosoft-foo warnings when it does so (warnings are ignored for system headers). You can choose to fix your code, or pass -Wno-microsoft-foo to Clang.

    link.exe can produce regular PDB files from the CodeView information that Clang writes.

    Project History

    We switched chrome/mac and chrome/linux to Clang a while ago. But on Windows, Clang was still missing support for parsing many Microsoft language extensions, and it didn’t have any Microsoft C++ ABI-compatible codegen at all. In 2013, we spun up a team to improve Clang’s Windows support, consisting half of Chrome engineers with a compiler background and half of other toolchain people. In mid-2014, Clang could self-host on Windows. In February 2015, we had the first fallback-free build of 64-bit Chrome, in July 2015 the first fallback-free build of 32-bit Chrome (32-bit SEH was difficult). In Oct 2015, we shipped a first clang-built Chrome to the Canary channel. Since then, we’ve worked on improving the size of Clang’s output, improved Clang’s debug information (some of it behind -instcombine-lower-dbg-declare=0 for now), and A/B-tested stability and telemetry performance metrics.

    We use versions of Clang that are pinned to a recent upstream revision that we update every one to three weeks, without any local patches. All our work is done in upstream LLVM.

    Mid-2015, Microsoft announced that they were building on top of our work of making Clang able to parse all the Microsoft SDK headers with clang/c2, which used the Clang frontend for parsing code, but cl.exe’s codegen to generate code. Development on clang/c2 was halted again in mid-2017; it is conceivable that this was related to our improvements to MSVC-ABI-compatible Clang codegen quality. We’re thankful to Microsoft for publishing documentation on the PDB file format, answering many of our questions, fixing Clang compatibility issues in their SDKs, and for giving us publicity on their blog! Again, Clang is not a replacement for MSVC, but a complement to it.

    Opera for Windows is also compiled with Clang starting in version 51.

    Firefox is also looking at using clang-cl for building Firefox for Windows.

    Next Steps

    Just as clang-cl is a cl.exe-compatible interface for Clang, lld-link is a link.exe-compatible interface for lld, the LLVM linker. Our next step is to use lld-link as an alternative to link.exe for linking Chrome for Windows. This has many of the same advantages as clang-cl (open-source, easy to update, …). Also, using clang-cl together with lld-link allows using LLVM-bitcode-based LTO (which in turn enables using CFI) and using PE/COFF extensions to speed up linking. A prerequisite for using lld-link was its ability to write PDB files.
    We’re also considering using libc++ instead of the MSVC STL – this allows us to instrument the standard library, which is again useful for CFI and Address Sanitizer.

    In Closing

    Thanks to the whole LLVM community for helping to create the first new production C++ compiler for Windows in over a decade, and the first-ever open-source C++ compiler that’s ABI-compatible with MSVC!

    by Nico Weber (noreply@blogger.com) at March 06, 2018 06:54 PM

    February 14, 2018

    LLVM Blog

    LLVM accepted to 2018 Google Summer of Code!

    We are excited to announce the LLVM project has been accepted to 2018 Google Summer of Code!

    What is Google Summer of Code?

    Google Summer of Code (GSoC) is a global program focused on introducing students to open source software development. Students work on a 3 month programming project with an open source organization during their break from university. There are several benefits to this program for both the students and LLVM:

    • Inspire students to get involved with open source, compilers and LLVM
    • Give students exposure to real-world software development while getting paid a stipend
    • Allow students to do paid work related to their academic pursuits versus getting an unrelated summer job
    • Bring new developers into the LLVM project
    • Some LLVM bugs get fixed or new features get added

    Students - Apply now! 

    Ok, so you can't apply right now as the official application to GSoC does not open until March 12, 2018, but you must begin discussing your project on the LLVM mailing lists well before that date. There are many open projects listed on our webpage. Once you have selected a project, you will discuss it on the appropriate mailing list.

    If you have an idea for a project that is not listed, you can always propose it on the list as well and seek out a mentor.

    Key Dates to Remember

    We have listed a few key dates here, but always consult the official GSoC timeline to confirm.

    • March 12 (16:00 UTC) - Applications open
    • March 27 (16:00 UTC) - Deadline to file your application
    • April 23 (16:00 UTC) - Accepted student proposals are announced
    • May 14 - Coding begins


    LLVM Developers - Consider being mentor!

    This program is not a success without our mentors. Thank you to all that have all who have already volunteered! If you have never mentored a GSoC project but are curious, it is not too late to volunteer! You can either select an open project without a mentor or propose your own. Make sure to get it listed on the webpage so that students can see it as an option.

    If mentoring just isn't an option for you at this time, consider helping the project out my spreading the word about GSoC.

    Questions?

    If you have questions about the program for the organizers, please email gsoc@lists.llvm.org. Project specific questions should be sent to the appropriate developer mailing list instead.

    by Tanya Lattner (noreply@blogger.com) at February 14, 2018 09:38 PM

    January 09, 2018

    LLVM Blog

    Improving Link Time on Windows with clang-cl and lld

    One of our goals in bringing clang and lld to Windows has always been to improve developer experience, and what is it that developers want the most?  Faster build times!  Recently, our focus has been on improving link time because it's the step that's the hardest to parallelize so we can't fall back on the time honored tradition of throwing more cores at it.

    Of the various steps involved in linking, generating the debug info (which, on Windows, is a PDB file) is by far the slowest since it involves merging O(# of linker inputs) sequences of type records, most of which are duplicate anyway.  For example, if two cpp files both include <string>, then both of those object files will have hundreds of duplicate type records that need to be de-duplicated during the link step.  This means you have to compute O(M x N) hash values, even though only a small fraction of those ultimately contribute to the final PDB.

    Several strategies have been invented to deal with this over the years and try to make linking faster.  Many years ago, Microsoft introduced the notion of a Type Server (enabled via /Zi compiler option in MSVC), which moves some of the work into the compiler (to take advantage of parallelism).  More recently we have been given the /DEBUG:FASTLINK linker option which attempts to solve the problem by not merging types at all in the linker.  However, each of these strategies has its own set of disadvantages, and neither can be considered perfect for all use cases.

    In this blog post, we'll first go over some technical background about CodeView so that we can understand the problem, followed by a summary of existing attempts to speed up type merging.  Then, we'll describe a novel extension to the PE/COFF file format which speeds up linking by offloading part of the work required to de-duplicate types to the compiler and using a new algorithm which uniquely identifies type records even across input files, and discuss the various tradeoffs of each approach.  Finally, we'll present some benchmarks and discuss how you can try this out in clang-cl and lld today.


    Background

    Consider a simple structure in C++, defined like this a header file:

         struct Node {
           Node *Next = nullptr;
           Node *Prev = nullptr;
           int Value = 0;
         };

    Since each compilation happens independently of every other compilation, the compiler cannot assume any other translation unit will ever emit the records necessary to describe this type.  As a result, to guarantee that the type makes it into the final PDB, every compiler instance that encounters this definition must emit type information for this type.  So the record will be serialized by the compiler into a series of records that looks roughly like this:

    0x1004 | LF_STRUCTURE [size = 40] `Node`
             unique name: `.?AUNode@@`
             vtable: <none>
             base list: <none>
             field list: <none>
             options: forward ref | has unique name
    0x1005 | LF_POINTER [size = 12]
             referent = 0x1004
             mode = pointer
             opts = None
             kind = ptr32
    0x1006 | LF_FIELDLIST [size = 52]
             - LF_MEMBER
               name = `Next`
               Type = 0x1005
               Offset = 0
               attrs = public
             - LF_MEMBER
               name = `Prev`
               Type = 0x1005
               Offset = 4
               attrs = public
             - LF_MEMBER
               name = `Value`
               Type = 0x0074 (int)
               Offset = 8
               attrs = public
    0x1007 | LF_STRUCTURE [size = 40] `Node`
             unique name: `.?AUNode@@`
             vtable: <none>
             base list: <none>
             field list: 0x1006
             options: has unique name
    The values on the left correspond to the types index in the type sequence and depend on what types have already been encountered, while other types can the refer to them (for example, referent = 0x1004) means that this record is a pointer to whatever the type at index 0x1004 was.

    As a result of this design, another compilation unit which includes the same header file will need to emit this exact same type, with the only difference being the indices (since the other compilation may encounter other types before this one, causing the ordering to be different).

    In short, type indices only make sense within the context of a single type sequence (i.e. compiland), but since the linker needs to see across all object files, it has to have some way of identifying whether a type from object file A is isomorphic to a different type from object file B, even if its type indices might be different numerically from any previously seen type. 

    This algorithm, henceforth referred to as type merging, is the primary consumer of CPU cycles during linking (measured in LLD, and estimated in MSVC linker by comparing /DEBUG:FULL vs /DEBUG:FASTLINK times), and as such it is the portion of the linking process which this blog post presents a new solution to.

    Existing Solutions

    It’s worthwhile to discuss some of the existing attempts to reduce the cost associated with type merging so that we can compare and contrast their various pros and cons.

    Type Servers (/Zi)


    The /Zi compiler option was one of the first attempts to address type merging speed, and it dates back many years.  The idea behind type servers is to offload the work of de-duplication from the linking phase to the compilation phase.  Most build systems already support parallel compilation, and even if they don’t cl.exe supports it natively via the /MP compiler switch, so there is no roadblock to anyone taking advantage of parallel compilation. 



    To implement type servers, each compilation process communicates via IPC with a single process (mspdbsrv.exe) whose job is to de-duplicate type records on the fly, and when a record is isomorphic to an existing record, the type server communicates back the previously saved index, and when it is new it sends back a new index.  This allows type deduplication to happen mostlyin parallel, but adding some overhead to each compilation (since there is contention over a global lock) in return for significantly reduced link times, since types will already have been merged.


    Type servers bring with them some disadvantages though, so we enumerate them here:
    1. Type servers add significant context switching and global lock contention to the compilation phase, reducing parallelism and degrading overall system performance while a build is in process.  While some performance is reclaimed from the linker, some is sacrificed due to the use of a global system lock.  It’s still a net win, but as it is not free, it leaves open the possibility that we may be able to achieve better parallelism using a different approach.
    2. The type server process itself (mspdbsrv.exe) introduces a single point of failure.  When it crashes (we see C1033 several times per day on Chrome, for example, which seems to indicate an mspdbsrv.exe crash) it could trigger a full rebuild if the type server PDB file is left in a corrupt state.
    3. mspdbsrv is incompatible with distributed builds, which is a show-stopper for large applications that can take several hours to build on normal workstations.  Type servers operate only via local IPC.  While multi-processing works well for small applications, many large products have build farms that distribute compilations among tens or hundreds of physical machines.  Type servers are incompatible with this scenario.

    Fastlink PDBs

    Fastlink PDBs are a relatively recent introduction, and the approach used by this solution is to eliminate type merging entirely.  To support this, special metadata is set in the PDB file to indicate to the tool that this is a fastlink PDB, and when the tool (e.g. debugger) encounters this metadata, it will fetch all type information from the original object file, rather than from the PDB.  As before, there are several disadvantages to this approach, enumerated here:
    1. The pdbcopy utility is almost unusable with fastlink PDBs for performance reasons.
    2. Since type merging doesn’t happen, indexing of type information also doesn’t happen (since the expensive part of building an index -- the hashing -- comes for free when you were hashing the record anyway).  This leads to degradation in the debugger user experience, since waits which previously happened only at build time now happen at debug-time.
    3. Fastlink PDBs are not portable.  The PDB references the object files by path, so if you copy the PDB and object files to a different machine (or even different path on the same machine) for archival purposes, they can no longer be debugged.  This is a deal-breaker for using it on production builds
    4. Symbols can’t be enumerated in a Fastlink PDB.  This is most obvious if you attempt to use DIA SDK on a Fastlink PDB, where it will simply refuse to do anything at all.  This means that the only externally supported way of querying debug info for users is impossible against a Fastlink PDB.  Beyond that, however, it also means that even Microsoft’s own tools which need to enumerate symbols cannot use any standard API for doing so.  For example, WinDbg doesn’t fully support Fastlink PDBs, and many workflows are broken by the use of them, even using supported Microsoft tools.
    5. It has several serious stability issues which make it unusable on large projects  [ref].  This is probably related to point 4 above, namely the fact that every tool that wants to be able to work with a Fastlink PDB needs to use different code than the SDK that has been tested and battle-hardened through years of development.
    6. When compiling with clang-cl and linking with /debug:fastlink the compiler has to be instructed to emit additional debug information, making .obj files about 29% larger.

    Clang's Solution - The COFF .debug$H section

    This new approach tries to combine the ideas behind type servers and fastlink PDBs.  Like type servers, it attempts to offload the work of de-duplication to the compilation phase so that it can be done in parallel.  However, it does so using an algorithm with the property that the resulting hash can be used to identify a type record even across type streams.  Specifically, if two records have the same hash, they are the same record even if they are from different object files.  If you can take it on faith that such an algorithm exists (which will be henceforth referred to as a global hash), then the amount of work that the linker needs to perform is greatly reduced.  And the work that it does still have to do can be done much quicker.  Perhaps most importantly, it produces a byte-for-byte identical PDB to when the option is not used, meaning all of the issues surrounding Fastlink PDBs and compatibility are gone.

    Previously, the linker would do something that looks roughly like this:

         HashTable<Type> HashedTypes;
         vector<Type> MergedTypes;
         for (ObjectFile &Obj : Objects) {
           for (Type &T : Obj.types()) {
             remapAllTypeIndices(MergedTypes, T);

             if (!HashedTypes.try_insert(T))
               continue;
             MergedTypes.push_back(T);
           }
         }
    The important observations here are:
    1. remapAllTypeIndices is called unconditionally for every type in every object file.
    2. A hash of the type is computed unconditionally for every type
    3. At least one full record comparison is done for every type.  In practice it turns out to be much more, because hash buckets are computed modulo table size, so there will actually be 1 full record comparison for every probe.
    Given a global hash function as described above, the algorithm can be re-written like this:
          HashMap<SHA1, int> HashToIndex;
          vector<Type> OrderedTypes;
          for (ObjectFile &Obj : Objects) {
            auto Hashes = Obj.DebugHSectionHashes;
            for (int I=0; I < Obj.NumTypes; ++I) {
              int NextIndex = OrderedTypes.size();
              if (!HashToIndex.try_emplace(Hashes[I], NextIndex))
                continue;
              remapAllTypeIndices(T);
              OrderedTypes.push_back(T);
            }
          }

    While this appears very similar, its performance characteristics are quite different.
    1. remapAllTypeIndices is only called when the record is actually new.  Which, as we discussed earlier, is a small fraction of the time over many linker inputs.
    2. A hash of the type is never computed by the linker.  It is simply there in the object file (the exception to this is mixed linker inputs, discussed earlier, but those are a small fraction of input files).
    3. Full record comparisons never happen.  Since we are using a strong hash function with negligible chance of false collisions, and since the hash of a record provides equality semantics across streams, the hash is as good as the record itself.

    Combining all of these points, we get an algorithm that is extremely cache friendly.  Amortized over all input files, most records during type merging are cache hits (i.e. duplicate records).  With this algorithm when we get a cache hit, the only two data structures that are accessed are:
    1. An array of contiguous hash values.
    2. An array of contiguous hash buckets.
    Since we never do full equality comparison (which would blow out the L1 and sometimes even L2 cache due to the average size of a type record being larger than a cache line) the algorithm here is very fast.

    We’ve deferred discussion of how to create such a hash up until now, but it is actually fairly straightforward.  We use what is known as a “tree hash” or “Merkle tree”.  The idea is to pass bytes from a type record directly to the hash function up until the point we get to a type index.  Then, instead of passing the numeric value of the type index to the hash function, we pass the previously computed hash of the record that is being referenced.

    Such a hash is very fast to compute in the compiler because the compiler must already hash types anyway, so the incremental cost to emit this to the .debug$H section is negligible.  For example, when a type is encountered in a translation unit, before you can add that type to the object file’s .debug$T section, it must first be verified that the type has not already been added.  And since this is happening naturally in the order in which types are encountered, all that has to be done is to save these hash values in an array indexed by type index, and subsequent hash operations will have O(1) access to all of the information needed to compute this merkle hash.
      

    Mixed Input Files and Compiler/Linker Compatibility

    A linker must be prepared to deal with a mixed set of input files.  For example, while a particular compiler may choose to always emit .debug$H sections, a linker must be prepared to link objects that for whatever reason do not have this section.  To handle this, the linker can examine all inputs up front and manually compute hashes for inputs with missing .debug$Hsections.  In practice this proves to be a small fraction and the penalty for doing this serially is negligible, although it should be noted that in theory this can also be done as a parallel pre-processing step if some use cases show that this has non-negligible cost.

    Similarly, the emission of this section in an object file has no impact on linkers which have not been taught to use it.  Since it is a purely additive (and optional) inclusion into the object file, any linker which does not understand it will continue to work exactly as it does today.

    The On-Disk Format

    Clang uses the following on-disk format for the .debug$H section.

               0x0     : <Section Magic>  (4 bytes)
         0x4     : <Version>        (2 bytes)
         0x6     : <Hash Algorithm> (2 bytes)
         0x8     : <Hash Value>     (N bytes)
         0x8 + N : <Hash Value>     (N bytes)
                        …

    Here, “Section Magic” is an arbitrarily chosen 4-byte number whose purpose is to provide some level of certainty that what we’re seeing is a real .debug$H section, and not some section that someone created that accidentally happened to be called that.   Our current implementation uses the value 0x133C9C5, which represents the date of the initial prototype implementation.  But this can be any reasonable value here, as long as it never changes.

    “Version” is reserved for future use, so that the format of the section can theoretically change.

    “Hash Algorithm” is a value that indicates what algorithm was used to generate the hashes that follow.  As such, the value of N above is also a function of what hash algorithm is used.  Currently, the only proposed value for Hash Algorithm is SHA1 = 0, which would imply N = 20 when Hash Algorithm = 0.  Should it prove useful to have truncated 8-byte SHA1 hashes, we could define SHA1_8 = 1, for example.

    Limitations and Pitfalls

    The biggest limitation of this format is that it increases object file size.  Experiments locally on fairly large projects show an average aggregate object file size increase of ~15% compared to /DEBUG:FULL (which, for clang-cl, actually makes .debug$H object files smallerthan those needed to support /DEBUG:FASTLINK).

    There is another, less obvious potential pitfall as well.  The worst case scenario is when no input files have a .debug$H section present, but this limitation is the same in principle even if only a subset of files have a .debug$H section.  Since the linker must agree on a single hash function for all object files, there is the question of what to do when not all object files agree on hash function, or when not all object files contain a .debug$H section.  If the code is not written carefully, you could get into a situation where, for example, no input files contain a .debug$H section so the linker decides to synthesize one on the fly for every input file.  Since SHA1 (for example) is quite slow, this could cause a huge performance penalty.

    This limitation can be coded around with some care, however.  For example, tree hashes can be computed up-front in parallel as a pre-processing step.  Alternatively, a hash function could be chosen based on some heuristic estimate of what would likely lead to the fastest link (based on the percentage of inputs that had a .debug$H section, for example).  There are other possibilities as well.  The important thing is to just be aware of this potential pitfall, and if your links become very slow, you'll know that the first thing you should check is "do all my object files have .debug$H sections?"

    Finally, since a hash is considered to be identical to the original record, we must consider the possibility of collisions.  That said, this does not appear to be a serious concern in practice.  A single PDB can have a theoretical maximum of 232 type records anyway (due to a type index being 4 bytes).  The following table shows the expected number of type records needed for a collision to exist as a function of hash size.
    Hash Size (Bytes)
    Average # of records needed for a collision
    4
    82,137
    6
    21,027,121
    8
    5,382,943,231
    12
    3.53 x 1014
    16
    2.31 x 1019
    20
    1.52 x 1024
    Given that this is strictly for debug information and not generated code, it’s worth thinking about the severity of a collision.  We feel that an 8-byte hash is probably acceptable for real world use.

    Benchmarks

    Here we will give some benchmarks on large real world applications (specifically, Chrome and clang).  The times presented are only for the linker.  gn args for each build of chromium are specified at the end..


    Toolchain 

    Mode
    Target
    blink_core.dll
    content.dll
    chrome.dll
    clang.exe
    MSVC
    /DEBUG:FULL
    553.11s
    205.45s
    507.17s
    62.45s
    MSVC
    /DEBUG:FASTLINK
    116.77s
    56.05s
    67.80s
    29.37s
    lld-link
    /DEBUG:FULL
    121.17s
    42.10s
    42.31s
    24.14s
    lld-link
    /DEBUG:GHASH
    88.71s
    33.30s
    34.76s
    17.99s


    The numbers here indicate a reduction in link time of up to 30% by enabling /DEBUG:GHASH in lld.

    It's worth mentioning that lld does not yet have support for incremental linking so we could not compare the cost of an incremental link with /DEBUG:GHASH versus MSVC.  We still expect incremental linking using MSVC under optimal conditions (e.g. change whitespace in a header file) to produce much faster links than lld is currently able to do.

    There are several possible avenues for further optimization though, so we will finish up by discussing them.

    Further Improvements

    There are several ways to improve the times further, which have yet to be explored.

    1. Use a smaller or faster hash.  We use a 20-byte SHA1 hash.  This is not a multiple of cache line size, and in any case the probability of collision is astronomically small even in the largest PDBs, considering that the theoretical limit of a PDB is just under 2^32 possible unique types (due to the 4-byte size of a type index).  SHA1 is also notoriously slow.  It might be interesting to try, for example, a Blake2 set to output an 8 byte hash.  This should give sufficiently low probability of a collision while improving cache performance.  The on-disk format is designed with this flexibility in mind, as different hash algorithms can be specified in the header.
    2. Hashes for compilands with missing .debug$H sections can be computed in parallel before linking.  Currently when we encounter an object file without a .debug$H section, we must synthesize one in the linker.  Our prototype algorithm does this serially for each input.
    3. Symbol records from .debug$S sections can be merged in parallel.  Currently in lld, we first merge type records into the TPI stream, then we iterate symbol records and remap types in each symbol record to correspond to the new type indices.  If we merge types from all modules up front, the symbol records (with the exception of global symbols) can be merged in parallel since they get written to independent streams).

    Try it out!

    If you're already using clang-cl and lld on Windows today, you can try this out.  There are two flags needed to enable this, one for the compiler and one for the linker:
    1. To enable the emission of a .debug$H section by the compiler, you will need to pass the undocumented -mllvm -emit-codeview-ghash-section flag to clang-cl  (this flag should go away in the future, once this is considered stable and good enough to be turned on by default).
    2. To tell lld to use this information, you will need to pass the /DEBUG:GHASH to lld.
    Note that this feature is still considered highly experimental, so we're interested in your feedback (llvm-dev@ mailing list, direct email is ok too) and bug reports (bugs.llvm.org).  

    by Zachary Turner (noreply@blogger.com) at January 09, 2018 06:49 PM

    September 21, 2017

    LLVM Blog

    2017 US LLVM Developers' Meeting Program

    The LLVM Foundation is excited to announce the selected proposals for the 2017 US LLVM Developers' Meeting!

    Keynotes:


    Talks:


    BoFs:


    Tutorials:


    Lightning Talks:


    Student Research Competition:


    Posters:

    If you are interested in any of these talks, you should register to attend the 2017 US LLVM Developers' Meeting! Tickets are limited, so register now!

    by Tanya Lattner (noreply@blogger.com) at September 21, 2017 08:14 AM

    September 20, 2017

    LLVM Blog

    Clang ♥ bash -- better auto completion is coming to bash



    Compilers are complex pieces of software and have a multitude of command-line options to fine tune parameters. Clang is no exception: it has 447 command-line options. It’s nearly impossible to memorize all these options and their correct spellings, that's where shell completion can be very handy. When you type in the first few characters of a flag and hit tab, it will autocomplete the rest for you.

    Background
    However, such a autocompletion feature is not available yet, as there's no easy way to get a complete list of the options Clang supports. For example, bash doesn’t have any autocompletion support for Clang, and despite some shells like zsh having a script for command-line autocompletion, they use hard coded lists of command-line options, and are not automatically updated when a new option is added to Clang. These shells also can’t autocomplete arguments which some flags take (-std=[tab] for instance).


    This is the problem we were working to solve during this year’s Google Summer of Code. We’re adding a feature to Clang so that we can implement a complete, exact command-line option completion which is highly portable for any shell. To start with, we'll provide a completion script for bash which uses this feature.

    Implementation
    Clang now has a new command line option called --autocomplete. This flag receives the incomplete user input from the shell and then queries the internal data structures of the current Clang binary, and returns a list of possible completions. With this API, we can always get an accurate list of options and values any time, on any newer versions of Clang.

    We built an autocompletion using this in bash for the first implementation. You can find its source code here. Also, here is the sample for Qt text entry autocompletion to give an example how to use this API from an UI application as seen below:

    final.gif

    You can always complete one flag at a time. So if you want to use the API, you have to select the flag that the user is currently typing. Then just pass this flag to the --autocomplete flag in the selected clang binary. So in the case below all flags start with `-tr` are displayed with their descriptions behind them (separated from the flag with a tab character).
    The API also supports completing the values of flags. If you have a flag for which value completion is supported, you can also provide an incomplete value behind the flag separated by a comma to get completion for this:
    If you provide nothing after the comma, the list of the all possible values for this flag is displayed.

    How to get it
    This feature is available for use now with LLVM/clang 5.0 and we’ll also be adding this feature to the standard bash completion package. Make sure you have the latest clang version on your machine, and source this script. If want to make the change permanent, just source it from your .bashrc and enjoy typing your clang invocations!

    by Yuka Takahashi (noreply@blogger.com) at September 20, 2017 04:11 PM

    August 23, 2017

    Sylvestre Ledru

    Rebuild of Debian using Clang 3.9, 4.0 and 5.0

    tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go.

    The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years.

    Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th).

    For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month.

    Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures.

    Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
    For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error.

    In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).

    VersionBuild failuresIgnoring haskell pkgs
    3.81367 / 5.6%
    3.92274 / 8.1%1618 / 5.8%
    4.02311 / 8.3%1655 / 5.9%
    5.01445 / 5.1%

    In parallel, new warnings and errors showed up in Clang.
    This is causing a new set of build failures (especially with the usage of -Werror).

    As few examples:
    * Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
    * Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
    * clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
    * With the same version, clang can trigger a new error when auto is used in function return type.

    Now, as a conclusion, having Debian being built with clang by default is still a long shot.
    First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
    and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
    As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
    Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
    Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs.

    Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects.

    Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).

    by Sylvestre at August 23, 2017 10:09 PM

    August 18, 2017

    LLVM Blog

    LLVM on Windows now supports PDB Debug Info

    For several years, we’ve been hard at work on making clang a world class toolchain for developing software on Windows.  We’ve written about this several times in the past, and we’ve had full ABI compatibility (minus bugs) for some time. One area that been notoriously hard to achieve compatibility on has been debug information, but over the past 2 years we’ve made significant leaps.  If you just want the TL;DR, then here you go: If you’re using clang on Windows, you can now get PDB debug information!


    Background: CodeView vs. PDB
    CodeView is a debug information format invented by Microsoft in the mid 1980s. For various reasons, other debuggers developed an independent format called DWARF, which eventually became standardized and is now widely supported by many compilers and programming languages.  CodeView, like DWARF, defines a set of records that describe mappings between source lines and code addresses, as well as types and symbols that your program uses.  The debugger then uses this information to let you set breakpoints by function name, display the value of a variable, etc.  But CodeView is only somewhat documented, with the most recent official documentation being at least 20 years old.  While some records still have the format documented above, others have evolved, and entirely new records have been introduced that are not documented anywhere.


    It’s important to understand though that CodeView is just a collection of records.  What happens when the user says “show me the value of Foo”?  The debugger has to find the record that describes Foo.  And now things start getting complicated.  What optimizations are enabled?  What version of the compiler was used?  (These could be important if there are certain ABI incompatibilities between different versions of the compiler, or as a hint when trying to reconstruct a backtrace in heavily optimized code, or if the stack has been smashed).  There are a billion other symbols in the program, how can we find the one named Foo without doing an exhaustive O(n) search?  How can we support incremental linking so that it doesn’t take a long time to re-generate debug info when only a small amount of code has actually changed?  How can we save space by de-duplicating strings that are used repeatedly?  Enter PDB.


    PDB (Program Database) is, as you might have guessed from the name, a database.  It contains CodeView but it also contains many other things that allow indexing of the CodeView records in various ways.  This allows for fast lookups of types and symbols by name or address, the philosophical equivalent of “tables” for individual input files, and various other things that are mostly invisible to you as a user but largely responsible for making the debugging experience on Windows so great.  But there’s a problem: While CodeView is at least kind-of documented, PDB is completely undocumented.  And it’s highly non-trivial.


    We’re Stuck (Or Are We?)
    Several years ago, we decided that the path forward was to abandon any hope of emitting CodeView and PDB, and instead focus on two things:
    1. Make clang-cl emit DWARF debug information on Windows
    2. Port LLDB to Windows and teach it about the Windows ABI, which would be significantly easier than teaching Visual Studio and/or WinDbg to be able to interpret DWARF (assuming this is even possible at all, given that everything would have to be done strictly through the Visual Studio / WinDbg extensibility model)
    In fact, I even wrote another blog post about this very topic a little over 2 years ago.  So I got it to work, and I eventually got parts of LLDB working on Windows for simple debugging scenarios.


    Unfortunately, it was beginning to become clear that we really needed PDB.  Our goal has always been to create as little friction as possible for developers who are embedded in the Windows ecosystem.  Tools like Windows Performance Analyzer and vTune are very powerful and standard tools in engineers’ existing repertoires.  Organizations already have infrastructure in place to archive PDB files, and collect & analyze crash dumps.  Debugging with PDB is extremely responsive given that the debugger does not have to index symbols upon startup, since the indices are built into the file format.  And last but not least, tools such as WinDbg are already great for post-mortem debugging, and frankly many (perhaps even most) Windows developers will only give up the Visual Studio debugger when it is pried from their cold dead hands.


    I got some odd stares (to put it lightly) when I suggested that we just ask Microsoft if they would help us out.  But ultimately we did, and… they agreed!  This came in the form of some code uploaded to the Microsoft Github repo which we were on our own to figure out.  Although they were only able to upload a subset of their PDB code (meaning we had to do a lot of guessing and exploration, and the code didn’t compile either since half of it was missing), it filled in enough blanks that we were able to do the rest.


    After about a year and a half of studying this code, hacking away, studying the code some more, hacking away some more, etc, I’m proud to say that lld (the LLVM linker) can finally emit working PDBs.  All the basics like setting breakpoints by line, or by name, or viewing variables, or searching for symbols or types, everything works (minus bugs, of course).


    For those of you who are interested in digging into the internals of a PDB, we also have been developing a tool for expressly this purpose.  It’s called llvm-pdbutil and is the spiritual counterpart to Microsoft’s own cvdump utility.  It can dump the internals of a PDB, convert a PDB to yaml and vice versa, find differences between two PDBs, and much more.  Brief documentation for llvm-pdbutil is here, and a detailed description of the PDB file format internals are here, consisting of everything we’ve learned over the past 2 years (still a work in progress, as I have to divide my time between writing the documentation and actually making PDBs work).


    Bring on the Bugs!
    So this is where you come in.  We’ve tested simple debugging scenarios with our PDBs, but we still consider this alpha in terms of debug info quality.  We’d love for you to try it out and report issues on our bug tracker.  To get you started, download the latest snapshot of clang for Windows.  Here are two simple ways to test out this new functionality:
    1. Have clang-cl invoke lld automatically
      1. clang-cl -fuse-ld=lld -Z7 -MTd hello.cpp
    2. Invoke clang-cl and lld separately.
      1. clang-cl -c -Z7 -MTd -o hello.obj hello.cpp
      2. lld-link -debug hello.obj
    We look forward to the onslaught of bug reports!


    We would like to extend a very sincere and deep thanks to Microsoft for their help in getting the code uploaded to the github repository, as we would never have gotten this far without it.


    And to leave you with something to get you even more excited for the future, it's worth reiterating that all of this is done without a dependency on any windows specific api, dll, or library.  It's 100% portable.  Do I hear cross-compilation?

    Zach Turner (on behalf of the the LLVM Windows Team)

    by Zachary Turner (noreply@blogger.com) at August 18, 2017 07:55 PM

    August 17, 2017

    LLVM Blog

    LLVM Weekly - #130, Jun 27th 2016

    Welcome to the one hundred and thirtieth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.
    If you're reading this on blog.llvm.org then do note this is LAST TIME it will be cross-posted there directly. There is a great effort underway to increase the content on the LLVM blog, and unfortunately LLVM Weekly has the effect of drowning out this content. As ever, you can head to http://llvmweekly.org, subscribe to get it by email, or subscribe to the RSS feed.
    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    After recently being taken down due to excessive resource usage, the LLVM apt repositories are now back.
    A detailed introduction to ThinLTO has been published on the LLVM blog. This covers the background, design, current status, and usage information for ThinLTO.
    A post on Reddit gives a summary of notable language features voted into the C++17 working draft at the Oulu meeting.

    On the mailing lists

    LLVM commits

    • The new representation for control-flow integrity and virtual call metadata has landed. The commit message further details the problems this change addresses. r273729.
    • The llvm.type.checked.load intrinsic was added. It loads a function pointer from a virtual table pointer using type metadata. r273576.
    • As part of the work on CFL-AA, interprocedural function summaries were added. These avoid recomputation for many properties of a function. r273219, r273596.
    • MemorySSA gained new APIs for PHI creation and MemoryAccess creation. r273295.
    • Metadata attachments are now allowed for declarations. r273336.
    • A new runtimes directory was added to the LLVM tree. r273620.
    • LLVM's dynamic loader gained basic support for COFF ARM. r273682.

    Clang commits

    • constexpr if support has been added to Clang. r273602.
    • clang-tidy has a new modernize-use-emplace check that will replace calls of push_back to emplace_back. r273275.
    • The CMake build system for Clang gained a ENABLE_X86_RELAX_RELOCATIONS option. r273224.

    Other project commits

    • Basic support for versioned symbols was added to LLD. r273143.
    • LLD now handles both single and double dashes for all options. r273256.

    by Alex Bradbury (noreply@blogger.com) at August 17, 2017 10:48 PM

    March 10, 2017

    LLVM Blog

    Devirtualization in LLVM and Clang

    This blog post is part of a series of blog posts from students who were funded by the LLVM Foundation to attend the 2016 LLVM Developers' Meeting in San Jose, CA. Please visit the LLVM Foundation's webpage for more information on our Travel Grants program. 

    This post is from Piotr Padlewski on his work that he presented at the meeting:

    This blogpost will show how C++ devirtualization is performed in current (4.0) clang and LLVM and also ongoing work on -fstrict-vtable-pointers features.

    Devirtualization done by the frontend


    In order to transform a virtual call into a direct call, the frontend must be sure that there are no overrides of vfunction in the program or know the dynamic type of object. Compilation proceeds one translation unit at a time, so, barring LTO, there are only a few cases when the compiler may conclude that there are no overrides:

    • either the class or virtual method is marked as final
    • the class is defined in an anonymous namespace and has no deriving classes in its translation unit

    The latter is more tricky for clang, which translates the source code in chunks on the fly (see: ASTProducer and ASTConsumer), so is not able to determine if there are any deriving classes later in the source. This could be dealt with in a couple of ways:
    • give up immediate generation
    • run data flow analysis in LLVM to find all the dynamic types passed to function, which has static linkage
    • hope that every use of the virtual function, which is necessarily in the same translation unit, will be inlined by LLVM -- static linkage increases the chances of inlining

    Store to load propagation in LLVM

    In order to devirtualize a virtual call we need:
    • value of vptr - which virtual table is pointed by it
    • value of vtable slot - which exact virtual function it is

    Because vtables are constant, the latter value is much easier to get when we have the value of vptr. The only thing we need is vtable definition, which can be achieved by using available_externally linkage.

    In order to figure out the vptr value, we have to find the store to the same location that defines it. There are 2 analysis responsible for it:

    • MemDep (Memory Dependence Analysis) is a simple linear algorithm that for each quered instruction iterates through all instructions above and stops when first dependency is found. Because queries might be performed for each instruction we end up with a quadratic algorithm. Of course quadratic algorithms are not welcome in compilers, so MemDep can only check certain number of instructions.
    • Memory SSA on the other hand has constant complexity because of caching. To find out more, watch “Memory SSA in 5minutes” (https://www.youtube.com/watch?v=bdxWmryoHak). MemSSA is a pretty new analysis and it doesn’t have all the features MemDep has, therefore MemDep is still widely used.
    The LLVM main pass that does store to load propagation is GVN - Global Value Numbering.



    Finding vptr store

    In order to figure out the vptr value, we need to see store from constructor. To not rely on constructor's availability or inlining, we decided to use the @llvm.assume intrinsic to indicate the value of vptr. Assume is akin to assert - optimizer seeing call to @llvm.assume(i1 %b) can assume that %b is true after it. We can indicate vptr value by comparing it with the vtable and then call the @llvm.assume with the result of this comparison.

    call void @_ZN1AC1Ev(%struct.A* %a) ; call ctor
     %3 = load {...} %a                  ; Load vptr
     %4 = icmp eq %3, @_ZTV1A      ; compare vptr with vtable
     call void @llvm.assume(i1 %4)


    Calling multiple virtual functions

    A non-inlined virtual call will clobber the vptr. In other words, optimizer will have to assume that vfunction might change the vptr in passed object. This sounds like something that never happens because vptr is “const”. The truth is that it is actually weaker than C++ const member, because it changes multiple times during construction of an object (every base type constructor or destructor must set vptrs). But vptr can't be changed during a virtual call, right? Well, what about that?

    void A::foo() { // virtual
    static_assert(sizeof(A) == sizeof(Derived));
    new(this) Derived;
    }

    This is call of placement new operator - it doesn’t allocate new memory, it just creates a new object in the provided location. So, by constructing a Derived object in the place where an object of type A was living, we change the vptr to point to Derived’s vtable. Is this code even legal? C++ Standard says yes.

    However it turns out that if someone called foo 2 times (with the same object), the second call would be undefined behavior. Standard pretty much says that call or dereference of a pointer to an object whose lifetime has ended is UB, and because the standard agrees that nuking object from inside ends its lifetime, the second call is UB. Be aware that this is only because a zombie pointer is used for the second call. The pointer returned by placement new is considered alive, so performing calls on that pointer is valid. Note that we also silently used that fact with the use of assume.

    (un)clobbering vptr

    We need to somehow say that vptr is invariant during its lifetime. We decided to introduce a new metadata for that purpose - !invariant.group. The presence of the invariant.group metadata on the load/store tells the optimizer that every load and store to the same pointer operand within the same invariant group can be assumed to load or store the same value. With -fstrict-vtable-pointers Clang decorates vtable loads with invariant.group metadana coresponding to caller pointer type. 

    We can enhance the load of virtual function (second load) by decorating it with !invariant.load, which is equivalent of saying “load from this location is always the same”, which is true because vtables never changes. This way we don’t rely on having the definition of vtable.

    Call like:

    void g(A *a) {
      a->foo();
      a->foo();
    }

    Will be translated to:

    define void @function(%struct.A* %a) {
     %1 = load {...} %a, !invariant.group !0
     %2 = load {...} %1, !invariant.load !1
     call void %2(%struct.A* %a)

     %3 = load {...} %a, !invariant.group !0
     %4 = load {...} %4, !invariant.load !1
     call void %4(%struct.A* %a)
     ret void
    }

    !0 = !{!"_ZTS1A"} ; mangled type name of A
    !1 = !{}

    And now by magic of GVN and MemDep:

    define void @function(%struct.A* %a) {
     %1 = load {...} %a, !invariant.group !0
     %2 = load {...} %1, !invariant.load !1
     call void %2(%struct.A* %a)
     call void %2(%struct.A* %a)
     ret void
    }

    With this, llvm-4.0 is be able to devirtualize function calls inside loops. 

    Barriers

    In order to prevent the middle-end from finding load/store with the same !invariant.group metadata, that would come from construction/destruction of dead dynamic object, @llvm.invariant.group.barrier was introduced. It returns another pointer that aliases its argument but is considered different for the purposes of load/store invariant.group metadata. Optimizer won’t be able to figure out that returned pointer is the same because intrinsics don’t have a definition. Barrier must be inserted in all the places where the dynamic object changes:
    • constructors
    • destructors
    • placement new of dynamic object

    Dealing with barriers

    Barriers hinder some other optimizations. Some ideas how it could be fixed:

    • stripping invariant.group metadata and barriers just after devirtualization. Currently it is done before codegen. The problem is that most of the devirtualization comes from GVN, which also does most of the optimizations we would miss with barriers. GVN is expensive therefore it is run only once. It also might make less sense if we are in LTO mode, because that would limit the devirtualization in the link phase. 
    • teaching important passes to look through the barrier. This might be very tricky to preserve the semantics of barrier, but e.g. looking for dependency of load without invariant.group by jumping through the barrier to find a store without invariant.group, is likely to do the trick.
    • removing invariant.barrier when its argument comes from alloca and is never used etc.
    To find out more details about devirtualization check my talk (http://llvm.org/devmtg/2016-11/#talk6) from LLVM Dev Meeting 2016.

    About author

    Undergraduate student at University of Warsaw, currently working on C++ static analysis in IIIT.

    by Tanya Lattner (noreply@blogger.com) at March 10, 2017 09:23 PM

    March 07, 2017

    LLVM Blog

    Some news about apt.llvm.org

    apt.llvm.org provides Debian and Ubuntu repositories for every maintained version of these distributions. LLVM, Clang, clang extra tools, compiler-rt, polly, LLDB and LLD packages are generated for the stable, stabilization and development branches.

    As it seems that we have more and more users of these packages, I would like to share an update about various recent changes.

    New features

    LLD
    First, the cool new stuff : lld is now proposed and built for i386/amd64 on all Debian and Ubuntu supported versions. The test suite is also executed and the coverage results are great.

    4.0
    Then, following the branching for the 4.0 release, I created new repositories to propose this release.
    For example, for Debian stable, just add the following in /etc/apt/sources.list.d/llvm.list

    deb http://apt.llvm.org/jessie/ llvm-toolchain-jessie-4.0 main
      deb-src http://apt.llvm.org/jessie/ llvm-toolchain-jessie main

    llvm-defaults
    Obviously, the trunk is now 5.0. If llvm-defaults is used, clang, lldb and other meta packages will be automatically updated to this version.
    As a consequence and also because the branches are dead, 3.7 and 3.8 jobs have been disabled. Please note that both repositories are still available on apt.llvm.org and won't be removed.

    Zesty: New Ubuntu
    Packages for the next Ubuntu 17.04 (zesty) are also generated for 3.9, 4.0 and 5.0.

    libfuzzer
    It has been implemented a few months ago but not clearly communicated. libfuzzer has also its own packages: libfuzzer-X.Y-dev (example: libfuzzer-3.9-dev, libfuzzer-4.0-dev or libfuzzer-5.0-dev).


    Changes in the infrastructure


    In order to support the load, I started to use new blades that Google (thanks again to Nick Lewycky) sponsored for an initiative that I was running for Debian and IRILL. The 6 new blades removed all the wait time. With a new salt configuration, I automated the deployment of the slaves. In case the load increases again, we will have access to more blades.

    I also took the time to fix some long ongoing issues:
    • all repositories are signed and verified that they are    
    • i386 and amd64 packages are now uploaded at once instead of being uploaded separately. This was causing checksum error when one of the two architectures built correctly and the second was failing (ex: test failing)
    Last but not least, the code coverage results are produced in a more reliable manner.


    More information about the implementation and services.

    As what is shipped on apt.llvm.org is exactly the same as in Debian and Ubuntu, packaging files are stored on the Debian subversion server.

    A Jenkins instance is in charge of the orchestration of the whole build infrastructure.

    The trunk packages are built twice a day for every Debian and Ubuntu packages. Branches (3.9 and 4.0 currently) are rebuilt only when the - trigger job found a change.

    In both case, the Jenkins source job will checkout the Debian SVN branches for their version, checkout/update LLVM/clang/etc repositories and repack everything to create the source tarballs and Debian files (dsc, etc).The completion of job will trigger the binaries job to start. These jobs, thanks to Debian Jenkins glue will create or update Debian/Ubuntu versions.

    Then builds are done the usual way through pbuilder for both i386 and amd64. All the test suites are going to be executed. If any LLVM test is failing on i386 or amd64, the whole build will fail. If both builds and the LLVM testsuite are successful, the sync job will start and rsync packages to the LLVM server to be replicated on the CDN. If one or both builds fail, a notification is sent to the administrator.

    Some Debian static analysis (lintian) are executed on the packages to prevent some packaging errors. From time to time, some interesting issues are found.

    In parallel, some binary builds have some special hooks like Coverity, code coverage or installation of more recent versions of gcc for Ubuntu precise.

    Report bugs

    Bugs can be reported on the bugzilla of the LLVM project in the product "Packaging" and the component "deb packages".
      

    Common issues

    Because packaging quickly moving projects like LLVM or clang, in some cases, this can be challenging to follow the rhythm in particular with regard to tests. For Debian unstable or the latest version of Ubuntu, the matrix is complexified by new versions of the basic pieces of the operating system like gcc/g++ or libtstdc++.

    This is also not uncommon that some tests are being ignored in the process.

    How to help


    Some new comers bugs are available. As an example:
    Related to all this, a Google Summer of Code 2017 under the LLVM umbrella has been proposed: Integrate libc++ and OpenMP in apt.llvm.org

    Help is also needed to keep track of the new test failures and get them fixed upstream. For example, a few tests have been marked as expected to fail to avoid crashes.

    by Ledru Sylvestre (noreply@blogger.com) at March 07, 2017 06:59 PM

    February 22, 2017

    LLVM Blog

    2016 LLVM Developers' Meeting - Experience from Johannes Doerfert, Travel Grant Recipient

    This blog post is part of a series of blog posts from students who were funded by the LLVM Foundation to attend the 2016 LLVM Developers' Meeting in San Jose, CA. Please visit the LLVM Foundation's webpage for more information on our Travel Grants program.

    This post is from Johannes Doerfert:
    2016 was my third time attending the US LLVM developers meeting and for the third year in a row I was impressed by the quality of the talks, the organization and the diversity of attendees. The hands on experiences that are presented, combined with innovative ideas and cutting edge research makes it a perfect venue for me as a PhD student. The honest interest in the presented topics and the lively discussions that include students, professors and industry people are two of the many things that I experienced the strongest at these developer meetings.

    For the last two years I was mainly attending as a Polly developer that talked about new features and possible applications of Polly. This year however my roles were different. First, I was attending as part of the organization team of the European LLVM developers meeting 2017 [0] together with my colleagues Tina Jung and Simon Moll. In this capacity I answered questions about the venue (Saarbruecken, Germany [1,2]) and the alterations in contrast to prior meetings. Though, more importantly, I advertised the meeting to core developers that usually do not attend the European version. Second on my agenda was the BoF on a parallel extension to the LLVM-IR which I organized with Simon Moll. In this BoF, but also during the preparation discussion on the mailing list [3], we tried to collect motivating examples, requirements as well as known challenges for a parallel extension to LLVM. These insights will be used to draft a proposal that can be discussed in the community.

    Finally, I attended as a 4th year PhD student who is interested in contributing his work to the LLVM project (not only Polly). As my current research required a flexible polyhedral value (and iterationspace) analysis, I used the opportunity to implement one with aninterface similar to scalar evolution. The feedback I received on this topic was strictly positive. I will soon post a first version of this standalone analysis and start a public discussion. Since I hope to finish my studies at some (not too distant) point in time, I seized the opportunity to inquire about potential options for the time after my PhD.

    As a final note I would like to thank the LLVM Foundation for their student travel grant that allowed me to attend the meeting in the first place.


    [0] http://llvm.org/devmtg/2017-03/
    [1] http://sic.saarland/
    [2] https://en.wikipedia.org/wiki/Saarbr%C3%BCcken
    [3] http://lists.llvm.org/pipermail/llvm-dev/2016-October/106051.html

    by Tanya Lattner (noreply@blogger.com) at February 22, 2017 07:29 AM

    December 14, 2016

    LLVM Blog

    LLVM's New Versioning Scheme

    Historically, LLVM's major releases always added "0.1" to the version number, producing major versions like 3.8, 3.9, and 4.0 (expected by March 2017). With our next release though, we're changing this.  The LLVM version number will now increase by "1.0" with every major release, which means that the first major release after LLVM 4.0 will be LLVM 5.0 (expected September 2017).
    We believe that this approach will provide a simpler and more standard approach to versioning.
    LLVM’s version number (also shared by many of its sub-projects, such as Clang, LLD, etc.) consists of three parts: major.minor.patch. The community produces a new release every six months, with "patch" releases (also known as "dot" or "stable" releases) containing bug fixes in between.
    Until now, the six-monthly releases would cause the minor component of the version to be incremented. Every five years, after minor reached 9, a more major release would occur, including some breaking changes: 2.0 introduced the bitcode format, 3.0 a type system rewrite.
    During the discussions about what to call the release after 3.9, it was pointed out that since our releases are time-based rather than feature-based, the distinction between major and minor releases seems arbitrary. Further, every release is also API breaking, so by the principles of semantic versioning, we should be incrementing the major version number.
    We decided that going forward, every release on the six-month cycle will be a major release. Patch releases will increment the patch component as before (producing versions like 5.0.1), and the minor component will stay at zero since no minor releases will be made.

    Bitcode Compatibility

    Before LLVM 4.0.0, the Developer Policy specified that bitcode produced by LLVM would be readable by the next versions up to and including the next major release. The new version of the Developer Policy instead specifies that LLVM will currently load any bitcode produced by version 3.0 or later. When developers decide to drop support for some old bitcode feature, the policy will be updated.

    API Compatibility

    Nothing has changed. As before, patch releases are API and ABI compatible with the main releases, and the C API is "best effort" for stability, but besides that, LLVM’s API changes between releases.

    What About the Minor Version?

    Since the minor version is expected to always be zero, why not drop it and just use major.patch as the version number?
    Dropping the minor component from the middle of the version string would introduce ambiguity: whether to interpret x.y as major.minor or major.patch would then depend on the value of x.
    The version numbers are also exposed through various APIs, such as LLVM's llvm-config.h and Clang's __clang_minor__ preprocessor macro. Removing the minor component from these APIs would break a lot of existing code.
    Going forward, since the minor number will be zero and patch releases are compatible, I expect we will generally refer to versions simply by their major number and treat the rest of the version string as details (just as Chromium 55 might really be 55.0.2883.76). Future versions of LLVM and Clang can generally be referred to simply as "LLVM 4" or "Clang 5".

    by Hans Wennborg (noreply@blogger.com) at December 14, 2016 11:38 PM

    September 12, 2016

    LLVM Blog

    Announcing the next LLVM Foundation Board of Directors

    The LLVM Foundation is pleased to announce its new Board of Directors:


    Chandler Carruth
    Hal Finkel
    Arnaud de Grandmaison
    David Kipping
    Anton Korobeynikov
    Tanya Lattner
    Chris Lattner
    John Regehr


    Three new members and five continuing members were elected to the eight person board. The new board consists of individuals from corporations and from the academic and scientific communities. They also represent various geographical groups of the LLVM community. All board members are dedicated and passionate about the programs of the LLVM Foundation and growing and supporting the LLVM community.


    When voting on new board members, we took into consideration all contributions (past and present) and current involvement in the LLVM community. We also tried to create a balanced board of individuals from a wide range of backgrounds and locations to provide a voice to as many groups within the LLVM community.


    We want to thank everyone who applied as we had many strong applications. As the programs of the LLVM Foundation grow we will be relying on volunteers to help us reach success. Please join our mailing list to be informed of volunteer opportunities.


    About the board of directors (listed alphabetically by last name):


    Chandler Carruth has been an active contributor to LLVM since 2007. Over the years, he has has worked on LLVM’s memory model and atomics, Clang’s C++ support, GCC-compatible driver, initial profile-aware code layout optimization pass, pass manager, IPO infrastructure, and much more. He is the current code owner of inlining and SSA formation.


    In addition to his numerous technical contributions, Chandler has led Google’s LLVM efforts since 2010 and shepherded a number of new efforts that have positively and significantly impacted the LLVM project. These new efforts include things such as adding C++ modules to Clang, adding address and other sanitizers to Clang/LLVM, making Clang compatible with MSVC and available to the Windows C++ developer community, and much more.


    Chandler works at Google Inc. as a technical lead for their C++ developer platform and has served on the LLVM Foundation board of directors for the last 2 years.
    Hal Finkel has been an active contributor to the LLVM project since 2011. He is the code owner for the PowerPC target, alias-analysis infrastructure, loop re-roller and the basic-block vectorizer.  


    In addition to his numerous technical contributions, Hal has chaired the LLVM in HPC workshop, which is held in conjunction with Super Computing (SC), for the last 3 years. This workshop provides a venue for the presentation of peer-reviewed HPC-related researching LLVM from both industry and academia. He has also been involved in organizing an LLVM-themed BoF session at SC and LLVM socials in Austin.


    Hal is Lead for Compiler Technology and Programming Languages at Argonne National Laboratory’s Leadership Computing Facility.


    Arnaud de Grandmaison has been hacking on LLVM projects since 2008. In addition to his open source contributions, he has worked for many years on private out-of-tree LLVM-based projects at Parrot, DiBcom, or ARM. He has also been a leader in the European LLVM community by organizing the EuroLLVM Developers’ meeting, Paris socials, and chaired or participated in numerous program committees for the LLVM Developers’ Meetings and other LLVM related conferences.


    Arnaud has attended numerous LLVM Developers’ meetings and volunteered as moderator or presented as well. He also moderates several LLVM mailing lists.  Arnaud is also very involved in community wide discussions and decisions such as re-licensing and code of conduct.


    Arnaud is a Principal Engineer at ARM.


    David Kipping has been involved with the LLVM project since 2010. He has been a key organizer and supporter of many LLVM community events such as the US and European LLVM Developers’ Meetings. He has served on many of the program committees for these events.


    David has worked hard to advance the adoption of LLVM at Qualcomm and other companies. One such example of his efforts is the LLVM track he created at the 2011 Linux Collaboration summit. He has over 30 years experience in open source and developer tools including working on C++ at Borland.


    David has served on the board of directors for the last 2 years and has held the officer position of treasurer. The treasurer is a time demanding position in that he supports the day to day operation of the foundation, balancing the books, and generates monthly treasurer reports.


    David is Director of Product Management at Qualcomm and has served on the LLVM Foundation board of directors for the last 2 years


    Anton Korobeynikov has been an active contributor to the LLVM project since 2006. Over the years, he has numerous technical contributions to areas including Windows support, ELF features, debug info, exception handling, and backends such as ARM and x86. He was the original author of the MSP430 and original System Z backend.


    In addition to his technical contributions, Anton has maintained LLVM’s participation in Google Summer of Code by managing applications, deadlines, and overall organization. He also supports the LLVM infrastructure and has been on numerous program committees for the LLVM Developers’ Meetings (both US and EuroLLVM).


    Anton is currently an associate professor at the Saint Petersburg State University and has served on the LLVM Foundation board of directors for the last 2 years.


    Tanya Lattner has been involved in the LLVM project for over 14 years. She began as a graduate student who wrote her master's thesis using LLVM, and continued on using and extending LLVM technologies at various jobs during her career as a compiler engineer.   


    Tanya has been organizing the US LLVM Developers’ meeting since 2008 and attended every developer meeting. She was the LLVM release manager for 3 years, moderates the LLVM mailing lists, and helps administer the LLVM infrastructure servers, mailing lists, bugzilla, etc. Tanya has also been on the program committee for the US LLVM Developers’ meeting (4 years) and the EuroLLVM Developers’ Meeting (1 year).


    With the support of the initial board of directors, Tanya created the LLVM Foundation, defined its charitable and education mission, and worked to get 501(c)(3) status.


    Tanya is the Chief Operating Officer and has served as the President of the LLVM Foundation board for the last 2 years.


    Chris Lattner is well known as the founder for the LLVM project and has a lengthy history of technical contributions to the project over the years.  He drove much of the early implementation, architecture, and design of LLVM and Clang.


    Chris has attended every LLVM Developers’ meeting, and presented at the majority. He helped drive the conception and incorporation of the LLVM Foundation, and has served as Secretary of the board for the last 2 years. Chris also grants commit access to the LLVM Project, moderates mailing lists, moderates and edits the LLVM blog, and drives important non-technical discussions and policy decisions related to the LLVM project.


    Chris manages the Developer Tools department at Apple Inc and has served on the LLVM Foundation board of directors for the last 2 years.



    John Regehr has been involved in LLVM for a number of years. As a professor of computer science at the University of Utah, his research specializes in compiler correctness and undefined behavior. He is well known within the LLVM community for the hundreds of bug reports his group has reported to LLVM/Clang.


    John was a project lead for IOC, a Clang based integer overflow checker that eventually became the basis for the integer parts of UBSan. He was also the primary developer of C-Reduce which utilizes Clang as a library and is often used as a test case reducer for compiler issues.

    In addition to his technical contributions, John has served on several LLVM-related program committees. He also has a widely read blog about LLVM and other compiler-related issues (Embedded in Academia).

    by Tanya Lattner (noreply@blogger.com) at September 12, 2016 05:10 PM

    August 17, 2016

    OpenMP Runtime Project

    New code release

    We are excited to announce the next release of the Intel® OpenMP* Runtime Library at openmprtl.org. This release aligns with Intel® Parallel Studio XE 2017 Composer Edition

    New Features:

    • OpenMP* 4.5 nonmonotonic modifier for schedule dynamic and guided support

    Bug Fixes:

    by mad\vishakh1 at August 17, 2016 07:06 PM

    July 22, 2016

    Aaron Ballman

    Here are Some Hints for Prospective Users about How to Select the Best Virtual Data Room for their Business

    Keeping and sharing of the confidential documents on the Internet are becoming an indispensable necessity of the modern business sphere. A wide range of providers offer businessmen highly protected digital venues that have to simplify the entire course of the transaction fulfillment – so-called VDRs. Their main function is to ensure safety to virtual versions of documents. A virtual repository might be utilized not merely as a storage for secret data but also as space where numerous partners have a possibility to exchange and discuss files, transactions, and deals. In such a way, virtual platforms reduced the need to finish the entire project merely during personal negotiations. In the event that you wish to study more about data room services, please, pay your attention to this site – Ideals data room. Considering the fact that VDRs are becoming more and more popular, a lot of vendors provide their services on the market. Undoubtedly, not all the existing platforms are reliable and decent enough to rely on. To choose a reliable platform, the one is expected to pay attention to various characteristics.

    1. Reputation and experience of the VDR

    Be attentive to the reputation of the vendor. Usually, the reputation might be assessed on the basis of the comments available in media. Both – the comments of regular room visitors and the investigation by experts – are handy when it comes to choosing a virtual data room vendor. In addition, it seems to be interesting to look through the details of the transactions which were completed with the help of the software developer and, possibly, even to meet with the deal-makers from a corporation which has already exploited services of the provider. In addition, the decent vendor will be capable of offering customized services to a VDR visitor.

    2. The functions the virtual room will provide you with

    In a case of searching out a virtual room, a certain set of crucially important options must be considered. Thus, a decent room concentrates on document and access protection. The data room is excepted to possess all the specific certificates (SSAE 16 and ISO 27001), provide encryption of the data, firewalls, multi-sided user verification system, watermarks, etc. In addition, the room administrator is expected to possess all the rights to manage access to the virtual room in general and to chosen documents, folders. Regular audit reports allow to keep an eye on all the actions in the virtual repository. Together with being secure, the virtual room is expected to be convenient to utilize. That is why, simple interface that simplifies navigation in the VDR proved to be crucial. Various upload and search instruments also simplify and speed up the work in the virtual room. The mentioned options do not constitute an exhaustive list of the instruments which the user must look for in the virtual data room: the demands will depend on the customer’s needs.

    3. Expected expenditures

    If there are various provides on the market, the potential clients may choose among repositories of different price: the cost of utilization varies significantly on the vendor, on the expected time needed for execution of the deal, on the particular options demanded, etc. Hence, the deal-maker is supposed to be down-to-earth and to determine how much he is willing to pay for the virtual room.

    4. The benefits you and your stakeholders will have a chance to experience

    A platform have to be not only cheap enough and comfortable for the owner but also meet the expectations of the owner’s present or potential partners. From time to time it is advisable to choose the more luxurious virtual room considering it has numerous features required by the other side of the deal.

    5. Required set of functions

    Before make a first payment for a virtual data room, critical evaluation of the demands and expectations is supposed to be performed: not a single deal-maker wants to pay for an expensive software provided with a bunch of useless tools. The client has to make sure he is paying for what he really needs and expects – not for trendy and catchy instruments which have totally nothing in common with information security and storage.

    If following these simple guidelines in mind, the potential VDR user will have just a few doubts when selecting a VDR. Although the selection process may require a considerable amount of time, it is better to waste a bit more time and to try demo versions of diverse virtual repositories than to choose the very first virtual platform which was on sale at affordable price. You are supposed to remember the fact that you are about to pay for your protection and comfort and the virtual data room should not be the ones to save money on.

    by joe at July 22, 2016 02:19 PM

    June 21, 2016

    LLVM Blog

    ThinLTO: Scalable and Incremental LTO

    ThinLTO was first introduced at EuroLLVM in 2015, with results shown from a prototype implementation within clang and LLVM. Since then, the design was reviewed through several RFCs, it has been implemented in LLVM (for gold and libLTO), and tuning is ongoing. Results already show good performance for a number of benchmarks, with compile time close to a non-LTO build.

    This blog post covers the background, design, current status and usage information.

    This post was written by Teresa Johnson, Mehdi Amini and David Li.

    LTO Background and Motivation

    LTO (Link Time Optimization) is a method for achieving better runtime performance through whole-program analysis and cross-module optimization. During the compile phase, clang will emit LLVM bitcode  instead of an object file. The linker recognizes these bitcode files and invokes LLVM during the link to generate the final objects that will constitute the executable. The LLVM implementation loads all input bitcode files and merges them together to produce a single Module. The interprocedural analyses (IPA) as well as the interprocedural optimizations (IPO) are performed serially on this monolithic Module.



    What this means in practice is that LTO often requires a large amount of memory (to hold all IR at once) and is very slow. And with debug information enabled via -g, the size of the IR and the resulting memory requirements are significantly larger. Even without debug information, this is prohibitive for very large applications, or when compiling on memory-constrained machines. It also makes incremental builds less effective, as everything from the LTO step on must be re-executed when any input source changes.

    ThinLTO Design

    ThinLTO is a new approach that is designed to scale like a non-LTO build, while retaining most of the performance achievement of full LTO.
    In ThinLTO, the serial step is very thin and fast. This is because instead of loading the bitcode and merging a single monolithic module to perform these analyses, it utilizes compact summaries of each module for global analyses in the serial link step, as well as an index of function locations for later cross module importing. The function importing and other IPO transformations are performed later when the modules are optimized in fully parallel backends.

    The key transformation enabled by ThinLTO global analyses is function importing, in which only those functions likely to be inlined are imported into each module. This minimizes the memory overhead in each ThinLTO backend, while maximizing the most impactful cross module optimization opportunities. The IPO transformations are therefore performed on each module extended with its imported functions.

    The ThinLTO process is divided into 3 phases:
    1. Compile: Generate IR as with full LTO mode, but extended with module summaries 
    2. Thin Link: Thin linker plugin layer to combine summaries and perform global analyses 
    3. ThinLTO backend: Parallel backends with summary-based importing and optimizations 

    By default, linkers that support ThinLTO (see below) are set up to launch the ThinLTO backends in threads. So the distinction between the second and third phases is transparent to the user.

    The key enabler for this process are the summaries emitted during phase 1. These summaries are emitted using the bitcode format, but designed so that they can be separately loaded without involving an LLVMContext or any other expensive construction. Each global variable and function has an entry in the module summary. An entry contains metadata that abstracts the symbol it is describing. For example, a function is abstracted with its linkage type, the number of instructions it contains, and optional profiling information (PGO). Additionally, every reference (address taken, direct call) to another global is recorded. This information enables building a complete reference graph during the Thin Link phase, and subsequent fast analyses using the global summary information.

    Current Status

    ThinLTO is currently supported in both the gold plugin as well as in ld64 starting with Xcode 8. Additionally, support is currently being added to the lld linker. The 3.9 release of clang will have ThinLTO accessible using the -flto=thin command line option.

    While tuning is still in progress, ThinLTO already performs well compared to LTO, in many cases matching the performance improvement. In a few cases ThinLTO even outperforms full LTO, most likely because the higher scalability of ThinLTO allows using a more aggressive backend optimization pipeline (similar to that of a non-LTO build).

    The following results were collected for the C/C++ SPEC cpu2006 benchmarks on an 8-core 2.6GHz Intel Xeon E5-2689. Each benchmark was run in isolation three times and results are shown for the average of the three runs.



    Critically, due to the scalable design of ThinLTO, this performance is achieved with a build time that stays within a non-LTO build scale. The following build times were collected on a 20 core 2.8GHz Intel Xeon CPU E5-2680 v2, running Linux and using the gold linker. The results are for an end-to-end build of clang (ninja clang) from a clean build directory, so it includes all the compile steps and links of intermediate binaries such as llvm-tblgen and clang-tblgen.

    Release build shows how ThinLTO build time is very comparable to a non-LTO build. Adding -gline-tables-only adds a very small overhead, and ThinLTO is again similar to the regular non-LTO build. However with full debug information, ThinLTO is still somewhat slower than a non-LTO build due to the additional overhead during importing. Ongoing improvements to debug metadata representation and handling are expected to continue to reduce this overhead. In all cases, full LTO is actually significantly slower.

    On the memory consumption side, the improvements are significant. Over the last two years, FullLTO was significantly improved, as shown on the chart below, but our measurement shows that ThinLTO keeps a large advantage.

    Usage Information

    To utilize ThinLTO, simply add the -flto=thin option to compile and link. E.g.
        % clang -flto=thin -O2 file1.c file2.c -c
        % clang -flto=thin -O2 file1.o file2.o -o a.out

    As mentioned earlier, by default the linkers will launch the ThinLTO backend threads in parallel, passing the resulting native object files back to the linker for the final native link.  As such, the usage model the same as non- LTO. Similar to regular LTO, for Linux this requires using the gold linker configured with plugins enabled or ld64 starting with Xcode 8.

    Distributed Build Support

    To take advantage of a distributed build system, the parallel ThinLTO backends can each be launched as a separate process. To support this, the gold plugin provides a thinlto_index_only option that causes the link to exit after creating the combined index and performing global analysis.

    Additionally, in this mode:
    • Instead of using a monolithic combined index, a separate individual index file is written per backend containing the necessary portions of the combined index for recording the imports and any other global summary based optimization decisions that should be acted on in the backend. 
    • A plain text listing of the bitcode files each module will import from is optionally emitted to aid in distributed build file staging (thinlto-emit-imports-files plugin option). 

    The backends can be launched by invoking clang on the bitcode and providing its index via an option. Finally, the resulting native objects are linked to generate the final binary. For example:

        % clang -flto=thin -O2 file1.c file2.c -c
        % clang -flto=thin -O2 file1.o file2.o -Wl,-plugin-opt,-thinlto-index-only
        % clang -O2 -o file1.native.o -x ir file1.o -c -fthinlto-index=./file1.o.thinlto.bc
        % clang -O2 -o file2.native.o -x ir file2.o -c -fthinlto-index=./file2.o.thinlto.bc
        % clang file1.native.o file2.native.o -o a.out

    Incremental ThinLTO Support

    With full LTO, only the initial compile steps can be performed incrementally. If any input has changed, the expensive serial IPA/IPO step must be redone.

    With ThinLTO, the serial Thin Link step must be redone if any input has changed, however, as noted earlier this is small and fast, and does not involve loading any module. And any particular ThinLTO backend must be redone iff:

    1. The corresponding (primary) module’s bitcode changed 
    2. The list of imports into or exports from the module changed 
    3. The bitcode for any module being imported from has changed 
    4. Any global analysis result affecting either the primary module or anything it imports has changed. 

    For single machine builds, where the threads are launched by the linker, incremental builds can be achieved by caching the module after applying the global summary based optimizations such as importing, using a hash of the information listed above as the key. This caching is already supported in libLTO’s ThinLTO handling, which is used by ld64. To enable it, the link step needs to be passed an extra flag: -Wl,-cache_path_lto,/path/to/cache

    For distributed builds, the above information in items 2-4 are all serialized into the individual index files. So the build system can compare the contents of the input bitcode files (the primary module’s bitcode and any it imports from) along with the combined index against those from an earlier build to decide if a particular ThinLTO backend must be redone. To make this process more efficient, the content of the bitcode file is hashed when emitted during the compile phase, and the result is stored in the bitcode file itself so that the cache can be queried during the Thin Link step without reading the IR.

    The chart below illustrates the full build time of clang in three different situations:
    1. The full link following a clean build.
    2. The developer fixes the implementation of DenseMap::grow(). This is a widely used header in the project, which forces to rebuild a large number of files.
    3. The developer fixes the implementation of visitCallInst() in InstCombineCalls.cpp. This an implementation file and incremental build should be fast.

    These results illustrate how full LTO is not friendly with incremental build, and show how ThinLTO is providing an incremental link-time very close to a non-LTO build.

    by Teresa Johnson (noreply@blogger.com) at June 21, 2016 05:24 PM

    June 20, 2016

    LLVM Blog

    LLVM Weekly - #129, Jun 20th 2016

    Welcome to the one hundred and twenty-ninth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Last week was WWDC, which featured talks on what's new in LLVM (slides) and what's new in Swift (slides). Note that the embedded video player suggests you need Safari or the WWDC app to stream the video, but you can find a downloadable version under the "resources" tab.

    On the mailing lists

    LLVM commits

    • FileCheck learnt the --check-prefixes option as a shorthand for multiple --check-prefix options. r272670.

    • A local_unnamed_addr attribute was introduced. This can be used by the code generator and LTO to allow the linker to decide whether the global needs to be in the symbol table. r272709.

    • The ScalarReplAggregates pass has been removed as it has been superseded by SROA by a long time. r272737.

    • LLVM's C API gained support for string attributes. r272811.

    • Assembly parsing and lexing has seem some cleanups. r273007.

    Clang commits

    • A new loop distribution pragma was added. Loop distribution is a transformation which attempts to break a loop in to multiple loops with each taking part of the loop body. r272656.

    • The nodebug attribute can now be applied to local variables. r272859.

    • The validity check for MIPS CPU/ABI pairings is now performed at initialisation time and a much clearer message is printed. r272645.

    Other project commits

    • A complete implementation of the C++ Filesystem TS has been checked in. r273034.

    • LLD's ARM port gained initial support for Thumb with ARMv7a. r272881.

    by Alex Bradbury (noreply@blogger.com) at June 20, 2016 11:23 AM

    June 16, 2016

    LLVM Blog

    Using LNT to Track Performance


    In the past year, LNT has grown a number of new features that makes performance tracking and understanding the root causes of performance deltas a lot easier. In this post, I’m showing how we’re using these features.

    LNT contains 2 big pieces of functionality:
    1. A server,
      a. to which you can submit correctness and performance measurement data, by sending it a json-file in the correct format,
      b. that analyzes which performance changes are significant and which ones aren't,
      c. that has a webui to show results and analyses in a number of different ways.
    2. A command line tool to run tests and benchmarks, such as LLVM’s test-suite, SPEC2000 and SPEC2006 benchmarks.
    This post focuses on using the server. None of the features I’ll show are LLVM-specific, or even specific to ahead-of-time code generators, so you should be able to use LNT in the same way for all your code performance tracking needs. At the end, I’ll give pointers to the documentation needed to setup an LNT server and how to construct the json file format with benchmarking and profiling data to be submitted to the server.
    The features highlighted focus on tracking the performance of code, not on other aspects LNT can track and analyze.
    We have 2 main uses cases in tracking performance:
    • Post-commit detection of performance regressions and improvements.
    • Pre-commit analysis of the impact of a patch on performance.
    I'll focus on the post-commit detection use case.

    Post-commit performance tracking

    Step 1. Get an overview of the "Daily Report" page

    Assuming your server runs at http://yourlntserver:8000, this page is located at http://yourlntserver:8000/db_default/v4/nts/daily_report
    The page gives a summary of the significant changes it found today.
    An example of the kind of view you can get on that page is the following
    In the above screenshot, you can see that there were performance differences on 3 different programs, bigfib, fasta and ffbench. The improvement on ffbench only shows up on a machine named “machine3”, whereas the performance regression on the other 2 programs shows up on multiple machines.

    The table shows how performance evolved over the past 7 days, one column for each day. The sparkline on the right shows graphically how performance has evolved over those days. When the program was run multiple times to get multiple sample points, these show as separate dots that are vertically aligned (because they happened on the same date). The background color in the sparkline represents a hash of the program binary. If the color is the same on multiple days, the binaries were identical on those days.

    Let’s look first at the ffbench program. The background color in the sparkline is the same for the last 2 days, so the binary for this program didn’t change in those 2 days. Conclusion: the reported performance variation of -8.23% is caused by noise on the machine, not due to a change in code. The vertically spread out dots also indicate that this program has been noisy consistently over the past 7 days.

    Let’s now look at the bigfib. The background color in the sparkline has changed since its previous run, so let’s investigate further. By clicking on one of the machine names in the table, we go to a chart showing the long-term evolution of the performance of this program on that machine.

    Step 2. The long-term performance evolution chart

    This view shows how performance has evolved for this program since we started measuring it. When you click on one of the dots, which each represent a single execution of the program, you get a pop-up with information such as revision, date at which this was run etc.
    When you click on the number after “Run:” in that pop-up, it’ll bring you to the run page.

    Step 3. The Run page

    The run page gives an overview of a full “Run” on a given machine. Exactly what a Run contains depends a bit on how you organize the data, but typically it consists of many programs being run a few times on 1 machine, representing the quality of the code generated by a specific revision of the compiler on one machine, for one optimization level.
    This run page shows a lot of information, including performance changes seen since the previous run:
    When hovering with the mouse over entries, a “Profile” button will show, that when clicked, shows profiles of both the previous run and the current run.

    Step 4. The Profile page

    At the top, the page gives you an overview of differences of recorded performance events between the current and previous run.
    After selecting which function you want to compare, this page shows you the annotated assembly:


    While it’s clear that there are differences between the disassembly, it’s often much easier to understand the differences by reconstructing the control flow graph to get a per-basic-block view of differences. By clicking on the “View:” drop-down box and selecting the assembly language you see, you can get a CFG view. I find showing absolute values rather than relative values helps to understand performance differences better, so I also chose “Absolute numbers” in the drop down box on the far right:
    There is obviously a single hot basic block, and there are differences in instructions in the 2 versions. The number in the red side-bar shows that the number of cycles spent in this basic block has increased from 431M to 716M. In just a few clicks, I managed to drill down to the key codegen change that caused the performance difference!

    We combine the above workflow with the llvmbisect tool available at http://llvm.org/viewvc/llvm-project/zorg/trunk/llvmbisect/ to also quickly find the commit introducing the performance difference. We find that using both the above LNT workflow and the llvmbisect tool are vital to be able to act quickly on performance deltas.

    Pointers on setting up your own LNT server for tracking performance

    Setting up an LNT server is as simple as running the half a dozen commands documented at http://lnt.llvm.org/quickstart.html under "Installation" and "Viewing Results". The "Running tests" section is specific to LLVM tests, the rest is generic to performance tracking of general software.

    The documentation for the json file format to submit results to the LNT server is here: http://lnt.llvm.org/importing_data.html.
    The documentation for how to also add profile information, is at http://lnt.llvm.org/profiles.html.


    by Kristof Beyls (noreply@blogger.com) at June 16, 2016 06:19 AM

    June 13, 2016

    LLVM Blog

    LLVM Weekly - #128, June 13th 2016

    Welcome to the one hundred and twenty-eighth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    LDC, a compiler for the D programming language with an LLVM backends has a major release with 1.0.0. The big news with this release is that the frontend is now completely written in D. Congratulations to everyone involved in this release. See the D website for more information about the D programming language.

    The minor release LLVM 3.8.1-rc1 has been tagged.

    On the mailing lists

    LLVM commits

    • Some of the work from the GSoC project on interprocedural register allocation has started to land. A RegUsageInfoCollector analysis was added that collects the list of clobbered registers for a MachineFunction. A new transformation pass was committed which scans the body of a function to find calls and updates the register mask with the one saved by RegUsageInfoCollector. r272403, r272414.

    • Chapter 2 of the tutorial on building a JIT with ORC has been fleshed out with a rough draft of the text. r271885.

    • The host CPU detection code for x86 has seen a large refactoring. r271921.

    • More documentation has been added about LLVM's CodeView support. r272057.

    • llvm-symbolizer will now be searched for in the same directory as the LLVM or Clang tool being executed. This increases the chance of being able to print pretty backtraces for systems where LLVM tools aren't installed in the $PATH. r272232.

    Clang commits

    • Clang analyzer gained a checker for correct usage of the MPI API in C and C++. r271907.

    • Documentation was added on avoiding static initializers when using profiling. r272067, r272214.

    Other project commits

    • A hardened allocator, 'scudo' was added to compiler-rt. It attempts to mitigate some common heap-based vulnerabilities. r271968.

    • Initial support for ARM has landed in LLD. This is just enough to link a hello world on ARM Linux. r271993.

    • Initial support for AddressSanitizer on Win64 was added. r271915.

    by Alex Bradbury (noreply@blogger.com) at June 13, 2016 12:08 PM

    June 06, 2016

    LLVM Blog

    LLVM Weekly - #127, June 6th 2016

    Welcome to the one hundred and twenty-seventh issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Graham Markall at Embecosm has been comparing the code size of RISC-V binaries produced by the GCC and LLVM ports, as well as compared to ARM. GCC is currently ahead, though it is worth noting the LLVM port has seen much less attention.

    Matthias Reisinger is a Google Summer of Code student working on enabling polyhedral optimisations for the Julia programming language. He's written a blog post detailing his initial steps and immediate future plans. Hopefully we'll see more posts over the summer.

    Loïc Hamot has been working on a C++ to D converter, implemented using Clang.

    The MSVC team have blogged about the latest release of Clang with Microsoft CodeGen, based on Clang 3.8.

    There is going to be a clang-tidy code dojo in Warsaw on Tuesday the 7th of June.

    On the mailing lists

    LLVM commits

    • LLVM gained support for 'SJLJ' (setjmp/longjmp) exception handling on x86 targets. r271244.

    • LLVM now requires CMake 3.4.3 to build r271325.

    • Support was added for attaching metadata to global variables. r271348.

    • The AArch64 backend switched to use SubtargetFeatures rather than testing for specific CPUs. r271555.

    Clang commits

    • The release notes have been updated to explain the current level of OpenMP support (full support of non-offloading features of OpenMP 4.5). r271263.

    • Clang's source-based code coverage has been documented. r271454.

    Other project commits

    • An -fno-exceptions libc++abi library variant was defined, to match the -fno-exceptions libc++ build. r271267.

    • LLDB's compact unwind printing tool gained support for ARMv7's compact unwind format. r271744.

    by Alex Bradbury (noreply@blogger.com) at June 06, 2016 03:02 PM

    May 30, 2016

    LLVM Blog

    LLVM Weekly - #126, May 30th 2016

    Welcome to the one hundred and twenty-sixth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    I've been moving house this weekend, so do accept my apologies if you find this issue to be a little less thorough than usual.

    News and articles from around the web

    Pyston, the LLVM-based Python compiler has released version 0.5. The main changes are a switch to reference counting and NumPy compatibility.

    I don't want to become "C++ weekly", but I think this audience appreciates a fun use of C++ features. Verdigris is a header-only library that allows you to use Qt5 without the moc preprocessor.

    The call for papers for the 3rd workshop on the LLVM compiler infrastructure in HPC has been published. The deadline for paper submission is September 1st. The workshop will take place on November 14th in Salt Lake City, and is held in conjunction with SC16.

    On the mailing lists

    • Vivek Pandya, a GSoC student working on interprocedural register allocation has shared a weekly status report.

    • Rafael Espíndola has proposed creating a bitcode symbol table.

    • There's been some updates on the progress of open-sourcing PGI's Fortran frontend.

    • Elena Lepilkina has proposed some enhancement to FileCheck. Some questions were raised about how useful the proposed extensions will be. Sergey Yakoushkin provided more background on how these features are used in a commercial codebase. As Elena notes, these features don't need to all be upstreamed at once (or at all), and are mostly independent.

    • Lang Hames has posted a heads-up about upcoming breaking API changes for ORC and MCJIT.

    • Sean Silva has kicked off a discussion on the state of IRPGO. You might ask what is IRPGO? This is profile-guided optimisation performed through instrumentation at the LLVM IR level, as opposed to FEPGO where instrumentation is added by the frontend (e.g. Clang), prior to lowering to IR. Sean would like to make IRPGO the default on all platforms other than Apple at the moment (who may require a longer deprecation period). A number of followup comments discuss possibilities for ensuring all platforms can move forward together, and ensuring a sensible flag exists to choose between frontend or middle-end PGO.

    • What exactly is a register pressure set? Both Quentin Colombet and Andrew Trick have answers for us.

    LLVM commits

    • New optimisations covering checked arithmetic were added. r271152, r271153.

    • Advanced unrolling analysis is now enabled by default. r270478.

    • The initial version of a new chapter to the 'Kaleidoscope' tutorial has been committed. This describes how to build a JIT using ORC. r270487, r271054.

    • LLVM's stack colouring analysis data flow analysis has been rewritten in order to increase the number of stack variables that can be overlapped. r270559.

    • Parts of EfficiencySanitizer are starting to land, notably instrumentation for its working set tool. r270640.

    • SelectionDAG learned how to expand multiplication for larger integer types where there isn't a standard runtime call to handle it. r270720.

    • LLVM will now report more accurate loop locations in optimisation remarks by reading the starting location from llvm.loop metadata. r270771.

    • Symbolic expressions are now supported in assembly directives, matching the behaviour of the GNU assembler. r271102.

    • Symbols used by plugins can now be auto-exported on Windows, which improves support for plugins in Windows. See the commit message for a full description. r270839.

    Clang commits

    • Software floating point for Sparc has been exposed in Clang through -msoft-float. r270538.

    • Clang now supports the -finline-functions argument to enable inlining separately from the standard -O flags. r270609.

    Other project commits

    • SectionPiece in LLD is now 8-bytes smaller on 64-bit platforms. This improves the time to link Clang with debug info by 2%. r270717.

    • LLD has replaced a use of binary search with a hash table lookup, resulting in a 4% speedup when linking Clang with debug info. r270999.

    • LLDB now supports AArch64 compact unwind tables, as used on iOS, tvos and watchos. r270658.

    by Alex Bradbury (noreply@blogger.com) at May 30, 2016 07:07 PM

    May 23, 2016

    LLVM Blog

    LLVM Weekly - #125, May 23rd 2016

    Welcome to the one hundred and twenty-fifth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Stephen Kelly has written a blog post about using Clang through the cindex API to automatically generate Python bindings. He also makes use of SIP.

    Krister Walfridsson has written a wonderfully clear post on C's type-based aliasing rules.

    This week I discovered the Swift Weekly Brief newsletter. Its author, Jesse Squires does a wonderful job of summarising mailing list traffic, recent commits, and discussions on swift-evolution proposals. If you have an interest in Swift development or language design in general I highly recommend it.

    Are you interested in writing for the LLVM blog? Or volunteering to help recruit content authors? If so, get in touch with Tanya.

    The next Cambridge LLVM Social will be held at 7.30pm on May 25th at the Cambridge Blue.

    On the mailing lists

    LLVM commits

    • llc will now report all errors in the input file rather than just exiting after the first. r269655.

    • The SPARC backend gained support for soft floating point. r269892.

    • Reloc::Default no longer exists. Instead, Optional<Reloc> is used. r269988.

    • An initial implementation of a "guard widening" pass has been committed. This will combine multiple guards to reduce the number of checks at runtime. r269997.

    Clang commits

    • clang-include-fixer gained a basic Vim integration. r269927.

    • The intrinsics headers now have feature guards enabled in Microsoft mode to combat the compile-time regression discussed last week due to their increased size. r269675.

    • avxintrin.h gained many new Doxygen comments. r269718.

    Other project commits

    • lld now lets you specify a subset of passes to run in LTO. r269605.

    • LLDB has replaced uses of its own Mutex class with std::mutex. r269877, r270024.

    by Alex Bradbury (noreply@blogger.com) at May 23, 2016 11:41 AM

    May 16, 2016

    LLVM Blog

    LLVM Weekly - #124, May 16th 2016

    Welcome to the one hundred and twenty-fourth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    The main news this week is the announcement of Scala-native, an ahead-of-time compiler for Scala using LLVM. Jos Dirkens has written a getting started guide if you want to compile it and try it out. There's also more information in the slides from the announcement talk.

    On the mailing lists

    LLVM commits

    • The outdated guide on cross-compiling LLVM has been brought up to date. r269054.

    • The WebAssembly backend gained preliminary fast instruction selection (fast-isel) support. r269083, r269203, r269273.

    • Loop unrolling (other than in the case of explicit pragmas) is now disabled at -Os in LLVM. You may recall last week it was enabled for -Os in Clang, but with different thresholds. r269124.

    • A new cost-tracking system has been implemented for the loop unroller. r269388.

    • LLVM's Sparc backend has seen the addition of more LEON-specific features, e.g. signed and unsigned multiply-accumulate. r268908.

    • llc's -run-pass option will now work with any pass known to the pass registry. Previously it would silently do nothing if you specify indirectly added analysis passes or passes not present in the optimisation pipeline. r269003.

    • WebAssembly register stackification and coloring are now run very late in the optimisation pipeline. The commit message suggests it's useful to think of these passes as domain-specific liveness-based compression rather than a conventional optimisation. r269012.

    • When declaring global in textual LLVM IR, you must now assign them with e.g. @0 = global i32 42. r269096.

    • The internal assembler is now enabled by default for 32-bit MIPS targets. r269560.

    Clang commits

    • Clang now supports __float128. r268898.

    • Clang gained a new warning that triggers when casting away calling conventions from a function. r269116.

    • The recently developed include-fixer tools now has documentation. r269167.

    Other project commits

    • compiler-rt's CMake build system can now build builtins without a full toolchain, allowing you to bootstrap a cross-compiler. r268977.

    • LLD will now sort relocations to optimise dynamic linker performance. r269066.

    by Alex Bradbury (noreply@blogger.com) at May 16, 2016 11:26 AM

    May 09, 2016

    LLVM Blog

    LLVM Weekly - #123, May 9th 2016

    Welcome to the one hundred and twenty-third issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    If you're in London tomorrow you may be interested in the NMI Open Source Conference. You can register until midday today. I'll be giving a brief talk on lowRISC. While on the subject of conferences, if you are interested in diversity and inclusion in computing education, you may want to check out the CAS #include diversity conference in Manchester on the 11th June.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Fabien Giesen has written a brief article explaining why compilers exploit undefined signed overflow.

    The Google Open Source blog has a short piece on the XRay function call tracing system that was proposed for upstreaming last week on the LLVM mailing list.

    On the mailing lists

    LLVM commits

    • LLVM's CppBackend has been removed. As the commit message says, this backend has bit-rotted to the extent that it's not useful for its original purpose and doesn't generate code that compiles. r268631.

    • The AVR backend has seen a large amount of code merged in to LLVM. r268722.

    • The MIPS backend has seen some large changes to how relocations are handled. These are now represented using MipsMCExpr instead of MCSymbolRefExpr. As someone who has done quite a lot of (out-of-tree) LLVM backend work, I've always found it odd how some architectures have globally visible enum members in include/llvm/MC/MCExpr.h. r268379.

    • LLVM builds should hopefully now be deterministic by default, as LLVM_ENABLE_TIMESTAMPS is now opt-in rather than opt-out. In fact, a follow-up patch removed the option altogether. r268441, r268670.

    • The AARch64 backend learned to combine adjustments to the stack pointer for callee-save stack memory and local stack memory. r268746.

    Clang commits

    • Clang now supports -malign-double for x86. This matches the default behaviour on x86-64, where i64 and f64 types are aligned to 8-bytes instead of 4. r268473.

    • Loop unrolling is no longer completely disabled for -Os. r268509.

    • Clang's release notes (reflecting the state of current trunk) have been updated to say more about the state of C++1z support. r268663.

    Other project commits

    • libcxx will now build a libc++experimental.a static library to hold symbols from the experimental C++ Technical Specifications (e.g. filesystem). This library provides no ABI compatibility. r268443, r268456.

    • All usage of pthreads in libcxx has been refactored in to the __threading_support header, with the intention of making it easier to retarget libcxx to platform that don't support pthreads. r268374.

    • libcxx gained support for the polymorphic memory resources C++ TS. r268829.

    by Alex Bradbury (noreply@blogger.com) at May 09, 2016 08:28 AM

    May 02, 2016

    LLVM Blog

    LLVM Weekly - #122, May 2nd 2016

    Welcome to the one hundred and twenty-second issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    GCC 6.1 has been released. Perhaps the most apparent user-visible change is that the C++ frontend now defaults to C++14.

    The Rust compiler has introduced a new intermediate representation, MIR, used for optimisations prior to lowering to LLVM IR.

    Tanya Lattner has written about the LLVM Foundation's plans for 2016. The LLVM Foundation has established 3 main programs: Educational Outreach, Grants and Scholarships, and Women in Compilers and Tools.

    On the mailing lists

    LLVM commits

    • LLVM now supports indirect call promotion based on value-profile information. This will promote indirect calls to a direct call guarded by a precondition. r267815.

    • The LLVM documentation has been extended with a CMake primer covering the basics of the CMake scripting language. r268096.

    • The PDB dumper has been refactored into a library. r267431.

    • The MinLatency attributed has been removed from SchedMachineModel. r267502.

    • CodeGenPrepare will now use branch weight metadata to decide if a select should be turned into a branch. r267572.

    • Support for llvm.loop.distribute.enable metadata was added. This indicates a loop should be split in to multiple loops. r267672.

    • The SystemZ backend now supports the Swift calling convention. r267823.

    • libFuzzer's documentation has been expanded and improved. r267892.

    Clang commits

    • clang-tidy gained a new checker for redundant expressions on both sides of a binary operator. r267574.

    • A new clang-tidy check will warn for use of functions like atoi and atol that don't report conversion errors. r268100.

    • The nodebug attribute on a global or static variable will now suppress all debug info for that variable. r267746.

    • A number of OpenMP features gained codegen support, such as the map clause and target data directive. r267808, r267811.

    Other project commits

    • LLD now supports an -O0 option to produce output as quickly as possible. Currently this disables section merging at the cost of a potentially much larger output. r268056.

    • The symbol table in LLD's ELF linker has been redesigned with the intent of improving memory locality. The new design produces measurable speedups for the binaries tested in the commit message. r268178.

    • LLD's linkerscript support expanded to encompass comparison operators. r267832.

    • LLD performance on large executables has been improved by skipping scanRelocs on sections that are never mapped to memory at runtime (e.g. debug sections). r267917.

    by Alex Bradbury (noreply@blogger.com) at May 02, 2016 03:31 PM

    April 27, 2016

    LLVM Blog

    LLVM Foundation 2016 Announcements

    With 2016 upon us, the LLVM Foundation would like to announce our plans for the year. If you are not familiar with the LLVM Foundation, we are a 501(c)(3) nonprofit that supports the LLVM Project and its community. We are best known for our LLVM Developers’ Meetings, but we are introducing several new programs this year. 

    The LLVM Foundation originally grew out of the need to have a legal entity to plan and support the annual LLVM Developers’ Meeting and LLVM infrastructure. However, as the Foundation was created we saw a need for help in other areas related to the LLVM project, compilers, and tools. The LLVM Foundation has established 3 main programs: Educational Outreach, Grants & Scholarships, and Women in Compilers & Tools.

    Educational Outreach 

    The LLVM Foundation plans to expand its educational materials and events related to the LLVM Project and compiler technology and tools. 

    First, the LLVM Foundation is excited to announce the 2016 Bay Area LLVM Developers’ Meeting will be held November 3-4 in San Jose, CA. This year will be the 10th anniversary of the developer meeting which brings together developers of LLVM, Clang, and related projects. For this year’s meeting, we are increasing our registration cap to 400 in order to allow more community members to attend.

    We also are investigating how we can support or be involved in other conferences in the field of compilers and tools. This may include things such as LLVM workshops or tutorials by sponsoring presenters, or providing instructional materials. We plan to work with other conference organizers to determine how the LLVM Foundation can be helpful and develop a plan going forward.

    However, we want to do more for the community and have brainstormed some ideas for the coming year. We plan to create some instructional videos for those just beginning with LLVM. These will be short 5-10 minute videos that introduce developers to the project and get them started. Documentation is always important, but we find that many are turning to videos as a way to learn. 


    Grants & Scholarships

    We are creating a grants and scholarships program to cover student presenter travel expenses to the LLVM Developers’ Meetings. However, we also hope to expand this program to include student presenter travel to other conferences where the student is presenting their LLVM related work. Details on this program will be published once they have been finalized. 

    Women in Compilers & Tools

    Grace Hopper invented the first compiler and yet women are severely underrepresented in the field of compilers and tools. At the 2015 Bay Area LLVM Developers’ Meeting, we held a BoF on this topic and brainstormed ideas about what can be done. One idea was to increase LLVM awareness at technical conferences that have strong female participation. One such conference is the Grace Hopper Conference (GHC). The LLVM Foundation has submitted a proposal to present about LLVM and how to get involved with the LLVM open source community. We hope our submission is accepted, but if not, we are exploring other ways we can increase our visibility at GHC. Many of the other ideas from this BoF are being considered and actionable plans are in progress.

    In addition, to these 3 programs, we will continue to support the LLVM Project’s infrastructure. The llvm.org server will move to a new machine to increase performance and reliability.  


    We hope that you are excited about the work the LLVM Foundation will be doing in 2016. Our 2016 Plans & Budget may be viewed here. You may also contact our COO & President, Tanya Lattner (tanyalattner@llvm.org) or the LLVM Foundation Board of Directors (board@llvm.org).

    by Tanya Lattner (noreply@blogger.com) at April 27, 2016 03:48 PM

    April 25, 2016

    LLVM Blog

    LLVM Weekly - #121, Apr 25th 2016

    Welcome to the one hundred and twenty-first issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Congratulations to the eight students who have been selected for LLVM projects on Google Summer of Code this year. There's about a month before they start coding. The time between now and then is the 'community bonding period', so please do make them feel welcome.

    The preliminary release schedule for LLVM/Clang 3.8.1 has been published. This would have a deadline of May 25th for requesting changes to be merged and would see the final release on June 15th.

    On the mailing lists

    LLVM commits

    • An implementation of optimisation bisection support has landed. This helps to track down bugs by allowing optimisations to be selectively disabled at compile-time to identify the one introducing a miscompile. r267022.

    • The AArch64 and ARM thread pointer intrinsics have been merged to make a target-independent llvm.thread.pointer intrinsic. r266818.

    • The llvm.load.relative intrinsic has been added. r267233.

    • There have been more changes to DebugInfo which will require a bitcode upgrade. A script to perform this upgrade is linked in the commit message. r27296.

    • The ORC JIT API improved its support for RPC, including support for calling functions with return values. r266581.

    • The patchable-function function attribution has been introduced, indicating that the function should be easily patchable at runtime. r266715.

    • The IntrReadArgMem intrinsic property has been split in to IntrReadMem and IntrArgMemOnly. r267021.

    • The MachineCombiner gained the ability to combine AArch64 fmul and fadd in to an fmadd. r267328.

    • Scheduling itineraries were added for Sparc, specifically for the LEON processors. r267121.

    Clang commits

    • A prototype of an include fixing tool was created. The indexer remains to be written. r266870.

    • A new warning has been added, which will trigger if the compiler tries to make an implicit instantiation of a template but cannot find the template definition. r266719.

    • Initial driver flags for EfficiencySanitizer were added. r267059.

    Other project commits

    • The initial EfficiencySanitizer base runtime library was added to compiler-rt. It doesn't do much of anything yet. r267060.

    • LLD learned to support the linkerscript ALIGN command. r267145.

    • LLDB can now parse EABI attributes for an ELF input. r267291.

    by Alex Bradbury (noreply@blogger.com) at April 25, 2016 11:20 AM

    April 18, 2016

    LLVM Blog

    LLVM Weekly - #120, Apr 18th 2016

    Welcome to the one hundred and twentieth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    This week has seen not one, but two articles about LLVM and profile-guided optimisation. Dig in John Engelen's article about optimising D's virtual function calls with PGO, then read Geoffroy Couprie's article about PGO with Rust.

    The next Cambridge (UK) social will be at 7.30pm on April 20th, at the Cambridge Blue.

    Alex Denisov has written a blog post around the idea of building a mutation testing system using LLVM.

    On the mailing lists

    LLVM commits

    • AtomicExpandPass learned to lower various atomic operations to __atomic_* library calls. The eventual aim is to move all atomic lowering from Clang to LLVM. r266115.

    • Targets can now define an inlining threshold multiplier, to e.g. increase the likelihood of inlining on platforms where calls are very expensive. r266405.

    • The ownership between DICompileUnit and DISubprogram has been reversed. This may break tests for your out-of-tree backend, but the commit has a link to a Python script to update your testcases. r266446.

    • llvm-readobj learned to print a histogram of an input ELF file's .gnu.hash . r265967.

    • More target-specific support for the Swift calling convention (on ARM, AARch64, and X86) has landed. Also, a callee save register is used for the swiftself parameter. r265997, r266251.

    • A new allocsize attribute has been introduced. This indicates the given function is an allocation function. r266032.

    • analyzeSiblingValues has been replaced with a new lower-complexity implementation in order to reduce compile times. r266162.

    • The AMDGPU backend gained a skeleton GlobalISel implementation. r266356.

    • Every use of getGlobalContext other than the C API has been removed. r266379.

    Clang commits

    • Clang gained support for the GCC ifunc attribute. r265917.

    • The __unaligned type qualifier was implemented for MSVC compatibility. r266415.

    • Support for C++ core guideline Type.6: always initialize a member variable was completed in clang-tidy. r266191.

    • A new clang-tidy checker for suspicious sizeof expressions was added. r266451.

    Other project commits

    • The way relocations are applied in the new ELF linker has been reworked. r266158.

    • ELF LLD now supports parallel codegen for LTO using splitCodeGen. r266484.

    • Support for Linux on SystemZ in LLDB landed. r266308.

    by Alex Bradbury (noreply@blogger.com) at April 18, 2016 01:05 PM

    April 11, 2016

    LLVM Blog

    LLVM Weekly - #119, Apr 11th 2016

    Welcome to the one hundred and nineteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Last week the slides from the recent EuroLLVM 2016 Developers' Meeting made it online. This week this has been followed by videos of the talks from the conference.

    John Regehr has written about efficient integer overflow checking in LLVM, looking at cases where LLVM can and cannot remove unnecessary overflow checks, and how this might be improved.

    Version 0.13 of Pocl, the portable OpenCL implementation has been released. This release works with LLVM/Clang 3.8 and 3.7, and adds initial OpenCL 2.0 support and improved HSA support.

    Serge Guelton at QuarksLab has written up a really useful guide to implementing a custom directive handler in Clang.

    Microsoft's Visual C++ team are looking for feedback on Clang/C2 (Clang with Microsoft CodeGen).

    On the mailing lists

    • James Molloy has posted an RFC on adding support for constant folding calls to math.h functions on long doubles. Currently these functions aren't constant-folded as the internal APFloat class doesn't implement them and long double operations aren't portable. Solutions include adding support to APFloat, linking against libMPFR to provide compile-time evaluation, or recognising when the long double format of the host and target are the same, so the host math library can be called. From the responses so far, there seems to be some push-back on adding the libMPFR dependency.

    • Sanjoy Das has an RFC on adding a patchable-prologue attribute. This would be used to indicate that the function's prologue is compiled so as to provide support for easy hot-patching.

    • Ulrich Weigand has shared a patch for supporting LLDB on Linux on SystemZ. The patchset contains many big-endian fixes, and may be of interest to others looking at porting LLDB.

    LLVM commits

    • The Swift calling convention as well as support for the 'swifterror' argument has been added. r265433, r265480.

    • Work on GlobalISel continues with many commits related to the assignment of virtual registers to register banks. r265445, r265440.

    • LLVM will no longer perform inter-procedural optimisation over functions that can be "de-refined". r265762.

    • The substitutions supported by lit are now documented. r265314.

    • Unrolled loops now execute the remainder in an epilogue rather than the prologue. This should produce slightly improved code. r265388.

    Clang commits

    • Clang gained necessary support for the Swift calling convention. r265324.

    • New flags -fno-jump-tables and -fjump-tables can be used to disable/enable support for jump tables when lowering switch statements. r265425.

    • TargetOptions is now passed through all the TargetInfo constructors. This will allow target information to be modified based on the ABI selected. r265640.

    • A large number of intrinsics from emmintrin.h now have Doxygen docs. r265844.

    Other project commits

    • clang-tidy gained a new check to flag initializers of globals that access extern objects, leading to potential order-of-initialization issues. r265774.

    • LLD's ELF linker gained new options --start-lib, --end-lib, --no-gnu-unique, --strip-debug. r265710, r265717, r265722.

    by Alex Bradbury (noreply@blogger.com) at April 11, 2016 01:03 PM

    April 04, 2016

    LLVM Blog

    LLVM Weekly - #118, Apr 4th 2016

    Welcome to the one hundred and eighteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Almost all slides from the recent EuroLLVM conference are now available online for your enjoyment.

    Some readers my be interested in a new paper about the 'LifeJacket' tool for verifying precise floating-point optimisations in LLVM.

    Christian Neumüller has written a new tool for syntax highlighting and cross-referencing C and C++ source using libclang.

    On the mailing lists

    LLVM commits

    • The Lanai backend has landed. r264578.

    • A new llvm.experimental.guard intrinsic has been added. As described in the accompanying documentation, along with deoptimization operand bundles this allows frontends to express guards or checks on optimistic assumptions made during compilation. r264976.

    • Support for a number of new Altivec instructions has been added. Amazingly, this includes BCD (Binary Coded Decimal) instructions. r264568.

    • The concept of MachineFunctionProperties has been introduced, with the first property being AllVRegsAllocated. This allows passes to declare that they require a particular property, in this case requiring that they be run after regalloc. r264593.

    • On X86, push will now be used in preference to mov at all optimisation levels (before it was only enabled for -Os). r264966.

    • LLVM's support library can now compute SHA1 hashes. This is used to implement a 'build-id'. r265094, r265095.

    • When metadata is only referenced in a single function, it will now be emitted just in that function block. The aim of this is to improve the potential of lazy-loading. r265226.

    Clang commits

    • The Lanai backend is now supported in the Clang driver. r264655.

    • libTooling gained a handy formatAndApplyAllReplacements function. r264745.

    Other project commits

    • Parts of LLD are starting to use the new Error handling. r264910, r264921, r264924, and more.

    • Infrastructure was added to LLD for generating thunks (as required on platforms like MIPS when calling PIC code from non-PIC). r265059.

    by Alex Bradbury (noreply@blogger.com) at April 04, 2016 11:22 AM

    April 01, 2016

    LLVM Blog

    My Little LLVM: Undefined Behavior is Magic!

    A horrible mashup between LLVM's old dragon logo and a My Little Pony inspired pegasus pony
    New LLVM logo

    There’s been lots of discussion online (and then quite some more) about compilers abusing undefined behavior. As a response the LLVM compiler infrastructure is rebranding and adopting a motto to make undefined behavior friendlier and less prone to corruption.


    The re-branding puts to rest a long-standing issue with LLVM’s “dragon” logo actually being a wyvern with an upside-down head, a special form of undefined behavior in its own right. The logo is now clearly a pegasus pony.


    Another great side-effect of this rebranding is increased security by auto-magically closing all vulnerabilities used by the hacker who goes by the pseudonym “Pinkie Pie”.


    These new features are enabled with the -rainbow clang option, in honor of Rainbow Dash’s unary name.


    A Few Examples


    C++’s memory model specifies that data races are undefined behavior. It is well established that no sane compiler would optimize atomics, LLVM will therefore supplement the Standard’s happens-before relationship with an LLVM-specific happens-to-work relationship. On most architectures this will be implemented with micro-pause primitives such as x86’s rep rep rep nop instruction.


    Shifts by bit-width or larger will now return a normally-distributed random number. This also obsoletes rand() and std::random_shuffle.


    bool now obeys the rules of truthiness to avoid that annoying “but what if it’s not zero or one?” interview question. Further, incrementing a bool with ++ now does the right thing.


    Atomic integer arithmetic is already specified to be two’s complement. Regular arithmetic will therefore now also be atomic. Except when volatile, but not when volatile atomic.


    NaNs will now compare equal, subnormals are free to self-classify as normal / zero / other, negative zero simply won’t be a thing, IEEE-754 has been upgraded to PONY-754, floats will still round with style, and generating a signaling NaN is now guaranteed to not be quiet by being equivalent to putchar('\a'). While we’re at it none of math.h will set errno anymore. This has nothing to do with undefined behavior but seriously, errno?


    Type-punning isn’t a thing anymore. We’re renaming it to type-pony-ing, but it doesn’t do anything surprising besides throw parties. AND WHO DOESN’T LIKE PARTIES‽ EVEN SECURITY PEOPLE DO! 🎉


    A Word From Our Sponsors



    The sanitizers—especially undefined behavior sanitizer, address sanitizer and thread sanitizer—are great tools when dealing with undefined behavior. Use them on your tests, combine them with fuzzers, try them as cupcake topping! Be warned: their runtimes aren’t designed to be secure and you shouldn’t ship them in production code!


    Cutie Marks


    To address the horse in the room: we’ve left the new LLVM logo’s cutie mark as implementation-defined. Different instances of the logo can use their own cutie mark to illustrate their proclivities, but must clearly document them.



    by Unknown (noreply@blogger.com) at April 01, 2016 07:02 AM

    March 29, 2016

    OpenMP Runtime Project

    New code release

    We are excited to announce the next release of the Intel® OpenMP* Runtime Library at openmprtl.org. This release aligns with Intel® Parallel Studio XE 2016 Composer Edition Update 3.

    New Features

    • OpenMP* 4.5 schedule(simd:static) support

    Bug Fixes

    • Hwloc topology discovery improved
    • Spin backoff mechanism fixed in lock code
    • Plain barrier performance improved on Intel(R) Xeon Phi

    Contributions

    by mad\tlwilmar at March 29, 2016 06:50 PM

    March 28, 2016

    LLVM Blog

    LLVM Weekly - #117, Mar 28th 2016

    Welcome to the one hundred and seventeenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Google Summer of Code applications are now closed. Applicants and interested third-parties can look forward to finding out which projects were selected on April 22nd.

    Ramkumar Ramachandra has written a blog post giving a whirlwind tour of the internals of LLVM's fast register allocator (FastRegAlloc.cpp).

    Alex Denisov has blogged about the various test suites used within the LLVM project.

    Version 1.13 of the TTA-based Co-design Environment (TCE) has been released. This adds support for LLVM 3.8.

    On the mailing lists

    LLVM commits

    • A new utility, update_test_checks.py was added to update opt or llc test cases with new FileCheck patterns. r264357.

    • Non-power-of-2 loop unroll count pragmas are now supported. r264407.

    • The NVPTX backend gained a new address space inference pass. r263916.

    • Instances of Error are now convertible to std::error_code. Conversions are also available between Expected<T> and ErrorOr<T>. r264221, r264238.

    • Hexagon gained supported for run-time stack overflow checking. r264328.

    Clang commits

    • Clang now supports lambda capture of *this by value. r263921.

    • The bitreverse builtins are now documented. r264203.

    Other project commits

    • LLDB will fix inputted expressions with 'trivial' mistakes automatically. r264379.

    • ThreadSanitizer debugging support was added to LLDB. r264162.

    • Polly gained documentation to describe how it fits in to the LLVM pass pipeline. r264446.

    • LLDB has been updated to handle the UTF-16 APIs on Windows. r264074.

    by Alex Bradbury (noreply@blogger.com) at March 28, 2016 01:22 PM

    March 21, 2016

    LLVM Blog

    LLVM Weekly - #116, Mar 21st 2016

    Welcome to the one hundred and sixteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    If you're a student and would like to get paid to work on an LLVM-related project over the summer then do consider applying for Google Summer of Code with LLVM. More details about Summer of Code are available here. The deadline for applications is this Friday, March 25th at 1900 GMT. I'd also encourage you to look at lowRISC's project ideas if you have an interest in open source hardware.

    Stephen Kelly has written about his new Clang-based tool for porting a C++ codebase to use almost-always-auto. As was pointed out on Twitter, Ryan Stortz from Trail of Bits has a tools that removes auto and does roughly the opposite.

    Honza Hubička has written up his experiments of building LibreOffice with GCC6 and LTO. This includes a comparison to a build using LLVM and Clang.

    Nick Clifton has shared an update for February and March on the GNU toolchain that may be of interest.

    The developer of the Capstone disassembly framework and the Unicorn multi-architecture simulator is running a funding campaign for the Keystone multi-architecture assembler framework. Like Capstone, this will build on LLVM but also aims to go beyond it.

    On the mailing lists

    LLVM commits

    • A new Error support class has been added to support structured error handling. See the associated updates to the LLVM programmer's manual for more info. r263609.

    • New documentation was committed for advanced CMake build configurations. r263834.

    • Support was added for MIPS32R6 compact branches. r263444.

    • The MemCpyOptimizer will now attempt to reorder instructions in order to create an optimisable sequence. r263503.

    • llvm-readobj learnt to print sections and relocations in the GNU style. r263561.

    Clang commits

    • Attributes have been added for the preserve_mostcc and preserve_allcc calling conventions. r263647.

    • clang-format will handle some cases of automatic semicolon insertion in JavaScript. r263470.

    • Clang learned to convert some Objective-C message sends to runtime calls. r263607.

    Other project commits

    • AddressSanitizer is now supported on mips/mips64 Android. r263261.

    • The documentation on the LLD linker has added a few numbers to give an idea of the sort of inputs it needs to handle. e.g. Chrome with debug info contains roughly 13M relocations, 6.3M symbols, 1.8M sections and 17k files. r263466.

    by Alex Bradbury (noreply@blogger.com) at March 21, 2016 12:06 PM

    March 14, 2016

    LLVM Blog

    LLVM Weekly - #115, Mar 14th 2016

    Welcome to the one hundred and fifteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    We have an LLVM-related research position currently being advertised here at the University of Cambridge Computer Lab. If you'd like an informal chat about what it's like working in this group or on this project please don't hesitate to get in touch with me.

    News and articles from around the web

    LLVM and Clang 3.8 have now been released. Check out the LLVM and Clang release notes for a run-down of the new features.

    It's GDC this week and if you're attending you may be interested that there's an LLVM meetup scheduled for Thursday.

    Felix Angell has a detailed blog post introducing generating LLVM IR from Go.

    On the mailing lists

    LLVM commits

    • Loop invariant code motion learnt the ability the exploit the fact a memory location is known to be thread-local. r263072.

    • A new llvm.experimental.deoptimize intrinsic has been added. r26328.

    • A ThinLTOCodeGenerator was added in order to provide a proof-of-concept implementation. r262977.

    • The Sparc backend gained support for co-processor condition branching and conditional traps. r263044.

    Clang commits

    • Clang gained support for the [[nodiscard]] attribute. r262872.

    • New AST matchers were added for addrLabelExpr, atomicExpr, binaryCondtionalOperator, designatedINitExpr, designatedInitExpr, designatorCountIs, hasSyntacticForm, implicitValueINitExpr, labelDecl, opaqueValueExpr, parenListExpr, predefinedExpr, requiresZeroInitialization, and stmtExpr. r263027.

    Other project commits

    • Error and warning messages in LLD are now more consistent. r263125.

    • Documentation on the new ELF and COFF LLD linkers has been updated. r263336.

    by Alex Bradbury (noreply@blogger.com) at March 14, 2016 11:56 AM

    March 07, 2016

    LLVM Blog

    LLVM Weekly - #114, Mar 7th 2016

    Welcome to the one hundred and fourteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    LLVM has been accepted as a mentoring organisation in Google Summer of Code 2016. See here for more about what that means. If you're a student who would like to get paid to work on LLVM over the summer, you should definitely consider applying. Also take a look at the full list of organisations in GSoC 2016. If you have an interest in open source hardware, in my (biased) opinion you should definitely look at lowRISC's listed project ideas.

    LLVM and Clang 3.8 'final' has been tagged. A release should be imminent.

    There was a big C++ committee meeting last week. You can find summaries here and here. If you were hoping for modules, concepts, UFCS, ranges, or coroutines in C++17 I'm afraid you're in for disappointment. Many new features will be available in C++ Technical Specifications though.

    llvmlite 0.9.0 has been released. llvmlite is a light-weight Python binding for LLVM. If you're wondering how to get started with llvmlite, then check out this recent blog post from Ian Bertolacci on writing fibonacci in LLVM with llvmlite.

    Andi McClure has written a really interesting blog post about writing software without a compiler. In this case, generating LLVM IR from LuaJIT.

    On the mailing lists

    LLVM commits

    • MemorySSA has gained an initial update API. r262362.

    • TableGen can now check at compile time that a scheduling model is complete. r262384.

    • New comments in PassBuilder give a description of what trade-offs are expected for each optimisation level. r262196.

    • LoopLoadElimination is now enabled by default. r262250.

    • A new patch adding infrastructure for profile-guided optimisation enhancements in the inline has landed. r262636.

    • Experimental ValueTracking code which tried to infer more precise known bits using implied dominating conditions has been removed. Experiments didn't find it to be profitable enough, but it may still be useful to people wanting to experiment out of tree. r262646.

    Clang commits

    • Clang's C API gained an option to demote fatal errors to non-fatal errors. This is likely to be useful for clients like IDEs. r262318.

    • clang-cl gained initial support for precompiled headers. r262420.

    • An -fembed-bitcode driver option has been introduced. r262282.

    • Semantic analysis for the swiftcall calling convention has landed. r262587.

    • Clang's TargetInfo will now store an actual DataLayout instance rather than a string. r262737.

    Other project commits

    • LLDB can now read line tables from Microsoft's PDB debug info files. r262528.

    • The LLVM test-suite gained the ability to hash generated binaries and to skip tests if the hash didn't change since a previous run. r262307.

    • LLVM's OpenMP runtime now supports the new OpenMP 4.5 doacross loop nest and taskloop features. r262532, r262535.

    by Alex Bradbury (noreply@blogger.com) at March 07, 2016 12:40 PM

    February 29, 2016

    LLVM Blog

    LLVM Weekly - #113, Feb 29th 2016

    Welcome to the one hundred and thirteenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    News and articles from around the web

    LLVM and Clang 3.8RC3 has been tagged.

    EuroLLVM 2016 is less than a month away. If you want to attend, be sure to register.

    The Red Hat blog has a summary of new features in the upcoming GCC 6 release.

    The Meeting C++ blog has a helpful summary of a subset of the proposals for the next C++ committee meeting.

    On the mailing lists

    LLVM commits

    • The Sparc backend now contains definitions for all registers and instructions defined in the Sparc v8 manual. r262133.

    • LLVM gained a basic LoopPassManager, though it currently only contains dummy passes. r261831.

    • A number of TargetInstrInfo predicates now take a reference to a MachineInstr rather than a pointer. r261605.

    • The WebAssembly backend gained redzone support for the userspace stack. r261662.

    Clang commits

    • Whole-program vtable optimisation is now available in Clang using the -fwhole-program-vtables flag. r261767.

    • Clang gained __builtin_canonicalize which returns the platform-specific canonical encoding of a floating point number. r262122.

    • A hasAnyName matcher was added. r261574.

    • The pointer arithmetic checker has been improved to report fewer false positives. r261632.

    Other project commits

    • The new ELF linker gained support for identical code folding (ICF). This reduces the size of an LLD binary by 3.6% and of a Clang binary by 2.7%. As described in the commit message, this is not a "safe" version of ICF as implemented in GNU gold, so will cause issues if the input relies on two distinct functions always having distinct addresses. r261912.

    • Polly's tree now contains an update_check.py script that may be useful to other LLVM devs. It updates a FileCheck-based lit test by updating the CHECK: lines with the actual output of the RUN: command. r261899.

    • LLDB gained a new set of plugins to help debug Java programs, specifically Java code JIT-ed by the Android runtime. r262015.

    • The new OpenMP 4.5 affinity API is now supported in LLVM's openmp implementation. r261915.

    • The new ELF linker gained support for the -r command-line option, which produces relocatable output (partial linking). r261838.

    • The CMake/lit runner for SPEC in the LLVM test-suite can now run the C CPU2006 floating point benchmarks (but not the Fortran ones). r261816.

    • The old ELF linker has been deleted from LLD. r262158.

    by Alex Bradbury (noreply@blogger.com) at February 29, 2016 02:58 PM

    February 25, 2016

    OpenMP Runtime Project

    New code release

    We are excited to announce the next release of the Intel® OpenMP* Runtime Library at openmprtl.org. This release aligns with Intel® Parallel Studio XE 2016 Composer Edition Update 2

    New Features:

    • Hwloc* 2.0 support added for affinity interface
    • OMPT support for windows
    • Support for untied tasks
    • OpenMP* 4.5 doacross, taskloop, and new affinity API

    Bug Fixes:

    by mad\jlpeyton at February 25, 2016 10:21 PM

    February 22, 2016

    LLVM Blog

    LLVM Weekly - #112, Feb 22nd 2016

    Welcome to the one hundred and twelfth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Filip Pizlo has written a fantastic article introducing the new B3 JIT compiler for WebKit's JavaScriptCore. This intends to replace LLVM as the optimising backend to their fourth-tier JIT. The article describes in detail their reasons for moving away from LLVM (mainly compile-time) and the design trade-offs made, such as in reducing memory allocations and minimising pointer-chasing in the IR. This reminds me of the trade-offs Mike Pall made in the LuaJIT 2.0 IR. Philip Reames also shared some initial thoughts on B3. I know some people have expressed disappointment about WebKit moving away from LLVM, but if you'll allow me to insert just a little bit of editorial I'd argue B3 is a very positive development for LLVM and the wider compiler community. B3 explores a different set of design trade-offs to those chosen for LLVM, and these sort of changes are probably easiest to explore in a fresh codebase.Thanks to this write-up (and hopefully future B3/AIR documentation), we can learn from the B3 developers' experiences and consider if some of their choices will make sense for LLVM. It's also good to remember that LLVM isn't the only feasible route for code generation and optimisation, and we shouldn't treat LLVM's design choices as the one-true way to do things. Impressively, B3 was developed to its current state in only 6 months of developer-time.

    Version 0.17.0 of LDC, the LLVM-based compiler for the D programming language has been released. You can view a detailed changelog here.

    GCC6 will feature a whole bunch of new warnings, and this blog post details many of them.

    The schedule for EuroLLVM 2016 has now been posted. This will be held March 17th-18th in Barcelona.

    On the mailing lists

    LLVM commits

    • The PPCLoopDataPrefetch pass has been moved to Transforms/Scalar/LoopDataPrefetch in preparation for it becoming a target-agnostic pass. r261265.

    • The cmpxchg LLVM instruction now allows pointer type operands. r261281.

    • The X86 backend gained support for a new stack symbol ordering optimisation. This is primarily intended to reduce code size, and produces small but measurable improvements across some SPEC CPU 2000 benchmarks. r260917.

    • The LLVM C API has been extended to allow it to be used to manipulate the datalayout. r260936.

    • Some major work on the LazyCallGraph has been checked in. r261040.

    • The AMDGPU backend gained a basic disassembler. r261185.

    • The PostOrderFuctionAttrs pass has been ported to the new pass manager. As described in the commit message, this actually represents a major milestone. r261203.

    • The Hexagon backend gained support for thread-local storage. r261218.

    Clang commits

    • A nullPointerConstant AST matcher was added. r261008.

    • Clang gained a -Wcomma warning, which will warn for most uses of the builtin comma operator. r261278

    Other project commits

    • LLD has sprouted a release notes document. r260960.

    • The LLVM test-suite's CMake build system saw a number of fixes for SPEC. r261470.

    by Alex Bradbury (noreply@blogger.com) at February 22, 2016 11:29 AM

    February 16, 2016

    Philip Reames

    Quick thoughts on WebKit’s B3

    I just finished reading Introducing the B3 JIT Compiler and wanted to jot down some thoughts.  Fair warning, this post is being written in a hurry.  I’m focusing on getting down my initial reactions rather than trying for a really well thought out post.  That may follow at a later time or it may not.

    The first key bit is that the goal of this effort appears to be strictly compile time, not peak performance.  Understanding that makes the effort make a lot more sense.  It’s still seems a bit odd to me for the compile time of your *fourth tier* JIT to be that important, but given I’m no expert in JavaScript, I’ll just accept that as a given.

    In that context, I find the claimed 4.7x reduction in compile time surprisingly unexciting.  There’s enough low hanging fruit in LLVM – in particular, a better representation for “check” nodes – that I would expect something on that magnitude being possible within the existing framework they had.  Achieving a ~5 improvement of compile time with an entirely new compiler (and all of the engineering that implies), seems slightly disappointing.  Now it’s possible (heck, even likely!) that the new architecture will allow them to further drop compile time, but still…

    The performance numbers quoted were uninformative at best.  From what I can gather in the write up, the JetStream benchmark is highly influenced by compile time.  While I understand the goal (it’s a useful one), it doesn’t really say anything about the peak performance of the code generated by the two compilers.  Given that, it’s really hard to tell if B3 is actually breakeven with the existing LLVM backend at peak. It have been really nice to see the same numbers with the compile time somehow removed or adjusted for.  (b3 with a sleep to be slower?  A longer warmup period in a modified benchmark?)

    Just to be clear, I’m not saying that the numbers presented are “wrong”.  Merely that they don’t answer the question I’d most like them to. 🙂

    Other things that jumped out at me:

    • The points about matching the IR to the source language in an effort to reduce the number of nodes (and thus memory, and time) are spot on.  If what you’re going for is compile time above all else, using an IR which closely matches your source language is absolutely the right approach.  This same general idea (remove memory/nodes where they don’t provide enough value) is what’s motivating the removal of pointer-to-pointer bitcasts in LLVM right now.
    • The emphasis on the importance of the “check” node (i.e. early OSR exit if condition fails) matches our experience as well.  You can also see this in Swift’s IR as well.  There is clearly an area that LLVM needs to improve.  I think we can do a lot better within LLVM, and I’m surprised they didn’t try to push that.  In particular, the aliasing problems mentioned could have been addressed with a custom AliasAnalysis instance.  Oh well.
    • The choice to use arrays (instead of lists) gives some interesting tradeoffs.  From a compile time perspective, a modify and compact scheme is likely a win.  Interestingly, this reminds me a lot of a mark-compact garbage collector (b3’s layout) vs a standard malloc/free allocator (llvm’s layout).  Given typical lifetimes in a compiler (short!), the collector approach is likely to be the right one.  It does raise some interesting challenges though: pointer equality can no longer be used, trivial dead code elimination is no longer trivial (which complicates various other transforms), and transforms have to deal with non-canonical forms due to extra identify nodes.  It’ll be really interesting to see where b3 goes on this basis alone.

    by reames at February 16, 2016 12:22 AM

    February 15, 2016

    LLVM Blog

    LLVM Weekly - #111, Feb 15th 2016

    Welcome to the one hundred and eleventh issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    There has been a new release of the CilkPlus compiler. This includes an update to the latest LLVM and Clang trunk. CilkPlus implements the Cilk Plus language extensions for data and task parallelism in Clang.

    There's been some more papers appearing from the C++ standards committee. P0225R0, or as you may prefer to call it "Why I want Concepts, and why I want them sooner rather than later" is worth a read. There's also been a few other recently published papers on iterator facades, the filesystem technical specification, and unified function call syntax.

    On the mailing lists

    LLVM commits

    • The WholeProgramDevirt pass has been added. This implements whole program optimization of virtual calls where the list of callees is known to be fixed. r260312.

    • The AVR backend upstreaming continues with the addition of the AVR tablegen instruction definitions. r260363.

    • There's been a bunch of other work on the new global instruction selection mechanism this week, but the commits I'd pick out are the addition of support for translating Add instructions and for lowering returns. It is currently being tests with the AArch64 backend. r260549, r260562, r260600.

    • The AArch64 backend gained support (including a scheduling model) for the Qualcomm Kryo CPU. r260686.

    • LoopUnrollAnalyzer has been abstracted out from LoopUnrollPass, and gained unit tests for its functionality. r260169.

    • llvm-config gained preliminary Windows support. r260263.

    • The details of the convergent attribute have been clarified in the language reference. The convergent attribute will now be removed on functions which provably don't converge or invoke any convergent functions. r260316, r260319.

    Clang commits

    • It is now possible to perform a 3-stage Clang build using CMake. It is suggested in the commit message this may be useful for detecting non-determinism in the compiler by verifying stage2 and stage3 are identical. r260261.

    • ARMv8.2-A can be targeted using appropriate Clang options. r260533.

    • Clang's CMake build system learned the CLANG_DEFAULT_CXX_STDLIB to set the default C++ standard library. r260662.

    Other project commits

    • The new LLD ELF linker gained initial link-time optimisation support. r260726.

    • LLDB has seen some more updates for Python 3 support, though not yet enough for a clean testsuite run. r260721.

    by Alex Bradbury (noreply@blogger.com) at February 15, 2016 11:58 AM

    February 08, 2016

    LLVM Blog

    LLVM Weekly - #110, Feb 8th 2016

    Welcome to the one hundred and tenth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Slides from the LLVM devroom at FOSDEM last weekend are now available online. Unfortunately there was an issue with the recording of the talks so videos will not be available.

    JavaScriptCore's FTL JIT is moving away from using LLVM as its backend, towards B3 (Bare Bones Backend). This includes its own SSA IR, optimisations, and instruction selection backend.

    Source tarballs and binaries are now available for LLVM and Clang 3.8-RC2.

    The Zurich LLVM Social is coming up this Thursday, February 11th at 7pm.

    Jeremy Bennett has written up a comparison of the Clang and GCC command-line flags. The headline summary is that 397 work in both GCC and LLVM, 433 are LLVM-only and 598 are GCC-only.

    vim-llvmcov has been released. It is a vim plugin to show code coverage using the llvm-cov tool.

    On the mailing lists

    • Mehdi Amini has posted an RFC on floating point environment and rounding mode handling in LLVM. The work started all the way back in 2014 and has a whole bunch of patches up for review. Chandler Carruth has responded with a detail description of his concerns about the current design, and his proposed alternative seems to be getting a lot of positive feedback.

    • Morten Brodersen has recently upgraded a number of applications from the old JIT to the new MCJIT under LLVM 3.7.1 but has found significant performance regressions. Some other respondents have seen similar issues, either in compilation time or in reduced code quality in the generated code. Some of the thread participants will be providing specific examples so they can be investigated. It's possible the issue is something as simple as a different default somewhere. Benoit Belley noted they saw regressions due to their frontend's use of allocas in 3.7.

    • Lang Hames kicked off a long discussion about error handling in LLVM libraries. Lang has implemented a new scheme and is seeking feedback on it. There's a lot of discussion that unfortunately I haven't had time to summarise properly. If error handling design interests you, do get stuck in.

    • Adrian McCarthy has written up details on the recent addition of minidump support to LLDB. Minidumps are the Windows equivalent of a core file.

    • Juan Wajnerman is looking at adding support for multithreading to the Crystal language, and has a question about thread local variables. LLVM won't re-load the thread local address, which causes issues when a thread local variable is read in a coroutine running on one thread which is then suspended and continued on a different thread. This is apparently a known issue, covered by PR19177.

    • Steven Wu has posted an RFC on embedding bitcode in object files. The intent is to upstream support that already exists in Apple's fork. Understandably some of the respondents asked how this relates to the .llvmbc section that the Thin-LTO work is introducing. Steven indicates it's pretty much the same, but for Mach-O rather than ELF and that he hopes to unify them during the upstreaming.

    LLVM commits

    • LLVM now has a memory SSA form. This isn't yet used by anything in-tree, but should form a very useful basis for a variety of analyses and transformations. This patch has been baking for a long time, first being submitted for initial feedback in April last year. r259595.

    • A new loop versioning loop-invariant code motion (LICM) pass was introduced. This enables more opportunities for LICM by creating a new version of the loop guarded by runtime checks to test for potential aliases that can't be determined not to exist at compile-time. r259986.

    • LazyValueInfo gained an intersect operation on lattice values, which can be used to exploit multiple sources of facts at once. The intent is to make greater use of it, but already it is able to remove a half range-check when performing jump-threading. r259461.

    • The SmallSet and SmallPtrSet templates will now error out if created with a size greater than 32. r259419.

    • The ability to emit errors from the backend for unsupported features has been refactored, so BPF, WebAssembly, and AMDGPU backends can all share the same implementation. r259498.

    • A simple pass using LoopVersioning has been added, primarily for testing. The new pass will fully disambiguate all may-aliasing memory accesses no matter how many runtime checks are required. r259610.

    • The way bitsets are used to encode type information has now been documented. r259619.

    • You can now use the flag -DLLVM_ENABLE_LTO with CMake to build LLVM with link-time optimisation. r259766.

    • TableGen's AsmOperandClass gained the IsOptional field. Setting this to 1 means the operand is optional and the AsmParser will not emit an error if the operand isn't present. r259913.

    • There is now a scheduling model for the Exynos-M1. r259958.

    Clang commits

    • Clang now has builtins for the bitreverse intrinsic. r259671.

    • The option names for profile-guided optimisations with the cc1 driver have been modified. r259811.

    Other project commits

    • AddressSanitizer now supports iOS. r259451.

    • The current policy for using the new ELF LLD as a library has been documented. r259606.

    • Polly's new Sphinx documentation gained a guide on using Polly with Clang. r259767.

    by Alex Bradbury (noreply@blogger.com) at February 08, 2016 04:44 PM

    LLVM Weekly - #109, Feb 1st 2016

    Welcome to the one hundred and ninth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    The GNU Tools Cauldron 2016 has been announced for the 9th-11th of September 2016, in Hebden Bridge, UK.

    The Sulong project has been announced. It is an LLVM IR interpreter using the Truffle framework and Graal on the JVM to support JIT compilation.

    Ehsan Akhgari has posted an updated on building Firefox with clang-cl. It is now possible to build a complete Firefox with Clang without using the MSVC fallback once.

    I've mentioned it down below in the list of notable commits, but it's worth calling out here too: the old autoconf build-system has now been removed from LLVM. 3.8 will be the last release to include it. Time to switch to CMake if you haven't already.

    John Regehr gave a talk about undefined behaviour in LLVM at the Paris LLVM meetup, and you can find the slides here.

    On the mailing lists

    LLVM commits

    • The autoconf build system for LLVM has been removed. r258861.

    • The WebAssembly backend gained support for unaligned loads and stores. r258779.

    • LLVM's MCAsmSreamer will now always use .p2align rather than .align, because .align's behaviour can differ between targets. r258750.

    • Intrinsic IDs are now looked up by binary search rather than the previous more complex mechanism. This improves the compile time of Function.cpp. r258774.

    • TargetSelectionDAGInfo has been renamed to SelectionDAGTargetInfo and now lives in CodeGen rather than Target. r258939.

    • A LoopSimplifyCFG pass was added to canonicalise loops before running through passes such as LoopRotate and LoopUnroll. r259256.

    Clang commits

    • The clang-cl driver will now warn for unknown arguments rather than erroring, to match the behaviour of MSVC. r258720.

    • The old autoconf build system was removed from Clang. r258862.

    • The 'sancov' (SanitizerCoverage) tool gained some documentation. r259000.

    Other project commits

    • libcxx gained an implementation of ostream_joiner. r259014, r259015.

    • lld gained a new error function which won't cause process exit. The hope is this can be used to provide a gradual path towards lld-as-a-library. r259069.

    • The lit runner for the LLVM test suite can now be passed --param=profile=perf which will cause each test to be run under perf record. r259051.

    by Alex Bradbury (noreply@blogger.com) at February 08, 2016 04:43 PM

    January 25, 2016

    LLVM Blog

    LLVM Weekly - #108, Jan 25th 2016

    Welcome to the one hundred and eighth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    LLVM 3.8 RC1 has been released. Now is the time to test it out with your favourite projects and report any issues.

    The deadline for the EuroLLVM call for papers is today.

    Version 1.6 of the Rust programming language was released las week. Rust uses LLVM for its code generation.

    The LLVM Social in Paris will be held this week on Wednesday.

    On the mailing lists

    LLVM commits

    • llvm::SplitModule gained a new flag which can be used to cause it to attempt to split the module without globalizing local objects. r258083.

    • The WebAssembly backend will now rematerialize constants with multiple uses rather than holding them live in registers, as there is no code size saving in using registers in for constants in most cases in the WebAssembly encoding. r258142.

    • Some small patches from the global instruction selection effort have started to land, such as the introduction of a generic machine opcode for ADD (G_ADD) and the all-important CMake support for building it. r258333, r258344.

    • getCacheLineSize was added to TargetTransformInfo. It's currently only used by PPCLoopDataPrefetch. r258419.

    • LoopIdiomRecognize improved in its ability to recognise memsets. r258620.

    Clang commits

    • A number of new AST matchers were added. r258042, r258072, and more.

    • The LeakSanitizer documentation has been updated with a usage example. r258476.

    Other project commits

    • The new ELF linker gained initial support for MIPS local GOT (global offset table) entries. r2583888.

    • The LLVM test suite now contains a ClangAnalyzer subdirectory containing tests for the static analyzer. r258336.

    by Alex Bradbury (noreply@blogger.com) at January 25, 2016 12:48 PM

    January 18, 2016

    LLVM Blog

    LLVM Weekly - #107, Jan 18th 2016

    Welcome to the one hundred and seventh issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    I have a very exciting piece of non-LLVM news to share this week. On Saturday I proposed to my partner Carrie Anne, and I'm delighted to report that she said yes. You may well question if this piece of personal news has any relevance to you, and in response I'd like to highlight just how important Carrie Anne is to this weekly newsletter. For over two years now, I've given up 2-3+ hours of my time every week without fail on evenings and weekends, time we could really be spending together as a couple. Without Carrie Anne's understanding and support LLVM Weekly couldn't exist. 2016 is going to be a very exciting year.

    News and articles from around the web

    Registration is now open for EuroLLVM 2016. The conference will be held in Barcelona on March 17th-18th. The call for papers closes on January 25th.

    Registration is open for the Clang/LLVM development sprint to be held on the weekend of Feb 6th/7th at Bloomberg's London and New York offices.

    The next Cambridge LLVM social will be held on Wednesday 20th January at 7.30pm, and will be colocated with the FreeBSD social.

    On the mailing lists

    LLVM commits

    • The ORC JIT API now supports remote JITing over an RPC interface to a separate process. The LLI tool has been updated to use this interface. r257305, r257343.

    • The Hexagon backend gained a target-independent SSA-based data flow framework for representing data flow between physical registers and passes using this to implement register liveness analysis, dead code elimination, and copy propagation. r257447, r257480, r257485, r257490.

    • The documentation on committing code reviewed on Phabricator to trunk has been improved. r257764.

    • WebAssembly gained a prototype instruction encoder and disassembler based on a temporary binary format. r257440.

    • LLVM's MathExtras gained a SaturatingMultiplyAdd helper. r257352.

    • llvm-readobj has much-expanded support for dumping CodeView debug info. r257658.

    • The code that finds code sequences implementing bswap or bitreverse and emits the appropriate intrinsic has been rewritten. r257875.

    • The AMDGPU backend gained a new machine scheduler for the Southern Islands architecture. r257609.

    Clang commits

    • A Python implementation of scan-build has been added. r257533.

    • The 'interrupt' attribute is now supported on x86. r257867.

    • Clang learned to respond to the -fsanitize-stats flag. It can currently only be used with control-flow integrity and allows statistics to be dumped. r257971.

    Other project commits

    • The compiler-rt CMake buildsystem gained experimental support for tvOS and watchOS. r257544.

    • Initial support was added for PPC and the new ELF linker. r257374.

    • The CMake and Lit runners in the LLVM test-suite can now support the integer C and C++ tests from SPEC CPU2006. r257370.

    by Alex Bradbury (noreply@blogger.com) at January 18, 2016 01:33 PM

    January 11, 2016

    LLVM Blog

    LLVM Weekly - #106, Jan 11th 2016

    Welcome to the one hundred and sixth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    Many readers may be interested that last week was the 3rd RISC-V Workshop. You can find slides from the two lowRISC talks here and here. You may also want to read my liveblog of the event.

    News and articles from around the web

    The BSD Now podcast recently interviewed Alex Rosenberg about his work on LLVM/Clang and FreeBSD.

    The folks at QuarksLab have shared a Clang hardening cheat sheet.

    LLDB 3.8 will feature initial Go debugging support.

    The next Paris LLVM Social will be held on January 27th and includes a talk from John Regehr.

    The next Zurich LLVM Social will be taking place on January 14th.

    On the mailing lists

    LLVM commits

    • LLVM gained the -print-funcs option which can be used to filter IR printing to only certain functions. r256952.

    • The LLVM ADT library gained a new sum type abstraction for pointer-like types and an abstraction for embedding an integer within a pointer-like type. r257282, r257284.

    • LLVM now recognises the Samsung Exynos M1 core. r256828.

    • InstCombine learned to expose more constants when comparing getelementptrs (GEPs) by detecting when both GEPs could be expressed as GEPs with the same base pointer. r257064.

    • SelectionDAGBuilder will set NoUnsignedWrap for an inbounds getelementptr and for load/store offsets. r256890.

    • AArch64 MachineCombine will now allow fadd and fmul instructions to be reassociated. r257024.

    • Macro emission in DWARFv4 is now supported. r257060.

    • llvm-symbolizer gained the -print-source-context-lines option to print source code around the line. r257326.

    Clang commits

    • Clang's CMake build system can now perform a multi-stage bootstrap build with profile-guided optimisation. r256873.

    • Clang's command line frontend learned to handle a whole bunch of -fno-builtin-* arguments. r256937.

    • The new ELF LLD linker will now be used for th AMDGPU target. r257175.

    Other project commits

    • The performance of string table construction in the LLD ELF linker has been improved. This improves link time of lld by 12% from 3.50 seconds to 3.08 seconds. r257017.

    • The LLD ELF linker gained support for the AMDGPU target. r257023.

    by Alex Bradbury (noreply@blogger.com) at January 11, 2016 01:15 PM

    January 04, 2016

    LLVM Blog

    LLVM Weekly - #105, Jan 4th 2016

    Welcome to the one hundred and fifth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    Happy new year! This issue marks the second anniversary of LLVM Weekly. It's rather short as the past week has been very quiet, with most LLVM developers seemingly taking a break over the holidays. My colleague Wei Song and myself will be presenting about lowRISC at the 3rd RISC-V workshop on Wednesday this week. Do say hi if you're going to be there.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Sanjoy Das has written a blog post about issues with LLVM's undef value. Interestingly, he provides an example where undef can actually inhibit optimisations.

    On the mailing lists

    LLVM commits

    • The -align-all-loops and -align-all-functions arguments have been introduced to force function or loop alignments for testing purposes. r256571.

    • The x86 backend has added intrinsics for reading and writing to the flags register. r256685.

    Clang commits

    • Various Clang classes have been converted to use the TrailingObjects helper. r256658, r256659, and more.

    • __readeflags and __writeeflags intrinsics are exposed in Clang. r256686.

    Other project commits

    • In libcxx, undefined behaviour in <list> has been fixed for builtin pointer types and support added for the next ABI version. r256652.

    by Alex Bradbury (noreply@blogger.com) at January 04, 2016 02:07 PM

    Philip Reames

    A perspective on friendly C

    I was talking about John Regehr’s Friendly C proposal and recent follow-on post tonight with a friend, and decide to jot down some thoughts in a sharable format.

    I believe the idea of a friendly C variant is entirely feasible, but it posses an incredibly challenging design problem.  Every change considered needs to be validated against a deep knowledge of the implementation of the associated compiler, runtime environment, and the underlying hardware.

    As a simple example, let’s consider trying to establish semantics for stray (i.e. out of bounds) read and writes.  We can start by trying to define what happens for a stray read.  That’s fairly easy, we can simply return an undefined value.  We could even be a bit more restrictive and say that the value must be one which is written to that address by some part of the program.

    (The vagueness in that last bit is to allow concurrent execution reordering.  However, we accidentally required atomic reads and writes since we disallowed wording tearing.  Is that a good thing or not?  There’s a cost to that, but maybe it’s a cost we’re willing to pay.  Or maybe not…)

    Now let’s consider how to handle stray writes.  We could simply define them to be erroneous, but that simply gets us back to undefined behavior in C/C++.  We’re trying to avoid that.  We either need to detect them, or provide a reasonable semantics.  Detecting arbitrary stray writes is a very hard problem.  We can easily handle specific categories of stray writes through techniques like implicit null checking, but detecting an arbitrary stray write requires something like a full address-sanitizer (or possibly even more expensive checks).  I doubt anyone is willing to pay 2x performance for their C code to be more friendly.  If they were, why are they writing in C?

    The challenge with having defined stray writes is what does a particular read return?  Does it return the last written value to a particular address?  Or the last value written to the particular field of the given object?  With out of bounds writes, these are not necessarily the same.

    It’s very tempting to have the read return the last value written to the underlying address, but that introduces a huge problem.  In particular, it breaks essentially all load-load forwarding.

    int foo(int* p_int, float p_float) {
     int a = *p_int;
     *p_float = 0.0;
     return a - *p_int;
    }

    In the example above, your normal C compiler could return “0” because it assumes the intervening write can’t change the value at p_int.  An implementation of a friendly C variant with the semantics we’ve proposed could not.  In practice, this is probably unacceptable from a performance perspective; memory optimization (load-load forwarding and associated optimizations) is a huge part of what a normal C/C++ compiler does.  (see: BasicAA, MDA. GVN, DSE, EarlyCSE in LLVM)

    If we want to avoid that problem, we could try to be more subtle in our definition.  Let’s say we instead defined a read as returning either the last value written to that field (i.e. in bounds write) or underlying memory address (i.e. stray write).  We still have the problem of requiring atomic memory access, but we seem to have allowed the compiler optimization we intended.

    The problem with this definition is that we’ve introduced a huge amount of complexity to our language specification and compiler.  We now have to have separate definitions of both our objects, their underlying addresses, and all the associated implementation machinery.

    Another approach would be to define a read as returning either the last value written to the field (if no stray write has occurred to that address) or an undefined value (if a stray write to that address has occurred).  Is that friendly enough?

    Moreover, what happens if we improve our ability to detect stray writes?  Are we allowed to make that write suddenly fail?  Is a program which functions only because of a stray write correct?

    (Before you dismiss this as ridiculous, I personally know of an emergency software release that did nothing but reintroduce a particular stray memory write in a C++ program because it happened to restore behavior that a client had been relying on for many many years.)

    Hopefully, I’ve given you a hint of the complexities inherent in any friendly C proposal.  These are the same complexities involved in designing any new language.  If anything designing a workable friendly-C proposal is harder than designing a new language.  At least with a new language you’d have the freedom to change other aspects of the language to avoid having to deal with garbage pointers entirely; in practice, that’s often the much easier approach.

     

    by reames at January 04, 2016 04:32 AM

    December 28, 2015

    LLVM Blog

    LLVM Weekly - #104, Dec 28th 2015

    Welcome to the one hundred and fourth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    The schedule for the LLVM devroom at FOSDEM has been published. This will be on January 30th 2016 in Brussels at FOSDEM.

    Andy Finnell spent some time over the Christmas vacation porting the LLVM Kaleidoscope tutorial to Erlang and has kindly shared the fruits of his labours.

    Richard Pennington has written another blog post about ELLCC, this time about using it to cross-compile the Linux kernel for the Raspberry Pi.

    Tim Jones (lecturer at the University of Cambridge Computer Laboratory) has written about the alias analysis used in the HELIX compiler. There's nothing LLVM-specific here, indeed it was implemented using ILDJIT but should be of general interest to compiler developers.

    On the mailing lists

    LLVM commits

    • An initial implementation of an LLVMCodeView library has landed. This implements support for emitting debug info in the CodeView format. r256385.

    • lit has gained support for a per-test timeout which can be set using --timeout=. r256471.

    • All uses of edge wights in BranchProbabilityInfo have been replaced with probabilities. r256263.

    • The LLVM project documentation on patch reviews via Phabricator now has advice on choosing reviewers. r256265.

    • The gc.statepoint intrinsic's return type is now a token type rather than i32. r256443.

    Clang commits

    • ASTtemplateKWAndArgsInfo and ASTTemplateArgumentListInfo have been converted to use the TrailingObjects header. This abstracts away reinterpret_cast, pointer arithmetic, and size calculations needed for the case where a class has some other objects appended to the end of it. r256359.

    Other project commits

    • Development of LLD's new ELF linker is continuing, with support for new relocations on x86, x86-64, and MIPS. r256143, r256144, r256172, r256416.

    by Alex Bradbury (noreply@blogger.com) at December 28, 2015 03:50 PM

    December 21, 2015

    LLVM Blog

    LLVM Weekly - #103, Dec 21st 2015

    Welcome to the one hundred and third issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    Regular readers will know about lowRISC, a not-for-profit project a group of us founded aiming to produce a complete open-source System-on-Chip in volume. We've just hit a new milestone with the untethering of the base SoC. If you're interested in contributing, the blog post contains a number of potential starting points.

    News and articles from around the web

    The 6th EuroLLVM conference will be held on March 17th-18th in Barcelona, Spain. The call for papers is now open and will remain open until January 25th 2016. EuroLLVM CFP

    Chandler Carruth's keynote, "Understanding compiler optimizations" from the Meeting C++ 2015 conference is now online.

    Richard Pennington has blogged about bootstrapping LLVM and Clang using pre-compiled ELLCC binaries.

    Bloomberg is going to be holding a weekend Clang and LLVM hackathon in NYC and in London on February 6th and 7th. The event will be open to everyone in the community and Bloomberg will provide space, power, food, beverages, and internet access.They're looking for experienced Clang and LLVM developers to help as mentors.

    On the mailing lists

    LLVM commits

    • LLVM IR now supports floating point atomic loads and stores. r255737.

    • New attributes have been introduced: InaccessibleMemOnly (a function may only access memory that is not accessible by the module being compiled) and InaccessibleMemOrArgMemOnly (a function may only access memory that is either not accessible by the module being compiled or is pointed to by its pointer arguments). r255778.

    • The PowerPC backend gained support for soft float operations on ppc32. r255516.

    • The terminatepad instruction has been removed from LLVM IR. r255522.

    • IR call instructions can now take a fast-math flags marker which indicates fast-math flags may allow otherwise unsafe optimisations. r255555.

    • LLVM gained a C++11 ThreadPool in its internal library. It is intended to be used for ThinLTO. r255593.

    • The default set of passes has been adjusted. mem2reg will not be run immediately after globalopt and more scalar optimization passes have been added to the LTO pipeline. r255634.

    • The llvm-profdata tool now supports specifying a weight when merging profile data. This can be used to give more relative importance to one of multiple profile runs. r255659.

    • For CMake builds, a compile_commands.json file will now be generated which tells tools like YouCompleteMe and clang_complete how to build each source file. r255789.

    • The Hexagon VLIW packetizer saw a large update (though unfortunately the changes aren't summarised in the commit message). r255807.

    • A number of LLVM's C APIs have been depreciated: LLVMParseBitcode, LLVMParseBitcodeInContext, LLVMGetBitcodeModuleInContext and LLVMGetBitcodeModule. These have been replaced with new versions of the functions which don't record a diagnostic. r256065.

    • The AVR backend (which is being imported incrementally) gained AVR.td and AVRRegisterInfo.td. r256120.

    Clang commits

    • A new checker has been introduced to detect excess padding in classes and structs. r255545.

    • A new control-flow integrity mode was introduced, cross-DSO CFI allows control flow to be protected across shared objects. It is currently marked experimental. r255694.

    • Clang's CMake build system now supports generating profile data for Clang. r255740, r256069.

    Other project commits

    • It is now possible to suppress reports from UndefinedBehaviourSanitizer for certain files, functions, or modules at runtime. r256018.

    • The llvm test-suite's CMake+Lit runner gained support for SPEC2000 and SPEC CPU95. r255876, r255878.

    by Alex Bradbury (noreply@blogger.com) at December 21, 2015 10:13 PM

    December 15, 2015

    LLVM Blog

    LLVM Weekly - #102, Dec 14th 2015

    Welcome to the one hundred and second issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    Version 1.5 of the Rust programming language has been released. Rust of course uses LLVM as its backend.

    George Balatsouras has written a blog post on compiling a project using autotools to LLVM bitcode.

    On the mailing lists

    LLVM commits

    • A new minimum spanning tree based method of instrumenting code for profile-guided optimisation was added. This guarantees the minimum number of CFG edges are instrumented. r255132.

    • MatchBSwap in InstCombine will now also detect bit reversals. r255334.

    • Sample-based profile-guided optimisation memory usage has been reduced by 10x by changing from using a DenseMap for sample records to a std::map. r255389.

    • An Instruction::getFunction method was added. It's perhaps surprising this didn't exist before. r254975.

    • FP16 vector instructions defined in ARMv8.2-A are now supported. r255010.

    • The EarlyCSE (common subexpression elimination) pass learned to perform value forwarding for unordered atomics. r255054.

    • Debug info in LLVM IR can now refer to macros. r255245.

    • LLVM's developer policy has been updated to detail the currently accepted C API stability policy and other guidelines. r255300.

    • A massive rework of funclet-oriented exception handling (needed for Windows exceptions) has landed. r255422.

    Clang commits

    • Clang gained an option to use the new ThinLTO pipeline. r254927.

    • Hexagon will use the integrated assembler by default. r255127.

    • dllexport and dllimport attributes are now exposed through the libclang API. r255273.

    Other project commits

    • ThreadSanitizer gained initial support for PPC64. r255057.

    by Alex Bradbury (noreply@blogger.com) at December 15, 2015 09:41 AM

    December 07, 2015

    LLVM Blog

    LLVM Weekly - #101, Dec 7th 2015

    Welcome to the one hundred and first issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

    The canonical home for this issue can be found here at llvmweekly.org.

    News and articles from around the web

    The implementation of the Swift programming language is now open source. Rather than being a simple code dump, development will now occur out in the open with external contributions encouraged. If you haven't already, now might be a good time to watch Joseph Groff and Chris Lattner's talk on the Swift Intermediate Language.

    Rui Ueyama wrote about the new LLD ELF linker on the official LLVM blog.

    The Visual C++ team have released Clang with Microsoft CodeGen. This uses the Clang parser along with the code generator and optimizer from Visual C++. The majority of the Clang and LLVM changes will be contributed back upstream.

    Alex Denisov wrote about using the LLVM API with Swift.

    If you haven't already submitted your talk proposal for the LLVM devroom at FOSDEM, you've now got a little more time. Get your submission in by this Friday.

    On the mailing lists

    LLVM commits

    • llc and opt gained an option to run all passes twice. This is intended to help show up bugs that occur when using the same pass manager to compile multiple modules. r254774.

    • An initial prototype for llvm-dwp has been committed. This will eventually be a tool for building a DWARF package file out of a number of .dwo split debug files. r254355.

    • All weight-based interfaces in MachineBasicBlock have now been replaced with probability-based interfaces. r254377.

    • LLVM's STLExtras gained a range-based version of std::any_of and std::find. r254391, r254390.

    • llvm.get.dynamic.area.offset.{i32,264} intrinsics have been added. These can be used to get the address of the most recent dynamic alloca. r254404.

    • The X86 backend gained a new pass to reduce code size by removing redundant address recalculations for LEA. r254712.

    • The WebAssembly backend now has initial support for varargs. r254799.

    Clang commits

    • Design docs have been added for forward-edge CFI for indirect calls. r254464.

    • The pass_object_size attribute was added to Clang. This intended to be used to work around cases where __builtin_object_size doesn't function. r254554.

    • Documentation was added for UndefinedBehaviorSanitizer. r254733.

    Other project commits

    • LLD now supports the R_MIPS_HI16/LO16 relocations. r254461.

    • libomp can now make use of libhwloc on Unix to discover topology of the host system. r254320.

    by Alex Bradbury (noreply@blogger.com) at December 07, 2015 11:52 AM