Planet Clang

Within the C committee, there’s been a lot of talk about what the C charter means and whether it should be updated or not. My personal musing on the topic is that the C charter needs to be updated in order for C to remain relevant long-term. Whether C should remain relevant long-term is an exercise left to the reader. Note, these thoughts may change over time as I gather more feedback from more places.

Existing code is important, existing implementations are not.
A standard is a treaty between implementor and programmer.
Migration of an existing code base is an issue.
Minimize incompatibilities with C90.

I think we could combine these principles into one bullet instead of four because they’re all effectively about code migration to newer standard versions.

Each new release of C should provide a simple upgrade path from the previous release of C so that users can gradually transition between adjacent versions of C, but incompatibilities between releases are acceptable. In other words, it’s fine to deprecate old functionality even without replacement, or remove previously deprecated functionality (again, even without replacement), but this should only be done with significant justification given the costs to users of C.

Changing the standard a code base compiles against is a new agreement between the programmer and the implementer; it is akin to changing your code (as is changing your compiler), and there should be no expectation that everything remains identical to how it was before. Older functionality will be deprecated, newer functionality will be added, bugs will be fixed, etc. New standard == new treaty.

I think we should be careful about blind adherence to this principle. It’s obviously true that there’s a lot of economic value tied up in existing C code bases and that rewriting that code is both expensive and error-prone. However, C has undergone multiple revisions in its long history. A good idea in 1978 is not necessarily even acceptable practice in 2024. I would love to see this principle updated to set a time limit, along the lines of: existing code that has been maintained to not use features marked deprecated, obsolescent, or removed in the past ten years is important; unmaintained code and existing implementations are not. If you cannot update your code to stop relying on deprecated functionality, your code is not actually economically important — people spend time and money maintaining things that are economically important.

C code can be portable.
C code can be non-portable.

I think both of these principles are still reasonable today. While commodity hardware has trended towards more homogeneity, there is still plenty of heterogeneous hardware solutions out there (DSPs, FPGAs, etc) and domains like AI, ML, cryptography, etc are pushing the boundaries for new hardware solutions. We should continue to aim to be as portable as possible while not constraining ourselves to the lowest common denominator of hardware.

Avoid quiet changes.

I still strongly believe in this principle. It directly impacts transitions between standard versions — quiet changes make it much harder for users to upgrade to the latest standard.

Keep the spirit of C.

I do not think this principle has much value because it is too subjective, though I understand the desire behind it. I think people want C to remain familiar between releases instead of being reinvented. For example, I think users do not want to see a Perl 5/Perl 6- or Python 2/Python 3-style bifurcation of the language and community. However, I think the principle that existing code is important is sufficient to cover that need.

Most especially, I think concepts like “trust the programmer”, “don’t prevent the programmer from doing what needs to be done”, “make it fast”, etc are problematic given the increased need for improved security in system’s level programming languages.

I would prefer to see this bullet removed because it’s not an actionable principle and I think other bullets cover the main thrust of what it was trying to say.

Support international programming.

I still strongly believe in this principle, though I worry that WG14 lacks enough expertise to really help validate designs in this space. I appreciate that WG21’s SG16 members have been willing to provide advice, and that the Unicode consortium folks are actively engaging with programming language committees as well. But there’s more to internationalization than simply text (for example, units of measure, date/time functionality, and input methods are also part of internationalization). I think the goal is excellent, but hopefully we have enough humility as a committee to recognize when we could use outside advice.

Codify existing practice to address evident deficiencies.
Unlike for C99, the consensus at the London meeting was that there should be no invention, without exception.

I think these bullets are saying the same thing and we only need to say this once if we wish to say it at all. However, I do not think the committee agrees with the idea of no invention without exception and we do everyone a disservice by stating it as a principle. For starters, we’ve shown we’re perfectly comfortable with invention during the C11 and C23 cycles. Invention is necessary if we want to unify divergent existing practice in the wild into something standardized. “No new invention” is why we still have no solution for a “pointer + size” type or a “string” type. There are a lot of different ways to solve those kinds of problems and until the committee picks one and says “This is The Way”, we’ll continue to see more ad hoc solutions.

That said, I think there are really important points hiding in these two bullets. For example, having a principled way to separate “need” from “want” is critical to avoid spending too much committee time on work that does not benefit enough users. What makes something a good feature to standardize vs not? How integrated should a feature be (for example, does adding a new arithmetic type to the language also require it be supported in I/O operations in the standard library, etc)? When two implementations support the same notional feature with slightly differing semantics, should the committee use undefined behavior to resolve the conflict so no users have to change code, or should the committee push for well-defined behavior despite knowing it will break some users? I think we want to have answers to these sorts of questions, but they’re less about charter and more about feature design guidance. Perhaps we want a second document to help authors understand what the committee needs in a proposal and why?

Minimize incompatibilities with C++

C++ inherits the C standard library almost directly from C, but the core C++ language is specified entirely independently of the C language specification. Despite the languages being distinct in terms of specification, every production C++ compiler also claims to be a conforming C compiler. Further, C++ standard library implementations eventually need to use C interfaces to communicate with the host operating system. So I think it is in the best interests of WG21, users of C++, and C++ implementers for WG14 to minimize incompatibilities with C++.

Based on my experience as the former C and C++ compatibility study group chair and from being on both of these committees for about a decade, I think WG14 spends a significantly larger percentage of committee time on C and C++ compatibility than WG21 does. WG14 has a charter that mandates we minimize these incompatibilities (WG21 has no such mandate as they do not have a charter). The WG14 committee spends a considerable amount of meeting time deliberating how changes to C will impact C++ and our convener regularly reminds us of our obligations to minimize incompatibilities where possible. Comparatively, WG21 spends relatively little committee time on C compatibility and WG21 leadership does not proactively attempt to steer the committee to minimize incompatibilities. Given the difference in size between our committees (WG21 is approximate 15x larger than WG14), this is not a tenable situation. The committee with the least resources is spending the most time on compatibility despite reaping the least benefits.

I believe compatibility with C++ should not be a chartered requirement for features brought into C that are present in C++. Without closer cooperation between WG21 and WG14 (such as a shared base document between the two languages), I do not think WG14, C implementers, or C users get sufficient benefit for the amount of WG14 committee effort spent on compatibility with C++. Unfortunately, an inability for the committees to effectively collaborate hurts the people least represented on either committee: users working in this shared space. Therefore, I do believe compatibility with C++ should continue to be a goal as a design principle, especially for features that are likely to be used in header files shared between a mixed C and C++ code base.

Maintain conceptual simplicity.

I still believe in this principle, though it’s a bit too ambiguous for my tastes. What one committee members finds to be simple, another might find to be too complex. For example, is a feature for overloading operators maintaining conceptual simplicity? I suspect asking five committee members that exact question will net about seven different answers.

Trust the programmer, as a goal, is outdated in respect to the security and safety programming communities.

I don’t think this goes far enough. I think “trust the programmer” has lead to decades of security vulnerabilities, to the point that major government agencies are now recommending against using C (and C++). I would prefer we take a stronger stance along the lines of: do not trust the programmer. Design facilities such that their misuse becomes almost impossible outside of pathological code. Prefer “defined” behavior (well-defined, implementation-defined, or unspecified behaviors) to undefined behavior where possible. When misuse is possible, require strong error detection mechanisms which can prevent undefined behavior. I think it’s perfectly reasonable to expect “trust me, I know what I’m doing” users to have to step outside of the language to accomplish their goals and use facilities like inline assembly or implementation extensions.

Application Programming Interfaces (APIs) should be self-documenting when possible.

I agree with the idea, but I don’t agree with the way it is presented regarding preference for VLA interfaces. That’s reasonable guidance for new APIs unrelated to any existing APIs, but it is not defensible when adding new APIs that are related to existing ones. e.g., adding memset_explicit should not (and thankfully did not) rearrange parameter order to fit this guidance, because it would have been very hard for people to migrate existing uses of memset to memset_explicit without introducing memory-related errors. I would prefer we made the guidance more general and less about VLAs.

What’s Missing?

To me, the biggest thing we’re missing is a plan. ISO standards committees are “obligated” to consider all proposals from committee members, and so the approach the committee takes is to take whatever papers arrive and try to give the author feedback so that they can make progress. That said, our obligations do not include a mandate on when to schedule discussion of papers — we could elect to consider papers that aren’t in the committee’s plan only after considering all other in-scope work. But the committee has never gone to those lengths and so we release standards with a hodge podge of unrelated features and bug fixes. I would love to see the committee charter require that we come up with a roadmap of what we want to accomplish in a given release cycle and then stick to that road map. I don’t mind considering occasional out-of-scope work during the cycle, but that should not come at the expense of in-scope work. I think having a roadmap will also help the relationship between the committee, implementers, and users because it gives everyone more time to plan for what’s coming in the next few years.

The #1 thing I want to see on that roadmap is a goal to address the memory safety issues that plague C. We don’t have to fix everything at once, but we do need to start making inroads into solving the problem.

Closing Thoughts

Regardless of whether others on the committee agree with my current thinking, I think it’s a great thing that committee is reflecting on our charter. I think C is at an inflection point in many ways the should spend time carefully considering what we want the future of the language to look like. I don’t think C or C++ will be irrelevant any time soon, but I do worry they’re heading that direction and will continue to do so without course correction. No programming language is “too big to fail”, not even something as venerable and ubiquitous as C.

The year is 2023 and you’re asking “What’s new in C?” Surely the answer is “absolutely nothing”, right?

Wrong!

C23 will be released in the coming months (likely in early 2024), and this is a whirlwind tour of what changes were made for the latest revision of one of the world’s most popular programming languages.

I originally prepared this talk for NDC Tech Town in Kongsberg, but I was unable to give the talk because a hurricane + poor airline services interrupted my travel plans. Oh well, their loss is your gain! I hope you enjoy the talk.

Errata

• The slide about underlying enumeration types did not make it clear that the choice of the underlying type for the enumeration type is implementation-defined, so the given types in the comments might be incorrect depending on your implementation.

• When mentioning that signed overflow is still implementation-defined, I misspoke. I was talking about this situation involving unsigned -> signed conversion:

unsigned int i = UINT_MAX;
signed int j = i; // Implementation-defined behavior

However, signed *overflow* is still undefined behavior, as it always was. e.g.,

signed int i = INT_MAX + 1; // Undefined behavior

C++20 introduced the likelihood attributes [[likely]] and [[unlikely]] as a way for a programmer to give an optimization hint to their implementation that a given code path is more or less likely to be taken. On its face, this seems like a great set of attributes because you can give hints to the optimizer in a way that is hopefully understood by all implementations and will result in faster performance. What’s not to love?

The attribute is specified to appertain to arbitrary statements or labels with the recommended practice “to optimize for the case where paths of execution including it are arbitrarily more likely|unlikely than any alternative path of execution that does not include such an attribute on a statement or label.” Pop quiz, what does this code do?

if (something) {
  [[likely]];
  [[unlikely]];
  foo(something);
}

Sorry, but the answer key for this quiz is currently unavailable. However, one rule you should follow about how to use these attributes is: never allow both attributes to appear in the same path of execution. Lest you think, “but who would write such bad code?”, consider this reasonable-looking-but-probably-very-unfortunate code:

#define MY_COOL_ASSERT(x) [[unlikely]] assert(x)

if (something) {
  [[likely]];
  MY_COOL_ASSERT(something > 0);
  foo(something);
}

Despite the name, these attributes do not mark whether the statement itself is likely, only whether the path leading to the statement is likely. This brings us to our second rule: only mark the dominating statement or label of the flow control path you want to optimize for. This will often mean you only mark the compound statement after a flow control statement, as in:

if (foo) [[likely]] {
  do_something(foo);
}

while (bar) [[unlikely]] {
  ;
}

switch (baz) {
[[likely]] case 0: whatever(); break;
[[unlikely]] case 1: something_else(); break;
default: break;
}

Speaking of code that looks reasonable when you apply the attribute to the dominating statement of flow control, what does this code do?

if (foo) [[likely]] { // A
  baz();
} else if (bar) [[likely]] { // B
  quux();
} else [[unlikely]] { // C
  bing();
}

It marks the true branch at A as being likely and says nothing about the false (else) branch (making it unlikely by default). It then marks the true (if) branch at B as being likely and has a redundant (but harmless) [[unlikely]] attribute at C. What it does not do is mark that A and B are equally likely and C is unlikely; it will optimize the path for A over B rather than treat them as equals. The issue is that the attribute is not written on the dominating statement of flow control, and the code should be written as:

if (foo) [[likely]] { // A
  baz();
} else [[likely]] if (bar) [[likely]] { // B
  quux();
} else [[unlikely]] { // C
  bing();
}

However, because of the duplicate likely branches at A and B (on the else), it’s not clear what the implementation will do with the construct from reading the code (not to mention that it’s super ugly and unintuitive code). Because of this, the initial rule should be augmented to be: Never allow multiple likelihood attributes to appear in the same path of execution. This sort of confusion comes up in other places as well. Pop quiz, which cases are likely and which cases are unlikely in the following?

switch (foo) {
case 0:
[[likely]] case 1:
[[unlikely]] case 2: bar(); break;
[[likely]] case 3:
default: baz(); break;

Sorry, I still can’t find the answer key. Given that case 1 looks to be likely, but it falls through to case 2 which looks to be unlikely, it’s hard to say what should happen here. Further, it’s hard to say whether the default case is likely given that case 3 is likely. The only unambiguous cases are that case 0 say nothing about whether it is or isn’t likely and case 3 is likely. Unfortunately, the wording from the standard leaves a bit to be desired when considering switch statements because it says “A path of execution includes a label if and only if it contains a jump to that label.” A switch statement contains a path of execution which can jump to any of its labels, so when you couple this recommended practice with the earlier one about applying to arbitrary statements, you have to work to answer whether this code path is likely, unlikely, or something else:

if (foo) { // is this branch likely or unlikely?
  switch (*foo) {
  [[likely]] case 0: bar(); break;
  [[unlikely]] case 1: baz(); break;
  [[likely]] default: quux(); break;
  }
} else {
  ...
}

Now, a sensible person would look at this and say “aha, those attributes shouldn’t impact the if statement because they’re within a different control flow statement with its own substatements.” Well, unfortunately the standard doesn’t say anything about how recursive these attributes should be treated. For instance, one would certainly hope that an implementation allowing attributes on arbitrary statements would do something reasonable with this:

if (foo) { // Is this path likely?
  {
    [[likely]];
    SomeRAIIObject Obj;
    Obj.whatever(foo);
  }
} else {
  ...
}

The standard also doesn’t say what happens when you follow my rule to only mark the dominating statement or label and that leads to a conflict like this (with thanks to Arthur O’Dwyer for the example):

if (ch == ' ') [[likely]] {
  goto whitespace;  // A
} else if (ch == '\n' || ch == '\t') [[unlikely]] {
  goto whitespace;  // B
} else {
  foo();
}
[[likely]] whitespace: bar();  // C

The [[likely]] attribute at C says the path at both A and B are likely, despite the path of B being marked as unlikely. Which attributes, if any, are ignored? Who knows — in all of these circumstances, the standard says nothing and so implementations will likely come up with different answers to different situations. This brings me to the next rule: assume no two implementations will behave the same way for optimizing using these attributes.

So given all of these odd issues with the attributes, why would you want to use them? In my mind, there are only two use cases for the likelihood attributes. Either you have an implementation which does not support profile-guided optimizations (which will generally do a far better job of predicting branch weights for optimization than a programmer ever could) or you need to optimize a code path in a strange way where you cannot use PGO. The first question is the easier one to address: can you point to a C++20 implementation that doesn’t support profile-guided optimizations? I can’t find one. Maybe these implementations really do exist, but the major vendors all support the concept, so this isn’t a very compelling argument for adding the attributes to your own code unless you’re in that situation. That is why my rule is: prefer profile-guided optimization over likelihood attributes. It is more suited to the purpose of optimizing flow control and is likely to result in better performing code.

The second situation is more interesting to talk about because it seems off-the-wall until you understand it. Sometimes you want to optimize the failure path that almost never gets hit rather than the common paths that do. Consider writing some safety-critical piece of code to control an elevator where you need the failure path to meet some real-time obligations in order to stop the elevator from dashing its occupants to death. In that situation, your optimization needs can’t be met by PGO and the likelihood attributes could be very useful. Consider this use case which came up during the standards discussions about the feature:

try {
  foo();
} catch (...) [[likely]] {
  dont_kill_people();
}

This is an attempt to convince the optimizer to optimize the catch statement control flow path, but it has three problems that may not be obvious from looking at the code. The first problem is a small one, the attribute is misnamed for its use in this case, which makes the code far harder to read than it needs to be. The second problem is that the C++ grammar doesn’t allow you to write the attribute at that position! You’d have to put the [[likely]] attribute inside of the catch block’s compound statement. The final problem is: implementations typically have no idea how to optimize the failure path for C++ exceptions. So these attributes failed to address the intended need in this circumstance, which is another rule: not all flow control paths can be optimized. Exception handling, setjmp/longjmp, the branches in a ?: operator are all examples of flow control where the likelihood attributes either cannot be written or may look like they’ll do something useful, but likely won’t (pun totally intended).

Let’s review the rules we’ve got so far:

0) Never allow multiple likelihood attributes to appear in the same path of execution.
1) Only mark the dominating statement or label of the flow control path you want to optimize for.
2) Assume no two implementations will behave the same way for optimization behaviors with these attributes.
3) Prefer profile-guided optimization over likelihood attributes.
4) Not all flow control paths can be optimized.

These attributes are starting to look a bit more like some other code constructs we’ve seen in the past: the register keyword as an optimization hint to put things in registers and the inline keyword as an optimization hint to inline function bodies into the call site. Using register or inline for these purposes is often strongly discouraged because experience has shown that optimizer implementations eventually improved to the point where they were consistently better at optimizing than the user trying to give their own hints. However, at least the register and inline keywords have other semantic impact (like not being able to take the address of a register variable in C). The likelihood attributes have no semantic impact beyond their optimization hints. Given how hard it is to use these attributes properly (especially if the code is being compiled by multiple implementations), how good profile-guided optimization is by comparison, and that there is no semantic impact from the attribute, my recommendation is to never use the likelihood attributes. They’re just not worth it.

Because of the lock-down in France and thanks to Lucas, I have been able to make some progress rebuilding Debian with clang instead of gcc.

TLDR

Instead of patching clang itself, I used a different approach this time: patching Debian tools or implementing some workaround to mitigate an issue.
The percentage of packages failing drop from 4.5% to 3.6% (1400 packages to 1110 - on a total of 31014).

I focused on two classes of issues:

• The qmake bug - 250 issues - https://clang.debian.net/status.php?version=9.0.1&key=FAILED_PARSE_DEFAULT
  Basically, Qmake refuses to build a package when gcc's binary is replaced by clang

• Different symbols generated in the libraries - 137 issues - https://clang.debian.net/status.php?version=9.0.1&key=CHANGE_SYM_LIB
  Clang and gcc differ in term of symbols generated in the libraries. While there isn't any impact on the execution or linking, the Debian tooling doesn't have the capability to manage the two cases.
  For more details, see https://bugs.llvm.org/show_bug.cgi?id=30441 & https://bugs.llvm.org/show_bug.cgi?id=45322.

Qmake

As I have no intention to merge the patch upstream, I used a very dirty workaround. I overwrote the g++ qmake file by clang's:
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/clang10#L44-47

I dropped the number of this failure to 0, making some packages build flawlessly (example: qtcreator, chessx, fwbuilder, etc).

However, some packages are still failing later and therefore increasing the number of failures in some other categories like link error. For example, qtads fails because of ordered comparison between pointer and zero or oscar fails on a -Werror,-Wdeprecated-copy error.

Breaking the build later also highlighted some new classes of issues which didn't occur with clang < 10.
For example, warnings related to C++ range loop or implicit int float conversion (I fixed a bunch of them in Firefox) .

Symbol differences

Historically, symbol management for C++ in Debian has been a pain. Russ Allbery wrote a blog post in 2012 explaining the situation. AFAIK, it hasn't changed much.
Once more, I took the dirty approach: if there new or missing symbols, don't fail the build.
The rational is the following: Packages in the Debian archive are supposed to build without any issue. If there is new or missing symbols, it is probably clang generating a different library but this library is very likely working as expected (and usable by a program compiled with g++ or clang). It is purely a different approach taken by the compiler developer.

In order to mitigate this issue, before the build starts, I am modifying dpkg-gensymbols to transform the error into a warning.
So, the typical Debian error some new symbols appeared in the symbols file or some symbols or patterns disappeared in the symbols file will NOT fail the build.

Unsurprisingly, all but one package (libktorrent) build.

Even if I am pessimistic, I reported a bug on dpkg-dev to evaluate if we could improve dpkg-gensymbol not to fail on these cases.

Next steps

The next offender is Imake.tmpl:2243:10: fatal error: ' X11 .rules' file not found with more than an hundred occurrences, reported upstream quite sometime ago.

Then, the big issues are going to be much harder to fix as they are real issues/warnings (with -Werror) in the code of the packages. Example: -Wc++11-narrowing & Wreserved-user-defined-literal... The list is long.
I will probably work on that when llvm/clang 11 are in RC phase.

For maintainers & upstream

Maintainer of Debian/Ubuntu packages? I am providing a list of failing packages per maintainer: https://clang.debian.net/maintainers.php
For upstream, it is also easy to test with clang. Usually, apt install clang && CC=clang CXX=clang++ <build step> is good enough.

Conclusion

With these two changes, I have been able to fix about 290 packages. I think I will be able to get that down a bit more but we will soon reach a plateau as many warnings/issues will have to fix in the C/C++ code itself.

Motivation

Using the _ExtInt Language Feature

History

Future Extensions

Conclusion

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions.

Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs

Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results

Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:

Errors found

The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the compiler.

The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.
I reported this bug upstream a while back: https://bugs.llvm.org/show_bug.cgi?id=30441

Current status

As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps

I see some potential next steps to decrease the number of failure:

• Workaround the Qt/Qmake issue
• Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
• Identify why clang generates more/less symbols that gcc in the library and try to fix that
• Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions.

Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs

Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results  

Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:

Errors found  

The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the compiler.

The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.

Current status  

As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps  

I see some potential next steps to decrease the number of failure:

• Workaround the Qt/Qmake issue
• Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
• Identify why clang generates more/less symbols that gcc in the library and try to fix that
• Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.

• Rust, with its lack of a language runtime and its low-level reach, has an almost unique ability to seamlessly integrate with an existing C/C++ codebase, and
• LLVM, as a language agnostic foundation, provides a common ground where the source language a particular piece of code was written in does not matter anymore.

• From a technical perspective it allows for codebases to be optimized without regard for implementation language boundaries, making it possible for important optimizations, such as function inlining, to be performed across individual compilation units even if, for example, one of the compilation units is written in Rust while the other is written in C++.
• From a psychological perspective, which arguably is just as important, it helps to alleviate the nagging feeling of inefficiency that many performance conscious developers might have when working on a piece of software that jumps back and forth a lot between functions implemented in different source languages.

Background - A bird's eye view of the LLVM compilation pipeline

1. The compiler front-end generates an LLVM bitcode module (.bc) for each compilation unit. In C and C++ each source file will result in a single compilation unit. In Rust each crate is translated into at least one compilation unit.
   

   
       .c --clang--> .bc
   
       .c --clang--> .bc
   
   
       .rs --+
             |
       .rs --+--rustc--> .bc
             |
       .rs --+
   
   

2. In the next step, LLVM's optimization pipeline will optimize each LLVM module in isolation:

   
       .c --clang--> .bc --LLVM--> .bc (opt)
   
       .c --clang--> .bc --LLVM--> .bc (opt)
   
   
       .rs --+
             |
       .rs --+--rustc--> .bc --LLVM--> .bc (opt)
             |
       .rs --+
   
   

3. LLVM then lowers each module into machine code so that we get one object file per module:

   
       .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o
   
       .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o
   
   
       .rs --+
             |
       .rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o
             |
       .rs --+
   
   

4. Finally, the linker will take the set of object files and link them together into a binary:

   
       .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o ------+
                                                                |
       .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o ------+
                                                                |
                                                                +--ld--> bin
       .rs --+                                                  |
             |                                                  |
       .rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o --+
             |
       .rs --+
   
   

Link time optimization in LLVM

• the compiler translates each compilation unit into LLVM bitcode (i.e. it skips lowering to machine code),
   
• the linker, via the LLVM linker plugin, knows how to read LLVM bitcode modules like regular object files, and
   
• the linker, again via the LLVM linker plugin, merges all bitcode modules it encounters and then runs LLVM optimization passes before doing the actual linking.

    .c --clang--> .bc --LLVM--> .bc (opt) ------------------+ - - +
                                                            |     |
    .c --clang--> .bc --LLVM--> .bc (opt) ------------------+ - - +
                                                            |     |
                                                            +-ld+LLVM--> bin
    .rs --+                                                 |
          |                                                 |
    .rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o -+
          |
    .rs --+

Cross-language link time optimization

    .c --clang--> .bc --LLVM--> .bc (opt) ---------+
                                                   |
    .c --clang--> .bc --LLVM--> .bc (opt) ---------+
                                                   |
                                                   +-ld+LLVM--> bin
    .rs --+                                        |
          |                                        |
    .rs --+--rustc--> .bc --LLVM--> .bc (opt) -----+
          |
    .rs --+

• The Rust compiler and Clang are both based on LLVM but they might be using different versions of LLVM. This was further complicated by the fact that Rust's LLVM version often does not match a specific LLVM release, but can be an arbitrary revision from LLVM's repository. We learned that all LLVM versions involved really have to be a close match in order for things to work out. The Rust compiler's documentation now offers a compatibility table for the various versions of Rust and Clang.
   
• The Rust compiler by default performs a special form of LTO, called ThinLTO, on all compilation units of the same crate before passing them on to the linker. We quickly learned, however, that the LLVM linker plugin crashes with a segmentation fault when trying to perform another round of ThinLTO on a module that had already gone through the process. No problem, we thought and instructed the Rust compiler to disable its own ThinLTO pass when compiling for the cross-language case and indeed everything was fine -- until the segmentation faults mysteriously returned a few weeks later even though ThinLTO was still disabled.
  
  We noticed that the problem only occurred in a specific, presumably innocent setting: again two passes of LTO needed to happen, this time the first was a regular LTO pass within rustc and the output of that would then be fed into ThinLTO within the linker plugin. This setup, although computationally expensive, was desirable because it produced faster code and allowed for better dead-code elimination on the Rust side. And in theory it should have worked just fine. Yet somehow rustc produced symbol names that had apparently gone through ThinLTO's mangling even though we checked time and again that ThinLTO was disabled for Rust. We were beginning to seriously question our understanding of LLVM's inner workings as the problem persisted while we slowly ran out of ideas on how to debug this further.
  
  You can picture the proverbial lightbulb appearing over our heads when we figured out that Rust's pre-compiled standard library would still have ThinLTO enabled, no matter the compiler settings we were using for our tests. The standard library, including its LLVM bitcode representation, is compiled as part of Rust's binary distribution so it is always compiled with the settings from Rust's build servers. Our local full LTO pass within rustc would then pull this troublesome bitcode into the output module which in turn would make the linker plugin crash again. Since then ThinLTO is turned off for libstd by default.
   
• After the above fixes, we succeeded in compiling the entirety of Firefox with cross-language LTO enabled. Unfortunately, we discovered that no actual cross-language optimizations were happening. Both Clang and rustc were producing LLVM bitcode and LLD produced functioning Firefox binaries, but when looking at the machine code, not even trivial functions were being inlined across language boundaries. After days of debugging (and unfortunately without being aware of LLVM's optimization remarks at the time) it turned out that Clang was emitting a target-cpu attribute on all functions while rustc didn't, which made LLVM reject inlining opportunities.
  
  In order to prevent the feature from silently regressing for similar reasons in the future we put quite a bit of effort into extending the Rust compiler's testing framework and CI. It is now able to compile and run a compatible version of Clang and uses that to perform end-to-end tests of cross-language LTO, making sure that small functions will indeed get inlined across language boundaries.

Using cross-language LTO: a minimal example

    # Compile the Rust static library, called "xyz"
    rustc --crate-type=staticlib -O -C linker-plugin-lto -o libxyz.a lib.rs

    # Compile the C code with "-flto"
    clang -flto -c -O2 main.c

    # Link everything
    clang -flto -O2 main.o -L . -lxyz

Conclusion

Acknowledgments

• Generating Optimized Code with GlobalISel - Volkan Keles, Daniel Sanders
• Even Better C++ Performance and Productivity: Enhancing Clang to Support Just-in-Time Compilation of Templates- Hal Finkel

• Using LLVM's portable SIMD with Zig - Shawn Landden
• Code-Generation for the Arm M-profile Vector Extension - Sjoerd Meijer
• Alive2: Verifying Existing Optimizations - Nuno Lopes
• The clang constexpr interpreter - Nandor Licker
• Souper-Charging Peepholes with Target Machine Info - Min-Yih Hsu
• Transitioning the Networking Software Toolchain to Clang/LLVM - Ivan Baev, Jeremy Stenglein, Bharathi Seshadri
• Link Time Optimization For Swift - Jin Lin
• Hot Cold Splitting Optimization Pass In LLVM - Aditya Kumar
• Making UB hurt less: security mitigations through automatic variable initialization - JF Bastien
• Propeller: Profile Guided Large Scale Performance Enhancing Relinker - Sriraman Tallam
• From C++ for OpenCL to C++ for accelerator devices - Anastasia Stulova
• LLVM-Canon: Shooting for Clear Diffs - Michal Paszkowski
• Better C++ debugging using Clang Modules in LLDB - Raphael Isemann
• Ownership SSA and Semantic SIL - Michael Gottesman
• arm64e: An ABI for Pointer Authentication - Ahmed Bougacha, John McCall
• Porting by a 1000 Patches: Bringing Swift to Windows - Saleem Abdulrasool
• The Penultimate Challange: Constructing bug reports in the Clang Static Analyzer - Kristóf Umann
• Address Spaces in LLVM - Matt Arsenault
• An MLIR Dialect for High-Level Optimization of Fortran - Eric Schweitz
• Loop-transformation #pragmas in the front-end - Michael Kruse
• Optimizing builds on Windows: some practical considerations - Alexandre Ganea
• LLVM-Reduce for testcase reduction - Diego Treviño Ferrer
• Memoro: Scaling an LLVM-based Heap profiler - Thierry Treyer
• The Attributor: A Versatile Inter-procedural Fixpoint Iteration Framework - Johannes Doerfert
• LLVM Tutorials: How to write Beginner-Friendly, Inclusive Tutorials - Meike Baumgärtner
• Maturing an LLVM backend: Lessons learned from the RISC-V target - Alex Bradbury

• Getting Started With LLVM: Basics - Jessica Paquette, Florian Hahn
• ASTImporter: Merging Clang ASTs - Gábor Márton
• Developing the Clang Static Analyzer - Artem Dergachev
• Writing an LLVM Pass: 101 - Andrzej Warzynski
• Writing Loop Optimizations in LLVM - Kit Barton, Ettore Tiotto, Hal Finkel, Michael Kruse, Johannes Doerfert
• The Attributor: A Versatile Inter-procedural Fixpoint Iteration Framework - Johannes Doerfert
• Getting Started with the LLVM Testing Infrastructure - Brian Homerding, Michael Kruse
• An overview of Clang - Sven Van Haastregt, Anastasia Stulova
• An overview of LLVM - Eric Christopher, Sanjoy Das, Johannes Doerfert
• How to Contribute to LLVM - Chris Bieneman, Kit Barton
• My First Clang Warning - Dmitri Gribenko, Meike Baumgartner

• Cross-Translation Unit Optimization via Annotated Headers - William S. Moses
• Quantifying Dataflow Analysis with Gradients in LLVM - Abhishek Shah
• Floating Point Consistency in the Wild: A practical evaluation of how compiler optimizations affect high performance floating point code - Jack J Garzella
• Static Analysis of OpenMP Data Mapping for Target Offloading - Prithayan Barua

• Panel: Inter-procedural Optimization (IPO) - Teresa Johnson, Philip Reames, Chandler Carruth, Johannes Doerfert
• The Loop Optimization Working Group - Kit Barton, Michael Kruse, TBD

• LLDB - Jonas Devlieghere
• Towards Better Code Generator Design and Unification for a Stack Machine - Leonid Kholodov, Dmitry Borisenkov
• Debug Info - Adrian Prantl

• GWP-ASan: Zero-Cost Detection of MEmory Safety Bugs in Production - Matt Morehouse
• When 3 Memory Models Aren’t Enough – OpenVMS on x86 - John Reagan
• FileCheck: learning arithmetic - Thomas Preud'homme
• -Wall Found Programming Errors and Engineering Effort to Enable Across a Large Codebase - Aditya Kumar
• Handling 1000s of OpenCL builtin functions in Clang - Sven van Haastregt
• NEC SX-Aurora as a Scalable Vector Playground - Kazuhisa Ishizaka
• Implementing Machine Code Optimizations for RISC-V - Lewis Revill
• Optimization Remarks Update - Francis Visoiu Mistrih
• Supporting Regular and Thin LTO with a Single LTO Bitcode Format - Matthew Voss
• Transitioning Apple’s Downstream llvm-project Repositories to the Monorepo - Alex Lorenz
• A Unified Debug Server For Deeply Embedded Systems and LLDB - Simon Cook
• State of LLDB and Deeply Embedded RISC-V - Simon Cook
• Supporting a Vendor ABI Variant in Clang - Paul Robinson
• Speculative Compilation in ORC JIT - Praveen Velliengiri
• Optimization Remarks for Human Beings - William Bundy
• Improving the Optimized Debugging Experience - Orlando Cazalet-Hyams
• Improving your TableGen Descriptions - Javed Absar
• Loom: Weaving Instrumentation for Program Analysis - Brian Kidney
• Clang Interface Stubs: Syntax Directed Stub Library Generation. - Puyan Lotfi
• Flang Update - Steve Scalpone
• Lowering tale: Supporting 64 bit pointers in RISCV 32 bit LLVM backend - Reshabh Sharma
• Virtual Function Elimination in LLVM - Oliver Stannard
• Making a Language Cross Platform: Libraries and Tooling - Gwen Mittertreiner
• Grafter - A use case to implement an embedded DSL in C++ and perform source to source traversal fusion transformation using Clang - Laith Sakka

• TON Labs Backend for TON Blockchain - Dmitry Borisenkov, Dmitry Shtukenberg, Leonid Kholodov
• LLVM Build Times Using a Program Repository - Rusell Gallop, Phil Camp
• RISC-V Bit Manipulation Support in the Clang/LLVM Toolchain - Scott Egerton, Paolo Savini
• Attributor, a Framework for Interprocedural Information Deduction - Johannes Doerfert, Hideto Ueno, Stefan Stipanovic
• Overflows Be Gone: Checked C for Memory Safety - Mandeep Singh Grang
• Cross-Translation Unit Optimization via Annotated Headers - William S. Moses
• Quantifying Dataflow Analysis with Gradients in LLVM - Abhishek Shah
• Floating Point Consistency in the Wild: A practical evaluation of how compiler optimizations affect high performance floating point code - Jack J Garzella
• Static Analysis of OpenMP Data Mapping for Target Offloading - Prithayan Barua
• NEC SX-Aurora as a Scalable Vector Playground - Kazuhisa Ishizaka
• A Unified Debug Server For Deeply Embedded Systems and LLDB - Simon Cook
• Speculative Compilation in ORC JIT - Praveen Velliengiri
• Loom: Weaving Instrumentation for Program Analysis - Brian Kidney
• Lowering tale: Supporting 64 bit pointers in RISCV 32 bit LLVM backend - Reshabh Sharma

The LLVM Project is Moving to GitHub

• Include Prior Experience: Do you have prior technical writing experience? We want to see this! Considering including links to prior documentation or attachments of documentation you have written. If you can’t include a link to the actual documentation, please describe in detail what you wrote, who the audience was, and any other important information that can help us gauge your prior experience. Please also include any experience with Sphinx or other documentation generation tools.
• Take your time writing the proposal: We will be looking closely at your application to see how well it is written. Take the time to proofread and know who your audience is.
• Propose your plan for our documentation project: We have given a rough idea of what changes or topics we envision for the documentation, but this is just a start. We expect you to take the idea and expand or modify it as you see fit. Review our existing documentation and see how it would compliment or replace other pieces. Optionally include an overview or document design or layout plan in your application.
• Become familiar with our project: We don’t expect you to become a compiler expert, but we do expect you read up on our project to learn a bit about LLVM.

• MLIR: Multi-Level Intermediate Representation for Compiler Infrastructure Tatiana Shpeisman (Google), Chris Lattner (Google)

• Switching a Linux distribution's main toolchains to LLVM/Clang Bernhard Rosenkränzer (Linaro, OpenMandriva, LinDev)
• Just compile it: High-level programming on the GPU with Julia Tim Besard (Ghent University)
• The Future of AST Matcher-based Refactoring Stephen Kelly
• A compiler approach to Cyber-Security François de Ferrière (STMicroelectronics)
• Compiler Optimizations for (OpenMP) Target Offloading to GPUs Johannes Doerfert (Argonne National Laboratory), Hal Finkel (Argonne National Laboratory)
• Handling massive concurrency: Development of a programming model for GPU and CPU Matthias Liedtke (SAP)
• Automated GPU Kernel Fusion with XLA Thomas Joerg (Google)
• The Helium Haskell compiler and its new LLVM backend Ivo Gabe de Wolff (University of Utrecht)
• Testing and Qualification of Optimizing Compilers for Functional Safety José Luis March Cabrelles (Solid Sands)
• Improving Debug Information in LLVM to Recover Optimized-out Function Parameters Nikola Prica (RT-RK), Djordje Todorovic (RT-RK), Ananthakrishna Sowda (CISCO), Ivan Baev (CISCO)
• LLVM IR in GraalVM: Multi-Level, Polyglot Debugging with Sulong Jacob Kreindl (Johannes Kepler University Linz)
• LLDB Reproducers Jonas Devlieghere (Apple)
• Sulong: An experience report of using the "other end" of LLVM in GraalVM. Roland Schatz (Oracle Labs), Josef Eisl (Oracle Labs)
• SYCL compiler: zero-cost abstraction and type safety for heterogeneous computing Andrew Savonichev (Intel)
• Handling all Facebook requests with JITed C++ code Huapeng Zhou (Facebook), Yuhan Guo (Facebook)
• clang-scan-deps: Fast dependency scanning for explicit modules Alex Lorenz (Apple), Michael Spencer (Apple)
• Clang tools for implementing cryptographic protocols like OTRv4 Sofia Celi (Centro de Autonomia Digital)
• Implementing the C++ Core Guidelines'; Lifetime Safety Profile in Clang Gabor Horvath (Eotvos Lorand University), Matthias Gehre (Silexica GmbH), Herb Sutter (Microsoft)
• Changes to the C++ standard library for C++20 Marshall Clow (CppAlliance)
• Adventures with RISC-V Vectors and LLVM Robin Kruppe (TU Darmstadt), Roger Espasa (Esperanto Technologies)
• A Tale of Two ABIs: ILP32 on AArch64 Tim Northover (Apple)
• LLVM Numerics Improvements Michael Berg (Apple), Steve Canon (Apple)
• DOE Proxy Apps: Compiler Performance Analysis and Optimistic Annotation Exploration Brian Homerding (Argonne National Laboratory), Johannes Doerfert (Argonne National Laboratory)
• Loop Fusion, Loop Distribution and their Place in the Loop Optimization Pipeline Kit Barton (IBM), Johannes Doerfert (Argonne National Lab), Hal Finkel (Argonne National Lab), Michael Kruse (Argonne National Lab)

• Tutorial: Building a Compiler with MLIR Amini Mehdi (Google), Jacques Pienaar (Google), Nicolas Vasilache (Google)
• Building an LLVM-based tool: lessons learned Alex Denisov
• LLVM IR Tutorial - Phis, GEPs and other things, oh my! Vince Bridgers (Intel Corporation), Felipe de Azevedo Piovezan (Intel Corporation)

• Safely Optimizing Casts between Pointers and Integers Juneyoung Lee (Seoul National University, Korea), Chung-Kil Hur (Seoul National University, Korea), Ralf Jung (MPI-SWS, Germany), Zhengyang Liu (University of Utah, USA), John Regehr (University of Utah, USA), Nuno P. Lopes (Microsoft Research, UK)
• An alternative OpenMP Backend for Polly Michael Halkenhäuser (TU Darmstadt)
• Implementing SPMD control flow in LLVM using reconverging CFGs Fabian Wahlster (Technische Universität München), Nicolai Hähnle (Advanced Micro Devices)
• Function Merging by Sequence Alignment Rodrigo Rocha (University of Edinburgh), Pavlos Petoumenos (University of Edinburgh), Zheng Wang (Lancaster University), Murray Cole (University of Edinburgh), Hugh Leather (University of Edinburgh)
• Compilation and optimization with security annotations Son Tuan Vu (LIP6), Karine Heydemann (LIP6), Arnaud de Grandmaison (ARM), Albert Cohen (Google)
• Adding support for C++ contracts to Clang Javier López-Gómez (University Carlos III of Madrid), J. Daniel García (University Carlos III of Madrid)

• LLVM IR Timing Predictions: Fast Explorations via lli Alessandro Cornaglia (FZI - Research Center for Information Technology)
• Simple Outer-Loop-Vectorization == LoopUnroll-And-Jam + SLP Dibyendu Das (AMD)
• Clacc 2019: An Update on OpenACC Support for Clang and LLVM Joel E. Denny (Oak Ridge National Laboratory), Seyong Lee (Oak Ridge National Laboratory), Jeffrey S. Vetter (Oak Ridge National Laboratory)
• Targeting a statically compiled program repository with LLVM Phil Camp (SN Systems (Sony Interactive Entertainment)), Russell Gallop (SN Systems (Sony Interactive Entertainment))
• Does the win32 clang compiler executable really need to be over 21MB in size? Russell Gallop (SN Systems), Greg Bedwell (SN Systems)
• Resolving the almost decade old checker dependency issue in the Clang Static Analyzer Kristóf Umann (Ericsson Hungary, Eötvös Loránd University)
• Adopting LLVM Binary Utilities in Toolchains Jordan Rupprecht (Google)
• Multiplication and Division in the Range-Based Constraint Manager Ádám Balogh (Ericsson Hungary Ltd.)
• Statistics Based Checkers in the Clang Static Analyzer Ádám Balogh (Ericsson Hungary Ltd.)
• Flang Update Steve Scalpone (NVIDA / PGI / Flang)
• Swinging Modulo Scheduling together with Register Allocation Lama Saba (Intel)
• LLVM for the Apollo Guidance Computer Lewis Revill (University of Bath)
• Catch dangling inner pointers with the Clang Static Analyzer Réka Kovács (Eötvös Loränd University)
• Cross translation unit test case reduction Réka Kovács (Eötvös Loränd University)

• RFC: Towards Vector Predication in LLVM IR Simon Moll (Saarland University), Sebastian Hack (Saarland University)
• IPO --- Where are we, where do we want to go? Johannes Doerfert (Argonne National Laboratory), Kit Barton (IBM Toronto Lab)
• LLVM binutils James Henderson (SN Systems (Sony Interactive Entertainment)), Jordan Rupprecht (Google)
• RFC: Reference OpenCL Runtime library for LLVM Andrew Savonichev (Intel), Alexey Sachkov (Intel)
• LLVM Interface Stability Guarantees BoF Stephen Kelly
• Clang Static Analyzer BoF Devin Coughlin (Apple), Gabor Horvath (Eotvos Lorand University)
• LLVM Numerics Improvements Michael Berg (Apple), Steve Canon (Apple)

• Clava: C/C++ source-to-source from CMake using LARA João Bispo (FEUP/INESCTEC)
• Safely Optimizing Casts between Pointers and Integers Juneyoung Lee (Seoul National University, Korea), Chung-Kil Hur (Seoul National University, Korea), Ralf Jung (MPI-SWS, Germany), Zhengyang Liu (University of Utah, USA), John Regehr (University of Utah, USA), Nuno P. Lopes (Microsoft Research, UK)
• Scalar Evolution Canon: Click! Canonicalize SCEV and validate it by Z3 SMT solver! Lin-Ya Yu (Xilinx), Alexandre Isoard (Xilinx)
• Splendid GVN: Partial Redundancy Elimination for Algebraic Simplification Li-An Her (National Tsing Hua University), Jenq-Kuen Lee (National Tsing Hua University)
• An alternative OpenMP Backend for Polly Michael Halkenhäuser (TU Darmstadt)
• Does the win32 clang compiler executable really need to be over 21MB in size? Russell Gallop (SN Systems), G Bedwell (SN Systems)
• Enabling Multi- and Cross-Language Verification with LLVM Zvonimir Rakamaric (University of Utah)
• Instruction Tracing and dynamic codegen analysis to identify unique llvm performance issues. Biplob (IBM)
• Handling all Facebook requests with JITed C++ code Huapeng Zhou (Facebook), Yuhan Guo (Facebook)
• Implementing SPMD control flow in LLVM using reconverging CFGs Fabian Wahlster (Technische Universität München), Nicolai Hähnle (Advanced Micro Devices)
• LLVM for the Apollo Guidance Computer Lewis Revill (University of Bath)
• LLVM Miner: Text Analytics based Static Knowledge Extractor Hameeza Ahmed (NED University of Engineering and Technology), Muhammad Ali Ismail (NED University of Engineering and Technology)
• Function Merging by Sequence Alignment Rodrigo Rocha (University of Edinburgh), Pavlos Petoumenos (University of Edinburgh), Zheng Wang (Lancaster University), Murray Cole (University of Edinburgh), Hugh Leather (University of Edinburgh)
• Compilation and optimization with security annotations Son Tuan Vu (LIP6), Karine Heydemann (LIP6), Arnaud de Grandmaison (ARM), Albert Cohen (Google)
• Cross translation unit test case reduction Réka Kovács (Eötvös Loränd University)
• Leveraging Polyhedral Compilation in Chapel Compiler Sahil Yerawar (IIT Hyderabad), Siddharth Bhat (IIIT Hyderabad), Michael Ferguson (Cray Inc.), Philip Pfaffe (Karlsruhe Institute of Technology), Ramakrishna Upadrasta (IIT Hyderabad)
• LLVM on AVR - textual IR as a powerful tool for making "impossible" compilers Carl Peto (Swift for Arduino/Petosoft)
• Vectorizing Add/Sub Expressions with SLP Vasileios Porpodas (Intel Corporation, USA), Rodrigo C. O. Rocha (University of Edinburgh, UK), Evgueni Brevnov (Intel Corporation, USA), Luís F. W. Góes (PUC Minas, Brazil), Timothy Mattson (Intel Corporation, USA)
• Adding support for C++ contracts to Clang Javier López-Gómez (University Carlos III of Madrid), J. Daniel García (University Carlos III of Madrid)
• Optimizing Nondeterminacy: Exploiting Race Conditions in Parallel Programs William S. Moses (MIT CSAIL)

• Should specialization be applied at the call level for edges in a call graph where the caller has special context to extend into the callee wrt to flags?
• Should the inliner apply something similar to calls that meet inlining criteria?
• What other part(s) of the compiler could make use of IR flags that are currently not covered?
• What work needs to be done regarding code debt wrt current areas of implementation.

• Roll your own compiler with LLVM (Kai Nacke)
• Rewriting Pointer Dereferences in bcc with Clang (Paul Chaignon)
• Building an LLVM-based tool (Alex Denisov)
• Debug info in optimized code - how far can we go? (Nikola Prica, Djordje Todorovic)
• Lessons in TableGen (Nicolai Hähnle)
• LLVM for the Apollo Guidance Computer (Lewis Revill)
• llvm.mix Multi-stage compiler-assisted specializer generator built on LLVM ( Eugene Sharygin)
• SMT-Based Refutation of Spurious Bug Reports in the Clang Static Analyzer (Mikhail Gadelha)
• What makes LLD so fast? (Peter Smith)
• Compiling the Linux kernel with LLVM tools (Nick Desaulniers and Bill Wendling)
• It was working yesterday! Investigating regressions with llvmlab bisect (Leandro Nunes)

Background

In the course of adding Microsoft Visual C++ (MSVC) compatible Windows support to Clang, we worked hard to make sure the dllexport and dllimport declspecs are handled the same way by Clang as by MSVC.

dllexport and dllimport are used to specify what functions and variables should be externally accessible ("exported") from the currently compiled Dynamic-Link Library (DLL), or should be accessed ("imported") from another DLL. In the class declaration below, S::foo() will be exported when building a DLL:

struct __declspec(dllexport) S {
  void foo() {}
};

and code using that DLL would typically see a declaration like this:

struct __declspec(dllimport) S {
  void foo() {}
};

to indicate that the function is defined in and should be accessed from another DLL.

Often the same declaration is used along with a preprocessor macro to flip between dllexport and dllimport, depending on whether a DLL is being built or consumed.

The basic idea of dllexport and dllimport is simple, but the semantics get more complicated as they interact with more facets of the C++ language: templates, inheritance, different kinds of instantiation, redeclarations with different declspecs, and so on. Sometimes the semantics are surprising, but by now we think clang-cl gets most of them right. And as the old maxim goes, once you know the rules well, you can start tactfully breaking them.

One issue with dllexport is that for inline functions such as S::foo() above, the compiler must emit the definition even if it's not used in the translation unit. That's because the DLL must export it, and the compiler cannot know if any other translation unit will provide a definition.

This is very inefficient. A dllexported class with inline members in a header file will cause definitions of those members to be emitted in every translation unit that includes the header, directly or indirectly. And as we know, C++ source files often end up including a lot of headers. This behaviour is also different from non-Windows systems, where inline function definitions are not emitted unless they're used, even in shared objects and dynamic libraries.

/Zc:dllexportInlines-

To address this problem, clang-cl recently gained a new command-line flag, /Zc:dllexportInlines- (MSVC uses the /Zc: prefix for language conformance options). The basic idea is simple: since the definition of an inline function is available along with its declaration, it's not necessary to import or export it from a DLL — the inline definition can be used directly. The effect of the flag is to not apply class-level dllexport/dllimport declspecs to inline member functions. In the two examples above, it means S::foo() would not be dllexported or dllimported, even though the S class is declared as such.

This is very similar to the -fvisibility-inlines-hidden Clang and GCC flag used on non-Windows. For C++ projects with many inline functions, it can significantly reduce the set of exported functions, and thereby the symbol table and file size of the shared object or dynamic library, as well as program load time.

On Windows however, the main benefit is not having to emit the unused inline function definitions. This means the compiler has to do much less work, and reduces object file size which in turn reduces the work for the linker. For Chrome, we saw 30 % faster full builds, 30 % shorter link times for blink_core.dll, and 40 % smaller total .obj file size.

The reduction in .obj file size, combined with the enormous reduction in .lib files allowed by previously switching linkers to lld-link which uses thin archives, means that a typical Chrome build directory is now 60 % smaller than it would have been just a year ago.

(Some of the same benefit can be had without this flag if the dllexport inline function comes from a pre-compiled header (PCH) file. In that case, the definition will be emitted in the object file when building the PCH, and so is not emitted elsewhere unless it's used.)

Compatibility

Using /Zc:dllexportInlines- is "half ABI incompatible". If it's used to build a DLL, inline members will no longer be exported, so any code using the DLL must use the same flag to not dllimport those members. However, the reverse scenario generally works: a DLL compiled without the flag (such as a system DLL built with MSVC) can be referenced from code that uses the flag, meaning that the referencing code will use the inline definitions instead of importing them from the DLL.

Like -fvisibility-inlines-hidden, /Zc:dllexportInlines- breaks the C++ language guarantee that (even an inline) function has a unique address within the program. When using these flags, an inline function will have a different address when used inside the library and outside.

Also, these flags can lead to link errors when inline functions, which would normally be dllimported, refer to internal symbols of a DLL:

void internal();

struct __declspec(dllimport) S {
  void foo() { internal(); }
}

Normally, references to S::foo() would use the definition in the DLL, which also contains the definition of internal(), but when using /Zc:dllexportInlines-, the inline definition of S::foo() is used directly, resulting in a link error since no definition of internal() can be found.

Even worse, if there is an inline definition of internal() containing a static local variable, the program will now refer to a different instance of that variable than in the DLL:

inline int internal() { static int x; return x++; }

struct __declspec(dllimport) S {
  int foo() { return internal(); }
}

This could lead to very subtle bugs. However, since Chrome already uses -fvisibility-inlines-hidden, which has the same potential problem, we believe this is not a common issue.

Summary

/Zc:dllexportInlines- is like -fvisibility-inlines-hidden for DLLs and significantly reduces build times. We're excited that using Clang on Windows allows us to benefit from new features like this.

More information

For more information, see the User's Manual for /Zc:dllexportInlines-.

The flag was added in Clang r346069, which will be part of the Clang 8 release expected in March 2019. It's also available in the Windows Snapshot Build.

Acknowledgements

/Zc:dllexportInlines- was implemented by Takuto Ikuta based on a prototype by Nico Weber.

• Lessons Learned Implementing Common Lisp with LLVM over Six Years - Christian Schafmeister
• Porting Function merging pass to thinlto - Aditya Kumar
• Build Impact of Explicit and C++ Standard Modules - David Blaikie
• Profile Guided Code Layout in LLVM and LLD - Michael Spencer
• Developer Toolchain for the Nintendo Switch - Bob Campbell, Jeff Sirois
• Methods for Maintaining OpenMP Semantics without Being Overly Conservative - Jin Lin, Ernesto Su, Xinmin Tian
• Understanding the performance of code using LLVM's Machine Code Analyzer (llvm-mca) - Andrea Di Biagio, Matt Davis
• Art Class for Dragons: Supporting GPU compilation without metadata hacks! - Neil Hickey
• Implementing an OpenCL compiler for CPU in LLVM - Evgeniy Tyurin
• Working with Standalone Builds of LLVM sub-projects - Tom Stellard
• Loop Transformations in LLVM: The Good, the Bad, and the Ugly - Michael Kruse, Hal Finkel
• Efficiently Implementing Runtime Metadata with LLVM - Joe Groff, Doug Gregor
• Coroutine Representations and ABIs in LLVM - John McCall
• Glow: LLVM-based machine learning compiler - Nadav Rotem, Jakob Olesen
• Graph Program Extraction and Device Partitioning in Swift for TensorFlow - Mingsheng Hong, Chris Lattner
• Memory Tagging, how it improves C++ memory safety, and what does it mean for compiler optimizations - Kostya Serebryany, Evgenii Stepanov, Vlad Tsyrklevich
• Improving code reuse in clang tools with clangmetatool - Daniel Ruoso
• Sound Devirtualization in LLVM - Piotr Padlewski, Krzysztof Pszeniczny
• Extending the SLP vectorizer to support variable vector widths - Vasileios Porpodas, Rodrigo C. O. Rocha, Luís F. W. Góes
• Revisiting Loop Fusion, and its place in the loop transformation framework. - Johannes Doerfert, Kit Barton, Hal Finkel, Michael Kruse
• Optimizing Indirections, using abstractions without remorse. - Johannes Doerfert, Hal Finkel
• Outer Loop Vectorization in LLVM: Current Status and Future Plans - Florian Hahn, Satish Guggilla, Diego Caballero
• Stories from RV: The LLVM vectorization ecosystem - Simon Moll, Matthias Kurtenacker, Sebastian Hack
• Faster, Stronger C++ Analysis with the Clang Static Analyzer - George Karpenkov, Artem Dergachev

• Updating ORC JIT for Concurrency - Lang Hames, Breckin Loggins
• Register Allocation: More than Coloring - Matthias Braun
• How to use LLVM to optimize your parallel programs - William S. Moses
• LLVM backend development by example (RISC-V) - Alex Bradbury

• Debug Info BoF - Vedant Kumar, Adrian Prantl
• Lifecycle of LLVM bug reports - Kristof Beyls, Paul Robinson
• GlobalISel Design and Development - Amara Emerson
• Migrating to C++14, and beyond! - JF Bastien
• Ideal versus Reality: Optimal Parallelism and Offloading Support in LLVM - Xinmin Tian, Hal Finkel, TB Schardl, Johannes Doerfert, Vikram Adve
• Implementing the parallel STL in libc++ - Louis Dionne
• Clang Static Analyzer BoF - Devin Coughlin
• LLVM Foundation BoF - LLVM Foundation Board of Directors

• Automatic Differentiation in C/C++ Using Clang Plugin Infrastructure - Vassil Vassilev, Aleksandr Efremov
• More efficient LLVM devs: 1000x faster build file generation, -j1000 builds, and O(1) test execution - Nico Weber
• Heap-to-Stack Conversion - Hal Finkel
• TWINS - This Workflow is Not Scrum: Adapting Agile for Open Source Interaction - Joshua Magee
• Mutating the clang AST from Plugins - Andrei Homescu, Per Larsen
• atJIT: an online, feedback-directed optimizer for C++ - Kavon Farvardin, Hal Finkel, Michael Kruse, John Reppy
• Repurposing GCC Regression for LLVM Based Tool Chains - Jeremy Bennett, Simon Cook, Ed Jones
• ThinLTO Summaries in JIT Compilation - Stefan Gränitz
• Refuting False Bugs in the Clang Static Analyzer using SMT Solvers - Mikhail R. Gadelha
• What’s New In Outlining - Jessica Paquette
• DWARF v5 Highlights - Why You Care - Paul Robinson, Pavel Labath, Wolfgang Pieb
• Using TAPI to Understand APIs and Speed Up Builds - Steven Wu, Juergen Ributzka
• Hardware Interference Size - JF Bastien
• Dex: efficient symbol index for Clangd - Kirill Bobyrev, Eric Liu, Sam McCall, Ilya Biryukov
• Flang Update - Steve Scalpone
• clang-doc: an elegant generator for more civilized documentation - Julie Hockett
• Code Coverage with CPU Performance Monitoring Unit - Ivan Baev, Bharathi Seshadri, Stefan Pejic
• VecClone Pass: Function Vectorization via LoopVectorizer - Matt Masten, Evgeniy Tyurin, Konstantina Mitropoulou
• ISL Memory Management Using Clang Static Analyzer - Malhar Thakkar, Ramakrishna Upadrasta
• Eliminating always_inline in libc++: a journey of visibility and linkage - Louis Dionne
• Error Handling in Libraries: A Case Study - James Henderson

• Gaining fine-grain control over pass management - serge guelton, adrien guinet, pierrick brunet, juan manuel martinez, béatrice creusillet
• Integration of OpenMP, libcxx and libcxxabi packages into LLVM toolchain - Reshabh Sharma
• Improving Debug Information in LLVM to Recover Optimized-out Function Parameters - Ananthakrishna Sowda, Djordje Todorovic, Nikola Prica, Ivan Baev
• Automatic Compression for LLVM RISC-V - Sameer AbuAsal, Ana Pazos
• Guaranteeing the Correctness of LLVM RISC-V Machine Code with Fuzzing - Jocelyn Wei, Ana Pazos, Mandeep Singh Grang
• NEC SX-Aurora - A Scalable Vector Architecture - Kazuhisa Ishizaka, Kazushi Marukawa, Erich Focht, Simon Moll, Matthias Kurtenacker, Sebastian Hack
• Extending Clang Static Analyzer to enable Cross Translation Unit Analysis - Varun Subramanian
• Leveraging Polyhedral Compilation in Chapel Compiler - Siddharth Bhat, Michael Ferguson, Philip Pfaffe, Sahil Yerawar

The LLVM Foundation is pleased to announce its new Board of Directors:

Two new members and seven continuing members were elected to the nine person board.

We want to thank David Kipping for his 2 terms on the board. David has been actively involved with the LLVM Developer Meetings and was the treasurer for the past 4 years. The treasurer is a time demanding position in that he supports the day to day operation of the foundation, balancing the books, and generates monthly treasurer reports.

We also want to thank all the applicants to the board. When voting on new board members, we took into consideration all contributions (past and present) and current involvement in the LLVM community. We also tried to create a balanced board of individuals from a wide range of backgrounds and locations to provide a voice to as many groups within the LLVM community. Given this criteria and strong applicants, we increased the board from 8 members to 9.

About the board of directors (listed alphabetically by last name):

Chandler Carruth has been an active contributor to LLVM since 2007. Over the years, he has has worked on LLVM’s memory model and atomics, Clang’s C++ support, GCC-compatible driver, initial profile-aware code layout optimization pass, pass manager, IPO infrastructure, and much more. He is the current code owner of inlining and SSA formation.

In addition to his numerous technical contributions, Chandler has led Google’s LLVM efforts since 2010 and shepherded a number of new efforts that have positively and significantly impacted the LLVM project. These new efforts include things such as adding C++ modules to Clang, adding address and other sanitizers to Clang/LLVM, making Clang compatible with MSVC and available to the Windows C++ developer community, and much more.

Chandler works at Google Inc. as a technical lead for their C++ developer platform and has served on the LLVM Foundation board of directors for the last 4 years.

Mike Edwards is a relative newcomer to the LLVM community, beginning his involvement just a few years ago while working for Sony Playstation. Finding the LLVM community to be an incredibly amazing and welcoming group of people, Mike knew he had to find a way to contribute. Mike’s previous work in DevOps led him to get involved in helping to work on the llvm.org infrastructure. Last year, with the help of the Board and several community members, Mike was able to get the llvm.org infrastructure moved onto a modern compute platform at Amazon Web Services. Mike is one of the maintainers of our llvm.org infrastructure.

Mike is currently working as a Software Engineer at Apple, Inc. working on the Continuous Integration and Quality Engineering efforts for LLVM and Clang development.

Hal Finkel has been an active contributor to the LLVM project since 2011. He is the code owner for the PowerPC target, the alias-analysis infrastructure, and other components.

In addition to his numerous technical contributions, Hal has chaired the LLVM in HPC workshop, which is held in conjunction with Super Computing (SC), for the last five years. This workshop provides a venue for the presentation of peer-reviewed HPC-related researching LLVM from both industry and academia. He has also been involved in organizing an LLVM-themed BoF session at SC and LLVM socials in Austin.

Hal is Lead for Compiler Technology and Programming Languages at Argonne National Laboratory’s Leadership Computing Facility.

Arnaud de Grandmaison has been hacking on LLVM projects since 2008. In addition to his open source contributions, he has worked for many years on private out-of-tree LLVM-based projects at Parrot, DiBcom, or Arm. He has also been a leader in the European LLVM community by organizing the EuroLLVM Developers’ meeting, Paris socials, and chaired or participated in numerous program committees for the LLVM Developers’ Meetings and other LLVM related conferences.

Arnaud has attended numerous LLVM Developers’ meetings and volunteered as moderator or presented as well. He also moderates several LLVM mailing lists. Arnaud is also very involved in community wide discussions and decisions such as re-licensing and code of conduct.

Arnaud is a Senior Principal Engineer at Arm.

Anton Korobeynikov has been an active contributor to the LLVM project since 2006. Over the years, he has numerous technical contributions to areas including Windows support, ELF features, debug info, exception handling, and backends such as ARM and x86. He was the original author of the MSP430 and original System Z backend.

In addition to his technical contributions, Anton has maintained LLVM’s participation in Google Summer of Code by managing applications, deadlines, and overall organization. He also supports the LLVM infrastructure and has been on numerous program committees for the LLVM Developers’ Meetings (both US and EuroLLVM).

Anton is currently an associate professor at the Saint Petersburg State University and has served on the LLVM Foundation board of directors for the last 4 years.

Tanya Lattner has been involved in the LLVM project for over 14 years. She began as a graduate student who wrote her master's thesis using LLVM, and continued on using and extending LLVM technologies at various jobs during her career as a compiler engineer.

Tanya has been organizing the US LLVM Developers’ meeting since 2008 and attended every developer meeting. She was the LLVM release manager for 3 years, moderates the LLVM mailing lists, and helps administer the LLVM infrastructure servers, mailing lists, bugzilla, etc. Tanya has also been on the program committee for the US LLVM Developers’ meeting (4+ years) and the EuroLLVM Developers’ Meeting.

With the support of the initial board of directors, Tanya created the LLVM Foundation, defined its charitable and education mission, and worked to get 501(c)(3) status.

Tanya is the Chief Operating Officer and has served as the President of the LLVM Foundation board for the last 4 years.

Chris Lattner is well known as the founder for the LLVM project and has a lengthy history of technical contributions to the project over the years. He drove much of the early implementation, architecture, and design of LLVM and Clang.

Chris has attended every LLVM Developers’ meeting, and presented at many of them. He helped drive the conception and incorporation of the LLVM Foundation, and has served as its secretary. Chris also grants commit access to the LLVM Project, moderates mailing lists, moderates and edits the LLVM blog, and drives important non-technical discussions and policy decisions related to the LLVM project.

Chris manages a team building machine learning infrastructure at Google and has served on the LLVM Foundation board of directors for the last 4 years.

John Regehr has been involved in LLVM for a number of years. As a professor of computer science at the University of Utah, his research specializes in compiler correctness and undefined behavior. He is well known within the LLVM community for the hundreds of bug reports his group has reported to LLVM/Clang.

John was a project lead for IOC, a Clang based integer overflow checker that eventually became the basis for the integer parts of UBSan. He was also the primary developer of C-Reduce which utilizes Clang as a library and is often used as a test case reducer for compiler issues.

In addition to his technical contributions, John has served on several LLVM-related program committees. He also has a widely read blog about LLVM and other compiler-related issues (Embedded in Academia).

Tom Stellard has been contributing to the LLVM project since 2012. He was the original author of the AMDGPU backend and was also an active contributor to libclc. He has been the LLVM project’s stable release manager since 2014.

Tom is currently a Software Engineer at Red Hat and is the technical lead for emerging toolchains including Clang/LLvm. He also maintains the LLVM packages for the Fedora project.

Keynotes

Tutorials

• Pointers, Alias & ModRef Analyses A. Sbirlea, N. Lopes
• Scalar Evolution - Demystified J. Absar

Talks

• A Parallel IR in Real Life: Optimizing OpenMP H. Finkel, J. Doerfert, X. Tian, G. Stelle
• An Introduction to AMD Optimizing C/C++ Compiler A. Team
• Analysis of Executable Size Reduction by LLVM passes V. Sinha, P. Kumar, S. Jain, U. Bora, S. Purini, R. Upadrasta
• Developing Kotlin/Native infrastructure with LLVM/Clang, travel notes. N. Igotti
• Extending LoopVectorize to Support Outer Loop Vectorization Using VPlan D. Caballero, S. Guggilla
• Finding Iterator-related Errors with Clang Static Analyzer Á. Balogh
• Finding Missed Optimizations in LLVM (and other compilers) G. Barany
• Global code completion and architecture of clangd E. Liu, H. Wu, I. Biryukov, S. McCall
• Hardening the Standard Library M. Clow
• Implementing an LLVM based Dynamic Binary Instrumentation framework C. Hubain, C. Tessier
• LLVM Greedy Register Allocator – Improving Region Split Decisions M. Yatsina
• MIR-Canon: Improving Code Diff Through Canonical Transformation. P. Lotfi
• New PM: taming a custom pipeline of Falcon JIT F. Sergeev
• Organising benchmarking LLVM-based compiler: Arm experience E. Astigeevich
• Performance Analysis of Clang on DOE Proxy Apps H. Finkel, B. Homerding
• Point-Free Templates A. Gozillon, P. Keir
• Protecting the code: Control Flow Enforcement Technology O. Simhon

BoFs

Student Research Competition

Posters

Introduction

import pydffi
CU = pydffi.FFI().cdef("int puts(const char* s);");
CU.funcs.puts("hello world!")

import pydffi
pydffi.dlopen("/path/to/libarchive.so")
CU = pydffi.FFI().cdef("#include <archive.h>")
a = funcs.archive_read_new()
assert a
...

$ pip install pydffi

Related work

• libffi does not support the Microsoft x64 ABI under Linux x64. It isn't that trivial to add a new ABI (hand-written ABI, get the ABI right, ...), while a lot of effort have already been put into compilers to get these ABIs right.
• PyCParser only supports a very limited subset of C (no includes, function attributes, ...).

• it uses Clang to parse header files, allowing direct usage of a C library headers without adaptation;
• it support as many calling conventions and function attributes as Clang/LLVM do;
• as a bonus, Clang and LLVM allows on-the-fly compilation of C functions, without relying on the presence of a compiler on the system (you still need the headers of the system's libc thought, or MSVCRT headers under Windows);
• and this is a good way to have fun with Clang and LLVM! :)

Creating an FFI library for C

Supporting C ABIs

typedef struct {
  short a;
  int b;
} A;

void print_A(A s) {
  printf("%d %d\n", s.a, s.b);
}

target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-pc-linux-gnu"

@.str = private unnamed_addr constant [7 x i8] c"%d %d\0A\00", align 1

define void @print_A(i64) local_unnamed_addr {
  %2 = trunc i64 %0 to i32
  %3 = lshr i64 %0, 32
  %4 = trunc i64 %3 to i32
  %5 = shl i32 %2, 16
  %6 = ashr exact i32 %5, 16
  %7 = tail call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([7 x i8], [7 x i8]* @.str, i64 0, i64 0), i32 %6, i32 %4)
  ret void
}

Finding the right type abstraction

• the function types, and their calling convention
• for structures: field offsets and names
• for union/enums: field names (and values)

[...]
|-RecordDecl 0x5561d7f9fc20 <a.c:1:9, line:4:1> line:1:9 struct definition
| |-FieldDecl 0x5561d7ff4750 <line:2:3, col:9> col:9 referenced a 'short'
| `-FieldDecl 0x5561d7ff47b0 <line:3:3, col:7> col:7 referenced b 'int'

target triple = "x86_64-pc-linux-gnu"
%struct.A = type { i16, i32 }
@.str = private unnamed_addr constant [7 x i8] c"%d %d\0A\00", align 1

define void @print_A(i64) local_unnamed_addr !dbg !7 {
  %2 = trunc i64 %0 to i32
  %3 = lshr i64 %0, 32
  %4 = trunc i64 %3 to i32
  tail call void @llvm.dbg.value(metadata i32 %4, i64 0, metadata !18, metadata !19), !dbg !20
  tail call void @llvm.dbg.declare(metadata %struct.A* undef, metadata !18, metadata !21), !dbg !20
  %5 = shl i32 %2, 16, !dbg !22
  %6 = ashr exact i32 %5, 16, !dbg !22
  %7 = tail call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([...] @.str, i64 0, i64 0), i32 %6, i32 %4), !dbg !23
  ret void, !dbg !24
}

[...]
; DISubprogram defines (in our case) a C function, with its full type
!7 = distinct !DISubprogram(name: "print_A", scope: !1, file: !1, line: 6, type: !8, [...], variables: !17)
; This defines the type of our subprogram
!8 = !DISubroutineType(types: !9)
; We have the "original" types used for print_A, with the first one being the
; return type (null => void), and the other ones the arguments (in !10)
!9 = !{null, !10}
!10 = !DIDerivedType(tag: DW_TAG_typedef, name: "A", file: !1, line: 4, baseType: !11)
; This defines our structure, with its various fields
!11 = distinct !DICompositeType(tag: DW_TAG_structure_type, file: !1, line: 1, size: 64, elements: !12)
!12 = !{!13, !15}
; We have here the size and name of the member "a". Offset is 0 (default value)
!13 = !DIDerivedType(tag: DW_TAG_member, name: "a", scope: !11, file: !1, line: 2, baseType: !14, size: 16)
!14 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
; We have here the size, offset and name of the member "b"
!15 = !DIDerivedType(tag: DW_TAG_member, name: "b", scope: !11, file: !1, line: 3, baseType: !16, size: 32, offset: 32)
!16 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
[...]

Internals

• create a type system that gathers only the necessary informations from the metadata tree (we don't need the whole debug informations)
• make the public headers of the DragonFFI library free from any LLVM headers (so that the whole LLVM headers aren't needed to use the library)

DFFI FFI([...]);
// This will declare puts as a function that returns int and takes a const

// char* as an argument. We could also create this function type by hand.
CompilationUnit CU = FFI.cdef("int puts(const char* s);", [...]);
NativeFunc F = CU.getFunction("puts");
const char* s = "hello world!";
void* Args[] = {&s};
int Ret;
F.call(&Ret, Args);

void call_puts(void* Ret, void** Args) {
  *((int*)Ret) = puts((const char*) Args[0]);
}

typedef void(*puts_call_ty)(void*, void**);
puts_call_ty Wrapper = /* pointer to the compiled wrapper function */;
Wrapper(Ret, Args);

void __dffi_wrapper_0(int32_t( __attribute__((cdecl)) *__FPtr)(char *), int32_t *__Ret, void** __Args) {
  *__Ret = (__FPtr)(*((char **)__Args[0]));
}

Issues with Clang

typedef struct {
  short a;
  int b;
} A;

void print_A(A s);

$ clang -S -emit-llvm -g -femit-all-decls -o - a.c |grep print_A |wc -l
0

void __dffi_force_decl_print_A(A s) { }

Python bindings

Project status

• user and developer documentations exist!
• another foreign language is supported (JS? Ruby?)
• the DragonFFI main library API is considered stable
• a non negligible list of tests have been added
• all the things in the TODO file have been done :)

Various ideas for the future

Parse embedded DWARF information

Lightweight debug info?

• on libarchive 3.3.2, DWARF goes from 1.8Mb to 536Kb, for an original binary code size of 735Kb
• on zlib 1.2.11, DWARF goes from 162Kb to 61Kb, for an original binary code size of 99Kb

• debug informations are well supported on every platform nowadays: tools exist to parse them, embed/extract them from binary, and so on
• we already got DWARD and PDB: https://xkcd.com/927/

JIT code from the final language (like Python) to native function code

• arguments are converted from Python to C according to the function type
• the function pointer and wrapper and gathered from DragonFFI
• the final call is made

Reducing DragonFFI library size

• compile DragonFFI, Clang and LLVM using (Thin) LTO, with visibility hidden for both Clang and LLVM. This could have the effect of removing code from Clang/LLVM that isn't used by DragonFFI.
• make DragonFFI more modular: - one core module that only have the parts from CodeGen that deals with ABIs. If the types and function prototypes are defined "by hand" (without DFFI::cdef), that's more or less the only part that is needed (with LLVM obviously) - one optional module that includes the full clang compiler (to provide the DFFI::cdef and DFFI::compile APIs)

Conclusion

Acknowledgments

• Inspire students to get involved with open source, compilers and LLVM
• Give students exposure to real-world software development while getting paid a stipend
• Allow students to do paid work related to their academic pursuits versus getting an unrelated summer job
• Bring new developers into the LLVM project
• Some LLVM bugs get fixed or new features get added

• March 12 (16:00 UTC) - Applications open
• March 27 (16:00 UTC) - Deadline to file your application
• April 23 (16:00 UTC) - Accepted student proposals are announced
• May 14 - Coding begins

Background

• Falcon: An optimizing Java JIT - Philip Reames
• Compiling Android userspace and Linux kernel with LLVM - Stephen Hines, Nick Desaulniers and Greg Hackmann

• Apple LLVM GPU Compiler: Embedded Dragons - Marcello Maggioni and Charu Chandrasekaran
• Bringing link-time optimization to the embedded world: (Thin)LTO with Linker Scripts - Tobias Edler von Koch, Sergei Larin, Shankar Easwaran and Hemant Kulkarni
• Advancing Clangd: Bringing persisted indexing to Clang tooling - Marc-Andre Laperle
• The Further Benefits of Explicit Modularization: Modular Codegen - David Blaikie
• eval() in C++ - Sean Callanan
• Enabling Parallel Computing in Chapel with Clang and LLVM - Michael Ferguson
• Structure-aware fuzzing for Clang and LLVM with libprotobuf-mutator - Kostya Serebryany, Vitaly Buka and Matt Morehouse
• Adding Index‐While‐Building and Refactoring to Clang - Alex Lorenz and Nathan Hawes
• XRay in LLVM: Function Call Tracing and Analysis - Dean Michael Berris
• GlobalISel: Past, Present, and Future - Quentin Colombet and Ahmed Bougacha
• Dominator Trees and incremental updates that transcend time - Jakub Kuderski
• Scale, Robust and Regression-Free Loop Optimizations for Scientific Fortran and Modern C++ - Tobias Grosser and Michael Kruse
• Implementing Swift Generics - Douglas Gregor, Slava Pestov and John McCall
• lld: A Fast, Simple, and Portable Linker - Rui Ueyama
• Vectorizing Loops with VPlan – Current State and Next Steps - Ayal Zaks and Gil Rapaport
• LLVM Compile-Time: Challenges. Improvements. Outlook. 
• - Michael Zolotukhin
• Challenges when building an LLVM bitcode Obfuscator - Serge Guelton, Adrien Guinet, Juan Manuel Martinez and Pierrick Brunet
• Building Your Product Around LLVM Releases - Tom Stellard
• The Type Sanitizer: Free Yourself from -fno-strict-aliasing - Hal Finkel

• Storing Clang data for IDEs and static analysis - Marc-Andre Laperle
• Source-based Code Coverage BoF - Eli Friedman and Vedant Kumar
• Clang Static Analyzer BoF - Devin Coughlin, Artem Dergachev and Anna Zaks
• Co-ordinating RISC-V development in LLVM - Alex Bradbury
• Thoughts and State for Representing Parallelism with Minimal IR Extensions in LLVM - Xinmin Tian, Hal Finkel, Tb Schardl, Johannes Doerfert and Vikram Adve
• BoF - Loop and Accelerator Compilation Using Integer Polyhedra - Tobias Grosser and Hal Finkel
• LLDB Future Directions - Zachary Turner and David Blaikie
• LLVM Foundation - Status and Involvement - LLVM Foundation Board of Directors

• Writing Great Machine Schedulers - Javed Absar and Florian Hahn
• Tutorial: Head First into GlobalISel - Daniel Sanders, Aditya Nandakumar and Justin Bogner
• Welcome to the back-end: The LLVM machine representation. - Matthias Braun

• Porting OpenVMS using LLVM - John Reagan
• Porting LeakSanitizer: A Beginner's Guide - Francis Ricci
• Introsort based sorting function for libc++ - Divya Shanmughan and Aditya Kumar
• Code Size Optimization: Interprocedural Outlining at the IR Level - River Riddle
• ThreadSanitizer APIs for external libraries - Kuba Mracek
• A better shell command-line autocompletion for clang - Yuka Takahashi
• A CMake toolkit for migrating C++ projects to clang’s module system. - Raphael Isemann
• Debugging of optimized code: Extending the lifetime of local variables - Wolfgang Pieb
• An LLVM based Loop Profiler - Shalini Jain, Kamlesh Kumar, Suresh Purini, Dibyendu Das and Ramakrishna Upadrasta
• Compiling cross-toolchains with CMake and runtimes build - Petr Hosek

• VPlan + RV: A Proposal - Simon Moll and Sebastian Hack
• Polyhedral Value & Memory Analysis - Johannes Doerfert and Sebastian Hack
• DLVM: A Compiler Framework for Deep Learning DSLs - Richard Wei, Vikram Adve and Lane Schwartz
• Leveraging LLVM to Optimize Parallel Programs - William Moses
• Exploiting and improving LLVM's data flow analysis using superoptimizer - Jubi Taneja and John Regehr

• Venerable Variadic Vulnerabilities Vanquished - Priyam Biswas, Alessandro Di Federico, Scott A. Carr, Prabhu Rajasekaran, Stijn Volckaert, Yeoul Na, Michael Franz and Mathias Payer
• Extending LLVM’s masked.gather/scatter Intrinsic to Read/write Contiguous Chunks from/to Arbitrary Locations. - Farhana Aleen, Elena Demikhovsky, Hideki Saito, and David Kreitzer
• An LLVM based Loop Profiler - Shalini Jain, Kamlesh Kumar, Suresh Purini, Dibyendu Das and Ramakrishna Upadrasta
• Leveraging Compiler Optimizations to Reduce Runtime Fault Recovery Overhead - Fateme S. Hosseini, Pouya Fotouhi, Chengmo Yang and Guang R. Gao
• Polyhedral Optimizations and transparent GPU offloading for Julia by Polly - Sanjay Srivallabh Singapuram
• Improving debug information in LLVM to recover optimized-out function parameters - Ananth Sowda and Ivan Baev
• Adding Debug Information and Merge Attribute to Merge-Functions LLVM passes - Anmol Paralkar
• ALLVM: LLVM All the Things! - Will Dietz and Vikram Adve
• Project Sulong - Executing LLVM IR on top of a JVM - Matthias Grimmer and Christian Wimmer
• JIT Fuzzing Solver: A LibFuzzer based constraint solver - Daniel Liew, Cristian Cadar and Alastair Donaldson
• Non-determinism in LLVM Code Generation - Mandeep Singh Grang

tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go.

The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years.

Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th).

For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month.

Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures.

Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error.

In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).

In parallel, new warnings and errors showed up in Clang.
This is causing a new set of build failures (especially with the usage of -Werror).

As few examples:
* Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
* Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
* clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
* With the same version, clang can trigger a new error when auto is used in function return type.

Now, as a conclusion, having Debian being built with clang by default is still a long shot.
First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs.

Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects.

Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).

tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go.

The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years.

Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th).

For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month.

Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures.

Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error.

In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).

In parallel, new warnings and errors showed up in Clang.
This is causing a new set of build failures (especially with the usage of -Werror).

As few examples:
* Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
* Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
* clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
* With the same version, clang can trigger a new error when auto is used in function return type.

Now, as a conclusion, having Debian being built with clang by default is still a long shot.
First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs.

Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects.

Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).

1. Make clang-cl emit DWARF debug information on Windows
2. Port LLDB to Windows and teach it about the Windows ABI, which would be significantly easier than teaching Visual Studio and/or WinDbg to be able to interpret DWARF (assuming this is even possible at all, given that everything would have to be done strictly through the Visual Studio / WinDbg extensibility model)

1. Have clang-cl invoke lld automatically
1. clang-cl -fuse-ld=lld -Z7 -MTd hello.cpp
2. Invoke clang-cl and lld separately.
1. clang-cl -c -Z7 -MTd -o hello.obj hello.cpp
2. lld-link -debug hello.obj

News and articles from around the web

On the mailing lists

• Sanjoy Das has written an RFC on strong GC references in LLVM. The motivating case for this proposal is supporting a precise, relocating garbage collector.
  
• LLVM version 3.8.1-final has been tagged.
  
• The Google Summer of Code mid-terms have snuck up on us already. The participating students have posted to the mailing lists with a summary of their work so far:
  
  • Finding and reporting bugs caused by copy and paste
  • Enabling LLVM's self-hosted modules builds using libstdc++
  • Capture tracking improvements
  • Improvement of vectorization process in Polly
  • Better alias analysis by default
  • Interprocedural register allocation
  • Polly as an analysis pass
  • Enabling polyhedral optimisations in Julia
• Vivek Pandya wrote the mailing list seeking advice on adding a new calling convention for interprocedural register allocation. Matthias Braun summarised some follow-up discussion.
  

LLVM commits

• The new representation for control-flow integrity and virtual call metadata has landed. The commit message further details the problems this change addresses. r273729.
  
• The llvm.type.checked.load intrinsic was added. It loads a function pointer from a virtual table pointer using type metadata. r273576.
  
• As part of the work on CFL-AA, interprocedural function summaries were added. These avoid recomputation for many properties of a function. r273219, r273596.
  
• MemorySSA gained new APIs for PHI creation and MemoryAccess creation. r273295.
  
• Metadata attachments are now allowed for declarations. r273336.
  
• A new runtimes directory was added to the LLVM tree. r273620.
  
• LLVM's dynamic loader gained basic support for COFF ARM. r273682.
  

Clang commits

• constexpr if support has been added to Clang. r273602.
  
• clang-tidy has a new modernize-use-emplace check that will replace calls of push_back to emplace_back. r273275.
  
• The CMake build system for Clang gained a ENABLE_X86_RELAX_RELOCATIONS option. r273224.
  

Other project commits

• Basic support for versioned symbols was added to LLD. r273143.
  
• LLD now handles both single and double dashes for all options. r273256.
  

• either the class or virtual method is marked as final
• the class is defined in an anonymous namespace and has no deriving classes in its translation unit

• give up immediate generation
• run data flow analysis in LLVM to find all the dynamic types passed to function, which has static linkage
• hope that every use of the virtual function, which is necessarily in the same translation unit, will be inlined by LLVM -- static linkage increases the chances of inlining

• value of vptr - which virtual table is pointed by it
• value of vtable slot - which exact virtual function it is

Bitcode Compatibility

API Compatibility

What About the Minor Version?

Welcome to the one hundred and twenty-ninth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

Last week was WWDC, which featured talks on what's new in LLVM (slides) and what's new in Swift (slides). Note that the embedded video player suggests you need Safari or the WWDC app to stream the video, but you can find a downloadable version under the "resources" tab.

On the mailing lists

• Jason Henline has announced the LLVM parallel-libs subproject which will "host the development of libraries which are aimed at enabling parallelism in code and which are also closely tied to compiler technology.Examples of libraries suitable for hosting within the parallel-libs subproject are runtime libraries and parallel math libraries. The initial candidates for inclusion in this subproject are StreamExecutor and libomptarget which would live in the streamexecutorand libomptarget subdirectories of parallel-libs, respectively."

• One of the most active threads this week was about whether the release following 3.9 should be 4.0. Much of the discussion was around whether the move from 3.9 to 4.0 should come with a large change breaking IR compatibility. Chris Lattner suggests a sliding window of IR compatibility may be better.

• TB Schardl has posted an RFC on upstreaming the CSI framework ("Comprehensive Static Instrumentation"). The code is now up for review. This framework makes it easy to implement dynamic analysis tools, often without needing compiler changes.

• Ashutosh Nema has shared an RFC on strided memory access vectorisation.

• In response to a question on the mailing list, Hubert Tong has given a brain dump on the status of work on concepts support in Clang including opportunities for getting involved.

• Paweł Bylica has asked for advice on dealing with LLVM as a project dependency. In particular, is it worth investigating CMake's ExternalProject module? Chris Bieneman has shared some advice.

• Michael Kuperstein has posted an RFC on allowing the loop vectorizer to choose vector widths that generate illegal types. The feedback appears to be possible so far.

LLVM commits

• FileCheck learnt the --check-prefixes option as a shorthand for multiple --check-prefix options. r272670.

• A local_unnamed_addr attribute was introduced. This can be used by the code generator and LTO to allow the linker to decide whether the global needs to be in the symbol table. r272709.

• The ScalarReplAggregates pass has been removed as it has been superseded by SROA by a long time. r272737.

• LLVM's C API gained support for string attributes. r272811.

• Assembly parsing and lexing has seem some cleanups. r273007.

Clang commits

• A new loop distribution pragma was added. Loop distribution is a transformation which attempts to break a loop in to multiple loops with each taking part of the loop body. r272656.

• The nodebug attribute can now be applied to local variables. r272859.

• The validity check for MIPS CPU/ABI pairings is now performed at initialisation time and a much clearer message is printed. r272645.

Other project commits

• A complete implementation of the C++ Filesystem TS has been checked in. r273034.

• LLD's ARM port gained initial support for Thumb with ARMv7a. r272881.

1. A server,
   a. to which you can submit correctness and performance measurement data, by sending it a json-file in the correct format,
   b. that analyzes which performance changes are significant and which ones aren't,
   c. that has a webui to show results and analyses in a number of different ways.
2. A command line tool to run tests and benchmarks, such as LLVM’s test-suite, SPEC2000 and SPEC2006 benchmarks.

• Post-commit detection of performance regressions and improvements.
• Pre-commit analysis of the impact of a patch on performance.

Post-commit performance tracking

Step 1. Get an overview of the "Daily Report" page

Step 2. The long-term performance evolution chart

Step 3. The Run page

Step 4. The Profile page

While it’s clear that there are differences between the disassembly, it’s often much easier to understand the differences by reconstructing the control flow graph to get a per-basic-block view of differences. By clicking on the “View:” drop-down box and selecting the assembly language you see, you can get a CFG view. I find showing absolute values rather than relative values helps to understand performance differences better, so I also chose “Absolute numbers” in the drop down box on the far right:

There is obviously a single hot basic block, and there are differences in instructions in the 2 versions. The number in the red side-bar shows that the number of cycles spent in this basic block has increased from 431M to 716M. In just a few clicks, I managed to drill down to the key codegen change that caused the performance difference!

We combine the above workflow with the llvmbisect tool available at http://llvm.org/viewvc/llvm-project/zorg/trunk/llvmbisect/ to also quickly find the commit introducing the performance difference. We find that using both the above LNT workflow and the llvmbisect tool are vital to be able to act quickly on performance deltas.

Pointers on setting up your own LNT server for tracking performance

Setting up an LNT server is as simple as running the half a dozen commands documented at http://lnt.llvm.org/quickstart.html under "Installation" and "Viewing Results". The "Running tests" section is specific to LLVM tests, the rest is generic to performance tracking of general software.

The documentation for the json file format to submit results to the LNT server is here: http://lnt.llvm.org/importing_data.html.

The documentation for how to also add profile information, is at http://lnt.llvm.org/profiles.html.

Welcome to the one hundred and twenty-eighth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

LDC, a compiler for the D programming language with an LLVM backends has a major release with 1.0.0. The big news with this release is that the frontend is now completely written in D. Congratulations to everyone involved in this release. See the D website for more information about the D programming language.

The minor release LLVM 3.8.1-rc1 has been tagged.

On the mailing lists

• Gor Nishanov has shared an RFC on adding support for coroutines to LLVM IR.

• Hans Wennborg has shared the release plan for the 3.9 release. This plan would see the release branch created on the 18th of July and the final shipping on the 22nd of August. Hans wonders whether, as with 2.9 and 1.9 the base version number will be incremented (making release 4.0). He'd also like to make LLVM's current release cadence "official" on the website and publicly list people who are currently committed to testing releases.

• Chris Bieneman is proposing some changes to the LLVM directory structure. Specifically, adding a 'runtimes' subdirectory and removing the 'projects' subdirectory. Chris has helpfully summarised some of the key feedback and proposed how to move forwards.

• Sebastian Pop is trying to get a feel for how many LLVM developers are attending HPCA/PPoPP/CGO in February 2017 in order to estimate potential numbers for an LLVM gathering.

• In answer to a question, Krzysztof Parzyszek explains how to use glue values.

• Philip Reames provides a handy explanation of what is means for allocations to "escape", be "captured" or be "thread local".

• Simon Brand is seeking feedback on how to enhance LLDB to better support HSA applications.

• Simon Cook describes how he has set up pseudo-registers representing register locations on a very register-poor architecture.

• Vikram TV has shared a proposal on adding a pass that calculates loop cost based on cache data. The prototype patch analyses references to determine which would be in the same cache line. This knowledge can then be used to calculate a more accurate loop cost and. A drawback of the current implementation is it uses a static cache line size.

• Sean Silva has kicked off a thread about the intended behaviour of the CGSCC pass manager. This manages passes over strongly connected components of the callgraph.

LLVM commits

• Some of the work from the GSoC project on interprocedural register allocation has started to land. A RegUsageInfoCollector analysis was added that collects the list of clobbered registers for a MachineFunction. A new transformation pass was committed which scans the body of a function to find calls and updates the register mask with the one saved by RegUsageInfoCollector. r272403, r272414.

• Chapter 2 of the tutorial on building a JIT with ORC has been fleshed out with a rough draft of the text. r271885.

• The host CPU detection code for x86 has seen a large refactoring. r271921.

• More documentation has been added about LLVM's CodeView support. r272057.

• llvm-symbolizer will now be searched for in the same directory as the LLVM or Clang tool being executed. This increases the chance of being able to print pretty backtraces for systems where LLVM tools aren't installed in the $PATH. r272232.

Clang commits

• Clang analyzer gained a checker for correct usage of the MPI API in C and C++. r271907.

• Documentation was added on avoiding static initializers when using profiling. r272067, r272214.

Other project commits

• A hardened allocator, 'scudo' was added to compiler-rt. It attempts to mitigate some common heap-based vulnerabilities. r271968.

• Initial support for ARM has landed in LLD. This is just enough to link a hello world on ARM Linux. r271993.

• Initial support for AddressSanitizer on Win64 was added. r271915.

Welcome to the one hundred and twenty-seventh issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

Graham Markall at Embecosm has been comparing the code size of RISC-V binaries produced by the GCC and LLVM ports, as well as compared to ARM. GCC is currently ahead, though it is worth noting the LLVM port has seen much less attention.

Matthias Reisinger is a Google Summer of Code student working on enabling polyhedral optimisations for the Julia programming language. He's written a blog post detailing his initial steps and immediate future plans. Hopefully we'll see more posts over the summer.

Loïc Hamot has been working on a C++ to D converter, implemented using Clang.

The MSVC team have blogged about the latest release of Clang with Microsoft CodeGen, based on Clang 3.8.

There is going to be a clang-tidy code dojo in Warsaw on Tuesday the 7th of June.

On the mailing lists

• Renato Golin has kicked off a discussion on moving LLVM's repository hosting to GitHub. Chris Lattner came out in favour, specifically motivated by GitHub's community aspects. Renato has very helpfully summarised the discussion.

• Peter Smith has shared his initial port of LLD to ARM along with his planned roadmap. The currently submitted patch is just enough to link a hello world executable on ARM Linux.

• The apt repo hosted at llvm.org has been temporarily turned off as it is resulting in excessive I/O and network activity. Some commenters ask about setting up an Ubuntu PPA or using the OpenSUSE build service.

• David Blaikie has written a GDB pretty printer script for some common LLVM types and described how to use it.

• Michael LeMay has posted an RFC on using segmentation to harden SafeStack.

• Rui Ueyama has been investigating using sendfile to copy file contents in LLD and shares his results. He concludes the performance improvement is too modest to be worth the change.

• If you're interested in register allocation, then delve in to this thread on LLVM's PBQP allocator and copy propagation.

• Daniel Dunbar is suggesting some changes to the lit default output.

• Steven Wu shares a follow-up RFC on embedded bitcode.

• There's a useful discussion in this thread on lowering loops to use a hardware loop instruction.

• Peter Collingbourne proposes renaming and slightly redesigning the bitset metadata.

LLVM commits

• LLVM gained support for 'SJLJ' (setjmp/longjmp) exception handling on x86 targets. r271244.

• LLVM now requires CMake 3.4.3 to build r271325.

• Support was added for attaching metadata to global variables. r271348.

• The AArch64 backend switched to use SubtargetFeatures rather than testing for specific CPUs. r271555.

Clang commits

• The release notes have been updated to explain the current level of OpenMP support (full support of non-offloading features of OpenMP 4.5). r271263.

• Clang's source-based code coverage has been documented. r271454.

Other project commits

• An -fno-exceptions libc++abi library variant was defined, to match the -fno-exceptions libc++ build. r271267.

• LLDB's compact unwind printing tool gained support for ARMv7's compact unwind format. r271744.

Welcome to the one hundred and twenty-sixth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

I've been moving house this weekend, so do accept my apologies if you find this issue to be a little less thorough than usual.

News and articles from around the web

Pyston, the LLVM-based Python compiler has released version 0.5. The main changes are a switch to reference counting and NumPy compatibility.

I don't want to become "C++ weekly", but I think this audience appreciates a fun use of C++ features. Verdigris is a header-only library that allows you to use Qt5 without the moc preprocessor.

The call for papers for the 3rd workshop on the LLVM compiler infrastructure in HPC has been published. The deadline for paper submission is September 1st. The workshop will take place on November 14th in Salt Lake City, and is held in conjunction with SC16.

On the mailing lists

• Vivek Pandya, a GSoC student working on interprocedural register allocation has shared a weekly status report.

• Rafael Espíndola has proposed creating a bitcode symbol table.

• There's been some updates on the progress of open-sourcing PGI's Fortran frontend.

• Elena Lepilkina has proposed some enhancement to FileCheck. Some questions were raised about how useful the proposed extensions will be. Sergey Yakoushkin provided more background on how these features are used in a commercial codebase. As Elena notes, these features don't need to all be upstreamed at once (or at all), and are mostly independent.

• Lang Hames has posted a heads-up about upcoming breaking API changes for ORC and MCJIT.

• Sean Silva has kicked off a discussion on the state of IRPGO. You might ask what is IRPGO? This is profile-guided optimisation performed through instrumentation at the LLVM IR level, as opposed to FEPGO where instrumentation is added by the frontend (e.g. Clang), prior to lowering to IR. Sean would like to make IRPGO the default on all platforms other than Apple at the moment (who may require a longer deprecation period). A number of followup comments discuss possibilities for ensuring all platforms can move forward together, and ensuring a sensible flag exists to choose between frontend or middle-end PGO.

• What exactly is a register pressure set? Both Quentin Colombet and Andrew Trick have answers for us.

LLVM commits

• New optimisations covering checked arithmetic were added. r271152, r271153.

• Advanced unrolling analysis is now enabled by default. r270478.

• The initial version of a new chapter to the 'Kaleidoscope' tutorial has been committed. This describes how to build a JIT using ORC. r270487, r271054.

• LLVM's stack colouring analysis data flow analysis has been rewritten in order to increase the number of stack variables that can be overlapped. r270559.

• Parts of EfficiencySanitizer are starting to land, notably instrumentation for its working set tool. r270640.

• SelectionDAG learned how to expand multiplication for larger integer types where there isn't a standard runtime call to handle it. r270720.

• LLVM will now report more accurate loop locations in optimisation remarks by reading the starting location from llvm.loop metadata. r270771.

• Symbolic expressions are now supported in assembly directives, matching the behaviour of the GNU assembler. r271102.

• Symbols used by plugins can now be auto-exported on Windows, which improves support for plugins in Windows. See the commit message for a full description. r270839.

Clang commits

• Software floating point for Sparc has been exposed in Clang through -msoft-float. r270538.

• Clang now supports the -finline-functions argument to enable inlining separately from the standard -O flags. r270609.

Other project commits

• SectionPiece in LLD is now 8-bytes smaller on 64-bit platforms. This improves the time to link Clang with debug info by 2%. r270717.

• LLD has replaced a use of binary search with a hash table lookup, resulting in a 4% speedup when linking Clang with debug info. r270999.

• LLDB now supports AArch64 compact unwind tables, as used on iOS, tvos and watchos. r270658.

Welcome to the one hundred and twenty-fifth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

Stephen Kelly has written a blog post about using Clang through the cindex API to automatically generate Python bindings. He also makes use of SIP.

Krister Walfridsson has written a wonderfully clear post on C's type-based aliasing rules.

This week I discovered the Swift Weekly Brief newsletter. Its author, Jesse Squires does a wonderful job of summarising mailing list traffic, recent commits, and discussions on swift-evolution proposals. If you have an interest in Swift development or language design in general I highly recommend it.

Are you interested in writing for the LLVM blog? Or volunteering to help recruit content authors? If so, get in touch with Tanya.

The next Cambridge LLVM Social will be held at 7.30pm on May 25th at the Cambridge Blue.

On the mailing lists

• Elena Demikhovsky is interested in extending scalar evolution (SCEV) analysis to include floating point support. This kicked off a pretty interesting discussion. Sanjoy Das highlighted what he sees as the most important issues to discuss. A number of follow-ups discussed whether enough code uses floating point values as an induction variable to be worth optimising. There was also the question of should vectorisation be pursued at any cost?. Even if a loop can be made vectorisable through loop-versioning with run-time checks, is it worth the code size? Is the cost of maintaining the compiler code worthwhile? Hideki Saito posted a useful summary of the discussion so far.

• Chandler Carruth is looking for feedback on the idea of supporting horizontal operations on vector types such as sum directly in LLVM IR. Everyone who has responsed so far is in favour.

• Jia Chen, GSoC student with LLVM, has noted the CFL-AA pass seems to be mostly working now and would appreciate reports from people trying it out on their codebases.So far, Geoff Berry reports no correctness issues but seemingly very limited changes in the generated code for SPEC and the LLVM test-suite.

• Adam Nemet is seeking feedback on the idea of adding optimisation remarks to indicate where non-temporal stores may be profitable.

• Quentin Colombet has summarised recent discussion on policies to help release management and detailed the automatic hooks he hopes to explore next for updating bugs when referenced in a commit message. The following discussion looked at how these hooks may be implemented and what level of rigidity would be most beneficial to the community.

• Dean Michael Berris is looking for a way of defining a default implementation for a pseudo-instruction. No answers yet, but hopefully that will change soon!

• Galina Kistanova is doing some cleanup work on zorg (the buildbot-based testing infrastructure of the LLVM project) and is interested whether anyone uses these seemingly stale modules.

LLVM commits

• llc will now report all errors in the input file rather than just exiting after the first. r269655.

• The SPARC backend gained support for soft floating point. r269892.

• Reloc::Default no longer exists. Instead, Optional<Reloc> is used. r269988.

• An initial implementation of a "guard widening" pass has been committed. This will combine multiple guards to reduce the number of checks at runtime. r269997.

Clang commits

• clang-include-fixer gained a basic Vim integration. r269927.

• The intrinsics headers now have feature guards enabled in Microsoft mode to combat the compile-time regression discussed last week due to their increased size. r269675.

• avxintrin.h gained many new Doxygen comments. r269718.

Other project commits

• lld now lets you specify a subset of passes to run in LTO. r269605.

• LLDB has replaced uses of its own Mutex class with std::mutex. r269877, r270024.

Welcome to the one hundred and twenty-fourth issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

The main news this week is the announcement of Scala-native, an ahead-of-time compiler for Scala using LLVM. Jos Dirkens has written a getting started guide if you want to compile it and try it out. There's also more information in the slides from the announcement talk.

On the mailing lists

• More of the students taking part in Google Summer of Code with LLVM-related projects have introduced themselves and their plans. Vivek Pandya will be working on interprocedural register allocation. Scott Egerton will be working on capture tracking improvements. Jie Chen will be working on better alias analysis, specifically improving cfl-aa. Matthias Reisinger will be working on enabling polyhedral optimisations in Julia, and Zhengyan Liu has plans for SAFECode memory hardening.

• Renato Golin kicked off a discussion about whether LLVM's release process could be better aligned with downstream users. This thread covered a broad range of topics and triggered a lot of discussion, but luckily there's no need to summarise it as Renato has done the job for us.

• Nicolai Hähnle notes that currently libLLVM.so contains about 1.7MB in its .data.rel.ro section, of which about 1.3MB comes from the MCInstrDesc tables created by tablegen representing a massive number of pointers to be relocated. He suggests reducing this by using offsets instead. Reducing the relocations will both reduce binary size and increase the portion of the binary that can be mapped as shared. So far, responses to the thread are supportive of the idea.

• James Knight has written a detailed post on how it's not really possible to write an LL/SC loop guaranteed to make forward progress in LLVM IR right now. There are restrictions on what you can do between a load-linked and a store-conditional instruction that the code generator may not meet.

• A public llvm-foundation mailing list has been announced, which to facilitate discussions related to the Foundation.

• As well as the long, technically detailed and precise threads each week it's nice to highlight cases where a simple question has a simple answer. How do you register a pass as being opt-in based on a command-line flag? Answer: have it run every time, but return immediately if the desired command line flag isn't present.

• Sanjoy Das has shared an RFC on adding a callee-saved register verifier. As is clarified later in the thread, the intention is to ensure that code not generated by LLVM (e.g. output from another JIT or hand-written assembly) properly adheres to the calling convention and doesn't clobber registers it shouldn't. The proposed pass would simply add code to check that the test values written to the callee-saved registers aren't modified.

• In response to questions about pass ordering, Mehdi Amini has written a helpful description of what exactly happens when you do opt -mymodulepass0 -myfunctionpass -mymodulepass1.

• Konstantin Vladimirov wonders if there's an option to force the register allocator to use as many architectural registers as possible to reduce dependencies. The short answer is there isn't currently, but it would be interesting to investigate.

• Diana Picus has shared an RFC on modifying llc so it no longer exits after the first error. Generally people are in favour, and the patch should hopefully land soon (it had to be temporarily backed out after exposing some test cases failures in lldb).

• Nico Weber has noted that now with AVX512, Clang's intrinsics headers are huge. This can cause compile time issues, for instance Nico reports building all of the v8 JS engine is 6% faster after removing the avx512 includes. The thread participants haven't yet decided on the best way forward to fix this, beyond the potential immediate step of adding include guards so AVX512 intrinsic headers aren't included when not compiling for AVX512 platforms.

LLVM commits

• The outdated guide on cross-compiling LLVM has been brought up to date. r269054.

• The WebAssembly backend gained preliminary fast instruction selection (fast-isel) support. r269083, r269203, r269273.

• Loop unrolling (other than in the case of explicit pragmas) is now disabled at -Os in LLVM. You may recall last week it was enabled for -Os in Clang, but with different thresholds. r269124.

• A new cost-tracking system has been implemented for the loop unroller. r269388.

• LLVM's Sparc backend has seen the addition of more LEON-specific features, e.g. signed and unsigned multiply-accumulate. r268908.

• llc's -run-pass option will now work with any pass known to the pass registry. Previously it would silently do nothing if you specify indirectly added analysis passes or passes not present in the optimisation pipeline. r269003.

• WebAssembly register stackification and coloring are now run very late in the optimisation pipeline. The commit message suggests it's useful to think of these passes as domain-specific liveness-based compression rather than a conventional optimisation. r269012.

• When declaring global in textual LLVM IR, you must now assign them with e.g. @0 = global i32 42. r269096.

• The internal assembler is now enabled by default for 32-bit MIPS targets. r269560.

Clang commits

• Clang now supports __float128. r268898.

• Clang gained a new warning that triggers when casting away calling conventions from a function. r269116.

• The recently developed include-fixer tools now has documentation. r269167.

Other project commits

• compiler-rt's CMake build system can now build builtins without a full toolchain, allowing you to bootstrap a cross-compiler. r268977.

• LLD will now sort relocations to optimise dynamic linker performance. r269066.

Welcome to the one hundred and twenty-third issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

If you're in London tomorrow you may be interested in the NMI Open Source Conference. You can register until midday today. I'll be giving a brief talk on lowRISC. While on the subject of conferences, if you are interested in diversity and inclusion in computing education, you may want to check out the CAS #include diversity conference in Manchester on the 11th June.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

Fabien Giesen has written a brief article explaining why compilers exploit undefined signed overflow.

The Google Open Source blog has a short piece on the XRay function call tracing system that was proposed for upstreaming last week on the LLVM mailing list.

On the mailing lists

• By far the most active thread on the mailing list this week was the resumption of discussion on adding an LLVM Code of Conduct. The draft text can be found here. As well as a number of messages offering a "+1" to the current text, concerns were raised by some about the implications of "violations of this code outside these spaces may affect a person's ability to participate within them", and about how the committee enforcing the CoC will be selected.

• Amos Robinson wrote to the mailing list with an optimisation missed by LLVM's current Global Value Numbering pass. Rather excitingly, Daniel Berlin reports he's working on a new GVN implementation.

• Chandler Carruth has written an update on the state of work to move to the new pass manager. He notes the major missing piece at the moment is the ability to communicate invalidation information between two parts of the pass manager.

• Jonas Hahnfield has shared an RFC on automatically generating non-temporal loads and stores. Some respondents are very strongly against this, suggesting it's something better left for the programmer to specify.

• Some of the students taking part in Google Summer of Code this year with LLVM-related projects have been introducing themselves on the mailing list. Utpal Bora will be working on implementing Polly as an analysis pass in LLVM. Bianca-Cristina Cristescu will be working in enabling LLVM's self-hosted modules builds using libstdc++, and Roman Gareev will be improving the vectorisation process in Polly.

• Chris Bieneman notes he recently introduced a new option in LLVM's CMake buildsystem that may be of particular interest to package maintainers. LLVM_DISTRIBUTION_COMPONENTS allows you to specify which components of LLVM you want to install.

• Peter Collingbourne has posted an RFC on extending ThinLTO to allow a bitcode module to embed another bitcode module containing summary information for CFI and whole-program devirtualisation.

• Adam Nemet is interested in feedback on the idea of filtering optimisation remarks by the hotness of the code region.

• Justin Bogner has given a heads-up to out-of-tree backend maintainers that he intends to change the API of SelectionDAGISel::Select so the function directly replaces nodes rather than returning the desired replacement.

• Quentin Colombet has shared an RFC on how LLVM contributors can better help release management. There's a lot of support for this direction, with most comments discussing ways of better tagging commit messages (post-commit in phabricator/bugzilla, or through getting committers to write commit messages in a certain format).

LLVM commits

• LLVM's CppBackend has been removed. As the commit message says, this backend has bit-rotted to the extent that it's not useful for its original purpose and doesn't generate code that compiles. r268631.

• The AVR backend has seen a large amount of code merged in to LLVM. r268722.

• The MIPS backend has seen some large changes to how relocations are handled. These are now represented using MipsMCExpr instead of MCSymbolRefExpr. As someone who has done quite a lot of (out-of-tree) LLVM backend work, I've always found it odd how some architectures have globally visible enum members in include/llvm/MC/MCExpr.h. r268379.

• LLVM builds should hopefully now be deterministic by default, as LLVM_ENABLE_TIMESTAMPS is now opt-in rather than opt-out. In fact, a follow-up patch removed the option altogether. r268441, r268670.

• The AARch64 backend learned to combine adjustments to the stack pointer for callee-save stack memory and local stack memory. r268746.

Clang commits

• Clang now supports -malign-double for x86. This matches the default behaviour on x86-64, where i64 and f64 types are aligned to 8-bytes instead of 4. r268473.

• Loop unrolling is no longer completely disabled for -Os. r268509.

• Clang's release notes (reflecting the state of current trunk) have been updated to say more about the state of C++1z support. r268663.

Other project commits

• libcxx will now build a libc++experimental.a static library to hold symbols from the experimental C++ Technical Specifications (e.g. filesystem). This library provides no ABI compatibility. r268443, r268456.

• All usage of pthreads in libcxx has been refactored in to the __threading_support header, with the intention of making it easier to retarget libcxx to platform that don't support pthreads. r268374.

• libcxx gained support for the polymorphic memory resources C++ TS. r268829.

Welcome to the one hundred and twenty-second issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

GCC 6.1 has been released. Perhaps the most apparent user-visible change is that the C++ frontend now defaults to C++14.

The Rust compiler has introduced a new intermediate representation, MIR, used for optimisations prior to lowering to LLVM IR.

Tanya Lattner has written about the LLVM Foundation's plans for 2016. The LLVM Foundation has established 3 main programs: Educational Outreach, Grants and Scholarships, and Women in Compilers and Tools.

On the mailing lists

• Dean Michael Berris has shared an RFC on upstraming Google's 'XRay' function call tracing system. For more information, you can read the XRay whitepaper.

• Sanjoy Das has suggested generalising the AssumptionCache to AxiomAche. He proposes maintaining separate lists of guards and assumptions within the AxiomCache.

• There's been some more activity in response to Phil Tomson's question about instruction scheduling. Christof Douma followed up with some advice.

• Chris Bieneman has suggested raising the CMake minimum version to 3.4.3. Renato Golin flagged up some concerns, with Chandler Carruth offering a counterpoint.

• Discussion has continued on the proposal to introduce a new LLVM sub-project for parallelism runtime and support libraries. This is probably best summarised by reading Hal Finkel's thoughts on the way forward, C Bergström's concerns, and Jason Henline's proposed charter for the subproject.

• Peter Collingbourne has shared an RFC on redesigning the LLD symbol table in order to improve memory locality.

LLVM commits

• LLVM now supports indirect call promotion based on value-profile information. This will promote indirect calls to a direct call guarded by a precondition. r267815.

• The LLVM documentation has been extended with a CMake primer covering the basics of the CMake scripting language. r268096.

• The PDB dumper has been refactored into a library. r267431.

• The MinLatency attributed has been removed from SchedMachineModel. r267502.

• CodeGenPrepare will now use branch weight metadata to decide if a select should be turned into a branch. r267572.

• Support for llvm.loop.distribute.enable metadata was added. This indicates a loop should be split in to multiple loops. r267672.

• The SystemZ backend now supports the Swift calling convention. r267823.

• libFuzzer's documentation has been expanded and improved. r267892.

Clang commits

• clang-tidy gained a new checker for redundant expressions on both sides of a binary operator. r267574.

• A new clang-tidy check will warn for use of functions like atoi and atol that don't report conversion errors. r268100.

• The nodebug attribute on a global or static variable will now suppress all debug info for that variable. r267746.

• A number of OpenMP features gained codegen support, such as the map clause and target data directive. r267808, r267811.

Other project commits

• LLD now supports an -O0 option to produce output as quickly as possible. Currently this disables section merging at the cost of a potentially much larger output. r268056.

• The symbol table in LLD's ELF linker has been redesigned with the intent of improving memory locality. The new design produces measurable speedups for the binaries tested in the commit message. r268178.

• LLD's linkerscript support expanded to encompass comparison operators. r267832.

• LLD performance on large executables has been improved by skipping scanRelocs on sections that are never mapped to memory at runtime (e.g. debug sections). r267917.

Welcome to the one hundred and twenty-first issue of LLVM Weekly, a weekly newsletter (published every Monday) covering developments in LLVM, Clang, and related projects. LLVM Weekly is brought to you by Alex Bradbury. Subscribe to future issues at http://llvmweekly.org and pass it on to anyone else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

The canonical home for this issue can be found here at llvmweekly.org.

News and articles from around the web

Congratulations to the eight students who have been selected for LLVM projects on Google Summer of Code this year. There's about a month before they start coding. The time between now and then is the 'community bonding period', so please do make them feel welcome.

The preliminary release schedule for LLVM/Clang 3.8.1 has been published. This would have a deadline of May 25th for requesting changes to be merged and would see the final release on June 15th.

On the mailing lists

• A series of RFCs have been posted around the idea of adding an EfficiencySanitizer tool. Like the existing sanitizer tools, this would rely on compiler-based dynamic instrumentation in order to detect problems in user code. The goal is to collect userful information with overhead less than 5x, ideally closer to 3x. Separate threads have been started to discuss an EfficiencySanitizer cache fragmentation tool and a working set tool.

• Sanjoy Patel has proposed removing the llvm.expect intrinsic, in favour of using metadata to represent the same information. There isn't currently full agreement on this.

• Richard Trieu is seeking feedback on which floating point to boolean conversions should trigger warnings.

• How do you add fixup information to a MachineInstruction? Tim Northover has the answer.

• Elena Lepilkina has shared an RFC on adding support for custom metrics and test parameterisation to LNT. The feedback so far seems positive.

• Phil Tomson is looking for advice on instruction scheduling in LLVM. As he notes in his email, it's part of LLVM that's seen a lot of changes over the past 8 years or so. I'm certainly interested in the answer here.