Contents


Controlling symbol visibility for shared libraries: Part 2 - Advanced knowledge with IBM XL C/C++ V13 compiler

Comments

IBM XL C/C++ V13 compiler: The support of visibility attribute

In the previous article, we have learnt the different methods to handle symbol visibilities. The most important ones include: GNU visibility attribute extension, IBM AIX® export files, and GNU version script. GNU visibility can enable a programmer to control symbol visibility from the source-code level, whereas the script for linking might be more useful for controlling symbol visibilities from the perspective of the build and release step.

However, as XL C/C++ for AIX and Linux® V13 compiler is released, GNU visibility attribute extension is accepted by the XL C/C++ compiler. Further, entities such as new pragma directive and new global compiler options might also be served for this purpose. In this part, we focus more on visibility attribute settings and its final resolutions. Further, we might get into some details on the differences between AIX and Linux visibility designs and the compatibility on AIX visibility that is achieved.

The visibility attribute specifier

Similar to the GNU visibility attribute extension, for XL C/C++ compiler, a programmer can set symbol visibility by the attribute specifier using the GNU attribute syntax. Listing 1 shows an example for function and variable declarations.

Listing 1. Visibility specifier for variable and functions
int __attribute__((visibility("hidden"))) m;  // m is set as "hidden"
void __attribute__((visibility("protected"))) fun(); // fun() is set as "protected"

The type of the visibility attribute can be default, hidden, protected or internal. That is aligned with the GNU syntax, which we have introduced in the previous article. However, in the default value, there is a subtle difference between AIX and Linux implementations. This is discussed later in this article.

The visibility attribute looks straightforward at the first glance. However, in programming practice, a programmer can have more questions. For example, one commonly raised question is about the multiple definition of the same symbol with different visibilities. This might not be rare for handing different header definitions and can cause a lot of trouble to the programmer. From the perspective of a compiler, this issue is caused by imperfect reality and the solution would rely more on the programmer's side. But, the compiler does provide rules and facility to help. The XL C/C++ compiler would take the visibility attribute that is first encounters. And, a warning is issued to the programmer when handling the rest (with a different visibility). Finally the visibility defined latter is ignored by compiler. A sample is provided in Listing 2.

Listing 2. Different visibility attributes specified for the same symbol
extern int __attribute__((visibility("hidden"))) m;  // "m" is "hidden".
int __attribute__((visibility("protected"))) m;       // Compiler warning - "protected" is ignored.

In this example, m is first defined as hidden. That means, the symbol for the variable would not be visible during dynamic linking. However, the code followed would set it again but, as protected, that is visible globally. The compiler would issue a warning and then ignore the protected attribute as it processes the code.

Some other questions on visibility would get type involved. In C++ coding, programmers might be able to create a type by themselves. Programmers might even wonder whether it is possible to specify the visibility attribute to such types. This is an interesting question because in C programming such issues never happens. All built-in types would not interfere with the visibility attribute. However, in C++ programming, this is no longer true. This is demonstrated in the following listing.

Listing 3. Visibility attributes specified for a class
class __attribute__((visibility("hidden"))) T  // class T has "hidden" visibility
{
public:
      static int gVal;  // T::gVal is "hidden"
      void Dosth();    // T::Dosth() is "hidden"
};  

T t1;  // t1 is "hidden".
T __attribute__((visibility("internal"))) t2; // t2 has "internal" visibility.

In Listing 3, we have defined a class T with hidden visibility. In C++ code, a programmer can define the visibility attribute to a class as well as to functions and objects. The syntax is the same but the effects would be different. Here, we can just assume that the type T is hidden. Later, the code has defined two objects of type T: t1 and t2. You might observe that t1 is defined without any visibility attribute specifier. Therefore, the visibility setting of t1 is intuitively taken from its type. And, for t2, the code defines it as internal visibility. Such definitions would be prior to the visibility attribute of its type T. So, the final visibility of t2 is internal. This is the basic rule. However, visibilities of entities such as member functions and static member, which are associated with the class, would not be changed due to different object visibility definitions. That means, in Listing 3, T::gVal and T::Dosth() would be with hidden visibility as the class T defines. Further, it also implies that the objects of same type (t1 and t2) still share the same function table no matter what the visibility attribute is specified for them. And, if a programmer intends to change the visibility of such members, taking the member function for example, then programmers need to specify the visibility attribute to the member function directly. Listing 4 shows how the visibility attribute of T::Dosth()can be changed to protected.

Listing 4. Change the visibility attribute for a class member
class __attribute__((visibility("hidden"))) T  // class T has "hidden" visibility
{
public:
      static int gVal;  // T::gVal is "hidden"
      void __attribute__((visibility("protected"))) Dosth();    // T::Dosth() is now "protected"
};  

T t1;  // t1 is "hidden".
T __attribute__((visibility("internal"))) t2; // t2 has "internal" visibility.

Visibility attribute can also be applied to union, enum, namespace, and template. Visibility on namespace would work similar to a class in scope or context. When the visibility attribute is specified for a namespace, the attribute is just applied to any symbol enclosed in the scope. But, in the meantime, the functionality is limited to the enclosed scope. Listing 5 shows an example.

Listing 5. Visibility attribute for namespaces
namespace std __attribute__((visibility("hidden"))) {
   void foo1() {};  // foo1() has "hidden" visibility.
}

namespace std {
   void foo2() {}   // The visibility of foo2() is "default"/"unspecified".
}

In this example, foo1() is hidden as expected. However, foo2() would not be a hidden symbol. That means, if you have defined the visibility for namespace std in your file, a programmer writing another file with the same namespace std would not be affected. As a result, such a design can potentially avoid the pollution of visibility setting across the code blocks (with the same namespace).

Template definition is also equally interesting. Programmers might always be curious about the rules and to understand how the visibility attribute can be set when both the template and the type parameter have visibility attribute settings. In brief, if the primary template definition has visibility, the visibility is implicitly propagated to template instantiation. Otherwise, the visibility of template instantiation is, by default, the same as its template argument. Refer to Listing 6 for an example.

Listing 6. Visibility attribute for namespaces
class __attribute__((visibility("internal"))) A {};
template <typename T> void foo1() {};
template <typename T> void __attribute__((visibility("hidden"))) foo2() {};

foo1<A>();  // The visibility of "foo1<A>()" is "internal" which is propagated from its argument "A".
foo2<int>();  // The visibility of "foo2<int>()" is "hidden" which is propagated from template.
foo2<A>();   // The visibility of "foo2<A>()" is "hidden".

This code defines a type A as internal, a common template function foo1, and a template function foo2 with hidden visibility. For the first instantiation foo1<A>, because A has the visibility attribute, the function would propagate the visibility attribute from this template argument. And for foo2<int>, it takes the visibility attribute from the template. And, so does foo2<A>. The instantiation ignores the visibility attribute from class type A. And as a result, the visibility attribute of the template takes precedence.

The pragmas

Besides the visibility attribute, there is still another way to set the visibility attribute for the symbols from the source-code level. The XL C/C++ compiler provides such functionality by introducing the visibility pragma. Pragma has some advantages over the visibility attribute. It allows a programmer to set visibility for several declarations without defining the visibility symbol by symbol. The visibility pragma syntax is shown in Listing 7.

Listing 7. The syntax of visibility pragma
#pragma GCC visibility push(hidden)
#pragma GCC visibility pop

The visibility pragma only includes two stack operations. The top of stack would show up the current visibility attribute. The current visibility can apply to all the variables defined after the code point. For example, on Linux, the default visibility of the current context is default. If a programmer uses pragma to push a hidden visibility, the visibility of all the declarations defined after the push pragma is hidden. It would not be changed until another visibility pragma is encountered. Listing 8 shows an example.

Listing 8. Visibility attribute for namespaces
#pragma GCC visibility push(hidden)
int m; // "hidden" visibility

#pragma GCC visibility push(protected)
void foo() // "protected" visibility

#pragma GCC visibility pop
class A{}; // "hidden" visibility ("protected" visibility is popped)

#pragma GCC visibility pop
int n; // "default" visibility

#pragma GCC visibility pop  // An error is emitted as there is no visibility pragma to pop.

Notice that if the stack of visibility is already empty, a pop pragma can cause an error during compilation as the case shows.

The compiler options

Sometimes, programmers might also want to set the visibility attribute from the compiler options. Using these options, they can change the visibilities of the symbols embedded in the compiling file. For the XL C/C++ compiler, you can specify the following options.

xlc -qvisibility=default | protected | hidden | internal | unspecified

Notice that, the unspecified option is available only for the AIX platform. And on Linux, -qvisibility would be equivalent to –qvisibility=default. Whereas on AIX, -qvisibility would be equivalent to –qvisibility=unspecified.

Visibility determination rules

As the declaration visibility can be set from the visibility attribute specifier, visibility pragma, or the visibility option, it is important to be aware of the rules on how the compiler can resolve conflicts between them.

In fact, compiler would finally resolve a declaration's visibility by checking in the following sequence:

  • The visibility attribute specifier, if there is any
  • The visibility of its primary template definition if it is a template instance
  • The visibility of its enclosing context (struct, class, union, enum, namespace, or pragma) if there is any
  • Otherwise, the visibility of its type

The rule is straightforward. First of all, the visibility attribute specifier would always take precedence if it is a part of declaration. If this is not true, and the code is a template instantiation, the visibility of primary template definition is considered. Listing 6 has shown us such an instance.

And otherwise, the visibility of the declaration is determined by its enclosing context. A declaration's enclosing context can be, for example, a namespace scope for a global variable or a class scope for its member, and so on. Refer to Listing 9 for an example.

Listing 9. Visibility attribute for enclosing context
#pragma GCC visibility push(hidden)
class A   // "hidden" from pragma
{
  static int m; // "hidden" from class definition
   void __attribute__((visibility("protected")))  foo()  // foo is "protected"
};

In Listing 9, class A is hidden because the pragma defines an enclosing context with the visibility, hidden. Its member also takes hidden as its visibility because its enclosing context is the definition of A. However, foo is set to protected visibility because an attribute specifier is there.

As the last rule, the declaration's visibility is determined by its type. Listing 4 has already shown an example about class type.

Whereas, for a function declaration without any visibility specifier, its visibility is determined by the following four elements.

  • Function parameters
  • Function return type
  • Function template arguments (type or non-type)
  • Visibility option passed from the compiler

By comparing the four elements, it is not hard to find the one with the least visible visibility. Here, we consider default as the most visible, protected as the secondary, hidden places third, and internal as the least visible. The function would take the least visible visibility from the four elements provided. Listing 10 shows an example.

Listing 10. Visibility determination for functions
class __attribute__((visibility("internal"))) A {};
A *foo (int a) {}; // Even with option -qvisibility=hidden, foo is "internal".

In this example, a user might observe that the foo function is returning a pointer type to class A, which is defined with internal visibility. And in global context, there is an option that defines that the visibility is hidden. The final resolution result of foo is internal as it is the least visible visibility.

And if no rule listed above can be applied to variable declarations or functions, a default visibility is set for them. The default visibility is described in the following section.

Visibility attribute and backward compatibility on AIX

As we know from the previous article, on AIX, symbols are not visible by default unless we export them at the linking stage, either manually or with the help of CreateExportList. However, on Linux, symbols are, by default, with export (namely default) visibility. This brings a gap between the AIX visibility attribute and the GNU visibility attribute. To be backward compatible, on AIX, XL C/C++ would not set all the symbols to be exported like Linux. It might consider symbol without any visibility setting to be an unspecific visibility, which aligns with an old AIX implementation. For such symbols, AIX compiler, linker, and related tools would handle it as before. However, unspecific visibility does not mean that the symbol is internal or invisible at all. It is just a visibility that is specially designed to keep the compatibility.

Consequently, as we have mentioned at the beginning of this article, if the programmer does not explicitly specify the visibility attribute for a symbol, on Linux, the visibility of the symbol could be the default. But on AIX, the visibility would be unspecified.

Visibility conflict at linking

Visibility conflict at linking on AIX

From the previous paper, we know that, on an AIX system, it is possible to determine whether a symbol in libraries / executables is exported by using the export file. However, as the IBM XL C/C++ V13 compiler introduces symbol visibility attributes, it is essential to know how the feature interacts with the export file.

Consider a typical definition in the a.cpp file, as shown in Listing 11.

Listing 11. file a.cpp
int sym __attribute__((visibility("default")));

The variable sym is defined with the default (exported) attribute in the source code. We can compile the a.cpp file to generate a dynamic shared library, named liba.so. And we expect that sym is an exported symbol in the library. However, if the exportlist (an export file) is given at the linking step (as shown in Listing 12), and we pass the -bE:exportlist option to the linker for the liba.so library generation (as shown in Listing 13), it is common to raise a question: is sym a global visible (exported) symbol finally in the shared library.

Listing 12. The exportlist file
sym hidden
Listing 13. The compile command
xlC -G -o liba.so a.c -qpic -bE:exportlist

Such a question brings challenge for linker/compiler design. But as designers, we noticed that:

  • Visibility attribute would be introduced by XL C/C++ V13 for AIX. It targets more for new code and ported code (that is, code that accepts the GNU visibility attribute).
  • Visibility in exportlist targets at the linking stage, which is commonly invoked after the compilation stage.

As a result, it is better for the linker to make the visibility keyword in the export file to take precedence. In fact, the linker might have to handle different visibility keyword/attribute combinations from the export file (which defines the visibility keyword for symbols) and modules such as object files (which provides the visibility attribute for symbols). Such handling is also known as resolution of visibility conflict. Table 1 describes how the linker works.

Table 1. Visibility conflict resolution by AIX linker
Keyword in export fileSymbol visibility attribute in the object file
ImportedExportedProtectedHiddenInternalNone
Export EXP EXP EXP EXP error EXP
Protected PROT PROT PROT PROT error PROT
Hidden error No No No error No
Internal error No No No No No
None (no keyword) No EXP PROT No No No
  • EXP: Symbol is exported
  • PROT: Symbol is exported with protected visibility
  • No: Symbol is not exported
  • Error: Symbol is not exported. An error message is printed and the link fails

In Table 1, the row headers show symbol visibility defined in the source module. And the column headers are the symbol's keywords defined in the export file. Note that in some versions of AIX, the keyword can be export instead of exported. Table 1 shows the final resolution result of the symbol visibility by linker, including:

  • Whether the symbol is visible in the target module (such module can be a library that is finally built)
  • Whether the exported symbol can be preempted

As Table 1 shows, a symbol is finally resolved as globally visible by the linker if the export file claims it as exported or protected, (The latter one can not be preempted). Similarly, the symbol would not be visible finally if the export file describes it as hidden or internal. Further, if no export file is specified and no linking option -bexpall/-bxpfull is specified, the symbol visibility in the object file takes precedence. For some non-rational combination, for example, the symbol has internal attribute in the object file while the export file expects it to be exported, the linker would raise an error. The reason is that the export file might potentially change the programmer's intention.

As a result, for the a.cpp file mentioned earlier, the symbol for the sym variable would finally be invisible globally.

Sometimes, a programmer might still choose to use the linker option, -bexpall/-bexpfull to export all the symbols. However, as the new design of visibility resolution, under this circumstance, the symbol visibility in the object file would take precedence unless no visibility (unspecific) is defined for the symbol.

Now, we learn that the export file can generally overwrite the visibility at the final linking step. However, we do not recommend using both, export file and visibility attribute, in the new code. The mixed use can potentially bring confusion for code development. However, sometimes this facility would still be very useful to ensure release-to-release binary compatibility of shared libraries.

Visibility conflict at linking on Linux

The visibility conflict at linking also happens on Linux platforms. However, the behavior of the GNU linker is slightly different when compared to the AIX linker because of the difference in the platform.

Listing 14 shows an example on the Linux platform, assuming that we have a variable in c.c:

Listing 14. file c.c
int bar __attribute__((visibility("protected")));

And we declare bar as global visible in the exmap export map file. The GNU linker also names it as version script. Notice that a programmer can either declare a symbol as global visible or local symbol in the export map.

Listing 15. exmap file
{
  global:
  bar;
  local:
  *;
};

Use the command shown in Listing 16 to build a dynamic shared object (DSO) libc.so.

Listing 16. The compile command
xlC -qmkshrobj c.c -o libc.so -Wl,--version-script=exmap

Finally, it is time to check the dynamic symbol table (DYNSYM) section (commonly called .dynsym) of libc.so.

Listing 17. Dump of symbol bar
5 0: 0001100c  4 OBJECT  GLOBAL PROTECTED  21 bar

Similarly, on the Linux platform, if a symbol is exported, it is picked up to the DYNSYM section of DSO. Table 2 shows how the export map and visibility attribute decide if a symbol should be exported (picked up to DYNSYM) or not.

Table 2. Visibility confliction resolution by Linux linker
Keyword in export fileSymbol visibility attribute in the object file
DefaultProtectedHiddenInternal
Global Yes Yes No No
Local No No No No

Similar to Table 1, the first row of Table 2 shows symbol visibility in the source module. Generally speaking, this is similar to AIX because keyword takes precedence. But the Linux version script is unable to export the hidden symbol out. That is caused due to the difference in the nature between these two platforms.

Another difference between AIX and Linux is that both symbol from DYNSYM and symbol from SYMTAB maintain the visibility attribute information. However, AIX does not keep the same attribute between the symbols in the symbol table and the loader symbol table. Visibility attribute information is valid only in the symbol table. The loader symbol table just reveals the final visibility resolution result. If a symbol is listed there, it is exported. Otherwise, the symbol is invisible. That is also a difference between the AIX and Linux platforms.

CreateExportList facility

In the previous article, we have introduced the CreateExportList facility that helps to generate an export file automatically. A programmer might still care how it can interact with the visibility attribute. The general rule for the CreateExportList tool is to add all the global visible symbols to the export file, together with a matching visibility attribute as a keyword. Table 3 shows if the symbol var can be picked up into the export file and how the keyword is generated for the symbol.

Table 3. New working rule for CreateExportList
Visibility attribute in .o Representation in the export file Comment
(unspecified) var This rule is kept the same for legacy code compatibility
exported var exported Keyword is exported
protected var protected Keyword is protected
hidden - Symbol of this visibility attribute is not picked up into exportlist
internal - Symbol of this visibility attribute is not picked up into exportlist

Referring to Table 1, readers might generally find it working well because the linker can properly handle the conflicts. And, this tool is very useful when we are building a module (library or an executable) from mixed objects (such as legacy objects and new objects with symbol visibility). Further, it ensures that the old building script does not break as the compiler version gets updated.

Assembly syntax

Programmers writing an assembly code might also be able to assign a symbol with different visibilities. Listing 14 provides a sample syntax for the .globl, .comm, .extern, and .weak pseudo operations.

Listing 14. Syntax for specifying visibility attribute in assembly
   .extern Name [ , Visibility ]
   .globl  Name [ , Visibility ]
   .weak   Name [ , Visibility ]
   .comm   Name, Expression [, Number [, Visibility ] ]

For .comm , if "Visibility" is specified, "Number" (alignment) can be omitted:
   .comm   Name, Expression , , Visibility

The keywords for Visibility listed here are defined similarly as shown in Listing 14 and can have four values: internal, hidden, protected and exported. If no visibility is defined, the symbol entry might not have any visibility attribute (unspecified). That is similar to the rules in C/C++ source code.

However, the assembler would not accept such syntaxes unless its version is up to date. Similar pre-requisition is needed for AIX linker. That means AIX 6.1 TL8/AIX 7.1 TL2 or later version would meet the requirement. And the XL C/C++ compiler for AIX, V13.1 or later version would be sufficient to support the visibility feature in C/C++ code.

Resources


Downloadable resources


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=982555
ArticleTitle=Controlling symbol visibility for shared libraries: Part 2 - Advanced knowledge with IBM XL C/C++ V13 compiler
publish-date=09092014