Difference between pages "Architecture" and "HowtoCompile"

From Fakeroot NG
(Difference between pages)
(Move page from old installation)
 
(Move page from old installation)
 
Line 1: Line 1:
This page documents the software architecture of fakeroot-ng. The documentation is relevant to the (as yet unreleased) multi-threaded debugger branch.
+
=Compiling Fakeroot-NG=
  
Throughout this page, the acronym "frng" is used to refer to fakeroot-ng.
+
Here are a few tips and solutions to possible compilation problems:
  
=The Daemon=
+
==Generic Compilation Instructions==
Note: some people think that daemons are, by necessity, system startup processes. This is not the case. A daemon of any process that disassociates itself from the environment that started it.
+
In general, as the README file says, compiling fakeroot-ng is just a matter of running:
 +
./configure
 +
make
 +
sudo make install
  
Fakeroot-ng act as a debugger (tracer) for the faked root process. To do this, it creates an extra process and disassociates it from the running environment. This is the fakeroot-ng daemon, and it is an unprivileged process, without any special permissions.
+
The project is an automake project, which means it supports out of tree builds, install targets and changing the installation path. Still, there are a few caveats that might be a problem, under certain conditions.
  
==File Lie Database==
+
==Passing Custom Compiler Flags==
Fakeroot-ng allows processes to carry out changes to the file system that would require root permission. These include, among other things, changing ownership and device files. Since no fakeroot-ng related process actually has privileges, this is done by faking success on the creation system calls, and then lying about the effect to all future requests. If you create a character device, and then ask for a list of files in the directory, fakeroot-ng will lie to you that the file is a character device.  
+
The following holds for any autoconf based project.
  
The lies frng tells the process are indexed in an [http://en.cppreference.com/w/cpp/container/unordered_map unordered_map]. The index key is the file's device and inode numbers. This allows the file lie database to persist across different processes, and also across invocations of frng. Also, since all of those lies are related to the [http://linux.die.net/man/2/stat stat] structure, this indexing is extremely convenient.
+
During the ''configure'' stage, the environment is read for environment variables for CFLAGS and CXXFLAGS (a full list is available by running '''./configure --help'''). These values are then coded into the generated Makefile, and need not be set later on.
  
The database is maintained in '''file_lie.h''' and '''file_lie.cpp'''.
+
For fakeroot-ng in particular, care must be taken that the ptlib library is written in C, while fakeroot-ng itself is written in C++. As a result, it is important to set any flags you want for both CFLAGS and CXXFLAGS.
  
=Startup=
+
For example, if you want to compile a version with higher optimization values, you should use the following command when running configure (assuming bourne shell or derived shell):
==Running the Daemon==
+
CFLAGS='-O0' CXXFLAGS="$CFLAGS" ./configure
Fakeroot-ng has two modes of operation. In the non-persistent mode any changes to the file system are discarded once the debugged process and all of its children exit. In the persistent mode (invoking frng with the ''-p'' option) the database is loaded from the state file (if it exists), and saved back to it once the daemon exits.
 
  
Running the daemon is managed through the '''daemonCtrl''' class. The standard mode of running a debugged process is fork+exec. In it, the parent forks, the child runs ptrace(PTRACE_TRACEME) and then performs execve to run the actual command. The parent is then responsible to wait for the child to finish.
+
In this particular case, it would have been shorter to just repeat the flags for the CXXFLAGS variable, but this mechanism ensures that the C and C++ compiler receive the same flags. Also note that this does not apply to preprocessor defines, which are shared between C and C++. So, if all we wanted was to disable all of the asserts in the code, we would have done:
 +
CPPFLAGS='-DNDEBUG' ./configure
  
This mode is not how frng works. Instead, the command to be run is a direct child of the shell that started frng. The daemon, if a new one is needed, detaches itself from the environment in the [http://en.wikipedia.org/wiki/Daemon_%28computing%29#Creation usual way]. The debugee is, therefor, the grandparent of the debugger. For debug purposes, it is sometimes beneficial for frng not to disassociate itself from the standard output and standard input of the calling shell, or to change directory to / (otherwise a core file will not be generated). The ''-d'' option to frng causes it to not perform full daemonization.
+
==Enabling Extra Warnings==
 +
It is possible to raise the warning level when compiling fakeroot-ng. It is written to be warning free when compiled with '''-Wall -Wextra'''. The only exception is the ''unused parameter'' warning, which I find unhelpful and counter-productive to eliminate. My debug environment is set up with the following command:
 +
CFLAGS='-O0 -Wall -Wextra -Wno-unused-parameter' CXXFLAGS="$CFLAGS" ./configure
  
Despite the non-standard relation, throughout this document the debugee is sometimes referred to as the "child" process. This is due to the fact that, working with ptrace, all debugees' status is retrieved via the [http://linux.die.net/man/2/wait wait] system call.
+
==Configure error: Fakeroot-ng needs a compiler that supports C++11==
 +
As of version 0.18 of Fakeroot-ng, a compiler with a fairly complete support for the new C++11 standard is required in order to compile the program. In particular, fakeroot-ng makes use of ''decltype''[http://en.wikipedia.org/wiki/Decltype], ''nullptr''[http://www.codeguru.com/cpp/cpp/article.php/c19093/C-2011-nullptr.htm] and ''std::unordered_map''[http://www.cplusplus.com/reference/unordered_map/unordered_map/].
  
This non-standard mode of operation allows fakeroot-ng to not affect script usage of commands. If the child itself daemonizes, running it under fakeroot-ng from a script will let the script continue execution as soon as the child finishes daemonizing. Compare this to running such a process through strace, where strace will only return when the last traced process finishes.
+
During the configure stage, fakeroot-ng tries to compile a small test program that utilizes all three capabilities, and checks which compiler flags are necessary in order to activate the relevant C++ language support. Mostly, this involves testing the "-std=c++11" flag, required for gcc.
  
==Daemon Control Sockets==
+
If no flag passes the compilation, your configure state will look something like this:
Before running the actual command, the child process establishes a socket with the debugger process. Through this socket the child sends two commands. "Reserve" and "Attach". Once the second command is done, the child is free to execve the actual command to be executed.
 
  
==Non-Persistent State Invocation==
+
checking for PTRACE_GETREGS... yes
In the non-persistent case, every invocation of frng must result in a new daemon running. The communication socket with the daemon is opened using [http://linux.die.net/man/2/socketpair socketpair]. Once the child performs execve, the only new processes to attach to the debugger are descendants of the original child process.
+
checking for PTRACE_PEEKUSER... yes
 +
checking what flags are needed for compiler support of the ISO C++11 standard... not supported
 +
configure: error: Fakeroot-ng needs a compiler that supports C++11. If you are using gcc, that
 +
                  means version 4.7 or higher.
  
In this mode of operation, as soon as the last such descendant exits, the daemon also exits.
+
If you are using gcc with a version lower than 4.7, then your only option is to upgrade. Versions before 4.7 simply did not support the C++11 standard to the extent required by fakeroot-ng. If you are using another compiler, in particular, one that does support ''nullptr'', ''decltype'' and ''std::unordered_map'', then you might be missing the correct compilation command line. Find out what those are. As a temporary workaround, pass them through the CXXFLAGS environment variable to configure. If that works, then do contact us on the [[support|mailing list]] and we'll gladly add that detection into the configure script.
  
==Persistent State Invocation==
+
==Building a i386 binary on an AMD64/X86_64 host==
If frng is invoked with the ''-p'' option, it is committed to providing a consistent view of the file system to all processes that asked to use the same state file. This is true whether these processes are descendants of the original child process or are a result of an unrelated invocation of frng that gave the same state file.
+
Sometimes it is desired to build a 32bit target on a computer that, itself, is running a 64bit operating system. This is possible, but is not as straight forward as with other projects.
  
This means that, on startup, frng needs to decide whether to create a new daemon, or whether to try to attach to an existing daemon. At any time a child process exists, exactly one daemon must be running to handle it.
+
First, make sure that your installed compiler knows how to generate 32bit binaries. On Debian based systems this requires installing the "g++-multilib" package. If you only installed the "gcc-multilib" package, then your compiler will know how to generate 32 binaries for C, but not C++.
  
There is a potential race with this strategy. If one instance of the daemon is in the process of exiting while another is just starting up, the new instance might try to connect to the old instance's socket after the old instance is no longer listening on it. This bug was found and diagnosed by [http://sourceforge.net/p/fakerootng/mailman/message/25749940/ Russell Yanofsky].
+
Once that's resolved, two things must be done. The first is to tell the build system you are cross compiling. This is done by passing ''configure'' the option '''--target=i686-pc-linux-gnu'''. This is important, as fakeroot-ng has specific code for each destination platform, and so must know which platform it is compiling for. This, however, does not actually causes make to generate 32 bit binaries (which is probably a bug in autoconf). To actually generate 32 bit binaries, the '''-m32''' flags must be passed to gcc. See the above section on passing custom compiler flags. The complete command line to compile 32 bit is:
 +
CFLAGS='-O2 -m32' CXXFLAGS="$CFLAGS" ./configure --target=i686-pc-linux-gnu
  
To prevent this race, the following sequence is employed:
+
===Error messages, and their meanings===
# Open the state file (creating it if not already existing), and try to lock it using [http://linux.die.net/man/2/flock flock].
+
In each error message, the important part is displayed in bold:
# If the lock succeeds, no other daemon is running (and none can start, as we now hold the lock), create the daemon which will:
 
## Create a SOCK_SEQPACKET Unix domain socket and start listening on it. The socket is bound to a file with the same name as the state file, with the extension ''.sock''.
 
# Whether or not we tried to create a daemon, close the state file and try to connect to the ''state.sock'' socket and send it the "Reserve" command.
 
# If the connection was refused or hung up without sending us a reply to the "Reserve" then the daemon has exited. Retry everything up to this point.
 
# If the Reserve succeeded, then the existing daemon is guaranteed to not exit until we do. Send it an Attach and continue to the fork.
 
  
This sequence serves two purposes. The first is to eliminate the race mentioned above. If the current daemon is in the process of shutting down it will not handle our Reserve request. Once it closes its socket, we'll get a hangup and retry the entire process. At this point we will, likely, get the lock, and launch our own daemon. This is done without busy waiting.
+
arch/linux/x86_64/platform.c:453:50: error: '''‘RIP’ undeclared (first use in this function)'''
 +
This means that you told the compiler to compile in 32 bit, but did not tell configure that your target is 32 bit.
  
The second purpose this serves is that it allows us to know the debugger's PID before it tries to attach to us. This is important on Linux systems with the Yama security module enabled. On those systems, the child must run a special [http://linux.die.net/man/2/prctl prctl] command to tell the system which PID is allowed to attach to it. This sequence allows frng to work on Yama ptrace_scope of 1 without any manual work by the user. Since that is the default on at least some versions of Ubuntu, this is rather important.
+
arch/linux/i386/platform.c:80:44: error: '''‘ORIG_EAX’ undeclared (first use in this function'''
 +
This means that you told fakeroot-ng you are compiling for a 32 bit target, but did not tell the compiler to compile for 32 bit.
 +
 
 +
parent.cpp: In function ‘void init_handlers()’:
 +
parent.cpp:210:1: error: '''‘sys_fstatat64’ was not declared in this scope'''
 +
parent.cpp: In function ‘bool hma_state0(int, pid_t, pid_state*)’:
 +
parent.cpp:986:29: error: '''‘SYS_mmap2’ was not declared in this scope'''
 +
parent.cpp: In function ‘bool hma_state3(int, pid_t, pid_state*)’:
 +
parent.cpp:1054:38: error: '''‘SYS_mmap2’ was not declared in this scope'''
 +
This means that you passed the flags to compile 32 bit to CFLAGS, but not to CXXFLAGS.

Latest revision as of 15:57, 22 April 2019

Compiling Fakeroot-NG

Here are a few tips and solutions to possible compilation problems:

Generic Compilation Instructions

In general, as the README file says, compiling fakeroot-ng is just a matter of running:

./configure
make
sudo make install

The project is an automake project, which means it supports out of tree builds, install targets and changing the installation path. Still, there are a few caveats that might be a problem, under certain conditions.

Passing Custom Compiler Flags

The following holds for any autoconf based project.

During the configure stage, the environment is read for environment variables for CFLAGS and CXXFLAGS (a full list is available by running ./configure --help). These values are then coded into the generated Makefile, and need not be set later on.

For fakeroot-ng in particular, care must be taken that the ptlib library is written in C, while fakeroot-ng itself is written in C++. As a result, it is important to set any flags you want for both CFLAGS and CXXFLAGS.

For example, if you want to compile a version with higher optimization values, you should use the following command when running configure (assuming bourne shell or derived shell):

CFLAGS='-O0' CXXFLAGS="$CFLAGS" ./configure

In this particular case, it would have been shorter to just repeat the flags for the CXXFLAGS variable, but this mechanism ensures that the C and C++ compiler receive the same flags. Also note that this does not apply to preprocessor defines, which are shared between C and C++. So, if all we wanted was to disable all of the asserts in the code, we would have done:

CPPFLAGS='-DNDEBUG' ./configure

Enabling Extra Warnings

It is possible to raise the warning level when compiling fakeroot-ng. It is written to be warning free when compiled with -Wall -Wextra. The only exception is the unused parameter warning, which I find unhelpful and counter-productive to eliminate. My debug environment is set up with the following command:

CFLAGS='-O0 -Wall -Wextra -Wno-unused-parameter' CXXFLAGS="$CFLAGS" ./configure

Configure error: Fakeroot-ng needs a compiler that supports C++11

As of version 0.18 of Fakeroot-ng, a compiler with a fairly complete support for the new C++11 standard is required in order to compile the program. In particular, fakeroot-ng makes use of decltype[1], nullptr[2] and std::unordered_map[3].

During the configure stage, fakeroot-ng tries to compile a small test program that utilizes all three capabilities, and checks which compiler flags are necessary in order to activate the relevant C++ language support. Mostly, this involves testing the "-std=c++11" flag, required for gcc.

If no flag passes the compilation, your configure state will look something like this:

checking for PTRACE_GETREGS... yes
checking for PTRACE_PEEKUSER... yes
checking what flags are needed for compiler support of the ISO C++11 standard... not supported
configure: error: Fakeroot-ng needs a compiler that supports C++11. If you are using gcc, that
                  means version 4.7 or higher.

If you are using gcc with a version lower than 4.7, then your only option is to upgrade. Versions before 4.7 simply did not support the C++11 standard to the extent required by fakeroot-ng. If you are using another compiler, in particular, one that does support nullptr, decltype and std::unordered_map, then you might be missing the correct compilation command line. Find out what those are. As a temporary workaround, pass them through the CXXFLAGS environment variable to configure. If that works, then do contact us on the mailing list and we'll gladly add that detection into the configure script.

Building a i386 binary on an AMD64/X86_64 host

Sometimes it is desired to build a 32bit target on a computer that, itself, is running a 64bit operating system. This is possible, but is not as straight forward as with other projects.

First, make sure that your installed compiler knows how to generate 32bit binaries. On Debian based systems this requires installing the "g++-multilib" package. If you only installed the "gcc-multilib" package, then your compiler will know how to generate 32 binaries for C, but not C++.

Once that's resolved, two things must be done. The first is to tell the build system you are cross compiling. This is done by passing configure the option --target=i686-pc-linux-gnu. This is important, as fakeroot-ng has specific code for each destination platform, and so must know which platform it is compiling for. This, however, does not actually causes make to generate 32 bit binaries (which is probably a bug in autoconf). To actually generate 32 bit binaries, the -m32 flags must be passed to gcc. See the above section on passing custom compiler flags. The complete command line to compile 32 bit is:

CFLAGS='-O2 -m32' CXXFLAGS="$CFLAGS" ./configure --target=i686-pc-linux-gnu

Error messages, and their meanings

In each error message, the important part is displayed in bold:

arch/linux/x86_64/platform.c:453:50: error: ‘RIP’ undeclared (first use in this function)

This means that you told the compiler to compile in 32 bit, but did not tell configure that your target is 32 bit.

arch/linux/i386/platform.c:80:44: error: ‘ORIG_EAX’ undeclared (first use in this function

This means that you told fakeroot-ng you are compiling for a 32 bit target, but did not tell the compiler to compile for 32 bit.

parent.cpp: In function ‘void init_handlers()’:
parent.cpp:210:1: error: ‘sys_fstatat64’ was not declared in this scope
parent.cpp: In function ‘bool hma_state0(int, pid_t, pid_state*)’:
parent.cpp:986:29: error: ‘SYS_mmap2’ was not declared in this scope
parent.cpp: In function ‘bool hma_state3(int, pid_t, pid_state*)’:
parent.cpp:1054:38: error: ‘SYS_mmap2’ was not declared in this scope

This means that you passed the flags to compile 32 bit to CFLAGS, but not to CXXFLAGS.