https://fakeroot-ng.lingnu.com/index.php?title=Special:NewPages&feed=atom&limit=50&offset=&namespace=0&username=&tagfilter=&size-mode=max&size=0Fakeroot NG - New pages [en]2024-03-29T11:59:21ZFrom Fakeroot NGMediaWiki 1.32.0https://fakeroot-ng.lingnu.com/index.php?title=PTRACE_LD_PRELOAD_comparisonPTRACE LD PRELOAD comparison2019-04-22T14:00:02Z<p>Shachar: Move page from old installation</p>
<hr />
<div>=LD_PRELOAD=<br />
Most C compiled programs do not communicate directly with the kernel. Instead, the program calls a function defined inside the runtime library (on Linux, typically, glibc), which then proceeds to perform whatever preparations are necessary (such as copying the arguments into registers) and trigger the system call. The connection between the program and the runtime library's functions is performed when the program is loaded by the run time linker (on Linux, typically called /lib/ld-linux.so.6 or similar name).<br />
<br />
If an environment variable named LD_PRELOAD is defined, any shared object linked by it is loaded by the dynamic linker first, before the dependencies explicitly stated in the program file. As a result, if this shared object has functions bearing the same name as functions defined in the run time library, the former's functions will be called. This allows a library to "wrap" the run time library's original functions, intercepting system calls. A shared object implementing LD_PRELOAD has the option to call the original system call, not call it, or even call several unrelated system calls, all using a standard C function.<br />
<br />
Programming an LD_PRELOAD library, while not trivial, is a relatively straight forward task.<br />
<br />
=PTRACE=<br />
Ptrace is the name of a system call that handles the case where one process acts as a debugger for another. The debugger process can attach to another process, or the debugee can ask for its parent to become its debugger. The debugger then controls the debugee, allowing it to query its state, control the delivery of signals, and to single step it, run it until a signal, or run it until the debugee performs a system call.<br />
<br />
While the function itself is fairly well documented, this documentation is very superficial. Almost everything about working with ptrace is extremely platform dependent. For example, the ptrace interface defines a command for retrieving a process' CPU registers. While this is a standard command, which registers exist is CPU dependent, the way they are numbered depends on the platform, and what each register means depends on both. Theoretically, you can write a cross platform program that manipulates the registers, but practically, anything you do has to be both operating system and CPU specific.<br />
<br />
The theory of wrapping system calls with ptrace is this: Run the program "until syscall". When a system call arrives, the debugger gets called twice. Once right before the kernel starts processing the system call, and one immediately after it finished. Ptrace allows you to change all the registers involved in the system call (including the one determining which system call is being carried out), but does not allow you to cancel the system call. Ptrace also does not allow you to generate a system call out of the blue.<br />
<br />
Of course, fakeroot-ng has to be able to do all three, plus much more. This is achieved by a host of tricks, described in the [[technical details]] page.</div>Shacharhttps://fakeroot-ng.lingnu.com/index.php?title=OutOfScopeOutOfScope2019-04-22T13:59:28Z<p>Shachar: Move page from old installation</p>
<hr />
<div>=Drawing the Line=<br />
Fakeroot-ng's design makes it possible to emulate just about any behavior that root can do, without escalating privileges. In fact, the "[http://user-mode-linux.sourceforge.net/ user mode linux]" project emulates an entire kernel using the very same ptrace technology employed by fakeroot-ng. Just because it is possible to run an entire kernel this way, however, does not mean this is desired.<br />
<br />
==Out of Scope==<br />
Some capabilities available for a root user are simply out of scope for the fakeroot-ng project. The following layers of emulation are, theoretically, possible in fakeroot-ng, but no such support is planned.<br />
* Containers<br />
** Mount namespaces<br />
* Xattributes<br />
* Capabilities<br />
* Privileges drop<br />
* Proc filesystem emulation (under consideration)</div>Shacharhttps://fakeroot-ng.lingnu.com/index.php?title=Fakeroot_vs_fakeroot-ng_comparisonFakeroot vs fakeroot-ng comparison2019-04-22T13:58:39Z<p>Shachar: Move page from old installation</p>
<hr />
<div>Some people are wondering whether fakeroot-ng is meant to kill off fakeroot. It is important to understand that fakeroot, developed by Clint Adams and Timo Savola, is an independent project that will live so long as they (or anyone else) decided to keep it alive. Trying to provide a quick answer, there are some area where it is almost impossible for fakeroot-ng to compete with fakeroot, just as there are some areas where fakeroot has no way to compete with fakeroot-ng. It can be said that fakeroot's coverage is wider, while fakeroot-ng's is deeper.<br />
<br />
=Design advantages of fakeroot=<br />
==Speed==<br />
Fakeroot-ng runs as a separate process. Every time the program performs a system call, the kernel performs at least two context switchs to fakeroot-ng (one before and one after the system call), even if the system call is not handled. As a result, at least under some scenarios, fakeroot-ng is considerably slower than fakeroot.<br />
<br />
For example, creating a copy of the /dev folder:<br />
<source lang=bash><br />
$ time fakeroot cp -a /dev/ /tmp/<br />
<br />
real 0m0.132s<br />
user 0m0.028s<br />
sys 0m0.076s<br />
$ rm -rf /tmp/dev<br />
$ time fakeroot-ng cp -a /dev/ /tmp/<br />
<br />
real 0m0.434s<br />
user 0m0.020s<br />
sys 0m0.076s<br />
</source><br />
<br />
building fakeroot-ng:<br />
<source lang=bash><br />
$ time fakeroot make -j2<br />
real 0m0.900s<br />
user 0m0.776s<br />
sys 0m0.284s<br />
$ time fakeroot-ng make -j2<br />
real 0m1.816s<br />
user 0m0.784s<br />
sys 0m0.384s<br />
</source><br />
<br />
==Platform Support==<br />
Fakeroot runs as part of the process, getting standard C interface function calls (glibc versioning trickery non-withstanding). Porting fakeroot to a new platform for an already supported operating system (say, a new CPU of Linux) is, in theory, just a matter of recompiling for the new platform. As a result, fakeroot easily spans all hardware platforms supported by, for example, Debian, including the non-Linux platforms.<br />
<br />
The need to know which registers are used, how system calls are performed and other parameters mean that fakeroot-ng needs to be ported to each new operating system/CPU combination individually. As a result, at the time of this writing, only Linux is supported, and there, only PowerPC and Intel's x86 and x86_64 platforms are supported. Further platforms support depends on someone familiar with the platforms writing the support code needed. The original author now has access to an ARM and a SPARC platforms, so support for these two platforms is not unlikely in the near future.<br />
<br />
Another aspect of the same problem is that, at least on Linux, the interface to the kernel is sometimes significantly different than the one defined by Posix or the Linux man pages. It is not uncommon to see a function defined in one way by the manual page, but to actually have that function translated into another by glibc, and implemented differently by the kernel. The different hooking locations mean that fakeroot has to implement the standard behavior, while fakeroot-ng needs to implement the platform dependent behavior.<br />
<br />
==Simplicity of Design==<br />
From the July 26, 1997 version of the fakeroot manual page:<br />
<blockquote>'''Bugs'''<br><br />
None so far. Be warned, though: although I've written quite a few much larger (and smaller) programs, I've never written anything that was as tiny as fakeroot, had as many bugs as fakeroot, and still was as usable as, say, fakeroot version 0.0_3, the first version that could be used to build itself.</blockquote><br />
<br />
This simplicity cannot be attributed to fakeroot-ng, no matter how you stretch it. Fakeroot-ng is built as a stateful non-blocking server, attempting to use a single process to handle as many simultaneous requests from different processes as possible. It uses hacks, tricks and, occasionally, coercion to make the debugged program do what it wants. There is nothing simple about it.<br />
<br />
=Fakeroot-ng advantages=<br />
Many of the advantages listed below boil down to one point - with ptrace, the control you have over the application is absolute. As such, there is nothing you can't do.<br />
<br />
==Static linked executables==<br />
Because fakeroot uses the dynamic linker to wrap the system calls, executables that are statically linked, use a non-standard dynamic linker (that does not honor LD_PRELOAD), or that do not use glibc in order to access the kernel are not visible to fakeroot, and cannot be wrapped by it. While most programs are dynamically linked, a major example of a statically linked binary that is very often used is the dynamic linker itself. Fakeroot does not, and cannot, wrap the system calls performed by it. Fakeroot-ng has no problem.<br />
<br />
==Fakeroot Limitations==<br />
The man page for fakeroot has a section titled "Limitations". They list three major limitations. At the time of writing, fakeroot-ng has none of those limitations. It does not depend on the version of any library used, does wrap "open" and "create" in a reliable race free manner, and does not care about environment variables or any other by-product semantics, and therefor runs "configure' without a problem.<br />
<br />
Fakeroot-ng does have its own limitations, but they are a result of how far the author wished to go emulating the kernel, and not any inherent limitations of the technology. At the moment, the most pressing limitation is that the dynamic linker is loaded by the kernel, and is therefor loaded from the real root of the system, and not from the chroot (if applicable). There is a plan laid out to fix this.<br />
<br />
==Handling open and chroot==<br />
As fakeroot-ng has no problem in handling open, file ownership for unknown files is their real ownership. As a result, immediately after running fakeroot-ng, there is no change in how old files are displayed. New files are, of course, owned by the fake uid of the process (actually, the fake file system uid). Fakeroot-ng also supports chroot environments. Both are unsupported by fakeroot, due to its inability to handle "open".<br />
<br />
Please note that there are other projects (most notable - fakechroot) that do wrap the open system call. Check out the [[fakeroot-ng vs fakechroot]] page for more details on it.<br />
<br />
==Library and Language Independence==<br />
Somewhat covered above, but worth expanding. Fakeroot needs to be linked with the precise same glibc version as the one the end program is linked. If the versions don't match, the best case is that fakeroot does not work at all, and the worse case is that subtle bugs appear in unexpected places. This may not be a huge problem for open source programs compiled as part of the distribution, but when 3<sup>rd</sup> party, and especially closed source precompiled programs, are involved, this might be a major drawback.<br />
<br />
==Program Address Space Presence==<br />
Fakeroot is a library that gets loaded into the program's address space. This means that any further libraries that fakeroot may need to load itself are also loaded into the program's address space. Furthermore, any data structures that fakeroot might need for normal operation are subtracted from the memory available for running the actual program. Fakeroot's solution (not only due to this reason) is to "outsource" all database required to an external daemon, that performs the actual data lookup (mostly the file fake owners and type).<br />
<br />
Fakeroot-ng's presence inside the programs's address space, on the other hand, is minimal. The program runs while totally oblivious to the fact it is being debugged. Fakeroot-ng is written in C++, but it does not load any C++ run time library into the program's address space. The entire program address space presence is limited to a handful of memory pages, typically four, allocated explicitly by fakeroot-ng.<br />
<br />
==Security==<br />
Security is a goal for neither fakeroot nor fakeroot-ng. Still, as far as the technology goes, there is a world of difference between the two. With fakeroot security is not a goal partly because, had it been a goal, it would be unobtainable. Bypassing the fakeroot syscall wrapping is as easy as compiling part of the program statically. As of this writing, fakeroot actually exhibits '''accidental''' breaking away of processes run by an autoconf generated configure script.<br />
<br />
Fakeroot-ng, on the other hand, uses a technology that leaves the actual program no say over whether it wants to be traced. The system calls hooks are enforced by the kernel, and the data stored in the pages mapped into the process is protected by the machine's MMU from alteration. While the system may not be full proof, it can be made so if desired.<br />
<br />
=Relative Roles=<br />
<br />
Fakeroot-ng is not likely to kill fakeroot, nor is it likely to any time soon. Fakeroot's wider platform support and better performance means it is likely to be the preferred tool wherever it gets the job done. Fakeroot is still continuously developed, and, in any case, is much more mature than fakeroot-ng.<br />
<br />
That said, fakeroot-ng does have capabilities that fakeroot does not, and where those are needed, it makes an excellent tool.</div>Shacharhttps://fakeroot-ng.lingnu.com/index.php?title=HowtoCompileHowtoCompile2019-04-22T13:57:54Z<p>Shachar: Move page from old installation</p>
<hr />
<div>=Compiling Fakeroot-NG=<br />
<br />
Here are a few tips and solutions to possible compilation problems:<br />
<br />
==Generic Compilation Instructions==<br />
In general, as the README file says, compiling fakeroot-ng is just a matter of running:<br />
./configure<br />
make<br />
sudo make install<br />
<br />
The project is an automake project, which means it supports out of tree builds, install targets and changing the installation path. Still, there are a few caveats that might be a problem, under certain conditions.<br />
<br />
==Passing Custom Compiler Flags==<br />
The following holds for any autoconf based project.<br />
<br />
During the ''configure'' stage, the environment is read for environment variables for CFLAGS and CXXFLAGS (a full list is available by running '''./configure --help'''). These values are then coded into the generated Makefile, and need not be set later on.<br />
<br />
For fakeroot-ng in particular, care must be taken that the ptlib library is written in C, while fakeroot-ng itself is written in C++. As a result, it is important to set any flags you want for both CFLAGS and CXXFLAGS.<br />
<br />
For example, if you want to compile a version with higher optimization values, you should use the following command when running configure (assuming bourne shell or derived shell):<br />
CFLAGS='-O0' CXXFLAGS="$CFLAGS" ./configure<br />
<br />
In this particular case, it would have been shorter to just repeat the flags for the CXXFLAGS variable, but this mechanism ensures that the C and C++ compiler receive the same flags. Also note that this does not apply to preprocessor defines, which are shared between C and C++. So, if all we wanted was to disable all of the asserts in the code, we would have done:<br />
CPPFLAGS='-DNDEBUG' ./configure<br />
<br />
==Enabling Extra Warnings==<br />
It is possible to raise the warning level when compiling fakeroot-ng. It is written to be warning free when compiled with '''-Wall -Wextra'''. The only exception is the ''unused parameter'' warning, which I find unhelpful and counter-productive to eliminate. My debug environment is set up with the following command:<br />
CFLAGS='-O0 -Wall -Wextra -Wno-unused-parameter' CXXFLAGS="$CFLAGS" ./configure<br />
<br />
==Configure error: Fakeroot-ng needs a compiler that supports C++11==<br />
As of version 0.18 of Fakeroot-ng, a compiler with a fairly complete support for the new C++11 standard is required in order to compile the program. In particular, fakeroot-ng makes use of ''decltype''[http://en.wikipedia.org/wiki/Decltype], ''nullptr''[http://www.codeguru.com/cpp/cpp/article.php/c19093/C-2011-nullptr.htm] and ''std::unordered_map''[http://www.cplusplus.com/reference/unordered_map/unordered_map/].<br />
<br />
During the configure stage, fakeroot-ng tries to compile a small test program that utilizes all three capabilities, and checks which compiler flags are necessary in order to activate the relevant C++ language support. Mostly, this involves testing the "-std=c++11" flag, required for gcc.<br />
<br />
If no flag passes the compilation, your configure state will look something like this:<br />
<br />
checking for PTRACE_GETREGS... yes<br />
checking for PTRACE_PEEKUSER... yes<br />
checking what flags are needed for compiler support of the ISO C++11 standard... not supported<br />
configure: error: Fakeroot-ng needs a compiler that supports C++11. If you are using gcc, that<br />
means version 4.7 or higher.<br />
<br />
If you are using gcc with a version lower than 4.7, then your only option is to upgrade. Versions before 4.7 simply did not support the C++11 standard to the extent required by fakeroot-ng. If you are using another compiler, in particular, one that does support ''nullptr'', ''decltype'' and ''std::unordered_map'', then you might be missing the correct compilation command line. Find out what those are. As a temporary workaround, pass them through the CXXFLAGS environment variable to configure. If that works, then do contact us on the [[support|mailing list]] and we'll gladly add that detection into the configure script.<br />
<br />
==Building a i386 binary on an AMD64/X86_64 host==<br />
Sometimes it is desired to build a 32bit target on a computer that, itself, is running a 64bit operating system. This is possible, but is not as straight forward as with other projects.<br />
<br />
First, make sure that your installed compiler knows how to generate 32bit binaries. On Debian based systems this requires installing the "g++-multilib" package. If you only installed the "gcc-multilib" package, then your compiler will know how to generate 32 binaries for C, but not C++.<br />
<br />
Once that's resolved, two things must be done. The first is to tell the build system you are cross compiling. This is done by passing ''configure'' the option '''--target=i686-pc-linux-gnu'''. This is important, as fakeroot-ng has specific code for each destination platform, and so must know which platform it is compiling for. This, however, does not actually causes make to generate 32 bit binaries (which is probably a bug in autoconf). To actually generate 32 bit binaries, the '''-m32''' flags must be passed to gcc. See the above section on passing custom compiler flags. The complete command line to compile 32 bit is:<br />
CFLAGS='-O2 -m32' CXXFLAGS="$CFLAGS" ./configure --target=i686-pc-linux-gnu<br />
<br />
===Error messages, and their meanings===<br />
In each error message, the important part is displayed in bold:<br />
<br />
arch/linux/x86_64/platform.c:453:50: error: '''‘RIP’ undeclared (first use in this function)'''<br />
This means that you told the compiler to compile in 32 bit, but did not tell configure that your target is 32 bit.<br />
<br />
arch/linux/i386/platform.c:80:44: error: '''‘ORIG_EAX’ undeclared (first use in this function'''<br />
This means that you told fakeroot-ng you are compiling for a 32 bit target, but did not tell the compiler to compile for 32 bit.<br />
<br />
parent.cpp: In function ‘void init_handlers()’:<br />
parent.cpp:210:1: error: '''‘sys_fstatat64’ was not declared in this scope'''<br />
parent.cpp: In function ‘bool hma_state0(int, pid_t, pid_state*)’:<br />
parent.cpp:986:29: error: '''‘SYS_mmap2’ was not declared in this scope'''<br />
parent.cpp: In function ‘bool hma_state3(int, pid_t, pid_state*)’:<br />
parent.cpp:1054:38: error: '''‘SYS_mmap2’ was not declared in this scope'''<br />
This means that you passed the flags to compile 32 bit to CFLAGS, but not to CXXFLAGS.</div>Shacharhttps://fakeroot-ng.lingnu.com/index.php?title=ArchitectureArchitecture2019-04-22T13:57:00Z<p>Shachar: Move page from old installation</p>
<hr />
<div>This page documents the software architecture of fakeroot-ng. The documentation is relevant to the (as yet unreleased) multi-threaded debugger branch.<br />
<br />
Throughout this page, the acronym "frng" is used to refer to fakeroot-ng.<br />
<br />
=The Daemon=<br />
Note: some people think that daemons are, by necessity, system startup processes. This is not the case. A daemon of any process that disassociates itself from the environment that started it.<br />
<br />
Fakeroot-ng act as a debugger (tracer) for the faked root process. To do this, it creates an extra process and disassociates it from the running environment. This is the fakeroot-ng daemon, and it is an unprivileged process, without any special permissions.<br />
<br />
==File Lie Database==<br />
Fakeroot-ng allows processes to carry out changes to the file system that would require root permission. These include, among other things, changing ownership and device files. Since no fakeroot-ng related process actually has privileges, this is done by faking success on the creation system calls, and then lying about the effect to all future requests. If you create a character device, and then ask for a list of files in the directory, fakeroot-ng will lie to you that the file is a character device. <br />
<br />
The lies frng tells the process are indexed in an [http://en.cppreference.com/w/cpp/container/unordered_map unordered_map]. The index key is the file's device and inode numbers. This allows the file lie database to persist across different processes, and also across invocations of frng. Also, since all of those lies are related to the [http://linux.die.net/man/2/stat stat] structure, this indexing is extremely convenient.<br />
<br />
The database is maintained in '''file_lie.h''' and '''file_lie.cpp'''.<br />
<br />
=Startup=<br />
==Running the Daemon==<br />
Fakeroot-ng has two modes of operation. In the non-persistent mode any changes to the file system are discarded once the debugged process and all of its children exit. In the persistent mode (invoking frng with the ''-p'' option) the database is loaded from the state file (if it exists), and saved back to it once the daemon exits.<br />
<br />
Running the daemon is managed through the '''daemonCtrl''' class. The standard mode of running a debugged process is fork+exec. In it, the parent forks, the child runs ptrace(PTRACE_TRACEME) and then performs execve to run the actual command. The parent is then responsible to wait for the child to finish.<br />
<br />
This mode is not how frng works. Instead, the command to be run is a direct child of the shell that started frng. The daemon, if a new one is needed, detaches itself from the environment in the [http://en.wikipedia.org/wiki/Daemon_%28computing%29#Creation usual way]. The debugee is, therefor, the grandparent of the debugger. For debug purposes, it is sometimes beneficial for frng not to disassociate itself from the standard output and standard input of the calling shell, or to change directory to / (otherwise a core file will not be generated). The ''-d'' option to frng causes it to not perform full daemonization.<br />
<br />
Despite the non-standard relation, throughout this document the debugee is sometimes referred to as the "child" process. This is due to the fact that, working with ptrace, all debugees' status is retrieved via the [http://linux.die.net/man/2/wait wait] system call.<br />
<br />
This non-standard mode of operation allows fakeroot-ng to not affect script usage of commands. If the child itself daemonizes, running it under fakeroot-ng from a script will let the script continue execution as soon as the child finishes daemonizing. Compare this to running such a process through strace, where strace will only return when the last traced process finishes.<br />
<br />
==Daemon Control Sockets==<br />
Before running the actual command, the child process establishes a socket with the debugger process. Through this socket the child sends two commands. "Reserve" and "Attach". Once the second command is done, the child is free to execve the actual command to be executed.<br />
<br />
==Non-Persistent State Invocation==<br />
In the non-persistent case, every invocation of frng must result in a new daemon running. The communication socket with the daemon is opened using [http://linux.die.net/man/2/socketpair socketpair]. Once the child performs execve, the only new processes to attach to the debugger are descendants of the original child process.<br />
<br />
In this mode of operation, as soon as the last such descendant exits, the daemon also exits.<br />
<br />
==Persistent State Invocation==<br />
If frng is invoked with the ''-p'' option, it is committed to providing a consistent view of the file system to all processes that asked to use the same state file. This is true whether these processes are descendants of the original child process or are a result of an unrelated invocation of frng that gave the same state file.<br />
<br />
This means that, on startup, frng needs to decide whether to create a new daemon, or whether to try to attach to an existing daemon. At any time a child process exists, exactly one daemon must be running to handle it.<br />
<br />
There is a potential race with this strategy. If one instance of the daemon is in the process of exiting while another is just starting up, the new instance might try to connect to the old instance's socket after the old instance is no longer listening on it. This bug was found and diagnosed by [http://sourceforge.net/p/fakerootng/mailman/message/25749940/ Russell Yanofsky].<br />
<br />
To prevent this race, the following sequence is employed:<br />
# Open the state file (creating it if not already existing), and try to lock it using [http://linux.die.net/man/2/flock flock].<br />
# If the lock succeeds, no other daemon is running (and none can start, as we now hold the lock), create the daemon which will:<br />
## Create a SOCK_SEQPACKET Unix domain socket and start listening on it. The socket is bound to a file with the same name as the state file, with the extension ''.sock''.<br />
# Whether or not we tried to create a daemon, close the state file and try to connect to the ''state.sock'' socket and send it the "Reserve" command.<br />
# If the connection was refused or hung up without sending us a reply to the "Reserve" then the daemon has exited. Retry everything up to this point.<br />
# If the Reserve succeeded, then the existing daemon is guaranteed to not exit until we do. Send it an Attach and continue to the fork.<br />
<br />
This sequence serves two purposes. The first is to eliminate the race mentioned above. If the current daemon is in the process of shutting down it will not handle our Reserve request. Once it closes its socket, we'll get a hangup and retry the entire process. At this point we will, likely, get the lock, and launch our own daemon. This is done without busy waiting.<br />
<br />
The second purpose this serves is that it allows us to know the debugger's PID before it tries to attach to us. This is important on Linux systems with the Yama security module enabled. On those systems, the child must run a special [http://linux.die.net/man/2/prctl prctl] command to tell the system which PID is allowed to attach to it. This sequence allows frng to work on Yama ptrace_scope of 1 without any manual work by the user. Since that is the default on at least some versions of Ubuntu, this is rather important.</div>Shachar