Adding some extra functionality into a running UNIX kernel in ad-hoc
fashion is quite easy; creating an extensible system for doing so is
very hard. As usual, the best way to see it is to study examples; we
first look at the simplest tricks used by rootkits, and progress to
advanced -- and highly successful -- kernel architectures like
Netfilter.

At its simplest, the kernel is just a program written mostly in C,
whose parts get called by other processes or interrupts. The locations
of its functions are known or can be deduced, and these functions can
be patched. Moreover, good software engineering designs collect the 
pointers to important top-level and worker functions in dispatch tables
(sys_call_table, VFS tables, Solaris memory segment tables and other
object-oriented techniques, etc.) -- and "hooking" a function becomes
as easy as replacing a pointer to it in the appropriate table. 

Such structure makes the kernel easier to maintain, update, and to
compose new functionality into it. In case of rootkits, this
functionality may be undesirable to the system's operator (a sysadmin
would arguably prefer any attacker attempts to modify the kernel to
crash the system than to succeed quietly without his knowledge), but
any gains from making composition harder tend to also make it harder
to apply bona-fide updates, and, so long as software is not perfect,
updates are a necessity for both engineering and economic reasons.

1. Simple rootkits

The simplest rootkits insert code into the kernel's syscalls, by
patching the syscall table, in order to hide files, processes, and
connections (sockets) from the administrator. Consider the examples
of rootkits against older Linux kernels in 
http://althing.cs.dartmouth.edu/secref/resources/kernel/rootkits/

Rial.c : start with init_module(). What does this rootkit do?

Knark/: much more mature code, a development of the ideas in
http://althing.cs.dartmouth.edu/secref/resources/kernel/rootkits/linux_module_heroin.txt

Read the sources in http://althing.cs.dartmouth.edu/secref/resources/kernel/rootkits/knark-2.4.3-release/src/

Note the neat trick that makes the process' hiding a hereditary feature
that transfers to any fork-ed children of a hidden process, by defining
and maintaining a special  PF_INVISIBLE flag in task_struct. 

But also note that there are no mutexes in any of the Knark code -- this
rootkit targeted 2.0 and 2.2 Linux kernels for uniprocessor systems. 

2. Advanced rootkits

Rootkits, simple though they may be, must solve a non-trivial problem:
whatever they do, they shouldn't crash the kernel. Crashes are unusual
in Linux, and may cause sysadmin to worry.

Therefore rootkits must look for most reliable yet non-obvious ways to
interpose their functionality into the kernel. So rootkit developers
keep finding and using kernel engineering mechanisms and artifacts
that few people besides their authors understand or know of.

This Phrack 59:5 article illustrates it brillianty:

      http://www.phrack.org/issues.html?issue=59&id=5

(the first part of this two-parter is
      http://www.phrack.org/issues.html?issue=58&id=6 )

See also overviews of kernel function hijacking (pioneered
by Silvio Cesare, who noticed that the kernel loaded in RAM
was still an ELF image, and could be patched as such even with
LKM support syscalls disabled):

     http://www.phrack.org/issues.html?issue=58&id=8
     http://www.phrack.org/issues.html?issue=58&id=7

Note that the interfaces that connect the kernel software
logic and the processor's hard-coded logic, such as the 
Interrupt Descriptor Table (IDT):

     http://phrack.org/issues.html?issue=59&id=4

3. Kernel defences

While it is *hard* to defend the kernel's Ring0 code and
data from attackers who achieve Ring0, is not impossible
to raise the bar. 

Here are some detection and mitigation ideas. These 
ideas have since been developed and incorporated
into production systems.
   
   http://packetstormsecurity.org/files/31513/Kfence.c

    http://phrack.org/issues.html?issue=59&id=10

   http://www.kernelhacking.com/rodrigo/docs/StMichael/guardian01.ppt
     (Rodrigo Branco was the maintainer of StMichael)

   For another summary of defences:
     http://www.kernelhacking.com/rodrigo/docs/StMichael/c17.pdf

4. Linux kernel's production hook systems

Even though rootkits can teach a lot about kernel architecture, 
their scope is limited to just those data flows that are used
on the kernel's various state reporting paths. 

By definition, rootkits are not meant to noticeably alter the paths
for network packet processing, other input processing, and other core
kernel functionality; but this is what we want to control and change
what the kernel is doing, to make it into a firewall or router
(Netfilter/IPTables), or to control the syscalls a process is allowed
to call (SELinux). 

For that, we need comprehensive hooking systems along the paths where
the relevant data is handled (packets, or system call arguments). The
hooks must be placed in just the right places to act on that data with
the minimal amounts of logic that leaves the data consistent for
further processing.  The choice makes or break the hook system design.

Examples of notable designs: 

A.) Netfilter:  http://www.netfilter.org/documentation/HOWTO//netfilter-hacking-HOWTO-3.html

   For an in-depth view of Netfilter hooks in the context of packet processing, see 
slides 5--9 in Joanna Rutkowska's talk:

   http://events.ccc.de/congress/2004/fahrplan/files/319-passive-covert-channels-slides.pdf


 If you are new to networking, read 
     http://www.netfilter.org/documentation/HOWTO//networking-concepts-HOWTO.html 
 and 
     http://www.netfilter.org/documentation/HOWTO//packet-filtering-HOWTO.html

B.) LSM, Linux Security Modules

  http://kernel.org/doc/htmldocs/lsm.html#framework

  http://static.usenix.org/event/sec02/full_papers/wright/wright.pdf

  SELinux uses LSM for its enforcing of labeling rules -- see selinux.txt notes.