1. A crash course in Solaris network configuration 1a. Use the right network port. 1b. If nothing works, see 1a. Seriously, though, see network-config-notes.txt I suspect that I initially chose the port bound to our VoIP VLAN. That would give me an IP address but no kind of reasonable routing or name resolution. 2. Exploring DTrace. Unlike "truss " which traces only system calls by the process created by , DTrace probes fire for events caused by all processes on the system, unless limited by predicates. Hint: as a rule, there is some event that happens a lot more often than others, and needs to be filtered out before the output becomes readable. Providers: syscall -- entry & return of all syscalls (argument knowledge) proc -- process creation and lifecycle; signal-related events fbt -- all kernel functions' boundaries sdt -- statically defined tracing: programmer-placed probes sched -- scheduler events that happen to threads io -- I/O subsystem pid -- user-level functions in processes vminfo -- VM events (based on kstat) sysinfo -- sys kstat kernel statistics profile -- profiler, runs periodic "tick" actions (e.g., profile:::tick-5s) D has its own built-in string type. Convert to it with stringof( char* ) for char* kernel strings. Argument strings located in userspace (such as syscall arguments), must be copied into kernel space before they can used, since all probe actions are executed within kernel space. Use "copyinstr" (see examples). #pragma D option quiet suppresses DTrace's own default print actions #pragma D option flowindent supplies indentation that follows entry and exit from functions 3. Examples. Lots of examples found here: http://www.brendangregg.com/DTrace/dtrace_oneliners.txt Run them and interpret the results. NOTE: DTrace tries to provide its action blocks with variables and structs that are most convenient to work with. Which variables will be made accessible in the action block depends on the probes that match the provider:module:probefunc:probename expression. But: - if a variable is not defined for some matching probe, the block won't compile; - always check the DTrace Guide for the tables of variables available for specific providers and probes: this can save a lot of time. # dtrace -n 'proc:::exec-success { printf("%s", curpsinfo->pr_psargs); }' dtrace: description 'proc:::exec-success ' matched 1 probe CPU ID FUNCTION:NAME 0 16132 exec_common:exec-success ls callout.d intr1.d proc1.d 0 16132 exec_common:exec-success hostname (output produced by "ls *.d" in another terminal window) Observe the curpsinfo variable pointing to a special "struct psinfo_t" filled with info about the "current" process (i.e., the process that caused the probe to fire), as described in "Table 25–1 proc Probes" at http://docs.sun.com/app/docs/doc/817-6223/chp-proc?a=view Observe that this struct's pr_psargs member contains the string of arguments to ls after the Bash shell expanded them. --- # Files opened by process, dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }' Observe the "copyinstr" used to copy the syscall's string argument into kernel space. --- # Syscall count by program, dtrace -n 'syscall:::entry { @num[execname] = count(); }' Observe special aggregation syntax: @num[execname] = count(); creates a counting table that increments the count of each individual execname when the probe fires. On exit (^C) DTrace prints the table, nicely formatted and sorted. "num" is the name of the table. If you only have one, you may omit it and just have @[execname] = count(); Note that execname (or any expression in @[...]) first gets evaluated and then the count() action is taken on the associated value in the table (i.e., execname is used as a key into the table, the value for that key is extraced and incremented). count() merely increments, whereas sum() will add that value. So @[execname] = sum(1); has the same effect as above. --- Suggestion: work thought the rest of the examples in http://www.brendangregg.com/DTrace/dtrace_oneliners.txt , using the DTrace Guide (http://docs.sun.com/app/docs/doc/817-6223) More examples: http://blogs.sun.com/uejio/entry/dtrace_tutorial_for_x_window http://developers.sun.com/solaris/articles/dtrace_example.html --- In class examples: Observe all signal sending activity on the system execept those sent by "sched": proc:::signal-send /execname != "sched" / { printf("%s -> %s %d\n", execname, args[1]->pr_fname, args[2] ); } --- Observe all write() syscalls on the system: syscall::write:entry { printf( "%-15s: %-10s(%d) %8x %s %8x %8x %8x %8x\n", probefunc, execname, pid, arg0, copyinstr(arg1), arg2, arg3, arg4, arg5); } OK, this is pretty bad, because - it would be nice to know the filename, not just the file descriptor arg0 - arg1 is a pointer to a buffer, containing arg2 characters, but copyinstr goes till it encounters a \0 because it is really intended for null-terminated C strings. - dtrace itself prints stuff to the terminal, and ends up tracing its own writes. Silly. Suggestion: fix it! use the DTrace Guide. --- Trace all keyboard (i8042 chip) interrupts. Turns out, it also picks up trackpad interrupts :-) Uncomment the fbt clause to trace all kernel functions called while processing an interrupt. Observe the use of self->ts thread variable in the actions and the predicate. It ensures that fbt probes only fire between the interrupt-start and interrupt-complete events (otherwise the DTrace would just choke). This is a standard trick. Also observe the reference to the kernel symbol devnamesp via ` . Using casts and kernel symbols, you can extract any information that the kernel itself uses! Very cool (the code snippet is copied from http://docs.sun.com/app/docs/doc/817-6223/chp-sdt?a=view)! stringof converts C-style strings into DTrace's own string type ("string"), so that these strings could then be compared with != . See see http://docs.sun.com/app/docs/doc/817-6223/6mlkidlrg?l=en&a=view&q=stringof (it also explains the runtime errors this script tends to produce and shows how to get rid of them). #pragma D option quiet #pragma D option flowindent sdt:::interrupt-start /stringof(`devnamesp[((struct dev_info *)arg0)->devi_major].dn_name) == "i8042"/ { printf("%s\n", stringof(`devnamesp[((struct dev_info *)arg0)->devi_major].dn_name)); @keypresses["i8042"]=count(); self->ts = 1; } sdt:::interrupt-complete /self->ts/ { self->ts = 0; } /* fbt:::entry,fbt:::return /self->ts/ { printf("%s %s %s\n", probemod, probefunc, probename) } */ Suggestion: extract key scan codes and character values of pressed keys! Your keyboard device may differ. ---- Extra: OpenSolaris has demo scripts in /usr/demo/dtrace/ . Study them. Extra: I found a really nice book on Solaris x86 internals: http://opensolaris.org/os/community/documentation/files/book.pdf Enjoy it! ---- SEND ME YOUR E-MAIL FOR THE CLASS LIST!