Chapter 10.2.7 explains the need for "page coloring" cache optimizations.

In the code, the relevant logic affects page_free (which puts pages on the
respective colored free lists) and page_create_va (which maps physical pages
for the page_t arrays, by taking them off the free lists).

More precisely:
http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/vm/vm_page.c#125

125/*
126 * The logical page free list is maintained as two lists, the 'free'
127 * and the 'cache' lists.
128 * The free list contains those pages that should be reused first.
129 *
130 * The implementation of the lists is machine dependent.
131 * page_get_freelist(), page_get_cachelist(),
132 * page_list_sub(), and page_list_add()
133 * form the interface to the machine dependent implementation.
134 *
135 * Pages with p_free set are on the cache list.
136 * Pages with p_free and p_age set are on the free list,
137 *
138 * A page may be locked while on either list.
139 */

page_create_va gets to selecting the pages off the freelist around line 2400,
trying to get them first from the free list and then the cache list:

2400 npp = page_get_freelist(vp, off, seg, vaddr, PAGESIZE,
2401                         flags | PG_MATCH_COLOR, lgrp);
2402 if (npp == NULL) {
2403        npp = page_get_cachelist(vp, off, seg,
2404                                 vaddr, flags | PG_MATCH_COLOR, lgrp);
2405        if (npp == NULL) {
2406                npp = page_create_get_something(vp, // <---- fallback, ignore color
2407                                                off, seg, vaddr,
2408                                                flags & ~PG_MATCH_COLOR);
2409        }

page_get_freelist goes to page_get_mnode_freelist and then to the architecture-specific
macro PAGE_FREELISTS, and similarly for the oage cache list. To wit:

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/vm/vm_dep.h#108

108/*
109 * Per page size free lists. Allocated dynamically.
110 * dimensions [mtype][mmu_page_sizes][colors]
111 *
112 * mtype specifies a physical memory range with a unique mnode.
113 */
114
115extern page_t ****page_freelists;
116
117#define      PAGE_FREELISTS(mnode, szc, color, mtype)                \
118             (*(page_freelists[mtype][szc] + (color)))
119
120/*
121 * For now there is only a single size cache list. Allocated dynamically.
122 * dimensions [mtype][colors]
123 *
124 * mtype specifies a physical memory range with a unique mnode.
125 */
126extern page_t ***page_cachelists;
127
128#define      PAGE_CACHELISTS(mnode, color, mtype)            \
129             (*(page_cachelists[mtype] + (color)))

Note that the page_cachelists page_t array is 4 levels deep!

Read through these functions to see where color comes from
(called "bin" in page_get_freelist, set by AS_2_BIN macro in
http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/vm/vm_dep.h#434

434/*
435 * hash as and addr to get a bin.
436 */
437
438#define      AS_2_BIN(as, seg, vp, addr, bin, szc)                               \
439             bin = (((((uintptr_t)(addr) >> PAGESHIFT) + ((uintptr_t)(as) >> 4)) \
440                 & page_colors_mask) >>                                       \
441                     (hw_page_array[szc].hp_shift - hw_page_array[0].hp_shift))
442

Suggestion: emulate this algorithm in your favorite programming language, and then
see how virtual address allocations would go on your Illumos system. Hint:
to see the variables involved, just do ::print -t in MDB, e.g.:
"page_colors_mask::print -t", "page_colors::print -t", "hw_page_array::print -t" )