Playing with xmodmap I encountered a modifier key I hadn't heard of: Mode_switch. It seems to have something to do with inserting special characters.

I assigned it to a key but it seems to have no effect. What is it for? Is it different from ISO_Level3_Shift (Alt Gr)?

Mode_switch is the old-style (pre-XKB) name of the key that is called AltGr on many keyboard layouts. It is similar to Shift, in that when you press a key that corresponds to a character, you get a different character if Shift or AltGr is also pressed. Unlike Shift, Mod_switch is not a modifier in the X11 sense because it normally applies to characters, not to function keys, so applications only need to perform a character lookup to obtain the desired effect.

ISO_Level3_Shift is the XKB version of this key. Generally speaking, XKB is a lot more complicated and can do some extra fancy stuff. XKB's mechanism is more general as it allows keyboard layouts to vary in which keys are influenced by which modifiers, it generalizes sticky (CapsLock-style) and simultaneous-press (Shift-style) modifiers and so on.

This answer is not meant to contradict @Gilles' answer, but is meant to describe another aspect of xkb Input handling which I feel is "under-illuminated" in his answer.

You can use both ISO_Level3_Shift(Alt-Gr) and Mode_switch at the same time with different results. It requires some configuration, but you can have them both, and use them to produce 6 different keys per physical keyboard key!

You can e.g. configure Tab to be Mode_switch when held with xcape ⌂ and xmodmap, then e.g. the key labeled "H" on the keyboard could produce following characters/keys:

h
H
← (Cursor)
← (Symbol)
┘
┐
with H

with ShiftH

with AltGrH

with AltGrShiftH

with TabH

with TabShiftH

So, to come back around to your question: IL3S and Mode_switch can be configured to mean different things, so you can have cursor keys and ~200 useful characters from the (accented) languages, box-drawing, and technical character set on the 35 keys of the 3 letter rows.

To accomplish the above:

# Initial Setup

xmodmap -e 'keycode 23 = Mode_switch ISO_Left_Tab Tab ISO_Left_Tab Tab ISO_Left_Tab'

xmodmap -e 'keycode any = Tab'

xcape -e "Mode_switch=Tab"

example for H:

xmodmap -pke | grep 'h H'

# Make note of the keycode, note the 8 columns after the `=`,

# then edit and feed back into xmodmap with:

xmodmap -e 'keycode  43 = h H lowrightcorner uprightcorner Left leftarrow h H'

Mode_shift in Xkb is a shift key for groups or layouts. If you have a
layout with multiple groups, you can hold down this key to access the
next group. If you have several layouts, you can hold down this key to
shift between layouts (†1).

So Mode_shift is to groups what Shift (or Shift_L and
Shift_R, to use the keysym names) is to levels.

This might become more clear if we consider that Mode_shift has an
alias called ISO_Group_Shift (†2). There are also keys called
ISO_Group_Latch and ISO_Group_Lock which correspond to
ISO_Level2_Latch (latch shift) and Shift_Lock, respectively (†3).

There are also a number of aliases for different scripts:

• kana_switch


• Arabic_switch


• Greek_switch


• Hebrew_switch


• Hangul_switch

†1: This is based on my understanding of Xkb’s “groups” and “layouts”.
It seems that they are treated the same as far as selecting between them
with keys like this.

†2: I am basing this on the file keysymdef.h in this Git repository
at revision ab8666661fc6 (“Add XF86XK_AudioPreset”, 2017-05-10). On
installations you will probably find this file at
/usr/include/X11/keysymdef.h.

†3: There is also a key ISO_Lock, a kind of general-purpose lock key.

The meaning of the AltGr key

Gilles mentioned in his answer the connection with the
AltGr key:

Mode_switch is the old-style (pre-XKB) name of the key that is
called AltGr on many keyboard layouts.

That lead me to suspect that “AltGr” might stand for
“alternate/alternative group”, since Mode_switch is used to shift
groups. This turned out to be wrong, however. According to the
Wikipedia page on the AltGr key, “AltGr” seems to come from
IBM keyboards, and in their manuals it says that it is an abbreviation
for “alternate graphics”. This makes sense, since it seems likely that
using the keyboard to draw “graphics” (box-drawing characters used in
text user interfaces) came well before there was good support for
switching quickly between multiple scripts.