The punctuation that runs the web
Everything after the scheme in a web address is a string being read by a parser, left to right, and the punctuation is the grammar it reads. Letters and digits carry the meaning; a dozen or so marks decide where one part of the address ends and the next begins. None of them ended up there by taste. Each was assigned a job by a specification, and several of them have a story about the job they used to have.
The specification in force is RFC 3986, published in 2005, and its first move is to sort punctuation into two piles. Reserved characters have work to do: the general delimiters are the colon (:), the solidus (/), the question mark (?), the number sign (#), the two square brackets ([ and ]) and the commercial at (@), and a second group — exclamation mark (!), dollar sign ($), ampersand (&), apostrophe ('), the parentheses ( and ), asterisk (*), plus sign (+), comma (,), semicolon (;) and equals sign (=) — is set aside for whatever an individual scheme wants to do with it. Unreserved characters have no job at all: letters, digits, and the four marks the standard names as hyphen (-), period (.), underscore (_) and tilde (~). An unreserved character means itself, always, and encoding it changes nothing; they are the only marks in an address doing no work, ordinary punctuation left alone.
The delimiters work because they never argue
What makes the reserved set usable is that its members appear in a fixed order and each one closes the section before it. The colon ends the scheme. The double solidus opens the authority — the host and its optional port, which the colon then separates again in a different role. Single solidi divide path segments, the same mark that mathematics reaches for when it wants a fraction on one line. The question mark ends the path and starts the query. The number sign ends everything else and starts the fragment. A parser can find the boundaries of a URL it has never seen before without knowing anything about the site, because the punctuation is doing the work in a fixed sequence rather than by convention.
The number sign is the one that surprises people. RFC 3986 says the fragment identifier is separated from the rest of the URI before the address is dereferenced, which means everything from the number sign onward is a private note from the page to the browser. The server never sees it. A tracking parameter hidden after a number sign is not hidden — it never left the building.
The query, by contrast, is the lawless part. The standard gives the query component a syntax but not a meaning: a sequence of ordinary characters, plus solidi and question marks if you want them, and nothing whatever about pairs. The habit of writing key=value and joining the pairs with ampersands arrived with HTML form submission, not with the URI standard, which is why the same page can accept a repeated parameter, or a semicolon instead of an ampersand, and be within its rights either way. The two most familiar characters in a modern URL are conventions rather than rules.
Two marks with a past
The commercial at is the one delimiter most people never type on purpose. It separates the user information from the host, and the standard is blunt about the version everyone remembers: use of the format with a password in it is deprecated, and applications are told to ignore such credentials when they arrive. A character kept in the grammar mainly so that parsers agree on how to throw it away.
The tilde had the opposite career. RFC 1738, the 1994 specification, listed it among the unsafe characters — grouped with the braces, the vertical line, the reverse solidus, the circumflex, the square brackets and the grave accent, all of them unsafe because gateways and other transport agents were known to modify them in passing. Eleven years later RFC 3986 lists it as unreserved, in the same breath as the hyphen and the underscore. In between, the shell convention of writing a tilde before a username to mean that user's home directory had been carried onto the web: Apache's mod_userdir maps a path beginning with a tilde and a name to files inside that account's home directory, which is why so many university pages have one in the middle.
The same 1994 document called the double quotation mark (") unsafe for a different reason again: some systems use it to delimit a URL written in running text, so an address containing one could not reliably be told apart from its own container. That is a rule about the world an address travels through rather than about the address, and it is why the older specification's list is longer than the newer one's.
The percent sign (%) is the escape hatch for everything else, and it is worth knowing what it escapes. Percent-encoding does not encode characters; it encodes bytes, as a percent followed by two hexadecimal digits. A character outside the unreserved set is first turned into bytes — UTF-8 is what the standard recommends for anything new — and each byte is then written out separately. That is why a single accented letter usually costs six characters in an address bar, and why a URL that has been encoded twice looks like an unfortunate accident rather than a slightly wrong one.