Mathematics runs on notation nobody quite agreed on
A schoolchild in Oslo and a schoolchild in Ohio could once read the same line of arithmetic and get different answers, because the sign between the numbers had been given two jobs in two places. The division sign (÷) is a horizontal bar with a dot above and below, and it means divide in British and American classrooms. In parts of northern Europe it was a subtraction sign, a usage that continued in Norway and, until fairly recently, Denmark. ISO 80000-2, the standard for mathematical signs, now says plainly that it should not be used for division at all, and points instead at the solidus (/), the fraction bar or the colon (:). The solidus is a borrowed mark — the same character that divides a web address into path segments — and Unicode keeps a division slash separate from it for anyone who wants the mathematical one.
That is the shape of most mathematical punctuation. Very little of it was designed; nearly all of it was proposed by one person, adopted by the people who read that person's book, and then had to be reconciled with whatever the neighbours were doing.
The cross that looks like an unknown
The multiplication sign (×) has the clearest paternity. William Oughtred used it in his 1631 algebra, Clavis Mathematicae, writing that multiplication connects both proposed magnitudes with the symbol "in" or with the cross; an earlier appearance, in an anonymous 1618 appendix to Napier's work on logarithms, is usually attributed to him too. The trouble arrived with algebra itself. Once the letter x became the standard name for the thing you are solving for, a multiplication sign shaped like a small x was competing with the most common variable in the subject. The objection is old and is usually credited to Leibniz. In Germany the dot operator (⋅) is preferred for exactly this reason, and in handwriting most people simply write nothing at all between two factors and let adjacency do the work. Unicode carries look-alikes that get pressed into the same service too: the multiplication x and the asterisk operator are separate characters again.
The minus sign (−) is a subtler case of the same disease. It has its own codepoint in Unicode, separate from the hyphen (-) that sits on every keyboard, because the two are not the same mark: the minus is drawn to match the plus it partners, and the hyphen is drawn to join words. Almost nobody types the real one. It is the most reliably wrong character in any equation copied out of a word processor.
Two signs that were somebody's private notation
Leibniz again, thirty-odd years later. The integral sign (∫) is an elongated long s, the letter ſ, chosen because he thought of an integral as an infinite sum of infinitesimal parts and s was the initial of summa. He wrote it in his private papers in 1675; it reached print in 1686, in a paper in Acta Eruditorum. A private shorthand that became the most recognisable mark in higher mathematics.
The infinity sign (∞) was introduced by John Wallis in 1655, in De sectionibus conicis, and he never said why he chose that shape. The candidates are all guesses, offered as guesses: a Roman numeral form for a thousand, written CIↃ; a variant of the Greek omega, the last letter and so the end of the line. The lemniscate itself is much older than the notation — the figure turns up as ornament long before it meant anything numerical — which is exactly what makes the question unanswerable. The loop on an emoji keyboard is not this character at all.
The marks that came from clocks
The prime (′) and the double prime (″) are the oldest idea in the set and the only one that is genuinely systematic. Sexagesimal division gave Latin the pars minuta prima, the first small part, and the pars minuta secunda, the second — one stroke for the first division of a degree or an hour, two strokes for the division of that. Our minutes and seconds are named after the marks. Imperial measurement borrowed the same pair for feet and inches, which is why a builder's drawing and an astronomer's table are using identical punctuation for unrelated quantities.
They are also the marks most often replaced by the wrong character. An apostrophe (') is not a prime and a double quotation mark (") is not a double prime; they are near enough in most typefaces to pass unnoticed, and the substitution is treated as unacceptable in typeset work for the same reason that a hyphen is not a minus. The neighbouring mistake is a straight quote where a curly one belongs, one family over and the same habit. The distinction survives because someone has to be able to search for it.
None of this is tidy, and the tidying is still going on. A standard can say the division sign should not be used for division; it cannot get it off the calculator keys, out of the textbooks or away from a generation that learned it at seven.