Higher-order functions in mathematics are functions that can take other functions as inputs or return them as outputs. This concept allows for more complex operations and manipulations of functions, enabling mathematicians to create new functions based on existing ones. An example of a higher-order function is the map function, which applies a given function to each item in a collection, producing a new collection of results. Higher-order functions are essential in various fields, including calculus and functional programming, as they facilitate more abstract and flexible approaches to problem-solving.