Nonscalar Mathematical Morphology

Mathematical morphology is not always straightforward to generalize to situations where values in an image do not admit a natural total order. This mostly due to the limitations of the underlying formalism. Three state-of-the-art solutions for bypassing those limitations are discussed, with example applications ranging from color images interpreted as vector spaces to periodic and hyperbolic value spaces, and categorical data stemming from per-pixel classification of remote sensing images.


INTRODUCTION
Mathematical morphology has traditionally mostly been used on binary and grayscale images.These kinds of images share a common theme: they admit a very natural total order on the value space.The theory behind mathematical morphology is equipped to deal with more general situations through the use of lattices, but this has only been used sparingly.It is reasonable to say that the main reason has been that lattices are ill-suited to many such situations.In particular, we will consider both higher dimensional data, as well as categorical data.
When we deal with multidimensional data, like vector-valued data, we usually have some ideas about what kinds of operations "should not matter".For example, if ρ(a) = b for some vector-valued images a and b, and a is another vector-valued image such that the vectors at all positions are rotated by some fixed rotation -a (x) = r(a(x)) -then we might expect ρ(a )(x) to simply be a rotated version of ρ(a)(x) as well.It can be shown that this type of constraint is incompatible with imposing a lattice structure on the vector space, at least if we assume the lattice is compatible with the vector space in a particular (and sensible) way.Section 3 shows how one can work around this by representing vectors using a frame rather than a basis.Unfortunately, this means using a larger and/or more complicated representation, and the answers you get might not correspond to the frame-based representation of any particular vector, leading to a possible loss of information, as well as a loss of the properties that usually characterize morphological operators.This motivates the introduction of sponges in Section 4: a generalization of lattices, sponges are easier to adapt to multidimensional spaces, and we present several examples of sponges.Interestingly, we can show that sponges allow us to recover at least some parts of the usual morphological theory based on lattices.
Categorical data presents another interesting challenge: there is no up or down, no large or small.While a vector at least has a magnitude, different categories are just different.We cannot even use something like a median algebra, since we also do not have a concept of "betweenness": different categories are just different.Section 5 shows how the concept of n-ary morphology can be used to deal with this case.We also briefly touch upon data where each value is not a single category, but a vector of likelihoods for each category.
Finally, in Section 6 we very briefly touch upon some other ideas people have used to apply morphology to non-traditional data, and when they should be preferred over the approaches detailed in the earlier sections.But first, we will have a closer look at the traditional morphological framework.
Code for all of the examples in this work can be found at http://bit.ly/ 2u95Z8W.

LATTICE-BASED MATHEMATICAL MORPHOLOGY
In its original form, mathematical morphology is a set-theoretical approach to image analysis (Matheron, 1975;Serra, 1982).The key idea is to define image transformations based on shape information.In the simplest case of binary images, small subsets, called structuring elements, of various forms and sizes are translated over the image plane to perform shape extraction.Such morphological image transformations have, on the one hand, an intuitive geometrical interpretation, and on the other hand can be precisely formulated as algebraic image operators.In contrast to traditional linear image processing based on concepts such as convolution and frequency analysis, the morphological image operators focus on the geometrical content of images and are nonlinear.
The mathematical description of morphological image operators has been extensively developed within the framework of complete lattice theory (Serra, 1988;Heijmans, 1994;Heijmans & Ronse, 1989;Ronse & Heijmans, 1991).Also, the case of vector-valued data, such as color or hyper-spectral images, has been addressed.An important tool in morphological image analysis is the algebraic decomposition and synthesis of image operators in terms of elementary operations.Once such an algebraic decomposition is available, it enables efficient implementations on digital computers, see e.g.(Giardina & Dougherty, 1988;Soille, 2003).

Morphology for Binary Images
Let E be the Euclidean space R n or the discrete grid Z n .By P(E) we denote the set of all subsets of E ordered by set-inclusion.A binary image can be represented as a subset X of E. The space E is a commutative group under vector addition: we write x + y for the sum of two vectors x and y, and −x for the inverse of x.The following two algebraic operations are fundamental for mathematical morphology of binary images: where X a = {x + a : x ∈ X} is the translate of the set X along the vector a.
Let the reflected or symmetric set of A be denoted by are called dilation and erosion by the structuring element A, respectively.There exists a duality relation with respect to set-complementation (X c denotes the complement of the set X): X ⊕ A = (X c A ∨ ) c , i.e. dilating an image by A gives the same result as eroding the background by A ∨ .To any mapping ψ : P(E) → P(E) we associate the dual mapping ψ : (3) By composition of dilation and erosion, other important transformations can be built; in particular, the opening α A , which is an erosion following by a dilation, and the closing φ A , which is a dilation followed by an erosion: Also opening and closing have an intuitive geometric interpretation, as can be observed from the above formulas.The opening of X is the union of all the translates of the structuring element which are included in X.The closing of X by A is the complement of the opening of X c by A ∨ .The dilation δ A , erosion ε A , opening α A , and closing φ A are all examples of translation-invariant binary image operators.For any operator ψ : P(E) → P(E) we say it is translation-invariant when ψ(X h ) = (ψ(X)) h for all h ∈ E.

Complete Lattice Framework
We now summarize the main concepts from lattice theory needed in this paper, cf.Heijmans (1994), Serra (1988).For a general introduction to lattice theory, see Birkhoff (1961).
A complete lattice (L, ≤) is a partially ordered set L with order relation ≤, a supremum or join operation written , and an infimum or meet operation written , such that every (finite or infinite) subset of L has a supremum (smallest upper bound) and an infimum (greatest lower bound).In particular there exist two universal bounds, the least element written O L and the greatest element I L .Barring such a least and greatest element, one can still have a conditionally complete lattice, in this case every subset of L that is bounded from below has a meet, and every subset that is bounded from above has a join.
For the case of binary images, the relevant lattice is the power lattice P(E) of all subsets of the set E, the order relation is set-inclusion ⊆, the supremum is the union of sets, the infimum is the intersection of sets, the least element is the empty set ∅, and the greatest element is the set E itself.The power lattice P(E) is an atomic complete Boolean lattice, and conversely any atomic complete Boolean lattice has this form.

Mappings
The composition of two mappings ψ 1 and ψ 2 on a complete lattice L is written ψ 1 ψ 2 or ψ 1 • ψ 2 , and instead of ψψ we also write • a closing, if it is increasing, extensive and idempotent; • an opening, if it is increasing, anti-extensive and idempotent.
Let L and M be complete lattices.A mapping ψ : L → M is called: Let ε : L → M and δ : M → L be two mappings, where L and M are complete lattices.Then the pair (ε, δ) is called an adjunction between L and M, if for every a ∈ M and b ∈ L, the following equivalence holds: If M coincides with L we speak of an adjunction on L. It has been shown (Gierz, Hofmann, Keimel, Lawson, Mislove, & Scott, 1980;Heijmans & Ronse, 1989;Ronse & Heijmans, 1991) that in an adjunction (ε, δ), ε is an erosion and δ a dilation.Also, for every dilation δ : M → L there is a unique erosion ε : L → M such that (ε, δ) is an adjunction between L and M; ε is given by ε(b) = {a ∈ M : δ(a) ≤ b}, and is called the upper adjoint of δ.Similarly, for every erosion ε : L → M there is a unique dilation δ : M → L such that (ε, δ) is an adjunction between L and M; δ is given by δ(a) = {b ∈ L : a ≤ ε(b)}, and is called the lower adjoint of ε.Finally, for any adjunction (ε, δ), the mapping δε is an opening on L and εδ is a closing on M. In the case that L and M are identical, one sometimes refers to such openings and closings as morphological or adjunctional (Heijmans, 1994).
As a specific case we may consider the lattice of gray scale functions.Let L = Fun(E, T ) denote the complete lattice of gray scale functions with domain E, whose range is a complete lattice T of gray values.Then the following gray scale dilation-erosion transform pair on L can be defined Gray scale opening and closing are defined by composition, as in the binary case.

Invariance
Invariance of image operators can be captured by group actions (Robinson, 1982;Suzuki, 1982).When T is an (automorphism) group of two lattices L and M, a mapping ψ : L → M is called T-invariant if it commutes with all τ ∈ T, i.e., if ψ(τ (a)) = τ (ψ(a)) for all a ∈ L, τ ∈ T.More generally, a mapping ψ : L → M, with L and M two distinct lattices, is said to be invariant to T if ψ(τ (a)) = ρ τ (ψ(a)) for all a ∈ L, τ ∈ T and some representation ρ : T → S of T on M, where S is a group on M.
For the case of binary images, the symmetry group is the Euclidean translation group.For other situations, different groups may be relevant.For example, some images have radial instead of translation symmetry (Serra, 1982, p. 17), requiring a polar group structure.The generalization of morphology for binary images with arbitrary abelian symmetry groups was worked out by Heijmans (1987), see also Roerdink and Heijmans (1988); for the general lattice case, see Heijmans and Ronse (1989), Ronse and Heijmans (1991).The further generalization to lattices with a non-abelian symmetry group was made by Roerdink (1993Roerdink ( , 2000)).
The general setup is to start from a group T acting transitively on the image space E. We say that T is transitive on E if for each x, y ∈ E there is a g ∈ T such that gx = y, and simply transitive when this element g is unique.
A homogeneous space is a pair (T, E) where T is a group acting transitively on E. Any transitive abelian group T is simply transitive.For the Boolean case, the object space of interest is again the lattice P(E ) of all subsets of E. The general strategy is to make use of the results for the simply transitive case (corresponding to the situation where T is the Euclidean translation group), by 'lifting' subsets of E to subsets of T, applying morphological operators on the lattice P(T), and then 'projecting' the results back to the original space E. The case of non-Boolean lattices, such as the lattice of gray value functions, requires the notion of sup-generating families (Heijmans & Ronse, 1989;Ronse & Heijmans, 1991;Heijmans, 1994).It turns out that only part of the invariance results for Boolean lattices carries over to the case of non-Boolean lattices with a non-abelian symmetry group; for details, see Roerdink (2000).

(Hyper)connected Filters
Connected filters are used to perform filtering based on various shape and size attributes (Salembier, Oliveras, & Garrido, 1998;Serra & Salembier, 1993;Salembier & Serra, 1995).A key property of connected filters is their edge preserving nature.Connected filters rely on an axiomatic definition of connectivity within a complete lattice framework (Serra, 1988;Heijmans, 1994Heijmans, , 1999;;Braga-Neto & Goutsias, 2003).To address some shortcomings, hyperconnectivity and hyperconnected filters were introduced by Serra (1998).In particular, hyperconnected filters can deal with overlapping objects as separate entities (Serra, 1998;Wilkinson, 2007Wilkinson, , 2009;;Ouzounis & Wilkinson, 2007, 2011) and can prevent the so-called "leakage" problem of connected filters.(Hyper)connected operators will be very useful for systematically simplifying complex oriented structures.Furthermore, connected filters are readily extended to the vector case (Salembier & Garrido, 2000), so it is expected that they extend to the tensor case as well.

Inf-Semilattices
We briefly mention Inf-semilattices here, as these are related to frames and sponges which will be discussed later.
The extension of mathematical morphology to complete semilattices was developed by Kresch (later called Keshet) (Kresch, 1998(Kresch, , 2000)).The main motivation for this extension is that there are cases (such as the difference between two real functions) where the existence of both a least and a greatest element is not obvious.Then the notion of complete semilattice would be useful.
An inf-semilattice is a partially ordered set P where every two-element subset {a, b} of P has an infimum a ∧ b, but not necessarily a supremum. 1An inf-semilattice is complete when every non-empty subset B ⊆ P has an infimum B.
Erosions on complete inf-semilattices can be defined as operators that distribute over infima.The difference with erosions on complete lattices is that the condition of preservation of the greatest element is dropped.Erosions on complete inf-semilattices retain many properties of their counterparts on complete lattices: they are increasing, and an infimum of erosions is itself an erosion.Algebraic openings can be defined in the same way as for complete lattices.Also the notion of adjoint dilation in an inf-semilattice can be defined, although in a less straightforward way.See Kresch (2000) for further details.

FRAMES
Marginal processing of vector-valued images assumes that the image is decomposed into channels correspond to a certain basis, and processes the channels independently.This does not make sense for proper vectors, as the results will be biased by the choice of basis, and when considering something as a vector, it is usually important to have everything invariant 1 A sup-semilattice is defined analogously.
to the choice of basis.However, instead of using a basis (a minimal set of vectors spanning the vector space), we can also pick a frame: a not necessarily minimal set of vectors spanning the vector space.A basis is thus a frame, but a frame might contain many more vectors, and can even be infinite for finite dimensional vector spaces.In particular, we can create a frame that contains all unit vectors, and thus essentially encompasses all possible bases (morphological operators are often invariant to scaling, so the magnitude of the vectors is usually irrelevant).
Another way of looking at this, is by considering the set of transformations of the basis or frame that should not matter.For example, the frame consisting of all unit vectors can also be viewed as the frame that is invariant to all rotations.Now we can see that, sometimes, instead of being invariant to all rotations, we may wish to be invariant only to certain rotations.For example, in an RGB color space, while within a plane of constant lightness it might be natural to pick any two vectors (corresponding to two hues) to span the space, it may not always make sense to be invariant to rotations that completely invert the order on luminance, for example.It is also possible to consider transformations other than rotations, but then we have to be extra careful about how we weigh the different vectors; an example is shown in Section 3.6.2.

Definitions
A real Hilbert space is a real vector space with a symmetric, bilinear, and positive definite binary operator '•': the inner product.The inner product gives rise to a metric by putting d(a, b) = (a − b) • (a − b), and a Hilbert space is complete with respect to this metric.
A frame (Christensen, 2008) is a set of vectors {f i } i∈I in a Hilbert space such that there are finite and positive constants A and B satisfying (for all a in the Hilbert space) where (F a) i = f i • a, and it is assumed that the range of F also admits an inner product, which is in practice not difficult to ensure.It should be noted that a frame has at least one dual frame { fi } i∈I , so that F * is a left-inverse of F: F * • F = id.One particularly important choice is the so-called canonical dual frame, for which F * is the Moore-Penrose pseudoinverse of F (Ben-Israel & Greville, 1974).

Lattices and Group Invariance
First, let us motivate why a frame is needed in the first place.If we have a Hilbert space that is also a lattice, such that the lattice is invariant to addition and multiplication by positive scalars, then it can be shown that it essentially has to be based on having lattices on the individual coefficients in some basis, and then combining those lattices using a "direct or lexicographical product"; loosely translated, that means we are limited to product orders and lexicographical orders (or, possibly, combinations of the two).Given that lexicographical orders lead to discontinuous joins and meets (Chevallier & Angulo, 2014), we focus on product orders.Typically, the group of transformations to which a basis is invariant are fairly limited, and in particular: a basis is never invariant to the group of all rotations.Using a frame allows us to sidestep this problem, as we can construct (infinite) frames in which the transformations are represented by permutations (van de Gronde & Roerdink, 2013a& Roerdink, , 2013b& Roerdink, , 2014b)).Now, we could also look explicitly for Hilbert spaces that are lattices, such that the lattice is rotation invariant, rather than invariant to vector addition and multiplication by positive scalars.However, in two dimensions it is clear that these two things are related to each other through the mapping (x, y) → (cos(x) e y , sin(x) e y ) and its set-valued inverse, and in higher dimensions it is easy to see that things do not get any easier.

Constructing a Group Invariant Lattice
To construct a group invariant lattice based on a particular Hilbert space, simply pick a set of vectors {f i } i∈I that is invariant to the transformation group T, verify that the chosen set of vectors is, in fact, a frame, and turn it into a product lattice, using the same lattice on all of the individual coefficients.Theorem 1 shows that this gives a lattice that is invariant to a representation of T, while Proposition 1 shows that the lattice constructed on R I may not give rise to a lattice on the original Hilbert space, but still induces a partial order invariant to the transformation group.
Theorem 1.If H is a Hilbert space, {f i } i∈I is a frame invariant to the transformation group T on H, then any lattice on R gives rise to a lattice on R I that is invariant to a representation of T on R I .
Proof.If the frame is invariant to T, then (due to each transformation being a bijection), T must act as a permutation on the frame.That is, for every τ ∈ T there is a bijection p τ : This permutation can be used as a representation of T on R I : Clearly, if we construct a lattice on R I as a direct product of some lattice on R, then this larger lattice is invariant to this representation of T.
Proposition 1.If H is a Hilbert space, {f i } i∈I is a frame with the analysis operator F, and R I is a partially ordered set invariant to a representation of the transformation group T on H, then this leads to a partial order on H invariant to T as follows: a ≤ b ⇐⇒ Fa ≤ Fb (for all a and b in H).
Proof.The given relation on H can be seen, from the definition, to inherit reflexivity, antisymmetry, and transitivity from the partial order on R I , and is thus a partial order.Furthermore, consider ρ τ to be the representation of τ ∈ T, and note that the partial order on R I is invariant to ρ τ for all τ ∈ T. Now, for all τ ∈ T, This shows that the given partial order on H is invariant to the group T.
Example 1 (Rotation around axis).Take the Euclidean space R 3 as Hilbert space, and take as frame the set { 1 √ 2 (cos(α), sin(α), 1) | α ∈ [0, 2π)}.Clearly, this frame is invariant to the group of rotations around the z-axis: {(0, 0, z) | z ∈ R}.Theorem 1 now implies that any lattice on R gives rise to a lattice on R [0,2π) that is invariant to a representation of the group of rotations around the z-axis.In particular, the lattice will be invariant to the representation that converts a rotation of β around the z-axis into a permutation that maps index α to α ± β mod 2π, with the sign determined by the "handedness" of the rotation.Due to Proposition 1, if we pick the usual lattice on R, the resulting lattice on R [0,2π) induces a partial order on the original Hilbert space R 3 that can be checked to be isomorphic to the Loewner order on 2 × 2 symmetric (real) matrices (van de Gronde & Roerdink, 2014a, §3.2).
It should be noted that instead of a lattice on the individual coefficients, we can also use a semilattice, like the one proposed by Heijmans and Keshet (2002), in which numbers are ordered by magnitude, with zero as the lowest number, and negative and positive numbers being incomparable.This approach is illustrated in the next example.
Example 2 (Complete rotation invariance).Take the Euclidean space R 3 as Hilbert space, and take as frame the set S 2 = {v | v = 1}.Clearly, this frame is invariant to the group of all rotations SO(3).Theorem 1 now implies that any lattice on R gives rise to a lattice on R S 2 that is invariant to a representation of SO(3).In particular, the lattice will be invariant to the representation that converts a rotation τ ∈ SO(3) into a permutation that maps index v to τ −1 (v).If we pick the usual lattice on R, the resulting lattice on R [0,2π) induces a partial order on the original Hilbert space R 3 that can be checked to be trivial, in the sense that no elements are comparable.In contrast, if we pick the inf-semilattice on R proposed by Heijmans and Keshet (2002), in which numbers are ordered by magnitude, with zero as the lowest number, and negative and positive numbers being incomparable, then the result is related to the inner product sponge discussed in Section 4.3.1 (also see the next section), although the partial order induced on R 3 is still only mildly non-trivial: elements are only comparable if the lie along the same ray beginning at the origin.

Going Back
The frames resulting from the procedure explained in the previous section tend to be infinite, as most of the interesting groups tend to be infinite.And even if we stick to a finite sampling of such a frame, the number of vectors needed to give reasonable results will typically still be quite a bit higher than the number of vectors in a basis.As a result, it is typically desirable to somehow go back to a representation using a basis, and several options are possible.
Perhaps the most generic, and often easiest, option is to use a dual frame.In particular, if we use the Moore-Penrose pseudoinverse (corresponding to the canonical dual frame), we get a least squares solution, in the sense that F † u = arg min a u − Fa 2 .However, other dual frames could also be used, if this suits the application.To make this work we need an inner product on the space of frame coefficients, and it has to be invariant to the representation of the transformation group to which the lattice on the frame coefficients is invariant (van de Gronde & Roerdink, 2013b, §IV, 2014b).In other words, there must be a representation that maps the given transformation to the intersection of automorphisms (of the lattice on the frame coefficients) and orthogonal operations (with respect to the inner product on the frame coefficients).
Another possibility is to go back in such a way that the result is either an upper bound or a lower bound.To this end, imagine we have a function It is often possible to give a suitable h that makes the above well-defined (such that there is always a solution, and it is unique).It can be verified that if the latter is the case, and holds for the same transformation group T as the frame is invariant to, then τ • P + = P + • ρ τ for all τ ∈ T (and similarly for P − ), and the representation (ρ) used in Theorem 1.
Example 4 (The inner product sponge).Continuing from Example 2, computing P + (Fa ∧ Fb), with h(x, y, z) = x 2 + y 2 + z 2 , gives the meet of the inner product sponge described in Section 4.3.1.Given that here the underlying structure is an inf-semilattice, we cannot really define the join in a similar way.

Practical Considerations
Many interesting groups are infinite, and this presents some issues when implementing the above.In particular, we obviously cannot directly compute an infinite number of inner products when computing Fa, nor can we simply store a list containing an infinite number of coefficients.There are at least two ways to get around this: • approximate an infinite frame by a finite one, for example selecting just a few dozen uniformly distributed vectors can give very good results in practice; • represent the frame coefficients implicitly, for example by using convex hull-related structures (van de Gronde, 2015, §5.3).The first option is easy, and typically gives good enough results, so is the one that was used so far in implementations.The second option is primarily interesting from a theoretical perspective, since it links frames to several other methods, but if high quality results are needed, it could also be faster/less memory intensive.

Hue or Rotation Invariant
Given an RGB color space, we can consider the group of hue rotations, which for the purposes of this example will be considered as the group of rotations around the gray axis.The construction of the frame follows Example 1. Fig. 1 shows the result on an image.Alternatively, we can consider all rotations in the RGB space.The construction of the frame follows Example 2. Fig. 2 shows the result on an image.In the dilation result on the left there are many areas where the join is ill-defined for some coefficients, indicated by the bright and contrasting colors.On the right, this is avoided by only using valid coefficients (so for which the join did exist) in the least squares procedure; the result is qualitatively comparable to the dilation in Fig. 5.

Hue and Saturation Invariant
Changing the saturation of a color can be modeled in the RGB space as scaling that part of the color that is orthogonal to the gray axis.Such scalings form a group, and making use of the fact that (flat) morphological operators are themselves invariant to monotone transformations of individual coefficients in product lattices, it is possible to define a frame that allows defining operators that are invariant to both hue rotations and saturation changes (van de Gronde & Roerdink, 2013b).Fig. 3 illustrates the effect on an image.

SPONGES
Frames can successfully enable the application of traditional morphological concepts on vector-and tensor-valued data, but at the cost of Figure 3 Top row: the result of an erosion and opening using a hue-and-saturationinvariant frame based on an inf-semilattice, with least squares backprojection.Bottom row: the result of a dilation using least squares backprojection and least squares backprojection on valid values only.In the dilation result on the left there are many areas where the join is ill-defined for some coefficients, indicated by the bright and contrasting colors.On the right, this is avoided by only using valid coefficients (so for which the join did exist) in the least squares procedure; the result is qualitatively comparable to the dilation in Fig. 4.
requiring a more complex representation of (intermediate) results.Apart from implying a longer processing time, this also means that when using frames we either have to save results in a different format then the input, or accept a certain amount of information loss, which can cause problems when chaining operations for example.An alternative is to use sponges (van de Gronde & Roerdink, 2015& Roerdink, , 2016)): generalizing lattices, sponges provide more flexibility in their definition, while still allowing us to recover at least some of the familiar results from lattice-based mathematical morphology.

Definitions
An oriented set is a set with a reflexive and antisymmetric relation ' ' (an orientation), such that a b is interpreted as a being "less than or equal to" b in some way, without this relation necessarily being transitive.We write A B for subsets A and B of an orientation if and only if ∀a ∈ A, b ∈ B : a b.A sponge is an oriented set in which there exists a supremum (infimum) for every nonempty and finite subset with at least one upper bound (lower bound).That is, if A is a finite and nonempty subset of a sponge, then there exists a supremum of A if and only if there is an element that bounds all elements of P from above.By definition, the supremum of A (if it exists) is the least upper bound, in the sense that it is less than all other upper bounds.
Suprema in a sponge S are given through the join function J : P(S) → P 1 (S), and infima through the meet function M : P(S) → P 1 (S), where P 1 (S) gives all subsets of S of size at most one (so the empty set and all singletons).If J(A) = ∅ for some subset A, then A has no supremum, and thus no upper bounds.The functions J and M satisfy the following properties (for all b ∈ S and A a finite, nonempty subset of the sponge S): and M(M(A) ∪{b}) = {b}, and the same properties with the roles of J and M reversed.Absorption is a direct adaptation of the same concept in lattices.Part preservation was adapted from weakly associative lattices (or trellises) (Skala, 1971;Fried & Grätzer, 1973a, 1973b) and specifies that any lower (upper) bound of a set is also a lower (upper) bound of the infimum (supremum) of this set.In a lattice, part preservation is implied by associativity.Commutativity is implied in sponges, as they operate on sets.Idempotence follows from combining the two absorption laws.2Note that the existence of a J and M with these properties is also sufficient to define a sponge.
A tournament is a totally oriented (sub)set.That is, it is an orientation (or subset of an orientation) such that every pair of elements is comparable.A subset A of a sponge S is tournament-sup complete if and only if J(T) ⊆ A for every non-empty tournament T ⊆ A. Note that this definition does not require the supremum of every tournament to exist (J(T) to be nonempty), but if it does exist in the original sponge, it should be in A. We briefly summarize some results derived in earlier work.
As a consequence, if an orientation is acyclic, any subtournament is also acyclic, and thus a chain.
Proposition 3 (Prop.3 in van de Gronde & Roerdink, 2016).A tournament never has more than one maximal (minimal) element, and if it has a maximal (minimal) element it is the supremum (infimum) of the tournament.The latter assumes the principle holds for partial orders, or, equivalently, that the axiom of choice holds (Birkhoff, 1961, §VIII.14).
Proposition 5 (Prop.5 in van de Gronde & Roerdink, 2016).A maximal totally oriented subset of a tournament-sup complete subset of a sponge contains its supremum, if it exists (in the sponge).
Proposition 6 (Prop.6 in van de Gronde & Roerdink, 2016).In a conditionally complete sponge, the set of lower bounds L(a) is tournament-sup complete for [every] a in the sponge.
Proposition 7 (Prop.7 in van de Gronde & Roerdink, 2016).The intersection of two tournament-sup complete subsets of a sponge is tournament-sup complete.
A subset A of a sponge S is nonredundant if and only if The reduction operator ψ N : P(S) → P(S) is defined by A subset A is said to refine a subset B (A B) if and only if every element in A is bounded from above by an element from B. The result of the reduction operator is nonredundant, but need not (in general) be refined by the original set (situations where this is true play an important role in Section 4.2).
A sponge S is called dry if it satisfies (for all a, b in S and nonredundant3 A ⊆ S, with A = ∅ and J(A) = ∅) {a} J(A) and A {b} =⇒ a b or ∃c ∈ A : a c If a sponge is not dry, it is wet.

Openings
In a lattice, an operator is considered an opening if it is anti-extensive, idempotent, and increasing.Increasingness likely cannot generalize directly to sponges, since even the join and meet are not necessarily "increasing", in the sense that a c and b d need not imply J({a, b}) J({c, d}), for example.However, we will see that it is possible to get familiar-looking operators that are anti-extensive and idempotent.Also, it is possible to replace increasingness by a weaker property that together with anti-extensivity and idempotence still characterizes openings in lattices (van de Gronde & Roerdink, 2016).The idea is that such a weaker property might be easier to apply to sponge-based operators, although so far this has not yet been shown to be true.
We start with a slightly reworked version of a result shown earlier (van de Gronde & Roerdink, 2016, Thm. 7).For this, consider the operator with I a subset of S that contains a lower bound for every element in S. 4It is not difficult to verify that γ N I is anti-extensive, in contrast to similar operators that are based on the Loewner order (van de Gronde & Roerdink, 2014a), which is not a sponge.On the other hand, since A ⊆ B, and

Theorem 2. In a dry conditionally complete sponge S, γ
, then we have a contradiction, because there then cannot be any element in ψ N (B) that bounds a from above (given that ψ N (B) is a subset of A, and that It was shown previously that structural "openings" formed by combining a structural "erosion" and a structural "dilation" can be viewed as an operator γ N I for use with Theorem 2, with Here it is assumed that a sponge Fun(E, S) is used, with S a conditionally complete sponge with a least element, E a vector space, and X ⊂ E a structuring element; X y is used to denote {x + y | x ∈ X}, and a X y should be interpreted as in Eq. ( 4).We now proceed to show that (trivial) connected openings based on flat zones can be defined based on a set I that can be used in Theorem 2 as well.A flat zone is simply a (connected) region of an image that has the same value everywhere within the region, and it will be represented as follows (for every position x in the domain E, and a (value) sponge S with a least element O S , and an element a ∈ S): x∈ X, and O S otherwise. (4) A connected opening γ N I can now be defined by picking as I a set of flat zones that satisfy certain criteria.For example, all flat zones whose support has an area of at least 100 pixels.It is assumed that the support sets of such flat zones are a suitable subset of a connectivity class (Heijmans, 1999), and in particular that the set of "valid" support sets D is closed for (infinite) unions of families of support sets that have a nonempty intersection: Theorem 3. If S is a conditionally complete sponge with a least element, and D is a set of nonempty subsets of E, closed for infinite unions of families of support sets that have a nonempty intersection, then I = {aX | a ∈ S and X ∈ D} satisfies Proof.Note that a flat zone is greater than or equal to another flat zone if both the support and the associated value are greater than or equal to those of the other.For each support set As a consequence, for each support set X ∈ D there is a maximal element in I ∩ L(f ) whose value is equal to the meet of the values in f within the support X, and whose support is Y ∈ D for some Y ⊇ X.The statement is now proved by noting that every element in I ∩ L(f ) is of the form aX, with X a member of D, and a ∈ S; given that aX f , we have {a} {f (x) | x ∈ X}, and xA is thus bounded from above by the maximal element corresponding to X (which has value Note that Theorem 2 does not necessarily apply to a sponge of the form Fun(E, S), where S is dry, as the direct product of dry sponges need not be dry.So, although we know that there are both dry sponges (see Section 4.3.1),and appropriate sets suitable for use with Theorem 2, it is not yet clear whether they indeed lead to idempotent openings.However, in practice it looks like sponges of the form Fun(E, S), where S is dry do indeed allow idempotent (structural) openings, so it remains to be seen whether it is simply very difficult to construct counterexamples, or if Theorem 2 can be further generalized (or changed).

The Inner Product Sponge
One of the main examples of a sponge is the so-called inner product sponge.The idea is quite simple: in an inner product (or, preferably, Hilbert) space, From left to right and top to bottom: the original, and the result of an erosion, dilation, and "opening" using the direct product of the inner product sponge on the chroma plane, and a traditional (totally ordered) lattice along the lightness axis.The images are processed at half the original resolution, and a 5 × 5 square structuring element is used.Note that the dilation shows many artifacts in regions where different hues meet, as then it can be that there is no well defined supremum.
It has been shown that this indeed gives rise to a sponge, and that this sponge is dry in two dimensions.5 Fig. 4 shows an example of such a sponge.Fig. 5 shows an example of using a three dimensional inner product sponge, illustrating that in practice it can still come quite close to exhibiting idempotent "openings", even if it is not dry.

The (Hemi)spherical Sponges
It is also possible to define sensible sponges on both a (hyper)sphere and a (hyper)hemisphere (van de Gronde & Roerdink, 2016), with the latter being interesting for working with discrete probability/likelihood distributions expressed in barycentric coordinates and mapped to a triangle on a (hyper)sphere by squaring the coordinates (Facchi, Kulkarni, Man'ko, Marmo, Sudarshan, & Ventriglia, 2010;Franchi & Angulo, 2015;Gromov, Figure 5 From left to right and top to bottom: the result of an erosion, dilation, "opening", and "opening" of the "opening" using the inner product sponge directly on the RGB space.Note that since here the input data is strictly within the positive cone, it can be seen that a supremum always exists, so the dilation is effectively artifact free.In addition, although strictly speaking the "opening" need not be idempotent, in practice the results are quite close.

2013
).However, these sponges can be related directly to the inner product sponge through various projections, so will not be discussed further here.

Periodic Sponges
A lattice is a partially ordered set, and given that a partial order is transitive, it cannot contain cycles.Sponges, however, can contain cycles.A simple example is 2 for all a, b ∈ S. It can be verified that this relation is a dry sponge (van de Gronde & Roerdink, 2016, §5.5).In higher dimensions, the powers of S are also sponges, but not necessarily dry.In particular, if take S 3 , then we see that if we take a = [(0.6,0.9, 1.0)], P = {[(0.8,0.8, 1.0)], [(0.8, 1.0, 0.8)], [(1.0, 0.8, 0.8)]}, and b = [(1.2,1.2, 1.2)], that a J(P) and P b, but also a b, and ∀p ∈ P : a p, the sponge is thus wet; here [(a, b, c)] ∈ S 3 can be identified with the set {(a + i, b + j, c + k) | i, j, k ∈ Z}.

The Hyperbolic Sponge
One more sponge that should be mentioned, is the hyperbolic sponge (van de Gronde & Roerdink, 2016; van de Gronde & Hesselink, submitted for publication).This is a sponge on hyperbolic space (in any number of dimensions), in which sets of lower/upper bounds (depending on the point of view) are defined using geodesics, similar in spirit to sets of upper bounds in the inner product sponge (although it looks like the two are, in principle, not directly related).For examples of its application, see the work by Angulo andVelasco-Forero (2013, 2014) (which uses the same structure under a different name, in part because this work predates the identification of sponges).

n-ARY MORPHOLOGY
Another class of data for which it is difficult to pick a sensible total order is categorical data.After all, each category, or class, is simply different from all others.For this situation n-ary morphology was proposed: in this framework we no longer just have up and down, we have as many directions as we have categories.These categories could come from a per-pixel classification or an image segmentation, for example.

Definitions
In discrete n-ary morphology, an image is given as a function f : ω → L, where L is a set of labels, and is some metric space with an (arbitrary) origin, and which allows translation; B x will denote the translated version of B, such that if B contained the origin, B x contains x.It will be assumed that B always contains the origin.The labels are assumed to be completely independent of each other.For each label ∈ L, both a dilation δ and an erosion ε are defined.In the binary case, the dilation of one is the erosion of the other, and vice versa.The definition of the structural dilation with (flat) structuring element B presents few difficulties6 : The erosion is defined similarly, but requires a method for dealing with ambiguities, represented by the function θ7 : Here, m stands for some arbitrary label satisfying the given condition, if one exists.At each position, at most one such m can exist.We put the following constraints on θ : 1. θ should always give a label other than ; 2. θ (f , x, B) only depends on the values of f at the positions B x (and their locations relative to x); 3. if θ (f , x, B) = m, and g is an image such that ∀x ∈ : f (x) = g(x) or g(x) = m, then θ (g, x, B) = m as well.The effect of the above definition of the erosion is that • values unequal to are not touched, since they are already "as far from as possible", • if the set f (B x ) = {f (y) | y ∈ B x } only contains one label (which could be ), then ε (f , B)(x) will equal this label, • if f (B x ) contains and exactly one other label, then ε (f , B)(x) will be equal to this other label, • otherwise, we do not a priori know which label to pick, and θ will be used to make a choice.Now, in the common case that B x is a ball of a certain radius around x, the easiest (and, arguably, most correct (Chevallier, Chevallier, & Angulo, 2016, §3)) approach for defining θ is to simply take the label assigned to the closest point to x that is not equal to .If this is not uniquely defined, we can either define some kind of arbitration rule, or assign a set of labels to the point, rather than a single label.Taking the closest label is exactly equivalent to considering the limit of implementing the erosion as repeated application of an erosion with a ball of smaller radius.8

Adjunctions, Openings, and Closings
Traditionally, each dilation has a corresponding erosion, such that the pair satisfies an adjunction: This is a very powerful relationship, that allows one to show, for example, that the composition of an erosion and a dilation is an opening or closing (depending on the order of the composition).
For n-ary morphology, instead of having a single partial order or lattice, we define a preorder ≤ consistent with δ B and ε B : The consistency is captured in Lemma 1.Since a preorder is not necessarily anti-symmetric, it is possible that a ≤ b and b ≤ a, but a = b, we say that the two are not equal but equivalent: a ≡ b.Since a preorder is transitive, the equivalence relation '∼ ' is transitive as well, and we can recognize equivalence classes: subsets of L that are all equivalent to each other.In the current case there are two such classes: { } and L \ { }.
Proof.This follows directly from the definitions, the assumption made at the start of the section that B contains the origin, and the constraint that the case using θ should never give .
Giving this consistency, we can show that a structural dilation and its corresponding erosion in the n-ary framework indeed form an adjunction.
Theorem 4. δ B and ε B form an adjunction in the preorder ≤ : (5) Proof.First, assume that δ B (f ) ≤ g holds.Then, f (x) ≤ g(y) for all x and y with y ∈ B x .As a result, for a given x, if f (x) = , then g(y) = for all y ∈ B x , and ε B (g)(x) = as well, and f (x) ≤ ε B (g)(x).On the other hand, if f (x) = , then f (x) ≤ ε B (g)(x) holds by definition.Since this argument holds for all x, f ≤ ε B (g).Now assume that f ≤ ε B (g) holds.Then, f (x) ≤ g(y) for all x and y with y ∈ B x .As a result, for a given y, if there exists an x such that y ∈ B x and f (x) = , then g(y) = δ B (f )(y) = .On the other hand, if (for a given y) no such x can be found, δ B (f )(y) = f (y), which is different from due to the assumption that B contains the origin.In either case, δ B (f )(y) ≤ g(y).Since this holds for all y, δ B (f ) ≤ g.This concludes the proof.Heijmans (1994, Prop. 3.14) proves several statements concerning the composition of a dilation and erosion that form an adjunction, Theorem 5 does much the same thing for the current setting.
Lemma 2. The operators δ B and ε B are increasing in the preorder ≤ .
Proof.First consider the dilation δ B and preorder ≤ .Note that f ≤ g if and only , where δ B is an ordinary binary dilation with structuring element B. Since a normal dilation is increasing, f ≤ g =⇒ δ B (f ) ≤ δ B (g), proving that δ B is increasing.That ε B is increasing in ≤ can be shown the same way.
Recall that an (algebraic) opening is an operator that is increasing, idempotent, and anti-extensive, and that a closing is increasing, idempotent, and extensive.
Corollary 1.The operator ε B • δ B is a closing, and the operator δ B • ε B is an opening.Proof.The composition of two increasing operators is increasing, hence both operators mentioned are increasing by Lemma 2. By Eqs. ( 6) and (7) of Theorem 5 the two operators are also extensive and anti-extensive, respectively.Finally, based on Eq. ( 8), it is quickly verified that both operators are idempotent, either by substituting ε B (f ) for f , or by applying ε B to both sides of the equation.

Example
This example uses the "Urban" hyperspectral data set (Zhu, Wang, Xiang, Fan, & Pan, 2014), where each pixel is labeled according to the largest abundance out of six classes at that point.9Fig. 6 shows the original, as well as the basic operations dilation and erosion.Fig. 7 shows the opening and closing of a particular label.

OTHER APPROACHES
In addition to sponges and n-ary morphology, there have been more approaches to achieve qualitatively similar results to traditional morphology in nonscalar settings.For tensor-valued images, the Loewner order has been used by Burgeth, Bruhn, Didas, et al. (2007) to define erosion-, dilation-, opening-, and closing-like operators.The operators allow for relatively efficient implementations, but do violate some of the basic properties that define the traditional operators (van de Gronde & Roerdink, 2014a).
Another approach is to observe that the usual maximum on reals (at least on the positive reals) is intimately related to certain types of norms.Attempts have been made to do morphology based on generalizing such constructions to non-scalar settings (Burgeth, Welk, Feddern, & Weickert, 2004;Angulo, 2013).It is not yet clear if, and if so how, the results relate to other approaches, nor what properties the resulting operators preserve.
Related to rotation-invariant frame-based methods and inner product sponges, are methods based directly on convex hulls (Gimenez & Evans, 2005), as well as the vector median filter (Astola, Haavisto, & Neuvo, 1990).
When the data at hand can be interpreted as a vector-or tensor-field, so when the value at each position is a vector or tensor that can usefully be interpreted in terms of the tangent space of the image domain, then it may make sense to interpret these vectors and tensors more as giving scalar values in certain directions rather than as opaque non-scalar objects.This leads to path-based morphology (van de Gronde, Lysenko, & Roerdink, 2015;van de Gronde, 2015).
Finally, it should be mentioned that marginal processing (for example using a product lattice on a basis for vector-valued data) is often quite adequate, especially if it does not really make sense to mix the different components (Serra, 1993).Alternatively, one can use Pareto morphology in such situations (Köppen & Franke, 2007;Köppen, Nowack, & Rösel, 1999), even though this will give less stable results than marginal processing.

SUMMARY AND CONCLUSIONS
In this paper we have discussed extensions of mathematical morphology beyond the classical cases of binary and grayscale images with the standard ordering of sets or real function values, for which the complete lattice framework is no longer adequate.In particular, for multidimensional images such as vector and tensor valued images, as well as categorical data there is no obvious ordering of the value domain.
Nevertheless, morphological operators on such images can be defined by focusing on the invariance properties one would like to maintain, such as invariance under rotations or scaling.Taking inspiration from the concepts defined in group morphology (Roerdink, 2000), we have shown that this can be achieved by representing vectors or tensors using a frame rather than a basis.For example, in the case of color images we saw that in order to avoid the false color problem one should not use a particular color basis, but simply consider all possible bases generated by the required invariance (for example all rotations of a specific RGB basis), perform some morphological operator in this lifted space, and in the end project the result back to the original domain.This more complicated representation comes at a price, leading to a possible loss of information, as well as a loss of the properties that usually characterize morphological operators.This motivated the introduction of sponges, which are easier to adapt to multidimensional spaces, and allow us to recover parts of the usual morphological theory based on lattices.
For categorical data, or data with a vector of likelihoods for each category, we have shown how the concept of n-ary morphology can be used.Finally, we discussed various other approaches that have ambitions similar to the methods in this work, but either sacrifice (even more) properties, or take a completely different point of view.
123 h : H → R such that Fa < Fb =⇒ h(a) < h(b), then we can define P + (u) = arg min a h(a) subject to u ≤ Fa, and P − (u) = arg max a h(a) subject to Fa ≤ u.

Figure 1
Figure 1 From left to right and top to bottom: the original, and the result of an erosion, dilation, and opening using a hue-invariant frame based on the usual lattice on the reals, with least squares backprojection.The images are processed at a resolution of 384 ×256 (half the original resolution), and a 5 × 5 square structuring element is used.(The parrot image is based on a photograph by Luc Viatour/www.Lucnix.be,used under the CC BY 2.0 license.)

Figure 2
Figure2Top row: the result of an erosion and opening using a rotation-invariant frame based on an inf-semilattice, with least squares backprojection.Bottom row: the result of a dilation using least squares backprojection and least squares backprojection on valid values only.In the dilation result on the left there are many areas where the join is ill-defined for some coefficients, indicated by the bright and contrasting colors.On the right, this is avoided by only using valid coefficients (so for which the join did exist) in the least squares procedure; the result is qualitatively comparable to the dilation in Fig.5.

Proposition 4 (
Prop. 4 in van de Gronde & Roerdink, 2016 -Hausdorff 's Maximal Principle for orientations).Every totally oriented subset T of an orientation O is contained in a maximal (in the sense of set inclusion) totally oriented subset M.
γ N I : S → S on a conditionally complete sponge S, defined by γ N I (a) ∈ J(ψ N ({b | b ∈ I and b a})), is the subset of I that is bounded by the join of A, itself a subset of I, A ⊆ B. Also, from the definitions of A and B, we get {b} J(A) and A {f } for all b ∈ B. Given that I ∩ L(f ) ψ N (I ∩ L(f )), and I ∩ L(f ) is guaranteed to be non-empty, A is non-empty.Now, since S is dry (and A is nonempty and nonredundant and J(A) = ∅), we have b f or ∃a ∈ A : b a for all b ∈ B. We now show that ψ N (B) ⊆ A. If b ∈ B satisfies b f , then it is in I ∩ L(f ), and if it is not in A, then it is also not in ψ N (B), since it is bounded from above by some element in A ⊆ B. Alternatively, if ∃a ∈ A : b a, then either b ∈ A or b / ∈ ψ N (B), since it is bounded from above by some element in A ⊆ B. Summarizing, every b ∈ B must either be in A or not in ψ N (B): ψ N (B) ⊆ A.

Figure 4
Figure 4From left to right and top to bottom: the original, and the result of an erosion, dilation, and "opening" using the direct product of the inner product sponge on the chroma plane, and a traditional (totally ordered) lattice along the lightness axis.The images are processed at half the original resolution, and a 5 × 5 square structuring element is used.Note that the dilation shows many artifacts in regions where different hues meet, as then it can be that there is no well defined supremum.

Figure 6
Figure 6 From left to right: the original, the dilation, and the erosion of the magenta label.

Figure 7
Figure 7 LEFT: opening of the magenta label.RIGHT: closing of the magenta label.See Fig. 6 for the original.