Crystallographic image processing (CIP) is traditionally understood as being a set of key steps in the determination of the atomic structure of crystalline matter from high-resolution electron microscopy (HREM) images obtained in a transmission electron microscope (TEM) that is run in the parallel illumination mode. The term was created in the research group of Sven Hovmöller at Stockholm University during the early 1980s and became rapidly a label for the "3D crystal structure from 2D transmission/projection images" approach. Since the late 1990s, analogous and complementary image processing techniques that are directed towards the achieving of goals with are either complementary or entirely beyond the scope of the original inception of CIP have been developed independently by members of the computational symmetry/geometry, scanning transmission electron microscopy, scanning probe microscopy communities, and applied crystallography communities.
HREM image contrasts and crystal potential reconstruction methods
Many beam HREM images of extremely thin samples are only directly interpretable in terms of a projected crystal structure if they have been recorded under special conditions, i.e. the so-called Scherzer defocus. In that case the positions of the atom columns appear as black blobs in the image (when the spherical aberration coefficient of the objective lens is positive - as always the case for uncorrected TEMs). Difficulties for interpretation of HREM images arise for other defocus values because the transfer properties of the objective lens alter the image contrast as function of the defocus. Hence atom columns which appear at one defocus value as dark blobs can turn into white blobs at a different defocus and vice versa. In addition to the objective lens defocus (which can easily be changed by the TEM operator), the thickness of the crystal under investigation has also a significant influence on the image contrast. These two factors often mix and yield HREM images which cannot be straightforwardly interpreted as a projected structure. If the structure is unknown, so that image simulation techniques cannot be applied beforehand, image interpretation is even more complicated. Nowadays two approaches are available to overcome this problem: one method is the exit-wave function reconstruction method, which requires several HREM images from the same area at different defocus and the other method is crystallographic image processing (CIP) which processes only a single HREM image. Exit-wave function reconstruction provides an amplitude and phase image of the (effective) projected crystal potential over the whole field of view. The thereby reconstructed crystal potential is corrected for aberration and delocalisation and also not affected by possible transfer gaps since several images with different defocus are processed. CIP on the other side considers only one image and applies corrections on the averaged image amplitudes and phases. The result of the latter is a pseudo-potential map of one projected unit cell. The result can be further improved by crystal tilt compensation and search for the most likely projected symmetry. In conclusion one can say that the exit-wave function reconstruction method has most advantages for determining the (aperiodic) atomic structure of defects and small clusters and CIP is the method of choice if the periodic structure is in focus of the investigation or when defocus series of HREM images cannot be obtained, e.g. due to beam damage of the sample. However, a recent study on the catalyst related material Cs0.5[Nb2.5W2.5O14] shows the advantages when both methods are linked in one study.
Brief history of crystallographic image processing Aaron Klug suggested in 1979 that a technique that was originally developed for structure determination of membrane protein structures can also be used for structure determination of inorganic crystals. This idea was picked up by the research group of Sven Hovmöller which proved that the metal framework partial structure of the K8−xNb16−xW12+xO80 heavy-metal oxide could be determined from single HREM images recorded at Scherzer defocus. (Scherzer defocus ensures within the weak-phase object approximation a maximal contribution to the image of elastically scattered electrons that were scattered just once while contributions of doubly elastically scattered electrons to the image are optimally suppressed.)
In later years the methods became more sophisticated so that also non-Scherzer images could be processed. One of the most impressive applications at that time was the determination of the complete structure of the complex compound Ti11Se4, which has been inaccessible by X-ray crystallography. Since CIP on single HREM images works only smoothly for layer-structures with at least one short (3 to 5 Å) crystal axis, the method was extended to work also with data from different crystal orientations (= atomic resolution electron tomography). This approach was used in 1990 to reconstruct the 3D structure of the mineral staurolite HFe2Al9Si4O4 and more recently to determine the structures of the huge quasicrystal approximant phase ν-AlCrFe and the structures of the complex zeolites TNU-9 and IM-5. As mentioned below in the section on crystallographic processing of images that were recorded from 2D periodic arrays with other types of microscopes, the CIP techniques were taken up since 2009 by members of the scanning transmission electron microscopy, scanning probe microscopy and applied crystallography communities.
Contemporary robotics and computer vision researchers also deal with the topic of "computational symmetry",Y. Liu, "Computational Symmetry" in: Symmetry 2000, Part I, eds. I. Hargittai and T. C. Laurent, chapter 21, p. 231–245, Portland Press, London, 2002, (Wenner-Gren International Series, vol. 80), https://www.cs.cmu.edu/~yanxi/images/computationalSymmetry.pdfY. Liu, H. Hel-Or, C. S. Kaplan, and L. V. Gool (2009) "Foundations and Trends in Computer Graphics and Vision" vol. 5, Nos. 1–2, p. 1–195, open access: http://vision.cse.psu.edu/publications/pdfs/liuCSinCV.pdf, also published as a book by Now Publishers Inc., Boston and Delft, 2010Y. Liu (2014) "Computational Symmetry", Computer Vision, A Reference Guide, Ikeuchi, K. (ed.), Springer, 2014 but have so far failed to utilize the spatial distribution of site symmetries that result from crystallographic Hahn T. (2005) International Tables for Crystallography, Brief Teaching Edition of Volume A, Space-group symmetry. 5th revised edition, Chester: International Union of Crystallography origin conventions. In addition, a well known statistician noted in his comments on "Symmetry as a continuous feature" that symmetry groups possess inclusion relations (are not disjoint in other words) so that conclusions about which symmetry is most likely present in an image need to be based on "geometric inferences".K. Kanatani, (1996) Statistical Optimization for Geometric Computation: Theory and Practice, Dover Books on Mathematics, Mineola, New York Such inferences are deeply rooted in information theory, where one is not trying to model empirical data, but extracts and models the information content of the data.K. P. Burnham and D R Anderson (2002) Model selection and multimodel inference: a practical information-theoretic approach, 2nd edition. Springer, New York The key difference between geometric inference and all kinds of traditional statistical inferences is that the former merely states the co-existence of a set of definitive (and exact geometrical) constraints and noise, whereby noise is nothing else but an unknown characteristic of the measurement device and data processing operations. From this follows that "in comparing two" (or more) "geometric models we must take into account the fact that the noise is identical (but unknown) and has the same characteristic for both" (all) "models". Because many of these approaches use linear approximations, the level of random noise needs to be low to moderate, or in other words, the measuring devices must be very well corrected for all kinds of known systematic errors.
These kinds of ideas have, however, only been taken up by a tiny minority of researchers within the computational symmetry and scanning probe microscopy / applied crystallography communities. It is fair to say that the members of computational symmetry community are doing crystallographic image processing under a different name and without utilization of its full mathematical framework (e.g. ignorance to the proper choice of the origin of a unit cell and preference for direct space analyses). Frequently, they are working with artificially created 2D periodic patterns, e.g. wallpapers, textiles, or building decoration in the Moorish/Arabic/Islamic tradition. The goals of these researchers are often related to the identification of point and translation symmetries by computational means and the subsequent classifications of patterns into groups. Since their patterns were artificially created, they do not need to obey all of the restrictions that nature typically imposes on long range periodic ordered arrays of atoms or molecules.
Computational geometry takes a broader view on this issue and concluded already in 1991 that the problem of testing approximate point symmetries in noisy images is in general NP-hard and later on that it is also NP-complete. For restricted versions of this problem, there exist polynomial time algorithms that solve the corresponding optimization problems for a few point symmetries in 2D.C. Dieckmann (2012) Symmetry Detection and Approximation, Dissertation zur Erlangung des Doktorgrades, Fachbereich Mathematik und Informatik der Freien Universität Berlin
Crystallographic image processing of high-resolution TEM images The principal steps for solving a structure of an inorganic crystal from HREM images by CIP are as follows (for a detailed discussion see X.D. Zou, T.E. Weirich & S. Hovmöller (2001) "Electron Crystallography - Structure determination by combining HREM, crystallographic image processing and electron diffraction." In: Progress in Transmission Electron Microscopy, I. Concepts and Techniques, X.F. Zhang, Z. Zhang Ed., Springer Series in Surface Science. Vol. 38, Springer 2001, 191 – 222. ).
Selecting the area of interest and calculation of the Fourier transform (= power spectrum consisting of a 2D periodic array of complex numbers) Determining the defocus value and compensating for the contrast changes imposed by the objective lens (done in Fourier space) Indexing and refining the lattice (done in Fourier space) Extracting amplitudes and phase values at the refined lattice positions (done in Fourier space) Determining the origin of the projected unit cell and determining the projected (plane group) symmetry Imposing constrains of the most likely plane group symmetry on the amplitudes an phases. At this step the image phases are converted into the phases of the structure factors. Calculating the pseudo-potential map by Fourier synthesis with corrected (structure factor) amplitudes and phases (done in real space) Determining 2D (projected) atomic co-ordinates (done in real space)
A few computer programs are available which assist to perform the necessary steps of processing. The most popular programs used by materials scientists (electron crystallographers) are CRISP, VEC, and the EDM package. There is also the recently developed crystallographic image processing program EMIA, but so far there do not seem to be reports by users of this program.