The Apodisation filter is a special, concentric gray gradient filter with maximum transmission in the middle and decreasing transparency to the edges.
There is an apodization filter near the aperture. It's basically a graduated neutral density filter that filters out more light peripherally and is clearer in the center. So instead of having an aperture that is solid with a sharp cut off of the focused or unfocused rays allowed to hit the sensor, it allows a graduated amount of the unfocused rays of light in.
In a perfect pinhole situation (as in infinitely small aperture), each point from the object casts only one point on the sensor, therefore you have a perfectly sharp image (ignoring diffraction) at any focal plane. With a larger aperture, only a certain range of focal planes will cast a sufficiently distinct image that will be sharp, everything else will be out of focus and blurred. If you have a really large aperture only one focal plane will be in focus (small depth of focus).
Each out of focus light ray (point source) will cast a circle instead of a perfect corresponding point. The larger the aperture and the further away from the plane of focus, the larger that circle. That circle should have a fairly distinct border if the aperture is solid (and the lens elements are theoretically perfect).
With the apodiztion filter, think of combining separate images. Lets simplify and say you combine an image with a pinhole, an image with a moderate sized aperture and an image with a large aperture. Lets consider single point that is out of the focus plane. With the pinhole, you get a perfect point still. with the moderate aperture you get a small circle with a distinct border. With the large aperture you get a large circle. If you combine the images you will get a point with 2 concentric circles around it. Now imagine an infinite number of apertures and combine the images so that the larger aperture images (the larger circles) are fainter than the smaller aperture images, you will get a point of light that gradually fades away the further out you go. So instead of a harsh circle bokeh, you get a smooth drop off. think of looking at a cut end of a cylinder head on versus looking down on a bell shaped curve with the peak of the curve brightest.
Note that a point that is in the focal plane will be a point with any of the sized apertures, so the objects in the focal plane (within the depth of focus) will be sharp. Only the out of focus areas will be gradually smoothed out.
So in essence the apodization filter is like a infinite number of partially transmitting apertures betweenwide open and stopped down.
The net effect is that you have a sharp in focus image and a smooth bokeh. You also lose light since that your essentially partially filtering out some of the light to smooth out the out of focus image. And finally autofocus doesn't work because you no longer have the distinct contrast of out of focus areas needed to autofocus.
Apodization literally means "removing the foot". It is the technical term for changing the shape of a mathematical function, an electrical signal, an optical transmission or a mechanical structure. An example of apodization is the use of the Hann window in the Fast Fourier transform analyzer to smooth the discontinuities at the beginning and end of the sampled time record.
In optical design jargon, an apodization function is used to purposely change the input intensity profile of an optical system, and may be a complicated function to tailor the system to certain properties. Usually it refers to a non-uniform illumination or transmission profile that approaches zero at the edges.
Apodization is used in telescope optics in order to improve the dynamic range of the image. For example, stars with low intensity in the close vicinity of very bright stars can be made visible using this technique. Generally, apodization reduces the resolution of an optical image; however, because it reduces diffraction edge effects, it can actually enhance certain small details.