The ArcPad Team Blog

Unofficial stuff from the team behind the World's leading mobile GIS platform

Wednesday, August 23, 2006

Raster Tips # 1 - Choosing a raster format

To make a proper raster format choice, you should read the entire blog entry, however, having said that, my favourite is TIFF/LZW because it is every where, it has a good compression (10:1), uses little memory and is fast.

Bits Lossy Files Memory Drawing
BMP
1:1
any
n
large
tiny
fast raw
CADRG
55:1
8
n
small
tiny
fast tiles
GIF/LZW
10:1
8
n
small
tiny
slow
JPEG
20:1
24
y
tiny
large
fast memory
JPEG2000
20:1
any
y
tiny
medium
fast tiles
MrSID
20:1
any
y
tiny
medium
fast tiles
PNG
10:1
any
n
small
tiny
slow
TIFF/none
1:1
any
n
large
tiny
fast raw
TIFF/CCITT
100:1
2
n
small
tiny
slow
TIFF/LZW
10:1
any
n
small
tiny
fast strip
TIFF/DEFLATE
10:1
any
n
small
tiny
fast strip
ECW *
20:1
24
y
tiny
medium
fast tiles

* ECW requires a 3rd party extension

In the table Compression, Memory and Drawing speed are a rough guide based on my personal experience. You should always check by running your own performance tests on your data. You may get different results which could and should affect your choice of raster format.

JPEG, JPEG2000, MrSID and ECW are lossy. They achieve high 20:1 compression by losing/replacing bits of image often undetected by the human eye. Choose lossy compression for captured imagery (e.g. scan images, digital camera photos and aerial photography) otherwise choose a non lossy compression for computer generated images such as a cartographic map. If you inadvertently choose a lossy compression for a cartographic map the edges of lines will distort noticeably producing a poorer looking map. JPEGs should be used for small photos like 1 mega pixel images. This is because when JPEGs are loaded into ArcPad they are decompressed in their entirety and can consume large amounts of memory. MrSID, JPEG2000 and ECW are hierarchical tile formats. They are fast when viewing the overall image but may slow down when you zoom in since higher detailed tiles are being retrieved and decompressed which consumes more memory and more CPU time. MrSID and JPEG2000 are supported out of the box by ArcPad whereas ECW requires a 3rd party extension downloadable from ERMapper's website: http://www.ermapper.com/downloads/plugins.aspx.

GIF and PNG are widely used on the internet and both achieve a decent compression of 10:1. These formats are ideal for small images, but are definitely not ideal for large images since both of these formats are sequential meaning they must be read from start to finish. If we zoom in to the centre of these images, ArcPad cannot jump to the right place in the file, it must always read from the beginning of the file and often has to decode large volumes of off screen data before it finds the portion required.

CADRG file format is used by the National Geospatial-Intelligence Agency and is claimed to have compression ratios up to 55:1. I suspect this figure is an overestimation due to mixing 8 bit and 24 bit file formats and the real figure lying between 10:1 and 20:1. The file sizes are small. It also does not use much memory or require much speed to draw in ArcPad. The only limitation is that the format is extremely restrictive in its geographic area and resolution.

BMP, TIFF/none are extremely fast to draw because ArcPad knows where to look in a file to retrieve exactly the right data to reconstruct an image. In addition no decompression step is needed. This format is suitable in terms of speed for both small and large images alike. Its only disadvantage is the file size and therefore it's impact on cost and size of memory cards needed to make such data portable.

TIFF/CCITT is tailored for black/white fax transmissions. It assumes that a large portion of your document is white and assigns a highly compressed binary code for such an area. Such a technique can yield up to 100:1 compression ratios, however, in practice, 50:1 may be more realistic. This format is often coded in a single strip which makes it behave like a sequential file format. This means its will suffer from the same limitations that exists in GIF and PNG in that when you zoom in to the centre of the image large portions of off screen imagery needs to be decoded before ArcPad reaches the portion needed.

TIFF/LZW and TIFF/DEFLATE are both good. They both achieve decent 10:1 compression, require small file sizes, use little memory to decompress and are relatively fast. They are strip based formats so that ArcPad can decompress the image that it needs by decompressing the right strips. The most ideal layout is to have exactly 1 row per strip. This will guarantee that ArcPad decompresses the least amount of data to reconstruct the image. TIFF/LZW is widely supported by various software applications whereas TIFF/DEFLATE (sometimes called ZIP-in-TIFF) is supported mainly by Adobe software.

7 Comments:

  • At Thursday, August 24, 2006 4:50:00 pm, Blogger oj said…

    Stephen,

    Thanks for the great input concerning raster formats.

    In tests, we did not get satisfying results with jpg2000, probably because of the used programm and settings we used to produce the jpg2000 files.
    Could you give us a hint on which program and settings you use to produce jpg2000 rasters.

    Thanks, oj

     
  • At Sunday, August 27, 2006 5:05:00 pm, Blogger Stephen Quan said…

    Hi oj, I've used LizardTech's GeoExpress encoder to create both MrSID and JPEG2000 images. I've received advice stating that MrSID images needs a minimum of 7 levels in order to not consume much memory on WM devices (the more the better). This should also be true for JPEG2000 images. Having more levels ensures that when you zoom in ArcPad will still be able to find a set of small tiles to decompress and render your image. Having more levels will also result in better decompression speeds.

     
  • At Saturday, January 20, 2007 7:09:00 am, Anonymous Anonymous said…

    I have had similar problems getting Jpeg2000 to work (using the ECW compressor). At the moment we use ecw exclusively, even for carto rasters. I know it does degrade quality, but Im happy to accept this for smaller file sizes.

    Thanks for this post, it is extremely useful. For me the hardest part of getting arcpad to perform was not the software itself, but optimizing supporting functions like this. Keep up the good work!

     
  • At Wednesday, April 30, 2008 2:29:00 pm, Anonymous Anonymous said…

    Stephen (or anyone),

    Thanks for the great tip on using LZW compression for TIFFs. However, I'm having a few difficulties in getting this compression format to actually reduce the file size in ArcGIS Desktop 9.2. I've gone into Go to C:\Program Files\Common Files\ESRI\Raster\defaults and edited the following tiff.pdf properties:

    compression( "8-bit Image Compression" ): "LZW"

    create_tiled_images("Create Tiled Images"): "false"

    (uncommented)
    rows_strip("Rows per strip"): 1

    (ArcGIS was restarted after saving the changes)

    However these settings resulted in a file size that is larger than the original uncompressed tiff file.

    What am I doing wrong?

     
  • At Wednesday, April 30, 2008 5:50:00 pm, Blogger Stephen Quan said…

    Hi, check out my new post "Raster Tips # 2" which shows you how to create faster, smaller and low memory usage TIFF images for ArcPad.

     
  • At Friday, May 02, 2008 11:10:00 am, Anonymous Anonymous said…

    Stephen,

    Thanks for the very detailed instructions in your new post. I followed the instructions step-by-step and they work fine, except for one thing: the resulting LZW tiff file was still larger than the original. Shouldn't I get compression ratios of up to 10:1? Just to clarify, my source is an 8-bit triple-band uncompressed tiff file created by exporting the current view of a larger tiff in ArcGIS 9.2. The resulting image reports (in ArcGIS Raster Dataset properties) to use LZW compression but it's size is 360MB as opposed to the 327MB original.

    Cheers, Darren

     
  • At Tuesday, April 07, 2009 11:14:00 pm, Anonymous Anonymous said…

    Hi everyone,

    In case you are using ECW files and you need the plug in, the link has moved from what is posted in the blog.

    http://www.ermapper.com/Products/ERDASDownloads/tabid/192/CurrentID/2526/Default.aspx

    or

    http://www.ermapper.com/Products/ERDASDownloads/tabid/192/CurrentID/2510/Default.aspx

     

Post a Comment

<< Home