problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Paul Hemmer

Hello!

The following code saves a TIF file which I can open and see correctly 
in a variety of image viewers. The problem is described following.

Note that in the code below, the "l_thumb" structure holds the filename, 
width, height, pixels_per_cm and buffer data for a 16bit grayscale image. 

I have compiled libTiff4.0 on a 64bit Win10 machine using Visual Studio 2012.



TIFF* tif = TIFFOpen(l_thumb->filename, "w");

TIFFSetField(tif, TIFFTAG_IMAGEWIDTH     , l_thumb->width        );
TIFFSetField(tif, TIFFTAG_IMAGELENGTH    , l_thumb->height       );
TIFFSetField(tif, TIFFTAG_BITSPERSAMPLE  , 16                    );
TIFFSetField(tif, TIFFTAG_COMPRESSION    , COMPRESSION_LZW       );
TIFFSetField(tif, TIFFTAG_PHOTOMETRIC    , PHOTOMETRIC_MINISBLACK);
TIFFSetField(tif, TIFFTAG_FILLORDER      , FILLORDER_MSB2LSB     );
TIFFSetField(tif, TIFFTAG_MAKE           , "My Company Name"     );
TIFFSetField(tif, TIFFTAG_MODEL          , "My Product"         );
TIFFSetField(tif, TIFFTAG_ORIENTATION    , ORIENTATION_TOPLEFT   );
TIFFSetField(tif, TIFFTAG_SAMPLESPERPIXEL, 1 );
TIFFSetField(tif, TIFFTAG_ROWSPERSTRIP , l_thumb->height   );
//TIFFSetField(tif, TIFFTAG_XRESOLUTION    , l_thumb->pixels_per_cm);
//TIFFSetField(tif, TIFFTAG_YRESOLUTION    , l_thumb->pixels_per_cm);
TIFFSetField(tif, TIFFTAG_PLANARCONFIG   , PLANARCONFIG_CONTIG   );
TIFFSetField(tif, TIFFTAG_RESOLUTIONUNIT , RESUNIT_CENTIMETER );

unsigned long l_buffer_size = l_thumb->width*2;
tdata_t       l_buffer;

l_buffer = _TIFFmalloc(l_buffer_size);

for (int row = 0; row < l_thumb->height; row++){
memcpy_s((byte*)l_buffer, l_buffer_size, &l_thumb->data[row*l_thumb->width], l_buffer_size);

int ret=TIFFWriteScanline(tif, l_buffer, row, 0);
if (ret==-1){
TIFFClose(tif);
return PACKET_TYPE_ERROR;
}
}

TIFFClose(tif);
free(l_thumb->data);
I can open the image and see what I expect to see. When I view the header tags, I can see 
all values set correctly including the final "Centimeter" resolution unit tag. 

However, notice I've commented out the X/Y resolution tags. If I put those lines
back, suddenly I continue to see XResolution, but Y Resolution and ResolutionUnit
are missing. Instead I see this and I am at a loss to explain why:

XResolution     (1 Rational): 115
0               (0 NoType): 
1               (2051838 NoType): 


I have tried hard-coding values like "115" and "115.00" and tried making 
l_thumb->pixels_per_cm double, float, int, and the results are always the same. 

I have also tried compiling and running against libTiff4.1 (BigTIFF) because ultimately I will need
files > 4GB in size. But with either version of the library, I simply cannot
seem to insert the resolution tags.  Also, TIFFSetField on those tags is always 
returning 1, success.

Any idea where I am going wrong?

Thanks!
-Paul




_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Paul Hemmer
My colleague helped in tracing the problem to a bug in the libtiff code, and to help any future searchers, I'll post his findings below that solved the problem.  I can't seem to tell if anybody actively maintains this library anymore? 

The issue is in tif_dirwrite.c, part of libtiff.  It exists in libtiff 3.8.0, 3.8.2; and bigtiff 4.1.

 

The basic issue is how the method to write out a pair of Rationals (WriteRationalPair) handles the directory pointer (dir).

 

The original code treats ‘dir’ as a regular pointer, which is incorrect.  Simply incrementing the pointer does not point to the next directory entry.  There is special method (TDIREntryNext) provided for handling the pointer. 

 

Changing WriteRationalPair to use TDIREntryNext resolved the problem of YResolution not be being written.  It also solved an issue found during debug where YPosition was not being correctly written.  [Setting YPosition was tested as both the resolution and position are written out by WriteRationalPair .]  Furthermore, the correction fixed an invalid free() call that was caught in the debug build.

 

The changes are highlighted below.

 

The original code (tif_dirwrite.x, line 67)

 

#define       WriteRationalPair(type, tag1, v1, tag2, v2) {          \

       TIFFWriteRational((tif), (type), (tag1), (dir), (v1))  \

       TIFFWriteRational((tif), (type), (tag2), (dir)+1, (v2))       \

       (dir)++;                                        \

}

 

The fixed code (tif_dirwrite.x, line 67)

 

#define       WriteRationalPair(type, tag1, v1, tag2, v2) {          \

       TIFFWriteRational((tif), (type), (tag1), (dir), (v1))  \

       TDIREntryNext(tif, (dir)); /* go to next entry */ \

       TIFFWriteRational((tif), (type), (tag2), (dir), (v2))  \

}

 



_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Even Rouault-2
Le jeudi 27 octobre 2016 17:55:55, Paul Hemmer a écrit :
> My colleague helped in tracing the problem to a bug in the libtiff code,
> and to help any future searchers, I'll post his findings below that solved
> the problem.  I can't seem to tell if anybody actively maintains this
> library anymore?
>
> The issue is in tif_dirwrite.c, part of libtiff.  It exists in libtiff
> 3.8.0, 3.8.2;

Those are old versions of libtiff. The latest one is 4.0.6 :
http://www.simplesystems.org/libtiff/
I cannot find trace of the code you mention in that version, so I assume this
has been fixed.

> and bigtiff 4.1.

Perhaps you are refering to a fork of libtiff, since there's no 4.1 version yet
in libtiff.
libtiff 4.0 or above incorporate bigtiff support.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Paul Hemmer
In reply to this post by Paul Hemmer
Hello,

Is the "BigTIFF" specification/functionality fully incorporated now into LibTiff?  If yes, great! Which version? If no, where can I find the sources to compile for my MSVC2015 project? I need to generate TIFF files that are larger than 4GB in size and want to use the most appropriate and up to date library to do so.

Regards,
Paul


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Kemp Watson-2
In reply to this post by Paul Hemmer
Yes, libtiff v4 (I think) and up, certainly the newest few support BigTIFF.

W. Kemp Watson


Objective Pathology Services Limited

8250 Lawson Road
Milton, Ontario
Canada  L9T 5C6

www.objectivepathology.com
tel. +1 (416) 970-7284


From: <[hidden email]> on behalf of Paul Hemmer <[hidden email]>
Date: Thursday, March 16, 2017 at 3:16 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: [Tiff] problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Hello,

Is the "BigTIFF" specification/functionality fully incorporated now into LibTiff?  If yes, great! Which version? If no, where can I find the sources to compile for my MSVC2015 project? I need to generate TIFF files that are larger than 4GB in size and want to use the most appropriate and up to date library to do so.

Regards,
Paul

_______________________________________________ Tiff mailing list: [hidden email] http://lists.maptools.org/mailman/listinfo/tiff http://www.remotesensing.org/libtiff/

_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: problem adding YRESOLUTION and RESOLUTIONUNIT to 16bit grayscale TIFF

Bob Friesenhahn
In reply to this post by Paul Hemmer
On Thu, 16 Mar 2017, Paul Hemmer wrote:

> Hello,
>
> Is the "BigTIFF" specification/functionality fully incorporated now
> into LibTiff?  If yes, great! Which version? If no, where can I find
> the sources to compile for my MSVC2015 project? I need to generate
> TIFF files that are larger than 4GB in size and want to use the most
> appropriate and up to date library to do so.

BigTIFF is supported starting in libtiff 4.0.  I don't recall
BigTIFF-specific issues in any release starting with 4.0.

See http://www.simplesystems.org/libtiff/ or
http://libtiff.maptools.org/ for current information on the project.

You should use the latest version (4.0.7).

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Determining use of "w" or "w8" in TIFFOpen

Paul Hemmer
In reply to this post by Paul Hemmer

Hello!


If I have a buffer of 16bit grayscale data, and a known width and height, I could calculate the size by w*h*2


If that size is greater than 4GB, I should specify "w8" and if less than, I should specify "w" when calling TIFFOpen() in order to get the BigTIFF format when needed.


But what if the use of COMPRESSION_LZW would result in a file that is much smaller than 4GB? I can't know that ahead of time (can I?)  Should I still use w8 if the uncompressed dimensions require more than 4GB?


I'm finding that I can't simply specify "w8" for all, as those that are less than 4GB end up failing to open.


What is the right way to approach this?


Thanks!

Paul




_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

jcupitt
Hello Paul,

On 26 August 2017 at 14:56, Paul Hemmer <[hidden email]> wrote:
> If that size is greater than 4GB, I should specify "w8" and if less than, I
> should specify "w" when calling TIFFOpen() in order to get the BigTIFF
> format when needed.
>
> But what if the use of COMPRESSION_LZW would result in a file that is much
> smaller than 4GB? I can't know that ahead of time (can I?)  Should I still
> use w8 if the uncompressed dimensions require more than 4GB?

I handle this by enabling bigtiff mode automatically if I'm writing an
uncompressed TIFF which will be over 4GB. If compression is turned on,
users have to tick a box (or pass an option) to enable bigtiff write.
It's a bit unsatisfactory.

> I'm finding that I can't simply specify "w8" for all, as those that are less
> than 4GB end up failing to open.

They should only fail to open if the program you are using does not
support bigtiff.

Sadly, rather few programs support bigtiff, hence the need to have it
as an option that users must explicitly turn on.

John
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paul Hemmer

Thank you John for your reply.


I see, I wasn't aware that only certain programs support BigTIFF. 


That being said, if my [height*width*2] calculation results in a value greater than 4GB and I specify "wb" as the open option, but the LZW compressed data ends up taking less than 4GB, the resulting file will still be a "BigTIFF File" and should be viewable inside an image viewer that does support BigTIFF? 


ie. can BigTIFF be used for files that are smaller than 4GB due to the compression?


Thanks!



-Paul





From: [hidden email] <[hidden email]>
Sent: Saturday, August 26, 2017 10:19 AM
To: Paul Hemmer
Cc: [hidden email]
Subject: Re: [Tiff] Determining use of "w" or "w8" in TIFFOpen
 
Hello Paul,

On 26 August 2017 at 14:56, Paul Hemmer <[hidden email]> wrote:
> If that size is greater than 4GB, I should specify "w8" and if less than, I
> should specify "w" when calling TIFFOpen() in order to get the BigTIFF
> format when needed.
>
> But what if the use of COMPRESSION_LZW would result in a file that is much
> smaller than 4GB? I can't know that ahead of time (can I?)  Should I still
> use w8 if the uncompressed dimensions require more than 4GB?

I handle this by enabling bigtiff mode automatically if I'm writing an
uncompressed TIFF which will be over 4GB. If compression is turned on,
users have to tick a box (or pass an option) to enable bigtiff write.
It's a bit unsatisfactory.

> I'm finding that I can't simply specify "w8" for all, as those that are less
> than 4GB end up failing to open.

They should only fail to open if the program you are using does not
support bigtiff.

Sadly, rather few programs support bigtiff, hence the need to have it
as an option that users must explicitly turn on.

John

_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paavo Helde

On 26.08.2017 17:54, Paul Hemmer wrote:
ie. can BigTIFF be used for files that are smaller than 4GB due to the compression?

Absolutely, you can encode a 100 byte file as BigTIFF, it's just a file format. Actually there is some wisdom in producing all TIFF files as BigTIFF regardless of size as this helps to ensure beforehand that the workflows using those files are prepared to work with large files.

hth
Paavo


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paul Hemmer

I see, thank you.


But if a small file is saved with BigTIFF format, then the viewing software still must support BigTIFF, correct?







From: [hidden email] <[hidden email]> on behalf of Paavo Helde <[hidden email]>
Sent: Saturday, August 26, 2017 11:14 AM
To: [hidden email]
Subject: Re: [Tiff] Determining use of "w" or "w8" in TIFFOpen
 

On 26.08.2017 17:54, Paul Hemmer wrote:
ie. can BigTIFF be used for files that are smaller than 4GB due to the compression?

Absolutely, you can encode a 100 byte file as BigTIFF, it's just a file format. Actually there is some wisdom in producing all TIFF files as BigTIFF regardless of size as this helps to ensure beforehand that the workflows using those files are prepared to work with large files.

hth
Paavo


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

jcupitt
In reply to this post by Paul Hemmer
On 26 August 2017 at 15:54, Paul Hemmer <[hidden email]> wrote:
> ie. can BigTIFF be used for files that are smaller than 4GB due to the
> compression?

Yes, bigtiff works fine for small files too. The only drawback with
bigtiff is patchy support.

Even programs which claim to support it will often fall over. I
believe irfanview64 (for example) struggles over about 60k x 60k
pixels.

John
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Roger Leigh
In reply to this post by Paul Hemmer
On 26/08/2017 13:56, Paul Hemmer wrote:

> Hello!
>
>
> If I have a buffer of 16bit grayscale data, and a known width and
> height, I could calculate the size by w*h*2
>
>
> If that size is greater than 4GB, I should specify "w8" and if less
> than, I should specify "w" when calling TIFFOpen() in order to get the
> BigTIFF format when needed.

Yes, but also take care to account for the additional space required by
the TIFF header and all IFDs.  There's an overhead of 16 bytes per tile
or strip using BigTIFF for the offset and size (8 for regular TIFF).
And there may be other metadata stored as well.

I use this logic:

https://github.com/ome/ome-files-cpp/blob/master/lib/ome/files/tiff/Util.cpp#L389

Basically, we add an extra 5% to all the pixel data sizes to ensure we
will have enough space with some headroom for safety.  This means we
switch automatically from TIFF to BigTIFF at approximately 3890 MiB,
which is already quite large.  Or allow the user to manually force one
or the other, with warnings when the limit is exceeded.

> But what if the use of COMPRESSION_LZW would result in a file that is
> much smaller than 4GB? I can't know that ahead of time (can I?)  Should
> I still use w8 if the uncompressed dimensions require more than 4GB?

Because it's hard to tell up front just how much space compression will
save (if any), I ignore it entirely to be safe.  Some image data
compresses well, other data very poorly.

> I'm finding that I can't simply specify "w8" for all, as those that are
> less than 4GB end up failing to open.

The 4GB limit is solely due to the offsets being 32-bit for regular TIFF
and 64-bit for BigTIFF.  You can have a BigTIFF of any size up to 2^64;
being less than 2^32 is not a problem.  Check that your software can
read BigTIFF; it's likely that rather than a problem with the file.


Regards,
Roger
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paul Hemmer
In reply to this post by Paul Hemmer

Thanks again. 

My only concern then goes back to the original question.. I'd like to be able to use BigTIFF only if the final file size demands it.. If the LZW compression results in a file smaller, it would be best to use regular TIFF (just knowing my end-users and their workflows).  

Is there a practical way to pre-compute what the compressed size would be before I open the file and begin writing it? (I suspect this question falls outside the scope of this ListServe.)



_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paul Hemmer
In reply to this post by Roger Leigh

Thank you Roger, this is clear. 


appreciate it!




From: [hidden email] <[hidden email]> on behalf of Roger Leigh <[hidden email]>
Sent: Saturday, August 26, 2017 11:31 AM
To: [hidden email]
Subject: Re: [Tiff] Determining use of "w" or "w8" in TIFFOpen
 
On 26/08/2017 13:56, Paul Hemmer wrote:
> Hello!
>
>
> If I have a buffer of 16bit grayscale data, and a known width and
> height, I could calculate the size by w*h*2
>
>
> If that size is greater than 4GB, I should specify "w8" and if less
> than, I should specify "w" when calling TIFFOpen() in order to get the
> BigTIFF format when needed.

Yes, but also take care to account for the additional space required by
the TIFF header and all IFDs.  There's an overhead of 16 bytes per tile
or strip using BigTIFF for the offset and size (8 for regular TIFF).
And there may be other metadata stored as well.

I use this logic:

https://github.com/ome/ome-files-cpp/blob/master/lib/ome/files/tiff/Util.cpp#L389



Basically, we add an extra 5% to all the pixel data sizes to ensure we
will have enough space with some headroom for safety.  This means we
switch automatically from TIFF to BigTIFF at approximately 3890 MiB,
which is already quite large.  Or allow the user to manually force one
or the other, with warnings when the limit is exceeded.

> But what if the use of COMPRESSION_LZW would result in a file that is
> much smaller than 4GB? I can't know that ahead of time (can I?)  Should
> I still use w8 if the uncompressed dimensions require more than 4GB?

Because it's hard to tell up front just how much space compression will
save (if any), I ignore it entirely to be safe.  Some image data
compresses well, other data very poorly.

> I'm finding that I can't simply specify "w8" for all, as those that are
> less than 4GB end up failing to open.

The 4GB limit is solely due to the offsets being 32-bit for regular TIFF
and 64-bit for BigTIFF.  You can have a BigTIFF of any size up to 2^64;
being less than 2^32 is not a problem.  Check that your software can
read BigTIFF; it's likely that rather than a problem with the file.


Regards,
Roger
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff


http://www.remotesensing.org/libtiff/

_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Paavo Helde
In reply to this post by Paul Hemmer

Yes, of course. If the intent is to reach the widest possible audience then standard TIFF would still be a better option than BigTIFF.


On 26.08.2017 18:22, Paul Hemmer wrote:

I see, thank you.


But if a small file is saved with BigTIFF format, then the viewing software still must support BigTIFF, correct?







From: [hidden email] [hidden email] on behalf of Paavo Helde [hidden email]
Sent: Saturday, August 26, 2017 11:14 AM
To: [hidden email]
Subject: Re: [Tiff] Determining use of "w" or "w8" in TIFFOpen
 

On 26.08.2017 17:54, Paul Hemmer wrote:
ie. can BigTIFF be used for files that are smaller than 4GB due to the compression?

Absolutely, you can encode a 100 byte file as BigTIFF, it's just a file format. Actually there is some wisdom in producing all TIFF files as BigTIFF regardless of size as this helps to ensure beforehand that the workflows using those files are prepared to work with large files.

hth
Paavo



_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Determining use of "w" or "w8" in TIFFOpen

Leonard Rosenthol-3
In reply to this post by Paul Hemmer

> Is there a practical way to pre-compute what the compressed size would be before I open the file and begin writing it? (I suspect this question falls outside the scope of this ListServe.)

> 

The exact size – no – not without actually performing the compression.

 

You can, of course, use some heuristics based on experience…

 

Leonard

 

From: <[hidden email]> on behalf of Paul Hemmer <[hidden email]>
Date: Saturday, August 26, 2017 at 11:35 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [Tiff] Determining use of "w" or "w8" in TIFFOpen

 

Thanks again. 

My only concern then goes back to the original question.. I'd like to be able to use BigTIFF only if the final file size demands it.. If the LZW compression results in a file smaller, it would be best to use regular TIFF (just knowing my end-users and their workflows).  

Is there a practical way to pre-compute what the compressed size would be before I open the file and begin writing it? (I suspect this question falls outside the scope of this ListServe.)

 


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/