Reading images with bits per pixel not a multiple of 8

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Reading images with bits per pixel not a multiple of 8

Dinesh Iyer
Hi everyone,
I am trying to use libtiff to read TIFF files and I am looking for clarification when it comes to the layout of images which have bits per pixel that are non-standard such as 2, 4, 11 etc. 

My understanding is that after calling TIFFReadEncodedStrip/Tile the data I get back is packed i.e. one sample might have all its bits stored across a byte-boundary. Is this true for all cases or does this apply only for PackBits compression?

I was able to read 4-bit files using my code but when I tried to extend this for a 11-bit file (yes, we have one in our test-suite), I got a garbled image. 

The image dimensions are: 
height = 154, width = 287, RowsPerStrip = 20, bitsPerSample = 11, SamplesPerPixel = 1.

So, the number of bytes in 1 row of a strip will be ceil(11*287/8) = 395. Hence the number of bytes per strip will be 395*20 = 7900. This value, 7900, matches what TIFFStripSize returned.

My understanding of the layout is as below:

Byte1, Byte2, Byte3, Byte4, Byte5, Byte6, ...

1st sample = Entire Byte1, Bit7,Bit6, Bit5 (of Byte2)
2nd sample: Bits4-0 (of Byte2),  Bits7-2 (of Byte3)
...

Is my understanding of the layout correct? 

The code I am using is below:

// I am using the term slice in my code to mean either strip/tile as appropriate.
uint32_T destSliceWidth = 287;
uint32_T destSliceHeight = 20;
uint16_T samplesPerPixel = 1;
uint16_T bitsPerSample = 11;

// Create a buffer to hold the appropriate number of bytes.
auto dataBuffer = createBuffer<uint16_T>(destSliceWidth*destSliceLength*samplesPerPixel);

// readRawSlice calls TIFFReadEncodedStrip().
std::unique_ptr<uint8_T[]> dataSlice = readRawSlice();

// Read 2-bytes at a time. The readUnitSize logic below should handle this correctly.
uint16_T* srcPtr = reinterpret_cast<uint16_T*>( dataSlice.get() );

uint16_T* destPtr = allocatedOutput<uint16_T>(destSliceWidth, destSliceHeight, samplesPerPixel);

// This will evaluate to 395
size_t srcNumBytesPerRow = static_cast<size_t>( 
                                std::ceil( _imageInfo.sliceWidth*samplesPerPixel*bitsPerSample / 8.0 ) );

// Number of samples (not bits or bytes) in this strip
size_t destTotalNumberOfSamples = destSliceLength*destSliceWidth*samplesPerPixel;

// Each read from the dataSlice will read 16-bits
uint16_T readUnitSize = sizeof(uint16_T)*8;    // In terms of bits

// Offset into the read-unit
uint16_T readUnitOffset = readUnitSize; // In terms of bits

// The actual computed 11-bit value of the sample
uint16_T srcSampleVal = 0;

for( size_t destCnt = 0; destCnt < destTotalNumberOfSamples; destCnt++ )
{
    // If end-of-row reached, then move 395*rowNum byes from the start of dataSlice to get to the next row
    if( destCnt % _imageInfo.sliceWidth == 0)
    {
        srcPtr = reinterpret_cast<uint16_T*>( dataSlice.get()  + ( destCnt / _imageInfo.sliceWidth )*srcNumBytesPerRow );
        readUnitOffset = readUnitSize;
    }
    uint16_T destSampleVal = 0;

    // Indicates that 11-bits need to be grabbed to compute the value of this sample completely.
    uint16_T reqNumBitsForSample = bitsPerSample;

    // If 16-bits have been completely processed, then read 16-bits more from the source.
    if( readUnitOffset == readUnitSize )
    {
        srcSampleVal = *srcPtr++;
        readUnitOffset = 0;
    }

    // Loop until 11-bits have been obtained.
    while( reqNumBitsForSample != 0 )
    {
        // Determine how many bits are available to read in the current 16-bit value read in
        uint16_T numReqBitsAvailInReadUnit = std::min<uint16_T>( reqNumBitsForSample, readUnitSize - readUnitOffset );

        // Create a bit-mask
        uint16_T bitMask = static_cast<uint16_T>( std::pow( 2.0, 
                                                static_cast<double>( readUnitSize - readUnitOffset ) ) - 1 );
                            
        // Mask and shift the values suitably.
        uint16_T value = srcSampleVal & bitMask;
        value >>= (readUnitSize - (readUnitOffset + numReqBitsAvailInReadUnit));

        // Add to already extracted bits
        destSampleVal |= (value << (reqNumBitsForSample - numReqBitsAvailInReadUnit));

        // Updated the ofset and the number of bits required
        readUnitOffset += numReqBitsAvailInReadUnit;
        reqNumBitsForSample -= numReqBitsAvailInReadUnit;

        if( readUnitOffset == readUnitSize )
        {
            srcSampleVal = *srcPtr++;
            readUnitOffset = 0;
        }
    }

    *destPtr++ = destSampleVal;
}

Appreciate any help.

Dinesh

_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Bob Friesenhahn
On Mon, 3 Apr 2017, Dinesh Iyer wrote:
>
> My understanding is that after calling TIFFReadEncodedStrip/Tile the data I
> get back is packed i.e. one sample might have all its bits stored across a
> byte-boundary. Is this true for all cases or does this apply only for
> PackBits compression?

It applies for all cases.

Except for depths 8, 16, and 24 (which are swapped to "native"
endianness by libtiff), the data is always ordered in
most-significant-bit order
so you need to start with the top bits.

I put sample files (with interesting strange depths) written by
GraphicsMagick out for ftp under
ftp://ftp.simplesystems.org/pub/tiff-samples/

In particular

ftp://ftp.simplesystems.org/pub/tiff-samples/tiff-sample-images-be.tar.gz

and

ftp://ftp.simplesystems.org/pub/tiff-samples/tiff-sample-images-le.tar.gz

There are also BigTIFF versions there.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Roger Leigh
On 03/04/2017 18:50, Bob Friesenhahn wrote:

> On Mon, 3 Apr 2017, Dinesh Iyer wrote:
>>
>> My understanding is that after calling TIFFReadEncodedStrip/Tile the data I
>> get back is packed i.e. one sample might have all its bits stored across a
>> byte-boundary. Is this true for all cases or does this apply only for
>> PackBits compression?
>
> It applies for all cases.
>
> Except for depths 8, 16, and 24 (which are swapped to "native"
> endianness by libtiff), the data is always ordered in
> most-significant-bit order
> so you need to start with the top bits.

What about 32, 48 and 64 bit variants which are all multiples of these
specially handled sizes?

Just wondering because I am not currently doing any special case
handling for 32-bit and higher sizes.


Thanks,
Roger
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Bob Friesenhahn
On Mon, 3 Apr 2017, Roger Leigh wrote:

>>
>> Except for depths 8, 16, and 24 (which are swapped to "native"
>> endianness by libtiff), the data is always ordered in
>> most-significant-bit order
>> so you need to start with the top bits.
>
> What about 32, 48 and 64 bit variants which are all multiples of these
> specially handled sizes?
>
> Just wondering because I am not currently doing any special case
> handling for 32-bit and higher sizes.

You are correct.  I should have mentioned 32 and 64 bit sizes as well.
I am not sure about 48.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Dinesh Iyer
Currently, in my code, when I read 16-bit images, I just treat the buffer returned by TiffReadEncodedStrip as uint16_T and read the data. It appears that is incorrect because it will give the wrong result for big-endian files. Am I correct?

Regards,
Dinesh

On Mon, Apr 3, 2017 at 2:46 PM, Bob Friesenhahn <[hidden email]> wrote:
On Mon, 3 Apr 2017, Roger Leigh wrote:
>>
>> Except for depths 8, 16, and 24 (which are swapped to "native"
>> endianness by libtiff), the data is always ordered in
>> most-significant-bit order
>> so you need to start with the top bits.
>
> What about 32, 48 and 64 bit variants which are all multiples of these
> specially handled sizes?
>
> Just wondering because I am not currently doing any special case
> handling for 32-bit and higher sizes.

You are correct.  I should have mentioned 32 and 64 bit sizes as well.
I am not sure about 48.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Bob Friesenhahn
On Tue, 4 Apr 2017, Dinesh Iyer wrote:

> Currently, in my code, when I read 16-bit images, I just treat the buffer
> returned by TiffReadEncodedStrip as uint16_T and read the data. It appears
> that is incorrect because it will give the wrong result for big-endian
> files. Am I correct?

I don't think so.  As mentioned before, libtiff automatically swaps
sample sizes of 16, 24, 32, and 64 to native host order before
returning the buffer.  It also swaps bit order if necessary.

Bob

>
> Regards,
> Dinesh
>
> On Mon, Apr 3, 2017 at 2:46 PM, Bob Friesenhahn <
> [hidden email]> wrote:
>
>> On Mon, 3 Apr 2017, Roger Leigh wrote:
>>>>
>>>> Except for depths 8, 16, and 24 (which are swapped to "native"
>>>> endianness by libtiff), the data is always ordered in
>>>> most-significant-bit order
>>>> so you need to start with the top bits.
>>>
>>> What about 32, 48 and 64 bit variants which are all multiples of these
>>> specially handled sizes?
>>>
>>> Just wondering because I am not currently doing any special case
>>> handling for 32-bit and higher sizes.
>>
>> You are correct.  I should have mentioned 32 and 64 bit sizes as well.
>> I am not sure about 48.
>>
>> Bob
>> --
>> Bob Friesenhahn
>> [hidden email], http://www.simplesystems.org/users/bfriesen/
>> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
>> _______________________________________________
>> Tiff mailing list: [hidden email]
>> http://lists.maptools.org/mailman/listinfo/tiff
>> http://www.remotesensing.org/libtiff/
>>
>

--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/
Reply | Threaded
Open this post in threaded view
|

Re: Reading images with bits per pixel not a multiple of 8

Charles Auer
In reply to this post by Dinesh Iyer

TIFFIsByteSwapped returns a non-zero value if the image data was in a different byte-order than the host machine. Zero is returned if the TIFF file and local host byte-orders are the same. Note that TIFFReadTile(), TIFFReadStrip() and TIFFReadScanline() functions already normally perform byte swapping to local host order if needed.

.

I'm not sure if this is helpful.

.






From: [hidden email] <[hidden email]> on behalf of Dinesh Iyer <[hidden email]>
Sent: Tuesday, April 4, 2017 2:00 PM
To: Bob Friesenhahn
Cc: TIFF mailing list
Subject: Re: [Tiff] Reading images with bits per pixel not a multiple of 8
 
Currently, in my code, when I read 16-bit images, I just treat the buffer returned by TiffReadEncodedStrip as uint16_T and read the data. It appears that is incorrect because it will give the wrong result for big-endian files. Am I correct?

Regards,
Dinesh

On Mon, Apr 3, 2017 at 2:46 PM, Bob Friesenhahn <[hidden email]> wrote:
On Mon, 3 Apr 2017, Roger Leigh wrote:
>>
>> Except for depths 8, 16, and 24 (which are swapped to "native"
>> endianness by libtiff), the data is always ordered in
>> most-significant-bit order
>> so you need to start with the top bits.
>
> What about 32, 48 and 64 bit variants which are all multiples of these
> specially handled sizes?
>
> Just wondering because I am not currently doing any special case
> handling for 32-bit and higher sizes.

You are correct.  I should have mentioned 32 and 64 bit sizes as well.
I am not sure about 48.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/


_______________________________________________
Tiff mailing list: [hidden email]
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/