No space for data buffer error... RRS feed

  • Question

  • While attempting to join 42 tiff tiles together, each of size 300,000 x 7143 pixels, the hdmake process gets to about image number 21 and then starts spitting out the error:

    TIFFReadBufferSetup: <full path to my image>: No space for data buffer at scanline -1.

    Then after repeatedly spitting out this error hdmake eventually gives up and reports:

    HDMake: ERROR - hr = 80070070

    I have 16Gb of RAM and was watching the memory usage and it didn't approach anything beyond a few gig.

    Do you know what this bottleneck could be? Is there some way I can circumvent this?

    Many thanks,
    Helgi Rudd

    Wednesday, September 5, 2012 8:15 AM

All replies

  • Those are really large input images which is causing our tiff decoder to run out of memory. Are you on 64bit Windows?

    Wednesday, September 5, 2012 6:39 PM
  • Hi Matt,

    Thank you for getting back to me :-)

    I recently switched to a 64 bit system... didn't even think about that side of things, I was using the original 32 bit install I had.

    I just downloaded the 64 bit version... the hdmake.exe was a different file size so I had high hopes it would handle the process. It got a lot further and a little more than 9,000,000K of RAM (I have about 14,000,000K available for it if it wants it).

    Different behaviour on the failure... it didn't report "TIFFReadBufferSetup: <full path to my image>: No space for data buffer at scanline -1." this time... instead it got to just over 62% and bombed out with HDMake: ERROR - hr = 80070070

    I have paste the result from the cmd.exe window below.

    I was hoping to end up with a 90 Gigapixel image in the end. Do you know what the limit is? I'm throwing 8 bit per pixel black and white tiff images at it (which could be 1 bit per pixel but as I understand it HDMake requires 8 bit per pixel)

    Thanks again :-)

    Helgi Rudd

    Microsoft Windows [Version 6.1.7601]
    Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

    C:\Users\Owner>"D:\300000x300000 Ulam Spiral\HD Make\hdmake.exe" -nozip -extension png -html -srcgrid 1 42 "D:\300000x300000 Ulam Spiral\300000.tiff_1.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_2.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_3.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_4.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_5.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_6.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_7.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_8.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_9.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_10.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_11.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_12.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_13.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_14.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_15.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_16.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_17.tiff" "D:\300000x300000 Ulam Spiral\30000
    0.tiff_18.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_19.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_20.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_21.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_22.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_23.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_24.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_25.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_26.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_27.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_28.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_29.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_30.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_31.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_32.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_33.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_34.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_35.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_36.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_37.tiff" "D:\300000x3000
    00 Ulam Spiral\300000.tiff_38.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_39.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_40.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_41.tiff" "D:\300000x300000 Ulam Spiral\300000.tiff_42.tiff"
    HDMake: Begin -> opening source image(s)
    HDMake:     0%  3%  5%  8% 10% 13% 15% 18% 21% 23% 26% 28% 31% 33% 36% 38% 41% 44% 46% 49% 51% 54% 56% 59% 62%HDMake: ERROR - hr = 80070070


    Thursday, September 6, 2012 12:11 AM
  • This error means that you that you ran out of disk space. hdmake is creating a copy of your entire input data set, so you need space for that as well as the output.
    Thursday, September 6, 2012 1:14 AM
  • Ah okay.

    The 42 source images add up to about 3.4Gb

    They are on a 2Tb hard disk... with a few other things on it... it's got 1.8Tb left.... this seems like a lot of space in comparison to the source files.

    I've copied hdmake.exe onto the same disk.

    Does hdmake.exe output to the same directory it's running on?

    Thursday, September 6, 2012 1:30 AM
  • hdmake will represent these as 8bit images - 300,000 x 7143 x 8 x 41 = 703GB  It does seem like you have adequate space even for this, so I'm not sure what is going on.  The temp storage for hdmake is in %temp% usually C:\Users\...\AppData\Local\Temp

    Thursday, September 6, 2012 3:29 AM
  • Okay, that could explain it then... my local / C disk doesn't have anything like that space (it's a solid state drive so much smaller).

    The 2Tb disc is my D drive... is there some way to change the temp storage location for hdmake?

    Thursday, September 6, 2012 3:38 AM
  • I'm going to try changing environment variable to set temp folders to my larger disc... will see how that goes :-)
    Thursday, September 6, 2012 3:42 AM
  • I just took a look at the source code.  If it is set, hdmake uses the environment variable ICE_CACHE_LOCATION to set the temp location, so you can also use this.

    Thursday, September 6, 2012 3:48 AM
  • Thank you so much for all your help Matt!

    It worked... took a while, but I now have a 90 Gigapixel image... ends up being just under 30GB and 1,863,278 files in the tile collection.

    Going to upload the sucker... might take a while :-)

    Thanks again, fantastic piece of software!

    Thursday, September 6, 2012 6:32 AM
  • Happy to hear that this worked for you. Let us know when you get it posted.
    Thursday, September 6, 2012 5:02 PM
  • Took a few days to FTP the 1,863,281 files up, but it's finally up.... the ridiculously large 90 Gigapixel Ulam Spiral:

    Here's a link below starting zoomed into the center... not much more than a grey blob when zoomed out.

    If each square on the Ulam Spiral was 1 millimeter square the image would be 300 meters wide and tall.

    Thank you so much for making this possible, it's great fun using HD View, couldn't get the image viewable otherwise!


    Monday, September 10, 2012 10:46 PM