Dink Software

Excuse the vague thread title, but i did not know how best to word it.

Basically... I've been wrapping a C library --specifically FastLZ-- for Python using the ctypes module. This it self hasn't been too problematic, i even managed to wrap fastlz_compress_level(). Now, my "issue" cropped up when started on fastlz_decompress(). I probably don't get it because of my lack of experience with C, but, i simply don't understand the parameters i'm meant to pass with the function call.

From the header file:

Code: Select all

/**
  Decompress a block of compressed data and returns the size of the
  decompressed block. If error occurs, e.g. the compressed data is
  corrupted or the output buffer is not large enough, then 0 (zero)
  will be returned instead.

  The input buffer and the output buffer can not overlap.

  Decompression is memory safe and guaranteed not to write the output buffer
  more than what is specified in maxout.
*/

int fastlz_decompress(const void* input, int length, void* output, int maxout);

Now i understand the const void* input and int length parameters. It starts confusing me when i get to void* output and int maxout.

Taking into account that i do not know, nor can reliably predict, the size of the end result, how am i meant to know what size to set the output buffer to? I also don't understand the purpose of int maxout. Even reading the comment in the header a few hundred times hasn't help make it any clearer.

If you can shed any light what so ever on the matter, i'd be highly grateful.

I know this is a vague question, but it reflect the vague look on my face when i try to understand what's going on.

I'm not really sure...

Isn't it just saying that the output will never be void (as even if an error occurs, 0 will be returned as the output) and is just that, a destination/file locality for the decompressed? I don't know what maxout would be, the maximum decompression rate (bits?) per second?

A void pointer (void*) essentially means you can pass any other type of pointer in it's favour. In my case, a char*. So you are right when you say it's the "destination" of the decompressed data.

However, the fundamental problem is, how do i know what size the block should be the pointer points to. With fastlz_compress_level() it told how big the output buffer would have to be: at least 5% larger than the input buffer and greater than 66 bytes.

Code: Select all

# Ouput buffer must be at least 105 per cent of the input buffer size
   output_buffer_length = int(round((input_buffer_length +
                           (input_buffer_length * 0.05)) + 0.5, 0))
   
   # Force the output buffer to be at least 66 bytes
   if output_buffer_length < 66:
      output_buffer_length += (66 - output_buffer_length)


...and that appears to work correctly.

I really don't know where to start with this. I might have to go ask someone who is more knowledgeable on such matters.

Right, i think i've finally got the compression function working correctly.

Code: Select all

# @wrap fastlz_compress_level()
def compress(string=str, level=FASTEST_LEVEL):
   """Compresses string using the specified level and returns it."""
   
   _load_library()
      
   ## int fastlz_compress_level(int level, const void* input, int length, void* output)
   
   string = str(string) # Ensure string is a byte-string

   c_level = ctypes.c_int(level)
   
   input_buffer_length = _calc_string_size(string)
   
   if input_buffer_length < 16:
      raise FastLZError("Input buffer (string) must be greater than 16 bytes")
   
   c_length = ctypes.c_int(input_buffer_length)
   c_input_buffer = ctypes.pointer(ctypes.c_char_p(string))

   output_buffer_length = int(round((input_buffer_length +
                              (input_buffer_length * 0.05)) + 0.5, 0))
   
   # Force the output buffer to be at least 66 bytes
   if output_buffer_length < 66:
      output_buffer_length += (66 - output_buffer_length)
   
   c_output_buffer = ctypes.create_string_buffer(output_buffer_length)

   returned = _libfastlz.fastlz_compress_level(c_level, c_input_buffer,
                                       c_length, c_output_buffer)
   
   if returned > input_buffer_length:
      raise FastLZError("""Returned buffer is greater than input buffer;
                                 compression likley failed.""")

   return c_output_buffer.raw[:returned]

Don't attempt to run it, as this is not the whole module. And besides, i'm 95 per cent sure it will only work on Linux and alike, and ya'll wont have the FastLZ library. I'm just hoping you could tell me if anything looks wrong.

However, i am still stumped on how to approach fastlz_decompress(). However, i was thinking: maybe it could just "guess" a size, and then increase it if need be. The only problem with this is, it would be incredibly slow and is most probably not the "right way" to do this.