PyTables File Format
====================
PyTables has a powerful capability to deal with native HDF5 files
created with another tools. However, there are situations were you may
want to create truly native PyTables files with those tools while
retaining fully compatibility with PyTables format. That is perfectly
possible, and in this appendix is presented the format that you should
endow to your own-generated files in order to get a fully PyTables
compatible file.

We are going to describe the *2.0 version of PyTables
file format* (introduced in PyTables version 2.0). As time
goes by, some changes might be introduced (and documented here) in order
to cope with new necessities. However, the changes will be carefully
pondered so as to ensure backward compatibility whenever is
possible.

A PyTables file is composed with arbitrarily large amounts of HDF5
groups (Groups in PyTables naming scheme) and
datasets (Leaves in PyTables naming scheme). For
groups, the only requirements are that they must have some
*system attributes* available. By convention, system
attributes in PyTables are written in upper case, and user attributes in
lower case but this is not enforced by the software. In the case of
datasets, besides the mandatory system attributes, some conditions are
further needed in their storage layout, as well as in the datatypes used
in there, as we will see shortly.

As a final remark, you can use any filter as you want to create a
PyTables file, provided that the filter is a standard one in HDF5, like
*zlib*, *shuffle* or
*szip* (although the last one can not be used from
within PyTables to create a new file, datasets compressed with szip can
be read, because it is the HDF5 library which do the decompression
transparently).

.. currentmodule:: tables

Mandatory attributes for a File
-------------------------------
The File object is, in fact, an special HDF5
*group* structure that is *root*
for the rest of the objects on the object tree. The next attributes
are mandatory for the HDF5 *root group* structure
in PyTables files:

* *CLASS*: This attribute should always be set to 'GROUP' for group structures.
* *PYTABLES_FORMAT_VERSION*: It represents the internal format version, and currently
  should be set to the '2.0' string.
* *TITLE*: A string where the user can put some description on what is this group used for.
* *VERSION*: Should contains the string '1.0'.



Mandatory attributes for a Group
--------------------------------
The next attributes are mandatory for *group*
structures:

* *CLASS*: This attribute should always be set to 'GROUP' for group structures.
* *TITLE*: A string where the user can put some description on what is this group used for.
* *VERSION*: Should contains the string '1.0'.


Optional attributes for a Group
-------------------------------
The next attributes are optional for *group*
structures:

* *FILTERS*: When present, this attribute contains the filter
  properties (a Filters instance, see section
  :ref:`FiltersClassDescr`)
  that may be inherited by leaves or groups created immediately
  under this group. This is a packed 64-bit integer structure,
  where

  - *byte 0* (the least-significant byte) is the compression level (complevel).
  - *byte 1* is the compression library used (complib): 0
    when irrelevant, 1 for Zlib, 2 for LZO and 3 for Bzip2.
  - *byte 2* indicates which parameterless filters are enabled
    (shuffle and fletcher32): bit 0 is for
    *Shuffle* while bit 1 is for*Fletcher32*.
  - other bytes are reserved for future use.


Mandatory attributes, storage layout and supported data types for Leaves
------------------------------------------------------------------------
This depends on the kind of Leaf. The format for each type follows.



.. _TableFormatDescr:

Table format
~~~~~~~~~~~~

Mandatory attributes
^^^^^^^^^^^^^^^^^^^^
The next attributes are mandatory for *table* structures:

* *CLASS*: Must be set to 'TABLE'.
* *TITLE*: A string where the user can put some description on what is this dataset used for.
* *VERSION*: Should contain the string '2.6'.
* *FIELD_X_NAME*: It contains the names of the different fields. The
  X means the number of the field,
  zero-based (beware, order do matter). You should add as many
  attributes of this kind as fields you have in your
  records.
* *FIELD_X_FILL*: It contains the default values of the different
  fields. All the datatypes are supported natively, except for
  complex types that are currently serialized using Pickle.
  The X means the number of the field,
  zero-based (beware, order do matter). You should add as many
  attributes of this kind as fields you have in your records.
  These fields are meant for saving the default values
  persistently and their existence is optional.
* *NROWS*: This should contain the number of *compound* data type entries in the
  dataset. It must be an *int* data type.


Storage Layout
^^^^^^^^^^^^^^
A Table has a *dataspace* with a *1-dimensional chunked* layout.

Datatypes supported
^^^^^^^^^^^^^^^^^^^
The datatype of the elements (rows) of Table must be the H5T_COMPOUND
*compound* data type, and each of these compound components must be built with only the next HDF5 data
types *classes*:

* *H5T_BITFIELD*: This class is used to represent the Bool type. Such a type must be build
  using a H5T_NATIVE_B8 datatype, followed by a HDF5
  H5Tset_precision call to set its precision to be just 1 bit.
* *H5T_INTEGER*: This includes the next data types:
    * *H5T_NATIVE_SCHAR*: This represents a *signed char* C type, but it is effectively used to
      represent an Int8 type.
    * *H5T_NATIVE_UCHAR*:  This represents an *unsigned char* C type, but it is effectively used to
      represent an UInt8 type.
    * *H5T_NATIVE_SHORT*: This represents a *short* C type, and it is effectively used to represent an
      Int16 type.
    * *H5T_NATIVE_USHORT*: This represents an *unsigned
      short* C type, and it is effectively used to represent an UInt16 type.
    * *H5T_NATIVE_INT*: This represents an *int* C type, and it is effectively used to represent an
      Int32 type.
    * *H5T_NATIVE_UINT*: This represents an *unsigned int* C type, and it is effectively used to
      represent an UInt32 type.
    * *H5T_NATIVE_LONG*: This represents a *long* C type, and it is effectively used to represent an
      Int32 or an Int64, depending on whether you are running a 32-bit or 64-bit architecture.
    * *H5T_NATIVE_ULONG*: This represents an *unsigned
      long* C type, and it is effectively used to represent an UInt32 or an
      UInt64, depending on whether you
      are running a 32-bit or 64-bit architecture.
    * *H5T_NATIVE_LLONG*: This represents a *long long* C type (__int64, if you are using a
      Windows system) and it is effectively used to represent an Int64 type.
    * *H5T_NATIVE_ULLONG*: This represents an *unsigned long
      long* C type (beware: this type does not have a correspondence on Windows systems) and it is
      effectively used to represent an UInt64 type.
* *H5T_FLOAT*: This includes the next datatypes:
    * *H5T_NATIVE_FLOAT*: This represents a *float* C type and it is effectively used to represent an
      Float32 type.
    * *H5T_NATIVE_DOUBLE*: This represents a *double* C type and it is effectively used to represent an
      Float64 type.
* *H5T_TIME*: This includes the next datatypes:
    * *H5T_UNIX_D32*: This represents a POSIX *time_t* C type and it is effectively used to represent a
      'Time32' aliasing type, which corresponds to an Int32 type.
    * *H5T_UNIX_D64*: This represents a POSIX *struct timeval* C type and it is effectively used
      to represent a 'Time64' aliasing type, which corresponds to a Float64 type.
* *H5T_STRING*: The datatype used to describe strings in PyTables is H5T_C_S1 (i.e. a *string* C type)
  followed with a call to the HDF5 H5Tset_size() function to set their length.
* *H5T_ARRAY*: This allows the construction of homogeneous, multidimensional arrays,
  so that you can include such objects in compound records. The types supported as elements
  of H5T_ARRAY data types are the ones described above.
  Currently, PyTables does not support nested H5T_ARRAY types.
* *H5T_COMPOUND*: This allows the support for datatypes that are compounds of compounds
  (this is also known as *nested types* along this manual).

  This support can also be used for defining complex numbers. Its format is described below:

  The H5T_COMPOUND type class contains two members. Both
  members must have the H5T_FLOAT atomic datatype class. The
  name of the first member should be "r" and represents the
  real part. The name of the second member should be "i" and
  represents the imaginary part. The
  *precision* property of both of the
  H5T_FLOAT members must be either 32 significant bits (e.g.
  H5T_NATIVE_FLOAT) or 64 significant bits (e.g.
  H5T_NATIVE_DOUBLE). They represent Complex32 and Complex64
  types respectively.



Array format
~~~~~~~~~~~~

Mandatory attributes
^^^^^^^^^^^^^^^^^^^^
The next attributes are mandatory for *array* structures:

* *CLASS*: Must be set to 'ARRAY'.
* *TITLE*: A string where the user can put some description on
  what is this dataset used for.
* *VERSION*: Should contain the string '2.3'.


Storage Layout
^^^^^^^^^^^^^^
An Array has a
*dataspace* with a *N-dimensional
contiguous* layout (if you prefer a
*chunked* layout see EArray
below).


Datatypes supported
^^^^^^^^^^^^^^^^^^^
The elements of Array must have either
HDF5 *atomic* data types or a
*compound* data type representing a complex
number. The atomic data types can currently be one of the next
HDF5 data type *classes*: H5T_BITFIELD,
H5T_INTEGER, H5T_FLOAT and H5T_STRING. The H5T_TIME class is also
supported for reading existing Array objects,
but not for creating them. See the Table format
description in :ref:`TableFormatDescr` for more info about these
types.

In addition to the HDF5 atomic data types, the Array format
supports complex numbers with the H5T_COMPOUND data type class.
See the Table format description in :ref:`TableFormatDescr` for more
info about this special type.

You should note that H5T_ARRAY class datatypes are not
allowed in Array objects.


CArray format
~~~~~~~~~~~~~

Mandatory attributes
^^^^^^^^^^^^^^^^^^^^
The next attributes are mandatory for *CArray* structures:

* *CLASS*: Must be set to 'CARRAY'.
* *TITLE*: A string where the user can put some description on
  what is this dataset used for.
* *VERSION*: Should contain the string '1.0'.


Storage Layout
^^^^^^^^^^^^^^
An CArray has a
*dataspace* with a *N-dimensional
chunked* layout.

Datatypes supported
^^^^^^^^^^^^^^^^^^^
The elements of CArray must have either
HDF5 *atomic* data types or a
*compound* data type representing a complex
number. The atomic data types can currently be one of the next
HDF5 data type *classes*: H5T_BITFIELD,
H5T_INTEGER, H5T_FLOAT and H5T_STRING. The H5T_TIME class is also
supported for reading existing CArray objects,
but not for creating them. See the Table format
description in :ref:`TableFormatDescr` for more info about these
types.

In addition to the HDF5 atomic data types, the CArray format
supports complex numbers with the H5T_COMPOUND data type class.
See the Table format description in :ref:`TableFormatDescr` for more
info about this special type.

You should note that H5T_ARRAY class datatypes are not
allowed yet in Array objects.


EArray format
~~~~~~~~~~~~~

Mandatory attributes
^^^^^^^^^^^^^^^^^^^^
The next attributes are mandatory for *earray* structures:

* *CLASS*: Must be set to 'EARRAY'.
* *EXTDIM*: (*Integer*) Must be set to the
  extendable dimension. Only one extendable dimension is
  supported right now.
* *TITLE*: A string where the user can put some description on
  what is this dataset used for.
* *VERSION*: Should contain the string '1.3'.


Storage Layout
^^^^^^^^^^^^^^
An EArray has a
*dataspace* with a *N-dimensional
chunked* layout.

Datatypes supported
^^^^^^^^^^^^^^^^^^^
The elements of EArray are allowed to
have the same data types as for the elements in the Array format.
They can be one of the HDF5 *atomic* data type
*classes*: H5T_BITFIELD, H5T_INTEGER,
H5T_FLOAT, H5T_TIME or H5T_STRING, see the
Table format description in :ref:`TableFormatDescr` for more
info about these types. They can also be a H5T_COMPOUND datatype
representing a complex number, see the Table
format description in :ref:`TableFormatDescr`.

You should note that H5T_ARRAY class data types are not
allowed in EArray objects.



.. _VLArrayFormatDescr:

VLArray format
~~~~~~~~~~~~~~

Mandatory attributes
^^^^^^^^^^^^^^^^^^^^
The next attributes are mandatory for *vlarray* structures:

* *CLASS*: Must be set to 'VLARRAY'.
* *PSEUDOATOM*: This is used so as to specify the kind of pseudo-atom
  (see :ref:`VLArrayFormatDescr`) for the VLArray. It can take
  the values 'vlstring',
  'vlunicode' or 'object'. If your atom is not a
  pseudo-atom then you should not specify it.
* *TITLE*: A string where the user can put some description on
  what is this dataset used for.
* *VERSION*: Should contain the string '1.3'.

Storage Layout
^^^^^^^^^^^^^^
An VLArray has a
*dataspace* with a *1-dimensional
chunked* layout.

Data types supported
^^^^^^^^^^^^^^^^^^^^
The data type of the elements (rows) of
VLArray objects must be the H5T_VLEN
*variable-length* (or VL for short) datatype,
and the base datatype specified for the VL datatype can be of any
*atomic* HDF5 datatype that is listed in the
Table format description :ref:`TableFormatDescr`.  That
includes the classes:

- H5T_BITFIELD
- H5T_INTEGER
- H5T_FLOAT
- H5T_TIME
- H5T_STRING
- H5T_ARRAY

They can also be a H5T_COMPOUND data type representing a
complex number, see the Table format
description in :ref:`TableFormatDescr` for a detailed description.

You should note that this does not include another VL
datatype, or a compound datatype that does not fit the description
of a complex number. Note as well that, for
object and vlstring
pseudo-atoms, the base for the VL datatype is always a
H5T_NATIVE_UCHAR
(H5T_NATIVE_UINT for
vlunicode). That means that the complete row
entry in the dataset has to be used in order to fully serialize
the object or the variable length string.


Optional attributes for Leaves
------------------------------
The next attributes are optional for *leaves*:

* *FLAVOR*: This is meant to provide the information about the kind of
  object kept in the Leaf, i.e. when the
  dataset is read, it will be converted to the indicated flavor.
  It can take one the next string values:

    * *"numpy"*: Read data (record arrays, arrays, records, scalars)
      will be returned as NumPy objects.
    * *"numarray"*: Read data will be returned as
      numarray objects (deprecated).
    * *"numeric"*: Read data will be returned as
      Numeric objects (deprecated).
    * *"python"*: Read data will be returned as Python lists, tuples, or scalars.

