Volume 4: OGC CDB Rules for Encoding CDB Vector Data using Shapefiles (Best Practice)

This CDB Best Practice volume defines the requirements and provides guidance on how to use Esri ShapeFiles in a CDB data store.

For ease of editing and review, the standard has been separated into 16 Volumes, one being a schema repository.

\\\\ For later https://github.com/opengeospatial/cdb-volume-1/blob/master/list_of_volumes.adoc \\\\

This standard defines conformance class for testing the use of Esri Shapefiles for storing vector data in a CDB data store.

Conformance with this standard shall be checked using all the relevant tests specified in Annex A (normative) of this document. The framework, concepts, and methodology for testing, and the criteria to be achieved to claim conformance are specified in the OGC Compliance Testing Policies and Procedures and the OGC Compliance Testing web site^[1].

All requirements-classes and conformance-classes described in this document are owned by the standard(s) identified.

The following normative documents contain provisions that, through reference in this text, constitute provisions of this document. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. For undated references, the latest edition of the normative document referred to applies.

Esri ShapeFile Technical Description (https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf)

This document uses the terms defined in Sub-clause 5.3 of [OGC 06-121r8], which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this standard.

Please see the CDB Volume 3: Terms and Definitions document. http://www.opengeospatial.org/standards/cdb

This section provides details and examples for any conventions used in the document. Examples of conventions are symbols, abbreviations, use of XML schema, or special notes regarding how to read the document.

A Shapefile is a specification for encoding and storing vector data storage format for storing the location, shape, and attributes of geographic features. A Shapefile is stored as a set of related files and contains one feature class. The Shapefile specification was developed by Esri.

In a CDB Shapefile, the position of all points is expressed using WGS-84 geographic coordinates (latitude, longitude, altitude), as explained in Volume 8: CDB Spatial Reference Systems Guidance.

As per Esri Shapefile Technical Description, the set of attributes of Vector features are stored in dBase III+ files. Refer to Annex E in Volume 2^[2] of the CDB standard: CDB Annexes for the dBase III file format description and recommended encodings. The CDB standard provides three attribution schemas to represent attribution data:

To completely represent the vector data and attributes in a given tile as a Shapefile, the CDB standard requires that a Vector dataset consists of some or all of the following files:

In addition to *.shp, *.dbf and *.shx files, the Shapefile specification also refers to a memo file with a *.dbt file that is used to store comment fields associated with the attribution *.dbf file.

All of the information that is needed to instance features is organized in accordance to the CDB tile structure. All the tiled Shapefile dataset files are located in the same directory. The dataset’s second component selector (CS2) is used to differentiate between files with the same extension or with the same Vector features. Table 6-1: Component Selector 2 for Vector Dataset, presents the list of codes that are allocated. Note that Vector datasets do not necessarily use all of the Dataset Component Selector 2 reserved codes. Users of the CDB standard should refer to the appropriate section for an enumeration of the supported File Component Selector 2 codes as well as the ones specific to the Dataset.

The Vector dataset concept and the feature code concepts overlap somewhat; some of the Vector datasets are generalizations or specializations of feature codes. Section 1.5 of the OGC CDB Core Standard: Model and Physical Data Store Structure (Volume 1) provides a recommended mapping of the feature attributes across the CDB compliant datasets. Note that the same feature should not have two representations.

Table 6-1: Component Selector 2 for Vector Datasets

Deprecation Note: In CDB Version 1.2, the Shapefile MultiPatch geometry type was deprecated. For backwards compatibility, this geometry type will remain in the CDB standard until such time as CDB Version 2.0 is approved as an OGC standard. AT that time, all MultiPatch references will be removed from all CDB volumes.

Notes about Shapefile Polygon Shapes

Even though the Shapefile standard is very versatile, it also enforces some guidelines with respect to the Polygon Shapes. Those guidelines are referred to in Volume 4: OGC CDB Use of Shapefiles for Vector Data Storage (Best Practice).

Was B.1.3 Annex B Volume 2 of the OGC CDB Best Practice

dBASE .DBF File Structure

by Borland Developer Support Staff

Technical Information Database

This document has been annotated to reflect the conventions established by the CDB standard.  Collectively, these conventions are referred to as dBASE/CDB.  The conventions define how dBASE files are interpreted by a CDB-compliant dBASE reader; the stated conventions supersede or replace related aspects of this annotated specification.  Unless stated otherwise, CDB-compliant dBASE readers will ignore any data that fails to conform to the stated conventions.

Note on directory and file names: Shape/CDB Readers: The CDB standard globally provides a set of directory and filename conventions.  The conventions do not limit filenames to the 8.3 naming convention

TI838D.txt   dBASE .DBF File Structure

Category   :Database Programming

Platform    :All

Product    :Delphi  All

Description:

Sometimes it is necessary to delve into a dBASE table outside the control of the Borland Database Engine (BDE). For instance, if the .DBT file (that contains memo data) for a given table is irretrievably lost, the file will not be usable because the byte in the file header indicates that there should be a corresponding memo file. This necessitates toggling this byte to indicate no such accompanying memo file. Or, you may just want to write your own data access routine.

Below are the file structures for dBASE table files. Represented are the file structures as used for various versions of dBASE: dBASE III PLUS 1.1, dBASE IV 2.0, dBASE 5.0 for DOS, and dBASE 5.0 for Windows.

The table file header:

Table Field Descriptor Bytes

Table Records

The records follow the header in the table file. Data records are preceded by one byte, that is, a space (20h) if the record is not deleted, an asterisk (2Ah) if the record is deleted. Fields are packed into records without field separators or record terminators. The end of the file is mark by a single byte, with the end-of-file marker, an OEM code page character value of 26 (1Ah). You can input OEM code page data as indicated below.

Allowable Input for dBASE Data Types

Binary, Memo, and OLE Fields And .DBT Files

Memo fields store data in .DBT files consisting of blocks numbered sequentially (0, 1, 2, and so on). The size of these blocks are internally set to 512 bytes. The first block in the .DBT file, block 0, is the .DBTfile header.

Memo field of each record in the .DBF file contains the number of the block (in OEM code page values) where the field’s data actually begins. If a field contains no data, the .DBF file contains blanks (20h) rather than a number.

When data is changed in a field, the block numbers may also change and the number in the .DBF may be changed to reflect the new location.

This information is from the Using dBASE III PLUS manual, Appendix C.

File Structure:

The field descriptor array:

Database records:

The records follow the header in the database file. Data records are preceded by one byte; that is, a space (20H) if the record is not deleted, an asterisk (2AH) if the record is deleted. Fields are packed into records without field separators or record terminators. The end of the file is marked by a single byte, with the end-of-file marker an ASCII 26 (1AH) character.

[underline]#Allowable Input for dBASE Data Types:#ß

Memo Fields And .DBT Files

Memo fields store data in .DBT files consisting of blocks numbered sequentially (0, 1, 2, and so on). SET BLOCKSIZE determines the size of each block. The first block in the .DBT file, block 0, is the .DBT file header.

Each memo field of each record in the .DBF file contains the number of the block (in OEM code page values) where the field’s data actually begins. If a field contains no data, the .DBF file contains blanks (20h) rather than a number.

When data is changed in a field, the block numbers may also change and the number in the .DBF may be changed to reflect the new location.

This information is from the dBASE IV Language Reference manual, Appendix D.

The table file header:

Byte  Contents    Description

----- --------    --------------------------------------------------------

n above is the last byte in the field descriptor array. The size of the array depends on the number of fields in the table file.

Table Field Descriptor Bytes

Table Records

The records follow the header in the table file. Data records are preceded by one byte, that is, a space (20h) if the record is not deleted, an asterisk (2Ah) if the record is deleted. Fields are packed into records without field separators or record terminators. The end of the file is marked by a single byte, with the end-of-file marker, an OEM code page character value of 26 (1Ah). You can input OEM code page data as indicated below.

Allowable Input for dBASE Data Types

Memo Fields And .DBT Files

Memo fields store data in .DBT files consisting of blocks numbered sequentially (0, 1, 2, and so on). SET BLOCKSIZE determines the size of each block. The first block in the .DBT file, block 0, is the .DBT file header.

Each memo field of each record in the .DBF file contains the number of the block (in OEM code page values) where the field’s data actually begins. If a field contains no data, the .DBF file contains blanks (20h) rather than a number.

When data is changed in a field, the block numbers may also change and the number in the .DBF may be changed to reflect the new location.

Unlike dBASE III PLUS, if you delete text in a memo field, dBASE 5.0 for DOS may reuse the space from the deleted text when you input new text. dBASE III PLUS always appends new text to the end of the .DBT file. In dBASE III PLUS, the .DBT file size grows whenever new text is added, even if other text in the file is deleted.

This information is from the dBASE for DOS Language Reference manual, Appendix C.

The table file header:

n above is the last byte in the field descriptor array. The size of the array depends on the number of fields in the table file.

Table Field Descriptor Bytes

Table Records (See above)

The records follow the header in the table file. Data records are preceded by one byte, that is, a space (20h) if the record is not deleted, an asterisk (2Ah) if the record is deleted. Fields are packed into records without field separators or record terminators. The end of the file is marked by a single byte, with the end-of-file marker, an OEM code page character value of 26 (1Ah). You can input OEM code page data as indicated below.

Allowable Input for dBASE Data Types

Binary, Memo, and OLE Fields And .DBT Files

Binary, memo, and OLE fields store data in .DBT files consisting of blocks numbered sequentially (0, 1, 2, and so on). SET BLOCKSIZE determines the size of each block. The first block in the .DBT file, block 0, is the .DBT file header.

Each binary, memo, or OLE field of each record in the .DBF file contains the number of the block (in OEM code page values) where the field’s data actually begins. If a field contains no data, the .DBF file contains blanks (20h) rather than a number.

When data is changed in a field, the block numbers may also change and the number in the .DBF may be changed to reflect the new location.

Unlike dBASE III PLUS, if you delete text in a memo field (or binary and OLE fields), dBASE for Windows (unlike dBASE IV) may reuse the space from the deleted text when you input new text. dBASE III PLUS always appends new text to the end of the .DBT file. In dBASE III PLUS, the .DBT file size grows whenever new text is added, even if other text in the file is deleted.

This information is from the dBASE for Windows Language Reference manual, Appendix C.