Harbour File Size Limits

 

Harbour File Size Limits

  • Max record size: 2^16-1 = 65535 byts ( 64 MB )
  • Max number of recors : 2^32-1 = 4,294,967,295 ( 4 Bilion )
  • Max .dbf file size : 2^48 = 256 TB
  • Max DBT memo file size : 2 TB
  • Max FPT memo file size : 256 GB
  • Max SMT memo file size : 128 GB
  • Max NTX file size (standard) : 4GB
  • Max NTX file size (incresead ) : 4TB
  • Max CDX file size : 4GB

 

Source : Harbour\doc\xhb-diff.txt :

### NATIVE RDDs ###

In both compilers maximal file size for tables, memos and indexes is limited only by OS and file format structures. Neither Harbour nor xHarbour introduce own limits here.

The maximal file size for DBFs is limited by number of records 2^32-1 = 4294967295 and maximal record size: 2^16-1 = 65535 what gives nearly 2^48 = 256TB as maximal .dbf file size.

The maximal memo format size depends on used memo type: DBT, FPT or SMT and size of  memo block. It’s limited by maximal number of memo blocks = 2^32 and size of memo block so it’s 2^32*<size_of_memo_block>.

The default memo block size for DBT is 512 bytes, FPT – 64 bytes and for SMT 32 bytes. So for standard memo block sizes the maximum are:

DBT->2TB, FPT->256GB, SMT->128GB. The maximal memo block size in Harbour is 2^32 and minimal is 1 byte and it can be any value between 1 and 65536 and then any number of 64KB blocks. The last limitation is introduced as workaround for some wrongly implemented in other
languages memo drivers which were setting only 16 bits in 32bit field in memo header. Most of other languages has limit for memo block size at 2^15 and the block size has to be power of 2. Some of them also introduce minimal block size limits. If programmers plans to share data with programs compiled by such languages then he should check their documentation to not create memo files which cannot be accessed by them.

Maximal NTX file size for standard NTX files is 4GB and it’s limited by internal NTX structures. Enabling 64bit locking in [x]Harbour change slightly used NTX format and increase maximum NTX file size to 4TB.

The NTX format in [x]Harbour has also many other extensions like support for multitag indexes or using record number as hidden part of index key and many others which are unique to [x]Harbour. In practice all of CDX extensions are supported by NTX in [x]Harbour.

The NSX format in [x]Harbour is also limited by default to 4GB but like in NTX enabling 64bit locking extend it to 4TB. It also supports common to NTX and CDX set of features.

The CDX format is limited to 4GB and so far [x]Harbour does not support extended mode which can increase the size up to 2TB with standard page length and it can be bigger in all formats if we introduce support for bigger index pages. Of course all such extended formats are not binary compatible with original ones and so far can be used only by [x]Harbour RDDs though in ADS the .adi format is such extended CDX format so maybe in the future it will be possible to use .adi indexes in our CDX RDD.

Of course all of the above sizes can be reduced by operating system (OS) or file system (FS) limitations so it’s necessary to check what is supported by environment where [x]Harbour applications are executed. 

Advertisements

C5DG-6 DBFNDX Driver

Clipper 5.x – Drivers Guide

Chapter 6

DBFNDX Driver Installation and Usage

DBFNDX is the dBASE III PLUS compatible RDD for Clipper. The DBFNDX driver uses the Clipper driver architecture to access dBASE III PLUS compatible index files within a Clipper program.

In This Chapter

This chapter explains how to install DBFNDX and how to use it in your applications. The following major topics are discussed:

. Overview of the DBFNDX RDD

. Installing DBFNDX Driver Files

. Linking the DBFNDX Driver

. Using the DBFNDX Driver

. Compatibility with dBASE III PLUS

Overview of the DBFNDX RDD

The DBFNDX database driver allows creation, access, and updating of dBASE III and dBASE III PLUS compatible index (.ndx) files. Index files (.ndx) created with Clipper are exactly the same as those created by dBASE III PLUS. All operations that can be performed on standard Clipper index (.ntx) files can be performed on (.ndx) files using the DBFNDX database driver.

In a network environment, the DBFNDX driver supports the Clipper file and record locking scheme. The multiuser behavior is the same as the default DBFNTX driver. This means that the DBFNDX database driver supports concurrent access to (.ndx) files between Clipper applications only. Concurrent access to (.ndx) files between dBASE III PLUS and Clipper programs is not supported.

Important! Updating database (.dbf) and index (.ndx) files shared between dBASE III PLUS and Clipper programs may corrupt the (.dbf) and any of its associated (.ndx) files.

Installing DBFNDX Driver Files

The DBFNDX database driver is supplied as the file, DBFNDX.LIB.

The Clipper installation program installs this driver in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Linking the DBFNDX Database Driver

To link the DBFNDX database driver into an application program, you must specify DBFNDX.LIB to the linker in addition to your application object files (.OBJ).

1. To link with .RTLink using positional syntax:

C>RTLINK <appObjectList> ,,, DBFNDX

2. To link with .RTLink using freeformat syntax:

C>RTLINK FI <appObjectList> LIB DBFNDX

Note: These link commands all assume the LIB, OBJ, and PLL environment variables are set to the standard locations. They also assume that the Clipper programs were compiled without the /R option.

Using the DBFNDX Database Driver

To use (.ndx) files in a Clipper program:

1. Place a REQUEST DBFNDX at the beginning of your application or at the top of the first program file (.prg) that opens a database file using the DBFNDX driver.

2. Specify the VIA “DBFNDX” clause if you open the database file with the USE command.

-OR-

3. Specify “DBFNDX” for the <cDriver> argument if you open the database file with the DBUSEAREA() function.

-OR-

4. Use RDDSETDEFAULT(“DBFNDX”) to set the default driver to DBFNDX.

Except in the case of REQUEST, the RDD name must be a literal character string or a variable. In all cases it is important that the driver name be spelled correctly.

The following program fragments illustrate:

REQUEST DBFNDX 
. 
. 
. 
USE Customers INDEX Name, Address NEW VIA "DBFNDX"

-OR-

REQUEST DBFNDX RDDSETDEFAULT( "DBFNDX" ) 
.
.
. 
USE Customers INDEX Name, Address NEW

Using (.ntx) and (.ndx) Files Concurrently

You can use (.ndx) and (.ntx) files concurrently in a Clipper program like this:

REQUEST DBFNDX

// (.ntx) file using default DBFNTX driver USE File1 INDEX File1 NEW

// (.ndx) files using DBFNDX driver USE File2 VIA “DBFNDX” INDEX File2 NEW

Note, however, that you cannot use (.ndx) and (.ntx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNDX" INDEX File1.ntx, File2.ndx

Compatibility with dBASE III PLUS

When accessing dBASE III PLUS (.ndx) files, there are several compatibility issues of which you must be aware. These issues are discussed below.

Supported Data Types

The DBFNDX database driver supports the following data types for key expressions:

. Character

. Numeric

. Date

This is consistent with dBASE III PLUS.

The DBFNDX database driver does not support indexing with logical key expressions as does the default DBFNTX database driver. This is actually a dBASE III PLUS limitation and is not supported by the DBFNDX driver in order to enforce compatibility with dBASE III PLUS.

To work around this limitation, index logical values by converting them to character values like this:

INDEX ON IIF(<lExp>, "T", "F") TO <logicalIndex>

Supported Key Expressions

When you create (.ndx) files using the DBFNDX driver, you must use only Clipper or user-defined functions compatible with dBASE III PLUS. Use of the other functions will render the (.ndx) file unreadable in dBASE III PLUS.

FIND vs SEEK

In Clipper, you can use the FIND command only to locate keys in indexes where the index key expression is character type. This differs from dBASE III PLUS where FIND supports character and numeric key values.

Note: In Clipper programs, always use the SEEK command or the DBSEEK() function to search an index for a key value.

The DBFNDX driver lets you recover from a data type error raised during a FIND or SEEK. However, since Error:canDefault, Error:canRetry, or Error:canSubstitute are set to false (.F.), you should use BEGIN SEQUENCE…END to handle a SEEK or FIND data type error. Within the error block for the current operation, issue a BREAK() using the error object the DBFNDX database driver generates, like this:

bOld := ERRORBLOCK({|oError| BREAK(oError)})
.
.
 .
BEGIN SEQUENCE
SEEK xVar
RECOVER USING oError
// Recovery code
END
.
.
 .
ERRORBLOCK(bOld)

There is an extensive discussion of the effective use of the Clipper error system in the Error Handling Strategies chapter of the Programming and Utilities guide.

Sharing Data on a Network

As mentioned above, the DBFNDX driver does not support dBASE III PLUS file and record locking schemes. Instead, the DBFNDX driver supports the DBFNTX file and record locking scheme. This means that if the same database and index files are open in Clipper and dBASE III PLUS, Clipper program locks are not visible to dBASE III PLUS and vice versa.

Warning! Database integrity is not guaranteed and index corruption will occur if Clipper and dBASE III PLUS programs attempt to write to a database or index file at the same time. For this reason, concurrent use of the same database (.dbf) and index (.ndx) files by dBASE III PLUS and Clipper programs is strongly discouraged and not supported.

Compatibility with dBASE IV

Specific compatibility with dBASE IV is provided through the DBFMDX driver. It includes (.dbf), (.mdx), and (.dbt) file format compatibility and is described in detail in the previous chapter.

Summary

In this chapter, you were given an overview of the features and benefits of the DBFNDX RDD. You learned how to link this driver and how to use it in your applications, and were given an overview of the compatibility issues.

C5DG-4 DBFCDX Driver

Clipper 5.x – Drivers Guide

Chapter 4

DBFCDX Driver Installation and Usage

DBFCDX is the FoxPro 2 compatible RDD for Clipper. As such, it connects to the low-level database management subsystem in the Clipper architecture. When you use the DBFCDX RDD, you add a number of new features including:

. FoxPro 2 file format compatibility

. Compact indexes

. Compound indexes

. Conditional indexes

. Memo files smaller than DBFNTX format

In This Chapter

This chapter explains how to install DBFCDX and how to use it in your applications. The following major topics are discussed:

. Overview of the DBFCDX RDD

. Installing DBFCDX Driver Files

. Linking the DBFCDX Driver

. Using the DBFCDX Driver

Overview of the DBFCDX RDD

The DBFCDX driver lets you create and maintain (.cdx) and (.idx) files with features different from those supplied with the original DBFNTX driver and is compatible with files created under FoxPro 2. The new features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically, you can:

. Create indexes smaller than those created with the DBFNTX
driver. The key data is stored in a compressed format that
substantially reduces the size of the index file.

. Create a compound index file that contains multiple indexes
(TAGs), making it possible to open several indexes under one file
handle. A single (.cdx) file may contain up to 99 index keys.

. Create conditional indexes (FOR / WHILE / REST / NEXT).

. Create files with FoxPro 2 file format compatibility.

Compact Indexes

Like FoxPro 2, The DBFCDX driver creates compact indexes. This means that the key data is stored in a compressed format, resulting in a substantial size reduction in the index file. Compact indexes store only the actual data for the index keys. Trailing blanks and duplicate bytes between keys are stored in one or two bytes. This allows considerable space savings in indexes with much empty space and similar keys. Since the amount of compression is dependent on many variables, including the number of unique keys in an index, the exact amount of compression is impossible to predetermine.

Compound Indexes

A compound index is an index file that contains multiple indexes (called tags). Compound indexes (.cdx)’s make several indexes available to your application while only using one file handle. Therefore, you can overcome the Clipper index file limit of 15. A compound index can have as many as 99 tags, but the practical limit is around 50. Once you open a compound index, all the tags in the file are automatically updated as the records are changed.

Once you open a compound index, all the tags contained in the file are automatically updated as the records are changed. A tag in a compound index is essentially identical to an individual index (.idx) and supports all the same features. The first tag (in order of creation) in the compound index is, by default, the controlling index.

Conditional Indexes

The DBFCDX driver can create indexes with a built-in FOR clause. These are conditional indexes in which the condition can be any expression, including a user-defined function. As the database is updated, only records that match the index condition are added to the index, and records that satisfied the condition before, but don’t any longer, are automatically removed.

Expanded control over conditional indexing is supported with the revised INDEX and REINDEX command options as in the new DBFNTX driver.

Installing DBFCDX Driver Files

The DBFCDX driver is supplied as the file, DBFCDX.LIB.

The Clipper installation program installs this driver in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Linking the DBFCDX Database Driver

To link the DBFCDX database driver into an application program, you must specify DBFCDX.LIB to the linker in addition to your application object files (.OBJ).

1. To link with .RTLink using positional syntax:

C>RTLINK <appObjectList> ,,,DBFCDX

2. To link with .RTLink using freeformat syntax:

C>RTLINK FI <appObjectList> LIB DBFCDX

Note: These link commands all assume the LIB, OBJ, and PLL environment variables are set to the standard locations. They also assume that the Clipper programs were compiled without the /R option.

Using the DBFCDX Database Driver

To use FoxPro 2 files in a Clipper program:

1. Place REQUEST DBFCDX at the beginning of your application or at the top of the first program file       (.prg) that opens a database file using the DBFCDX driver.

2. Specify the VIA “DBFCDX” clause if you open the database file with the USE command.

    -OR-

3. Specify “DBFCDX” for the <cDriver> argument if you open the database file with the DBUSEAREA()       function.

   -OR-

4. Use ( “DBFCDX” ) to set the default driver to DBFCDX.

    Except in the case of REQUEST, the RDD name must be a literal character string or a variable. In all       cases it is important that the driver name be spelled correctly.

The following program fragments illustrate:

  REQUEST DBFCDX
  .
  .
  .
  USE Customers INDEX Name, Address NEW VIA "DBFCDX"

     -OR-

  REQUEST DBFCDX
  RDDSETDEFAULT( "DBFCDX" ) .
  . 
  .
  USE Customers INDEX Name, Address NEW

Using (.idx) and (.ntx) Files Concurrently

You can use both (.idx) and (.ntx) files concurrently in a Clipper program like this:

// (.ntx) file using default DBFNTX driver
 USE File1 INDEX File1 NEW
// (.idx) files using DBFCDX driver
 USE File2 VIA "DBFCDX" INDEX File2 NEW

Note, however, that you cannot use (.idx) and (.ntx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNTX" INDEX File1.ntx, File2.idx

Using (.cdx) and (.idx) Files Concurrently

You may use (.cdx) with (.idx) files concurrently (even in the same work area); however, in most cases it is easier to use a single (.cdx) index for each database file or separate (.idx) files. When using both types of index at the same time, attempting to select an Order based on its Order Number can be confusing and will become difficult to maintain.

File Maintenance under DBFCDX

When an existing tag in a compound index (.cdx) is rebuilt using INDEX ON…TAG… the space used by the original tag is not automatically reclaimed. Instead, the new tag is added to the end of the file, increasing file size.

You can use the REINDEX command to “pack” the index file. REINDEX rebuilds each tag, eliminating any unused space in the file.

If you rebuild your indexes on a regular basis, you should either delete your (.cdx) files before rebuilding the tags or use the REINDEX command to rebuild them instead.

DBFCDX and Memo Files

The DBFCDX driver uses FoxPro compatible memo (.fpt) files to store data for memo fields. These memo files have a default block size of 64 bytes rather than the 512 byte default for (.dbt) files.

DBFCDX memo files can store any type of data. While (.dbt) files use an end of file marker (ASCII 26) at the end of a memo entry, (.fpt) files store the length of the entry. This not only eliminates the problems normally encountered with storing binary data in a memo field but also speeds up memo field access since the data need not be scanned to determine the length.

Tips For Using DBFCDX

1. Make sure index extensions aren’t hard-coded in your application. The default extension for DBFCDX indexes is (.idx), not (.ntx). You can still use (.ntx) as the extension as long as you specify the extension when you create your indexes. The best way to determine index extensions in an application is to call ORDBAGEXT().

For example, if you currently use the following code to determine the existence of an index file:

IF .NOT. FILE("index.ntx")
    INDEX ON field TO index
ENDIF

Change the code to include the INDEXEXT() function, as follows:

IF .NOT. FILE("index"+ORDBAGEXT())
   INDEX ON field TO index
ENDIF

2. If your application uses memo fields, you should convert your (.dbt) files to (.fpt) files.

There are some good reasons for using (.fpt) files. Most important is the smaller block size (64 bytes). Clipper’s (.dbt) files use a fixed block size of 512 bytes which means that every time you store even 1 byte in a memo field Clipper uses 512 bytes to store it. If the data in a memo field grows to 513 bytes, then two blocks are required.

When creating (.fpt) files, the block size is set at 64 bytes to optimize it for your needs. A simple conversion from (.dbt) files to (.fpt) files will generally shrink your memo files by approximately 30%.

3. Add DBFCDX.LIB as a library to your link command or link script.

Summary

In this chapter, you were given an overview of the features and benefits of the DBFCDX RDD. You also learned how to link this driver and how to use it in your applications.

Tools — Database Functions

Introduction Database Functions
DBFDSKSIZE() Determines the size of a selected (.dbf) file on a disk drive
DBFSIZE()    Determines the size of a selected (.dbf) file in memory
FIELDDECI()  Determines the number of decimal places in a field
FIELDNUM()   Determines field number for a specific field in a database file
FIELDSIZE()  Determines the size of a field
FIELDTYPE()  Determines the data type for a field
ISDBT()      Determines if a memo file (.dbt) is present

Copy Protection

How I can protect my application against copying?

May be exist many ways. One of theme is using HD serial number.

1. Read volume serial number of hard disk(s) of your customer. ( Apply this procedure while installation, once only one and preferably manually.  Of course you can obtain this info by online accessing to system of your customer. )

2. Record this info ( after crypt ) anywhere you like in the customer system: in a .MEM file, a .dbf / dbt, in registry, even directly into your application as a literal constant.

3. Your application will begin to run by reading this recorded data and compare it with actual HD serial number of customer; if match continue working, else …

Here an example to get  HD serial number.

Format of xBase Files

xBase File Format Description

xBase File Types

xBase File  Types and Extensions

Ext. File Type  Introduced or used by
.$$$ temporary file  dBASE III
.$db temporary file  dBASE IV
.act FoxDoc Action Diagrams  FoxPro
.app application object file  dBASE Appl. Generator
.app generated application  FoxPro
.bak Backup file  dBASE
.bar horizontal bar menu object file  dBASE Appl. Generator
.bin Binary files  dBASE
.bch batch process object file  dBASE Appl. Generator
.cac executable when caching on/off  dBASE IV
.cat catalog  dBASE III, IV
.cdx compound index  FoxPro
.ch header file  Clipper
.cht interface file for ChartMaster  dBASE
.clp compiler script file (clip list)  Clipper
.cmd command  dBASE – Waffle
.cod template source file  dBASE Appl. Generator
.cpt encrypted memo file  dBASE
.crp encrypted database  dBASE IV
.ctl control file  dBASE IV – Aldus Setup
.cvt backup file for CONVERTed database file  dBASE IV
.db configuration  dBASE
.db$ temporary file  dBASE
.db2 database  dBASE II
.db3 database  dBASE III
.dbf database file  dBASE – FoxPro
.dbk database backup  dBASE IV
.dbo compiled program  dBASE IV
.dbt FoxBASE+ style memo  FoxPro
.dbt memo file for database w/same name  dBASE – Clipper
.def Definitions list  dBASE
.dif Data Interchange Format. For APPEND FROM, COPY  dBASE – VisiCal
.doc Documentation text file  dBASE Appl. Generator
.fil files list object file  dBASE Appl. Generator
.fky macro file  FoxPro
.fmo compiled format file  dBASE IV
.fmt format file  dBASE – FoxPro – Clipper
.fp configuration file  FoxPro
.fpc catalog  FoxPro
.fpt memo  FoxPro
.fr3 renamed dBASE III+ form file  dBASE IV
.frg uncompiled report file, code fragment file  dBASE IV
.frm report file  dBASE – Clipper
.fro compiled report file  dBASE IV
.frt report memo  FoxPro
.frx report  FoxPro
.fw2 Framework spreadsheet or database file  Framework – dBASE
.fxp compiled format  FoxPro
.gen compiled template  dBASE Appl. Generator
.idx index (many)  FoxPro
.ind include index  dBASE IV
.inx index  FoxBase
.key Key macro library  dBASE
.lbg label generator data  dBASE IV
.lbl label  dBASE – Clipper
.lbo compiled label  dBASE IV
.lbt label memo  FoxPro
.lbx label  FoxPro
.ld1 overlay file  dBASE
.log Transaction log file  dBASE
.mbk multiple index file backup  dBASE IV
.mdx multiple index file  dBASE IV
.mem memory variable save file  dBASE – FoxPro
.mnt menu memo  FoxPro
.mnx menu  FoxPro
.mpr generated program  FoxPro
.mpx compiled menu program  FoxPro
.ndx index file  dBASE
.npi source for DGEN.EXE interpreter  dBASE Appl. Generator
.ntx index file  Clipper
.pjt project memo  FoxPro
.pjx project  FoxPro
.plb library  FoxPro
.pll pre-linked library  Clipper
.plt pre-linked transfer file  Clipper
.pop pop-up menu object  dBASE Appl. Generator
.ppo pre-processor output  Clipper
.pr2 printer driver  dBASE IV
.pr3 PostScript printer driver  dBASE IV
.prf printer driver  dBASE IV
.prg program source file  dBASE – FoxPro – Clipper
.prs procedure  dBASE IV
.prt Print dump  dBASE
.prx compiled program  FoxPro
.qbe saved query (Query By Example)  dBASE IV – Quattro Pro
.qbo compiled query  dBASE IV
.qpr generated query program  FoxPro
.qpx compiled query program  FoxPro
.qry query  dBASE IV
.res dBASE resources  dBASE IV
.rpd Rapid file. For IMPORT/EXPORT,APPEND  FROM, COPY  dBASE
.sc3 renamed dBASE III screen mask file  dBASE IV
.scr screen – screen snapshot  dBASE IV
.sct screen memo  FoxPro
.scx screen  FoxPro
.spr generated screen program  FoxPro
.spx compiled screen program  FoxPro
.str structure list object file  dBASE Appl. Generator
.t44 Temporary file for Sort or Index  dBASE IV
.tbk memo backup  dBASE IV – FoxPro
.tvf table view settings  dBASE
.upd update data  dBASE
.upo compiled update data  dBASE
.val values list object file  dBASE Appl. Generator
.vew view file  Clipper, Lotus Approach
.vue view  dBASE IV – FoxPro
.w44 temporary file for Sort or Index  dBASE
.wfm form object  dBASE Form Designer
.win window file  FoxPro – dBASE

Source :  http://www.clicketyclick.dk/databases/xbase/format/index.html

Notes:

–          Such a list can’t be perfect. Some item may be obsolete / forgotten and something may not exist when this list compiled.

–          “Clipper” may not include all versions of Clipper.

–          Most of Clipper files are supported by Harbour.