Index fundemantals

Question : I just would like to understand a bit what is the difference between DBFNSX and SIXCDX, please. Are not  they the same thing?

Answer : I described it few times in the past.

Lets forget about SIXCDX – it’s slighlt modified DBFCDX not worth to talk about.

DBT, FMT, SPT are different MEMO formats. All of them are supported by Harbour and automatically recognized when DBF file is open. NTX, CDX and NSX are different index formats. They can be used in any combinations of MEMO formats, i.e. DBFCDX perfectly well works with DBT memo files just like with FPT and SMT ones.

In Harbour all core RDDs using above index formats (DBFNSX, DBFCDX, DBFNSX) have nearly the same functionality which cover nearly all index feautres known in xbase world and many of them are unique to [x]Harbour only so they are not supported by other drivers.

With all above RDDs user can use all ord*(), db*(), sx_*(), hsx_*() , … functions, can create multitag indexes (many orders in single file, also for NTX format), autooreder, autoopen, production indexes, etc. so for programmer used RDD should not create any difference. It’s also possible to enable disable some features using RDDI_* interface, i.e. this code change default DBFNTX behavior so it behaves just like DBFCDX and even uses “.cdx” as default file extnesion (of course internally it’s still NTX format with Harbour extenssions – we support CTX format from CLIP)

 // default index extenssion 
 rddInfo( RDDI_ORDBAGEXT, ".cdx", "DBFNTX" ) 
 //support multi tag in single index file 
 rddInfo( RDDI_MULTITAG , .t. , "DBFNTX" ) 
 // structural indexes support 
 rddInfo( RDDI_STRUCTORD, .t. , "DBFNTX" ) 
 // record number is hidden trailing part of key duirng sorted 
 rddInfo( RDDI_SORTRECNO, .t. , "DBFNTX" )

There are only few minor excpetions rather unimportant for most of users unique to given RDDs. The most important are two:

1. I implemented dynamic unique indexes only in DBFCDX It means that ordUnique( ,, .t. ) -> can enable/disable unique mode only in DBFCDX (and SIXCDX)

2. Only in DBFNTX and DBFNSX I implement special mode which allows to use page numbers instead of file offsets in index pages. In this mode indexes are not binary compatible with other languages but their maximal size has been greatly extended and for NTX and NSX files is 2^32 * index_page_size what gives 2^42 for default 1024 pages in this formats – 2^42 is 4TB I haven’t implemented it in DBFCDX so far and for this format maximum index size is still 4GB. Maybe in some spare time I’ll do that and also add support for different page sizes. BTW in ADS .adi indexes are slightly modified CDX files where page numbers are used instead of offsets and index page size can be changed.

Of course there are very serious differences in low level implementation and structures used by these formats.

NTX is simple BTREE without any compression. The operation are extremely fast but indexes are much bigger then in CDX or NSX format so performance is usually storngly reduced by cost of IO operations. Anyhow theoretically having very strong server application with a lot of RAM so all data are accessed from memory not from harddisks this is the best choice.

CDX and NSX compress leaf nodes so total size of index files is much smaller then in NTX format. NSX uses simple BTREE when CDX uses three of the most significant keys. It means that update in CDX files can be more expensive then in NSX format especially if we are adding keys which should be sorted as last (it’s the most common situation, i.e. when we are adding records with current date) because all nodes from leaf to root have to be updated. All keys are repeated in leaf nodes and there are internal bindings for nodes on the same level (BTW some RDDs like SIX3 do not update them correctly for interior nodes) so CDX format is also a little bit redundant. Anyhow all keys are repeated in leafs nodes and we have internal bindings between all leaves so skipping can be a little bit faster, etc.

Page size in CDX is smaller then in NSX so for very long keys, i.e. over 100 bytes NSX format should be much more efficient. It also uses different compression method which should be better for keys having long space substrings inside, i.e. due to concatenation of few longer fields like: FNAME[40] + LNAME[40]

In general this is to big subject to describe it here in few words.

In include/hbrddnsx.h I’ve made some small description of NSX format when I was implementing it.

best regards,
Przemek
Source : https://groups.google.com/d/msg/harbour-devel/9nT9lZmtztk/Q3X-s81UpYYJ

Harbour File Size Limits

 

Harbour File Size Limits

  • Max record size: 2^16-1 = 65535 byts ( 64 MB )
  • Max number of recors : 2^32-1 = 4,294,967,295 ( 4 Bilion )
  • Max .dbf file size : 2^48 = 256 TB
  • Max DBT memo file size : 2 TB
  • Max FPT memo file size : 256 GB
  • Max SMT memo file size : 128 GB
  • Max NTX file size (standard) : 4GB
  • Max NTX file size (incresead ) : 4TB
  • Max CDX file size : 4GB

 

Source : Harbour\doc\xhb-diff.txt :

### NATIVE RDDs ###

In both compilers maximal file size for tables, memos and indexes is limited only by OS and file format structures. Neither Harbour nor xHarbour introduce own limits here.

The maximal file size for DBFs is limited by number of records 2^32-1 = 4294967295 and maximal record size: 2^16-1 = 65535 what gives nearly 2^48 = 256TB as maximal .dbf file size.

The maximal memo format size depends on used memo type: DBT, FPT or SMT and size of  memo block. It’s limited by maximal number of memo blocks = 2^32 and size of memo block so it’s 2^32*<size_of_memo_block>.

The default memo block size for DBT is 512 bytes, FPT – 64 bytes and for SMT 32 bytes. So for standard memo block sizes the maximum are:

DBT->2TB, FPT->256GB, SMT->128GB. The maximal memo block size in Harbour is 2^32 and minimal is 1 byte and it can be any value between 1 and 65536 and then any number of 64KB blocks. The last limitation is introduced as workaround for some wrongly implemented in other
languages memo drivers which were setting only 16 bits in 32bit field in memo header. Most of other languages has limit for memo block size at 2^15 and the block size has to be power of 2. Some of them also introduce minimal block size limits. If programmers plans to share data with programs compiled by such languages then he should check their documentation to not create memo files which cannot be accessed by them.

Maximal NTX file size for standard NTX files is 4GB and it’s limited by internal NTX structures. Enabling 64bit locking in [x]Harbour change slightly used NTX format and increase maximum NTX file size to 4TB.

The NTX format in [x]Harbour has also many other extensions like support for multitag indexes or using record number as hidden part of index key and many others which are unique to [x]Harbour. In practice all of CDX extensions are supported by NTX in [x]Harbour.

The NSX format in [x]Harbour is also limited by default to 4GB but like in NTX enabling 64bit locking extend it to 4TB. It also supports common to NTX and CDX set of features.

The CDX format is limited to 4GB and so far [x]Harbour does not support extended mode which can increase the size up to 2TB with standard page length and it can be bigger in all formats if we introduce support for bigger index pages. Of course all such extended formats are not binary compatible with original ones and so far can be used only by [x]Harbour RDDs though in ADS the .adi format is such extended CDX format so maybe in the future it will be possible to use .adi indexes in our CDX RDD.

Of course all of the above sizes can be reduced by operating system (OS) or file system (FS) limitations so it’s necessary to check what is supported by environment where [x]Harbour applications are executed. 

C5DG-2 RDD Architecture

Clipper 5.x – Drivers Guide

Chapter 2

Replaceable Database Driver Architecture

Clipper supports a driver architecture that allows Clipper applications to use Replaceable Database Drivers (RDDs). The RDD system makes Clipper applications data-format independent. Such applications can, therefore, access the data formats of other database systems, including the dBASE IV (.mdx), FoxPro (.cdx), and Paradox (.db) formats on a variety of equipment. This driver architecture can even support database drivers that are not file-based, although all of the drivers supplied with Clipper 5.x are file-based.

The concept of replaceable drivers is not new to this version of Clipper. In previous versions, the use of the default database driver (DBFNTX.LIB) was hidden by the fact that it was automatically linked into your application. In fact, this is still the case. The DBFNTX driver has been replaceable since it was first introduced in version 5.0. Before this version, the DBFNTX driver was the only RDD supplied as part of the system.

In This Chapter

With the introduction of the new RDDs, Clipper provides many new and enhanced commands and functions that access and manipulate databases. These language elements can enable your applications to access data regardless of the RDD under which it is ordered. There are also commands and functions that give you specific information about the RDDs in use.

The Language Implementation section of this chapter includes tables that summarize these new and enhanced language elements. This chapter also covers basic terminology, implementation principals, and general concepts of the Order Management System.

The following major topics are discussed:

. RDD Basics

. Basic Terminology

. The Language Implementation

. Order Management System

RDD Basics

The cornerstone of the replaceable database driver system is the Clipper work area. All Clipper database commands and functions operate in a work area through a database driver that actually performs the access to the stored database information. The layering of the system looks like this:

                      +———————————+

                      | Database Commands and Functions |
                      ----------------------------------|
                      |          RDD Interface          |
                      |---------------------------------|
                      |         Database driver         |
                      |---------------------------------|
                      |           Stored Data           |
                      +---------------------------------+

 In this system, each work area is associated with a single database driver. Each database driver, in turn, is supplied as a separate library file (.LIB) you link into your application programs. Within an application, you specify the name of the database driver when you open or access a database file or table with the USE command or DBUSEAREA() function. If you specify no database driver at the time a file is opened, the default driver is used. You may select which driver will be used as the default driver.

Once you open a database in a work area, the RDD used for that work area is automatically used for all operations on that database (except commands and functions that create a new table). Any command or function that creates a new table (i.e., SORT, CREATE FROM, DBCREATE(), etc.) uses the default RDD. Most of the new commands and functions let you specify a driver other than the default driver.

The normal default database driver, DBFNTX (which supports the traditional (.dbf), (.ntx), and (.dbt) files) is installed into your \CLIPPER5\LIB directory. This driver is linked into each program automatically to provide backwards compatibility.

To use any of the other supplied drivers, either as an additional driver or an alternate driver, you must use the REQUEST command to assure that the driver will be linked in. You must also include the appropriate library on the link line.

All Clipper applications will automatically include code generated by RDDSYS.PRG from the \CLIPPER5\SOURCE\SYS subdirectory. If you wish to automatically load another RDD, you must modify and compile RDDSYS.PRG and link the resulting object file into your application. The content of the default RDDSYS.PRG is shown below. Only the portion in bold should be modified

 
     //  Current RDDSYS.PRG
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit
        REQUEST DBFNTX                   // Force link for DBFNTX RDD
        RDDSETDEFAULT( "DBFNTX" )        // Set up DBFNTX as default
                                         // driver

        RETURN

     // eof: rddsys.prg

To change the default to a new automatically-loading driver, modify the bold lines in RDDSYS.PRG to include the name of the new driver. For example:

     //  Revised RDDSYS.PRG
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit
        REQUEST DBFCDX                   // Force link for DBFCDX RDD
        RDDSETDEFAULT( "DBFCDX" )        // Set up DBFCDX as default
                                         // driver

        RETURN

     // eof: rddsys.prg

If you change this file, all Clipper applications in which it is linked will automatically include the new RDD.

To use any RDD other than the default, you must explicitly identify it through use of the VIA clause of the USE command.

You need not disable the automatic DBFNTX loading to use other RDDs in your applications, but if your application will not use any DBFNTX functionality, you can save its code overhead by disabling it.

To completely disable the automatic loading of a default RDD, remove the two lines shown above in bold. For example:

     //  New Revised RDDSYS.PRG
     //  disables auto-loading
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit

        RETURN
     // eof: rddsys.prg

Basic Terminology

The RDD architecture introduces several new terms and concepts that are key to the design and usage of RDDs. You should familiarize yourself with these concepts and terms as you begin to use the RDD functionality. The meaning of some earlier terminology is also further defined. The following RDD functional glossary defines the terminology for all RDDs.

. Key Expression : A valid Clipper expression that creates a key value from a single record.

. Key Value : A value that is based on value(s) contained within database fields, associated with a particular record in a database.

. Identity : A unique value guaranteed by the structure of the data file to reference a specific record in a database even if the record is empty. In the Xbase file (.dbf), the identity is the record number; but it could be the value of a unique primary key or even the offset of an array in memory.

. Keyed-Pair : A pair consisting of a key value and an identity.

. Identity Order : Describes a database arranged by identity. In Xbase, this refers to the physical arrangement of the records in the database in the order in which they were entered (natural order).

. Tag : A set of keyed-pairs that provides ordered access to the table based on a key value. Usually, an Order in a multiple-Order index (Order). An Order.

. Order : A named mechanism (index) that provides logical access to a database according to the keyed-pairs. This term encompasses both single indexes and the Tags in multiple-Tag indexes.

Orders are not, themselves, data files. They provide access to data that gives the appearance of an ordering of the data in a specific way. This ordering is defined by the relationships between keyed- pairs. An Order does not change the physical (the natural or entry) order of data in a database.

. Controlling Order : The active Order (index) for a particular work area. Only one Order may control a work area at any time, and it controls the order in which the database is accessed during paging and searching.

. Order List : A list of all the Orders available to the database in the specified work area.

. Order Bag : A container that holds zero or more Orders. Normally a disk or memory file. A traditional index like (.ntx) is an Order Bag that holds only one Order. A multiple-Tag index (.mdx or .cdx) is an Order Bag that holds zero or more Orders. Though Order Bags may be a memory or disk file, Clipper 5.x only supports Order Bags as disk files.

. Record : A record in the traditional database paradigm is a row of one or more related columns (fields) of data. In the expanded architecture of Clipper, a record could be data that does not exactly fit this definition.

A record is, in this expanded context, data associated with a single identity. In an Xbase data structure, this corresponds to a row (fields associated with a record number); in other data structures, this may not be the case.

In this document we use “record” in the traditional sense, but you should be aware that Clipper permits expansion of the meaning of record.

. single-Order Bag : An Order Bag that can contain only one Order. The (.ntx) and (.ndx) files are examples of single-Order Bags.

. multiple-Order Bag : An Order Bag that can contain any number of Orders; a multiple-Tag index. The (.cdx) and (.mdx) files are examples of multiple-Order Bags.

. maintainable scoped Orders : Scoped (filtered) Orders created using the FOR clause. The FOR condition is stored in the index header. Orders of this type are correctly updated using the expression to reflect record updates, deletions and additions.

. non-maintainable/temporary Orders : Orders created using the WHILE or NEXT clauses. These Orders are useful because they can be created quickly. However, the conditions in these clauses are not stored in the index header. Therefore, Orders of this type are not correctly updated to reflect record updates, deletions and additions. They are only for temporary use.

. Lock List : A list of the records that are currently locked in the work area.

The Language Implementation

To support the RDD architecture and let you design applications that are independent of the data format you are using, many existing Clipper commands and functions have been enhanced, and several new language elements have been added. The following tables summarize these changes and additions. See the Reference chapter of this guide for more detailed information on a particular item.

     Enhanced Commands and Functions
     ------------------------------------------------------------------------
     Command/Function  Changes
     ------------------------------------------------------------------------
     APPEND FROM       VIA clause
     COPY TO           VIA clause
     DBAPPEND()        Terminology
     GO                Terminology
     DBAPPEND()        Terminology
     INDEX             ALL, EVAL, EVERY, NEXT, RECORD, REST, TAG, and
                       UNIQUE clauses
     SEEK              SOFTSEEK option
     SET INDEX         ADDITIVE clause
     SET ORDER         IN, TAG clauses
     DBSETINDEX()      Terminology
     RECNO()           Terminology
     ------------------------------------------------------------------------

     New Commands and Functions
     ------------------------------------------------------------------------
     Command/Function    Description
     ------------------------------------------------------------------------
     DELETE TAG          Delete a Tag (Order)
     DBGOTO()            Position record pointer to a specific identity
     DBRLOCK()           Lock the record at the current or specified identity
     DBRLOCKLIST()       Return an array of the currently locked records
     DBRUNLOCK           Release all or specified record locks
     ORDBAGEXT()         Return the Order Bag file extension
     ORDBAGNAME()        Return the Order Bag name of a specific Order
     ORDCREATE()         Create an Order in an Order Bag
     ORDDESTROY()        Remove a specified Order from an Order Bag
     ORDFOR()            Return the FOR expression of an Order
     ORDKEY()            Return the Key expression of an Order
     ORDLISTADD()        Add Order Bag contents or single Order to the Order
                         List
     ORDLISTCLEAR()      Clear the current Order List
     ORDLISTREBUILD()    Rebuild all Orders in the Order List of the current
                         work area
     ORDNAME()           Return the name of an Order in the work area
     ORDNUMBER()         Return the position of an Order in the current Order
                         List
     ORDSETFOCUS()       Set focus to an Order in an Order List
     RDDLIST()           Return an array of the available Replaceable
                         Database Drivers
     RDDNAME()           Return the name of the RDD active in the current or
                         specified work area
     RDDSETDEFAULT()     Set or return the default RDD for the application
     ------------------------------------------------------------------------

User Interface Levels

We want to make it easy for you to quickly take advantage of the added functionality provided in Clipper 5.x. In order to effectively use the RDDs, you should read the following discussions. They are provided as a means of identifying the degree of programming knowledge or Clipper experience that will let you effectively use the RDD features.

For this purpose the RDD feature set is arbitrarily divided into levels A and B. Tables listing the commands or functions that comprise these access levels are also supplied. In addition, an RDD Features Summary is provided in table form which outlines the features available in each driver. The commands and functions in both of these levels of access are described in the Reference chapter of this guide.

Level A – Command-Level Interface

Level A. a simple command-level interface very similar to those found in other languages (e.g., dBASE IV, FoxPro). This is the primary access for new Clipper users who may or may not be familiar with other languages.

The following table lists the commands and functions accessible by the Clipper programmer with background in languages such as dBASE or FoxPro. The commands and functions in this table provide access to the additional features without requiring an advanced knowledge of Clipper or other programming concepts.

     Basic Commands and Functions
     ------------------------------------------------------------------------
     Command/Function  Changes
     ------------------------------------------------------------------------
     DELETE TAG        Delete a Tag
     GOTO              Move the pointer to the specified identity
     INDEX             Create an index file
     SEEK              Search an Order for a specified key value
     SET INDEX         Open one or more Order Bags in the current work area
     SET ORDER         Select the controlling Order
     DBAPPEND()        Append a new record to the current Lock List
     DBRLOCK()         Lock the record at the current or specified identity
     DBRLOCKLIST()     Return an array of the current Lock List
     DBRUNLOCK         Release all or specified record locks
     ------------------------------------------------------------------------

Level B – Function-Level Interface

Level B. Clipper also adds a function level interface that not only allows access to the enhanced functionality of the drivers, but permits the building of higher-level functions using these composing behaviors. This level is meant for more experienced Clipper users who need to take advantage of the full power of the driver and Order Management System.

The following table lists the DML and Order Management functions recommended to the intermediate to advanced Clipper programmer. These functions provide the greatest flexibility in accessing the extended features of these drivers

     Advanced Functions (including Order Management)
     ------------------------------------------------------------------------
     Command/Function    Description
     ------------------------------------------------------------------------
     DBAPPEND()          Append a new record to the current Lock List
     DBRLOCK()           Lock the record at the current or specified identity
     DBRLOCKLIST()       Return an array of the current Lock List
     DBRUNLOCK()         Release all or specified record locks
     ORDBAGEXT()         Return the default Order Bag RDD extension
     ORDBAGNAME()        Return the Order Bag name of a specific Order
     ORDCREATE()         Create an Order in an Order Bag
     ORDDESTROY()        Remove a specified Order from an Order Bag
     ORDFOR()            Return the FOR expression of an Order
     ORDKEY()            Return the Key expression of an Order
     ORDLISTADD()        Add Order Bag contents or single Order to the Order
                         List
     ORDLISTCLEAR()      Clear the current Order List
     ORDLISTREBUILD()    Rebuild all Orders in the Order List of the current
                         work area
     ORDNAME()           Return the name of an Order in the work area
     ORDNUMBER()         Return the position of an Order in the current Order
                         List
     ORDSETFOCUS()       Set focus to an Order in an Order List
     RDDLIST()           Return an array of the available Replaceable
                         Database Drivers
     RDDNAME()           Return the name of the RDD active in the current or
                         specified work area
     RDDSETDEFAULT()     Set or return the default RDD for the application
     ------------------------------------------------------------------------

RDD Features

The following decision table summarizes the availability of key features across RDDs. It lists the features available in each RDD so you can use it as an aid in correct RDD implementation and data access.

     RDD Features Summary
     ------------------------------------------------------------------------
     Item                                NTX   NDX   MDX   CDX  DBPX
     ------------------------------------------------------------------------
     Implicit record unlocking in        Yes   Yes   Yes   Yes  Yes
     single lock mode
     Multiple Record Locks               Yes   Yes   Yes   Yes  No
     Number of Concurrent Record Locks   *1    *1    *1    *1   1
     Order Management (Tag support)      Yes   Yes   Yes   Yes  No
     Orders (Tags) per Order Bag (File)  1     1     47    50   N/A
     Number of Order Bags (Files)        15    15    15    15   N/A
     per work area
     Conditional Indexes (FOR clause)    Yes   No    Yes   Yes  No
     Temporary (Partial) Indexes         Yes   No    No    Yes  No
     (WHILE, ... )
     Descending via DESCENDING clause    Yes   No    Yes   Yes  No
     Unique via the UNIQUE clause        Yes   Yes   Yes   Yes  No
     EVAL and EVERY clause support       Yes   No    No    Yes  No
     Production/Structural Indexes       No    No    Yes   Yes  No
     Maximum Key Expression length       256   256   220   255  N/A
     (bytes)
     Maximum FOR Condition length        256   N/A   261   255  N/A
     (bytes)
     ------------------------------------------------------------------------

     *1 determined by available memory.

Clipper 5.x Order Management

Clipper includes a new Order Management System which provides a more effective and flexible way of indexing data. The main objective of the new Order Management implementation is to raise the Xbase indexing paradigm from a low level of abstraction (Xbase database specific) to a higher, more robust, level. This higher level of abstraction allows the user to build new commands and functions.

Low level abstraction refers to manipulation of discrete elements in the database architecture (i.e., field names and sizes, methods of handling controlling indexes, etc.).

High level abstraction refers to manipulation of general elements in a data source. It lets us, for example, set a controlling Order without explicitly addressing the character of the data file structure. This higher level of abstraction was achieved by reviewing all the processes that indexes have in common.

The Order Management function set was generically named (i.e. non-dbf specific) to provide a semantic that could encompass future RDD implementations that may not be file-bound. For example, an RDD could easily be created that orders (indexes) on a memory array, or other data structure, instead of a database. Therefore, all Order Management functions simply begin with ORD (for Order). You will find the function names to be self-explanatory (e.g., ORDCREATE() creates an Order, and ORDDESTROY() destroys an Order).

Concept

An Order is a set of keyed-pairs that provides a logical ordering of the records in an associated database file. Each key in an Order (index) is associated with a particular identity (record number) in the data set (database file). The records can be processed sequentially in key order, and any record can be located by performing a SEEK operation with the associated key value. An Order never physically changes the data that it’s applied against, but creates a different view of that data.

There are at least four basic types of processes that you can perform with an Order:

1. Ordering: Changes the sequence in which you view the data records.

2. Scoping: Constrains the visibility of data to specified upper and lower bounds. Determines the range of data items included, through a scoping rule, like the WHILE clause.

3. Filtration: Visibility of data is subject to conditional evaluation. Filtration determines which items of data are included, through a filter rule, like the FOR clause.

4. Translation: Values in underlying data source are translated (or converted) in some form based on a selection criteria. For example:

INDEX ON IIF(CUSTID > 1000, "NEW", "OLD")

The difference between scope and condition as it applies to FOR and WHILE is that the WHILE clause provides scope, but not filtering, but a FOR clause can provide both.

There are three primary elements in Order Management:

. Order: An Order is a set that has two elements in it: an Order Name, which is a logical name that can be referenced, and an Order Expression which supplies the view of the data. The Order Name provides logical access to the expression and the Order Expression provides a way of viewing the underlying data source. Data ordering can also be modified to ascending or descending sequence.

– Order Name: An Order Name is a symbolic name, that you use to manipulate an Order, like a file’s alias. The difference between an Order Name and the Order Number with which you would normally access indexes (Orders), is that the Order Name is stored in the index file. It is available each time you run the program, and is maintained by the system. The Order Number is generated each time the Order is added to an Order List and may change from one program execution to another. This makes Order Name the preferred means of referencing Orders.

– Order Expression: Is any valid Clipper expression. This is an index expression such as:

CUSTLIST->CUSTID

This expression produces the ordered view of the data. The values derived from this expression are sorted, and it is the relationship of these values to one another that provides the actual ordering.

. Order Number: An Order Number is provided by the Order List. An Order Number is only valid as long as the work area to which it belongs is open.

– Order Numbers provide one of the services performed by Order Names, allowing you to access a specific Order. In general, you should avoid accessing Orders by number.

– The ORDNUMBER() function returns the ordinal position of the specified <orderName> within the specified <orderList>.

. Order Bag: Unsorted collection of Orders. Each Order contains two elements (Order Name and Order Expression). Each Order Bag may have zero to n Orders. The maximum is determined by the RDD driver being used. Order Bags are similar to multiple-index files in that there’s no guarantee of any specific order within the container or Bag. Within an Order Bag you can access specific Orders by referencing a particular Order Name. Order Bags have persistence between activations of the program.

. Order List: An Order List orders the collection of Orders that are associated with and active in the current work area. It provides an access to the Orders active within a given work area. Each work area has an Order List, and there is only one Order List per work area. An Order List is created when a new work area is opened, and exists only as long as that work area is active. Once you close a work area, the Order List ceases to exist.

When you SET INDEX TO, the contents of the Order Bag are emptied into the Order List. At this point, the Orders in the Order List are active in the work area, where they will be updated as the data associated with the work area is modified. You may access an Order in the list by its Order Number or by its Order Name. You should access an Order by its name rather than a hard-coded ordinal position. You can make any Order in the Order List the controlling Order by giving it focus, as explained below.

. Order List Focus: Order List Focus is, essentially, a pointer to the Order that is used to change the view of the data. It is synonymous with controlling Order or controlling index, and defines the active index order. The SET ORDER TO command does not modify the Order List in any way. It does not clear the active indexes. It only changes the Order List Focus (the controlling order in the Order List).

Notes

The following list contains specific information regarding Order Bag usage and limitations with DBFNDX and DBFNTX index files:

. Single-Order Bags: With DBFNDX and DBFNTX you can explicitly assign the Order Name within the Order creation syntax. You can then use the Order Name in any command or function that accepts an Order Name (Tag) as a parameter.

. Single-Order Bag with INDEX ON: Single-Order Bags may retain the Order Name between activations. During creation, DBFNTX stores an optionally supplied Order Name in the file’s header for subsequent use. Therefore, the Order Name is not necessarily the same as that of the file. By contrast, DBFNDX cannot store an Order Name since this would prevent dBASE from accessing the file. By default DBFNDX Orders inherit the name of their index file.

Summary

This chapter has introduced you to the RDD concept, giving you specific information on the architecture that implements RDDs in Clipper. The basic terminology of RDDs has also been defined.

Finally, you have seen an overview of the language enhancements designed to make using RDDs straightforward and to let you build applications that do not depend on the RDD in use. The next chapter elaborates on these language enhancements, discussing syntax and usage in detail.

Introduction Harbour Extensions

Harbour is 100% Clipper-compatible and supports many language syntax extensions

including greatly extended run-time libraries such as OLE, Blat, OpenSSL, FreeImage,

GD, TIP, Tpathy, PCRE, HbZip (zlib and bzip2), cURL, Cairo, its own implementation of

CTools and NanFor libraries and many others. Harbour has an active development

community and extensive third party support.

 

Harbour includes various extensions of Clipper language; such as class creation,

support  for date constants, FOR EACH and SWITCH structures, many extensions to

RDD and language features and much more.

HbRun

HbRun is a console interpreter and program ( command file / script file / .prg / .hrb ) runner for the Harbour Language.

Addendum: a clarification by Przemek:

HBRUN is a simple wrapper to Harbour compiler so the same syntax as in
Cl*pper is supported:

DO <filename>[.prg]

only .prg is accepted as extension and it’s default so you do not
have to set it explicitly.

( In Harbour Users Google group, under “hbmk2 and the Dot Prompt” topic:

It can work as interpreter when invoked without parameters or can execute xBase / Harbour source code in .prg file or compiled Harbour Portable Objects (.hrb) file given as parameter.

Type of file is recognized by extension used with <file> parameter. If not given then .hrb is used.

In other words, HbRun can be use in both interpret  and batch mode.

Regarding parameter given or not, when calling HbRun this ‘mode’ determined by program itself. If a parameter ( usually a .prg or .hrb file name ) given, program run in ‘batch’ mode, runs (executes) given script file and end. If no parameter given, program enter interpreter mode.

Using HbRun as an interpreter, may be very useful, productive, and educative for xBase programmers. Too many xBase programmers was learned everything, including   DBF file system and xBase programming language by famous “dot prompt”. Today many xBase programmers uses HbRun daily basis.

When HbRun begin, open a console screen with two basic area: status bars at top and dot prompt line at bottom.

Status bars :

 hbrunStatLines

Dot prompt is quite simple visually: a dot and a  line in inverse color beginning with a blinking cursor :

 hbRunDotPrompt

You may enter here a command to see the result.

For example “DIR” command will give a list of database (.dbf) files in current directory:

hbRun_Dir

SET COLO TO “GR+/N” command will remember you old days :

hbRun_Dir2

The DIR command can be used with DOS style “filter / skeleton” parameter :

DIR *.PRG
DIR *.*

etc.

Inspecting any table ( .dbf file ) is very simple:

USE CLIENTS
BROWSE ()

 hbrunBrowse

 Expand a little:

SET COLO TO “GB+/N”
USE CLIENTS
BROWSE( 3, 10, 24, 60 )

hbrunBrowse2

If you plan to use this snap frequently, make a .prg file (say brwclien.prg) with this three line and run it with DO command:

DO BRWCLIEN

Sometime LIST command may be better:

LIST CL_ID, CLI_SNAM, CLI_NAME, CLI_TLF

hbrun_list

 You can add FOR clause to the LIST command:

LIST CL_ID, CLI_SNAM, CLI_NAME, CLI_TLF FOR RECN() < 10

or

LIST CL_ID, CLI_SNAM, CLI_NAME, CLI_TLF FOR EMPTY( CLI_TLF )

The structure info of a table frequently requires while daily work to xBase Programmers.

Here three small programs for obtain structure info of a table. Usage is quite simple: open ( USE ) your table and enter DO <prgFileName>; for example:

USE CLIENT
DO LISTSTRU
or
DO DISPSTRU
or
DO SAVESTRU

 Notes :

–      To avoid some possible screen metric conflicts caused by default console (DOS box) settings of OS, may be useful some adjusting before invoke HbRun; such as:

MODE CON LINES=48 COLS=128

–       “?” command may be useful as a built-in calculator :

? 2*2           // 4
? 2**8          // 256
? SQRT( 81 )    // 9

–      HbRun keep a “history” for commands entered (for a limited count of commands of course). You can access (and re-enter when required) by using up and down keys. Moreover this history may be usable after re-invoke HbRun.

–      Though Harbour Language is essential, some legal Harbour commands / functions may be un-recognizable by HbRun.

–      Though some legal statements works in interpret mode, may not works in batch mode (such as Browse() ).

Last Note : No further explanation required for experienced xBase programmers; try, see and learn.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Examples :

 
/*
 DispStru.prg 

 Display structure of current table ( .dbf file ) on screen.

*/
MEMVAR ASTRUCT, NTOTLEN
IF EMPTY( ALIAS() )
   SETCOLOR( "R/N" )
   ? "No active table in the current work area !", LTRIM( STR( SELECT() ) )
ELSE 
   @ 3, 0 CLEA TO MAXROW() - 1, MAXCOL()
   aStruct := DBSTRUCT()
   nTotLen := 1
   AEVAL( aStruct, { | a1Field | nTotLen += a1Field[ 3 ] } )
   AEVAL( aStruct, { | a1Field, n1FieldNo | ;
   aStruct[ n1FieldNo ] := STR( n1FieldNo, 3 ) + " " +;
                           PADR( a1Field[ 1 ], 12 ) +;
                           PADC( a1Field[ 2 ], 4 ) +;
                           PADL( a1Field[ 3 ], 5 ) +;
                           PADL( a1Field[ 4 ], 3 ) } )
   ? "Structure of database :", DBINFO( 10 )
   ? "Number of data records :", LTRIM( STR( LASTREC() ) )
   ? "Date of last update :", LUPDATE()
   ? "Fld Name Type Width Dec"
   ? "--- ---------- ---- ----- ---"
   @ 21,0 SAY "** Total ** " + PADL( nTotLen, 6 )
   ACHOICE( 8, 0, 20, 30, aStruct ) 
ENDIF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/*
 ListStru.prg 

 List structure of current table ( .dbf file ) on screen.

*/
MEMVAR ASTRUCT, NTOTLEN
IF EMPTY( ALIAS() )
   SETCOLOR( "R/N" )
   ? "No active table in the current work area !", LTRIM( STR( SELECT() ) )
ELSE 
   @ 3, 0 CLEA TO MAXROW() - 1, MAXCOL()
   aStruct := DBSTRUCT()
   nTotLen := 1
   AEVAL( aStruct, { | a1Field | nTotLen += a1Field[ 3 ] } ) 
   AEVAL( aStruct, { | a1Field, n1FieldNo | ;
   aStruct[ n1FieldNo ] := STR( n1FieldNo, 3 ) + " " +;
                           PADR( a1Field[ 1 ], 12 ) +;
                           PADC( a1Field[ 2 ], 4 ) +;
                           PADL( a1Field[ 3 ], 5 ) +;
                           PADL( a1Field[ 4 ], 3 ) } )
   ? "Structure of database :", DBINFO( 10 )
   ? "Number of data records :", LTRIM( STR( LASTREC() ) )
   ? "Date of last update :", LUPDATE()
   ? "Fld Name Type Width Dec"
   ? "--- ---------- ---- ----- ---"
   AEVAL( aStruct, { | c1Field | QOUT( c1Field ) } ) 
   ? "** Total ** ", PADL( nTotLen, 5 )
ENDIF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/*
SaveStru.prg

 Save structure of current table ( .dbf file ) to a file.

 Notes :

 - This program uses ListStru.prg
 - Name of target file constructed at line 18; 
   if required you may use alternate ways or
   simply using a constant.
*/
MEMVAR AlteFName
IF EMPTY( ALIAS() )
   SETCOLOR( "R/N" )
   ? "No active table in the current work area !", LTRIM( STR( SELECT() ) )
ELSE 
   AlteFName := LEFT( ALIAS(), 4 ) + "STRU" 
   SET ALTE TO &AlteFName
   SET ALTE ON
   DO LISTSTRU
   SET ALTE OFF
   SET ALTE TO
ENDIF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Clipper conversion process

The Clipper conversion process (.pdf)

Memo File Extension

#include "dbinfo.ch"
PROC Main()
* Test .prg for MemoFExtn() function

 SetMode( 25, 80 ) 
 CLS
 ? "Memo File extension :", MemoFExtn() // NIL
 ? REPL( "-", 80 )

 cTfName := "MFInTst1"
 aStruct := { { "FLD1", "C", 10, 0 } }

 DBCREATE( cTfName, aStruct ) 
 USE ( cTfName )

 ? "Memo File extension :", ">" + MemoFExtn() + "<" // ><
 ? REPL( "-", 80 )

 cTfName := "MFInTst2"
 aStruct := { { "FLD1", "M", 10, 0 } }
 DBCREATE( cTfName, aStruct ) 
 USE ( cTfName )
 ? "Memo File extension :", MemoFExtn() // .dbt
 ? REPL( "-", 80 )

 cTfName := "MFInTst3"
 aStruct := { { "FLD1", "M", 10, 0 } }
 DBCREATE( cTfName, aStruct, "DBFFPT" ) 
 USE ( cTfName )
 ? "Memo File extension :", MemoFExtn() // .fpt
 ? REPL( "-", 80 )

 ?
 WAIT "EOF MemoFExtn.prg"

RETURN // MemoFExtn.Main()
/*
Function MemoFExtn() -> Memo File Extension of current table

 Return : If current work area is empty : NIL 
 elseIf current table not include MEMO field : ""
 else .dbt / or .fpt depending RDD used in creation of current table.

 Required :
 #include "dbinfo.ch"
Aug 2012

*/
FUNCTION MemoFExtn() // Memo file extension
LOCAL xRetVal

 IF !EMPTY( ALIAS() )
 IF DBINFO( DBI_MEMOHANDLE ) < 0
 xRetVal := ""
 ELSE 
 xRetVal := DBINFO( DBI_MEMOEXT ) 
 ENDIF 
 ENDIF !EMPTY( ALIAS() )

RETURN xRetVal // MemoFExtn()