Data Encryption

Data Encryption

Top  Previous  Next


QM supports three data encryption methods:


Ad hoc encryption is provided by three QMBasic functions, ENCRYPT(), ENCRYPTX() and DECRYPT(). These take two arguments; the string to be processed and the encryption key (not the name of a key defined with CREATE.KEY as discussed below). Internally, these use the AES 128 bit encryption algorithm but the encrypted data is further processed to ensure that it can never contain the mark characters or ASCII C0 control characters and can, therefore, be stored as a field within a data record or in a text file. The ENCRYPT() function uses a fixed initialisation vector whereas ENCRYPTX() uses a random initialisation vector for improved security but returns a slightly longer result. The DECRYPT() function handles both forms automatically.


Because encryption is a byte level operation, on an ECS mode system, the data to be encrypted must first be transformed to a byte string using the BS conversion code if it may contain ECS characters. Conversely, the decrypted data must be transformed back to ECS form using the same conversion code. It is, therefore, necessary for an ECS capable application to know whether the ECS to byte string transformation has been done. An important point about this process is that the encrypted form of data is identical on ECS mode and non-ECS mode systems so long as the plain text form contains only data from the 8-bit character set.


It is the user's responsibility to provide a mechanism to prevent disclosure of the key string. The key provided by the user can be of any length and may contain any characters. QM will automatically transform this into a form that is valid for use with the AES algorithm. For best security, avoid very short encryption keys which could be determined by repeated attempts to decrypt the data.


Because ad hoc encryption is performed outside of QM's control, it is unlikely to be practical to build alternate key indices on encrypted data.



Record level encryption encrypts an entire record using a single, user defined key and the AES 128, 192 or 256 bit encryption algorithm. Because this encryption occurs deep inside the QM file system, it is possible to have alternate key indices on fields within a file that uses record level encryption. Note, however, that unless the index file itself is also encrypted, indexed fields are partially exposed outside of the encryption system.


Field level encryption allows users to encrypt specific fields within a file, possibly using different keys or algorithms for each encrypted field. It is not possible to build an alternate key index on an encrypted field mainly because not all users might have access to the field and hence the index cannot be maintained. An index built on a non-encrypted field may itself be encrypted. Field level encryption results in a slight increase in record size because of a transformation performed to ensure that the encryption process can never produce data that contains the mark characters.


With either level of encryption, an application may use many different encryption keys and each key can be made accessible to a selected set of users. A user cannot open a file that uses record level encryption unless they have access to the key. When using field level encryption, read operations will return null strings for fields to which the user has no access and write operations will preserve the previous content of these fields when updating an existing record.


The encryption system is managed by a user (or multiple users) known as the security administrator. This user is responsible for management of the key vault, a file that defines the names and actual values of all encryption keys used on the system. The key vault is itself encrypted using the master key which should be known only by the security administrator.  This key value is entered when the key vault is created by first use of CREATE.KEY. If the key vault is moved to another system or a new licence is applied, the master key must be re-entered either via the licence entry screen or by use of the RESET.MASTER.KEY command. Note that there is no way to recover the master key or the data keys if they are lost. Always keep a written copy of these keys in a secure off-site location.


Because QM uses key names rather than the actual keys in operations such as creating files or setting encryption rules, there is no need to restrict knowledge of the key names. The security administrator sets the actual encryption key value when the key is entered into the key vault and this can subsequently be applied to data files without knowing what encryption is being used. For example, the security administrator might define a key named CARDNO for use on fields containing credit card numbers. Developers can freely apply this without knowing the encryption key. To ensure that data is not lost in the event of a major system failure, the master key and details of each key name / key value pair should be stored securely off-site in some way. All security administrator commands require entry of the master key though this is remembered for the duration of the user's session.


A new encryption key is created using the CREATE.KEY command, available only to users with administrative rights in the QMSYS account. This command assigns the actual encryption key and the algorithm to be associated with the key name. The AES 128, 192 and 256 bit algorithms require a key length of 16, 24 or 32 bytes respectively, however, to enable administrators to use convenient keys of any length up to 64 characters, QM will transform the key entered by the user into a form that is valid for the AES algorithms. Specifying a key that is approximately the required internal length or a multiple thereof gives best security.

Key names are case insensitive but the encryption key itself is case sensitive.


Defining a key name makes it accessible to the user that defined it. To make it accessible to other users, the security administrator uses the GRANT.KEY command, specifying the key name and the login names of the users or user groups (not Windows) to whom access is to be granted. Access to a key can subsequently be removed using the REVOKE.KEY command. If a key is no longer used, it can be removed from the key vault using DELETE.KEY.


Keys created using the CREATE.SECURE.KEY command can only be accessed after entry of the associated password. This can be done using the ENABLE.KEY command or the corresponding ENABLE.KEY QMBasic statement. If required, access can be denied again by use of the DISABLE.KEY command or the corresponding DISABLE.KEY QMBasic statement. A user with administrative rights can update the password by use of the CHANGE.KEY.PASSWORD command.


The security administrator can use the LIST.KEYS command to report the name, algorithm and access rights of each encryption key in the key vault. There is no way to report the key string associated with the key. This same command can be used with a filename to report the encryption key names used by the file.


The actual encryption process uses the key string defined in the key vault. If a file that uses encryption is moved to another system, the data in that file will be accessible if the encryption key names used by the file are added to the new system's key vault with the same encryption algorithm and key string.


The master key is used only to encrypt the key vault. The master key must be re-entered if the key vault is moved to another system or possibly as a result of relicensing the system. This is an automatic part of the QM licensing process and helps to ensure security if, for example, a backup tape of the system is stolen. If the master key is forgotten, there is no way to retrieve it from the key vault and hence the vault and all encrypted files would become inaccessible. It is therefore essential that a paper copy of the encryption keys is stored securely.


A file is created for record level encryption by use of the ENCRYPT keyword to the CREATE.FILE command, specifying the name of the encryption key to be used. This style of encryption is supported for hashed files and directory files but note that when applied to directory files, the data cannot be read from outside of QM and any record written from outside will be meaningless within a QM session as the system will attempt to decrypt an unencrypted record. Also, encryption and replication cannot be used together on a directory file. The $ENCR X-type VOC record can be used to apply record level encryption automatically when creating a hashed file. See CREATE.FILE for more details.


For field level encryption, the developer creates the file, populates the dictionary and then uses the ENCRYPT.FILE command to specify the names of each field to be encrypted and the name of the key to be applied. Field level encryption is supported only for hashed files. If the file is not empty, the newly defined encryption is applied to the data. This command can also be used to apply record level encryption to an existing file or to many files in a single operation.


QM also provides optional protection that makes the data inaccessible even if the entire system is stolen - a situation that is more likely with portable computers than with corporate servers. This optional feature requires that the master key is entered every time that QM is started (not for each user entry into the system). Whilst potentially inconvenient, this mode of operation gives the highest level of security and is recommended for portable systems. The process of creating the master key includes a prompt asking if it must be entered to unlock the key vault after system restart. When this mode is enabled, any user with administrator rights can use the UNLOCK.KEY.VAULT command to enable access to encrypted files. Until access is unlocked, files using either field or record level encryption cannot be opened. The need for use of UNLOCK.KEY.VAULT can be enabled or disabled at any time by using RESET.MASTER.KEY.


To allow files to be moved to systems where the encryption key name clashes with a name already defined on that system, the SET.ENCRYPTION.KEY.NAME command can be used to update the key name index stored in the encrypted data file.


When used with QMClient or QMNet, access to encrypted data is based on the access rights of the user name under which the process connects to the server system. Because QMClient is a general library that can be used in various programming environments, it is not possible to apply the encryption at the client side. Although the data will be decrypted on the remote system (assuming that the user has access to it), the actual network traffic of QMClient and QMNet uses its own separate encryption.


ACCOUNT.SAVE, FILE.SAVE and T.DUMP operate within the security rules imposed by use of QM's encryption features. Files that use record level encryption will not be saved if the user performing the save does not have access to the encryption key. The data saved for files that use field level encryption will have all fields for which the user is denied access set to null strings. All saved data is recorded in decrypted form and hence storage of media created using these tools may reduce system security. The media format is an industry standard that does not provide a way to record details of data encryption. Restoring a save in which encrypted fields have been omitted is unlikely to yield a usable file. Backup of accounts that include encrypted data should be performed using operating system level tools.


Encryption can be removed from a file or specific field by use of the DECRYPT.FILE command. This requires that the user has access to the data to be decrypted and also prompts for the master key for improved security.



Disaster Recovery with Encryption


If it is necessary to move a QM system that uses encryption to a different system, the following steps are required:


Copy the data files to the new system, probably restoring them from a backup.


Then do one of the following:


If the new system does not already use encryption, copy/restore the $VAULT subdirectory of the QMSYS account to the new system. Then use the RESET.MASTER.KEY to update the key vault.


If the new system already uses encryption but not with the same key names use CREATE.KEY to create keys to match those of the original system..


If the new system already uses encryption but has the same key names referencing different encryption details, use CREATE.KEY to create replacements for the original encryption keys and use SET.ENCRYPTION.KEY.NAME to amend the key name details stored in each affected file.


In all cases, it may be necessary to use GRANT.KEY to amend access rights to encryption keys if the user names or groups on the new system do not match those of the original system.



Recovery from Loss of the Key Vault


If the key vault is accidentally deleted or damaged in some way, it can be reconstructed as follows:

1.Shut down QM.

2.Delete the $VAULT directory from the QMSYS account directory.

3.Restart QM.

4.Use the CREATE.KEY command to re-enter the data keys using the same key names, encryption modes and key strings as before the failure. When entering the first key, you will be prompted to set the master key. This does not have to be the same as before the failure.