Binary file comparison

Introduction

The binary files comparison will compare, in two folders, files with the same structure. More precisely, it will compare files with the same path inside the two folders.          

Binary files are arbitrary files with a format that is described in another file named a fileformat file. The fileformat files are in another dedicated fileformat folder. 

Fileformat files

The fileformat files are xml description files that explain the various fields in the file, their length and their format. A simplest file would look like below, though other attributes are allowed: 

<?xml version="1.0" encoding="ASCII"?> 
<FileFormat ConversionTable="Cp037"> 
<RecordFormat> 
<FieldFormat Name="NAME" Type="X" Size="15" /> 
<FieldFormat Name="CODE" Type="9" Size="5" /> 
<FieldFormat Name="AMOUNT" Type="9" Size="12" Decimal="2" Signed="true" /> 
</RecordFormat> 
</FileFormat> 

Field formats

The field formats use a terminology similar as in Cobol. The main difference is that the “Size” attribute corresponds to the total size.          
For instance, this definition: NUM1 PIC S9(6)V9(2) corresponds to (notice the 8): <FieldFormat Name="NUM1" Type="9" Size="8" Decimal="2" Signed="true" />          

The handled types are: X, 9, 1, 2, 3, 5, 7, 8, B, H, T, Z.          
 

Repeated data

The “Occurs” attribute enables to define that a field is repeated.          
<FieldFormat Name="ADDRLINE" Type="X" Size="30" Occurs="3" />          

The “DependingOn” attributes can be used if the occurs depends on another field.          
<FieldFormat Name="CNT" Type="9" Size="2" />          
<FieldFormat Name="ARRAY" Type="X" Size="20" Occurs="50" DependingOn="CNT" />          

In this case, the “Occurs” value corresponds to the maximum value.          
If the file uses RDW (Record Descriptor Word) or BDW (Block Descriptor Word), the tool can read their values and make them available as reserved field names that can be used with DependingOn. See Variable-length record files (V and VB) for details.       

These attributes can also apply to fields group          
<FieldFormat Name="CNT" Type="9" Size="2" />          
<FieldsGroup Name="ARRAY" Occurs="50" DependingOn="CNT">          
<FieldFormat Name="KEY" Type="9" Size="1" />          
<FieldFormat Name="VALUE" Type="X" Size="10" />          
</FieldsGroup>          
 

Variable-length record files (V and VB)

The tool supports comparing variable-length (V) and variable-length blocked (VB) binary files by handling Block Descriptor Words (BDW) and Record Descriptor Words (RDW).

• BDW (Block Descriptor Word): A 4-byte prefix at the beginning of each block in VB files. The first 2 bytes contain the block length. 
• RDW (Record Descriptor Word): A 4-byte prefix at the beginning of each record in V and VB files. The first 2 bytes contain the record length. 

When hasBdw or hasRdw is enabled (either in the configuration file or in the fileformat file), the tool reads the descriptor words and makes their values available through reserved field names that can be used in the fileformat file:

Reserved field names    
When hasBdw is enabled:

When hasRdw is enabled:

These reserved field names should be declared as FieldFormat entries in the fileformat file. The Type and Size attributes are present for structural consistency but their values are ignored — the actual values come from the parsed BDW/RDW. The RDW reserved names (_RDW_ and _RECORD_SIZE_) can also be used in the DependingOn attribute of another field to define variable-length records. The BDW reserved names (_BDW_ and _BLOCK_SIZE_) are available for display in the comparison report but cannot be used in DependingOn.

Example: VB file (variable-length blocked)    
For files that have both block and record descriptors, set hasBdw="true" and hasRdw="true":

<?xml version="1.0" encoding="ASCII"?>
<FileFormat ConversionTable="IBM-1147" hasBdw="true" hasRdw="true">
 <RecordFormat cobolRecordName="RAW-RECORD-VARIABLE">
   <FieldFormat Name="_BDW_" Type="B" Size="2" />
   <FieldFormat Name="_BLOCK_SIZE_" Type="B" Size="2" />
   <FieldFormat Name="_RDW_" Type="B" Size="2" />
   <FieldFormat Name="_RECORD_SIZE_" Type="B" Size="2" />
   <FieldFormat Name="REC-DATA" Type="X" Size="1" Occurs="80" DependingOn="_RECORD_SIZE_" />
 </RecordFormat>
</FileFormat>

In this example, REC-DATA has a variable length that depends on _RECORD_SIZE_ (the RDW value minus 4). The Occurs="80" defines the maximum possible size.

Example: V file (variable-length, not blocked)    
For files that have record descriptors but no block descriptors, set hasBdw="false" (or omit it) and hasRdw="true":

<?xml version="1.0" encoding="ASCII"?>
<FileFormat ConversionTable="IBM-1147" hasRdw="true">
 <RecordFormat cobolRecordName="RAW-RECORD-VARIABLE">
   <FieldFormat Name="_RDW_" Type="B" Size="2" />
   <FieldFormat Name="_RECORD_SIZE_" Type="B" Size="2" />
   <FieldFormat Name="REC-DATA" Type="X" Size="1" Occurs="80" DependingOn="_RECORD_SIZE_" />
 </RecordFormat>
</FileFormat>

Notes

• When hasBdw is enabled, hasRdw is automatically enabled as well (VB files always have both BDW and RDW).
• The hasBdw and hasRdw settings can be specified at two levels:
  • Configuration file (bacmp): Applies globally to all files in the comparison.
  • Fileformat file (XML): Applies to the specific file described by that fileformat. The fileformat-level setting takes precedence.
• The reserved field names are case-insensitive when matched.

Multiple record formats

When the record can have various formats, multiple record formats can be used. In this case, the record will first be parsed by the tool to determine which record format is suitable, thanks to a pattern.          

By default, one HTML page will be generated, as a report, for each record. This way the HTML table describing the columns will be consistent in each page.          

The amount of data that must be read to verify the pattern must be setup at the <FieldFormat> level, in the distinguishedFieldSize attribute and each <RecordFormat> must bear a pattern in its distinguishedFieldValue attribute.          

In below example, the tool is instructed to look if the 16^th character of the record is a A or a B (in this case it will match record RF1), or a Z (in this case it will match RF2). In this case, 16 characters must be read, but the 15 first can be anything. 

<FileFormat ConversionTable="Cp037" distinguishFieldSize="16"> 
<RecordFormat cobolRecordName="RF1" distinguishFieldValue=".{15}(A|B)"> 
... 
</RecordFormat> 
<RecordFormat cobolRecordName="RF2" distinguishFieldValue=".{15}(Z)"> 
... 
</RecordFormat> 
</FileFormat> 

Configuration file

A minimal bacmp configuration file for binary files will be as below. It will scan the default left and right folders and compare each file found with same name, using the fileformat file with the same name found in the default fileformat folder. 

{"comparisonType“: "Binary“} 

The files to compare and the corresponding fileformat files will be matched this way: 

• Try to match a file like ‘test.ebc’ with a fileformat named ‘test.ebc.fileformat’
• Try to match a file like ‘test.ebc’ with a fileformat named ‘test.fileformat’
• Use the fileformat in the folder, whatever its name, if it is unique in its folder

Business keys, removed and rejected columns

You can specify, for each file, columns which are business keys, rejected columns or removed columns. You can also manually specify the fileformat file, thus overriding the logic explained in previous chapter.          

Business keys allows you to specify which records are the same in left and right file. (See Business keys for an example, it works the same as in databases).          

Removed columns will not appear at all in comparison, rejected columns will appear but values will be replaced by ‘(REJECTED)’.

Please note that the file list that appears in the ‘details’ here does not restrict the list of compared files. All files in the folders are compared, and these files only need additional details. In order to compare only some files, you can change the left and right folder, or use a file filter. See Additional properties for more details.

Additional properties

In addition to the configurations above, there are additional properties. Most have default values that are used if the property does not appear in the file, but can also be specified to another value. Others are not used if left unspecified.          

Default values are written in green below when they exist, a red ‘no value’ otherwise. 

Note: hasBdw and hasRdw can be set both at the configuration file level (bacmp) and at the fileformat file level (XML). The fileformat-level setting takes precedence for individual files.