FMDiff™   

This site is W3C compliant:
Valid XHTML - Valid CSS
Last modified February 24 2016, 21:19:04 CET.

File Corruption

It happens! You never can tell when you will be hit, so be prepared. File corruption can happen with every file, not just FileMaker files, but since FileMaker has a complicated and sensitive data structure, one single wrong bit can do a lot of harm.

How to protect your files from corruption?

Back up! In a server - client environment, depending on importance, size and usage, this should be done every day, every few hours or every 30 minutes. Save your backups separate from the server. Keep backups in increasing timely distances to now. For example all of today's backups, one per day of one week, one for every week, and one for every month anyway. Read the white papers provided from FMI and follow suit.

Can you trust your backups?

Despite the fact that you are saving backups, file corruption still may happen. While you think you can just revert to a backup, you may already have been hit silently and your backups are already damaged. FileMaker mailing lists are full of reports like this, just to convince you it really happens all the time.

There are several methods people try to salvage their files and data with the least effort. The best known is the FileMaker built-in Recover command. Maybe you are lucky, Recover your files and continue working with them as if nothing had happened. But that's not always the case - and it's not recommended either.

Some use a clone (best would be one from the time of installing the solution) and import the data from the damaged or recovered file. But you can not rely on the data being complete here either.

FileMaker Server 9 and up claim to verify the backup.

Basically a great feature to make sure the backup does not differ from the production file. But does this exclude any file damages? There is no guarantee the production file and/or the backup are free of corruption.

Why does Recover not always work?

To better understand what may happen, you have to have some knowledge about how such a file is structured. It consists of a linked series of data blocks. Linked means, that one block points to the next block and the next block points to the following one (and the previous one). During normal operation blocks are constantly added to the file whenever one of the blocks is becoming full. The new block is added to the end of the file and the links within the old and the new block are being set appropriately.

When data is removed from the file, blocks bare of data are subsequently "deleted" by first marking them to be unused and linking them to the free blocks list. They can be reused when new data is entered or they are completely removed when the file is closed, or saving a compacted copy.

The Recover menu command does not ask a lot for what may be within such a block, it just reads the headers of a block to find what structural IDs it starts off with. It ignores blocks which are marked "deleted". One bit can make the difference here.

Due to some damage it might be that this bit is no longer set. Nevertheless the file stays fully functional - until someone does a Recover, which then completely removes these blocks and leaves the file damaged. There is no distinction between "structure" and "data" information, it happens where it happens.

There are still other bytes that can have immediate influence on the integrity and functionality of the file that no tool can salvage.

Is this a catch-22 situation?

No! In November 2005 we introduced FMDiff, a tool to compare two FileMaker files. Meanwhile we published eleven important updates. The latest version can compare complete folders in one go at an incredible speed: typically 1 GB in 10 minutes.

FMDiff - as the name implies - was primarily intended to find differences between two FileMaker files, but it soon became apparent its by far more important use was to detect file corruption.

With the help of FMDiff you can make sure a file is in good shape:

• compare the structure (tables, fields, layouts. etc.) to the original, a backup, or a clone
• check files for being well-formed
• check files for special cases, like strictly ascending order of IDs

as such violations are a definite sign of corruption.

A Reliable Corruption Test with FMDiff:

1. Save a Compacted Copy of the file and compare it to the original
   The file should run through the comparison without reporting any differences or errors.
2. Recover the original file and compare the recovered one to the original
   No deleted ( [–] ) items should be reported.
   In case a Recovered Table or a Recovered Library appears ( [+] ), this indicates that there are some "loose ends" in the file, data blocks that are not needed. Unfortunately they can't be deleted in the original, only within the recovered file. Small changes ( [~] ) (modification count, language settings, format changes) appear due to the nature of Recover.

Once these tests are passed you can be 99.9% sure there is no "hidden" corruption within the file.

How do these tests work?

1. FMDiff reads the files logically sequential. If it can't read through the whole file, there is corruption which will be reported.
   There were additional test added, like circular block references, ascending order of IDs, and correct lengths within blocks and segmented lists. If any of these tests fails, there is corruption.
2. A Compacted Copy means the content of a file is completely reordered and compacted and written to a new file. Comparing the original to the compacted copy would discover any different "views" of the content by either FMDiff or FileMaker, i. e. FileMaker copies exactly these elements to the copy that FMDiff assumes it should.
3. Finally the comparison to a Recover of the file shows whether all blocks marked as "used" are really used and free blocks are really free. Otherwise there would be either something missing or superfluous, like a "Recovered Library" with a Blob element. A Recover will utilize all blocks marked as used and skip those marked as unused. FMDiff will show these differences.
   Please note: There will always be differences when comparing to a recovered file. These are mainly triggered from the language information that is not present after a recover, but will be added when the file is opened with FileMaker Pro for the first time.
   Important: The language (date, time, and number) format applied after a recover depends on the current system settings. For this test you can ignore these differences as long as the size [in bytes] of the script or layout is the same.

Having done all these tests you can subsequently safely use File Maintenance provided by FileMaker Pro Advanced 8.5 and 9.

Not knowing the condition of your files can turn out as a great risk. It has been reported that even backups made weeks ago contained some kind of damage which made it impossible to further modify the file structure, despite the file worked flawlessly.

Important note about the extent of the comparison with FMDiff:

Some differences between files are currently not reported - for various reasons.

• The text version of calculations. Only the compiled (working) byte code is compared.
• The list order of tables, fields, scripts, layouts, etc.
• and some other elements

These additional differences are usually undesirable in a report and would require a set of preferences to switch them on or off. These preferences and additional tests will be added with future versions.

This, for example, makes it possible that a calculation definition may not be accessible (like in a runtime) or contain garbage when opened, although the calculation as such works perfectly. In this case you have to revert to your "golden clone" or a backup.

You may however, copy the calculation from the FMDiff html report and paste it into a calculation definition in FileMaker Pro - provided there is a difference and hence it shows up.

We test your suspected corrupt files for free

Please check out our Repair Service

Article to be continued ...