Database corruption questions/rant

I have a P4 with a ~200gb library on 1tb internal that I [intended to] use for mobiles. I say intended to because, aside from the unwieldy workflow involved in actually getting files onto the unit, I experienced a problem this weekend that makes me question whether I can rely on this gear.

After spending the better part of a week preparing for a wedding, a few hours before the gig yesterday I went to update the library on the internal drive. 20 or 30 minutes later, the status bar shows no jobs running and I press the button on the Prime to switch it back to standalone. Woops. The database is corrupt. Please reformat the drive and reload the library. Are. You. Serious. I ended up doing the gig using my laptop standalone with vdj. God I love that program.

Iā€™m assuming the only thing that went wrong was that I failed to press the eject button in Engine. This is an absolutely horrible design. I can understand if a drive gets pulled in the middle of a transfer, but if the system is literally sitting there doing nothing and you unplug a drive, that must not destroy the library. You should do things to protect the library.

If there is a situation where the library does get corrupted, users need some resolutions other than ā€œreformat the drive and rebuild the library.ā€ Thatā€™s a catch all, itā€™s impractical and itā€™s overkill in most situations. I know the music made it on to the drive. Why canā€™t I just rebuild the database?!

Would love it if someone from Denon can weigh in on what causes the database to get corrupted. Are there known bugs that cause corruption? Is there any way to rebuild the database without wiping the drive? Are there any plans to give us tools to do this? Is the only actual solution to wipe the drive and reload the library?

3 Likes

Man, I feel sorry for what happened I really doā€¦ but yeah you are right that itā€™s because you didnā€™t eject the drive, it corrupted it. The reason for this, even though the drive may appear that itā€™s doing nothing, it that it actually might be! Itā€™s called write caching, and when you eject a drive that forces the host to finish writing anything itā€™s doing so itā€™s totally safe. This is why it can sometimes take a few more seconds than you expect for the drive to be unmounted/ejected. Itā€™s because even though you think itā€™s done everythingā€¦ it actually hasnā€™t.

Now, it may be the partition map/directory that became corrupted, or it may have corrupted just the library file if it hadnā€™t finished with the drive. I donā€™t know if you are Mac or PC, but you can mount the drive and run a disk fixing utility on the drive. This will be useful because if it is the partition map or directory thatā€™s corrupted, that can usually be easily fixed and boom, you have everything back. If itā€™s just the library file then, sorry, bad luck Iā€™m afraid.

Having written all that, it might be possible for Denon, in a future version of the firmware, to include a function that if you are standalone mode, and switch it back to controller mode, then it automatically sends an eject signal to the drive. I donā€™t know if this is technically possible as really, the P4 is just a client to the Mac/PC host at this point.

I hope that shines a light onto why it happened, a possible fix for you, and a future possible solution.

2 Likes

And this is why I come here. Some really knowledgeable people on this forum that have great ways of explaining things that I want to learn about.

2 Likes

Youā€™re very welcome. Iā€™d like to know if the disk fixing utility you try fixes it for you though, let us all know! Best of luck :slight_smile:

1 Like

I donā€™t get why the databases on the Denons get corrupted so easily. Sure, you should always dismount your drives before ejecting them, but Iā€™m not sure this is normal. I work with sql databases every day and this does not feel statistical normal to me.

Indeed, I donā€™t know if a database that Denon recognises as ā€œcorruptedā€ is corrupted at all. I have tried to open ā€œcorruptedā€ databases in SQL browser programs, and they are simply OK.

Also, the message of ā€œDatabase corrupted, reformat your driveā€ is misleading to say the least. One thing is the databases files in the Engine Library (there are more than one, what a strange design) are faulty, another is the drive partition to be faulty. Formatting (rebuilding the drive partition) makes sense when the disk is corrupted, but not when the database is corrupted. Indeed, disks with databases that Denon flags as corrupted can perfectly be mounted in a laptop and are operable, hence, there is simply no need to reformat the disk (indeed, as said before even the database looks healthy).

If it helps, it the decks says that the db is corrupted by you can still read the drive in the laptop, Iā€™d recommend to simply delete the ā€œEngine Libraryā€ directory, run Engine, and drag all the songs in the drive to Engine, so the Library is built out of the songs in the drive and that way at least you save the time transferring the files from the laptop to the drive (process that Denon calls ā€œpackagingā€).

To me, the management of the drives in the Prime system looks very unstable and not robust at all. Transferring files to the drive or adding new songs in Engine takes ages for no reason. Search and navigation with big libraries is slower than is should be, databases get corrupted way more that it would look reasonable to me, searching libraries sometimes simply hangs indefinitely, my linked SC5000 sometimes crashes when searching in a linked databaseā€¦

I suspect the code handling the db stuff and disk access is not as robust as it should be. The rest is fine, but everything related to drive access is not stable enough in my opinion.

1 Like

I would recomment denon just save a copy of its database on the drive (yea it doubles the amount of space but its saver!) - If the loaded database find any corruptions, it could load the faulty data from its backup and repair itself. so we dont need to reformat the drive. but safety first - backup your library to a second external hdd.

2 Likes

Hi @jalderwood, thanks for posting. Iā€™m sorry to hear about this corruption issue you are having, typically ejecting without ā€˜ejectingā€™ shouldnā€™t have caused this problem. We definitely want to help you out, and weā€™d like to do so by investigating this corruption on our own. If possible, can you send us all of the database files (not your music folder, but everything else in the Engine Library folder on your hard drive) so that we can run tests on our end?

1 Like

@Nekoro_DenonDJ Unfortunately I already formatted. Doubt this helps but p.db was just over 2gb. Are there any cached copies stored locally?

After formatting I dragged my collection back onto it in EP. Half a day later I received a message about not being able to pack some files, which also happened previously when the drive got corrupted. There are a little over 100 tracks in the ā€˜Packing Jobā€™ list and there is no obvious reason for why the files failed. They do exist on disk. Can you tell us what a packing list is and what can cause EP to choke on files? (These same files did not fail the first time I put my library on this drive.)

Tobes - thereā€™s nothing wrong with the disk, itā€™s a Samsung SSD that Iā€™ve had no problems with. I initially put the library onto it back in June and had updated it at least once prior to this without issue.

Thatā€™s a bummer on the files, no worries. Let me talk to the Dev Team and see what they can do with the addition info you gave us, @jalderwood. Stand by.

Ok, Dev team needs more info on those 100 tracks that fail to pack, @jalderwood. Can you manually drag and drop each of those files from their original source directly into your storage device via ā€˜Devicesā€™ in Engine PRIME? In other words, import those songs into your HD via Engine PRIME without importing those songs into the softwareā€™s library and making crates first.

Iā€™m not going to go through this exercise of searching for 100+ individual tracks and individually putting them on a drive just to have a similar thing happen again next time I try to update music. You still havenā€™t given me any information about why this happens.

I did about 10 tracks. I dragged them on the drive, they are searchable and they load from the drive. The same tracks still appear in red in the main library. What would be the process to have those restored?

In the interests of helping the situation why donā€™t you put 50 of the hundred files in a folder ā€œAā€ and the other 50 of the hundred files in folder B and see which folder wonā€™t work. This way, if thereā€™s only one unliked file then you narrow it down quickly.

Take the folder than wonā€™t work and split its 50 files into two folders and try loading those folders to see which wonā€™t work - now youā€™re down to 25 files ā€¦ then 12 or 13 then 6 or 7

Sometimes itā€™s a case of helping people to help yourself

1 Like

I right click > removed one of the broken files from the main collection.

Restarted EP just in case.

Imported same file back into collection. Says itā€™s analyzing, then takes a little more time to finish the job (update the db I assume)

File is not searchable.

@DJ_Didi I am not sure what you mean about splitting files in half. They are all over my library in artist folders (itunes). Ah, iTunes, that was how I initially imported. Maybe that helps.

Out of curiosityā€¦ And as far as I can see, its not mentioned anywhere (maybe I have missed it)ā€¦

The 100 files - which format are they? Mp3, wave etc etc?

all of the files are mp3

It seems like engine is susceptible to corrupting easilyā€¦ I turned the sc5000 power on this past sundayā€¦ and remembered I hadnt added new tunes to the USB, so at the ā€œload profile screenā€ I just removed the usbā€¦ and then had to spend the next hour formatting and re importing everythingā€¦its a pain!

it doesnt, ive just had the same issueā€¦ windows says its fine but engine says still corrupt. Only a 32gb USB drive in my case so not too bad,

Hi All,

Sorry to see the issues with database corruption. This is something we take very seriously but need to collect as much detail as possible to help us better understand what could be causing the behavior.

For those experiencing this issue please help to answer the following. We appreciate all of your help!

What version of Engine Prime are you using?

Did you experience similar issues with previous versions of Engine Prime?

Are you able to consistently reproduce the behavior with specific workflow steps? If so, please outline the exact steps to reproduce.

Is this happening with the internal HDD or external media devices or both?

What brand media device and capacity?

If internal, what OS are you running on your computer?

What is you eject process on computer?

  1. Safely eject in Engine Prime.
  2. Close Engine Prime then safely eject from Finder/Explorer.

What is your eject process for media devices on Prime hardware?

  1. Power off hardware without ejecting, remove drive.
  2. Safely eject on hardware, remove drive power off.
  3. Remove the drive while hardware is on without ejecting.
  4. Internal HDD.

Are you using DECU software to import your 3rd party collections?

Lastly, if you see the database corrupt message, it would be very helpful if you could zip all of the files in the Engine Library folder from your computer or media device (which ever collection is corrupt). Just zip everything in the Engine Library folder except the Music folder and itsā€™ contents. This will be very beneficial in helping us understand and identifying the root cause.

It may be worth trying to run the Library Optimization Cleanup in the Preferences of Engine Prime preferences. Be sure your problematic media device is connected when running the optimization.

where does one submit a corrupt library zip?

Hi @jalderwood,

Please contact our Engine DJ Technical Support team for further assistance with this matter. A member of the team will help you identify the needed files and provide a link for upload.

Support | Engine DJ