Не обрабатывает ZIP архивы на NAS

9 years 4 months ago - 9 years 4 months ago #350 by mypill
mypill created the topic: Не обрабатывает ZIP архивы на NAS
Добрый день!
Установил Вашу замечательную прогрумму для NAS D-Link DNS-320L. На нем есть python3 и коннектор к бд и mysql. При сканировании обычных fb2 не запакованных все нормально. Есть доступ к книгам по веб-интерфейсу 8081. При сканировании выводит ошибки в лог:
2014-12-10 17:00:35,693 INFO     sopdsDaemon __init__()...
2014-12-10 17:00:35,696 INFO     sopdsDaemon start()...
2014-12-10 17:00:36,412 INFO     sopdsDaemon entering in main loop...
2014-12-10 17:00:36,415 INFO      ***** Starting sopds-scan...
2014-12-10 17:00:36,417 DEBUG    OPTIONS SET
2014-12-10 17:00:36,418 DEBUG    configfile = /mnt/HD/HD_a2/sopds-master/conf/sopds.conf
2014-12-10 17:00:36,440 DEBUG    root_lib = /mnt/HD/HD_b2/0
2014-12-10 17:00:36,441 DEBUG    temp_dir = /ffp/tmp
2014-12-10 17:00:37,032 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173578.fb2
2014-12-10 17:00:37,044 DEBUG    Attempt to add book 999.zip/173578.fb2
2014-12-10 17:00:37,230 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173578.fb2
2014-12-10 17:00:37,231 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173579.fb2
2014-12-10 17:00:37,235 DEBUG    Attempt to add book 999.zip/173579.fb2
2014-12-10 17:00:37,286 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173579.fb2
2014-12-10 17:00:37,288 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173580.fb2
2014-12-10 17:00:37,321 DEBUG    Attempt to add book 999.zip/173580.fb2
2014-12-10 17:00:37,412 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173580.fb2
2014-12-10 17:00:37,414 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173581.fb2
2014-12-10 17:00:37,417 DEBUG    Attempt to add book 999.zip/173581.fb2
2014-12-10 17:00:37,504 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173581.fb2
2014-12-10 17:00:37,506 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173582.fb2
2014-12-10 17:00:37,509 DEBUG    Attempt to add book 999.zip/173582.fb2
2014-12-10 17:00:37,627 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173582.fb2
2014-12-10 17:00:37,629 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173583.fb2
2014-12-10 17:00:37,652 DEBUG    Attempt to add book 999.zip/173583.fb2
2014-12-10 17:00:37,755 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173583.fb2
2014-12-10 17:00:37,757 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173584.fb2
2014-12-10 17:00:37,760 DEBUG    Attempt to add book 999.zip/173584.fb2
2014-12-10 17:00:37,834 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173584.fb2
2014-12-10 17:00:37,835 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173585.fb2
2014-12-10 17:00:37,838 DEBUG    Attempt to add book 999.zip/173585.fb2
2014-12-10 17:00:37,936 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173585.fb2
2014-12-10 17:00:37,938 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173586.fb2
2014-12-10 17:00:37,961 DEBUG    Attempt to add book 999.zip/173586.fb2
2014-12-10 17:00:38,192 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173586.fb2
2014-12-10 17:00:38,193 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173587.fb2
2014-12-10 17:00:38,196 DEBUG    Attempt to add book 999.zip/173587.fb2
2014-12-10 17:00:38,229 ERROR    Error processing ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173587.fb2
2014-12-10 17:00:38,284 INFO     Books added      : 0
2014-12-10 17:00:38,286 INFO     Books skipped    : 0
2014-12-10 17:00:38,287 INFO     Books DB entries deleted : 0
2014-12-10 17:00:38,289 INFO     Books in archives: 0
2014-12-10 17:00:38,310 INFO     Archives scanned : 1
2014-12-10 17:00:38,312 INFO     Archives skipped : 0
2014-12-10 17:00:38,313 INFO     Bad archives     : 0
2014-12-10 17:00:38,315 INFO     Time estimated:0 hours, 0 minutes, 1 seconds.
2014-12-10 17:00:38,316 INFO     Starting mark_double proc with DUBLICATES_FIND param = 2
2014-12-10 17:00:38,371 INFO     Finishing mark_double proc in 0 hours, 0 minutes, 0 seconds.

/tmp - папка указана в /ffp/tmp - свободного места там больше 20 ГБ.

Архив не битый. Проверено. Zip на NAS упаковываются и распаковываются нормально. Как быть?

P.S. Ваша программа лучшая в своем классе. Спасибо за Ваш труд.
Last Edit: 9 years 4 months ago by mypill.

Please Войти or Create an account to join the conversation.

  • mypill
  • mypill's Avatar Topic Author
  • Offline
  • Новый участник
  • Новый участник
More
9 years 4 months ago - 9 years 4 months ago #351 by mitshel
mitshel replied the topic: Не обрабатывает ZIP архивы на NAS
Попробуйте в файле sopdscan.py в функции processzip закомментировать строки как показано ниже и запустить ресканирование zip-архива. Ну и лог снова прислать:
def processzip(self,name,full_path,file):
        rel_file=os.path.relpath(file,self.cfg.ROOT_LIB)
        if self.cfg.ZIPRESCAN or self.opdsdb.zipisscanned(rel_file,1)==0:
            cat_id=self.opdsdb.addcattree(rel_file,1)
            try:
                z = zipf.ZipFile(file, 'r', allowZip64=True)
                filelist = z.namelist()
                for n in filelist:
#                    try:
                        self.logger.debug('Start process ZIP file = '+file+' book file = '+n)
                        file_size=z.getinfo(n).file_size
                        self.processfile(n,file,z.open(n),1,file_size,cat_id=cat_id)
#                    except:
#                        self.logger.error('Error processing ZIP file = '+file+' book file = '+n)
                z.close()
                self.arch_scanned+=1
            except:
                self.logger.error('Error while read ZIP archive. File '+file+' corrupt.')
                self.bad_archives+=1
        else:
            self.arch_skipped+=1
            self.logger.debug('Skip ZIP archive '+rel_file+'. Already scanned.')
Last Edit: 9 years 4 months ago by mitshel.

Please Войти or Create an account to join the conversation.

  • mitshel
  • mitshel's Avatar
  • Offline
  • Администратор
  • Администратор
More
9 years 4 months ago #352 by mypill
mypill replied the topic: Не обрабатывает ZIP архивы на NAS
Закомментировал и запустил вот лог:
2014-12-10 20:32:23,841 INFO     sopdsDaemon __init__()...
2014-12-10 20:32:23,844 INFO     sopdsDaemon start()...
2014-12-10 20:32:24,411 INFO     sopdsDaemon entering in main loop...
2014-12-10 20:32:24,414 INFO      ***** Starting sopds-scan...
2014-12-10 20:32:24,415 DEBUG    OPTIONS SET
2014-12-10 20:32:24,417 DEBUG    configfile = /mnt/HD/HD_a2/sopds-master/conf/sopds.conf
2014-12-10 20:32:24,418 DEBUG    root_lib = /mnt/HD/HD_b2/0
2014-12-10 20:32:24,440 DEBUG    temp_dir = /ffp/tmp
2014-12-10 20:32:24,813 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173578.fb2
2014-12-10 20:32:24,824 DEBUG    Attempt to add book 999.zip/173578.fb2
2014-12-10 20:32:24,895 ERROR    Error while read ZIP archive. File /mnt/HD/HD_b2/0/999.zip corrupt.
2014-12-10 20:32:24,955 INFO     Books added      : 0
2014-12-10 20:32:24,956 INFO     Books skipped    : 0
2014-12-10 20:32:24,957 INFO     Books DB entries deleted : 0
2014-12-10 20:32:24,959 INFO     Books in archives: 0
2014-12-10 20:32:24,961 INFO     Archives scanned : 0
2014-12-10 20:32:24,962 INFO     Archives skipped : 0
2014-12-10 20:32:24,963 INFO     Bad archives     : 1
2014-12-10 20:32:24,965 INFO     Time estimated:0 hours, 0 minutes, 0 seconds.
2014-12-10 20:32:24,966 INFO     Starting mark_double proc with DUBLICATES_FIND param = 2
2014-12-10 20:32:25,031 INFO     Finishing mark_double proc in 0 hours, 0 minutes, 0 seconds.
А можно вопрос, что использует Ваша программа: внутренний механизм или внешний для архивов? Например на NAS установлен Zlib 1.2.8. Но может это вообще не в тему... Я не спец.

Please Войти or Create an account to join the conversation.

  • mypill
  • mypill's Avatar Topic Author
  • Offline
  • Новый участник
  • Новый участник
More
9 years 4 months ago - 9 years 4 months ago #353 by mitshel
mitshel replied the topic: Не обрабатывает ZIP архивы на NAS
Используется модуль Питона под названием zipfile
В общем Ваш зип-архив какой-то нестандартный, в итоге corrupt.
Может быть он с паролем? поскольку ясно что содержимое файла считывается а сами данные нет.

Порпобуйте заменить всю процедуру processzip на приведенную ниже. И вновь попробовать запустить сканирование.

def processzip(self,name,full_path,file):
        rel_file=os.path.relpath(file,self.cfg.ROOT_LIB)
        if self.cfg.ZIPRESCAN or self.opdsdb.zipisscanned(rel_file,1)==0:
            cat_id=self.opdsdb.addcattree(rel_file,1)
            z = zipf.ZipFile(file, 'r', allowZip64=True)
            filelist = z.namelist()
            for n in filelist:
                        self.logger.debug('Start process ZIP file = '+file+' book file = '+n)
                        file_size=z.getinfo(n).file_size
                        self.processfile(n,file,z.open(n),1,file_size,cat_id=cat_id)
            z.close()
            self.arch_scanned+=1
        else:
            self.arch_skipped+=1
            self.logger.debug('Skip ZIP archive '+rel_file+'. Already scanned.')
Last Edit: 9 years 4 months ago by mitshel.

Please Войти or Create an account to join the conversation.

  • mitshel
  • mitshel's Avatar
  • Offline
  • Администратор
  • Администратор
More
9 years 4 months ago - 9 years 4 months ago #354 by mypill
mypill replied the topic: Не обрабатывает ZIP архивы на NAS
Нет, мой архив стандартный. Пробовал из либрусека - ошибка. И сам пробовал паковать. Ошибка одна. Еще немного информации: Nas armhf, установлен fun_plug, доступен ipkg. Может есть еще варианты? Может быть нужны какие-то модули для питона? :(

Только увидел ваше дополнение к ответу. Буду ждать.
Last Edit: 9 years 4 months ago by mypill.

Please Войти or Create an account to join the conversation.

  • mypill
  • mypill's Avatar Topic Author
  • Offline
  • Новый участник
  • Новый участник
More
9 years 4 months ago - 9 years 4 months ago #355 by mypill
mypill replied the topic: Не обрабатывает ZIP архивы на NAS
Немного пошаманил:

Заметил, что в папке py есть файл zipf.py - он напомнил мне файл из папки lib в питоне zipfile.py.
Переименовал zipf.py в orig_zipf.py
Скопировал zipfile.py и переименовал в zipf.py
Запустил демона, вот лог:
2014-12-11 09:51:45,183 INFO     sopdsDaemon __init__()...
2014-12-11 09:51:45,189 INFO     sopdsDaemon start()...
2014-12-11 09:51:45,787 INFO     sopdsDaemon entering in main loop...
2014-12-11 09:51:45,791 INFO      ***** Starting sopds-scan...
2014-12-11 09:51:45,796 DEBUG    OPTIONS SET
2014-12-11 09:51:45,797 DEBUG    configfile = /mnt/HD/HD_a2/sopds-master/conf/sopds.conf
2014-12-11 09:51:45,799 DEBUG    root_lib = /mnt/HD/HD_b2/0
2014-12-11 09:51:45,800 DEBUG    temp_dir = /ffp/tmp
2014-12-11 09:51:46,335 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173578.fb2
2014-12-11 09:51:46,393 DEBUG    Attempt to add book 999.zip/173578.fb2
2014-12-11 09:51:46,881 DEBUG    Book 999.zip/173578.fb2 Added ok.
2014-12-11 09:51:46,948 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173579.fb2
2014-12-11 09:51:46,981 DEBUG    Attempt to add book 999.zip/173579.fb2
2014-12-11 09:51:47,443 DEBUG    Book 999.zip/173579.fb2 Added ok.
2014-12-11 09:51:47,511 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173580.fb2
2014-12-11 09:51:47,514 DEBUG    Attempt to add book 999.zip/173580.fb2
2014-12-11 09:51:47,862 DEBUG    Book 999.zip/173580.fb2 Added ok.
2014-12-11 09:51:47,893 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173581.fb2
2014-12-11 09:51:47,896 DEBUG    Attempt to add book 999.zip/173581.fb2
2014-12-11 09:51:47,981 DEBUG    Book 999.zip/173581.fb2 Added ok.
2014-12-11 09:51:48,009 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173582.fb2
2014-12-11 09:51:48,013 DEBUG    Attempt to add book 999.zip/173582.fb2
2014-12-11 09:51:48,117 DEBUG    Book 999.zip/173582.fb2 Added ok.
2014-12-11 09:51:48,147 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173583.fb2
2014-12-11 09:51:48,150 DEBUG    Attempt to add book 999.zip/173583.fb2
2014-12-11 09:51:48,239 DEBUG    Book 999.zip/173583.fb2 Added ok.
2014-12-11 09:51:48,314 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173584.fb2
2014-12-11 09:51:48,322 DEBUG    Attempt to add book 999.zip/173584.fb2
2014-12-11 09:51:48,390 DEBUG    Book 999.zip/173584.fb2 Added ok.
2014-12-11 09:51:48,420 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173585.fb2
2014-12-11 09:51:48,423 DEBUG    Attempt to add book 999.zip/173585.fb2
2014-12-11 09:51:48,544 DEBUG    Book 999.zip/173585.fb2 Added ok.
2014-12-11 09:51:48,570 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173586.fb2
2014-12-11 09:51:48,573 DEBUG    Attempt to add book 999.zip/173586.fb2
2014-12-11 09:51:48,865 DEBUG    Book 999.zip/173586.fb2 Added ok.
2014-12-11 09:51:48,893 DEBUG    Start process ZIP file = /mnt/HD/HD_b2/0/999.zip book file = 173587.fb2
2014-12-11 09:51:48,897 DEBUG    Attempt to add book 999.zip/173587.fb2
2014-12-11 09:51:48,961 DEBUG    Book 999.zip/173587.fb2 Added ok.
2014-12-11 09:51:49,010 INFO     Books added      : 10
2014-12-11 09:51:49,011 INFO     Books skipped    : 0
2014-12-11 09:51:49,013 INFO     Books DB entries deleted : 0
2014-12-11 09:51:49,015 INFO     Books in archives: 10
2014-12-11 09:51:49,018 INFO     Archives scanned : 1
2014-12-11 09:51:49,022 INFO     Archives skipped : 0
2014-12-11 09:51:49,023 INFO     Bad archives     : 0
2014-12-11 09:51:49,025 INFO     Time estimated:0 hours, 0 minutes, 3 seconds.
2014-12-11 09:51:49,026 INFO     Starting mark_double proc with DUBLICATES_FIND param = 2
2014-12-11 09:51:49,124 INFO     Finishing mark_double proc in 0 hours, 0 minutes, 0 seconds.
Книги в БД есть и они открываются.
В дополнение Питон 3.2 (вроде бы), установлен из ipkg.
Код нового zipf.py
Warning: Spoiler! [ Click to expand ]
Last Edit: 9 years 4 months ago by mypill.

Please Войти or Create an account to join the conversation.

  • mypill
  • mypill's Avatar Topic Author
  • Offline
  • Новый участник
  • Новый участник
More
Time to create page: 0.198 seconds