unpack_from() не работает с большими файлами

Question

Я пытаюсь читать данные из файлов с фиксированной шириной (от here), используя python3.

Если я предварительный выбор всего несколько строк, это работает отлично, но если я хочу пойти через файл отверстие (около 1000 строк и 611 блоков в строку а 4 символов = 2444 символы) питон говорит мне, что struct.Struct(bytes).unpackFrom(bytes)a buffer of at least 2444 bytes, в настоящее время я не знаю, почему у него нет такого большого буфера.

Возможно, это полезно, что я работаю на 64-битной Linux с 4-гигабайтной оперативной памятью и 20-гигабайтной своп.

Фрагмент кода это:

#edit 
"""rowMask is 611 times 4s, just to prevent you from counting it... """ 
rowMask="4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s4s" 
def readUsableFields(filename,stdPath): 
    usableFields=[] 
    with open(stdPath+filename,"r") as f: 
      count_line=0 
      for line in f: 
        count_col=0 
        fields=struct.Struct(bytes(rowMask,"UTF-8")).unpack_from(bytes(line,"UTF-8")) 
        for field in fields: 
          if(field!=-999): 
            usableFields.append([count_line,count_col]) 
          count_col+=1 
        count_line+=1 
      return usableFields 

Я также посмотрел на this и this, но они оба не ответ на мою проблему.

Некоторая помощь была бы хорошей, если мой вопрос является дубликатом (что я не нашел), пожалуйста, скажите мне.

Answer 1

Как много фиксированной шириной файлов имеют колонтитул (или заголовок) код

подведет на сноске, поскольку она не может иметь правильную длину.

Таким образом, вы должны будете проверить правильность linelength:

rowMask="4s"*611 
def readUsableFields(filename,stdPath): 
    usableFields=[] 

    with open(stdPath+filename,"r") as f: 
      count_line=0 
      for line in f: 
        count_col=0 
        # len(line) = 611 * 4 +1 
        # as there is a trailing '\0' 
        if(len(line)!=2445): 
          continue 
        fields=struct.Struct(bytes(rowMask,"UTF-8")).unpack_from(bytes(line,"UTF-8")) 
        for field in fields: 
          if(field!=-999): 
            usableFields.append([count_line,count_col]) 
          count_col+=1 
        count_line+=1 

      f.close() 
    return usableFields