Многопроцессорный пул apply_async

Question

Я играю с многопроцессорной обработкой и, в частности, с бассейном.

import multiprocessing 
import os 
import time 

results = [] 

def task(args): 
    print "Start Process", os.getpid(), "with args", args, " timing", time.time(
) 
    time.sleep(10) 

    print "End Process", os.getpid(), "with args", args, " timing", time.time(
) 
    return os.getpid(), args, timestamp 


def log_result(result): 
    results.append(result) 

if __name__ == "__main__": 
    num_processes=64 
    pool = multiprocessing.Pool(processes=num_processes) 

    for i in range(num_processes): 
     pool.apply_async(task, args=(i,), callback=log_result) 

    pool.close() 
    pool.join() 

Моя машина должна быть в состоянии работать 32 потока (в соответствии с файлом /proc/cpuinfo, у меня есть 32 процессоров).

Когда я бегу этот сценарий, я получаю следующие результаты:

Start Process 87166 with args 0 timing 1414054820.26 
Start Process 87167 with args 1 timing 1414054820.26 
Start Process 87168 with args 2 timing 1414054820.26 
Start Process 87169 with args 3 timing 1414054820.26 
Start Process 87170 with args 4 timing 1414054820.26 
Start Process 87171 with args 5 timing 1414054820.26 
Start Process 87172 with args 6 timing 1414054820.26 
Start Process 87173 with args 7 timing 1414054820.26 
Start Process 87174 with args 8 timing 1414054820.26 
Start Process 87175 with args 9 timing 1414054820.26 
Start Process 87176 with args 10 timing 1414054820.26 
Start Process 87177 with args 11 timing 1414054820.26 
Start Process 87178 with args 12 timing 1414054820.26 
Start Process 87179 with args 13 timing 1414054820.26 
Start Process 87180 with args 14 timing 1414054820.26 
Start Process 87181 with args 15 timing 1414054820.26 
Start Process 87182 with args 16 timing 1414054820.26 
Start Process 87183 with args 17 timing 1414054820.26 
Start Process 87184 with args 18 timing 1414054820.26 
Start Process 87185 with args 19 timing 1414054820.26 
Start Process 87186 with args 20 timing 1414054820.26 
Start Process 87187 with args 21 timing 1414054820.26 
Start Process 87188 with args 22 timing 1414054820.26 
Start Process 87189 with args 23 timing 1414054820.26 
Start Process 87190 with args 24 timing 1414054820.26 
Start Process 87191 with args 25 timing 1414054820.26 
Start Process 87192 with args 26 timing 1414054820.26 
Start Process 87193 with args 27 timing 1414054820.26 
Start Process 87194 with args 28 timing 1414054820.26 
Start Process 87195 with args 29 timing 1414054820.26 
Start Process 87196 with args 30 timing 1414054820.26 
Start Process 87197 with args 31 timing 1414054820.26 
Start Process 87198 with args 32 timing 1414054820.26 
Start Process 87199 with args 33 timing 1414054820.26 
Start Process 87200 with args 34 timing 1414054820.26 
Start Process 87201 with args 35 timing 1414054820.26 
Start Process 87202 with args 36 timing 1414054820.26 
Start Process 87203 with args 37 timing 1414054820.26 
Start Process 87204 with args 38 timing 1414054820.26 
Start Process 87205 with args 39 timing 1414054820.26 
Start Process 87206 with args 40 timing 1414054820.26 
Start Process 87207 with args 41 timing 1414054820.26 
Start Process 87208 with args 42 timing 1414054820.26 
Start Process 87209 with args 43 timing 1414054820.26 
Start Process 87210 with args 44 timing 1414054820.26 
Start Process 87211 with args 45 timing 1414054820.26 
Start Process 87212 with args 46 timing 1414054820.26 
Start Process 87213 with args 47 timing 1414054820.26 
Start Process 87214 with args 48 timing 1414054820.26 
Start Process 87215 with args 49 timing 1414054820.26 
Start Process 87216 with args 50 timing 1414054820.26 
Start Process 87217 with args 51 timing 1414054820.26 
Start Process 87218 with args 52 timing 1414054820.26 
Start Process 87219 with args 53 timing 1414054820.26 
Start Process 87220 with args 54 timing 1414054820.26 
Start Process 87221 with args 55 timing 1414054820.26 
Start Process 87222 with args 56 timing 1414054820.27 
Start Process 87223 with args 57 timing 1414054820.27 
Start Process 87224 with args 58 timing 1414054820.27 
Start Process 87225 with args 59 timing 1414054820.27 
Start Process 87226 with args 60 timing 1414054820.27 
Start Process 87227 with args 61 timing 1414054820.27 
Start Process 87228 with args 62 timing 1414054820.27 
Start Process 87229 with args 63 timing 1414054820.27 
End Process 87166 with args 0 timing 1414054830.27 
End Process 87179 with args 13 timing 1414054830.27 
End Process 87229 with args 63 timing 1414054830.27 
End Process 87225 with args 59 timing 1414054830.27 
End Process 87223 with args 57 timing 1414054830.27 
End Process 87221 with args 55 timing 1414054830.27 
End Process 87219 with args 53 timing 1414054830.27 
End Process 87215 with args 49 timing 1414054830.27 
End Process 87191 with args 25 timing 1414054830.27 
End Process 87195 with args 29 timing 1414054830.27 
End Process 87171 with args 5 timing 1414054830.27 
End Process 87183 with args 17 timing 1414054830.27 
End Process 87189 with args 23 timing 1414054830.27 
End Process 87181 with args 15 timing 1414054830.27 
End Process 87185 with args 19 timing 1414054830.27 
End Process 87217 with args 51 timing 1414054830.27 
End Process 87175 with args 9 timing 1414054830.27 
End Process 87178 with args 12 timing 1414054830.27 
End Process 87213 with args 47 timing 1414054830.27 
End Process 87169 with args 3 timing 1414054830.27 
End Process 87193 with args 27 timing 1414054830.27 
End Process 87209 with args 43 timing 1414054830.27 
End Process 87177 with args 11 timing 1414054830.27 
End Process 87197 with args 31 timing 1414054830.27 
End Process 87187 with args 21 timing 1414054830.27 
End Process 87173 with args 7 timing 1414054830.27 
End Process 87172 with args 6 timing 1414054830.27 
End Process 87201 with args 35 timing 1414054830.27 
End Process 87176 with args 10 timing 1414054830.27 
End Process 87199 with args 33 timing 1414054830.27 
End Process 87205 with args 39 timing 1414054830.27 
End Process 87203 with args 37 timing 1414054830.27 
End Process 87207 with args 41 timing 1414054830.27 
End Process 87224 with args 58 timing 1414054830.27 
End Process 87208 with args 42 timing 1414054830.27 
End Process 87211 with args 45 timing 1414054830.27 
End Process 87212 with args 46 timing 1414054830.27 
End Process 87170 with args 4 timing 1414054830.27 
End Process 87167 with args 1 timing 1414054830.27 
End Process 87204 with args 38 timing 1414054830.27 
End Process 87227 with args 61 timing 1414054830.27 
End Process 87200 with args 34 timing 1414054830.27 
End Process 87216 with args 50 timing 1414054830.27 
End Process 87220 with args 54 timing 1414054830.27 
End Process 87196 with args 30 timing 1414054830.27 
End Process 87192 with args 26 timing 1414054830.27 
End Process 87184 with args 18 timing 1414054830.27 
End Process 87188 with args 22 timing 1414054830.27 
End Process 87180 with args 14 timing 1414054830.27 
End Process 87174 with args 8 timing 1414054830.27 
End Process 87168 with args 2 timing 1414054830.27 
End Process 87182 with args 16 timing 1414054830.27 
End Process 87186 with args 20 timing 1414054830.27 
End Process 87190 with args 24 timing 1414054830.27 
End Process 87202 with args 36 timing 1414054830.27 
End Process 87206 with args 40 timing 1414054830.27 
End Process 87210 with args 44 timing 1414054830.27 
End Process 87198 with args 32 timing 1414054830.27 
End Process 87194 with args 28 timing 1414054830.27 
End Process 87214 with args 48 timing 1414054830.27 
End Process 87228 with args 62 timing 1414054830.27 
End Process 87226 with args 60 timing 1414054830.27 
End Process 87222 with args 56 timing 1414054830.27 
End Process 87218 with args 52 timing 1414054830.27 

Таким образом, Есть две вещи, которые я не понимаю:

• Как я могу запустить эти процессы 64 в параллели ? (так как все начальные и конечные времена одинаковы)
• Каково максимальное значение processes в конструкторе пула?

Answer 1

Ответ 1: Вы смотрите секунды, а не миллисекунды. Повысьте точность.

Ответ 2: Нет ограничений, но он назначит processes/n_processors каждому процессору.

Редактировать: Обратите внимание, что на вашем компьютере работает более 32 процессов, просто используйте команду ps -A. Это не означает, что вы фактически запускаете все параллельно, диспетчер процессов ОС поворачивает очередь процессов для каждого процессора. См. process management для получения дополнительной информации.