Файл журнала парсинга с python pandas

Question

Проблема 1

Я пытаюсь прочитать и проанализировать файл журнала из симуляции. В идеале я хотел бы сделать это с помощью панд, но у меня возникли проблемы. Этот файл называется log, и образец содержится ниже. Теперь я пытаюсь сделать

 import pandas as pd 
    import numpy as np 
    import csv 

    # This method works: 
    results = [] 
    with open('log') as inputfile: 
      for row in csv.reader(inputfile): 
       results.append(row) 


     # This method doesn't work: 
     pd.read_csv('log') 

Ошибка

pandas.parser.CParserError: Error tokenizing data. C error: Expected 1 fields in line 77, saw 4 

Почему это происходит?

Задача 2

Что является наиболее эффективным способом для анализа лог-файл? Имейте в виду, что файл на самом деле будет очень большим. В основном я хочу анализировать остатки для каждого временного шага (каждый шаг времени обозначается Time= x) в массив numpy для построения графика. Например

smoothSolver: Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2 

Я хотел бы добавить 0.813857557031 в массив res_gamma_init

Что является наиболее эффективным способом сделать это?

/*---------------------------------------------------------------------------*\ 
| =========     |             | 
| \\ /F ield   | OpenFOAM: The Open Source CFD Toolbox   | 
| \\ / O peration  | Version: 2.3.x         | 
| \\/ A nd   | Web:  www.OpenFOAM.org      | 
| \\/  M anipulation |             | 
\*---------------------------------------------------------------------------*/ 
Build : 2.3.x-590d57f32fed 
Exec : simpleFoam 
Date : Oct 30 2015 
Time : 17:24:08 
Host : "jack" 
PID : 4034 
Case : /home/jack/OpenFOAM/jack-2.3.x/run/sim 
nProcs : 1 
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE). 
fileModificationChecking : Monitoring run-time modified files using timeStampMaster 
allowSystemOperations : Allowing user-supplied system call operations 

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // 
Create time 

Create mesh for time = 0 

Reading field p 

Reading field U 

Reading/calculating face flux field phi 

Selecting incompressible transport model Newtonian 
Selecting RAS turbulence model kOmegaSST 
Using gammaReThetat-correlations by Langtry and Menter (2009) 
gammaReThetatSSTCoeffs 
{ 
    ca1    2; 
    ce1    1; 
    ca2    0.06; 
    ce2    50; 
    cThetat   0.03; 
    sigmaf   1; 
    sigmaThetat  2; 
    s1    2; 
    dUds   false; 
    alphaK1   0.85034; 
    alphaK2   1; 
    alphaOmega1  0.5; 
    alphaOmega2  0.85616; 
    gamma1   0.5532; 
    gamma2   0.4403; 
    beta1   0.075; 
    beta2   0.0828; 
    betaStar  0.09; 
    a1    0.31; 
    c1    10; 
    kInf   0; 
    omegaInf  0; 
} 

No finite volume options present 


SIMPLE: convergence criteria 
    field p tolerance 1e-06 
    field U tolerance 1e-06 
    field "(k|omega|ReThetaTilda|nut|gamma)"  tolerance 1e-06 


Starting time loop 


forceCoeffs forceCoeffs: 
    Not including porosity effects 

Time = 1 

smoothSolver: Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2 
smoothSolver: Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2 
DICPCG: Solving for p, Initial residual = 1, Final residual = 0.00976261323556, No Iterations 326 
time step continuity errors : sum local = 8.88353600919, global = -0.154492041821, cumulative = -0.154492041821 
smoothSolver: Solving for omega, Initial residual = 1, Final residual = 0.0110458877993, No Iterations 2 
smoothSolver: Solving for k, Initial residual = 0.999999999804, Final residual = 0.022014322402, No Iterations 2 
bounding k, min: 0 max: 0.415754458684 average: 0.410526729282 
smoothSolver: Solving for ReThetatTilda, Initial residual = 1, Final residual = 0.0259070897862, No Iterations 2 
smoothSolver: Solving for gamma, Initial residual = 1, Final residual = 1.51565684699e-05, No Iterations 2 
ExecutionTime = 1.41 s ClockTime = 1 s 

forceCoeffs forceCoeffs output: 
    Cm = 0 
    Cd = -19811.1501384 
    Cl = 4.01861806726e-15 
    Cl(f) = 2.00930903363e-15 
    Cl(r) = 2.00930903363e-15 

Time = 2 

smoothSolver: Solving for Ux, Initial residual = 0.510606164486, Final residual = 0.0151751146151, No Iterations 2 
smoothSolver: Solving for Uy, Initial residual = 0.570639175137, Final residual = 0.0164524779792, No Iterations 2 
DICPCG: Solving for p, Initial residual = 0.160667758287, Final residual = 0.00156013775625, No Iterations 143 
time step continuity errors : sum local = 88.8245306208, global = 11.6476210117, cumulative = 11.4931289699 
smoothSolver: Solving for omega, Initial residual = 0.531389189498, Final residual = 0.00899532326598, No Iterations 2 
smoothSolver: Solving for k, Initial residual = 0.562103650426, Final residual = 0.0138022250836, No Iterations 2 
bounding k, min: 0 max: 0.425403200237 average: 0.411310758596 
smoothSolver: Solving for ReThetatTilda, Initial residual = 0.301017401945, Final residual = 0.00693946297375, No Iterations 2 
smoothSolver: Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2 
ExecutionTime = 2.13 s ClockTime = 2 s 

forceCoeffs forceCoeffs output: 
    Cm = 0 
    Cd = -20283.0256988 
    Cl = 3.99364280461e-15 
    Cl(f) = 1.9968214023e-15 
    Cl(r) = 1.9968214023e-15 

Time = 3 

Answer 1

Чтобы ответить на ваш второй вопрос. Это может помочь, хотя регулярное выражение может быть приятнее.

line1 = 'smoothSolver: Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2' 
line2 = 'smoothSolver: Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2' 
test = [line1, line2] 
import re 
for line in test: 
    if line.startswith('smoothSolver'): 
     print re.findall(r'[residual = ][\d][.][\d]+', line) 

В результате в консоли:

[' 0.999999999388', ' 0.00692443749034'] 
[' 0.999999994742', ' 0.0027182321684']