фильтра Значения, которые больше, чем AVG значение в Свиньи/Hive

Question

Это моя выборка данных:

+---------------------------+------+-----------+--------------+------------+--------+--------------+-------+------------------+ 
|   Car   | MPG | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model |  Origin  | 
+---------------------------+------+-----------+--------------+------------+--------+--------------+-------+------------------+ 
| Chevrolet Chevelle Malibu | 18.0 |   8 | 307.0  | 130.0  | 3504 | 12.0   | 70 | US Buick   | 
| Skylark 320    | 15.0 |   8 | 350.0  | 165.0  | 3693 | 11.5   | 70 | US Plymouth  | 
| Satellite     | 18.0 |   8 | 318.0  | 150.0  | 3436 | 11.0   | 70 | US AMC Rebel  | 
| SST      | 16.0 |   8 | 304.0  | 150.0  | 3433 | 12.0   | 70 | US Ford   | 
| Torino     | 17.0 |   8 | 302.0  | 140.0  | 3449 | 10.5   | 70 | US Ford Galaxie | 
| 500      | 15.0 |   8 | 429.0  | 198.0  | 4341 | 10.0   | 70 | US Chevrolet  | 
| Impala     | 14.0 |   8 | 454.0  | 220.0  | 4354 | 9.0   | 70 | US Plymouth Fury | 
| iii      | 14.0 |   8 | 440.0  | 215.0  | 4312 | 8.5   | 70 | US    | 
+---------------------------+------+-----------+--------------+------------+--------+--------------+-------+------------------+ 

Я хочу, чтобы выяснить эти MPG и HorsePower на основе каждого автомобиля, значения которых превышают их AVG стоимость. Как mpg> AVG (mpg) и HorsePower> AVG (HorsePower).

Что я сделал:

r = load '/user/CarData/cars.csv' using PigStorage(',') as (car:chararray,mpg:float,cyl:INT,disp:DOUBLE,hp:DOUBLE,weight:INT,acc:DOUBLE,model:INT,org:chararray); 
r1 = group r by car; 
r2 = foreach r1 generate group,AVG(r.mpg) as avg_mpg,AVG(r.hp) as avg_hp,r.mpg,r.hp; 

Это даст мне carname, средний и мешок {} миль на галлон, теперь я перед проблемой фильтрации от r2. Я пытаюсь что-то вроде:

FILTER r2 by r.mpg > AVG(mpg) and r.hp > AVG(hp)

Пожалуйста, помогите мне. Благодаря

Answer 1

В улье, было бы что-то вроде

Select Car, MPG, Cylinders, Displacement, Horsepower, Weight, Acceleration, Model, Origin 
FROM cars_table 
JOIN (Select AVG(mpg) as a_m, AVG(hp) as a_h) averages ON (1 = 1) 
WHERE Horsepower > a_h AND MPG > a_m; 

Answer 2

Вход:

Chevrolet Chevelle Malibu,18.0,8,307.0,130.0,3504,12.0,70,US Buick 
Skylark 320,15.0,8,350.0,165.0,3693,11.5,70,US Plymouth 
Satellite,18.0,8,318.0,150.0,3436,11.0,70,US AMC Rebel 
SST,16.0,8,304.0,150.0,3433,12.0,70,US Ford 
Torino,17.0,8,302.0,140.0,3449,10.5,70,US Ford Galaxie 
500,15.0,8,429.0,2.0,4341,10.0,70,US Chevrolet 
500,45.0,8,429.0,198.0,4341,10.0,70,US Chevrolet 
500,10.0,8,429.0,40.0,4341,10.0,70,US Chevrolet 
Impala,14.0,8,454.0,220.0,4354,9.0,70,US Plymouth Fury 
iii,14.0,8,440.0,215.0,4312,8.5,70,US 

Код:

r = load 'test.data' using PigStorage(',') as car:chararray,mpg:float,cyl:INT,disp:DOUBLE,hp:DOUBLE,weight:INT,acc:DOUBLE,mod el:INT,org:chararray); 
r1 = group r by car; 
r2 = foreach r1 generate FLATTEN(group) as (car_grp:chararray),(float)AVG(r.mpg) as (avg_mpg:float), 
          (DOUBLE)AVG(r.hp) as (avg_hp:DOUBLE); 
j = JOIN r2 BY car_grp, r BY car; 
r3 = foreach j generate r2::car_grp as (car:chararray),r::mpg as (mpg:float),r::cyl as (cyl:INT),r::disp as (disp:DOUBLE),r::hp as (hp:DOUBLE),r::weight as (weight:INT),r::acc as (acc:DOUBLE),r::model as (model:INT),r::org as (org:chararray),r2::avg_mpg as (avg_mpg:float),r2::avg_hp as (avg_hp:DOUBLE); 
r4 = FILTER r3 BY mpg > avg_mpg AND hp > avg_hp; 

Выход:

(500,45.0,8,429.0,198.0,4341,10.0,70,US Chevrolet,23.333334,80.0) 

Answer 3

Как и в приведенных выше ответах, вам не нужно присоединяться к таблицам. Я считаю, что это будет оптимизированная версия.

Входные данные:

Chevrolet Chevelle Malibu,18.0,8,307.0,130.0,3504,12.0,70,US Buick 
Skylark 320,15.0,8,350.0,165.0,3693,11.5,70,US Plymouth 
Satellite,18.0,8,318.0,150.0,3436,11.0,70,US AMC Rebel 
SST,16.0,8,304.0,150.0,3433,12.0,70,US Ford 
Torino,17.0,8,302.0,140.0,3449,10.5,70,US Ford Galaxie 
500,15.0,8,429.0,198.0,4341,10.0,70,US Chevrolet 
Impala1,14.0,8,454.0,220.0,4354,9.0,70,US Plymouth Fury 
Impala2,25.0,8,454.0,270.0,4354,9.0,70,US Plymouth Fury 
Impala3,30.0,8,454.0,290.0,4354,9.0,70,US Plymouth Fury 
iii,14.0,8,440.0,215.0,4312,8.5,70,US 

Pig Сценарий:

input_data = LOAD '/pigsamples/carinfo' USING PigStorage (',') 
      AS (car:CHARARRAY, mpg:FLOAT, cyl:INT, disp:DOUBLE, hp:DOUBLE, weight:INT, acc:DOUBLE, model:INT, org:CHARARRAY); 

group_data = GROUP input_data ALL; 
average_values = FOREACH group_data GENERATE AVG(input_data.mpg) AS avg_mpg, AVG(input_data.hp) AS avg_hp; 
filter_data = FILTER input_data BY mpg > average_values.avg_mpg AND hp > average_values.hp; 

Выход:

(Impala2,25.0,8,454.0,270.0,4354,9.0,70,US Plymouth Fury) 
(Impala3,30.0,8,454.0,290.0,4354,9.0,70,US Plymouth Fury) 

Answer 4

Я изменил mpg для Impala и iii - 19.0, чтобы запрос возвращал что-то. Вы хотите избежать самообучения здесь; его можно эффективно выполнить с помощью hive windowing functions.

Hive:

select car, mpg, avg_mpg, horsepower, avg_hrspwr 
from (
    select car, mpg, horsepower 
    , avg(mpg) over() as avg_mpg 
    , avg(horsepower) over() as avg_hrspwr 
    from db.table) x 
where horsepower > avg_hrspwr and mpg > avg_mpg 

Выход:

Impala 19.0 17.125 220.0 171.0 
iii  19.0 17.125 215.0 171.0 

Насколько Pig обеспокоен я думаю, решение @Sai Киран Neelakantam является довольно твердым.

Answer 5

r = load 'A' using PigStorage(',') as (car:chararray,mpg:float,cyl:INT,disp:DOUBLE,hp:DOUBLE,weight:INT,acc:DOUBLE,model:INT,org:chararray); 
r1 = group r by car; 
r2 = foreach r1 generate group,FLATTEN(AVG(r.mpg)) as avg_mpg,AVG(r.hp) as avg_hp,FLATTEN(r.mpg) as mpg ,FLATTEN(r.hp) as hp; 
r3 = FILTER r2 by mpg > avg_mpg and hp > avg_hp; 
r3 = distinct r3; 
dump r3;