Как сопоставить записи между двумя столами истории?

Question

У меня есть 5 таблиц (изобретенные и упрощенных для вопроса):

Device 
--------------- 
DeviceID - INT IDENTITY - PRIMARY KEY 
DeviceGUID - UNIQUEIDENTIFIER - UNIQUE CLUSTERED (used for lookups) 
OtherInfo 


Device_State 
--------------- 
DeviceStateID - INT IDENTITY - PRIMARY KEY 
DeviceID - INT - FOREIGN KEY - UNIQUE CLUSTERED (used for lookups) 
Pump1 - TINYINT 
Pump2 - TINYINT 
Pump3 - TINYINT 
Pump4 - TINYINT 
Pump5 - TINYINT 
Heater1 - TINYINT 
Heater2 - TINYINT 
Lights - BIT 
TimeStamp - SMALLDATETIME 

Device_State_History 
--------------- 
DeviceStateHistoryID - INT IDENTITY - PRIMARY KEY 
DeviceStateID - INT - FOREIGN KEY - NONCLUSTERED INDEX (used for lookups) 
Pump1 - TINYINT 
Pump2 - TINYINT 
Pump3 - TINYINT 
Pump4 - TINYINT 
Pump5 - TINYINT 
Heater1 - TINYINT 
Heater2 - TINYINT 
Lights - BIT 
TimeStamp - SMALLDATETIME 


Peripheral_State 
---------------- 
PeripheralStateID - INT IDENTITY - PRIMARY KEY 
DeviceID - INT - FOREIGN KEY - UNIQUE CLUSTERED (used for lookups) 
Pump1 - TINYINT 
Pump2 - TINYINT 
PH - SMALLINT 
ORP - SMALLINT 
ELECTRODEID - TINYINT 
ELECTRODEPOLARITY - TINYINT 
TimeStamp - SMALLDATETIME 


Peripheral_State_History 
---------------- 
PeripheralStateHistoryID - INT IDENTITY - PRIMARY KEY 
PeripheralStateID - INT - FOREIGN KEY - NONCLUSTERED INDEX (used for lookups) 
Pump1 - TINYINT 
Pump2 - TINYINT 
PH - SMALLINT 
ORP - SMALLINT 
ELECTRODEID - TINYINT 
ELECTRODEPOLARITY - TINYINT 
TimeStamp - SMALLDATETIME 

Вот ситуация:

У меня есть ~ 3,8 миллиона записей в Device_State_History таблице и у меня есть ~ 100K записей в Peripheral_State_History Таблица.

Я хотел бы написать запрос, который выбирает все Peripheral_State_history и Device_State_History для данного DeviceGUID между датой даты дат. Я хотел бы получить записи, которые имеют Somthing так:

Select 
Peripheral_State_History.PH, 
Peripheral_State_History.ORP, 
( 
if(Device_State_History.Pump1 == 1 || 
    Device_State_History.Pump2 == 1 || 
    Device_State_History.Pump3 == 1 || 
    Device_State_History.Pump4 == 1 || 
    Device_State_History.Pump5 == 1 
    ) 
    { 1 } 
    else 
    { 0 } 
) AS PumpsOn, 
    TimeStamp 
FROM CombinedData 

Кроме того, Timestamps между Device_State_History и Peripheral_State_History не всегда одинаковы. Под этим я имею в виду, что временные метки записи для 1 устройства в таблице Device_State_History может быть такой:

Mar-1-2013 12:31 
Mar-1-2013 12:33 
Mar-1-2013 12:36 
Mar-1-2013 12:38 
Mar-1-2013 12:41 

И тогда будет записей в Peripheral_State_History таблице:

Mar-1-2013 12:29 
Mar-1-2013 12:33 
Mar-1-2013 12:34 
Mar-1-2013 12:38 
Mar-1-2013 12:39 
Mar-1-2013 12:41 

Как вы можете видеть, временные метки не всегда накладываются.

Я читал об этом и принимал во внимание запись хранимой процедуры для этого, однако это выходит за рамки моих возможностей SQL, руководство и примеры действительно оценены. Спасибо.

EDIT:

Следующая хранимая процедура я написал, чтобы получить данные из таблицы Peripheral_State_History. Он не включает данные Pump, которые я хочу получить из таблицы Device_State_History.

Пара примечаний: я получаю только 1 запись за отметку времени, так как есть несколько марок времени с несколькими записями. Эта хранимая процедура захватывает текущую запись из таблицы состояний, а также все записи из таблицы истории.

USE my_db 
GO 
IF EXISTS (SELECT * FROM sys.objects WHERE type = 'P' AND name = 'SP_Select_Stuff') 
DROP PROCEDURE 'SP_Select_Stuff') 
GO 
CREATE PROCEDURE 'SP_Select_Stuff') 
( 
    @DeviceGUID uniqueidentifier 
) 
AS 
DECLARE @RetCode INT 
;WITH combined AS 
(
    SELECT 
    ROW_NUMBER() OVER 
    (PARTITION BY [TimeStamp] ORDER BY [TimeStamp]) as num, ORP, PH, [TimeStamp] 
    FROM Peripheral_State 
    INNER JOIN Device 
     ON Peripheral_State.DeviceID = Device.DeviceID 
     WHERE Device.DeviceGUID = @DeviceGUID 
    UNION ALL 
    SELECT 
    ROW_NUMBER() OVER 
    (PARTITION BY [TimeStamp] ORDER BY [TimeStamp]) as num, ORP, PH, [TimeStamp] 
    FROM Peripheral_State_History 
    INNER JOIN Device 
     ON Peripheral_State.DeviceID = Device.DeviceID 
     WHERE Device.DeviceGUID = @DeviceGUID 

) 
SELECT ORP, PH, [TimeStamp] 
FROM combined 
WHERE ORP > 0 
AND PH > 0 
AND combined.num = 1 
ORDER BY [TimeStamp] ASC 

IF @@ERROR <> 0 
    BEGIN 
     SET @RetCode = 5 
     RETURN @RetCode 
    END 
ELSE 
    BEGIN 
     SET @RetCode = 0 
     RETURN @RetCode 
    END 
GO 

EDIT:

Вот что я придумал.

USE my_db 
--"combined" gets all of the data. 
--A row from the peripheral history will look like: ORP-Value, PH-Value, Current-Value, NULL, TimeStamp. 
--A row from the device history will look like: NULL, NULL, NULL, PumpsOn-Value, TimeStamp 
-- I also get a num for filtering 1 TimeStamp per minute. My TimeStamps are accurate to the minute, so if one minute has 30 records, I will use the first record. 
;WITH combined AS 
( 
    SELECT 
    ROW_NUMBER() OVER 
    (PARTITION BY ph.[TimeStamp] ORDER BY ph.[TimeStamp]) as num, ph.ORP, ph.PH, ph.[Current], NULL as PumpsOn, ph.[TimeStamp] 
    FROM Device as d 

    LEFT JOIN Peripheral_State as ps 
    ON ps.DeviceID = d.DeviceID 

     LEFT JOIN Peripheral_State_History as ph 
     ON ph.DeviceID = d.DeviceID 

     WHERE d.DeviceGUID = @DeviceGUID 
     AND ph.TimeStamp BETWEEN '2013-03-02 00:00:00' AND GETDATE() 

    UNION ALL 

    SELECT 
    ROW_NUMBER() OVER 
    (PARTITION BY dh.[TimeStamp] ORDER BY dh.[TimeStamp]) as num, NULL as ORP, NULL as PH, NULL as [Current], IsNull(COALESCE(dh.Pump1, dh.Pump2, dh.Pump3, dh.Pump4, dh.Pump5),0) as PumpsOn, dh.[TimeStamp] 
    FROM Device as d 

    LEFT JOIN Device_State as ds 
    ON ds.DeviceID = d.DeviceID 

     LEFT JOIN Device_State_History as dh 
     ON dh.DeviceID = d.DeviceID 

     WHERE d.DeviceGUID = @DeviceGUID 
     AND ph.TimeStamp BETWEEN '2013-03-02 00:00:00' AND GETDATE()    
), 
--"FilteredAndAddID" gets a record set that has an ID added to the row. This is required for the 3rd step. 
--It also filters the data so that records with 0 orp/ph are omitted. 
FilteredAndAddID AS 
(
    SELECT 
     ROW_NUMBER() OVER (ORDER BY [TimeStamp]) as ID, 
     ORP, 
     PH, 
     [Current], 
     PumpsOn, 
     [TimeStamp] 
    FROM combined 
    WHERE 
    (ORP is NULL OR ORP > 0) AND (PH is NULL OR PH > 0) 
    AND combined.num = 1 
), 
--"filled" is used to fill the "FilteredAddID" record set NULLS with the previous row that has a value instead of NULL. 
filled AS 
(
    SELECT Curr.ID, 
      ISNULL(Curr.ORP, (SELECT TOP 1 ORP FROM FilteredAddID WHERE ID < Curr.ID AND ORP IS NOT NULL ORDER BY ID)) ORP, 
      ISNULL(Curr.PH, (SELECT TOP 1 PH FROM FilteredAddID WHERE ID < Curr.ID AND PH IS NOT NULL ORDER BY ID)) PH, 
      ISNULL(Curr.[Current], (SELECT TOP 1 [Current] FROM FilteredAddID WHERE ID < Curr.ID AND [Current] IS NOT NULL ORDER BY ID)) [Current], 
      ISNULL(Curr.PumpsOn, (SELECT TOP 1 PumpsOn FROM FilteredAddID WHERE ID < Curr.ID AND PumpsOn IS NOT NULL ORDER BY ID)) PumpsOn, 
      Curr.[TimeStamp] 
    FROM FilteredAddID Curr 
) 
SELECT ID, ORP, PH, [Current], PumpsOn, [TimeStamp] 
FROM filled 
ORDER BY [TimeStamp] ASC 

Первые 2 запросы «комбинированные» и «FilterAndAddID» выполнить в миллисекундах, однако «заполненный» запрос занимает примерно 6 минут, чтобы заполнить пустые значения в ~ 1700 записей.

Есть ли способ ускорить это?

Я подхожу к этому правильным способом?

Идеально мне нужно, чтобы он работал в считанные секунды.

Answer 1

Как долго работает этот запрос?

;with combined as 
(
select 
    psh.PH, psh.ORP, psh.TimeStamp, 
    max(dsh.TimeStamp) as Device_Stamp 
from Peripheral_State_History psh 
    join Peripheral_State ps on ps.PeripheralStateID = psh.PeripheralStateID 
    join Device d on d.DeviceID = ps.DeviceID 
    left join Device_State ds on ds.DeviceID = d.DeviceID 
    left join Device_State_History dsh on dsh.DeviceStateID = ds.DeviceStateID 
    and dsh.TimeStamp <= psh.TimeStamp 
where d.DeviceGUID = @DeviceGUID 
    and psh.TimeStamp between @date1 and @date2 
group by psh.PH, psh.ORP, psh.TimeStamp 
) 
select 
    c.PH, c.ORP, c.TimeStamp, 
    case 
    when dsh.Pump1 = 1 and dsh.Pump2 = 1 and dsh.Pump3 = 1 
     and dsh.Pump4 = 1 and dsh.Pump5 = 1 
    then 1 else 0 end as PumpsOn 
from combined c 
    left join Device_State_History dsh on dsh.TimeStamp = c.Device_Stamp 
    left join Device_State ds on ds.DeviceStateID = dsh.DeviceStateID 
    left join Device d on ds.DeviceID = d.DeviceID 
where d.DeviceGUID = @DeviceGUID 

EDIT

Решение проблемы с несколькими записями с той же отметкой времени в Device_State_History.

Если у вас есть для большей DeviceStateHistoryID всегда больше или равно timestamp, вы можете попробовать этот запрос. Он выбирает одну запись с максимальным DeviceStateHistoryID, где записи имеют ту же метку времени в Device_State_History

;with combined as 
(
select 
    psh.PH, psh.ORP, psh.TimeStamp, 
    max(dsh.DeviceStateHistoryID) as DeviceStateHistoryID 
from Peripheral_State_History psh 
    join Peripheral_State ps on ps.PeripheralStateID = psh.PeripheralStateID 
    join Device d on d.DeviceID = ps.DeviceID 
    left join Device_State ds on ds.DeviceID = d.DeviceID 
    left join Device_State_History dsh on dsh.DeviceStateID = ds.DeviceStateID 
    and dsh.TimeStamp <= psh.TimeStamp 
where d.DeviceGUID = @DeviceGUID 
    and psh.TimeStamp between @date1 and @date2 
group by psh.PH, psh.ORP, psh.TimeStamp 
) 
select 
    c.PH, c.ORP, c.TimeStamp, 
    case 
    when dsh.Pump1 = 1 and dsh.Pump2 = 1 and dsh.Pump3 = 1 
     and dsh.Pump4 = 1 and dsh.Pump5 = 1 
    then 1 else 0 end as PumpsOn 
from combined c 
    left join Device_State_History dsh on dsh.DeviceStateHistoryID = c.DeviceStateHistoryID 
    left join Device_State ds on ds.DeviceStateID = dsh.DeviceStateID 
    left join Device d on ds.DeviceID = d.DeviceID 
where d.DeviceGUID = @DeviceGUID