Author |
Topic  |
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 20 September 2003 : 11:32:53
|
Huh. It appears that this archive actually DID finish. Why the *exact same* routine I've run multiple times to no avail worked this time is beyond me. Might have been less of a server load this time or something... ? I woke up this morning and found the range to have been archived. So, my script doesn't contain the completion...and was cut short at 30 minutes. I'll try again tonight for the next month's worth of topics and see what happens. <sigh> |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
Posted - 20 September 2003 : 15:45:12
|
Hmmm... that would probably mean that server load may have some effect on what is happening. That was my initial guess. Anyway I want to know where does the script get stuck, to see if we can overcome that. |
Snitz 3.4 Readme | Like the support? Support Snitz too |
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 20 September 2003 : 18:09:21
|
So...I just created a new DB and was going to archive on THAT one so I could get a trace that was 100% clean. Why is it that the filter...
DatabaseName ---Like ------BLAHBLAHBLAH
...still returns queries from my forum database? I can put whatever garbage string I want in that field, and it does NOT filter correctly at all. Seems to just return data queried from any database. If I can fix this, I'll have a completely clean trace with only archive queries in it.
??? <confused> ??? |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 20 September 2003 : 18:21:27
|
Bizarre. The DatabaseName field apparently comes in blank no matter what database is used. <confused look>
Anyway, I figured out the DatabaseID of the new DB and am running an archive on it now. So, we'll have nothing but archive queries to look at when it gets done... |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
|
Jeepaholic
Average Member
  
USA
697 Posts |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
Posted - 22 September 2003 : 14:45:57
|
Got the trace and it seems clear to me why the script times out. Basically you have about 4860 rows of replies and the script was navigating through all these replies, accessing them through a recordset and navigating the recordset. This navigation is known to be slow.
The only way out for this, as I see it, would be to change the code. Probably using getrows can help speed up the navigation within the recordset, though I don't really know whether than won't require a lot of server memory.
Al, do you think you can change the code to use getRows? I think it can have a dramatic effect on the speed of the script, specially in big forums like yours... I'm in a bit of a squeeze right now, so I can't make any promises on when I'll be able to handle this change. |
Snitz 3.4 Readme | Like the support? Support Snitz too |
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 22 September 2003 : 14:55:46
|
Wouldn't have a clue where to begin, Rui. I know a lot more about .NET than I do about classic ASP methods, so I'm kinda confused by this stuff.
Is there any reason it doesn't just do a full query of the old posts/replies, retrieve a single recordset - and insert the entire recordset into the archive table? Maybe even using a temp table or something? I could run a manual query in that case, then just have it update the counts afterwards.
Thoughts? |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
Posted - 22 September 2003 : 17:54:36
|
Al, I guess that could be done, indeed. You would need to use an INSERT INTO statement that also used a SELECT statement to select the relevant records from the FORUM_REPLY table. Of course, then you would need to move the topics individually as well and, after that, DELETE the topics and replies from the live tables. After that you'd need to update counts.
Need help with the INSERT INTO statement?
P.S.: Of course, this could also be done from the code, but again you'd need a different script for Access/SQL Server and MySQL, though I guess you'd probably not worry too much about it. The script could be changed to do this, IMO. It would probably be fun to do it, just can do it now. I can help with the SQL statements though. |
Snitz 3.4 Readme | Like the support? Support Snitz too |
Edited by - ruirib on 22 September 2003 17:56:58 |
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 22 September 2003 : 21:06:46
|
I'm pretty good with SQL, I just don't know squat about ASP's getRows or navigation of recordsets. I'll mess with QA and see what I can come up with...
So, to clarify:
1) Copy Replies 2) Delete Replies 3) Copy Topics 4) Delete Topics 5) Update Counts (which times out too, btw...haha)
Thanks Rui, I'll post what I come up with. |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 22 September 2003 : 22:54:56
|
Ok Rui, poke some holes in this. I've done some pretty extensive testing and think I've nailed it. I did not include modifying the counts, haven't gotten to that yet. This simply does steps #1-#4 listed above. Find bugs!
BTW, this code archived 310 TOPICS (with varying replies) in 30 seconds.
DECLARE
@ArchiveDate nvarchar(14),
@ArchiveForumID int
SET
@ArchiveDate = '20030701000000'
SET
@ArchiveForumID = 2
-- Create temp table that will hold TOPIC_ID's to archive
CREATE TABLE #TOPIC_IDs
(
TOPIC_ID int
)
-- Fill temp table with TOPIC_ID's to archive
INSERT INTO
#TOPIC_IDs
-- Query finds topics started (and edited) before archive date
SELECT
TOPIC_ID
FROM
FORUM_TOPICS AS FT
WHERE
T_DATE < @ArchiveDate
AND
-- If no last edit date (NULL), return lowest date possible
ISNULL(T_LAST_EDIT,'00000000000000') < @ArchiveDate
AND
FORUM_ID = @ArchiveForumID
AND
-- Subquery to determine if any replies (or edits) are
-- AFTER the archive date. If so, do not archive topic.
(SELECT
COUNT(*)
FROM
FORUM_REPLY AS FR
WHERE
(R_DATE >= @ArchiveDate
OR
-- If no last edit date (NULL), return lowest date possible
ISNULL(R_LAST_EDIT,'00000000000000') >= @ArchiveDate)
AND
FR.TOPIC_ID = FT.TOPIC_ID
) = 0
-- Copy TOPIC's to archive table
INSERT INTO
FORUM_A_TOPICS
(CAT_ID,
FORUM_ID,
TOPIC_ID,
T_STATUS,
T_MAIL,
T_SUBJECT,
T_MESSAGE,
T_AUTHOR,
T_REPLIES,
T_UREPLIES,
T_VIEW_COUNT,
T_LAST_POST,
T_DATE,
T_LAST_POSTER,
T_IP,
T_LAST_POST_AUTHOR,
T_LAST_POST_REPLY_ID,
T_ARCHIVE_FLAG,
T_LAST_EDIT,
T_LAST_EDITBY,
T_STICKY,
T_SIG,
T_EVENT_DATE,
T_ISEVENT)
SELECT
CAT_ID,
FORUM_ID,
TOPIC_ID,
T_STATUS,
T_MAIL,
T_SUBJECT,
T_MESSAGE,
T_AUTHOR,
T_REPLIES,
T_UREPLIES,
T_VIEW_COUNT,
T_LAST_POST,
T_DATE,
T_LAST_POSTER,
T_IP,
T_LAST_POST_AUTHOR,
T_LAST_POST_REPLY_ID,
T_ARCHIVE_FLAG,
T_LAST_EDIT,
T_LAST_EDITBY,
T_STICKY,
T_SIG,
T_EVENT_DATE,
T_ISEVENT
FROM
FORUM_TOPICS
WHERE
TOPIC_ID
IN
(SELECT
TOPIC_ID
FROM
#TOPIC_IDs)
-- Copy REPLY's to archive table
INSERT INTO
FORUM_A_REPLY
(CAT_ID,
FORUM_ID,
TOPIC_ID,
REPLY_ID,
R_MAIL,
R_AUTHOR,
R_MESSAGE,
R_DATE,
R_IP,
R_STATUS,
R_LAST_EDIT,
R_LAST_EDITBY,
R_SIG)
SELECT
CAT_ID,
FORUM_ID,
TOPIC_ID,
REPLY_ID,
R_MAIL,
R_AUTHOR,
R_MESSAGE,
R_DATE,
R_IP,
R_STATUS,
R_LAST_EDIT,
R_LAST_EDITBY,
R_SIG
FROM
FORUM_REPLY
WHERE
TOPIC_ID
IN
(SELECT
TOPIC_ID
FROM
#TOPIC_IDs)
-- Delete TOPIC's from main table
DELETE
FROM
FORUM_TOPICS
WHERE
TOPIC_ID
IN
(SELECT
TOPIC_ID
FROM
#TOPIC_IDs)
-- Delete REPLY's from main table
DELETE
FROM
FORUM_REPLY
WHERE
TOPIC_ID
IN
(SELECT
TOPIC_ID
FROM
#TOPIC_IDs)
DROP TABLE #TOPIC_IDs
-- TO DO: UPDATE FORUM COUNTS!!
GO
|
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
Edited by - Jeepaholic on 28 September 2003 15:02:58 |
 |
|
ruirib
Snitz Forums Admin
    
Portugal
26364 Posts |
Posted - 23 September 2003 : 05:59:48
|
Ok Al, I'll have a busy day today, will try to have a look at this later on. A quick look makes think the insertion code could be simpler, but I'll go through that tonight.
Anyway, at first sight, looks that it is doing what it should. Seems like the archival went as expected as well so, even if you can make it simpler, this one does work and that's what ya want, right ? |
Snitz 3.4 Readme | Like the support? Support Snitz too |
 |
|
Jeepaholic
Average Member
  
USA
697 Posts |
Posted - 23 September 2003 : 13:26:42
|
Yep. I'm just hoping that I didn't miss something... My first version didn't take the Edited Date into account, and I was lucky to find that one in my testing... SOooo, just looking for a couple more sets of eyes before I hit my production forum with it.
Eventually, I'd love to see how you'd improve my code!  |
Al Bsharah Aholics.com
Jeepaholics Anonymous Broncoholics Anonymous Network Insight
|
 |
|
Topic  |
|
|
|