Snitz Forums 2000
Snitz Forums 2000
Home | Profile | Register | Active Topics | Members | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Help Groups for Snitz Forums 2000 Users
 Help: Database: MS SQL Server
 Faster MS SQL Archive Script (BETA!)
 New Topic  Topic Locked
 Printer Friendly
Previous Page | Next Page
Author Previous Topic Topic Next Topic
Page: of 3

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 20 September 2003 :  09:08:42  Show Profile  Send ruirib a Yahoo! Message
Email sent.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 20 September 2003 :  11:32:53  Show Profile  Visit Jeepaholic's Homepage
Huh. It appears that this archive actually DID finish. Why the *exact same* routine I've run multiple times to no avail worked this time is beyond me. Might have been less of a server load this time or something... ? I woke up this morning and found the range to have been archived. So, my script doesn't contain the completion...and was cut short at 30 minutes. I'll try again tonight for the next month's worth of topics and see what happens. <sigh>

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 20 September 2003 :  15:45:12  Show Profile  Send ruirib a Yahoo! Message
Hmmm... that would probably mean that server load may have some effect on what is happening. That was my initial guess. Anyway I want to know where does the script get stuck, to see if we can overcome that.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 20 September 2003 :  18:09:21  Show Profile  Visit Jeepaholic's Homepage
So...I just created a new DB and was going to archive on THAT one so I could get a trace that was 100% clean. Why is it that the filter...

DatabaseName
---Like
------BLAHBLAHBLAH

...still returns queries from my forum database? I can put whatever garbage string I want in that field, and it does NOT filter correctly at all. Seems to just return data queried from any database. If I can fix this, I'll have a completely clean trace with only archive queries in it.

??? <confused> ???

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 20 September 2003 :  18:21:27  Show Profile  Visit Jeepaholic's Homepage
Bizarre. The DatabaseName field apparently comes in blank no matter what database is used. <confused look>

Anyway, I figured out the DatabaseID of the new DB and am running an archive on it now. So, we'll have nothing but archive queries to look at when it gets done...

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 20 September 2003 :  20:11:34  Show Profile  Send ruirib a Yahoo! Message
okie dokie. Email me the data when you're done.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 20 September 2003 :  21:59:53  Show Profile  Visit Jeepaholic's Homepage
Email sent...

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 21 September 2003 :  11:37:03  Show Profile  Send ruirib a Yahoo! Message
I've been unable to login to the site. Have a look at what may be wrong and let me know.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 22 September 2003 :  14:45:57  Show Profile  Send ruirib a Yahoo! Message
Got the trace and it seems clear to me why the script times out. Basically you have about 4860 rows of replies and the script was navigating through all these replies, accessing them through a recordset and navigating the recordset. This navigation is known to be slow.

The only way out for this, as I see it, would be to change the code. Probably using getrows can help speed up the navigation within the recordset, though I don't really know whether than won't require a lot of server memory.

Al, do you think you can change the code to use getRows? I think it can have a dramatic effect on the speed of the script, specially in big forums like yours... I'm in a bit of a squeeze right now, so I can't make any promises on when I'll be able to handle this change.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 22 September 2003 :  14:55:46  Show Profile  Visit Jeepaholic's Homepage
Wouldn't have a clue where to begin, Rui. I know a lot more about .NET than I do about classic ASP methods, so I'm kinda confused by this stuff.

Is there any reason it doesn't just do a full query of the old posts/replies, retrieve a single recordset - and insert the entire recordset into the archive table? Maybe even using a temp table or something? I could run a manual query in that case, then just have it update the counts afterwards.

Thoughts?

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 22 September 2003 :  17:54:36  Show Profile  Send ruirib a Yahoo! Message
Al, I guess that could be done, indeed. You would need to use an INSERT INTO statement that also used a SELECT statement to select the relevant records from the FORUM_REPLY table. Of course, then you would need to move the topics individually as well and, after that, DELETE the topics and replies from the live tables. After that you'd need to update counts.

Need help with the INSERT INTO statement?

P.S.: Of course, this could also be done from the code, but again you'd need a different script for Access/SQL Server and MySQL, though I guess you'd probably not worry too much about it. The script could be changed to do this, IMO. It would probably be fun to do it, just can do it now. I can help with the SQL statements though.


Snitz 3.4 Readme | Like the support? Support Snitz too

Edited by - ruirib on 22 September 2003 17:56:58
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 22 September 2003 :  21:06:46  Show Profile  Visit Jeepaholic's Homepage
I'm pretty good with SQL, I just don't know squat about ASP's getRows or navigation of recordsets. I'll mess with QA and see what I can come up with...

So, to clarify:

1) Copy Replies
2) Delete Replies
3) Copy Topics
4) Delete Topics
5) Update Counts (which times out too, btw...haha)

Thanks Rui, I'll post what I come up with.

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 22 September 2003 :  22:54:56  Show Profile  Visit Jeepaholic's Homepage
Ok Rui, poke some holes in this. I've done some pretty extensive testing and think I've nailed it. I did not include modifying the counts, haven't gotten to that yet. This simply does steps #1-#4 listed above. Find bugs!

BTW, this code archived 310 TOPICS (with varying replies) in 30 seconds.


DECLARE
	@ArchiveDate nvarchar(14),
	@ArchiveForumID int

SET
	@ArchiveDate = '20030701000000'
SET
	@ArchiveForumID = 2


-- Create temp table that will hold TOPIC_ID's to archive
CREATE TABLE #TOPIC_IDs
(
	TOPIC_ID int
)

-- Fill temp table with TOPIC_ID's to archive
INSERT INTO
	#TOPIC_IDs
-- Query finds topics started (and edited) before archive date
SELECT
	TOPIC_ID
FROM
	FORUM_TOPICS AS FT
WHERE
	T_DATE < @ArchiveDate
AND
	-- If no last edit date (NULL), return lowest date possible
	ISNULL(T_LAST_EDIT,'00000000000000') < @ArchiveDate
	
AND
	FORUM_ID = @ArchiveForumID
AND
	-- Subquery to determine if any replies (or edits) are 
	--  AFTER the archive date.  If so, do not archive topic.
	(SELECT
		COUNT(*)
	FROM
		FORUM_REPLY AS FR
	WHERE
		(R_DATE >= @ArchiveDate
		OR
		-- If no last edit date (NULL), return lowest date possible
		ISNULL(R_LAST_EDIT,'00000000000000') >= @ArchiveDate)
	AND
		FR.TOPIC_ID = FT.TOPIC_ID
	) = 0


-- Copy TOPIC's to archive table
INSERT INTO
	FORUM_A_TOPICS
	(CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	T_STATUS,
	T_MAIL,
	T_SUBJECT,
	T_MESSAGE,
	T_AUTHOR,
	T_REPLIES,
	T_UREPLIES,
	T_VIEW_COUNT,
	T_LAST_POST,
	T_DATE,
	T_LAST_POSTER,
	T_IP,
	T_LAST_POST_AUTHOR,
	T_LAST_POST_REPLY_ID,
	T_ARCHIVE_FLAG,
	T_LAST_EDIT,
	T_LAST_EDITBY,
	T_STICKY,
	T_SIG,
	T_EVENT_DATE,
	T_ISEVENT)
SELECT
	CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	T_STATUS,
	T_MAIL,
	T_SUBJECT,
	T_MESSAGE,
	T_AUTHOR,
	T_REPLIES,
	T_UREPLIES,
	T_VIEW_COUNT,
	T_LAST_POST,
	T_DATE,
	T_LAST_POSTER,
	T_IP,
	T_LAST_POST_AUTHOR,
	T_LAST_POST_REPLY_ID,
	T_ARCHIVE_FLAG,
	T_LAST_EDIT,
	T_LAST_EDITBY,
	T_STICKY,
	T_SIG,
	T_EVENT_DATE,
	T_ISEVENT
FROM
	FORUM_TOPICS
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)


-- Copy REPLY's to archive table
INSERT INTO
	FORUM_A_REPLY
	(CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	REPLY_ID,
	R_MAIL,
	R_AUTHOR,
	R_MESSAGE,
	R_DATE,
	R_IP,
	R_STATUS,
	R_LAST_EDIT,
	R_LAST_EDITBY,
	R_SIG)
SELECT
	CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	REPLY_ID,
	R_MAIL,
	R_AUTHOR,
	R_MESSAGE,
	R_DATE,
	R_IP,
	R_STATUS,
	R_LAST_EDIT,
	R_LAST_EDITBY,
	R_SIG
FROM
	FORUM_REPLY
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

-- Delete TOPIC's from main table
DELETE
FROM
	FORUM_TOPICS
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

-- Delete REPLY's from main table
DELETE
FROM
	FORUM_REPLY
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

DROP TABLE #TOPIC_IDs

-- TO DO:  UPDATE FORUM COUNTS!!

GO

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight

Edited by - Jeepaholic on 28 September 2003 15:02:58
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 23 September 2003 :  05:59:48  Show Profile  Send ruirib a Yahoo! Message
Ok Al, I'll have a busy day today, will try to have a look at this later on. A quick look makes think the insertion code could be simpler, but I'll go through that tonight.

Anyway, at first sight, looks that it is doing what it should. Seems like the archival went as expected as well so, even if you can make it simpler, this one does work and that's what ya want, right ?


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 23 September 2003 :  13:26:42  Show Profile  Visit Jeepaholic's Homepage
Yep. I'm just hoping that I didn't miss something... My first version didn't take the Edited Date into account, and I was lucky to find that one in my testing... SOooo, just looking for a couple more sets of eyes before I hit my production forum with it.

Eventually, I'd love to see how you'd improve my code!

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page
Page: of 3 Previous Topic Topic Next Topic  
Previous Page | Next Page
 New Topic  Topic Locked
 Printer Friendly
Jump To:
Snitz Forums 2000 © 2000-2021 Snitz™ Communications Go To Top Of Page
This page was generated in 0.19 seconds. Powered By: Snitz Forums 2000 Version 3.4.07