Snitz Forums 2000
Snitz Forums 2000
Home | Profile | Register | Active Topics | Members | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Help Groups for Snitz Forums 2000 Users
 Help: Database: MS SQL Server
 Faster MS SQL Archive Script (BETA!)
 New Topic  Topic Locked
 Printer Friendly
Next Page
Author Previous Topic Topic Next Topic
Page: of 3

Jeepaholic
Average Member

USA
697 Posts

Posted - 17 September 2003 :  05:44:18  Show Profile  Visit Jeepaholic's Homepage
04/29/04 UPDATE:
Realized that the Events Calendar mod database fields are included in this release. If you do not have this mod, you need to eliminate all references to T_EVENT_DATE and T_ISEVENT.

As well, if you have other mods that only add columns to the TOPIC and/or REPLY tables, you could just as easily add them to this script in order to assure that information is archived as well.

11/22/03 UPDATE:
It just dawned on me that the script does not lock the topics as they are archived. This is a requirement for the system. Please make the change (in red) to the script below.

11/22/03 UPDATE:
An UN-Archive script has been created as well, and is located in THIS TOPIC

11/19/03 UPDATE:
The code appears to be functional and accurate. It is posted below in green. You can use Query Analyzer to run this query on your forums, and it will archive and update the forum counts VERY quickly compared to the default forum archiving functionality. My forums would actually time-out and fail trying to archive a single month, however, with this query it took around 30 seconds to archive nearly 6 months. Your mileage will, of course, vary based on hardware. Simply enter the date in the format shown (to archive from), and the Forum ID to archive.

As stated, I believe it's accurate in what it does...but as always...backup your database first, test it on a copy of your database if you can, and use at your own risk.

ORIGINAL POST
Hey there. I've archived my forum a number of times - but for some reason, I can't seem to archive anything. I click "Yes" that I'm sure I want to archive the selected topics, and the browser just spins and spins. MS SQL Server doesn't seem to be showing any form of a "hit" regarding the archiving process.

I'll try to provide some stats, but I guess I'm looking for a way to debug this. My site performance is taking a hit, and am in desperate need of archiving again.

STATS:
224,407 Posts in 20,610 Topics
156,966 Archived Posts in 13,517 Archived Topics

I have tried to archive a forum with only 100 posts, which seems to have worked...HOWEVER, the archive topics admin screen never updated to tell me it was complete. Still spinning, in fact.

In my larger forums, I've also tried to only archive one month at a time, even though I ultimately need to archive 3-4 months worth. No avail, same issue.

Any help or words of wisdom would be appreciated. Thanks!
Al

CODE

DECLARE
	@ArchiveDate nvarchar(14),
	@ArchiveForumID int

SET
	@ArchiveDate = '20030701000000'  -- Date to archive from (YYYYMMDDHHMMSS)
SET
	@ArchiveForumID = 35  -- Forum ID to Archive


-- Create temp table that will hold TOPIC_ID's to archive
CREATE TABLE #TOPIC_IDs
(
	TOPIC_ID int
)

-- Fill temp table with TOPIC_ID's to archive
INSERT INTO
	#TOPIC_IDs
-- Query finds topics started (and edited) before archive date
SELECT
	TOPIC_ID
FROM
	FORUM_TOPICS AS FT
WHERE
	T_DATE < @ArchiveDate
AND
	-- If no last edit date (NULL), return lowest date possible
	ISNULL(T_LAST_EDIT,'00000000000000') < @ArchiveDate
	
AND
	FORUM_ID = @ArchiveForumID
AND
	-- Subquery to determine if any replies (or edits) are 
	--  AFTER the archive date.  If so, do not archive topic.
	(SELECT
		COUNT(*)
	FROM
		FORUM_REPLY AS FR
	WHERE
		(R_DATE >= @ArchiveDate
		OR
		-- If no last edit date (NULL), return lowest date possible
		ISNULL(R_LAST_EDIT,'00000000000000') >= @ArchiveDate)
	AND
		FR.TOPIC_ID = FT.TOPIC_ID
	) = 0


-- Copy TOPIC's to archive table
INSERT INTO
	FORUM_A_TOPICS
	(CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	T_STATUS,
	T_MAIL,
	T_SUBJECT,
	T_MESSAGE,
	T_AUTHOR,
	T_REPLIES,
	T_UREPLIES,
	T_VIEW_COUNT,
	T_LAST_POST,
	T_DATE,
	T_LAST_POSTER,
	T_IP,
	T_LAST_POST_AUTHOR,
	T_LAST_POST_REPLY_ID,
	T_ARCHIVE_FLAG,
	T_LAST_EDIT,
	T_LAST_EDITBY,
	T_STICKY,
	T_SIG,
	T_EVENT_DATE,
	T_ISEVENT)
SELECT
	CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	0,
	T_MAIL,
	T_SUBJECT,
	T_MESSAGE,
	T_AUTHOR,
	T_REPLIES,
	T_UREPLIES,
	T_VIEW_COUNT,
	T_LAST_POST,
	T_DATE,
	T_LAST_POSTER,
	T_IP,
	T_LAST_POST_AUTHOR,
	T_LAST_POST_REPLY_ID,
	T_ARCHIVE_FLAG,
	T_LAST_EDIT,
	T_LAST_EDITBY,
	T_STICKY,
	T_SIG,
	T_EVENT_DATE,
	T_ISEVENT
FROM
	FORUM_TOPICS
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)


-- Copy REPLY's to archive table
INSERT INTO
	FORUM_A_REPLY
	(CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	REPLY_ID,
	R_MAIL,
	R_AUTHOR,
	R_MESSAGE,
	R_DATE,
	R_IP,
	R_STATUS,
	R_LAST_EDIT,
	R_LAST_EDITBY,
	R_SIG)
SELECT
	CAT_ID,
	FORUM_ID,
	TOPIC_ID,
	REPLY_ID,
	R_MAIL,
	R_AUTHOR,
	R_MESSAGE,
	R_DATE,
	R_IP,
	R_STATUS,
	R_LAST_EDIT,
	R_LAST_EDITBY,
	R_SIG
FROM
	FORUM_REPLY
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

-- Delete TOPIC's from main table
DELETE
FROM
	FORUM_TOPICS
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

-- Delete REPLY's from main table
DELETE
FROM
	FORUM_REPLY
WHERE
	TOPIC_ID
	IN
	(SELECT
		TOPIC_ID
	FROM
		#TOPIC_IDs)

DROP TABLE #TOPIC_IDs

-- Update FORUM_TOTALS Counts
UPDATE
	FORUM_TOTALS
SET
	P_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_REPLY
		),
	P_A_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_A_REPLY
		),
	T_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_TOPICS
		),
	T_A_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_A_TOPICS
		)

-- Update Specific Forum Counts
UPDATE
	FORUM_FORUM
SET
	F_TOPICS = (
		SELECT
			COUNT(*)
		FROM
			FORUM_TOPICS
		WHERE
			FORUM_ID = @ArchiveForumID
		),
	F_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_REPLY
		WHERE
			FORUM_ID = @ArchiveForumID
		),
	F_A_TOPICS = (
		SELECT
			COUNT(*)
		FROM
			FORUM_A_TOPICS
		WHERE
			FORUM_ID = @ArchiveForumID
		),
	F_A_COUNT = (
		SELECT
			COUNT(*)
		FROM
			FORUM_A_REPLY
		WHERE
			FORUM_ID = @ArchiveForumID
		)
WHERE
	FORUM_ID = @ArchiveForumID
								

GO

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight

Edited by - Jeepaholic on 29 April 2004 21:55:36

Jeepaholic
Average Member

USA
697 Posts

Posted - 17 September 2003 :  05:46:17  Show Profile  Visit Jeepaholic's Homepage
Oh yeah...running the latest and greatest with security patches.

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

Nikkol
Forum Moderator

USA
6907 Posts

Posted - 17 September 2003 :  09:41:38  Show Profile
any mods?

Nikkol ~ Help Us Help You | ReadMe | 3.4.03 fixes | security fixes ~
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 17 September 2003 :  13:55:13  Show Profile  Visit Jeepaholic's Homepage
Sorry, yes. NO new mods have been installed since the last archive, however. The only changes would have been security updates. MOD's, off the top of my head:

Calendar
Active Users
File Upload

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 17 September 2003 :  17:32:33  Show Profile  Send ruirib a Yahoo! Message
I know this does not help much, but I wouldn't be surprised if this was due to the number of records in the forum. Al, maybe try selecting the shortest amount of time possible and choose the smaller forum, just to see if it works.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 17 September 2003 :  17:38:41  Show Profile  Visit Jeepaholic's Homepage
Hi Rui!

From my first post:
quote:
I have tried to archive a forum with only 100 posts, which seems to have worked...HOWEVER, the archive topics admin screen never updated to tell me it was complete. Still spinning, in fact.

In my larger forums, I've also tried to only archive one month at a time, even though I ultimately need to archive 3-4 months worth. No avail, same issue.



I have tried both of those items, with the results shown above. Any thoughts on how to resolve/debug this? If it's due to the number of records in the forum (which seems very likely), how can I go about archiving? I've limited the number of records to archive as much as I possibly can, and it's still not working...

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 17 September 2003 :  18:47:51  Show Profile  Send ruirib a Yahoo! Message
Hi Al,

I don't know the file that codes this that well. Let me have a look to see what I can come up with.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 17 September 2003 :  22:25:51  Show Profile  Visit Jeepaholic's Homepage
Thanks Rui. Any chance using the Profiler would give us more info? I'm not very familiar with how to use it, but with some guidance I could set something up and see what comes of it. Let me know if there's anything else I can provide...thanks!

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 18 September 2003 :  09:47:15  Show Profile  Send ruirib a Yahoo! Message
Well clearly we need to findout what part of the script is being run, what part is getting stuck. With the profiler we may able to find that. So, if you want to run a trace on that, it could help. Otherwise we'll have to resort to old-fashioned debugging, which can make it hard for me to help you, since I don't have a DB with that much data.

Anyway I think I've found room for improvement in the script, but it does not make any sense to do it if the parts in question are not causing the current problems. Let's identify the causes and act with that info on hand.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 18 September 2003 :  13:51:41  Show Profile  Visit Jeepaholic's Homepage
Ok, I'm going to need some help setting up the Profiler. What should I filter on to make sure I only get Archiving Information? I've never really set up anything other than a basic "watch everything" with the profiler.

Thanks Rui!

Completely side note, but for my own curiosity, how does the script work - in general? It appears that it doesn't just copy each line from the main table to the archive table? What are the steps it takes?

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 18 September 2003 :  17:38:40  Show Profile  Send ruirib a Yahoo! Message
SQL:BatchCompleted events are enough for me to find out where it gets stuck.

The script basically gets the replies, navigates the recordset in order to insert them one by one and writes each one into the archived replies table. Then navigates the recordset again to delete the replies from the live replies table(improvement can be achieved here). Then gets the topics and inserts them in the archived topics table one by one. After that deletes the topics from the live table. It also updates counts after doing this.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 18 September 2003 :  23:25:42  Show Profile  Visit Jeepaholic's Homepage
Ok...started running one and it's up to 2000 lines after only about 5 minutes. Shall I continue with it, or is there a way to filter futher?

I can also try late at night when there's less traffic, if there's no better way to filter.

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight

Edited by - Jeepaholic on 18 September 2003 23:27:25
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 19 September 2003 :  01:49:22  Show Profile  Visit Jeepaholic's Homepage
Ok Rui, I've got like 25000 lines in a 9MB trace file on a failed archive. If you want that shoot me an email and I'll give you FTP access. Or, if there's a better way to filter let me know and we'll start over. Thanks again for any help...

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

ruirib
Snitz Forums Admin

Portugal
26364 Posts

Posted - 19 September 2003 :  08:01:51  Show Profile  Send ruirib a Yahoo! Message
Al, is that a 9 MB trace just for the archive function? Or does it include the usual forum traffic?

It would help if you could choose a period of less traffic at the forum and then run the archive procedure and stop the trace after a couple minutes. Of course, I'm mainly interested in seeing how the SQL statements related to the archiving script are being handled, to see where the script is hanging.


Snitz 3.4 Readme | Like the support? Support Snitz too
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 19 September 2003 :  12:51:44  Show Profile  Visit Jeepaholic's Homepage
Yes, it includes normal forum traffic...I couldn't figure out how to trace just the archive function. I'll do another one late tonight and see if we get a smaller trace file.

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page

Jeepaholic
Average Member

USA
697 Posts

Posted - 20 September 2003 :  03:01:44  Show Profile  Visit Jeepaholic's Homepage
Shoot me an email, Rui. I'll reply with the new file. Ran for about 30 minutes...2700 lines (1.7MB I think). It was an attempt to archive less than 1 months worth of data... If that's too big, let me know and I'll try something else.

Al Bsharah
Aholics.com

Jeepaholics Anonymous
Broncoholics Anonymous
Network Insight
Go to Top of Page
Page: of 3 Previous Topic Topic Next Topic  
Next Page
 New Topic  Topic Locked
 Printer Friendly
Jump To:
Snitz Forums 2000 © 2000-2021 Snitz™ Communications Go To Top Of Page
This page was generated in 0.16 seconds. Powered By: Snitz Forums 2000 Version 3.4.07