If this is the incorrect forum I apologize, I thought this was the best fit.
A big problem I have and probably everyone else is when forum members link to external images that disappear later on. Many topics now have red X's where the now non-existant images should have been.
I have proeders file attachment mod being used but still many members find it easier to just throw up a link to inkfrog or another image hosting company.
My idea is this. I would like an admin utility that will go through a forum at a time and detect all links to external images, all of the IMG tagged lines which are followed by http, indicative of an external image. I would like it to then get that image and place it in the users uploaded folder. The final step would be to change the line of code to point to the forum stored image.
This would be a god send for most of us. Thanks in advance.

are you able to use .net on your web server? I already have some code in .net that would do this with a little tinkerring, and is a lot easier to do in .net than classic ASP

Thanks for the response. I'm not sure how to answer your question so I will tell you what I do and do not have.
The server is running Windows 2003 Server with SP2. The .Net Framework for versions 2, 3 and 3.5 are installed, all are at the latest service pack. I am using IIS and ASP.Net v2.0.50707 is installed.
I do not have any .net developer tools installed.
What do I need if anything? In short, to answer your question are you able to use .net on your web server? I will do whatever it takes to make this work.
Also to get another question out of the way, does your code do this for the entire forum at one shot or can I narrow it down to individual forums? I would then set up a test forum to test and troubleshoot it. I would be hesitant to try this out against 5 years worth of work.

Hi,

i would be intreasted in this code you have HuwR, Sounds like it will come in handy with a gallery project im working on.
do you mind sharing it?
Thanks

MaD2ko0l

Todd, all you need is the .net frameworks installed, I will try and throw something together over the weekend, bit pushed for time in the evenings at the moment, but hassle me after the weekend if you haven't heard anything.(I will make it work on individual forumsas it could take a while to do a whole Forum in one go)

maD2ko0l, sure no problem I will post the code next week, do you want the raw grab and save code or the finished code I will make for Todd ?

Thank you so much HuwR!.. Whenever it's ready is fine, no rush. I will keep checking here. Do you need my contact information?

the raw grab and save code would be great if you dont mind.
thanks very much

Originally posted by MaD2ko0l
the raw grab and save code would be great if you dont mind.
thanks very much

Actually both would be great. Looking at the 2 I'm sure there's a lot I can learn.

Originally posted by MaD2ko0l
the raw grab and save code would be great if you dont mind.
thanks very much

this is the grab and save code, and shows the simplicity of using .Net (code is C#)

Grabbing the image is simple, first we declare a byte array to store the image in
Byte[] imagearray; next we grab the image into the array using the webclient class

using (WebClient client = new WebClient())
{
imagearray = client.DownloadData(urlofimage);
} next we just save the imagearray to disk using a filestream

using (FileStream writer = new FileStream(destinationpath + filename, FileMode.Create))
{
writer.Write(image, 0, image.Length);

}
I will post the full code for parsing the img tags and grabbing the images later

Todd, what database do you use ?
what would you like it to do if it fails to retreive the image ?

Originally posted by HuwR
Todd, what database do you use ?
what would you like it to do if it fails to retreive the image ?

I'm using an Access 2000 database.
If it were to fail to get the image I would imagine you are referring to when that particular image has already "diappeared" from it's externally hosted site. Maybe it would be good to replace that with a generic image called PleaseFixLink.jpg That could be an image that all snitz forum admins can decide they want it to be. Thinking about this more that would mean we would be changing the link to point to that generic image thus losing the original non working link.
Ideas?
Maybe just logging the poster name, forum/topic and missing image link? I could then PM the user's asking them to fix it. The next time I run the utility it could pick up the image.

Maybe just logging the poster name, forum/topic and missing image link? I could then PM the user's asking them to fix it. The next time I run the utility it could pick up the image.

That may not help, in some of the tests I ran the image no longer existed because the domain had "diappeared", so it is likely that the user may no longer be contactable either

Maybe just logging the poster name, forum/topic and missing image link? I could then PM the user's asking them to fix it. The next time I run the utility it could pick up the image.

That may not help, in some of the tests I ran the image no longer existed because the domain had "diappeared", so it is likely that the user may no longer be contactable either

Yes that could be the case but... I know many of my users are still active on the forums and there is a good chance they would upload the pictures again if I asked them to. They would probably do it the "proper" way which would be using the file attachment mod. Right now there are just too many missing images and I have no efficient means of pinpointing all of them. I would say only 5% of the images aren't available for viewing for all of the reasons discussed but with thousands of posts... tough job for anyone.

I have it almost ready. just doing some tests on it now, any images it can't pick up are logged to a file which you can load up when the transfer is finished, it will tell you why it couldn't grab it and provide a link to the post.
Do you want to be able to do this with archived posts as well ?

Originally posted by HuwR
I have it almost ready. just doing some tests on it now, any images it can't pick up are logged to a file which you can load up when the transfer is finished, it will tell you why it couldn't grab it and provide a link to the post.
Do you want to be able to do this with archived posts as well ?

Excellent, glad I'm online at the same time you are for once. As far as archived posts goes I have very few archived posts. The only posts I've archived are ones from me addressing the forum members with changes I've made. Some are no longer relevant so I archive them. That might however be useful for other forum admins.
Looking forward to trying this out, it's a good day for me to do it.

I'm just adding some other checks to try and reduce the time it takes, say for example if it has already logged a url as missing there is no point in looking if it crops up again

one more question, should have asked before, do you want to store them in directories using the username or id ?

Username please, that's how it's been done since I put the forums up.
The webpath to my forum files is http://www.tycoforums.com/tyco/forum

The webpath to an image I upload using the filea attachment mod would end up being http://www.tycoforums.com/tyco/forum/uploaded/MYUSERNAME/MYUPLOADEDIMAGE.JPG

ok great, I have done it so that the upload root is read from a config file, so you would put /tyco/forum/uploaded/ in the config file.
I think I have it pretty much ready to go, but won't get chance to upload it today, got to nip out for a few hours but will upload the code with some instructions for you tomorrow.

Originally posted by HuwR
ok great, I have done it so that the upload root is read from a config file, so you would put /tyco/forum/uploaded/ in the config file.
I think I have it pretty much ready to go, but won't get chance to upload it today, got to nip out for a few hours but will upload the code with some instructions for you tomorrow.

Thank you very much, it's appreciated.

Not sure if you considered the following in your code.
What happens if a poster replies to a topic with a quote (reply with quote) and that quoted material includes an image from the original post?
I don't think we would want to capture that image also and put it into the responder's upload folder. We would however want to capture all images outside the quoted picture, maybe even if the target image outside the forum is the same.
Thoughts?

not a great deal I can do about that, the code just grabs [img] tags and replaces the content if it is an external link to do something different if it is wrapped in a quote would make the code virtually un-runable (it would just time out) unless you did it topic by topic, there are too many images to try and keep track of every url it replaces, if it were a desktop app it would be different but you are limited to what you can do with web.

Originally posted by HuwR
not a great deal I can do about that, the code just grabs [img] tags and replaces the content if it is an external link to do something different if it is wrapped in a quote would make the code virtually un-runable (it would just time out) unless you did it topic by topic, there are too many images to try and keep track of every url it replaces, if it were a desktop app it would be different but you are limited to what you can do with web.

I'm sure you're right, plus I'll have backups of some of the images. Actually, no, it's going to parse the urls, do it's job and mark that url as being taken care of. In most cases it wouldn't hit a quoted image that hasn't been taken care of already assuming most quoted images are likely to be in the same topic. Is that how it progresses through a topic? Re-reading your explanation above, maybe not.
Please realize I'm only trying to understand how things work and you're a great one to learn from.
What you have mentioned it will do is going to be awesome! I've already notified my forum members that I would be doing something with the forums and they might notice some sluggishness.

no, it only tracks missing images (since they are the ones that generally take the longest to get a response from) images it finds it just downloads and replaces the img tag with a local reference, it does not keep track of them, obviously it could do things a lot better, but only if you don't mind waiting a few weeks for me to write the code
it grabs the image tags by doing some regex matches on the message, it doesn't check anything other than image tags so does not know if it is within a quote tag or not. Like I said, it could be done, but it is a lot more coding and a lot more time.

Ignore last post, I think I just had a brain wave, just doing some quick testing.

OK , here goes. If you need any help with any of this just drop me an email.
You need to do the following. 1) In your website you need to create an application directory for .net 2
Say something like imagegrab, so it's url would be http://www.tycoforums.com/imagegrab
2) in the zip you will find a file called web.config, open it in notepad and adjust any of the setting that need tweaking (i think they should be ok apart from the connection string which will need pointing at your database)
3)upload the files in the zip to the newly created app directory
4) open the default.aspx page in your browser and off you go.
I tested it on a copy of this database, it ran through the whole forum without an issue, but then I was running it locally so it had nothing to contend with.
files for download Image Grabber

Originally posted by HuwR
OK , here goes. If you need any help with any of this just drop me an email.
You need to do the following. 1) In your website you need to create an application directory for .net 2
Say something like imagegrab, so it's url would be http://www.tycoforums.com/imagegrab
2) in the zip you will find a file called web.config, open it in notepad and adjust any of the setting that need tweaking (i think they should be ok apart from the connection string which will need pointing at your database)
3)upload the files in the zip to the newly created app directory
4) open the default.aspx page in your browser and off you go.
I tested it on a copy of this database, it ran through the whole forum without an issue, but then I was running it locally so it had nothing to contend with.
files for download Image Grabber

I made a folder in the wwwroot called imagegrab and copied your files into it. Went into IIS and created application directory for it. Set to ASP.NET 2.0 and changed the path to database starting at C:\Intetpub\etc\etc\etc\etc\databasefile I also used a relative webpath such as /tyco/fo..../databasefile

Went to open the default.aspx of the application and got this. Might the bolded part be the cause?

Server Error in '/imagegrab' Application. --------------------------------------------------------------------------------

Keyword not supported: 'provider'.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: Keyword not supported: 'provider'.
Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:


[ArgumentException: Keyword not supported: 'provider'.] System.Data.Common.DbConnectionOptions.ParseInternal(Hashtable parsetable, String connectionString, Boolean buildChain, Hashtable synonyms, Boolean firstKey) +4907604
System.Data.Common.DbConnectionOptions..ctor(String connectionString, Hashtable synonyms, Boolean useOdbcRules) +98
System.Data.SqlClient.SqlConnectionString..ctor(String connectionString) +55
System.Data.SqlClient.SqlConnectionFactory.CreateConnectionOptions(String connectionString, DbConnectionOptions previous) +24
System.Data.ProviderBase.DbConnectionFactory.GetConnectionPoolGroup(String connectionString, DbConnectionPoolGroupOptions poolOptions, DbConnectionOptions& userConnectionOptions) +200
System.Data.SqlClient.SqlConnection.ConnectionString_Set(String value) +62
System.Data.SqlClient.SqlConnection.set_ConnectionString(String value) +4
System.Data.SqlClient.SqlConnection..ctor(String connectionString) +24
SnitzImageGrab.DataAccess.GetForumList() in D:\Development\SnitzImageGrab\SnitzImageGrab\App_Code\DataAccess.cs:20
SnitzImageGrab._Default.BindForumList() in D:\Development\SnitzImageGrab\SnitzImageGrab\Default.aspx.cs:27
SnitzImageGrab._Default.Page_Load(Object sender, EventArgs e) in D:\Development\SnitzImageGrab\SnitzImageGrab\Default.aspx.cs:20 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +14
System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +35
System.Web.UI.Control.OnLoad(EventArgs e) +99
System.Web.UI.Control.LoadRecursive() +50
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +627




--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:2.0.50727.3603; ASP.NET Version:2.0.50727.3082

it doesn't like the connection string, since I don't use access I have no ide what it should be, that was taken from a website, you will have to do some goggle searches to find out what the correct connection string should be

seems a lot of people are having this issue, I will see if I can figure it out

thanks. I do think I've put in the right path. It looks as if something you've hardcoded is pointing to your D: Drive, possibly from when you were developing and testing?

ok, new code coming in a few minutes, figured out what I did wrong, it is nothing to do with the path, the d: is from the debug file and is just showing where in the code the error is occuring

ok, second attemp
Image Grabber
replace all the files with these new ones, and change the database path etc in web.config, should now be ok. Just tested it on a snitz access db and it connects ok

Thanks, I'm going to try the new code now. It took me 2 hours to get home from work in sleet and snow, whew, I made it home.

UPDate

It's almost completely working for me. It is reading the forums correctly and scouring the internet for those external images. It is also downloading them and putting them in the appropriate users uploaded folder.
But it isn't modifying the code to point to the new local images which were successfully brought in.

well that's a bit odd, it definately changed them for me.

How large is your database ? any chance you can give me ftp access to download it ? I don't have an access database with any data in so could only test it on sql

Originally posted by HuwR
well that's a bit odd, it definately changed them for me.

How large is your database ? any chance you can give me ftp access to download it ? I don't have an access database with any data in so could only test it on sql

Yes I will give you ftp access and email the information to you. Thank you. Database is about 50MB. Incidently, I've run this against the entire forum and it captured over 1000 pics and put them in the appropriate folders. Not sure how the code works but my fear is that it's too late to now actually make the link changes. If it finds an external link and tries to copy to uploaded/user folder and that image is already there, will it still change the topic code?

yes it will, it will simply overwrite the image, that is the bit I don't understand, in my test it definitely changed the links since when I ran it a second time it no longer found any external images other than the failed ones, so not quite sure why it didn't do that for you.

figured out the problem, just not used to connecting to access databses in .net have tested against a copy of your db and it seemed to work ok.
Image Grabber update (the updated zip does not contain the web.config file, so use the one you currently have

I've been trying to get this to work and I come up with this error on debugging

Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.
Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:


[NullReferenceException: Object reference not set to an instance of an object.] SnitzImageGrab.DataAccess..cctor() in D:\Development\SnitzImageGrab\SnitzImageGrab\App_Code\DataAccess.cs:12

[TypeInitializationException: The type initializer for 'SnitzImageGrab.DataAccess' threw an exception.] SnitzImageGrab.DataAccess.GetForumList() in D:\Development\SnitzImageGrab\SnitzImageGrab\App_Code\DataAccess.cs:41
SnitzImageGrab._Default.BindForumList() in D:\Development\SnitzImageGrab\SnitzImageGrab\Default.aspx.cs:27
SnitzImageGrab._Default.Page_Load(Object sender, EventArgs e) in D:\Development\SnitzImageGrab\SnitzImageGrab\Default.aspx.cs:20
System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +14
System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +35
System.Web.UI.Control.OnLoad(EventArgs e) +99
System.Web.UI.Control.LoadRecursive() +50
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +627


Any pointers please??

line 12 assigns a value from the web.config file to variable, so the only thing that could cause the error is if you don't have the web.config file.

Thanks Huw, I have just redone all changes made and started again... I now get this error..
Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level. This error can be caused by a virtual directory not being configured as an application in IIS.
Source Error:


Line 13: <system.web>
Line 14: <compilation debug="true"></compilation>
Line 15: <authentication mode="Windows"/> Line 16: </system.web>
Line 17: <system.codedom>

Looking on the net this is something to do with having 2 web.config files??

Looking on the net this is something to do with having 2 web.config files??

or by a virtual directory not being configured as an application in IIS

I'm using an Access 2000 database and this utility worked very well. I do however still have one issue. Some people have decided to type the img tags themselves and they did it in all CAPS. The utility is case sensitive and converted every single lowercase instance but the uppercase instances are untouched.
I've tried using Access 2000 to open the database and do replace operations. I have no idea what's going on but it seems anything within the brackets isn't found when doing a find (or replace) So if I search for img or img within brackets, to cut it short I've tried all combinations of case, limitations, etc..

I'm at a loss. Anyone know how to work around this or of another editor that would be capable of just replacing all the uppercase img tags with lowercase?

that could be because [] is reserved in SQL to do a regular sql search to find [] you probably need to do something like

WHERE COLUMN_NAME LIKE '%[IMG]%' ESCAPE '['

or

WHERE COLUMN_NAME LIKE '%[[]IMG]%'

anyway, I'm pretty sure I fixed the case sensitivity, will dig out my latest copy of the code and post it later today

Originally posted by HuwR anyway, I'm pretty sure I fixed the case sensitivity, will dig out my latest copy of the code and post it later today

Thanks, I didn't want to keep bothering for it, look forward to trying it out.

I think this should be the latest version, but haven't had chance to check if it fixes the case issue, bit busy building a polytunnel at the moment.
ImageGrab dll

I replaced the dll and had problems.
New default.aspx listed below by HUWR solved problem with it not working with just the new DLL.

I put back the old dll from Feb 24 and it worked ok again, so it must be the .dll

There was one problem I've had all along. A few months back I modded the forum so that when the image tags are placed around the image location it is also centered. After running the imagegrab utility, the code would be changed from


to


removing the img tags altogether and leaving nothing but the text

/tyco/forum/uploaded/New_York_Central/train_in_spain.jpg

displayed in the post.

This happens ONLY on images which are located on the forum server already, it does NOT happen to images that are offsite. They are grabbed and the code is changed properly.

ok, you may need a new default.aspx, I assumed I had just made a change in the dll, i will zip up the other files and post them later

try this default.aspx with the new dll I posted.
default.aspx

Originally posted by HuwR
try this default.aspx with the new dll I posted.
default.aspx

Worked well with new default.aspx file. It does the job and it's even working with the uppercase IMG tags. Thank you very much for this utility, it's a godsend.
However, (and this existed with the old dll and default.aspx) all pictures that are locally hosted on the forum and are within nested code like;

center img /tyco/forums/uploaded/user/picture.jpg /img /center

get changed to;

/tyco/forums/uploaded/user/picture.jpg


UPDATE: I think I fixed it. I went into inc_code.asp and changed the function back to it's default snitz code. Like I said, I modded this forum alot. I had changed the function for insert image to one that also put the code for centering the image. Now it just inserts the picture. Users will have to center the pics in another step, no big deal.
I think this might be 100% now. I'll follow up with my successes soon, going to try out a few of the real forums now.

Sorry, I forgot to come back and give an update. Thanks HUWR, it's working correctly. Wonderful utility!

this is the grab and save code, and shows the simplicity of using .Net (code is C#)

Grabbing the image is simple, first we declare a byte array to store the image in
Byte[] imagearray; next we grab the image into the array using the webclient class

using (WebClient client = new WebClient())
{
imagearray = client.DownloadData(urlofimage);
} next we just save the imagearray to disk using a filestream

using (FileStream writer = new FileStream(destinationpath + filename, FileMode.Create))
{
writer.Write(image, 0, image.Length);

}
I will post the full code for parsing the img tags and grabbing the images later

Thanx for sharing!!

---

I have a lovely cat named GHD.Barin

Issue with Image Grabber. Image grabber breaks, actually it removes the img code when run for a second time against a post only IF that post also contains a link to an external webpage.
Update: Done testing. If there is a link to an external webpage it interferes when starting with http:// If I remove the http:// the image grabber seems to ignore it and process the post correctly.
See this post where I worked out the issue.
http://www.tycoforums.com/tyco/forum/topic.asp?TOPIC_ID=14039

CAN THIS BE FIXED?