| Author |
 Topic  |
|
skwayb
Starting Member
17 Posts |
 Posted - 03 November 2006 : 15:26:28
|
| Can't wait to see this... Any ideas of when you will have something we can play with and help test? |
 |
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 03 November 2006 : 15:40:20
|
It is getting closer  I don't want to give a date yet as it would put me under pressure to try to get it done, and I am busy doing "real work" at the moment so am not able to do as much with it currently as I would like, but it will definitely be before the end of the year, that is about as firm as I can be at present, however initially it will be for online testing only, it will be some time before it is ready for others to install and test. |
|
|
|
wildfiction
Junior Member
167 Posts |
Posted - 03 November 2006 : 22:16:14
|
| All your hard work is much appreciated Huw! |
|
|
|
StephenD
Senior Member
Australia
1044 Posts |
Posted - 11 December 2006 : 04:08:41
|
| Looking forward to this Huwr! |
|
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 11 December 2006 : 04:58:41
|
Update:
Not had much time to do anything recently (need to earn some money ) however I am hoping to get a test site up ready for testing/preview in the new year, no promises, as real work is quite hectic currently, but that is plan if all goes well.
It will not be available for you to download at that time as it still has quite a bit of work that needs doing, but would like to get what I have done upto that point fully tested before continuing. |
|
|
|
janaf
Starting Member
Sweden
19 Posts |
Posted - 13 December 2006 : 10:13:51
|
Hi,
I would like to Beta test when it's time.
When this project advances, I think it should have it's own topic/forum, as this one is '...ASP.NET (Non_Forum Related)...'
I also open a can of worms by asking for core support for binary content in the DB. IMO, at some point, binaries including video, should be in (or linked from) normal postings, not in separate galleries/archives. IMO, storing binaries (with can be enabled / disabled and size limited per upload and per user) is easier to managage than separate file directories if the structre allows simple cleanups, for example for expired accounts, deleted content, archived content, and other dead binaries. Stopping executables, scripts etc, is of course vital. |
|
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 13 December 2006 : 10:35:17
|
| rubbish, if you knew anything about db performance, you would know that storing binaries in a database is an extremely bad idea |
|
|
|
pdrg
Support Moderator
United Kingdom
2897 Posts |
Posted - 13 December 2006 : 11:27:46
|
quote:
Originally posted by janaf
I also open a can of worms by asking for core support for binary content in the DB. IMO, at some point, binaries including video, should be in (or linked from) normal postings, not in separate galleries/archives. IMO, storing binaries (with can be enabled / disabled and size limited per upload and per user) is easier to managage than separate file directories if the structre allows simple cleanups, for example for expired accounts, deleted content, archived content, and other dead binaries. Stopping executables, scripts etc, is of course vital.
Too right this is a can of worms, but with the exception of a configuration management system, I do not see a reason to ever do this. Each binary you save has to spool back to disk to do anything with it anyway so you've got additional steps of streaming it into the field, storing huge BLOBS, streaming it (multiple times?!) back out all on top of the regular overhead of binaries. Think harder about your system design, and you can tidy up files far easier in the filesystem.
The only caveats I mentioned are config management, but also winfs, although it does it invisibly so it's more for completeness than an exception! |
|
|
|
janaf
Starting Member
Sweden
19 Posts |
Posted - 13 December 2006 : 13:27:48
|
quote:
Originally posted by HuwR
rubbish, if you knew anything about db performance, you would know that storing binaries in a database is an extremely bad idea
True, I have not seen the perfomance issues, because on the small & medium size systems I have worked with, typiaclly less than a thousand visitors / 10.000 page views per day, performance (due to BLOBs or other) has not been an issue. The BLOBs (50-200K) handle in a blink and I can not see a big percieved performance difference between the Access file saved BLOBs and file system based binary file handling. This may be completely different for large systems.
quote:
Originally posted by kentk...
huge BLOBS, streaming it (multiple times?!) back out all on top of the regular overhead of binaries
We have two issues here, one is if we allow binary uploads or not and the other is how to store it.
As far as I have seen, storage overhead for a BLOB is not large, compared to saving in the FS (wich btw is also a kind of database...) if done right, perhaps 20% space overhead. I just checked with a small DB I have with about 100 BLOB files. The uncompressed Access file size is 32MB, and the sum of the size of BLOB files in it is 28.8MB. I should add 10% for thumbnails so the main BLOB would not nead to be read all the time. Done wrong, the file size can easily increase by 1000% in size when BLOB:ed into the DB.
The disk price cost is now around 1$ per GB and steadily decreasing. By the time a project like this would be ready to run, price wil most likely have gone down a lot more.
For me, and I think many small medium size systems, ease of management both for administrators and site users, is far more important than disk space and performance.
If you think management can be done as well as files on the FS rather than in the DB file, fine with me. The important issue is that the binary file handling is easy to manage and presentation is well integrated, not a loose add-on.
|
Edited by - janaf on 13 December 2006 13:51:47 |
|
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 13 December 2006 : 14:07:24
|
1) While Filesytems may share some characteristics of a database,it is debatable whether a file system can be classified as a special-purpose database
2) disk cost per Gb is a lot cheaper and far less restricted (in web hosting terms) than database size/cost, and you certainly wouldn't be using an access db for web hosting if it's size was measured in Gb.
3) Performance of binaries stored in a db vs FS is not a disputable issue, it is a well known fact, try doing some research on google, there are hundreds of articles which will tell you storing files in db is a bad idea
4) This is a forum, it is not a file management or a content management system and therefore management of binary files whether they be in the db or in the File system is not an important issue in the slightest. |
|
|
|
pdrg
Support Moderator
United Kingdom
2897 Posts |
Posted - 13 December 2006 : 14:33:33
|
| Ahhh - do you mean like storing a photo in Access/Rich client like Northwind does? If you want a web app/thin client/hefty db you have a very different case, I feel. Then you have to manage streaming. Not really worried about disk space, but db bloat can be an issue I'm sure your DBA will have comments about :-D |
|
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 13 December 2006 : 14:35:52
|
quote:
do you mean like storing a photo in Access/Rich client like Northwind does?
This is still an extremely bad thing to do, especially where web sites are concerned, IIS and all other web servers cache image files for a reason, they do not cache images streamed from a db hence the reason it is a very bad thing to do. |
|
|
|
janaf
Starting Member
Sweden
19 Posts |
Posted - 13 December 2006 : 16:05:18
|
Why so offended?
1) I never said it was (but guess the comment was for me).
2) Yes.
3) After searching the web a little, I do not come to the same conclusion. There are plenty of people who LIKE one or the other, but is seems few have actually compared and most references I find talk about the pros and cons of each solution, including terrabyte databases with blobs.
Here is a wiki with summaries by 12 authors:
http://en.wikibooks.org/wiki/Programming:WebObjects/Web_Applications/Development/Database_vs_Filesystem
Here is a june 2006 Microsoft Research paper where they have measured, with SQL Server and the result is a bit "it depends", and the performance difference is not big either way. The Paper also has a chapter named Folklore which starts "There is a wealth of anecdotal experience with applications that use large objects". ftp://ftp.research.microsoft.com/pub/tr/TR-2006-45.pdf
The result summary is: "The study indicates that if objects are larger than one megabyte on average, NTFS has a clear advantage over SQL Server. If the objects are under 256 kilobytes, the database has a clear advantage. Inside this range, it depends on how write intensive the workload is, and the storage age of a typical replica in the system"
Myself, I do not know, I never made a comparison of BLOB performance, and I did not claim I did. I only say that in the limited BLOB use I have done, performance was not a problem or a limiting factor.
4) It seems some of the most visited threads on the mod pages on this site are the file sharing mod thread by korayem, the file sharing mod thread by proeder, the photo album and photo gallery mod threads. |
|
|
|
HuwR
Forum Admin
United Kingdom
20579 Posts |
Posted - 13 December 2006 : 16:58:53
|
1) you did quote:
compared to saving in the FS (wich btw is also a kind of database...)
sorry, but if you don't even acknowledge what you write then there isn't much point in pursueing it.
2) yes what ?
3) blob performance may not be a limiting factor in certain circumstances, but when it comes to web servers it is since as I said IIS and other web servers cache static files and images, streamed blob fields are not static by there nature are not cached, this is extremely limiting when you are paying for your bandwidth.
4) So what,because a thread is the most visited does not necesarily imply that it is the most popular Mod, it could mean that it is either the most buggy or the most difficult to implement, so you should not read anything into the viewing numbers of a topic, and besides just because people utilise those mods does not make it the business of a forum to be a file management system, you may well find that most people who utilise these mods do so as part of a portal system, Snitz is not a portal system it is a forum there is a distinction. |
|
|
|
janaf
Starting Member
Sweden
19 Posts |
Posted - 13 December 2006 : 18:39:55
|
quote:
Originally posted by HuwR
1) While Filesytems may share some characteristics of a database,it is debatable whether a file system can be classified as a special-purpose database
I did not write it was could be "classified as a special-purpose database", but "wich btw is also a kind of database..." which is something else.
quote:
Originally posted by HuwR
2) disk cost per Gb is a lot cheaper and far less restricted (in web hosting terms) than database size/cost, and you certainly wouldn't be using an access db for web hosting if it's size was measured in Gb.
2) yes, I agree to your 2) statement
quote:
Originally posted by HuwR
3) blob performance may not be a limiting factor in certain circumstances, but when it comes to web servers it is since as I said IIS and other web servers cache static files and images, streamed blob fields are not static by there nature are not cached, this is extremely limiting when you are paying for your bandwidth.
4) So what,because a thread is the most visited does not necesarily imply that it is the most popular Mod, it could mean that it is either the most buggy or the most difficult to implement, so you should not read anything into the viewing numbers of a topic, and besides just because people utilise those mods does not make it the business of a forum to be a file management system, you may well find that most people who utilise these mods do so as part of a portal system, Snitz is not a portal system it is a forum there is a distinction.
3) I can not see how caching in the web server changes the bandwidth use, unless it is browser caching? But then I do not know much of how IIS works. But both the OS and SQL server and even the disk also cache, so I would like to see some real life tests before drawing conclusions. I just may do some tests one day. To some, bandwidth cost is limitig, and you would always have the possibility to shut binary files off or on, regardless of how they are stored. And like I wrote before, if this can be solved better by storing in the FS, fine with me.
4) Could be, I don't know. It could also be that they simply are the most popular. One indication is the mod downloads, where at Snitzbitz the Avatar Mod is topping the downloads list right now.
A .net port should i.m.o. aim for some guestimate of what the situation (performance, demands, costs), will be in something like 5 years from now, not how it is now. What has been the last years, and how it is right now will be passed tense when a .net port is available. Look back a few years and extrapolate forward a few years from now.
My guess, based on my own needs and requests I see coming from customers and users, is that a more or less text focused forum system will not be very interesting in five years, whether we like it or not.
And again there are two separate issues, one if binary content should be intergarted/promoted/accepted, the other is how to store it.
But I as I can not spend time on development right now, I will have to sit back and see what comes, and try to like it  !
My two cents worth.... |
|
|
|
Topic |
|
|
|
Topic Locked
