ext3 directory limit

[TYSON]

Just reworking the PSB image caching so that it works in the background instead of when you make the post as large posts with lots of images bomb out.

We are currently at 2700 odd cached images after 2 weeks and am looking to come up with some form of directory structure for storing them.

Reading here ext3 - Wikipedia, the free encyclopedia

The maximum number of inodes (and hence the maximum number of files and directories) is set when the file system is created. If V is the volume size in bytes, then the default number of inodes is given by V/213 (or the number of blocks, whichever is less), and the minimum by V/223. The default was deemed sufficient for most applications. The max number of subdirectories in one directory is fixed to 32000.

Im thinking of a folder structure like this..

images/year/forum_id/thread_id/img_id-original-filename.ext

Hopefully including the year and forumID would overcome when we get more than 32k threads with images in a year.

Can anyone see any flaw in this?

[thro]

Looked into any of the alternative filesystems?

I know EXT4 overcomes the 32k subdirectory limit, but its not really production ready yet.

I hear Reiser is a killer filesystem....


edit:
I guess what I'm saying is that if it was me, i'd take the lazy way out and move the complexity out of the app and into the filesystem... not having seen the forum code i've got no idea how much hacking you'll have to do otherwise?

[MikeC]

+1 if it's a limit you realistically expect to see reached, use a different filesystem. It'll save untold pain later.

[TYSON]

It's all custom code so complexity isn't an issue, just a matter of specifying a different file path and of course some logic to create the directory if it doesn't exist.

ie

$start_dir = "/home/psb/www/imgcache/".date("Y")."/".$forum_id."/".$thread_id."/";
if(!is_dir($start_dir))
{
mkdir($start_dir, 0777,1);
}

I'd rather not have to fuck around with filesystems as it's just another thing to do when an upgrade occurs.

Basically i can over come the limit with folder structures but just wanted to be sure.

[TYSON]

lol, the forum strips out variables... what use is the php bbcode then.

[Klink]

store it all as binary in a db? (just thinking out loud)

[TYSON]

contemplated that, just worried about cpu. however it may be the best way for when we bring multiple servers online.

[Klink]

Once again... thinking out loud here... mount an NTFS disk for storage?

[MikeC]

Could you not just re-format a partition EXT4 and mount it as the images directory? i'm aware i'm probably solving a problem you don't have here but...

[MikeC]

Once again... thinking out loud here... mount an NTFS disk for storage?

BWAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!two!!!




You are joking, right

[Klink]

Actually no...

My last employment saw a system storing a bazillion small files and the engineers of the system had it on an NTFS drive specifically. Believe it or not... it worked.

[MikeC]

Well shit.


i wouldn't have relied 1) on NTFS to do anything right at the extremes of performance, or 2) on a write mounted NTFS partition from anything *NIX. i guess i'll have to look again.

[Klink]

There is a Linux NTFS-3G driver which would do the job. Now I wasnt saying that it was the best option... or even a good one. It was a thinking outside of the box (which I do quite often).

The maximum number of files on an NTFS volume is 4,294,967,295. Which would fit the bill on that attribute.

[TYSON]

store it all as binary in a db? (just thinking out loud)

Actually,think i might try this option for a bit and see how it goes.

Test shows no performance hit.

Comparison between storing images/files in mySQL and on filesystem. | Sitek Technology Blog

[XSorXpire]

Excuse me for being dumb, but can you store it in the users name who posted it under a common named file in their profile?

[thro]

^^ i guess you could.

but then you'd perhaps be open to some user deciding to upload 32,769 1 pixel images just to be a pratt

[JiggyJigs]

^^At which point you ban their arses for being said pratt. If someone is malicious enough to do that then do ya really want them on your site anyway?

[TYSON]

Well it's now running in parallel with the file system. The good part of this is being able to index them by forum, user, thread, etc etc Will make for a good image gallery based purely on what has been posted aswell as be able to provide a friendly path ie

http://www.perthstreetbikes.com/30/Dream-garage/IMG_1010.sized.jpg

Also discovered that wsftp won't list over 3k files in a directory as we are now at 4500+ images and it will only show me 3k of them

[thro]

^^At which point you ban their arses for being said pratt. If someone is malicious enough to do that then do ya really want them on your site anyway?

Granted, but it still doesn't stop it happening in the first place, either through accident, negligence or malice.

Or script from backpackistan hacker cafe, or whatever.

You don't design systems with the mindset of "nah, no one would ever do that to me?", or you'll get fucked

[Klink]

Well it's now running in parallel with the file system. The good part of this is being able to index them by forum, user, thread, etc etc Will make for a good image gallery based purely on what has been posted aswell as be able to provide a friendly path ie

http://www.perthstreetbikes.com/30/Dream-garage/IMG_1010.sized.jpg

Also discovered that wsftp won't list over 3k files in a directory as we are now at 4500+ images and it will only show me 3k of them

Ahh actually come to think of it bro, you will run into problems with shell commands before you run into file system limitations.

We had a system where I used to work, that had a crap load of files (in the 1000's) and it wasnt before long that we started having major problems with commands like: ls, grep, rm etc.

The filesystem didnt seem to have any problems adding more and more files to the directory... but it started becoming very difficult to manage the files inside of the directory.

Even if we piped ls thru grep for a particular date or file... grep still has to parse the entire ls listing before it can give you the output.