Community Project: Retrieval of "Lost Images"

[Berry Syedow]

I thought it might be a good idea if we had a thread devoted to rescuing MOC photos. To maintain some order specify the theme and then provide links under it. Although it might go without saying, limit the pictures to MOCs that aren't yours, unless you think your MOC is particularly cool. ;-) And hey, if you're lucky some of your missing pictures might pop up!

Now I just need to find a place to store all o' my pictures...

(P.S. ~I take it maj.com moderates content as well?)

[JINZONINGEN73]

Cajun (Bryce McGlone) of In_the_Bricks pointed out that webarchive.org is a good way to get at least SOME of your photos.

Be warned though, alot will be missing.

Even more, alot will have thumbnails, but no pics and no thumbnails and HAVE pics.

Prepare yourself for a whole lot of hitting "next".

(As a bonus, I noticed older saves on webarchive produced pics I'd long since deleted off Brickshelf! What an awesome place).

[Berry Syedow]

Well, I got a new account at flickr. My user name is "Bob The Shop". I uploaded most of my miscellaneous LEGO creations by other builders: LINK

I'm disappointed that cheapskates only get three folders to store all of their junk (and I'm not about to subscribe to flickr just yet ;-)).

For those of you who are interested:

Miscellaneous Medieval Structures

Moko's Beds: 1 2 3 4 5 6

Bone Crusher Knight

Piggies and Sheep Cattle

Tin Foil Hat Man (Too cool for words! X-D)

Spaceship, Delorean, Battleship, and Neat Building Technique

Have a good night, I'm pooped!

[Brainbox]

(P.S. ~I take it maj.com moderates content as well?)

Maj does moderate content, but all uploads are instantly viewable. It's still up, and my Brickshelf login still works.

Thankfully, I've backed up all my MOC photos myself anyway, some on CD, some are still on my PC.

[Grand Moff Viceroy]

There is a lot of wasted stuff, but I only had an acccount, I didn't have any pics, haha.

[Starwars4J]

Well I don't know if I can help, but over the last few months I've saved to my computer a lot of my favorite MOCs that really inspired me. I could upload them all to a maj folder or something, and let the word spread so that anyone missing a MOC could look there

[Dryw Filtiarn]

I'm currently working on storing everything that can be found on brickshelf. This means a 100% backup, so this way there will at least be more people beside the site owner having all the images. When it's done it will initially serve as a recovery backup. So people who did not make it to retrieve their image (i.e. due to holidays) can give me a message to get their stuff back. What is to be done with the content in the future is to be determined at this point.

It's likely that video's will be excluded though for the fact that they take a long time to capture as well as that they take up a lot of storage space. And in my opinion it's the images and ldraw files which are of most importance.

Edited July 17, 2007 by Dryw Filtiarn

[Asuka]

I'm currently working on storing everything that can be found on brickshelf. This means a 100% backup, so this way there will at least be more people beside the site owner having all the images.

What a noble, marvelous and simply stunning project.... 8-|

Thanks a lot for all that incredible work, Dryw Filtiarn.... just great! :'-)

[ups]

I'm currently working on storing everything that can be found on brickshelf. This means a 100% backup, so this way there will at least be more people beside the site owner having all the images. When it's done it will initially serve as a recovery backup. So people who did not make it to retrieve their image (i.e. due to holidays) can give me a message to get their stuff back. What is to be done with the content in the future is to be determined at this point.

It's likely that video's will be excluded though for the fact that they take a long time to capture as well as that they take up a lot of storage space. And in my opinion it's the images and ldraw files which are of most importance.

Is that feasable? I guess we are talking about Terabytes of imformation to be downloaded and organized in just a few days!

[Dryw Filtiarn]

Organizing the data is not much of a problem, that's just a matter of downloading the stuff as is. When I leave out huger files (like video's) you will probably not really be talking about TB's of data. Considering the fact that there are aprox. 1.9 million files on BS, of which probably 5% will be video (yes there are actually some folders which are nearly entirely packed with video), and of which aprox. 10% will be textfiles/ldraw files. Then we will be talking about the following amount of data:

1.9 million minus 5%: 1.8 million

of which: 180.000 will be textfiles/ldraw (or other small text type files)

and: 1.6 million will be images

The average image size will somewhere in between 150 and 250KB (i'm talking AVERAGE here), the average file size for text/ldraw will be in between 5KB and 25KB (once again AVERAGE). Then I come to a total sum of:

1600000 * 200KB = 320000000KB / 1024 = 312500MB / 1024 =~ 305GB

180000 * 15KB = 2700000KB / 1024 =~ 2636MB / 1024 =~ 2.6GB

So we are talking about 310GB + or - 100GB (to be safe) is somewhere between 210GB and 410GB.

Which is pretty doable :)

[ups]

So we are talking about 310GB + or - 100GB (to be safe) is somewhere between 210GB and 410GB.

Which is pretty doable :)

With that figures, yes I agree it is pretty doable! I would guess that the library was much bigger! (My guess is that the image sizes are a bit bigger (even in average) since I see a lot of pictures easily hitting the 1M mark.

Still you have about 10 days to download everyting, which makes about 40G per day. Good luck, and If you need any help just say so.

(Ok.. just before submitting this I went to see my backup of BS folders and calculated the average of image sizes and it sticked under 150K.. I guess that maybe your calculations migth not be too far from the truth!)

(Maybe event pictures are larger than mocs..)

[Dryw Filtiarn]

I'm strictly monitoring the progress and averages on filesizes at the moment.

The statistics I can currently give you are the following:

At this point the average filesize over all files already downloaded is about 270kB. Which will result in about 420GB of data when everything is downloaded (theoreticly). I must say though, that I start to doubt the statistics on the site a bit regarding the 1.9 million shared files that would be on BS.

The initial thing that I have been doing is retrieve all user accounts that exist on BS, and I can't get any further then a total of 34569 users, which I'm pretty sure of to be correct. When I compare that to the average amount of files per user (based upon the already downloaded content, this results in aprox. 1 million files, where I must admit that the following filetypes are being excluded: avi, mpg, mov, wmv, wma, mp3, ogg and for safety also: exe, com and pif).

Anyway, everything is going pretty well right now :)

BTW this whole operation is being done by selfwritten tooling using the command line, and it's currently running 10 parallel processes, which uses my available bandwidth pretty well (nearly maximum capacity).

Edited July 18, 2007 by Dryw Filtiarn

[ZeekyBoogeyDoog]

Very noble effort Dryw! Good luck!

[Dryw Filtiarn]

I'll keep you posted over the coming few days (as this will be a multi-day effort to be able to retrieve everything).

Regarding my previous post, I might have to correct the amount of users that I had been able to find on BS. I'm currently rerunning the scan (doens't have any effect on the timeplan though!) to figure if I was correct. It appeared that there was an oddity in my script that scanned the useraccounts. Will get back on that issue.

[Dryw Filtiarn]

So far copying the data from BS. Kevin apparently decided to ban my IP on the server, meaning that I can no longer visit BS nor Maj.com.

Edit: Typo

Edited July 18, 2007 by Dryw Filtiarn

[ups]

So far copying the data from BS. Kevin apparently decided to ban my IP on the server, meaning that I can no longer visit BS nor Maj.com.

Edit: Typo

Ouch! That is a big draw back!

Still.. you alone must be giving him a big headache concerning bandwith usage! I would continue the project if I hadn't just 20G free :P

[Siegfried]

So far copying the data from BS. Kevin apparently decided to ban my IP on the server, meaning that I can no longer visit BS nor Maj.com.

Edit: Typo

Darn. You were obviously doing too good a job. I am downloading a lot as well. I hope I don't get banned...

[Asuka]

So far copying the data from BS. Kevin apparently decided to ban my IP on the server, meaning that I can no longer visit BS nor Maj.com.

Ouch. That

[Quarryman]

So far copying the data from BS. Kevin apparently decided to ban my IP on the server, meaning that I can no longer visit BS nor Maj.com.

Edit: Typo

heh.. me too. Seems like three computers I've used to fetch images from BS have been banned. We should form a club or something :-P

[ups]

heh.. me too. Seems like three computers I've used to fetch images from BS have been banned. We should form a club or something :-P

And something tells me that you both have almost the same data not being enough to build a full replica!

[Dryw Filtiarn]

I've been getting to about 30.000 images, which is far from enough unfortunatly :(

[Quarryman]

I've been getting to about 30.000 images, which is far from enough unfortunatly :(

I think I have a bit more than that.. at least over 50k, but I'm really not sure. I hope to get as many as possible though.. why? because nobody else seems to be doing it ;-)

Edited July 18, 2007 by Quarryman

[Dryw Filtiarn]

I'm currently somewhere around 64.000 files and 14GB

My processes are running again. I managed to play around a bit with some other webservers to create a file-proxy there, which allows me to access brickshelf :)

[ups]

... 400G.. it will take ages!

I wonder if after you could sell a hard drive with the information...

[Berry Syedow]

Here's hoping you save everything before being permanently barred from brickshelf. *y*